Sync Observability Settings

Options to store connection level sync logs and retry sending failed records.

This guide explains the different observability settings that let you specify:

  • How RudderStack retains the sync log and snapshot tables for your Reverse ETL connection.
  • Whether RudderStack should continually retry sending the failed records in the future syncs.

Sync settings

You can access the following sync settings in the Settings tab of your Reverse ETL connection:

SettingDescription
Retain sync logsThis setting is toggled on by default and instructs RudderStack to store the sync logs in your warehouse. You can also configure the below settings:

SettingDescription
Sync log retentionSpecify the retention period of the sync logs in your warehouse.

If you set it to 1, then RudderStack deletes any sync log older than a day (in UTC time).
Snapshot table retentionSpecify the number of snapshot tables to retain.
Retry failed recordsThis setting is toggled on by default and causes RudderStack to continually retry sending the failed records.
warning
Storing sync logs and snapshot tables may incur additional warehouse costs.

Sync logs

RudderStack supports storing Reverse ETL sync logs in your warehouse. These logs are stored in a certain format in the RudderStack schema (used for storing the state of each sync). You can explore these logs using SQL or any BI tool to debug any issues or failures during syncs.

warning
Storing the sync logs in your warehouse will incur additional costs.

Sync log table schema

The sync log table schema is as follows:

ColumnDescription
connection_idConnection ID for the sync run.
sync_run_idUnique identifier for the sync run.
primary_keyValue of the primary key column selected for the sync.
operationNature of the operation performed on the row. It can be either insert, update, or delete.
statusResult of the operation. It can be either succeeded or failed.
error_reasonReason for failure in case status is failed.
sync_started_atSync start time in UTC.
sync_finished_atSync finish time in UTC.

Snapshot table schema

RudderStack generates a snapshot table in the RudderStack schema for each sync. It contains rows (barring invalid rows like duplicate primary keys and null primary keys) from the data source like a warehouse table, view, SQL model, or audience. It also includes the details of the changes that happened in each row during the sync.

warning
Storing the snapshot tables in your warehouse will incur additional costs.

The snapshot tables are stored in the snapshot_<connection_id>_<sync_run_id> format, where:

  • <connection_id> is the Reverse ETL connection ID.
  • <sync_run_id> is the unique identifier for the sync run.
RETL connection sync run ID

The snapshot table schema is as follows:

ColumnDescription
rudder_operation_typeType of operation performed on the row when compared with the last synced data. It can be either insert, update, or delete.
<column_name>Name of the columns selected in the mappings for the sync.

Debug issues with sync logs

This section lists some queries you can run on your sync logs for debugging issues in your Reverse ETL syncs.

  • Check how a particular primary key has changed over time for a Reverse ETL connection:
select * from _rudderstack.sync_log where primary_key='<primary_key>' and connection_id='<connection_id>';
  • Check the data for failures in the sync run:
SELECT *
FROM   _rudderstack.sync_log sl
       LEFT JOIN
       _rudderstack.snapshot_<connection_id>_<sync_run_id>
       sn
              ON sl.primary_key = sn.<primary_key>
WHERE  sl.sync_run_id = '<sync_run_id>'
       AND sl.status = 'failed'; 
warning
Make sure to replace the placeholders in the above queries with the actual values.

Retry failed records

The Retry failed records toggle in the sync settings lets you determine whether RudderStack should continually retry sending the failed records in the previous syncs to the downstream destination.

info
If a failed record has undergone any changes, RudderStack retries syncing the updated record instead.

Questions? Contact us by email or on Slack