Configuration Parameters

Various types of configuration parameters explained with their types, descriptions, and default values

  •   8 minute read  

This document describes the various configuration parameters for the config.yaml file. You can fine-tune them to suit your application’s needs.

info
These configuration parameters are also applicable for the config.toml file, in case you have an older RudderStack deployment.
success
To best understand some of the terms covered in this guide, we recommend going through the RudderStack Architecture first.

Global Parameters

Parameter nameTypeDescriptionDefault value
maxProcessIntNumber of parallel threads used in server. Should be set to number of cores.12
gwDBRetentionStringThe time for which the events are stored in the Gateway database after they have been processed. Examples: 1h, 30m, 35s, etc.0h
routerDBRetentionStringThe time for which the events are stored in the Router database after they have been processed. Examples: 1h, 30m, 45s, etc.0h
enableProcessorBooleanThis variable enables or disables the Processor module. It will be set to false when running in Degraded mode.true
enableRouterBooleanThis variable enables or disables the Router module. Will be set to false when running in Degraded mode.true
enableStatsBooleanThis variable enables or disables stats.true
info
For more information on the Processor, Gateway, and Router modules of the RudderStack backend, refer to the RudderStack Architecture.

[Gateway]

Parameter nameTypeDescriptionDefault value
webPortIntThe port on which the RudderStack server runs.8080
maxUserWebRequestWorkerProcessIntRudderStack spawns this specified number of processes to consume the events at a user level.64
maxDBWriterProcessIntAs requests come in to the Gateway and are batched, RudderStack runs the maxDBWriterProcess writers to send these batches to the database and the Configuration Backend.256
CustomValStringFor the creation of a job in the backend PostgreSQL database (jobsDB), the value of this variable will be assigned to the CustomVal column.GW
maxBatchSizeIntThe batch size used in the Gateway. The requests are batched up to this size before writing to the database.32
batchTimeoutStringIn case the request batches do not meet the maxBatchSize, the batches are are sent to the database and the Configuration Backend in this interval.20ms
maxReqSizeInKBIntAn error message ("Request size exceeds max limit") is thrown for a particular request is when its size in KB crosses this value*.*4000
enableDedupBooleanEnables or disables deduplication of events. RudderStack uses message_id to de-dup. The duplicate events are dropped at the Gateway.false
dedupWindowStringEvents with the same message_id within this timeframe are considered duplicate and are dropped.3600s
enableRateLimitBooleanRate limits the number of requests accepted by the Gateway. This is used for running the RudderStack-hosted service.false

[SourceDebugger]

Parameter nameTypeDescriptionDefault value
disableEventUploadsBooleanEnables or disables the event schema upload.false
maxBatchSizeIntThe maximum size of the live events batch sent to the Configuration Backend.32
maxESQueueSizeIntThe maximum size of the live events queue in the memory.1024
maxRetryIntThe maximum number of attempts RudderStack makes to upload the request batches, in case of any errors.3
batchTimeoutStringIn case the request batches do not meet the maxBatchSize, the request batches are uploaded to Configuration Backend in this time interval.2s
retrySleepStringIn case of errors while uploading the request batches, RudderStack waits for this time interval before retrying. This is done until the maxRetry limit is reached.100ms

[JobsDB]

Parameter nameTypeDescriptionDefault value
jobDoneMigrateThresFloat64If (deletedJobsCount / totalJobsCount) >{" "} jobDoneMigrateThres, RudderStack migrates the table of the jobs that have been processed.0.8
jobStatusMigrateThresFloat64

If (statusCount / totalCount) > jobStatusMigrateThres,

RudderStack migrates the table of jobs that have been processed.

5
maxDSSizeIntThe maximum size of a table. If the maximum size is reached, RudderStack migrates all jobs in that table.100000
maxMigrateOnceIntThe maximum number of tables that can be migrated together.10
maxTableSizeInMBIntThe maximum size of the tables in MB. The tables are migrated if they cross this limit.300
migrateDSLoopSleepDurationString
The time RudderStack waits before migrating the datasets.		30s
maxMigrateDSProbeInt64The maximum number of tables probed to find if they can be migrated.10
addNewDSLoopSleepDurationStringThe time RudderStack waits before adding a dataset.5s
backupCheckSleepDurationStringRudderStack waits for this time interval before checking if a dataset needs backing up.5s
enableBackupBooleanEnables or disables the backup. This is set to false in the [Degraded](https://www.rudderstack.com/docs/user-guides/administrators-guide/high-availability/#rudderstack-server-running-modes) mode.true

[Router]

Parameter nameTypeDescriptionDefault value
jobQueryBatchSizeIntThe size of a jobs batch to get from the database. This includes the retry list, processing list, unprocessed list, or executed list.10000
updateStatusBatchSizeIntThe minimum size needed to update the status of a batch of jobs.1000
readSleepStringThe time RudderStack waits before fetching the next jobs batch from the database, in case the length of unprocessed and retry list is 0.1000ms
noOfWorkersIntRudderStack starts this number of workers to send events to the destinations.64
noOfJobsPerChannelIntThe number of jobs a channel in each worker can contain.1000
maxSleepStringThe time to wait when the response status code is not 200. This is to give RudderStack some time before the next retry.60s
minSleepStringThe time to sleep when the response status code is 200.0s
maxStatusUpdateWaitStringThe time to sleep before ending one round of stats collection.5s
useTestSinkBooleanRuns internal tests if set to true.false
maxFailedCountForJobIntThe maximum number of times a job can fail before it is marked as aborted.8
guaranteeUserEventOrderBooleanRudderStack maintains the order of user events if set to true.true
retryTimeWindowStringThe minimum retry window in case of 5XX, 429 errors.180m
minRetryBackoffStringThe minimum time before the next retry in case of 5XX, 429 errors.10s
maxRetryBackoffStringThe maximum allowed time between the errors in case of 5XX, 429 errors.300s

[BatchRouter]

Parameter nameTypeDescriptionDefault value
mainLoopSleepStringThe timeout while running the main loop.2s
noOfWorkersIntThe number of workers to batch jobs before deletion.8
jobQueryBatchSizeIntThe number of events picked up from the batch router’s database (Jobs DB) in each query.100000
uploadFreqStringThe frequency with which the batch router dumps the events to the storage destinations.30s
maxFailedCountForJobIntThe maximum number of times a job can fail before marking it as aborted.128

[Warehouse]

info
You can ignore these settings if you don’t have any Warehouse Destinations configured in RudderStack.
Parameter nameTypeDescriptionDefault value
stagingFilesTableStringTable name of the staging files.wh_staging_files
loadFilesTableStringTable name of load files.wh_load_files
uploadsTableStringTable name of uploads.wh_uploads
schemasTableStringTable name of schemas.wh_schemas
uploadFreqStringThe frequency of the upload in seconds.1800s
noOfWorkersIntNumber of concurrent writes to the warehouse.8
mainLoopSleepStringThe time RudderStack waits between multiple warehouse writes.5s
stagingFilesBatchSizeIntThe batch size of the staging files.960

[Processor]

Variable nameTypeDescriptionDefault value
loopSleepStringIn case the length of the user jobs process queue is 0 or the unprocessed and retry list is empty, RudderStack sleeps for this specified time.10ms
maxLoopSleepStringMaximum loop sleep time for the Processor.5000ms
dbReadBatchSizeIntThe total number of events to get as a batch from the database.10000
transformBatchSizeIntBatch size of the events added to the request queue before sending them for transformation.100
userTransformBatchSizeIntBatch size of the events added to request queue before sending them to the custom transformation server. NOTE: This is used only when a user transformation function is connected to a destination.200
sessionThresholdEventsIntThe minimum number of events needed to be process further.20
sessionThresholdStringThe minimum time needed before a new session is created.10s
maxChanSizeIntThe maximum channel size for the request and response queue in the transformer.2048
processSessionsBooleanIf set to true, the status of the job in the database is updated.false
numTransformWorkerIntSpecifies the number of Go transform workers.8
maxRetryIntThe maximum number of times a transformer retries hitting the API in case of an error.30
retrySleepStringThe sleep time in case of a transformer error while hitting the API, before retrying.100ms

[BackendConfig]

Variable nameTypeDescriptionDefault value
pollIntervalStringThe frequency of updating data from the Configuration Backend.5s
configFromFileBooleanWhen set to true, RudderStack reads the backend workspace configuration from a JSON file instead of fetching it from the API.false
configJSONPathStringThe path of the JSON file which contains the backend workspace configuration./etc/rudderstack /workspaceConfig.json

[RateLimit]

Variable nameTypeDescriptionDefault value
eventLimitInt64The maximum number of events to be allowed in a time interval.1000
rateLimitWindowStringThe rolling time interval used to limit the allowed number of events.60m
noOfBucketsInWindowInt32The number of buckets rateLimitWindow is broken down into.12

[Diagnostics]

Parameter nameTypeDescriptionDefault value
enableDiagnosisBooleanRudderStack sends the server diagnostics report to the user. Disabling this will disable sending all diagnostics information.true
gatewayTimePeriodStringThe time interval to send the Gateway requests report60s
routerTimePeriodStringThe time interval to send the Router requests report.60s
batchRouterTimePeriodStringThe time interval to send the Batch router requests report.10m
enableServerStartMetricBooleanSends the server start event.true
enableConfigIdentifyMetricBooleanSends the workspace config received event.true
enableServerStartedMetricBooleanSends the successful server start event.true
enableConfigProcessedMetricBooleanSends the workspace config details.true
enableGatewayMetricBooleanSends the Gateway request metrics.true
enableRouterMetricBooleanSends the Router request metrics.true
enableBatchRouterMetricBooleanSends the Batch Router request metrics.true
enableDestinationFailuresMetricBooleanSends the destination failures metrics.true

Questions? Contact us by email or on Slack