Bucket Configuration Settings for Event Backups

Configure cloud-specific buckets for your event backups.

Depending on your data retention policy, RudderStack stores the following two types of events:

  • Raw events ingested by RudderStack.
  • The final event payload along with the error in case of delivery failures.
info

Note that:

  • RudderStack deletes the events from the bucket upon successful delivery.
  • RudderStack does not persist any of the customer data.

Follow the steps in this guide if you want RudderStack to back up the events in your own cloud-specific bucket.

Bucket configuration settings

If you are using RudderStack Open Source and want to use your own bucket to store the events, you need to enable and set certain variables in your RudderStack backend.

Docker setup

Kubernetes setup

Similar to the Docker setup, you can configure your bucket settings by changing the values in the values.yaml file.

Permissions for Amazon S3

Follow these steps to use your own S3 bucket for RudderStack to store the events:

  1. Create a new S3 bucket.
  2. Create a new customer-managed policy with the following JSON:
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "s3:GetObject",
      "s3:PutObject",
      "s3:ListBucket",
      "s3:ListObjectsV2",
      "s3:AbortMultipartUpload"
    ],
    "Resource": "arn:aws:s3:::{BUCKET_NAME}/*",
    "Resource": "arn:aws:s3:::{BUCKET_NAME}"
  }]
}
  1. Create a new group and add the above policy to this group.
  2. Create a new user in Identity and Access Management (IAM) with programmatic access and add the user to the above group.
  3. Download the Access key ID and Secret Access Key.
  4. Specify the credentials during the S3 bucket configuration.

Permissions for GCS

This section lists the steps to use your own GCS bucket for RudderStack to store the events:

Under Roles in your GCP dashboard, create a role with the below permissions:

  • storage.objects.create
  • storage.objects.get
  • storage.objects.list
warning
Make sure to add the permissions one after the other.

Create a service account

  1. Assign a name to the service account:
Assign a name
  1. Add the role you created above.
  1. Create a key of JSON type and save this file locally:
Create a key
  1. Create a bucket with the bucket access control set to Uniform:
Create a bucket

Add the required permissions to your GCS bucket by following the below steps:

  1. Go to the Permissions tab.
  2. Add the member with the service account created above.
  3. Add the role.
Go to Permissions
  1. Download the JSON file containing the required permissions.
  2. Specify the JSON file location during the GCS bucket configuration.


Questions? Contact us by email or on Slack