Azure Data Lake is Microsoft’s secure and scalable data lake functionality that lets you store data of varying sizes and complexity and facilitates fast, cross-platform data processing and analytics.
Refer to the Warehouse Schema guide for more information on how the events are mapped to the tables in the Azure data lake.
Find the open source code for this destination in the GitHub repository.
Configuring Azure data lake destination in RudderStack
To send event data to Azure data lake, you first need to add it as a destination in RudderStack and connect it to your data source. Once the destination is enabled, events will automatically start flowing to Azure data lake via RudderStack.
To configure Azure data lake as a destination in RudderStack, follow these steps:
In your RudderStack dashboard, set up the data source. Then, select Azure Data Lake from the list of destinations.
Assign a name to your destination and then click Next.
Connection settings
Azure Blob Storage Container Name: Enter the name of your Blob Storage container used to store the data before loading it into the data lake.
Prefix: If specified, RudderStack will create a folder in the bucket with this prefix and push all data within that folder. For example, https://<account_name>.blob.core.windows.net/<container_name>/<prefix>/
Namespace: If specified, all data for the destination will be pushed to https://<account_name>.blob.core.windows.net/<bucketName>/<prefix>/rudder-datalake/<namespace>. If you don’t specify a namespace in the settings, RudderStack sets it to the source name, by default.
Azure Blob Storage Account Name: Enter your Azure Blob Storage account name.
Azure Blob Storage Account Key: Enter your Blob Storage account key.
Use shared access signature (SAS) Tokens: Enable this setting to grant limited access to your Azure Storage resources. When enabled, you can use a Shared Access Signatures (SAS) token instead of your storage account access key:
Azure Blob Storage SAS: Enter the generated SAS token.
For more information on setting up your Azure Blob Storage account, refer to the Azure Blob Storage guide.
Sync Frequency: Specify how often RudderStack should sync the data to your Azure data lake.
Sync Starting At: This optional setting lets you specify the particular time of the day (in UTC) when you want RudderStack to sync the data to the warehouse.
Advanced settings
Skip User Table: Toggle on this setting to send events exclusively to the identifies table and skip the users table, eliminating the need for a merge operation on the users table.
Skip Tracks Table: Toggle on this setting to skip sending events to the tracks table.
Finding your data in the Azure data lake
RudderStack converts your events into Parquet files and dumps them into the configured Azure bucket. Before dumping the events, RudderStack groups the files by the event name based on the time (in UTC) they were received.
The folder structure is similar to the following format:
<prefix> is the Azure prefix used while configuring the Azure data lake destination in RudderStack. If not specified, RudderStack will omit this value.
<namespace> is the namespace specified in the destination settings. If not specified, RudderStack sets it to the source name.
<tableName> is set to the event name.
YYYY, MM, DD, and HH are replaced by the actual time values.
A combination of the YYYY, MM, DD, and HH values represents the UTC time.
Use case
Suppose that RudderStack tracks the following two events:
Event Name
Timestamp
Product Purchased
"2019-10-12T08:40:50.52Z" UTC
Cart Viewed
"2019-11-12T09:34:50.52Z" UTC
RudderStack then converts these events into Parquet files and dumps them into the following folders:
This site uses cookies to improve your experience while you navigate through the website. Out of
these
cookies, the cookies that are categorized as necessary are stored on your browser as they are as
essential
for the working of basic functionalities of the website. We also use third-party cookies that
help
us
analyze and understand how you use this website. These cookies will be stored in your browser
only
with
your
consent. You also have the option to opt-out of these cookies. But opting out of some of these
cookies
may
have an effect on your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. This
category only includes cookies that ensures basic functionalities and security
features of the website. These cookies do not store any personal information.
This site uses cookies to improve your experience. If you want to
learn more about cookies and why we use them, visit our cookie
policy. We'll assume you're ok with this, but you can opt-out if you wish Cookie Settings.