Create Custom Source Integrations With Webhooks and Transformations

Why Webhooks for custom integrations?

RudderStack supports a growing list of sources and destinations, but we also make it easy to build custom integrations to both internal tooling as well as integrations that we don’t support yet. Our users accomplish this by leveraging our Webhook Sources and Destinations in combination with our powerful Transformations feature.

There’s a reason why developers love webhooks: it’s because they open up opportunities for flexibility in creating all kinds of integrations as well as testing and automating events.

Webhooks make building custom integrations easy because you can ingest server-side events from sources that don’t support our SDKs, or deliver events to any compatible API endpoint. Best of all, you can do anything you want to the payload in between using RudderStack’s Transformations.

We’ve actually leveraged Webhooks for that exact scenario in our internal production stack. Check out this guide on how we used Basin Webhooks to pull in web form lead data from our Marketing website.

Guide: creating a custom integration with Webhooks and Transformations

The steps for creating a custom integration are simple, and only require basic JavaScript to build out a user Transformation that reshapes the incoming JSON payload to conform to RudderStack’s Event Spec. Conveniently, much of this basic transformation happens out of the box and gives you a standardized starting point, but as we will see in our example, different types of methods, like the identify method, required a bit more shaping because of certain required fields.

In this example, we are going to create a custom source integration that streams incoming events from Drip to Mixpanel.

Drip is an e-commerce marketing automation and segmentation tool and we don’t currently have an out of the box Cloud Event source integration for it. Not to worry, though—with RudderStack we can easily set up a Webhook Source that streams events from Drip when a new subscriber gets created and add that subscriber as a new user in Mixpanel.

Additionally, though, we can use Transformations to do something even more powerful: split the incoming webhook event into both an .identify call and a .track call that represents the user action of actually subscribing. The end result is that in Mixpanel, our product team will see not only the new user who subscribed, but have that subscription as an event on their user journey timeline, enabling all kinds of more advanced analysis.

Step 1: Create a Webhook Source in RudderStack

Step 2: Configure the Webhook Source to ingest events from Drip

Step 3: Create a Transformation to split the incoming webhook event into distinct .track and .identify calls

Step 4: Connect the Mixpanel destination and Test the Transformation to ensure that events are being delivered properly

Before we begin, make sure you are signed into RudderStack. If you don’t already have an account you can create one for free here. If you are new to RudderStack, check out this detailed guide on how to get started.

Step 1: Create a Webhook Source in RudderStack

In your RudderStack Connection dashboard, click on “+ ADD source” and search for “Webhook Source.” Select that source and name it.

Once that source is created you’ll see within the source Settings tab that a Webhook URL was provided to you. This is what we’ll be using as an endpoint for receiving any events from an external source, in this case Drip, so make sure to copy the URL or keep it handy.

Note: One question you may have is, “how do I know if the custom integration I want to create can send events to a webhook? Not all SaaS tools support this functionality, so you’ll want to check before going through the entire setup process. This information is usually available in developer or integration documentation. Thankfully, many modern SaaS tools do support webhooks.

Step 2: Configure the Webhook Source to ingest events from Drip

Next we’ll want to create a webhook integration in Drip. (If you want to test this with Drip specifically, you can create a Drip account here.).

In Drip, we’ll go to Settings and from there select “Webhooks”, and in the upper right corner we’ll click on “+ New webhook.” This is where you will paste the webhook URL from RudderStack that you created earlier.

When integrating with a webhook, some tools like Drip allow you to specify which events you want to send. For now, we will check the box that says “All events,” but if you don’t need everything, you can select the ones you want from the list. .

The RudderStack Webhook source makes integration easier because it will add the necessary fields to the incoming payload from Drip, you can view an example of the payload provided by Drip here. RudderStack will add an anonymous ID, message ID as well as a default event type which will be a track event. Again, this makes it very convenient to afterwards add or change the event type and level to route that event to a specific destination. When we create a new subscriber in Drip, this will generate an incoming webhook event that will have a payload that looks like this in RudderStack:

JSON
{
   "anonymousId": "a0a484b7-5a0c-488e-94ac-13c22d5333cd",
   "event": "webhook_source_event",
   "messageId": "550a149f-dc3d-440a-865a-582594158970",
   "properties": {
       "data": {
           "account_id": "6514652",
           "properties": {
               "source": "drip"
           },
           "subscriber": {
               "base_lead_score": null,
               "created_at": "2022-12-01T21:07:54Z",
               "custom_fields": {},
               "email": "jane@rudderstack.com",
               "id": "b3561zwxmux0pkm903oa",
               "ip_address": null,
               "landing_url": null,
               "lead_score": null,
               "lifetime_value": null,
               "original_referrer": null,
               "prospect": false,
               "status": "active",
               "tags": [],
               "time_zone": null,
               "user_agent": null,
               "user_id": null,
               "utc_offset": 0
           }
       },
       "event": "subscriber.subscribed_to_email_marketing",
       "occurred_at": "2022-12-01T21:07:54Z"
   },
   "rudderId": "2942d042-2d50-4eb9-ad8d-2cf47c0301f9",
   "type": "track"
}

In this example, we want the event to be an ‘identify’. Mixpanel will also look for a context object which will have traits in it, those will be used to add additional information that can further add context to the identified user. In order to create a new user in Mixpanel, in addition to what we have, we’d still need:

Event type to be identify
First name and last name to be passed as traits.
Event name to be in its own field at root level.

Step 3: Create a Transformation to split the webhook event into distinct .track and .identify calls

Now let’s write a transformation that will transform some of the fields and add any necessary additional ones.

In your RudderStack dashboard and under Enhance, click on Transformations.

The payload above is what we’ll be starting with. We’ll want to transform that to a payload that will be ingested into Mixpanel and mapped to a new user.

JAVASCRIPT
export function transformEvent(event, metadata) {
   const updatedEvent = event
   const { properties } = event
   const context ={}
   const meta = metadata(event);
   if (meta.sourceId==="source-id-of-your-webhook-source")
  {
       if (event.event && event.properties.event==="subscriber.created" || event.properties.event==="subscriber.updated_custom_field" )
      
      {
    
         let traits={
             email: properties.data.subscriber.email,
             firstName: properties.data.subscriber.custom_fields.first_name,
             lastName: properties.data.subscriber.custom_fields.last_name,
         }
 
       if (properties) {
         updatedEvent.event = properties.event
         updatedEvent.userId = properties.data.subscriber.email
         updatedEvent.type =  "identify"
       
         updatedEvent.context = { traits: traits };
       
         delete properties.event
        
        }      
      }
    let trackEvent=
         {
             "userId": properties.data.subscriber.email,
             "event": properties.event,
             "type": "track",
             properties: properties.data.properties
           
         }
   
    return [updatedEvent,trackEvent]
 }
      return
 }

This transformations is doing the following:

Applying this transformation should be applied to drip webhook source events.
Searching for events of type subscriber.created and subscriber.updated_custom_field

Note: Make sure here when creating a new subscriber in Drip, to update the custom fields with a first name and optionally a last name, otherwise users will have names as blank in Mixpanel.

Creating a traits object from the first and last name in custom fields.
Changing event type from Track to Identify.

Note: all events streamed via webhook are ingested as Track calls within RudderStack.

Adding event name to the root level

Note: all events streamed via a webhook have the name “webhook _source_event” in RudderStack by default, this step would be to provide a meaningful name for the event.

Returning two different events:
- An identify event: to create the user in Mixpanel
- A track event: to list actions and events related to this user.

Step 4: Connect the Mixpanel destination and Test the Transformation to ensure that events are being delivered properly

In order to begin testing the events, head over to Drip and create a new subscriber, provide them with a first and last name as well as any additional details you want to use to describe them.

One of the useful benefits of User Transformations, is you can test the event payloads before and after applying the Transformation on your live incoming payloads. This lets you detect any issues and correct them on the spot within the Transformation UI.

Once this step is done, we’ll also want to verify that our events have successfully created users in Mixpanel.

If you don’t already have a Mixpanel destination setup, you can set one up on RudderStack using the instructions here.

Now that our destination is all set up, we can head over to our Mixpanel Dashboard, select the users tab and verify that our events have successfully mapped subscribers as new users in Mixpanel.

We can see that the user "Jane" was now created in Mixpanel

And the events associated with this user are now also tracked

Conclusion

With RudderStack, we want to enable developers to take advantage of built in integrations that enable them to perform their customer data analytics in no time. We also want to make sure there is enough flexibility provided for Developers to build their own custom integrations but still benefit from the out-of-the box reliability that we provide.

This example is just one of many on how that can be achieved. For any feedback or questions, feel free to reach out to our team on the Community Slack. Thank you for reading!