Transformation Action
Create, test, and publish JavaScript/Python transformations and libraries directly from your development repository.
RudderStack’s Transformation Action lets you create, update, test, and publish transformations and libraries written in JavaScript or Python - directly from your development repository. Internally, this action leverages the Transformations API.
This action is currently available for GitHub.
Prerequisites
For the action to work, you need the email address associated with your RudderStack workspace. Also, you need to generate a service access token.
Note that:
- For production use cases, RudderStack recommends using a service access token instead of personal access token.
- For security purposes, it is highly recommended to use GitHub secrets to store your service access token.
Usage
name: RudderStack Transformer Test and Publish
uses: rudderlabs/rudder-transformation-action@<current_action_version>
with:
metaPath: './code/meta.json'
email: 'test@rudderlabs.com'
accessToken: ${{ secrets.ACCESS_TOKEN }}
uploadTestArtifact: true
See the sample workflow for more information on using this action.
For more information on how to use uploadTestArtifact
, see FAQ.
Action fields
Field | Description |
---|
metapath Required | Path to the meta file that:
- Lets the action know about what transformations/libraries to test based on the input events and the expected output.
- Publishes the transformations/libraries if the test passes.
See Meta file structure for more information. |
email Required | Email address associated with the RudderStack workspace. |
accessToken Required | Your service access token obtained from the RudderStack dashboard. |
uploadTestArtifact | Boolean flag that determines:
- Whether to upload the individual transformation outputs after running the transformation on the test events.
- The difference from the expected output.
See uploadTestArtifact parameter for more information.
Default value: False |
As mentioned above, a meta file lets the transformation action know what transformations or libraries to test depending on the input events and the expected output.
A generic meta file schema is shown below:
// Meta file schema
{
"transformations": <array of transformation schema>,
"libraries": <array of library schema>
}
A sample meta file is shown below:
// example meta.json
{
"transformations": [
{
"file": "./code/t1.js",
"name": "action-T1",
"language": "javascript",
"description": "action-T1",
"test-input-file": "./code/testevents.json",
"expected-output": "./code/expectedoutput.json"
},
{
"file": "./code/t2.py",
"name": "action-T2",
"language": "pythonfaas",
"description": "action-T2",
"test-input-file": "./code/events.json",
"expected-output": "./code/expected.json"
}
],
"libraries": [{
"file": "./code/lib1.js",
"name": "lib1",
"language": "javascript",
"description": "action-lib1"
},
{
"file": "./code/lib2.py",
"name": "getFinanceDataPy",
"language": "pythonfaas",
"description": "Python library to get finance data"
}
]
}
The path mentioned in the file
field should be relative to the base repository path.
The transformations
parameter in the meta file contains an array of transformation schemas. A single transformation schema contains the following parameters:
Parameter | Description |
---|
file Required | Path to the transformation code. |
name Required | Transformation name. |
language Required | Transformation language. Permissible values are javascript and pythonfaas . |
description | Transformation description. |
test-input-file | Path to the JSON file containing an array of events to test the transformation. |
expected-output | Path to the JSON file containing the expected output for the above input, after running the transformation code. |
Library schema
The libraries
parameter in the meta file contains an array of library schemas. A single library schema contains the following parameters:
Parameter | Description |
---|
file Required | Path to the library code. |
name Required | Library name that you must specify to import it in any transformation. |
language Required | Library language. Permissible values are javascript and pythonfaas . |
description | Library description. |
FAQ
How do I use the uploadTestArtifact
parameter?
When you set uploadTestArtifact
to true
in the transformation action, RudderStack:
- Runs the transformation on the test events present in
test-input-file
. - Generates and stores the output in a test-outputs folder contained within a zipped artifact.
- Validates the output against the contents of the file specified in
expected-output
.
Note that:
- RudderStack stores the output from Step 1 in a
[transformation name in camel case]_output.json
file. - It stores the validation differences from Step 3 in the respective
[transformation name in camel case]_diff.json
file. - If you do not set
uploadTestArtifact
to true
in the transformation action, RudderStack does not create the output zip containing the above files.
For more context, see the use case below.
Consider the following transformation schema:
// single transformationSchema
{
"file": "./code/t1.js",
"name": "action-T1",
"language": "javascript",
"description": "action-T1",
"test-input-file": "./code/testevents.json",
"expected-output": "./code/expectedoutput.json"
}
The workflow when you set uploadTestArtifact
to true
is as follows:
RudderStack runs the transformation action-T1
(with its code present in t1.js
) on the testevents.json
file.
RudderStack then:
- Generates and stores the output from Step 1 in a file named
actionT1_output.json
. This file is stored in a test-outputs folder contained within a zipped artifact. - Validates the output with the contents present in the
expectedoutput.json
file. It stores any differences in a file named actionT1_diff.json
.
The test-outputs folder contains the two files - actionT1_output.json
and actionT1_diff.json
.
Questions? Contact us by email or on
Slack