You are viewing documentation for an older version.

Click here to view the latest documentation.

Commands

The Profile Builder tool supports specific commands, making executing the usual operations easier. The basic syntax of executing a command is:

$ pb <command> <subcommand> [parameters]

Supported commands

You can use the following Profile Builder commands:

cleanup

Displays and removes materials, older than the retention time period specified by the user (default value is 180 days).

pb cleanup materials -r <number of days>

Optional Parameter

compile

Generates SQL queries from models.

pb compile

It creates SQL queries from the models/profiles.yaml file, storing the generated results in the Output subfolder in the project’s folder. With each run, a new folder is created inside it. You can manually execute these SQL files on the warehouse.

Optional parameters

discover

Discovers elements in the warehouse, such as models, entities, features and sources.

pb discover

It allows you to discover all the registered elements in the warehouse.

Subcommands

Discover all the models, entities, features, sources, and materials in the warehouse.

$ pb discover models
$ pb discover entities
$ pb discover features
$ pb discover sources
$ pb discover materials

Optional parameters

Examples

# Discover all the models
$ pb discover models

# Discover a model with specific name
$ pb discover -m 'RUDDER_WEB_EVENTS.PROD_SCHEMA.feature_profile'

# Discover all features having 'max' in their name
$ pb discover features -u %max%

# Discover all the entities for a specific profile
$ pb discover entities -c siteconfig.yaml

# Discover all materials for target dev
$ pb discover materials -t dev

# Export output of discover command to a CSV file in output folder
$ pb discover -f my-custom-name.csv

# Export all sources to a CSV file in output folder
$ pb discover sources -f my-custom-name.csv

help

Provides list information for any command.

$ pb help

Subcommand

Get usage information for a specific command, with subcommands, and optional parameters.

$ pb help <command_name>

init

Creates connection and initializes projects.

pb init

Subcommands

Inputs values for a warehouse connection and then stores it in the siteconfig.yaml file.

pb init connection

Generates files in a folder named HelloPbProject with sample data. You can change it as per project information, models, etc.

pb init pb-project

Optional parameters

insert

Allows you to store the test dataset in your (Snowflake) warehouse . It creates the tables sample_rs_demo_identifies and sample_rs_demo_tracks in your warehouse schema specified in the test connection.

# Select the first connection named test having target and output as dev, of type Snowflake.
$ pb insert
# By default it'll pick up connection named test. To use connection named red:
$ pb insert -n red
# To pick up connection named red, with target test .
$ pb insert -n red -t test

This command is supported only for Snowflake currently.

migrate

Migrate your project to the latest schema.

Subcommands

Based on the current schema version of your project, it enlists all the steps needed to migrate it to the latest one.

pb migrate manual

Automatically migrate from one version to another.

pb migrate auto 

To migrate your models:

Schema 44 onwards

Navigate to the folder where your project files are stored. Then execute one of the following:

• pb migrate auto --inplace: Replaces contents of existing folder with the migrated folder.
• pb migrate auto -d <MigratedFolder>: Keeps the original project intact and stores the migrated project in another folder.

Schema 43 -> 44:

Use {{entity-name.Var(var-name)}} to refer to an entity-var or an input-var.

For example, for entity_var user_lifespan in your HelloPbProject, change select: last_seen - first_seen to select: '{{user.Var("last_seen")}} - {{user.Var("first_seen")}}'.

Note that:

• You must use two curly brackets.
• Anything contained within double curly brackets must be written in double quotes (" "). If you use single quotes within double quotes, then use the escape character (\) that comes when using macros.

Further, navigate to the folder where your project files are stored. Then execute one of the following:

• pb migrate auto --inplace: Replaces contents of existing folder with the migrated folder.
• pb migrate auto -d <MigratedFolder>: Keeps the original project intact and stores the migrated project in another folder.

Linear dependency

Specify this parameter when entity as vars migration is not done (till version 43). After the migration is done, it’s not necessary to mention this parameter and can be removed.

  compatibility_mode:
    linear_dependency_of_vars: true

Optional parameters

run

Creates identity stitcher or feature view model in the warehouse.

pb run

It generates the SQL files from models and executes them in the warehouse. Once executed, you can see the output table names, which are accessible from the warehouse.

Optional parameters

The run command shares the same parameters as the compile command, in addition to the following ones:

show

Obtains a comprehensive overview of models, id_clusters, packages, and more in a project. Its capacity to provide detailed information makes it particularly useful when searching for specific details, like all the models in your project.

$ pb show

Subcommands

1. pb show models

This command lets you view information about the models in your project. The output includes the following information about each model:

• Warehouse name: Name of the table/view to be created in the warehouse.
• Model type: Whether its an identity stitching, feature view, SQL model etc.
• Output type: Whether the output type is ephemeral, table, or view.
• Run type: Whether the model’s run type is discrete or incremental.
• SQL type: Whether the SQL type of the model is single_sql or multi_sql.

2. pb show models --json

This subcommand saves all model details in a JSON file.

3. pb show dependencies

This subcommand generates a DAG in a graph file (dependencies.png) highlighting the dependencies of all models in your project.

4. pb show dataflow

This subcommand generates a DAG in a graph file (dataflow.png) highlighting the data flow of all models in your project.

5. pb show idstitcher-report --id_stitcher_model models/<ModelName> --migrate_on_load

This subcommand creates a detailed report about the identity stitching model runs. To know the exact modelRef to be used, you can execute pb show models. By default, it picks up the last run, which can be changed using flag -l. The output consists of:

• ModelRef: The model reference name.
• Seq No: Sequence number of the run for which you are creating the report.
• Material Name: Output name as created in warehouse.
• Creation Time: Time when the material object was created.
• Model Converged: Indicates a successful run if true.
• Pre Stitched IDs before run: Count of all the IDs before stitching.
• Post Stitched IDs after run: Count of unique IDs after stitching.

Profile Builder also generates a HTML report with relevant results and graphics including largest cluster, ID graph, etc. It is saved in output folder and the exact path is shown on screen when you execute the command.

6. pb show entity-lookup -v '<trait value>'

This subcommand lists all the features associated with an entity using any of the traits (flag -v) as ID types (email, user id, etc. that you are trying to discover).

Optional parameters

query

Executes SQL query on the warehouse and prints the output on screen (10 rows by default).

pb query <query>

For example, if you want to print the output of a specific table/view named user_id_stitcher, run the following query:

pb query "select * from user_id_stitcher"

To reference a model with the name user_default_id_stitcher for a previous run with seq_no 26, you can execute:

pb query "select * from {{this.DeRef('path/to/user_default_id_stitcher')}} limit 10" --seq_no=26

Optional parameters:

validate

Validates aspects of the project and configuration.

$ pb validate

It allows you to run various tests on the project-related configurations and validate those. This includes but is not limited to validating the project configuration, privileges associated with the role specified in the site configuration of the project’s connection, etc.

Subcommands

Runs tests on the role specified in the site configuration file and validates if the role has privileges to access all the related objects in the warehouse. It throws an error if the role does not have required privileges to access the input tables or does not have the permissions to write the material output in the output schema.

$ pb validate access

version

Shows the Profile Builder’s current version along with its GitHash and native schema version.

pb version

Questions? Contact us by email or on Slack