Pipeline Import and Export

The Pipeline Import/Export feature provides a simple way to transfer pipeline configurations between accounts or create backups with a single click. The generated file uses a standardized JSON format (a .pipeline file) that is both machine-readable and human-understandable.

Features

Import

With the Import you have:

• Instant setup: Import complete pipeline configurations with one click
• Validation: Automatic verification of the imported pipeline structure
• Idempotent operation: Can be safely repeated with consistent results

Export

With the Export you have:

• One-button operation: Export the entire pipeline configuration with a single click
• Standardized format: Generates a .pipeline file in JSON format
• Complete representation: Includes all pipeline components, connections, and settings

File Format Specification

A pipeline that looks like this:

flowchart LR
    p25581kv["Reddit<br>(ingress)"] --> yra9w3ng
    yra9w3ng["Unify Transform<br>(connector)"] --> 47he3421
    47he3421["Emotion Classifier<br>(connector)"] --> 4y0t4f9w
    4y0t4f9w["Intent Classifier<br>(connector)"] --> 7dlncjjg
    7dlncjjg["Inspector<br>(egress)"]

Can be represented in a .pipeline file using JSON format with the following structure:

{
  "version": 1,
  "exported_date": "2025-06-13T17:59:35Z",
  "pipeline_name": "reddit_genai_01",
  "pipeline_description": "",
  "steps": [
    {
      "type": "ingress",
      "component_name": "socialgist-reddit-ingress",
      "step_id": "p25581kv",
      "properties": {},
      "next_steps": [
        "yra9w3ng"
      ]
    },
    {
      "type": "connector",
      "component_name": "unify-transform",
      "step_id": "yra9w3ng",
      "properties": {},
      "next_steps": [
        "47he3421"
      ]
    },
    {
      "type": "connector",
      "component_name": "genai-emotion-classifier",
      "step_id": "47he3421",
      "properties": {
        "text": "content.body",
        "destination_path": "enrichment.emotions"
      },
      "next_steps": [
        "4y0t4f9w"
      ]
    },
    {
      "type": "connector",
      "component_name": "genai-intent-classifier",
      "step_id": "4y0t4f9w",
      "properties": {
        "text": "content.body",
        "destination_path": "enrichment.intent"
      },
      "next_steps": [
        "7dlncjjg"
      ]
    },
    {
      "type": "egress",
      "component_name": "inspector-egress",
      "step_id": "7dlncjjg",
      "properties": {}
    }
  ]
}

Usage Instructions

Exporting a Pipeline:

1. Navigate to the Pipelines page
2. Choose the Pipeline that you want to export
3. Click the "Export" button available in the Pipeline available actions menu
4. The system will generate and download a .pipeline file
5. Save the file to your desired location

Importing a Pipeline:

1. Navigate to the Pipelines page
2. Click the "Import" button in the Pipelines page interface
3. Select the .pipeline file you want to import
4. The system will validate and load the pipeline configuration
5. The system will open the imported pipeline, ready for use

Best Practices

• Version Control: The .pipeline files are perfect for version control systems
• Backups & Copies. Export pipelines to:
  • Create external backups (beyond Datastreamer pipeline versioning) for compliance/archiving.
  • Duplicate pipelines for experimentation—test changes risk-free without affecting the original pipeline.
• Sharing: Use the export/import feature to share pipelines between team members
• Templates: Create template pipelines for common workflows and import them when needed, applying your own customizations

Limitations

• Organization Dependencies: Some components may require specific organization dependencies that need to be present on the target organization, like Secrets
• Version Compatibility: Pipelines exported a long time ago may not be compatible with the current version of our services