What is Direct Data Upload?

Direct Data Upload allows you send documents through the UI or via a HTTPS endpoint. This allows you to integrate without building custom configurations, allowing you to simply and quickly add needed data to your systems.

How to add "Direct Data Upload" to your Pipelines.

To begin, add a Direct Data Upload Ingress component to the beginning of your pipeline. Save and deploy the configuration for changes to take effect.

🚢

Deploy your Pipeline to continue!

When a Pipeline is running with "Direct Data Upload" as an Ingress, additional features and uploading of data are only available on the deployed version.

Uploading data using the "Direct Data Upload" via Portal

(API options are listed below.)

The deployed component has an Upload button, which opens a dedicated menu containing relevant options.

You can interact with the HTTPS Endpoint in one of two ways:

• Upload a JSON file or JSON Lines file directly.
• Alternatively, you can paste or type your document into the text area below and send.

The text area supports either a single document as can be seen in the example text, or an array of documents, for example:

[
  {
    "id": "123456",
    "content":{
    	"body": "First document"
    }
  },
  {
    "id": "789012",
    "content":{
    	"body": "Second document"
    }
  }
]

Uploading data using the "Direct Data Upload" via API

After pasting or typing valid JSON into the text area, the Code button will become available. Click it to view a code snippet with a curl example that demonstrates how to send documents programmatically via a POST request. The generated code looks like this:

curl --location 'https://portal.datastreamer.io/api/pipelines/<your-pipeline-id>/steps/<your-component-id>/documents' \
      --header 'apikey: <your-api-key>' \
      --header 'Content-Type: application/json' \
      --data \
        '[
          {
            "property": "value"
          }
        ]'