You can create a Dynamic Pipeline using Portal. It is the visual interface of the platform at portal.datastreamer.io. Regardless of where you create a Pipeline, you will be able to view, edit, manage, and interact with your pipeline from either method.

Deploying

What happens when you deploy?

When you deploy your Pipeline for the first time, or any changes to an existing; the Platform will bring online all the necessary components to handle your Pipeline steps. This process is automatic and instant. You will be able to see all steps of the Pipeline coming online in the user interface.

Upon deployment, this Pipeline is fully ready to consume and process data. If you have connected your Pipeline to a live feed of data, you will notice data begin to flow upon deployment. Deploying generally takes 10-30 seconds for full rollout and diagnostics.

Deployment Checks

The Platform will perform a check to ensure that all Components are configured. If a Component is not configured (green checks), then the Platform will not allow that Pipeline to be deployed. You can however save your in-progress Pipeline (see: Saving).

Deploying to a Pipeline actively processing

Deploying a new version goes into immediate effect. Existing content that is being managed in the Pipeline will follow the new Pipeline's direction without the need to resend data. For more control, we recommend Pausing the Pipeline, or Stopping it.

Pausing, Stopping, and Deleting a Pipeline

Pausing an active Pipeline allows each component to finish the data it may be actively processing, and data in different steps are preserved. When you resume your Pipeline, data continues from the state it is in.

Stopping an active Pipeline will bring a stop to each Component, even if that Pipeline was actively processing data. All data being processed by the Pipeline is stopped and the Pipeline is purged.

Deleting a Pipeline will Stop and destroy the Pipeline. This can not be recovered. It should be considered a final step.

Saving an Incomplete Pipeline

If your Pipeline is not yet ready to Deploy, you can save the current status. This Incomplete pipeline will not be an active Pipeline, and no data will be processed until it is deployed.

Pipeline Versioning

After deploying a Pipeline, alongside "Resume", "Stop", "Edit" buttons, is the "Versions" button. When you save a modification to a Pipeline, you will have the option to add a Changelog. This information will be present alongside the version number for a Pipeline.

Deploying new changes to the Pipeline will show as versions. These versions are visible by clicking the version number.