What is Datastreamer's Platform?

The Platform providers the underlying systems on which customers run hundreds of Pipelines powering market-leading products in many marketing, threat, risk, and brand intelligence sectors. It provides the infrastructure and tools to rapidly create, manage, deploy, and run those data Pipelines!

What is a Pipeline?

A pipeline is the logic and architecture of the paths you want data to follow on its way to your products.

For example: Let's say we wanted to take Tiktok, convert it to a usable schema, apply sentiment, and push into BigQuery. That pipeline may look like this.

flowchart LR
    Tiktok --> Transformation["fa:fa-spinner Transformation"]
    Transformation --> Sentiment["fa:fa-check Sentiment"]
    Sentiment --> BigQuery["fa:fa-ban BigQuery"]

All of these individual parts come together in Datastreamer's Platform. Data then flows according to this path in a high-speed and flexible manner, delivering key data into the Pipeline creator's products.

Pipelines can be much more complex than this, sometimes these pipelines can contain dozens of individual steps with complex logic routing and filtering the data. In addition, these Pipelines can be enriching data with AI capabilities, translating text, or even stripping videos into thousands of image frames.

What are the elements of a Pipeline?

Datastreamer's Platform has two main elements:

Components

Each source, stage, and destination with the Datastreamer platform are self-contained building blocks called "Components". Similar to Legos, Components allow you to assemble the ideal flow and stages of the journey of data through the platform.

In this example each of these boxes are their own components!

flowchart LR
    Tiktok --> Transformation["fa:fa-spinner Transformation"]
    Transformation --> Sentiment["fa:fa-check Sentiment"]
    Sentiment --> BigQuery["fa:fa-ban BigQuery"]

Dynamic Pipelines

Unlike the pipelines created internally by many companies, Datastreamer allows you to create truly Dynamic Pipelines. You can assemble Components in any shape/order/method to achieve your goals.

🤯

22,000,000,000,000,000,000,000,000 options

Datastreamer's Dynamic Pipelines allows interchangeability of components, and there are also no limits to size of pipelines! As a result, there are over 2.281 x 10^49 (22 septillion) unique pipeline layouts possible.

Why use a dynamic pipeline?

Dynamic Pipelines means that you can:

• Build pipelines with multiple parts and complex routing in minutes. (So you can focus on your product, not your plumbing).
• Add new enrichments or data sources without any integration effort. (So you can ship faster).
• Full transparency and insight into the data pipelines operations. (So you can sleep better).
• Allow you to avoid any data leakage or unnecessary data noise with unlimited pipelines, not jamming everything into one technical solution (So you can look 4-5 years ahead).
• Data process in a dedicated and scalable environment.

What's Next?

You've got the foundations of the platform, dynamic pipelines, and components. Now let's make your product shine.

• Learn about pipeline anatomy: Understand all the parts of a successful pipeline
• Create your first pipeline: Generate a pipelines in the platform and play around to see what options are best suited to your needs.
• Stay Current: Keep your changelog updated with the latest platform changes

Need a hand? Our friendly support team is just a chat away