Executive "Launch" Guide

For our users focused on the non-technical and administrative aspects of data pipelines, here is a "launch" guide of things you should know to gain the most benefit out of the Datastreamer platform.

👍
Assumptions
This Guide assumes you are familiar with Datastreamer, its purpose, product, and the tools your organization now has available to it.
As a quick reminder, Datastreamer is a pipeline platform and therefore this guide is focused on such, and does not address 3rd party components.

What does Datastreamer now solve for me?

Datastreamer is the infrastructure layer to power marketing, threat intelligence, brand, risk, and many other platforms that use web and social data.

Using Datastreamer, you benefit from:

Drastically lowered maintenance of data sources and data pipelines.
Rapid access to additional sources and enrichments to power your product capabilities.
Near-instant integration of 3rd party capabilities.
Offloaded infrastructure costs and maintenance.

Selection of scenarios where this impact is the most visible:

Scenario	The Pain	Why Datastreamer
Working with multiple web/social data sources.	Web/Social data sources are extremely varied in structure, format, and access methods. Causing high maintenance costs and effort.	Pipelines running on Datastreamer are built to stabilize and auto-adapt to the changes.
Ingesting social and web data at high volumes	Social and web sources generate high volumes of noisy data, and vary greatly	Pipelines running on Datastreamer were built on adaptive scaling, adapting to volumes in realtime.
Needing multiple providers for coverage	Juggling the operation of multiple vendors for the complete sources multiples effort due to competing standards and operations.	Your pipelines and Jobs systems standardize the data providers and the data sources, allowing central control.
Expansion and growth requires long projects	The average organization has 10-12 data sources on their roadmaps with an average implementation time of 6-10 weeks per source (not including discovery and procurement).	Adding new sources and enrichments takes minutes due to Datastreamer's partner ecosystem (for instant procurement) and component architecture (drag and drop integration).
Existing internal pipelines restrict new feature exploration.	Established internal data pipelines for organization rapidly gather tech debt due to web/social data complexities, removing nimbleness and adaptiveness.	Your engineers are able to modify/add/upgrade Pipelines using pre-existing capabilities, wiring into legacy pipelines to combine capabilities.

There are many other capabilities which you can explore across Datastreamer.io and in greater detail throughout this documentation.

How does pricing and billing work?

Datastreamer is priced based on usage-based pricing, and the usage of various components are measured in Data Volume Units (DVUs).

There are a few key things to note:

The number of pipelines, components, or users does not affect pricing. Only volumes and any 3rd parties you may have integrated.
Auto Source usage draws from one DVU pool. Enrichments and Direct Integrations are billed at their own per-component rates. For rates, see Datastreamer Pricing.
The billing cycle is monthly, and you can adjust your needs and Commits monthly.
Committed Usage Discounts (Commits) can be used on a monthly basis to lower the costs of pipeline resources by pre-committing to expected volumes upfront.
There are a number of pricing and billing tools available, such as alerts, projections, estimation tools, and detailed analytics. All providing detailed insights into your usage.

What can I add or integrate into Datastreamer?

Datastreamer has a component-focused architecture, so adding new sources, queries, enrichments, databases, and more are often a 1-2 minute effort. You can read more on the overall Pipeline architecture here or view a more detailed version more technical overview here.

Data Streams or Direct Integrations?

Datastreamer offers two paths, and you can run both in the same account. Data Streams use managed Auto Sources: you choose the platform and query, and Datastreamer selects the provider for each Job and fails over automatically, with one pooled commit across your Auto Sources. Direct Integrations connect a specific provider, either with your own key or with one Datastreamer procures for you, for provider-specific needs, with per-source commits.

What tools/features should I be using on an ongoing basis?

As a non-technical focused user, you should use the following features and resources:

Set up "Budget Alerts" to be alerted when certain spend milestones are reached.
View the "Billing Dashboard" regularly to see current, projected, and historical usage and spend.
Use the "Pricing Calculator" to price and scope out new projects.
Use the Volume Health Alerting in critical pipelines and sources to detect if volumes are outside of expected bands.

How to get support or guidance?

Datastreamer takes a consultative approach with our customers. Our team members are subject matter experts, ready to support and help with any question. Contact [email protected].

Other FAQ

Data ingestion is product-critical, does Datastreamer cap usage?

We know that market-leading companies rely on Datastreamer to power their platforms. Datastreamer, as an infrastructure platform, provides the tools for detailed insights. Pipelines and Jobs running on Datastreamer contain the tools to manage and cap volumes and spend: document limits on Jobs, budget alerts, and Query and Pipeline Budget Limits (early access; contact [email protected] for access).

Do I need to procure capabilities from Datastreamer's ecosystem?

No. You can use Datastreamer's managed Auto Sources through Data Streams, or connect a specific provider through Direct Integrations with your own key. See "Data Streams or Direct Integrations?" above.