Private data sources fall into two categories:

• Filtered Data collected from one or more available providers via Datastreamer.
• Directly Ingested Data, supplied by the user (BYOD).

In both cases the data will be stored in Datastreamer Searchable Storage

Data Collected from a Provider

When collecting data from a provider and storing it in searchable storage, the corresponding data source is automatically named using the format private.<datasource> (e.g., private.wsl_instagram).

The private prefix is required when querying via the Search API, but it can be omitted when using the Datastreamer Searchable Storage Ingress.

Bring Your Own Data (BYOD)

If you're ingesting your own documents into a pipeline, you can optionally create a data source to associate with this data. However, if you're using Datastreamer Searchable Storage, creating a data source is mandatory.

There are two mechanisms for creating a data source

Storage & Data Page

Navigate to the Storage & Data page of the Datastreamer portal and select the Create New option

Enter the details for your data source and click Save

📘

Data Source Name

Data source names must be globally unique across the entire Datastreamer platform, similar to how bucket names work in cloud storage services like S3.

Names can only contain lowercase letters (a–z), numbers (0–9), and underscores (_).

A recommended naming convention is: <organization>_<datatype>

The data source is now ready to use.

When You Need It

Alternatively you can create your data source directly from the control that is used to select a data source. Enter the new data source name and click the highlighted button.