Private Data Sources
Private data sources contain data collections that are visible only to your organization.
Private data sources fall into two categories:
- Filtered Data collected from one or more available providers via Datastreamer.
- Directly Ingested Data, supplied by the user (BYOD).
In both cases the data will be stored in Datastreamer Searchable Storage
Data Collected from a Provider
When collecting data from a provider and storing it in searchable storage, the corresponding data source is automatically named using the format private.<datasource>
(e.g., private.wsl_instagram
).
The private prefix is required when querying via the Search API, but it can be omitted when using the Datastreamer Searchable Storage Ingress.
Bring Your Own Data (BYOD)
If you're ingesting your own documents into a pipeline, you can optionally create a data source to associate with this data. However, if you're using Datastreamer Searchable Storage, creating a data source is mandatory.
There are two mechanisms for creating a data source
Storage & Data Page
Navigate to the Storage & Data page of the Datastreamer portal and select the Create New option

Enter the details for your data source and click Save

Data Source Name
Data source names must be globally unique across the entire Datastreamer platform, similar to how bucket names work in cloud storage services like S3.
Names can only contain lowercase letters (
a–z
), numbers (0–9
), and underscores (_
).A recommended naming convention is:
<organization>_<datatype>
The data source is now ready to use.
When You Need It
Alternatively you can create your data source directly from the control that is used to select a data source. Enter the new data source name and click the highlighted button.

Updated 2 days ago