About Auto Source Selection & FAQ

What is Auto Source Selection?

"Auto" source selection is available for many major data categories, and handles the selection and ingestion of web and social data! Selecting "Auto" components offloads to the platform to select the premium 3rd party data source provider best fit for the query, and with the highest reliability based on current demand.

"Auto" can also detect degraded performance or outages, and automatically process your queries through another available provider to solve it.

Why use Auto Source Selection?

For Pipeline users focused on outcomes, "Auto" source selection simplifies the ingestion of web and social data. Using the Auto Source Selection component for any data source simplifies the process of selecting, testing, maintaining, and migrating between various providers.

"Auto" sources also ensure greater reliability and quality of data by selecting the right providers for your queries based on their own capabilities and availability. This provides the highest uptime and coverage for the least effort.

How is usage measured?

Usage is measured in DVUs (Data Volume Units) and visible in Portal, the same as with any other integrated data source.

What data source providers are used?

Within each of the Auto Source selection offerings, the providers may adapt and change as the capabilities of each providers may change as well. Providers are selected based on functionality, cost, reliability, and integrated billing capabilities.

Many of the providers are chosen from our Registry: View the Datastreamer Registry.

What if I want to integrate a specific provider?

If you need a more customized experience, or have your own provider keys, you can implement any of the providers directly. Over 100+ 1st and 3rd party data source providers are supported.

You are not able to add your own API keys into the Auto sources.

Can I customize the providers leveraged based on specific requirements?

Within the "Auto" sources, the providers are not able to be customized. However, we are working to release customization capabilities that allow you to indicate preferences on latency and costs. These will help to ensure the best providers are used.

If you require greater customization, you can implement any of the providers directly. Many of the available components and providers are listed on our Registry here: View the Datastreamer Registry.