AI Agent Guides

This page is written for an AI agent, or the developer pointing one at Datastreamer. Read this page first, then open only the guide your task needs.

Which Guide Do You Need?

Your task	Guide
Collect data. Create and manage Jobs on Auto Sources, choose a query, set limits, estimate and check DVU usage.	Collect Data with the Jobs API
Read results back. Search, count, and aggregate the documents a pipeline wrote to Searchable Storage.	Read Results from Searchable Storage

The Searchable Storage guide applies only if the pipeline writes to Searchable Storage. Other destinations (warehouses, cloud storage, webhooks) receive the data directly, and you read it with the customer's own tools.

Shared Basics

Both guides use the same API basics.

Base URL: https://api.platform.datastreamer.io
Authentication: send the API key in a header named apikey on every request.
Content type: Content-Type: application/json for requests with a body.
Dates: ISO 8601 in UTC, for example 2026-06-01T00:00:00Z.

The API key comes from the Datastreamer Portal, Keys & Secrets page, API Keys section. See API Key Management. An agent cannot create a key itself; if it is missing, ask the user for it.

How the Two Connect

A Job's data_source (for example twitter_auto) is the value you pass in Search's data_sources array. The source you collected from is the source you read back.
A Job's query_from / query_to collection window corresponds to query_start_date / query_end_date on Search.
Documents can take a short time after a Job reports succeeded to become searchable, so a search run immediately after may return fewer documents than the Job collected.

What Else Exists (Not Covered Here)

These are available on the platform. They are outside these guides, but worth knowing about so you can point your user to them.

Direct Integrations: Jobs on a specific provider using the user's own credentials, with provider-specific parameters and per-source pricing. Direct Integrations
Enrichments: sentiment, entities, categories, language, location, translation and custom AI steps, added to the pipeline in the Portal. Enrichments
Destinations: BigQuery, Snowflake, Databricks, Elasticsearch, S3, Azure Blob, GCS, webhooks and more, configured in the Portal.
Pipeline building and deployment: done in the Portal. What is a Dynamic Pipeline?
Work items: the units each Job is broken into, searchable at POST /api/pipelines/work-items/search. Advanced Job Search
Monitoring: alerts, component logs, the Document Inspector and pipeline analytics, in the Portal. Document Inspector
Billing and commits: the Billing Dashboard and committed usage. Committed Usage Discounts

AI Agent Guides

Which Guide Do You Need?

Shared Basics

How the Two Connect

What Else Exists (Not Covered Here)

Related