Read Results from Searchable Storage (AI Agent Guide)

These APIs work only when the pipeline writes to Searchable Storage. Other destinations (warehouses, cloud storage, webhooks) receive the data directly and are read with the customer's own tools. Read the AI Agent Guides index first for the shared basics (base URL, apikey header, dates). To collect the data in the first place, see Collect Data with the Jobs API.

Search

POST https://api.platform.datastreamer.io/api/search

{
  "query": {
    "from": 0,
    "size": 100,
    "query": "\"ev charging\"",
    "query_start_date": "2026-06-01T00:00:00Z",
    "query_end_date": "2026-06-02T00:00:00Z",
    "data_sources": [ "twitter_auto" ]
  }
}

query uses Apache Lucene syntax: AND, OR, NOT, phrases in quotes, wildcards (test*, not as the first character), and field ranges.
Dates. If you set no dates, results cover everything in the index. Set query_start_date and query_end_date (ISO 8601) inside query to limit the window. A doc_date:[2026-06-01 TO 2026-06-02] range inside the query string also works. For Search, either date method works.
data_sources is required. Use the Auto source name the Job used, for example twitter_auto. A name the key cannot see returns 403 "Cannot view data source". Provider names (for example a specific vendor's name) are not valid here.
The response returns results (documents in the unified schema), total, and search_after.

Reference: Search API · Performing a Query

Pagination

size defaults to 10. A size above 100 is capped at 100 without an error. from is at most 9,900.
For deeper result sets, pass the search_after value from the previous response in the next request. This is available for stream-integrated sources.

Reference: API Pagination

Count

POST https://api.platform.datastreamer.io/api/count with the same query and data_sources as Search. Returns { "total": { "value": N } }. Use it to size a result set before paging through it.

Reference: Count API

Aggregations

Add an aggregations object to a Search request (with size: 0 if you only need the aggregation). Supported types are Terms, Significant Terms (trends) and Date Histogram. Aggregate on .keyword fields, for example enrichment.language.keyword.

/api/aggregate/term and /api/aggregate/date_histogram also exist as separate endpoints (see the Search API reference); their request bodies are not documented here.

Reference: Aggregations with Search API

When Something Goes Wrong

Symptom	What to check
`401` or `403`	The `apikey` header is missing or the key is wrong, or `data_sources` names a source the key cannot see (`403 "Cannot view data source"`).
Search returns nothing for a new Job	Check that (1) the pipeline writes to Searchable Storage, (2) `data_sources` matches the Job's `data_source`, and (3) the date fields cover the Job's collection window.