Read Results from Searchable Storage (AI Agent Guide)

These APIs work only when the pipeline writes to Searchable Storage. Other destinations (warehouses, cloud storage, webhooks) receive the data directly and are read with the customer's own tools. Read the AI Agent Guides index first for the shared basics (base URL, apikey header, dates). To collect the data in the first place, see Collect Data with the Jobs API.

Search

POST https://api.platform.datastreamer.io/api/search

{
  "query": {
    "from": 0,
    "size": 100,
    "query": "\"ev charging\"",
    "query_start_date": "2026-06-01T00:00:00Z",
    "query_end_date": "2026-06-02T00:00:00Z",
    "data_sources": [ "twitter_auto" ]
  }
}

query uses Apache Lucene syntax: AND, OR, NOT, phrases in quotes, wildcards (test*, not as the first character), and field ranges.
Dates. If you set no explicit dates, a date range in the query or a data source default may still limit results. Set query_start_date and query_end_date (ISO 8601) inside query to limit the window. A doc_date:[2026-06-01 TO 2026-06-02] range inside the query string also works. For Search, either date method works.
data_sources is required. Use the Auto source name the Job used, for example twitter_auto. A name the key cannot see returns 403 "Cannot view data source". Provider names (for example a specific vendor's name) are not valid here.
The response contains results and may include total and search_after. For indexed storage, set track_total_hits: true to request total; with false, it is omitted. If total.relation is gte, its value is a lower bound. External provider responses handle total differently; see Search API.

Reference: Search API · Performing a Query · API Reference: Search

Pagination

size defaults to 10. A size above 100 is capped at 100 without an error. from is at most 9,900.
For deeper result sets, pass the search_after value from the previous response in the next request. This is available for stream-integrated sources.

Reference: API Pagination

Count

POST https://api.platform.datastreamer.io/api/count with the same query and data_sources as Search. Returns { "total": { "value": N } }. Count supports internal data sources and rejects external data sources. Use it to size a result set before paging through it.

Reference: Count API · API Reference: Count

Aggregations

Add an aggregations object to a Search request (with size: 0 if you only need the aggregation). All requested data sources must be private, for example private.twitter_auto. Supported types are Terms, Significant Terms (trends) and Date Histogram. Aggregate on .keyword fields, for example enrichment.language.keyword.

/api/aggregate/term and /api/aggregate/date_histogram are deprecated separate endpoints (see the Search API reference); their request bodies are not documented here.

Reference: Aggregations with Search API · API Reference: Aggregate Term · Aggregate Date Histogram

When Something Goes Wrong

Symptom	What to check
`401` or `403`	The `apikey` header is missing or the key is wrong, or `data_sources` names a source the key cannot see (`403 "Cannot view data source"`).
Search returns nothing for a new Job	Check that (1) the pipeline writes to Searchable Storage, (2) `data_sources` matches the Job's `data_source`, and (3) the date fields cover the Job's collection window.