Search API

Searchable Storage's full-text search API is based on Apache Lucene and provides advanced search facilities on top of a high-quality content index. Use it to read back the documents your pipeline wrote to Searchable Storage.

This API allows you to search for arbitrary text strings, search with complex boolean logic, use filters, and other advanced features. The results are then returned as ordinary JSON documents.

📘

Requires Searchable Storage

The Search API is an API offering of Searchable Storage, and requires Searchable Storage in your pipeline.

Endpoint Details

Send your API key in the apikey header. You can generate your own API Key in the Keys & Secrets page, API Keys section, in your Portal screens.

Request Details

A sample request looks like:

curl --request POST \
  --url https://api.platform.datastreamer.io/api/search \
  --header 'apikey: <API_KEY>' \
  --header 'content-type: application/json' \
  --data '{
  "query": {
    "from": 0,
    "size": 1,
    "track_total_hits": true,
    "query": "washington AND football",
    "query_start_date": "2026-06-01T00:00:00Z",
    "query_end_date": "2026-06-30T00:00:00Z",
    "data_sources": [
      "twitter_auto"
    ]
  }
}'

API Reference: Search

Data Sources, Query and API Pagination are used within the JSON request body to filter the results as needed. Apache Lucene based syntax is used to write the queries within the query parameter of the request body.

Parameters

All parameters go inside the query object of the request body, except keep_alive_seconds, which is a query string parameter.

To aggregate results (terms, date histogram, or significant terms), add an aggregations object inside query and use only private data sources, such as private.twitter_auto. See Aggregations with Search API.

Data Sources

data_sources is required. Use the Auto source names your Jobs used, for example twitter_auto or linkedin_auto. For private data, add the private. prefix, for example private.twitter_auto. A name your API key cannot see returns 403 with Cannot view data source: <name>. A provider name, rather than the Auto source name, returns no results.

Date Range

If you set no explicit dates, a date range in the query or a data source default may still limit results. Set query_start_date and query_end_date (ISO 8601) inside query to limit the window, as in the sample request above. A doc_date:[2026-06-01 TO 2026-06-30] range inside the query string also works.

Response Details

Based on the data_sources provided in the request body, the response of the Search API may vary, as each data source has its own set of values alongside the standard datasets like Source, Content, Author, and Enrichment, which are retrieved for each query result. Here is a shortened sample response for the request above.

{
    "results": [
        {
            "id": "1802345678901234567-twitter_auto",
            "doc_date": "2026-06-15T18:22:10Z",
            "data_source": "twitter_auto",
            "source": {
                "link": "https://x.com/example_user/status/1802345678901234567"
            },
            "content": {
                "body": "Washington football preseason starts next week.",
                "published": "2026-06-15T18:22:10Z"
            },
            "author": {
                "name": "Example User",
                "handle": "example_user"
            },
            "enrichment": {
                "language": "en",
                "sentiment": "neutral"
            }
        }
    ],
    "total": {
        "value": 898,
        "relation": "eq"
    },
    "search_after": [
        1781547730000
    ]
}

The response contains results and may include total and search_after. When search_after is returned for a stream-integrated source, pass its values back unchanged in the next request.

For indexed storage, set track_total_hits: true to include total; false omits it. A relation of gte means the value is a lower bound. External provider responses can include total regardless of the flag. In the current implementation, their total.value counts provider entries rather than documents; source_totals gives counts of returned documents by provider.