Aggregations with Search API

Using Aggregation in the Search API, you can power dashboards and run high-level analysis on the results of your searches. Aggregations use the same Lucene-based logic, and results are returned as ordinary JSON documents.

👍

Search API Endpoint Refresh

The Search API endpoint (https://api.platform.datastreamer.io/api/search) is the same endpoint for aggregations. The aggregation field (used within the query) is what specifies the aggregation actions to be performed.

📘

Requires Searchable Storage

Aggregations are an offering of Searchable Storage, and require Searchable Storage in your pipeline.

There are three (3) different types of aggregation:

• Terms (terms): Terms Aggregate enables the user to query for an aggregated view of the results, based on the provided parameters. This retrieves the counts for the desired field of the results against the query. Mostly used to generate dashboards, reports, and graphs.
• Significant Terms (significant_terms, for trends): Compared to the Terms Aggregation, the Significant Terms Aggregation does not return the most popular terms in a set. Instead, the trend aggregation highlights the results of a field that has undergone a significant change in popularity measured between a foreground and background set.
• Date Histogram (date_histogram): Date Histogram Aggregate enables the user to query the aggregated result based on a time-frame. Mostly used to generate dashboards, reports, and graphs.

The use cases for the aggregations are many:

• Searching for which organizations have more results for "fraud" than usual
• Detecting micro-trends within hashtags, locations, or other enriched fields.
• Seeing patterns of sentiment towards company products over a time period.

Using the Search API, the user can use the aggregations field to send the query aggregation. This is an example of a request using terms aggregation:

{
    "query": {
        "from": 0,
        "size": 0,
        "data_sources": [
            "twitter_auto"
        ],
        "query": "pizza",
        "aggregations": {
            "posts_by_company": {
                "terms": {
                    "field": "enrichment.company_entities.keyword",
                    "size": 5
                }
            }
        }
    }
}

This search returns no documents ("size": 0) and an aggregated view of the posts mentioning "pizza", grouped by the companies detected in them (enrichment.company_entities). This shows which companies come up most in pizza-related discussion.

Add the suffix .keyword to the fields used in an embedded aggregation, as in the example: enrichment.company_entities + .keyword.

The response will look like this:

{
  "total": {
    "value": 20,
    "relation": "eq"
  },
  "aggregations": {
    "posts_by_company": {
      "buckets": [
        {
          "key": "Papa Johns",
          "doc_count": 2
        },
        {
          "key": "PizzaPizza",
          "doc_count": 2
        },
        {
          "key": "Ginos",
          "doc_count": 1
        },
        {
          "key": "Pizza Hut",
          "doc_count": 1
        },
        {
          "key": "Dominos",
          "doc_count": 1
        }
      ]
    }
  }
}

Date Histogram Example

This request counts the posts mentioning "pizza" per day:

{
    "query": {
        "size": 0,
        "data_sources": [
            "twitter_auto"
        ],
        "query": "pizza",
        "aggregations": {
            "posts_per_day": {
                "date_histogram": {
                    "field": "doc_date",
                    "calendar_interval": "day"
                }
            }
        }
    }
}

Results return under aggregations in the response, as in the Terms example above.

Separate Aggregate Endpoints

/api/aggregate/term and /api/aggregate/date_histogram also exist as separate endpoints. See Aggregate Term and Aggregate Date Histogram in the API Reference. On the separate term endpoint, field takes the field name without .keyword, for example enrichment.language. The embedded aggregations approach on this page works through the regular Search endpoint.