Search API

Our full-text search API is based on Apache Lucene and provides advanced search facilities on top of a high-quality content index. If you’re getting up and running with Datastreamer for the first time you probably want to be using our Search API. This API allows you to search for arbitrary text strings, search with complex boolean logic, use filters, and other advanced features. The results are then returned as ordinary JSON documents.

Endpoint Details

Endpoint	Type	Description	Status
Search API	`POST`	Central endpoint for Search API which enables developer to retrieve relevant results based on the provided parameters.	Live

Request Object Details

A sample request body looks like:

{
  "query": {
    "from": 0,
    "size": 1,
    "query": "New York",
    "data_sources": [
         "wsl_twitter"
    ]
  }
}

Data Source , Query and API Pagination are extensively used within the JSON request body to filter out the results as per the requirements. Apache Lucene based syntax is used to write down the queries within the query parameter of the request body.

Response Details

Based on the data_source provided in the request body, the response of the Search API may vary as each data source have its own set of values alongside the standard datasets like Source, Content, Author, and Enrichment which are retrieved for each query result data. Here is a sample response against the above request body.

{
    "results": [
        {
            "id": "1646766821028700399-artemis",
            "data_source": "wsl_twitter",
            "source": {
                "link": "https://twitter.com/npnewamericans/status/1501274294852096000"
            },
            "content": {
                "body": "#ICYMI: We celebrated #InternationalWomensDay with @TXOrgProject and the City of San Antonio with an all-women Mariachi band Las Alteñas and remarks from powerful women activists, naturalized citizens and members of the San Antonio City Council! If you missed it, watch below!",
                "found": "2022-03-08T19:13:41Z",
                "published": "2022-03-08T19:10:58Z",
                "hashtags": [
                    "InternationalWomensDay",
                    "ICYMI"
                ],
                "favorites": 3802,
                "followers": 4043,
                "following": 1929,
                "mentions": [
                    "TXOrgProject"
                ]
            },
            "author": {
                "name": "NPNA",
                "bio": "<p>We represent the nation's largest immigrant coalitions. Aim to achieve vibrant, just democracy for all. Immigrants are the soul of our org, inspire our work</p>",
                "location": "",
                "profile_image_source": "https://pbs.twimg.com/profile_images/1501225151668445184/IUKHGkge_normal.jpg",
                "gender": "UNKNOWN",
                "url": "https://twitter.com/npnewamericans",
                "bio_links": [
                    "http://partnershipfornewamericans.org"
                ],
                "handle": "npnewamericans"
            },
            "enrichment": {
                "sentiment": "POSITIVE",
                "language": "en",
                "reported_violence": {
                    "label": "No violence",
                    "confidence": 0.9921875
                },
                "location_inference": {
                    "label": "Las Vegas",
                    "confidence": 0.833333334
                }
            },
            "twitter": {
                "tweet_type": "POST",
                "retweet_type": "REPLY",
                "post_identifier": "1501274294852096000",
                "user_verified": true,
                "user_id": "469708047"
            }
        }
    ],
    "total": {
        "value": 10000,
        "relation": "gte"
    }
}