Search API

Our full-text search API is based on Apache Lucene and provides advanced search facilities on top of a high-quality content index. If you’re getting up and running with Datastreamer for the first time you probably want to be using our Search API.

This API allows you to search for arbitrary text strings, search with complex boolean logic, use filters, and other advanced features. The results are then returned as ordinary JSON documents.

Endpoint Details

Endpoint	Type	Description
/api/search	`POST`	Central endpoint for Search API which enables developer to retrieve relevant results based on the provided parameters.

A request to Search API will look like this:

curl --request POST \
  --url https://api.platform.datastreamer.io/api/search \
  --header 'apikey: <API_KEY>' \
  --header 'content-type: application/json' \
  --data '{
  "query": {
    "from": 0,
    "size": 1,
    "query": "washington AND football",
    "data_sources": [
      "wsl_instagram"
    ]
  }
}'

You can generate your own API Key in the Keys & Secrets page, API Keys section, in our Portal.

Request Details

A sample request body looks like:

{
  "query": {
    "from": 0,
    "size": 1,
    "query": "washington AND football",
    "data_sources": [
         "wsl_instagram"
    ]
  }
}

Data Source , Query and API Pagination are extensively used within the JSON request body to filter out the results as per the requirements. Apache Lucene based syntax is used to write down the queries within the query parameter of the request body.

If you want to use the Search API with your own Datastreamer storage, you just need to add the prefix private. before the name of your Datastreamer storage data source:

{
  "query": {
    "from": 0,
    "size": 10,
    "query": "pizza AND burguer",
    "data_sources": [
         "private.socialgist_reddit"
    ]
  }
}

Response Details

Based on the data_source provided in the request body, the response of the Search API may vary as each data source have its own set of values alongside the standard datasets like Source, Content, Author, and Enrichment which are retrieved for each query result data. Here is a sample response against the above request body.

{
    "results": [
        {
            "id": "3369086690004949772-wsl_instagram",
            "doc_date": "2024-05-16T08:00:52Z",
            "data_source": "wsl_instagram",
            "source": {
                "link": "https://instagram.com/p/C7BZn6PPdsM/"
            },
            "content": {
                "body": "Throw🔙| 15 years ago today, Taylor arrived at the key arena for the fearless tour with Selena Gomez in Seattle, Washington. (May 15, 2009)1\n\nFollow me (@taylorswiftfit ) for more updates ‼️\n\n #taylorswift #taylorswiftedit #taylorswift13 #taylorsversion #theerastour #erastour #joealwyn #taylor #swift #tayloralisonswift #taylahschild #swifties #swiftie  #traviskelce #football #explorepage #explore #trending #trend #viral #fyp #selenagomez #taylena #slaylor #throwback #fearless #taylahschild #tayloralisonswift",
                "image_urls": [
                    "https://scontent.cdninstagram.com/v/t51.29350-15/436278198_724536656300509_8680948375493467172_n.heic?se=7&stp=dst-jpg_e35&_nc_ht=scontent.cdninstagram.com&_nc_cat=101&_nc_ohc=IBYH4zP02nAQ7kNvgHV8QSj&edm=APs17CUBAAAA&ccb=7-5&ig_cache_key=MzM2OTA4NjY4NDE3NDg1NzU0OQ%3D%3D.2-ccb7-5&oh=00_AYAtLxe-uu1ysudsd-3gqQltVVrQjF_WYuC2CodvDyT4AQ&oe=664B8FD8&_nc_sid=10d13b",
                    "https://scontent.cdninstagram.com/v/t51.29350-15/444224445_1190537922302561_4255294051900406561_n.heic?se=7&stp=dst-jpg_e35&_nc_ht=scontent.cdninstagram.com&_nc_cat=105&_nc_ohc=hvQJ8onPL9cQ7kNvgEoT8Ku&edm=APs17CUBAAAA&ccb=7-5&ig_cache_key=MzM2OTA4NjY4NDE4MzI3NjgzNQ%3D%3D.2-ccb7-5&oh=00_AYA8BwS__JoVLcrmNlWNawkmzXYKlNOmyZaqRATSexiTPg&oe=664B9584&_nc_sid=10d13b",
                    "https://scontent.cdninstagram.com/v/t51.29350-15/444124856_956835229216995_2057344118970909470_n.heic?se=7&stp=dst-jpg_e35&_nc_ht=scontent.cdninstagram.com&_nc_cat=103&_nc_ohc=3vV3bkLdcoIQ7kNvgEwWl04&edm=APs17CUBAAAA&ccb=7-5&ig_cache_key=MzM2OTA4NjY4NDA0MDc3ODEzNg%3D%3D.2-ccb7-5&oh=00_AYB29bMg97uTOXP7cnx2RqJYER-szpnAi9YUcgxGcqAoUA&oe=664B912A&_nc_sid=10d13b"
                ],
                "images": [
                    {
                        "url": "https://scontent.cdninstagram.com/v/t51.29350-15/436278198_724536656300509_8680948375493467172_n.heic?se=7&stp=dst-jpg_e35&_nc_ht=scontent.cdninstagram.com&_nc_cat=101&_nc_ohc=IBYH4zP02nAQ7kNvgHV8QSj&edm=APs17CUBAAAA&ccb=7-5&ig_cache_key=MzM2OTA4NjY4NDE3NDg1NzU0OQ%3D%3D.2-ccb7-5&oh=00_AYAtLxe-uu1ysudsd-3gqQltVVrQjF_WYuC2CodvDyT4AQ&oe=664B8FD8&_nc_sid=10d13b",
                        "alternative_text": "Photo by Taylor Swift on May 16, 2024. May be an image of 2 people, people standing, sundress and miniskirt."
                    },
                    {
                        "url": "https://scontent.cdninstagram.com/v/t51.29350-15/444224445_1190537922302561_4255294051900406561_n.heic?se=7&stp=dst-jpg_e35&_nc_ht=scontent.cdninstagram.com&_nc_cat=105&_nc_ohc=hvQJ8onPL9cQ7kNvgEoT8Ku&edm=APs17CUBAAAA&ccb=7-5&ig_cache_key=MzM2OTA4NjY4NDE4MzI3NjgzNQ%3D%3D.2-ccb7-5&oh=00_AYA8BwS__JoVLcrmNlWNawkmzXYKlNOmyZaqRATSexiTPg&oe=664B9584&_nc_sid=10d13b",
                        "alternative_text": "Photo by Taylor Swift on May 16, 2024. May be an image of 2 people and miniskirt."
                    },
                    {
                        "url": "https://scontent.cdninstagram.com/v/t51.29350-15/444124856_956835229216995_2057344118970909470_n.heic?se=7&stp=dst-jpg_e35&_nc_ht=scontent.cdninstagram.com&_nc_cat=103&_nc_ohc=3vV3bkLdcoIQ7kNvgEwWl04&edm=APs17CUBAAAA&ccb=7-5&ig_cache_key=MzM2OTA4NjY4NDA0MDc3ODEzNg%3D%3D.2-ccb7-5&oh=00_AYB29bMg97uTOXP7cnx2RqJYER-szpnAi9YUcgxGcqAoUA&oe=664B912A&_nc_sid=10d13b",
                        "alternative_text": "Photo by Taylor Swift on May 16, 2024. May be an image of 4 people."
                    }
                ],
                "found": "2024-05-16T08:14:12Z",
                "found_by": "wsl_instagram_tag_robot",
                "published": "2024-05-16T08:00:52Z",
                "hashtags": [
                    "taylorswift",
                    "taylorswiftedit",
                    "taylorswift13",
                    "taylorsversion",
                    "theerastour",
                    "erastour",
                    "joealwyn",
                    "taylor",
                    "swift",
                    "tayloralisonswift",
                    "taylahschild",
                    "swifties",
                    "swiftie",
                    "traviskelce",
                    "football",
                    "explorepage",
                    "explore",
                    "trending",
                    "trend",
                    "viral",
                    "fyp",
                    "selenagomez",
                    "taylena",
                    "slaylor",
                    "throwback",
                    "fearless",
                    "taylahschild",
                    "tayloralisonswift"
                ],
                "favorites": 0,
                "followers": 2834,
                "following": 0,
                "mentions": [
                    "taylorswiftfit"
                ],
                "last_updated": "2024-05-16T08:14:18.157Z",
                "likes_count": 84
            },
            "author": {
                "name": "Taylor Swift",
                "bio": "Just a Fan Account\nnot impersonating anyone\nJust sharing #taylorswift  content out of context 💁❤️\nTAYLOR SWIFT STORE\nSHOP NOW\n👇👇👇👇",
                "profile_image_source": "https://scontent.cdninstagram.com/v/t51.2885-19/353562128_949432242773041_740200117086601067_n.jpg?stp=dst-jpg_e0_s150x150&_nc_ht=scontent.cdninstagram.com&_nc_cat=1&_nc_ohc=7qFXZHruDb0Q7kNvgHNPBls&edm=APs17CUBAAAA&ccb=7-5&oh=00_AYCjxvs57TG0F6Q5GqAkpFwRvPI9VQEi4taeh0JZCH25Xg&oe=664BA7A9&_nc_sid=10d13b",
                "gender": "UNKNOWN",
                "url": "https://www.instagram.com/taylorswiftfit/",
                "handle": "taylorswiftfit"
            },
            "enrichment": {
                "language": "en",
                "location_inference_country": {
                    "label": "FR",
                    "confidence": 0.6138
                },
                "sentiment": "neutral",
                "emoji_sentiment": "neutral"
            },
            "instagram": {
                "content_type": "POST",
                "user_id": "7954205346"
            }
        }
    ],
    "total": {
        "value": 898,
        "relation": "eq"
    },
    "search_after": [
        1715846452000
    ]
}