About Partner Data Sources

Datastreamer`s data sources are primary elements responsible for retrieving relevant data from the sources effectively to ensure correct data is retrieved with fractions. To assist users of our platform in expanding the data in their pipeline, we have integrated and ready to use the following sources.

Data sources are used within search endpoints to retrieve data accurately from the relevant sources. The below example indicates to retrieve data from Twitter tweets and Instagram feeds based on the remaining query parameters available in the request body.

{
   "query":{
      "from":0,
      "size":1,
      "query":"New York",
      "data_sources":[
         "wsl_twitter",
         "wsl_instagram"
      ]
   }
}

🚧

Integration with any data source

Any data source can be integrated with the Datastreamer pipeline platform.

Sample Data Sources

Select data sources have samples that can be tested free of charge by prefixing sample_ before the data source name, for example:

• sample_wsl_instagram for wsl_instagram (Currently with 10,000 documents available)
• sample_wsl_twitter for wsl_twitter (Currently with 10,000 documents available)

Based on the previous example, the same query but targeting sample data sources will look like the following:

{
   "query":{
      "from":0,
      "size":1,
      "query":"New York",
      "data_sources":[
         "sample_wsl_twitter",
         "sample_wsl_instagram"
      ]
   }
}

📘

Dates on sample documents

To keep the documents easily searchable, the published_data, doc_date and found_date of documents older than 30 days are automatically updated to the current date.