Data365 Collection Tasks

❗️

Deprecated

This page is now outdated. Please view all updated documentation for Data365 at: https://docs.datastreamer.io/docs/data365

This documentation covers how to setup and use Data365 collection tasks.

🚧

Have you registered your token?

If you have registered your Data365 Token in the Datastreamer platform, proceed to step 3. If not, being at step 1.

1. Getting Started

To get started with Data365, you will first need to have an access token from Data365 (please contact us if you don’t already have one).

NOTE: for the cURL examples below, we've injected twitter_keywords as a possible DataSourceName. Feel free to use any of the possible Data365 sources available that are found within the Response Body of View your new connected sources

2. Register your Data365 Token with Datastreamer.

📘

Full documentation:

Full documentation is available here: https://docs.datastreamer.io/docs/connecting-compatible-sources. A snippet is listed below.

2.1. Register your Token.

curl --location --request PUT 'https://api.platform.datastreamer.io/api/data-sources/data365' \
--header 'Content-Type: application/json' \
--header 'apikey: {YourDatastreamerApiKey}' \
--data '{ 
    "token": "{YourData365AccessToken}"
}'

2.2. View your new connected sources

curl --location 'https://api.platform.datastreamer.io/api/data-providers/data365' \
--header 'apikey: {YourDatastreamerApiKey}'

Using the above command you can see all the current supported sub-data sources from Data365.

3. Create your tasks

Creating your tasks for Data365 collection involves specifying the source in your URL, and needed data within the body.

You can use the following command to create your first batch of tasks.

curl --location 'https://api.platform.datastreamer.io/api/data-providers/data365/twitter_keywords' \
--header 'Content-Type: application/json' \
--header 'apikey: {YourDatastreamerApiKey}' \
--data '{
    "tasks": [
        {
            "value": "hellokitty",
            "from_date": "2023-09-01",
            "to_date": "2023-09-12",
            "update_interval": 300
        }
    ]
}'

Some explanations:

• Depending on the data source, value could be the profile, keywords, location_id, music name or hashtag (without #).
• You will need to specify a from_date and to_date in yyyy-mm-dd format.
• If you want it to be a repeated task, you’ll need to specify the update_interval
• If the upstream data365 has more parameters you want to use, put them under parameters and it will be added to the request.
• You can add multiple items to the same source.
• You can add multiple times the same or different items, it will always add to the current list.

Here are some of the sample value for each data source

• For instagram_location, you can try to find out the location id via this guide

4. Listing Your Tasks

You can list all the tasks under a certain source (putting the data source name in the URL)

curl --location 'https://api.platform.datastreamer.io/api/data-providers/data365/twitter_keywords' \
--header 'apikey: {YourDatastreamerApiKey}'

Some notes:

• The list of all tasks also has task status for each item, but it might be different than the status you get from individual status - this is caused by upstream (data365) task management. Generally speaking, the status in the list reflects the last time a task was executed and the status in the individual reflects the newest/next time the task is executed.

5. Start/Stop and Check tasks

Once you have the list of tasks, you need to start them in order to get the data. You can either start all of them (put the correct data source name in the URL)

a. Start All

curl --location --request PUT 'https://api.platform.datastreamer.io/api/data-providers/data365/twitter_keywords/start-all' \
--header 'Content-Type: application/json' \
--header 'apikey: {YourDatastreamerApiKey}'

b. Start a Single task

Start an individual task (using the TaskId from listing or creation)

curl --location --request PUT 'https://api.platform.datastreamer.io/api/data-providers/data365/twitter_keywords/{TaskId}/start' \
--header 'apikey: {YourDatastreamerApiKey}'

c. Cancel All Tasks

Similarly, you can cancel the tasks you started either in batch

curl --location --request POST 'https://api.platform.datastreamer.io/api/data-providers/data365/twitter_keywords/cancel-all' \
--header 'apikey: {YourDatastreamerApiKey}'

d. Cancel a Single Task

or individually

curl --location --request POST 'https://api.platform.datastreamer.io/api/data-providers/data365/twitter_keywords/{TaskId}/cancel' \
--header 'apikey: {YourDatastreamerApiKey}'

e. Check Task Status

Usually, the task takes 10-30 minutes to get executed, and you can check the status of the tasks via

curl --location 'https://api.platform.datastreamer.io/api/data-providers/data365/twitter_keywords/{TaskId}' \
--header 'apikey: {YourDatastreamerApiKey}'

Once the status becomes “complete” or “finished” your data is ready to be consumed.

f. Remove a task

Where if you want to remove a task, use the following, putting the correct data source name and task id in URL):

curl --location --request DELETE 'https://api.platform.datastreamer.io/api/data-providers/data365/twitter_keywords/{TaskId}' \
--header 'apikey: {YourDatastreamerApiKey}'

6. Consume the data you have received

Once your tasks are completed, the data are ready to be consumed via normal Datastreamer search/count APIs. For example, the following query will be searching in the data you have in data365_instagram_location. (You’ll need to add data365_ before the name of the data source).

Even though there is no organization on the data source name, the search API implementation ensures it hits the proper index.

curl --location 'https://api.platform.datastreamer.io/api/search?keep_alive_seconds=60' \
--header 'Content-Type: application/json' \
--header 'apikey: {YourDatastreamerApiKey}' \
--data '{
    "query": {
        "from":0,
        "size": 100,
        "query": "hellokitty",
        "data_sources": ["data365_twitter_keywords"]               
    }
}'

👍

You are all set!

If you want to explore with a sample source before diving into production usage, try out these sample sources: https://docs.datastreamer.io/docs/sample-sources.

Task Restrictions

The following limits are in place for task creation:

• 498 characters for twitter search query
• When you create a task with data365_twitter_profile, make sure you don't use operators. It's best to create one-task-per-profile.
• When you create a task with data365_twitter_keywords, you can use operators (AND, OR)
  for Twitter, up to 300 posts per search