Guides
Guides
Get Started

data365_twitter_keywords

This component enables real-time and historical data collection of X (Twitter) content based on keywords

Suggest Edits

Compatible Metadata Fields

Applicable Metadata CategoriesCompatible
SourceYes
ContentYes
AuthorYes
PersonNo
EnrichmentYes
OrganizationNo
Data source-specific fields?Yes, please see the Metadata page.

Metadata Example

{
    "data_source": "data365_twitter_keywords",
    "meta_data": {
        "job_label": "job-label",
        "job_id": "job-id"
    },
    "id": "1704492629574991926",
    "doc_date": "2023-09-20T13:48:05.000",
    "twitter": {
        "user_id": "134198871",
        "post_identifier": "1704492629574991926",
        "quote_count": 0,
        "reply_count": 0,
        "retweet_count": 0,
        "tweet_type": "POST",
        "retweet_type": "NONE"
    },
    "author": {
        "handle": "bikulmishra",
        "url": "https://twitter.com/bikulmishra"
    },
    "content": {
        "links": [
            "https://youtu.be/jMpiBeDq-jY"
        ],
        "published": "2023-09-20T13:48:05.000",
        "favorites": 0,
        "body": "TTOO Stock - T2 Biosystems Inc Stock Breaking News Today | TTOO Stock Price Prediction | TTOO Stock Target $183USD. Watch The Video Now.\nhttps://t.co/Ug9TLL09T2",
        "found": "2023-09-22T04:51:17.000Z",
        "found_by": "c557fdc5-2581-4597-bb65-30049e6c6164",
        "likes_count": 0
    },
    "enrichment": {
        "language": "en"
    },
    "source": {
        "link": "https://twitter.com/bikulmishra/status/1704492629574991926"
    }
}

Compatible Filters

Filter NameDescription
querySupports Language and logical operators (AND, OR). Limited to 400 characters in total length
query_fromFilter dates from/since - Example '2024-05-01T00:00:00Z'
query_toFilter dates to - Example '2024-08-01T00:00:00Z'
max_documentsThe maximum number of documents to be returned. Due to the asynchronous nature of pipelines this is a soft limit and more documents may be returned

Creating Data Collection Job

Example for creating a data collection job from data365_twitter_keywords data source, for content having keyword "cats" and published between 2023-10-01 and 2023-10-02 only.

curl --location 'https: //api.platform.datastreamer.io/api/pipelines/{PIPELINE_ID}/components/{COMPONENT_ID}/jobs?ready=true' \
      --header 'apikey: <your-api-key>' \
      --header 'Content-Type: application/json' \
      --data \
        '{
            "job_name": "my twitter data collection",
            "data_source": "data365_twitter_keywords",
            "from": "2023-10-01",
            "to": "2023-10-02",
            "query": {
                "query": "cats"
            },
            "job_type": "oneTime"
        }'

For more details on creating data collection jobs, see Job Management

Search Query

Data365 Twitter API keywords search query accepts list of keywords or phrase to search. It also accepts logical operators like (AND, OR) and stop-words.

🚧

Maximum allowed length of keywords string is 400 characters

For more details on supported search operators, see Data365 Twitter API

Search Query Examples

Example 1 - cats or dogs since '2024-10-01' until now

{
    "data_source": "data365_twitter_keywords",
    "job_type": "oneTime",
    "query_from": "2024-10-01T00:00:00Z",
    "query_to": null,
    "query": "(\"cats\" OR \"dogs\")"
}

Example 2 - cats and no dogs since '2024-10-01' until '2024-10-05'

{
    "data_source": "data365_twitter_keywords",
    "job_type": "oneTime",
    "query_from": "2024-10-01T00:00:00Z",
    "query_to": null,
    "query": "cats -\"dogs\""
}

Example 3 - cats since '2024-10-01' until now with 'Arabic' language filter

{
    "data_source": "data365_twitter_keywords",
    "job_type": "oneTime",
    "query_from": "2024-10-01T00:00:00Z",
    "query_to": null,
    "query": "((\"cats\") lang:ar)"
}

📘

Language filter supports ISO-639

Updated 1 day ago