Guides
Guides
Get Started

Vetric X Search

This component allows the creation of jobs to collect data from Vetric's X search system.

Suggest Edits

Compatible Metadata Fields

Applicable Metadata CategoriesCompatible
SourceYes
ContentYes
AuthorYes
PersonNo
EnrichmentYes
OrganizationNo
Data source-specific fields?Yes, please see the Metadata page.

Metadata Example

{
    "data_source": "vetric_x_search_latest",
    "id": "1813930239422239232",
    "twitter": {
        "user_id": "219699434",
        "post_identifier": "1813930239422239232",
        "quote_count": 0,
        "reply_count": 1,
        "retweet_count": 0,
        "user_verified": true,
        "tweet_type": "REPLY",
        "retweet_type": "NONE"
    },
    "author": {
        "handle": "HarryTurney",
        "userid": "219699434",
        "bio": "Website\nhttps://t.co/DXwbh4DSSn\nhttps://t.co/sv10Nb7quf\n\nHelp the website:\nhttps://t.co/04ZVp8HDzA\n\nDiscord\nhttps://t.co/xaTBDrA2ZC",
        "name": "Asuna",
        "profile_image_source": "https://pbs.twimg.com/profile_images/1768656974961385472/89Y1bZXn_normal.jpg",
        "likes_count": 26275,
        "verified": true,
        "bio_links": [
            "https://pbs.twimg.com/profile_banners/219699434/1710707766"
        ],
        "protected": false,
        "has_nft_avatar": false,
        "url": "https://twitter.com/HarryTurney"
    },
    "content": {
        "body": "@_Avsece_ GLB has some rate limits that stopped me from getting stats. I've disabled it and will give it a look tomorrow to see if I can get it working. For now, just use the JPN side.",
        "mentions": [
            "_Avsece_"
        ],
        "followers": 27670,
        "following": 661,
        "favorites": 0
    },
    "enrichment": {
        "language": "en"
    },
    "source": {
        "link": "https://twitter.com/HarryTurney/status/1813930239422239232"
    }
}

Creating Data Collection Job

Example for creating a data collection job from vetric-x-search-latest data source, for content having exact phrase "cats" and published between 2023-10-01 and 2023-10-02 only.

curl --location 'https: //api.platform.datastreamer.io/api/pipelines/{PIPELINE_ID}/components/{COMPONENT_ID}/jobs?ready=true' \
      --header 'apikey: <your-api-key>' \
      --header 'Content-Type: application/json' \
      --data \
        '{
            "job_name": "my twitter data collection",
            "data_source": "vetric_x_search_latest",
            "from": "2023-10-01",
            "to": "2023-10-02",
            "query": {
                "query": "cats"
            },
            "job_type": "oneTime"
        }'

Search Query

X (Twitter) Search API operators are supported, below are some examples for search queries to use with data collection jobs, described above. For more details on supported search operators, see Vetric APIs.

🚧

'since' and 'until' operators are not supported, please use 'from' and 'to' as shown examples below.

Example 1

{
    "data_source": "vetric_x_search_latest",
    "query": {
        "from": "2024-07-05",
        "to": null,
        "query": {
            "query": "copppa filter:replies lang:ar"
        }
    }
}

Example 2

{
    "data_source": "vetric_x_search_latest",
    "query": {
        "from": "2020-10-01",
        "to": "2020-10-28",
        "query": {
            "query": "#black_cats filter:images"
        }
    }
}

Updated about 21 hours ago