Vetric X Search
This component allows the creation of jobs to collect data from Vetric's X search system.
Compatible Metadata Fields
Applicable Metadata Categories | Compatible |
---|---|
Source | Yes |
Content | Yes |
Author | Yes |
Person | No |
Enrichment | Yes |
Organization | No |
Data source-specific fields? | Yes, please see the Metadata page. |
Metadata Example
{
"data_source": "vetric_x_search_latest",
"id": "1813930239422239232",
"twitter": {
"user_id": "219699434",
"post_identifier": "1813930239422239232",
"quote_count": 0,
"reply_count": 1,
"retweet_count": 0,
"user_verified": true,
"tweet_type": "REPLY",
"retweet_type": "NONE"
},
"author": {
"handle": "HarryTurney",
"userid": "219699434",
"bio": "Website\nhttps://t.co/DXwbh4DSSn\nhttps://t.co/sv10Nb7quf\n\nHelp the website:\nhttps://t.co/04ZVp8HDzA\n\nDiscord\nhttps://t.co/xaTBDrA2ZC",
"name": "Asuna",
"profile_image_source": "https://pbs.twimg.com/profile_images/1768656974961385472/89Y1bZXn_normal.jpg",
"likes_count": 26275,
"verified": true,
"bio_links": [
"https://pbs.twimg.com/profile_banners/219699434/1710707766"
],
"protected": false,
"has_nft_avatar": false,
"url": "https://twitter.com/HarryTurney"
},
"content": {
"body": "@_Avsece_ GLB has some rate limits that stopped me from getting stats. I've disabled it and will give it a look tomorrow to see if I can get it working. For now, just use the JPN side.",
"mentions": [
"_Avsece_"
],
"followers": 27670,
"following": 661,
"favorites": 0
},
"enrichment": {
"language": "en"
},
"source": {
"link": "https://twitter.com/HarryTurney/status/1813930239422239232"
}
}
Creating Data Collection Job
Example for creating a data collection job from vetric-x-search-latest data source, for content having exact phrase "cats" and published between 2023-10-01 and 2023-10-02 only.
curl --location 'https: //api.platform.datastreamer.io/api/pipelines/{PIPELINE_ID}/components/{COMPONENT_ID}/jobs?ready=true' \
--header 'apikey: <your-api-key>' \
--header 'Content-Type: application/json' \
--data \
'{
"job_name": "my twitter data collection",
"data_source": "vetric_x_search_latest",
"from": "2023-10-01",
"to": "2023-10-02",
"query": {
"query": "cats"
},
"job_type": "oneTime"
}'
Search Query
X (Twitter) Search API operators are supported, below are some examples for search queries to use with data collection jobs, described above. For more details on supported search operators, see Vetric APIs.
'since' and 'until' operators are not supported, please use 'from' and 'to' as shown examples below.
Example 1
{
"data_source": "vetric_x_search_latest",
"query": {
"from": "2024-07-05",
"to": null,
"query": {
"query": "copppa filter:replies lang:ar"
}
}
}
Example 2
{
"data_source": "vetric_x_search_latest",
"query": {
"from": "2020-10-01",
"to": "2020-10-28",
"query": {
"query": "#black_cats filter:images"
}
}
}
Updated about 1 month ago