Vetric X (Twitter) Search
Collect latest data from Vetric's X (Twitter) search system
The Vetric X (Twitter) Search allows you to retrieve the latest results for given keywords from X (Twitter).
The complete Vetric documentation about their endpoint can be found in their site here.
New to Datastreamer? Start here.
Unify Schema
This data source already use Unify Schema.
API Key
You will need a Vetric API key to use this component. If you don't have one reach out to [email protected] and our team will help you with that.
How To Use?
The component is powered by the Jobs System, when interacting with the component you have the option to define your jobs queries.
Search Query
Filters
| Filter Name | Description |
|---|---|
| query | The keyword or phrase you wish to search. |
| max_documents | Set a limit for the number of documents that will be fetched for the search. |
The query can be used to specify the field to search using the different X (Twitter) search operators, as described here. See below some query examples.
Words:
watching now
Phrase:
"happy hour"
Mentioning an account:
@NASA
Examples
Retrieves tweets using 'NASA'
Retrieves the most recent tweets for 'NASA' keyword.
You also have the option to use the API. You can use the Code button to extract this example:
curl --location 'https://api.platform.datastreamer.io/api/pipelines/{PIPELINE_ID}/components/{COMPONENT_ID}/jobs?ready=true' \
--header 'apikey: <your-api-key>' \
--header 'Content-Type: application/json' \
--data \
'{
"job_name": "{JOB_NAME}",
"data_source": "vetric_x_search_latest",
"query": {
"include": [
{
"data_source": "vetric_x_replies"
}
],
"query": "NASA"
},
"job_type": "oneTime",
"query_from": "2025-01-01T00:00:00.000Z",
"query_to": "2025-01-08T00:00:00.000Z",
"max_documents": 10
}'
For more details on creating data collection jobs, see Job Management.
Additional Details
Compatible Metadata Fields
| Applicable Metadata Categories | Compatible |
|---|---|
| Source | Yes |
| Content | Yes |
| Author | Yes |
| Person | No |
| Enrichment | Yes |
| Organization | No |
| Data source-specific fields? | Yes, please see the Metadata page. |
Metadata Example
{
"data_source": "vetric_x_search_latest",
"id": "1813930239422239232",
"twitter": {
"user_id": "219699434",
"post_identifier": "1813930239422239232",
"quote_count": 0,
"reply_count": 1,
"retweet_count": 0,
"user_verified": true,
"tweet_type": "REPLY",
"retweet_type": "NONE"
},
"author": {
"handle": "HarryTurney",
"userid": "219699434",
"bio": "Website\nhttps://t.co/DXwbh4DSSn\nhttps://t.co/sv10Nb7quf\n\nHelp the website:\nhttps://t.co/04ZVp8HDzA\n\nDiscord\nhttps://t.co/xaTBDrA2ZC",
"name": "Asuna",
"profile_image_source": "https://pbs.twimg.com/profile_images/1768656974961385472/89Y1bZXn_normal.jpg",
"likes_count": 26275,
"verified": true,
"bio_links": [
"https://pbs.twimg.com/profile_banners/219699434/1710707766"
],
"protected": false,
"has_nft_avatar": false,
"url": "https://twitter.com/HarryTurney"
},
"content": {
"body": "@_Avsece_ GLB has some rate limits that stopped me from getting stats. I've disabled it and will give it a look tomorrow to see if I can get it working. For now, just use the JPN side.",
"mentions": [
"_Avsece_"
],
"followers": 27670,
"following": 661,
"favorites": 0
},
"enrichment": {
"language": "en"
},
"source": {
"link": "https://twitter.com/HarryTurney/status/1813930239422239232"
}
}
Creating Data Collection Job
Example for creating a data collection job from vetric-x-search-latest data source, for content having exact phrase "cats" and published between 2023-10-01 and 2023-10-02 only.
curl --location 'https: //api.platform.datastreamer.io/api/pipelines/{PIPELINE_ID}/components/{COMPONENT_ID}/jobs?ready=true' \
--header 'apikey: <your-api-key>' \
--header 'Content-Type: application/json' \
--data \
'{
"job_name": "my twitter data collection",
"data_source": "vetric_x_search_latest",
"from": "2023-10-01",
"to": "2023-10-02",
"query": {
"query": "cats"
},
"job_type": "oneTime"
}'
Search Query
X (Twitter) Search API operators are supported, below are some examples for search queries to use with data collection jobs, described above. For more details on supported search operators, see Vetric APIs.
Warning
'since' and 'until' operators are not supported, please use 'from' and 'to' as shown examples below.
Example 1
{
"data_source": "vetric_x_search_latest",
"query": {
"from": "2024-07-05",
"to": null,
"query": {
"query": "copppa filter:replies lang:ar"
}
}
}
Example 2
{
"data_source": "vetric_x_search_latest",
"query": {
"from": "2020-10-01",
"to": "2020-10-28",
"query": {
"query": "#black_cats filter:images"
}
}
}
Including Replies
The Vetric X (Twitter) Search connector now has an additional feature that allows the inclusion of replies associated with each post (Tweet) found. Users can retrieve up to a specified number of replies for each post.
Data Collection Job Example
This example demonstrates a job query that will fetch up to 50 X Posts (Tweets) for the keyword raspberrypi, and then retrieve up to 20 replies for each of those posts found.
Note
You can specify the
max_documentsparameter to control the volume of data retrieved in both the main and included queries.
curl --location 'https://dev.api.platform.datastreamer.io/api/pipelines/{PIPELINE_ID}/components/{COMPONENT_ID}/jobs?ready=true' \
--header 'apikey: <your-api-key>' \
--header 'Content-Type: application/json' \
--data \
'{
"job_name": "65a085a5-69ec-461e-98c8-4713f33eb724",
"data_source": "vetric_x_search_latest",
"query": {
"include": [
{
"data_source": "vetric_x_replies",
"max_documents": 20
}
],
"query": "raspberrypi"
},
"job_type": "oneTime",
"query_from": "2024-10-01T00:00:00.000Z",
"query_to": "2024-10-29T00:00:00.000Z",
"max_documents": 50
}'
Important: Replace the PIPELINE_ID and COMPONENT_ID values with those corresponding to your pipeline.
Data Collection Job Results
Once X Search results have been fetched, the query will then retrieve up to 20 replies from the vetric_x_replies data source for each post. These replies are linked back to the related post by matching the post ID.
Here's an example of the response for the comments retrieved:
{
"documents": [
{
"data_source": "vetric_x_search_latest",
"meta_data": {
"job_label": "Label",
"job_id": "job-id"
},
"id": "1851041965691003142",
"doc_date": "2024-10-28T23:22:49Z",
"twitter": {
"user_id": "1280340517361012737",
"post_identifier": "1851041965691003142",
"quote_count": 0,
"reply_count": 0,
"retweet_count": 0,
"user_verified": false,
"tweet_type": "POST",
"retweet_type": "NONE"
},
"author": {
"handle": "DJFERALCATGIRL",
"userid": "1280340517361012737",
"bio": "★ lavender starlight | tkaronto | ae/aer | 28 ★\n\nkünstlerin, archivarin, musikerin, foto/videografin, DJ & softwareentwicklerin.\n\nbassist in boxcutter.",
"name": "🌸 CHXRRY STARLIGHT 🌸",
"profile_image_source": "https://pbs.twimg.com/profile_images/1769097772706226176/wpqHnyFD_normal.png",
"location": "CYBERIA:SOUNDKOLLEKTIV",
"likes_count": 7654,
"verified": false,
"bio_links": [
"https://pbs.twimg.com/profile_banners/1280340517361012737/1708657299"
],
"protected": false,
"created_at": "2020-07-07T03:19:08Z",
"url": "https://twitter.com/DJFERALCATGIRL"
},
"content": {
"published": "2024-10-28T23:22:49Z",
"body": "i need a raspberry pi 4/5 so bad. i have an idea for a custom SD card image that's themed to look like Windows 2K and runs games of the era w/ almost no setup from the end user using WINE but. i do Not have a Pi to actually test this with.",
"views_count": 58,
"found_by": "bc0b297f-adab-401f-b8f9-3add99c5d663",
"followers": 197,
"following": 142,
"favorites": 2
},
"enrichment": {
"language": "en"
},
"source": {
"link": "https://twitter.com/DJFERALCATGIRL/status/1851041965691003142"
}
}
]
}
Important: Only a single include object can be requested per query.
Updated 12 days ago