Guides
Guides
Get Started

Include vetric_x_replies

Include X (Twitter) replies from search latest results

Suggest Edits

The vetric_x_search_latest connector now has an additional feature that allows the inclusion of replies associated with each post (Tweet) found. Users can retrieve up to a specified number of replies for each post.

Data Collection Job Example

This example demonstrates a job query that will fetch up to 50 X Posts (Tweets) for the keyword raspberrypi, and then retrieve up to 20 replies for each of those posts found.

📘

Note: You can specify the max_documents parameter to control the volume of data retrieved in both the main and included queries.

curl --location 'https://dev.api.platform.datastreamer.io/api/pipelines/{PIPELINE_ID}/components/{COMPONENT_ID}/jobs?ready=true' \
      --header 'apikey: <your-api-key>' \
      --header 'Content-Type: application/json' \
      --data \
        '{
          "job_name": "65a085a5-69ec-461e-98c8-4713f33eb724",
          "data_source": "vetric_x_search_latest",
          "query": {
            "include": [
              {
                "data_source": "vetric_x_replies",
                "max_documents": 20
              }
            ],
            "query": "raspberrypi"
          },
          "job_type": "oneTime",
          "query_from": "2024-10-01T00:00:00.000Z",
          "query_to": "2024-10-29T00:00:00.000Z",
          "max_documents": 50
        }'

Important: Replace the PIPELINE_ID and COMPONENT_ID values with those corresponding to your pipeline.

Data Collection Job Results

Once X Search results have been fetched, the query will then retrieve up to 20 replies from the vetric_x_replies data source for each post. These replies are linked back to the related post by matching the post ID.

Here's an example of the response for the comments retrieved:

{
  "documents": [
    {
      "data_source": "vetric_x_search_latest",
      "meta_data": {
        "job_label": "Label",
        "job_id": "job-id"
      },
      "id": "1851041965691003142",
      "doc_date": "2024-10-28T23:22:49Z",
      "twitter": {
        "user_id": "1280340517361012737",
        "post_identifier": "1851041965691003142",
        "quote_count": 0,
        "reply_count": 0,
        "retweet_count": 0,
        "user_verified": false,
        "tweet_type": "POST",
        "retweet_type": "NONE"
      },
      "author": {
        "handle": "DJFERALCATGIRL",
        "userid": "1280340517361012737",
        "bio": "★ lavender starlight | tkaronto | ae/aer | 28 ★\n\nkünstlerin, archivarin, musikerin, foto/videografin, DJ & softwareentwicklerin.\n\nbassist in boxcutter.",
        "name": "🌸 CHXRRY STARLIGHT 🌸",
        "profile_image_source": "https://pbs.twimg.com/profile_images/1769097772706226176/wpqHnyFD_normal.png",
        "location": "CYBERIA:SOUNDKOLLEKTIV",
        "likes_count": 7654,
        "verified": false,
        "bio_links": [
          "https://pbs.twimg.com/profile_banners/1280340517361012737/1708657299"
        ],
        "protected": false,
        "created_at": "2020-07-07T03:19:08Z",
        "url": "https://twitter.com/DJFERALCATGIRL"
      },
      "content": {
        "published": "2024-10-28T23:22:49Z",
        "body": "i need a raspberry pi 4/5 so bad. i have an idea for a custom SD card image that's themed to look like Windows 2K and runs games of the era w/ almost no setup from the end user using WINE but. i do Not have a Pi to actually test this with.",
        "views_count": 58,
        "found_by": "bc0b297f-adab-401f-b8f9-3add99c5d663",
        "followers": 197,
        "following": 142,
        "favorites": 2
      },
      "enrichment": {
        "language": "en"
      },
      "source": {
        "link": "https://twitter.com/DJFERALCATGIRL/status/1851041965691003142"
      }
    }
  ]
}

Important: Only a single include object can be requested per query.

Updated about 2 months ago