Twitter/X
The Twitter/X source collects posts, replies, and public account content from Twitter/X as part of a Data Stream.
Datastreamer selects from available providers automatically based on your Job configuration. You do not need to manage provider accounts or credentials to use this source.
API data_source: twitter_auto
Configuring a Job
Every X/Twitter Job sets query.platform to twitter and passes a search expression in query.query, using X/Twitter search operators (see X/Twitter Query Syntax below). This source has no query_type.
Job options
| Field | Description |
|---|---|
query.prioritization | freshness favors newer documents; coverage favors collecting a greater volume. See Freshness or Coverage. |
job_type | onetime runs once over a date range; periodic runs on a schedule. |
query_from, query_to | One-time Jobs only. Start and end of the date range, in ISO 8601 format. Both are required. |
max_documents | Optional. Maximum documents to return. This is a soft limit: because pipelines are asynchronous, a few more documents may be returned. |
query.per_day_limit | Optional. Maximum documents to return per day, for date ranges longer than 24 hours. |
query.include[].max_documents | Optional. Collects comments with each post, up to this many per post (a soft limit). Example: "include": [{ "max_documents": 10 }]. |
label, tags | Optional. A label for grouping Jobs, and tags for filtering. |
Example Job request
curl --request POST \
--url 'https://api.platform.datastreamer.io/api/pipelines/{PIPELINE_ID}/components/{COMPONENT_ID}/jobs?ready=true' \
--header 'apikey: <your-api-key>' \
--header 'Content-Type: application/json' \
--data '{
"job_name": "twitter-example",
"component_name": "twitter-auto",
"query": {
"platform": "twitter",
"query": "(electric OR hybrid) vehicle lang:en",
"prioritization": "coverage",
"include": [
{
"max_documents": 10
}
],
"per_day_limit": 500
},
"data_source": "twitter_auto",
"job_type": "onetime",
"query_from": "2026-10-01T00:00:00Z",
"query_to": "2026-10-02T00:00:00Z",
"max_documents": 1000
}'Refer to the Job creation documentation for full configuration details: Creating Jobs
API Reference: Create a Job
Do not include dates in your query string.
The Job system handles date ranges and historical searches. Including dates inside the query may cause errors or limit provider selection.
Twitter/X Query Syntax
Queries for the Twitter/X source use Twitter/X advanced search formatting. The table below covers the most common operators. For how query types are interpreted across sources, see Query Handling.
| Operator | Example | Description |
|---|---|---|
| Phrase | "dog house" | Exact phrase match |
| NOT | -mouse | Exclude this term |
| From | from:cat | Posts from this handle (no @ symbol) |
| To | to:dog | Posts directed at this handle (no @ symbol) |
| Hashtag | (#dogsrule) | Hashtags in parentheses, always with # symbol |
| Language | lang:en | ISO 2-letter language code |
| Replies only | filter:replies | Filter to replies only (default includes both posts and replies) |
| Min replies | min_replies:100 | Minimum reply count |
| Min likes | min_faves:100 | Minimum like count |
| Min retweets | min_retweets:100 | Minimum retweet/repost count |
| Has link | filter:links | Only posts containing links |
Twitter/X treats spaces as AND and uses negation with a minus sign rather than NOT. Examples:
| Intent | Query | Boolean equivalent |
|---|---|---|
| Either term | cat OR dog | cat OR dog |
| Both terms | cat dog | cat AND dog |
| Exclude a term | (cat OR dog) -mouse | (cat OR dog) NOT mouse |
| Combined | (cat OR dog) ferret | (cat OR dog) AND ferret |
| Complex | (cat OR dog) mouse from:parrot | (cat OR dog) AND mouse AND from:parrot |
A query is limited to 400 characters.
For a full reference, see X Advanced Search.
What is Collected
Each document returned from the Twitter/X source represents a post or reply. Fields are mapped to the Datastreamer unified schema and include content text, author metadata, engagement metrics (likes, retweets, replies, quotes), and post date.
Platform-specific fields are available under the twitter schema namespace. See the Schema Reference for field details.
Troubleshooting
Job fails or returns no data
- Check that the query syntax is valid
- Verify the date range contains data
- Confirm filters are not too restrictive
- Review Job logs for specific errors
Unexpectedly high document counts
- Add engagement filters (e.g. min_faves, min_retweets) to reduce volume
- Set a document limit on the Job
- Review the DVU pricing page
Provider switch noted in Job logs
- This is expected behavior. If a provider is unavailable, the Job is routed to an alternative automatically. No action is required.
Related
- Sources Overview
- Creating Jobs
- Data Volume Units
- Direct Integrations: use if you need a specific provider directly
Updated 12 days ago