Collect Data with the Jobs API (AI Agent Guide)

This guide covers creating and managing data collection Jobs on Auto Sources through the API. Read the AI Agent Guides index first for the shared basics (base URL, apikey header, dates) and where the user finds the API key. To read collected data back, see Read Results from Searchable Storage.

How Datastreamer Fits Together

• A Pipeline (Data Stream) is a configured flow: sources, then transformation and enrichment, then destinations. Pipelines are built in the Datastreamer Portal, not through this guide.
• An Ingress Component (also called a Step) is the part of a pipeline that collects data. Jobs are created against a specific Pipeline ID and Component ID.
• A Job is one collection task: a source, a query, a time window or schedule, and limits. Jobs are what an agent creates and manages.
• Auto Sources are the managed sources of a Data Stream (twitter_auto, reddit_auto and so on). You choose the platform and the query. Datastreamer picks the best provider for each Job and fails over automatically. The underlying provider is not exposed by default.

Reference: Data Streams Overview · Sources Overview · Glossary

What You Need From Your User

An agent cannot create these itself. If any are missing, ask the user for them. Do not guess IDs.

Tip for users: in the Portal's Manage Jobs view, configure a Job once and select Code. It shows the exact API request for that component, which is the most reliable template for an agent to copy.

Default Guardrails for Agents

Creating and running Jobs uses DVUs, which the user pays for. Apply these rules by default, and tell your user at the start that you are applying them unless they say otherwise. For example: "I'll follow Datastreamer's default agent guardrails: a document limit on every Job, a small test run first, and your confirmation before any recurring or bulk Jobs. Tell me if you want any of these changed."

1. Always set max_documents. Start around 1,000 or lower, and raise it only after reviewing results.
2. Test before you scale. Run a one-time Job on a short window (for example, one day) before creating periodic or bulk Jobs.
3. Confirm before creating periodic Jobs, bulk Jobs, or anything large. State the source, query, schedule, max_documents, and the estimated DVUs per run (see Usage and Cost below) and wait for the user's go-ahead.
4. Never delete Jobs unless the user asks. To stop collection, cancel instead (see Cancel a Job).
5. Poll gently. When waiting on a Job, check its state no more often than every 30 seconds.
6. Keep dates out of query strings. Use query_from / query_to for time windows.
7. Report what you created. Give the user the job_id, job_name, and label of every Job you create.

Managing Jobs

All Job endpoints sit under:

https://api.platform.datastreamer.io/api/pipelines/{PIPELINE_ID}/components/{COMPONENT_ID}/jobs

Create a Job

POST .../jobs?ready=true

curl --location 'https://api.platform.datastreamer.io/api/pipelines/{PIPELINE_ID}/components/{COMPONENT_ID}/jobs?ready=true' \
  --header 'apikey: <your-api-key>' \
  --header 'Content-Type: application/json' \
  --data '{
    "job_name": "twitter-ev-charging-test",
    "data_source": "twitter_auto",
    "query": {
      "query": "\"ev charging\" lang:en"
    },
    "job_type": "onetime",
    "query_from": "2026-06-01T00:00:00Z",
    "query_to": "2026-06-02T00:00:00Z",
    "max_documents": 1000,
    "label": "ev-charging-test"
  }'

Responses return job_type as oneTime or periodic.

The response includes the new Job's job_id. Keep it: every other Job operation uses it.

Reference: Creating Jobs via Jobs API

Create Jobs in Bulk

POST .../jobs/bulk?ready=true with a JSON array of Job objects in the same format as Create a Job. Confirm with the user first (guardrail 3).

Get a Job

GET .../jobs/{JOB_ID}

Key response fields:

Reference: Get Job Details

List Jobs

GET .../jobs?count=50

Returns { "start": 0, "count": N, "records": [ ... ] }. Each record has the same shape as Get a Job. The list supports start as well as count for paging, for example ?count=50&start=50 for the next page.

Reference: Listing Jobs

Cancel a Job

PATCH .../jobs/{JOB_ID}/state/cancelled

Stops further collection. Work already in progress is not cancelled, so some data may still arrive. This is the preferred way to stop a Job.

Reference: Cancelling Jobs

Delete a Job

DELETE .../jobs/{JOB_ID}

Returns an empty success response. Only do this when the user explicitly asks (guardrail 4).

Reference: Deleting Jobs

Find Jobs Across Pipelines

POST https://api.platform.datastreamer.io/api/pipelines/jobs/search

Searches Job history with a Lucene query over fields such as pipeline_id, job_id, job_name, data_source, job_type, state and label. Operators (AND, OR, NOT, TO) must be upper case.

{
  "query": {
    "sort": [ { "field": "updated", "order": "DESC" } ],
    "from": 0,
    "size": 100,
    "query": "label:ev-charging-test AND state:failed"
  }
}

Reference: Advanced Job Search

A Typical Agent Flow

1. Confirm you have the API key, Pipeline ID and Component ID, and tell the user which guardrails you are applying.
2. Create a one-time test Job on a short window with a low max_documents.
3. Poll Get a Job every 30 seconds or more until state is succeeded or failed. On failed, read error and adjust the query.
4. Check results: if the pipeline writes to Searchable Storage, use the Searchable Storage guide; otherwise check the destination with the user's own tools. Check DVUs used (see Usage and Cost below) and show the user.
5. With the user's confirmation, create the periodic or larger Jobs.

Auto Sources

Use the Auto Source name as data_source. Query syntax and available search targets differ by platform, so check the source page before building a query.

Query rules that apply across Auto Sources:

• A bare term is treated as a keyword search. Other query types (such as boolean) must be specified explicitly. See Query Handling.
• Support for AND / OR / NOT varies by platform. On Reddit, Threads and TikTok, complex queries can return older results.
• X/Twitter uses its own operators: spaces mean AND, -term excludes, lang:en filters language, and a query is limited to 400 characters.
• Keep keyword lists short (8 or fewer per Job). Split broad topics into several Jobs.
• For long historical windows, create one Job per day rather than one Job over the whole range.

Reference: Data Stream Sources Capability Table · Best Practices for Data Collection Jobs

Usage and Cost (DVUs)

Usage is measured in Data Volume Units (DVUs).

Formula per run: 1 + max(0, ceil((documents - 100) / 100)).

Estimating before you create a Job: with max_documents: 1000, one run uses about 10 DVUs (max_documents is a soft cap, so a run can slightly exceed it), plus any enrichment DVUs, which are billed separately. A periodic Job every 6 hours runs 4 times a day, so about 40 DVUs a day before enrichments. Give the user this kind of estimate before creating periodic Jobs.

Checking actual usage: POST https://api.platform.datastreamer.io/api/v2/customer-usage/jobs

{ "job_ids": [ "<job-id-1>", "<job-id-2>" ] }

Returns { "jobs": [ { "job_id": ..., "dvu_count": N } ] }. Add "group_by": "DataSource" to group by source. Up to 500 records per request.

Reference: What are Data Volume Units? · Jobs DVU Count API · Optimizing Your Usage

When Something Goes Wrong

Failed work items can be retried with PUT https://api.platform.datastreamer.io/api/pipelines/work-items/retry?type=restart (or type=recover). See Job Failure Handling.

Related

• Creating Jobs via Jobs API
• Sources Overview
• Read Results from Searchable Storage
• Data Volume Units