Creating Jobs via Jobs API
Learn how to create Jobs for data collection via Jobs API
API calls to create Jobs are interacting with this Pipeline Component; so it is required to have both correct Pipeline ID and Step ID.
In the Portal, open your pipeline's Ingress Component and go to its "Manage Jobs" view to retrieve the Step ID and Pipeline ID, which form part of the request URL.
Quick TipWhile creating an initial Job in the UI is not required, the UI provides visibility of the API request "Code" for generating your API request.
By configuring the job in the UI you can use this to provide a completed API request visible by selecting the "{} Code" button. This will provide a code snippet that you can use as a template for generating additional Jobs via the API endpoint. Selecting "Add" will manually add a Job for immediate processing.
Understanding Job Creation API Request
Here is a sample Job Creation API Request.
curl --location 'https://api.platform.datastreamer.io/api/pipelines/3f8be588/components/d5wykwcclfi/jobs?ready=true' \
--header 'apikey: <your-api-key>' \
--header 'Content-Type: application/json' \
--data \
'{
"job_name": "bf5912e9-5e09-45c8-9a61-3ea75fbc7248",
"data_source": "socialgist_blogs",
"query": {
"query": "cats AND dogs",
"country": "FR",
"language": "fr"
},
"job_type": "periodic",
"schedule": "0 0 0/6 1/1 * ? *",
"label": "Test Label"
}'API Reference: Create a Job
URL (Required)
'https://api.platform.datastreamer.io/api/pipelines/3f8be588/components/d5wykwcclfi/jobs?ready=true' \The location is the same for each new job for that Step (Ingress Component). It contains the ID for the Pipeline, as well as the Ingress Component.
Job Name (Required)
"job_name": "bf5912e9-5e09-45c8-9a61-3ea75fbc7248",Job Name is used to identify this job in analytics and job management screens. It does not have specific requirements in format, and can be used to organize and track the names. Created jobs in the UI will have a randomly generated Job Name.
Data Source (Required)
"data_source": "socialgist_blogs",Some components may allow multiple sources, however most will only operate with a single data source. This is required.
Query (Required)
"query": {
"query": "cats AND dogs",
"country": "FR",
"language": "fr"The Query is required and contains the criteria of data collection. The available additional filters are optional and can vary by source. Common filters include language, country, content type, etc. All of the available additional filters are visible in the Ingress Component 'New Job' UI.
Prioritization (optional): for Auto Source Jobs, query.prioritization can be set to freshness or coverage. See Freshness or Coverage.
Job Type (Optional)
"job_type": "periodic",
"schedule": "0 0 0/6 1/1 * ? *",Job type specifies if the Job is onetime or periodic. If you do not specify it, the Job defaults to onetime and runs a single time. Responses return job_type as oneTime or periodic.
Periodic: This job will run on the specified schedule (a 7-field cron expression, as in the example), and collect new data between runs. If the specified schedule is more frequent than the source supports, the lowest supported interval is used.
One time: This job will run once and collect data from a specified time frame where supported by data sources. Time frames are determined by a "From" and "To" dates in this format: "2024-06-01T00:00:00Z"
"query_from": "2024-05-01T00:00:00Z",
"query_to": "2024-06-01T00:00:00Z"Labels
"label": "Test Label"As data is received from a Job, you can apply a Label, as a single string, to allow for additional sorting and analytics downstream.
Tags
As data is received from a Job, you can apply Tags, as an array of key value pairs.
"tags": [
{
"name": "default-folder",
"value": "test"
}
]Additionally when using Azure Blob Storage Egress, Google Cloud Storage Egress and Amazon S3 Storage Egress components tags can be used to control how the files are created:-
- Any tag can be used to specify the folder that files are created in. The Egress components can be configured to use Tag "value" for the folder name. For more information: Data Streams Overview
- The tag
collation_idcan be used instead of the work id when collating documents into a single file. This allows documents generated by different jobs to be collated together.
Max Documents
Max documents is available for Jobs, and acts as a soft cap on the documents ingested. It operates by measuring the amount of documents ingested, and ceasing additional collection within that Job after the max has been reached. Depending on the number of documents returned from data requests by providers the maximum count might be slightly higher. Integers are the supported input.
"max_documents": 10000Ready Parameter
By default, Jobs are created in the NotReady state and stay inactive until they are started explicitly using the state update endpoint. Add ?ready=true to the request URL to create Jobs ready to run, as the examples on this page do. See Job State Management.
Creating Jobs in Bulk via API
Jobs API supports creating many jobs using a single API call.
- See Understanding Job Creation API Request to obtain pipeline jobs URL
- Prepare a Collection of Jobs
- Similar to previous steps on creating a single Job via API, prepare many jobs as a collection
- Create and start jobs
Example - Creating Jobs in Bulk
curl --location 'https://api.platform.datastreamer.io/api/pipelines/3f8be588/components/d5wykwcclfi/jobs/bulk?ready=true' \
--header 'apikey: <your-api-key>' \
--header 'Content-Type: application/json' \
--data '[
{
"job_name": "bf5912e9-5e09-45c8-9a61-3ea75fbc7248",
"data_source": "socialgist_blogs",
"query": {
"query": "dogs",
"country": "FR",
"language": "fr"
},
"job_type": "periodic",
"schedule": "0 0 0/6 1/1 * ? *",
"label": "Job 1 Label"
},
{
"job_name": "8709610B-93AC-48BD-990B-BA3249F561D0",
"data_source": "socialgist_blogs",
"query": {
"query": "cats",
"country": "FR",
"language": "fr"
},
"job_type": "periodic",
"schedule": "0 0 0/6 1/1 * ? *",
"label": "Job 2 Label"
}
]'API Reference: Create Jobs in bulk
Updated 7 days ago