Listing Jobs via Jobs API

This page provides instructions on how to list jobs using the Jobs API. The API allows you to retrieve information about jobs associated with a specific pipeline and Ingress Component within your Datastreamer dynamic-pipelines environment.

Required Parameters

Before using the Jobs API, ensure you have the following:

Pipeline ID: The ID of the pipeline for which you want to list jobs.
Component ID: The ID of the component within the pipeline.

GET Jobs

To list jobs, you can use the following endpoint. Make sure to replace the placeholders with your actual values

curl --location 'https://api.platform.datastreamer.io/api/pipelines/:PipelineId/components/:ComponentId/jobs?count=50' \
--header 'apikey: API_KEY'

Example

Here is an example of how the curl command might look after replacing the placeholders:

curl --location 'https://api.platform.datastreamer.io/api/pipelines/12345/components/67890/jobs?count=50' \
--header 'apikey: abcdef123456'

The API will return a JSON response containing details of the jobs, such as job IDs, statuses, and timestamps.

{
    "jobs": [
        {
            "organization_id": "129706a7-0af1-4dc5-8eae-06404a89c3ca",
            "pipeline_id": "data365-twitter-keywords-1",
            "step_id": "docsin0",
            "job_id": "7c4623a3-10da-4638-b90f-995b7ca6d20e",
            "job_name": "test_twitter_keywords",
            "work_item_id": "aa7ec53c-b1df-43c2-b403-785e4ceb3c4a",
            "data_source": "twitter_keywords",
            "job_type": "oneTime",
            "job_state": "succeeded",
            "label": null,
            "tags": null,
            "created": "2024-05-01T21:23:36.627326Z",
            "updated": "2024-05-01T21:23:36.627326Z",
            "preferred_start_time": "2024-05-01T21:27:38.945018Z",
            "query_from": "2024-01-01T00:00:00Z",
            "query_to": "2024-02-01T00:00:00Z",
            "query": "raspberry pi",
            "max_documents": 0,
            "work_time_offset": 0,
            "start_time_offset": 0
        },
        {
            "organization_id": "129706a7-0af1-4dc5-8eae-06404a89c3ca",
            "pipeline_id": "data365-twitter-keywords-1",
            "step_id": "docsin0",
            "job_id": "a4a7f219-1eaf-42ff-bfb4-5a637d80fc44",
            "job_name": "test_twitter_keywords",
            "work_item_id": "c08aa1ab-3f7b-465b-9a68-f9cbb01d3697",
            "data_source": "twitter_keywords",
            "job_type": "oneTime",
            "job_state": "running",
            "label": null,
            "tags": null,
            "created": "2024-05-01T22:17:05.653031Z",
            "updated": "2024-05-01T22:17:05.653031Z",
            "preferred_start_time": "2024-05-02T16:08:22.335522Z",
            "query_from": "2024-01-01T00:00:00Z",
            "query_to": "2024-02-01T00:00:00Z",
            "query": "raspberry pi",
            "max_documents": 0,
            "work_time_offset": 0,
            "start_time_offset": 0
        },
        {
            "organization_id": "129706a7-0af1-4dc5-8eae-06404a89c3ca",
            "pipeline_id": "data365-twitter-keywords-2",
            "step_id": "docsin0",
            "job_id": "082109a9-32a7-4bfe-946d-e91b22a4eb2c",
            "job_name": "test_twitter_keywords",
            "work_item_id": "c39e24e5-1fbd-4aee-a789-9c996b6b0e5f",
            "data_source": "twitter_keywords",
            "job_type": "oneTime",
            "job_state": "failed",
            "label": null,
            "tags": null,
            "created": "2024-05-02T19:20:52.310094Z",
            "updated": "2024-05-02T19:20:52.310094Z",
            "preferred_start_time": "2024-05-02T19:21:19.140537Z",
            "query_from": "2024-01-01T00:00:00Z",
            "query_to": "2024-02-01T00:00:00Z",
            "query": "dot matrix",
            "max_documents": 0,
            "work_time_offset": 0,
            "start_time_offset": 0
        }
    ],
    "page": 1,
    "page_count": 50,
    "total_count": 3
}

📘
The count parameter specifies the number of jobs to return. Adjust this number based on your needs.