Advanced Job Search

The Find Jobs REST endpoint allows you job history to be searched using powerful Lucene queries.

Job Search Endpoint

The endpoint accepts a HTTP POST request with a JSON payload that contains search information

{
  "query": {
    "sort": [
      {
        "field": string
        "order": string
      }
    ],
    "from": int
    "size": int
    "track_total_hits": bool
    "query": string
  }
}

Where:

The payload can be sent to the endpoint as follows:

curl --request POST \
  --url http://localhost:5028/api/pipelines/jobs/search \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/json' \
  --header 'apikey: *'
  --data '{
  "query": {
    "sort": [
      {
        "field": "updated"
        "order": "DESC"
       }
    ],
    "from": 0,
    "size": 100,
    "track_total_hits": false,
    "query": ""
  }
}'

Results

The API will return a JSON response containing details of the jobs, such as job IDs, statuses, and timestamps. e.g:-

{
  "total": 127,
  "records": [
    {
      "organization_id": "129706a7-0af1-4dc5-8eae-06404a89c3ca",
      "pipeline_id": "data365-twitter-keywords-1",
      "step_id": "docsin0",
      "job_id": "7c4623a3-10da-4638-b90f-995b7ca6d20e",
      "job_name": "test_twitter_keywords",
      "work_item_id": "aa7ec53c-b1df-43c2-b403-785e4ceb3c4a",
      "data_source": "twitter_keywords",
      "job_type": "oneTime",
      "job_state": "succeeded",
      "label": null,
      "tags": null,
      "created": "2024-05-01T21:23:36.627326Z",
      "updated": "2024-05-01T21:23:36.627326Z",
      "preferred_start_time": "2024-05-01T21:27:38.945018Z",
      "query_from": "2024-01-01T00:00:00Z",
      "query_to": "2024-02-01T00:00:00Z",
      "query": "raspberry pi",
      "max_documents": 0,
      "work_time_offset": 0,
      "start_time_offset": 0
    },
     ...
  ]
}

Lucene Search

Lucene queries are described on this page: Apache Lucene - Query Parser Syntax

❗

Keywords such as AND, OR, NOT and TO must be in upper case.

Not all features of Lucene are supported (such as boosting and ~).

The fields available for searching are

Notes:

1. If no field is specified, then query, label and tags will be searched.
2. All string fields can match on the wildcard character * .
3. All string fields can be quoted with "..." for a match on the entire string without wildcards
4. * can be escaped with \ in an unquoted string if you need to find the * character.
5. null Can be used to find jobs where the value is not set. You can also use the negation character ! to find non-null values, e.g: label: !null

Date Format

Dates can be entered as follows:

1. UTC time in ISO8601 Date/Time format (YYYY-MM-DDTHH:MM:SS)

2. Date in ISO8601 Date format (YYYY-MM-DD)

3. Offset expression: [+|-]<number><unit> (e.g. -1h).

   Value	Unit
   s	Seconds
   mi	Minutes
   h	Hours
   d	Days
   w	Weeks
   m	Months
   y	Year

   
   

4. Fixed expression

   Value	Description
   now	The current date
   noon	The current evaluated date at 12:00:00
   midnight	The current evaluated date at 00:00:00
   eod	The current evaluated date at 1 tick before midnight
   fom	Midnight on the first of the current evaluated month.
   eom	The current evaluated date set 1 ticket before midnight on the last day of the month
   sow	00:00:00 on the preceding Sunday for the current evaluated date.
   eow	1 ticket before midnight on the next Saturday after the current evaluated date.
   today	Sets the current evaluated date to today's date. Time of day is unchanged
   yesterday	Sets the current evaluated date to yesterday’s date. Time of day is unchanged
   tomorrow	Sets the current evaluated date to tomorrow’s date. Time of day is unchanged
   at “hh:mm”	Sets the current time to hh:mm

Examples

1. state: succeeded AND updated: [-3m to -1m] Return all successfully jobs completed between 1 and 3 months ago.
2. cat OR dog AND data_source: wsl_instagram Return all jobs searching for cat or dog on the wsl_instagram data source.
3. label: tiger AND created: [fom-1m TO eom-1m] Return all jobs with label tiger created in the previous calendar month
4. fish AND chips AND [sow-1w TO eow-1w] Return all jobs searching for fish and chips for the previous week (Sunday 00:00.00 to Saturday 23:59:59.999)