Job Failure Handling

A job is a collection of work items that are executed asynchronously and as such can succeed or fail independently. This page describes how to handle failures where the root cause is transient such as service outages etc.

There are two modes for handling for work item failures.

Recover

Continue a failed work item from where the failure occurred.

📘

Tip

Using this mode when the document ingestion has distinct collection and ingestion phases (e.g bright data) can help to prevent collecting previously collected data.

Restart

Start the work item from the beginning, collecting and processing all documents including any previously collected.

Retry from the Jobs Page

Retrying a job in the portal UI by either selecting the Retry option from the menu, or selecting the failed work item and clicking retry.

Using the REST API

The following request can be used to retry failed work items. The type query parameter can be restart or recover

curl --request PUT \  
  --url 'https://api.platform.datastreamer.io/api/pipelines/work-items/retry?type=restart' \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/json' \
  --header 'apikey: ***' \
  --data '{
	"work_item_ids": [
		"<work-item-id1>",
		"<work-item-id2",
		"<work-item-id3",
    ...
	]
}'

This will return the number of work items where the retry was started.

{
	"total": 1
}