Troubleshooting & FAQ

What is Apify, and how does it work with Datastreamer?

Apify is a cloud platform for web scraping using prebuilt “actors.” You can use Apify actors to extract data from websites and send that data into Datastreamer pipelines for transformation and delivery.

How do I connect an Apify actor to a Datastreamer pipeline?

You can view the Apify Actor Integration setup guide here: Link

Can I use any actor from the Apify Store?

Yes, most public actors can be used as ingresses. Custom actors are also supported.

Does Datastreamer support streaming real-time output from Apify?

Apify runs actors in batch mode, and does not currently offer streaming data actors. However, you can simulate near-real-time by scheduling periodic Jobs on the Apify Actor Integration. Running these Jobs at short intervals can provide similar benefits.

Can I enrich or normalize data from Apify actors after ingestion?

Yes. Once Apify data enters the Datastreamer pipeline, you can apply any enrichment, transformation, or routing rules as you would with other sources.

My custom Apify actor collects documents, but they fail at the Unify Transformer. Why?

Custom (non-Store) actors can return a JSON shape that Unify does not recognize as a known source schema, so the documents fail at the Unify step instead of continuing downstream. Add a JSON Schema Transformer before Unify to pre-map the actor's fields into the Datastreamer Default Schema. See Does Unify map every source automatically?.

What happens when an Apify actor returns a "no results" or error object?

Some actors return an error object instead of content when a search term yields no results, for example:

{"error": "No videos found for the search query", "errorCode": "SEARCH_QUERY_NOT_FOUND", "input": "example query"}

This payload has no content to map, so it will fail at a downstream transform such as Unify. To keep these out of your pipeline:

• Add a Lucene Document Filter after the Apify component to drop documents that contain an errorCode (or error) field before they reach Unify.
• Alternatively, narrow the actor's searchQueries to terms that reliably return results.

The volume is usually small (one object per empty query per run), but filtering keeps your Failed Items list clean and your counts accurate.