The Pipeline Import/Export feature provides a simple way to transfer pipeline configurations between accounts or create backups with a single click.

The generated file uses a standardized JSON format (a .pipeline file) that is both machine-readable and human-understandable, and contains the functionality required to configure a new pipeline!

Read more: https://docs.datastreamer.io/docs/pipeline-export-and-import#/

Additional data source components have been added to support new Vetric data source endpoints.

• Greater support for custom data sources: https://docs.datastreamer.io/update/docs/private-data-sources#/
• Improvements to Pipeline validation, improving the error messages when running misconfigured pipelines.
• Removed empty and unnecessary fields from Pipeline export files.
• Fixed issues with query builder when using select Socialgist sources.
• Improved chunking when using GenAI Translation component.

Within Datastreamer, you can now integrate any Apify actor into your pipelines!

About Apify: Apify provides a marketplace of scrapers designed for large-scale web scraping, data extraction, and browser automation. This marketplace of community-made scrapers, let's you run 5,000+ different scrapers known as "Actors".

The new Apify Actor Integration component allows you to use any of the Actors from Apify within your pipeline.

To get started, check out this Setup Guide: Apify Integration Setup.

Not sure what to start with? Registry offers a selection of the 5,000+ Apify Actors:

• Apify's Instagram Profile Actor
• Apify's Facebook Post Actor
• Apify's Facebook Comment Actor
• Apify's Facebook Groups Actor
• Apify's Community Actors
• Apify's Amazon Scraper
• Apify's AI Website Crawler
• Apify's YouTube Scraping Actor
• Apify's Google Maps Scraping Actor
• Apify's Google Search Actor
• Apify's TikTok Hashtag Scraping Actor
• Apify's TikTok Comments Scraping Actor
• Apify's Tiktok Profile Scraping Actor
• Apify's Instagram Post Scraping Actor

We've released a fix that was causing per-document storage in AWS S3 storage.

You can now connect your pipelines running on Datastreamer to Cloudflare R2 storage.

With supporting connections to both ingress and egress components.

• Cloudflare R2 Ingress: https://docs.datastreamer.io/docs/cloudflare-r2-ingress-setup-guide#/
• Cloudflare R2 Egress: https://docs.datastreamer.io/docs/cloudflare-r2-storage-egress#/

Query validation has been added to select Vetric data sources.

This query validation makes Job creation easier by highlighting any query issues where users are using fields and formats that are not accepted.

A number of video and audio enrichment models and classifiers have been released into Preview mode. These models and classifiers from Social Voice excel at short-form and long-form internet video content.

• Social Voice Transcription: Speech-to-text transcription of voice in a video or audio file. https://datastreamer.io/catalog/social-voice-transcription-model/
• Social Voice Toxicity Classifier: Determines metrics on different toxic elements which may be present in the transcription. https://datastreamer.io/catalog/social-voice-toxicity-classifier/
• Social Voice IAB Category Classifier: Analyzes the content of the video or audio transcription and categorizes it using IAB categorization. https://datastreamer.io/catalog/social-voice-iab-category-classifier/
• Social Voice Personality Model: Analyzes the content of the video or audio transcription and returns values corresponding to different Myers–Briggs Type Indicators. https://datastreamer.io/catalog/social-voice-personality-model/
• Social Voice Brand Safety Model: Analyzes different risks to brand safety according to GARM categorizations. https://datastreamer.io/catalog/social-voice-brand-safety-model-garm/
• Social Voice Political Leaning Model: Determines the political leaning of the language used in the transcription. https://datastreamer.io/catalog/social-voice-political-leaning-model/
• Social Voice Tonality Classifier: Determines if the tone of the transcription is more personal or corporate. https://datastreamer.io/catalog/social-voice-tonality-classifier/
• Social Voice Direction Focus Classifier: Analyzes text to determine the standpoint or directional emphasis of the transcription. https://datastreamer.io/catalog/social-voice-direction-focus-classifier/
• Social Voice On-Screen Text Detection Model: Detection of text on-screen in a video scene or image. https://datastreamer.io/catalog/social-voice-on-screen-text-detection-model/
• Social Voice On-Screen Logo Detection Model: Detection of brand logos in a video scene or image. https://datastreamer.io/catalog/social-voice-on-screen-logo-detection-model/

The OpenAI text prompt completion model has been updated to match the "Secrets" patterns of adding API keys in a secure manner within the platform.

A number of text enrichment models and classifiers have been released into Preview mode. These models from Tisane excel at analysis of forum, chat, and live conversation analysis.

• Tisane Sentiment Analysis: Tisane sentiment analysis supports 35+ languages, including slang and obfuscated text. https://datastreamer.io/catalog/tisane-sentiment-analysis/
• Tisane Entity Extraction: Detect mentions of people, organizations, locations, filenames, phone numbers, crypto addresses, and more. Tisane extracts both standard entities and those relevant to trust & safety/law enforcement applications. https://datastreamer.io/catalog/tisane-entity-extraction/
• Tisane Topic Extraction: Also known as: theme identification, subject detection, or key topic recognition. Topic extraction determines the dominant topics in the text. Supports IPTC, IAB, and Wikidata IDs. https://datastreamer.io/catalog/tisane-topic-extraction/