For AI agents: visit https://docs.datastreamer.io/llms.txt for an index of all pages formatted in Markdown and endpoints in OpenAPI.
Jump to Content
View Interactive Demo ↗
Book a call ↗
Go to Datastreamer.io ↗
Documentation
Recipes & Templates
Product Updates
View Interactive Demo ↗
Book a call ↗
Go to Datastreamer.io ↗
Documentation
Documentation
Recipes & Templates
Product Updates
Violence Detection
Getting Started
👋 Welcome to Datastreamer!
Platform Overview
Creating Your Account
Platform Glossary
Architecture Overview
Platform FAQs
DATA STREAMS
Data Streams Overview
Sources
Sources Overview
Data Stream Sources Capability Table
Auto Profiles
Twitter/X
Threads
Facebook
YouTube
Reddit
Instagram
TikTok
🔜
Bluesky
Transformations & Operations
Unify Transformer
Unified Data Dictionary
Routing & Filtering
Concat
JSON Router
Splitters
Lucene Document Filter
Document Batcher
Document Deduplication
Custom Transformations
JSON Schema Transformer
Transformation Schema
Transformation Operations
JSON Conditions
Searchable Storage
Adding & Using Searchable Storage
Managing your Searchable Storage
Searchable Storage APIs
Search API
Performing a Query
Sample Search Request & Response
API Pagination
Aggregations with Search API
Count API
Destinations
Firehose
Webhook
Datastreamer Searchable Storage Ingress
Direct Data Upload
SFTP
SFTP Storage Egress
Event Streaming
Google Pub/Sub Ingress
Google Pub/Sub Egress Connector
File Storage
Datastreamer File Storage Egress
Datastreamer File Storage Ingress
Moving to Data Streams
JOBS
Connector Automation (Jobs)
Creating Jobs via Jobs API
Managing Jobs
Stopping a Job (Portal)
Listing Jobs via Jobs API
Get Job Details via Jobs API
Cancelling Jobs via Jobs API
Deleting Jobs via Jobs API
Updating Jobs (Portal, API)
Advanced Job Search
Job Failure Handling
Jobs Creation with AI/Agents
Best Practices for Data Collection Jobs
Jobs DVU Count API
Connector Automation (AI Tools)
MCP Server Setup Guide
Job Creation Agent
DIRECT INTEGRATIONS
Direct Integrations Overview
Data Sources
Apify
Apify Setup Guide
Troubleshooting & FAQ
Brightdata
Brightdata Account Setup
Brightdata Amazon Products
Brightdata CNN News
Brightdata Crunchbase Business
Brightdata Ebay Products
Brightdata Etsy Products
Brightdata G2 Reviews
Brightdata Github Code
Brightdata Glassdoor Jobs
Brightdata Google Shopping
Brightdata Indeed Jobs
Brightdata Instagram Posts
Brightdata Pinterest Posts
Brightdata Reddit Posts
Brightdata Shein Products
Brightdata Target Products
Brightdata Trustradius Reviews
Brightdata Walmart Products
Brightdata Yahoo Finance Business
Brightdata Youtube Posts
Bluesky Live Feed
DarkOwl Search
Datashake Social Media
Datashake Bluesky
Datashake Facebook
Datashake Instagram
Datashake LinkedIn
Datashake Reddit
Datashake X (Twitter)
Datashake YouTube
Exorde
Exorde X
Exorde Threads
Exorde Reddit
Exorde TikTok
Exorde YouTube
Opoint News
Socialgist
Socialgist Blogs
Socialgist Blog Links
Socialgist Boards
Socialgist Boards Compliance
Socialgist News
Socialgist News Compliance
Socialgist Quora
Socialgist Reddit
Socialgist Reddit Links
Socialgist Reviews
Socialgist Tencent Weibo
Socialgist Tiktok
Socialgist Videos
Socialgist VK
Talkwalker
Talkwalker Widget
Talkwalker Search
WebSightLine
WebSightLine Augmented Instagram
WebSightLine Instagram
WebSightLine Threads
Private Data Sources
Cloud Storage
Cloudflare R2 Storage Ingress
Cloudflare R2 Setup Guide
Cloudflare R2 Storage Egress
Azure Blob Storage Ingress
Azure Blob Storage Ingress Setup Guide
Azure Blob Storage Egress
Google Cloud Storage Ingress
Google Cloud Storage Ingress Setup Guide
Google Cloud Storage Egress
Amazon S3
Amazon S3 Ingress Setup Guide
Amazon S3 Storage Egress Connector
Enrichments & Models
GenAI ESG Classifier
Language Detection (Google Service)
Language Detection (Datastreamer)
Named Entity Recognition
GenAI Brand Recognition Classifier
GenAI Entity Recognition Classifier
GenAI Sentiment Classifier
GenAI Category Classifier
Market Interest Categorization Taxonomy
IPTC Media Topic Categorization Taxonomy
GenAI Emotion Detection Classifier
Product Sentiment Classifier
Sentiment Classification (short content)
Custom Functions
Content Similarity Clustering
GenAI Intent Classifier
Violence Detection
Hard News
Influence Classification
Private AI PII Redaction
Gemini Translate (Large Language Model)
Google Translate
Open AI Completion
Content Enrichment
X/Twitter Post Metrics Collector
X/Twitter Retweets Collector
File Operations
PDF Table Extraction to Unified Schema
PDF to JSON Text Extraction
WebSightLine File Fetcher
WebSightLine Profile Fetcher
Location Classification
Location Inference Models
Dominant Location Classifier
Social Voice
Social Voice Video On Demand Transcription
Databases
Big Query
Big Query Writer
Big Query Fixed Schema Writer
Snowflake
Snowflake Setup Guide
Databricks
Databricks File Egress
Databricks SQL Egress
ElasticSearch
Elasticsearch Egress
ETL Platforms
Fivetran
Fivetran Setup Guide
PRICING & BILLING
How Data Streams are Priced
Direct Integrations Pricing
What are Data Volume Units?
Committed Usage Discounts (Commits)
Billing Management
Billing Dashboard
Detailed Billing Views (Using Tags)
Pricing Calculator
Budget Alerts
Tips: Optimizing Your Datastreamer Usage
Provider Bundles for Direct Integrations
What are Bundles?
NLP Classifiers (Datastreamer-provided)
AI Classifiers (Datastreamer-provided)
Location Inference Bundle (Datastreamer-provided)
Bright Data Specialty Sources
Bright Data High Result Source Bundle
Pipeline Management, Security, APIs
What is a Dynamic Pipeline?
Pipeline Deployment
Managing Pipelines
Pipeline Versioning
Pausing, Stopping, and Deleting
Pipeline Import and Export
Tracking & Monitoring
Pipeline Document Inspector Component
Volume Health Monitoring & Alerting
Pipeline Metrics & Analytics
Component Log Viewer
Failed Items Viewer
Quick Diagnostics
Creating Jobs
Creating Jobs by Importing a Jobs CSV File
Security & Compliance
Pipeline Regional Deployment
Platform Transparency Overview
Platform Security Overview
API Key Regeneration
Compliance-Sensitive Usage
Other
Quick Start Pipeline Templates
Sentiment, Location, and Language Enriched Pipeline Template
Multi-source Entity Recognition Pipeline Template
Tips & Tricks
How To Enrich Your Own Data (Bring Your Own Data)
Blacklist Filtering
Estimating external data volumes
Google Sheet Integration
📘
Executive "Launch" Guide
MCP Server
MCP
Powered by
Violence Detection
Copy Page
