# Datastreamer API Documentation

> Datastreamer's API allows you to quickly building products using text conversation data.

Append .md to any documentation page URL to get its markdown version.

## Guides
- [👋 Welcome to Datastreamer!](https://docs.datastreamer.io/docs/welcome-to-datastreamer.md)
- [Platform Overview](https://docs.datastreamer.io/docs/platform-overview.md): A Data Stream is the core unit of work in Datastreamer
- [Creating Your Account](https://docs.datastreamer.io/docs/getting-started.md)
- [Platform Glossary](https://docs.datastreamer.io/docs/glossary.md): A reference for terms used in this documentation and across the Datastreamer platform.
- [Architecture Overview](https://docs.datastreamer.io/docs/datastreamer-architecture-overview.md): This overview describes the high-level overview of Datastreamer's architecture
- [Platform FAQs](https://docs.datastreamer.io/docs/platform-faqs.md): Answers to common questions about Datastreamer.
- [Data Streams Overview](https://docs.datastreamer.io/docs/data-streams-overview.md): This section covers everything involved in building and running a Data Stream: sources, transformation, enrichment, and destinations.
- [Sources Overview](https://docs.datastreamer.io/docs/auto-source-selection-faq.md): Sources are the data inputs for a Data Stream. When you configure a source in a Data Stream, Datastreamer handles provider selection and data retrieval automatically.
- [Data Stream Sources Capability Table](https://docs.datastreamer.io/docs/auto-capability-table.md)
- [Auto Profiles](https://docs.datastreamer.io/docs/auto-profiles.md): The Auto Profiles source collects content using Auto source selection, routing each Job to the best available provider.
- [Twitter/X](https://docs.datastreamer.io/docs/twitter-auto.md): The Twitter/X source collects posts, replies, and public account content from Twitter/X as part of a Data Stream.
- [Threads](https://docs.datastreamer.io/docs/threads-auto.md): The Threads source collects posts and replies from Threads (Meta) as part of a Data Stream.
- [Facebook](https://docs.datastreamer.io/docs/facebook-auto.md): The Facebook source collects posts, comments, and public page content from Facebook as part of a Data Stream.
- [YouTube](https://docs.datastreamer.io/docs/youtube-auto.md): The YouTube source collects videos and comments from YouTube as part of a Data Stream.
- [Reddit](https://docs.datastreamer.io/docs/reddit-auto.md): The Reddit source collects posts and comments from Reddit as part of a Data Stream.
- [Instagram](https://docs.datastreamer.io/docs/instagram-auto.md): The Instagram source collects posts, reels, and public profile content from Instagram as part of a Data Stream.
- [TikTok](https://docs.datastreamer.io/docs/tiktok-auto.md): The TikTok source collects videos and public account content from TikTok as part of a Data Stream.
- [Bluesky](https://docs.datastreamer.io/docs/copy-of-instagram.md): The Bluesky source collects posts, reels, and public profile content as part of a Data Stream.
- [Unify Transformer](https://docs.datastreamer.io/docs/metadata-fields.md): A Datastreamer-designed schema to use as a basis or as your platform schema.
- [Routing & Filtering](https://docs.datastreamer.io/docs/routing-filtering.md)
- [Concat](https://docs.datastreamer.io/docs/concat.md): Combines the text from multiple fields into a single field.
- [JSON Router](https://docs.datastreamer.io/docs/json-document-router.md)
- [Splitters](https://docs.datastreamer.io/docs/splitters.md)
- [Lucene Document Filter](https://docs.datastreamer.io/docs/lucene-document-filter.md)
- [Document Batcher](https://docs.datastreamer.io/docs/document-batcher.md)
- [Document Deduplication](https://docs.datastreamer.io/docs/document-deduplication.md)
- [JSON Schema Transformer](https://docs.datastreamer.io/docs/json-transform.md)
- [Transformation Schema](https://docs.datastreamer.io/docs/transformation-schema.md): The transformation schema defines how to transform a source JSON document into a destination format. It consists of a list of mapping properties, each defining a specific transformation rule.
- [Transformation Operations](https://docs.datastreamer.io/docs/transformation-operations.md)
- [JSON Conditions](https://docs.datastreamer.io/docs/json-conditions.md)
- [Searchable Storage](https://docs.datastreamer.io/docs/searchable-storage-egress.md): Datastreamer-managed and in-pipeline searchable database solution.
- [Adding & Using Searchable Storage](https://docs.datastreamer.io/docs/adding-using-searchable-storage.md)
- [Managing your Searchable Storage](https://docs.datastreamer.io/docs/managing-your-searchable-storage.md)
- [Searchable Storage APIs](https://docs.datastreamer.io/docs/searchable-storage-apis.md)
- [Search API](https://docs.datastreamer.io/docs/search.md): An Overview of what Search API offers within Datastreamer
- [Performing a Query](https://docs.datastreamer.io/docs/query.md): An overview of Apache Lucene implementation within Datastreamer API
- [Sample Search Request & Response](https://docs.datastreamer.io/docs/sample-request.md)
- [API Pagination](https://docs.datastreamer.io/docs/api-pagination.md): Pagination is only relevant to the Search API
- [Aggregations with Search API](https://docs.datastreamer.io/docs/aggregations_with_search_api.md): Allows users to perform high-level analysis, providing different types of aggregations like Terms, Significant Terms, and Date Histogram to detect trends and generate insights
- [Count API](https://docs.datastreamer.io/docs/count-api.md): An Overview of what the Count API offers within Datastreamer
- [Firehose](https://docs.datastreamer.io/docs/firehose-egress-connector.md): Pipeline Firehose Egress
- [Webhook](https://docs.datastreamer.io/docs/webhook-egress.md): Sends pipeline data to an external URL via customizable HTTP requests
- [Datastreamer Searchable Storage Ingress](https://docs.datastreamer.io/docs/searchable-storage-ingress.md): Reads data from existing searchable storages using Lucene queries
- [Direct Data Upload](https://docs.datastreamer.io/docs/https-endpoint-ingress.md): Direct Data Upload allows you send documents through the UI or via a HTTPS endpoint.
- [SFTP](https://docs.datastreamer.io/docs/sftp.md)
- [SFTP Storage Egress](https://docs.datastreamer.io/docs/sftp-storage-egress.md)
- [Google Pub/Sub Ingress](https://docs.datastreamer.io/docs/google-pubsub-ingress.md)
- [Google Pub/Sub Egress Connector](https://docs.datastreamer.io/docs/google-pubsub-egress-connector.md): Pipeline Egress to Google Pub/Sub
- [Datastreamer File Storage Egress](https://docs.datastreamer.io/docs/datastreamer-file-storage-egress.md): The Datastreamer File Storage Egress component allows you to create files in Datastreamer Storage from your Pipeline documents, with options to customize file creation
- [Datastreamer File Storage Ingress](https://docs.datastreamer.io/docs/datastreamer-file-storage-ingress.md): Read files from Datastreamer File Storage and process them in your pipeline.
- [Moving to Data Streams](https://docs.datastreamer.io/docs/moving-to-data-streams.md): What changes when you move sourcing onto Data Streams, what stays the same, and how to switch.
- [Connector Automation (Jobs)](https://docs.datastreamer.io/docs/connector-automation.md)
- [Creating Jobs via Jobs API](https://docs.datastreamer.io/docs/creating-jobs-portal-api.md): Learn how to create Jobs for data collection via Jobs API
- [Managing Jobs](https://docs.datastreamer.io/docs/view-jobs.md)
- [Stopping a Job (Portal)](https://docs.datastreamer.io/docs/stopping-a-job.md)
- [Listing Jobs via Jobs API](https://docs.datastreamer.io/docs/listing-jobs.md): Instructions on using Jobs API to list previously created jobs
- [Get Job Details via Jobs API](https://docs.datastreamer.io/docs/get-job-details.md): Instructions on getting a previously created Job details
- [Cancelling Jobs via Jobs API](https://docs.datastreamer.io/docs/cancelling-jobs-via-jobs-api.md): Details on how to cancel a job using the Jobs REST API
- [Deleting Jobs via Jobs API](https://docs.datastreamer.io/docs/deleting-jobs-via-jobs-api.md): Details on how to delete a job using the Jobs REST API
- [Updating Jobs (Portal, API)](https://docs.datastreamer.io/docs/creating-jobs-portal-api-copy.md): Learn how to edit scheduled Jobs for data collection.
- [Advanced Job Search](https://docs.datastreamer.io/docs/searching-jobs.md)
- [Job Failure Handling](https://docs.datastreamer.io/docs/job-failure-handling.md): This page describes how to handle failures where the root cause is transient such as service outages etc.
- [Best Practices for Data Collection Jobs](https://docs.datastreamer.io/docs/best-practices-for-data-collection-jobs.md): This guide outlines best practices to help you get the most from your Job, manage the costs, and maintain multi-source data.
- [Jobs DVU Count API](https://docs.datastreamer.io/docs/jobs-dvu-count-api.md): Instructions of how to use Jobs DVU Count API.
- [MCP Server Setup Guide](https://docs.datastreamer.io/docs/mcp-server-setup-guide.md): A guide on how to connect the Datastreamer MCP server, and access the Agent tools available.
- [Job Creation Agent](https://docs.datastreamer.io/docs/job-creation-agent.md)
- [Direct Integrations Overview](https://docs.datastreamer.io/docs/direct-integrations.md): Direct Integrations give you access to individual data providers, cloud storage systems, and other external services through dedicated connectors.
- [Data Sources](https://docs.datastreamer.io/docs/ingress-connectors.md)
- [Apify](https://docs.datastreamer.io/docs/apify-actor-integration.md): Apify helps you build reliable web scrapers, fast, and automate anything you can do in a web browser.
- [Apify Setup Guide](https://docs.datastreamer.io/docs/apify.md): How to integrate Apify actors to your Datastreamer pipelines.
- [Troubleshooting & FAQ](https://docs.datastreamer.io/docs/apify-troubleshooting-and-faq.md): Common questions and troubleshooting for integrating Apify to Datastreamer
- [Brightdata](https://docs.datastreamer.io/docs/brightdata.md): How to connect a Brightdata account to the Datastreamer Platform.
- [Brightdata Account Setup](https://docs.datastreamer.io/docs/brightdata-account-setup.md): How to connect a Brightdata account to the Datastreamer Platform.
- [Brightdata Amazon Products](https://docs.datastreamer.io/docs/brightdata-ingress-amazon-products.md): Ingest Brightdata Amazon Products content into Datastreamer Pipeline
- [Brightdata CNN News](https://docs.datastreamer.io/docs/brightdata-ingress-cnn-news.md): Ingest Brightdata CNN News content into Datastreamer Pipeline
- [Brightdata Crunchbase Business](https://docs.datastreamer.io/docs/brightdata-ingress-crunchbase-business.md): Ingesting Brightdata Crunchbase into your Datastreamer Pipeline
- [Brightdata Ebay Products](https://docs.datastreamer.io/docs/brightdata-ingress-ebay-products.md): Ingest Brightdata Ebay Products content into Datastreamer Pipeline
- [Brightdata Etsy Products](https://docs.datastreamer.io/docs/brightdata-ingress-etsy-products.md): Ingest Brightdata Etsy Products content into Datastreamer Pipeline
- [Brightdata G2 Reviews](https://docs.datastreamer.io/docs/brightdata-ingress-g2-reviews.md): Ingest Brightdata G2 Reviews content into Datastreamer Pipeline
- [Brightdata Github Code](https://docs.datastreamer.io/docs/brightdata-ingress-github-code.md): Ingest Brightdata Github Code content into Datastreamer Pipeline
- [Brightdata Glassdoor Jobs](https://docs.datastreamer.io/docs/brightdata-ingress-glassdoor-jobs.md): Ingest Brightdata Glassdoor Jobs content into Datastreamer Pipeline
- [Brightdata Google Shopping](https://docs.datastreamer.io/docs/brightdata-ingress-google-shopping.md): Ingest Brightdata Google Shopping content into Datastreamer Pipeline
- [Brightdata Indeed Jobs](https://docs.datastreamer.io/docs/brightdata-ingress-indeed-jobs.md): Ingest Brightdata Indeed Jobs content into Datastreamer Pipeline
- [Brightdata Instagram Posts](https://docs.datastreamer.io/docs/brightdata-ingress-instagram-posts.md): Ingest Brightdata Instagram Posts content into Datastreamer Pipeline
- [Brightdata Pinterest Posts](https://docs.datastreamer.io/docs/brightdata-ingress-pinterest-posts.md): Ingest Brightdata Pinterest Posts content into Datastreamer Pipeline
- [Brightdata Reddit Posts](https://docs.datastreamer.io/docs/brightdata-ingress-reddit-posts.md): Ingest Brightdata Reddit Posts content into Datastreamer Pipeline
- [Brightdata Shein Products](https://docs.datastreamer.io/docs/brightdata-ingress-shein-products.md): Ingest Brightdata Shein Products content into Datastreamer Pipeline
- [Brightdata Target Products](https://docs.datastreamer.io/docs/brightdata-ingress-target-products.md): Ingest Brightdata Target Products content into Datastreamer Pipeline
- [Brightdata Trustradius Reviews](https://docs.datastreamer.io/docs/brightdata-ingress-trustradius-reviews.md): Ingest Brightdata Trustradius Reviews content into Datastreamer Pipeline
- [Brightdata Walmart Products](https://docs.datastreamer.io/docs/brightdata-ingress-walmart-products.md): Ingest Brightdata Walmart Products content into Datastreamer Pipeline
- [Brightdata Yahoo Finance Business](https://docs.datastreamer.io/docs/brightdata-ingress-yahoo-finance-business.md): Ingesting Brightdata Yahoo Finance Business into your Datastreamer Pipeline
- [Brightdata Youtube Posts](https://docs.datastreamer.io/docs/brightdata-ingress-youtube-posts.md): Ingest Brightdata Youtube Posts content into Datastreamer Pipeline
- [Bluesky Live Feed](https://docs.datastreamer.io/docs/bluesky.md): A live firehose feed of posts sourced directly from Bluesky's official stream.
- [DarkOwl Search](https://docs.datastreamer.io/docs/darkowl_search.md): DarkOwl Search allows you to query DarkOwl Vision’s DARKINT data collection
- [Datashake Social Media](https://docs.datastreamer.io/docs/datashake.md): Search and scrape social media content from Reddit, X (Twitter), LinkedIn, Instagram, Facebook, YouTube, and Bluesky using keyword searches, content URLs, or profile scraping
- [Datashake Bluesky](https://docs.datastreamer.io/docs/datashake_bluesky.md): Search and scrape Bluesky posts and profiles using keyword searches or content URLs
- [Datashake Facebook](https://docs.datastreamer.io/docs/datashake_facebook.md): Search and scrape Facebook posts and pages using keyword searches or content URLs
- [Datashake Instagram](https://docs.datastreamer.io/docs/datashake_instagram.md): Search and scrape Instagram posts and profiles using keyword searches or content URLs
- [Datashake LinkedIn](https://docs.datastreamer.io/docs/datashake_linkedin.md): Search and scrape LinkedIn posts and profiles using keyword searches or content URLs
- [Datashake Reddit](https://docs.datastreamer.io/docs/datashake_reddit.md): Search and scrape Reddit posts, comments, and profiles using keyword searches or content URLs
- [Datashake X (Twitter)](https://docs.datastreamer.io/docs/datashake_x.md): Search and scrape X (Twitter) posts, tweets, and profiles using keyword searches or content URLs
- [Datashake YouTube](https://docs.datastreamer.io/docs/datashake_youtube.md): Search and scrape YouTube videos and channels using keyword searches or content URLs
- [Exorde](https://docs.datastreamer.io/docs/exorde.md): Ingest social content collected by Exorde across X (Twitter), Threads, Reddit, TikTok, and YouTube into your Datastreamer pipeline
- [Exorde X](https://docs.datastreamer.io/docs/exorde_x.md): Ingest X (Twitter) content collected by Exorde into your pipeline
- [Exorde Threads](https://docs.datastreamer.io/docs/exorde_threads.md): Ingest Threads (Meta) content collected by Exorde into your pipeline
- [Exorde Reddit](https://docs.datastreamer.io/docs/exorde_reddit.md): Ingest Reddit content collected by Exorde into your pipeline
- [Exorde TikTok](https://docs.datastreamer.io/docs/exorde_tiktok.md): Ingest TikTok content collected by Exorde into your pipeline
- [Exorde YouTube](https://docs.datastreamer.io/docs/exorde_youtube.md): Ingest YouTube content collected by Exorde into your pipeline
- [Opoint News](https://docs.datastreamer.io/docs/opoint_news_historical.md): Provides a historical view into public international news content
- [Socialgist](https://docs.datastreamer.io/docs/socialgist-1.md): Socialgist is a social media analytics platform that provides businesses with insights into what's being said about them and their competitors online.
- [Socialgist Blogs](https://docs.datastreamer.io/docs/socialgist-blog.md): The Blogs search allows you to search for blog posts and/or comments.
- [Socialgist Blog Links](https://docs.datastreamer.io/docs/socialgist-blog-link.md): The Blog Links Search Web Service allows you to find outbound links that were contained in blog posts.
- [Socialgist Boards](https://docs.datastreamer.io/docs/socialgist-boards.md): The Message Board Search allows you to search for message board posts.
- [Socialgist Boards Compliance](https://docs.datastreamer.io/docs/socialgist-boards-compliance.md): Socialgist Compliance Boards Search allows users to search for message board posts with the results Personally Identifiable Information (PII) redacted.
- [Socialgist News](https://docs.datastreamer.io/docs/socialgist-news.md): The News Search Web Service from Socialgist allows you to search for news posts.
- [Socialgist News Compliance](https://docs.datastreamer.io/docs/socialgist-news-compliance.md): Socialgist Compliance News Search allows you to search for message board posts with the results Personally Identifiable Information (PII) redacted.
- [Socialgist Quora](https://docs.datastreamer.io/docs/socialgist-quora.md): The Quora Search allows you to search for Quora posts.
- [Socialgist Reddit](https://docs.datastreamer.io/docs/socialgist-reddit.md): The Reddit Search allows you to search for Reddit posts.
- [Socialgist Reddit Links](https://docs.datastreamer.io/docs/socialgist-blog-reddit-links.md): Reddit Links Search allows you to find outbound links that were contained in Reddit posts.
- [Socialgist Reviews](https://docs.datastreamer.io/docs/socialgist-reviews.md): Socialgist Reviews Search allows you to search for reviews.
- [Socialgist Tencent Weibo](https://docs.datastreamer.io/docs/socialgist-tencent-weibo.md): The Tencent Weibo Search allows you to search for micro blog messages.
- [Socialgist Tiktok](https://docs.datastreamer.io/docs/socialgist_tiktok.md): The Socialgist Tiktok allows you to search for short videos on Tiktok
- [Socialgist Videos](https://docs.datastreamer.io/docs/socialgist-videos.md): The Video Search allows you to search for videos and/or comments.
- [Socialgist VK](https://docs.datastreamer.io/docs/socialgist-vk.md): The VK Search allows you to search for vk.com posts.
- [Talkwalker Widget](https://docs.datastreamer.io/docs/talkwalker-widget.md): Ingest public Talkwalker widget data into your Datastreamer pipeline
- [Talkwalker Search](https://docs.datastreamer.io/docs/talkwalker-search.md): Ingest social listening data from Talkwalker's Search API into your Datastreamer pipeline
- [WebSightLine Augmented Instagram](https://docs.datastreamer.io/docs/websightline-augmented-instagram.md): The most advanced WebSightLine solution, this component allows you to configure jobs that search directly by user or location
- [WebSightLine Instagram](https://docs.datastreamer.io/docs/wsl_instagram.md): The WebSightLine Instagram allows you to search for Instagram posts and/or comments
- [WebSightLine Threads](https://docs.datastreamer.io/docs/wsl_threads.md): WebSightLine (WSL) Threads is a high-sampling of near-time public Threads content.
- [Private Data Sources ](https://docs.datastreamer.io/docs/private-data-sources.md): Private data sources contain data collections that are visible only to your organization.
- [Cloudflare R2 Storage Ingress](https://docs.datastreamer.io/docs/cloudflare-r2-storage-ingress.md): Pipeline component to read data from an SFTP server.
- [Cloudflare R2 Setup Guide](https://docs.datastreamer.io/docs/cloudflare-r2-ingress-setup-guide.md)
- [Cloudflare R2 Storage Egress](https://docs.datastreamer.io/docs/cloudflare-r2-storage-egress.md): Export pipeline data to Cloudflare R2
- [Azure Blob Storage Ingress](https://docs.datastreamer.io/docs/azure-blob-storage-ingress.md): Stream data into your pipelines from Azure Blob Storage with Datastreamer’s Ingress component.
- [Azure Blob Storage Ingress Setup Guide](https://docs.datastreamer.io/docs/azure-blob-storage-ingress-setup-guide.md): Step-by-step of how to ingest data from Azure Blob Storage into your Datastreamer pipeline.
- [Azure Blob Storage Egress](https://docs.datastreamer.io/docs/azure-blob-storage-egress.md): Use Datastreamer’s Azure Blob Storage Egress to save pipeline data securely at scale.
- [Google Cloud Storage Ingress](https://docs.datastreamer.io/docs/google-cloud-storage-ingress.md): Ingest data into your pipelines directly from Google Cloud Storage buckets!
- [Google Cloud Storage Ingress Setup Guide](https://docs.datastreamer.io/docs/google-cloud-storage-ingress-setup-guide.md): Step-by-step of how to bring data from Google Cloud Storage (GCS) into your Datastreamer pipeline.
- [Google Cloud Storage Egress](https://docs.datastreamer.io/docs/google-cloud-storage-egress.md)
- [Amazon S3](https://docs.datastreamer.io/docs/amazon-s3-storage-ingress.md): Use Amazon S3 with Datastreamer to bring data into a pipeline from an S3 bucket, or to deliver processed data from a pipeline to an S3 bucket.
- [Amazon S3 Ingress Setup Guide](https://docs.datastreamer.io/docs/amazon-s3-ingress-setup-guide.md): Step-by-step of how to ingest data from Amazon S3 into your Datastreamer pipeline.
- [Amazon S3 Storage Egress Connector](https://docs.datastreamer.io/docs/aws-s3-egress-connector.md): Pipeline Egress to Amazon S3 Storage buckets
- [Language Detection (Google Service)](https://docs.datastreamer.io/docs/language-detection-google.md): Detecting language used in any field of given inputs
- [Language Detection (Datastreamer)](https://docs.datastreamer.io/docs/language-detection-datastreamer.md): Detecting language from any field of a given input
- [Named Entity Recognition](https://docs.datastreamer.io/docs/named-entity-recognition.md)
- [Market Interest Categorization Taxonomy](https://docs.datastreamer.io/docs/market-interest-categorization.md): This is an available Categorization taxonomy for the AI Category Classifier
- [IPTC Media Topic Categorization Taxonomy](https://docs.datastreamer.io/docs/category.md)
- [Product Sentiment Classifier](https://docs.datastreamer.io/docs/micro-classifier-product-sentiment.md): Instantly detect brands, assess sentiment, and uncover insights from social media content in a single step!
- [Sentiment Classification (short content)](https://docs.datastreamer.io/docs/sentiment-classification-short-content.md): Sentiment detection for English short-form content
- [Custom Functions](https://docs.datastreamer.io/docs/custom-function.md): Enable document editing in the pipeline with Python using customized functions
- [Content Similarity Clustering](https://docs.datastreamer.io/docs/content-similarity-clustering.md): Cluster (group) similar content together from a query.
- [Violence Detection](https://docs.datastreamer.io/docs/violence-detection.md)
- [Hard News](https://docs.datastreamer.io/docs/hard-news.md)
- [Influence Classification](https://docs.datastreamer.io/docs/influence-classification.md): Influence Classification estimates and predicts influence scores of social media content.
- [Private AI PII Redaction](https://docs.datastreamer.io/docs/private-ai-pii-redaction.md): Private AI PII (Personally Identifiable Information) Redaction component
- [Google Translate](https://docs.datastreamer.io/docs/text-translation-google-translate.md): Google Translate component. Translate any text field to 100+ languages with auto-detect source language.
- [Gemini Text Generation](https://docs.datastreamer.io/docs/gemini-text-generation.md): Generate text and structured AI classifications with your own Gemini API key.
- [Open AI Completion](https://docs.datastreamer.io/docs/open-ai-completion.md): Integrate OpenAI Chat Completion into Datastreamer Pipelines
- [X/Twitter Post Metrics Collector](https://docs.datastreamer.io/docs/xtwitter-post-metrics-collector.md): Fetches metrics for X/Twitter posts and enriches incoming JSON documents with the collected data
- [X/Twitter Retweets Collector](https://docs.datastreamer.io/docs/xtwitter-retweets-collector.md): Fetches list of users that retweeted a post and enriches incoming JSON documents with the collected data
- [PDF Table Extraction to Unified Schema](https://docs.datastreamer.io/docs/pdf-table-extraction-copy.md): Detection, extraction  structuring to a specified schema component of tables from PDF Documents
- [PDF to JSON Text Extraction](https://docs.datastreamer.io/docs/pdf-to-json-text-extraction.md): Transforming PDF Unstructured Data into Structured Insights
- [WebSightLine File Fetcher](https://docs.datastreamer.io/docs/wsl-file-fetcher.md): WebSightLine file fetcher downloads images into client storage automatically from JSON paths.
- [WebSightLine Profile Fetcher](https://docs.datastreamer.io/docs/profile-fetcher.md): Fetch and integrate detailed social media profile data into JSON documents via WebSightLine
- [Location Classification](https://docs.datastreamer.io/docs/location.md)
- [Location Inference Models](https://docs.datastreamer.io/docs/location-inference.md): Location Inference Models work to infer the location of origin of a media post.
- [Dominant Location Classifier](https://docs.datastreamer.io/docs/dominant_location.md)
- [Social Voice](https://docs.datastreamer.io/docs/social-voice.md): AI-powered text analysis components for content moderation, sentiment analysis, entity detection, and content classification.
- [Big Query](https://docs.datastreamer.io/docs/big-query.md): Easily integrate Google BigQuery into your data pipelines for scalable analytics and warehousing.
- [Big Query Writer](https://docs.datastreamer.io/docs/big-query-writer.md): The Big Query Writer allow you to automatically ingests JSON data into tables (auto-schemas or predefined columns), dynamically creating/updating tables into Google BigQuery tables
- [Big Query Fixed Schema Writer](https://docs.datastreamer.io/docs/big-query-fixed-schema-writer.md): The Big Query Fixed Schema Writer ingests structured data into Google BigQuery using explicitly defined column schemas, enforcing strict type validation and  consistent data formats through predefined column definitions
- [Snowflake](https://docs.datastreamer.io/docs/snowflake-egress.md): Import content into Snowflake using Datastreamer Pipeline.
- [Snowflake Setup Guide](https://docs.datastreamer.io/docs/snowflake-setup-guide.md): Step-by step guide of how to import data into Snowflake using Datastreamer pipeline.
- [Databricks](https://docs.datastreamer.io/docs/databricks-egress.md): Import content into Databricks using Datastreamer Pipeline.
- [Databricks File Egress](https://docs.datastreamer.io/docs/databricks-file-egress.md): Step-by step guide of how to export JSON content from your Datastreamer pipeline to Databricks using the Databricks files API.
- [Databricks SQL Egress](https://docs.datastreamer.io/docs/databricks-sql-egress.md): Import JSON documents into your your Databricks table from your Datastreamer pipeline.
- [Elasticsearch Egress](https://docs.datastreamer.io/docs/elasticsearch-egress.md): This component connects your system to an Elasticsearch server, allowing you to send documents to be indexed and stored
- [Fivetran Setup Guide](https://docs.datastreamer.io/docs/fivetran-egress.md): Fivetran Egress Connector
- [How Data Streams are Priced](https://docs.datastreamer.io/docs/platform-pricing-overview.md): An overview of how pricing and costs are measured and displayed for your pipelines.
- [Direct Integrations Pricing](https://docs.datastreamer.io/docs/direct-integrations-pricing.md): This page covers pricing for pipelines built on Direct Integrations.
- [What are Data Volume Units?](https://docs.datastreamer.io/docs/data-volume-units.md): A unifying model for multiple different data consumption metrics.
- [Committed Usage Discounts (Commits)](https://docs.datastreamer.io/docs/committed-usage-discounts-commits.md): A Committed Usage Discount (Commit) is a pre-purchased volume of DVUs for a billing cycle.
- [Billing Dashboard](https://docs.datastreamer.io/docs/billing-dashboard-portal.md): Learn about the billing dashboard on the Datastreamer platform.
- [Detailed Billing Views (Using Tags)](https://docs.datastreamer.io/docs/billing-tags.md): Categorize and filter pipeline usage costs with custom billing tags
- [Pricing Calculator](https://docs.datastreamer.io/docs/pricing-estimator.md): You can use the pricing calculator to assess the cost for future roadmap expansions with ease.
- [Budget Alerts](https://docs.datastreamer.io/docs/budget-alerts.md): Monitor spending by your organization with real-time email alerts when reaching budget thresholds.
- [Tips: Optimizing Your Datastreamer Usage](https://docs.datastreamer.io/docs/understanding-and-optimizing-your-datastreamer-usage.md): This guide outlines the best practices to optimization of costs at various levels of usage.
- [What are Bundles?](https://docs.datastreamer.io/docs/component-bundles.md): Learn how component bundles can give access to many features without requiring multiple commits.
- [NLP Classifiers (Datastreamer-provided)](https://docs.datastreamer.io/docs/nlp-classifiers-datastreamer-provided.md): Component and source bundles are a grouping of capabilities present in pricing, estimation, billing, and access.
- [AI Classifiers (Datastreamer-provided)](https://docs.datastreamer.io/docs/ai-classifiers-datastreamer-provided.md): Component and source bundles are a grouping of capabilities present in pricing, estimation, billing, and access.
- [Location Inference Bundle (Datastreamer-provided)](https://docs.datastreamer.io/docs/location-inference-bundle.md): Component and source bundles are a grouping of capabilities present in pricing, estimation, billing, and access.
- [Bright Data Specialty Sources](https://docs.datastreamer.io/docs/bright-data-specialty-sources.md): Component and source bundles are a grouping of capabilities present in pricing, estimation, billing, and access.
- [Bright Data High Result Source Bundle](https://docs.datastreamer.io/docs/bright-data-high-result-source-bundle.md): Component and source bundles are a grouping of capabilities present in pricing, estimation, billing, and access.
- [What is a Dynamic Pipeline?](https://docs.datastreamer.io/docs/what-is-a-dynamic-pipeline.md): Understand Dynamic Pipelines: Explore Datastreamer's flexible, component-based data processing, covering core concepts, lifecycle, and use cases
- [Pipeline Deployment](https://docs.datastreamer.io/docs/saving-and-deploying-a-dynamic-pipeline.md): Learn how to save, manage, and deploy a Dynamic Pipeline
- [Pipeline Versioning](https://docs.datastreamer.io/docs/pipeline-versioning.md)
- [Pausing, Stopping, and Deleting](https://docs.datastreamer.io/docs/deployment-versioning-copy.md): Learn how to save, manage, and deploy a Dynamic Pipeline
- [Pipeline Import and Export](https://docs.datastreamer.io/docs/pipeline-export-and-import.md): The Import/Export feature lets you share and set up pipelines instantly with a single click
- [Pipeline Document Inspector Component](https://docs.datastreamer.io/docs/inspector.md)
- [Volume Health Monitoring & Alerting](https://docs.datastreamer.io/docs/alerts.md)
- [Pipeline Metrics & Analytics](https://docs.datastreamer.io/docs/pipeline-analytics.md): Understanding and viewing metrics associated with a Pipeline.
- [Component Log Viewer](https://docs.datastreamer.io/docs/view-component-logs.md): Built-in panel that lets you inspect every execution log produced by an individual component in your pipeline
- [Failed Items Viewer](https://docs.datastreamer.io/docs/failed-items-viewer.md): Built-in panel that lets you inspect, retry, or discard failed items from a component in your pipeline
- [Quick Diagnostics](https://docs.datastreamer.io/docs/quick-diagnostics.md): Here are some quick diagnostics that you can do to review your pipeline performance.
- [Creating Jobs](https://docs.datastreamer.io/docs/creating-jobs.md): Learn how to create Jobs for data collection.
- [Creating Jobs by Importing a Jobs CSV File](https://docs.datastreamer.io/docs/creating-jobs-by-importing-a-jobs-csv-file.md): This feature enables users to upload a list of jobs using CSV files.
- [Pipeline Regional Deployment](https://docs.datastreamer.io/docs/regional-deployment.md): Deploy your Pipelines to a supported Region, supporting location-specific compliance requirements.
- [Platform Transparency Overview](https://docs.datastreamer.io/docs/pipeline-platform-transperancy-overview.md): Datastreamer's in-platform transparency and data integrity overview
- [Platform Security Overview](https://docs.datastreamer.io/docs/security-faq.md): Datastreamer's encyption, access, logging, and security measures.
- [API Key Management](https://docs.datastreamer.io/docs/api-key-regeneration.md): Create and manage multiple Datastreamer API keys
- [Organization Management](https://docs.datastreamer.io/docs/organization-management.md)
- [Manage Organization Users](https://docs.datastreamer.io/docs/manage-organization-users.md): Add, remove and change roles of your users
- [Compliance-Sensitive Usage](https://docs.datastreamer.io/docs/usage-in-compliance-sensitive-enviroments.md): Usage of Datastreamer in compliance-sensitive enviroments.
- [Sentiment, Location, and Language Enriched Pipeline Template](https://docs.datastreamer.io/docs/sentiment-location-and-language-analysis.md): Import this pipeline template to enrich a single data source with sentiment, location, and language.
- [Multi-source Entity Recognition Pipeline Template](https://docs.datastreamer.io/docs/multi-source-entity-recognition-template.md): Import this pipeline template to Unify and detect entities in multiple social sources.
- [Tips & Tricks](https://docs.datastreamer.io/docs/helpful-resources.md)
- [How To Enrich Your Own Data (Bring Your Own Data)](https://docs.datastreamer.io/docs/enrich-your-own-data-bring-your-own-data.md)
- [Blacklist Filtering](https://docs.datastreamer.io/docs/blacklist-filtering.md): Remove unwanted web content through blacklist filtering.
- [Estimating external data volumes](https://docs.datastreamer.io/docs/3rd-party-volume-estimations.md): Estimate the content in a 3rd party data source using extrapolation methods.
- [Google Sheet Integration](https://docs.datastreamer.io/docs/google-sheet-integration.md): This is a unofficially supported way to interact with Searchable Storage from within Google Sheets.
- [Executive "Launch" Guide](https://docs.datastreamer.io/docs/executive-launch-guide.md): A starting guide for users who will be administering the business aspects of your data pipelines.
- [MCP](https://docs.datastreamer.io/docs/mcp.md)

## Recipes
- [Custom Function: Adjective Phrase Extraction](https://docs.datastreamer.io/recipes/custom-function-adjective-phrase-extraction.md)
- [Custom Function: Auto Summary](https://docs.datastreamer.io/recipes/custom-function-auto-summary-1.md)
- [Custom Function: Bot and AI Linguistic Detection ](https://docs.datastreamer.io/recipes/custom-function-bot-and-ai-linguistic-detection.md)
- [Custom Function: Character Counting](https://docs.datastreamer.io/recipes/custom-function-character-counting.md)
- [Custom Function: Emotion Detection with Emojis](https://docs.datastreamer.io/recipes/custom-function-emotion-detection-with-emojis.md)
- [Custom Function: Extract Common Hashtags](https://docs.datastreamer.io/recipes/custom-function-extract-common-hashtags.md)
- [Custom Function: Informal and Misspelling Detection](https://docs.datastreamer.io/recipes/custom-function-informal-and-misspelling-detection.md)
- [Custom Function: Keyword Extraction](https://docs.datastreamer.io/recipes/custom-function-keyword-extraction.md)
- [Custom Function: Link Extraction](https://docs.datastreamer.io/recipes/custom-function-link-extraction.md)
- [Custom Function: Passive Voice Detection](https://docs.datastreamer.io/recipes/custom-function-passive-voice-detection.md)
- [Custom Function: PII Extraction](https://docs.datastreamer.io/recipes/custom-function-pii-extraction-this-custom-function-recipe-helps-you-extract-all-urls-from-the-given-text-useful-for-brand-discussion-analysis-and-finding-misleading-links-in-threat-monitoring-tweak-as-you-need.md)
- [Custom Function: Question Extraction](https://docs.datastreamer.io/recipes/custom-function-question-extraction.md)
- [Custom Function: Readability Score](https://docs.datastreamer.io/recipes/custom-function-readability-score.md)
- [Custom Function: Sentiment Analysis](https://docs.datastreamer.io/recipes/custom-function-sentiment-analysis.md)
- [Custom Function: Spam Detection](https://docs.datastreamer.io/recipes/custom-function-spam-detection.md)
- [Custom Function: Subjectivity Score](https://docs.datastreamer.io/recipes/custom-function-subjectivity-score.md)
- [Custom Function: Text Cleaning](https://docs.datastreamer.io/recipes/custom-function-text-cleaning.md)
- [Custom Function: Urgency Detection](https://docs.datastreamer.io/recipes/custom-function-urgency-detection.md)
- [Gemini Prompt: Brand Recognition from Hashtags](https://docs.datastreamer.io/recipes/gemini-brand-recognition-hashtag-prompt.md)
- [Gemini Prompt: Brand Recognition](https://docs.datastreamer.io/recipes/gemini-brand-recognition-prompt.md)
- [Gemini Prompt: Category](https://docs.datastreamer.io/recipes/gemini-category-prompt.md)
- [Gemini Prompt: Emotions](https://docs.datastreamer.io/recipes/gemini-emotions-prompt.md)
- [Gemini Prompt: Entity Recognition](https://docs.datastreamer.io/recipes/gemini-entity-recognition-prompt.md)
- [Gemini Prompt: ESG Classification](https://docs.datastreamer.io/recipes/gemini-esg-prompt.md)
- [Gemini Prompt: Intent](https://docs.datastreamer.io/recipes/gemini-intent-prompt.md)
- [Gemini Prompt: Sentiment](https://docs.datastreamer.io/recipes/gemini-sentiment-prompt.md)
- [Gemini Prompt: Translation](https://docs.datastreamer.io/recipes/gemini-translation-prompt.md)
- [Welcome to Recipes!](https://docs.datastreamer.io/recipes/welcome-to-recipes.md)

## Changelog
- [Improved: Auto Profiles region-based provider selection](https://docs.datastreamer.io/changelog/improved-auto-profiles-region-based-provider-selection.md)
- [New: Multiple API keys and key management](https://docs.datastreamer.io/changelog/new-multiple-api-keys-and-key-management.md)
- [New: User Management](https://docs.datastreamer.io/changelog/new-user-management.md)
- [Deprecated: GenAI classifiers sunset](https://docs.datastreamer.io/changelog/deprecated-genai-classifiers-sunset.md)
- [Improved: Socialgist TikTok](https://docs.datastreamer.io/changelog/improved-socialgist-tiktok.md)
- [Removed: Google Marketplace integration](https://docs.datastreamer.io/changelog/deprecated-google-marketplace-integration.md)
- [New: Data Streams (generally available September 1)](https://docs.datastreamer.io/changelog/new-data-streams-generally-available-september-1.md)
- [Fixed: Auto Profile errors](https://docs.datastreamer.io/changelog/fixed-auto-profile-errors.md)
- [Fixed: Social Voice Transcription](https://docs.datastreamer.io/changelog/fixed-social-voice-transcription.md)
- [Improved: Bulk download of billing tags as CSV](https://docs.datastreamer.io/changelog/improved-bulk-download-of-billing-tags-as-csv.md)