Metadata Fields

Datastreamer's pipeline contains a high-powered transformation service that can convert billions of individual fields into a common schema per minute. This unlocks the ability to easily generate features and machine learning capabilities that use multiple sources concurrently; as well easy integration of new data partners and pipeline capabilities.

View "Integrated Data Sources" and "Integrated Enrichments" to see which metadata categories apply to individual sources.

Datasource Response Bodies

Each data source has its own specific response parameters based on the source from which information is being fetched. Each data source response is merged with common information like Source, and Enrichment details.

Source

Field	Description
source.link	Common for all sources, this is the URL link directly to the content and can serve as a UID for the content.
source.kind	For sources with subtypes (such as dark web content), this covers the subtypes.

Content

Field	Description
content.body	Full-text content
content.found	When the content was discovered
content.published	When the content was originally published
content.last_updated	When the content was last updated
content.hashtags	Any hashtags within the content
content.links	Any links found within the content
content.favorites	A count of social media favorites of the content at the time of index
content.followers	A count of social media followers of the content at the time of index
content.following	A count of the social media following count of the content at the time of index
content.mentions	Profiles that are being mentioned in content or directed to with said pieces of content
content.image_urls	Contains all URLs to images that were present within or associated to the content.
content.title	The header (title) of the content.
content.summary	A summarization or excerpt of the content.
content.domain	The associated domain of the content record.
content.cashtags	Any cashtags within the content
content.comments_count	A count of number of comments on the social media post
content.location	Location mentioned with social media post
content.video_urls	Video url links within social media post [*]
content.shares_count	A count of number of times social media post was shared
content.plays_count	A count of number of times social media post was played
content.views_count	A count of number of times social media post was viewed
content.likes_count	A count of number of times social media post was liked
content.images	Contains images information associated with the content

images

Field	Description
url	URL to image found in the content
alternative_text	Alternative text or accessibility caption associated with image found in the content

Author

Field	Description
author.name	Name of the author
author.handle	Handle of the user
author.bio	A personal description of the author
author.location	Location of the author
author.profile_image_source	Link to the profile image of the author or profile.
author.gender	Gender of the author
author.url	URL to the author's profile
author.bio_links	URLs mentioned in the author profile
author.userid	User id mentioned in author profile
author.verified	Is author profile verified ? Boolean True or False
author.videos_count	A count of number of times author posted videos
author.phone	Author phone number in the author profile

Person

Field	Description
person.name	Full name of the individual (if available).
person.handle	Profile handle
person.image	Avatar of image associated with the person.
person.aliases	Any aliases
person.alternative_names	Alternative or maiden names of the individual
person.addresses	Address associated with the record.
person.country	Country of person
person.region	Region of person
person.place_of_birth	Place or location of birth.
person.education	Education history
person.work_current	Current employment with associated title (if available), at time of document date.
person.work_history	Employment history
person.volunteer_experience	Volunteer experience of person.
person.bio	Available biography of person.
person.certifications	Any certifications received.
person.languages	Any languages spoken.
person.verified	If the profile has been verified as accurate by partner.
person.profile_create_date	Creation date of the profile.
person.dob	Date of birth, of the person.
person.dod	Date of death, of the person.
person.convicted	Convictions (if available) of the person.
person.family_relations	Any family relations present in the result.
person.business_relations	Any business relations present in the result.
person.business_partnerships	Any business partnerships present in the result.
person.contributions	Contributions and awards of the person.
person.lobbying	Any lobbying partnerships or relationships.
person.stakeholder	Any stakeholder partnerships or relationships.
person.contributions	Any contributions by the person
person.certifications	Any certifications obtained by person
person.followers	Number of followers for the person
person.following	Number of following for the person
person.headline	Headline or title of the person
person.honors	Honors mentioned for the person
person.userid	user id for the person in the metadata
person.hashtags	Hashtags related to the person
person.maiden_name	Prior first or last name of the person
person.organizations	Person working/affiliated with the organizations
person.patents	Patents obtained by the person
person.projects	Projects worked on by the person
person.publications	Publications by the persons
person.bio	Bio summary of a person
person.test_scores	Exam test scores of a person
person.skills	Technical skills of a person
person.courses	Courses created or completed by person
person.url	Person's website url

Offenses

Field	Description
offenses.case_number	Case number of the offense. Ex: "095978201010"
offenses.case_type	Case type of the offense as defined by the source. Ex: "felony", "gross misdemeanor", "Offense Infraction"
offenses.date	Date the offense occurred. Ex: "2003-07-28"
offenses.class	Class of the offense
offenses.subclass	Subclass of the offense
offenses.description	Offense description from the source. Ex: "SEXUAL ASSAULT CHILD", "BATTERy', "AGGRAV STALKING"
offenses.degree	Degree of the offense from the source. Ex: "FELONY", "Misdemeanor", "Infraction"
offenses.level	Feld classifying offense level.Ex: "FELONY", "MISDEMEANOR", "UNKNOWN", "INFRACTION/VIOLATION/ORDIANCE/TRAFFIC"
offenses.sentence_date	Date the defendant was sentenced. Ex: "2004-06-03"
offenses.sentence_type	Type of sentence as defined by the source. Ex: "fined", "failed", "probation"
offenses.sentence_description	Description of the sentence.
offenses.incarceration_date	Date the defendant was incarcerated. Ex: "2007-06-03"
offenses.arrest_date	Date the defendant was arrested. Ex: "2003-07-28"
offenses.arresting_agency	Arresting agency. ExL UNITED STATES MARSHALL'S SERVICE", "Elwood Police Department", "Madison County Sheriff"
offenses.court_case_number	Case number of the offense. Example: "MCRDCRTR03-0001124-008"
offenses.court_name	Name of the court that tried the defendant. Ex: "CA Shasta Superior Court", "GA Dept of Corrections (Web)"
offenses.release_date	Release date of the offender. Ex: "2019-03-26"
offenses.record_date	Feld to calculate the age of the record. Ex: "2003-02-19"
offenses.case_filing_date	Date the case was filed. Ex: "2003-02-19"
offenses.disposition	Disposition of the offense. Ex: "DISM OTHER", "DEFERRED ADJUDICATION TERMINATED", "Adjudicated Guilty", "NOLLE PROSEQUI", "DISMISSED CASEREFILED"
offenses.disposition_type	Internal field classifying the disposition. Ex: "GUILTY", "NOT GUILTY", "WITHHELD", "NOT DISPOSED", "NO VERDICT", "UNKNOWN"
offenses.disposition_date	Date the offense was disposed. Ex: "2013-03-28"
offenses.county	County associated with the offense. Ex: "Benton", "JACKSON", "PORTER"
offenses.conviction_jurisdiction	Jurisdiction associated with the offense City, State, or County. Ex: "Woodbury", "Black Hawk", "Hudson", "Jersey City Municipal Court"
offenses.conviction_date	Date of conviction. Ex: "2004-06-03"

Product

Field	Description
product.title	The title of the product in the product record
product.brand	The brand name of the product (ie: Nike)
product.description	The description on the product record
product.listed_date	The date the product was listed
product.price	The listed price
product.currency	The currency of the price, using ISO standard (ie: USD)
product.seller_name	The displayed name of the product seller
product.seller_url	The URL to the profile or site of the seller
product.location	The location of the product
product.image_urls	Links to images of the product from the product record

Organization

Field	Description
organization.acquired_by	List of organizaitions that was acquired by the Company through a merger, consolidation, combination, exchange of shares, acquisition
organization.affiliated_companies	List of affiliated company's which is a minority shareholder of another company
organization.hashtags	Hashtags used for company
organization.company_type	Following company types : Public company, self-employed, government agency, non-profit
organization.bio	Description of the organization
organization.headquarter	Location of organization headquarters
organization.followers	Number of followers for the organization
organization.founded_date	Founded date for the organization
organization.userid	user id of the organization within metadata
organization.locations	Location of the organization within metadata
organization.name	Name of the organization
organization.parent_company	Parent company of the organization
organization.image	Profile image of the organization
organization.specialities	Specialities of the organization
organization.tagline	Organization headline or tagline
organization.handle	Handle of the organization
organization.url	Organization website url
organization.employees_count_range	A count of range of number of employees of the organization

Enrichment

Field	Description
enrichment.sentiment	The extracted sentiment of the source content
enrichment.language	Per ISO: https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes#Partial_ISO_639_table
enrichment.reported_violence.label	Display label representing the violence: 'No violence', 'Threat of physical violence', 'Reported violence', 'Sexual violence'
enrichment.reported_violence.confidence	Displays a score of reported violence (both directed and indirected) arising from our in-house violence classifier. Can be filtered by confidence (.confidence) of 0.0 to 1.0
enrichment.location_inference.label	Displays the infered city location of the top 31 cities in the United states. Currently only available on English wsl_Twitter
enrichment.location_inference.confidence	Displays a score of location inference confidence. Can be filtered by confidence (.confidence) of 0.0 to 1.0
enrichment.location_entities	Displays NLP-extracted locations from within the content.body of the media.
enrichment.location_entities.confidence	Displays a score of Location entity recognition confidence. Can be filtered by confidence (.confidence) of 0.0 to 1.0
enrichment.person_entities	Displays NLP-extracted persons from within the content.body of the media.
enrichment.person_entities.confidence	Displays a score of Person entity recognition confidence. Can be filtered by confidence (.confidence) of 0.0 to 1.0
enrichment.organization_entities	Displays NLP-extracted organizations from within the content.body of the media.
enrichment.organization_entities.confidence	Displays a score of Organization entity recognition confidence. Can be filtered by confidence (.confidence) of 0.0 to 1.0
enrichment.hard_news.label	Displays label of whether content has been determined as 'Hard News'.
enrichment.hard_news.confidence	Displays a score of Hard News classification confidence. Can be filtered by confidence (.confidence) of 0.0 to 1.0

For more information on Enrichments, view the dedicated page linked in the sidebar.

Data source-specific fields

🚧
Data source-specific fields
The below sections contain fields that are not common to every data source, and apply only to specific data sources.

Twitter-specific fields

Field	Description
twitter.tweet_type	The type of content (POST, REPLY or RETWEET)
twitter.retweet_type	Retweet type (NONE, RAW, or QUOTE)
twittter.retweet_link	Link of the original tweet post
twitter.post_identifier	The identification string of that piece of content
twitter.user_verified	If the user account has been verified
twitter.user_id	Unique Twitter ID of the user on Twitter (as usernames can be changed)
twitter.quote_count	A count of number of times Twitter post was tweeted with quote
twitter.reply_count	A count of number of times Twitter post was give reply
twitter.retweet_count	A count of number of times Twitter post was retweeted
twitter.reply_link	Link of the replied post or comment

Instagram-specific fields

Field	Description
instagram.content_type	The type of content (comment, post)

Threads-specific fields

Field	Description
threads.content_type	The type of content (TEXT, IMAGE, CAROUSEL, or VIDEO)
threads.post_type	The type of post (POST, QUOTE, REPOST, or REPLY)
threads.source_link	The original post URL if post_type is REPOST, QUOTE or REPLY
threads.post_identifier	The identification string of that piece of content
threads.user_verified	If the user account has been verified
threads.user_id	Unique Meta ID of the user on Threads (as usernames can be changed)
threads.reply_count	Number of replies at the moment this post was found.

Tiktok-specific fields

Field	Description
tiktok.is_ad	True/False if the post is the ad
tiktok.is_official_item	True/False if the post is official
tiktok.is_original_item	True/False if the post is original
tiktok.is_share_enabled	True/False if the post is shared enable
tiktok.is_stitch_enabled	True/False if the post is stitch option enable
tiktok.is_activity_item	True/False if the post is type of activity
tiktok.is_duet_enabled	True/False if the post is duet option enable.
tiktok.is_for_friend	True/False if the post is for friend
tiktok.stickers	List of stickers in Tiktok social media post
tiktok.duet	List of duet video tiktok social media post
tiktok.music	List of music details showing title, album name, url, duration etc.

Facebook-specific fields

Field	Description
facebook.post_type	Type of Facebook post
facebook.reactions_angry_count	A count of number of angry emoji reactions on Facebook post
facebook.reactions_haha_count	A count of number of haha emoji reactions on Facebook post
facebook.reactions_like_count	A count of number of likes emoji reactions on Facebook post
facebook.reactions_love_count	A count of number of love emoji reactions on Facebook post
facebook.reactions_sad_count	A count of number of sad emoji reactions on Facebook post
facebook.reactions_support_count	A count of number of support emoji reactions on Facebook post
facebook.reactions_wow_count	A count of number of wow emoji reaction on Facebook post
facebook.reactions_total_count	A count of number of reactions on Facebook post
facebook.members_count	A count of number of members on Facebook page

LinkedIn-specific fields

Field	Description
linkedin.industries	List of industries mentioned on LinkedIn company and member profiles. This allow member/company to choose an industry that best matches their interest in company or the type of work that they do
linkedin.interests_companies	The Interests section displays Companies, Schools, Newsletters, and Groups that a LinkedIn member is following, subscribed to or joined, as applicable
linkedin.interests_top_voices	The Interests top voices section displays Top Voices followed, subscribed to or joined, as applicable by LinkedIn member
linkedin.is_creator	Boolean True/False field for creator mode profile setting that can help creator member to grow your reach and influence on LinkedIn.
linkedin.is_influencer	Boolean True/False field showing if a member is a LinkedIn influencer who is a thought leader within a particular industry who shares organic content to a large LinkedIn audience
linkedin.is_premium	Boolean True/False field showing a company or member is premium subscriber
linkedin.interest_count	A count of number of interest subscribed by member
linkedin.praise_count	A count of number of times kudos on the post
linkedin.emapthy_count	A count of number of times empathy or support emoji on the post
linkedin.carousel	List of multiple images and/or videos in a single post
linkedin.employees_on_linkedin_count	A count of number of employees of organization on LinkedIn
linkedin.open_to	Boolean True/False field shows open to new job opportunities on member profile
linkedin.connections_count	A count of number of member connections

News-specific fields

Field	Description
news.global_rank	Rank of the news site in the world
news.country_rank	Rank of the news site in the country
news.word_count	Length of the article, as determined by wordcount

Darknet-specific fields

Field	Description
enrichment.hackishness	A DarkOwl-unique field, measuring the Hackishness of any piece of content.
enrichment.ccn_entities.label	Used to search for a desired credit card number in the body of a result. Input is 6-20 digits and supports trailing wildcards.
enrichment.ssn_entities.label	Used to search for an individual SSN record.
enrichment.email_entities.label	Used to search for an individual email address. Note: search the keyword of "thedomain.com" to find email addresses within an entire domain.
enrichment.cryptocurrency_entities.label	Used to search for a cryptocurrency address in the body of a result. Supported types: bitcoin, dash, ethereum, litecoin, monero, zcash.
enrichment.ipaddress_entities.label	Used to search for an ip address in the body of a result.

Forum-specific fields

Field	Description
forum.thread_title	The title of the thread for the content
forum.thread_start	True/False on if the post is the first post (head) of a forum thread.

Others

Field	Description
doc_date	The date of the original creation of content. While this parallels content.published in many cases, some sources that do not provide content.published. This date is always the date of document creation at its source.