Default Metadata Fields
The primary source of information regarding metadata within API responses.
Datastreamer's pipeline contains a high-powered transformation service that can convert billions of individual fields into a common schema per minute. This unlocks the ability to easily generate features and machine learning capabilities that use multiple sources concurrently; as well easy integration of new data partners and pipeline capabilities.
View "Integrated Data Sources" and "Integrated Enrichments" to see which metadata categories apply to individual sources.
Datasource Response Bodies
Each data source has its own specific response parameters based on the source from which information is being fetched. Each data source response is merged with common information like Source, and Enrichment details.
Source
| Field | Description |
|---|---|
| source.link | Common for all sources, this is the URL link directly to the content and can serve as a UID for the content. |
| source.kind | For sources with subtypes (such as dark web content), this covers the subtypes. |
Content
| Field | Description |
|---|---|
| content.body | Full-text content |
| content.found | When the content was discovered |
| content.published | When the content was originally published |
| content.last_updated | When the content was last updated |
| content.hashtags | Any hashtags within the content |
| content.links | Any links found within the content |
| content.favorites | A count of social media favorites of the content at the time of index |
| content.followers | A count of social media followers of the content at the time of index |
| content.following | A count of the social media following count of the content at the time of index |
| content.mentions | Profiles that are being mentioned in content or directed to with said pieces of content |
| content.image_urls | Contains all URLs to images that were present within or associated to the content. |
| content.title | The header (title) of the content. |
| content.summary | A summarization or excerpt of the content. |
| content.domain | The associated domain of the content record. |
| content.cashtags | Any cashtags within the content |
| content.comments_count | A count of number of comments on the social media post |
| content.location | Location mentioned with social media post, usually provided by the author. |
| content.video_urls | Video url links within social media post [*] |
| content.shares_count | A count of number of times social media post was shared |
| content.plays_count | A count of number of times social media post was played |
| content.views_count | A count of number of times social media post was viewed |
| content.likes_count | A count of number of times social media post was liked |
| content.images | Contains images information associated with the content |
images
| Field | Description |
|---|---|
| url | URL to image found in the content |
| alternative_text | Alternative text or accessibility caption associated with image found in the content |
Author
| Field | Description |
|---|---|
| author.name | Name of the author |
| author.handle | Handle of the user |
| author.bio | A personal description of the author |
| author.location | Location of the author |
| author.profile_image_source | Link to the profile image of the author or profile. |
| author.gender | Gender of the author |
| author.url | URL to the author's profile |
| author.bio_links | URLs mentioned in the author profile |
| author.userid | User id mentioned in author profile |
| author.verified | Is author profile verified ? Boolean True or False |
| author.videos_count | A count of number of times author posted videos |
| author.phone | Author phone number in the author profile |
Person
| Field | Description |
|---|---|
| person.name | Full name of the individual (if available). |
| person.handle | Profile handle |
| person.image | Avatar of image associated with the person. |
| person.aliases | Any aliases |
| person.alternative_names | Alternative or maiden names of the individual |
| person.addresses | Address associated with the record. |
| person.country | Country of person |
| person.region | Region of person |
| person.place_of_birth | Place or location of birth. |
| person.education | Education history |
| person.work_current | Current employment with associated title (if available), at time of document date. |
| person.work_history | Employment history |
| person.volunteer_experience | Volunteer experience of person. |
| person.bio | Available biography of person. |
| person.certifications | Any certifications received. |
| person.languages | Any languages spoken. |
| person.verified | If the profile has been verified as accurate by partner. |
| person.profile_create_date | Creation date of the profile. |
| person.dob | Date of birth, of the person. |
| person.dod | Date of death, of the person. |
| person.convicted | Convictions (if available) of the person. |
| person.family_relations | Any family relations present in the result. |
| person.business_relations | Any business relations present in the result. |
| person.business_partnerships | Any business partnerships present in the result. |
| person.contributions | Contributions and awards of the person. |
| person.lobbying | Any lobbying partnerships or relationships. |
| person.stakeholder | Any stakeholder partnerships or relationships. |
| person.contributions | Any contributions by the person |
| person.certifications | Any certifications obtained by person |
| person.followers | Number of followers for the person |
| person.following | Number of following for the person |
| person.headline | Headline or title of the person |
| person.honors | Honors mentioned for the person |
| person.userid | user id for the person in the metadata |
| person.hashtags | Hashtags related to the person |
| person.maiden_name | Prior first or last name of the person |
| person.organizations | Person working/affiliated with the organizations |
| person.patents | Patents obtained by the person |
| person.projects | Projects worked on by the person |
| person.publications | Publications by the persons |
| person.bio | Bio summary of a person |
| person.test_scores | Exam test scores of a person |
| person.skills | Technical skills of a person |
| person.courses | Courses created or completed by person |
| person.url | Person's website url |
Offenses
| Field | Description |
|---|---|
| offenses.case_number | Case number of the offense. Ex: "095978201010" |
| offenses.case_type | Case type of the offense as defined by the source. Ex: "felony", "gross misdemeanor", "Offense Infraction" |
| offenses.date | Date the offense occurred. Ex: "2003-07-28" |
| offenses.class | Class of the offense |
| offenses.subclass | Subclass of the offense |
| offenses.description | Offense description from the source. Ex: "SEXUAL ASSAULT CHILD", "BATTERy', "AGGRAV STALKING" |
| offenses.degree | Degree of the offense from the source. Ex: "FELONY", "Misdemeanor", "Infraction" |
| offenses.level | Feld classifying offense level.Ex: "FELONY", "MISDEMEANOR", "UNKNOWN", "INFRACTION/VIOLATION/ORDIANCE/TRAFFIC" |
| offenses.sentence_date | Date the defendant was sentenced. Ex: "2004-06-03" |
| offenses.sentence_type | Type of sentence as defined by the source. Ex: "fined", "failed", "probation" |
| offenses.sentence_description | Description of the sentence. |
| offenses.incarceration_date | Date the defendant was incarcerated. Ex: "2007-06-03" |
| offenses.arrest_date | Date the defendant was arrested. Ex: "2003-07-28" |
| offenses.arresting_agency | Arresting agency. ExL UNITED STATES MARSHALL'S SERVICE", "Elwood Police Department", "Madison County Sheriff" |
| offenses.court_case_number | Case number of the offense. Example: "MCRDCRTR03-0001124-008" |
| offenses.court_name | Name of the court that tried the defendant. Ex: "CA Shasta Superior Court", "GA Dept of Corrections (Web)" |
| offenses.release_date | Release date of the offender. Ex: "2019-03-26" |
| offenses.record_date | Feld to calculate the age of the record. Ex: "2003-02-19" |
| offenses.case_filing_date | Date the case was filed. Ex: "2003-02-19" |
| offenses.disposition | Disposition of the offense. Ex: "DISM OTHER", "DEFERRED ADJUDICATION TERMINATED", "Adjudicated Guilty", "NOLLE PROSEQUI", "DISMISSED CASEREFILED" |
| offenses.disposition_type | Internal field classifying the disposition. Ex: "GUILTY", "NOT GUILTY", "WITHHELD", "NOT DISPOSED", "NO VERDICT", "UNKNOWN" |
| offenses.disposition_date | Date the offense was disposed. Ex: "2013-03-28" |
| offenses.county | County associated with the offense. Ex: "Benton", "JACKSON", "PORTER" |
| offenses.conviction_jurisdiction | Jurisdiction associated with the offense City, State, or County. Ex: "Woodbury", "Black Hawk", "Hudson", "Jersey City Municipal Court" |
| offenses.conviction_date | Date of conviction. Ex: "2004-06-03" |
Product
| Field | Description |
|---|---|
| product.title | The title of the product in the product record |
| product.brand | The brand name of the product (ie: Nike) |
| product.description | The description on the product record |
| product.listed_date | The date the product was listed |
| product.price | The listed price |
| product.currency | The currency of the price, using ISO standard (ie: USD) |
| product.seller_name | The displayed name of the product seller |
| product.seller_url | The URL to the profile or site of the seller |
| product.location | The location of the product |
| product.image_urls | Links to images of the product from the product record |
Organization
| Field | Description |
|---|---|
| organization.acquired_by | List of organizaitions that was acquired by the Company through a merger, consolidation, combination, exchange of shares, acquisition |
| organization.affiliated_companies | List of affiliated company's which is a minority shareholder of another company |
| organization.hashtags | Hashtags used for company |
| organization.company_type | Following company types : Public company, self-employed, government agency, non-profit |
| organization.bio | Description of the organization |
| organization.headquarter | Location of organization headquarters |
| organization.followers | Number of followers for the organization |
| organization.founded_date | Founded date for the organization |
| organization.userid | user id of the organization within metadata |
| organization.locations | Location of the organization within metadata |
| organization.name | Name of the organization |
| organization.parent_company | Parent company of the organization |
| organization.image | Profile image of the organization |
| organization.specialities | Specialities of the organization |
| organization.tagline | Organization headline or tagline |
| organization.handle | Handle of the organization |
| organization.url | Organization website url |
| organization.employees_count_range | A count of range of number of employees of the organization |
Enrichment
| Field | Description |
|---|---|
| enrichment.sentiment | The extracted sentiment of the source content |
| enrichment.language | Per ISO: https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes#Partial_ISO_639_table |
| enrichment.reported_violence.label | Display label representing the violence: 'No violence', 'Threat of physical violence', 'Reported violence', 'Sexual violence' |
| enrichment.reported_violence.confidence | Displays a score of reported violence (both directed and indirected) arising from our in-house violence classifier. Can be filtered by confidence (.confidence) of 0.0 to 1.0 |
| enrichment.location_inference.label | Displays the infered city location of the top 31 cities in the United states. Currently only available on English wsl_Twitter |
| enrichment.location_inference.confidence | Displays a score of location inference confidence. Can be filtered by confidence (.confidence) of 0.0 to 1.0 |
| enrichment.location_entities | Displays NLP-extracted locations from within the content.body of the media. |
| enrichment.location_entities.confidence | Displays a score of Location entity recognition confidence. Can be filtered by confidence (.confidence) of 0.0 to 1.0 |
| enrichment.person_entities | Displays NLP-extracted persons from within the content.body of the media. |
| enrichment.person_entities.confidence | Displays a score of Person entity recognition confidence. Can be filtered by confidence (.confidence) of 0.0 to 1.0 |
| enrichment.organization_entities | Displays NLP-extracted organizations from within the content.body of the media. |
| enrichment.organization_entities.confidence | Displays a score of Organization entity recognition confidence. Can be filtered by confidence (.confidence) of 0.0 to 1.0 |
| enrichment.hard_news.label | Displays label of whether content has been determined as 'Hard News'. |
| enrichment.hard_news.confidence | Displays a score of Hard News classification confidence. Can be filtered by confidence (.confidence) of 0.0 to 1.0 |
For more information on Enrichments, view the dedicated page linked in the sidebar.
Data source-specific fields
Data source-specific fields
The below sections contain fields that are not common to every data source, and apply only to specific data sources.
Twitter-specific fields
| Field | Description |
|---|---|
| twitter.tweet_type | The type of content (POST, REPLY or RETWEET) |
| twitter.retweet_type | Retweet type (NONE, RAW, or QUOTE) |
| twittter.retweet_link | Link of the original tweet post |
| twitter.post_identifier | The identification string of that piece of content |
| twitter.user_verified | If the user account has been verified |
| twitter.user_id | Unique Twitter ID of the user on Twitter (as usernames can be changed) |
| twitter.quote_count | A count of number of times Twitter post was tweeted with quote |
| twitter.reply_count | A count of number of times Twitter post was give reply |
| twitter.retweet_count | A count of number of times Twitter post was retweeted |
| twitter.reply_link | Link of the replied post or comment |
Instagram-specific fields
| Field | Description |
|---|---|
| instagram.content_type | The type of content (comment, post) |
| instagram.user_id | Unique Meta ID of the user on Instagram (as usernames can be changed) |
Threads-specific fields
| Field | Description |
|---|---|
| threads.content_type | The type of content (TEXT, IMAGE, CAROUSEL, or VIDEO) |
| threads.post_type | The type of post (POST, QUOTE, REPOST, or REPLY) |
| threads.source_link | The original post URL if post_type is REPOST, QUOTE or REPLY |
| threads.post_identifier | The identification string of that piece of content |
| threads.user_verified | If the user account has been verified |
| threads.user_id | Unique Meta ID of the user on Threads (as usernames can be changed) |
| threads.reply_count | Number of replies at the moment this post was found. |
Tiktok-specific fields
| Field | Description |
|---|---|
| tiktok.is_ad | True/False if the post is the ad |
| tiktok.is_official_item | True/False if the post is official |
| tiktok.is_original_item | True/False if the post is original |
| tiktok.is_share_enabled | True/False if the post is shared enable |
| tiktok.is_stitch_enabled | True/False if the post is stitch option enable |
| tiktok.is_activity_item | True/False if the post is type of activity |
| tiktok.is_duet_enabled | True/False if the post is duet option enable. |
| tiktok.is_for_friend | True/False if the post is for friend |
| tiktok.stickers | List of stickers in Tiktok social media post |
| tiktok.duet | List of duet video tiktok social media post |
| tiktok.music | List of music details showing title, album name, url, duration etc. |
Facebook-specific fields
| Field | Description |
|---|---|
| facebook.post_type | Type of Facebook post |
| facebook.reactions_angry_count | A count of number of angry emoji reactions on Facebook post |
| facebook.reactions_haha_count | A count of number of haha emoji reactions on Facebook post |
| facebook.reactions_like_count | A count of number of likes emoji reactions on Facebook post |
| facebook.reactions_love_count | A count of number of love emoji reactions on Facebook post |
| facebook.reactions_sad_count | A count of number of sad emoji reactions on Facebook post |
| facebook.reactions_support_count | A count of number of support emoji reactions on Facebook post |
| facebook.reactions_wow_count | A count of number of wow emoji reaction on Facebook post |
| facebook.reactions_total_count | A count of number of reactions on Facebook post |
| facebook.members_count | A count of number of members on Facebook page |
LinkedIn-specific fields
| Field | Description |
|---|---|
| linkedin.industries | List of industries mentioned on LinkedIn company and member profiles. This allow member/company to choose an industry that best matches their interest in company or the type of work that they do |
| linkedin.interests_companies | The Interests section displays Companies, Schools, Newsletters, and Groups that a LinkedIn member is following, subscribed to or joined, as applicable |
| linkedin.interests_top_voices | The Interests top voices section displays Top Voices followed, subscribed to or joined, as applicable by LinkedIn member |
| linkedin.is_creator | Boolean True/False field for creator mode profile setting that can help creator member to grow your reach and influence on LinkedIn. |
| linkedin.is_influencer | Boolean True/False field showing if a member is a LinkedIn influencer who is a thought leader within a particular industry who shares organic content to a large LinkedIn audience |
| linkedin.is_premium | Boolean True/False field showing a company or member is premium subscriber |
| linkedin.interest_count | A count of number of interest subscribed by member |
| linkedin.praise_count | A count of number of times kudos on the post |
| linkedin.emapthy_count | A count of number of times empathy or support emoji on the post |
| linkedin.carousel | List of multiple images and/or videos in a single post |
| linkedin.employees_on_linkedin_count | A count of number of employees of organization on LinkedIn |
| linkedin.open_to | Boolean True/False field shows open to new job opportunities on member profile |
| linkedin.connections_count | A count of number of member connections |
News-specific fields
| Field | Description |
|---|---|
| news.global_rank | Rank of the news site in the world |
| news.country_rank | Rank of the news site in the country |
| news.word_count | Length of the article, as determined by wordcount |
Darknet-specific fields
| Field | Description |
|---|---|
| enrichment.hackishness | A DarkOwl-unique field, measuring the Hackishness of any piece of content. |
| enrichment.ccn_entities.label | Used to search for a desired credit card number in the body of a result. Input is 6-20 digits and supports trailing wildcards. |
| enrichment.ssn_entities.label | Used to search for an individual SSN record. |
| enrichment.email_entities.label | Used to search for an individual email address. Note: search the keyword of "thedomain.com" to find email addresses within an entire domain. |
| enrichment.cryptocurrency_entities.label | Used to search for a cryptocurrency address in the body of a result. Supported types: bitcoin, dash, ethereum, litecoin, monero, zcash. |
| enrichment.ipaddress_entities.label | Used to search for an ip address in the body of a result. |
Forum-specific fields
| Field | Description |
|---|---|
| forum.thread_title | The title of the thread for the content |
| forum.thread_start | True/False on if the post is the first post (head) of a forum thread. |
Others
| Field | Description |
|---|---|
| doc_date | The date of the original creation of content. While this parallels content.published in many cases, some sources that do not provide content.published. This date is always the date of document creation at its source. |
Updated about 2 months ago