Datastreamer Default Schema and Unify

Datastreamer's pipeline contains a high-powered AI-enriched transformation service that can convert billions of individual fields into a common schema per minute. This unlocks the ability to easily generate features and machine learning capabilities that use multiple sources concurrently; as well easy integration of new data sources, and pipeline capabilities.

Using the Datastreamer Default Schema is suggested if you do not have a set schema that you wish to use. This eases adoption, support, and ability to leverage default and examples present in components.

Unify Transformer

Within the Datastreamer Platform, "Unify Transform" converts integrated sources from data partners into the Datastreamer Default Schema. This component also leverages Datastreamer's own schema generation product that reads the available schema of existing content and auto-aligns to the default schema.

Note: Unify can be used with sources not listed in the Datastreamer Default Schema, but the output and structure is not verified for those sources. If you require additional transformation, use the JSON schema transformer component for full customization.

Using Unify within Portal

In the Datastreamer Portal, you can use Unify Transformer component after adding an Ingress component to your pipeline. After adding an Ingress component, you can search for "Unify Transformer" in the Component Picker and select the component from there. All Ingress content will be automatically converted to the unified Datastreamer schema.

Does Unify map every source automatically?

Unify Transform auto-aligns recognized source schemas to the Datastreamer Default Schema. Sources integrated through the platform's standard connectors are recognized out of the box.

Custom or non-standard sources, for example a custom Apify actor that returns its own JSON shape, may not be recognized. When Unify does not recognize the incoming shape:

• The document is not force-fit into the Default Schema.
• Unrecognized documents can fail at the Unify step rather than continuing downstream.

How do I map a custom source that Unify doesn't recognize?

Add a JSON Schema Transformer before the Unify Transformer to pre-map the custom fields into the Default Schema shape:

• Map the custom source's fields (for example, authorMeta.name, videoMeta.duration) to their Default Schema destinations (author.name, content.duration, and so on).
• Once the document is in the expected shape, Unify aligns the rest as normal.

If you expect a source to be recognized automatically but its documents are failing at Unify, inspect a failed item in the Failed Items Viewer to confirm the shape, then either add a pre-mapping step or contact support with the sample document.

Schema Format

The Datastreamer Schema is nested JSON. This nested JSON is set by field type. Fields regarding a similar focus of the document (Source, Author, Content, Enrichment, etc...) are nested in categories.

            "id": "3420227781307210486-wsl_instagram",
            "doc_date": "2024-07-25T21:29:05Z",
            "data_source": "wsl_instagram",
            "source": {
                "link": "https://instagram.com/p/C93FwvDvRr2/"
            },
            "content": {
                "body": "Ruélia vermelha ou Ruélia da Amazônia \n\n🤓 Nome científico..."

View Current Datastreamer Default Schema

As the Datastreamer Default Schema is constantly expanded with new fields, you can view the Datastreamer Default Schema at the link below. This link also shows many of the common data partner sources and if Unify has a verified conversion for that source.

Datastreamer Default Schema (Google Sheet)