Unify Schema and Unify

Datastreamer's pipeline contains a transformation service that converts documents from many sources into one common schema. This makes it easier to build features and machine learning capabilities that use multiple sources together, and to integrate new data sources.

Using the Unify Schema is suggested if you do not have a set schema that you wish to use. This eases adoption and support, and lets you use the defaults and examples present in components.

Unify Transformer

Within the Datastreamer Platform, "Unify Transform" converts integrated sources from data partners into the Unify Schema. This component also uses Datastreamer's own schema generation product that reads the available schema of existing content and auto-aligns to the Unify Schema.

Using Unify within Portal

In the Datastreamer Portal, you can use Unify Transformer component after adding an Ingress component to your pipeline. After adding an Ingress component, you can search for "Unify Transformer" in the Component Picker and select the component from there. All Ingress content will be automatically converted to the unified Datastreamer schema.

Does Unify map every source automatically?

Unify Transform auto-aligns recognized source schemas to the Unify Schema. Sources integrated through the platform's standard connectors are recognized out of the box.

Custom or non-standard sources, for example a custom Apify actor that returns its own JSON shape, may not be recognized. When Unify does not recognize the incoming shape:

• The document is not force-fit into the Unify Schema.
• Unrecognized documents can fail at the Unify step rather than continuing downstream.

How do I map a custom source that Unify doesn't recognize?

Add a JSON Schema Transformer before the Unify Transformer to pre-map the custom fields into the Unify Schema shape:

• Map the custom source's fields (for example, authorMeta.name, videoMeta.duration) to their Unify Schema destinations (author.name, content.duration, and so on).
• Once the document is in the expected shape, Unify aligns the rest as normal.

If you expect a source to be recognized automatically but its documents are failing at Unify, inspect a failed item in the Failed Items Viewer to confirm the shape, then either add a pre-mapping step or contact support with the sample document.

Schema Format

The Datastreamer Schema is nested JSON. This nested JSON is set by field type. Fields regarding a similar focus of the document (Source, Author, Content, Enrichment, etc...) are nested in categories.

            "id": "3420227781307210486-wsl_instagram",
            "doc_date": "2024-07-25T21:29:05Z",
            "data_source": "wsl_instagram",
            "source": {
                "link": "https://instagram.com/p/C93FwvDvRr2/"
            },
            "content": {
                "body": "Ruélia vermelha ou Ruélia da Amazônia \n\n🤓 Nome científico..."

View Current Unify Schema

As the Unify Schema is constantly expanded with new fields, you can view the Unify Schema at the link below. This link also shows many of the common data partner sources and if Unify has a verified conversion for that source.

Unify Schema (Google Sheet)