Transformation Schema

Schema

A transformation schema is of the form

{
  "name": "string",
  "description": "string",
  "run_in_place": "boolean",
  "mappings": [
    {}
  ],
  "transformations": [  
    {}
  ],
  "condition": {
  }
}

Field	Description
`name`	An optional name for this transformation.
`description`	An optional description for this transformation.
`run_in_place`	If set to true then the transformation will mutate the input document rather than create a new document.
`mappings`	An array of mappings that will be performed in the order they are in the array.
`transformations`	An array of child transformations (with this structure) that can be conditionally performed.
`condition`	A JSON condition that is evaluated on the input JSON document before this transformation is performed.

Mapping Structure

Mappings contain the following JSON fields

{
  "type": "string|integer|float|boolean|date|json|const|operation",
  "source_path": "string",
  "format": "string",
  "destination_path": "string",
  "destination_format": "string",
  "array": "string",
	"name": "string",
  "parameters": {
    "field": "value",
		...
	}
}

`type`

Specifies the data type in the destination JSON. The default is string

Type	Description
`string`	A text string
`integer`	An integer numeric value
`float`	A decimal numeric value
`boolean`	true \| false
`date`	A date value depending on `destination_format`. By default this is an ISO8601 date of the form `YYYY-MM-DDTHH:MM:SS.SSSZ`
`const`	Sets a constant value. The value is taken from `source_path` (interpreted as a literal) or generated based on `format`. See `format` below
`json`	A JSON structure. This will duplicate the JSON at `source_path` at `destination_path` without change.
`operation`	Specifies that this mapping is a named operation that will be performed on the document JSON

`source_path`

The dot notation JSON path to extract data from the input document.

Array Wildcard: To process all elements in an array, use the [*] wildcard. For example, a path of items[*].name will extract the name from every object within the items array, applying the transformation to each one.
Condition Elements: To process specific elements within an array use [?(condition)], e.g.
- reactions[?(@.count >= 50)]
- files[?(@.content_type == 'application/json')]

`destination_path` (Required)

The JSON path where the transformed value will be placed in the output document.

Use dot notation for objects: person.name
Use array notation for lists: users[0].id or users[*].id for multiple elements.

`format`

Specifies the input format or parsing logic, depending on the type:

Type	`format` Value
`string`	A Regular Expression (Regex) used to extract parts of the source string.
`date`	The date format (e.g. `MM/DD/YYYY`) or `truncate`: Removes nanosecond precision from a timestamp before parsing. `unix`: The unix time in seconds `unix-ms`: The unix time in milli-seconds
`const`	`date`: Sets the value to the current UTC time `bool`: Parses `source_path` as a boolean (`"true"` or `"false"`).Parses `source_path` `int`: Parses `source_path` as an integer.

Type

format Value

string

A Regular Expression (Regex) used to extract parts of the source string.

date

The date format (e.g. MM/DD/YYYY) or

truncate: Removes nanosecond precision from a timestamp before parsing.
unix: The unix time in seconds
unix-ms: The unix time in milli-seconds

const

date: Sets the value to the current UTC time
bool: Parses source_path as a boolean ("true" or "false").Parses source_path
int: Parses source_path as an integer.

`destination_format`

Specifies the output format or parsing logic, depending on the type:

Type	`format` Value
`string`	A format string used to construct the output from Regex matches defined in `format`. Use `{0}`, `{1}`, etc., to reference captured groups. If `format` does not contain a Regex then use `{0}` to refer to the entire input string, e.g. `https://twitter.com/{0}`
`date`	The date format (e.g. `MM/DD/YYYY`) or `unix`: The unix time in seconds `unix-ms`: The unix time in milli-second

Type

format Value

string

A format string used to construct the output from Regex matches defined in format. Use {0}, {1}, etc., to reference captured groups. If format does not contain a Regex then use {0} to refer to the entire input string, e.g.

https://twitter.com/{0}

date

The date format (e.g. MM/DD/YYYY) or

unix: The unix time in seconds
unix-ms: The unix time in milli-second

`array`

This is a list of values that describes how array elements are added to the destination JSON. If set then this list should contain a value for each level of nesting in the destination JSON. Values are:

Value	Description
`map`	Array elements are added at the same position as in the source JSON
`append`	Array elements are appended to the values currently in the destination JSON

`source_type`

Specify the source JSON; values are:

`source_type`	Description
`input`	`source_path` references values in the input document JSON
`output`	`source_path`references values in the currently building destination document JSON

`name`

When type is operation this field specifies the name of the operation.

`parameters`

Key-value pairs of parameters passed to custom operations when type is "operation".

`condition`

Type: object

A filter condition that must be met for this mapping to execute.

Schema

Mapping Structure

type

source_path

destination_path (Required)

format

destination_format

array

source_type

name

parameters

condition