Transformation Operations

The following operations can be used within the transformation mapping by setting the type to "operation" and specifying the name and parameters.

map

Maps a source value to a different destination value based on a predefined dictionary.

Mapping Fields

Additional Parameters

Example

Map product types to a description

{
  "type": "operation",
  "name": "map",
  "source_path": "product.type",
  "destination_path": "product.description",
  "parameters": {
    "alt": "Other Products",
    "map": [
      {
        "from": "^toys.*$",
        "to": "Toys & Games"
      },
      {
        "from": "^audio.*$",
        "to": "Audio Electronics"
      },
      {
        "from": "^pc.*$",
        "to": "Laptop & Desktop Computers"
      }
    ]
  }
}

hash

Generates a hash from the concatenated values of one or more fields.

Mapping Fields

Additional Parameters

Example

Generate an id from a handle and time stamp.

{
  "type": "operation",
  "name": "hash",
  "destination_path": "id",
  "destination_format": "hexidecimal",
  "parameters": {
    "type": "MD5",
    "fields": [
      "handle",
      "timestamp"
    ]
  }
}

concat

Joins the string values of multiple fields together.

Mapping Fields

Additional Parameters

Example

Generate an author full name from the first and last name

{
  "type": "operation",
  "name": "concat",
  "destination_path": "author.full_name",
  "parameters": {
    "separator": " ",
    "fields": [
      "author.first_name",
      "author.last_name"
    ]
  }
}

format

Creates a formatted string from the values of one or more fields.

Mapping Fields

Additional Parameters

Example

Generate an author full name from the first and last name

{
  "type": "operation",
  "name": "format",
  "destination_path": "author.full_name",
  "destination_format": "{0} {1}",
  "parameters": {
    "fields": [
      "author.first_name",
      "author.last_name"
    ]
  }
}

extract

Extracts a value from a string using a regular expression.

Mapping Fields

Additional Parameters

No additional parameters

Example

Extract @xyz mentions from field containing free format text.

{
  "type": "operation",
  "name": "extract",
  "source_path": "main_text",
  "destination_path": "content.mentions[*]",
  "format": "@([A-Za-z0-9_]{4,15})",
  "destination_format": "{*}"
}

trim_array

Removes null or empty values from an array.

Requiressource_path, format and destination_path to be present in the mapping JSON.

Mapping Fields

Additional Parameters

No additional parameters

Example

Remove empty hashtag values from the output document.

{
  "type": "operation",
  "name": "trim_array",
  "source_path": "content.hashtags",
  "destination_path": "content.hashtags",
  "format": "null, empty",
  "source_type": "output"
}

key_to_value

Transforms an object with dynamic keys into an array of objects. This is useful when the number of unique key names is not defined but there are limits on the number of unique keys (e.g. an Elasticsearch index)

Mapping Fields

Additional Parameters

Example

Convert :

• {"metrics": {"clicks": 10, "views": 100}}

To

• {"metrics": [{"metric": clicks", "value": 10}, {"metric": "views": "value": 100}]}

{
  "type": "operation",
  "name": "key_to_value",
  "source_path": "metrics",
  "destination_path": "metrics",
  "parameters": {
    "key_name": "metric",
    "value_name": "value"
  }
}

i18n_language_shorthand

Converts a full language name (e.g., "English") to its two or three-letter ISO code.

Mapping Parameters

Additional Parameters

No additional parameters

Example

{
  "type": "operation",
  "name": "i18n_language_shorthand",
  "source_path": "language_name",
  "destination_path": "language",
  "format": "two_letter_code"
}