JSON Transform
The JSON Transform feature enables modification and enrichment of incoming content within the pipeline. This component allows the application of mappings, operations, and conditions to customize and transform the pipeline data.
Mappings
Mappings can be used to rename JSON properties, remove unnecessary content, or add new data. Expression conditions control when these mapping actions are applied. For more details, see the following section Operators.
Here is an example of JSON Transform configuration with mappings.
{
"mappings": [
{
"source_path": "post.id",
"destination_path": "id",
"type": "string"
},
{
"source_path": "post.created_at",
"destination_path": "doc_date",
"type": "date"
},
{
"source_path": "post.comment",
"destination_path": "content.body",
"type": "date",
"condition": {
"path": "type",
"operator": "eq",
"value_type": "string",
"value": "comment"
}
}
]
}
Operations
Operations are a key feature of the JSON Transformation component, allowing content manipulation. Available operations enable formatting, concatenation, mapping, content extraction, and more. All operations support conditions.
{
"mappings": [
{
"source_path": "post.id",
"destination_path": "id",
"type": "string"
},
{
"source_path": "post.created_at",
"destination_path": "doc_date",
"type": "date"
},
{
"source_path": "post.comment",
"destination_path": "content.body",
"type": "date",
"condition": {
"path": "type",
"operator": "eq",
"value_type": "string",
"value": "comment"
}
}
],
"operations": [
{
"stage": "transformation",
"name": "format",
"destination_path": "source.link",
"parameters": {
"fields": [
"post.id"
],
"format": "https://example.com/comments/{0}"
},
"condition": {
"path": "type",
"operator": "eq",
"value_type": "string",
"value": "comment"
}
}
]
}
Concat
Join two or more field values using a separator.
{
"operations": [
{
"name": "concat",
"stage": "transformation",
"destination_path": "author.full_name",
"parameters": {
"fields": [
"author.first_name",
"author.last_name"
],
"separator": " "
}
}
]
}
Format
Build a string value using a template with placeholders. The format is similar to a string interpolation in computer programing.
- In fields, one or more JSON source paths can be defined, and the values will be used to replace the format placeholders.
- The format represents the template with the placeholders {0}, {1} ...
{
"operations": [
{
"stage": "transformation",
"name": "format",
"destination_path": "source.link",
"parameters": {
"fields": [
"post.id"
],
"format": "https://example.com/comments/{0}"
},
"condition": {
"path": "type",
"operator": "eq",
"value_type": "string",
"value": "comment"
}
}
]
}
Hash
Concat fields and return a hash representation. This operation is useful for cases where there is no unique ID in the content or when you need to hide sensitive information but, still need a hash for comparison.
- type: SHA-256 or MD5 (default)
- fields values to hash
{
"operations": [
{
"name": "hash",
"destination_path": "id",
"parameters": {
"type": "SHA-256", // Default MD5
"fields": [
"post.created_at",
"post.comment"
]
}
}
]
}
Key_to_value
Map a JSON Object to a key-value-pair array.
{
"operations": [
{
"name": "key_to_value",
"destination_path": "attributes",
"parameters": {
"source": "attributes",
"value_name": "value",
"key_name": "key"
}
}
]
}
Input
{
"product_id": "12345",
"name": "Wireless Bluetooth Headphones",
"description": "High-quality wireless Bluetooth headphones with noise-cancellation, up to 30 hours of battery life, and built-in microphone.",
"attributes": {
"battery_life": "30 hours",
"has_microphone": true,
"colors": [
"Black",
"White",
"Blue"
],
"features": [
"Bluetooth 5.0",
"Noise-cancellation"
]
}
}
Output
{
"id": "12345",
"attributes": [
{
"key": "battery_life",
"value": "30 hours"
},
{
"key": "has_microphone",
"value": true
},
{
"key": "colors",
"value": [
"Black",
"White",
"Blue"
]
},
{
"key": "features",
"value": [
"Bluetooth 5.0",
"Noise-cancellation"
]
}
]
}
i18n_language_shorthand
The Internationalization Language Shorthand converts a language string to an ISO 639 language code.
Examples
two_letter_code: es-ES -> es
three_letter_code: es-ES -> esp
{
"operations": [
{
"name": "i18n_language_shorthand",
"parameters": {
"format": "two_letter_code",
"field": "lang"
},
"destination_path": "enrichment.language"
}
]
}
Map
Perform different map actions based on different conditions.
- Alt: the default alternative value if none of the conditions match
- From: regex condition from source value
Configuration
{
"operations": [
{
"stage": "transformation",
"name": "map",
"destination_path": "new_category",
"parameters": {
"source": "category",
"alt": "other",
"map": [
{
"from": "^arts",
"to": "entertainment"
},
{
"from": "^culture",
"to": "entertainment"
},
{
"from": "^entertainment",
"to": "entertainment"
}
]
}
}
]
}
Input
[
{
"title": "The Mona Lisa by Leonardo da Vinci, the world's most famous portrait, could get a room of its own in the Louvre, the museum's president said on Saturday.",
"category": "entertainment"
},
{
"title": "Reviving Traditions: Global Cultures Embrace Ancient Art Forms in Modern Times",
"category": "entertainment"
},
{
"title": "Streaming Wars Intensify: New Platforms and Big Releases Reshape Entertainment Landscape",
"category": "entertainment"
},
{
"title": "Electric Revolution: Automakers Race to Redefine the Future of Cars",
"category": "automobile"
}
]
Output
[
{
"title": "The Mona Lisa by Leonardo da Vinci, the world's most famous portrait, could get a room of its own in the Louvre, the museum's president said on Saturday.",
"category": "art",
"new_category": "other"
},
{
"title": "Reviving Traditions: Global Cultures Embrace Ancient Art Forms in Modern Times",
"category": "culture",
"new_category": "entertainment"
},
{
"title": "Streaming Wars Intensify: New Platforms and Big Releases Reshape Entertainment Landscape",
"category": "entertainment",
"new_category": "entertainment"
},
{
"title": "Electric Revolution: Automakers Race to Redefine the Future of Cars",
"category": "automobile",
"new_category": "other"
}
]
Extract
Extract content that matches the regex. Use the format field to set the destination of every match.
Configuration
{
"operations": [
{
"stage": "transformation",
"name": "extract",
"destination_path": "content.author_related_post",
"parameters": {
"source_path": "url",
"regex": "@([^\/]+)",
"format": "{1}"
}
}
]
}
Input
{
"source": {
"link": "https://www.example.com/@mycat/comments/12345"
}
}
Output
{
"content.author_related_post": "mycat",
"source": {
"link": "https://www.example.com/@mycat/comments/12345"
}
}
Updated 16 days ago