Datastreamer File Storage Egress
The Datastreamer File Storage Egress component allows you to create files in Datastreamer Storage from your Pipeline documents, with options to customize file creation
The Datastreamer File Storage Egress allows you to create files in the Datastreamer Storage from your Pipeline documents.
This component can be placed at the end of your Pipeline and you can choose how the component will transform the documents in files. Later, you can find these created files in the Storage & Data page.
Setup the Component Properties
The Component Properties section of this component should look like this:
The configuration starts in the Component Properties. You will see the Collation Type, which has 3 different options:
- File Collation: with the File Collation option, you can choose how big each file will be before a new file is created.
- You can set the desired file size (in bytes) in the Collation Size field (the default value is 100.000.000 bytes, which is equivalent to 100 megabytes). If no data arrives within 2 minutes after the collection started, the file is created with the current collected documents.
- Per Message: Create one file for every message.
- Per Document: Create one file for every document.
In the Folder/Prefix field you can set any prefix for the documents/files that will be saved in the file storage.
- You also have the option to Use Metadata Tag. With this option, File Storage will look in the tags in the
metadataof the message and use the value that you provided as thenamefor the tag, obtaining the folder name in thevalueof the tag. If not found, the value provided in this field will be used as the folder name.
In the Egress Data field, you have 3 options:
- Documents: copy the content of the documents to the created file in the storage.
- Files: copy the content of the cached files of your pipeline to the storage.
- Documents & Files: apply both actions described above.
In the Output Format, you have two options:
- application/json-lines: the JSON lines format will keep each document in a single line in the created file.
- application/json: all documents in the file will be in an JSON array.
File Storage
As mentioned before, you can find the created files in the Datastreamer File Storage, found in the Storage & Data in our Portal menu.
You can Download or Delete each file created. You can also Filter the files, using the available field for this.
Updated 2 days ago