SFTP Storage Egress

Component Configuration

You can use SFTP Egress for your Pipeline outputs by adding this component at the end of the Pipeline. It is recommended to use a transform operation (JSON Transform or Unify Transform) to standardize the fields before this component.

Collation Type

It is recommended to use File Collation (default). This groups documents into files for the job. Alternative options are to collate based on messages (internal process for managing requests into manageable units for pipeline processing) or individual files for each document received.

Use Metadata Tag (Optional)

Specify the Metadata Tag "name" to be used for the output folder in the bucket. The Tag "value" is configured as part of job creation. See Creating Jobs (Portal, API). If the Tag is not present on the document/file received by the SFTP Egress component the Metadata Tag value will be used by default as the folder name.

Collation Size

Integer (bytes) specifying the collation size of the output JSON file to be created in the Cloudflare R2 bucket. While processing a job, the Cloudflare R2 Storage Egress component will collate results until the file size is reached. Once the size is reached the file will be uploaded to the Cloudflare R2 bucket. Where the job generates more results additional files will be created with an incrementing number appended to the file name i.e. "-1", "-2".

The Cloudflare R2Storage Egress component will wait for 60 seconds for new documents to collate, if no more are received in that time, the collated file is uploaded to the Cloudflare R2 bucket even if size limit is not reached.

Egress Data

It is recommended to use Documents (default configuration). For Ingress and Operation Components (i.e. WebSightLine File Fetcher) that process file objects (i.e. images, PDFs) these objects can be retained in cache for additional processing and egressed at the end of the pipeline using the alternative options: Files & Documents or Files only.

Output Format

Options for JSON collation format.

SFTP Host

The hostname or IP address of the SFTP server, e.g sftp.example.com.

SFTP Port

The port number used for the SFTP connection. This is usually 22.

Username

The username used to authenticate with the SFTP server.

Password

The password for the specified username. Used if password authentication is enabled.

Private Key

The private key (PEM format) used for authentication. Leave blank if using password authentication.

Private Key Passphrase

The passphrase for the private key (if required). Leave blank if not applicable.