Google Cloud Storage Egress

Component Configuration

Export data to your Google Cloud Storage (GCS) bucket and folder using Datastreamer Pipeline. The component supports different egress types and Data Collation.

Collation Type

It is recommended to use File Collation (default). This groups documents into files for the job. Alternative options are to collate based on messages (internal process for managing requests into manageable units for pipeline processing) or individual files for each document received.

Bucket (required)

Specify the Google Cloud Storage bucket name for egress.

Use Metadata Tag (Optional)

Specify the Metadata Tag "name" to be used for the output folder in the bucket. The Tag "value" is configured as part of job creation. See Creating Jobs (Portal, API). If the Tag is not present on the document/file received by the Google Cloud Storage Egress component, the Metadata Tag value will be used by default as the folder name.

Collation Size

Integer (bytes) specifying the collation size of the output JSON file to be created in the Google Cloud Storage bucket. While processing a job, the Google Cloud Storage Egress component will collate results until the file size is reached. Once the size is reached the file will be uploaded to the Google Cloud Storage bucket. Where the job generates more results additional files will be created with an incrementing number appended to the file name i.e. "-1", "-2".

The Google Cloud Storage Storage Egress component will wait for 60 seconds for new documents to collate, if no more are received in that time, the collated file is uploaded to the Google Cloud Storage Egress bucket even if size limit is not reached.

Egress Data

It is recommended to use Documents (default configuration). For Ingress and Operation Components (i.e. WebSightLine File Fetcher) that process file objects (i.e. images, PDFs) these objects can be retained in cache for additional processing and egressed at the end of the pipeline using the alternative options: Files & Documents or Files only.

Output Format

Options for JSON collation format.

GCP Service Account (Required)

Add your GCP Service Account in the "Keys & Secrets" page from the Portal menu. Make sure your Service Account has write permission to the destination bucket.

Here is how to get a Google Cloud Service Account https://cloud.google.com/iam/docs/keys-create-delete.