Google Cloud Storage Ingress Setup Guide
Step-by-step of how to bring data from Google Cloud Storage (GCS) into your Datastreamer pipeline.
A guide to set up your pipeline with Google Cloud Storage (GCS) Ingress.
Prerequisites
To use Google Cloud Storage within your Pipelines, you will need to have the following:
- Datastreamer Portal account with access to add a new component.
- Google Cloud Service Account.
- The bucket exists. Bucket names are globally unique.
- Ensure the provided Service Account has the following access permissions to the bucket:
- Storage Object Viewer: View objects, their metadata (excluding ACLs), and list bucket contents.
- Storage Legacy Bucket Reader: List bucket contents, read bucket metadata (excluding IAM policies), and read object metadata when listing.
Here is how to get a Google Cloud Service Account https://cloud.google.com/iam/docs/keys-create-delete.
Setup Instructions
Step 1: Open the target Pipeline within Portal
You can create a new Pipeline or open an existing Pipeline within the Portal.

Step 2: Add a Google Cloud Storage Ingress component from the selection menu
Click on the Ingress (+) button and select the component Google Cloud Storage on the side panel menu.
Step 3: Configure the component
Next, click on the Configure button, and a list of properties will be displayed on the side panel.
The configuration is divided into 2 parts: the Component Properties and the Jobs. Component Properties must be configured before deploying the pipeline. Jobs, on the other hand, can be configured before or after the pipeline is deployed.

Component Properties configuration.

New Job Form.
Step 4: Deploy your Pipeline
Upon deployment, this Pipeline is fully ready to consume and process data. Deploying generally takes 10-30 seconds for full rollout and diagnostics.
More details on Pipeline deployment is available here: Docs: Pipeline Deployoment
Updated about 18 hours ago