Amazon S3 Ingress Setup Guide

To set up your pipeline S3 Ingress, you will need to follow just a few steps.

Prerequisites

To use Amazon S3 within your Pipelines, you will need to have the following:

1. Datastreamer Portal account with access to add a new component.
2. Amazon Access Key.
3. Amazon Secret Key.

Setup Instructions

Step 1: Open the target Pipeline within Portal

You can create a new Pipeline or open an existing Pipeline within the Portal.

Step 2: Add a Amazon S3 Ingress component from the selection menu

Click on the Ingress (+) button and select the component Amazon S3 Storage on the side panel menu.

Step 3: Configure the component

Next, click on the Configure button, and a list of properties will be displayed on the side panel.

The configuration is divided into 2 parts: the Component Properties and the Jobs. Component Properties must be configured before deploying the pipeline. Jobs, on the other hand, can be configured before or after the pipeline is deployed.

Step 4: Deploy your Pipeline

Upon deployment, this Pipeline is fully ready to consume and process data. Deploying generally takes 10-30 seconds for full rollout and diagnostics.

More details on Pipeline deployment is available here: Docs: Pipeline Deployoment