An AWS lambda listening to events from S3 and running Ylem pipelines using the uploaded file content as input.
After installation, the lambda listens to S3 events about created objects (i.e. uploaded files). If the file path matches the configured expression, the lambda reads their contents and runs the configured pipelines for each file.
Create an OAuth client for the lambda here: https://app.datamin.io/api-clients and copy the client ID and client secret key.
Follow this guide to install the lambda from a zip archive.
- Navigate to AWS Lambda → Functions section.
- Click "Create function".
- Choose "Container image".
- Enter function name.
- Enter container image URI:
241485570393.dkr.ecr.eu-central-1.amazonaws.com/datamin-integration:latest - Architecture:
x86_64. - Execution role: pick
Create a new role with basic Lambda permissions.
First, make sure that the lambda has read access to objects in all S3 buckets you plan to use.
Second, follow this guide to set up a S3 trigger for the lambda.
Then, you need to configure several environment variables for the lamba to work.
Navigate to Configuration → Environment variables section of the lambda.
| Variable | Description |
|---|---|
| DTMN_API_CLIENT_ID | OAuth client ID. Create a new client here: https://app.ylem.co/api-clients |
| DTMN_API_CLIENT_SECRET | OAuth client secret |
| DTMN_S3_MAPPING | Mapping of path expressions to pipeline UUIDs. See below. |
The format is:
<path expression #1>:<pipeline UUID #1>,<pipeline UUID #2>;<path expression #2>:<pipeline UUID #3>
Path expression: A Glob-like expression to match the full object path against, including the bucket name.
If the object path matches the expression, lambda will read the object contents and send them as input to all pipelines which UUIDs are enumerated in the config.
Pipeline UUIDs: a comma-separated list of pipelines to run if the object path matches the expression.
my-bucket-name/logs/*.json:18d27bb8-f886-4072-9165-485baa3e332d;my-bucket-name/*/daily.json:2f0cfe9e-3571-49a4-9103-cbd9d3d58c2b