Iceberg Tables Schema Drift Alerts

Overview

This Lightweight solution monitors Iceberg table schema changes in AWS Glue catalog and sends notifications via Slack webhooks when changes are detected.

Prerequisites

• AWS CLI configured with appropriate permissions
• Slack workspace with webhook integration capability
• Access to AWS Glue catalog with Iceberg tables
• Python 3.11+ for local development/testing

Setup Instructions

1. Slack Webhook Setup

1. Go to your Slack workspace settings
2. Navigate to Apps → Manage → Custom Integrations → Incoming Webhooks
3. Click Add Configuration
4. Select the channel where you want notifications
5. Copy the webhook URL (format: https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX)

2. Deploy Infrastructure using AWS cloudformation

# Clone or download the CloudFormation template
aws cloudformation create-stack \
  --stack-name iceberg-tables-schema-drift-alerts \
  --template-body file://./deployment/cloudformation.yaml \
  --parameters \
    ParameterKey=TargetDatabase,ParameterValue=DATABASE_NAME \
    ParameterKey=SlackWebhookUrl,ParameterValue=SLACK_WEBHOOK_URL \
    ParameterKey=ScheduleExpression,ParameterValue="rate(1 hour)" \
  --capabilities CAPABILITY_NAMED_IAM

3. Deploy Lambda Code

Prepare Deployment Package

# Create deployment directory
sh deploy_lambda.sh

4. Verify Deployment

Test Lambda Function

# Test the Lambda function
aws lambda invoke \
  --function-name iceberg-tables-schema-drift-alerts \
  --payload '{}' \
  response.json

# Check the response
cat response.json

Check CloudWatch Logs

# View recent logs
aws logs describe-log-streams \
  --log-group-name /aws/lambda/iceberg-schema-monitor \
  --order-by LastEventTime \
  --descending