rdi-cloud-automation

Terraform modules to configure producer databases and network connectivity for Redis Data Integration (RDI).

🚀 Overview

This repository provides production-ready Terraform modules to deploy and configure source databases for Redis Data Integration (RDI) with secure AWS PrivateLink connectivity. It supports multiple database engines with automatic CDC (Change Data Capture) user provisioning and optional sample data loading.

Supported Database Engines

Database	Engine	CDC Method	Auto User Creation	High Availability
PostgreSQL	Aurora PostgreSQL	Logical Replication	❌ (uses admin)	✅ Multi-AZ
MySQL	Aurora MySQL 8.0	Debezium (binlog)	✅ debezium user	✅ Multi-AZ
SQL Server	RDS SQL Server SE	Change Tracking	✅ rdi_user	✅ Multi-AZ

📋 Prerequisites

• Terraform >= 1.5.7
• AWS CLI configured with credentials
• Redis Cloud account with RDI enabled
• Database client tools (optional, for testing):
  • psql for PostgreSQL
  • mysql for MySQL
  • sqlcmd for SQL Server

📚 Examples

The examples directory contains complete, ready-to-deploy examples:

aws-ec2-privatelink

Creates a PostgreSQL database on EC2 exposed with PrivateLink. This example creates a VPC and can be used to try RDI quickly with no existing resources.

Use case: Quick testing and development with PostgreSQL

aws-rds-privatelink-failover

Creates production-ready RDS databases (PostgreSQL, MySQL, or SQL Server) with automatic failover support via AWS PrivateLink.

Features:

• ✅ Multi-engine support: Choose PostgreSQL, MySQL, or SQL Server
• ✅ Automatic CDC user creation: MySQL and SQL Server users created automatically
• ✅ High availability: Multi-AZ deployment with automatic failover
• ✅ Lambda-based failover: Automatically updates NLB targets during RDS failover
• ✅ Optional sample data: Chinook database for testing
• ✅ Secure connectivity: AWS PrivateLink for private VPC-to-VPC connections

Quick Start:

cd examples/aws-rds-privatelink-failover

# For PostgreSQL
terraform apply -var-file example-postgres.tfvars

# For MySQL
terraform apply -var-file example-mysql.tfvars

# For SQL Server
terraform apply -var-file example-sqlserver.tfvars

See examples/aws-rds-privatelink-failover/README.md for detailed documentation.

🧩 Modules

The modules directory contains reusable Terraform modules which can be composed together to build custom database infrastructure.

Database Modules

Module	Description	Database Type	Use Case
aws-rdi-quickstart-postgres	VPC and EC2 instance with PostgreSQL	PostgreSQL on EC2	Quick testing and development
aws-rds-chinook	Aurora PostgreSQL RDS cluster	Aurora PostgreSQL	Production PostgreSQL with HA
aws-rds-mysql-chinook	Aurora MySQL RDS cluster	Aurora MySQL 8.0	Production MySQL with HA
aws-rds-sqlserver-chinook	RDS SQL Server instance	SQL Server SE 2022	Production SQL Server with HA

Infrastructure Modules

Module	Description	Purpose
aws-privatelink	Network Load Balancer + PrivateLink	Secure VPC-to-VPC connectivity
aws-rds-lambda	Lambda function for RDS event handling	Automatic failover detection and NLB updates
aws-secret-manager	KMS Key + Secrets Manager	Secure credential storage for RDI

Key Features by Module

Database Modules:

• ✅ VPC with public, private, and database subnets
• ✅ Security groups with self-referencing rules
• ✅ Multi-AZ deployment for high availability
• ✅ CDC-enabled parameter groups
• ✅ Automatic engine version selection (MySQL)

aws-privatelink:

• ✅ Network Load Balancer (internal or public)
• ✅ Cross-zone load balancing
• ✅ PrivateLink VPC Endpoint Service
• ✅ Automatic principal whitelisting

aws-rds-lambda:

• ✅ SNS topic for RDS events
• ✅ Lambda function to detect writer changes
• ✅ Automatic NLB target group updates
• ✅ CloudWatch logging

aws-secret-manager:

• ✅ KMS encryption for secrets
• ✅ IAM policy for Redis Cloud access
• ✅ Automatic credential rotation support

🔐 CDC User Management

The Terraform modules automatically create CDC (Change Data Capture) users with appropriate permissions for MySQL and SQL Server. PostgreSQL uses the admin user directly.

Automatic User Creation

Database	User	Password	Permissions	Created By
PostgreSQL	postgres	Admin password	Superuser (logical replication)	RDS (admin user)
MySQL	debezium	Auto-generated	SELECT, RELOAD, REPLICATION SLAVE, REPLICATION CLIENT	Terraform null_resource
SQL Server	rdi_user	Auto-generated	dbcreator, VIEW SERVER STATE, VIEW ANY DEFINITION	Terraform null_resource

How It Works

MySQL:

# Automatically runs during terraform apply
mysql -h <nlb_hostname> -u admin -p <<SQL
CREATE USER IF NOT EXISTS 'debezium'@'%' IDENTIFIED BY '<auto-generated>';
GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT, LOCK TABLES ON *.* TO 'debezium'@'%';
FLUSH PRIVILEGES;
SQL

SQL Server:

-- Automatically runs during terraform apply
CREATE LOGIN rdi_user WITH PASSWORD = '<auto-generated>';
CREATE USER rdi_user FOR LOGIN rdi_user;
ALTER SERVER ROLE [dbcreator] ADD MEMBER rdi_user;
GRANT VIEW SERVER STATE TO rdi_user;
GRANT VIEW ANY DEFINITION TO rdi_user;

Credentials Storage

All credentials are automatically stored in AWS Secrets Manager with:

• ✅ KMS encryption at rest
• ✅ IAM policy allowing Redis Cloud access
• ✅ Automatic rotation support (optional)

Access credentials via Terraform outputs:

terraform output rdi_username  # CDC username for RDI
terraform output rdi_password  # CDC password (sensitive)
terraform output secret_arn    # Secrets Manager ARN for Redis Cloud

🏗️ Architecture

Network Flow

Redis Cloud RDI
    ↓
AWS PrivateLink (VPC Endpoint)
    ↓
Network Load Balancer (NLB)
    ↓
RDS Database (Multi-AZ)

Failover Handling

For Aurora clusters (PostgreSQL and MySQL):

1. RDS emits failover event to SNS
2. Lambda function detects writer instance change
3. Lambda updates NLB target group with new writer IP
4. RDI connections automatically route to new writer
5. Zero configuration changes needed in Redis Cloud

Security Architecture

• ✅ Private connectivity: AWS PrivateLink (no internet exposure)
• ✅ Encryption at rest: RDS storage encryption enabled
• ✅ Encryption in transit: TLS support (optional)
• ✅ Credential management: AWS Secrets Manager with KMS
• ✅ Network isolation: VPC with private subnets for databases
• ✅ Least privilege: IAM policies scoped to specific resources

💡 Common Use Cases

1. Quick Testing with Sample Data

Deploy PostgreSQL and manually load Chinook sample database:

cd examples/aws-rds-privatelink-failover

# Deploy infrastructure
terraform apply -var-file example-postgres.tfvars

# Load sample data (requires network access to RDS)
# Option A: If nlb_internal = false (public NLB)
./psql.sh  # Then manually load Chinook SQL

# Option B: From bastion host or VPN connection
# See example README for detailed setup instructions

2. Production MySQL with CDC

Deploy Aurora MySQL with automatic debezium user creation:

cd examples/aws-rds-privatelink-failover
terraform apply -var-file example-mysql.tfvars
# debezium user is automatically created with CDC permissions
# Credentials stored in AWS Secrets Manager

3. SQL Server with Change Tracking

Deploy SQL Server with automatic rdi_user creation:

cd examples/aws-rds-privatelink-failover
terraform apply -var-file example-sqlserver.tfvars
# rdi_user is automatically created with CDC permissions
# Ready for SQL Server Change Tracking

4. Multi-Region Deployment

Deploy databases in multiple AWS regions for disaster recovery:

# Region 1 (us-east-1)
cd examples/aws-rds-privatelink-failover
terraform workspace new us-east-1
terraform apply -var-file example-postgres.tfvars -var region=us-east-1

# Region 2 (us-west-2)
terraform workspace new us-west-2
terraform apply -var-file example-postgres.tfvars -var region=us-west-2

🔧 Troubleshooting

CDC User Creation Issues

Problem: MySQL debezium user creation fails

Solution:

• Ensure mysql client is installed locally
• Check network connectivity to NLB (set nlb_internal = false for testing)
• Verify admin password is correct
• Check CloudWatch logs for detailed error messages

Problem: SQL Server rdi_user creation fails

Solution:

• Ensure sqlcmd is installed locally (installation guide)
• Check network connectivity to NLB
• Verify sa password is correct
• SQL Server may take 60+ seconds to be ready after creation

Connection Issues

Problem: Cannot connect to database from laptop

Solution:

• Set nlb_internal = false in tfvars to make NLB public
• Ensure security group allows your IP address
• Use the correct port (5432 for PostgreSQL, 3306 for MySQL, 1433 for SQL Server)
• Get connection details: terraform output

Problem: RDI cannot connect via PrivateLink

Solution:

• Verify PrivateLink service is whitelisted for Redis Cloud principal
• Check that the correct secret ARN is configured in Redis Cloud RDI
• Ensure the secret contains the correct username and password
• Test connectivity using ./connect.sh script

Failover Issues

Problem: Lambda not updating NLB after failover

Solution:

• Check Lambda CloudWatch logs for errors
• Verify SNS topic is subscribed to RDS events
• Ensure Lambda has IAM permissions to modify NLB target group
• Test failover manually: aws rds failover-db-cluster --db-cluster-identifier <cluster-id>

🔒 Security

This repository uses automated secret scanning to prevent accidental credential leaks:

• Gitleaks - Fast regex-based secret detection
• TruffleHog - High-entropy string detection with verification
• detect-secrets - Baseline-based secret scanning

Secret scanning runs automatically on:

• Every push to main branches
• Every pull request
• Weekly scheduled scans

For more information, see:

• Security Policy
• Secret Scanning Guide

Quick Start - Local Scanning

# Install Gitleaks
brew install gitleaks  # macOS

# Scan before committing
gitleaks detect --no-git

# Install pre-commit hook
curl -sSfL https://raw.githubusercontent.com/gitleaks/gitleaks/master/scripts/pre-commit.py -o .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit

🤝 Contributing

Contributions are welcome! Please follow these guidelines:

1. Fork the repository and create a feature branch
2. Test your changes thoroughly with terraform plan and terraform apply
3. Run security scans before committing:

   gitleaks detect --no-git

4. Update documentation if you add new features or modules
5. Submit a pull request with a clear description of changes

Development Setup

# Clone the repository
git clone https://github.com/redis/rdi-cloud-automation.git
cd rdi-cloud-automation

# Install pre-commit hooks
curl -sSfL https://raw.githubusercontent.com/gitleaks/gitleaks/master/scripts/pre-commit.py -o .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit

# Initialize Terraform
cd examples/aws-rds-privatelink-failover
terraform init

📄 License

This project is licensed under the terms specified in the repository.

🆘 Support

For issues, questions, or contributions:

• GitHub Issues: Report a bug or request a feature
• Redis Documentation: Redis Data Integration (RDI) Docs

📚 Additional Resources

• AWS PrivateLink Documentation
• Redis Data Integration (RDI) Overview
• Terraform AWS Provider Documentation
• Debezium Documentation
• AWS RDS Best Practices