Summary
We're excited to announce the Kafka Consumer utility, which transparently handles message deserialization, provides an intuitive developer experience, and integrates seamlessly with the rest of the Powertools for AWS Lambda ecosystem.
Key features
- Automatic deserialization of Kafka messages (JSON, Avro, and Protocol Buffers)
- Simplified event record handling with intuitive interface
- Support for key and value deserialization
- Support for ESM with and without Schema Registry integration
- Proper error handling for deserialization issues
Getting started
To get started, depending on the schema types you want to use, install the library and the corresponding libraries:
- JSON schemas
dotnet add package AWS.Lambda.Powertools.Kafka.Json - Avro schemas
dotnet add package AWS.Lambda.Powertools.Kafka.Avro - Protocol Buffer schemas
dotnet add package AWS.Lambda.Powertools.Kafka.Protobuf
Processing Kafka events
You can use Kafka consumer utility to transform raw Kafka events into an intuitive format for processing.
The Handler function can deserialize both keys and values independently based on your schema configuration. This flexibility allows you to work with different data formats in the same message.
Working with Avro:
Working with Protocol Buffers:
Working with JSON messages:
Error handling
You can handle errors when processing Kafka messages to ensure your application maintains resilience and provides clear diagnostic information.
The Kafka consumer utility integrates with standard C# exception handling patterns.
To learn more about the launch, read the blog post published alongside the release.
🌟New features and non-breaking changes
📜 Documentation updates
- docs: add Kafka Consumer utility documentation (#906) by @hjgraca
- chore: Update mkdocs.yml (#898) by @hjgraca
- chore: Feature/bedrock agent function resolver (#894) by @hjgraca
- chore: Add Instil as a customer ref (#896) by @hjgraca
- chore: Add llms.txt to documentation (#893) by @hjgraca
🔧 Maintenance
- chore(deps): bump urllib3 from 2.3.0 to 2.5.0 (#907) by @dependabot[bot]
- chore(deps): bump github/codeql-action from 3.28.19 to 3.29.0 (#902) by @dependabot[bot]
- chore(deps): bump requests from 2.32.3 to 2.32.4 (#901) by @dependabot[bot]
- chore: Feature/kafka consumer (#905) by @hjgraca
- chore(deps): bump github/codeql-action from 3.28.18 to 3.28.19 (#899) by @dependabot[bot]
- chore(deps): bump ossf/scorecard-action from 2.4.1 to 2.4.2 (#897) by @dependabot[bot]
- chore: Update mkdocs.yml (#898) by @hjgraca
- chore: Feature/bedrock agent function resolver (#894) by @hjgraca
- chore: Add Instil as a customer ref (#896) by @hjgraca
- chore: Add llms.txt to documentation (#893) by @hjgraca
This release was made possible by the following contributors:
@dependabot[bot], @hjgraca and dependabot[bot]
