Rare IdempotencyPersistenceLayerError before DynamoDB call with makeIdempotent in Step Functions Lambda (~1/1000 executions)

[AlfGoto]

Rare IdempotencyPersistenceLayerError before DynamoDB call with makeIdempotent in Step Functions Lambda (~1/1000 executions)

Summary

We have a Lambda invoked by Step Functions that uses makeIdempotent with a DynamoDBPersistenceLayer.
On very rare occasions (estimated ~1 in 1000 executions, sometimes less than 2 times a week), it fails before calling DynamoDB, with the error:

IdempotencyPersistenceLayerError: Failed to save in progress record to idempotency store

The DynamoDB table exists and the Lambda has full permissions.
This same Lambda processes thousands of executions without error.
It only seems to happen when using an eventKeyJmesPath in object form, but we cannot reproduce reliably — the vast majority of those executions succeed.

---

Environment

• Powertools for AWS Lambda (TypeScript):** "2.24.1"
• AWS Lambda runtime: Node.js 20.x
• Architecture: arm64
• Trigger: AWS Step Functions
• Idempotency store: DynamoDB (PK/SK schema)

---

Invocation Flow

Step Functions is configured to invoke the Lambda like this (simplified CDK construct):

new tasks.LambdaInvoke(this, "SendMailLambda", {
  lambdaFunction: props.handler,
  payloadResponseOnly: true,
  payload: sfn.TaskInput.fromObject({
    event: sfn.JsonPath.objectAt("$.event"),
    emailType: props.emailType,
    executionArn: sfn.JsonPath.stringAt("$$.Execution.Id"),
  }),
  resultPath: "$.data",
}).addRetry({ errors: ["Lambda.Unknown"] })

---

Lambda Handler

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: process.env.IDEMPOTENCY_TABLE_NAME!,
  keyAttr: "PK",
  sortKeyAttr: "SK",
});

export const lambdaHandler: lambda.Handler<EventType> = async (event, _ctx, callback) => {
  let config: IdempotencyConfig
  let handlerFunction: lambda.Handler<EventType>

  if (event.emailType === EmailType.AddSlotsRequest) {
    config = new IdempotencyConfig({
      throwOnNoIdempotencyKey: true,
      eventKeyJmesPath: "event.activityId",
      expiresAfterSeconds: 7 * 24 * 60 * 60,
    })
    config.registerLambdaContext?.(_ctx)
    handlerFunction = createAddSlotsRequestHandler(config)
  } else {
    config = new IdempotencyConfig({
      throwOnNoIdempotencyKey: true,
      eventKeyJmesPath: '{ "emailType": emailType, "eventId": event.id }',
      expiresAfterSeconds: 30 * 24 * 60 * 60,
    })
    config.registerLambdaContext?.(_ctx)
    handlerFunction = createBaseHandler(config)
  }
  return handlerFunction(event, _ctx, callback)
}

export const handler = middy(lambdaHandler)
  .use(captureLambdaHandler(tracer))
  .use(injectLambdaContext(logger, { logEvent: true }));

function createBaseHandler(config: IdempotencyConfig): lambda.Handler<EventType> {
  return makeIdempotent(
    async (event) => {
      // ... business logic ...
    },
    { persistenceStore, config }
  );
}

---

Logs (redacted)

The error occurs immediately, before any DynamoDB request in X-Ray:

IdempotencyPersistenceLayerError: Failed to save in progress record to idempotency store
    at #saveInProgressOrReturnExistingResult (...)
    at async IdempotencyHandler.handle (...)

No PutItem call is seen in X-Ray, which suggests the failure occurs before the persistence layer tries to store the record.

---

Observations

• Happens extremely rarely (~1/1000 executions or less).
• Only seen so far when using the eventKeyJmesPath object form:

  '{ "emailType": emailType, "eventId": event.id }'

• Never seen for the string path ("event.activityId") variant, but those executions are much fewer.
• DynamoDB table exists and Lambda has correct permissions.
• When it happens, the Lambda execution fails instantly (tens of ms), without any business logic running.

---

Expected behavior

• Either the in-progress record should be saved in DynamoDB before failing, or
• If the idempotency key computation fails, provide a more specific error message rather than IdempotencyPersistenceLayerError.

---

Additional context

When this happen its always in the same lambda execution, the first call of the lambda work but all the other one fails

We suspect this might be related to:

• A rare parsing failure for the object-form eventKeyJmesPath
• An unexpected Step Functions payload variation causing key computation to return undefined or invalid data.

[dreamorosi]

Hi @AlfGoto, thanks for opening this - we'll try to reproduce the issue but given the conditions you described it might take a while before we can artificially reproduce it.

Is there any chance that you could also share an example of a payload that fails and the exact JMESPath expression that you're using? If you prefer not to paste it here, could you send it privately via email at [email protected]?