Chainhook silently stops triggering events while showing "active" status

[Rapha-btc]

Bug Description

A chainhook predicate configured to monitor Stacks smart contract events stops triggering webhook calls after a certain block height, despite:

• The chainhook status showing as "active"
• The "last evaluated" timestamp updating to current time
• Matching events continuing to occur on-chain

Environment

• Hiro Platform: Mainnet
• Chainhook Type: Stacks smart contract event monitoring
• Predicate Scope: print_event
• Target: HTTP webhook endpoint

Expected Behavior

• Chainhook should continue triggering webhook calls for matching events
• If chainhook fails internally, status should reflect the failure state
• "Last evaluated" timestamp should indicate actual processing status

Actual Behavior

• Chainhook stops calling webhook after Bitcoin block ~913018
• Status remains "active" with current timestamps
• No error indicators in the Hiro dashboard
• On-chain events matching the predicate continue to occur but are not delivered

Reproduction Steps

1. Create chainhook predicate monitoring smart contract print events
2. Verify initial webhook deliveries work correctly
3. Wait for extended period (several days/blocks)
4. Observe webhook calls stop arriving despite matching on-chain events
5. Note that chainhook status still shows "active"

Impact

This creates a false sense of reliability for applications depending on real-time event processing. Applications cannot distinguish between:

• No matching events occurring
• Chainhook silently failing to deliver events

Configuration Details

{
  "predicate": {
    "scope": "print_event",
    "contract_identifier": "SP29D6YMDNAKN1P045T6Z817RTE1AC0JAA99WAX2B.burn-bob-faktory",
    "contains": "daily-burn"
  }
}

Proposed Solutions

1. Status accuracy: Chainhook status should reflect actual delivery capability
2. Error reporting: Clear error messages when webhook delivery fails
3. Health checks: Periodic validation that chainhooks are actually processing events
4. Retry mechanism: Automatic restart/recovery for failed chainhooks
5. Monitoring: Better observability into chainhook processing pipeline

Workaround

Currently requiring manual chainhook restarts, which is not viable for production applications requiring reliable event processing.

Additional Context

This appears to be an infrastructure-level issue rather than application code, as:

• No changes made to webhook endpoint
• Endpoint remains accessible and functional
• Same predicate configuration worked previously
• Issue resolved temporarily by restarting chainhook

Please prioritize fixing the reliability and status reporting of chainhooks, as this affects production applications depending on real-time blockchain event processing.

[Rapha-btc]

As a temporary fix, I've restarted the chainhook from Stacks block 3255343. The 'unknown' status actually provides clearer visibility than the misleading 'active' status displayed while the system was broken. This issue is impacting production users in an active daily burn community on Stacks (https://fak.fun/bob). Please prioritize addressing the underlying reliability and status reporting issues.

[Rapha-btc]

added a banner to warn users fwiw - thxs in advance fellows

https://fak.fun/bob

[deantchi]

@Rapha-btc is this still an on-going issue? The Chainhooks service was restored this morning.