Add automatic retry functionality for failed report executions

[eschutho]

Implement error handling controls in the report and alert configuration flow that allow users to enable automatic retries for failed chart/dashboard executions. When a scheduled report encounters failures during generation, the system should automatically retry up to a specified maximum number of attempts using an exponential backoff strategy before declaring the report as failed.

Retries should be triggered when:

• The report itself encounters an error (existing failure scenario)
• Any individual chart within the report has an error

The Error Handling section should appear in the "Add report" modal and include:

• Enable Retries checkbox to toggle retry behavior on/off
• Maximum Retry Attempts field allowing users to specify the number of retry attempts (with min/max limits)
• Send Failed Reports checkbox/toggle to control whether failed reports are sent to all recipients (email and Slack) after retries are exhausted
• Failure Notifications checkboxes for selecting who receives notifications on each retry attempt (Owners and/or Report Recipients)
• Informational note explaining that recipients only receive reports when all charts successfully load

On each retry attempt, notification emails should be sent to selected recipients including explanatory text indicating the failure occurred, how many retry attempts remain, the error message (if available), and the screenshot of the chart/dashboard (matching the current report email format).

When all retries are exhausted and "Send Failed Reports" is enabled, the failed report should be sent to all configured recipients including:

• Email recipients (Report Recipients list)
• Slack channels (if configured for the report)

Email Notification Text

During Retry Attempts

Subject Line:
Report Retry [X of Y]: [Report Name] - [Date/Time]

Email Body:

---

Report Generation Failed - Retry in Progress

Your scheduled report "[Report Name]" encountered an error during generation. The system is automatically retrying.

Retry Status:

• Retries remaining: [X]

Error Details:
[Error message from the system, if available]

What's happening:
The system will automatically retry report generation. You will receive an update email after each retry attempt.

[If configured to send failed reports: "If all retry attempts are exhausted, you will receive the failed report with error details."]

[If NOT configured to send failed reports: "If all retry attempts are exhausted, the report will not be sent. You can manually regenerate the report or contact support for assistance."]

Report Preview:
Below is a screenshot of the chart/dashboard that failed to generate:

[Screenshot of chart/dashboard]

---

Final Failure Email/Slack Message (All Retries Exhausted - If Configured to Send)

Subject Line (Email):
Report Failed - All Retries Exhausted: [Report Name] - [Date/Time]

Email Body:

---

Report Generation Failed

Your scheduled report "[Report Name]" failed to generate after [X] retry attempts.

Error Details:
[Error message from the system, if available]

What happened:
The system automatically retried [X] times, but was unable to successfully generate all charts in this report.

Next Steps:

• Review the error details above to identify the issue
• Check your data source connections and filters
• Verify that all charts are properly configured
• Try generating the report manually to troubleshoot
• Contact support if the issue persists

Report Preview:
Below is a screenshot of the chart/dashboard that failed to generate:

[Screenshot of chart/dashboard]

---

Slack Message:
Similar format to email, adapted for Slack message formatting with error details and screenshot/chart preview.

---

Acceptance Criteria

• Users can enable or disable automatic retries via the "Enable Retries" checkbox in the Error Handling section
• When retries are enabled, users can configure the maximum number of retry attempts within defined min/max limits
• Users can select who receives retry notification emails (Owners, Report Recipients, or both)
• Users can enable/disable sending failed reports to all recipients (email and Slack) via a "Send Failed Reports" toggle/checkbox
• System triggers retries when the report itself encounters an error or when any individual chart has an error
• Retry attempts use an exponential backoff strategy (increasing wait time between attempts)
• A notification email is sent on each retry attempt to selected recipients (per Failure Notifications configuration)
• Each notification email includes the number of remaining retry attempts
• Each notification email indicates whether the failed report will be sent if all retries fail (based on "Send Failed Reports" configuration)
• Recipients only receive the final successful report when all charts successfully load
• When "Send Failed Reports" is enabled and all retries are exhausted, the failed report is sent to all configured email recipients
• When "Send Failed Reports" is enabled and all retries are exhausted, the failed report is sent to all configured Slack channels
• Notification emails include text explaining that the report failed, how many retries remain, the error message (when available), and the screenshot of the chart/dashboard in the same format as current report emails

Dependencies

• Existing report scheduling and execution system
• Notification delivery system for failure alerts
• Chart/dashboard execution engine
• Screenshot capture functionality for charts/dashboards
• Slack integration for report delivery

[eschutho]

Designs to come. We have someone working on this.