You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Enhanced Alert Configuration with Multiple, Escalating Alerts per Status
#494
I'd like to propose an enhancement to the alert configuration system that would significantly improve our ability to manage service status changes and automate appropriate responses.
Current Limitation:
Currently, we are limited to a single alert configuration per status type (e.g., one "DOWN" alert, one DEGRADED alert). This can restrict our ability to implement nuanced and escalating responses to service degradation or outages.
Proposed Feature: Multiple, Configurable Alerts per Status
My suggestion is to allow users to configure zero to N (0-N) distinct alerts for a given service status, such as DOWN or DEGRADED. Each of these individual alerts would possess its own independent configuration settings, including:
Type: The status type it monitors (e.g., DOWN, DEGRADED).
Failure Threshold: The number of consecutive failures (or duration) required to trigger this specific alert.
Success Threshold: The number of consecutive successes required to resolve this specific alert.
Create Incident: A flag to determine if this alert should automatically create a formal incident within the system.
Message: A custom message for this specific alert.
Triggers: A list of actions to execute when this alert is triggered (e.g., send email, fire a webhook, send Slack message).
Benefits of this Approach:
This flexible configuration would enable us to implement sophisticated, escalating alert patterns tailored to the severity and duration of an issue. This offers several key advantages:
Nuanced Response: Differentiate between a brief, transient blip and a sustained outage, triggering different actions for each.
Automated Escalation: Ensure the right teams are notified and the appropriate response actions are initiated as an issue persists or worsens.
Tailored Notifications: Send targeted notifications to different stakeholders (e.g., internal teams, on-call engineers, management) based on the severity.
Illustrative Example: Escalating "DOWN" Alerts
Consider the following scenario for a service experiencing DOWN status, demonstrating how multiple alerts could work in an escalating fashion:
Create Incident: NO (as the primary incident should already be active)
Triggers:webhook (to an incident management system like PagerDuty or Opsgenie, triggering on-call rotations and critical response procedures)
Purpose: For severe and prolonged outages, this alert ensures immediate, high-priority action is taken by the on-call team and triggers full incident response workflows.
This proposed functionality would empower users with much greater control and automation capabilities for their monitoring and incident response strategies.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
I'd like to propose an enhancement to the alert configuration system that would significantly improve our ability to manage service status changes and automate appropriate responses.
Current Limitation:
Currently, we are limited to a single alert configuration per status type (e.g., one "DOWN" alert, one DEGRADED alert). This can restrict our ability to implement nuanced and escalating responses to service degradation or outages.
Proposed Feature: Multiple, Configurable Alerts per Status
My suggestion is to allow users to configure zero to N (0-N) distinct alerts for a given service status, such as
DOWNorDEGRADED. Each of these individual alerts would possess its own independent configuration settings, including:DOWN,DEGRADED).Benefits of this Approach:
This flexible configuration would enable us to implement sophisticated, escalating alert patterns tailored to the severity and duration of an issue. This offers several key advantages:
Illustrative Example: Escalating "DOWN" Alerts
Consider the following scenario for a service experiencing
DOWNstatus, demonstrating how multiple alerts could work in an escalating fashion:First Alert - Initial Blip Notification
DOWNSecond Alert - Sustained Issue Notification
DOWNemail(to a broader team distribution list for awareness, but not yet a high-priority PagerDuty alert)DOWNstatus persists beyond a minor blip, this alert notifies a wider group that the problem is not self-correcting.Third Alert - Critical Outage Escalation
DOWNwebhook(to an incident management system like PagerDuty or Opsgenie, triggering on-call rotations and critical response procedures)This proposed functionality would empower users with much greater control and automation capabilities for their monitoring and incident response strategies.
All reactions