A notification channel can pass its connection test while a real outage remains silent. The channel test answers whether Warden can send a message to that destination. It does not exercise the monitor, its expected response, confirmation policy or sustained-outage window.
Use a disposable endpoint and a destination you control. Keep production running throughout this test. These steps follow Warden v0.9.0 notification behavior.
Write down the expected behavior
Before changing anything, record the monitor interval, expected HTTP status, timeout, confirmation count and sustained window. Also record whether the monitor is muted and whether Down and Recovered events are enabled.
Warden’s default sustained window is 180 seconds after the confirmed outage opens. Polling and confirmation add time before that. A 30-second test failure may therefore be correctly recorded without interrupting anyone.
Test the destination first
In Settings → Notifications, add or edit your email, Slack, Discord, Telegram or webhook channel. Send a test and inspect the actual destination. Check the recipient or channel, message readability and whether your device surfaces it as expected. Save the channel and confirm it is enabled.
Channels are global in this version. Use a separate test instance if a real outage drill would disturb production recipients. Selecting a monitor does not give it an independent set of channels.
Test a sustained failure
Use the endpoint from the slow-API exercise, initially in fast mode. Add the monitor and wait for successful checks. Switch the endpoint to down mode and leave it there until the configured policy should announce it.
Capture four times: when you changed the endpoint, the first failed check, the confirmed outage and the notification’s arrival. Those distinguish polling delay from policy delay and delivery delay. Do not infer all four from the message timestamp.
Restore fast mode. Confirm recovery in Warden and at the destination. Recovery notifications depend on the outage having been announced; a short unannounced blip should not produce an unexpected recovery message.
Test a short blip separately
After the monitor stabilizes, repeat with a failure shorter than the sustained window. Expect history to preserve what happened even if no notification is sent. Compare this with the sustained run rather than treating silence as a failed test by itself.
If results differ from the policy, inspect event toggles, maintenance, muting, flapping, correlation and repeat-offender damping. Repeated drills can themselves activate noise controls. Keep the original settings so you can restore them.
Keep a small evidence record
Save the version, settings, timestamps and redacted messages. Exclude webhook URLs, tokens and private endpoints. Recheck delivery after changing providers or recipients, and after restoring a backup. Start with one monitor if you have not yet completed a full failure-and-recovery cycle.