HTTP 200 but a Slow API: A Repeatable Monitoring Exercise

A successful HTTP status tells you that a request completed according to the server. It does not tell you that the response arrived quickly enough. A service that takes 1.2 seconds to return a health response may remain green in a status-only check.

This exercise separates three conditions: fast success, slow success and failure. It uses a disposable Python endpoint, not customer traffic. The Warden configuration references v0.9.0; timings below are configured delays, not benchmark results.

Run the endpoint

Download and inspect latency-lab.py. It uses Python 3’s standard library, serves only /health and binds to loopback by default.

python3 latency-lab.py

In another terminal in the same directory, select a mode and measure it:

printf 'fast' > latency-mode.txt
curl -s -o /dev/null -w 'status=%{http_code} seconds=%{time_total}\n' http://127.0.0.1:8088/health
printf 'slow' > latency-mode.txt
curl -s -o /dev/null -w 'status=%{http_code} seconds=%{time_total}\n' http://127.0.0.1:8088/health
printf 'down' > latency-mode.txt
curl -s -o /dev/null -w 'status=%{http_code} seconds=%{time_total}\n' http://127.0.0.1:8088/health

Fast and slow both return 200; down returns 503. Fast adds 25 ms of sleep and slow adds 1,200 ms. Scheduling and network time add to those delays.

Make the target reachable from Warden

For a native Warden process on the same host, use http://127.0.0.1:8088/health. Inside Docker, that address refers to the Warden container. Run the lab and Warden on an isolated shared container network, or deliberately expose the lab on a host interface reachable from that container with --bind. Do not expose this exercise as a production service.

Install Warden, add an HTTP monitor and expect status 200. Set a timeout longer than 1.2 seconds so the slow mode measures latency instead of timing out. Set an explicit latency threshold of 500 ms for this exercise. Record the confirmation count and sustained notification window before testing.

Observe each transition

Start in fast mode until the monitor reports up. Switch to slow and leave it there through the configured confirmation and sustained window. Look for degraded state while checks still receive 200. Restore fast, then repeat with down mode to distinguish failure from slowness.

Use a test notification destination you control. Verify delivery there; a dashboard state alone does not prove a message arrived. Record timestamps, response status, measured latency and messages. If there is no alert, check muted monitors, disabled events, channel settings and the sustained window before changing thresholds.

Repeat with learning

An explicit threshold takes precedence over the learned baseline. Remove it only when ready to evaluate adaptive behavior. Defaults require 200 successful checks and the next hourly calculation; do not describe a fresh monitor as already trained.

The baseline documentation explains the formula. This lab isolates response time; it does not measure browser rendering, user-request P99 or the health of every application dependency. Stop the Python process when finished and remove the disposable monitor.