A local DNS cache that does not lie
Every machine here used to resolve names through whatever the hosting provider handed out. It worked until the day one upstream started answering slowly, and every outbound request quietly grew a two second tail.
Symptoms
Deploys felt sluggish, package updates stalled at “connecting”, and the graphs showed nothing because the latency lived below the application.
$ time getent hosts example.org
real 0m2.014s
The setup that held
- One caching resolver per host, bound to loopback only.
- Two upstreams from different networks, queried in parallel, first answer wins.
- Negative answers cached for at most sixty seconds.
- Serving stale records is allowed for an hour, but each stale answer is logged.
What I would do differently
Measure name resolution separately from the request it belongs to. A single histogram for lookup time would have shortened the investigation from an evening to five minutes.