The deploy that worked until it did not

In August 2026 a deploy pipeline started failing on a stage that had passed on every merge for days. Nothing in the code had changed. The job sat waiting, then timed out and cancelled itself.

That stage connected from a build server to the target machine by SSH over our private network. The network's access rule for that machine asked a person to confirm in a browser every so often. A pipeline has no browser. While an earlier confirmation was still fresh, the job got through. Once it lapsed, the job waited on a login link nobody would ever open.

It was slow to find because SSH from my own machine worked instantly every time. My session had its own fresh confirmation, so the failure looked like a fault on that one server, or a flaky network.

It was neither. A login step that expects a person cannot work with a job that has none, and a retry or a longer timeout only changes when it fails.

The fix moved the job instead of the connection. A build agent now runs as a service on the target machine, and the stages that touch that machine run there, locally, with no SSH step. Image builds stay on the build server.