<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Field notes by Bala Sista</title><link>https://balasista.com/notes</link><atom:link href="https://balasista.com/feed.xml" rel="self" type="application/rss+xml"/><description>The real mistakes behind the rules for building and running AI systems in production.</description><language>en-us</language><item><title>I added one word and the menu broke</title><link>https://balasista.com/notes/i-added-one-word-and-the-menu-broke</link><guid isPermaLink="true">https://balasista.com/notes/i-added-one-word-and-the-menu-broke</guid><pubDate>Tue, 13 Oct 2026 12:00:00 GMT</pubDate><description>A new link passed every check. The menu it sat in was running off the screen at sizes nobody checked.</description><content:encoded><![CDATA[<p>On the evening of October 13 an agent added a link called Now to the menu on this site, and checked its work the usual way: the new page loaded, the link went to it, and the menu fit on a phone and on a laptop. It went live.</p><p>The menu did not fit between those two. On any screen from about 480 to 560 pixels wide, which covers a phone turned sideways and some small tablets, it ran past the right edge of the page. Nothing flagged it, because nothing looked there.</p><p>It turned up 45 minutes later, by luck. I asked for one more link, and this time the agent checked the menu at a wider range of sizes. The overflow showed up at widths where the new link was not even on screen. The fix went live the same evening.</p><p>The new link was fine. What broke was the thing next to it, which I had not touched and so had not checked. That is the most common way my work goes wrong. In my own working history, breaking something that used to work comes up more often than any other complaint.</p><p>So the question after a change is no longer whether the new thing works. It is whether everything around it still works, including the parts nobody asked me to change.</p><h2>What I check now</h2><ul><li>After a layout change, the page at every common screen size, not only the smallest and the largest.</li><li>After a back-end change, the screens that read from it. After a screen change, the calls it makes.</li><li>The features that worked before the change, not only the one that was added.</li><li>If a fix does not work the first time, the cause was wrong. I go back to the cause instead of fixing the same thing again.</li></ul>]]></content:encoded></item><item><title>The deploy that worked until it did not</title><link>https://balasista.com/notes/the-deploy-that-worked-until-it-didnt</link><guid isPermaLink="true">https://balasista.com/notes/the-deploy-that-worked-until-it-didnt</guid><pubDate>Tue, 13 Oct 2026 12:00:00 GMT</pubDate><description>A pipeline stage passed for days, then timed out with no code change. It had been waiting for a person to click a link.</description><content:encoded><![CDATA[<p><em>From client work with the team at <a href="https://codvo.ai">CODVO.AI</a>. Client details are left out.</em></p><p>In August 2026 a deploy pipeline started failing on a stage that had passed on every merge for days. Nothing in the code had changed. The job sat waiting, then timed out and cancelled itself.</p><p>That stage connected from a build server to the target machine by SSH over our private network. The network's access rule for that machine asked a person to confirm in a browser every so often. A pipeline has no browser. While an earlier confirmation was still fresh, the job got through. Once it lapsed, the job waited on a login link nobody would ever open.</p><p>It was slow to find because SSH from my own machine worked instantly every time. My session had its own fresh confirmation, so the failure looked like a fault on that one server, or a flaky network.</p><p>It was neither. A login step that expects a person cannot work with a job that has none, and a retry or a longer timeout only changes when it fails.</p><p>The fix moved the job instead of the connection. A build agent now runs as a service on the target machine, and the stages that touch that machine run there, locally, with no SSH step. Image builds stay on the build server.</p><h2>What changed</h2><ul><li>A pipeline never reaches a deploy target by SSH over the private network.</li><li>Each target that needs deploy steps gets its own build agent, in a pool of its own.</li><li>Image builds stay off machines that serve live traffic.</li></ul>]]></content:encoded></item><item><title>The laptop did the work while the server sat idle</title><link>https://balasista.com/notes/the-laptop-did-the-work</link><guid isPermaLink="true">https://balasista.com/notes/the-laptop-did-the-work</guid><pubDate>Tue, 13 Oct 2026 12:00:00 GMT</pubDate><description>A two-day data job pinned my laptop. A far bigger server on the same network had nothing to do.</description><content:encoded><![CDATA[<p><em>From client work with the team at <a href="https://codvo.ai">CODVO.AI</a>. Client details are left out.</em></p><p>Over two days in August 2026 we regenerated a large training dataset: 30 equipment families, 11.8 billion rows and about 96 GB, followed by 60 model trainings.</p><p>It ran on my laptop with 16 to 20 worker processes at once. The machine sat at a load average of 15 and was useless for anything else. Partway through, I deleted finished datasets so the disk would not fill.</p><p>A server on the same private network had several times the memory, a GPU and hundreds of gigabytes of free disk. Its load average for those two days was 0.01.</p><p>Nobody chose the laptop. It was the terminal that happened to be open, so the agents started there, and nothing asked where the work belonged.</p><p>The GPU was not the reason to move. Most of that job was array arithmetic that a GPU does nothing for. The reasons were disk space, memory and a laptop I could still use.</p><h2>The rule now</h2><ul><li>Before a job that will run for more than a few minutes or write more than a few GB, the agent asks where it should run. The default answer is the server.</li><li>The laptop is for editing, tests and short checks.</li><li>Code reaches the server through git: push from the laptop, pull on the server. Nothing gets copied across by hand.</li><li>A job moves for the GPU only when it can use one.</li></ul>]]></content:encoded></item><item><title>Three days of work lived in /tmp</title><link>https://balasista.com/notes/three-days-in-tmp</link><guid isPermaLink="true">https://balasista.com/notes/three-days-in-tmp</guid><pubDate>Mon, 12 Oct 2026 12:00:00 GMT</pubDate><description>A laptop restarted and took five branches with it. They had been committed, but never pushed.</description><content:encoded><![CDATA[<p><em>From client work with the team at <a href="https://codvo.ai">CODVO.AI</a>. Client details are left out.</em></p><p>On October 7, 2026 my laptop restarted. The operating system cleared its temporary folder, as it does on every restart.</p><p>One of the things in that folder was a clone of a repository. Agents had been working in it for three days across five related projects, each on its own branch, plus a branch that integrated them. Every branch had commits. None had been pushed.</p><p>All of it was gone. The work came back only because the agents' transcripts still held the specifications they had been building from, and we rebuilt the branches from those. Repositories that lived in my normal workspace folder lost nothing.</p><p>The cost was a day. The cause was a clone made in the wrong folder and branches that were never pushed.</p><h2>What changed</h2><ul><li>Clones and working copies live under the project folder, never in a temporary one.</li><li>A branch is pushed as soon as it has one commit worth keeping.</li><li>A workflow with several agents pushes after every stage, not at the end.</li><li>Before anyone says the work is safe, they check that no commit exists only on the laptop.</li></ul>]]></content:encoded></item><item><title>Fast everywhere except one place</title><link>https://balasista.com/notes/fast-everywhere-except-one-place</link><guid isPermaLink="true">https://balasista.com/notes/fast-everywhere-except-one-place</guid><pubDate>Mon, 12 Oct 2026 12:00:00 GMT</pubDate><description>During an incident, an agent changed a production database without asking. Now every write waits for a yes.</description><content:encoded><![CDATA[<p><em>From client work with the team at <a href="https://codvo.ai">CODVO.AI</a>. Client details are left out.</em></p><p>I let agents move quickly. They read code, run tests, open branches and rebuild containers without waiting for me, and most of the time that is the right setting.</p><p>During one incident an agent ran several updates and a schema change against a production database with no confirmation step. Nobody had looked at them first, and a production write is the one kind of mistake a revert does not undo.</p><p>I did not respond by banning writes. An agent that cannot change production data is not much use in an incident. What I took away was its ability to do it quietly.</p><p>The rule also covers the moment that caused it. An incident is when the pressure to skip the step is highest, and it is also when a wrong write does the most damage. So the rule states that incident pressure does not waive it.</p><h2>The gate</h2><ul><li>Reads run freely. The agent says what it is about to run, then runs it.</li><li>Before any write, the agent shows the exact statement and names the database it points at.</li><li>Before an update or delete, it runs a count with the same filter, so the number of affected rows is known first.</li><li>Approval is per statement. A yes to one write says nothing about the next.</li><li>Where the database supports it, the write goes inside a transaction.</li><li>Local and test databases need none of this.</li></ul>]]></content:encoded></item><item><title>The count was right and the pins were 400 km away</title><link>https://balasista.com/notes/pins-400-km-away</link><guid isPermaLink="true">https://balasista.com/notes/pins-400-km-away</guid><pubDate>Mon, 12 Oct 2026 12:00:00 GMT</pubDate><description>A demo dataset gave every record a believable county and a believable location. The two had never met.</description><content:encoded><![CDATA[<p><em>From client work with the team at <a href="https://codvo.ai">CODVO.AI</a>. Client details are left out.</em></p><p>In July 2026 a map sandbox answered a simple question, how many wells are in one county, with a number that looked right. Then it drew the pins. They landed about 400 km away, near a large city.</p><p>The data was synthetic, and the generator had assigned each record a county name at random. It had also assigned each record a position. Each value was plausible alone. Together they described wells that sat in one county and claimed to be in another. Every county's records covered the same rectangle of the map.</p><p>The part that cost the most time was the test. An acceptance check compared the count in the app with the count in the database, and they agreed every time. Both were reading the same invented column, so the check passed while every geographic answer was wrong.</p><p>The fix was small. The positions were fine and the labels were wrong, so we threw the labels away and worked out each county from the point, using real county boundaries. A county should never have been a value someone typed in. It is a fact about where the point is.</p><p>A person who knows the region sees this kind of error in one glance, and after that they stop trusting the count, the chart and the next demo as well.</p><h2>How we build demo data now</h2><ul><li>If one field depends on another, we compute it. County comes from the point and age comes from the birth date.</li><li>We look up the real totals and ranges before generating anything, then scale to them.</li><li>We give every region a user might ask about some data, unless the real total is so small that most regions truly have none. Then zero is the honest answer.</li><li>We seed the generator so the same input gives the same dataset on every rebuild.</li><li>We validate against something outside the system: geometry, a reference layer or a published figure.</li></ul>]]></content:encoded></item><item><title>A 200 that carried nothing</title><link>https://balasista.com/notes/a-200-that-carried-nothing</link><guid isPermaLink="true">https://balasista.com/notes/a-200-that-carried-nothing</guid><pubDate>Mon, 12 Oct 2026 12:00:00 GMT</pubDate><description>Four features were called done in one day. All four had been checked against a stand-in for the real thing.</description><content:encoded><![CDATA[<p><em>From client work with the team at <a href="https://codvo.ai">CODVO.AI</a>. Client details are left out.</em></p><p>In one working day, four pieces of work were reported as done and were not. We went back through them expecting four different causes and found one.</p><p>The first was an API feature checked with a command that printed only the HTTP status. It printed 200. The response body was empty, and nobody had read it.</p><p>The second was wired to one endpoint and tested there. The product calls a different one, the streaming version, so the test passed on a path no user ever takes.</p><p>The third was a build of two services chained with a start command. The second build failed, the chain stopped, and the test ran against a stale copy of the first service.</p><p>The fourth was a change to a screen, declared finished three times without anyone opening a browser. The first version shipped a control that was invisible. The fix shipped one that covered the text underneath it. An element can be present in the page structure and still be invisible to a person.</p><p>Each check was real and each one passed. Every one of them tested something next to the feature: a status line, a sibling endpoint, a stale container, a tree of elements. Since then the rule has been to check the thing itself.</p><h2>What we check now</h2><ul><li>For a screen, we open it in a browser and look. Present in the page structure is not the same as visible to a person.</li><li>For an API, we read the payload, and we call the endpoint the product calls.</li><li>For configuration, we read the value back from the running service, never from the source file or the deploy log.</li><li>For a count, we compare it with a figure computed somewhere else.</li></ul>]]></content:encoded></item></channel></rss>