Your Mac Is the Datacenter Now
I shipped local AI in Azynote 1.3.0 two days ago. Transcription on your machine, summaries on your machine, search on your machine. Nothing about your work leaving your Mac at any point.
I had been building and testing it on a 64 GB MacBook Pro, where it felt finished.
Then I opened it on my 16 GB one.
Same app, same model, same settings. But here I would run a single Smart Action, proofread one paragraph, watch it come back in a second and a half, and then watch seven gigabytes stay gone for the rest of the day. The model had loaded itself and there was nothing in the product that could make it let go. Quitting Azynote was the only way to get my memory back.
It was not a bug in the sense of something being broken. Every line did what it was written to do. The feature was simply unusable on the machine most people would run it on, and I could not see that from the machine I built it on.
The bill nobody puts on the invoice
The conversation about local-first is almost entirely about privacy, and privacy is the part I care about most. But privacy is the benefit. It is not the thing you take on.
What you take on is operations.
When you call a cloud model, someone else is running a datacenter for you, and a large part of what they do is lifecycle management. Your model gets loaded onto a GPU before your request and evicted after it. Capacity is shared across thousands of people, so nothing sits idle holding memory it is not using. You never think about any of this, because the entire point of the service is that you do not have to.
Move it on-device and you inherit all of it. Except the datacenter is now a laptop, the only tenant is you, and the operator is an app that was never designed to be one.
There is a second thing you inherit, and it is the one that got me. In the cloud, the hardware is uniform and it is yours. Here the hardware belongs to the user, and it varies by a factor of four in the same product line. The machine is no longer a detail underneath the software. It is part of the software's behaviour.
Azynote already knew this in one direction. It sizes the model's context window from the memory each machine actually has, so a bigger Mac gets a much larger working window out of the same model file. Which produces a result I find genuinely funny in hindsight: my 64 GB Mac was holding 13.3 GB for that idle model, more than twice what the 16 GB one held at 6.3.
The machine carrying the heavier load was the machine with no reason to care. The one that would have told me immediately was the one I was not using.
Two wrong answers
The first fix that came to mind was to unload after every action. You proofread a paragraph, we hand the memory straight back. Clean.
The numbers say no. Loading the weights takes between 2.6 and 11.9 seconds depending on what else is competing for the GPU. Smart Actions are exactly where that hurts most, because a text transform has almost no other cost to hide it behind. You would be trading a memory problem you notice once a day for a stall you notice twenty times a day.
The second idea was to show the running processes and let people stop them. A little panel, a list, a stop button.
I sat with that one longer than it deserved. It is wrong for a quieter reason: it answers a question nobody asked. Nobody wants to manage llama-server processes. The unit people think in is not a process, it is "the AI is loaded and it is costing me memory". Building a process manager means exporting my implementation into someone else's head and asking them to care about a word they have never needed to know.
What it actually needed
Release the model when it goes unused. Five minutes without a request and the memory goes back. This is not clever. Ollama has done it for years. It is simply the piece I had not written, because on the machine I was testing on, nothing ever made me want it.
Make the cost visible while it is being paid. This is the part I had wrong in my own head. I assumed the complaint was "I cannot stop the model". Sitting with it properly, the real complaint came earlier than that: I did not know it was running.
So there is now a quiet line at the top of the window whenever a model is loaded, and it says how much memory it is holding right now. Not in settings. Not behind a click. On screen, while it is true. Clicking it shows what is loaded and offers to free it immediately.
If I am asking you to be the operator of your own inference, the least I can do is show you the one number an operator needs.
Neither of these is a hard thing to build. That is sort of the point. The real texture of local-first is not moving a call from their machine to yours. It is inheriting a set of small problems that used to be somebody else's full-time job, and then solving them inside an app that also has to be pleasant to use.
Where I think this goes
I do not think I am finished. Releasing on a timer is the simple answer, and simple answers are usually the first half of something.
What I would want eventually is for the app to respond to actual memory pressure rather than to a clock. If you open Xcode and a simulator and your Mac starts breathing hard, the model should get out of the way without being asked. I have not built that, because I do not yet have evidence the timer is insufficient, and I have learned to distrust the guard I add before the problem shows up.
The broader lesson I am taking is duller and more useful: if the user's machine is the infrastructure, then testing on one machine is not testing. I will be doing a lot more of my own work on the 16 GB Mac.
If you run Azynote with the Local model, on a 16 GB machine, this is exactly where I would like to hear from you. Does five minutes feel right, or does it let go while you are still thinking? Does the memory line read as useful, or as one more thing on your screen? I built both from one afternoon of my own annoyance, which is a sample size of one.
Happy meetings, and may your RAM be free.