Joe Freeman: Tech, Teams, Trade-offs

 Your content is only as good as it presents 💬 I've had plenty of blog articles that I thought were good enough to put out on the Internet. It was only after I tried to use them as a basis for a video that I realized how chaotic or incomplete they were. AI-Assisted Timeline 2 hours wall clock getting an LLM to create outlines for 6 blog articles 1 hour to generate a blog article from the outline 2 wall clock hours of iteration and ideation and proofreading 4 days of coffee decompression Return to project 6 wall clock hours of iteration with the AI and proofreading Published the article. Immediately found problems 2 wall clock hours of iteration and republished 1 day later LLM found problems while working on other article. 1 wall clock hour of proofreading and iteration 4 days of other stuff including coffee decompression Started video recording Unrecorded hours of swearing 1 wall clock hour of iteration and republished Unrecorded hours of moaning and banging head on table. 2 wall ...

This article was revised with later test data. Earlier figures and tables on this page were replaced. Each Ollama runner keeps a prefix cache that can skip re-evaluating a shared system prompt. Nine of seventeen local builds reused the cached system prompt on nearly every new conversation. Four others re-evaluated a 1,024-token batch of it each time. Four more re-evaluated all of it once per system prompt, up to 37 seconds on the largest. The model's memory type and runner set the group, not the prompt. In freemansoft/Flutter-AdaptiveCards a demonstration Dart chat server asks a local Ollama model for an answer as Adaptive Card JSON. That is a strict, closed-vocabulary schema, which a Flutter app renders as interactive UI. The card system prompt that describes the element vocabulary runs to about 3,800 to 4,400 tokens, by the model's tokenizer. The chat server replays up to ten prior exchanges on every turn. Each Ollama runner keeps a prefix cache, so a request whose open...

This article was revised with later test data. Earlier figures and tables on this page were replaced. A long conversation in the context window lowers structured-output scores on five of eight local models. Two Nemotron 30B models lose 8 of 25 cases each, and a parse check catches only one of their two failure types. Rank models at the context length you will run at, and check each reply against the specific question asked. In freemansoft/Flutter-AdaptiveCards a demonstration Dart chat server hands a question to a local Ollama model. It asks for the answer as Adaptive Card JSON, a strict, closed-vocabulary schema that a Flutter client renders as interactive UI rather than as text. A set of probes in that repository puts identical questions to different local models. context_fill_probe.dart produced every figure in this article. It runs 25 test cases, one question each. A test case passes only if the reply used an element type that would answer t...

This article contains updated prose and later test data than the original. Earlier figures and tables on this page were replaced. In freemansoft/Flutter-AdaptiveCards a demonstration Dart chat server asks a local Ollama model for Adaptive Card JSON, a strict, closed-vocabulary schema that a Flutter client renders as interactive UI. A set of probes measures how well local models manage that. Every probe ran with a nearly empty context window: a system prompt, one question, and at most a short seed exchange. A real conversation fills the context, and this article measures what Ollama does when it is full. The measurements ran on an Apple M1 Max with 64 GB under Ollama 0.35.1, and on an Apple M5 with 16 GB under Ollama 0.33.3. They produced three findings: Every window Ollama allocated in these runs was min(requested, trained window) . That is often not what the request asked for. The same text is 4.30 characters per token on one...

This article was completely regenerated using updated prose and later test data. Earlier figures and tables on this page were replaced. See the revision history at the bottom In freemansoft/Flutter-AdaptiveCards a demonstration Flutter client sends questions to a Dart chat server. The server passes each one to a local Ollama model and asks for the answer as Adaptive Card JSON. The client renders the card that comes back. To measure which models manage that, a directory of probes sends a fixed set of questions straight to Ollama, one model at a time. Each probe builds its requests the way the chat server does and judges every reply with the server's own card detector. Some recorded results came from the test setup rather than from a model. A busy machine caused one incident, the Ollama runtime others, the harness itself the rest. Each looked like model behavior. Each rule below stops one such mistake. Five rules come from a measurement that went wrong or near...