Posts

My blog whispered 'genius'. Turning the content into a video shouted 'chaotic dreck'

Image
 Your content is only as good as it presents 💬 I've had plenty of blog articles that I thought were good enough to put out on the Internet. It was only after I tried to use them as a basis for a video that I realized how chaotic or incomplete they were. AI-Assisted Timeline 2 hours wall clock getting an LLM to create outlines for 6 blog articles 1 hour to generate a blog article from the outline 2 wall clock hours of iteration and ideation and proofreading 4 days of coffee decompression Return to project 6 wall clock hours of iteration with the AI and proofreading Published the article. Immediately found problems 2 wall clock hours of iteration and republished 1 day later LLM found problems while working on other article. 1 wall clock hour of proofreading and iteration 4 days of other stuff including coffee decompression Started video recording Unrecorded hours of swearing 1 wall clock hour of iteration and republished Unrecorded hours of moaning and banging head on table. 2 wall ...
Read more

On Ollama, a model's memory type and its runner decide how much of the cached prompt a new conversation reuses - 8

Image
In freemansoft/Flutter-AdaptiveCards a demonstration Dart chat server asks a local Ollama model for an answer as Adaptive Card JSON. That is a strict, closed-vocabulary schema, which a Flutter app renders as interactive UI. The card system prompt that describes the element vocabulary is about 3,755 tokens. The chat server replays up to ten prior exchanges on every turn. Each Ollama runner keeps a prefix cache, so a request whose opening tokens match an earlier one does not have to process them again. Ollama 0.33.3 and later report, in prompt_eval_cached_count , how many of a reply's prompt tokens came from that cache. A chat server can read from that field how much of each prompt the cache served and where it missed. Every reading below comes from ModelBehavior.md , the lab notebook in that repository. A new conversation reuses a cached system prompt on some models and not on others. The split follows how the model stores its context. A pr...
Read more

A full context breaks three local models, each in a different way - 7

Image
In freemansoft/Flutter-AdaptiveCards a demonstration Dart chat server hands a question to a local Ollama model. It asks for the answer as Adaptive Card JSON, a strict, closed-vocabulary schema that a Flutter client renders as interactive UI rather than as text. A set of probes in that repository puts identical questions to different local models. Probe context_fill_probe.dart  produced every figure in this article. It runs 25 test cases, one question each. A test case passes only if the reply used an element type that would answer the question. Every other probe in that set had been asking into a nearly empty window. That window holds a system prompt, one question, and at most a short canned exchange showing the model the shape of a card. The literature documents that a long context degrades model behavior. What it does to a model's adherence to an output format is measured less often. The test probe simulates a long conversation by filling three q...
Read more

Ollama silently drops a history message larger than its context window - 6

Image
In freemansoft/Flutter-AdaptiveCards a demonstration Dart chat server hands a question to a local Ollama model. It asks for the answer as Adaptive Card JSON, a strict, closed-vocabulary schema that a Flutter client renders as interactive UI rather than as text. A directory of probes measures which local models manage that and how well. Every one of those probes had been asking its question into a nearly empty context window. Each call carried a system prompt, one question, and at most a short seed exchange. A real conversation fills the window. What changes when it is full? Two findings came out of filling it, measured on an Apple M1 Max with 64 GB and an Apple M5 with 16 GB. Both hosts ran Ollama 0.33.3, and the M1 Max results reproduced under Ollama 0.34.0. What the runner allocates follows one rule , with no counterexample in fifty-one runs across the two Ollama versions. It is often not what the request asked for. A history m...
Read more

Eight measurement rules from a local-model benchmark on Ollama - 5

Image
In freemansoft/Flutter-AdaptiveCards  a demonstration Flutter client sends questions to a Dart chat server. The server passes each one to a local Ollama model and asks for the answer as Adaptive Card JSON. The client renders the card that comes back. To measure which models manage that, a directory of probes sends a fixed set of questions straight to Ollama, one model at a time. Each probe builds its requests the way the chat server does and judges every reply with the server's own card detector. Several of those results looked like something a model did when the cause was the test setup: the machine, the Ollama runtime, the harness or the probe. Each rule below checks for one of those mistakes before it reaches a published number. Five of the eight rules come from a measurement that went wrong. The other three guard against a known weakness in the setup: the per-call time limit, the point in a long run where a model is measured, and what the card detector cann...
Read more