Posts

Benchmarking local model LLM generated Adaptive Card JSON on a 64 GB M1 Max and 16 GB M5

Image
In freemansoft/Flutter-AdaptiveCards  a demonstration Dart chat server hands a question to a local Ollama model and asks for the answer as Adaptive Card JSON, a strict, closed-vocabulary schema, which a Flutter client renders as interactive UI rather than as text. We built a set of test probes that measure how well fifteen local models manage that well and how fast. We built the probes on a 64 GB machine and then compared the results to a machine with a quarter the memory. Two machines, one set of test probes Two Apple machines ran the same probes and the same prompts on a 64 GB M1 Max MacBook Pro 14-inch ( MacBookPro18,4 ) and a fanless 16 GB M5 MacBook Air ( Mac17,3 ) used for comparison. The goal was to validate execution and measure performance differences when running the same models on two Apple Silicon chips with different memory sizes and bandwidth. Both hosts ran models back-to-back for hours, so each median below carries whatever position in that r...
Post a Comment
Read more

I accidentally used Fable to do a system upgrade and it was Fableous

Image
A Flutter SDK upgrade I've done several times has a long manual tail that I've never been able to hand off.  The tail includes deciding which golden-image failures are real and handling the checked-in images. It includes shuttling CI-generated Linux images back into the repo as new goldens. This time the agent did the whole workflow unprompted. It is the same repo, same skills , no new guidance . The only difference was that the session was left set to Fable from previous work instead of Opus or Sonnet. I upgraded a multi-platform Flutter library, all its tools and examples , to the latest Dart (3.13) , Flutter (3.47) , and mobile integration patterns using Claude Code .  Using an LLM to do an upgrade in itself isn't that interesting. The interesting part is that I had switched to Fable, and Fable did way more with the same repo than I had seen in the past, with zero repo or skill changes. It felt like I handed off the work to someone. The work itself was routine: a multi...
Post a Comment
Read more

We tried 14 levers to get reliable SDUI card JSON from a local LLM model

Image
In freemansoft/Flutter-AdaptiveCards a demonstration Dart chat server hands a question to a local Ollama model, asks for the answer as Adaptive Card JSON, and a Flutter client app renders the reply. Everything below is an attempt to make that card generation more reliable and more faithful to what was asked. The figures are transcribed from ModelBehavior.md , a lab notebook in that repository. Giving the explanation a place in the card worked; banning prose did not qwen2.5-coder:7b  answered a request to explain a snippet of code with a valid Adaptive Card, then appended the explanation after it. A reply is either a card or prose, with nothing in between: the client renders a card only when the entire reply is one, so appending the explanation demoted the whole thing to text and the user saw raw JSON. The obvious repair was to tell the model harder not to write anything after the card. That did not work. It scored the same and stopped producing cards at all,...
Post a Comment
Read more