Blog de Joe Freeman

This is about running VSCode AI code assist locally replacing Copilot or some other service.  You may run local models to guarantee none of your code ends up on external servers. Or, you may not want to maintain an ongoing AI subscription. We are going to use  LM  Studio and VS Code.  This was tested on Windows 11 with an RTX 3060 TI with 8GB of VRAM. 8GB really limits the number and size of the models we can use. LM Studio's simple hosting model of 1 LLM and an embedding works for us in this situation.  You want a big card. 8GB is a tiny card. Related blog articles and videos Several related blogs and videos that cover VSCode and local LLMs Blog  Get AI code assist VSCode with local LLMs using Ollama and the Continue.dev extension - Mac Get AI code assist VSCode with local LLMs using LM Studio and the Continue.dev extension - Windows Rocking an older Titan RTX 24GB as my local AI Code assist on Windows 11, Ollama and VS Code YouTube Video  Using loc...

This is about running VSCode AI code assist locally as a replacement for Copilot or some other service.  You may run local models to guarantee none of your code ends up on external servers. Or, you may not want to maintain an ongoing AI subscription. We are going to use Ollama  as our LLM service and the continue.dev  VSCode extension as the language service inside VSCode.   This was tested on a MacBook using the Apple GPU. Macs are an interesting platform for running local AI code assist and LLMs because you can treat much of main memory as GPU VRAM. Related blog articles and videos Several related blogs and videos that cover VSCode and local LLMs Blog  Get AI code assist VSCode with local LLMs using Ollama and the Continue.dev extension - Mac Get AI code assist VSCode with local LLMs using LM Studio and the Continue.dev extension - Windows Rocking an older Titan RTX 24GB as my local AI Code assist on Windows 11, Ollama and VS Code YouTube Video ...

Newer LLM models are built around the bfloat16  data type that has different types of precision than the older float16.  My Tesla vintage GPU supports the lower precision  float16, not the newer bfloat16.   you can coerce the model from bfloat16 to float16.  Disclaimer: The difference in precision can result in errors. Your mileage and accuracy may vary depending on the model  Hacking config.json for FP16 The basic steps are  Download the model either manually or as part of an attempted run.   Find the location of the model on disk. This typically is in the model cache directory ~/.cache/nvidia/nvidia-nims/ngc/<some_path>/config.json ~/.cache/nim/ngc/<some_path>/config.json ~/.cache/nvidia-nims/ngc/hub/<some-model>/snapshots/config.json Edit the config.json found for the model. Find the line  "torch_dtype":"bfloat16" Change the value of  torch_dtype  to one supported by the card, float16  in my cas...

Verify that your container can communicate with the GPU and the type of NVIDIA GPUs your Docker containers can access using one of NVIDIA's container images. docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi  Sample Output The containers on this server have access to a single Titan RTX card Source This came from  NVIDIA getting started with large language models Revision  Created 2024/07  Corrected NVIDIA captialization 2025/08

NVIDIA AI Workbench runs inside a containerized environment, and I wanted an environment check that verifies the container has access and that Docker/podman, the NVIDIA driver, and Workbench are all on compatible versions.   Containerization has no effect on performance. The CUDA code is pretty much a direct pass-through to the card. Environment NVIDIA AI Workbench One local GPU Windows 11  Docker Desktop Program running in a container built by the Workbench based on the PyTorch/Cuda image Program I access the containerized environment via a Jupyter Notebook visible to the browser on the Windows machine. This is a snapshot of the Jupyter Notebook. numpy_cupy_sort.ipynb Gist It found a problem The program demonstrated that there was a container adapter (or something) mismatch that recently happened.  cupy returned that it had access to the GPU. It turns out that it really did not, and that a Docker Desktop upgrade was needed to fix something driver-related.  The ...

We try to prevent or catch failures in all kinds of everyday tasks.  The cat escapes. The dog eats the socks.  Someone leaves the garage door up at night. Failure mode analysis (FMEA) works for all kinds of problems.  We can apply a version of FMEA to the garage door problem and use it as an example that everyone understands and potentially solves an everyday problem. The talk I'm finally getting around to posting the content to this 2022 talk:  Working through all the ways to fix the problem that I don't always close the garage door Talk content These are the contents of the Keynote presentation used in the talk.  See the talk for an explanation. Related This was part of a series of FMEA talks  Blog Articles Throwing down failures  http://joe.blog.freemansoft.com/2021/01/failure-mode-analysis-ste-one-throwing.html Detection and remediation  https://joe.blog.freemansoft.com/2021/02/failure-mode-analysis-step-two.html Videos Step 1: Throwing do...