Blog de Joe Freeman

Data science and AI projects are typically run on rented CPU/GPU resources or in self-managed data centers. By using containerization, we can separate our code and configuration from the underlying hardware, making it easier to switch between different systems for cost or performance reasons. However, containerization and remote execution introduce new networking and connectivity challenges. In this talk, we will explore how to manage connectivity for both the control plane and the user experience plane, including requirements like web access. AI transcription of the talk This section contains Gemini's rewrite of the YouTube Video's transcript. The raw text is down below.  The Role of Containerization in Local and Remote Data Science When designing a GPU-bound data exploration or analytics environment , the key to maintaining consistency—whether you're working locally or remotely —is a containerized workflow . For quick prototyping, we often leverage a few local GPUs witho...

 I wanted to run an ML project in a bigger environment than I have in my homelab.  I don't have any cloud accounts but I can rent GPU capacity via the Brev GPU/CPU marketplace. I just needed to get it there. The project itself is fully containerized and available on git as an AI Workbench project.  This means I just needed to install the AI Workbench server code on a Brev-rented GPU instance and then attach to that server over the provided network tunnel.  My local AI Workbench can then manage and run my project remotely from my MacBook. All the steps Some of these steps will be automated in future NVIDIA AI workbench releases.  Use this diagram to follow along in the video. Video Revision History Created 2024/11 Fixed NVIDIA capitalization 2025/08

The NVIDIA AI Workbench team has created a prototypical workspace for those wanting to participate in Kaggle competitions while running on their own hardware.  NVIDIA created a workbench project,  NVIDIA Competition Kernel on GitHub , that integrates Kaggle's GPU container images along with Jupyter Notebooks that interface with Kaggle. The Kaggle images are the same ones they use in their data scientist environment. The project is configured to participate in the continuous, training, Digits competition where you train a model to recognize handwritten numerical digits. NVIDIA created the project in a way that can be used in other competitions of the same class.  Three Notebooks Downloads the Training and Submission data Creates a model and trains the model using the training data. The notebook tunes the model against the training data. Then, it evaluates the Test data to generate what it thinks are the matches. The matches are written to a file. Uploads the CSV file conta...

I struggled initially to manage my ML and data science projects when running them in the NVIDIA Workbench. The built-in GIT interface was basic  and fine for basic operations but I found it frustrating for branching and merging.  I ended up using tools that ran inside the Workbench Container instead of the Workbench itself. Workbench SVC support continues to improve but I use the other tools discussed while watching Workbench improve. Here are the 4 ways I've explored. The AI Workbench UI This NVIDIA AI Workbench interface is a work in progress.  The latest version adds git diff  capabilities, a big improvement.  This UI is still simple and intended for basic file commits.  Use something else if you need advanced git control or merge functionality. Jupyter Notebook Extensions There are a few different Jupyter Notebook extensions. Some, like jupyterlab-git,   add themselves to the Notebook UI toolbar. Here is the left gutter interface for extension:...

Kaggle runs different machine learning or data science competitions. You can participate using their containerized environments or by coding locally. NVIDIA simplified working locally, or in your own cloud, with an AI Workbench-compatible  example Kaggle competition kernel . Their project contains everything needed to download competition data from Kaggle, run train/test cycles, and then upload the results for evaluation. I love this dockerized project because it lets me play in a competition sandbox on my local machine with no local configuration changes to my development machine The  Handwritten Digits Recognizer  competition is an open-ended competition trainer. Kaggle provides images of handwritten digits. You train against the training dataset and test against the testing dataset. Then run your trained model against the candidate digits of the competition submission set and upload the results to the Kaggle competition. The  NVIDIA Workbench example Kaggle c...

Newer LLM models are built around the bfloat16  data type that has different types of precision than the older float16.  My Tesla vintage GPU supports the lower precision  float16, not the newer bfloat16.   you can coerce the model from bfloat16 to float16.  Disclaimer: The difference in precision can result in errors. Your mileage and accuracy may vary depending on the model  Hacking config.json for FP16 The basic steps are  Download the model either manually or as part of an attempted run.   Find the location of the model on disk. This typically is in the model cache directory ~/.cache/nvidia/nvidia-nims/ngc/<some_path>/config.json ~/.cache/nim/ngc/<some_path>/config.json ~/.cache/nvidia-nims/ngc/hub/<some-model>/snapshots/config.json Edit the config.json found for the model. Find the line  "torch_dtype":"bfloat16" Change the value of  torch_dtype  to one supported by the card, float16  in my cas...

NVIDIA AI Workbench runs inside a containerized environment, and I wanted an environment check that verifies the container has access and that Docker/podman, the NVIDIA driver, and Workbench are all on compatible versions.   Containerization has no effect on performance. The CUDA code is pretty much a direct pass-through to the card. Environment NVIDIA AI Workbench One local GPU Windows 11  Docker Desktop Program running in a container built by the Workbench based on the PyTorch/Cuda image Program I access the containerized environment via a Jupyter Notebook visible to the browser on the Windows machine. This is a snapshot of the Jupyter Notebook. numpy_cupy_sort.ipynb Gist It found a problem The program demonstrated that there was a container adapter (or something) mismatch that recently happened.  cupy returned that it had access to the GPU. It turns out that it really did not, and that a Docker Desktop upgrade was needed to fix something driver-related.  The ...

NVidia AI Studio creates and manages containerized ML environments that isolate ML projects on local and remote machines.  You no longer have to switch environments  or remember which version of Python or Anaconda you are using in your global machine environment. NVIDIA simplifies the initial configuration by providing predefined image definitions containing Python, PyTorch, and other tools to be used with or without NVIDIA graphics cards. The actual development is done via browser-based tools like JupyterLab notebooks. Workbench spins up local proxies that port forward into the development container. See videos below NVIDIA Workbench runs in a WSL instance NVIDIA Workbench runs in its own WSL instance.  Each project runs in its own Docker container.  You can look at the NVIDIA main WSL instance by opening a shell into that WSL instance. The following command can be run in a Windows terminal window. wsl -d NVIDIA-Workbench NVIDIA projects live in the WSL instance in...