Python comparison - JIT, CUDA and DASK

Python has become the language of choice for data scientists and data analysts. It is easy to use with a lot analytical support libraries. Python programs aren't particularly fast. This has driven people to create a set of tools that help Python programs scale-up and scale-out. 

 We can compare approaches with a simple program that I adapted to JIT, GPU and distributed computing.

Demonstration Caveat

Distributed computing works best when there are problems with lots of I/O that can be spread across workers.  This program has no I/O and a high data transfer to computation ratio.

GPUs have a high cost for data load and unload data with GPU memory. This means they work best when there is a significantly higher computation to data transfer ratio than this program has.

Demonstration

The sample Python program creates 10,000,00 3 variable rows that it then uses as inputs for 10,000,000 iterations of log(x)*log(y)*log(Z).  The results are returned in a 10,000,000 long result array.  The times for the various options are shown in the following drawing.
  • Non-JIT: is the default execution model for a standard Python program.  
  • JIT: represents the Numba JIT compiled code.  
  • JIT Parallel: represents Numba JIT compiled code where iterating loop methods have been marked for JIT auto-parallelization if possible. In the test app, this means the 10,000,000 long iteration loop is spread across as many worker threads as it can.
  • GPU: moves computational code from the CPU to the Nvidia GPU.  The 10,000,000 calculations are distributed across the GPU's 2,000 CPUS. Computation iterator loops are replaced with a mapping of the calculation across the 2000+ GPU cores. The mapped functions execute on the cores until all are done. The results are aggregated by collecting the data from GPU memory.
  • DASK: distributes computational work across a set of threads either in process or on DASK worker processes.  The 10,000,000 input values are allocated across the worker cluster. Remote task invocation causes each of the worker nodes to iterate across the data allocated to that DASK worker thread. The results are aggregated by collecting data from each worker node.
The Parallel JIT is the easiest to implement with simple annotations. Parallel JIT changes the least amount of code and can provide the highest single machine performance. GPU and DASK overtake Just JIT if provided larger or more complex calculations.
Test run results

Sample Repository

Recordings

Video Part 1 - Base, JIT and JIT Parallel

Video Part 2 - Numba and JIT for CUDA

Video Part 3 - DASK


Modification Log

Created 2020 03 29

Comments

Post a Comment

Popular posts from this blog

Understanding your WSL2 RAM and swap - Changing the default 50%-25%

Image
The Windows Subsystem for Linux operates as a virtual machine that can dynamically grow the amount of RAM to a maximum set at startup time.  Microsoft sets a default maximum RAM available to 50% of the physical memory and a swap-space that is 1/4 of the maximum WSL RAM.  You can scale those numbers up or down to allocate more or less RAM to the Linux instance. The first drawing shows the default WSL memory and swap space sizing. The images below show a developer machine that is running a dev environment in WSL2 and Docker Desktop.  Docker Desktop has two of its own WSL modules that need to be accounted for.  You can see that the memory would actually be oversubscribed, 3 x 50% if every VM used its maximum memory.  The actual amount of memory used is significantly smaller allowing every piece to fit. Click to Enlarge The second drawing shows the memory allocation on my 64GB laptop. WSL Linux defaults to a maximum RAM size of 5
Read more

Installing the RNDIS driver on Windows 11 to use USB Raspberry Pi as network attached

Image
I do a lot of my development and configuration via ssh into my Raspberry Pi Zero over the RNDIS connection. Some models of the Raspberry PIs can be configured with  gadget drivers  that let the Raspberry pi emulate different devices when plugged into computers via USB. My favorite gadget  is the network profile that makes a Raspberry Pi look like an RNDIS-attached network device.  All types of network services travel over an RNDIS device without knowing it is a USB hardware connection. A Raspberry Pi shows up as a Remote NDIS (RNDIS) device when you plug the Pi into a PC or Mac via a USB cable. The gadget  in the Windows Device Manager picture shows this RNDIS Gadget connectivity between a Windows machine and a Raspberry Pi. The Problem Windows 11 and Windows 10 no longer auto-installs the RNDIS driver that makes magic happen. Windows recognizes that the Raspberry Pi is some type of generic USB COM device. Manually running W indows Update  or  Update Driver  does  not  install the RNDI
Read more

DNS for Azure Point to Site (P2S) VPN - getting the internal IPs

Image
I wanted to access all my Azure resources without making any of them visible to the Internet.   The easiest give my local machine access to everything on my Azure Virtual Network (VNET) was to connect to it over VPN. It turns out creating Azure VPN gateways and connecting to Azure VPN endpoints is easy.  There are some subtleties in getting DNS name resolution to work that can confuse when first starting out. Setting the Stage There are a few ways to get to Azure endpoints and resources that are blocked from the internet.  We can  Create a Point-to-Site connection from our local machines to Azure Network Gateways Create a Site-to-Site network connection from our local networks to Azure Network Gateways. Use Bastion Hosts Use Cloud Shell Leave everything open to the internet. I chose a Point-to-Site (P2S) VPN connection that connects from my laptop to a VNet Gateway.  That joins my laptop
Read more