Python comparison - JIT, CUDA and DASK

Python has become the language of choice for data scientists and data analysts. It is easy to use with a lot analytical support libraries. Python programs aren't particularly fast. This has driven people to create a set of tools that help Python programs scale-up and scale-out. 

 We can compare approaches with a simple program that I adapted to JIT, GPU and distributed computing.

Demonstration Caveat

Distributed computing works best when there are problems with lots of I/O that can be spread across workers.  This program has no I/O and a high data transfer to computation ratio.

GPUs have a high cost for data load and unload data with GPU memory. This means they work best when there is a significantly higher computation to data transfer ratio than this program has.

Demonstration

The sample Python program creates 10,000,00 3 variable rows that it then uses as inputs for 10,000,000 iterations of log(x)*log(y)*log(Z).  The results are returned in a 10,000,000 long result array.  The times for the various options are shown in the following drawing.
  • Non-JIT: is the default execution model for a standard Python program.  
  • JIT: represents the Numba JIT compiled code.  
  • JIT Parallel: represents Numba JIT compiled code where iterating loop methods have been marked for JIT auto-parallelization if possible. In the test app, this means the 10,000,000 long iteration loop is spread across as many worker threads as it can.
  • GPU: moves computational code from the CPU to the Nvidia GPU.  The 10,000,000 calculations are distributed across the GPU's 2,000 CPUS. Computation iterator loops are replaced with a mapping of the calculation across the 2000+ GPU cores. The mapped functions execute on the cores until all are done. The results are aggregated by collecting the data from GPU memory.
  • DASK: distributes computational work across a set of threads either in process or on DASK worker processes.  The 10,000,000 input values are allocated across the worker cluster. Remote task invocation causes each of the worker nodes to iterate across the data allocated to that DASK worker thread. The results are aggregated by collecting data from each worker node.
The Parallel JIT is the easiest to implement with simple annotations. Parallel JIT changes the least amount of code and can provide the highest single machine performance. GPU and DASK overtake Just JIT if provided larger or more complex calculations.
Test run results

Sample Repository

Recordings

Video Part 1 - Base, JIT and JIT Parallel

Video Part 2 - Numba and JIT for CUDA

Video Part 3 - DASK


Modification Log

Created 2020 03 29

Comments

Post a Comment

Popular posts from this blog

Installing the RNDIS driver on Windows 11 to use USB Raspberry Pi as network attached

Image
I do a lot of my development and configuration via ssh into my Raspberry Pi Zero over the RNDIS connection. Some models of the Raspberry PIs can be configured with  gadget drivers  that let the Raspberry pi emulate different devices when plugged into computers via USB. My favorite gadget  is the network profile that makes a Raspberry Pi look like an RNDIS-attached network device.  All types of network services travel over an RNDIS device without knowing it is a USB hardware connection. A Raspberry Pi shows up as a Remote NDIS (RNDIS) device when you plug the Pi into a PC or Mac via a USB cable. The gadget  in the Windows Device Manager picture shows this RNDIS Gadget connectivity between a Windows machine and a Raspberry Pi. The Problem Windows 11 and Windows 10 no longer auto-installs the RNDIS driver that makes magic happen. Windows recognizes that the Raspberry Pi is some type of generic USB COM device. Manually running W indows Update  or  Upd...
Read more

Understanding your WSL2 RAM and swap - Changing the default 50%-25%

Image
The Windows Subsystem for Linux operates as a virtual machine that can dynamically grow the amount of RAM to a maximum set at startup time.  Microsoft sets a default maximum RAM available to 50% of the physical memory and a swap-space that is 1/4 of the maximum WSL RAM.  You can scale those numbers up or down to allocate more or less RAM to the Linux instance. The first drawing shows the default WSL memory and swap space sizing. The images below show a developer machine that is running a dev environment in WSL2 and Docker Desktop.  Docker Desktop has two of its own WSL modules that need to be accounted for.  You can see that the memory would actually be oversubscribed, 3 x 50% if every VM used its maximum memory.  The actual amount of memory used is significantly smaller allowing every piece to fit. Click to Enlarge The second drawing shows the memory allocation on my 64GB laptop. WSL Linux defaul...
Read more

Almost PaaS Document Parsing with Tika and AWS Elastic Beanstalk

Image
The Apache Tika project provides a  library capable of parsing and extracting data and meta data from over 1000 file types.  Tika is available as a single jar file that can be included inside applications or as a deployable jar file that runs Tika as a standalone service. This blog describes deploying the Tika jar as an auto-scale service in Amazon AWS Elastic Beanstalk.  I selected Elastic Beanstalk because it supports jar based deployments without any real Infrastructure configuration. Elastic Beanstalk auto-scale should take care of scaling up and down for for the number of requests you get. Tika parses documents and extracts their text completely in memory. Tika was deployed for this blog using EC2 t2.micro instances available in the AWS free tier. t2.micro VMs are 1GB which means that you are restricted in document complexity and size. You would size your instances appropriately for your largest documents.   Preconditions An AWS account. AWS ac...
Read more