Blog de Joe Freeman

Some people buy computing power for self-training or self-edification projects. Others rent computing power. I like owning the gear I work on. I purchased a desktop to be used as a gaming machine, a development system, a containerized workload machine, a Data Science machine, and a Machine Learning platform. I didn't understand that it would turn into a series of hardware upgrades bound by PC architecture constraints. This was a great learning experience, but not the best raw dollars investment from a pure cost/capabilities point of view. The 3 years of upgrades cost $1200. I saved some money by purchasing previous-generation hardware. Buying current-generation hardware upgrades would cost $1900.  I could have stopped anywhere on the path. Apple wasn't a player in the gaming, M/L, or GPU market when I made my purchase. Apple has caught up for most of my use cases with its large shared memory architecture and performant CPUs. A MacBook might be the simplest approach for someone ...

Process and decision repeatability and accountability is a requirement for large enterprises and entities operated in regulated industries.  Machine Learning decision justification and auditability and privacy related data tracking are two areas pushing organizations to improve the way they track data movement, transformation and usage.  This drives the need for  Data Lineage  tracking and reporting. Organizations have to trade off the  ease of creating and capturing data lineage,    the amount of data captured  and the  ease of reporting and auditability   Data lineage includes the data origin, what happens to it and where it moves over time. [1]  Data lineage information includes technical metadata involving data transformations .   [2] This diagram shows a simple data movement where data originates in one system, is transformed, stored in a database, then transformed again and used by a machine mode.  The re...

What is a Feature? A feature is data that has been prepared to be used as input to a Machine mode.  The feature can be a data set or scalar value or an aggregation. It is created by transforming, categorizing or aggregating original source data.  Features can be created and used in almost any type of application, and can be calculated a priori or calculated as part of model execution. What is an Enterprise Feature? ML/AI model usage in regulated industries often includes proof of data lineage used in training the model and in feeding the model in production. The models themselves must often be registered as they are trained and retained for audit purposes.  The retained features and retained models can be used later for bias or fraud investigations as part of the normal regulated industry audit process. An enterprise feature is a feature that meets regulatory, legal and compliance requirements required in regulated industries.  Data, and transformation r...

Big pieces of the 2018 Microsoft Build conference were about applied machine models and secure IOT. One of the keynote demos was called "Scott or Not" where a Raspberry PI used a machine learning vision model to determine if the person in front of a camera "looked like Scott". Some of the most interesting parts of the demo were not obvious without a both demo later in the day. Hobbyist are often ok with a hand crafted build using manual script instructions. That approach doesn't work in a commercial environment with 100s or 1000s of units. Microsoft took a more enterprise approach by creating a modular demo that supported easier automation. The demonstration code is organized in a modular fashion using Docker images. Individual functions of the pipeline are isolated to their own containers. This makes it possible to update tools, languages and code without any updates to the core system. Demonstration Flow The diagram on the right shows the flow for the vi...