Showing posts from December, 2019

What do you know and why do you know it - Lineage for ML

Process and decision repeatability and accountability is a requirement for large enterprises and entities operated in regulated industries.  Machine Learning decision justification and auditability and privacy related data tracking are two areas pushing organizations to improve the way they track data movement, transformation and usage.  This drives the need for  Data Lineage  tracking and reporting. Organizations have to trade off the  ease of creating and capturing data lineage,    the amount of data captured  and the  ease of reporting and auditability   Data lineage includes the data origin, what happens to it and where it moves over time. [1]  Data lineage information includes technical metadata involving data transformations .   [2] This diagram shows a simple data movement where data originates in one system, is transformed, stored in a database, then transformed again and used by a machine mode.  The resulting calculation is then stored again. The small circles call out what

Run applications on a Mac using ASP.NET Core and with MongoDB on Docker

It is easy to develop ASP.NET Core applications on a Mac with VSCode, and run them while they rely Docker based data services. The first part is easy because of ASP.NET core's cross platform compatibility. The is very simple even though it sounds impressive.  Services like MongoDB expose the same ports whether they are Docker deployed or directly installed to the operating system. Our application is an ASP.NET core micro-service example put out by Microsoft. The bookstore REST service API support implements basic CRUD operations. The net result is an ASP.NET web service that uses MongoDB as its' persistent store. I build this on a Mac because both .NetCore is cross platform and MongoDB is running in a Linux Docker container isolated from the underlying system. You should be able to run this without changes on a PC without modification. We are using Visual Studio code because it is cross platform and runs on Mac, PC and Linux.  I have run it in the Linux Subsystem on Ch