Blog de Joe Freeman

Data scientists and others want to analyze and manipulate lots of data without having to be Software developers. Jupyter notebooks, Python and no DASK are vehicles that help make that happen. DASK is a distributed Python library that can run in pretty much any Python environment.  It was designed to help data scientists scale up their data analysis by providing an easy to use distributed compute API and paradigm. DASK/Python can be run both inside Jupyter notebooks and as standalone Python programs. Environment The DASK environment consists of a Dask Scheduler and any number of worker nodes. The Python program sends a set of tasks to the Dask Scheduler which then distributes those across the worker nodes.  The worker node results are then aggregated  and returned to the original program. Docker The DASK team provides samples how to use docker-compose to create a DASK development and execution environment The environment consists of A development no...

Azure Sphere is another example of  how vendors can create great end to end solutions. Time will tell if it will become another cautionary tale about vendor lock in and corporate attention spans. Why Azure Sphere? Azure Sphere is an end-to-end IOT platform to build applications, deploy them, manage them, securely communicate with them and and remotely update them. Microsoft spent the last few years learning the complexity of building and running secure IOT environments. The continual stream of IOT device hacks and attacks shows that building and running secure, scale-able IOT clouds is beyond the capabilities of many companies. Azure Sphere simplifies this by pushing some of the responsibilities onto Microsoft itself. Azure  Microsoft's Software Effort  Microsoft is putting in a real effort create a professional grade  IOT ecosystem with their Azure Sphere project. They have built a hardware and software platform where security and cloud integration ...

This post explains how Hyper-V and Docker reserve memory on Windows machines that is outside of the normal process allocations.  It shows how you can see those reservations. Docker and Hyper-V Docker Desktop on Windows runs inside a Hyper-V virtual machine.  The virtual machine's memory is sized specified on the Memory  setting in the Docker Control panel's Advanced  tab. Docker Desktop memory reservation. That memory sizing maps directly to the Docker virtual machine's  Assigned Memory value that is displayed in the Hyper V Manager  Virtual Machines view. Hyper-V locks the virtual machine into memory by Drover Locking  the VM's memory.  Memory used for VMs is partitioned away from the rest of the host's processes. Hyper-V Manager VM's view Hyper-V guest memory does not appear in TaskManager  because it is reserved outside of the normal process structures. Memory allocation by type with RamMap Hyper-V virtual machine me...

I've been struggling with my Dell laptop running Windows 10.  The machine seems to shutdown overnight or after a few hours even when it's plugged in. It turns out my machine is hibernating which doesn't actually work right on my machine.  My temporary fix is to extend the  Hibernate after  value to 5 days, 7200 minutes I used this forum posting  as a guide. Default  Power Options values for Balanced  profile Steps Press the Windows Start key or the  Start menu  button  Enter "power and sleep" and press Enter Click on Additional power settings Click on Change plan settings for the selected plan Click on Change when the Computer sleeps Click Change Advanced Settings Expand Sleep  (click on +) Expand Hibernate after Change the  Plugged in:  value to  7200 min Click the  Apply  button

Do you want to try out new technologies without installing a bunch of software inside your local machine?  Do you wish to edit, build and debug code on commonly available Docker containers without having to build custom development containers? Visual Studio Code with  remote extensions  let you do just that. I wanted to play with Machine Learning, Jupyter Notebooks, and TensorFlow without mucking up my local machine. ML tooling is large and complicated to install. The easiest way to do this is to run Anaconda/Jupyter/tensor-flow in containers derived from unadulterated  publicly available  Docker images. Jupyter notebooks are great but sometimes I want an IDE. VS Code Remote  lets me use a standard IDE while developing in a remote sandbox. It provides command line access without forcing me to command line-based tooling. Visual Studio Code can build, run and debug on a remote machine via its various remote extensions .  Remote Extension for Docke...

Process and decision repeatability and accountability is a requirement for large enterprises and entities operated in regulated industries.  Machine Learning decision justification and auditability and privacy related data tracking are two areas pushing organizations to improve the way they track data movement, transformation and usage.  This drives the need for  Data Lineage  tracking and reporting. Organizations have to trade off the  ease of creating and capturing data lineage,    the amount of data captured  and the  ease of reporting and auditability   Data lineage includes the data origin, what happens to it and where it moves over time. [1]  Data lineage information includes technical metadata involving data transformations .   [2] This diagram shows a simple data movement where data originates in one system, is transformed, stored in a database, then transformed again and used by a machine mode.  The re...

It is easy to develop ASP.NET Core applications on a Mac with VSCode, and run them while they rely Docker based data services. The first part is easy because of ASP.NET core's cross platform compatibility. The is very simple even though it sounds impressive.  Services like MongoDB expose the same ports whether they are Docker deployed or directly installed to the operating system. Our application is an ASP.NET core micro-service example put out by Microsoft. The bookstore REST service API support implements basic CRUD operations. The net result is an ASP.NET web service that uses MongoDB as its' persistent store. I build this on a Mac because both .NetCore is cross platform and MongoDB is running in a Linux Docker container isolated from the underlying system. You should be able to run this without changes on a PC without modification. We are using Visual Studio code because it is cross platform and runs on Mac, PC and Linux.  I have run it in the Linux Subsystem on C...

We wanted our 25 person team tell us who delivered value to them .  The one-time exercise was totally opened to being gamed and manipulated as are all systems.  We attempted to limit risk by keeping the stakes low.   The gift card experiment Proposal: We had a pile of $10 gift cards targeted to be used for incentives.  We gave everyone two cards.  They were to keep one as a reward and give away the other as a thank you to someone else for their help.   Process: Every person was given two gift cards. Each person kept one. This meant no one walked away empty handed. Each person had 7 days to give one away as a thank you for that person's help during the year. Supervisors and team leads were excluded. I did an informal survey to find out who people gave their thank you card to. No records were kept Results A couple people kept there give away cards.  This was disappointing but not a surprise. About 1/3 of the earm...

Did you ever wake up, look at your development box and wonder "when did that happen"?  I've started using Docker for deployed services like databases, messages brokers etc.  At the same time, I've been trying to use Kali Linux for hackathons and general security work.   Windows gaming, my pathetic mobile efforts and windows docker development are done using Windows 10.  Windows must run under a Hypervisor or as a dual boot.  That is how you end up with three hypervisors, three operating systems and two docker environments on the same machine. The following diagram shows the underlying complexity of all this. Hypervisors in action HyperKit on OS/X:  Docker for Mac desktop runs docker containers inside a HyperKit virtual machine that leverages the Mac OS/X Hypervisor.framework.  Docker named drives live inside this virtual machine. VMWare Fusion on OS/X:  VMWare Fusion can host Windows and Linux virtual machines. Fusion suppor...

Data lakes exist to store and expose data in its native format without size or format constraints. Cloud data storage makes it possible to store large amounts of data without worrying about costs or data loss.  Corporate lakes often store the same data multiple in transformed or enriched formats making them easier to use.   My last two employers each had over 20 Petabytes of data in their lakes. A well-managed lake organizes data based on usage, data quality, data trust levels, governance policies, data sensitivity and information lifecycles. Lake architects can spread their data across horizontal zones for purpose and/or vertical organization zones .   The actual zones for purpose vary by industry or company. Zone Based Data Organization This diagram demonstrates a zone structure that might be fit for a financial services company.  It assumes that company generates its' own data and receives data from external organizations.  Data exists in unstru...