Joe Freeman

  Containers make it easy to set up a complex data scientist development environment.  A developer can just spin up a Python, Jupyter Notebook, Spark, Hadoop, or another type of container on a local machine in minutes. Containers can be confusing when you first work with them. Here we talk a little about how you can get code and data into your container environment and how you can get it back out. I want to write code  local  to my laptop and run the code inside a fully configured Anaconda container. And, I'm lazy. Two ways to get code onto a container for development Containers are standalone  mini machines  with private disk space, CPU, networking  and other services.  They are not intended to retain state, something that we definitely want to do in a development environment. We need to get our code inside the container. We can do the same thing with data or we can have our code pull the data in at runtime. We plan on doing all  development on  the container for this discussion.

Containers make it easy to set up a complex data scientist development environment.  A developer can just spin up a Python, Jupyter Notebook, Spark, Hadoop, or another type of container on a local machine in minutes. Containers can be confusing when you first work with them. Here we talk a little about how you can get code and data into your container environment and how you can get it back out. I want to write code local to my laptop and run the code inside a fully configured Anaconda container. And, I'm lazy. Two ways to get code onto a container for development Containers are standalone mini machines with private disk space, CPU, networking  and other services.  They are not intended to retain state, something that we definitely want to do in a development environment. We need to get our code inside the container. We can do the same thing with data or we can have our code pull the data in at runtime. There are two primary ways of getting code onto a machine.  We can copy our cod

We can make it better if we measure or analyze it. Let's analyze a small program in order to determine how it might fail and what we can do about it.  We will break down a software program into smaller modules and look at how each phase or component might fail.  We will also look for silent failures or a lack of success metrics where something didn't occur at a time when there should have been some activity. Sample System Under Analysis Our example system is a data lake sink that  Reads streaming data  Validates the data Bundles the data into micro-batch sets Writes the data to a data lake.  Each lake write has a corresponding metrics push that updates our metrics store statistics and other features. YouTube recorded session In this video, we throw down as many failures as we can think of. We can worry about detection and remediation in a later phase Worksheet Template We will record the identified failure modes using a worksheet like this one. Component Fault Severity L

I've wanted to record with both cameras while doing a walkthrough or doing video interviews or other purposes but had no idea how to do it. It turns out it is super easy now that Apple added support for multi-cam recording to their more recent iPhones. I used DoubleTake for this video on my iPhone SE to make this video Other programs claim to require X, 11 or, 12 derivatives. DoubleTake supports the Phone SE 2 which has similar CPU power. Multi-cam recording feels like a great tool for Vlogging, interviews or, other purposes. Video Video recorded on an iPhone SE One App I used DoubleTake because it explicitly said it supported the iPhone SE Capture Modes The application saves the recording in .mov files.  I renamed them to .mp4 files and they loaded without issues in my editor. Discrete Each camera recording is saved to its' own track. The sound is recorded into both video tracks. Editing software can manipulate the two video feeds separately.

I wanted to use a green cloth as a video backdrop for video calls and videos. This stand is actually a portable wardrobe rack like the ones you buy when you live in an apartment with no closets. It makes a decent green screen stand with a couple minor additions. This rack looks like https://www.target.com/p/adjustable-single-rod-garment-rack-black-room-essentials-8482/-/A-51688630 The product description says 64" high which is 5'4". That is shorter than I guessed in the video. I added a couple pipe extensions to make it wider and a couple clothespins to temporarily fasten the cloth. Video Description Video created on an iPhone SE with DoubleTake https://apps.apple.com/app/doubletake-by-filmic-pro/id1478041592# Note: Camtasia and other software can remove the green background so you can stand "in front of" anything you wish to show wall talking.

Graph Databases have a different paradigm than traditional relational databases. They have two major first-class entities, the Node , and the Edge .  Nodes are entities or things. Edges are relationships that can be typed  and have properties.   The video does a quick demonstration of how we can model a person and sales transactions using  Nodes  and Edges. It shows how there can be multiple Relation Types  between the same two Node Types and other interesting effects that can be done with graph models. Video Walk Through Diagram

Graph databases seem to lend themselves to modeling people and their interactions and relationships. You tend to see them used in Customer 360 type systems. Let's walk through how a Graph Database might be applied to identifying coupling between fraud incidents. Video A person and their transactions This is an incomplete model for discussion purposes Overlapping people and transactions Coalescing People 

The internet age and cloud competing moved data size, ownership, and privacy to an entirely new level. This is driving organizations to re-thinking where data is moved, how it is transformed, who can see it and, who owns it.   Let's talk about how data moves from producer systems through data pipelines and, into Consumable locations with correct access controls. Target State We want to create a well-organized data ecosystem that makes information available to data consumers. The resulting data should be organized for consumers and made available in a way that works with consumer tools. Data starts in source systems owned by applications and organized for the data producer's use cases. The data is extracted from the producer systems in a raw form and stored in a data ingestion zone or data lake. This stage represents the end of the Producer's involvement. The data is converted from producer oriented models to consumption independent models. Traditionally this has been a stan

Graph style databases can be used to model some data sets in a more intuitive fashion. This lets us leverage the structure for special purpose applications and data exploration.  Standard analytics tools often work best against relational record-oriented data.  There are a couple strategies for moving graph modeled data to relational stores for analytics. We will walk through some very basic options. There are more sophisticated approaches than what we will discuss Video - YouTube The Initial Graph DB Here we have a simple graph modeling a person with their various addresses and phone numbers.  We have modeled two different address types, mailing, and billing.  The model supports multiple phone numbers but does not break them out by type and purpose. Each of the graph node types can have additional business and technical properties.  The Person  might have FirstName , LastName, last modified at. The relationships  or edges  can also have additional properties.  They might all have some