Blog de Joe Freeman

We evaluate the identified possible faults and issues to determine how we can detect the failure and how we can remediate it.    For this discussion, we will bucket the failure modes into three types which can help us determine how they can be detected.   We will categorize failures as technical, design time and business types of failures.  We can use the category to determine how we wish to remediate the failures. Some of the business rule failures will be "by policy" and their remediation will be in the business departments. The other failures will be remediated via technical means. Capturing - Detection and Remediation We want to fill in the  Detection  and  Remediation  columns.  You can tune the meanings of these columns to your use case.  For this walkthrough We sweep across all the faults to determine how the fault would actually be detected and then how we would permanently, tactically, manually, transiently remediate that....

Containers make it easy to set up a complex data scientist development environment.  A developer can just spin up a Python, Jupyter Notebook, Spark, Hadoop, or another type of container on a local machine in minutes. Containers can be confusing when you first work with them. Here we talk a little about how you can get code and data into your container environment and how you can get it back out. I want to write code  local  to my laptop and run the code inside a fully configured Anaconda container. And, I'm lazy. Two ways to get code onto a container for development Containers are standalone  mini machines  with private disk space, CPU, networking  and other services.  They are not intended to retain state, something that we definitely want to do in a development environment. We need to get our code inside the container. We can do the same thing with data or we can have our code pull the data in at runtime. We plan on doing all  development on ...

Containers make it easy to set up a complex data scientist development environment.  A developer can just spin up a Python, Jupyter Notebook, Spark, Hadoop, or another type of container on a local machine in minutes. Containers can be confusing when you first work with them. Here we talk a little about how you can get code and data into your container environment and how you can get it back out. I want to write code local to my laptop and run the code inside a fully configured Anaconda container. And, I'm lazy. Two ways to get code onto a container for development Containers are standalone mini machines with private disk space, CPU, networking  and other services.  They are not intended to retain state, something that we definitely want to do in a development environment. We need to get our code inside the container. We can do the same thing with data or we can have our code pull the data in at runtime. There are two primary ways of getting code onto a machine.  We c...