Blog de Joe Freeman

We evaluate the identified possible faults and issues to determine how we can detect the failure and how we can remediate it.    For this discussion, we will bucket the failure modes into three types which can help us determine how they can be detected.   We will categorize failures as technical, design time and business types of failures.  We can use the category to determine how we wish to remediate the failures. Some of the business rule failures will be "by policy" and their remediation will be in the business departments. The other failures will be remediated via technical means. Capturing - Detection and Remediation We want to fill in the  Detection  and  Remediation  columns.  You can tune the meanings of these columns to your use case.  For this walkthrough We sweep across all the faults to determine how the fault would actually be detected and then how we would permanently, tactically, manually, transiently remediate that....

We can make it better if we measure or analyze it. Let's analyze a small program to determine where it might fail and what we can do about it.  We will break down a software program into smaller modules and look at how each phase or component might fail.  We will also look for silent failures or a lack of success metrics where something didn't occur at a time when there should have been some activity. Sample System Under Analysis Our example system is a data lake sink that  Reads streaming data  Validates the data Bundles the data into micro-batch sets Writes the data to a data lake.  Each lake write has a corresponding metrics push that updates our metrics store, statistics and other features. Video Walkthrough In this video, we throw down as many failures as we can think of. We can worry about detection and remediation in a later phase Worksheet Template We will reco...