Joe Freeman: Tech, Teams, Trade-offs

Data producers often create multiple versions of their data across time. Some of those changes are additive and easy to push from operational to analytical stores.  Other changes are transformational breaking changes. We want to integrate these changes in a way that reduces the amount of magic required by our consumers.  All these changes should be captured in data catalogs where consumers can discover our datasets, their versions and, the datasets they feed. Managing incompatible Producer Schemas We can take this approach when providing consumers a unified consumer-driven data model. Version all data sets and schema changes. Minor versions represent backward-compatible changes like table, column, or property additions. Major version numbers represent breaking changes from the previous versions. Data should be stored in separate raw zone data sets ba...

Process and decision repeatability and accountability is a requirement for large enterprises and entities operated in regulated industries.  Machine Learning decision justification and auditability and privacy related data tracking are two areas pushing organizations to improve the way they track data movement, transformation and usage.  This drives the need for  Data Lineage  tracking and reporting. Organizations have to trade off the  ease of creating and capturing data lineage,    the amount of data captured  and the  ease of reporting and auditability   Data lineage includes the data origin, what happens to it and where it moves over time. [1]  Data lineage information includes technical metadata involving data transformations .   [2] This diagram shows a simple data movement where data originates in one system, is transformed, stored in a database, then transformed again and used by a machine mode.  The re...