Blog de Joe Freeman

ElasticSearch is an awesome extensible text search engine. It provides methods for loading data, customizing the data, applying analyzers, changing search weightings, and enriching data by merging subsets of multiple datasets. We can merge pieces of different datasets (indexes) into customized indexes to meet our data analysis needs.  We want to do all of that in a repeatable and automatable fashion with some level of flexibility.  The Python code lets us define pipelines that support multiple steps and customized operations.   This diagram shows a 3-step pipeline that represents data being loaded into two indexes (1,3) with an enrichment and resource manipulation step (2). Each step is driven from a config file that describes the phase processors and other configuration information.  {     "steps" : [         {             "name" : "doctors-clinicians" ,             "phase...

PII tokenization is a way of protecting Personally Identifiable Information (PII) with similar impacts as field-level encryption but without the overhead of key management or rotation.  Tokenization and Encryption both have an impact on data equality checking inside applications and on the fidelity of the data. Plaintext data can be searched and matched us case-sensitive or case-insensitive techniques. Tokenization and Encryption make that impossible because the data is transformed into a form where the case and punctuation are embedded in the data blob.  We can transform the data to a canonical form prior to tokenization.  That makes equality matches easy but makes it difficult to return the originally formatted value because the token/encryption had to same case or character set prior to tokenization in the source data. Video Presentation  Slides and Speaker Notes Slides are provided here for Video reference. Speaker notes to be added Speaker notes t...

Data is a collection of attributes. Information is data that is invested in or drives or has meaning within some type of business or another process.  The same data can represent or have different information contained within the scope of different bounded contexts.  My target here is operational systems where we partition information within business operational contexts. Analytical stores are a bit of a different beast because they tend to be large collections of data that are converted into information through data aggregation and promotion. What is the difference between Information and Data?  Why do we call it Information Science but a Data Lake? I have no idea why they call it the last two but I'm going to use those two labels to bolster my case. Video Slides used in the Video Speaker's notes are to be added at a future date. Revision History Created 2023 02

Python Pandas are targeted at data science applications but they are useful for everyday data conversion. I needed to convert a wide column sheet of NFRs in a TSV to Markdown for display. It was easy with just 4 lines of code This code  Opens the delimited file,  Fills in all the empty cells with empty strings,  Writes out the .md file df = pd . read_csv ( args .csvFile, engine = "python" , sep = args .sep, header = args .header) with open ( args .mdFile, "w" ) as md :     df . fillna ( "" , inplace = True )     df . to_markdown ( buf = md , index = False ) Usage Example https://github.com/freemansoft/Non-Functional-Requirements Complete Program w Command Line Arguments __doc__ = """ This converts a delimited csv file to a markdown table using pandas. Run with the -h option to see arguments """ import pandas as pd import argparse csvFileDefault = "NFRs.tsv" mdFileDefault = "NFRs.md" headerRowDefau...

Cloud Key/Value databases have some interesting features and limitations that can change the way we model our databases. There is a class of key/value stores that have native change feed support that is in a form that is easy to connect to and operate against. In CQRS, we capture an event stream in a primary store and then materialize that in a query store.  An alternative to the CQRS pattern is to create an updated version of a document and then append that updated version of the document to the database.  We're going to look at the drivers and patterns for the latter approach. Video Presentation Content  Speaker's Notes to be added later Published 2022/10

We're flooded with data.  Customers and fraudsters have the ability to submit data, purchase services or products, and interact with corporate edge systems.  Every transaction should be wrapped with the following questions. Who are we talking to?  What is the risk if we don't know? Who are we talking about?   What is the risk if we tie it to someone we already know about? What is the risk of poisoning other data? Be skeptical my friend. Video Presentation Presentation Content Speakers notes will be added later