Blog de Joe Freeman

A few years ago I tried to build a probabilistic/soft entity resolution system a few years ago using Elasticsearch with its ability to match "like" terms. The idea was to create a candidate list of companies that closed and re-opened a short time later, a pattern often used to avoid some type of consequences. The project was more work than I felt like doing at the time. Recently, I decided to try to push the project forward using Claude Code as my development team.  The LLM filled in the blank spaces, created tests, and helped me analyze the data to understand what I needed to do.  All within just a couple of days. Entitopia as a working project Entitopia  is a proof of concept for using semantic matching and multi-property similarity as part of probabilistic entity resolution. It looks for an _indicator_: a carrier that shut down under one DOT registration while a closely-resembling "new" carrier registered shortly after. That pattern has plenty of legitimate expla...

Google Drive is one of those cloud technologies that democratized cloud access to data storage. It lets you securely push all kinds of data into and out of the cloud via Google-provided APIs. APIs and documents are bound to permissions, roles, and identities. Programs accessing Google Docs require credentials, preferably least-privilege credentials, which exist just for a single program's needs. Google IAM supports Service Accounts that are not tied to any human .  They can be enabled and disabled without impacting individual users.  Accessing Google Docs via API means you have to enable  Google Drive API  in a project, create an identity/credentials for the program, and then give the identity access to the docs or the API.  There are plenty of good tutorials that walk you through setting up an account. They are often light on the overall process or how the steps tie together.  It can be confusing the first time or 10 you go through it. Google services are...

We're flooded with data.  Customers and fraudsters have the ability to submit data, purchase services or products, and interact with corporate edge systems.  Every transaction should be wrapped with the following questions. Who are we talking to?  What is the risk if we don't know? Who are we talking about?   What is the risk if we tie it to someone we already know about? What is the risk of poisoning other data? Be skeptical my friend. Video Presentation Presentation Content Speakers notes will be added later

Customer identifiers are the keys we use to bind different bits of information together that we believe represent the same person. This can be more complex than it sounds when you take into account the changing amount of what we know, corporate acquisitions, and partner company interactions. Video Presentation Content Speaker's notes to be added at some future date. A corporation, with divisions that each have apps and with partners. The corporation runs with a single id per person and handles any merge or separation actions by updating all parties. Conway's law: Architecture and communication paths align with organizational structure. Every team does their own thing. Division IDs are bound to the corporate ID at the corporate level. Corporate IDs are bound to the division at the division level.   The corporation creates link ids there shared with the other orgs. The other orgs create link ids and share with the parent corporation.