Create a grammar before defining event models

Software systems stream business events to be seen by customers, used by processes, or written to analytical stores. Sometimes the best way to initiate a design is to create a grammar, a language, that can be used to generate the data models used in the events.

Our first step is to define the grammar or syntax for Business Events / Audit Messages in a generalized form. We use this grammar to validate design time event definitions. We can implement that grammar in whatever technology we choose.

Business Events are different from technical events. Programs and subsystems generate technical events about invocation and process failure events. Technical events are targeted at production support, system triage, and data store replication. Customers and other users expect to see activities in human-readable form. These activities are referred to as Business Events or Business Audits. They define operations one level higher than Technical Events.

Structured Grammar

We're going to start defining our Events with the fixed/structured portion. It will have some basic standard fields and an extension area for customized payload fragments. Structured fields have the advantage that they are always there in tradeoff against customized flexibility.

Events describe an action taken. In this case, our grammar could be something like

[Actor] did [Action] on [Target] with [Value] at [ThisTime]

Individual parameters can be something like

Attribute	Purpose
Actor (Required)	The party that initiated the Action
Action (Required)	The business transaction or event.
Target (Required)	The party or object this action operated against.
Value (Optional)	A value that is meaningful in this event
This Time (Required)	The time the event occurred. May be used for event sorting.

Example purchase event for "Customer X" on "Card X" for an amount of "$50.23"

[Customer X] did [Purchase] on [Card X] with [$50.23] at [2000/01/01-01:01:01]

Video

Multi-part Actions

Some systems generate enough events of related types that they may require a primary and secondary action type. This also lets us group related SubActions which can be helpful for reporting aggretations. We can specify the grammar for this Action/SubAction as

[Actor] did [Action : SubAction] on [Target] with [Value] at [ThisTime]

Example: where a customer purchases an item for $50.23.

[Customer X] did [Purchase : Debit] on [Card X] with [$50.23] at [2000/01/01-01:01:01]

We hardcoded for a maximum of two parts in our compound action. We could either add another field or put it in the context that will be discussed below.

Multi-part Targets

A lot of actions are multi-party, having multiple targets or a source and target. I'm just going to implement this as a multi-part target. Consumers will just have to know the meaning of Target1 and Target2 for any given Action:SubAction

[Actor] took [Action : SubAction] on [Target1 : Target2] with [Value] at [ThisTime]

Example customer purchases, card/account linkage, and bank account linkages.

[CustomerX ] did [Purchase : Debit] on [Card X : Store 4321] with [$50.23] at [2000/01/01-01:01:01]

[CustomerX] did [LinkPay : Card] on [Card X : Phone P] with [Link43] at [2000/01/01-01:01:01]

[CustomerE] did [Link : Accounts] on [Account 1 : Account 2] with [<empty>] at [2000/01/01-01:01:01]

We created a two-part compound target. There may be situations where more actors or more targets must be supported. We could either add another field or put it in the context that will be discussed below.

Multi-part fields

Parameters/fields can be multi-part as shown above for Actions and Targets.

We can implement the multi-part fields in some delimited format as shown above.
We can implement multi-part fields as individual properties if we believe they will be a fixed number of them, Action1, Action2.
Some fields may have an indeterminate number of sub-properties and may be better implemented in JSON

Fixed fields have the advantage of being very explicit about what is expected. This can simplify storage and queries.

Metadata: Time

Our samples show a single event time. Some systems may need more than one time. These additional elements could be broken out specifically or be added to the context discussed below.

The Context - Bag of Stuff

Some events have additional context required to describe an event.

[Actor] took [Action : SubAction] on [Target1 : Target2] with [Value] at [ThisTime] in context [Context]

It could be some kind of to-many detail record like the list of products that were purchased. This sample has a context containing 3 items.

[CustomerX ] did [Purchase : Debit] on [Card X : Store 4321] with [$50.23] at [2000/01/01-01:01:01] in context [itemx : itemy : itemz]

The actual JSON Context would be something like

  {
    values: [ itemx, itemy, itemz ]
  }

Metadata: Tracing and Tracking Identifiers

We can store additional metadata like tracking identifiers in the context. We just need to define a standard pattern for those identifiers. Well-known identifiers can be put into structured fields or stored in the context of well-known property names.

This is a sample JSON with both the purchase list and correlation/tracking identifiers.

  {
    values: [ itemx, itemy, itemz ]
    span-id: ab-df
    trace-id: d3l2-2323-sd32w
  }

Modeling and Persisting Events

The data model can be any old model object or struct. I've tended to use a POJO/POCO with the mandatory properties and a dictionary for context.

Wire protocols are often some form of data model serialization. This means the event is serialized to something like JSON AVRO.

The storage model can be optimized for capture or query.

Relational databases tend to put each structured field in its own database columns.
Document DBs tend to just stuff everything into a single object.

Software Implementation

My personal preference is to create model/view objects that contain all of the structured data types as properties. Everything else can go in a dictionary that is attached to the model/view objects.

Sample JSON

{
    "actor": "",
    "actionPrimary":"",
    "actionSecondary": "",
    "target1":"",
    "target2":"",
    "value":"",
    "eventTime":"",
    "context":{
    }
}

Created 2022 05

Blog de Joe Freeman