What are we running - API changes expressed as versions

What API version is out there and how do we know it?

The software world is continually moving along the path to more API-driven solutions in an attempt to break down monolithic software systems into more manageable size pieces.  We stitch together business processes with a collection of synchronous API endpoints, asynchronous messages, and massive event streams.  Every one of these connection points is a data and behavior contract that can be changed every time one of these endpoints is updated or upgraded.  Today's agile means that the individual services change often and at a rate decoupled from the API consumers.  We need to apply software engineering discipline to capture the changes and make it possible for our consumers to know which version they are talking to.

Video

Part 1 Change Types and Scale, Synchronous/Asynchronous APIs, Design Time vs Run time Considerations


Part 2 Run Time versioning options for consumers and operations

All APIs: Synchronous and Asynchronous

Synchronous APIs can be implemented with a variety of technologies, REST, GraphQL, gRPC, RPC, etc.  They all have some type of resource address and a data contract.  RESTs can very the resource address as part of the invocation.  Most of the others change the payload as part of the invocation. Changes in either the resource address, in the data contract, or in metadata can all represent new variations or versions.  We need to be able to create versions based on resource, data, or metadata changes. The API caller should know immediately if they invoke an old synchronous API version.

Asynchronous APIs can be implemented in a variety of technologies, RabbitMQ, ActiveMQ, Kinesis Streams, Kafka Streams, Event Hub, Azure Service bus, etc.  Asynchronous APIs primarily evolve with changes to their data contract or payload. We need to be able to create versions based on data or metadata changes. Asynchronous invocations are usually fire and forget.  

The caller may never know whether or not they passed the correct payload to the consumer.  This means we may need to implement an out-of-band validation function to verify that the request is valid for the targeted endpoint.  Kafka schema registry is an example of an out-of-band validation tool.
Click to Expand

Every API change is a new version

APIs implicitly version every time the API changes no matter how small the change.  Teams need to define a process for capturing those changes at design time and for exposing the versions at run time to consumers and tools.  API consumers must be able to determine the version of any endpoint they communicate in every deployment situation at any time.

Reliable service-based systems demand stable API contracts and well-understood behavior changes.  API providers must strive to avoid breaking consumers. They must provide a way of informing consumers which version they are using and how far this version is from any other version.

Scope Delivery / Execution Governance
Design TimePackage or Contract VersioningSchema Registration
Run Time TimeAPI VersionsInterrogation Endpoints

Be explicit about versioning

Every API team needs to have an API governance plan:
  1. Design Time: How do you capture that change at design time for later analysis?
  2. Run Time: How do you expose the fact that there is an API change to consumers and to version validation tools?

Compatibility: Major and Minor Versions

Minor Changes: Changes that do not break well-constructed consumers of the API or message.  Additive changes where we add new functionality or new endpoints or topics are typically minor changes. They don't affect the existing APIs or their consumers. Minor changes are represented by the second position in the major.minor.patch semantic versioning standards.

Major Changes: Changes that break anyone consuming that endpoint or stream. Breaking changes are major changes.  ARN/URI address changes are breaking changes.  Incompatible data model changes are breaking changes.  I say may break here because it is possible that some consumers don't use that endpoint or feature.  Major changes are indicated by the value in the first position in the major.minor.patch semantic versioning standards.

Major and Minor changes by API and change type

Major version breaking changes often result in a new API channel: endpoint, URI, stream, queue.  Minor version changes can usually be supported on the same endpoint, stream, queue as the previous version.

Capturing Changes - Design Time

Design-time and build-time may be used interchangeably in this section. Design-time refers to tracking the versions in source code or in some metadata catalog through the deploymentprocess.

For deployment packages containing a single endpoint or lockstep endpoints, the package's semantic version major and minor numbers represent the API's effective version number. There are a few options including the following.
  1. The package's major version number tracks the endpoint version number.  The package minor version number increments as needed possibly having the side effect of incrementing the API's endpoint minor number even if that part of the deployable didn't change.
  2. The package's version number increments only in the patch number if changes are released unrelated to the API's functionality.  This leaves the package's version number as a copy of the API version number.

For deployment packages containing multiple endpoints that change at different rates, there are two major options.  
  1. Lock the semantic version and the major version number are always in sync.  The package's sematic major number changes for every breaking change which means a change in any of the API contracts.  In this case, we just force all the endpoints to increase their major version number. 

    Note that it should be easy in this situation to implement alias services on the previous major version number for the endpoints that didn't actually change.  This gives backward compatibility for consumers of the unchanged APIs. 
  2. The package's semantic version number is decoupled from the API endpoint's version numbers.  The package's version major number does increment when any of the endpoints have breaking changes but the other endpoints are unaffected.

Exposing Changes via Versioning - Run Time

Select the option that works best for your work style and API style. We have two main needs
  1. Provide a way for the caller to invoke a specific version of the service.  The major number must always specifiable in some fashion.  The minor may be provided.  Some teams don't include the minor version as they are backward compatible with previous consumers. We are assuming that it won't be needed in most environments.
  2. Provide a way to interrogate some metadata API to find the implemented versions on an instance. This could be a direct query or a query to some metadata catalog tied to the instance.  The query API will provide the major number and either a minor number or a revision number for the number of schemas that have been registered under that major number.
This diagram describes how we implement major (breaking) and minor (non-breaking) changes across different paradigms and their implementations.
Click to Enlarge

Synchronous resource-based endpoints

Requirement Handling
Major Version Request - AMajor version number in resource path
Major Version Request - BMajor version specified in a request header
Minor Version RequestNot specified. Compatibility model documented
Versions Documentation - AMetadata endpoint exposes all services Major/Minor
Versions Documentation - BCloud Tags

Synchronous payload-based endpoints

Requirement Handling
Major Version Request - AMajor version number in resource path
Major Version Request - BMajor version number in a request header
Major Version Request - CMajor version specified in the request payload
Minor Version Request - APayload.
Minor Version Request - BNot specified. Compatibility model documented
Versions Documentation - AMetadata endpoint exposes all services Major/Minor
Versions Documentation - BCloud Tags

Asynchronous payload-based endpoints

Requirement Handling
Major Version Request - AMajor version encoded in Queue or Stream ARN
Major Version Request - BMajor version number in a message header
Major Version Request - CMajor version specified in the payload
Minor Version Request - APayload.
Minor Version Request - BCompatability verified by schema registry.
Versions Documentation - AMetadata endpoint exposes all services Major/Minor
Versions Documentation - BCloud Tags

Typically there is a schema registry for each queue or stream.  If we support major and minor changes on the same topic then the schema registry would have to support breaking changes on the queue/stream.  Typically we don't allow breaking changes on a topic/stream and create a new one for each major version. If we change the name of the queue/stream then each schema registry would only contain the minor revisions for that single major version.

Breaking vs non-breaking changes in model-driven APIs

Model-driven APIs can leverage database-style rules when determining whether something is a major or minor version difference. This table describes Avro schema registry versioning rules  It can be found on the Confluent website at https://docs.confluent.io/platform/current/schema-registry/avro.html

Compatibility TypeChanges allowedCheck against which schemasUpgrade first
BACKWARD
  • Delete fields
  • Add optional fields
Last versionConsumers
BACKWARD_TRANSITIVE
  • Delete fields
  • Add optional fields
All previous versionsConsumers
FORWARD
  • Add fields
  • Delete optional fields
Last versionProducers
FORWARD_TRANSITIVE
  • Add fields
  • Delete optional fields
All previous versionsProducers
FULL
  • Add optional fields
  • Delete optional fields
Last versionAny order
FULL_TRANSITIVE
  • Add optional fields
  • Delete optional fields
All previous versionsAny order
NONE
  • All changes are accepted
Compatibility checking disabledDepends

Technical Articles on Implementing Version API

  • https://www.hanselman.com/blog/aspnet-core-restful-web-api-versioning-made-easy


Created 2022 01 22

Comments

Popular posts from this blog

Accelerate Storage Spaces with SSDs in Windows 10 Storage Pool tiers

Java 8 development on Linux/WSL with Visual Studio Code on Windows 10

Understanding your WSL2 RAM and swap - Changing the default 50%-25%