Sunday, February 26, 2017

Time Warp: Business Cycle Testing

"Let's do the time warp again..."


A video version of this blog

Business Cycle with Time dependencies?

What is a business cycle and why do I need to test it?  I'm really talking about any type of business process that has time based business rules.  The rules can periodic in that they fire on a regular basis or they can one-time based on some time based criteria.  Most of the ones I've worked with are contract oriented or billing cycle oriented. Examples include telecom contracts, home mortgage servicing systems, term based insurance to just name a few.   They usually have some time based sequence of operations, date based rules and may have some type of state machine.  

Testing is complicated by the fact that data may need to be of a certain age before processing begins.  Loan payments may need to be delinquent.  An insurance policy may start the renewal process some time before expiration.  Collateralize debt may have payment, reimbursement  and equity distribution segments based on the time of the month.

Compressing the clock

The length of the business process and the length of a test are different.  In this picture we see that we have 30 hours to run simulated processing for a 30 day month. We are compressing a 30 day process into a 30 hour test without changing any of the business rules or system behavior.  

Understanding dates and time

Computer software can involve a lot of dates. They are used to represent action in the system, time of events in log output and are analyzed as part of business rules.  Three relatively common date concepts are 
  • "Current Date" meaning "today".  
  • "Activity Date" an event occurs and is recorded in the system. This usually must be within the cycle of the test.  This is different from an "Effective Date" when there is a delay in processing when there is a data fix or when something is back dated.
  • "Effective Date" when an operation actually takes effect.  This is may be different than the "Activity Date" when something is processed out of order.  A system that runs only on effective date may not need any date/time manipulation.
Business data and information may use "any" or "all" of these data types. Integrated business cycle testing must manipulate all three to maintain the correct relationships.

Manipulating Time

We want to put our data in the "right place" in our timeline for each portion of the test cycle. The data needs to be coherent and must interact correctly with users, business rules and other systems.. 

Let's talk about four different ways of manipulating time while testing.  There are probably plenty of others. These are four that I've seen in person or heard about.

Explicit Effective Date

It is possible to run a full cycle test using just effective dates if all of the APIs support passing in effective dates. This lets the test back or forward date the operations. This approach will probably not work with most user interfaces since they will not allow user input in effective date for all operations.

Manipulating System Time

Change the system clock on the operating systems where the business logic runs. This approach works best in systems with few connecting components and few servers.

No application changes are required
Works easily with single machine applications
Records the internal test timestamps in logs and databases

Doesn't work for shared machines
Doesn't work for SaaS applications
Doesn't work in the cloud.
Logs will show the in-test time and not the external world time.

Manipulating Data Time

Change the recorded time-stamps for all data at rest between stages.

Works if all data is in mutable data stores.
Requires no code or system changes

Hard to test with partner systems.
File and log time will show different times
Partner data must also be modified
Hard to find all impacted systems.

Manipulating Program Time

Alter the program's concept of current time.  This can be done by adding crosscutting code on Date and Time libraries.  The aspects go to a central time server to obtain the current "system test" time and use that in their calculations.

No downstream manipulation.
Downstream systems see the test time

Must reliably shim whole application
Doesn't work with services owned by other teams.
Will not work with SaaS components. 

Isolating Data by Time

Create unique data sets for each test phase.  

No system changes
No application changes
Test can run overlapping
Works for SaaS
Works for cloud

No true end to end test with same data
Data must be synthesized for all non-initial phases


Full process lifecycle can be difficult, especially if there are time based rules that cannot be changed to run a test. Try some of these approaches and see how they work for you.  I've used Program Time manipulation via libraries, Separate Data for each phase, Data Fixes and Effective Date testing when we planned on it early in the project.  The choice depends on the business problem, the operational model and other factors.

Good luck with your full lifecycle testing!

Created Feb 2017

Wednesday, February 8, 2017

AWS Relationships between EC2, ELB and ASG

This post describes the basic relationships of ELBs (now ALBs), EC2 instances and ASGs.  I used AWS for over a year before I realized how Auto Scaling Groups actually interacted with ELBs and EC2 instances.


EC2: An Amazon virtual machine used to host applications and services.  EC2 instances can be pooled for scale or failover.  EC2 instances can be based on any of the Amazon EC2 machine types.

Elastic Load Balancer (ELB): The basic load balancer provided by Amazon.  They are used as a reverse proxy servers for pools of EC2 instances.  ELBs determine instance health via basic health check operations.

Auto Scaling Group (ASG): A control mechanism that manages how many EC2 instances make up a pool. ASGs will create new EC2 instances based on configured pool sizes. They can also auto-scale up and auto-scale down the pool sizes based on load.  ASGs can register created EC2 instances with associated ELBs.

Availability Zones (AZ): An Amazon region is made up of various data centers that have isolated power and communication.  Those data centers are referred to as Availability Zones. An ASG can spread its' EC2 instances across those data centers. This increases the availability of the EC2 instance pool and reduces the impact of data center failure.

Standard Deployment 

Auto Scaling Groups (ASG) and Availability Zones (AZ)

An ASG controls the number and location of EC2 instances that make up a worker pool.

The ASG creates new instances when needed based on a Launch Configuration. It creates new instances when a new ASG is first created, when an ASG needs to "size up" based on configuration changes or increased demand.

It destroys all associated instances when an ASG is destroyed. It individual instances when an ASG's pool size settings are changed or when demand falls below the load required for the current number of instances.

An ASG can distirbute instances across one or more availability zones. This means the ASG can configure the worker pool for basic HA.

Elastic Load Balancer (ELB) and ASG

An ELB acts as a load balancer acts as a reverse proxy server for a set of EC2 instances that have been registered with the load balancer.  The ELB knows nothing about the type or purpose of nodes in the worker pool.  An ELB can forward to nodes in different Availability Zones (AZ)

ASGs know how to register instances with an ELB. This means an ALB can automatically add and remove worker nodes as they are created or destroyed.

ELBs know about instances.  ASGs know about instances and ELBs.


  • ELB
    • Distributes traffic across target EC2 instances
  • Application Scaling Group
    • Creates Instances
    • Distributes Instances across Availability Zones
    • Registers instances with ELB
  • ELB
      • Knows about Instances
      • Does not know about ASG

    ELB Configuration with Cloud Formation

    Application components are best created as part of a Cloud Formation template.  Basic ELB Cloud Formation templates define the load balancer type, the availability zones and the configuration of the listeners.  The ELB configuration does not define the instances that the listener forwards to. Note that the cloud formation template defines a HealthCheck used by the ELB to determine if a target instances is "healthy" enough to accept traffic.

    Here is a sample ELB configuration from the AWS tutorial:
     "ElasticLoadBalancer" : {      "Type" : "AWS::ElasticLoadBalancing::LoadBalancer",      "Properties" : {        "AvailabilityZones" : { "Fn::GetAZs" : "" },        "CrossZone" : "true",        "Listeners" : [ {          "LoadBalancerPort" : "80",          "InstancePort" : "80",          "Protocol" : "HTTP"        } ],        "HealthCheck      }    },

    ASG Behavior and ELB Linkage

    ASGs create, destroy and generally manage a pool of worker nodes / instances.  They allocate created instances across a set of subnets or Availability zones to create low cost highly available applications.  The diagram at right describes a three node network. It does not describe any set of Availability zones.

    ASGs are configured with minimum, maximum and desired number of instances.  They manage there worker pool to meet those numbers.  Some teams manage costs by problematically adjusting their min/max pool sizes in the evening or during times of low demand. It is possible to take all of an ASG's nodes off line by setting the pool sizes to "0".

    Dynamic autoscaling is not a mandatory "must use" feature of an ASG.

    This is separate from autoscaling behavior where the pool sizes are adjusted based on CPU utilization or some other metrics.  Monitors can listen for resource consumption events and adjust the ASG pool size accordingly.. Here is an ASG that will maintain from 1-3 worker nodes.

    "WebServerGroup" : {      "Type" : "AWS::AutoScaling::AutoScalingGroup",      "Properties" : {        "AvailabilityZones" : { "Fn::GetAZs" : ""},        "LaunchConfigurationName" :
                { "Ref" : "LaunchConfig" },        "MinSize" : "1",        "MaxSize" : "3",        "LoadBalancerNames" :
                 [ { "Ref" : "ElasticLoadBalancer" } ],

    EC2 Names and other stuff

    EC2 Instances have Instance Id's that define individual instances that are unique in an Accouint.  You will often also see a "Name" value in various EC2 dashboards. This represents the "component" or "common" name for a pool of machines doing the same function.  Amazon implenets the "Name" attribute as an EC2 Tag. It is a well known tag that has psuedo-special meaning.  I highly recommend adding a Name tage in your Cloiud Formation template or whenever manually creating machines.
    • EC2 Names
      • Name field on EC2 instances, just a Tag
      • Name is convention so that related instances have a common value
      • Do not define instance relationships in AWS
      • Same EC2 Name could be used for different roles. Don’t do that.
    • ASGs create instances with same Name but different Instance IDs
    • ELB is a proxy for a pool of instances
      • Could target instances with different names.
      • ALB Target Group behaves like an ELB.

    Example Cloud Formation Tempalate

    Amazon has a sample Cloud Formation template that sets up a web app with an ELB, ASG and some scale up and scale down behavior.  You can find this example and the associated Cloud Formation json file at

    The Cloud Formation section of the AWS Console has a "design" function that creates a picture of how your Cloud Formation template actually hangs together.  The example one is shown below.  The primary area of focus above was the Elastic Load Balancer, the Web Server auto-scaling group and the Launch Configuration used to create new instances to populate the ASG and be configured into the ELB.

    It also contains some Notification monitors and dynamic ScalingPolicies for scale-up and scale-down.  AWS pretty much standardizes on using message based notifications for communicating with either AWS components or appropriately configured cusctom components.

    Created 2017 Feb 01