Friday, October 28, 2016

Deploying DotNet Core in Azure with GIT and Kudu

I starting this project trying to build and deploy the ASP.NET Core example application first on my local box, then in Microsoft Azure via Web Deploy, Microsoft Azure via local Azure GIT integration and finally via Visual Studio Team Services (VSTS) via SCM integration.

Deployment Types

  1. Local deployment into a local IIS is pretty straightforward. We won't talk about it here. 
  2. Remote web deployments are the legacy way of pushing Web applications to the (Azure) cloud that works with IDE, CI or command line. Compiled and static application artifacts that are then sent to the remote application servers via FTP.   The servers unpack the archive and deploys it.
  3. Remote SCM deployments are a relatively new and interesting way to support automated deployments and to support multiple branches with little work. The IDE or build system pushes source code to a monitored repository.  Azure (Kudu) monitors the source code repository, runs a build and deploys the resulting artifacts to the cloud.

There are a lot of many variations in deployment models.  These are some of the most common. The diagram may be getting carried away when creating visuals.

Any build server would work in place of Visual Studio Team Services:
Jenkins, Team City, TFS, etc.


Every Azure web site has an associated, Kudu, SCM service. The SCM can bind to a variety of repositories. Kudu has a micro build server and source code repository. It can build a variety of platforms. This makes it possible to create a CI/CD pipeline with no external components.  



A single application can be deployed multiple ways. A team might support web deployments for individual developers, build server web deploy when running tests and repository (Kudu) deployments for Continuous Delivery environments  This is often a philosophical decision build-once vs build-every.  Note: Kudu deployments look like "rebuild every" as of 10/2016.

Repository based deployment with Kudu in Azure

Some folks may be surprised to know that Microsoft has added an integrated Source Code Management system and build server with every Azure Web App.   This means you can build and deploy code in Azure from inside Azure without standing up additional services.  

Kudu supports ASP.NET, nodes.js, python and basic web sites.   It also has Deployment , Web Integration and Post Deployment hooks that let you customize or supplement the Continuous Integration environment.  Deployments are done in a windows environment.

I tested this with the example ASP.NET core application.
  1. Create an Azure web site.
  2. Create a local GIT repository.
  3. Create an application. 
  4. Commit local changes to local GIT
  5. Push changes to GIT tied to your web site.
  6. Test the web site after Kudu builds and deploys your application

This next diagram shows a couple alternative ways of integrating an external SCM with Kudu and Azure SCM services.  The same model works with any build system or external repository.  I used Visual Studio Team Services.  You could use others like Team City or Jenkins. External lets you implement more sophisticated processes based on your existing system. 

I tested this with the following flow:
  1. Create an Azure web site.
  2. Create a local GIT repository
  3. Create an application
  4. Commit local changes to GIT
  5. Push changes to the develop branch in VSTS.
  6. Run a ASP.NET core build in VSTS.
  7. Have the build merge the changes to a GIT branch in VSTS.
  8. Push the merged branch changes to GIT in Azure
  9. Test the web site after Kudu builds and deploys the application.

There are plenty of internet sites that show how to deploy an application in Azure based on this SCM integration.

Integration is configured in the Azure portal under App Deployment --> Deployment Options.  The image on the right shows an application configured to deploy an application that is stored in a GIT repository managed by Visual Studio Team Services (VS Online)

Web Deploy via IDE and build servers

Azure still supports standard Web App (FTP?) deployments from developer workstations and from build servers.  Azure doesn't care where the deployments come from.

Here we have a simple web application that is deployed to a local IIS server and to an Azure web application. The developer/deployer can use Visual Studio's "run application" for IIS and "Publish" to deploy to Azure.

Here we have a simple web application that is deployed to Azure.  The build server, VSTS in this case, runs a deployment phase that pushes the application to Azure. 


References

Create 2016.10.28

Monday, October 17, 2016

Visual Studio Team Services Git your build face on

This page describes configuration settings required to enable GIT integration when building code in Visual Studio team Services.  It will show you how to
  • Enable CI builds when a specific GIT repository and branch are updated
  • Provide the CI build with permissions required to make changes to a GIT repository
  • Provide the CI build with credentials required to make changes to a GIT repository
This diagram shows how GIT and Visual Studio Team Services (VSTS) might be used to implement a CI build triggered on check-in that merges code into another branch and deploys it.  The actual deployment commands are out of scope for this document.



The following changes must be made on the Repositories configuration at the Project (Team) level and on the affected individual build definitions. We first show project level configuration and then Build Definition configuration.

Let VSTS Builds Update GIT Repository

Some builds may need to update a GIT repository upon build success.  This could be to merge with another branch, add test results or some other process. This change occurs at the VS Team Project level.
The build account must have GIT update permissions.This is a repository level control that must be set in the VS TS Control panel. 

This can be applied at the repository or branch level.  The picture to the right shows where to click to set this at the repository level.

All branches inherit permissions set at this level




Access control attributes "Branch creation" and "Contribute" have default values of "Not Set".  Change these to "Allow".

Access control attributes "Read" and "Tag Creation" have default values of "Inherited allow".  That level is sufficient for our needs.  Do not change them.




Enabling GIT Build Integration

Builds are bound to code repositories.  Builds must be configured to point to the correct Git repository and branch. This change is made on each build that requires Git access. This must be done for every build that is triggered by check-in changes.

Configure the Repository on the Build -> Definitions tab tied to he build itself.  This is not done on the project Control Panel.

Triggering Builds on Changes

Continuous Integration builds trigger whenever the source code repository is updated.  This must be done for every build that is triggered by check-in changes.

Configure build triggers on the Triggers pane of the Build -> Definitions tab tied to the build itself.  This is not done on the project Control Panel.






Provide Credentials for GIT Updates


Some builds may need to update a GIT repository upon build success.  This could be to merge with another branch, add test results or some other process.

The build account must have GIT update permissions as described above.

The build itself must present the build account's OAuth token to GIT when running any GIT update commands, usually "GIT push"

Enable OAuth tokens in scripts on the Definitions -> Option pane.

Troubleshooting

This error means you have not enabled credentials propagation.  You must enable the OATH Token.
##[error]bash: /dev/tty: No such device or address

References

The following web pages or blog articles were helpful while learning this information
  • https://www.visualstudio.com/en-us/docs/build/scripts/git-commands#enable
  • http://stackoverflow.com/questions/38670306/executing-git-commands-inside-a-build-job-in-visual-studio-team-services-was-vs
  • https://www.visualstudio.com/en-us/docs/setup-admin/permissions

Created 2016 Oct 16
Updated 2016 Oct 26 Added doc pipeline

Tuesday, October 4, 2016

Classifying your return codes

Document the meaning, ownership and handling behavior of your Service return codes.  Do not assume your partner teams and calling systems have any expectations or understanding beyond success and not-success. Ask other teams, you call, for their Service return code documentation. Force them to document their expectations.

Proposed Return Code Category Types

Create response categories.  Determine the owner and expected behavior (possibilities) for each category for services you build.  The following is a simple proposed of categories


HTTP Code CategoryRemediation OwnerRemediation
SuccessEveryoneApplication or n/a
Business ErrorBusinessManual process or Application rule
Technical ErrorIT / TechnologyManual
Embedded RedirectIT / TechnologyApplication Library
NACK / RetryIT / TechnologyLibrary and/or delayed retry mechanism

Asynchronous Messaging.

You can create the same types of categories for message driven systems.  They can post return codes to reply / response queus or to event log capture.

Proposed Return Code Categories

Map each response code to one of the categories.  Make your application return the correct code for each service it provides. Implement the correct behavior for each service it invokes. The following is a sample list of proposed HTTP return code categories.

HTTP CodeReturn Code DescriptionDefault Contract Behavior
100ContinueTechnical Error
101Switching protocolTechnical Error
200OKSuccess
201CreatedSuccess
202AcceptedSuccess
203Non-Authorative InformationSuccess
204No ContentSuccess
205ResetSuccess
206Partial ContentSuccess
207Partial SuccessBusiness Error
300Multiple ChoicesTechnical Error
301Moved PermanentlyEmbedded Redirect
302FoundEmbedded Redirect
303See OtherEmbedded Redirect
304Not ModifiedTechnical Error
305Use ProxyTechnical Error
306UnusedTechnical Error
307Temporary RedirectEmbedded Redirect
308Permanent RedirectEmbedded Redirect
400Bad RequestBusiness Error
401UnauthorizedTechnical Error
402Payment RequiredTechnical Error
403ForbiddenTechnical Error
404Not FoundTechnical Error
405Method Not AllowedTechnical Error
406Not AcceptableTechnical Error
407Proxy Auth RequiredTechnical Error
408Request TimeoutTechnical Error
409ConflictTechnical Error
410GoneTechnical Error
411Length RequiredTechnical Error
412Precondition FailedTechnical Error
413Request Entity Too LargeTechnical Error
414Request-URI Too LongTechnical Error
415Unsupported Media TypeTechnical Error
416Request Ranage Not SatisfyableTechnical Error
417Expectation FailedTechnical Error
500Internal Server ErrorNack / Retry
501Not ImplementedTechnical Error
502Bad GatewayTechnical Error
503Service UnavailableNack / Retry
504Gateway TimeoutNack / Retry
505HTTP Version Not SupportedTechnical Error
N/AFailed to Route?

Final

blah blah and blah blah

Created 2016 Oct 04

Monday, October 3, 2016

Success and Failure in the world of Service Calls and Messages


Victory has 100 fathers. No-one wants to recognize failure.    The Italian Job
Developers and designers spend a fair amount of time creating API and Service contracts.  The primary focus tends to be around happy path API invocation with a lot of discussion about the parameters and the data that is returned.  Teams seem to spend little time on the multitude of failure paths often avoiding any type of failure analysis and planning.

At the API level: Some groups standardize on void methods that throw Exceptions to declare errors. The list of possible exceptions and their causes is not documented.  Exceptions contain messages but no structured machine readable data.

At the REST web service level: Some groups standardize on service return values other than 200 for errors. They do not document all of the HTTP return codes they can generate or their meanings.  API consumers are left to their own devices to figure out who owns failures and how they should handled.

Note:  I get a special kind of heartburn when discussing APIs folks who have no concept of partial success or success with soft errors.  Yeah, CRUD means you commit or you don't.  More complex applications have more subtle nuances.

Infrastructure Integration

We can do this on a per service basis or try and come up with some type of standard.  This is generally a good idea and makes it possible to plug behavior into infrastructure components without them having to know any details of the invocation or business process.
  • Can you build a system that only looks at the envelope (headers, return codes)?
  • Does it work for impartial systems like service buses or API routers?
  • Can we create a standard that lets us get statistics from cloud components like Load Balancers

Conceptual Baseline

Let's take broader view of service invocation and define possible end states for API invocation. APIs can end in success, partial-success fatal failure, recoverable failure and possible other states. Failure can be due to technical issues, bad code, defects, business rules or broken business processes. Failures and errors must be owned by someone either from the business or by some technology team.  

Lets use the following diagram as a starting point. We divide failure based on the owner of the triage and remediation.


Success

CompletePartial / Soft Errors
DB inserts should probably return a transaction receipt or the key to the updated data or the URL to retrieve the modified data. Success may return confirmation codes, correlation IDs, operation codes, text messages or message parameters. Some of these are used for audit and some are used to build a better user experience. This one seems to give some folks stomach acid problems.  They view success as something absolute. That is true for DB operations This may not be true for any type of call that can have soft errors where the back-end service can be partially successful or blocked in some way that doesn't immediately impact the caller.  We'll talk about the meaning of "success" later.


Business Failure

Business users own the rules, triage and repair of business failures.  Some business failures are common and may be handled through normal application behavior or some type of rework/manual business processes., In other cases they may they may rely on the technical team to acquired failure details and to apply a fix.  The best case is that business users can fix, restart AND terminate the business processes themselves.

Business rules are a normal part of computer programs.  Business rules may terminate an operational request in response to a business rule failure.  This is not a Business Failure for the purposes of this discussion if it is an expected type of behavior.

Business processes can terminate because of a business rule that may actually pass at some time in the future.  This could be due to a asynchronous workflow some type of failed requests or behavior related to soft edits or something else.Business processes can terminate because of unexpected data or state.  They may fail deliberately due to policy rules. 

Retry ReadyBusiness Triage
Retry Ready. Business Failures and Technical Failures may be indistinguishable from each other. Some business failures can be immediately retried or retired some time later. There are a whole range of reasons a business transaction can fail. Some business processes  fail, routing work to application error handlers or some type of manual rework queue. There may or may not be tools to fix data found in the Business triage when the business failure was unanticipated..


Technical Failure

IT users own the triage and fixing of technical failures.  Systems and monitoring needs to be built assuming there will be failures. These types of failures are often due to design defects, dependency problems, infrastructure issues or version mismatch. IT should detect and automatically fixes errors before business users know his is an issue.

Poison data failures may be either Business data or technical in nature.  Some technical triage may be required to determine the owner of service call failures that fail on retry.

Retry ReadyTechnical Triage
processes can fail technically because of network issues, remote system problems, resource constraints, asynchronous timing issues or other reasons.   Systems should build in automated retry that handles the bulk of these situations. Remote system exceptions, network connectivity,, unknown hosts poison messages and failed retires can force triage by the development or technical support teams.  Teams need to build in the logging and message captured needed to determine the causes of technical failures. 

Retry and Triage queues

Asynchronos messsaging make great resources for implementing automated retries and Technical and Business rework queues.  

Two Computers Meet in a Bar

  • How do they know if they are successfully communicating?
  • What tells them if they their conversation is succeeding?
  • Can they partially successful?
  • Who owns failure?
  • Who owns a processing failure

Are We Successful?

  • When another system receives the message?
  • When another system accepts the message?
  • When another system returns success?
  • When some other part of the processing is delayed?

Did We Fail?

  • When a business rule finds an error?
  • When there are soft errors?
  • When a 3rd party is down and we use default behavior?
  • When the receiver has to retry?

Plan Ahead

Owning failures is painful.  I've worked with teams where no one owned failure. Production issues were discovered by users and handled via email.  Everything was reactive. We just sort of planned as if our system could never have problems.  This was a ridiculous approach.
  • Identify Success Scenarios including partial success
  • Identify Business Failure Scenarios and whether they can be remediated by people or systems
  • Identify Technical Failure Scenarios and whether they can be remediated by people or systems

  • Build monitoring, statistics and remediation into the first release
Created 2017 Oct 3
Last Edited 2017 Oct 4
Added AutoHandle 2017 Oct 5