As most Ops guys know Code deployment can be a challenge, especially when the code relase is dropping 100’s or 1000’s of changes to the product into production in one large code push. While I’m a huge fan of Devops and continuous deployments, realistically a lot of organizations are still deploying code the old fashioned way. Nate over at TechOpsGuys recently ranted about organizations that his company relies on pushing production code on friday and breaking his companies website (http://www.techopsguys.com/2012/02/03/dont-push-code-on-a-friday-damnit/). This rant brings up a few things i’ve been thinking about for a long time and felt needs to be expressed more clearly in a blog post:
1. Friday Code Deployments or Large code pushes are going the way of the dinosaur, more frequent smaller code releases with feature flags, continual integration testing, etc are becoming the norm.
2. If your software product is relying on services via API’s for your product to function or provide availability you need to start thinking differently.
Code Deployment in Devops
With the push for Devop’s in SAAS and B2C Business you are seeing more and more companies going to continous integration and continous deployment. In fact some software companies won’t even show a new developer where the bathroom is until they push code into production. (http://www.scottporad.com/2010/11/01/cheezburger-network-doesnt-show-its-new-employees-the-bathroom-until-theyve-checked-in-code/)
Having been involved in code deployment for over 5 years in a SAAS environment, I much prefer the smaller code deployment without taking major downtime of a site or service. Rolling in features bit by bit and turning them on when their fully deployed or turning them on to a select set of beta users makes it so much easier to test features, functions, etc with real production load and know how these systems operate in production. Plus if you find an issue once you’ve deployed the code you can roll back to the previous code by redeploying from source or fix it and push the code out.
Relying on Third Party Services
Once you understand that code from a lot of startups is being deployed all the time (Etsy pushed 10,000 code changes in 2011 (http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/). You need to start thinking about how you integrate with these services differently. If you main home page pulls in a twitter status updates from your CEO you need to make sure the following happens:
1. When the API is functioning, display the data (Yeah i know Duh)
2. When the API is not functioning that the following happens:
A. Your website still loads and doesn’t throw a nasty 500 error
B. Your website doesn’t have an obnoxious hole with a 404 or 500 error in the sidebar or header of the site.
This means your developers must be planning for the inevitable situation that the API is going to break, the way you call the API will change, or plagues of locusts have infested their datacenters. Your developers need to focus on thinking about all states of the service and the behavior they want their application to exhibit when the failure occurs.