We’ve all been there. We’ve broken something. We know it’s our fault, we know we are responsible and yet we can’t bring ourselves to admit it.
What next? Hide it? Bury it? Hope someone doesn’t notice? When they do notice we feign surprise and wrinkle our brow and look into it alongside our colleagues.
“How could this happen? Who would do this?” We fear the inevitable barbs. We suspect though that in time, once we have it fixed, once the release is done, the bosses are happy, everyone has understood that they will forgive us. Perhaps.
So what went wrong? How do we stop this from happening again?
Many times we don’t take the time to properly understand the failures in process that cause failures in our development or deployment process.
Despite all the process advances in Agile and Scrum sometimes we just ignore the ceremonies or use the time for something else instead. Rather than improving our processes we bury our heads and pretend it doesn’t affect us directly so why should we care?
The boss wants to talk about Continuous Integration and Continuous Delivery. Everyone wants software development to go more smoothly, to go faster, to go more predictably. We see the examples of Google, Facebook, Twitter, Toyota Production System – “they do X hundred deployments a second, we should too! We can do this faster, smoother, safer than ever!”
The boss implements a top-down approach to process improvement:
“We’re Agile now so this means we don’t have to write anything down!”
“We’re Agile now so we don’t need to keep track of anything!”
These type of jokes go around and help lighten the mood that we can’t help feeling .. that of helplessness in the face of decisions and prognostications coming down from on high.
Continuous Delivery has an image problem.
Continuous Integration/Continuous Delivery (CI/CD) is not a one-size-fits-all solution. To take some systems to full CD it would be almost pointlessly expensive. Only true scale demands extreme measures (meaning the extreme effort it takes to accomplish) and most enterprises are not the same scale as Google or Yahoo or Facebook.
So then ask your CI/CD expert – why would you base your processes on theirs?
I often relate this to the Inverted Pyramid Paradigm. The Inverted Pyramid is what the big guys are doing – they have a incredibly complex product which is supported by many thousands of servers, architectures, systems, CI/CD, developers and serving it all to lots more people. So the size of organisation serving the customers is big but it’s still smaller than the audience – so it’s heavier at the top than at the bottom – Inverted Pyramid.
For most enterprises their most complex and expensive systems are not actually their customer facing websites – rather it’s the back offices and integrations. This is especially prevalent in finance and accounting systems, CRM and HR. Most of these systems have more development and support staff than they do actual users .
The Amazons or Facebooks of this world are website companies – they have a website and they sell or share stuff through it. They have millions of customers to support and the Continuous Delivery systems they have in place are specifically designed to deliver new functionalities rapidly to those millions of users with no downtime.
So sure, if you have millions of customers then put effort into Continuous Delivery for your website – this will pay dividends and you can copy the examples above. But for your book-keeping, accounting, HR and regulatory reporting solutions you have to be smarter. For most of your systems you have to be way smarter with your application of technology.
So where do we start with any CI/CD journey?
CI/CD is about control, repeatability and accountability. We want to make sure our systems are under our control at all times, we can repeatably apply changes across all parts of the system and we know who is doing what to our systems – this last piece is maybe less important when you have a Facebook style approach to product development (because changes are usually small) however in enterprise there is so much baked-in domain specific knowledge in developer changes it is always important to know who made them.
So – the first step with this journey is always source control. Find a system (git is a good choice) and put everything you have into it. Source code, text files, icons, images, documentation, tests, SQL, DDL, whatever you can lay your hands on. Don’t be shy – get it into source control.
Why source control? Well, without controlling the inputs to your system you won’t be able to do any of the rest regarding repeatability, control and responsibility. All changes need to be linked to a version in a source control system and that version can be built into an artefact which can be applied to a system.
This may sound blindingly obvious to the GitHub generation but you might be surprised to hear that a lot of professional IT people in the 21st Century still don’t use any source control for some or all of their code. Or if they do then they fear it and only use it as a last repository for their perfect code. Or they consider it to be something that cannot make a difference to the success of their deployments.
None of this is true.
Get a source control system. Use it. Build your unit tests, your integration tests, and your deployments all about what is in your source control system.