DevOps and Segregation of Duties
It depends on your definition of “duties”
This question almost always comes up when I present the culture of DevOps to customers. After I talk about cross-functional DevOps teams that are responsible to design, develop, build, deploy, and run their own applications they invariably ask, “How do you implement segregation of duties when developers have access to production?” Customers shutter at the thought of giving developers that much power. I don’t blame them, you should not have anyone on your team pushing code to production without the appropriate checks and balances in place anymore than you would push untested code to production. (…you don’t do that do you?)
This is where having an automated Continuous Integration and Continuous Delivery (CI/CD) pipeline offers the opportunity to run tests and scan for vulnerabilities and add other gates that must be passed before anything reaches production regardless of who is requesting it. The problem is that none of these automated gates can detect nefarious code. That requires something smarter, and until we have AI that can detect this — and we will eventually — we need to get a human involved.
What does Segregation of Duties really mean?
Segregation of duties has nothing to do with not letting developers have access to production. Segregation of duties is about getting a second set if eyes on everything that goes to production so that no single individual can sabotage the system. This is why there are two physical keys to launch a nuclear missile and why the keyholes are further apart than the human body can possibly reach. It ensures that at least two people agree that launching a intercontinental balistic missile and risking thermal nuclear war is a good idea. (…let that sink in for a moment)
Traditional solutions are too limiting
The first thing that many companies do is assume that “duty” equals “job role” so they restrict access to production by developers allowing only operations people access to it. But this just slows things down. Some may only allow developers to use CI/CD pipelines to deploy to production and then they further restrict developers from modifying any pipeline definitions as an extra precaution.
That works until two operations employees decide to sabotage the system. Then what? Maybe we only give operations employees half of the password like the nuclear key so that it takes two of them to change something. I’m joking, of course, but you get the idea that job role has nothing to do with limiting the risk of a bad actor.
It’s not Dev vs. Ops… it’s requestor vs. approver
The “duties” in segregation of duties doesn’t have to be a job title. It just has to be a different actor. The person who is requesting that something be done cannot be the person that approves the action. It’s really that simple.
One way to do this is to adopt practices like “GitOps” following these three simple rules:
- Nothing gets changed in production unless it is checked into GitHub
- Nothing gets merged to a trunk branch in GitHub without a Pull Request
- Pull Requests cannot be approved by the person who created them
That’s all you need really. You can have truly cross-functional DevOps teams that design, code, build, deploy, and run all of their own work and still have segregation of duties where every Pull Request is an opportunity for a code review and approval by someone other than the requestor.
Conclusion
Using this workflow, it doesn’t matter if the change is coming from your cross-functional DevOps team, a dedicated development team, or a dedicated operations team, every change gets looked at by someone else before it gets approved. Voilà… Segregation of duties with DevOps!
