Break the Glass

I’ve been helping one of our clients with hiring a new devops person. I mentioned to a teammate that one of the candidates taught me something new, which is always a good sign. This jogged a memory of a previous colleague’s interview that taught me about a technique they called “breaking the glass”.

It’s 2 AM. Do you know where your org chart is?

Your phone starts blowing up with alerts. You rub the sleep out of your eyes and begin to dig into the problem. Hopefully, this is a rare occurrence for you, but anyone who has been on call can commiserate that it’s not usually a fun time.

The only thing that makes it worse is realizing you don’t have enough access to actually fix the problem!

So you have to wake someone else up. Who may even have to wake a third person up to approve your access or take care of the problem for you.

Controlling access accidentally became the problem

In this situation, the reasonable practice of controlling access is getting in the way of the job. Giving out minimal access is a best practice, but there has to be a workable middle ground. This is where this technique comes in.

The technique is to give your staff a quick way to temporarily grant themselves elevated credentials and still have accountability. When a team member needs more access they must provide a reason for the temporary access, which is quietly communicated to several other team members.

Quietly, in the sense that no one is woken up, but not so quietly that it goes unnoticed. It should be an in your face email to the whole team and a couple levels of supervisors. Probaby should also blast it into an approprite Slack channel.

This accountability ensures no one can secretly abuse the feature to get around their limited access for nefarious purposes. It’s obvious who broke the glass and if there is any question everyone knows to audit the system.

In Case of Emergency

This technique was named after the venerable “break glass in case of emergency” boxes containing axes, fire hoses, and/or jokes, but it can serve a really useful purpose beyond just fixing the outage.

With this emergency release valve of sorts, your team can confidently keep individual access way tighter and more restrictive than would otherwise be practical safe in the knowledge it won’t ACTUALLY get in the way of day to day operations.

Now, this should stay for emergency use only and if it is used often it is a good sign your team’s access is too restrictive and should be re-evaluated.

Useful in other situations

While we have been mostly talking about computer security credentials and access in the realm of operations, this technique can be useful in internal applications or processes as well.

Sure the HR Manager is usually the one to approve something, but in a pinch give a “break the glass” way for the entire HR Department to do it. Solves a lot of access juggling issues when the manager needs to take unexpected PTO.

Can you think of any other interesting ways to use this technique? Reach out on Twitter or Mastodon and let me know!

Frank Wiles

Founder of REVSYS and former President of the Django Software Foundation . Expert in building, scaling and maintaining complex web applications. Want to reach out? Contact me here or use the social links below.