Can Solving Problems All Day Make You Go Mad?

What is the happiness of an engineer who only does triage and solving problems all day? How does this impact their well being and satisfaction with their job? In this article I share my experience on what it means to have a balance as an engineer who triages solves issues coming in from customer facing people in your organization.

Your days should be split equally between solving direct customer facing issues and fixing the underlying infrastructure so they never happen again.

Impacting Factors

---

When looking at any normal day, the amount of time one can spend in an operations role is based on events which arise. This list is about issues above and beyond what would be considered normal tasks.

• Were there more or less customer problems that day? Was there a holiday or is it a Monday?

• Are there environmental factors such as recent IT or other infrastructure changes?

• Was there a new product launch?

• Were there any incidents?

These things can impact what would constitute a "normal day" for someone woking Tier 3. It can be hard to predict what will come up in your day based on these factors. Start each day with good intentions and understand interrupts do happen.

What percentage of the time your Tier 3 engineers spend on triage is for you to decide with team structures and staffing levels. When interrupted days like this happen, as long as they are not the norm, you have a balance. If you feel like more days than not are taken up by Tier 3 work, it is time to make a change.

It can weigh on you

---

When people asked me what I did while working mostly Tier 3, my answer was usually: solve problems other people created. This is not always the most satisfying job as usually you are dealing with negative things. It takes a certain mentality to be able to deal with this kind of work. Hiring decisions should be partially based on the ability to handle negative issues for at least half their days. When interviewing for Tier 3 positions questions I ask often are:

• What is your emotional reaction when someone presents you with a problem?

• What is your response when you see the same problem coming up again and again?

Answers to these questions are crucial as it can tell you if they will find the job too stressful or not. I usually am looking for responses in the realm of being intrigued by problems and eager to automate solutions to issues.

Find the Balance and Stick to it

---

Look at the balance in your days to discover how much of your time is usually being dedicated to Tier 3 work. Try and adopt the Google SRE mentality where no more than 50% of your days should be spent solving problems. There is a reason why larger organizations have adopted this mentality: it works!

Work with the development organization to understand you have limits as well. When you have used up your "error budget" of 50% of your day on problems. The overflow should be taken on by development. As the organization sees this push of work impacting development they are more likely to want to work with you on why these problems arise and solve them.

Get Proactive!

---

One solution which gets positive results: empower the Tier 3 engineers to be able to quickly triage and / or solve the problems they see each day. Once they have the ability to say: we have seen something like this before, we can temporarily solve but here is our long term fix; this will make them happiest. This is usually accomplished via runbooks.

Make sure Tier 3 understands as much as possible of the software they are supporting and its backend. This will allow them to solve the problems faster as well as increase accountability and the need to reach out for assistance. When a Tier 3 team does not have the knowledge they should have a framework in place for being able to reach out to development and get the knowledge they need easily and be able to document into a ready made format.

Does the team need additional training on the backend to be able to more effectively triage and understand the real issue? This might better enable them to look at problems from a more technical angle and increase the accuracy of their diagnosis. As their skills increase you will likely hear things like: we could probably solve this if we just tried X. This is what you are looking for in an ideal Tier 3 engineer.