As a developer, it’s easy to get caught up in writing good code instead of writing good software.
We all love new feature work and the process of creating things, but the reason we show up for work every day is to build a product for our customers. That’s why it’s so important that development and CX work side by side: Without constant communication with the people who are actually talking to your customers, the wrong things get done.
The purpose of a support engineer in a development team is to have someone in place to quickly address the customer-facing product concerns that come up on a daily basis, and serve as the bridge between the support team and the development team.
Whether a customer is having problems because of a bug that’s been in the system forever or because of something that just broke, it helps to have a person on the frontlines who’s ready to respond, and who can harness customer feedback into thoughtful output. By bridging the gap between developers and customers—and having a focused plan for what to prioritize—a support engineer can ensure that the right problems get solved at the right time.
Currently, every member of Nutshell’s development team takes two-day shifts rotating into our support engineer chair. Here’s how we do it, and what we’ve learned along the way.
The origin story
When we first tested out a support engineer role in late 2014, there were only seven people on our Engineering team, and most of the requests coming in from our Support team were being handled by our senior engineers because they knew the code and could jump in more effectively. But that also pulled them out of the context of whatever they were working on.
A lot of the time when a new issue pops up, all the engineers want to jump on it because it's a shiny new problem to solve. But if something goes wrong in a part of the product that's not well-traveled and isn't necessarily very exciting—customers having problems with their own integrations or setup, for example—engineers tend to avoid it because they know it will be a pain to fix.
Basically, we were running into a situation where our Support team felt guilty asking senior engineers to fix a problem, but they had no other choice. And similarly, since no one was directly responsible for fixing these problems, engineers would feel guilty just letting them sit there, and we weren’t giving the Support team the resources they deserved.
Furthermore, since there wasn't anyone responsible for managing the queue, we'd treat every customer problem equally, which is never a good thing to do.
The idea was, "Let's specifically dedicate one person per week to handle these requests, on a rotating basis." This would encourage newer engineers to talk to our Support team more often, and it would also force them to deal with the hairier parts of our code. Looking at old code and finding ways to fix it can be a fantastic learning experience, no matter how long you’ve been on the team.
The rotation helped alleviate the load that was on Support, but we quickly realized that one week was too long for a role like this. For one thing, there was more of an expectation back then that the support engineer could tackle a larger issue during their rotation—something that might take four days to fix, perhaps.
The reality is, bigger projects like that don't work very well for a single person in isolation. Someone still has to review the code, which requires context-switching from other members of the team. And as great as it is for newer engineers to dig into old code and learn from that process, we don’t want them going into production data or modifying customer information without someone more senior being there to help them along.
A two-day rotation allows us to focus on well-scoped work that a support engineer can handle independently. We eventually decided that if something will require more than two days to complete, it shouldn’t fall on the support engineer’s shoulders.
How it works
As a support engineer, your day starts with a morning stand-up where you and the Support team review the current queue of one-off customer requests, which we call “escalations.” Obviously, if a fire breaks out and there’s a significant customer-facing issue that’s affecting a lot of people, that immediately becomes the priority.
The way we prioritize our escalation queue has evolved over time, and it’s something we're continually trying to get better at. A typical customer request can be something like, "Hey, there’s an obscure third-party integration that isn't working well for me in Nutshell." The question then becomes, do we spend two hours trying to solve that issue and make that one customer’s life better, or spend four hours fixing an issue that multiple customers have reported in the past two weeks, and could potentially have a positive impact for a lot more of our customers?
Ultimately, not all of those one-off requests from customers will be worthy of the support engineer’s attention, and won’t be solved at a code level or product level in the short term. We deal with that internally by splitting escalations into two batches. If we decide a request isn’t worth acting on immediately, we’ll let the customer know that we heard their concern, but we won’t be able to have a fix for them in the near future. As long as we’re candid and communicative about the issue, we consider the request to be essentially resolved in that conversation.
Then, we have a separate queue where escalations go when we’ve determined that 1) they’re legitimate concerns that should be fixed in the product, and 2) they’re small enough for a support engineer to complete during their rotation. These tickets tend to be quick feature iterations, and the idea is that they should take less than a day to resolve.
Typically, the support engineer spends half of their day debugging and working to fix these escalations, and the other half fixing the long-running, smaller issues in the product that we want to get to.
Over time, I’d like our support engineers to spend closer to 80% of their time on those long-running issues, and reduce the amount of priority we put on escalations. That will inevitably require our Support team to let down angry customers more often, but I think a support engineer can have the greatest impact if they’re focusing on problems that all our customers are facing, rather than fixing issues that only affect a handful of users.
What we've gained
For me, the most valuable result of putting our support engineer rotation into practice is having a more cross-functional team approach. A big part of the role is simply building camaraderie between the development team and the CX team, breaking down our departmental silos, and creating a space for us all of us to work together.
Our Support team has to deal with extremely difficult problems raised by customers, and we’re dealing with extremely difficult problems of our own. If we’re just working with our headphones on all the time, isolated from the customer-facing team, we’ll never understand each other. Having someone in that support engineer chair at all times has helped us build team empathy, which is huge for company culture.
Plus, we get fired up when we actually solve customer problems. The support engineer role is a good grounding experience to remind us why we’re here, doing what we do.
Of course, there will always be challenges inherent to a role like this. We have 10 developers on the team now—which means you're a support engineer once every three or four weeks—and I think everyone on the team gets frustrated that they can't accomplish more in that two-day stretch. It can also be difficult when you’re heavily involved in a large project, and then you have to take a two-day break to completely switch gears, going from heavy design and development to just putting out fires. We’re very flexible about the schedule, and we’ve skipped people in the rotation if they're in a good flow on an important feature, but it’s still a jarring transition when you’re pulled away from something you’re doing deep work on.
A new perspective
On the surface, it may seem like putting all your developers on a support rotation is a net productivity loss, and that’s probably the biggest reason why a lot of companies either don't do it at all, or commit a single developer to that role at all times. (We tried doing it that way ourselves for a while.) But sometimes looking at the straight numbers doesn’t paint the whole picture.
What we’ve learned is that building a support engineer rotation not only makes the development team far more effective in dealing with the CX team's constant influx of customer issues, it also keeps engineers focused on what's actually important, which is building a product for your customers.
The best part is seeing that user focus carry over beyond each developer’s two-day stint in the support seat. When you return to your regular routine, your approach to developing code is far more customer-facing than it was before, which has a positive impact on all your development work, not just the portion you spend on fixing what’s broken.