Here are 3 keys to faster application troubleshooting

Observability experts deep dive into common cloud native challenges, and how you can help your engineers avoid lengthy bridge calls.

Sophie Kohler | Social Media and Content Manager | Chronosphere

Sophie Kohler is a Content Writer at Chronosphere where she writes blogs as well as creates videos and other educational content for a business-to-business audience. In her free-time, you can find her at hot yoga, working on creative writing or playing a game of pool.

On: Jan 15, 2025

39 MINS READ

Tired of endless bridge calls that keep you up at night? Today, modern teams and organizations are facing mounting complexity in a cloud native and microservices world. Oftentimes, this results in the dreaded 400-person bridge call –  where engineers may find themselves facing long application troubleshooting sessions, scrambling to diagnose or resolve an incident.

In this blog, learn three tricks from Chronosphere’s observability experts for speeding up problem diagnosis:

1. Aligning your priorities
2. Embracing change
3. Adopting hypothesis-driven troubleshooting

You can check out the on-demand webinar here, or catch a transcript of the conversation below.

Current challenges we see our customers face

Julia Blase: Thanks everyone for joining us today. I’m Julia Blase, really excited to have you. Here’s a quick overview of what we’re going to talk about in this webinar. So, first, we’re going to talk a little bit about the market landscape and challenges moving to cloud native and working in microservices environments. We spend a lot of time talking to people in this industry about their problems, and what they’re doing. We seem to find that they fall into kind of three main groups, and we wanted to share what we’ve learned with you all, and hear how that resonates with you as well. We’re going to end that section with kind of where we see people getting stuck today, which is, as we call it, the dreaded 400-person bridge call.

We talk to customers who say they do end up in this kind of situation once or twice a week. Obviously none of us want to be there, and we want to talk through how that comes to be the case. And then, we want to move into how you can avoid that. So we see three keys to faster application troubleshooting, and really avoiding that bridge call altogether. Once we go through those, we’re going to touch briefly on how Chronosphere can help, and how the tools we build help address those three key components of faster troubleshooting. We’ll give you a quick demo of what that looks like in action. All right, let’s go ahead and get started. So first, let’s talk about the market landscape and challenges.

Today, what we see is companies operating with a ton of really high scale workflows, and that is high scale both in infrastructure, just like running on a very large set of infrastructure to handle the load, and high scale in terms of users or customers, right? So, if you’re facing a large customer base, you’ve got a lot of individual users who can all have different experiences with your platform. That can also be high-scale in terms of the data that you’re generating. If you are going through a typical day, you might generate petabytes of data where you’re used to generating terabytes. And in this new world, what we see people struggling with are three kinds of problems.

1. Poor reliability: You have a system, you are trying to manage this high scale workload, and it goes down at exactly the wrong time, right? You’re an e-commerce retailer, it’s Black Friday, and all of a sudden in the middle of Black Friday you lose insight into what’s happening, which can cause lost revenue.
2. Data explosion: We see data exploding, like I said, both in volume and in complexity, higher cardinality, higher rates of change, which can be difficult to deal with. Even if you do have a system that’s very reliable and standing up, if you’ve got a lot of data in there and you don’t know what to do with it, that can be a huge challenge.
3. Change: We see things changing because developers are being asked to fill more of that DevOps role. In the past, those roles were separated, but we hear a lot of people talking about the shift left. Individual developers are being asked to do on-call platform rotations and monitoring, and are responsible for system health. But, it’s becoming increasingly hard for them to do that effectively.

Finding success in a microservices environment

Julia Blase: We all want to be building code. How, given this microservices world and given all of these changes, do we manage this? So at this point, Nate, I think I’m going to hand it over to you and you’re going to talk a little bit about what we’ve learned in terms of keys to success in this new environment.

Nate Heinrich: Thanks, Julia. All right, let’s get into it. The first thing we’re going to talk about is aligning your priorities. And I like to start this with just a question: Would you rather alert on CPU, memory, network, disk usage of your service, or when your customers are actually having a bad experience? Now, that’s a bit of a false dichotomy, it’s probably a little bit of the former, but mostly the latter. So, this is what I’m going to be talking about – is that most of you will probably answer that as, I want to get notified when customer experience is suffering.

But how do we do that? We know that customer experience needs to be on target. We know the stakes are high and we know that those moments where we miss a good experience, there’s an outage, there’s something that’s impacting customers consuming our products and services. Those are the moments where they’re thinking to themselves: “Maybe I should go elsewhere. Maybe I should use something else.”

And those are the things we want to avoid, but it can be tricky to determine how to know when they’re having a bad experience and what we should target as a good experience. It comes down to understanding what’s good enough. What should we aim for? And a lot of the time, your first instinct is: “Oh, we should be perfect.” But it turns out that isn’t actually the right answer. We’ll talk a little bit more about that. And you need to understand, what level of performance your customers expect, what level of reliability we should target for our systems so that we can navigate that space between perfect and the expectations of our customers.

The myth about 100 percent reliability and maintaining your SLOs

Nate Heinrich: A couple of things to consider: The first is, there’s some really good research around the cost of every nine that you add to your reliability. Roughly speaking, each nine you add in terms of: “I want to be 99.9 percent available in these types of things,” and providing a good experience is 10X more costly for your team to achieve. So, 10X more in terms of time, effort, and cost. When you can start to quickly see the balance of cost-to-benefit, cost and time, benefit to customers, the more time you spend on reliability, the less time you’re spending on building the things that they want for your product to evolve.

There’s a common kind of comparison to help folks think about being okay with less than 100 percent and it’s just this interesting fact about pacemakers. Pacemakers seem like they should be very reliable, 100 percent reliable, but in fact they’re not. You can look this up on Wikipedia. In 2005, there was this research done and it turns out that pacemakers are 99.6 percent reliable and we as a society are okay with that. And there isn’t a huge problem with something so critical, having a reliability of 99.6. So, this is a commonly used fact. Understand that, 100 percent doesn’t need to be the target, and how might you figure out what your target is, and what can you do with that once you understand it.

So, we’ll talk about two types of work that are in the balance here: Only working on reliability doesn’t seem like the right answer. Customers: Your customers won’t feel like you’re listening to them. Competition can get ahead. These are the two things that could happen if you only focused on reliability and we’re targeting too high of a target. It’s also clear that only working on innovation, new features, these types of things, isn’t the right answer either. Reliability will suffer if that’s the case.

How to prioritize your customer needs

Nate Heinrich: That’s right. I love it. So I think I’ve convinced you that less than a hundred percent is the target, but what is it? And then how do you pick the things that your customers care about to measure to ensure that they’re getting that good customer experience? There’s a couple of ways you can go about this one, and we’ll talk a little bit later about some statistics. A lot of people have adopted, you know, service level objectives and measuring that customer experience. If you’re new to it, that’s totally fine. But you can look at competition. You can look at adjacent markets. There’s expectations out there that you can draw from to just get you in the ballpark. Even better, talk to your customers, talk to your end users.

It is a discussion, right? It is a friendly negotiation because what you’re trying to figure out is, what are you willing to tolerate in terms of a customer experience? We want it to be good for you at all times. And we also want to not spend all of our time on that reliability so that we can spend more time on innovation and get you the things that you’re asking for, being able to take those feature requests and spend as much time as we can on them and get those into your hands as quickly as possible.

These are two things that I start with. When you start to get those parameters from your customers and you start laying them out, you get that picture of: “Okay, this could be our first SLO. This is the thing that we are going to react to when there is a problem.” And to illustrate this very clearly, what these parameters look like, you can start to think about: This is actually an SLO that one of our customers is targeting. And it’s a pretty common one, right? You’ve got a few pieces of information here, a few things that you have to decide on.

Defining the right SLOs for your team

Julia Blase: I wanted to chime in there with one other comment. I think like what you’re talking about here with being able to work on and define that with customers, it also helps you like define it for your organization and I’ve at least seen customers have this realization where they start to find good SLOs, especially on latency, which feels so qualitative like people can write in and say, hey, your page feels slow. And it’s what do you mean, feels slow? And that can be really hard to debug and respond to as a developer or on a product team. And I see people over and over starting to really define these. And then the next time a ticket comes in saying checkout felt slow, they can actually verify that. They can track down that specific customer ID or request, look at how it fits into those parameters and say: “Please try to scram the jets to go fix something,” or “You know what, it was actually 192 milliseconds. So while it might’ve felt slow, that might just be you.”

Embracing change in the remediation process

Nate Heinrich: All right, moving on. Let’s talk about the second thing here: Embracing change. We just talked about SLOs, how they are excellent at providing high quality alerts, those high quality signals to let you know when customers are having a problem. But then what do you do? When you respond, get a couple things here. I think this is probably the most underrated yet most valuable thing you could do to help your on-call responders. I call it the kind of shortcut to mitigation and it all starts with this increase in change over the years, right?

DevOps, agile software development, it’s all about change, constant change to drive improvement, lots of ideally smaller incremental changes with a feedback loop to allow some course correction along the way. And with that increase in change, you can see it in the stats that over eighty percent of all user impacting incidents are due to change. So with that increase in change, with that statistic, what are some things that we can do to help with that?

If we put ourselves in our SREs shoes here, and you see the error rate of a service increase, the data is trying to tell you a story. With the telemetry itself, the metric we’re seeing here of this increased error rate is maybe a couple of chapters of that story. There’s more it’s trying to tell you. And being able to bring in some of that change information completely changes how you interpret and fill in the gaps of that story.

Centralize your changes

Nate Heinrich: I think the first one’s pretty obvious is you need to put all of your changes into one place. Ideally, you need to centralize them and you need to put them with your telemetry data so that they’re accessible to everyone as they’re consuming it. There’s a number of different types of changes. We talked about two of them, so being able to categorize them and tell them apart. So whether it’s a backend engineer deploying a service, whether it’s a front end engineer changing a feature flag or an infrastructure engineer making some changes to Kubernetes or data store upgrade, being able to quickly tell the difference between them goes a long way in understanding that story.

Managing change in a repeatable way

The data always changes - but your process doesn’t have to

Julia Blase: Because yeah, the data always changes. The system always changes. The only constant thing is that there are a lot of changes like Nate just said. So, we can’t guarantee that the data is going to look exactly the same, but we can give you a really good answer, process for triaging those changes and figuring out what’s happening. We can take that deep food of the expert SRE and codify it so that anyone can follow it.

And as we look at those steps, I can tell you what they are right here. So, this was pretty much it, right? As we talk to the people who do really well in these environments, they always start with asking what happened: “Was there a system change like a deploy? Was there an observability change? Like maybe I added a new label or a bunch of new values for a label in my system. Was there a data or workload change? Did I onboard a new region? Did a bunch of people start coming in from a new region? Cluster, that I hadn’t seen a lot of traffic in before.”

So, is there an actual workload or data change underneath my system? And then they take the thing that changed or maybe the set of things. So they take those three things that are the hypotheses. It’s one of these three.

The power of hypothesis-based troubleshooting

Julia Blase: And they start comparing, and they start to see if that change is common in one category or the other. So is that feature flag always in the errors and always in the slow things or not? Sometimes it actually can be easier to disprove like: “Oh, that feature flag is in both errors versus successes. Great. That’s not actually the cause of my change. Let me go back to deploy. And let me do the same thing with the deploy. Oh, when I do it for the deploy, I actually see something else. I see that there’s that region that’s actually different between those. Now I have a new thesis. My thesis is about the region. Now I can go do the same thing and I see, okay, that region is definitely associated with things that are slow.” So they take this hypothesis, they run these comparisons, see if it’s common or not, and work to prove it or disprove it. And they’re looking for outliers.

Conclusion

We talk about lowering the mean time to sleep or mean time to return to sleep, but it’s so much easier to process through some complex data, a lot of different hypotheses, change events, workload events, and get back to sleep if you have something that’s repeatable and systematic.

Catch the full webinar here and watch a demo of our features.