Do You Know the Real Cost of Fixing Bugs?

Understanding the impacts of different test strategies

We’ve got a bug! Let’s fix it now.

How many times have you faced this scenario? For me, it would be at least a million times. But not every time was the same; bugs can happen at different moments, and how you deal with them contributes to the cost of fixing them.

Over the years, we’ve advanced quite a lot in software quality. We benefit from many techniques that avoid unpleasant bugs in production—for example, unit testing, automated testing, integration testing, etc. The question is, why do we still struggle with software quality?

It’s a vicious cycle. You’ve always got more to do than your capacity. The options for dealing with this situation are simple: you accept reality, prioritize mindfully, or compromise. Sadly, quality assurance is often the first to suffer. 

You often don’t see the costs of compromising testing immediately, but it’ll dramatically hurt your performance when the bill comes.

I’ve experienced many ways of dealing with defects. I’ve learned my lesson, and I’d like to share how you could sustainably deal with quality while being Agile. By the end of this article, you’ll get practical ideas regarding quality assurance which you’ll be able to apply from now on.

Understanding the Impacts of Quality Assurance

Before we jump into how you could work, reflect on the following questions:

• How is your current quality assurance fairing?
• How do you define your quality standards?
• How and when do you deal with identified defects?

Take these questions seriously. They’re pretty important for how you work. Now, let’s look at the following graphic and reflect on how much it costs to fix your issues.

This graph is powerful. Most defects are introduced during coding, and sadly they’re often identified later in the process. This scenario leads to ineffective bug fixing. 

Crafting a strategy that helps you identify bugs quickly is invaluable. I cannot overstate the importance of it. 

“Quality is not an act. It is a habit.” — Aristotle

Let’s understand a common way of testing software:

1. Code-reviews: Developers review the code from their colleagues. Yet, code reviews focus on code quality instead of functionality itself. Here lies the most significant opportunity for avoiding issues.
2. Unit test: Developers write a unit test to ensure a minimum level of test coverage. This helps avoid many errors, but it’s unfortunately one of the first parts to be compromised when deadlines get closer.
3. Functional test: Someone other than developers evaluate how the feature delivers the expected behavior. A typical scenario is identifying a design flaw or a missing requirement. This way, developers have to throw work away and redo the job. At this level, bug fixing has become more expensive because everything has already been done.
4. Acceptance test: Traditional companies require formal approval from stakeholders before releasing anything. Acceptance tests are ineffective and demotivating because it causes rework and confusion. Any defect found at this stage will be costly to fix because it means going through everything again.
5. Smoke test: After the release, teams manually test critical features and changes to ensure everything works as expected. After release, bugs put a lot of pressure on the team to fix them immediately or to roll back to the previous version. You better hope nothing is found during these tests. 

Quality assurance is vital. Without it, you’ll spend too much valuable time fixing issues that should not exist.

I don’t mean products should be completely issue-free. By nature, the software will have bugs, and that’s fine—but the team should have a minimum quality standard and create a meaningful strategy to avoid paying more than they should fix eventual issues.

Combining Agility with Quality Assurance

If you were to reflect on the traditional way quality assurance happens, which improvement opportunities would you see?

Traditional ways have too many handovers and lack collaboration.

The core of agility is accelerating learning time and reducing waste. I believe you can only do that when you collaborate closely with key people. Let me give you an example.

Some time ago, I joined a company to replace an experienced Product Manager. With fresh eyes, something caught my attention. Jira had a gigantic workflow. As I remember, it was something like this:

• Backlog
• Ready to Sprint
• To do
• In progress
• Code review
• Browser testing
• Quality assurance
• Product Manager Approval
• Business Approval
• Done

That annoyed me. A lot. Too many things had to happen before I’d interact with the product. I was confident this scenario would entail rework and low collaboration. I was new to the team and didn’t have any evidence it’d be a problem, but my guts told me it’d be a big issue.

My approach wasn’t to convince them to change, but rather to act differently. I sat closer to developers and attended all daily stand-ups. I’d always suggest taking a quick look at their work and providing feedback. This attitude allowed them to adapt immediately instead of waiting for everything else to happen.

Within time, we removed the column “Product Manager Approval.” Then, I encouraged developers to exchange with key business people as often as needed. This simple attitude gave them a chance to learn what mattered for business. And then, we removed the column “Business Approval.”

By increasing our collaboration, we reduced the number of bugs and improved our capacity. We still ensured our minimum quality standards were respected, but prioritized short feedback cycles. 

We learned that the faster you receive feedback, the more waste you avoid.

“Be a yardstick of quality. Some people aren’t used to an environment where excellence is expected.” — Steve Jobs

Be Careful. Quality Is Non-negotiable

At some point in time, you’ll face a tough decision. Choosing between pissing off stakeholders by not meeting their output expectations or pleasing them and compromising quality. No matter how intense the pressure is, don’t compromise quality. This can trap you forever.

The following are the effects of compromising on quality:

• Context Switch: Identifying defects too late will cause massive context switch costs. For example, when a developer finds an issue with the code review, the knowledge is fresh, making it easy to fix. However, when you find a defect three months later, maybe another person has to work on it, or the other doesn’t even remember.
• Firefighter Effect: How often do you encounter urgent issues? The more often you have, the lower your quality is. I’ve been part of some teams where we’d expect critical issues every week. The system was a low-quality legacy, and we’re doing our best to establish an acceptable quality level, but the critical bugs often got in our way.
• Fragmented Sprints: When your quality level is low, you’ve got no choice but to mix your Sprint with frequent bug fixes. This is confusing and fragments the team: someone is doing the cool new feature, while somebody else is dealing with an annoying bug. 

Compromising quality will lead to problems you’ll hate dealing with. Ultimately, you’ll reduce your chances of creating value faster.

“Without changing our patterns of thought, we will not be able to solve the problems that we created with our current patterns of thought.” — Albert Einstein

Final Thoughts

Uncovering bugs as early as possible will save you from a lot of trouble. Does that mean you should always invest in high-quality features? No. You’ve got to be mindful.

Remember, the key to Agile is accelerating learning and avoiding waste. The reality is that you’ll lack evidence that features deliver the value you expect from it. The less evidence you have, the less you should invest.

My approach is straightforward: 

1. Build to learn: Accept creating tech debt when necessary. Your goal is to understand whether that feature will yield the expected outcome. When the outcome is too shy, you better drop the initiative and not invest more time into it.
2. Build to last: Once you’ve enough evidence, you’ll build the feature correctly. You won’t compromise on quality.

Invest in winners, and drop the losers. The key is learning how to identify the two.