The Physics of Software Bugs

I graduated from Caltech 33 years ago, so I'm not sure if I've always been this way, but I was definitely trained to think this way there. So from first principles, this is how to cut your bug backload in half.

Principle 0: You will have bugs.

Humans err, computers do what you tell them, not what you want. Being a software engineer is a bit of a racket, you get paid to make the bugs, then paid again to fix the bugs. AI assist lets you generate weirder bugs faster, it doesn't help yet. Deal with it.

Principle 1: The business of your company is business.

This is a riff off of The business of America is business, a phrase long used to explain why our laws are the way they are. It's a tautology, so it's obviously true, but its relevance is that you need to remember that it's not personal, it's your job. The processes of your company is for making money. Companies that don't make money, don't stay in business. Processes that make it harder to make money should die.

A specific example from a QA contact, he was frustrated because there was an engineer who only worked on bugs marked "release critical".

Of course the engineer did, and he got away with it, because that was what the business wants. That said, it drives us to my next three principles.

Principle 2: Your bug database is about communication

The business needs to know what's going on. Don't lie in your bug database. The business needs to track where your development is at, and the number of bugs and their lifecycle is an obvious way to do that.

The bug database is how you do that. Or rather, the metrics on the bug database are.

Don't be stupid though. I worked for a large stupid-national that had a whole bunch of voodoo around their bugs, and the incentives were fake. Rather than encourage honesty, it encouraged each engineer, software or QA to mark the bug state for the minimum amount of time.

At the individual level, a bug has to be reproducible or it can't be worked on. Can't reproduce it? Pierce Wetter fixed it with his mental powers, prove me wrong... Still there, ok, how did you do that?

At the individual level, bugs are the way for QA and software engineer to communicate back and forth about a bug.

Principle 3: Humans have leaky brains.

To work on most software, you have to pull the code into your head and simulate it through the bug. Its easiest to do that the more recently you've worked on that code. Cosmetic bug from 2 years ago? Fuck a duck, I dunno, let me grep...

Principle 4: Bug Priorities are nonsense, use FIFO for 95% of all bugs.

Most businesses who use bug priorities use it as an excuse to not work on certain bugs.

There are really only 3 useful priorities: Holy Shit!, FIFO, and Duly Noted/Not gonna happen. The problem with having different priorities was illustrated for me at the textbook rental company because there was a low priority bug relating to printing out the shipping label for returning the textbook at the end of the semester. After 2 years, the bug hadn't been fixed, meaning many customers had to call, costing the company $30 in tech support call fees each cycle and almost each customer.

The CTO and CIO went apeshit, and instituted 0/1/close, added a service level agreement that all bugs would be fixed in 90 days, and saw engineering productivity go up. All that fucking around with bug priority was wasted effort. Forcing bugs into the 0/1/close mindset made everyone communicate honestly about the bug.

In practice, the 0 status was never used, because everyone in engineering all knew about the Holy Shit bug, so going into the bug database to mark it priority 0 wasn't useful work. Marking things closed forced engineering and product management to be more honest with each other, which was a good thing.

Principle 5: Features are up to the business to schedule, Bugs are up to engineering to schedule

It's common to have a "bug" that's really a feature. If something needs to be improved, that's not the same as needing to be fixed. That's not up to engineering to decide, it's up to the wider business. Though I always allow for inspiration with my engineers, if the business knows that feature X is going to be replaced by feature Y, it doesn't make any sense to improve feature X unless feature Y is delayed.

In UI/UX, improving something often means working with design, so that means its not strictly software, and it's a multiple step process.

So one of the first things I do when seeing a bug database is to scrub the features into a separate category, and push them to product management for stack ranking, then I pull them back into the feature backlog and schedule them (see principle 7 ahead).

Principle 6: Bugs need to be actionable

Actionable is a term I stole from the book Getting Things Done, the book is about time management and realizing that if something is actually a bunch of things or a process, it can't be worked on. So you have to split multi-step tasks into individual actions. Once that's done, it's faster to get large things done, because you can interleave them.

Making things actionable also relates to the feature thing I just mentioned, because if the verb for the next action is anything other than "fix", it's not a bug. Often, bugs won't be actionable because they're a side effect of some other sub system that needs to be improved. That's ok, make an entry for the other subsystem, mark the one bug as a sub-bug of the subsystem bug, and you will be organically driving improvements to that subsystem.

I use the labeling feature in Jira to label the bug appropriately when this happens:

next-reproduce: bug isn't reproducible from description.

next-infrastructure: bug requires system X to be built, linked to this bug

next-design: bug needs design help (UI/UX bugs)

next-other: bug dependent on other department, see bug linked to this bug

Then engineers only have to work on bugs without labels. This saves them the friction of scrolling through a bunch of stuff they're NOT working on. This seems like no big deal, but humans like to see their worklog shrink, not stay the same, so by making everything actionable, the engineers know they had a good day when they go home with a smaller list than they started with.

Principle 7: Humans can only do one thing at a time

This is a stupid manager problem. Yes, engineers can often fix bugs in parallel with other development. Throwing bugs in the database and hoping they will be somehow fixed by osmosis is just stupid. If you want bugs fixed, schedule them. Kanban is a useful technique for this.

Worked at many companies where the bug database was irrelevant to software engineering because no one looked at it except QA. Don't be one of those companies.

Principle 8: Use Bugs to Break Silos, have happier engineers

It's boring to work on the same thing every day. It's natural to fall into the Tar Baby trap of always assigning the same type of bug to the same engineer. It may be more convenient in the moment, but it screws you long term. If Bob is the only one who knows the print queue, Bob is the only one who knows the print queue.

Assign Alice to a few print queue bugs. Will she be as fast as Bob? No. Does it matter? No. Sure, she'll have to talk to Bob for 5 minutes, maybe 10 minutes. But now you have 2 engineers who know the print queue. It took Alice and Bob 10 extra minutes out of their day. BFD.

Don't fall into the trap of only giving the small bugs to Alice, or not giving the small bugs to Alice. A small bug is the perfect learning vehicle, use it to ramp Alice or any other co-worker up.

Now when you are scheduling your next sprint, you can divide the print queue tasks between Bob and Alice in parallel instead of serial. Finally, once Bob knows other engineers will be looking at his code, he'll write better code, because he knows Alice might be reading it.

This is better for Alice as well, because she was bored at working on the framzizzle system anyways, working on the print queue is a nice change.

Conclusion

If you follow these 8 principles, your development speed will accelerate, you will have fewer issues, and your backlog will get crushed.

I extended this article here: The Physics of Bugs, round 2, some implications