Thoughts On Measuring Software Quality
When you think of the word “high quality”, what comes to mind? As I wrote that to you, I looked at the notebook on my desk.
It’s an endlessly reusable notebook, and it’s my all time favorite purchase under $100 for the following reasons:
- I love the feeling of starting a new notebook, but have a history of abandoning them. This notebook lets me start a ‘new’ notebook any time I want, just by erasing what’s there.
- I identify as a minimalist and hate wasting paper. This notebook lets me write as much as I want without wasting paper.
- Not being afraid to waste paper has given me the freedom to scribble ideas and draw diagrams of my thought process, which has helped me endlessly at work.
If I were to rate the quality of this notebook, I’d give it 6 out of 5 stars. I wouldn’t change anything about it. In fact, I am mildly offended when I read Amazon reviews about it that are less than 5 stars. That being said, the reviews cover some really good points, and demonstrate that a product can meet some people’s needs whilst falling short of meeting other people’s needs. Some examples:
- “The spiral is on the right of the notebook. I’m left-handed and the pages can’t be turned upside down, which means it’s uncomfortable for me to use.” (they now have a top spiral book).
- “I feel embarrassed taking this to business meetings because the cover is plastic, I would prefer something like leather which would look more professional”.
- ”I’d love for the app to send line drawings instead of PDFs or JPGs, so I could then use both OneNote or EverNote natively”.
Asides from practically waving the Rocketbook at you, this story has implications for how the tech world tends to think about and measure software quality.
In the tech world, software quality is often described it terms of how well a software project does what the requirements says it should do.
None of the reviews above, good or bad, rated the quality of the product in terms of how well it conformed to the product description. No one wrote “Yep, this product is exactly what the product description described. It DOES have 36 pages. You CAN wipe it clean with a damp cloth and you CAN save your notes as PDFs or JPGs. 5 stars!” Instead, the reviews rated the quality of the product in terms of how well it let them to do the things that mattered to them.
So why do we rate the quality of our software like this? Why does our measuring stick start from the ‘solution’ level, instead of the problem or person level?
Measuring the quality of our software by the number of defects it has, the number of features, cost, time, best practices, performance etc tells us nothing about why we’re trying to do these things, and so it’s difficult to prioritize, eliminate or make thoughtful trade-offs between them.
Knowing what matters to people and why, allows us to make better decisions down the line when it comes to making trade-offs that impact customers, teams and organizations.
- Zero defects is critical to surgeon performing heart surgery with the help of medical software than it is for most people using social media. So we really don’t want to cut corners, no matter the deadline.
- Highly performant, optimized code matters more to organizations facing competition for a high traffic system, like Google or Amazon etc, than it does to your average developer writing about tech on their personal blog.
- User friendliness is more important to people who use software for most of the day, like cashiers who use cash registers in a fast-paced environment than it is to people who only use the software for ten minutes a day.
- The number of features matter more to people who actively need them (e.g. text editors for programmers) than it does to those who only care about a handful of those features (text editors for capturing thoughts).
So, whenever you see/hear a requirement or a quality statement, ask yourself and your team the following questions:
- Who is the person behind the requirement or statement of quality?
- What impact would not meeting this objective have for this person?
- Who are we excluding and is that acceptable?
- How can we work towards a win-win situation for everyone (what trade-offs can we make to achieve this)?
P.S. If you get a chance, read the Quality Software Series by Gerald Weinberg, it inspired this article. Some books are gold mines in that there are a few great nuggets. Other books are coal mines, where every shovel has something useful. As one of the reviewers in the preface says, this series is a bunch of coal mines. It amazes me how the very best ideas about software come from the oldest software books, here’s to hoping they become mainstream.