Author: Mo

Our team have been working on a report. It’s a fairly simple report driven by a SQL stored procedure that collates a list of accounts, and the customer(s) that hold them. Every account will have at least one customer but could have up to four customers associated with it.

Our test approach was fairly light-weight; make sure the report contains the number of rows we expect and then dip-check some rows at random* to make sure the content written out is correct.

^{*random samples not entirely random, but from identifying different classifications of test data and checking a few of each.}

Some years ago I attended a talk by Michael Bolton (no, not the famous singer) on testing ‘deep vs shallow‘. He described an approach to risk-based testing where you cast your test scope as broadly as possible but with shallow touches and then dive deeper on any areas of higher risk identified through error-guessing, specific areas of risk to the business/your stakeholder and customers or from spotting issues in your shallow testing that need more thorough investigation. It was an incredibly energising talk that amongst other things allowed me to put words to the risk based testing approach that I had been using for a good number of years knowing that software projects rarely have the time, budget or will for exhaustive testing.

Anyway our sample checks on the per-row content were fine, but our record counts were not. From what we had predicted we were expecting far more rows than were actually being written out in the report, which is a fairly fundamental problem to the user story.

So we took a deeper look, why weren’t the counts marrying up? First we checked the SQL query we were using to assert the number of rows to be returned, and had to check the query we were using against the requirements of the user story. Did we have our filtering criteria correct? Were there statuses or soft deletes that we hadn’t properly considered? Was there a type of account or customer that should have been included that we had excluded? We made a couple of subtle revisions to our test query and the discrepancy came down, but the difference still wasn’t 0. Someone asked me “is a <1% discrepancy OK? Can we sign this off?” but in our operating environment an unexplained discrepancy of any amount wasn’t acceptable.

Confident that the requirements driving our query matched those in the story, we looked at the stored procedure itself to see if that was filtering anything that we weren’t (correctly or not), but there was no observable difference.

The next step in our troubleshooting was to look at the data itself. Compare the accounts written out from our query to those from the SQL stored proc, and that’s when the penny dropped for me.

I kicked myself, hard, because I realised a PICNIC error and burnt at least an afternoon on it than I wish I hadn’t.

I’m sure some of you reading this will have already sussed it given clues from the title of the blog and the requirements I’ve explained.

But yes, when I compared the data in the report to the detail underneath my checking query I realised that some accounts were shown more than once. I had realised that my count(*) was missing the ‘distinct’ SQL operator. Thanks to the join in my query to return all the customer details where there was more than one customer per account then the account was being counted [customer] times in my SQL, whereas in the report the account was only being written out once as per requirements.

Or in simpler terms, my test assertion query was using

select count(accountNumber)

where it should have used

select count distinct(accountNumber)

and boy, did I feel like a dunce.

But why have I bothered writing out a long-ass blog for such a simple mistake? There are two major reflections that I want to share or reinforce:

1. Not all ‘defects’ are software faults. If a defect is considered an unexpected outcome, this is a great example as we expected a certain count of rows but received a different one. The issue wasn’t due to the software under test (or the SQL stored procedure in our case) but due to the test expectation. A test defect is just as valid as a software or environmental defect and needs to be highlighted for future learning. After all we don’t raise defect work items for shits and giggles, but as a future reference and learning item.
   
2. To err is human; mistakes are how we learn. I kicked myself for missing the ‘distinct’ condition but I won’t make that mistake again (any time soon, anyway). Following on from point 1, talking about that mistake and highlighting it helps others to learn so that if they’re facing similar inconsistencies it gives them an avenue to consider. I’m lucky that I feel a high level of psychological safety in my role and I hope that making my “silly mistakes” visible will help enable others to discuss their doubts or mistakes.

If you were in my shoes, would you have discussed this error among your team or publicly? When was the last time you kicked yourself for what you think was a silly mistake? Did you let anyone know, or keep it to yourself and carry on?

If you’ve taken anything from this blog or have any disagreements chuck them in the comments, or if you find yourself agreeing with the sentiment or nodding along feel free to share the post with your local QA or development community

Many of us will have encountered the project management triangle in our time. The idea behind it is that any project is constrained by 3 factors (Scope, Cost/Budget, Time) with the quality of the delivery determined by them. If the scope increases, you also need to increase the budget and/or time to deliver the project to the expected quality. If the available time is decreased, you also need to either decrease the scope or increase the budget else there is a risk to quality of the delivery.

It’s not a new concept. According to Wikipedia the triangle has been referenced in various ways and in multiple publications for over 60 years. I’ve discussed the ideas behind it with project and programme managers multiple times, but rarely when the project is running to plan. “Choose two” I’d often say “the project is behind plan, so you need to be prepared to flex on time (give us more of it), scope (expect less of it) or budget (we need more people/tooling.)” This works the other way as well, if you want to increase the scope, you’ll also need to increase either/or the available time and budget to ensure you have a delivery of the expected quality. Such is the beauty of a triangle.

Often I’d find that cost/resource was a favourite vertex for managers to look at increasing but there’s definitely a horizon above which throwing more people at a problem won’t lead to it being solved any quicker (which also leads to one of my favourite retorts “if you gather a team of 9 women we still won’t be able to make a baby in a month”).

Failure as an (unfortunate) outcome

In my time I’ve seen many teams (including mine) fail to deliver projects on time, to quality and budget. In fact I’d go so far as to say that seeing projects delivered satisfying all of these is a rarity and I think the reasons for this (broadly) fall into 1 or 2 categories:

• Insufficient planning
• Ineffective management

If you don’t plan your feature deliveries well enough, you risk not realising all the tasks needed to deliver your full scope; you risk not identifying your risks and dependencies fully; you risk setting unrealistic expectations to your stakeholders about your overall delivery. Poor planning by itself can risk all 3 of the triangle’s vertices. (I’m not even going to get into ‘right-to-left’ planning [planning your project to an immovable end date before all the work is understood] in an Agile environment here, that feels like an entirely different thought-dump.)

If your deliveries aren’t managed well enough, you’ll likely encounter poor stakeholder management/expectation setting; poor resource management; risk scope creep and waste; and risk losing sight of your overall goal(s).

I have sadly worked projects (as I’m sure many QAs have) that have fallen into both camps. I’ve had the displeasure of programme managers on projects that are contractually obliged to their scope and quality ask me to stop raising bugs; to delete the tests that are failing and write some that will pass purely to meet their contractual goals. As much as my reaction(s) may have gained me a reputation at the time for being a ‘straight talker’ with upper management, it also gave me a drive to ensure that quality is embedded at all stages of the project lifecycle and in a way fortified my desire for professional integrity.

How we Squared the Triangle

The title of this post is stolen from the idea of ‘squaring the circle’ which means attempting the impossible, and that’s because recently we’ve delivered a UK regulatory project on time, to budget and to the agreed scope. ‘Squaring the triangle’ if you will. Something I’ve rarely seen.

I’ve mentioned already how right-to-left planning isn’t a great idea in Agile and given what I’ve already laid out about the project management triangle that should make sense; if you need to flex your resource or scope then you also need to be able to flex your time or if not it’s at the cost of the overall quality. However our project had a stone-set date from the Payment Services Regulator so ‘time’ was an immovable vertex.

So how did we do it?

1. Understanding Requirements Early

Led by our Product Owner we worked to really understand all possible requirements from both the regulator and the business stakeholders for this feature.

It was important that everyone in the team understood what we were striving for; us as QAs, the developers, the business analyst (ok most of this was their work) and even our scrum master to a point. If you’re not all behind the goal it makes it harder to achieve.

2. Agreeing The MVP

Once we identified all possible requirements from all quarters we had to understand what was the Minimum Viable Product we could deliver to satisfy all parties, extraneous requirements be damned (for now). Knowing what we absolutely had to deliver to satisfy our stakeholders (and in that case it was regulator first, business second) means we can focus in on that and everything else comes after. Hopefully you have a Business Analyst and/or Product Owner who is comfortable applying the MoSCoW approach and if not, maybe read up on it to try and lead that conversation.

3. Refine, refine, refine

I think refinement is probably the single most important part of the delivery lifecycle. It’s the first opportunity to really understand the detail between each component part of the overall solution and what needs to be delivered.

A salient point and I took away from a QA meetup before was the idea that “QA” could stand for “Question Askers”; we bring value by exploring possibilities, extremities, and the unconsidered. That’s why I find story refinement invaluable – anyone can learn to ‘test’ but looking at things through a broader lens than the specific requirements being discussed and asking the questions no one else is thinking about is where we can truly bring a lot of benefit in QA.

Having everyone involved in the user story (whether they’re directly delivering it, will be consuming it, or even just outsiders looking in) really drives the important conversations about what’s being delivered. It’s also extremely important that if not enough is discovered during a refinement that you can say “we don’t know enough yet, we need to go away and think more about this” rather than just finish the meeting with open questions and make the story Ready anyway; This Way Lies awkward tech debt and failed deliveries. You can always come back to it later.

4. Dependency Management

Different teams work differently. If the piece of work you’re delivering will require external people or teams to assist (maybe it’s for a specialist ability like your DevOps engineers or IT Service Desk, third party technology partners or a subject matter expert) you can’t necessarily expect them to be available at your beck and call, these folk will have their own deliverables, their own processes for new work, their own iteration cadences, their own priorities. If there’s a chance that the feature or story you are delivering needs support externally the sooner you identify this, the better.

Uncertainties in your timeline are ok. Saying to them “we will need your expertise at some point soon just not yet” makes it easier for these tertiary teams to prepare and make sure the right people are ready than springing it on them and saying “we need you right now other priorities be damned”, it also helps build productive future relationships with them. Engage with them early and keep them updated. Better yet, ask them to join your refinements.

5. Protect The Work

This is a big one, it’s double-headed, and nuanced so interpret this section as fits your own business. But if you’re being distracted by other work or faced with scope creep (remember your MVP!) then you’ll struggle to meet your goals.

When we squared our particular triangle our QA colleagues in other teams and the delivery function at large relied on us a lot for various desk-side asks. Be that mentoring, maintaining our test automation estate, helping to deliver software releases or even helping out on other business priorities. It will rarely be on us as QAs to say if our particular piece of work is the priority so it’s important to make any other deskside favours ‘visible’. It may be that a scrum master, product owner or other stakeholder asks us to ignore the new ask in favour of the existing priority. After all there are only 24 hours in a day (and I like to sleep for at least 8 of them!). Despite trying to do the right thing and help our colleagues and friends, not making the extra asks visible risks our key deliverables.

The other side of protecting the work is something I touched on before, scope creep. Remembering the project management triangle if we lose sight of our MVP and accept bells and whistles that are only ‘nice to haves’ rather than ‘must haves’ then we risk being unable to deliver to quality without increasing the budget and/or available time. Even once you have a clear idea of your MVP during Feature discover and refinement, it’s not unusual to realise new requirements during delivery and it’s important to discuss these as a team to understand how they fit to the MVP and MoSCoW and whether it fundamentally changes or risks your overall delivery or delivery date because these too can risk the overall delivery target.

6. Automate First

According to traditional regression test thought, each time you iterate on a feature you should test the existing functions to ensure they haven’t been broken by the change. I could easily take 10-15 minutes (if not more) to execute a single manual test case. As the feature grows to completion and tests are added that time grows in an uncomfortably linear fashion. An automated test takes far less time to execute, if our tests take longer than 2 or 3 minutes then we’ve done something very wrong.

It is usually far more efficient to make the investment in automating these tests as early as possible even if that means they need amending as we iterate on our build; the time saving in repeated manual regression testing combined with the confidence in the overall feature quality brought by button-push regression testing easily demonstrates the benefit of automating our tests rather than manually testing.

For that reason we adopt an ‘automate first’ approach. Automate everything we can (and where possible using a TDD approach) so that we never need to worry about manually executing those tests again. Not every test will need repeating so it doesn’t make sense automating everything (e.g. edge cases) so we do still use manual tests but those are in the minority and where possible we would save time (and therefore Lead Time) by writing Exploratory Test Charters to cover these test scenarios rather than traditional long form test cases (I think I just found another future blog post subject).

7. Future Phases

I’ve mentioned a few times now the focus on the MVP. Delivering the MVP doesn’t mean the end of the work, it just means that value is being realised and you’ve delivered the most important parts first. There will 100% be requirements that aren’t considered to be MVP, there will be bugs your stakeholders have accepted into production, there will be enhancements you identified during build that won’t be a part of this initial MVP. For all the will in the world, unless you’re in a NASA clean-room environment there will eventually be escaped defects covering a use case no one realised existed until the users broke it. That’s OK! From the outset the team and your stakeholders should be aware and accepting of the fact that squaring the triangle is the goal and there could be a phase 2, 3 or 4 of the delivery. If not, then you should speak up that there will almost certainly be future tech debt on the backlog to consider.

Triangle Squared?

I won’t claim that this was all down to me as a QA, clearly everyone in the team has a part to play in ensuring a project is delivered well (I guess another future thought-dump will be around the concept of Quality Engineering) but I see asking the right questions as and when they come up and having the right discussions off the back of them as being essential to achieving project goals. I see our role as QAs not ‘just’ as testers but to help coach and encourage quality and direction at all stages of a project and if you’ve made it this far, hopefully this post has given you something you can take away and use yourself. Please feel free to let me know in the comments if you’ve taken any value with any of it or if you disagree with any of it.