The myth of the intermittent bug

It’s a term we will have all heard in the Quality Assurance world. “It’s an ‘intermittent bug’.”

intermittent: occurring at irregular intervals; not continuous or steady.

There is no such thing as an ‘intermittent’ bug. To me the word intermittent means something is irregular and inconsistent with no discernible pattern to it’s occurrence.

We will all have seen bugs that do not present themselves every time we execute a test case. They are perceivably intermittent but that doesn’t mean that they cannot be predictable.

We don’t live under a constant full moon at night, a full moon arises every 29(ish) days but you never hear anyone call the occurrence of a full moon ‘intermittent’. It’s predictable. It’s regular.

If my car starts regularly unless it’s a particularly cold morning, I don’t tell the mechanic that it’s an intermittent problem; I tell them the car won’t start when it’s a cold morning. Again, a predictable occurrence given we know the state that causes the car not to start.

Computing software is governed by 1s and 0s. States are boolean; true or false. There is no true random external influence, it isn’t possible for something to happen ‘intermittently’. If we are tempted to call an unexpected outcome as intermittent it just means we don’t know enough about the system’s state and conditions that allowed the bug to manifest.

We had an instance recently in our test pipelines. Where possible we create test data to consume by firing data into our environments via the same APIs that we use in production for creating new customers or accounts to create a repeatable starting point for each test using random yet realistic-seeming data (using a faker.Net library to make the data look realistic and human readable).

One of the developers flagged that there was an ‘intermittent bug’ in our customer creation process – they had seen that sometimes the API request that the framework used to create a customer would return a 400 Bad Request response. They had found a solution though, if they put a retry mechanism onto the customer create request that retried with fresh faker data, it always passed within 3 retries and removed the flakiness from the test.

The problem was that we hadn’t identified or address why the issue occurred, we merely plastered over the crack. While their solution seemed to solve the perceivably intermittent problem it had only masked it and the laws of probability meant that it could appear again.

We simply didn’t know enough about what was causing certain requests to return a BadRequest response and tried to mitigate it with spam-retries.

When we dug into the problem we found an issue in the way our framework was using the faker data to create the JSON payload. If there was an apostrophe in the name (e.g. O’Donnell) the apostrophe was being parsed to a quotation mark which miss-formed the JSON and triggered the BadRequest response. Armed with this information we addressed the way we formed the payload using the library data to solve it in a way that didn’t need a retry mechanism, it just worked (and all credit to the developer in question for taking my breadcrumbs of information and my best guesses at a time when I couldn’t give it my full attention.)

It’s just one example but one that I feel illustrates the point well – there was a bug that was perceived to be intermittent, but once we looked a bit deeper we understood the cause and it wasn’t intermittent, it was just a bug that we didn’t always allow to materialise. It’s a pattern I’ve seen many times in my life in Quality and led me to realise that if someone (including myself) refers to an issue as being ‘intermittent’ it means it’s time to look at the issue and behaviour a bit more closely.

There is no such thing as an ‘intermittent bug’. If we perceive an issue to be intermittent it just tells us that we don’t know enough about the states and conditions that cause it to be able to consistently and reliably reproduce it.