Testing software quality – open your mind

I recently attended a two day course called “Softwaretest – when it’s best” in Copenhagen. The reason I found it worth going to was that we recently had a major breakdown in our production environment because we pushed a bugfix into production without having properly testing what we were actually releasing. Thus having a new refactored decoupling of of our AspNetEmail component released without it being properly tested… You can imagine what people think of the guys in the IT departement when they either don’t get their mails or get them multiple times at random… Luckily we didn’t send anybody an email that they shouldn’t have recieved but we sustained major damage to our reputation within the organization because it’s not the first time we committed crimes like these before. So – we definately need improvement in our testing phase. We’re a team of developers – 5, to be exact. We know our TDD-drill and Continuous Integration is a first class citizen in our office so testing our own software isn’t totally unknown to us. I just feel we need to fill in the gaps to get to the next level - ideally without having to build and maintain an extensive and bureaucratic testing process nobody buys into 110 percent.

So I went there with a collegue to learn about this testing stuff - and I feel I am a little wiser now. Here's the eye-openers I have decided to share with you:

• Developers live and breathe to make things work. Testers live and breathe to make things break. It’s a state of mind you need to be aware of as a developer. You can’t sit down and develop – and then test your doings without having said to yourself “I’m not a developer right now – I’m now a tester and want to break stuff”. I tried it today and WHAM!!! I found 3 critical bugs in a piece of software I recently wrote on one of my hobby projects. Each of them could have been found if I sat down and reviewed the code - but I found them nevertheless by simply WANTING to find errors.
• Evolution is better than revolution. Nobody likes to be hazzled into something if they don't believe it provides value. Don't start out with preaching about "the god-awesome Testing Maturity Model (TMM)" and your visions about reaching level 5 before the end of the year. Get everybody involved before making any decisions - something about people and interactions over processes and tools... You know the drill for sure but it's easy to forget.
• If you don't understand the product you won't find the error! How often have you reviewed code and signed off on it just to find out a month later that it had bugs in it - functional as well as nonfunctional? I have more times than I actually would like to think of... If you don't know what the code is supposed to do - you can only verify what it actually does. You'll never get to the point where you realize what's missing in the code if you don't know what the customer expects from it.
• Test the deliverables not the specifications! If you have a clear set of specifications and a testplan for those you should of course use that testplan. But - your focus must always be on what is being delivered. You should as a tester try to gain as much understanding of the domain you're testing as possible - because who knows if the testplan (if one exists) is adequate? If you during development find out that performance is critical for a specific part of the software it is your responsibility to take this into account when testing even if performance is not an issue in the specifications. If the customer percieves the software as buggy it IS buggy regardless of whatever tests you've completed and signed during development and accepttesting.
• ...plus a number of other related goodies such as "How to write a test plan", equivalence numbers, conflict handling between testers and developers, the difference between Blackbox and Whitebox testing etc.

The very vital issue while testing is that testers have a different mindset. Their best days at work is when they find themselves "in the zone" registering bugs like maniacs while developers think success and failure in terms of solving problems and implementing features. If you’re a developer you will never become much better than poor (at best) in a testing phase if you don't acknowledge that you need to change some things in your head in order to improve your testing capabilities. That's probably the biggest eyeopener I've had since I wrote my first unittests and after an hour or two of "Why am I writing so much code for nothing" suddenly got a failing test totally unexpected...

Tomorrow, before you contact your Product Owner because you have released something to your staging environment - say to yourself: “The next hour I'm a success ONLY if I can find bugs in what I just released”. Sit down, seek - and you will find. Promise :o)