Testing for what you can't see

So - we delivered a product by the end of last sprint. October 1. was the deadline and we made it fine. Or to be precise: The code that we delivered worked fine. Nothing crashed, for instance. Well - was the release succesful? Not exactly...

It turned out that when we put code up to our staging environment we didn't have much of a plan for saying: "What does it mean to put something onto our staging environment?" We have always focused on being able to automate our releases so a release to our staging server never takes more than a few minutes and can be performed by anyone who has the password to our buildserver. In other words: Our Continuous Integration setup rocks - but what is it all worth if nontechnical errors are not caught until somebody is actually starting to USE our code? When is a feature Done - how do we test and assert that Done is really Done?

I will for the rest of my life never let nontechnical requirements to a feature be part of what IT can be blamed for if something gets released and doesn't work the way it was intended. That's just not what we are hired to do as software engineers - we build software. We are not supposed to have the deep insight in a business domain because we are hired to express the business needs towards their domain in programs and websites which provide the business with solutions to their problems. However - that's a dreamworld scenario because in a lot of companies IT has become the knowledge base of your business domain because of the insight you get into a domain when it is mapped to code. You have become bloated with too much information. How many of you have tried this - hands up everyone. I have both mine up right now ;o)

What usually happens is that the person implementing software also begins to take excessive responsibility for the business domain and while being responsible is good, being overly responsible is bad because people tend to think that "That's not our problem - that's something for the IT guys. They know what to do". When the business starts to react like this is the sound of an emergencyalarm going off.

I have a feeling that this is the usual case in most small companies and in quite a few large ones as well. IT knows what to do and how things are supposed to work. When this happens IT becomes responsible for the business domain and suddenly you have the business coming to the developers to ask for insight into their own domain.

I have a feeling that when we in collaboration with the business were unable to catch that a number of non-functional requirement caught us off-guard a few days before October 1. it is due to a number of reasons

• IT departement is supervising the business domain - but developers can't test their own code. Developers are in my opinion the lousiest testers the world will ever see - and that goes for both technical and nontechnical requirements.
• The IT guys takes the majority of responsibility because they know the business domain best - there can be a number of reasons for this (business culture for instance) but the business should never be excused for not being Master Jedi in their own domain and for not knowing their own domain at least as good as the developers.
  
• We tested what we could see - nobody tested for the things that wasn't visible.
  

What are your experiences in this matter? How do you lay off responsibility to the business domain experts and convince them that they are truly experts in their field and should take that responsibility on them? What are your Def. of Done criterias for completing a story? Lots of question asked, I know - drop a comment if you have something to share with me :o)

Regards K.

Define user satisfaction

Quote from Software facts and Fallacies by Robert L. Glass - page 133:

"A user will be satisfied if he gets a product that meets needs, is available at an appropriate time, doesn't cost a fortune and is of reasonable quality".

We had a couple of real world users testing the website that we have been building for the past 3-4 months. I noticed a few things walking around yesterday evening while they were testing:

• Nothing crashed. I never had users come in to test any of my code on an entirely new website without anything crashing with an ugly ASP.NET stacktrace showing up.
• Nobody gave up and needed a lot of guidance help to complete their testing. There were a lot of questions of course and quite a few logical errors but people seemed to have a basic understanding of what they were supposed to do to keep moving forward.
• 4 out of 5 questions could be solved with a textual changes / graphics instead of written text or layouting the page a little differently. We could really use a dedicated tester which knows something about design and usability in our team.
• Nothing is as painful and as to see users revealing stupid logical errors. In Denmark we usually write the zipcode and the city on the same line: "7100 Vejle". We had an inputfield, some 30 characters allowed but we had clientside validation which threw an error if you entered any non-numeric value or more than 4 characters (all zipcodes in this country consists of max. 4 numeric characters. Small country, you know...). During our own day-to-day testing during development we had always just entered the 4 numeric characters which would pass validation check - and we would gladly have put this in production if users hadn't pointed out our own stupidity to us in advance.

As the wise men say: "You learn the most by your biggest mistakes". Point taken - we had a few things to correct. And - given the quote from above: We actually delivered something the users testing our system felt was useful - we tested on the night scheduled a month ago - we haven't spent more than a few months developing and it provided a solution to our Product Owners problem. How about that? :o)

Regards K.