I suggest that all projects should have a documented test strategy.

Here are five good [...]]]>

I am sometimes asked, “Do we really need a test strategy?” and often such a question is followed immediately with a statement such as, “We have a test plan.” “No one will read it” or “We won’t follow it anyway”.

I suggest that all projects should have a documented test strategy.

Here are five good reasons why I feel that having an appropriate test strategy, documented at the right level makes sense and represents good practice.

Please feel free to challenge them or supply other good reasons….

1: Documenting a test strategy gives you time to think about what exactly the testing has to achieve, what and who is required to achieve it, how long it might take and what stages you will have to go through. Particularly if intelligently following a checklist of items to consider it helps manage and formalise your thinking so that all elements are covered.

2: Circulating a documented test strategy allows others to review, to cross reference sections and validate the content. This is particularly useful where you do not have the level of knowledge you would like.

3: Documenting a strategy assists with the overall project planning, providing the project manager with a reference document that provides a comprehensive list, explanation and justification for the various test phases and requirements, and can provide evidence to support the testing budget you are asking for.

4: Having a documented test strategy provides a point of reference both for yourself and others as the project progresses, allowing you to be reminded of the overall goals that have been agreed as you adapt the strategic response to each new challenge.

5:: Having a documented strategy allows all stakeholders to have a say in the testing process and allows the final approach to be signed up to by all relevant parties.

Six hat thinking applied to testing

Or maybe you have actually worked on a project were you tell the powers to be that the software should not be released, but lo and behold, they go and do it [...]]]>

Have you ever come across a piece of software that just  does not work and wonder, ‘How the heck did this get released?’

Or maybe you have actually worked on a project were you tell the powers to be that the software should not be released, but lo and behold, they go and do it anyway. I had cause to reflect on this question just recently …….. why does bad code get released?

There are of course a myriad of reasons and circumstances that lead to poor code being released, however once cause can be the company culture.

If developers are highly valued, and testers are seen of less value, then it is easy to see why the voice of the tester can go unheard and their view ignored or trodden over.

If testers are no tsufficiently technical or unable to write a coherent and convincing report, their message can get shot down in the general debate, or considered of little value due to the poor presentation.

If the release decision is taken place behind closed doors with no opportunity for the test report to be presented and discussed then the information regarding the true
state of the software might never be made available to those making the decisions.

Of course, there may be a number of reasons why, even though the test report is fully understood, the release is still sanctioned and bad code is released.  However these tend to be few and far between
in my experience, the main reason is that the test department of one reason or another is excluded from the process.

Who do I blame?   The test department. We need to do more to ensure we are respected, valued and listened to.

The chart below takes a slightly tongue in cheek look at an all too familiar release decision tree.

This blog lists some of my favourite spots, old and new.

New on the block is TechWell (http://test.techwell.com), from the guys at Stickyminds (which is also a favourite of mine.) This stuff [...]]]>

.Since I started to blog, and more recently tweet I have started to discover some really great places to visit.

This blog lists some of my favourite spots, old and new.

New on the block is TechWell (http://test.techwell.com), from the guys at Stickyminds (which is also a favourite of mine.) This stuff is high quality and covers a wealth of topics. I really love the look of the site as well.

The Software Testing Club (www.SoftwareTestingClub.com) is a site that has grown at an impressive rate over the past couple of years and Rosie (the founder) should be very proud of what she has achieved. It has a relaxed home grown feel about it. Some of the content can be a bit benign, but it is a great place to go for a quick answer from your peers if you get stuck.

Utest has a nice blog page (http://blog.utest.com) and manages to provide a very lively flow of content including interviews, articles information and posts. It has a nice clean layout that makes browsing easy on the eye.

The only blog that has caused me to write ‘fan mail’ is the fantastically honest QAHatesYou (http://qahatesyou.com/wordpress). As you can imagine from the name, he pulls very few punches. Add to that he has a certain warped mind that means that some of his post are incredible funny …….check out his songs for testers videos!

A nice site from the quality frog Ben Simo, is dedicated to software failures,, and should provide material to liven up a slide or two if you need to explain why testing is important, check it out. (http://blog.isthereaproblemhere.com)

I could go on, but that’s enough of that for now I think. I have set up a Software Bloggers Group on LinkedIN (http://www.linkedin.com/groups/Software-Testing-Bloggers-3945121?trk=myg_ugrp_ovr). I am hoping a bunch of bloggers might join and use it to spark off each other. I would like to issue a blogging challenge from time to time via the group. Check it out and let me know what you think.

Also why not post you own favourite testing sites via the comment box.

Let me justify that stamen by starting with a little quiz.

I will give you the question, and you mark the options below as ‘pass’ or ‘Fail’ as if this were a software test,

1: What was King George VI’s first [...]]]>

Testing is more about knowing the answer than it is about knowing the question!

Let me justify that stamen by starting with a little quiz.

I will give you the question, and you mark the options below as ‘pass’ or ‘Fail’ as if this were a software test,

1: What was King George VI’s first name?

A: William
B: George
C: Albert
D: Charles

2: How long did the ‘100 year war’ last?

A: 93 Years
B: 100 years
C: 101 years
D: 116 years

3: What is the acceptable load time for a standard web page

A: 2 seconds or less
B: Under 4 seconds
C: 10 seconds if containing rich media
D: 20 seconds

Of course if you know the answers, the decision to mark any result as a pass or a fail is easy, however if you just have the questions, you may not be so sure. The question may seem obvious to you, but how do you actually know what the correct answer is to generate a ‘pass’

The correct ‘pass’ criteria for question 1 is answer ‘C’ and for 2, it is ‘D’, however for question 3, there is no universally accepted ‘correct’ answer.

So you see, knowing the answer is much more important than knowing the question when it comes to testing. Requirements can sometimes hint at the answer. Lets work though an example;

‘Website links must be obvious and accessible’

Ok so my test would be,

‘Is this link obvious?’

But what is the answer?

This link has an’ under line ….. Pass
This link does not…. Fail…. Except that the link itself is blue and all the other text is black … so Pass … or fail .. er no pass!

You see the dilemma, unless it has been agreed what ‘obvious’ for each class of link will mean, then the result is open to interpretation, just like question 3 above.

The problem is further compounded by the fact that there will exceptions from time to time, so…

‘All textual links will be underlined and coloured blue’ is great, until you decided that actually, the small footer links (privacy policy etc.) will be underlined and grey. Again, knowing the answer for each link is more important than knowing the question (Is this link obvious?).

Of course in any real project you seldom get all the answers upfront, and so asking the question is an important way of getting the answer and I am certainly not saying you should wait till everything is nailed down before you write your tests. However I am suggesting that as testers we should not get too wedded to our tests and that if as a result of a test, an answer gets changed, then we should keep in mind that the answer is the important thing that drives us.

Get the right answer and you can ask the right question.

Tony Simms is Principal Consultant at Roque Consulting (www.roque.co.uk). He can be contacted via tony.simms@roque.co.uk.

The current Mrs Simms and I have now reached the age where the highlight of a romantic night in will be two or three rounds of Boggle. You probably know the game, where you have a 3 x 3 grid of random letters and you have to try to make as many words [...]]]>

Ever Played Boggle?

The current Mrs Simms and I have now reached the age where the highlight of a romantic night in will be two or three rounds of Boggle. You probably know the game, where you have a 3 x 3 grid of random letters and you have to try to make as many words as possible in a given time. It always amazes me that we can come to the last few seconds and neither of us can find any more words, yet when we add up the scores, she has five or six words I did not see and I have seven or eight she did not see.  So what does that teach us about testing?

Two heads are better than one, three are better than two.

I have just received back review comments on my DR Test Plan from four or five different people. Many make the same comment about this or that, but they all have additional comments none of the others have. They not only see the same things differently, they also have different agendas, responsibilities and experiences that influence that review process.

The same is true of test execution, having just had a report back from a session of crowd testing its clear that things have been spotted and logged as bugs that none of the previous reviews and test cycles picked out. Again it has to do with how our brains are all wired differently and how that influences what we see.

Results from the testing of common pieces of code is the combination of scientiffic experience of years suffixed by years of personal growth and development.

If you ask people to count the number of times the letter ‘f’ appears in the sentence above, it is interesting to see how many different answers you get. (by the way with the double ‘f’ in scientific I am pretty sure there are 9 or is it 10?) 

 Different is Good, but… Different can be Helped

Having agreed that different people see things differently and that brings value to the review and test process, that is not to say that you can’t teach some uniformity in to what to look for.

Below is a set of guidelines for reviewing documents, taken from the excellent book

‘Software testing In the real World’  by Edward Kit.

The check list below is adapted from the Boeing Computer Services Requirements Checklist

1. Complete.          All items needed to specify the solutions to the problem have been included.
2. Correct.              Each item is free from error.

1. Precise,               unambiguous , and clear. Each item is exact and not vague; there is a single interpretation; the meaning of each item is understood; the specification is easy to read.

4. Consistent.       No item conflicts with another item in the specification.
5. Relevant.           Each item is pertinent to the problem and its solution.
6. Testable.            During program development and acceptance testing, it will be possible to determine whether the item has been satisfied.
7. Traceable.        Each item can be traced to its origin in the problem environment.
8. Feasible.            Each item can be implemented with the available techniques, tools, resources, and personnel, and within the specified cost and schedule constraints.
9. Free of unwarranted design detail.          The requirement specifications are a statement of the requirements that must be satisfied by the problem solution, and they are not obscured by proposed solutions to the problem.
10. Manageable.  The requirements specification can be controlled in such a way that each item can be changed without excessive impact on other items. Changes to the completed requirements specification can be controlled; each proposed change can be traced to an existing requirement; and the impact of the proposed change can be accessed.

Some methods of transforming specifications to reveal ambiguities, errors and/or misunderstandings:

1. Read the sentence several times, varying the stress pattern to reveal possible ambiguities.
2. When a term is defined explicitly somewhere, try substituting that definition in place of the term to see if the definition fits in this context
3. When a structure is described in words, try to sketch a picture of the structure being described.  This is especially useful where hierarchical structures are being described.
4. When a picture describes a structure, see if redrawing the picture reveals different possible meanings
5. When there is an equation, try expressing the meaning of the equation in words.
6. When a calculation is specified or implied in words, try expressing it in an equation.
7. When a calculation is specified, work at least two examples by hand and give them as examples in the specifications.
8. Look for statements that in any way imply certainty and then look for proof.  Words such as ‘always’, ‘every’, ‘all’, ‘none’ and ‘never’ are useful clues to indicate unproved certainty. 
1. When you are searching behind certainty statements, PUSH THE SEARCH BACK as many levels as are needed to achieve the kind of certainty a computer will need. 
2. Be on the lookout for words that are supposed to be persuasive, such as CERTAINLY, THEREFORE, CLEARLY, OBVIOUSLY, or ‘AS ANY FOOL CAN PLAINLY SEE’.  If its not obvious to you then there is a good chance that others will interpret it differently from the author’s intended meaning.
3. Watch for vague words, such as SOME, SOMETIMES, OFTEN, USUALLY, ORDINARILY, CUSTOMARILY, MOST, or MOSTLY.  These must be backed by  conditions that you can test (and the developers can code to)
4. When lists are given, but not completed, make sure that there is a complete understanding of the nature of the subsequent items.  Watch out for ETC., AND SO FORTH, AND SO ON, or SUCH AS.
5. In attempting to clarify lists, as in (12), we sometimes state a rule.  Be sure that the rule doesn’t contain unstated assumptions.
6. Look for lists without examples or examples that are too few or too similar to each other to explicate the rule.
7. Beware of vague verbs such as HANDLED, PROCESSED, REJECTED, SKIPPED, or ELIMINATED.
8. PASSIVE VOICE constructions are also traps.  Since the passive voice does not name an actor, it’s easy to overlook who is doing the work.  An example is ‘The log file is deleted later’.  Who or what deletes the log file?
9. Be especially on the lookout for comparatives without referents.
10. Pronouns are often clear to the writer and not to the reader.

(By the way, I mentioned some time ago that QA&Test [http://www.qatest.org/en/] is a great conference for first time speakers and they have extended their call for papers, till April 15th, so still time [...]]]>

I have just sent off my proposal for a talk for QA & Test this October.

(By the way, I mentioned some time ago that QA&Test [http://www.qatest.org/en/] is a great conference for first time speakers and they have extended their call for papers, till April 15th, so still time to get an application in).

I hope to be speaking on the need to manage third party suppliers.

Running Deep and Silent

Left to their own devices, many suppliers have the tendency to go into what I call “RED OCTOBER” mode.

 In the movie “The Hunt for Red October”, the Soviet submarine commander, Ramius, played by Sean Connery, is hunted by both the Soviet and US navy. Everyone knows the sub is out there, they know where it is heading, but until it surfaces no one knows where it is, how far it has travelled, which course it is following and quite what to expect when it surfaces. When something goes wrong with the engine there is a loud noise, a dramatic explosion and all the ‘big wigs’ get very excited but then it all goes quiet again.

 Here’s a secret: most supplier Project Managers secretly think they are Sean Connery!

They don’t want you to see what’s going on. They hope that they can start the project, work away in isolation and deliver the solution at the end, with as little ‘interference’ as possible from the customer. The attitude appears to be: “Problems? Yes, of course there are problems, but as long as we sort them before the customer finds out, what’s the issue?” The issue, of course, is that if the problem is not sorted before the delivery date, then the solution is late, or defective, or over budget. Either way, the client has little time to take mitigating action and is often then held to ransom.

If we are to manage the overall quality and delivery of our project we need to know the status of the project at each stage of its journey. We can’t wait for the big disaster to hit and then look surprised and upset. We have to be able to measure progress at appropriate stages, report back to our managers and decide on appropriate action.

Test managers have a role to play.

As a Test Manager, I believe you can play an important role in ensuring the delivery from your supplier. Below are my Top 3 Tips for making a difference;

1:   You need to establish your role in managing third parties early on.

If joining at the beginning of the project, ensure you have input into the contract and write your own Test Strategy to make your role clear and documented. If joining late, remember that the third party has no idea who you are. You are free to present yourself as you wish so make sure that early on, in the first week or two you take time to set the ground rules with your third party Test Manager. Make it clear that it does not matter what has gone before, you are here now, this is the way you do things, and this is the way you want him to do things.

 2:    Get it in the Contract

Suppliers are only obliged to do what they are contracted to do and often they want to do as little as possible, for as much as possible and want to get paid, as quickly as possible. This is what drives their activity. If the test report is a contractual delivery and payment depends on it being signed off then by and large, you can expect to get a test report.

3:    Insist on Witnessing

There can be no substitute for actually sitting alongside the supplier’s testers and witnessing what is going on. Is the environment set up correctly? Are scripts being followed or are tests being ticked off without thought? Are issues being found but not reported, or worse still, fixed on the fly? These are all questions it is very difficult to answer by reading reports. Witnessing allows you not only to ask the question, but see, real time, the answer.

Tony Simms is the principal consultant at Roque Consulting ( www.roque.co.uk ) and can be contacted via email; tony.simms@roque.co.uk.