Advertisements
My essay "What is Exploratory Testing?" provoked some interesting reactions. One came from Yogita Sahoo, who agreed that ET can be useful, then asked "But can we get others to agree on such an unplanned process?" Ms. Sahoo's question raises some very important issues, such as:
* What is "planning"?
* Is exploratory testing necessarily unplanned?
* Why might we need to do things that aren't rigorously planned?
* How do we coordinate and support each other, if not with a plan?
Here's my short answer: Exploratory testing, as I practice it, usually proceeds according to a conscious plan. But not a rigorous plan. Rigor requires certainty and implies completeness, but I perform exploratory testing precisely because there's so much I don't know about the product and I know my testing can never be fully complete. Exploratory testers coordinate with each other and gain support in the same way executives run large companies: by talking with each other and building credibility over time. I want to say it's not rocket science, except that rocket scientists and all scientists use exploratory tactics in their work.
Exploratory testing may well be planned...
The American Heritage Dictionary offers us four definitions of plan:
* A scheme, program, or method worked out beforehand for the accomplishment of an objective: a plan of attack.
* A proposed or tentative project or course of action: had no plans for the evening.
* A systematic arrangement of important parts; an outline or a sketch: the plan of a story.
* A drawing or diagram made to scale showing the structure or arrangement of something.
Notice that a plan can be a plan without necessarily being a specific, rigorous, complete, or recorded plan. Thus, one answer to the Ms. Sahoo's question might be that good exploratory testing is a planned activity, even though it's not scripted in detail. I find it useful to define a test plan as "the set of ideas that guide the test process". This definition is consistent with at least the first two American Heritage efinitions. In order to guide testing, the test plan must exist prior to the testing, but how long prior? What if it comes into existence only moments before the testing? No problem. Nothing in the nature and notion of planning prohibits simultaneous planning and execution. In football, when a quarterback finds his receivers are too well covered, he may quickly make a new plan to run with the ball himself. An instant later, he executes his plan. Thinking fast is no crime against plankind.
...But, learning comes before the plan...
Still, I may be missing the point. Another meaning of the planning question could be "How can we get others to agree on a process with such an open-ended and unpredictable plan?" To address that issue it helps to imagine a very rigorous, repeatable activity, perhaps playing a flawless piano concerto, or executing a computer program. Now ask yourself, is there a plan in those activities? I would have to say yes. Now ask yourself, what happened before there was a plan? I would have to say some kind of composing and practicing process. If so, then was that prior process guided by the same rigorous sort of plan? I suspect not. In fact, if we keep peeling back the layers of how things came to be, we soon get to processes that obviously were not rigorously planned (at least not by humans): our education, society, childhood, life, the creation of the universe, etc. Anything explicitly planned must have arisen, at some point in history, from a less explicitly planned activity. Exploratory testing is exactly the kind of thing that comes before rigor and makes rigor possible. ET is primarily creative. It has value in terms of the problems we find while doing it, but it also has value because it is a way to create other kinds of tests. It's a great way to learn about a product. And if I did want to make a rigorous test plan, I'd better start exploring the thing I intend to rigorously test. That gives us the answer to the planning question: an open-ended plan may be the only plan available. If you already know for sure what the next test is that would be worth executing, by all means follow that plan. Otherwise, you'll have to figure that out for yourself: and that's an exploratory process.
You get someone to agree to such a process the same way you get someone to agree to allow a house to be built before they try to live in it. If they understand the difference between a house and an empty lot, and they want a house, and they see that there is no house, they may well agree that someone ought to build one.
...and dealing with a complex world requires skill, above all.
Still another way to think about Ms. Sahoo's question is that it's a query about coordinating people on a project and gaining support from management to do your work. How can a relatively lightly planned activity be organized and managed? This is where the myth of planning truly rears its head. The truth is, we organize and manage all kinds of things, all over the place, all through our lives, without detailed plans. In the absence of pre-scripted moves, what do we do? We pay attention. We react to the situation at hand. We anticipate problems. Because we don't know the right answers in advance, we find ways to come up with the answers on the fly. Think about this the next time you're merging onto a busy freeway.
In "The Effective Executive," Peter Drucker defines an executive as someone who manages the value of his own time. Testers are like executives. We can train testers in a manner similar to training executives: start small, provide coaching, learn from failure and build on successes. I don't send a tester on a complex exploratory mission until that tester has gained credibility on the easier missions. The tester must show an ability to design a good test strategy, execute good tests, find important problems, and report them in a compelling way. Furthermore, the tester must be able to explain his work. Only under those conditions do I come to trust that tester's work.
Remember it's the skill of the tester, the social system of the project team, and the testing situation at any given moment that enables excellent exploratory testing. Don't dwell too much on planning. If you're challenged to defend exploratory testing on the basis that it's not planned, reply that when the developers tell you where all the bugs are, precisely and accurately, you will happily write a precise and accurate script that visits each one. Until then you must spend some of your time in an unscripted search.
