Anything written in code must be QA (Quality Assurance) tested. It is a Fact Of Life. If there are possible errors or behavior that could be improved (and assuredly, these always is), they must be discovered and correctly. However, the only way to find these is through QA testing. Yes, there are also unit tests, a programmatic way of checking if certain parts of some code are working, but they primarily help guard against regressions, not ideal behavior. In this area, good old-fashioned humans excel over automated tests.
However, what are the rules of application QA testing? Surely there are some guidelines that must be followed. While the exact rules will vary depending on your application type and context, but there are some rules that are valid no matter what.
Here now is my personal set of QA Testing rules, which I have titled The Rules of Application QA Testing (I know, such a creative title 😛 ).
- Give Details! Details are a required part of QA testing. This should go without saying, yet many people fail to provide what is needed for the developer(s) to thoroughly investigate and sometimes even reproduce the issue so a proper fix may be written. This requirement is so important it is my first rule. I cannot stress the importance of details enough.
Now when I say details, unless otherwise stated I want the following information:
- Operating system, version, and bit (e.g. Windows 7 x64 or Mac OS X 10.8)
- Application version
- A log file, if present
- The exact error that occurred, full message and all
- The exact steps you took before encountering this
- If you have successfully and purposefully reproduced the issue
- If you have tried it on a different computer, preferably with a different OS but also with the same OS
- If it is a web app, the browser and browser version you used and
- If it reproduces or works in other browsers
By providing these things, you go a long way of helping track down the root cause of the issue and for me applying a good and proper fix.
- I, the developer, might not have ever heard of the issue before.
This point goes right along with giving details. Come to think of it, every single rule is based on giving details. 😛
I have learned a Fact of Life in my short time of programming: I will overlook something. It is an absolute truth. I do not know how many times my beta testers have reported bugs, missing features or usability issues I have completely overlooked because I am the developer. Therefore, if you find any issue at all, do not assume I already know about it. Report it in all its glorious detail as if I had never heard of the bug. Chances are that may very well be the case.
- Do (reasonably) everything you can do to break it. Commonly shortened to simply “Break it” to those who know what I mean, this is my strongest QA test. It gives testers free rein to do absolutely whatever they want in an attempt to break it. “Reasonably” refers to not doing absurd actions that would obviously make it break, like ripping away critical application files. This would be likened to a brute force attack to test a network’s security.
- If file manipulation is involved, use the file to break it. Make the file read-only, try to save in a restricted area, mess up the file layout, remove critical information from the file, and rename the file. Use it or multiple files to bring down the program, again, all in reason.
- Run the application in a restricted area. While some may see this as a convenience action by developers and not totally required by the application (wave hello abused iOS and Android permissions systems), there are legitimate reasons for an application to require elevated privileges in order to operate. Drop the program in a restricted area and see how it functions. If it is a web app, run it behind a firewall or proxy and observe the behavior.
- What was your expected experience? Did you expect a dialog box to come up when a certain thing happened? Was a useless message shown? Are some directions not clear enough or too dumb? Did it do something opposite to what you desired or expected? Is something too small to see on your 1080p monitor? Is something you want to do missing? Clearly state what happened, what is wrong and what you expected.
I know these six rules might seem like a lot to remember, and truthfully it kinda is. The fact of the matter is you will not remember all of these in the beginning. Heh, even I forget my own rules at times! However, the more you use them the more you will remember them, just like developing a new habit.
Now that you know the rules of Application QA Testing, you must wear them around your neck and hide them in your brain, that you may never
sin report an issue in an incorrect manner every again. 😛