The purpose of trialling is to determine whether the program meets its planned aims and objectives with its target audience and in its target environment. User feedback is important at all stages of the planning and development process but, until the product is at a prototype stage, it is always possible to treat the results of trialling with suspicion. User trials at the end of the production phase and just before final mastering and distribution is called beta testing.
Frequently, if the same people designing the materials are involved in trialling, there is a tendency to hang on to preconceived notions. Nor is this entirely unreasonable. Testing an unfinished product is fraught with complications that undermine the reliability of any results. Of course, any feedback is better than none but results should always be interpreted judiciously. It is best if trialling procedures are devised and supervised by those outside the actual origination of the design, who do not have a sense of ownership on the ideas and concepts embodied in the program under test. Often this is not possible, for reasons of expense or confidentiality.
No opportunity for trialling should be ignored. Everytime someone outside the team looks at the product, this is a form of informal trialling and feedback should be noted. Note that demonstrations of the software are a very weak form of informal trialling, since the audience does not experience the software in an interactive sense and their reactions are often affected by their relationship with the demonstrator. Nevertheless, information gleaned from the process should not be ignored and may prove valuable. With a robust version of the program, the outsider may be asked to use the package normally, controlling their interactions. Again, comments should be recorded. Attention should be paid to what the users say in these instances and to their body language. Because such tests are informal, specific, formal questions will not usually be put to the user.
In the case of formal testing, a representative sample of users will be invited to trial the program. The sample group should be selected with regard to the end-user model devised in the planning stages of the project. This will take into account their age, sex and computer literacy, among other things. Trials should include environment testing, being conducted in as near to planned end-user conditions as possible. Users may be invited, singly or in groups of a defined size, to explore the package and then comment on it. Comments may be verbal and recorded, or written in response to either written or verbal questions from the person conducting the test. Users' general body language, as well as eye-movement and other body-part movements can provide useful feedback, although this data can be time-consuming and expensive to gather and interpret.
For trialling to be effective, it should be treated as a self-contained project. It must be well planned, carefully documented and well supported.
Planning and Documentation This involves putting together the trialling team, drawing up procedures, documenting those procedures, agreeing schedules, fixing a schedule and affirming the availability of users, setting writing and producing user questionnaires, and agreeing how the results will be analysed and how the results will communicated to the design team as feedback for program revision.
Support Technical support should ensure that software has been correctly installed, that software and hardware has been tested just before the trials and that hardware continues to function during the trial. If the trials are to recorded, then the recording hardware should be tested beforehand and sufficient recording media, such as video or audio cassettes, made available. Those involved in the trialling should be familiar with the workings of all the hardware and software to be used. Support should be immediately contactable and available at all times during trials and should be able to provide backup systems in case of total system failure.
Trialling procedures should always leave room for the end-user to add their own comments, thoughts and feelings, and should not push the end-user into answering questions they regard as irrelevant to their experiences. There must also be the willingness to take and treat the results dispassionately.