Chapter

X

A Principled Approach to the

Evaluation of SV: a case-study

in Prolog

by

Paul Mulholland

Knowledge Media Institute

Open University

A large amount of Software Visualization (SV) technology has been developed This is particularly the case for the Prolog programming language whose execution model causes particular difficulties for the learner (e.g. Taylor, 1988; Fung et al; 1990; Schertz et al, 1990). As a result, a wide range of Prolog SVs (or tracers) exist and though many claims are made regarding their usefulness and suitability for various potential user populations there is little empirical evidence. This paper reports an empirical investigation into the suitability of four tracers for an early novice population.

The methodology aims to learn from the lessons of SV and Computer Based Training (CBT) evaluation which has provided many conflicting results that cannot be clearly interpreted. The empirical approach uses protocol analysis (Ericsson and Simon, 1984) to develop a fine-grained account of the user, identifying information access, the use of strategies, and misunderstandings of the SV and execution. This approach allows differences in performance to be more confidently explained.

The results show overall performance differences across subjects using the SVs which can be interpreted using the protocols. The interpretation permits

Introduction