
Copyright ? 1991, L. Hamey.
Appears in Advances in Neural Information Processing Systems 4
Morgan Kaufmann Publishers, 1992, pages 11671174.
Benchmarking FeedForward Neural Networks:
Models and Measures
Leonard G. C. Hamey
Computing Discipline
Macquarie University
NSW 2109
AUSTRALIA
Abstract
Existing metrics for the learning performance of feedforward neural networks do not provide a satisfactory basis for comparison because the choice of the training epoch limit can determine the results of the comparison. I propose new metrics which have the desirable property of being independent of the training epoch limit. The efficiency measures the yield of correct networks in proportion to the training effort expended. The optimalepoch limit provides the greatest efficiency. The learning performance is modelled statistically, and asymptotic performance is estimated. Implementation details may be found in (Hamey, 1992).
1 Introduction
The empirical comparison of neural network training algorithms is of great value in the development of improved techniques and in algorithm selection for problem solving. In view of the great sensitivity of learning times to the random starting weights (Kolen and Pollack, 1990), individual trial times such as reported in (Rumelhart, et al., 1986) are almost useless as measures of learning performance.
Benchmarking experiments normally involve many training trials (typically N = 25 or 100, although Tesauro and Janssens (1988) use N = 10000). For each trial i, the training time to obtain a correct network ti is recorded. Trials which are not successful within a limit of T epochs are considered failures; they are recorded as ti = T . The mean successful training time tT is defined as follows.