page 1  (14 pages)
2to next section

A Comparison of Back Propagation Implementations

Jondarr Gibb and Leonard Hamey

Department of Computing

Macquarie University

Technical Report C/TR95-06

September, 1995

Abstract

Back propagation training algorithms have been implemented by many researchers for their own purposes and provided publicly on the internet for others to use in verification of published results and for reuse in unrelated research projects. Often, the source code of a package is used as the basis for a new package for demonstrating new algorithm variations, or some functionality is added specifically for analysis of results. However, there are rarely any guarantees that the original implementation is faithful to the algorithm it represents, or that its code is bug free or accurate. This report attempts to look at a few implementations and provide a test suite which shows deficiencies in some software available which the average researcher may not be aware of, and may not have the time to discover on their own. This test suite may then be used to test the correctness of new packages.

1 Introduction

There are many neural network packages freely available on the Internet which allow the investigator to reuse an implementation developed and tested elsewhere. However, what guarantees are there that such an implementation is a working package? Many researchers in the area seem to have come to the conclusion that it is more worthwhile to develop an in-house package for the specific needs of the research at hand, and then make that package available when it has been used for some time, and its usefulness has been proven. If everyone follows this policy, then we have a proliferation of public packages which no-one else can use, some of which are not tested beyond the application for which they were written. Each of these packages will have some useful features, but may not be perfectly suited to an average researcher's task at hand (and may therefore need to be further modified by the researcher).

Not only is it imperative that a researcher know if a public domain package is correctly implementing the desired algorithm, is accurate, and is flexible enough to do what is required, it is also important that any package developed in-house be verifiable before it is used to demonstrate results in comparative studies and support arguments for newly-developed algorithms.

This report attempts to codify a series of tests for the researcher/developer to use on back propagation simulation packages of any origin, including pitfalls for the developer. We also look at a few commonly used and freely available packages.

In addition, the way in which implementation variations may affect results is implied by the application of these tests. Whether such effects are within the tolerance of experimental error, or are statistically insignificant in the realm of Neural Network simulations is an important question. Kolen and Pollack [7] have already investigated the effects of initial conditions (that is, weight configuration) on the outcome of experiments, and we intend to show how implementation differences may likewise have an effect on the final outcome of training a network. Xie and Jabri [26], and also Sakaue, et al [17] have investigated low-precision networks (around 10 bits), and found that the