1

page 1 (13 pages)
		2

Analysis of the Error Surface of the XOR Network with Two

Hidden Nodes

Leonard G. C. Hamey

Department of Computing

Macquarie University

NSW 2109 Australia

[email protected]

Computing Report 95{167C

9 Feb 1995

Abstract

The exclusive-or learning task in a feed-forward neural network with two hidden nodes is investigated. Constraint equations are derived which fully describe the finite stationary points of the error surface. It is shown that the stationary points occur in a single connected union of eighteen manifolds. A Taylor series expansion is applied to the network error surface and it is shown that all points within the enumerated manifolds are arbitrarily close to points of lower error. It follows that the finite stationary points of the exclusive-or task are not relative minima. This result is surprising in view of the commonly held belief that the exclusive-or task exhibits local minima. The present result complements a recent result of the author's which proves the absence of regional local minima in the exclusive-or task.

1 Introduction

It is well known that back-propagation learning can become trapped when being trained on the exclusiveor task with two hidden nodes (figure 1). However, the occurrence of trapped networks, which are commonly called local minima, has been observed to be rare [1] while depending upon the initial conditions and the network learning parameters [2]. The present paper presents a theoretical analysis of the error surface of the exclusive-or task.

The study of the error surfaces of feed-forward neural networks is hampered by high dimensionality and the difficulty of theoretical analysis. Although some results have been forthcoming, these are for restricted cases. Analyses exist for networks without hidden nodes [3, 4, 5], networks comprised of linear nodes [6] and networks with as many hidden nodes as training patterns [7]. In general, networks with less hidden nodes than training patterns appear not to be amenable to analysis. A significant exception is the exclusive-or network (figure 1).

Blum [8] proved the existence of solutions in the exclusive-or learning task. They attempted to prove the existence of a manifold of relative local minima in the error surface, but their proof was flawed as shown previously by the author in [9]. Lisboa and Perantonis [10] characterise the stationary points of the error surface, obtaining four classes. Their classes (b){(d) occur only as points with infinite weight values but class (a) occurs for finite weight values. Hamey [9] proves that the exclusive-or task does not have any regional local minima. Other analysis of the exclusive-or network and related learning tasks may be found in [11, 12, 13, 14].
The present paper extends the results of [9] by considering finite relative local minima. A point w0 is said to be a relative minimum of a function f(w) if there exists ffl > such that f(w0 +?w) <= f(w0) for
all j?wj < ffl. As discussed in [9], this definition is unsuitable for the consideration of minima that occur with infinite weights, hence the adoption of an alternate definition of local minimum in that paper. In