Learning With Feed Forward Neural Networks Three Schemes To Deal With The Bias Variance Trade Off

Eighth International Work-Conference on Artificial and Natural Neural Networks

We present in this volume the collection of finally accepted papers of the eighth edition of the “IWANN” conference (“International Work-Conference on Artificial Neural Networks”). This biennial meeting focuses on the foundations, theory, models and applications of systems inspired by nature (neural networks, fuzzy logic and evolutionary systems). Since the first edition of IWANN in Granada (LNCS 540, 1991), the Artificial Neural Network (ANN) community, and the domain itself, have matured and evolved. Under the ANN banner we find a very heterogeneous scenario with a main interest and objective: to better understand nature and beings for the correct elaboration of theories, models and new algorithms. For scientists, engineers and professionals working in the area, this is a very good way to get solid and competitive applications. We are facing a real revolution with the emergence of embedded intelligence in many artificial systems (systems covering diverse fields: industry, domotics, leisure, healthcare, ... ). So we are convinced that an enormous amount of work must be, and should be, still done. Many pieces of the puzzle must be built and placed into their proper positions, offering us new and solid theories and models (necessary tools) for the application and praxis of these current paradigms. The above-mentioned concepts were the main reason for the subtitle of the IWANN 2005 edition: “Computational Intelligence and Bioinspired Systems.” The call for papers was launched several months ago, addressing the following topics: 1. Mathematical and theoretical methods in computational intelligence.

Learning with Feed-forward Neural Networks: Three Schemes to Deal with the Bias/Variance Trade-off

In terms of the Bias/Variance decomposition, very flexible (i.e., complex) Supervised Machine Learning systems may lead to unbiased estimators but with high variance. A rigid model, in contrast, may lead to small variance but high bias. There is a trade-off between the bias and variance contributions to the error, where the optimal performance is achieved. In this work we present three schemes related to the control of the Bias/Variance decomposition for Feed-forward Neural Networks (FNNs) with the (sometimes modified) quadratic loss function: 1. An algorithm for sequential approximation with FNNs, named Sequential Approximation with Optimal Coefficients and Interacting Frequencies (SAOCIF). Most of the sequential approximations proposed in the literature select the new frequencies (the non-linear weights) guided by the approximation of the residue of the partial approximation. We propose a sequential algorithm where the new frequency is selected taking into account its interactions with the previously selected ones. The interactions are discovered by means of their optimal coefficients (the linear weights). A number of heuristics can be used to select the new frequencies. The aim is that the same level of approximation may be achieved with less hidden units than if we only try to match the residue as best as possible. In terms of the Bias/Variance decomposition, it will be possible to obtain simpler models with the same bias. The idea behind SAOCIF can be extended to approximation in Hilbert spaces, maintaining orthogonal-like properties. In this case, the importance of the interacting frequencies lies in the expectation of increasing the rate of approximation. Experimental results show that the idea of interacting frequencies allows to construct better approximations than matching the residue. 2. A study and comparison of different criteria to perform Feature Selection (FS) with Multi-Layer Perceptrons (MLPs) and the Sequential Backward Selection (SBS) procedure w.

Learning with feed-forward neural networks[