Change Of Representation And Inductive Bias

Change of Representation and Inductive Bias

Change of Representation and Inductive Bias

Change of Representation and Inductive Bias One of the most important emerging concerns of machine learning researchers is the dependence of their learning programs on the underlying representations, especially on the languages used to describe hypotheses. The effectiveness of learning algorithms is very sensitive to this choice of language; choosing too large a language permits too many possible hypotheses for a program to consider, precluding effective learning, but choosing too small a language can prohibit a program from being able to find acceptable hypotheses. This dependence is not just a pitfall, however; it is also an opportunity. The work of Saul Amarel over the past two decades has demonstrated the effectiveness of representational shift as a problem-solving technique. An increasing number of machine learning researchers are building programs that learn to alter their language to improve their effectiveness. At the Fourth Machine Learning Workshop held in June, 1987, at the University of California at Irvine, it became clear that the both the machine learning community and the number of topics it addresses had grown so large that the representation issue could not be discussed in sufficient depth. A number of attendees were particularly interested in the related topics of constructive induction, problem reformulation, representation selection, and multiple levels of abstraction. Rob Holte, Larry Rendell, and I decided to hold a workshop in 1988 to discuss these topics. To keep this workshop small, we decided that participation be by invitation only.

Structure Level Adaptation for Artificial Neural Networks

63 3. 2 Function Level Adaptation 64 3. 3 Parameter Level Adaptation. 67 3. 4 Structure Level Adaptation 70 3. 4. 1 Neuron Generation . 70 3. 4. 2 Neuron Annihilation 72 3. 5 Implementation . . . . . 74 3. 6 An Illustrative Example 77 3. 7 Summary . . . . . . . . 79 4 Competitive Signal Clustering Networks 93 4. 1 Introduction. . 93 4. 2 Basic Structure 94 4. 3 Function Level Adaptation 96 4. 4 Parameter Level Adaptation . 101 4. 5 Structure Level Adaptation 104 4. 5. 1 Neuron Generation Process 107 4. 5. 2 Neuron Annihilation and Coalition Process 114 4. 5. 3 Structural Relation Adjustment. 116 4. 6 Implementation . . 119 4. 7 Simulation Results 122 4. 8 Summary . . . . . 134 5 Application Example: An Adaptive Neural Network Source Coder 135 5. 1 Introduction. . . . . . . . . . 135 5. 2 Vector Quantization Problem 136 5. 3 VQ Using Neural Network Paradigms 139 Vlll 5. 3. 1 Basic Properties . 140 5. 3. 2 Fast Codebook Search Procedure 141 5. 3. 3 Path Coding Method. . . . . . . 143 5. 3. 4 Performance Comparison . . . . 144 5. 3. 5 Adaptive SPAN Coder/Decoder 147 5. 4 Summary . . . . . . . . . . . . . . . . . 152 6 Conclusions 155 6. 1 Contributions 155 6. 2 Recommendations 157 A Mathematical Background 159 A. 1 Kolmogorov's Theorem . 160 A. 2 Networks with One Hidden Layer are Sufficient 161 B Fluctuated Distortion Measure 163 B. 1 Measure Construction . 163 B. 2 The Relation Between Fluctuation and Error 166 C SPAN Convergence Theory 171 C. 1 Asymptotic Value of Wi 172 C. 2 Energy Function . .