The Introduction to the Theory of Neural Computation by Hertz, Krogh and Palmer ( The book fulfills its mission as an introduction for neural network novices. Introduction to Neural Networks. R. Beale & T. Jackson IOP Publishing, Former recommended book. An Introduction to the Theory of. Neural Computation. Introduction to the Theory of Neural Computation. American Journal of Physics 62, (); ronaldweinland.info · John Hertz, Anders Krogh.
|Language:||English, Spanish, Japanese|
|Genre:||Politics & Laws|
|ePub File Size:||21.88 MB|
|PDF File Size:||10.75 MB|
|Distribution:||Free* [*Register to download]|
PDF | Scitation is the online home of leading journals and conference The Introductionto theTheoryof NeuralComputation by Hertz, Kroghand. applications. It also provides coverage of neural. Introduction To The Theory Of Neural Computation DownloadPDF MB. Keywords. Introduction to the theory of neural computation. HOME · Introduction to the Download PDF Neural computation and self-organizing maps, an introduction .
Neuroscience: Exploring the brain 3rd ed. Rieke, D. Warland, R. De Ruyter Van Steveninck, and W. Neuroanatomy for the neuroscientist. Modern Techniques in Neuroscience Research. Dynamical Systems in Neuroscience.
We propose that this Temporal Structure of Spikes is very important in the processing, coding and transfer of information in the cerebral cortex. The temporally structured firing activity enables information to be processed and coded in a way that downstream networks can compute. Accordingly, the existence of this specific temporal structure implies that failures in that precision of spikes will result in processing dysfunctions. An example of the importance of the temporal structure is the spike-timing-dependent plasticity Caporale and Dan, This temporal structure is not fixed.
It can be dynamically adjusted for example by sensory input or by top-down influence to meet the finest processing resolution depending on perceptual, task or attentional demands.
Furthermore, the structure could be adjusted by neuromodulators. Accordingly, variations in the temporal structure will produce changes in the rate and temporal precision of PCs firing in the Ensemble. PCs spikes latencies and synchronization between them will vary accordingly.
Consequently, changes in that temporal precision will code different content. Temporal structure of spikes. Spike timing of PCs is constrained by FS inhibitory interneurons.
These cells set the timing and rate of action potentials produced by PCs limiting the temporal window during which they can be generated Pouille and Scanziani, Moreover, PCs cannot discharge when they are shunted by strong inhibition. Rows represent hypothetical spike rasters for an Ensemble of four PCs.
Spikes from these PCs occur independently but organized inside that temporal structure top. These cycles would result in the oscillations observed in the brain bottom left.
The temporal structure can be dynamically adjusted to meet the finest processing resolution depending on perceptual, task or attentional demands bottom right.
Experimental data provide support for this proposal. It is known that spike timing of PCs is constrained by the inhibitory cells. FS interneurons quickly limit the temporal window during which action potentials can be generated Pouille and Scanziani, ; Li et al. Consequently, PCs are more likely to fire at precise points in time Cardin et al. More importantly, PCs cannot discharge when they are shunted by strong inhibition.
Our hypothesis suggests that these precise Silent Gaps are very important for cortical computation.
It is also known that FS interneurons generate synchronized networks by mutual chemical and electrical connections in the neocortex Whittington et al. Electrical synapses generate highly precise transmission between interneurons of these networks. We propose that this coupling promotes the harmonized firing of connected neurons Jones et al. Different synchronized networks of FS interneurons create different sets of possible Ensembles. The simultaneous firing of the FS interneurons in the inhibitory network generates a synchronized inhibitory activity at their postsynaptic PCs in the Ensemble.
The synchronized spiking of the inhibitory network could be fast enough to adjust the onset spiking of PCs Woodruff et al. The rhythmic functioning of this network creates a sequence of temporal discrete events. This network rhythmically concentrates PCs discharges to particular discrete moments providing observable oscillation cycles at population level.
In sum, the inhibitory network forms a spatial structure of synchronized FS inhibitory cells and then this synchronized network generates a temporal structure of firing in the Ensemble. Discrete Result: A Functional Spatio-Temporal Unit of Computation Spatio-temporal activity patterns play an important role in cortical mechanisms of information processing Ayzenshtat et al.
Consequently, we propose that PCs compute and communicate information by using specific spatio-temporal patterns of spiking. Our hypothesis suggests that the cortex generates and employs these precise patterns to perform its computations. Thus cortical processing depends on the precise temporally structured relations among the respective spikes of PCs of the Ensemble. Information is encoded in the precise relations between temporal structured discharges.
Individual spikes of PCs in the Ensemble take functional relevance when inserted into that temporal structure, forming a Discrete Result. Precise silent periods Silent Gaps inside the structure discretize the processing and allow for the formation of these discrete spatio-temporal functional units. Therefore, all PCs in the Ensemble have the opportunity to fire.
Spikes from these PCs occur independently but organized inside the temporal structure. Each Discrete Result emerges transiently formed by the combination of the firing and silent responses no firing of all PCs in the Ensemble Figure 4. Thus, PCs silent responses are also important in cortical computation and codification.
The same Ensemble can form multiple Discrete Results. This means that the same PCs of the Ensemble, process and encode multitude of contents. Consequently, there are great potential neural possibilities that could form a Discrete Result and also enormous possibilities that could form different sequences of Discrete Results.
Therefore, the number of possible representations that can be formed is titanic. This mechanism could explain why the cortex is so robust to damage. Moreover, the content coded by a Discrete Result depends also on the resolution of the Temporal Structure of Spikes. Discrete Results: neural computational units. PCs belonging to the same Ensemble participate in the Discrete Result. Each potential Discrete Result emerges transiently formed by the combination of the firing and silent responses no firing of all PCs in the Ensemble.
Therefore, PCs silent responses are also important in cortical computation and codification. Moreover, the same Ensemble can form multiple Discrete Results. This means that the same PCs of the Ensemble process and encode multitude of contents.
In this hypothetical example, the Ensemble is formed by seven PCs and the synchronized network of FS interneurons by three cells. Five possible Discrete Results are shown. PCs spikes are displayed as green dots and silent responses no spikes as red dots. On the right, a digital binary representation of these hypothetical Discrete Results is shown. Individual PCs could participate in different Ensembles and be potentially implicated in multiple representations.
Furthermore, different synchronized networks of FS interneurons create different sets of possible Ensembles. Accordingly, the cortex performs computations using multiple Ensembles in parallel creating a multitude of Discrete Results simultaneously. Moreover, different sets of possible Ensembles are created by different synchronized networks of FS interneurons along the cortical processing hierarchy.
Discrete Results at higher levels integrate computational results from previous stages. Therefore, each Discrete Result constitutes a functional unit that has the ability to process, integrate and represent specific content Discrete Results from previous computations Figure 5.
Consequently, in sensory processing, they functionally contribute to unified stimulus codification. Discrete Results: functional units of neural computational integration. Different sets of possible Ensembles are created by different synchronized networks of FS cells along the cortical processing hierarchy.
Experimental studies support this idea. Distinct clusters of FS interneurons have been identified in the cortex. For example, in the rat barrel cortex, one layer 4 FS interneuron type has an axonal domain strictly confined to a barrel Koelbl et al.
Therefore, each Discrete Result constitutes a functional unit that has the ability to process, integrate and represent specific content Discrete Results from previous computations.
Therefore, the Discrete Result concept could explain the binding of separate features enabling perceptual unity. The Discrete Result concept has the ability to explain how complex neural computations underlying cortical processing could be temporally discrete. Consequently, we propose that sensory information would need to be quantized to be computed by the cerebral cortex.
Therefore, processing of sensory information must be temporally discrete and information flow in the cortex must be quantized allowing for the formation of Discrete Results. Therefore, in sensory processing, they can be defined as each neural computational functional unit resulting in quantization of the continuous flow of sensory information Figure 6. Increasing the number of Discrete Results per temporal unit allows resolution enhancement. It could be dynamically adjusted by sensory input or by top-down influence to meet the finest processing resolution depending on perceptual, task or attentional demands.
Discrete Results in cortical sensory processing. Hypothetical spatial maps of cortical neurons for each computational event resulting in quantization of the continuous flow of sensory information are shown.
Green cells show a representative Ensemble of PCs organized by a specific synchronized network of FS interneurons. Individual spikes of these PCs take functional relevance inserted into a temporal structure forming discrete spatiotemporal functional units Discrete Results.
In this hypothetical example, relevant sensory information predator movements can be extracted by computing differences between the Discrete Results. Experimental studies have increased our knowledge about how this sequential activity is generated in the brain Harris et al. However, untangling its functional computational significance is still a formidable challenge today.
We propose that precise sequences of Discrete Results are the mechanism used by the cortex to perform computations. The computation of the Discrete Results sequence is the mechanism used by the cortex to extract, code, memorize and transmit neural information.
This proposal is a neuronal population mechanism to compute and code. Dynamic sequences of Discrete Results generate representations. Different sequences codify different contents. The rhythmic functioning of the synchronized inhibitory network creates a sequence of Discrete Results Figure 7. Computations between successive Discrete Results in the sequence produce the power of the cortical processing.
Experimental data provide support for this hypothesis.
Sequential activity of multineuronal spiking has been well described in the cortex Fujisawa et al. Dynamic sequences of Discrete Results are the mechanism used by the cortex to perform computations. Different sequences of Discrete Results codify different content.
In this example, both types of predator trajectories involve different sequentially activated Discrete Results. One sequence codifies the predator approaching and the other leaving. Moreover, cortical processing by dynamic sequences of Discrete Results could be the neural source of some rhythmic signals observed at population level.
This hypothesis of neural processing could be applied to other structures and nuclei of the brain. Moreover, we propose that cortical processing is produced by the computation of discrete spatio-temporal functional units.
But what could be the neuronal elements underlining this computation? The cerebral cortex is composed of many types of neurons.
Although all of them play a key role in cortical processing, our hypothesis suggests that there must be a specific type of inhibitory cell that may be implicated in the creation of a spatio-temporal structure supporting discrete cortical computation. We propose that FS interneuron may be a key element in our hypothesis providing the basis for this computation.
These cells forming a synchronized spatially distributed cortical network may impose a temporal spike restriction in PCs creating functionally coupled units of computation. Their rhythmic activity may create a sequence of spatio-temporal functional units Discrete Results , discretizing the information processing.
In sum, we propose that they are able to integrate the spatial and temporal dimension of cortical computation. FS cells Kawaguchi and Kubota, are the largest population of interneurons in the neocortex. They play a key role as pacemakers for oscillations Whittington et al. However, it is still unclear how these cells functionally contribute to the operations performed by the cortex.
It is known that they form dense matrices covering PCs Packer and Yuste, extending a blanket of inhibition onto them Karnani et al. They shape the precise timing and dynamic range of action potentials produced by PCs Pouille and Scanziani, ; Cardin et al.
They generate synchronized networks by mutual chemical and electrical connections Galarreta and Hestrin, ; Gibson et al. Accordingly, they fire in high synchrony Jones et al. An extended review Weigend, discusses more recent developments in supervised feedforward neural networks, some of which are part of the helpful trend towards analyzing the networks as a class of statistical likelihood models. We conclude our review with a brief summary of the contents of each chapter.
Chapter 1 provides a brief review of the main characteristics of neurons in the brain, and how these might be related to artificial neurons. After a brief review of the history of research in neural networks, the chapter concludes with a discussion of some important practical issues regarding neural networks research.
Chapter 2 outlines the problem of associative memory describes the binary, discrete symmetric autoassociator as described by Hopfield, and analyzes its storage capacity. This chapter includes an illuminatingdescription of the similarity between the simple networks of McCulloch and Pitts, and the Ising spin model from statistical mechanics.
The chapter summarizes in simple terms how mean field theory can be used to analyze the behavior of a collection of simple, interacting elements, be they atoms in a lattice-like material or binary neurons in a fully connected network. Chapter 3 describes several modifications of Hopfield's autoassociator, including the extension to continuous units. The chapter also briefly discusses hardware implementations, and application of these networks to the generation of temporal sequences of patterns.
Chapter 4 illustrates several applications of this class of models, focusing on optimization problems. The chapter begins by describing how one might construct a meaningful energy function for a simple problem, and how the handcrafted energy function can be used to derive a network structure to solve the problem. The chapter then describes how the continuous autoassociator can solve the classical traveling salesman problem.
After one more example, the chapter closes with a description of applications in image processing. Chapter 5 moves to the class of supervised, error-based, feedforward networks, beginning with the simple perceptron. The authors do a nice job of summarizing simple concepts of classification, and then describing in an intuitive fashion the workings of the simple perceptron.
After a stand-alone section proving perceptron convergence when inputs are linearly separable , the chapter describes extensions to include nonlinear or stochastic units. Chapter 6 extends the analysis of Chapter 5 to the realm of multilayer perceptrons and back propagation. Palmer gradient descent on the error surface in weight space.
Section 6. The chapter concludes with an overview of methods for modifying the network architecture to improve performance. Chapter 7 focuses on a variety of recurrent or feedback networks, including the Boltzmann Machine an interesting cross between a multilayer perceptron and an autoassociator , recurrent back propagation, and other models for learning time sequences.
The closing section describes reinforcement learning models. It is unclear why this class of model appears in this chapter, as the "feedback" here is simply a signal indicating how the network is performing. This section could easily be extended and turned into an independent chapter. Chapter 8 gives a very nice analysis of feedforward networks that use associative, or Hebbian learning.
The chapter provides a useful discussion of how Hebbian networks are related to Principal Component Analysis, a technique for extracting information about the dimensions along which some input data exhibit maximum variance. The closing section discusses superficially the concept of self-organizing feature extraction. This topic serves as a lead-in to the material in the next chapter. Chapter 9 treats material that could easily take up one or two textbooks, under the heading of unsupervised competitive learning.
In spite of its condensed format, the chapter does a reasonable job of summarizing some of the main computational points of relevant models based on competitive learning, such as adaptive resonance theory and self-organizing feature maps. The chapter includes a theoretical analysis of self-organizing feature maps, and a classical example in which they are applied to the traveling salesman problem.
Chapter 9 concludes with a section on normalized radial basis function networks, which, as an important error-based supervised model, might better have been covered in Chapter 6.
Chapter 10 caps the book with a much more formal mathematical treatment of two problems: This chapter could probably have been absorbed as special sections in two earlier chapters.
In conclusion, we have found ITNC to be one of the best books around for the material that it covers. In a novel scientific field that is constantly being redefined and shaped by new advances, this book is likely to become a "classic. Rosen Ctr. References not listed below appear in the original text by Hertz, Krogh, and Palmer. Albus, J. A new approach to manipulator control: The cerebellar model articulation controller CMAC.
Journal of Dynamic Systems, Measurement, and Control, 37, Carpenter, G. Pattern recognition by self-organizing neural networks. Cambridge, MA: MIT Press. Levine, D. Introduction to neural and cognitive modeling.
Hillsdale, N J: Lawrence Erlbaum Associates. Weigend, A. Book review: Artificial intelligencejournal in press.