.

Netlab: Algorithms for Pattern Recognition

  • Chapter 1: Introduction
    Introduction to Matlab: Matlab Basics; Matlab Plotting; Programming in Matlab; Programming Facilities.
    The Netlab Toolbox: Overview of Netlab; Generic Functions.
    Worked Example: Data Normalisation

  • Chapter 2: Parameter Optimisation Algorithms
    • Controlling the Algorithms: Information Display; Termination Criteria; Extra Arguments and Return Values; Demonstration Program. Quadratic Approximation at a Minimum Line Search: Precision; Line Minimisation Algorithms; Bracketing the Minimum. Batch Gradient Descent Conjugate Gradients Scaled Conjugate Gradients Quasi-Newton Methods Optimisation and Neural Networks: General Purpose Optimisation Algorithms; On-line Gradient Descent. Worked Example: Constrained Optimisation

  • Chapter 3: Density Modelling and Clustering
    Gaussian Mixture Models: Theory; Implementation.
    Computing Probabilities: GMM Activations; GMM Probabilities; GMM Posteriors.
    EM Training Algorithm: Algorithm Theory; Initialisation; Netlab EM Implementation.
    Demonstrations of GMM: EM Algorithm; Density Modelling.
    K-means Clustering: Algorithm and Netlab Implementation; Demonstration Program.
    K-nearest-neighbour: Algorithm and Netlab Implementation; Demonstration Program.
    Worked Examples: Classification with Density Models; Novelty Detection.

  • Chapter 4: Single Layer Networks
    • The Single Layer Feed-forward Network: Netlab Implementation; Forward Propagation. Error Functions Error Gradient Calculation Evaluating Other Derivatives: Network Activation Derivatives; The Hessian Matrix. Iterated Re-weighted Least Squares Training Demonstration Programs: Two-class Problem; Three-class Problem. Worked Example: Training Regularised Models

  • Chapter 5: The Multi-layer Perceptron
    • The Two-layer Feed-forward Network: Definition; Network Creation and Initialisation; Manipulating Weights; Forward Propagation. Error Functions and Network Training Error Gradient Calculation Evaluating other Derivatives The Hessian Matrix: Fast Multiplication by the Hessian; Netlab Implementation. Demonstration Programs: Regression Demonstration; Classification Demonstration. Mixture Density Networks: Model Structure; Network Creation and Initialisation; Forward Propagation; Conditional Probabilities; Error and Gradient Calculation; Demonstration Program. Worked Example: Adding Direct Connections

  • Chapter 6: Radial Basis Functions
    • The RBF Network: Theory; Netlab}\ Implementation. Special Purpose Training Algorithms: Basis Function Optimisation; Output Weight Optimisation; Netlab Implementation. Error and Error Gradient Calculation Evaluating Other Derivatives: Output Derivatives; Network Jacobian; Network Hessian. Demonstration Program Worked Examples: Linear Smoothing; Dual Basis Functions; Characterising Network Complexity.

  • Chapter 7: Visualisation and Latent Variable Models
    • Principal Component Analysis: Theory; Netlab Implementation; Examples. Probabilistic Principal Component Analysis: Probabilistic PCA; PPCA Implementation; Mixture of PPCA; Mixture of PPCA Implementation. Generative Topographic Mapping: The GTM Model; Model Creation and Initialisation; Computing Probabilities; EM Algorithm; Magnification Factors; Demonstration Programs. Topographic Projection: Neuroscale Algorithm; Neuroscale Implementation; Demonstration of Neuroscale. Worked Example: Canonical Variates.

  • Chapter 8: Sampling
    • Monte Carlo Integration Basic Sampling: Random Number Generators; Transformation Methods; Rejection Sampling; Importance Sampling. Markov Chain Sampling: Markov Chain Fundamentals; Gibbs Sampling; Metropolis-Hastings Algorithm; Hybrid Monte Carlo. Demonstration Programs: Metropolis-Hastings Sampling; Hybrid Monte Carlo Sampling. Worked Example: Convergence Diagnostics

  • Chapter 9: Bayesian Techniques
    • Principles of Bayesian Inference Priors for Neural Networks: Theory of Priors; Netlab Implementation of Priors; Demonstration of Gaussian Priors; Masks and Weight Manipulation. Computing Error and Gradient Functions The Evidence Procedure: Theory; Netlab Implementation. Predictions and Error Bars: Predictions for Regression; Predictions for Classification; Netlab Implementation. Demonstrations of Evidence Procedure: The Evidence Procedure for Regression; The Evidence Procedure for Classification; Automatic Relevance Determination. Monte Carlo Methods Demonstration of Hybrid Monte Carlo for MLPs Worked Example: Improved Classification Approximation

  • Chapter 10: Gaussian Processes
    • Bayesian Regression: Weight Space View; Function Space View. Theory of Gaussian Processes: Model Definition; Learning Hyperparameters. Netlab Implementation: Model Creation and Initialisation; Making Predictions; Model Training. Demonstration Programs: Regression Demonstration; ARD Demonstration. Worked Example: GPs for Classification

  • Appendices: Linear Algebra and Matrices; Algorithm Error Analysis.