The Boltzmann Machine Learning Algorithm

The Goal of Learning

In the context of Boltzmann Machines, the primary objective of the learning process is to empower the model to accurately represent the underlying probability distribution of the training data. This is achieved by adjusting the network's weights and biases to maximize the likelihood that the machine assigns to the observed data.

Maximizing Probabilities for Training Data

• Log-Probability Maximization: The learning algorithm seeks to maximize the product of the probabilities that the Boltzmann machine assigns to the binary vectors present in the training set. Mathematically, this is equivalent to maximizing the sum of the log probabilities for these training vectors, which simplifies optimization through gradient ascent.
• Statistical Replication: An intuitive way to understand this goal is to imagine repeating an experiment: if we allowed the Boltzmann machine to settle to its stationary distribution `N` different times (where `N` is the number of training cases) without any external input, and then sampled a visible vector once each time, we would aim for these `N` samples to exactly match our original `N` training cases. This implies that the model has learned the data's distribution perfectly.

Why the Learning Could Be Difficult

While the goal is clear, the practical implementation of Boltzmann Machine learning presents significant computational challenges, primarily due to the intricate dependencies among the units, especially within hidden layers. These dependencies lead to a complex energy landscape that is difficult to navigate during optimization.

The Interdependence of Weights

Consider a simplified scenario: a chain of units where visible units are situated at the ends, connected through a series of hidden units. Let's denote the weights connecting these units as w1, w2, w3, w4, w5 along the chain.

The gradient of the log-likelihood function for a Boltzmann Machine involves two terms: a positive term representing the "data-driven" correlation and a negative term representing the "model-driven" correlation. Calculating these correlations accurately requires running the network until it reaches thermal equilibrium, a process that can be computationally intensive and slow, especially in networks with many hidden units or complex connectivity. This "sampling problem" is a core difficulty in Boltzmann Machine learning.