|
LDA++
|
#include <SecondOrderLogisticRegressionApproximation.hpp>
Public Member Functions | |
| SecondOrderLogisticRegressionApproximation (const MatrixX &X, const std::vector< MatrixX > &X_var, const Eigen::VectorXi &y, VectorX Cy, Scalar L) | |
| SecondOrderLogisticRegressionApproximation (const MatrixX &X, const std::vector< MatrixX > &X_var, const Eigen::VectorXi &y, Scalar L) | |
| Scalar | value (const MatrixX &eta) const |
| void | gradient (const MatrixX &eta, Eigen::Ref< MatrixX > grad) const |
SecondOrderLogisticRegressionApproximation is a second order taylor approximation to the expectation of the logistic loss function of a random variable.
We use this class to approximate the following equation in the lower bound of the likelihood of an LDA model. \(q\) is the variational distribution used and \(\bar{z}\) is a random variable (the mean of the topic assignments). The equation below is for a single document.
\[ \mathbb{E}_q\left[ \eta_{y_n}^T \bar{z} - \log\left( \sum_{\hat{y}=1}^Y \exp(\eta_{\hat{y}}^T \bar{z}) \right) \right] \approx \eta_{y_n}^T \mathbb{E}_q[\bar{z}] - \log \sum_{\hat{y}=1}^Y \exp(\eta_{\hat{y}}^T \mathbb{E}_q[\bar{z})\left( 1 + \frac{1}{2} \eta_{\hat{y}}^T \mathbb{V}_q[\bar{z}] \eta_{\hat{y}} \right) \]
| ldaplusplus::optimization::SecondOrderLogisticRegressionApproximation< Scalar >::SecondOrderLogisticRegressionApproximation | ( | const MatrixX & | X, |
| const std::vector< MatrixX > & | X_var, | ||
| const Eigen::VectorXi & | y, | ||
| VectorX | Cy, | ||
| Scalar | L | ||
| ) |
| X | The documents defining the minimization problem ( \(X \in \mathbb{R}^{D \times N}\)) |
| X_var | A vector containing the variance matrix for each document ( \(X_{\text{var}} \in \mathbb{R}^{N \times D \times D}\)) |
| y | The class indexes for each document ( \(y \in \mathbb{N}^N\)) |
| Cy | A different weight for each class in the optimization problem |
| L | The L2 regularization penalty for the weights |
| ldaplusplus::optimization::SecondOrderLogisticRegressionApproximation< Scalar >::SecondOrderLogisticRegressionApproximation | ( | const MatrixX & | X, |
| const std::vector< MatrixX > & | X_var, | ||
| const Eigen::VectorXi & | y, | ||
| Scalar | L | ||
| ) |
| X | The documents defining the minimization problem ( \(X \in \mathbb{R}^{D \times N}\)) |
| X_var | A vector containing the variance matrix for each document ( \(X_{\text{var}} \in \mathbb{R}^{N \times D \times D}\)) |
| y | The class indexes for each document ( \(y \in \mathbb{N}^N\)) |
| L | The L2 regularization penalty for the weights |
| void ldaplusplus::optimization::SecondOrderLogisticRegressionApproximation< Scalar >::gradient | ( | const MatrixX & | eta, |
| Eigen::Ref< MatrixX > | grad | ||
| ) | const |
The gradient of the objective function implemented in value().
We use \(I(y) \in \mathbb{R}^Y\) as the indicator vector of \(y\) (a vector with all the values 0 except at the yth position).
\[ \nabla_{\eta} J = - \sum_{n=1}^N C_{y_n} \left( X_n I(y_n)^T - \frac{ \sum_{\hat{y}=1}^Y \left(\left( X_n \exp(\eta_{\hat{y}}^T X_n) \left( 1 + \frac{1}{2} \eta_{\hat{y}}^T X_n^{\text{var}} \eta_{\hat{y}} \right)\right) + \left( \frac{1}{2} \exp(\eta_{\hat{y}}^T X_n) \eta_{\hat{y}}^T \left( X_n^{\text{var}} + \left(X_n^{\text{var}}\right)^T \right)\right) \right) I(\hat{y})^T } { \sum_{\hat{y}=1}^Y \exp(\eta_{\hat{y}}^T X_n) \left( 1 + \frac{1}{2} \eta_{\hat{y}}^T X_n^{\text{var}} \eta_{\hat{y}} \right) } \right) + L \eta \]
| eta | The weights of the linear model ( \(\eta \in \mathbb{R}^{D \times Y}\)) |
| grad | A matrix of dimensions equal to \(\eta\) that will hold the result |
| Scalar ldaplusplus::optimization::SecondOrderLogisticRegressionApproximation< Scalar >::value | ( | const MatrixX & | eta | ) | const |
The value of the objective function to be minimized.
\(N\) is the number of documents (different vectors), \(X_n \in \mathbb{R}^D\) is the nth document, \(\eta_y \in \mathbb{R}^D\) is the weights vector for the class \(y\) defining the hyperplane that separates class \(y\) from all the other, \(y_n\) is the class of the nth document and finally \(X_n^{\text{var}} \in \mathbb{R}^{D \times D}\) is the variance of the nth document (see the class description).
\[ J = - \sum_{n=1}^N C_{y_n} \left( \eta_{y_n}^T X_n - \log \sum_{\hat{y}=1}^Y \exp(\eta_{\hat{y}}^T X_n) \left( 1 + \frac{1}{2} \eta_{\hat{y}}^T X_n^{\text{var}} \eta_{\hat{y}} \right) \right) + \frac{L}{2} \left\| \eta \right\|_F^2 \]
| eta | The weights of the linear model ( \(\eta \in \mathbb{R}^{D \times Y}\)) |
1.8.11
