LDA++
ldaplusplus::optimization::SecondOrderLogisticRegressionApproximation< Scalar > Class Template Reference

#include <SecondOrderLogisticRegressionApproximation.hpp>

## Public Member Functions

SecondOrderLogisticRegressionApproximation (const MatrixX &X, const std::vector< MatrixX > &X_var, const Eigen::VectorXi &y, VectorX Cy, Scalar L)

SecondOrderLogisticRegressionApproximation (const MatrixX &X, const std::vector< MatrixX > &X_var, const Eigen::VectorXi &y, Scalar L)

Scalar value (const MatrixX &eta) const

## Detailed Description

### template<typename Scalar> class ldaplusplus::optimization::SecondOrderLogisticRegressionApproximation< Scalar >

SecondOrderLogisticRegressionApproximation is a second order taylor approximation to the expectation of the logistic loss function of a random variable.

We use this class to approximate the following equation in the lower bound of the likelihood of an LDA model. $$q$$ is the variational distribution used and $$\bar{z}$$ is a random variable (the mean of the topic assignments). The equation below is for a single document.

$\mathbb{E}_q\left[ \eta_{y_n}^T \bar{z} - \log\left( \sum_{\hat{y}=1}^Y \exp(\eta_{\hat{y}}^T \bar{z}) \right) \right] \approx \eta_{y_n}^T \mathbb{E}_q[\bar{z}] - \log \sum_{\hat{y}=1}^Y \exp(\eta_{\hat{y}}^T \mathbb{E}_q[\bar{z})\left( 1 + \frac{1}{2} \eta_{\hat{y}}^T \mathbb{V}_q[\bar{z}] \eta_{\hat{y}} \right)$

## Constructor & Destructor Documentation

template<typename Scalar >
 ldaplusplus::optimization::SecondOrderLogisticRegressionApproximation< Scalar >::SecondOrderLogisticRegressionApproximation ( const MatrixX & X, const std::vector< MatrixX > & X_var, const Eigen::VectorXi & y, VectorX Cy, Scalar L )
Parameters
 X The documents defining the minimization problem ( $$X \in \mathbb{R}^{D \times N}$$) X_var A vector containing the variance matrix for each document ( $$X_{\text{var}} \in \mathbb{R}^{N \times D \times D}$$) y The class indexes for each document ( $$y \in \mathbb{N}^N$$) Cy A different weight for each class in the optimization problem L The L2 regularization penalty for the weights
template<typename Scalar >
 ldaplusplus::optimization::SecondOrderLogisticRegressionApproximation< Scalar >::SecondOrderLogisticRegressionApproximation ( const MatrixX & X, const std::vector< MatrixX > & X_var, const Eigen::VectorXi & y, Scalar L )
Parameters
 X The documents defining the minimization problem ( $$X \in \mathbb{R}^{D \times N}$$) X_var A vector containing the variance matrix for each document ( $$X_{\text{var}} \in \mathbb{R}^{N \times D \times D}$$) y The class indexes for each document ( $$y \in \mathbb{N}^N$$) L The L2 regularization penalty for the weights

## Member Function Documentation

template<typename Scalar >
 void ldaplusplus::optimization::SecondOrderLogisticRegressionApproximation< Scalar >::gradient ( const MatrixX & eta, Eigen::Ref< MatrixX > grad ) const

The gradient of the objective function implemented in value().

We use $$I(y) \in \mathbb{R}^Y$$ as the indicator vector of $$y$$ (a vector with all the values 0 except at the yth position).

$\nabla_{\eta} J = - \sum_{n=1}^N C_{y_n} \left( X_n I(y_n)^T - \frac{ \sum_{\hat{y}=1}^Y \left(\left( X_n \exp(\eta_{\hat{y}}^T X_n) \left( 1 + \frac{1}{2} \eta_{\hat{y}}^T X_n^{\text{var}} \eta_{\hat{y}} \right)\right) + \left( \frac{1}{2} \exp(\eta_{\hat{y}}^T X_n) \eta_{\hat{y}}^T \left( X_n^{\text{var}} + \left(X_n^{\text{var}}\right)^T \right)\right) \right) I(\hat{y})^T } { \sum_{\hat{y}=1}^Y \exp(\eta_{\hat{y}}^T X_n) \left( 1 + \frac{1}{2} \eta_{\hat{y}}^T X_n^{\text{var}} \eta_{\hat{y}} \right) } \right) + L \eta$

Parameters
 eta The weights of the linear model ( $$\eta \in \mathbb{R}^{D \times Y}$$) grad A matrix of dimensions equal to $$\eta$$ that will hold the result
template<typename Scalar >
 Scalar ldaplusplus::optimization::SecondOrderLogisticRegressionApproximation< Scalar >::value ( const MatrixX & eta ) const

The value of the objective function to be minimized.

$$N$$ is the number of documents (different vectors), $$X_n \in \mathbb{R}^D$$ is the nth document, $$\eta_y \in \mathbb{R}^D$$ is the weights vector for the class $$y$$ defining the hyperplane that separates class $$y$$ from all the other, $$y_n$$ is the class of the nth document and finally $$X_n^{\text{var}} \in \mathbb{R}^{D \times D}$$ is the variance of the nth document (see the class description).

$J = - \sum_{n=1}^N C_{y_n} \left( \eta_{y_n}^T X_n - \log \sum_{\hat{y}=1}^Y \exp(\eta_{\hat{y}}^T X_n) \left( 1 + \frac{1}{2} \eta_{\hat{y}}^T X_n^{\text{var}} \eta_{\hat{y}} \right) \right) + \frac{L}{2} \left\| \eta \right\|_F^2$

Parameters
 eta The weights of the linear model ( $$\eta \in \mathbb{R}^{D \times Y}$$)

The documentation for this class was generated from the following files: