A unified statistical approach to non-negative matrix factorization and probabilistic latent semantic indexing

Karthik Devarajan, Guoli Wang, Nader Ebrahimi

Research output: Contribution to journalArticlepeer-review

14 Scopus citations


Non-negative matrix factorization (NMF) is a powerful machine learning method for decomposing a high-dimensional nonnegative matrix V into the product of two nonnegative matrices, W and H, such that V ∼ WH. It has been shown to have a parts-based, sparse representation of the data. NMF has been successfully applied in a variety of areas such as natural language processing, neuroscience, information retrieval, image processing, speech recognition and computational biology for the analysis and interpretation of large-scale data. There has also been simultaneous development of a related statistical latent class modeling approach, namely, probabilistic latent semantic indexing (PLSI), for analyzing and interpreting co-occurrence count data arising in natural language processing. In this paper, we present a generalized statistical approach to NMF and PLSI based on Renyi’s divergence between two non-negative matrices, stemming from the Poisson likelihood. Our approach unifies various competing models and provides a unique theoretical framework for these methods. We propose a unified algorithm for NMF and provide a rigorous proof of monotonicity of multiplicative updates for W and H. In addition, we generalize the relationship between NMF and PLSI within this framework. We demonstrate the applicability and utility of our approach as well as its superior performance relative to existing methods using real-life and simulated document clustering data.

Original languageEnglish
Pages (from-to)137-163
Number of pages27
JournalMachine Learning
Issue number1
StatePublished - Apr 2015
Externally publishedYes


  • Biomedical informatics
  • EM algorithm
  • Nonnegative matrix factorization
  • Probabilistic latent semantic indexing
  • Renyi’s divergence
  • λ-log-likelihood


Dive into the research topics of 'A unified statistical approach to non-negative matrix factorization and probabilistic latent semantic indexing'. Together they form a unique fingerprint.

Cite this