A statistical model for topically segmented documents

Giovanni Ponti, Andrea Tagarelli, George Karypis

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

Generative models for text data are based on the idea that a document can be modeled as a mixture of topics, each of which is represented as a probability distribution over the terms. Such models have traditionally assumed that a document is an indivisible unit for the generative process, which may not be appropriate to handle documents with an explicit multi-topic structure. This paper presents a generative model that exploits a given decomposition of documents in smaller text blocks which are topically cohesive (segments). A new variable is introduced to model the within-document segments: using this variable at document-level, word generation is related not only to the topics but also to the segments, while the topic latent variable is directly associated to the segments, rather than to the document as a whole. Experimental results have shown that, compared to existing generative models, our proposed model provides better perplexity of language modeling and better support for effective clustering of documents.

Original languageEnglish (US)
Title of host publicationDiscovery Science - 14th International Conference, DS 2011, Proceedings
Pages247-261
Number of pages15
DOIs
StatePublished - Oct 17 2011
Event14th International Conference on Discovery Science, DS 2011, Co-located with the 22nd International Conference on Algorithmic Learning Theory, ALT 2011 - Espoo, Finland
Duration: Oct 5 2011Oct 7 2011

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume6926 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other14th International Conference on Discovery Science, DS 2011, Co-located with the 22nd International Conference on Algorithmic Learning Theory, ALT 2011
CountryFinland
CityEspoo
Period10/5/1110/7/11

Fingerprint Dive into the research topics of 'A statistical model for topically segmented documents'. Together they form a unique fingerprint.

  • Cite this

    Ponti, G., Tagarelli, A., & Karypis, G. (2011). A statistical model for topically segmented documents. In Discovery Science - 14th International Conference, DS 2011, Proceedings (pp. 247-261). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 6926 LNAI). https://doi.org/10.1007/978-3-642-24477-3_21