Distributed representation of multi-sense words: A loss driven approach

Saurav Manchanda; George Karypis

doi:10.1007/978-3-319-93037-4_27

Distributed representation of multi-sense words: A loss driven approach

Saurav Manchanda, George Karypis

Computer Science and Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

1 Scopus citations

Abstract

Word2Vec’s Skip Gram model is the current state-of-the-art approach for estimating the distributed representation of words. However, it assumes a single vector per word, which is not well-suited for representing words that have multiple senses. This work presents LDMI, a new model for estimating distributional representations of words. LDMI relies on the idea that, if a word carries multiple senses, then having a different representation for each of its senses should lead to a lower loss associated with predicting its co-occurring words, as opposed to the case when a single vector representation is used for all the senses. After identifying the multi-sense words, LDMI clusters the occurrences of these words to assign a sense to each occurrence. Experiments on the contextual word similarity task show that LDMI leads to better performance than competing approaches.

Original language	English (US)
Title of host publication	Advances in Knowledge Discovery and Data Mining - 22nd Pacific-Asia Conference, PAKDD 2018, Proceedings
Editors	Bao Ho, Dinh Phung, Geoffrey I. Webb, Vincent S. Tseng, Mohadeseh Ganji, Lida Rashidi
Publisher	Springer Verlag
Pages	337-349
Number of pages	13
ISBN (Print)	9783319930367
DOIs	https://doi.org/10.1007/978-3-319-93037-4_27
State	Published - 2018
Event	22nd Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining, PAKDD 2018 - Melbourne, Australia Duration: Jun 3 2018 → Jun 6 2018

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	10938 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Other

Other	22nd Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining, PAKDD 2018
Country/Territory	Australia
City	Melbourne
Period	6/3/18 → 6/6/18

Bibliographical note

Funding Information:
Acknowledgments. This work was supported in part by NSF (IIS-1247632, IIP-1414153, IIS-1447788, IIS-1704074, CNS-1757916), Army Research Office (W911NF-14-1-0316), Intel Software and Services Group, and the Digital Technology Center at the University of Minnesota. Access to research and computing facilities was provided by the Digital Technology Center and the Minnesota Supercomputing Institute.

Publisher Copyright:
© 2018, Springer International Publishing AG, part of Springer Nature.

Publisher link

10.1007/978-3-319-93037-4_27

Find It

Find It at U of M Libraries

Cite this

Manchanda, S., & Karypis, G. (2018). Distributed representation of multi-sense words: A loss driven approach. In B. Ho, D. Phung, G. I. Webb, V. S. Tseng, M. Ganji, & L. Rashidi (Eds.), Advances in Knowledge Discovery and Data Mining - 22nd Pacific-Asia Conference, PAKDD 2018, Proceedings (pp. 337-349). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10938 LNAI). Springer Verlag. https://doi.org/10.1007/978-3-319-93037-4_27

Distributed representation of multi-sense words : A loss driven approach. / Manchanda, Saurav; Karypis, George.

Advances in Knowledge Discovery and Data Mining - 22nd Pacific-Asia Conference, PAKDD 2018, Proceedings. ed. / Bao Ho; Dinh Phung; Geoffrey I. Webb; Vincent S. Tseng; Mohadeseh Ganji; Lida Rashidi. Springer Verlag, 2018. p. 337-349 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10938 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Manchanda, S & Karypis, G 2018, Distributed representation of multi-sense words: A loss driven approach. in B Ho, D Phung, GI Webb, VS Tseng, M Ganji & L Rashidi (eds), Advances in Knowledge Discovery and Data Mining - 22nd Pacific-Asia Conference, PAKDD 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10938 LNAI, Springer Verlag, pp. 337-349, 22nd Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining, PAKDD 2018, Melbourne, Australia, 6/3/18. https://doi.org/10.1007/978-3-319-93037-4_27

Manchanda S, Karypis G. Distributed representation of multi-sense words: A loss driven approach. In Ho B, Phung D, Webb GI, Tseng VS, Ganji M, Rashidi L, editors, Advances in Knowledge Discovery and Data Mining - 22nd Pacific-Asia Conference, PAKDD 2018, Proceedings. Springer Verlag. 2018. p. 337-349. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-93037-4_27

Manchanda, Saurav ; Karypis, George. / Distributed representation of multi-sense words : A loss driven approach. Advances in Knowledge Discovery and Data Mining - 22nd Pacific-Asia Conference, PAKDD 2018, Proceedings. editor / Bao Ho ; Dinh Phung ; Geoffrey I. Webb ; Vincent S. Tseng ; Mohadeseh Ganji ; Lida Rashidi. Springer Verlag, 2018. pp. 337-349 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{f5e2748c84d644ddb98521b65bc6435c,

title = "Distributed representation of multi-sense words: A loss driven approach",

abstract = "Word2Vec{\textquoteright}s Skip Gram model is the current state-of-the-art approach for estimating the distributed representation of words. However, it assumes a single vector per word, which is not well-suited for representing words that have multiple senses. This work presents LDMI, a new model for estimating distributional representations of words. LDMI relies on the idea that, if a word carries multiple senses, then having a different representation for each of its senses should lead to a lower loss associated with predicting its co-occurring words, as opposed to the case when a single vector representation is used for all the senses. After identifying the multi-sense words, LDMI clusters the occurrences of these words to assign a sense to each occurrence. Experiments on the contextual word similarity task show that LDMI leads to better performance than competing approaches.",

author = "Saurav Manchanda and George Karypis",

note = "Funding Information: Acknowledgments. This work was supported in part by NSF (IIS-1247632, IIP-1414153, IIS-1447788, IIS-1704074, CNS-1757916), Army Research Office (W911NF-14-1-0316), Intel Software and Services Group, and the Digital Technology Center at the University of Minnesota. Access to research and computing facilities was provided by the Digital Technology Center and the Minnesota Supercomputing Institute. Publisher Copyright: {\textcopyright} 2018, Springer International Publishing AG, part of Springer Nature.; 22nd Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining, PAKDD 2018 ; Conference date: 03-06-2018 Through 06-06-2018",

year = "2018",

doi = "10.1007/978-3-319-93037-4_27",

language = "English (US)",

isbn = "9783319930367",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "337--349",

editor = "Bao Ho and Dinh Phung and Webb, {Geoffrey I.} and Tseng, {Vincent S.} and Mohadeseh Ganji and Lida Rashidi",

booktitle = "Advances in Knowledge Discovery and Data Mining - 22nd Pacific-Asia Conference, PAKDD 2018, Proceedings",

}

TY - GEN

T1 - Distributed representation of multi-sense words

T2 - 22nd Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining, PAKDD 2018

AU - Manchanda, Saurav

AU - Karypis, George

N1 - Funding Information: Acknowledgments. This work was supported in part by NSF (IIS-1247632, IIP-1414153, IIS-1447788, IIS-1704074, CNS-1757916), Army Research Office (W911NF-14-1-0316), Intel Software and Services Group, and the Digital Technology Center at the University of Minnesota. Access to research and computing facilities was provided by the Digital Technology Center and the Minnesota Supercomputing Institute. Publisher Copyright: © 2018, Springer International Publishing AG, part of Springer Nature.

PY - 2018

Y1 - 2018

N2 - Word2Vec’s Skip Gram model is the current state-of-the-art approach for estimating the distributed representation of words. However, it assumes a single vector per word, which is not well-suited for representing words that have multiple senses. This work presents LDMI, a new model for estimating distributional representations of words. LDMI relies on the idea that, if a word carries multiple senses, then having a different representation for each of its senses should lead to a lower loss associated with predicting its co-occurring words, as opposed to the case when a single vector representation is used for all the senses. After identifying the multi-sense words, LDMI clusters the occurrences of these words to assign a sense to each occurrence. Experiments on the contextual word similarity task show that LDMI leads to better performance than competing approaches.

AB - Word2Vec’s Skip Gram model is the current state-of-the-art approach for estimating the distributed representation of words. However, it assumes a single vector per word, which is not well-suited for representing words that have multiple senses. This work presents LDMI, a new model for estimating distributional representations of words. LDMI relies on the idea that, if a word carries multiple senses, then having a different representation for each of its senses should lead to a lower loss associated with predicting its co-occurring words, as opposed to the case when a single vector representation is used for all the senses. After identifying the multi-sense words, LDMI clusters the occurrences of these words to assign a sense to each occurrence. Experiments on the contextual word similarity task show that LDMI leads to better performance than competing approaches.

UR - http://www.scopus.com/inward/record.url?scp=85049375813&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85049375813&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-93037-4_27

DO - 10.1007/978-3-319-93037-4_27

M3 - Conference contribution

AN - SCOPUS:85049375813

SN - 9783319930367

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 337

EP - 349

BT - Advances in Knowledge Discovery and Data Mining - 22nd Pacific-Asia Conference, PAKDD 2018, Proceedings

A2 - Ho, Bao

A2 - Phung, Dinh

A2 - Webb, Geoffrey I.

A2 - Tseng, Vincent S.

A2 - Ganji, Mohadeseh

A2 - Rashidi, Lida

PB - Springer Verlag

Y2 - 3 June 2018 through 6 June 2018

ER -

Distributed representation of multi-sense words: A loss driven approach

Abstract

Publication series

Other

Bibliographical note

Publisher link

Find It

Other files and links

Fingerprint

Cite this