Abstract
Learning from source code usually requires a large amount of labeled data. Despite the possible scarcity of labeled data, the trained model is highly task-specific and lacks transferability to different tasks. In this work, we present effective pre-training strategies on top of a novel graph-based code representation, to produce universal representations for code. Specifically, our graph-based representation captures important semantics between code elements (e.g., control flow and data flow). We pre-train graph neural networks on the representation to extract universal code properties. The pre-trained model then enables the possibility of fine-tuning to support various downstream applications. We evaluate our model on two real-world datasets – spanning over 30M Java methods and 770K Python methods. Through visualization, we reveal discriminative properties in our universal code representation. By comparing multiple benchmarks, we demonstrate that the proposed framework achieves state-of-the-art results on method name prediction and code graph link prediction.
Original language | English (US) |
---|---|
Title of host publication | Advances in Knowledge Discovery and Data Mining - 25th Pacific-Asia Conference, PAKDD 2021, Proceedings |
Editors | Kamal Karlapalem, Hong Cheng, Naren Ramakrishnan, R. K. Agrawal, P. Krishna Reddy, Jaideep Srivastava, Tanmoy Chakraborty |
Publisher | Springer Science and Business Media Deutschland GmbH |
Pages | 16-28 |
Number of pages | 13 |
ISBN (Print) | 9783030757670 |
DOIs | |
State | Published - May 8 2021 |
Externally published | Yes |
Event | 25th Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2021 - Virtual, Online Duration: May 11 2021 → May 14 2021 |
Publication series
Name | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
---|---|
Volume | 12714 LNAI |
ISSN (Print) | 0302-9743 |
ISSN (Electronic) | 1611-3349 |
Conference
Conference | 25th Pacific-Asia Conference on Knowledge Discovery and Data Mining, PAKDD 2021 |
---|---|
City | Virtual, Online |
Period | 5/11/21 → 5/14/21 |
Bibliographical note
Publisher Copyright:© 2021, Springer Nature Switzerland AG.
Keywords
- Code representation
- Graph neural network
- Pre-training