Abstract
Repeating experiments is an important instrument in the scientific toolbox to validate previous work and build upon existing work. We present two concrete use cases involving key techniques in the NLP domain for which we show that reproducing results is still difficult. We show that the deviation that can be found in reproduction efforts leads to questions about how our results should be interpreted. Moreover, investigating these deviations provides new insights and a deeper understanding of the examined techniques. We identify five aspects that can influence the outcomes of experiments that are typically not addressed in research papers. Our use cases show that these aspects may change the answer to research questions leading us to conclude that more care should be taken in interpreting our results and more research involving systematic testing of methods is required in our field.
| Original language | English (US) |
|---|---|
| Title of host publication | Long Papers |
| Publisher | Association for Computational Linguistics (ACL) |
| Pages | 1691-1701 |
| Number of pages | 11 |
| ISBN (Print) | 9781937284503 |
| State | Published - 2013 |
| Externally published | Yes |
| Event | 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013 - Sofia, Bulgaria Duration: Aug 4 2013 → Aug 9 2013 |
Publication series
| Name | ACL 2013 - 51st Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference |
|---|---|
| Volume | 1 |
Other
| Other | 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013 |
|---|---|
| Country/Territory | Bulgaria |
| City | Sofia |
| Period | 8/4/13 → 8/9/13 |
Bibliographical note
Funding Information:Este art?culo se ha elaborado a partir de la ayuda recibida por el Programa de millora i innovaci? en la formaci? de mestres MOB-MIF. Agencia de Gesti?n de Ayudas Universitarias y de Investigaci?n (AGAUR). Resoluci?n ECO/1972/2014, de 28 de julio, (Ref., MOB-MIF 2014-00008).