Open AccessPosted Content
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search.
TL;DR: The methodology used to obtain the corpus and expert labels, as well as a number of simple baseline solutions for the task are described.
read more
Abstract: Semantic code search is the task of retrieving relevant code given a natural language query. While related to other information retrieval tasks, it requires bridging the gap between the language used in code (often abbreviated and highly technical) and natural language more suitable to describe vague concepts and ideas.
To enable evaluation of progress on code search, we are releasing the CodeSearchNet Corpus and are presenting the CodeSearchNet Challenge, which consists of 99 natural language queries with about 4k expert relevance annotations of likely results from CodeSearchNet Corpus. The corpus contains about 6 million functions from open-source code spanning six programming languages (Go, Java, JavaScript, PHP, Python, and Ruby). The CodeSearchNet Corpus also contains automatically generated query-like natural language for 2 million functions, obtained from mechanically scraping and preprocessing associated function documentation. In this article, we describe the methodology used to obtain the corpus and expert labels, as well as a number of simple baseline solutions for the task.
We hope that CodeSearchNet Challenge encourages researchers and practitioners to study this interesting task further and will host a competition and leaderboard to track the progress on the challenge. We are also keen on extending CodeSearchNet Challenge to more queries and programming languages in the future.
read more
Chat with Paper
AI Agents for this Paper
Find similar papers on Google Scholar, PubMed and Arxiv
Write a critical review of this paper
Analyze citations of this paper to find unaddressed research gaps
Citations
BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval
Hongjin Su,Howard Yen,Mengzhou Xia,Weijia Shi,Niklas Muennighoff,Han-yu Wang,Haisu Liu,Quan Shi,Zachary S. Siegel,Michael Tang,Ruoxi Sun,Jinsung Yoon,Sercan O. Arik,Danqi Chen,Tao Yu +14 more
TL;DR: BRIGHT, a novel text retrieval benchmark, evaluates models' ability to perform intensive reasoning on diverse real-world queries, revealing poor performance of state-of-the-art models and improved results with Chain-of-Thought reasoning augmentation.
7
Cross-Modal Contrastive Learning for Code Search
Zejian Shi,Yun Xiong,Xiaolong Zhang,Yao Zhang,Shanshan Li,Yangyong Zhu +5 more
- 01 Oct 2022
TL;DR: Xia et al. as mentioned in this paper proposed a cross-modal contrastive learning method for code search, which considers not only the similarity between modalities, but also the similarity within modalities.
6
SimSCOOD: Systematic Analysis of Out-of-Distribution Behavior of Source Code Models
TL;DR: This work contributes the first systematic approach that simulates various OOD scenarios along different dimensions of data properties and investigates the model behaviors in such scenarios and provides insights and sheds light for future research in terms of generalization, ro-bustness, and inductive biases of source code models.
6
•Posted Content
Contrastive Learning for Source Code with Structural and Functional Properties
Yangruibo Ding,Luca Buratti,Saikat Chakraborty,Saurabh Pujar,Alessandro Morari,Baishakhi Ray +5 more
TL;DR: BOOST as mentioned in this paper is a self-supervised model to focus pre-training based on the characteristics of source code, which can generate functionally equivalent code and textually and syntactically very similar code.
Data journeys: Explaining AI workflows through abstraction
Enrico Daga,P. Groth +1 more
TL;DR: In this article , the authors propose an ontology to capture the essential elements of a data journey and an approach to extract such data journeys using a corpus of Python notebooks from Kaggle.
References
•Proceedings Article
Attention is All you Need
Ashish Vaswani,Noam Shazeer,Niki Parmar,Jakob Uszkoreit,Llion Jones,Aidan N. Gomez,Lukasz Kaiser,Illia Polosukhin +7 more
- 12 Jun 2017
TL;DR: This paper proposed a simple network architecture based solely on an attention mechanism, dispensing with recurrence and convolutions entirely and achieved state-of-the-art performance on English-to-French translation.
•Posted Content
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
TL;DR: A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
81.7K
Convolutional Neural Networks for Sentence Classification
Yoon Kim
- 25 Aug 2014
TL;DR: The CNN models discussed herein improve upon the state of the art on 4 out of 7 tasks, which include sentiment analysis and question classification, and are proposed to allow for the use of both task-specific and static vectors.
•Book
Introduction to Information Retrieval
Christopher D. Manning,Prabhakar Raghavan,Hinrich Schütze +2 more
- 01 Jan 2008
TL;DR: In this article, the authors present an up-to-date treatment of all aspects of the design and implementation of systems for gathering, indexing, and searching documents; methods for evaluating systems; and an introduction to the use of machine learning methods on text collections.
Neural Machine Translation of Rare Words with Subword Units
Rico Sennrich,Barry Haddow,Alexandra Birch +2 more
- 12 Aug 2016
TL;DR: This paper introduces a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units, and empirically shows that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English-German and English-Russian by 1.3 BLEU.