Code summarization, the task of generating useful comments given the code, has long been of interest. Most of the existing code summarization models are trained and validated on widely-used code comment benchmark datasets. However, little is known about the quality of the benchmark datasets built from real-world projects. Are the benchmark datasets as good as expected? To bridge the gap, we conduct a systematic research to assess and improve the quality of four benchmark datasets widely used for code summarization tasks. First, we propose an automated code-comment cleaning tool that can accurately detect noisy data caused by inappropriate data preprocessing operations from existing benchmark datasets. Then, we apply the tool to further assess the data quality of the four benchmark datasets, based on the detected noises. Finally, we conduct comparative experiments to investigate the impact of noisy data on the performance of code summarization models. The results show that these data preprocessing noises widely exist in all four benchmark datasets, and removing these noisy data leads to a significant improvement on the performance of code summarization. We believe that the findings and insights will enable a better understanding of data quality in code summarization tasks, and pave the way for relevant research and practice.

/pdf/are-we-building-on-the-rock-on-the-importance-of-data-208mz0u3.pdf

Are we building on the rock? on the importance of data preprocessing for code summarization

Provenance variations in kernel oil content, fatty acid profile and biodiesel properties of Xanthoceras sorbifolium Bunge in northern China

As a fundamental component of software documentation, code comments could help developers comprehend and maintain programs. Several datasets of method header comments have been proposed in previous studies for machine learning-based code comment generation. As part of code comments, inline code comments are also crucial for code understanding activities. However, unlike method header comments written in a standard format and describing the whole method code, inline comments are often written in arbitrary formats by developers due to timelines pressures and different aspects of code snippets in the method are described. Currently, there is no large-scale dataset used for inline comments generation considering these. Hence, this naturally inspires us to explore whether we can construct a dataset to foster machine learning research that not only performs fine-grained noise-cleaning but conducts a taxonomy of inline comments. To this end, we first collect inline comments and code snippets from 8000 Java projects on GitHub. Then, we conduct a manual review to obtain heuristic rules, which could be used to clean the data noise in a fine-grained manner. As a result, we construct a large-scale benchmark dataset named ICG with 5,740,770 pairs of inline comments and code snippets. We then build a comprehensive taxonomy and conduct a statistical and manual analysis to explore the performances of different categories of inline comments, such as helpfulness in code understanding. After that, we provide and compare several baseline models to automatically generate inline comments, such as CodeBERT, to enhance the usability of the benchmark for researchers. The availability of our benchmark and baselines can help develop and validate new inline comment generation methods, which would also further facilitate code understanding activities. 

ICG: A Machine Learning Benchmark Dataset and Baselines for Inline Code Comments Generation Task

Deep learning-based code processing models have shown good performance for tasks such as predicting method names, summarizing programs, and comment generation. However, despite the tremendous progress, deep learning models are often prone to adversarial attacks, which can significantly threaten the robustness and generalizability of these models by leading them to misclassification with unexpected inputs. To address the above issue, many deep learning testing approaches have been proposed, however, these approaches mainly focus on testing deep learning applications in the domains of image, audio, and text analysis, etc., which cannot be directly applied to neural models for code due to the unique properties of programs. In this paper, we propose a coverage-based fuzzing framework, CoCoFuzzing, for testing deep learning-based code processing models. In particular, we first propose ten mutation operators to automatically generate valid and semantically preserving source code examples as tests; then we propose a neuron coverage-based approach to guide the generation of tests. We investigate the performance of CoCoFuzzing on three state-of-the-art neural code models, i.e., NeuralCodeSum, CODE2SEQ, and CODE2VEC. Our experiment results demonstrate that CoCoFuzzing can generate valid and semantically preserving source code examples for testing the robustness and generalizability of these models and improve the neuron coverage. Moreover, these tests can be used to improve the performance of the target neural code models through adversarial retraining. 

CoCoFuzzing: Testing Neural Code Models With Coverage-Guided Fuzzing

Abstract Source code summarization (SCS) refers to the natural language description on what a source code is performing. It can help developers understand programs and maintain software efficiently. Retrieval methods (RM) generate SCS reorganize terms selected from source code or use SCS of similar code snippets. Generative methods (GM) generate SCS via attentional encoder-decoder architecture. However, a GM can generate SCS for any code, but sometime the accuracy is still far from expectation (due to the lack of numerous high-quality training set). A RM is considered with a higher accuracy, but usually fail to generate SCS for a source code in the absence of a similar candidate in the database. In order to effectively combine the advantages of RM and GM, we propose a new method Re_Trans. For a given code, we first utilize RM to get the most similar code w.r.t sematic and its SCS (S_RM). Then, we input the given code and similar code into the trained discriminator. If the discriminator outputs 1, we take S_RM as the result, otherwise, we utilize the GM, transformer, to generate the given code’ SCS. Particularly, we use AST-augmented and code sequence-augmented information to make the source code semantic extraction more fully. Furthermore, we build a new SCS retrieval library through the public dataset. We evaluate our method on a data set of 2.1 million Java code-comment pairs, and experimental results show improvement over the state-of-the-art (SOTA) benchmarks, which demonstrates the effectiveness and efficiency of our method. 

Re_Trans: Combined Retrieval and transformer model for Source Code Summarization

Code comment generation techniques aim to generate natural language
descriptions for source code. There are two orthogonal approaches for this
task, i.e., information retrieval (IR) based and neural-based methods. Recent
studies have focused on combining their strengths by feeding the input code and
its similar code snippets retrieved by the IR-based approach to the
neural-based approach, which can enhance the neural-based approach's ability to
output low-frequency words and further improve the performance. However,
despite the tremendous progress, our pilot study reveals that the current
combination is not generalizable and can lead to performance degradation. In
this paper, we propose a straightforward but effective approach to tackle the
issue of existing combinations of these two comment generation approaches.
Instead of binding IR- and neural-based approaches statically, we combine them
in a dynamic manner. Specifically, given an input code snippet, we first use an
IR-based technique to retrieve a similar code snippet from the corpus. Then we
use a Cross-Encoder based classifier to decide the comment generation method to
be used dynamically, i.e., if the retrieved similar code snippet is a true
positive (i.e., is semantically similar to the input), we directly use the
IR-based technique. Otherwise, we pass the input to the neural-based model to
generate the comment. We evaluate our approach on a large-scale dataset of Java
projects. Experiment results show that our approach can achieve 25.45 BLEU
score, which improves the state-of-the-art IR-based approach, neural-based
approach, and their combination by 41%, 26%, and 7%, respectively. We propose a
straightforward but effective dynamic combination of IR-based and neural-based
comment generation, which outperforms state-of-the-art approaches by a
substantial margin.

Yet Another Combination of IR- and Neural-based Comment Generation

Probability distribution and selection of seed and fruit traits of different fruit types of Xanthoceras sorbifolium.

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of {limit} columns in the table. Remove at least 1 column to add or create another one.

Wang Qing

Author Tools

Chat about Author

Papers

Yet Another Combination of IR- and Neural-based Comment Generation

Probability distribution and selection of seed and fruit traits of different fruit types of Xanthoceras sorbifolium.