Abstract: Model building in machine learning is an iterative process. The performance analysis and debugging step typically involves a disruptive cognitive switch from model building to error analysis, discouraging an informed approach to model building. We present ModelTracker, an interactive visualization that subsumes information contained in numerous traditional summary statistics and graphs while displaying example-level performance and enabling direct error examination and debugging. Usage analysis from machine learning practitioners building real models with ModelTracker over six months shows ModelTracker is used often and throughout model building. A controlled experiment focusing on ModelTracker's debugging capabilities shows participants prefer ModelTracker over traditional tools without a loss in model performance.

Abstract: Almost every complex software system today is configurable. While configurability has many benefits, it challenges performance prediction, optimization, and debugging. Often, the influences of individual configuration options on performance are unknown. Worse, configuration options may interact, giving rise to a configuration space of possibly exponential size. Addressing this challenge, we propose an approach that derives a performance-influence model for a given configurable system, describing all relevant influences of configuration options and their interactions. Our approach combines machine-learning and sampling heuristics in a novel way. It improves over standard techniques in that it (1) represents influences of options and their interactions explicitly (which eases debugging), (2) smoothly integrates binary and numeric configuration options for the first time, (3) incorporates domain knowledge, if available (which eases learning and increases accuracy), (4) considers complex constraints among options, and (5) systematically reduces the solution space to a tractable size. A series of experiments demonstrates the feasibility of our approach in terms of the accuracy of the models learned as well as the accuracy of the performance predictions one can make with them.

Abstract: ContextEye-tracking is a mean to collect evidence regarding some participants' cognitive processes. Eye-trackers monitor participants' visual attention by collecting eye-movement data. These data are useful to get insights into participants' cognitive processes during reasoning tasks. ObjectiveThe Evidence-based Software Engineering (EBSE) paradigm has been proposed in 2004 and, since then, has been used to provide detailed insights regarding different topics in software engineering research and practice. Systematic Literature Reviews (SLR) are also useful in the context of EBSE by bringing together all existing evidence of research and results about a particular topic. This SLR evaluates the current state of the art of using eye-trackers in software engineering and provides evidence on the uses and contributions of eye-trackers to empirical studies in software engineering. MethodWe perform a SLR covering eye-tracking studies in software engineering published from 1990 up to the end of 2014. To search all recognised resources, instead of applying manual search, we perform an extensive automated search using Engineering Village. We identify 36 relevant publications, including nine journal papers, two workshop papers, and 25 conference papers. ResultsThe software engineering community started using eye-trackers in the 1990s and they have become increasingly recognised as useful tools to conduct empirical studies from 2006. We observe that researchers use eye-trackers to study model comprehension, code comprehension, debugging, collaborative interaction, and traceability. Moreover, we find that studies use different metrics based on eye-movement data to obtain quantitative measures. We also report the limitations of current eye-tracking technology, which threaten the validity of previous studies, along with suggestions to mitigate these limitations. ConclusionHowever, not withstanding these limitations and threats, we conclude that the advent of new eye-trackers makes the use of these tools easier and less obtrusive and that the software engineering community could benefit more from this technology.

Abstract: Debugging often takes much effort and resources. To help developers debug, numerous information retrieval (IR)-based and spectrum-based bug localization techniques have been proposed. IR-based techniques process textual information in bug reports, while spectrum-based techniques process program spectra (i.e., a record of which program elements are executed for each test case). Both eventually generate a ranked list of program elements that are likely to contain the bug. However, these techniques only consider one source of information, either bug reports or program spectra, which is not optimal. To deal with the limitation of existing techniques, in this work, we propose a new multi-modal technique that considers both bug reports and program spectra to localize bugs. Our approach adaptively creates a bug-specific model to map a particular bug to its possible location, and introduces a novel idea of suspicious words that are highly associated to a bug. We evaluate our approach on 157 real bugs from four software systems, and compare it with a state-of-the-art IR-based bug localization method, a state-of-the-art spectrum-based bug localization method, and three state-of-the-art multi-modal feature location methods that are adapted for bug localization. Experiments show that our approach can outperform the baselines by at least 47.62%, 31.48%, 27.78%, and 28.80% in terms of number of bugs successfully localized when a developer inspects 1, 5, and 10 program elements (i.e., Top 1, Top 5, and Top 10), and Mean Average Precision (MAP) respectively.

Abstract: Software debugging is tedious and time consuming. To reduce the manual effort needed for debugging, researchers have proposed a considerable number of techniques to automate the process of fault localization; in particular, techniques based on information retrieval (IR) have drawn increased attention in recent years. Although reportedly effective, these techniques have some potential limitations that may affect their performance. First, their effectiveness is likely to depend heavily on the quality of the bug reports; unfortunately, high-quality bug reports that contain rich information are not always available. Second, these techniques have not been evaluated through studies that involve actual developers, which is less than ideal, as purely analytical evaluations can hardly show the actual usefulness of debugging techniques. The goal of this work is to evaluate the usefulness of IR-based techniques in real-world scenarios. Our investigation shows that bug reports do not always contain rich information, and that low-quality bug reports can considerably affect the effectiveness of these techniques. Our research also shows, through a user study, that high-quality bug reports benefit developers just as much as they benefit IR-based techniques. In fact, the information provided by IR-based techniques when operating on high-quality reports is only helpful to developers in a limited number of cases. And even in these cases, such information only helps developers get to the faulty file quickly, but does not help them in their most time consuming task: understanding and fixing the bug within that file.

Abstract: Software energy profilers are the tools to measure the energy consumption of mobile devices, applications running on those devices, and various hardware components. They adopt different modeling and measurement techniques. In this article, we aim to review a wide range of such energy profilers for mobile devices. First, we introduce the terminologies and describe the power modeling and measurement methodologies applied in model-based energy profiling. Next, we classify the profilers according to their implementation and deployment strategies, and compare the profiling capabilities and performance between different types. Finally, we point out their limitations and the corresponding challenges.

Abstract: Heterogeneous Computing with OpenCL 2.0 teaches OpenCL and parallel programming for complex systems that may include a variety of device architectures: multi-core CPUs, GPUs, and fully-integrated Accelerated Processing Units (APUs). This fully-revised edition includes the latest enhancements in OpenCL 2.0 including: Shared virtual memory to increase programming flexibility and reduce data transfers that consume resources Dynamic parallelism which reduces processor load and avoids bottlenecks Improved imaging support and integration with OpenGL Designed to work on multiple platforms, OpenCL will help you more effectively program for a heterogeneous future. Written by leaders in the parallel computing and OpenCL communities, this book explores memory spaces, optimization techniques, extensions, debugging and profiling. Multiple case studies and examples illustrate high-performance algorithms, distributing work across heterogeneous systems, embedded domain-specific languages, and will give you hands-on OpenCL experience to address a range of fundamental parallel algorithms. Updated content to cover the latest developments in OpenCL 2.0, including improvements in memory handling, parallelism, and imaging support Explanations of principles and strategies to learn parallel programming with OpenCL, from understanding the abstraction models to thoroughly testing and debugging complete applications Example code covering image analytics, web plugins, particle simulations, video editing, performance optimization, and more

Abstract: The Message Passing Interface (MPI) 3.0 standard, introduced in September 2012, includes a significant update to the one-sided communication interface, also known as remote memory access (RMA). In particular, the interface has been extended to better support popular one-sided and global-address-space parallel programming models to provide better access to hardware performance features and enable new data-access modes. We present the new RMA interface and specify formal axiomatic models for data consistency and access semantics. Such models can help users reason about details of the semantics that are hard to extract from the English prose in the standard. It also fosters the development of tools and compilers, enabling them to automatically analyze, optimize, and debug RMA programs.

Abstract: This paper presents a cross-age study exploring children's performance on robot introductory programming activities with one tangible and one isomorphic graphical system. Both subsystems are parts of an innovative system, namely the PROTEAS kit. The tangible subsystem consists of cube-shaped blocks that represent simple and more advanced programming structures. Users may interconnect the cubic-shaped commands and so create the robot programming code. The graphical subsystem presents onscreen an isomorphic to the tangible programming space. Children (N = 109) of five different aged groups were let to interact in pairs with the two operationally equivalent programming subsystems with the scope to program a NXT Lego robot. Three variables associated with children performance upon tasks and four variables related with performance during free interaction were studied. Data analysis based on computer logs and video recordings showed that children produced fewer errors, made more effective debugging and younger children in particular needed less time to accomplish the robot programming tasks with the tangible subsystem. Moreover, during free interaction, elder children were more engaged, created more complicated programs and explored different commands and parameters more actively in the tangible case. Finally, interpretation of the findings is provided and the advantages of tangible user interfaces in children's introductory programming are discussed.

Abstract: Software crash reproduction is the necessary first step for debugging. Unfortunately, crash reproduction is often labor intensive. To automate crash reproduction, many techniques have been proposed including record-replay and post-failure-process approaches. Record-replay approaches can reliably replay recorded crashes, but they incur substantial performance overhead to program executions. Alternatively, post-failure-process approaches analyse crashes only after they have occurred. Therefore they do not incur performance overhead. However, existing post-failure-process approaches still cannot reproduce many crashes in practice because of scalability issues and the object creation challenge. This paper proposes an automatic crash reproduction framework using collected crash stack traces. The proposed approach combines an efficient backward symbolic execution and a novel method sequence composition approach to generate unit test cases that can reproduce the original crashes without incurring additional runtime overhead. Our evaluation study shows that our approach successfully exploited 31 (59.6 percent) of 52 crashes in three open source projects. Among these exploitable crashes, 22 (42.3 percent) are useful reproductions of the original crashes that reveal the crash triggering bugs. A comparison study also demonstrates that our approach can effectively outperform existing crash reproduction approaches.

Abstract: Programmers maintain and evolve their software in a variety of programming languages to take advantage of various control/data abstractions and legacy libraries. The programming language ecosystem has diversified over the last few decades, and non-trivial programs are likely to be written in more than a single language. Unfortunately, language interfaces such as Java Native Interface and Python/C are difficult to use correctly and the scope of fault localization goes beyond language boundaries, which makes debugging multilingual bugs challenging. To overcome the aforementioned limitations, we propose a mutation-based fault localization technique for real-world multilingual programs. To improve the accuracy of locating multilingual bugs, we have developed and applied new mutation operators as well as conventional mutation operators. The results of the empirical evaluation for six non-trivial real-world multilingual bugs are promising in that the proposed technique identifies the buggy statements as the most suspicious statements for all six bugs.

Abstract: With the rapid proliferation of malware attacks on the Internet, understanding these malicious behaviors plays a critical role in crafting effective defense. Advanced malware analysis relies on virtualization or emulation technology to run samples in a confined environment, and to analyze malicious activities by instrumenting code execution. However, virtual machines and emulators inevitably create artifacts in the execution environment, making these approaches vulnerable to detection or subversion. In this paper, we present MALT, a debugging framework that employs System Management Mode, a CPU mode in the x86 architecture, to transparently study armored malware. MALT does not depend on virtualization or emulation and thus is immune to threats targeting such environments. Our approach reduces the attack surface at the software level, and advances state-of-the-art debugging transparency. MALT embodies various debugging functions, including register/memory accesses, breakpoints, and four stepping modes. We implemented a prototype of MALT on two physical machines, and we conducted experiments by testing an array of existing anti-virtualization, anti-emulation, and packing techniques against MALT. The experimental results show that our prototype remains transparent and undetected against the samples. Furthermore, our prototype of MALT introduces moderate but manageable overheads on both Windows and Linux platforms.

Abstract: Debugging is a crucial yet expensive activity to improve the reliability of software systems. To reduce debugging cost, various fault localization tools have been proposed. A spectrum-based fault localization tool often outputs an ordered list of program elements sorted based on their likelihood to be the root cause of a set of failures (i.e., their suspiciousness scores). Despite the many studies on fault localization, unfortunately, however, for many bugs, the root causes are often low in the ordered list. This potentially causes developers to distrust fault localization tools. Recently, Parnin and Orso highlight in their user study that many debuggers do not find fault localization useful if they do not find the root cause early in the list. To alleviate the above issue, we build an oracle that could predict whether the output of a fault localization tool can be trusted or not. If the output is not likely to be trusted, developers do not need to spend time going through the list of most suspicious program elements one by one. Rather, other conventional means of debugging could be performed. To construct the oracle, we extract the values of a number of features that are potentially related to the effectiveness of fault localization. Building upon advances in machine learning, we process these feature values to learn a discriminative model that is able to predict the effectiveness of a fault localization tool output. In this work, we consider an output of a fault localization tool to be effective if the root cause appears in the top 10 most suspicious program elements. We have evaluated our proposed oracle on 200 faulty versions of Space, NanoXML, XML-Security, and the 7 programs in Siemens test suite. Our experiments demonstrate that we could predict the effectiveness of 9 fault localization tools with a precision, recall, and F-measure (harmonic mean of precision and recall) of up to 74.38 %, 90.00 % and 81.45 %, respectively. The numbers indicate that many ineffective fault localization instances are identified correctly, while only few effective ones are identified wrongly.

Abstract: Energy efficiency is a key concern in the design of modern computer systems. One promising approach to energy-efficient computation, approximate computing, trades off output accuracy for significant gains in energy efficiency. However, debugging the actual cause of output quality problems in approximate programs is challenging. This paper presents dynamic techniques to debug and monitor the quality of approximate computations. We propose both offline debugging tools that instrument code to determine the key sources of output degradation and online approaches that monitor the quality of deployed applications. We present two offline debugging techniques and three online monitoring mechanisms. The first offline tool identifies correlations between output quality and the execution of individual approximate operations. The second tracks approximate operations that flow into a particular value. Our online monitoring mechanisms are complementary approaches designed for detecting quality problems in deployed applications, while still maintaining the energy savings from approximation. We present implementations of our techniques and describe their usage with seven applications. Our online monitors control output quality while still maintaining significant energy efficiency gains, and our offline tools provide new insights into the effects of approximation on output quality.

Abstract: Temporal matching based debugging methods are disclosed. In one embodiment, the method is implemented in a software-defined networking (SDN) controller coupled to a SDN system. The SDN system contains a set of network devices managed by the SDN controller, the method includes recording, at the SDN controller, messages between the SDN controller and a network device. The method further includes obtaining at the SDN controller from the network device, a set of statistics from the network device, where the set of statistics indicates characteristics of packet processing at the network device. The method further includes temporally matching the set of statistics obtained from the network device with the messages recorded, and identifying, among the messages recorded at the SDN controller, one or more messages occurred when the set of statistics indicates a packet processing anomaly in the network device.

Abstract: Various technologies described herein pertain to performing time travel debugging. A computer-executable program can be executed. The computer-executable program can be executable under control of a virtual machine. The virtual machine can interact with a browser system during execution of the computer-executable program. Moreover, nondeterministic events can be logged via an interrogative virtual machine interface (VMI) during the execution of the computer-executable program. The nondeterministic events can be logged as part of event logs. Moreover, the interrogative VMI is between the virtual machine and the browser system. Further, snapshots of the virtual machine can be captured during the execution of the computer-executable program. The snapshots can be captured via the interrogative VMI. At least a portion of the execution of the computer-executable program can be replayed based at least in part on a snapshot and at least a portion of the event logs.

Abstract: Several techniques and tools have been proposed for the automatic generation of test cases. Usually, these tools are evaluated in terms of fault-revealing or coverage capability, but their impact on the manual debugging activity is not considered. The question is whether automatically generated test cases are equally effective in supporting debugging as manually written tests.We conducted a family of three experiments (five replications) with humans (in total, 55 subjects) to assess whether the features of automatically generated test cases, which make them less readable and understandable (e.g., unclear test scenarios, meaningless identifiers), have an impact on the effectiveness and efficiency of debugging. The first two experiments compare different test case generation tools (Randoop vs. EvoSuite). The third experiment investigates the role of code identifiers in test cases (obfuscated vs. original identifiers), since a major difference between manual and automatically generated test cases is that the latter contain meaningless (obfuscated) identifiers.We show that automatically generated test cases are as useful for debugging as manual test cases. Furthermore, we find that, for less experienced developers, automatic tests are more useful on average due to their lower static and dynamic complexity.

Abstract: Side-channel analysis has become a well-established topic in the scientific community and industry over the last one and a half decade. Somewhat surprisingly, the vast majority of work on side-channel analysis has been restricted to the "use case" of attacking cryptographic implementations through the recovery of keys. In this contribution, we show how side-channel analysis can be used for extracting code from embedded systems based on a CPU's electromagnetic emanation. There are many applications within and outside the security community where this is desirable. In cryptography, it can, e.g., be used for recovering proprietary ciphers and security protocols. Another broad application field is general security and reverse engineering, e.g., for detecting IP violations of firmware or for debugging embedded systems when there is no debug interface or it is proprietary.A core feature of our approach is that we take localized electromagnetic measurements that are spatially distributed over the IC being analyzed. Given these multiple inputs, we model code extraction as a classification problem that we solve with supervised learning algorithms. We apply a variant of linear discriminant analysis to distinguish between the multiple classes. In contrast to previous approaches, which reported instruction recognition rates between 40--70%, our approach detects more than 95% of all instructions for test code, and close to 90% for real-world code. The methods are thus very relevant for use in practice. Our method performs dynamic code recognition, which has both advantages (only the program parts that are actually executed are observed) but also limitations (rare code executions are difficult to observe).

Abstract: Renew is a continuously developed extensible Petri net tool, which enables modeling and simulating of various Petri net formalisms One unique characteristic of the tool is the full support for Java reference nets [2], which combine the concepts of nets-within-nets and synchronous channels with a reference semantics using a pattern/instance mechanism analogously to object oriented programming languages Furthermore Java can be used as inscription language whereby the formalism is well-suited for the implementation of concurrent software systems Since Renew is written in Java it is available for multiple platforms (including Windows, Linux and Mac) The current version 242 is available for download free of charge including the source code [3] Renew provides an easy to use graphical editor for Petri net models and other types of models and a simulation engine, which is seamlessly integrated into this editor It has a plug-in architecture, which makes it easily extensible The core plug-ins are provided as part of the Renew distribution Many advanced features are supplied by optional plug-ins The editor has been improved over the last years and received many small usability enhancements and has evolved into an integrated development environment (IDE) for net based software development It contains a syntax check during editing and debugging tools, such as breakpoints or manual transitions Furthermore the editor features desktop integration, a file navigator and image export to various formats The simulator is capable of handling different formalisms The main formalism is the Java reference net formalism, for which different extensions exist, such as inhibitor, reset and timed arcs The workflow net formalism, provided by an optional plug-in, adds a task transition, which can be canceled during execution, so that its effect on the net can be reverted Other formalisms provide simulation of P/T nets, feature structure nets and bool nets Simulation is available in different modes In the interactive simulation mode the user may control the simulation by choosing the transitions to fire and inspect each single step The automatic simulation mode is usable for system execution and can be run with and without graphical feedback Renew features dynamic loading of nets on demand and configurable logging of simulation events The monitoring plugins facilitate the inspection of remote simulations With an integration of the LoLA verification tool [1] Renew is also suited for verification tasks during modeling The first official version of Renew was released in 1999 and has since then been continuously developed as a Petri net editing and simulation environment The plugin system, introduced with the major release 20 in 2004 [4], enabled

Abstract: High-level synthesis (HLS) for FPGA designs has received considerable attention in recent years. To make this design methodology mainstream, improved debugging technologies are essential. Ideally, a user should be able to debug their design using the original source code, without detailed knowledge of the underlying hardware, while the circuit executes in-situ. Although recent work has made progress toward this goal, existing solutions are unable to provide visibility into circuits that have been heavily optimized by the compiler. HLS compilers typically perform many optimizations, including moving variable values out of memories and into registers distributed throughout the design. Debugging such circuits typically requires either understanding the hardware and probing the appropriate RTL level registers, or ignoring these variables while debugging the design, neither of which is desirable. In this work we present a new signal-tracing technique, specifically designed for circuits that have been optimized by an HLS tool. Information is extracted from the HLS process to determine which signals are relevant to record each cycle. We automatically embed circuitry which dynamically selects the relevant signals, cycle-by-cycle, and records them into on-chip memories. In addition, we explore techniques to balance tracing between cycles to further improve memory efficiency. For each 100Kb of memory allocated to trace buffers, our technique can, on average, record and replay 4322 lines of source code, versus 141 lines using traditional tracing methods.

Abstract: Spreadsheets are by far the most prominent example of end-user programs of ample size and substantial structural complexity. They are usually not thoroughly tested so they often contain faults. Debugging spreadsheets is a hard task due to the size and structure, which is usually not directly visible to the user, i.e., the functions are hidden and only the computed values are presented. A way to locate faulty cells in spreadsheets is by adapting software debugging approaches for traditional procedural or object-oriented programming languages. One of such approaches is spectrum-based fault localization (Sfl). In this paper, we study the impact of different similarity coefficients on the accuracy of Sfl applied to the spreadsheet domain. Our empirical evaluation shows that three of the 42 studied coefficients (Ochiai, Jaccard and Sorensen-Dice) require less effort by the user while inspecting the diagnostic report, and can also be used interchangeably without a loss of accuracy. In addition, we illustrate the influence of the number of correct and incorrect output cells on the diagnostic report.

Abstract: Rust, a new systems programming language, provides compile-time memory safety checks to help eliminate runtime bugs that manifest from improper memory management. This feature is advantageous for operating system development, and especially for embedded OS development, where recovery and debugging are particularly challenging. However, embedded platforms are highly event-based, and Rust's memory safety mechanisms largely presume threads. In our experience developing an operating system for embedded systems in Rust, we have found that Rust's ownership model prevents otherwise safe resource sharing common in the embedded domain, conflicts with the reality of hardware resources, and hinders using closures for programming asynchronously. We describe these experiences and how they relate to memory safety as well as illustrate our workarounds that preserve the safety guarantees to the largest extent possible. In addition, we draw from our experience to propose a new language extension to Rust that would enable it to provide better memory safety tools for event-driven platforms.

Abstract: UNICORN is an automated dynamic pattern-detection-based technique that finds and ranks problematic memory access patterns for non-deadlock concurrency bugs. It monitors pairs of memory accesses, combines the pairs into problematic patterns and ranks the patterns by their suspiciousness scores. It detects significant classes of bug types, including order violations and both single-variable and multivariable atomicity violations, which have been shown to be the most important classes of non-deadlock concurrency bugs. This paper describes the UNICORN approach, its implementations in Java and C++, and evaluates these implementations empirically. The evaluation shows that UNICORN can effectively compute and rank the patterns that represent concurrency bugs, and perform computation and ranking with reasonable efficiency. Copyright © 2014 John Wiley & Sons, Ltd.

Abstract: Omniscient debugging is a promising technique that relies on execution traces to enable free traversal of the states reached by a system during an execution. While some General-Purpose Languages (GPLs) already have support for omniscient debugging, developing such a complex tool for any executable Domain-Specific Modeling Language (xDSML) remains a challenging and error prone task. A solution to this problem is to define a generic omniscient debugger for all xDSMLs. However, generically supporting any xDSML both compromises the efficiency and the usability of such an approach. Our contribution relies on a partly generic omniscient debugger supported by generated domain-specific trace management facilities. Being domain-specific, these facilities are tuned to the considered xDSML for better efficiency. Usability is strengthened by providing multidimensional omniscient debugging. Results show that our approach is on average 3.0 times more efficient in memory and 5.03 more efficient in time when compared to a generic solution that copies the model at each step.

Abstract: IEEE 1687-2014 Standard provides an effective method for accessing on-chip instruments for testing, debugging and board configuration. The standard, however, causes a safety problem because anyone can access the chip instruments, set inputs and obtain safety critical information. In recent work, a lock in the segment insertion bit (SIB) and a corresponding unlocking key application procedure have been proposed for securing the 1687. This paper provides a linear feedback shift register (LFSR) based key generation mechanism that enhances the security of 1687 very significantly. By reconfiguring m (a small number) scan flip-flops into an LFSR that generates the key to unlock the SIB, we show a substantial increase in the expected break-in time.

Abstract: Spreadsheets can be viewed as a highly flexible programming environment for end users. Spreadsheets are widely adopted for decision making and may have a serious economical impact for the business. However, spreadsheets are staggeringly prone to errors. Hence, approaches for aiding the process of pinpointing the faulty cells in a spreadsheet are of great value. We present a constraint-based approach, ConBug, for debugging spreadsheets. The approach takes as input a (faulty) spreadsheet and a test case that reveals the fault and computes a set of diagnosis candidates for the debugging problem. Therefore, we convert the spreadsheet and a test case to a constraint satisfaction problem. We perform an empirical evaluation with 78 spreadsheets from different sources, where we demonstrate that our approach is light-weight and efficient. From our experimental results, we conclude that ConBug helps end users to pinpoint faulty cells.