Mini-track: empirical software evaluation

TL;DR: This minitrack focuses on bringing together the research work from various R&D groups around the world on empirical software evaluation, and demonstrates that by applying scientific approaches to designing repeatable experiments, significant insights in to the technologies can be obtained.

Abstract: In all areas of software technology, competing products battle to gain market share and dominance. While this situation promotes innovation in technologies and products, it creates a confusing landscape for IT organizations faced with selecting an appropriate technology that satisfies their requirements. It is well documented that the evaluation phase of software product procurement is typically expensive and time-consuming for organizations. It is also fraught with danger, as the selection of an inappropriate tool or technology will be at best a waste of money (shelfware), and in many cases it will lead to ultimate project and even business failure. The co-chairs of this minitrack have been working for 5 years to develop methods and techniques for empirically evaluating the strengths and weaknesses of various middleware and software infrastructure technologies. The work has had significant impact both in research and practice [1-3]. It has demonstrated that by applying scientific approaches to designing repeatable experiments, significant insights in to the technologies can be obtained. The results are of even greater value to potential adopters as they are repeatable and indisputable, rendering marketing messages ineffective in the face of cold hard facts. This minitrack focuses on bringing together the research work from various R&D groups around the world on empirical software evaluation. Devising empirical methods that can meaningfully differentiate between competing products in a wide range of technology classes is a new and growing research area, and one which has the potential to be of great importance to the IT industry as a whole.