Behaviour driven development: A systematic mapping study

Question

1. What is Behaviour Driven Development (BDD)?

2. How many papers were included after individual screening?

3. What classification scheme was used for research type?

4. What are the major themes in BDD research?

Accepted Answer

Behaviour Driven Development (BDD) is an agile technique where software requirements are specified in a semi-structured natural language using Given-When-Then to express examples of expected software behaviour. Scenarios act as test cases to verify software behaviour. Glue code connects scenarios to production code, enabling scenario execution to identify correct and problematic parts of the System Under Test (SUT). BDD has been studied in various domains and is an established industry practice. However, systematic mapping studies and literature reviews on BDD are limited, creating a need to identify research gaps and inform future studies.

Accepted Answer

After individual screening, 147 papers were included. This number also consists of the papers included during the pilot screening. To ensure no valuable papers were missed, forward and backward snowballing on all 147 included papers led to an addition of 19 papers not captured by the initial search. The screening process involved three phases, with the first phase removing duplicates and non-English papers, resulting in 600 papers. The second phase used Abstrackr for collaborative review, with a pilot phase and consensus on 14 papers. The final set of included papers was determined through individual screening and subsequent discussions.

Accepted Answer

The classification scheme used for research type was adopted by Wieringa et.al. [10] and recommended by Petersen et al. [7]. It consists of six types of research: validation research, evaluation research, solution proposal, philosophical papers, opinion papers, and experience papers. This classification scheme helps categorize papers based on the type of research conducted in each study.

Accepted Answer

The major themes in BDD research, as identified in the systematic mapping, are presented in Figure 8. These themes are categorized using sub-themes, which provide more specific insights. The themes and sub-themes are summarized in references [111, 112, 30, 113, 114]. The research highlights that most studies have utilized example applications or case studies to demonstrate the applicability of solutions or insights generated in BDD research. This approach helps in understanding the practical implications and effectiveness of BDD methodologies in real-world scenarios.

Accepted Answer

BDD has been applied in various settings, including the development of multi-agent systems, systems with hardware components, software systems for the health sector, teaching software engineering, computer networking, product configuration systems, mobile applications, and enterprise systems. Studies have shown its effectiveness in these diverse contexts, contributing to advancements in technology and software development. The versatility of BDD allows it to be utilized in different domains, making it a valuable tool for researchers and practitioners alike. By exploring the application of BDD in these settings, researchers can gain insights into its potential benefits and limitations, leading to further improvements and innovations in the field.

Accepted Answer

Ontologies aid BDD scenarios by formalizing BDD vocabularies used in user stories, scenarios, and GUI interactions. They enable the creation of model-driven tools for modelling BDD scenarios and evaluating their consistency. Ontologies also reduce ambiguity in defining BDD scenarios using natural language, facilitating consistency checking between system functionality and business-related tasks. Studies have shown the use of ontologies in the higher education and business information systems domains, enhancing BDD scenarios' effectiveness and reliability.

Accepted Answer

The publication frequency trend for BDD shows an increase after its introduction in 2006, with a peak in 2019. From 2016 to 2021, there was a notable increase in the number of BDD papers. The research focuses on solution proposals, validation research, and the lack of models and metrics in conducted validation research. Additionally, there is a scarcity of BDD research evaluated in industry settings and philosophical/experience papers for all types of contributions.

Accepted Answer

Most BDD studies have concentrated on creating tools, processes, methods, empirical insights, and models to enhance software engineering processes. These contributions aim to improve the BDD process and the resulting artifacts. Empirical insights and tools are the primary contributions in research on improving BDD. Studies on applying BDD in various settings have mainly contributed processes, empirical insights, and tools, although to a lesser extent. There is a significant gap in the availability of models, metrics, and methods to guide the use of BDD in different settings. Existing BDD research often includes ontologies accompanied by models, processes, methods, and tools, but with limited empirical evidence. When BDD is studied alongside other software engineering processes, the focus is more on empirical insights and processes rather than models, methods, and metrics. Integrating BDD with other technologies has resulted in fewer contributions of processes, methods, and tools.

Accepted Answer

Publication venues for BDD papers are scattered across various journals, conferences, and workshops. BDD research is not concentrated in specific venues, indicating a lack of dedicated platforms for BDD research. The International Conference on Agile Software Development is one of the popular venues, having published 7 papers over 10 years. However, there are no established BDD-specific publication venues or workshops yet. Establishing such venues would consolidate BDD research, informing the state-of-the-art and state-of-the-practice. This would also help in identifying the majority of BDD research publications.

Accepted Answer

The dominance of solution proposals and validation research in BDD research has led to a focus on proposing solutions for specific software engineering problems and lab evaluations of these solutions. However, the scarcity of industry-evaluated studies affects the applicability of proposed solutions due to a lack of evidence demonstrating their effectiveness. This highlights the need for more BDD experience reports and philosophical papers to guide research and practice, as well as opinion papers to understand practitioners' and academics' perspectives on BDD aspects.

Accepted Answer

Contributions in BDD research include studies on metrics for measuring various aspects of BDD specifications and the processes for producing BDD specifications. However, there is a shortage of studies on metrics for measuring other aspects of the BDD technique and the resulting specifications. The lack of studies on metrics indicates that practitioners lack ways to measure the quality of the BDD process and its associated aspects, and ways to measure the impact and value that BDD brings. Out of the three studies that proposed metrics, two focus on measuring the quality of individual scenarios in a BDD specification, while one focuses on measuring the overall quality of a BDD specification. This highlights the need for further research in this area to provide practitioners with the necessary tools to evaluate and improve their BDD practices.

Accepted Answer

Most BDD research studies use example applications to demonstrate solutions or insights, but this method is considered immature and not scientifically rigorous. Chen and Babar [13] suggest that BDD researchers should invest more in rigorous empirical evaluation methods to increase the chances of transferring research results into practice. However, there is a lack of studies reporting industry experiences with BDD, limiting the understanding of its practical use. Only three studies [37, 119, 120] have performed rigorous analysis through formal methods, such as hardware testing and software requirements tracing. More research focusing on formal aspects of BDD is needed to fully benefit from formal methods in software engineering.

Accepted Answer

The evolution of BDD research over time has seen an increase in publications covering a wide range of topics. Initially, BDD research focused on using BDD to facilitate software documentation. However, over the years, researchers have expanded their focus to other aspects of BDD, such as its application in various contexts like enterprise systems, health, education, and more. Studies on combining BDD with other software development processes, techniques, and technologies have also emerged, although they are relatively less popular compared to other BDD topics. The evolution and diversity of BDD research topics suggest that new and diverse topics could be explored in the future, enhancing our understanding of BDD in software development. Additionally, there has been a sustained interest in improving the BDD process and the artefacts produced, indicating potential advancements in the future. However, further research is needed to produce more mature insights to guide BDD practitioners.

Accepted Answer

The distribution of contributions in BDD research is dominated by studies proposing processes and associated artefacts as solutions for software engineering problems. Most BDD studies focus on artefacts resulting from the BDD process, similar to the TDD process. There is a lack of specific types of contributions such as metrics and a scarcity of BDD research evaluated in industry settings. The scarcity of philosophical and experience papers poses challenges in guiding BDD research and sharing lessons about its practical use.

Accepted Answer

BDD researchers need to focus more on producing mature and substantial studies, publishable by journals, to better inform the BDD state-of-art and state-of-practice. This will increase the understanding and application of knowledge in conference and workshop papers. More journal publications about BDD will give a clear understanding of research achievements in the area. Additionally, there is a need for more empirical evidence and insights informed by actual practices in the software industry. Experience reports on BDD should be shared to inform the user and research community about how BDD is actually used in the industry. Future research should also consider philosophical aspects of BDD, develop metrics for measuring various aspects of BDD specifications, and focus on improving the BDD process and associated artefacts. Studies on BDD alongside other software processes and technologies, as well as enhancing BDD with emerging technologies, are also needed.

Accepted Answer

Biases in primary study identification can be mitigated by formulating a generic search string, conducting snowballing, and having a pilot stage for agreement on exclusion criteria. The generic search string aims to avoid missing relevant studies, while snowballing accounts for missed papers. The pilot stage involves reviewing 20 papers by researchers to agree on exclusion or inclusion, and doubts are marked as 'maybe' for further discussion. This approach, as described by Petersen et al., aims to provide a general overview rather than capturing all evidence, reducing the impact of biases in the identification process.

Accepted Answer

To mitigate biases during data extraction and mapping, two strategies were employed. Firstly, established classification schemes by Wieringa et al., Mujtaba et al., and Chen and Babar were used to categorize papers into research types, contribution types, and research evaluation methods. Secondly, an inductive approach was adopted to categorize papers into specific themes, with the emergence of themes discussed by reviewers. To address potential discrepancies, a pilot workshop was conducted where 50 papers were categorized collaboratively by reviewers, ensuring agreement on themes. Subsequently, individual categorization was followed by a workshop to discuss and harmonize themes and address doubts regarding categorization. This comprehensive approach aimed to minimize biases and enhance the accuracy of data extraction and mapping.

Accepted Answer

Abushama et al.'s study, conducted in 2017, focused on the effect of Test-Driven Development (TDD) and Behavior-Driven Development (BDD) on project success. The study found that BDD was more likely to achieve higher customer satisfaction compared to TDD. However, both BDD and TDD consumed more development time and cost than traditional software testing approaches. The study highlighted the need for further research on the effectiveness of BDD and TDD in software development.

Accepted Answer

Behavior-Driven Development (BDD) enables software teams to specify software requirements using a structured natural language format. This approach facilitates understanding among all project stakeholders. The natural language specifications also serve as test cases, allowing for the verification of software behavior. Introduced about two decades ago, BDD has been extensively researched in the software engineering community. However, there is a scarcity of secondary studies published in peer-reviewed scientific outlets, making it challenging to comprehensively understand the current state of BDD research, its strengths, and potential gaps.

Behaviour driven development: A systematic mapping study

Chat with Paper

AI Agents for this Paper

Most frequently asked questions

1. What is Behaviour Driven Development (BDD)?

2. How many papers were included after individual screening?

3. What classification scheme was used for research type?

4. What are the major themes in BDD research?

5. Where has BDD been applied?

6. How ontologies aid BDD scenarios?

7. What is the publication frequency trend for BDD?

8. What are the main focuses of BDD studies over the years?

9. What are the publication venues for BDD papers?

10. What is the impact of solution proposals in BDD research?

11. What are the types of contributions in BDD research?

12. What evaluation methods are commonly used in BDD research?

13. What is the evolution of BDD research over time?

14. What is the distribution of contributions in BDD research?

15. What is the need for BDD researchers?

16. How can biases in primary study identification be mitigated?

17. What strategies were used to mitigate biases during data extraction and mapping?

18. What are the findings of Abushama et al.'s study on TDD and BDD?

19. What is the role of BDD in software requirements specification?

Citations

An Industrial Experience Leveraging the iv4XR Framework for BDD Testing of a 3D Sandbox Game

References

Systematic mapping studies in software engineering

Guidelines for conducting systematic mapping studies in software engineering : An update

Requirements engineering paper classification and evaluation criteria: a proposal and a discussion

Evidence-based software engineering for practitioners

Science and substance: a challenge to software engineers

Related Papers (5)

Rapid reviews for health policy and systems decision-making: more important than ever before

Crowdsourcing Paper Screening in Systematic Literature Reviews

Managing Scarcity: The German Housing Shortage and the Controlled Economy 1914–1990

Systematic Reviews: Questions, Methods and Usage

The scarcity school of economics, and the shortages it has wrought