Demystifying the adoption of behavior-driven development in open source projects

Context:Behavior-Driven Development (BDD) features the capability, through appropriate domain-specific languages, of specifying acceptance test cases and making them executable. The availability of frameworks such as Cucumber or RSpec makes the application of BDD possible in practice. However, it is unclear to what extent developers use such frameworks, and whether they use them for actually performing BDD, or, instead, for other purposes such as unit testing. Objective:In this paper, we conduct an empirical investigation about the use of BDD tools in open source, and how, when a BDD tool is in place, BDD specifications co-evolve with source code. Method:Our investigation includes three different phases: (i) a large-scale analysis to understand the extent to which BDD frameworks are used in 50,000 popular open-source projects written in five programming languages; (ii) a study on the co-evolution of scenarios, fixtures and production code in a sample of 20 Ruby projects, through the Granger's causality test, and (iii) a survey with 31 developers to understand how they use BDD frameworks. Results:Results of the study indicate that ≃ 27% of the sampled projects use BDD frameworks, with a prevalence in Ruby projects (68%). In about 37% of the cases, we found a co-evolution between scenarios/fixtures and production code. Specifically, changes to scenarios and fixtures often happen together or after changes to source code. Moreover, survey respondents indicate that, while they understand the intended purpose of BDD frameworks, most of them write tests while/after coding rather than strictly applying BDD. Conclusions:Even if the BDD frameworks usage is widespread among open source projects, in many cases they are used for different purposes such as unit testing activities. This mainly happens because developers felt BDD remains quite effort-prone, and its application goes beyond the simple adoption of a BDD framework.