Authors
Adil Mukhtar, Birgit Hofer, Dietmar Jannach, & Franz Wotawa
Abstract
Spreadsheets are widely used in organizations for various purposes such as data aggregation, reporting and decision-making. Since spreadsheets, like other types of software, can contain faulty formulas, it is important to provide developers with appropriate methods to find and fix such faults.
Recently, various heuristic and statistics-based fault identification methods were proposed, which point developers to potentially faulty parts of the spreadsheets. Due to their heuristic nature, these methods might, however, miss some faults. As a result, if spreadsheet developers rely too strongly on these methods, they might not pay sufficient attention to problems that are not pinpointed by the methods.
In this research, we are the first to study this potential problem of over-reliance in spreadsheet debugging, which may lead to limited debugging effectiveness. We report the outcome of a controlled experiment where 59 participants were tasked to find faulty formulas in a given spreadsheet with and without support of a novel spreadsheet debugging tool.
Our results indicate that tool over-reliance can indeed result as a phenomenon of using heuristic debugging techniques. However, the study also provides evidence that making users aware of potential tool limitations within the debugging environment may help to address this problem.
Sample
Participants were tasked to locate faults in a given spreadsheet. Participants in two treatment groups were supported through the debugging functionality of SmellChecker; participants in the control group did not receive such support. As the main dependent variable, we measured the fault identification performance of the participants.
The findings of this research have several practical implications:
- It is important for tool developers to understand that tool over-reliance phenomena may occur in practice. Therefore, it is crucial to communicate to developers what the tool is capable of and which types of problems it might not find.
- Raising awareness regarding tool limitations may have to be done in a very explicit way. Highlighting the limitations in a manual or during the installation was not effective in our study.
- Respective prompts within the application itself, in contrast, proved to be helpful to raise user awareness.
- In practice, of course, the question remains when and how often such warnings should be displayed, as too many prompts may distract and annoy users sooner or later.
- SmellChecker, even though being a research prototype, is promising from the perspective of end user acceptance and usability.
Publication
2022, The Journal of Systems & Software, Volume 184
