Over the course of the last year, there have been a huge number of changes to the Xojo compiler (just under 800 commits). We made large refactorings, like rewriting how unqualified name lookup works. We fixed around 35 bugs, some of them dating back years. We added major new features to the language, including ‘Using’, Iterators, and new data types. To top it all off, we shipped support for a completely new platform, iOS, and then met Apple’s deadline for building 64-bit iOS apps.
And after all of that, we ended up with around eight regressions in the compiler. While not the perfect zero, I think this is just as impressive as the changes themselves.
One of the biggest factors in the low regression rate is the compiler’s testing setup. For each bug we fix or feature we add, we create additional tests. Due to how the compiler is structured, not all parts can be tested, but we still added over a hundred tests in the course of the year.
The test suite itself is simply a collection of raw Xojo source files on disk. Each test case can have expectations, which specify what the compiler should emit for the line in terms of errors or warnings. The expectations are specified in specially-formatted comments inside of the script itself. The end result is that test cases are self-contained, play nicely with version control, and are easy to write (which is by far the most important factor).
Here’s a simple test case, which accompanied the fix for Feedback case #21462:
Function Pointless() As Object Return -Nil // expected-error End Function
Of course, not all test cases are quite so straightforward or short. For example, the test for Feedback case #34479 is over 200 lines long, due to the edge cases that were found while fixing it. There are also times where test cases have to be somewhat contorted:
Sub Foo(param As Int32, _ // no-error
param As Int32) // expected-error
End Sub
The test harness is a fairly simple Xojo console application. The harness recursively searches through the folder of test cases, executes the test via the XojoScript class, and compares the XojoScript results against expected results.
The expectations are extracted from the script via a simple parser that understands just enough of the Xojo language to read all of the comments. The compiler’s own functions were intentionally avoided because they are a fundamental part of what is being tested.
Overall, the test harness took about two weeks to write and polish off. The time spent was well worth it and I would encourage the use of automated testing, even if you can’t test every part of your product. Every bug that the test suite catches is one less bug that ends up in your customers’ hands.