Xojo has an extensive testing period where actual users test a pre-release version with their projects but if you are wondering what kind of testing we do internally before each pre-release of Xojo, we have quite a bit of automated testing processes. There are over 400 tests just for the compiler alone. Already, we are approaching 300 tests for our Android framework. In total, across all supported platforms, there are over 2500 automated tests. Most tests run every time we do a full build of Xojo (also a fully automated process) which occurs 2 to 5 times a week. When one of these tests fails, the unlucky engineer responsible is informed and gets to work fixing it. More automated tests are being added to our system regularly.
Primarily, we focus on the issues having the greatest impact on the Xojo community. However, that focus is not absolute. The engineering team at Xojo is using Xojo all-day, every day. They will often catch bugs in the early stages of the development of a new feature or as a side effect of fixing some other bug, long before that release gets to pre-release testers. The number of combinations of interactions between various parts of the Xojo frameworks is effectively infinite and no system can test everything. That’s why pre-release testing by actual users is so important. When you look at the bugs that are reported, 95.6% have only one user reporting them. Though we provide, in fact require, users to search Feedback before submitting a case, users often describe the same bug in varying ways resulting in duplicates. Given both the infinite number of interactions and some number of duplicate reports, the fact that overwhelming majority of reports have just one user is understandable. That we focus on the reports that have multiple users is also understandable. And of course there are exceptions, if it’s obvious that a new bug is going to effect a lot of users, we will get to work on it ASAP. In addition, if an engineer is working in a particular part of the IDE or a particular framework, since they have take the time to refamiliarize themselves with that code, they will often look for bugs that have been reported against it.
Development tools are special because the type of people who use them are more likely to report bugs and are more likely to want to search the list of open cases, get updates about bugs and more. While it’s important to report bugs, and we can’t stress enough how much we appreciate the time and care you take in doing so, even if the bug is fixed, you still have to wait for the release in which it’s fixed. Even our engineers have to wait for bugs fixes just like you do. Still, when you run into a bug, please search Feedback first to see if it’s been reported, perhaps think of a few ways it could have been reported and do a few searches. If you find a match, add yourself to the case. If you don’t, please file a case. If you haven’t found a workaround, ask on the forum or contact us directly and we see what we can do to help.
There is a lot of automated testing that goes on before we put out a pre-release to Testers. And we will continue to add more because automated testing is a terrific and efficient way to test anything that can be automatically tested. We are always on the lookout for ways to improve efficiency in our processes.