automated test systems

The Bullant system consisted of a virtual machine with its own Java-like language allowing multiple inheritance.

Each night, a series of tests was farmed out to all the unattended PC's. Unattended PC's were defined as being any PC that starting running an in-house developed screen saver. This screen saver would connect to a central server and request a workload. The 'idle' PC would then process this work load and then request another. The work loads were randomly selected, with the only overriding restriction being that all tests must be processed at least once during the night. It was similar to the farming carried out by seti@home.

The benchmarking section of the overnight runs ran a series of tests and then plotted the results of these tests against previous runs. This allowed quick inspections of the changes made on the previous day that affected various throughput parameters of the Bullant virtual machine.

The tests involved in the benchmarking simulated up to 1000's of simultaneous users and thus had it's own set of 8 dual process servers. All testing had to be carried out across multiple operating systems and implementations. The processors used were X86 and itaniums, as Intel was a shareholder in the company. There was some testing also done on a 24 node SPARC cluster from Sun.

The results from both these sets of tests were published each morning on the office intranet for all employees to see. Drill down techniques were used in order to show the details of the top level summaries.