automated test systems

A simple gaming machine was being produced based on a dual processor design. The graphics and display engines were built around a standard Intel x86 configuration while the IO processing of buttons, switches and money collection mechanisms was based around a Xilinx FPGA.

The main interface between the graphics subsystem and the IO sub-system was through dual ported ram. Access to this ram was by a two level locking mechanism that ensured controlled access. The IO subsystem monitored the condition of the IO and would report all events through a notification mechanism to the graphics subsystem. Specific commands were passed from the graphics subsystem to the IO system again through the dual ported ram.

There were two overriding requirements for the IO subsystem. A need to maintain consistent state information across power cycles and to report report events once and only once again across power cycles. These requirements were further was complicated by a mandatory requirement to record specific regardless of whether main power was active.

The complete system was built around Linux 2.6 kernels. With a dedicated controller handling the IO.

The Xilinx tool chain was used to build the FPGA image including the Xilinx Microblaze processor core. The IO core was a super loop calling specific routines to monitor the specific IO points and was built using the toolset from the University of Queensland. The pseudo/soft real-time aspects were controlled through interrupts both within the Microblaze processor core and interrupts to the X86. All the IO could be addressed by either the Microblaze code or the the Linux kernel with specific code design to disable one side of the other.

Specific problems encountered within this project were.
Serial port interrupt stealing between the x86 and the Microblaze.
Timing issues to do with events within the IO across power cycles.
VHDL timing issues which caused long delays on bus access.
A three stage boot process that allowed VHDL updates to the core of the Microblaze.

This project also had some severe risk mitigation problems which resulted in delays to the hardware and a cycle of continuous changes to the software for each new hardware release.

Device Drivers
The overall interface between the IO subsystem and the graphics subsystem was through a PCI interface. This IO Subsystem was seen as a simple PCI device providing several windows into the IO subsystems address space. This includes battery backed common memory, dual ported ram for communication and direct access to the IO devices and the instruction space for the FPGA internal Microblaze processor. This direct access to the IO devices was only used to allow direct access to the IO subsystems serial ports of which at last count there were 15 serial ports. The direct access to the Microblaze instruction ram meant that a CRC checked image could be loaded into the IO processor by the Linux subsystem at boot.
This one PCI device was in reality three specialised devices. A problem with this design is that each PCI device can only have one owner and thus the original design of three device drivers was not going to work. To solve this problem a fourth simple master device driver was written that owned the PCI device and then provided services to the other three secondary device driver's. These services consisted of providing the address ranges required by the secondary device drivers for their own specific tasks and an interface and call back mechanism to provide interrupt services. The three secondary device drivers would connect to this primary device driver and then request their own unique memory interface and register any interrupts these secondary device drivers needed. (As the final system provided only a single interrupt than these secondary device drivers simply registered Interrupt Service Routines.

Boot loaders
A mechanism was required whereby the code within the IO processor could be easily upgraded. This upgrade could also include changes to the FPGA firmware.

The FPGA image was first loaded from Linux. The FPGA then executed a first stage bootloader which spun until a command from Linux graphics subsystem was published. While the FPGA was spinning waiting for graphics subsystem to allow the FPGA to proceed, the second stage boot sequence was loading the Microblaze instructions and data into the FPGA instruction space. Once this stage of the boot sequence was complete and the FPGA was told to proceed and the third stage of booting commences. The FPGA would initialise the system default jump table and move the data into the correct memory locations. The FPGA would then proceed to boot the IO subsystem.

One of the circular peculiarities of this loading sequence was Linux would not see the IO PCI device until after the FPGA firmware was loaded. Linux then had to be told a new PCI device existed and to include it in its own internal tables before the device drivers associated with the IO subsystem could be started.