TOPBENCH: The Oldskool PC Benchmark

A new benchmark for oldskool PC gaming, retrocomputing, and emulator tuning

The Oldskool PC Benchmark (TOPBENCH) is a benchmarking and profiling utility designed to help the vintage PC gaming enthusiast find the best machine speed for a particular game, whether running in an emulator or on real hardware. Because it performs short measurements continuously, it is also ideal for "dialing" the speed of your favorite emulator, such as DOSBox, in realtime to more accurately match the speed of a particular system.

TOPBENCH works by profiling your system and recording a synthetic benchmark score into its database.  It then compares your system's score to the other systems in the database, and shows you what systems are closest to yours.  It can also compare any two systems in the database against each other, to illustrate the speed difference between them.  This database, coupled with published source code, can help emulator authors check their emulation against data collected from the real hardware itself.

TOPBENCH is still in development, but you can help seed the database right now by running the profiling stub.  The stub will calculate the benchmark score for your system, record microsecond timings of various metric suites, and detect what hardware you have.  It then writes this to a file that you can email to me for inclusion in both the first release of TOPBENCH as well as regular benchmark database releases.  The stub runs on any PC with 128K RAM or higher, running DOS 2.x or higher (DOS 3.3 preferred), and you can download the stub here: ftp://ftp.oldskool.org/pub/misc/stub91.zip
Run the stub from a freshly-booted machine and it will print instructions and start profiling your system.

Site index:

• Why another benchmarking utility?
• Feature List
  
• Downloads
• Submitting Results (Contact Information)
• Background
• Questions and Answers
• Known Bugs
• Thanks
• Development History

Why another benchmarking utility?

TOPBENCH was born out of a love/hate relationship with PC emulators.  For example, I love DOSBOX's ability to run old PC games on modern hardware, but I hate how its emulation speed is wildly inconsistent with how the real hardware behaves.  It is  impossible to know what you're getting when you set a particular speed.  I also like how MESS and other emulators attempt to be cycle-exact, but am frustrated than none of them truly are.

If emulators were the fathers of TOPBENCH, then older benchmarks themselves were the mothers.  Of all the early PC DOS benchmarks (Landmark Speed Test, Norton SI, C&T MIPS), not a single one got everything "right":

• Norton SI used a laughably tiny code metric
• Landmark confusingly reported its metric as a function of "80286 MHz speed"
• MIPS "instructions per second" did not indicate actual computing throughput

...and so on.  The worst problem with all DOS-era benchmarks is that their metric code was completely unknown.  You had no idea what the benchmark score you got was based on, and whether or not it applied to your situation.

To address these issues, I designed TOPBENCH.

Feature List

For retrogaming emulator users:

• Uses actual DOS 3-D game opcode distribution as part of its metric code. All benchmark scores are synthetic numbers, but with TOPBENCH, the benchmark score for your system is more relevant for gaming than other benchmarks.
  
• Dial the speed of DOSBox to more closely match a given target. Because TOPBENCH's measurements can be run continuously, you can run it in DOSBox and then use Ctrl-F11/Ctrl-F12 to raise/lower the speed of DOSBox to more closely match a given target machine, and see the benchmark comparison change in real time.
  

For emulator authors:

• The metric assembler source code is provided with the benchmark.  If you're having trouble getting certain opcode combinations or speeds in your emulator working, you can check the actual code being executed without having to resort to disassembly.  The code is verbosely commented for full disclosure.
  
• Each part of the metric code has its execution time in microseconds recorded in the database.  This lets you hone your emulator to more closely match the actual timings of your target processor/system.
  
• Every opcode that is practical to exercise is exercised.  Only instructions that would result in variable execution times (IN, OUT, HLT, etc.) or are unsafe to run on any machine (ie, LOCK, WAIT, ESC, etc.) are skipped.
  

For retrocomputing enthusiasts:

• Identify the core hardware of your vintage PC without opening it. TOPBENCH uses a mixture of libraries and techniques to inventory your machine so that some meaningful descriptive information can be attached to a measurement database entry. This information includes CPU make/model, speed in MHz, BIOS maker/date/revision, and make/model of video card.
• Answer some interesting historical questions. Ever wondered if a 20MHz 80286 was faster than a 16MHz 80386?  Just how much of a boost did NEC V20/V30 processors give over the original 8088/8086?  Did 0-wait-state RAM make that much of a performance difference? Just how much slower than an original IBM PC was the IBM PCjr?  TOPBENCH can  answer these questions, because every item in TOPBENCH's local database was sampled from real hardware.
• See how your clone PC measures up to other machines. This is the primary purpose of a benchmark, after all. Not all clones were created equal; see how yours stacks up against others.

The Performance Metric

There are three test suites that measure individual performance of certain areas of a machine (CPU, system memory, and video adapter memory), and a fourth suite that combines the first three. This fourth suite creates what is considered a system's DOS gaming performance "fingerprint", and is the primary basis of comparision between machines in TOPBENCH's database (although the individual suites can also be used if specified by the user).

The three individual test suites are:

• CPU instruction mix:
1. Executes every x86 instruction once, from AAA to XOR. (Uncommon instructions/behavior, such as AAM with a custom divisor or NEC V20/V30-specific instructions, are NOT measured.)
2. Executes MOV with every combination of the modr/m byte (this is primarily to expose differences in emulator implementations)
3. Copies data to/from registers, such that each register has data read from it and written to it at least once. (this is primarily to expose differences in emulator implementations verses CPU optimizations (ie. moves involving the accumulator are somewhat more optimized than generic register moves))
• System Memory test:
1. Executes REP MOVS/STOS/LODS/SCAS/CMPS. This test repeats with both byte-sized operation (ie. REP MOVSB) and word-sized operation. It then does the same thing but with the direction flag reversed. The byte/word variations expose differences in 8088 vs. 8086 processors. The direction flag variations expose differences in emulator implementations.
• Video Adapter Memory test:
1. Reads the first 4K contents of the video adapter as preperation, then writes them back, twice, using REP MOVSW and uses that count to determine the KB/s speed. (The test uses the first 4K of video adapter memory contents because that is the only value guaranteed to be on all video adapters (original MDA only had 4K of video memory). "Read once but write twice" was chosen to stress caches (if any) while being fair to the nature of video ram which is usually optimized for write speed, not read speed.

All suites are run in a loop for a certain length of time (currently 55ms, the maximum period allowed by interrupt-on-terminal-count), and the number of iterations counted. All results are measured by the 8253 timer, then the results are combined into a meaningful "fingerprint" in differentiating systems from each other. The formula for creating the fingerprint is the combination of:

• CPU: 40%
• System RAM: 40%
• Video RAM: 20%

This allows for better/worse video adapters to affect the "fingerprint" without completely dominating it.

When comparing your machine's performance to other systems, the fingerprint is used by default, but the individual test suites are also selectable as a comparision in case you don't agree with the methodology behind the fingerprint.

Downloads

None yet, sorry. Need to write the benchmark first ;-)

Questions and Answers

My vintage PC only has 128KB RAM, I can't run the TOPBENCH program.  How can I profile my machine and get it into the database?  The TOPBENCH "stub" (TOPBSTUB.EXE) can run (barely) on a 128K machine booted with DOS 3.3.  When it runs, it dumps information into a text file called output.ini which you can then import into TOPBENCH running on a different computer, or you can email me the file (see contact information above) to include into the next database release.

Why is it important to run games at the proper emulation speed? The earlier a game was written, the more susceptible it was to machine speed.  Most PC games made before 1985 will only run properly on one single speed!  So for maximum enjoyment, speed needs to match what the developer of the game expected when they wrote it.

It's also important to get speed right when doing historical research.  It's not fair to judge a work if that work is not being represented properly.

Known Bugs

• The profiling stub has not been tested in DOS 1.x.
  
• Running the stub under DOS 2.x mostly works except the CPU speed detection.
  

Thanks

I would like to thank:

• The creator of DOSBox, as well as everyone who has worked on it
• Everyone at comp.lang.asm.x86, comp.lang.pascal.borland, and the Vintage Computer forums
  
• The authors of the DETECT and TMi0SDGL detection libraries, which served as a good base to expand on.
• Andrew Jenner
  

Development History

20111111: Released v0.91 of the profiling stub for testing.

• Removed: CPU speed testing under DOS 2.x and lower (I need to debug it).  For now, please run the stub on DOS 3.x or higher for all functions to be tested.