Gambit benchmarks
From Gambit wiki
The following tables contain the execution time of the Gambit benchmarks on various implementations of Scheme in different situations. For a given benchmark and situation, the entry in green indicates which Scheme system has the fastest execution and the number given is the CPU time in milliseconds. Other entries give the execution time relative to the green entry. Blank entries indicate that this benchmark was not executed (possibly because the system did not accept to compile the program, or because a given situation could not be simulated with that implementation of Scheme).
The benchmarks were run in four different situations. This was done by using compiler options and/or declarations.
-
Safe with generic arithmetic and mutable bindings.
This situation approximates the R5RS semantics:
- generic arithmetic operations
- mutable bindings for the definitions in the benchmark
- mutable predefined bindings (for +, car, ...)
- safe execution (i.e. an exception must be signalled on errors)
-
Safe with generic arithmetic.
This situation corresponds to the proposed R6RS semantics:
- generic arithmetic operations
- overflow detection on fixnum arithmetic (either produce a bignum or signal an exception)
- immutable bindings for the definitions in the benchmark
- immutable predefined bindings (for +, car, ...)
- safe execution (i.e. an exception must be signalled on errors)
-
Safe with fixnum/flonum specializations.
This situation corresponds to the proposed R6RS semantics, with the
use of fixnum and flonum specific functions for arithmetic:
- arithmetic operations are specialized to fixnum or flonum arguments as appropriate (for example turning + into fl+) and the fixnum operations may wrap on overflow
- immutable bindings for the definitions in the benchmark
- immutable predefined bindings (for +, car, ...)
- safe execution (i.e. an exception must be signalled on errors)
-
Unsafe with fixnum/flonum specializations.
This situation corresponds to the proposed R6RS semantics, with the
use of fixnum and flonum specific functions for arithmetic and the use
of the declaration (declare unsafe):
- like "Safe with fixnum/flonum specializations" but errors are not checked
Note: the tables are best viewed with a wide window.
Scheme systems used: Bigloo 2.8c, Chicken 2 Build 41, Gambit 4.0 beta 18, MzScheme 352, Scheme48 1.3
Safe with generic arithmetic and mutable bindings |
|
Safe with generic arithmetic |
|
Safe with fixnum/flonum specializations |
|
Unsafe with fixnum/flonum specializations |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
(R5RS) |
|
(R6RS) |
|
(R6RS fixflo) |
|
(R6RS fixflo unsafe) |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
|
|
- The test machine is a MacBook Pro, 2 GHz Intel Core, 2 GB DDR2 SDRAM, Mac OS X 10.4.7 .
-
The following options were given to each system:
- Bigloo Version 2.8c
- R5RS: situation not possible (car, +, etc cannot be mutated)
- R6RS: bigloo -O6 -copt -O3 -copt -fomit-frame-pointer
- R6RS fixflo: bigloo -Obench -copt -O3 -copt -fomit-frame-pointer
- R6RS fixflo unsafe: bigloo -Obench -copt -O3 -copt -fomit-frame-pointer
Note that with these options the compiler emits run time type checks but does not detect fixnum arithmetic overflows.
- Chicken Version 2, Build 41
- R5RS: csc -d0 -no-trace -no-usual-integrations
- R6RS: csc -d0 -O3 -no-trace -block
- R6RS fixflo: situation not possible (fixnum/flonum specific operations are unsafe)
- R6RS fixflo unsafe: csc -d0 -O3 -no-trace -block -unsafe -unsafe-libraries
The following program options were used for all situations: -:hi10M -:hs0
- Gambit Version 4.0 beta 18
- R5RS: no declarations
- R6RS: (declare (standard-bindings) (extended-bindings) (block))
- R6RS fixflo: (declare (standard-bindings) (extended-bindings) (block))
- R6RS fixflo unsafe: (declare (standard-bindings) (extended-bindings) (block) (not safe))
The programs were compiled with gsc -dynamic. The following program options were used for all situations: -:m10000,d- (this gives a 10 MByte heap)
- MzScheme Version 352
- R5RS: flat program (toplevel definitions) loaded with "load"
- R6RS: program wrapped in a module and loaded with "require"
- R6RS fixflo: situation not possible (no fixnum/flonum specific operations)
- R6RS fixflo unsafe: situation not possible (no unsafe mode)
- Scheme48 Version 1.3
- R5RS: ,bench off & ,open time posix bitwise ascii & ,load bench.scm
- R6RS: ,bench on & ,open time posix bitwise ascii & ,load bench.scm
- R6RS fixflo: situation not possible (no fixnum/flonum specific operations)
- R6RS fixflo unsafe: situation not possible (no unsafe mode)
The following program options were used for all situations: -h 5000000
- Bigloo Version 2.8c
- Most of the benchmarks take parameters that affect the computation that is performed (number of iterations, initial arguments, etc). The benchmarks were carefully written to defeat some compiler optimizations such as partial or total evaluation of the benchmark at compile time when the benchmark parameters are known. Our position is that each benchmark should be thought of as a function exported from a separately compiled module, and the function is called from another module with the benchmark parameters (so the compilation of the benchmark module should not take into account the specific values that are passed to the function or even the type of value passed). The benchmark parameters are passed to the function using a call to apply. For the systems tested this seems to achieve the desired effect (at least for the versions used). It should be noted that some of the systems could generate faster code if this approach was not used. In particular, for the Bigloo compiler, if the types of the benchmark parameters are known, better code can often be generated. Moreover it is common in Bigloo to specify the type of parameters on exported functions, so the performance obtained with this experimental setup may not be representative of the performance commonly achieved by programmers accustomed to Bigloo.