Actions

Gambit benchmarks

From Gambit wiki

The following tables contain the execution time of the Gambit benchmarks on various implementations of Scheme in different situations. For a given benchmark and situation, the entry in green indicates which Scheme system has the fastest execution and the number given is the CPU time in milliseconds. Other entries give the execution time relative to the green entry. Blank entries indicate that this benchmark was not executed (possibly because the system did not accept to compile the program, or because a given situation could not be simulated with that implementation of Scheme).

The benchmarks were run in four different situations. This was done by using compiler options and/or declarations.

  1. Safe with generic arithmetic and mutable bindings. This situation approximates the R5RS semantics:
    • generic arithmetic operations
    • mutable bindings for the definitions in the benchmark
    • mutable predefined bindings (for +, car, ...)
    • safe execution (i.e. an exception must be signalled on errors)
  2. Safe with generic arithmetic. This situation corresponds to the proposed R6RS semantics:
    • generic arithmetic operations
    • overflow detection on fixnum arithmetic (either produce a bignum or signal an exception)
    • immutable bindings for the definitions in the benchmark
    • immutable predefined bindings (for +, car, ...)
    • safe execution (i.e. an exception must be signalled on errors)
  3. Safe with fixnum/flonum specializations. This situation corresponds to the proposed R6RS semantics, with the use of fixnum and flonum specific functions for arithmetic:
    • arithmetic operations are specialized to fixnum or flonum arguments as appropriate (for example turning + into fl+) and the fixnum operations may wrap on overflow
    • immutable bindings for the definitions in the benchmark
    • immutable predefined bindings (for +, car, ...)
    • safe execution (i.e. an exception must be signalled on errors)
  4. Unsafe with fixnum/flonum specializations. This situation corresponds to the proposed R6RS semantics, with the use of fixnum and flonum specific functions for arithmetic and the use of the declaration (declare unsafe):
    • like "Safe with fixnum/flonum specializations" but errors are not checked

Note: the tables are best viewed with a wide window.

Scheme systems used: Bigloo 2.8c, Chicken 2 Build 41, Gambit 4.0 beta 18, MzScheme 352, Scheme48 1.3


Safe with generic arithmetic and mutable bindings

 

Safe with generic arithmetic

 

Safe with fixnum/flonum specializations

 

Unsafe with fixnum/flonum specializations

(R5RS)

 

(R6RS)

 

(R6RS fixflo)

 

(R6RS fixflo unsafe)

Program

Bigloo

Chicken

Gambit

MzScheme

Scheme48

boyer

14.11

2050

5.48

28.08

browse

8.78

5445

5.93

19.24

cpstak

6.14

2267

7.83

17.08

ctak

2.54

1760

87.60

9.46

dderiv

6.51

3306

7.92

18.22

deriv

6.59

1934

10.64

19.55

destruc

10.41

1957

6.41

32.17

diviter

13.11

1911

10.68

39.62

divrec

7.94

3096

7.87

17.94

puzzle

9.85

2537

7.50

24.90

takl

10.36

3936

4.38

27.04

triangl

9.53

3491

5.48

24.29

fft

10.21

1446

7.06

20.95

fib

5.24

4917

4.41

12.27

fibfp

4.56

2499

8.28

11.31

mbrot

6.74

2123

10.79

18.45

pnpoly

10.06

1903

6.52

crash

sum

12.30

1453

8.88

35.51

sumfp

6.76

1518

14.00

19.30

tak

8.02

3207

5.16

19.81

ack

10.86

246

6.54

22.80

array1

9.31

1029

5.83

22.76

cat

2.83

1.55

1122

1.83

string

135

2.10

1.73

61.26

sum1

2.75

217

1.16

11.47

sumloop

13.01

1437

10.28

36.00

tail

2.21

1277

1.24

4.94

wc

3.97

863

1.71

6.14

conform

4.24

2628

3.29

13.44

dynamic

3.18

1282

1.24

7.68

earley

11.00

1337

6.79

25.20

fibc

3.27

1744

64.70

10.15

graphs

7.83

2170

8.45

20.73

lattice

13.47

3295

5.08

38.31

matrix

8.84

2162

5.36

21.27

maze

5.87

3138

4.12

12.96

mazefun

6.61

2475

4.65

18.21

nqueens

11.33

2482

6.85

29.58

paraffins

17.38

2164

4.77

14.91

peval

9.09

1895

5.10

21.86

primes

12.83

2006

8.04

25.70

ray

4.00

1980

5.68

104.72

scheme

5.54

2282

3.51

13.76

simplex

7.51

2279

5.82

55.46

slatex

1.65

1528

1.11

2.79

Program

Bigloo

Chicken

Gambit

MzScheme

Scheme48

boyer

650

2.48

1.56

3.81

28.77

browse

2940

2.64

1.27

5.44

14.53

cpstak

5.72

1.95

1775

8.35

12.03

ctak

60.67

1.85

1729

70.26

8.34

dderiv

3.11

3.27

1579

15.54

23.04

deriv

4.21

4.03

1111

19.00

21.70

destruc

1.20

2.35

1482

3.32

20.72

diviter

5.41

814

1.43

22.89

23.87

divrec

3.15

1.62

1599

14.08

13.63

puzzle

2.32

2.57

1909

3.32

10.75

takl

1.63

491

2.45

4.86

74.03

triangl

1.26

2.99

2243

2.24

14.96

fft

3.19

2.06

1291

3.50

8.81

fib

2.39

3.74

2288

1.25

11.45

fibfp

4.15

2.29

1733

9.58

7.81

mbrot

4.86

1.44

1816

12.20

10.51

pnpoly

8.20

2.64

1522

3.83

crash

sum

5.22

5.06

1.43

740

19.66

sumfp

5.30

1.74

1331

12.48

9.96

tak

2.17

3.73

1536

1.52

14.41

ack

3.38

6.20

133

1.47

18.20

array1

1.75

2.65

830

1.47

9.52

cat

230

11.85

7.14

3.66

2.74

string

30

4.50

9.50

9.57

275.33

sum1

80

7.25

2.04

2.96

30.62

sumloop

2.37

4.31

1037

2.83

15.22

tail

640

2.86

1.92

2.09

6.39

wc

230

7.38

3.36

3.77

7.43

conform

800

2.95

1.49

6.92

33.72

dynamic

520

4.72

1.98

2.13

16.67

earley

1.61

5.26

1127

4.62

19.36

fibc

33.08

2.14

1365

70.66

10.56

graphs

2.08

2.08

1358

9.86

20.09

lattice

1.50

1.82

2721

3.26

34.55

matrix

1.37

3.03

1727

4.33

16.03

maze

1410

3.03

1.13

3.53

14.74

mazefun

2.02

2.41

1359

2.44

15.54

nqueens

1.73

2.59

1733

3.12

15.74

paraffins

2.78

5.32

1748

4.50

8.63

peval

880

4.53

1.21

5.47

22.76

primes

2.22

8.24

1339

8.98

19.24

ray

3.42

1.22

1127

5.82

177.25

scheme

1460

1.75

1.26

3.08

14.34

simplex

1.93

2.40

1325

4.70

78.38

slatex

930

1.78

1.57

1.60

3.52

Program

Bigloo

Chicken

Gambit

MzScheme

Scheme48

boyer

640

1.60

browse

2990

1.27

cpstak

5.31

1660

ctak

59.02

1768

dderiv

2.90

1707

deriv

4.08

1158

destruc

1360

1.06

diviter

3.72

1201

divrec

3.16

1620

puzzle

1.90

1745

takl

830

1.50

triangl

1440

1.56

fft

1.11

1210

fib

1000

2.08

fibfp

1.60

1739

mbrot

240

7.67

pnpoly

4.10

1433

sum

100

6.64

sumfp

100

13.80

tak

1090

1.21

ack

1.30

92

array1

1.35

853

cat

240

6.29

string

40

7.32

sum1

70

2.40

sumloop

220

3.40

tail

630

1.99

wc

160

4.90

conform

800

1.54

dynamic

510

2.05

earley

1.42

1016

fibc

34.71

1339

graphs

1.95

1293

lattice

1.45

2877

matrix

1.17

1772

maze

1.08

973

mazefun

1.16

1279

nqueens

1.01

1563

paraffins

2.70

1829

peval

880

1.21

primes

2.15

1360

ray

1.02

1091

scheme

1490

1.20

simplex

1.02

1145

slatex

930

1.59

Program

Bigloo

Chicken

Gambit

MzScheme

Scheme48

boyer

470

3.39

1.43

browse

2500

2.57

1.50

cpstak

6.02

1.62

1438

ctak

49.19

1.65

1728

dderiv

3.00

2.72

1500

deriv

3.72

2.91

1179

destruc

1200

1.12

1.06

diviter

5.17

785

1.29

divrec

3.03

1.84

1409

puzzle

3.58

4.28

660

takl

450

1.10

1.96

triangl

950

4.63

1.20

fft

3.17

4.66

328

fib

770

1.49

1.24

fibfp

2.26

2.55

1173

mbrot

230

5.30

4.68

pnpoly

190

14.73

1.51

sum

100

1.48

1.70

sumfp

100

11.15

12.04

tak

740

1.26

1.42

ack

2.05

44

1.34

array1

1.28

2.75

431

cat

200

10.44

7.86

string

20

6.55

14.55

sum1

70

6.26

2.36

sumloop

100

9.55

3.85

tail

560

2.51

2.20

wc

130

9.89

5.58

conform

630

3.75

1.50

dynamic

500

4.25

1.93

earley

1.49

6.55

767

fibc

29.12

1.71

1358

graphs

1.90

2.39

979

lattice

1.75

2.39

2067

matrix

1.17

2.39

1481

maze

1.47

4.38

524

mazefun

1.26

1.25

1000

nqueens

1.23

1.59

1081

paraffins

2.65

5.53

1736

peval

780

4.43

1.10

primes

2.15

1.43

1079

ray

130

10.81

2.94

scheme

1060

2.31

1.44

simplex

350

crash

1.33

slatex

880

1.70

1.60

  • The test machine is a MacBook Pro, 2 GHz Intel Core, 2 GB DDR2 SDRAM, Mac OS X 10.4.7 .
  • The following options were given to each system:
    • Bigloo Version 2.8c
      • R5RS: situation not possible (car, +, etc cannot be mutated)
      • R6RS: bigloo -O6 -copt -O3 -copt -fomit-frame-pointer
      • R6RS fixflo: bigloo -Obench -copt -O3 -copt -fomit-frame-pointer
      • R6RS fixflo unsafe: bigloo -Obench -copt -O3 -copt -fomit-frame-pointer

      Note that with these options the compiler emits run time type checks but does not detect fixnum arithmetic overflows.

    • Chicken Version 2, Build 41
      • R5RS: csc -d0 -no-trace -no-usual-integrations
      • R6RS: csc -d0 -O3 -no-trace -block
      • R6RS fixflo: situation not possible (fixnum/flonum specific operations are unsafe)
      • R6RS fixflo unsafe: csc -d0 -O3 -no-trace -block -unsafe -unsafe-libraries

      The following program options were used for all situations: -:hi10M -:hs0

    • Gambit Version 4.0 beta 18
      • R5RS: no declarations
      • R6RS: (declare (standard-bindings) (extended-bindings) (block))
      • R6RS fixflo: (declare (standard-bindings) (extended-bindings) (block))
      • R6RS fixflo unsafe: (declare (standard-bindings) (extended-bindings) (block) (not safe))

      The programs were compiled with gsc -dynamic. The following program options were used for all situations: -:m10000,d- (this gives a 10 MByte heap)

    • MzScheme Version 352
      • R5RS: flat program (toplevel definitions) loaded with "load"
      • R6RS: program wrapped in a module and loaded with "require"
      • R6RS fixflo: situation not possible (no fixnum/flonum specific operations)
      • R6RS fixflo unsafe: situation not possible (no unsafe mode)
    • Scheme48 Version 1.3
      • R5RS: ,bench off & ,open time posix bitwise ascii & ,load bench.scm
      • R6RS: ,bench on & ,open time posix bitwise ascii & ,load bench.scm
      • R6RS fixflo: situation not possible (no fixnum/flonum specific operations)
      • R6RS fixflo unsafe: situation not possible (no unsafe mode)

      The following program options were used for all situations: -h 5000000

  • Most of the benchmarks take parameters that affect the computation that is performed (number of iterations, initial arguments, etc). The benchmarks were carefully written to defeat some compiler optimizations such as partial or total evaluation of the benchmark at compile time when the benchmark parameters are known. Our position is that each benchmark should be thought of as a function exported from a separately compiled module, and the function is called from another module with the benchmark parameters (so the compilation of the benchmark module should not take into account the specific values that are passed to the function or even the type of value passed). The benchmark parameters are passed to the function using a call to apply. For the systems tested this seems to achieve the desired effect (at least for the versions used). It should be noted that some of the systems could generate faster code if this approach was not used. In particular, for the Bigloo compiler, if the types of the benchmark parameters are known, better code can often be generated. Moreover it is common in Bigloo to specify the type of parameters on exported functions, so the performance obtained with this experimental setup may not be representative of the performance commonly achieved by programmers accustomed to Bigloo.