The Unofficial Guide to Running SPEC CPU2000/2006 in 2017 and onwards

Last update: 13-May-2017

This document is a collection of my experiences running SPEC CPU2000/CPU2006 benchmark suites for my research on computer architecture. I have ran SPEC CPU2000/2006 on around 70 configurations using 50+ computers so far (the result is here). It was not straightforward to run some benchmarks on certain configurations and took a lot of time to resolve problems. So I want to share my experiences with anybody who is having difficulty running them.

This document is not endorsed by or affiliated with Standard Performance Evaluation Corporation and the solutions proposed here are not warranted by SPEC. Please do not send them questions about this document. I'll try to reply as much questions as I can (see the bottom of this page for my e-mail address).


GCC version considerations

When I started this research GCC 4.5 was the latest stable version available. I tried several versions of compilers, and here's what I've experienced:

GCC 4.0.*
No Fortran 95 support. It's impossible to compile CFP2000 or CFP2006.
GCC 4.3.*
Buggy! Sometimes emits wrong code and results miscompare. Avoid it.
GCC 4.4.*
Probably OK, but the executables are slightly slower than those compiled with GCC 4.5.
GCC 4.5.*
This is unbeatable. Perfect. Sometimes better than vendor compilers (Sun/IBM).

Watch out for library versions!

Recent versions of GCC depends on GMP, MPFR, MPC and ZLIB for compilations. Of these ZLIB is the easiest-to-diagnose component and it just sometimes makes GCC fail to build. Passing --with-system-zlib to the GCC configure script solves this problem in most cases (do not forget to install zlib headers using your package manager).

GMP, MPFR and MPC are a bit more troublesome as they make the benchmarks to miscompare the results. Always grab the latest source tarballs from their site, and never forget to run ‘make check’ after each compilation. If it fails, just keep going and compile GCC anyway, then use the newly-built compiler to build the libraries again. Pass --with-gmp=, --with-mpfr and --with-mpc to the configure script of each package.

Compiling GCC is in itself a hard task for beginners, and often include extensive patching work on minor (sorry!) platforms like AIX and IRIX. I've done them the ad-hoc way, so there's not much I can tell you about it.


252.eon (CPU2000) doesn't compile on various configurations using GCC

Compilation fails like this:

./ In function 'ggRaster<T>* ggConvolveWithGaussian(ggRaster<T>, double)':
./ error: there are no arguments to 'memset' that depend on a template parameter, so a declaration of 'memset' must be available
./ note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated)
specmake: *** [mrSurfaceTexture.o] Error 1

As the error message suggests I needed to put ‘-fpermissive’ to CXXPORTABILITY to avoid this problem.

Miscompare of results in 255.vortex (CPU2000) when compiled using GCC

I needed to put ‘-fno-strict-aliasing’ to CPORTABILITY to avoid this issue in GCC 4.5.1.

Parallelizing compilation does not work in CPU2000

In CPU2006, adding options like ‘-j32’ to makeflags enables the use of additional processors. You can't do this in CPU2000 (at least when using GCC). Some Fortran programs in CFP2000 fails to build because GCC cannot find *.mod files that are generated on compilation. Just avoid using makeflags that way.

300.twolf (CPU2000) miscompares in ARM systems

On ARM, compiler defaults to unsigned char. Add -DHAVE_SIGNED_CHAR to CPORTABILITY.

Can I use GCC on IRIX for CPU2000?

So far I have successfully ran all benchmarks in CPU2000 except 187.facerec under IRIX using GCC 4.5.3. It took quite some time to figure out the flags though. Here are my flags for IRIX using GCC 4.5.3:

178.galgelFPORTABILITY= -ffixed-form

Need to adjust precision for some benchmarks to finish (x86)

Some benchmarks in CPU2000 and CPU2006 fall into an infinite loop unless proper precision flags are passed when using GCC 4.5. For CPU2000, passing -mpc64 is required for 32-bit x86 platforms. For CPU2006, do NOT pass -mpc64 as it will cause some of the CPU2006 benchmarks to fall in another infinite loop.

CPU2006 483.xalancbmk and 447.dealII fails to build

Build of 483.xanancbmk dies with:

FormatterToHTML.cpp: In member function 'void xalanc_1_8::FormatterToHTML::initCharsMap()':
FormatterToHTML.cpp:139:42: error: 'memset' was not declared in this scope
specmake: *** [FormatterToHTML.o] Error 1

and 447.dealII with: In constructor 'Quadrature<dim>::Quadrature(const std::vector<Point<dim> >&)': error: 'atof' is not a member of 'std'
specmake: *** [quadrature.o] Error 1

Both are caused by missing include files. GCC has a nice option for this problem that lets you compile these benchmarks without modification to source. Append ‘-include cstdlib -include cstring’ to CXXPORTABILITY, then problems go away (hopefully).

187.facerec (CPU2000) and 462.libquantum (CPU2006) segfaults on my PowerMac G5

When compiling these benchmarks on Apple PowerMac G5 computer with MacOS X 10.[45] using GNU GCC 4.5.2, I encountered this problem. GCC 4.5 (with or without multilib support) changes its behavior when compiling with option ‘-m32’ (it produces 32-bit binary with or without ‘-m32’). When compiled with ‘-m32’, this problem goes away.

The OpenMP implementation of NAS Parallel Benchmarks version 3.3.1 also suffers from this problem. Just add ‘-m32’ to CFLAGS and FFLAGS.

This problem was not encountered when 64-bit binary is compiled using ‘-m64’ flag.

On AMD K7 (Athlon MP/Duron Morgan), 410.bwaves miscompares!

For some reason these processors need the -mieee-fp for 410.bwaves when -O3 -march=athlon-mp optimization flags are used.

400.perlbench fails to build on OpenSolaris (Solaris 11) using GCC!

This problem is due to changes in a system header. You can avoid it with -U__sun__. miscompares on ARMHF (or any other architecture) invokes undefined behavior with signed integer overflow. Append -fno-strict-overflow to portability flags. For details see [].

ld complains a lot and fails to link 400.perlbench when using GCC5

The linker ld complains a lot about duplicate definitions when using GCC 5 to compile 400.perlbench. The default mode for C is now -std=gnu11 instead of -std=gnu89 []. Append -std=gnu89 to portability flags, and the problem goes away.

Working around problems in SPEC tools

How can I compile SPEC tools under platforms unknown by the tool scripts?

When compiling the spec tools for platforms unknown by the tool scripts (such as Tilera TILE-Gx), copying new config.sub file is necessary for the autotools to work. For CPU2006:

and for CPU2000:

Just google for the latest config.sub file. Just about any GNU package has the file.

Perl in SPEC is broken!

Older version of perl included in SPEC tool does not work well with new platforms. On running make, it immediately complains that:

make: *** No rule to make target `<command-line>'

In this case you need to add a line in tools/src/perl-*/makedepend.SH:

             -e '/^#.*<builtin>/d' \
             -e '/^#.*<built-in>/d' \
             -e '/^#.*<command line>/d' \
+            -e '/^#.*<command-line>/d' \
             -e '/^#.*"-"/d' \
             -e '/: file path prefix .* never used$/d' \

For dynamic linking (which is REQUIRED for running SPEC with their tools) to work you need to edit tools/src/buildtools script. My favorite options to pass to perl Configure script are:

    ./Configure -dOes -Ud_flock $PERLFLAGS -Ddosuid=undef -Dprefix=$INSTALLDIR 
-Dd_bincompat3=undef -A ldflags="-ldl -lm -L${INSTALLDIR}/lib" -A ccflags=-I${INSTALLDIR}/include 
-Ui_db -Ui_gdbm -Ui_ndbm -Ui_dbm -Uuse5005threads 
-Dcccdlflags="-fPIC -shared" -Dlddlflags="-shared -fPIC" -Duseshrplib=true ; 
testordie "error configuring perl"

Perl references old Linux header asm/page.h which isn't there already. Remove the reference by editing tools/src/perl-*/ext/IPC/SysV/SysV.xs. Just remove lines #include'ing asm/page.h.

All unit tests in Compress-Bzip2 in CPU2006 tools fail!

You need to pass -fPIC when compiling bzip2. If you're using tools/src/buildtools script set environment variable BZIP2CFLAGS=-fPIC and everything works fine.

specmd5sum refuses to be built

My dumb solution for specmd5sum not compiling right is to comment out lines in tools/src/specmd5sum/lib/getline.h. Just remove the getline and getdelim lines.

Tar in SPEC tools fails to build when I'm root (reported by a reader)

It's likely that you're the root user if the system you're benchmarking is an experimental one. In that case tar fails to build (in config.log):

configure:35146: error: in `/path/to/cpu2006/tools/src/tar-1.25':
configure:35149: error: you should not run configure as root (set FORCE_UNSAFE_CONFIGURE=1 in environment to bypass this check)
See `config.log' for more details.

Try setting the said environmental variable, or just create a non-root acount.

When compiling SPEC tools, perl complains You haven't done a "make depend" yet!

And when you manually run make depend in the last directory, it says:

../makedepend: 1: ../makedepend: Syntax error: Unterminated quoted string

This is because of a bug in the dash shell that are installed by default as /bin/sh in Debian and Ubuntu systems. You can confirm that you are using dash by:

# ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Nov  8 23:09 /bin/sh -> dash

A workaround is to replace the symlink with another symlink to /bin/bash:

# dpkg-reconfigure dash