Last update: 13-May-2017
This document is a collection of my experiences running SPEC CPU2000/CPU2006 benchmark suites for my research on computer architecture. I have ran SPEC CPU2000/2006 on around 70 configurations using 50+ computers so far (the result is here). It was not straightforward to run some benchmarks on certain configurations and took a lot of time to resolve problems. So I want to share my experiences with anybody who is having difficulty running them.
This document is not endorsed by or affiliated with Standard Performance Evaluation Corporation and the solutions proposed here are not warranted by SPEC. Please do not send them questions about this document. I'll try to reply as much questions as I can (see the bottom of this page for my e-mail address).
When I started this research GCC 4.5 was the latest stable version available. I tried several versions of compilers, and here's what I've experienced:
Recent versions of GCC depends on
GMP,
MPFR,
MPC
and ZLIB for compilations. Of these ZLIB is the
easiest-to-diagnose component and it just sometimes makes GCC fail to build.
Passing --with-system-zlib to the GCC configure script solves this
problem in most cases (do not forget to install zlib headers using your package
manager).
GMP, MPFR and MPC are a bit more troublesome as they make the benchmarks to
miscompare the results. Always grab the latest source tarballs from their site,
and never forget to run ‘make check’
after each compilation. If it fails, just keep going and compile GCC anyway,
then use the newly-built
compiler to build the libraries again. Pass --with-gmp=,
--with-mpfr and --with-mpc to the configure
script of each package.
Compiling GCC is in itself a hard task for beginners, and often include extensive patching work on minor (sorry!) platforms like AIX and IRIX. I've done them the ad-hoc way, so there's not much I can tell you about it.
Compilation fails like this:
./ggRaster.cc: In function 'ggRaster<T>* ggConvolveWithGaussian(ggRaster<T>, double)': ./ggRaster.cc:439:64: error: there are no arguments to 'memset' that depend on a template parameter, so a declaration of 'memset' must be available ./ggRaster.cc:439:64: note: (if you use '-fpermissive', G++ will accept your code, but allowing the use of an undeclared name is deprecated) specmake: *** [mrSurfaceTexture.o] Error 1
As the error message suggests I needed to put ‘-fpermissive’ to CXXPORTABILITY to avoid this problem.
I needed to put ‘-fno-strict-aliasing’ to CPORTABILITY to avoid this issue in GCC 4.5.1.
In CPU2006, adding options like ‘-j32’ to
makeflags enables the use of additional processors.
You can't do this in CPU2000 (at least when using GCC).
Some Fortran programs in CFP2000 fails to build because GCC cannot find *.mod files that are generated on compilation. Just avoid using makeflags that way.
On ARM, compiler defaults to unsigned char. Add -DHAVE_SIGNED_CHAR to CPORTABILITY.
So far I have successfully ran all benchmarks in CPU2000 except 187.facerec under IRIX using GCC 4.5.3. It took quite some time to figure out the flags though. Here are my flags for IRIX using GCC 4.5.3:
| 176.gcc | CPORTABILITY = -DMIPS -DHOST_WORDS_BIG_ENDIAN |
|---|---|
| 178.galgel | FPORTABILITY= -ffixed-form |
| 186.crafty | CPORTABILITY = -DSGI |
| 252.eon | CXXPORTABILITY=-fpermissive -DHAS_ERRLIST |
| 253.perlbmk | CPORTABILITY = -DSPEC_CPU2000_SGI -DSPEC_CPU2000_NEED_BOOL |
| 254.gap | CPORTABILITY = -DSYS_IS_USG -DSYS_HAS_TIME_PROTO -DSYS_HAS_SIGNAL_PROTO -DSYS_HAS_IOCTL_PROTO -DSYS_HAS_ANSI -DSYS_HAS_CALLOC_PROTO |
Some benchmarks in CPU2000 and CPU2006 fall into an infinite loop unless
proper precision flags are passed when using GCC 4.5. For CPU2000, passing
-mpc64 is required for 32-bit x86 platforms. For CPU2006,
do NOT pass -mpc64 as it will cause some of the
CPU2006 benchmarks to fall in another infinite loop.
Build of 483.xanancbmk dies with:
FormatterToHTML.cpp: In member function 'void xalanc_1_8::FormatterToHTML::initCharsMap()': FormatterToHTML.cpp:139:42: error: 'memset' was not declared in this scope specmake: *** [FormatterToHTML.o] Error 1
and 447.dealII with:
quadrature.cc: In constructor 'Quadrature<dim>::Quadrature(const std::vector<Point<dim> >&)': quadrature.cc:64:26: error: 'atof' is not a member of 'std' specmake: *** [quadrature.o] Error 1
Both are caused by missing include files. GCC has a nice option for this problem that lets you compile these benchmarks without modification to source. Append ‘-include cstdlib -include cstring’ to CXXPORTABILITY, then problems go away (hopefully).
When compiling these benchmarks on Apple PowerMac G5 computer with MacOS X 10.[45] using
GNU GCC 4.5.2, I encountered this problem.
GCC 4.5 (with or without multilib support) changes its behavior when compiling with
option ‘-m32’
(it produces 32-bit binary with or without ‘-m32’).
When compiled with ‘-m32’, this problem goes away.
The OpenMP implementation of NAS Parallel Benchmarks version 3.3.1 also suffers from this problem.
Just add ‘-m32’ to CFLAGS and FFLAGS.
This problem was not
encountered when 64-bit binary is compiled using ‘-m64’ flag.
For some reason these processors need the -mieee-fp for 410.bwaves
when -O3 -march=athlon-mp optimization flags are used.
This problem is due to changes in a system header. You can avoid it with
-U__sun__.
254.gap invokes undefined behavior with signed integer overflow.
Append -fno-strict-overflow to portability flags.
For details see [bugs.launchpad.net].
ld complains a lot and fails to link 400.perlbench when using GCC5The linker ld complains a lot about duplicate definitions when using GCC 5 to compile 400.perlbench.
The default mode for C is now -std=gnu11 instead of -std=gnu89 [gcc.gnu.org].
Append -std=gnu89 to portability flags, and the problem goes away.
When compiling the spec tools for platforms unknown by the tool scripts (such as Tilera TILE-Gx), copying new config.sub file is necessary for the autotools to work. For CPU2006:
and for CPU2000:
Just google for the latest config.sub file. Just about any GNU package has the file.
Older version of perl included in SPEC tool does not work well with new platforms. On running make, it immediately complains that:
make: *** No rule to make target `<command-line>'
In this case you need to add a line in tools/src/perl-*/makedepend.SH:
-e '/^#.*<builtin>/d' \
-e '/^#.*<built-in>/d' \
-e '/^#.*<command line>/d' \
+ -e '/^#.*<command-line>/d' \
-e '/^#.*"-"/d' \
-e '/: file path prefix .* never used$/d' \
For dynamic linking (which is REQUIRED for running SPEC with their tools) to work you need to edit tools/src/buildtools script.
My favorite options to pass to perl Configure script are:
./Configure -dOes -Ud_flock $PERLFLAGS -Ddosuid=undef -Dprefix=$INSTALLDIR
-Dd_bincompat3=undef -A ldflags="-ldl -lm -L${INSTALLDIR}/lib" -A ccflags=-I${INSTALLDIR}/include
-Ui_db -Ui_gdbm -Ui_ndbm -Ui_dbm -Uuse5005threads
-Dcccdlflags="-fPIC -shared" -Dlddlflags="-shared -fPIC" -Duseshrplib=true ;
testordie "error configuring perl"
Perl references old Linux header asm/page.h which isn't there already.
Remove the reference by editing tools/src/perl-*/ext/IPC/SysV/SysV.xs.
Just remove lines #include'ing asm/page.h.
You need to pass -fPIC when compiling bzip2.
If you're using tools/src/buildtools script set environment variable BZIP2CFLAGS=-fPIC and everything works fine.
My dumb solution for specmd5sum not compiling right is to comment out lines in tools/src/specmd5sum/lib/getline.h. Just remove the getline and getdelim lines.
It's likely that you're the root user if the system you're benchmarking is an experimental one.
In that case tar fails to build (in config.log):
configure:35146: error: in `/path/to/cpu2006/tools/src/tar-1.25': configure:35149: error: you should not run configure as root (set FORCE_UNSAFE_CONFIGURE=1 in environment to bypass this check) See `config.log' for more details.
Try setting the said environmental variable, or just create a non-root acount.
perl complains You haven't done a "make depend" yet!
And when you manually run make depend in the last directory, it says:
../makedepend: 1: ../makedepend: Syntax error: Unterminated quoted string
This is because of a bug in the dash shell that are installed by default as /bin/sh in Debian and Ubuntu systems.
You can confirm that you are using dash by:
# ls -l /bin/sh lrwxrwxrwx 1 root root 4 Nov 8 23:09 /bin/sh -> dash
A workaround is to replace the symlink with another symlink to /bin/bash:
# dpkg-reconfigure dash