Applying the Roofline Model

The roofline model includes two platform-specific performance ceilings: the processor’s peak performance and a ceiling derived from the memory bandwidth, which is relevant for code with low operational intensity. The model thus makes more precise the notions of memory- and compute-bound and, despite its simplicity, can provide an insightful visualization of bottlenecks. As such it can be valuable to guide manual code optimization as well as in education. The picture below sketches such a roofline plot for a platform with peak performance π = 2 flops/cycle and bandwidth β = 1 bytes/cycle.

Different levels of BLAS and FFT from Intel MKL on a Core i7-3930K (Windows), single threaded

We run a set of kernels that are known to have different operational intensities: the BLAS 1–3 functions and the FFT. The figure above shows the performance of the set of kernels when executed sequentially on a Core i7-3930K running Windows as OS. The set of input sizes n is chosen to use different levels of the memory hierarchy. For daxpy we use n = 10000 + 30000i*i, (0 <= i < 10), for dgemv and dgemm n = 100 + 300i (0 <= i < 10), and for FFT, n = 2k (5 <= k < 23). Daxpy and dgemv are memory- bound as expected, reaching about 95% and 90% of the peak. Dgemm is compute-bound for all sizes reaching also about 95% of the peak p. Note that the FFT is plotted with measured flops and not with pseudo- flops (fixing the op count to 5n log (n), an overestimation), as commonly done.

Comparing various FFT implementations on a Xeon X5680 (Linux)

We compare various FFT kernels in the figure above. As in the previous plots, we use sizes n = 2k (5 <= k < 23), except for the Spiral-generated kernels where we use an upper bound of 4096 (the code aviable via the software section of the spiral.net page). The order of the kernels, in terms of performance, is roughly as expected, with MKL and one version of Spiral using the SSE instruction set, and the rest of the implementations being unvectorized. The only surprise is the performance of the straightforward Numerical Recipes code. The Numerical Recipes code seems to perform surprisingly well, due to the fact that it calculates the twiddle constants at runtime for all sizes, resulting in an increased flop count and thus increased performance and operational intensity.

Comparing different optimization levels of MMM on a Xeon E5-2660 (

The figure above shows the roofline plot for the dgemm kernel at different levels of optimizations. In addition to ATLAS and MKL we consider a hand-coded standard triple loop and a six-fold loop obtained through blocking with block size 50. The MKL and ATLAS kernels provide improved ILP, SIMD vectorization, and multi-thread parallelism.