2.6 Benchmarks and Spec

COS2621 - Computer Organisation · 2. Performance Issues

Benchmarks and Spec

Understanding Benchmarks

Benchmarks are standard tests used to measure the performance of a computer system. They provide a way to compare different systems or components. Benchmarks can be synthetic, meaning they simulate workloads, or real-world, based on actual applications.

For example, a synthetic benchmark might measure how quickly a processor can perform a specific mathematical operation. A real-world benchmark might measure how quickly a computer can render a video or compile code.

Types of Benchmarks

Benchmarks can be categorized into several types:

  • Micro-benchmarks: Focus on specific components, like CPU or memory performance.
  • System benchmarks: Assess overall system performance, including CPU, memory, and I/O.
  • Application benchmarks: Evaluate performance based on specific applications, like databases or graphics software.

SPEC Benchmarks

The Standard Performance Evaluation Corporation (SPEC) is a well-known organization that develops benchmarks. SPEC benchmarks are widely used to evaluate the performance of various computer systems. The most recognized SPEC benchmark suite is SPEC CPU, which measures the performance of the CPU using a set of compute-intensive tasks.

For example, SPEC CPU might include tests that measure how fast a CPU can execute floating-point calculations or handle integer operations. The results are typically reported in a score, which allows for comparison between different systems.

Calculating Benchmark Scores

To calculate a benchmark score, you must run the benchmark on the system and record the time taken to complete each test. The score is often calculated as follows:

Score = (Reference Time / Measured Time) x Reference Score

For example, if a system takes 50 seconds to complete a benchmark that has a reference time of 100 seconds and a reference score of 1000, the calculated score would be:

Score = (100 / 50) x 1000 = 2000

Watch Out

Watch out: Ensure that you are using the correct reference time and score when calculating benchmark scores. Incorrect values will lead to inaccurate results.

Using Benchmarks for Comparison

When comparing systems, it is important to use the same benchmark suite. Different benchmarks can yield different results based on the specific workloads they test. For example, a system that performs well on CPU-intensive tasks might not perform as well on I/O-intensive tasks.

Additionally, consider the context in which the system will be used. A high benchmark score might not correlate with better performance for your specific applications.

Summary

  • Benchmarks are essential for measuring and comparing computer performance.
  • SPEC benchmarks are widely recognized and provide standardized testing methods.
  • Calculating scores requires accurate reference times and scores.
  • Always use the same benchmark suite for fair comparisons.

Check your understanding

  • What are the different types of benchmarks?
  • Explain how to calculate a benchmark score.
  • Why is it important to use the same benchmark suite when comparing systems?

Test yourself

4 quick questions on this lesson. Pick an answer to see if you're right.

  1. 1. What is the primary purpose of benchmarks in computer performance evaluation?

  2. 2. When calculating a benchmark score, what is the formula used?

  3. 3. What does SPEC stand for in the context of benchmarks?

  4. 4. Which of the following is NOT a type of benchmark?