The Embench group announces the first full release of its free and open source benchmark suite for IoT class devices. The benchmark suite, comprising 19 real programs, has proved popular across industry and academia since the pre-release version 0.5 was made available in February 2020. The project is developed by an open group, with contributors from companies and universities worldwide. The code is available under a free and open source license on GitHub (see github.com/embench/embench-iot).
Announcing the release, the Chair of the Embench Group, Prof David Patterson of Google and UC Berkeley said: “Embench 1.0 is a milestone for embedded benchmarks, leveraging the best ideas from benchmarking large systems and adapting them to the embedded ecosystem.”
One of the leading industrial users of Embench is Seagate Technology, Inc. “Seagate has been using the Embench benchmarks to measure the performance of our recently announced high-performance and area-optimized RISC-V cores,” said Seagate’s Engineering Director Richard Bohn. “We have found that the various workloads of the benchmarks provide us with useful guidance in the design tradeoffs among execution performance, memory footprint, and power estimations that we have to make in our RISC-V designs. Embench provides us with an excellent industry-standard, open-source, evolutionary set of benchmarks that enable us to make complex design decisions.”
Rice University has been one of the academic groups making major contributions to the development of Embench. Professor in the Practice, Ray Simar of Rice University said: “Without a doubt, Embench will have a positive, transformative, and lasting impact on the benchmarking of embedded microprocessors. Industry and academia have struggled to address this need in the past. This time, working together, we will succeed”.
The Embench project intends to make new releases every two years, to ensure the benchmark suite stays relevant and up to date.
The Embench group, part of the Free and Open Source Silicon Foundation, was set up by Turing Award winner, Prof David Patterson in late 2018. It’s goal is to provide best in class benchmarks as an open source collaboration between industry and academia. Priority has been given to using real programs and providing a statistically robust method of determining the benchmark score.
After two years of development by the collaborating business and universities, this is the release of the first full version of the code.
Embench is an exciting project and many people are curious about the history and destiny of it. We have interviewed the two founders of Embech, Dave Patterson and Jeremy Bennett, to learn more about it.
Why did you start Embench?
Dave Patterson: The embedded computing community suffers from terrible benchmark methodology. The old synthetic programs CoreMark (circa 2009) and Dhrystone (circa 1984) are by far the most widely quoted, despite widespread agreement in computer architecture that a single synthetic program can’t predict performance of real programs.
Perhaps this wasn’t a tragedy when embedded computing was a cottage industry using 8-bit microcontrollers, but the Internet of Things (IoT) market is now worth $1.7T and it is projected to grow by 3X to 6X by 2025 and cover everyone on the planet with increasingly sophisticated CPUs. They will likely be 95% of all 32- and 64-bit processors made in 2025.
Amazingly, in 2020 architects of billions of embedded computers must decide between designs that help real programs and designs that help marketing by accelerating these old, fake programs. It’s long past time to fix long-lasting embarrassment of indefensible benchmarking driving an increasingly important piece of the information technology industry.
What have you achieved and learned from the first year?
Jeremy Bennett: We’ve managed to get release Embench 0.5 published with very modest engineering resource and it has proved immensely popular. From emails I receive, I know many companies are using Embench in-house to evaluate processors, tools and libraries, even if they are not (yet) publishing results.
I have had my belief in the power of cooperation between industry and academia reinforced by the Embench experience. We each bring unique perspectives to the project, and the project is all the more successful as a result.
How well does the current Embench suite cover the shortcomings of previous benchmarks?
Dave Patterson: First, Embench is a suite of real (small) programs. A realistic workload is easier to represent as a suite than as a single program like CoreMark or Dhrystone. Also, a single program is hard to evolve. Third, real programs are more likely to be representative of important workloads than synthetic ones. Embench is a suite of 20 real programs, which makes it easy to drop the weak ones and add enhancements over time. Since it’s a suite, Embench reports a single summarizing performance score, along with the geometric mean and geometric standard deviation.
Unlike Dhrystone, Embench has a supporting organization (FOSSi Foundation) to maintain its relevance over time. Compilers and hardware change rapidly, so benchmarks need an organization that over time will deprecate pieces that are newly irrelevant and add missing features that speak to current challenges.
Jeremy Bennett: We are getting a reasonably amount of (mostly constructive) criticism, which will help the project. Much of it is about wanting benchmarks with larger programs than Embench 0.5 (which is aimed at small IoT class devices). We want that too - please join in and help us.
What are you currently working on?
Dave Patterson: We want to create a floating point benchmark suite. The target is again about 20 programs, at least partly from digital signal processing and machine learning.
Jeremy Bennett: Ray Sinar at Rice University and his graduate class are providing the first version of the floating-point Embench suite, which is targeted at machines with up to 1MByte memory.
What are your plans for the next 6 months?
Dave Patterson: We hope to complete and release Embench 0.6.
Jeremy Bennett: Much of my time is currently taken up as Chair of the Software Task Group for the Open Hardware Group. We are trying to build a vibrant commercial ecosystem for software providers in the RISC-V ecosystem.
But I still hope to find time to get to Embench release 1.0 and I’m looking forward to seeing the new floating point benchmark suite.
Where do you need help most urgently and how can people get involved?
Jeremy Bennett: There is some cleaning up needed around the existing IoT benchmark suite. However the big areas where we need help are: 1) more assistance alongside Ray Sinar with floating point benchmarks; and 2) someone to lead the development of an Embench suite for application class embedded devices - machines with lots of RAM running full-fat operating systems.
Plus looking forward, we want to renew the benchmark suites every two years, so we need people to put together new sets of programs.
Rick Merrit from EETimes has posted an article detailing the Embench call for support.
Embench aims to create a new benchmark suite aimed at evaluating micro-controller processors. It was founded by a small group led by Dave Patterson six months ago. Embench is based around a set of core ideas:
Benchmark suites must be available free of cost and available under an open-source license.
Benchmark suites must consist of a range of real programs, while providing a single summary number.
Benchmark suites must be regularly updated to stay relevant in a fast-changing world.
The full article can be found at EETimes.