Samsung’s Bizarre Benchmark-Boosting Explanation

Samsung has denied allegations that it designed the international version of its Galaxy S4 smartphone to perform better in benchmark tests than in real-world use.

Responding to an investigation by AnandTech that found evidence that seemingly suggested that the Exynos 5 Octa-based Galaxy S4 uses a hard-coded white list for certain benchmarking apps that allows them — and only them — to take advantage of the full power of its chipset, Samsung denied that this is the case. “The maximum GPU [graphical processing unit] frequencies for the GALAXY S4 have been varied to provide optimal user experience for our customers, and were not intended to improve certain benchmark results,” the company said in a statement.

Interestingly, in issuing that denial, Samsung also confirmed AnandTech’s findings — albeit somewhat bizarrely.

“… A maximum GPU frequency of 533MHz is applicable for running apps that are usually used in full-screen mode, such as the S Browser, Gallery, Camera, Video Player, and certain benchmarking apps, which also demand substantial performance.”

So, here the company acknowledges that the Exynos 5 Octa-based Galaxy S4 does shift into a sort of high-performance mode for certain first-party apps and those benchmarking tools AnandTech noted in its investigation.

The thing is, none of those first-party apps use the GPU for anything appreciable. In a follow-up post responding to Samsung’s statement, AnandTech explains that S Browser, Gallery and Video Player all top out at a GPU frequency of 266MHz. And while the Camera app does hit 532MHz, it only does so infrequently, under duress (aggressively switching between filters) and for very, very short periods of time.

In other words, that “maximum GPU frequency of 533MHz” might be available to those first-party apps, but none of them really use it. Meanwhile, it’s also available to a number of third-party benchmarking tools that do use it, and deliver inflated scores as a result. Caveat: Those first-party apps may not be able to sustain 533MHz for a significant period of time, because they’re governed by the same thermal limitations as normal apps. Still … (If you’re really interested in this stuff, AnandTech’s analysis is definitely worth your time.)

So … Samsung says the Galaxy S4’s optimizations “were not intended to improve certain benchmark results.” But it also explicitly says that it has made the S4’s maximum GPU frequency available to certain benchmarking apps. Meanwhile, it ignores completely AnandTech’s discovery of what appeared to be hard-coded “BenchmarkBooster” exceptions, seemingly intended to produce high and repeatable performance benchmarks.

“Samsung uses the same tool to boost their in-house applications as they do the benchmarks,” Patrick Moorhead, Principal Analyst, Moor Insights & Strategy, told AllThingsD after reviewing Samsung’s statement and AnandTech’s follow-up. “The reality is, most apps won’t be boosted in a similar way, users won’t experience the same benefits, and therefore on the whole, these benchmarks of the Samsung phones in question don’t reflect real-world application performance.”

Quite the conundrum, and one that speaks more to the futility of using synthetic benchmarks to test smartphones and other devices than anything else.

“Mobile benchmarking needs real-world application benchmarks, and not the synthetic ones we see today,” Kevin Krewell, senior analyst at the Linley Group and a senior editor of Microprocessor Report, told AllThingsD. “Real-world application benchmarking will best inform the consumer of the experience they should expect to find on the device.”