Benchmarks
When a new device (especially a company's flagships) are announced or rumored, everyone seems to be obsessed over every bit of specification.
And if the device doesn't offer the latest technologies thoroughly, we quickly lose our interest and the whole excitement goes away. If it does however, then every single mobile blog and website inevitably goes through every single benchmark application created for one purpose only - to squeeze the last bit of juice from its components. After all we want our devices to perform to the highest possible levels right? |
So they run AnTuTu, Quadrant Standard, Geekbench, Smartbench, Linpack, and more, and run multiples tests of every benchmark and run it again, until they've got an average score out for the full review. I know it because I was running benchmarks for the same purpose in my previous job.
Quite literally, for hours with the phone, you keep pressing a Begin test button, waiting for the test to finish, record (screen shot) the result, and start over. Some benchmark tests take a split-second, such as Linpack. Others take upward of four hours, like the AnTuTu Tester battery benchmark, not fun at all.
However, the more you test devices, the more times you hit the Test again button, the more you see how useful benchmark tests truly are, how well they correlate with real world performance. And believe me, its not as often as you would wish it would be. Benchmarks and the real world performance don't go in pair.
Quite literally, for hours with the phone, you keep pressing a Begin test button, waiting for the test to finish, record (screen shot) the result, and start over. Some benchmark tests take a split-second, such as Linpack. Others take upward of four hours, like the AnTuTu Tester battery benchmark, not fun at all.
However, the more you test devices, the more times you hit the Test again button, the more you see how useful benchmark tests truly are, how well they correlate with real world performance. And believe me, its not as often as you would wish it would be. Benchmarks and the real world performance don't go in pair.
So who is it for?
For people who read this stuff and get exited and plan their next purchases based on the synthetic benchmarks.
It is true that without the benchmarks, we wouldn't be able to scientifically measure the rough power of our devices. If we based it only on our daily experience with the phone, its smooth operations, transitions, speed and overall performance, every single modern smartphone would be more than competent for everyone's needs. If you hear people boasting about how powerful their new phones are, judging from the benchmarks and scores, then don't waste your time to get involved. Their knowledge is very narrowly limited, and simply wrong. If they knew that those benchmarks are being carried in the perfect conditions, with the bare software, with the handsets restored to manufacturer settings every so often, they would understand that bragging about their phones is simply stupid. They should also know that having those perfect conditions, the tests are being repeated numerous times, to get just average score, because every time you run the test you get a different, often seriously different results. Also with Android, one of the toughest challenges for benchmark developers is maintaining support for various chipsets, software versions, and new processor technologies, which basically means that the benchmarks are often not optimized yet at the time the new phone is going live.
For people who read this stuff and get exited and plan their next purchases based on the synthetic benchmarks.
It is true that without the benchmarks, we wouldn't be able to scientifically measure the rough power of our devices. If we based it only on our daily experience with the phone, its smooth operations, transitions, speed and overall performance, every single modern smartphone would be more than competent for everyone's needs. If you hear people boasting about how powerful their new phones are, judging from the benchmarks and scores, then don't waste your time to get involved. Their knowledge is very narrowly limited, and simply wrong. If they knew that those benchmarks are being carried in the perfect conditions, with the bare software, with the handsets restored to manufacturer settings every so often, they would understand that bragging about their phones is simply stupid. They should also know that having those perfect conditions, the tests are being repeated numerous times, to get just average score, because every time you run the test you get a different, often seriously different results. Also with Android, one of the toughest challenges for benchmark developers is maintaining support for various chipsets, software versions, and new processor technologies, which basically means that the benchmarks are often not optimized yet at the time the new phone is going live.
A single application installed on your phone, chosen settings, layout, widgets and so on, would badly affect your benchmarks. But even if we assume that our tests are conclusive, the performance of your phone in real life if far from the score it got in tests.
Recently I've come across a handful high-powered handsets. Courtesy to my Vodafone friends, I've tested them for a whole day - the Optimus G Pro, HTC One and Samsung Galaxy S4. They chew synthetic benchmarks up and spit them out with the most impressive results. Despite those scores on G Pro and S4, I experienced stuttering and tiny hints of lag throughout the usage. The HTC One, on the other hand, powered by the same chipset (lower clock speed), managed to withstand similar abuse with no trace of lag.
Recently I've come across a handful high-powered handsets. Courtesy to my Vodafone friends, I've tested them for a whole day - the Optimus G Pro, HTC One and Samsung Galaxy S4. They chew synthetic benchmarks up and spit them out with the most impressive results. Despite those scores on G Pro and S4, I experienced stuttering and tiny hints of lag throughout the usage. The HTC One, on the other hand, powered by the same chipset (lower clock speed), managed to withstand similar abuse with no trace of lag.
And there is another thing worth consideration. Cheating on benchmarks – almost everyone is doing it as it turns out. Manufacturers came with a clever method of cheating the benchmarks. It doesn't involve boosting the CPU or GPU speed – all it does is disable the lower clock speeds and sleep modes on all CPU cores. It means that when the device detects a benchmark on its list, all CPU cores are locked at their max frequency, while if the benchmark is renamed (so that the device doesn't detect it), there are natural drops in CPU frequency and efficiency.
Again, we need the benchmarks to check the progress of the hardware, but relying on it as a way to show superiority of the phone is simply stupid.
If you ever get into argument with this kind of "techy guy", I hope you'll find the right arguments. On the other hand...is it really worth to talk to such guys.
If you ever get into argument with this kind of "techy guy", I hope you'll find the right arguments. On the other hand...is it really worth to talk to such guys.