Saturday, November 13, 2010

Flaws in Quadrant demonstrate the need for care when analyzing benchmarks

Android benchmark enthusiasts, check out this article: http://briefmobile.com/cyanogen-demonstrates-quadrants-flaws

I'm not sure how I've missed this article, but it's quite interesting. I knew that the problem with the Quadrant I/O performance in the Galaxy S phones (causing them to score significantly below phones like the Droid X, which do not have as powerful a CPU or GPU) could be remedied on the I9000 Galaxy S and the Captivate using a fix from XDA developers.

The above article demonstrates however that the issue may go beyond the way the file system is set up in the Galaxy S. While the fix does produce actual performance gains on the hardware, phones like my Epic 4G apparently have no need for the fix. While Quadrant scores are still affected, Samsung appears to have made changes to the file system on the Epic 4G to allow for quicker loading of data from NAND flash instead of the SD card.

TL;DR - An I/O bug in other Galaxy S phones was rectified in the Epic 4G, but the performance gain is not reflected by Quadrant benchmarks.

1 comment:

  1. The point is that how do you get benchmarks reflective of real world performance? It is easy I suppose to get a game and measure the FPS - do it for a variety of games and you should have a pretty good idea of how fast its going to be. It seems that these benchmarks don't often reflect what you get in the real world.

    ReplyDelete