Wednesday, November 28, 2007

Benchmarks*: Why the Asterisk?

You may have noticed that all the "benchmark" results posted have an asterisk next to them. Why?
The main reason is that these results are preliminary. I am publishing them on this blog as they arrive, in the order they arrive, and asking for comment. This is to avoid any accusation of "cooking" the results to favour a particular product. They are interesting in and of themselves, but they don't prove anything.
The second reason is that they are incomplete. The tests measure the "read time" of 800 files, all listed in the layout.ini file in Windows XP. These are not user data files, and don't tell you what you are likely to experience when using your PC on a regular basis. The comments received will assist in developing additional benchmark tests, in order to arrive at a more complete picture.
Some of the comments and criticisms received are valid and need to be taken seriously. One of the criticisms is that the timing program itself may be generating inaccurate results, and its one I am checking out as carefully as possible.
Another problem is that I do not know how accurate these results are: i.e. how does one calculate the critical margin of error? Is it 0.1%, 1% or 10%? Clearly this would make a big difference to how the results should be interpreted.

I therefore urge all readers not to base any purchasing decisions on these results until all the testing has been completed. In the meantime, try the products for yourself, because your experience may be completely different to the test machine.

3 comments:

Anonymous said...

Prof D,
How's about one of Your splendid roadtests on 'Defraggler '?
http://www.defraggler.com/features
cheers

Donn Edwards said...

It's on the list for a Benchmark, and I'll wait until the first public release for a review. At the moment it's in beta.

Anonymous said...

I love these benchmarks you are doing. And indeed as you already point out your results are just temporary. My main comment is that you are taking a single point in time to measure the products. As you are doing these in your own time, this not a criticism, just a reflection on the validaty of your results.
Because for me the best defragger is not an utility that will optimize the disk just after doing an installation, or once a week. It needs to be a tool that will move the files I use daily to the optimum location of the disk. And there are 3 criteria: Fast boot-up, Fast shutdown, and fast access to the 'program files' and 'data' I use most often.
This indeed means moving archives, lots of unused Windows windows dll's, uninstall files, etc. to the slowest part of the disk to make room for the files one needs most often on the fast part of the disk.

I would also like that I can install a tool, set it to Auto and have the tool continuously analyze my disk usage, and optimize the disk accordingly. Sort of fire and forget.
So Diskkeeper seems to promise that (i-Faast, InvisiTask), but its very obscure in what and how it optimizes the disk. DiskTrix is very clear on how and what it can optimize, but just too many configuration options and I still need to schedule it, and analyze my own disk usage. I'm now trying JKDefrag, but helas here also, it needs to be scheduled.

Ideally, this tool would just be in Windows, but MS just doesn't get it, it has invented some band-aid solutions like 'ReadyBoost'.