A Closer Look at the DK2008 Review on 3DProfessor.org - Part 1

In my efforts to figure out what makes Diskeeper 2008 Pro Premier work, I thought I had found a useful review at 3D Professor.org, but there are aspects of it that are frankly misleading without further analysis. I think the testing was done in good faith, and there was no intention to mislead, but the method is sloppy and if this was an academic research paper subject to peer review it would not be approved for publication.
A lot of the tests are beyond my area of expertise, but one test in particular puzzled me. It involves the use of a disk testing utility called HD Tach from Simpli Software. What puzzled me about it is that, like HD Tune, it is used to calibrate the hardware performance of a hard drive at the sector level, and has nothing to do with the file system, and so it should show no performance change at all from the use of defrag software. When I downloaded a copy I noticed a section in the readme file that is relevant:

"SECTION 3 – HOW DO I RECORD GOOD BENCHMARKS?
Before benchmarking, reboot your computer. This will clear system caches and unload many unnecessary programs. Upon booting quit all programs that may have been loaded during the boot process, task bar icons, disable virus scanners. Advanced users should unplug the network cable and disable the swap file.
Give your system time to ‘stabilize’ – Windows does a lot of work in the background, so do not use HD Tach until you see your hard drive light stop blinking for 10-15 seconds. This may take one to two minutes after boot. During the benchmark do not move the mouse or press any keys."

The analysis of the tests imply that this was not done. To quote the Professor's report

"Going straight to the main points of the benchmark we ran the full test. As you are aware this benchmark has 3 types of test; short, long and full. The full test completely stretches the disc I/O. Average read speed in the full test increases by with Diskeeper 2007 installed by 1.2Mb/s with the actual burst speeds with nearly a complete 12Mb/s increase which is quite a lot to recover on a drive of this magnitude. With Diskeeper 2008 installed the results completely took our breath away."

What makes it worse is that the graph scale on the results changes from one test to another, and this was not pointed out. I have used Paint Shop Pro to correct the graphs published by the Professor, so that a visual comparison can be made.
This is the graph for Windows XP with no defrag.
This is the graph for Windows XP with Diskeeper 2008, adjusted to show a scale of 0 to 160, instead of 0 to 130 published here. The visual effect of the change in scale is quite dramatic, and appears to point to a huge change.
In fact the change is minor. The average read speed changes from 122.8MB/sec to 124MB/sec (1.6%), but the "burst speed" result goes from 278.6MB/sec to 650.9MB/sec. Clearly if this performance improvement of 134% is to be believed it should be repeatable. I believe it to be an error in testing methodology, because no hard drive on earth can run twice as fast by installing software. It defies the laws of physics.
I downloaded my own copy and ran the HD Tach benchmarks on my own laptop, with Diskeeper 2008 Pro Premier installed. In each case I ran the "short" test because the "full" test isn't available on my copy.
This is with Windows XP running all its services, discouraged by Simpli Software.
This is the same test with the bare minimum of services running, and with the mouse and network cables unplugged.
This is the same test, but run from a BartPE environment, which is a minimal version of Windows that runs from a boot CD.
Here is the same dive after reorganising the data completely using JkDefrag. As expected, the results are almost identical.
Finally, here is the BartPE test run after a complete defrag done by Diskeeper 2008. The "Go" button in the bottom left of the screen is part of the BartPE environment, and replaces the "Start" button. The average read is 30MB/s in each test except one, where it is 29.8MB/s, a variation of 0.6%, which doesn't really say anything. The burst speed remained the same thoughout.
The professor's testing in Part II does not make these mistakes, but no comment is made and no conclusions drawn. The tests were all run within a few days of one another.
My conclusion: the HD Tach test is completely irrelevant to the defrag process, because HD Tach is reading hard drive sectors, not NTFS files. The hard drive could be newly formatted and give the same result. How then can the 3D Professor conclude "... with Diskeeper 2008 installed the results completely took our breath away"? Unfortunately this is not the only flaw in the review published at 3dprofessor.org.

A Closer Look at the DK2008 Review on 3DProfessor.org: Part 1 | Part 2 | Part 3 | How Fast is I-FAAST™? | Diskeeper 2008 Professional: Preliminary Results | First Impressions | Diskeeper 2007 Review | Benchmarks: DK2008 and DK2007