A Closer Look at the DK2008 Review on 3DProfessor.org - Part 3

In trying to make sense of the 3D Professor review of Diskeeper 2008 I have examined the results and tried to make sense of them. The final set of tests uses Autodesk 3ds Max 9 SP2, a program that is used for 3D modelling, rendering and animation. This is a typical application for high-end graphics workstations. In the review, the professor states:

"[The] Radiosity test is one of the most disc intensive render taken from the SPECapc benchmark. It really does sort out the wheat from the chaff in system builds. The uptake in performance speaks volumes by themselves ... [and] is a substantial saving in time and money to the end user."

The chart shows the results for both Windows XP and Windows Server 2003. No explanation was offered as to why the CBALLS test was not run on the server. The performance improvement in the latter case is 1.9%, while on the XP workstation the claimed improvement is 15.8% for DK2007 and 26.4% for DK2008. Following the logic outlined in Part 2, all this really tells me is that DK2008 is 12.5% more effective than DK2007, and not much more.
The performance improvement of 1.9% on the server is ascribed to the fact that RAID drives are slower. But the previous test had a 6% performance gain, so the hard drive activity must have been enormous, since the graphics rendering speeds on both systems are almost identical.
On the other hand, the average read speed of the XP machine's hard drive is 124MB/sec, but the server's RAID system clocked only 52.9MB/sec. One would therefore expect the server's times to be 58% slower, yet they are faster, with or without DK2008. It doesn't make sense, and no explanation is provided.

Fawning Introduction

The introductory paragraph of this "independent review" could have been written by the same spin doctors who write the Diskeeper web site:

This is an exciting time for the crew at Diskeeper as this new product release brings with it an abundance of new features which enhance its elder of Diskeeper 2007. For years now we have been using the product exclusively primarily down to the fact that it maintains good maintenance within system performance. For over a decade, Diskeeper has been synonymous in setting new standards in the way that Microsoft Windows system defragmentation technology performs. In their true professional form, Diskeeper 2008 continues with its tradition of a true “Set It and Forget It®” performance enhancement that is the hallmark of the Diskeeper name.

Compare that with the following sentence

"For over a decade, Diskeeper has led the way in Windows system defragmentation technology, and Diskeeper 2008 continues with the tradition of true 'Set It and Forget It' performance enhancement that is a hallmark of the Diskeeper name"

which comes from the "What’s New in Diskeeper 2008" section (Chapter 4) of the "Diskeeper 2008 Administrator User's Manual".
In the "Product Enhancements" section of the review, he quotes 7 paragraphs directly from the same manual, without any reference to its source. I had a sense of deja vu reading this section, because I'd already encountered the same phrases in the DK2008 help file, and found them puzzling there too. If this was an academic paper the issue of plagiarism (or at last bad referencing) would be raised.

Selective Conclusions

The Conclusions drawn by the professor are highly selective. Take this paragraph, for example:

"The team at Diskeeper most certainly have an exciting new product that in most areas showed up to and in certain applications a 28% increase in performance over a Windows system with no defragmentation tool installed. Up to 14% increase in performance in certain areas over Diskeeper 2007 which is quite a remarkable achievement today, the development team from Diskeeper must be jumping around with glee here!"

The best result achieved is the radiosity test for XP is 28.8%, but it is hardly a representative sample of all the tests done, and no further mention is made of the HD Tach results ".. which took our breath away". How about explaining the other results as well?

Exclusive Review and Timing

I'm not complaining that the review could be published on the same day the product was launched. The beta testing phase closed on 3rd October, and from the HD Tach results we know that testing was done between 7th October and 12th October, in time for the launch date of 23rd October. What worries me is that no explanation if offered for this situation, nor is there any mention that the tests were done using the beta version of DK 2008 (The final build date of version 12.0.758.0 is 16 October 2007, 20:04:12).
When I stated in another posting that it was not clear whether DK had sponsored the review or not, I got the following email from Celine Bouquillon at Diskeeper Europe:

"I wanted to point out that the review published in "3dprofessor.org" has absolutely not been paid for by our company. And he is not affiliated to us in anyway. He is just a big fan of our product :-) If I have sent the link to you, it is precisely because he is a third party, as you are. It would not have made sense for me to send it to you if it was not the case. May I ask you to do some research on such claims before publishing them?"

I must therefore take it on face value that this is the case, but the reviewer has not been scientific or independent. Given my rants about DK 2007 I can hardly be the one to expect the professor to be impartial or objective, but when material is copied and pasted from the help file, and irrelevant tests are performed that prove nothing but still manage to "take our breath away", I start questioning the usefulness of the review in its entirety.

My conclusions: the 3D Professor is a well-meaning hobbyist who ran a battery of standard tests used for all his reviews, without much thought to their relevance or scientific merit. His enthusiasm for the product is clear, but the only valid conclusions that can be drawn from the tests are that the automatic defragmentation in DK2008 gives faster results than DK2007. The value of the review is diminished by hype and lack of clarity. I am not convinced that the correct tests were carried out.

A Closer Look at the DK2008 Review on 3DProfessor.org: Part 1 | Part 2 | Part 3 | How Fast is I-FAAST™? | Diskeeper 2008 Professional: Preliminary Results | First Impressions | Diskeeper 2007 Review | Benchmarks: DK2008 and DK2007