The Great Defrag Shootout Round 2 - Benchmarks

The image above shows a snapshot of a newly-installed version of Windows XP SR2, with Internet Explorer 7 and .NET Framework 1.1 and 2.0 installed. There are plenty of fragmented files, many of which are caused by the downloaded security updates and patches. The yellow files are fragmented, and the start of the hard drive is the bottom left. The pink area is the reserved space for the Master File Table (MFT).
Here is a more familiar picture of the drive, using WDD. The red files are fragmented, and the MFT reserved space is just shown as empty space.
I have set up my test laptop, an IBM ThinkPad R31 with a 20GB hard drive, and using Acronis True Image Home 11 recovery CD I have made a protected partition for backups, and a main NTFS partition (C:) of 8,497 MB for Windows. There is a smaller partition of around 1GB just for storing results. The C: partition is backed up and restored using a "sector-by-sector" method, so that I can reproduce the messy arrangement of the drive whenever I want to run a new set of tests.
I have installed the following software on the basic test machine:
The idea is to create a "typical" workstation that reflects currently available software. Some of the utilities are used to take measurements, and others were used to set up the workstation the way I normally set up computers I install software on.

Testing the system

The data collection process works like this:

1. Restore the full drive image, including all fragments.
2. Install and updated version of prefetch.exe with the correct testing files.
3. Install the required test defrag program.
4. Reboot and run the first round of tests.
5. Allow the defrag program to do its best to defrag the drive. This may take several attempts and several reboots.
6. Reboot and run the second round of tests.
7. Install Microsoft Office 2007 Professional trial edition.
8. Reboot and run the third round of tests.
9. Open a test document as part of "normal usage"
10. Allow the defrag program to do its best to defrag the drive. This may take several attempts and several reboots.
11. Reboot and run the fourth round of tests.
12. Run HD Tune to test the hard drive.
    
13. Copy all the test results off the drive and analyse them.

Each "round of tests" consists of running Prefetch.exe, using this layout.txt file. This is followed by JkDefrag in "Analyse" mode, and storing a screen shot of the drive image, as well as the JkDefrag log file.
The idea is to emulate the normal process of software installation and use. I chose Office 2007 Professional (Trial Edition) because it is a large, legal install, creating numerous new folders and adding a lot of stuff to the system as a whole, including DLLs, fonts, and so on. The defrag program should be able to cope with these changes.
The current Prefetch.exe install also includes an installation of my "Delay Launch" program. A shortcut is placed in the Startup menu, and 240 seconds (4 minutes) after the system reboots it loads the "prefetch.exe" program. The program attempts to measure the read time of 842 files, 39 belonging to Office 2007, and the remaining 802 belonging to Windows and the other preinstalled software.
The benchmark is run at the same time after each reboot, to ensure consistency. If a program like Diskeeper is being tested, the Diskeeper service is disabled before the reboot, so the measurements are not affected by the software being tested.
I used HD Tune to test the speed of the drive under normal conditions after a reboot, and the graph is shown here. I will post the results as they become available, complete with the raw data in spreadsheet form, so others can examine the data and provide comments.

Spammer Wins Darwin Award

This has to be the dumbest spammer on the planet. Granted, he did manage to get through two anti-spam systems to my inbox, but that's where the cleverness ends. First of all, the "From" part is described as "VIAGRA ® Official Site" which is a dead giveaway. Does he think I really believe that Pfizer needs to do direct marketing? To add insult to injury the "From" address is my own address, making it pretty easy to set up an email rule to send all mail from me back to me into the "Deleted Items" folder. DUH!
But the clincher is the subject. A few months ago when the messages started, the discount was in the 50% range, and then increased to 60% and so on. Now the "discount" is going up by a few % a day. At this rate they'll be paying me to buy the stuff in early January. What kind of an idiot gives this kind of discount in the first place, and how long does he want me to hold out for a better discount?
I guess this is a severe case of "It is morally wrong to allow a sucker to keep his money", but with a twist. The sad thing is that someone, somewhere, is going to buy from this site, and wonder why the pills never arrive. Anyone want to help transfer money from Nigeria in order to buy the Brooklyn Bridge?

How Viagra Spam Works

Finally! Barclaycard Apologises

It took them long enough, but I eventually got an apology from Barclaycard for their illegal marketing activities and invasion of privacy.
The original offending letter is dated 26th July 2007, and arrived in the mail the following Monday. I complained to Barclaycard enquiries on 30th July 2007, i.e. 14 weeks ago.
This is the letter of apology I received today from Chris Sweeney, Managing Executive of Absa Card. His letter refers to a complaint of 22 October: clearly someone at ABSA has misled him to avoid being fired, since my complaint was not made 4 weeks ago, but 14 weeks ago. Even my complaint to the DMA was lodged in mid September.
I guess I should be thankful for an apology at all, even though no apology is made for buying my data from Experian/P-Cubed/Effective Intelligence, or for failing to check this data against the DMA "Do Not Call" list.

Part I: BarclayCard Breaks the Law | Part II: BarclayCard Bungles Again | Part III: BarclayCard and P:Cubed - more ducking and diving | Part IV: Finally! BarclayCard Apologises

The Mysteries of Defrag Testing

I have spent several days testing various defrag programs, and I'm now completely baffled by the results. (Shorter times are better.) I'm not surprised that PerfectDisk 8, Puran Defrag 3, and JkDefrag 3 all produced better results than a fragmented installation of Windows XP. What is baffling is why the other programs do worse, particularly Diskeeper and UltimateDefrag.
I have done my best to correct the results. I took measurements of 655 files, and then discarded all but 483 of them, because the numbers were too low in at least one set of results, implying that the file was being read from cache RAM, not the hard drive. What I don't understand is why the final WDD result would be so different from the first one. The "raw" data is shown below.
Clearly something else is changing. I did not do a complete Windows reinstall after each test, because of the problems with getting a sector-based backup to work. Perhaps that has something to do with it. The UltimateDefrag (UD) result was obtained using its "Auto" setting, and I'm sure that further experimentation with the other options will give a more realistic result.
Part of the problem could be that programs like Diskeeper and UD move "old" files to the slower part of the drive. Since the oldest files on the drive include a lot of WinXP system files, this may not be such a great idea. PerfectDisk puts newer files at the end, which would include new WinXP patches, but I haven't checked. It "knows" which files are used at startup, so this may account for a better performance than DK and UD. Perhaps I'll include the file dates in my next round of testing. In the meantime, I wouldn't give these results too much significance.
Update: I have found a replacement hard drive without sector errors, and installed Windows XP again. Tests are beginning with an updated version of the "Prefetch File Processor" program that will note the file creation date as well. I will run tests on most of the programs reviewed in this blog, and publish them as the results become available.

The Difference Even a Basic Defrag Can Make

We all love to criticise the built-in Windows Disk Defragmenter (WDD) program as inadequate, clunky or even just useless, but even the humble WDD can make a difference to your system. How about 32% faster read times? I was shocked.
Before we get to the results, a word of explanation. After doing a Windows XP install and running the many updates and reboots required by Microsoft Update, I deleted the contents of the "c:\windows\prefetch" folder, and rebuilt the "layout.ini" file using the "rundll32.exe advapi32.dll,ProcessIdleTasks" command. This required several reboots, but it eventually rebuilt. This data became the basis for the "layout.txt" file used by my new "Prefetch File Processor" program.
The image above shows the before and after effect of WDD being run several times on a newly installed system: Windows XP SR2, with IE7, .NET Frameworks 1.1 and 2, several small utilities, drivers and all relevant updates from Microsoft Update. The "before" situation was a mess: 1975 files were fragmented, and there were over 10 000 free space gaps.
I ran the Analyze and Defragment options once from the "normal" program, then ran "defrag c: -f" five times from the command prompt, and then the "normal" defrag once more. The image above is a composite of the first and last screen shots.
Now the results: my "Prefetch File Processor" program read the files in 14.587 seconds before the defrag, and 9.779 seconds after the defrag. Each time the measurement was taken at least 2 minutes after a reboot, with no other programs running. That's an improvement of 32.96% in the file read time. It's the first of a series of tests I am running that will include Diskeeper 2007, Diskeeper 2008, Puran Defrag 3.0, JkDefrag 3.28, and PerfectDisk 8. The tests aim to show the kind of performance improvement that can be obtained during boot time. it is only one benchmark, and it has its flaws. These will be discussed once all the tests have been run. Of course, it's not just the fragmentation that is an issue: file placement is important too. By moving all the files and directories to the other end of the disk, I doubled the read time to 29 seconds. It was an exercise in weirdness, but still it shows how badly performance can be affected by sloppy file placement.

A Closer Look at the DK2008 Review on 3DProfessor.org - Part 3

In trying to make sense of the 3D Professor review of Diskeeper 2008 I have examined the results and tried to make sense of them. The final set of tests uses Autodesk 3ds Max 9 SP2, a program that is used for 3D modelling, rendering and animation. This is a typical application for high-end graphics workstations. In the review, the professor states:

"[The] Radiosity test is one of the most disc intensive render taken from the SPECapc benchmark. It really does sort out the wheat from the chaff in system builds. The uptake in performance speaks volumes by themselves ... [and] is a substantial saving in time and money to the end user."

The chart shows the results for both Windows XP and Windows Server 2003. No explanation was offered as to why the CBALLS test was not run on the server. The performance improvement in the latter case is 1.9%, while on the XP workstation the claimed improvement is 15.8% for DK2007 and 26.4% for DK2008. Following the logic outlined in Part 2, all this really tells me is that DK2008 is 12.5% more effective than DK2007, and not much more.
The performance improvement of 1.9% on the server is ascribed to the fact that RAID drives are slower. But the previous test had a 6% performance gain, so the hard drive activity must have been enormous, since the graphics rendering speeds on both systems are almost identical.
On the other hand, the average read speed of the XP machine's hard drive is 124MB/sec, but the server's RAID system clocked only 52.9MB/sec. One would therefore expect the server's times to be 58% slower, yet they are faster, with or without DK2008. It doesn't make sense, and no explanation is provided.

Fawning Introduction

The introductory paragraph of this "independent review" could have been written by the same spin doctors who write the Diskeeper web site:

This is an exciting time for the crew at Diskeeper as this new product release brings with it an abundance of new features which enhance its elder of Diskeeper 2007. For years now we have been using the product exclusively primarily down to the fact that it maintains good maintenance within system performance. For over a decade, Diskeeper has been synonymous in setting new standards in the way that Microsoft Windows system defragmentation technology performs. In their true professional form, Diskeeper 2008 continues with its tradition of a true “Set It and Forget It®” performance enhancement that is the hallmark of the Diskeeper name.

Compare that with the following sentence

"For over a decade, Diskeeper has led the way in Windows system defragmentation technology, and Diskeeper 2008 continues with the tradition of true 'Set It and Forget It' performance enhancement that is a hallmark of the Diskeeper name"

which comes from the "What’s New in Diskeeper 2008" section (Chapter 4) of the "Diskeeper 2008 Administrator User's Manual".
In the "Product Enhancements" section of the review, he quotes 7 paragraphs directly from the same manual, without any reference to its source. I had a sense of deja vu reading this section, because I'd already encountered the same phrases in the DK2008 help file, and found them puzzling there too. If this was an academic paper the issue of plagiarism (or at last bad referencing) would be raised.

Selective Conclusions

The Conclusions drawn by the professor are highly selective. Take this paragraph, for example:

"The team at Diskeeper most certainly have an exciting new product that in most areas showed up to and in certain applications a 28% increase in performance over a Windows system with no defragmentation tool installed. Up to 14% increase in performance in certain areas over Diskeeper 2007 which is quite a remarkable achievement today, the development team from Diskeeper must be jumping around with glee here!"

The best result achieved is the radiosity test for XP is 28.8%, but it is hardly a representative sample of all the tests done, and no further mention is made of the HD Tach results ".. which took our breath away". How about explaining the other results as well?

Exclusive Review and Timing

I'm not complaining that the review could be published on the same day the product was launched. The beta testing phase closed on 3rd October, and from the HD Tach results we know that testing was done between 7th October and 12th October, in time for the launch date of 23rd October. What worries me is that no explanation if offered for this situation, nor is there any mention that the tests were done using the beta version of DK 2008 (The final build date of version 12.0.758.0 is 16 October 2007, 20:04:12).
When I stated in another posting that it was not clear whether DK had sponsored the review or not, I got the following email from Celine Bouquillon at Diskeeper Europe:

"I wanted to point out that the review published in "3dprofessor.org" has absolutely not been paid for by our company. And he is not affiliated to us in anyway. He is just a big fan of our product :-) If I have sent the link to you, it is precisely because he is a third party, as you are. It would not have made sense for me to send it to you if it was not the case. May I ask you to do some research on such claims before publishing them?"

I must therefore take it on face value that this is the case, but the reviewer has not been scientific or independent. Given my rants about DK 2007 I can hardly be the one to expect the professor to be impartial or objective, but when material is copied and pasted from the help file, and irrelevant tests are performed that prove nothing but still manage to "take our breath away", I start questioning the usefulness of the review in its entirety.

My conclusions: the 3D Professor is a well-meaning hobbyist who ran a battery of standard tests used for all his reviews, without much thought to their relevance or scientific merit. His enthusiasm for the product is clear, but the only valid conclusions that can be drawn from the tests are that the automatic defragmentation in DK2008 gives faster results than DK2007. The value of the review is diminished by hype and lack of clarity. I am not convinced that the correct tests were carried out.

A Closer Look at the DK2008 Review on 3DProfessor.org: Part 1 | Part 2 | Part 3 | How Fast is I-FAAST™? | Diskeeper 2008 Professional: Preliminary Results | First Impressions | Diskeeper 2007 Review | Benchmarks: DK2008 and DK2007

Testing ... Testing ... 1 2 3

How do you test whether a given defrag program has sped up your system or not? It's all very well to look at the drive layout and say "this looks neat", but how can you tell if the machine boots faster or programs load faster?
While I was analysing the "review" by the 3d Professor, I tried to think of a method of objectively testing the performance of the file system, as improved by a given defrag program. I tried gathering data from the readfile program, but the maths doesn't work correctly. I'm not sure why, and I don't understand the C source code enough to figure it out. Also, readfile only accepts a single wildcard, so you can do *.* or *.exe but not *.exe;*.dll;*.sys for example.
I did an exercise in measuring the time it took to read all the files in the c:\windows\system32 directory, and the results were almost what I expected, but it still wasn't an accurate enough reflection of how a defrag program affects the performance of the system as a whole. Clearly there are some files in the system32 folder that are seldom if ever opened or run, and if the system was optimised for files that are used often, these slower files would skew the results.
So I gave up on readfile and decided to write my own program in Visual Basic 6. It's called "Prefetch.exe" or the "Prefetch File Processor" and I'm busy in the initial stages of running tests on an old ThinkPad R31 laptop. The program is freeware and you can examine all the source code as well to see in detail how it works. The idea is that the results should be repeatable on any given system, subject to the limitations of the package.
The program has three stages:

• In the first stage the "layout.ini" file is interpreted and checked and the results copied to a "layout.txt" file, which can be edited.
• In the second stage this file is opened and each file name it contains is opened and read, and the time this process takes is calculated in milliseconds.
• In the third stage the results are saved in a CSV file for further analysis and checking.

There are some limitations to this process. The biggest limitation is that Visual Basic 6 can only read the first 2,147,483,646 bytes (2GB) of any given file. Usually that's enough, and I'm not that interested in huge data files, so it's good enough. The second limitation is that certain system files are opened and locked by the system, so these can't be timed. This is a limitation, but equal across all programs tested. The third limitation is that it can't test the time it takes to open a folder, only a file.
Another problem is that the "layout.ini" file includes some junk files and it keeps changing. That's why I created the "layout.txt" file, and edited it to remove files in the temporary folders, references to cookies, log files, critical updates, and other miscellaneous junk. The test file I am using is here, and hopefully it will deliver a fair test of the system.
The timing starts just before the file is opened, and finishes when the file is closed. The timer works in msecs, and does not include the time it takes to read the name of the file from the "layout.txt" file, or the time to save the result in the "timing.csv" file. Later I may extend the program to do some read-write tests, but for now it's read only.

My Test System

I have created a 5.86GB partition on Penny's old ThinkPad R31 laptop. With Windows XP Professional and IE7 and the .NET 1.1 and 2.0 frameworks and all service packs loaded, there is 1.86GB free space, i.e. 32% free. I then made a full sector-by-sector image backup using the Acronis True Image Home 11 Recovery CD, and this image is stored on another partition of the disk that is not included in the testing. This backup image is fragmented, exactly as it was created during the install process, with absolutely no attempt to defragment the drive in any way. Here are some facts about the system, as reported by JkDefrag in "analyse" mode:

Total disk space: 6,293,757,952 bytes (5.86 gigabytes), 1,536,562 clusters
Bytes per cluster: 4,096 bytes
Number of files: 20,110
Number of directories: 2,464
Total size of analyzed items: 4,268,003,328 bytes (3.97 gigabytes), 1,041,993 clusters
Number of fragmented items: 1,342; 5.94% of all items
Total size of fragmented items: 952,217,600 bytes, 232,475 clusters, 22.31% of all items, 15.13% of disk
Free disk space: 1,262,436,352 bytes, 308,212 clusters, 20.06% of disk
Number of gaps: 4,722
Number of small gaps: 3,978; 84.24% of all gaps
Size of small gaps: 78,368,768 bytes, 19,133 clusters, 6.21% of free disk space
Number of big gaps: 744 (15.76% of all gaps)
Size of big gaps: 1,184,067,584 bytes, 289,079 clusters, 93.79% of free disk space
Average gap size: 65.27 clusters
Biggest gap: 926,806,016 bytes, 226,271 clusters, 73.41% of free disk space

Not chaotic, but hardly optimal. Each program tested will be given a chance to defragment and optimise this data, and once it has done its best, the system will be rebooted and the Prefetch File Processor will read the list of files and time the process. I will also record screen shots of the drive image before and after, and use the JkDefrag analyse log file to note other aspects of the defrag process.
I would welcome any comments or criticisms of this process, and feel free to download the program and run your own tests, and inspect the Visual Basic 6 code. I will document it more fully in the next few days, so the code is easier to read.
Update: My TrueImage backup file is corrupt, and I have to reinstall everything. I'll update the numbers published above once this has been done. Using the WDD defragger improved the read times by 15%, but I will re-run the tests and publish the results in full.
Update: For technical reasons associated with a large bad spot on my drive, I can't do a sector-based backup, only a complete file backup. This is going to complicate matters slightly, but hopefully the results won't be too skewed. It's a lot of work reinstalling XP, not to mention tons of bandwidth during updates.

Time Magazine Gets Sucked Into Reality Distortion Field

I'm glad that Time's international editions managed to retain a little more perspective than the US Edition, but the hype about the iPhone is getting completely out of hand.

"The iPhone is more than just a gadget. It's a genuine handheld computer, the first device that really deserves the name." - Time Magazine

What kind of nonsense is this? It's no more of a computer than a Palm or Blackberry, or for that matter, my Nokia bottom-of-the-range phone. Or does the writer mean that because the phone can be hacked that it is now a "real" computer? No one needed to hack the others to install apps! Only on the iPhone - only from Apple. I guess Time fails this week's gullibility test.