Windows Disk Defragmenter 5.1.2600.5512 in the 2009 Defrag Shootout

Windows Disk Defragmenter 5.1.2600.5512 is part of Windows XP, and is often known as WDD or just defrag.exe. When comparing the performance of a fragmented XP system and the defragmented one, the results are roughly as expected: in most cases the system performs faster, with a top improvement of 25% faster, an average around 12% faster, and a worst case of 0.83% slower.
I tested all 4 defrag options: the graphic interface (shown above), and 3 command-line options:
defrag c: -f -v
defrag c: -b
rundll32.exe advapi32.dll,ProcessIdleTasks
The first forces a defrag (-f), and gives statistics of the state of the file system (-v). The second (undocumented) option (-b) does a "bootup" defrag, where it attempts to optimise the system. The last one sets off a "boot time" defrag option, similar to the second, but it occurs at boot time on every 3rd reboot or so. I created a batch file that included all 3 options, and let it run for several hours, since WDD is a multi-pass defragger: it doesn't get everything done on the first pass.
I used CCleaner to remove old deleted files, tidy up the registry, clear the temporary Internet files, and generally fix up the system. This made a number of empty spaces near the start of the drive, and allowed the defrag process to optimise the system.
The benchmark results for the reference WinXP system are available in a PDF file, and the results of the WDD tests are also available.
Probably the most useful results are the time it takes to open a Word 2007 document and an Excel 2007 spreadsheet. The Excel spreadsheet improved from 9.45 seconds to 8.48 seconds, or 10% faster. Similarly the Word document time dropped from 17.82 seconds to 16.66 seconds, or 6.5% faster. The FileAccessTimer results show a 24% and 18% improvement on data files and program files, and the IOZone result is 25% faster. This last result depends on the availability of free space for temporary files. Even though the free space is in the slower half of the drive, the lack of fragmentation shows up in faster response times.
The SpecViewPerf tests showed only slight performance changes, and the Access Stress Test result was marginally slower. The AVG full computer scan result was only slightly faster. I would have expected it to be faster because all the files are now defragmented and better positioned. It remains to be seen whether other defrag programs have any effect on these benchmarks.
Conclusions: About the only safe conclusion we can draw is that the system runs the same or faster when the system is defragged from time to time. Perhaps when a number of other results have come in, we can start drawing more detailed conclusions.

Windows Disk Defragmenter gets the stress test

Don't expect any test results for a few days, because this is what the Windows XP Disk Defragmenter (WDD) is having to deal with: 75% full hard drive with a lot of fragmented files, both large and small, including big compressed files with lots of fragments.
The test PC has been set up to look like a home PC that hasn't seen a defrag in its life before. This isn't too far off the mark. Since WDD is supplied with WinXP, it gets to go first. Then I will move on to Vista's built-in defrag utility, and take a peek at the Windows 7 version.
There is no time limit on how long the defrag will run. The goal is better performance, since most people leave the defrag running overnight or when they are out.
Update Sunday 1 March: As I start noting the results of the benchmarks, I have coined the term "the slouch factor" to describe what happens when a hard drive gets full. We've all noticed how a Windows machine usually feels slower after a long time of use, and as the drive gets full. There is an explanation for this, best described by Robert Ferraro of DiskTrix, in the "Hard Drive Performance Theory" section of the UltimateDefrag help file.
You can see it in the graph above: data transfer rates drop from 76MB/sec down to around 40MB/sec as you move from the start of the drive to the end. When I ran my previous Defrag Shootout, I was using a drive that had two partitions, courtesy of Acer. The first (and therefore faster) partition had the OS and temporary file space, while the second had data files. It seems that Acer was being clever by doing this, because the slouch factor is caused by the position of working (or temporary) files. When there is free space near the start of the drive, the temporary files can be manipulated there, with faster seek times and data transfer rates.
As the drive fills up and is kept "tidy" by a traditional defrag program like WDD, the position of the available free space gradually moves towards the slow end of the drive, affecting the seek time and transfer rates of new and temporary files. The computer starts to slouch, hence the slouch factor.

Benchmarks*: Windows XP Disk Defragmenter

The first benchmark results are for the built-in Windows XP Disk Defragmeter (WDD). This composite image shows the "before" and "after" file layout of the test system. The defrag program was run numerous times between these two results, so don't expect to achieve this in one pass, or even in one day.
This graph shows the results in a more accessible form. The first test is at the bottom, and shorter lines mean faster times.

• "Basic XP" refers to the standard install, without Office 2007, so there are only 710 files to be tested.
• "WDD" refers to the read times of the same 802 files, after several defrag passes using the WDD Defrag button. A 26% performance improvement is measured. You could expect this result on a "clean" install machine.
• "Basic Office" refers to the read time of all 802 test files, where no defragmentation has been done whatsoever, before or after the installation of Microsoft Office 2007 Professional.
• "Full WDD" refers to the read time of all 802 test files after several reboots and several passes using the WDD defrag button. A 28% performance improvement has been measured.
• "Auto" is an interesting result. The system was left without any defragmentation, but rebooted once or twice a day for 6 days. This was done by changing the date on the laptop, and allowing several hours between each reboot.

The only difference between the "Auto" and "Full WDD" is a change in one of the Windows registry values (the default is 0):
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft
\Windows\CurrentVersion\OptimalLayout]
"EnableAutoLayout"=1
You can set this using a simple registry file, or by using the "UnityPro Disk Idle Optimiser" program. This registry entry causes Windows XP to optimise the layout of regularly used program and system files only, not data, so it isn't quite the same as an automatic defrag. Also make sure the following registry key is set (the default is "Y")
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft
\Dfrg\BootOptimizeFunction]
"Enable"="Y"
Conclusion: WDD can reduce program load times by around 27% compared to a badly maintained system. Scheduling a regular defrag (once a month for low-usage systems) will help prevent the system from experiencing major slowdowns caused by file fragmentation. if your system starts getting sluggish, try running WDD's Defrag manually. Also use a program like CCleaner to clean out junk from your system. Another useful option is the undocumented "defrag c: -b" command, which is supposed to do the boot optimize on demand.

*See: Benchmarks*: Why the Asterisk?

The Great Defrag Shootout Round 2 - Benchmarks

The image above shows a snapshot of a newly-installed version of Windows XP SR2, with Internet Explorer 7 and .NET Framework 1.1 and 2.0 installed. There are plenty of fragmented files, many of which are caused by the downloaded security updates and patches. The yellow files are fragmented, and the start of the hard drive is the bottom left. The pink area is the reserved space for the Master File Table (MFT).
Here is a more familiar picture of the drive, using WDD. The red files are fragmented, and the MFT reserved space is just shown as empty space.
I have set up my test laptop, an IBM ThinkPad R31 with a 20GB hard drive, and using Acronis True Image Home 11 recovery CD I have made a protected partition for backups, and a main NTFS partition (C:) of 8,497 MB for Windows. There is a smaller partition of around 1GB just for storing results. The C: partition is backed up and restored using a "sector-by-sector" method, so that I can reproduce the messy arrangement of the drive whenever I want to run a new set of tests.
I have installed the following software on the basic test machine:
The idea is to create a "typical" workstation that reflects currently available software. Some of the utilities are used to take measurements, and others were used to set up the workstation the way I normally set up computers I install software on.

Testing the system

The data collection process works like this:

1. Restore the full drive image, including all fragments.
2. Install and updated version of prefetch.exe with the correct testing files.
3. Install the required test defrag program.
4. Reboot and run the first round of tests.
5. Allow the defrag program to do its best to defrag the drive. This may take several attempts and several reboots.
6. Reboot and run the second round of tests.
7. Install Microsoft Office 2007 Professional trial edition.
8. Reboot and run the third round of tests.
9. Open a test document as part of "normal usage"
10. Allow the defrag program to do its best to defrag the drive. This may take several attempts and several reboots.
11. Reboot and run the fourth round of tests.
12. Run HD Tune to test the hard drive.
    
13. Copy all the test results off the drive and analyse them.

Each "round of tests" consists of running Prefetch.exe, using this layout.txt file. This is followed by JkDefrag in "Analyse" mode, and storing a screen shot of the drive image, as well as the JkDefrag log file.
The idea is to emulate the normal process of software installation and use. I chose Office 2007 Professional (Trial Edition) because it is a large, legal install, creating numerous new folders and adding a lot of stuff to the system as a whole, including DLLs, fonts, and so on. The defrag program should be able to cope with these changes.
The current Prefetch.exe install also includes an installation of my "Delay Launch" program. A shortcut is placed in the Startup menu, and 240 seconds (4 minutes) after the system reboots it loads the "prefetch.exe" program. The program attempts to measure the read time of 842 files, 39 belonging to Office 2007, and the remaining 802 belonging to Windows and the other preinstalled software.
The benchmark is run at the same time after each reboot, to ensure consistency. If a program like Diskeeper is being tested, the Diskeeper service is disabled before the reboot, so the measurements are not affected by the software being tested.
I used HD Tune to test the speed of the drive under normal conditions after a reboot, and the graph is shown here. I will post the results as they become available, complete with the raw data in spreadsheet form, so others can examine the data and provide comments.

The Difference Even a Basic Defrag Can Make

We all love to criticise the built-in Windows Disk Defragmenter (WDD) program as inadequate, clunky or even just useless, but even the humble WDD can make a difference to your system. How about 32% faster read times? I was shocked.
Before we get to the results, a word of explanation. After doing a Windows XP install and running the many updates and reboots required by Microsoft Update, I deleted the contents of the "c:\windows\prefetch" folder, and rebuilt the "layout.ini" file using the "rundll32.exe advapi32.dll,ProcessIdleTasks" command. This required several reboots, but it eventually rebuilt. This data became the basis for the "layout.txt" file used by my new "Prefetch File Processor" program.
The image above shows the before and after effect of WDD being run several times on a newly installed system: Windows XP SR2, with IE7, .NET Frameworks 1.1 and 2, several small utilities, drivers and all relevant updates from Microsoft Update. The "before" situation was a mess: 1975 files were fragmented, and there were over 10 000 free space gaps.
I ran the Analyze and Defragment options once from the "normal" program, then ran "defrag c: -f" five times from the command prompt, and then the "normal" defrag once more. The image above is a composite of the first and last screen shots.
Now the results: my "Prefetch File Processor" program read the files in 14.587 seconds before the defrag, and 9.779 seconds after the defrag. Each time the measurement was taken at least 2 minutes after a reboot, with no other programs running. That's an improvement of 32.96% in the file read time. It's the first of a series of tests I am running that will include Diskeeper 2007, Diskeeper 2008, Puran Defrag 3.0, JkDefrag 3.28, and PerfectDisk 8. The tests aim to show the kind of performance improvement that can be obtained during boot time. it is only one benchmark, and it has its flaws. These will be discussed once all the tests have been run. Of course, it's not just the fragmentation that is an issue: file placement is important too. By moving all the files and directories to the other end of the disk, I doubled the read time to 29 seconds. It was an exercise in weirdness, but still it shows how badly performance can be affected by sloppy file placement.

Testing ... Testing ... 1 2 3

How do you test whether a given defrag program has sped up your system or not? It's all very well to look at the drive layout and say "this looks neat", but how can you tell if the machine boots faster or programs load faster?
While I was analysing the "review" by the 3d Professor, I tried to think of a method of objectively testing the performance of the file system, as improved by a given defrag program. I tried gathering data from the readfile program, but the maths doesn't work correctly. I'm not sure why, and I don't understand the C source code enough to figure it out. Also, readfile only accepts a single wildcard, so you can do *.* or *.exe but not *.exe;*.dll;*.sys for example.
I did an exercise in measuring the time it took to read all the files in the c:\windows\system32 directory, and the results were almost what I expected, but it still wasn't an accurate enough reflection of how a defrag program affects the performance of the system as a whole. Clearly there are some files in the system32 folder that are seldom if ever opened or run, and if the system was optimised for files that are used often, these slower files would skew the results.
So I gave up on readfile and decided to write my own program in Visual Basic 6. It's called "Prefetch.exe" or the "Prefetch File Processor" and I'm busy in the initial stages of running tests on an old ThinkPad R31 laptop. The program is freeware and you can examine all the source code as well to see in detail how it works. The idea is that the results should be repeatable on any given system, subject to the limitations of the package.
The program has three stages:

• In the first stage the "layout.ini" file is interpreted and checked and the results copied to a "layout.txt" file, which can be edited.
• In the second stage this file is opened and each file name it contains is opened and read, and the time this process takes is calculated in milliseconds.
• In the third stage the results are saved in a CSV file for further analysis and checking.

There are some limitations to this process. The biggest limitation is that Visual Basic 6 can only read the first 2,147,483,646 bytes (2GB) of any given file. Usually that's enough, and I'm not that interested in huge data files, so it's good enough. The second limitation is that certain system files are opened and locked by the system, so these can't be timed. This is a limitation, but equal across all programs tested. The third limitation is that it can't test the time it takes to open a folder, only a file.
Another problem is that the "layout.ini" file includes some junk files and it keeps changing. That's why I created the "layout.txt" file, and edited it to remove files in the temporary folders, references to cookies, log files, critical updates, and other miscellaneous junk. The test file I am using is here, and hopefully it will deliver a fair test of the system.
The timing starts just before the file is opened, and finishes when the file is closed. The timer works in msecs, and does not include the time it takes to read the name of the file from the "layout.txt" file, or the time to save the result in the "timing.csv" file. Later I may extend the program to do some read-write tests, but for now it's read only.

My Test System

I have created a 5.86GB partition on Penny's old ThinkPad R31 laptop. With Windows XP Professional and IE7 and the .NET 1.1 and 2.0 frameworks and all service packs loaded, there is 1.86GB free space, i.e. 32% free. I then made a full sector-by-sector image backup using the Acronis True Image Home 11 Recovery CD, and this image is stored on another partition of the disk that is not included in the testing. This backup image is fragmented, exactly as it was created during the install process, with absolutely no attempt to defragment the drive in any way. Here are some facts about the system, as reported by JkDefrag in "analyse" mode:

Total disk space: 6,293,757,952 bytes (5.86 gigabytes), 1,536,562 clusters
Bytes per cluster: 4,096 bytes
Number of files: 20,110
Number of directories: 2,464
Total size of analyzed items: 4,268,003,328 bytes (3.97 gigabytes), 1,041,993 clusters
Number of fragmented items: 1,342; 5.94% of all items
Total size of fragmented items: 952,217,600 bytes, 232,475 clusters, 22.31% of all items, 15.13% of disk
Free disk space: 1,262,436,352 bytes, 308,212 clusters, 20.06% of disk
Number of gaps: 4,722
Number of small gaps: 3,978; 84.24% of all gaps
Size of small gaps: 78,368,768 bytes, 19,133 clusters, 6.21% of free disk space
Number of big gaps: 744 (15.76% of all gaps)
Size of big gaps: 1,184,067,584 bytes, 289,079 clusters, 93.79% of free disk space
Average gap size: 65.27 clusters
Biggest gap: 926,806,016 bytes, 226,271 clusters, 73.41% of free disk space

Not chaotic, but hardly optimal. Each program tested will be given a chance to defragment and optimise this data, and once it has done its best, the system will be rebooted and the Prefetch File Processor will read the list of files and time the process. I will also record screen shots of the drive image before and after, and use the JkDefrag analyse log file to note other aspects of the defrag process.
I would welcome any comments or criticisms of this process, and feel free to download the program and run your own tests, and inspect the Visual Basic 6 code. I will document it more fully in the next few days, so the code is easier to read.
Update: My TrueImage backup file is corrupt, and I have to reinstall everything. I'll update the numbers published above once this has been done. Using the WDD defragger improved the read times by 15%, but I will re-run the tests and publish the results in full.
Update: For technical reasons associated with a large bad spot on my drive, I can't do a sector-based backup, only a complete file backup. This is going to complicate matters slightly, but hopefully the results won't be too skewed. It's a lot of work reinstalling XP, not to mention tons of bandwidth during updates.

The Great Defrag Shootout XXIX: fragdown 2.4

Fragdown 2.4 is a front-end for the built-in Windows defrag.exe program. It's a highly economical 50kb download, and provides a simple facility whereby you can tell defrag.exe to defrag the hard drive and then shut down. The default is to start the defrag immediately and then shut down the PC. You can modify the shortcut to use option 2, in which case fragdown will show as a system tray icon. When you double-click on the icon the defrag starts.
This may be a useful utility in certain circumstances, but its scope is limited and it relies on WDD too much. It was written in 2004 and the documentation is not particularly clear for a novice user.

The Great Defrag Shootout: Part I | II | III | IV | V | VI | VII | VIII | IX | X | XI | XII | XIII | XIV | XV | XVI | XVII | XVIII | XIX | XX | XXI | XXII | XXIII | XXIV | XXV | XXVI | XXVII | XXVIII | XXIX| winner | all | why

The Great Defrag Shootout XXVII: Windows PowerTools 1.3

This is the worst waste of time I have ever had the misfortune of reviewing: "Windows PowerTools 1.3" is a batch file 3kb long, packaged in a 100kb self-extracting EXE. There is no uninstall: you just delete the folder containing the batch file and the desktop shortcut.
The batch file is reasonably easy to read, as batch files go, and some of the options are too buggy to work. There is NO documentation, other than the one-line explanations shown in the graphic above. When you select option 6 above the batch file calls the built-in Windows Disk Defrag utility. This "utility" gets a big thumbs down for sheer waste of time. Thanks for nothing.

The Great Defrag Shootout: Part I | II | III | IV | V | VI | VII | VIII | IX | X | XI | XII | XIII | XIV | XV | XVI | XVII | XVIII | XIX | XX | XXI | XXII | XXIII | XXIV | XXV | XXVI | XXVII | XXVIII | XXIX| winner | all | why

The Great Defrag Shootout XXV: UnityPro Disk Idle Optimiser Pro 4.0

Disk Idle Optimiser Pro 4.0 from UnityPro software is not a defrag utility in the conventional sense of the word. It's more like a control panel tweak. It allows you to control some built-in aspects of Microsoft Windows XP that normally just happen (or not) in the background.
I installed this utility after downloading it over 2 weeks ago. Since then I have not noticed any fundamental change in the way my laptop boots up or works. On the other hand it hasn't degraded either. Either way, the impact on a well-organised system is unlikely to be major. But as tweaks go, it isn't going to slow down your system, and it has the potential of keeping your system better organised by allowing it to "optimise" the files used during boot-up and in the "prefetch" cache.
It's an addition to your existing system, and once the settings have been changed it seems safe to uninstall it, since it only tweaked the Windows registry anyway. If you are using a defrag utility that controls the prefetch file (such as PerfectDisk), don't bother with this utility since it won't make any difference anyway.
The download page explains what is done, and why. There is nothing to see, and it would be quite tricky to measure what impact it has. As a result, this review gets neither a thumbs up or down. But I have kept a copy on my system, and not uninstalled it. I intend to use it on systems where I have not purchased PerfectDisk.

The Great Defrag Shootout: Part I | II | III | IV | V | VI | VII | VIII | IX | X | XI | XII | XIII | XIV | XV | XVI | XVII | XVIII | XIX | XX | XXI | XXII | XXIII | XXIV | XXV | XXVI | XXVII | XXVIII | XXIX| winner | all | why

The Great Defrag Shootout: Why Defrag at all?

There are a few interesting myths and misconceptions out there about PCs and defragmentation. The first myth is that NTFS drives don't need to be defragmented. The second is that "modern" operating systems don't need to be defragmented at all. The third is that simple defragmentation on its own keeps your PC working fast.
The last one is the easiest to deal with. Recently I converted my brother's old Pentium II machine running WindowsXP into a "backup server". The WDD defrag program had been run occasionally, and the drive was pretty full, so most files were not too seriously fragmented.
As you can see from the "before" picture, the green files are all defragmented, but not particularly well organised. This was partly because I had just uninstalled a whole load of games and other junk from what was at one time a family PC.
Then I set to work on the drive using JkDefrag, allowing it to do its standard default defrag. Here is the result:
Given the choice between the two drive layouts, which one do you think will work more efficiently? The answer is pretty obvious. It was easily demonstrated on the PC in question because of its low processing horsepower in the first place. By the time I had defragmented the drive, cleaned the registry with CCleaner, and then compressed it with NTREGOPT, the machine was starting to "run" instead of "walk".
Defragmentation isn't the only strategy to make a PC run faster, but it is one of the strategies that are useful. "Regular" defragmentation is better than "random" defragging, but how often is "regular"? It depends on your PC. If you are generating lots of files, or editing databases, a daily defrag may be required. If your files are pretty small, a monthly defrag is probably good enough.
My own rule of thumb: if the defrag takes longer than an hour you need to do it more often. If it takes less than 10 minutes, defrag less often. You don't want to waste a lot of time in order to save a little time. Do the defrag during a quiet period, such as when you're away from your desk for a meeting, or after hours. In that way you don't waste productive time.
The other myths take longer to explain, but every drive on the planet runs the risk of fragmented files. Consider the following scenario: I install a 10GB drive on a server, and connect 2 PCs to the server. User A saves a 4GB data file to the server. The next day user B saves a 3GB file. So the drive has 3GB free. Now user A modifies his data file so it grows to 4.1GB. Even if he deletes the old file before saving the new one, there are only 2 gaps of 3GB each on the drive, unless the server moved the files around, which it isn't likely to do. So the 4.1GB file will be fragmented. Now it's up to the server to sort out the fragmentation. Different operating systems will do this in different ways, but every OS has to have a solution, even Linux.

The Great Defrag Shootout: Part I | II | III | IV | V | VI | VII | VIII | IX | X | XI | XII | XIII | XIV | XV | XVI | XVII | XVIII | XIX | XX | XXI | XXII | XXIII | XXIV | XXV | XXVI | XXVII | XXVIII | XXIX| winner | all | why