A Closer Look at the DK2008 Review on 3DProfessor.org - Part 2

This graph is taken from the 3D Professor review of Diskeeper 2008: they show SPECviewperf® 10 results. I had no idea what those were, so I went to the SPEC site and read up about them. They measure the speed with which a given system can draw things in 3D. This is important when you are configuring or comparing high-end graphics workstations, because a fast result hopefully means you get more work done.

"SPECviewperf parses command lines and data files, sets the rendering state, and converts data sets to a format that can be traversed using OpenGL rendering calls. It renders the data set for a pre-specified amount of time or number of frames with animation between frames. Finally, it outputs the results.
SPECviewperf reports performance in frames per second. Other information about the system under test -- all the rendering states, the time to build display lists (if applicable), and the data set used -- are also output in a standardized report."

In the section "What SPECviewperf® Does and Doesn't Do" the writers state:

"Nearly all benchmarks are designed for a specific purpose. Quite often, however, users broaden that purpose beyond what the benchmark is designed to do, or in some cases they assume the benchmark can't do something that it actually can do. The SPECviewperf® benchmark is no different: it has been both overextended and underappreciated, sometimes reducing its overall value to the user."

Once again the results posted show little, if anything, to do with file fragmentation, and the graphs look quite scientific, but mean very little. It certainly doesn't show any clear difference between the products. I'm not sure what the professor means by

"Although minor points gains in many areas across the board each gained X.X point are in higher circles valuable scoring patterns and mean more sales to the resellers as corporate buying patterns are heavily based upon these results shown."

It sounds like too much caffeine to me. I can't decode it at all.
Also, the results are shown "Without Fragmentation" in Windows, but I think the label meant to read "Without Defragmentation", since there is no mention of using the built-in Windows defragger in competition with Diskeeper, and the previous test refers to tests "... without any hard drive defragmentation tool enabled".
In the other section the professor lists the elapsed times measured when running "SPECapc for 3ds Max™ 9", which

"... measures performance based on the workload of a typical user, including functions such as wireframe modeling, shading, texturing, lighting, blending, inverse kinematics, object creation and manipulation, editing, scene creation, particle tracing, animation and rendering. The benchmark runs under both OpenGL and DX implementations of 3ds Max 9, and tests all the components that come into play when running the application."

The 3D Professor review states these results but doesn't graph or tabulate them. I did the chart above in Excel, and calculate a 3.4% time saving with DK2007, and a 6.4% time saving with DK2008. The professor's conclusion is "A substantial improvements [sic] across the board, biggest improvement overall with Diskeeper 2008 installed on a fresh system."
As I understand it, the professor is saying that compared to a fragmented system, DK 2008's 6% improvement is substantial. What is missing from this analysis is the time the tests would have taken if the fresh system had been defragmented first. So far this is only an argument for using a defragger, not for using a specific defragger.
I am also extremely suspicious of the results for the Windows 2003 Server. They are exactly the same as the Windows XP results, down to the last second. Each machine has a different processor, different amounts of RAM, and different hard drive speeds. The only aspect in common is the "ATI FireGL V3600 Professional Graphic's [sic] Card". If the results are the same because the same graphics card is used, then why would a defrag product make the graphics card run 6.4% faster? I assume the results obtained are different from those published, and this is a cut and paste error.
Another point that needs clarification: was Diskeeper simply installed and enabled, but the manual defrag not run? The conclusion wording is vague (bold added):

"The amount of data and time needed to be collated and cross checked has been tremendous. At times when we undertake reviews here we just take for granted the overall performance that the products tested produce some remarkable results. Though on reflection we sit back and realise on just how much performance is obtained with no degeneration in system performance with the Diskeeper running on “the fly”!
... With Diskeeper 2008 we had all the possible functionality enabled during both Windows XP and Server 2003 tests. The intelligent mode in which Diskeeper functions utilising ... Windows API's is spectacular.
... Sincerely for the non-user's of this product, with this sort of performance increase to be gained from a simple effect software tool working away quietly in the background!

From this we can be sure that the background defrag was enabled, but no mention is made of an initial defrag. The "Software & Benchmarks Utilised" section lists the products used, and then states

"A mixed combination of some 10GBs software and benchmarks - for those who are aware; fragment the hard drive into a complete and utter mess once all of the above is installed. Each set of tests were completed on a fresh installation of all the above software, each time we low level formatted the hard drives to clear the MBR’s etc. Therefore the time to complete each set of results has been substantial as results are triple checked to ensure that no anomalies are seen. This combination mix of benchmarks and software would therefore ensure that the whole system I/O was fully stretched to its limits and ensure that the new software from Diskeeper would function as its fullest workload capacity.

Again, no mention of any manual defrag to fix the "complete and utter mess". This is troubling from several aspects:

• The results ignore the built-in Windows Disk Defragmenter, making DK appear better, because the tests compare a well-configured Diskeeper system with a badly configured Windows system.
  
• The background defrag would stop during the tests, but it may not have completed yet, making the results erratic, and prejudicing DK's true abilities.
• By starting from scratch each time, the I-FAAST file placement system is unable to work, because it needs several days to get data on the usage of the system in order to place the files correctly. Since I-FAAST constitutes 50% of the cost of DK2008 Pro Premier, this is a serious omission.
  

If these omissions were made, it calls into question the professor's understanding of what defragmentation is, and how Diskeeper works.
Then there is an interesting snippet in the latest white paper by Joe Kinsella, "The Impact of Disk Fragmentation", published on diskeeper.com. Figure 5 from that paper is shown below.
According to these figures, a fresh install of Windows XP followed by SP2 would create roughly 636 fragmented files with 3582 fragments. That doesn't include the 75 critical downloads of 185MB that follow SP2. The table above implies that this would generate several hundred more fragmented files.
For a fair test of Windows XP all these files would need to be defragmented, along with the 10GB of other software installed. Ideally this would be done by WDD. After this is done it could be backed up using Acronis True Image or DriveImage XML, to speed up reconfiguring the machines each time a fresh install was performed. The professor's review does not mention any details of this sort, which is worrying. Hopefully this is an oversight and the comparisons were fair after all. I have made several attempts to contact Professor Brian Robinson, and even searched for him on Google, without success.
Another look at the chart above could explain the 3.4% and 6.4% time savings as follows: DK2008 does appear to have a more effective background defrag capability than DK2007, and this is touted as “‘defragmentation intelligence’ enhancements” in the press release and the review. In that case the only effective conclusion we can reach is that DK2008 is 3.1% more effective than DK2007. I'm not sure that this was the professor's intention, but it is the only reliable conclusion I can make, based on the data he presents in these tests.

A Closer Look at the DK2008 Review on 3DProfessor.org: Part 1 | Part 2 | Part 3 | How Fast is I-FAAST™? | Diskeeper 2008 Professional: Preliminary Results | First Impressions | Diskeeper 2007 Review | Benchmarks: DK2008 and DK2007

More Malware File Scanners

I recently wrote about a site that scans files for virsues and malware, and tells you the results. Now I have found a few more. The list has grown to:

• VirusTotal at www.virustotal.com
• Virscan.org at www.virscan.org
• Online Malware Scan at virusscan.jotti.org
• Virus Chief at www.viruschief.com
• Virus.org at www.virus.org
• Dr Web offers to scan a URL as well
• Website Planet will scan a URL (without the adverts).

If I have missed any others, please let me know. There are a number of single-vendor scanners listed at Prospector.

Enterprise Architecture gets Agile and goes Open Source

Software development in large enterprises is a tricky business. Organisations change, needs change, and big projects suffer feature creep. In response to this a methodology known as "agile" development was born. Now there is a wiki site for people (like my brother, Charles Edwards) who work in or are interested in this kind of thing. Check out http://agileea.wikidot.com/. The main aim of the site is to build up an Eclipse Process Framework (EPF) Process for Agile Enterprise Architecture.



Enterprise Architecture (CEO, CIO (enterprise or holisitc view)) = the management and definition of the Architecture of the Enterprise (set of
organisations) includes everything from the Business Strategy, to the Business Operation Architecture (static and dynamic) for the purposes of optimizing the enterprise; the Information systems (made up of Applications, Services and Data) and the Technology and Infrastructure that this runs on.
EA also includes aspects such as Security, Data, People and Performance. The primary purpose of creating an enterprise architecture is to ensure that business strategy and IT investments are aligned.
Enterprise architecture models allow traceability from the business strategy down to the underlying technology, in order to do impact analysis and have the ability to react to changes quickly, govern the Architecture and guide outsourcing. To contain the Knowledge and the memory of the Enterprise (Architecture) in a single point of truth.

How Fast is I-FAAST™?

According to the Diskeeper.com web site, Diskeeper 2008 Pro Premier is "specially designed for high-end systems and power users! Get an EXTRA performance boost above and beyond anything you have ever experienced before - even better than when your system was new!"
The only "extra" feature is this version is I-FAAST, and it costs an extra $50 per workstation, more than the cost of any other defrag utility. According to the web site

Diskeeper's I-FAAST 2.0 (Intelligent File Access Acceleration Sequencing Technology) accelerates file access times to meet the heavy workloads of file-intensive applications. Utilizing a specially formulated technology, I-FAAST closely monitors file usage and organizes the most commonly accessed files for the fastest possible access, boosting file access and creation to speeds above and beyond the capabilities of your system when it was new, up to 80% faster.

I found an interesting 2005 document entitled "Benchmarking Diskeeper’s I-FAAST" which explains that Diskeeper monitors all files being opened and modified, and uses this statistical information to work out which files are more important.

"I-FAAST was designed specifically for real world environments where data is a heterogeneous mix of often used or “Hot” files, and stale or duplicate and unused data. In real world scenarios, it is also true that data goes through “hot and cold” phases, and that a significant amount of data is, or must be kept, for regulatory or archival purposes; often intermixed on active storage media. All these and other common scenarios are integrated into I-FAAST’s heuristic algorithm."

For that reason it takes a few days for the I-FAAST stats to be gathered before any noticeable change happens to your data. So far I have noticed that some of my files have been moved to the slowest part of the volume. Apart from that not much has changed.
In order to accelerate the effect, I created a utility to load several applications that I use on a regular basis. The utility is run at startup, and over the course of several minutes loads Word, Excel, Access, Paint Shop Pro, Outlook, VB6, Firefox, Access97, etc. etc. The timing between each one is set to around 30 seconds, so each app gets a chance to load before the next application starts loading. I didn't want to overwork my poor machine!
To get DK to learn about these common apps, I set the Task Scheduler to do a reboot every 30 minutes, and left my laptop doing reboots for the next 2 nights, effectively running each app around 30 times, in addition to normal use during the day.
The document about I-FAAST also refers to a freeware utility called readfile, which can measure the speed with which a given file can be opened/read. I have been using it to measure the effect of the I-FAAST placement on my system. So far the results have been inconclusive. Readfile doesn't work in a BartPE environment, so I have had to use Process Explorer from Sysinternals to make sure that no extraneous tasks are running during the measurement process. This is easier said than done.
Here is the "before" situation, with dozens of programs and processes running.
I shut down as many as possible before using readfile. I had to experiment a few times before getting reliable results. In each case I made sure the laptop booted up from scratch, and that none of the application data being tested was loaded on startup.
Here are my initial preliminary results. (Lower values are faster). The column "Access97" refers to the folder
"C:\Program Files\Office97\Office"
which contains the installation of Access97 that I use on a daily basis. The column "OfficeXP" refers to the folder
"C:\Program Files\OfficeXP\Office10"
which contains the bulk of the EXE and DLL files for Word, Excel, Outlook, Powerpoint and Access that is part of Microsoft Office 2002/XP. I usually only use the first 3 applications.
The column labelled "Data" refers to the folder
"C:\SQL\MSSQL\Data"
This contains 8.19GB of Microsoft SQL Server 2000 data that I use regularly. At the time I made the first DK2008 measurement, one of the files had 2 fragments that DK2008 decided not to defragment, for its own reasons. This was confirmed to me by a senior support person at Diskeeper Europe.

• The blue results are for Diskeeper 2008 Pro Premier with I-FAAST enabled, as well as automatic defragmentation, directory consolidation and statistics gathering all enabled.
• The purple results were generated after I ran JkDefrag 3.28 several times until it had moved and defragmented all the files on the drive. The total read time is 5.6% faster than the DK2008 result.
• The yellow results were generated after I ran PerfectDisk 8 after the JkDefrag changes. I did a boot time defrag first, and then a SmartPlacement defrag. The resulting total read time is 14.3% faster than the DK2008 result.

Are these results significant? I think it's too early to tell conclusively.
This is DK2008's drive map. The large red file is the SQL data file in two segments, as already mentioned.
After JkDefrag's complete defrag, all the files have been reorganised with the directories first, then the tell-tale 1% free space at the beginning of the drive, then the important files, then another 1% space, then the "spacehogs" at the end.
This is the disk arrangement after PerfectDisk 8 did its boot time defrag and SmartPlacement. Notice how the bulk of the free space is at the end, not spread all over the volume.
Each program has its own distinctive pattern. After running DK2008's defrag again, it moved some files to the end of the drive, as shown below.
Further testing will tell if more I-FAAST history and analysis can make a significant performance difference or not.
Update Tuesday 6th Nov: I was wondering if a readfile scan of the entire directory would not prejudice the DK2008 results because this would include "slow" files as well as "fast" ones, so I totalled up the read times for recently accessed EXE and DLL files only, i.e the ones I-FAAST is supposed to be optimising. Readfile has a problem calculating times for files larger than 4GB, so the "Data" directory is excluded.
The graph shown above provides the readfile times for EXE and DLL files only, using the same measurements as the first graph. The DK2008 data fared worse, not better; JkDefrag proved to be 13.4% faster, and PerfectDisk 8 is 20.1% faster. I will wait a few more days to see if the file placement improves, and then run the test again for DK2008. I cannot explain the difference in performance, because file placement by itself should not account for more than 15% at the most. Since we are dealing with single contiguous directories, the position of the directory shouldn't make such a difference either.
Craig Jensen's document on I-FAAST states

"This new technology adds even greater performance gains than previously available through defragmentation alone, improving file access and creation up to an additional 80%, with typical added gains between 10-20%."

My data only refers to read access of defragmented files, so I still can't explain what is happening, and why I-FAAST read access would be 20% slower, not faster. More testing is needed.

A Closer Look at the DK2008 Review on 3DProfessor.org: Part 1 | Part 2 | Part 3 | How Fast is I-FAAST™? | Diskeeper 2008 Professional: Preliminary Results | First Impressions | Diskeeper 2007 Review | Benchmarks: DK2008 and DK2007

A Closer Look at the DK2008 Review on 3DProfessor.org - Part 1

In my efforts to figure out what makes Diskeeper 2008 Pro Premier work, I thought I had found a useful review at 3D Professor.org, but there are aspects of it that are frankly misleading without further analysis. I think the testing was done in good faith, and there was no intention to mislead, but the method is sloppy and if this was an academic research paper subject to peer review it would not be approved for publication.
A lot of the tests are beyond my area of expertise, but one test in particular puzzled me. It involves the use of a disk testing utility called HD Tach from Simpli Software. What puzzled me about it is that, like HD Tune, it is used to calibrate the hardware performance of a hard drive at the sector level, and has nothing to do with the file system, and so it should show no performance change at all from the use of defrag software. When I downloaded a copy I noticed a section in the readme file that is relevant:

"SECTION 3 – HOW DO I RECORD GOOD BENCHMARKS?
Before benchmarking, reboot your computer. This will clear system caches and unload many unnecessary programs. Upon booting quit all programs that may have been loaded during the boot process, task bar icons, disable virus scanners. Advanced users should unplug the network cable and disable the swap file.
Give your system time to ‘stabilize’ – Windows does a lot of work in the background, so do not use HD Tach until you see your hard drive light stop blinking for 10-15 seconds. This may take one to two minutes after boot. During the benchmark do not move the mouse or press any keys."

The analysis of the tests imply that this was not done. To quote the Professor's report

"Going straight to the main points of the benchmark we ran the full test. As you are aware this benchmark has 3 types of test; short, long and full. The full test completely stretches the disc I/O. Average read speed in the full test increases by with Diskeeper 2007 installed by 1.2Mb/s with the actual burst speeds with nearly a complete 12Mb/s increase which is quite a lot to recover on a drive of this magnitude. With Diskeeper 2008 installed the results completely took our breath away."

What makes it worse is that the graph scale on the results changes from one test to another, and this was not pointed out. I have used Paint Shop Pro to correct the graphs published by the Professor, so that a visual comparison can be made.
This is the graph for Windows XP with no defrag.
This is the graph for Windows XP with Diskeeper 2008, adjusted to show a scale of 0 to 160, instead of 0 to 130 published here. The visual effect of the change in scale is quite dramatic, and appears to point to a huge change.
In fact the change is minor. The average read speed changes from 122.8MB/sec to 124MB/sec (1.6%), but the "burst speed" result goes from 278.6MB/sec to 650.9MB/sec. Clearly if this performance improvement of 134% is to be believed it should be repeatable. I believe it to be an error in testing methodology, because no hard drive on earth can run twice as fast by installing software. It defies the laws of physics.
I downloaded my own copy and ran the HD Tach benchmarks on my own laptop, with Diskeeper 2008 Pro Premier installed. In each case I ran the "short" test because the "full" test isn't available on my copy.
This is with Windows XP running all its services, discouraged by Simpli Software.
This is the same test with the bare minimum of services running, and with the mouse and network cables unplugged.
This is the same test, but run from a BartPE environment, which is a minimal version of Windows that runs from a boot CD.
Here is the same dive after reorganising the data completely using JkDefrag. As expected, the results are almost identical.
Finally, here is the BartPE test run after a complete defrag done by Diskeeper 2008. The "Go" button in the bottom left of the screen is part of the BartPE environment, and replaces the "Start" button. The average read is 30MB/s in each test except one, where it is 29.8MB/s, a variation of 0.6%, which doesn't really say anything. The burst speed remained the same thoughout.
The professor's testing in Part II does not make these mistakes, but no comment is made and no conclusions drawn. The tests were all run within a few days of one another.
My conclusion: the HD Tach test is completely irrelevant to the defrag process, because HD Tach is reading hard drive sectors, not NTFS files. The hard drive could be newly formatted and give the same result. How then can the 3D Professor conclude "... with Diskeeper 2008 installed the results completely took our breath away"? Unfortunately this is not the only flaw in the review published at 3dprofessor.org.

A Closer Look at the DK2008 Review on 3DProfessor.org: Part 1 | Part 2 | Part 3 | How Fast is I-FAAST™? | Diskeeper 2008 Professional: Preliminary Results | First Impressions | Diskeeper 2007 Review | Benchmarks: DK2008 and DK2007

Award Winning JkDefrag 3.34 Freeware Bundle

I have five defrag programs installed on my laptop at present: Diskeeper 2008 Pro Premier ($100) for testing, PerfectDisk 8 ($40) for emergencies and metadata defragmentation, JkDefrag 3.34 for daily tidying up, PageDefrag for quick boot-time defragmentation of critical system files, and Contig, for a quick defrag of Access data files that I'm working on. The last 3 programs are free: two from SysInternals (now part of Microsoft) and one from Jeroen Kessels.
Whenever I fix up a home PC for someone who can't afford PerfectDisk, or if it is already installed, I also install the last 3, along with NTREGOPT for getting rid of excess registry space. So I keep a setup file with all these utilities included, made with Inno Setup. The latest one is listed below. It's freeware and virus-free. And it works better than any version of Diskeeper, from their cheapie Home version ($30) to their eye-watering Server version ($300).
The large yellow blob in the middle of the screen shown above is a 4.88GB SQL Server critical data file that DK2008 doesn't want to defrag. The light green files are ones that JKD would want to put in Zone 2, and the dark green ones in the "spacehogs" zone 3. Notice how the free space is divided into chunks by DK2008's "NEW! COMPLETE FILE AND FREE SPACE DEFRAGMENTATION" feature. I guess I shouldn't mock them because they're new to free space defragging, while PerfectDisk has been getting it right for years.

Download JkDefrag 3.34 + NTREGOPT + PageDefrag + Contig, all in a simple setup program (759,489 bytes). The install program is unsigned but works fine.

Update 18 Feb 2008: The setup program now includes JkDefrag 3.34, and leaves out the JkDefrag GUI included before. If you want a GUI instead, get it here (3,726,575 bytes)

Update 28 January 2010: If you are using your computer for private or non-commercial purposes, then check out Puran Defrag Free Edition, which is awesome.

A Failure of Power

The powers that be at Eskom have failed us in the past year or so. This cartoon at www.ditwits.co.za and on the front page of The Times captures the frustration of the country pretty well.
So far we have had national rolling blackouts, pollution caused by sewerage works dumping untreated sewerage into rivers because the power was off, traffic chaos, lost productivity and of course denials that anything is wrong and general finger pointing. I guess it would be too much to ask anyone to actually admit responsibility.