Backing up your Clarion projects

Whenever you start work on a new project, it's too easy to forget to make backups. Even if you save your work often, what do you do if Clarion crashes, or you break something and need to go back to the previous version of the project? For US$99 DeveloperTeam has a nifty template you can use for automatic backups every time you do a compile. It's pretty automatic once you set it up, but it doesn't back up everything, just the really important bits (app file as txa, dictionary file as dctx) so it is very fast.

Another option is to use Zippy Backup Assist, a free utility I wrote for my Access97 development work after I threw away a whole day's work by mistake. You copy the Zippy.exe program into your project folder, as well as zip.bat, which does most of the work. Then you edit the zip.bat file and assign a name for the project. In this case, the name is "learn" and the project is called "Learning Clarion".

Zippy.exe calls the zip.bat file in the same folder as it is running from, and passes it a parameter, comprising the project name (set this using the ZIPPY= command in zip.bat) plus today's date (and time). It's written in Visual Basic 6, and runs fine in Windows 10 and 11.

Installation

Please download ZippySetup.exe which will also install the command line version of PKZIP called PKZIPC. (My installer and Zippy.exe is not signed. Sorry! But you can upload it to VirusTotal to check for malicious code.) There are two versions of the zip.bat file available:

• zip-daily.bat, which creates a new file for each day it is run; and
• zip-hourly.bat, which creates a new file for each (even numbered) hour of the day it is run. Use "Zippy.exe HOURLY" in the command line when you use this file.

Whichever one you choose, rename it to zip.bat, or it won't work. The backup files are stored in a "zip" folder
Update: I discovered that Clarion has installed 7z.exe in the Clarion\bin folder, so instead of using pkzipc.exe you can use 7z instead. I have updated the batch files to reflect this.

Backup with every build

If you want to make a backup after every successful build, then there is an additional step you should take:

From the "Project" menu, select "Project Options..." and go to the "Build Events" tab. Use the Ellipsis button to select the Zippy.exe in the same folder as your project. Make sure the post-build event is set to "On Successful Build" so you don't back up a bad set of data. If you do a lot of builds in a day, then it may be a backup too often. The Developer Team template makes a backup before the build, and it gives each backup a sequential number, so you can work backwards if needed. Or you can do both.

Another Option: Batch file

Carl Barnes told me of his solution: a single batch file that you copy and modify for each project. It uses RoboCopy and PowerShell to do some of the heavy lifting. These are included in Windows, so no extra installation required. He has shared it on GitHub. He offers various options, including a batch file that copies all your stuff to another drive. Worth checking out.
He also includes another batch file contributed by Rich Fulham during a ClarionLive webinar, which uses 7-Zip. So there are plenty of options.

Backup to the Cloud or another PC

Making zip copies of the project is a good way to keep a timeline, but if your hard drive crashes, you don't have a backup. A backup is only a backup if you have two other copies of the file, one on another media like a CD ROM or external hard drive (try SyncTrayzor or FreeFileSync), and one in the cloud or offline storage somewhere. Something like Google Drive, Dropbox, iDrive, Carbonite, etc.

[ Introducing Clarion FileManager (Part 1 of 2) ]
[ How to keep your useful CapeSoft accessories up to date ]
[ Adding Extensions to Clarion ]
[ Learning Clarion (Part 1 of 7) ]
[ Backing up your Clarion projects ]
[ Getting Started with Clarion (Part 1 of 2) ]
[ Introduction to Clarion 11 ]
[ Installing Clarion 11 on Windows 10 ]

[ Clarion Resources and Products ]
[ Clarion Build Events ]

Update Tuesday 2 October 2023: Newer version of Zippy.exe which allows you to kill or run applications using a file called zippy.ini in the same folder as zippy.exe. The file structure is simple. Each program you may want to run starts on a line by itself:

run="C:\Program Files (x86)\DebugView++\DebugView++.exe"

and each program/process to kill is just on a line without anything further:

iexplorer.exe

Use this in conjunction with the Clarion Build Events technique. You can automatically stop and start the debug program when you build your app.

Spinrite's Operations and "Bit Rot" explained

I have just posted this video on YouTube in order to share it here and on my Spinrite product page on the Fact-Reviews site. Steve Gibson explains what Spinrite does and how it operates, including how it "begs" for data from dodgy areas of the disk in order to recover it.
He also mentions the concept of "Bit Rot", where the defects on a hard drive can grow over time, gradually affecting more bits in a sector, until the sector becomes unreadable. Fortunately DiskFresh can also help detect these problems early without taking the entire PC offline for maintenance. With Spinrite the entire PC has to boot off a special boot CD or flash drive in order for Spinrite to do the recovery.
Of course if the PC isn't booting up anyway this is not really an issue, but with preventive maintenance it can be a big issue. That's why I was delighted when DiskFresh was developed, because now I can find out whether my hard drives (including removable USB drives) are developing problems before it is "too late".
Another issue I was not aware of is when and how often drives fail, including brand new "out of the box" drives. The authors of "Protecting Digital Assets for the Long Term" cite research from Google and others which suggests that new drives should be tested between 200 and 2000 hours before being put into active service, to prevent what they term "infant mortality" failures. The problem gets bigger as the size of the drive gets bigger, too.

In the case of Windows it usually takes around 200 hours to install and set up Windows and all the software updates and packages, but I have had a hard drive die on me after 7 days of use. It was not pleasant.
It is also not a good idea to leave a drive switched off for too long, because the mechanism can get stuck and the drive may never spin again. If you are using an external hard drive for backups, make sure you do a regular backup (monthly is more than adequate) to that drive, just to make sure that the drive is still working properly. And remember to do your quarterly disk refresh with DiskFresh (see article below).

DiskFresh stops data fading away

My development server has two huge 2TB drives in it, which makes backups a nightmare. But what has worried me even more is that these drives are too big to do proper disk maintenance. No drive is "too big to fail", and the more data it contains, the bigger the disaster when it does fail. It's just a matter of time.
All magnetic disks suffer from the same problem: the analogue magnetic signal decays over time. If the file is being regularly updated, then this isn't a problem, because each update rewrites the analogue signal. But what about those old music files that you haven't listened to in 2 years? How do they get updated? When was the last time your hard drive even read those files to see if they are OK or not?
Until now I have used Spinrite to maintain my hard drives. On my laptop this isn't a hassle, because the drive is only 160GB big, and I can run Spinrite overnight in "refresh mode", where it reads and rewrites every sector on the disk, whether it has been used or not. If any sectors are damaged or faulty, Spinrite will attempt to recover the data. It usually succeeds.
In addition, I use Carbonite to make an automatic backup to "the cloud" of all the files I need on a regular basis. This includes all the programming projects I am working on. So if I file gets damaged or corrupted, or I just screw up and need an older copy, I can get it back in a few minutes. For bigger files like my two 4GB Truecrypt volumes, I use SyncToy to make copies to the server, and to an external USB hard drive.
But this is where things start getting difficult. The 500GB USB drives are slow, and it can take ages to run Spinrite on them. Each of the 2TB drives takes Spinrite nearly two weeks to check thoroughly (level 4), but I simply cannot expect my server to be offline when this happens, and Spinrite only works in offline mode. Gotcha. So I have had several hard drives lying around the office with important data on them, and no way of ensuring that the integrity of the data is being maintained. I can't afford to wait for the drive to fail or the data to be lost. That's a luxury for big companies like Google.

Google have so many hard drives in their data centres that they are always removing and replacing faulty hard drives. They have special machines (shown above) to destroy drives once they fail. Watch from about 3 minutes into this YouTube video.
A few weeks ago, the author of Puran Defrag and the Puran Utilities, Vishal Gupta, wrote to me to say he had started working on a program to solve this problem. I gladly agreed to do some testing, and make suggestions. I made a complete image backup of my test machine, and set to work.
DiskFresh is the result of this collaboration. Vishal did all the engineering, wrote all the code, and put up with my nitpicking, wild ideas, and occasionally useful suggestions. The result is elegantly simple, but extremely useful. You select a drive (either the entire physical disk or a drive letter) and the program methodically reads each sector, and assuming it gets no error messages from the drive, rewrites the data back to the same sector. Bingo! A fresh analogue signal on the drive, and you can rest easy that (for the next few months at least) the magnetic signal is not going to fade away and lose your data with it.
DiskFresh also has a "read-only" mode which is useful for SSD drives and USB Flash drives. Flash memory also has a "data fade" problem, but rewriting the data doesn't help, because you can only write data on a flash drive a limited number of times. Instead, you just read all the data fairly often (4 times a year) and the drive controller should pick up any problems before it is too late.

On my server's scheduler I have set DiskFresh to run every 13 weeks, on a Saturday. It will refresh all 3 physical drives, a total of 4.3TB of storage space. The server can still run while this is happening, and the DiskFresh software detects how much other hard drive activity is happening, and will scale back if the drive gets too busy. It is very well behaved, and doesn't consume lots of computer resources or disk space. On completion of the refresh, you get a text report listing useful information and any errors encountered. FWIW, each 2TB drive took about 22 hours to refresh during testing.
There is a free version that you can use on your home PC or for testing in a business environment. There is even an option to schedule it to run quarterly. The commercial version has some extra features, and the installer allows you to also install Puran Defrag, Puran File Recovery and/or Puran Duplicate File Finder. All of these are essential tools for modern Windows PCs, and it is great to have them in a single setup program. It makes life easier for the IT technician. Choose from a 5PC license for small businesses, or a site license for 20 or more PCs.
Update: For a more detailed look at why DiskFresh is so useful, see my Fact-Reviews article on DiskFresh and why you need it.
Update 4 July 2013: DiskFresh is now at version 1.1, which fixes a bug encountered when using encryption in some cases.

Full disclosure: I was not paid to do any testing, nor do I receive any money for any sales of this product. I am grateful that Vishal listened to my rantings about the hard drive signal fading away, and I am glad to be able to use this utility on my computers: Windows 2012 Server, Windows Vista (laptop) and Windows 7 Home Premium (Test PC).

I just screwed up my blog!

I was fiddling around with some new blog settings and I totally wiped out all my carefully constructed blog settings.
Sorry folks! It's going to take a while to get it all back. Some of the items will take some time to figure out too. Please bear with me.
Update Friday 2 Sept: Not everything is fixed, but at least the site is usable again. I guess it was ready for a redesign, but not like that! I really must stop fiddling and get back to work.

Zero Day novel is scary and an exciting read

Mark Russinovich probably knows more about rootkits than most people on the planet, since he found some and wrote the first major tool to detect them. Now he has written a novel about viruses, computer security and government incompetence. And it is scary.
I thought my laptop was secure, but it isn't. The BIOS can be reprogrammed, the hard drive can we wiped out and any software running on it can be attacked and disabled, all by some cleverly written code. I called my antivirus company and asked them if they'd read the book. One of the guys had "heard about it".
So, my question is one that computer users hate to hear: when was the last time you checked your computer for security updates of ALL installed programs? When was the last time you made a backup? How about an off-line backup? Or an off-site backup? Will it be enough? Probably not.

Carbonite gets a new look

No sooner had the ink dried on my Carbonite review, than they updated their software. I guess that's the speed of the internet for you. The new version has a much better look, and is more useful. What I like the most is that there is now a progress bar that shows the current file it is backing up, and how far it has progressed.
There are other useful things, like the ability to freeze your backup while you are restoring files to the hard drive. I learnt of the update because there was an automatic update message on my machine. If you haven't seen the message, just go to the Carbonite web site, log in to your account, and choose the "Reinstall" button, and run through the instructions on screen. Simple.

Carbonite Backup Review

When you make backups, you should always have three copies: (1) the original, (2) one local backup and (3) one remote backup. A local copy is there for convenience and safety, and for when you upgrade your computer. Typically this would be stored on a CD or DVD, or an external hard drive that can be stored in a safe place.
The remote copy is for disasters like your building burning down or being flooded, or all your computer equipment being stolen. The remote copy is often ignored or neglected, because it's a hassle. Not any more, provided you have enough bandwidth (i.e. uncapped) and a reasonable speed internet connection. Read the rest of my Carbonite review here.
In doing research for this review I discovered some customers who thought that Carbonite could replace their external HDD backup, and/or could be used to migrate data from a PC to a Mac. I don't think so.
Note: While I am writing these reviews I am also running some new benchmarks for Windows 7, 64-bit and 32-bit. I haven't forgotten or abandoned the benchmarks for fact-reviews.com.

Backups? Who needs backups?

I make backups. Regularly. I have learnt the hard way, and rescued years of data as a result, in spite of two laptop hard drive failures. I have all my emails going back to 2001. Still, in the last two years I have wasted two weeks of valuable programming time as a result of bad backups. Ten years ago I threw away an entire day's worth of intensive programming by overwriting a file. That's when I developed Zippy. It has saved me from countless disasters simply by creating zip file backups on my hard drive. I also use an external hard drive to archive copies of important files, using SyncToy. And I have purchased Acronis TrueImage Home. But what happens if/when this drive dies or gets stolen? It sits next to my desk so it's not exactly a secure backup. Fortunately I am also a regular listener of the Security Now! podcast. So I started hearing about Carbonite, and after my last fiasco I decided to splash out and spend $54.95 (ZAR 431.77) for Carbonite's backup service for a year. And because I used the offer code in the podcast adverts, my subscription is renewable in May 2011. And If I install it on my wife's machine as well I get 50% off by using a link in the purchase receipt. South African readers will probably drool at the screen shot above. So far I have backed up nearly 6GB of data, with another 7GB to go. For most "broadband" users in South Africa that's double their monthly 3GB cap, and wireless/3G users are lucky to have a 1GB cap. Fortunately I have a bigger cap, and it is possible to manage the backup process better to restrict the size of the backup. If you have an "unlimited" data connection, consider yourself extremely lucky. You have no excuse for not doing backups. Carbonite is simple to install, easy to set up and works quietly in the background. It just gets on with the job, without interfering with normal work. You can pause it or stop it completely for a while, and restrict the hours when the backup runs. I haven't figured out whether it makes incremental backups of my Outlook data files, but I suspect that it does. Now I can sleep better, knowing that my most important files are also backed up automatically "in the cloud".

Update: M-Web has introduced "uncapped" shaped ADSL, so on Saturday I upgraded my R299 5GB account to a cheaper R129 uncapped 384kbps account. Finally SA enters the 21st century. Update 4 April: All the files are now backed up. It took a while but there were some big ones to do, particularly Outlook email archives and database files.

Oops! Backup not found

I managed to corrupt my network adapter settings on my Acer Extensa 5620 laptop. No problem, I do regular backups and I have an Acronis True Image sector-by-sector backup of my laptop.
Er, no. (!?) I have a backup file labelled "Donn5620.tib", but it turns out to be a backup of another machine. So now I have to use the factory restore CDs. Like I need this in my life! Maybe it's a blessing in disguise, but right now the disguise is pretty effective.
Update 2am: Leaving True Image to do a backup of the broken software, just in case things get any worse. Reinstall will happen later.
Update 11.30am: Backup finished overnight sometime. Now the Acer "restore to factory default" begins, but is interrupted by lunch, etc.
2pm: Windows boot for the "first" time. And the second time, and the third time ...
Now to uninstall the following:

• McAfee Antivirus
• Yahoo Toolbar
• Adobe Reader 8.1.0
• MS Business Contact Manager
• Office 2003 Web Components
• Office Small Business Connectivity Components
• Acer eDataSecurity Management

3pm: Next, driver downloads from Acer Web site, and start the long and painful process of Windows Update: 104 important and 7 optional (500MB) updates.
Install software:

• CCleaner 2.29
• NUAC, the only Symantec product I have liked in the past 10 years.
• ESET Smart Security
• Google Chrome 5, with Adthwart, FlashBlock, Xmarks Bookmark Sync, Blog This, Search the current site, Chrome Flags.
• TeraCopy 2.12
• DS Clock 2.4
• Puran Defrag 7
• Sun Java 6.0.180.74
• Prio - Priority Saver
• Adobe Reader 9.3.1
• TrueCrypt
• AutoHotKey 1.0.48.03
• PrimoPDF 3.2
• Droid Fonts, which work particularly well with DS Clock and Chrome.
• PeerBlock
• Core FTP LE 2.1
• KatMouse
• Fun Desktop Wallpaper Changer
• HD Tune 2.55

8pm: Break for dinner and allow Microsoft Update to finish.

10pm: Office 2007 Professional SP2 install.

Installed software:

• WinRar 3.92
• WinZip 9
• WinAmp 5.572
• SysInternals Suite
• SyncToy 2.1
• Search and Replace
• EditPad Pro
• Remote Administrator 3.4
• RescueTime
• GoldWave 5.55
• Microsoft DirectX (for Winamp)
• Vista Battery Saver
• Audible Manager
• Total Recorder Standard Edition

Midnight: I've had enough for one day. Will try more tomorrow. Downloaded 771MB today.

Monday 2am: Woke up in front of the computer. Ran Windows Update for the umpteenth time, to install Vista Service Pack 2 and numerous other updates. Used up another 219MB.

Monday 1pm: Returned to the office after a long sleep. Feeling like a bear with a sore head. Installing "last 3" Windows updates, including Windows Live Essentials, which is "only" 1.1Mb (I guess they mean the installer), but ends up being much more (90MB) once you choose the apps you want. Microsoft are such liars. And they use language badly: Windows keeps referring to "your" computer instead of "this" computer. They just don't think.

3pm: Re-reading my SQL Server 2008 Express checklist article in order to install it. Also installed a trial edition of Carbonite in case my backup drives get lost or stolen. Thanks to the adverts on TWiT I save some cash.

5pm: Finally finished installing and updating SQL Server 2008 Express SP1. Installation list:

• SM Player 0.6.8
• Putty
• ExamDiff
• PlaySound
• OverDrive Media Console
• Microsoft Access XP (2002)
• AllMusicConverter Platinum.
• Google Earth
• Google Photos Screensaver with Picasa
• HostsMan 3.0.45
• Jasc Paint Shop Pro 8.10
• Unlocker 1.8.8
• AllSnap 1.33

6.30pm: Back to Windows Update for Office XP SP3.

9pm: Installing Access 97, and SP2. Then Visual Basic 6 and its service packs, followed by yet another 22 security updates, some of which just refuse to install. Finally, a quick defrag and backup to call it a night.

Tuesday: forgot to install Nokia phone software, InnoSetup and ISTool. Restored stuff from the TrueImage backup and corrupted the entire file system. Copied data files to backup drive.

Wednesday: Restored a sector-by-sector backup, reinstalled missing apps, copied some data files back.

Thursday: Finally I have a working system to do a complete backup.

Acronis True Image Home 2009 and the feature that is really a bug (Updated)

I promised the Acronis support team that I would test Acronis True Image Home 2009 build 9709 on my laptop because they had said

I understand your issue and assure you of a quick resolution for the same. The time variation in progress bar might take place because of particular software/hardware configuration. There may be some applications running on the background that may have higher priority level than Acronis software.

Clearly they did not understand the issue because I was running from their boot CD and so there were no background applications. But to disprove the "particular software/hardware configuration" nonsense I ran the software on my Acer Extensa 5620 last night. (See below for the same bug on my FRAGG desktop machine.)
My first problem was getting the software to see the drive, because I protect it with a password. After removing the password we were ready to go. I selected all the partitions on the drive, and asked for a sector-by-sector backup, as before. Here is the summary screen:
The progress bar appears to work OK, with the top one moving faster than the bottom one, but not for long.
The overall time estimate has dropped from 10 minutes to 17 seconds, when it still has over 4 hours to go.
Notice how the total progress bar resets itself for the verification operation. Why?
To add further lunacy to the equation, the verification estimate jumps from 10 days to 3 minutes. It actually took 65 minutes.
Now for the final amazing feature: I selected the "Shutdown the computer after completion" option because it was late and it would have been nice to leave the computer and go to bed, knowing it would shut down when done.
Of course I had forgotten about this incredibly stupid dialog box. Even though it defaults to the "shutdown" option, if you don't click it during the few seconds it gives you (i.e. you've gone to sleep or left the machine unattended expecting it to shut down) then instead of shutting down it "reverts" the shutdown option and reboots instead.
What kind of feature is that? You select it expecting it to do a shutdown, but unless you're standing by the machine it doesn't. According to the "What's New" article,

Automatic computer shutdown after backup or restoration finishes. You can now perform a backup at night and go to sleep without bothering about turning off the computer - the program will do this on its own.

Except it doesn't do this on its own. You have to be there to confirm the operation within a few seconds or it "reverts" and does something else.

Update 4th April 2009: Acronis Customer Support have allowed me to change my purchase from the 2009 version to 11.0.8101 with a legit serial number. They also gave me a link to the boot CD ISO image, and it all works fine. The problems described still occur, and will probably be fixed in a future release.