How Fair is Load Shedding?

Two years ago an Electrical Engineering consultant was talking to my wife in the bookshop, and told her that Eskom had no contingency plans, but they had been warned about their lack of capacity and the effects of their non-maintenance policies. Now that we are experiencing this chaos first hand, the new web site www.poweroutages.co.za has helped to make it clear just how bad the planning is, and how unfair the power cuts really are. Of course it goes without saying that these timetables are purely on paper, and the actual situation is a lot worse.
I selected a few suburbs at random from a map of Johannesburg. The results are curious, but clearly unfair. Perhaps suburbs are graded according to usage patterns, but it looks pretty arbitrary to me.
Stage 1: (red) Eskom is overloaded by 1500 MW. This is the most likely type of power cut
Stage 2: Eskom is overloaded by 3000 MW. More locations will have no electricity
Stage 3: (brown) Eskom is overloaded by 4500 MW. Many areas are without power
Click on any graph for a full size view

Paulshof:
Lonehill:
Ferndale:
Cresta:
Zola:
Pimville:
Mondeor:
Jeppe:
Parkview:
Melrose:
Bramley:
Linbro:
Glen Austin A.H:
Alexandria:
Linksfield:
Sandringham:

Click to read the cartoon

Perhaps we should change the name to "Power Outrages", or just call it what it is: a failure of power.

BBC: South Africa cuts power to its neighbours

DMA: Privacy on a Shoestring

The illustrious Direct Marketing Association seems to be incapable of protecting my privacy, even among its own members. There is a form you can fill in, but only if you use Internet Explorer. Firefox and Opera aren't supported, even though most privacy-savvy users prefer Firefox with the NoScript extension enabled, and script blocked by default.
If you update your details you get an error message to say "You are already registered", and when I asked, they were unable or unwilling to tell me whether this meant that the data was thrown away or updated. Their "Do Not Call" database has me listed 3 times: twice with the same ID number (records 151759 and 185048) and once without my ID number but with my current address (record 193686). My wife is listed twice, but only with her maiden name, and we've been married for 7 years. So much for data accuracy.
Since DMA members are under no obligation to report back to the DMA, all my requests to FNB, Nedbank, Absa, Sanlam, Standard Bank, Cell-C, Autopage, Discovery, MTN, and the many other companies I have warned against sending me junk mail or SMS mssages, have not been reported back to the DMA, so my information is still woefully out of date.
As a database programmer I understand the complexities of keeping information up to date and accurate, but this is a simple flat file database with no field for first name, only initials. (Huh?) These guys are supposed to be protecting my privacy, but in my experince it is all window dressing. If not, why would MTN, Barclaycard and now Woolworths, all sent me unwanted marketing messages without even bothering to check to see if my name is on the list?

South African Google Earth

(Click for a larger picture)

South Africans have developed the ability to laugh at their problems, even under the most adverse circumstances. Our crime problems are being surpassed only by the random power blackouts that have been going on for months and are likely to continue for the next 8 years or so.
Today I visited my friend Dennis who is recuperating from gunshot wounds he received from 3 armed men, while out walking his dogs with his wife on Boxing day. One bullet narrowly missed his heart, went through his colon and liver. The second bullet injured his left leg, before lodging behind the knee of his right leg. Three weeks later he can barely walk, so I popped round to his house to help him set up some computers in his wife's study.
We couldn't test everything immediately, because of the 3 hour power blackout (Eskom calls it "Load Shedding") from 4pm to 7pm (their second one today) so I did a lot of running around with torches and candles first. I also sat through 90 minutes of traffic congestion to get there, caused by the power outages and rainy weather.
In case your're wondering what "Google-letu" is all about, Guguletu is a well known "township" area.

Power Cuts are a National Disgrace | Fifteen Years Too Late

Flash Drive Defrag Warning

I admit I've done it myself without giving it a second thought, but defragging your flash drive makes no sense at all. What's more, it will shorten the life of your drive.
Fragmentation is an issue for hard drives because they are mechanical devices with seek times and (to a lesser extent) varying read speeds. If you could reduce the seek time of a hard drive to zero, there would be no need to defrag the drive, because every sector of the drive would take just as long to read, so the performance improvement of rearranging the sectors would be zero. In a mechanical drive this will never happen, simply because the laws of physics tell us that any object with mass and velocity requires time to reach its destination, and time to change course.
In any RAM-based system, the rules change. It takes just as long to read one memory address as another. If it didn't, then our software would behave very differently. So if you want to read 2k from memory address A00000 it will take as long as if you wanted to read it from address B00000 or 000000. There is no "seek" time, just a "read" time. Similarly, reading 4k from two different addresses (2 x 2k) will take as long as reading them from adjacent addresses.
I have ignored the effects of caching for the purposes of this example, because I am assuming the worst case, where no caching has occurred, either for the hard drive or the RAM drive.
So why did I defrag it the first time round? I guess I associate "defrag" with "tidy up". There used to be a Norton Utility called DS for Directory Sort, and on slow DOS systems with a lot of file names in a single folder, DS helped quite a bit. Nowadays Windows does the sorting for you, so provided the directories are not too long, you don't even notice the sort order of a folder.
Now for the fun part: Flash Drives have a limited lifespan: typically 100,000-300,000 writes. There is no restriction on the number of times you can read the drive, just write it. On a well-designed drive, this means you would have to write new information to the entire drive 27 times a day for 10-30 years before the drive is exhausted. On a cheap or badly-designed drive, the story is different. Any change to the size (in clusters) or location of the file affects the File Allocation Table (FAT), so the FAT gets rewritten each time such a change is made. On a badly-designed flash drive, the FAT is rewritten to the same memory location each time, so that individual memory area will fail after 100000 writes, resulting in data loss. Most flash drives use the FAT32 file system.
Bear in mind that a defrag program changes the FAT after each cluster (or group of clusters) is moved, so that defragmenting a file with 27 fragments with resut in 27 changes to the FAT, and therefore 27 writes to the same area of memory. This is done to prevent data loss due to a reset or power failure. Defrag programs typically remove hundreds if not thousands of fragments, so a defrag pass has the potential of creating thousands of write requests to the FAT, as well as all the other reads and writes going on in other areas of the drive.
If your defrag program removes 100000 fragments, it has the potential to permanently damage the flash drive! Without wanting to sound alarmist, it is something to consider, and since you have no way of knowing whether your flash drive is well designed or not, it isn't worth the risk. Why waste time defragging to gain 0% performance improvement in exchange for the risk of 100% data loss? The technique for avoiding this situation is called "wear levelling", but not all manufacturers use it. I found the following description

Longevity/Lifespan
Unlike DRAM, flash memory chips have a limited lifespan. Further, different flash chips have a different number of write cycles before errors start to occur. Flash chips with 300,000 write cycles are common, and currently the best flash chips are rated at 1,000,000 write cycles per block (with 8,000 blocks per chip). Now, just because a flash chip has a given write cycle rating, it doesn't mean that the chip will self-destruct as soon as that threshold is reached. It means that a flash chip with a 1 million Erase/Write endurance threshold limit will have only 0.02 percent of the sample population turn into a bad block when the write threshold is reached for that block.
The better flash solid state flash drive manufacturers have two ways to increase the longevity of the drives: First, a "balancing" algorithm is used. This monitors how many times each disk block has been written. This will greatly extend the life of the drive. The better manufacturers have "wear-leveling" algorithms that balance the data intelligently, avoiding both exacerbating the wearing of the blocks and "thrashing" of the disk: When a given block has been written above a certain percentage threshold, the solid state flash drive will (in the background, avoiding performance decreases) swap the data in that block with the data in a block that has exhibited a "read-only-like" characteristic.
Second, should bad blocks occur, they are mapped out as they would be on a rotating disk. With usage patterns of writing gigabytes per day, each flash-based solid state flash drive should last hundreds of years, depending on capacity. If it has a DRAM cache, it'll last even longer.

This quote sounds optimistic, because it is written by a flash drive manufacturer. I've used several USB Flash drives, (and lost 2 of them) and the chance of data loss by misplacing the entire device is higher than wearing it out, but the point I'm trying to make is that flash drives are not designed or intended to be defragmented.
Hard drives wear out too, but defragging can help extend the life of a mechanical hard drive, not to mention improve performance. The same is not true for flash drives. You have been warned!

Merry Christmas from Musica - NOT!

This is the first and last time I ever try to buy anything from Musica.co.za. I ordered Johnny Clegg's "One Life" CD and a live DVD from them on 22nd December, hoping to have it delivered to my brother in Port Elizabeth for Christmas. Granted, it was a last minute purchase, but after 2 weeks they have still not delivered anything. Thanks for screwing up a Christmas gift for me, Musica! This is the way to grow your business and build customer loyalty!
By contrast, I had ordered the same CD from Amazon.co.uk, had it gift wrapped and delivered to my other brother in the UK. Both orders were placed within 30 minutes of one another. The Amazon web site told me there were only 2-3 items in stock, and promised delivery before 1pm on 24th December. Since it was Christmas Eve I thought this was a little optimistic, but it arrived, gift wrapped with a card, on 27th December (the next working day). Pretty darn good for an order placed on another side of the globe during the Christmas rush. Dozens of things could have gone wrong; but Amazon got it right.
That reminds me of kalahari.net, our local bookstore and music outlet. We have bought several books from them, and their service is fantastic. I looked first on their site, but they didn't have the Johnny Clegg albums. I also tried the Look & Listen site, but it told me they didn't have stock, and their aren't any outlets in PE.
I used the Musica site to find shops in the PE area and phoned them to find out if they had stock: they didn't. That's when I made the mistake of ordering from their web site. I assumed that since it took my order that they had stock and would be able to deliver. How wrong I was.
After placing the order I got an order confirmation on Saturday 22nd at 9am, stating

"Your order has been received and is being processed. Your gift recipient's parcel will be shipped within 1-2 WORKING DAYS depending on availability."

On Monday at 1pm (their first working day) I got another email that said

" ... our online warehouse is unfortunately out of stock of both these titles.
"We have placed an order to obtain the stock directly from our stores thus there may be a delay in the shipping of your order."

Since then I have heard nothing from them. I called on Thursday 3rd January (their 6th working day) and was told that the DVD was available but the CD was out of stock, and they offered to send the DVD in the meantime. I decided to wait and see if I could get the CD myself.
I looked on the Look & Listen site and found they still didn't have stock, so I phoned our local outlet and then their Hyde Park store. They had one CD left, which they kept aside for me until I could pop in and buy it.
This morning I phoned Musica Customer Services (0860-687422) again to find out the status of order number 50070. The lady who answered the phone was much more helpful and explained that the order is automatically cancelled after 7 days, but they had had the CD in stock for a few days but the DVD was unavailable until their suppliers reopened. I asked why I hadn't been told any of this and all she could do was apologise. Their was no explanation for why they hadn't bothered to contact me to let me know that the order had expired, or that they had located the CD.
I asked her to get the Managing Director to call me. She said he was still on holiday but their IT Director would call me on Monday; she said she was the Customer Services Manager and that customer feedback was important to them. She also said they had a banner on their web site advising customers to place their Christmas orders before 16th December, to give them 5 working days to deliver everything. There isn't much point telling me that on 5th January!
I count several system failures:

• The web site didn't/couldn't tell me that there was no stock. The best it can manage is a hugely over-optimistic "Usually shipped within 1-2 working days". Don't believe it.
  
• The second email didn't warn me that there may be a much longer delay in obtaining stock. "There may be a delay" says nothing.
  
• I was not told that my order had expired after 7 days
• I was not told that the CD had been located and was available for shipping.
  
• I was incorrectly told that the DVD was available, and was not warned that the order had already expired. I only discovered this by calling them.
  
• There was no record of my first call when I made the second call.
  

These guys clearly have no idea how to run a web site. Rather use kalahari.net (which is brilliant) or try Look & Listen. You have been warned!
Update Mon 7th January: The MD contacted me, and promised to send me his contact details and I asked him to post a comment on this blog once the system has been improved. Stand by.