Internet Solutions/ISDSL Epic Failure

On Friday I went to work for my biggest client, who uses Internet Solutions as their "service" provider. More specifically, they use ISDSL, the ADSL part of the company.
On Friday basically nothing was working: you couldn't use Carbonite, DropBox or even Google Search, Google Drive or Gmail. Twitter.com, like most dotcom sites, was inaccessible. It turns out that one of the major undersea cables, SAT3, was either broken or under "maintenance", and a number of ISPs were affected. Why single out Internet Solutions? Because they are the biggest commercial ISP, and at one time they were the best ISP. Sadly no more.
As a client we received no notification from ISDSL, and they wouldn't reply to my questions because I'm not on their Technical Support list. Instead, the only public acknowledgement of the problem was on Twitter, but you needed to use a different ISP to see the message.
The graphs above show the story pretty well: each graph shows the download speed for several web sites I monitor: www.fishwisepro.com, www.fact-reviews.com, and www.openaccess.co.za. It measures how long an HTML page takes to download. These graphs are the average of all 4 sites. The top graph is for 2 weeks ago, the bottom graph is the past week. They grey area is the weekend. Notice the large dip on Friday, and another on Saturday. Clearly they don't have a backup plan for this kind of disaster.
My own ISP, M-Web, also uses the SAT3 cable, along with the SEACOM cable. They suffered downtime some time back, but clearly they have learnt how to deal with this issue. My home ADSL service was totally unaffected. That's how I was able to read the brain-dead tweets from @IS_global. It was imposible to see these tweets from an ISDSL connection. They clearly have no idea what the word "revert" means.

Update Tuesday 3rd July: Duncan Alfers from IS has been in touch, and I have provided him with contact details for my main client, so we can be warned about future problems.

Another Day, Another Armed Robbery (Again)

They say Africa isn't for sissies: today I was visiting the home of Dennis and Sally Polack, to work on their Fishwise Pro web site, We had just finished lunch when Sally started screaming: an armed robbery was in progress at their house.
They had already tied up two of the farm workers and robbed them of cash and cell phones, and were now intent on taking whatever they could from the main house. One of the bastards pointed a firearm at me, although somewhat briefly. Then he ran away when he heard me calling to Dennis to get his firearm.
I called 10111 and the cops were here in minutes. They were amazing. They might even have got the guys once they ran away. It turns out there were 8 of them. The problem is to know when or whether they will be back.
Crime affects us all: I have been mugged twice, and my car broken into twice. My wife has been mugged, and been in 2 armed robberies when she worked at the bank. She saw a security guard shot and killed in front of her. Dennis was shot on his smallholding 4 years ago. My brother has been hijacked, and his house has been robbed. My in-laws have had their house burgled twice. My main client has had several laptops stolen during armed robberies, and had a few after-hours breakins as well. Cars have been stolen from the flats where we live, and three flats have been burgled.
And so the list goes on, while our glorious leaders try to deny the problem and make lame excuses. They are lazy, corrupt, greedy and inept. They deliberately under-fund the police, and siphon off government money in "arms deals" and dodgy tenders that lead to a lack of service delivery. One of these days we'll wake up and vote the criminals out of office. But by then the economy will be in tatters and all the good people will be dead or living in other countries.
Update: The male worker who was attacked first was threatened by being told the name of his 6-year-old son, who is currently visiting his granny. When I saw him he was in tears, shattered and scared for his safety and that of his family. The cold-blooded bastards who did this care nothing about the impact of their actions. They are usually high on drugs when they try this kind of stunt anyway. A pox on the lot of them!
Update 9pm: Dennis retuned from hospital with 22 stitches. He fell down the stairs during the robbery. He's fine but has a sore back.
Update 30 Jan 2012: The bastards came back, dressed in provincial police uniforms, and gained access to the house under the guise of wanting to take further statements. They made off with money and a pistol, and trashed Dennis' office. They also made off with two bunches of keys, so Dennis and Sally have to replace all the locks on all the buildings and outbuildings. Even their safe keys were taken. Hopefully these guys won't be back, but no-one is holding their breath.

FishWisePro gets a search facility

I have been working on the development of the FishWisePro web site for some time. I only get to work on it one day a week, with plenty of breaks when the owner goes on underwater photographic expeditions. He has published several GB of his fish pictures.
The web site is mostly data, with around 800MB of fish data, and a lot of pictures. The problem until now has been locating the data of interest to a visitor. It has been relatively easy to browse the data, although quite inefficient. In the past fortnight I have been able to add a "Quick Search" facility that makes it simple to run a search on the data. I'm quite proud of it.
Hopefully this will make the site a lot more usable, and will encourage visitors to return again for more information. Until now we have been relying on Google search results, but it seems to get bogged down in the data and its results are not good.
FishWisePro is a comprehensive, fully relational fish database of more than 99,200 scientific species name combinations and just over 34,000 fish pictures. It has been specifically developed for Academics, Students, Marine Biologists, Authors, Ichthyologists and all other serious users interested in Fish Taxonomy. I would be interested in hearing your feedback, impressions, and design suggestions. Please bear in mind we are trying to keep the site design simple and functional.

How to find the real Nemo

Nemo belongs to a group of fish commonly known as Clownfish. The scientists have given Nemo the scientific name, Amphiprion percula. Clownfish have a very interesting relationship with the anemones in which they live and produce their young. Because of this relationship, Clownfish are often also known as Anemone Fish. Although Nemo is now bred in large numbers in tanks, owning a Nemo clownfish impacts on the fish populations of the oceans, because some are still captured from the wild for the aquarium trade. I know this because I have been helping Dennis Polack with Fishwise Pro, an extensive fish database that he is publishing on the web at www.fishwisepro.com.
To quote the instructions on the web site: Go to Common Names, click in the "Used In" column filter and type "Disney". This will show all the Disney cartoon characters that are fish. Look for "Nemo" and click on the record. Click on the "Back: Species" link to find out about Nemo's Species. Click on the species record and choose "Pictures" to find 11 photos of Nemo's real life relatives, Amphiprion percula ("Clownfish").
Sally Polack took the best picture, shown here. Dennis would also appreciate any comments or feedback on the site. There are currently 33,200 pictures on the site, covering most of the known fish species.

After reading the October 2010 edition of National Geographic, I wonder how long it will be before Nemo and most other fishes are extinct. Read "Time for a Sea Change" and buy the magazine to see the maps of how we are over-fishing the oceans. Then check out their 2007 coverage of the oceans, and weep.

WebHost4Lies: incompetence as a business model

WebHost4Life proudly proclaims two outright lies on the front page of their web site. The first is that they provide "99% uptime" by which they mean (presumably) that you can access your web site 99% of the time if they host it. Not so. See my actual measurements below:
Nowhere near 99%. That means that if you tried to visit a page on FishWisePro.com then in some cases you would have waited longer than two minutes for the page. That's when internetsupervision.com logs a "timeout". These measurements are made by an independent company with multiple testing servers around the world.
The second lie is that they are "powered by 100% wind energy". Wrong. They have diesel generators for backup and they get their power from conventional sources. Their "Green Hosting" page states

Since generating wind energy on site isn't feasible, we're offsetting all of our electricity use with wind-generated Renewable Energy Certificates

That's very different from being powered by wind energy. It's a bit like saying "I'm not a racist because I know some darkies who aren't racist."
So from now onwards I am calling them "WebHost4Lies" because that's what they do: they lie, either directly or by omission. It's part of their business model. If you don't like it then they are quite happy if you leave, because they have too many customers anyway, and their tech support staff are too overloaded to bother. They aren't delivering on their promise of "an outstanding customer experience" because they aren't enjoying "an outstanding employee experience".

WebHost4Life? Not on your life! They suck

I originally thought they were just plain incompetent, but now I think they are slow, unreliable and incompetent. Which is a polite way of saying that WebHost4Life sucks. Badly.
The graph above shows that they have failed to provide a continuous service on the Fishwisepro.com site for the past month. Then check out the speed in the right hand column:
Now compare this with my new hosting company:
So WebHost4Life is slower. Much slower. When I measured this I also measured the "time to first byte" as 175 seconds, and a total load time of 298 seconds. (The numbers in are in the status bar on this screen shot):
Since 300 seconds is 5 minutes, we are talking about a very slow load time indeed. Most users would have cancelled the page after 10 to 15 seconds, so they would never have seen the page at all. I'm told this was caused by their "migration scripts". The ones that screw up your web site for you.
Maybe I should also add useless to the list. It certainly describes their help system, and their much-vaunted "migration" process. Incidentally, the "new platform" doesn't support multiple web sites. They proved it to me because they couldn't get it to work, although they wouldn't actually admit that it couldn't be done. They are under the false impression that it can be done, but when push comes to shove it doesn't work. How sad is that?
Update: We can add "losers" to the list. According to www.web-hosting-top.com, WebHost4Life's hosted domains grew for 2007-2009 from 62,076 to 70,669 domains. In 2010 the decline has set in, so that by May 17, 2010 they are down to 60,912 domains, a loss of 16% in just 5 months. It boggles the mind that they could throw away 10,000 customers so fast.

Another satisfied WH4L customer (not!)

Mustang waves goodbye to WebHost4Life

The Mustang web site has found a new home, thanks to the technical incompetence and bad planning of the Endurance International Group, the new owners of WebHost4Life. The DNS transfer from WebHost4Life to Arvixe should be completed in the next 24 hours. This blog uses graphics from the Mustang site, so there might be a hiccup or two, but I'm not expecting anything like the fiasco that occurred when WebHost4Life screwed up the FishWisePro web site.
The difference between WebHost4Life and Arvixe is like chalk and cheese. Arvixe sends you an email that explains how to set up everything. Their control panel is easy to use and mostly intuitive. Their database setup is totally straightforward, and even the domain transfer instructions are clear and easy to follow.
Two of my support questions were already answered in the support knowledgebase, and two other questions were answered by a technical support person who actually knew what was going on. Quite a change from the idiotic and nonsensical responses I got from even the most senior people at WebHost4Life.
They automatically created a test domain called http://mustang.co.za.lilac.arvixe.com that allows me to test the setup before changing the DNS settings. I'm a very satisfied customer.
Update Thursday 2pm: The DNS changes have been implemented, and the switchover went completely without hitch. Unlike the balls-up with rate-my.biz (now fixed) and rate-my-biz.com ("cancelled" by WebHost4Life). Why is it that they break everything they touch?

WebHost4Life breaks FishWisePro web site for 12 days

FishWisePro is a web site that provides information on all known species of fish. It is published by the author of www.fishwise.co.za, Dennis Polack. I have been privileged to help Dennis by developing the web site pages that extract the data from a SQL database and display it for the user.
We chose WebHost4Life last year because they offered enough disk space and database space to support a large database such as this. Now they have "transitioned" the site to their "new platform" and predictably it doesn't work.
I visited the site today only to discover the site is completely blank. Fortunately the data upload is going according to schedule, because we reorganised the database structure, and I re-generated all the data support pages to match the structure a week ago. I wonder how long the site will be down this time?
Update 28 March (day 1): I logged ticket no 6883865 and called international long distance to the USA to their "support" number. So far all I have is the following boilerplate response:

Hello,
Thank you for contacting Support.
We apologize for any inconvenience this may have caused you. I have checked your website http://fishwisepro.com and I was able to duplicate the issue. I have asked a member of our team who specializes in website issue to review your account. You should be hearing from this specialists within 24-48 hours. If you have any questions in the meantime, please let us know and be sure to refer to the link http://www.webhost4life.com/member/sconsole for the quickest service.
Thank you for choosing WebHost4Life, we appreciate your support.
Sincerely,
Bryan Smith
Customer Support

So basically they haven't admitted it but they have screwed up and now I must wait for 2 days for them to look into it. I'm willing to bet it takes longer than 48 hours.
Update Tuesday 30 March (day 3): as expected, the issue is not resolved, so I have changed the front page on the old platform in the hope of embarrassing them into changing to the new platform. My wife calls this "guerilla warfare". I call it revenge. As soon as they let me edit the files on the "new platform" I will be able to fix any errors, or at least tell them what's broken. Right now they are just wasting everyone's time.
Update Thursday (day 5): It's now 5 days since they stuffed it up, and finally we are on the new platform after much begging and pleading. Overnight they screwed up the site again, redirecting everything to a blank page. But at least now I have access to the files so I can repair the damage. The migration is 34 days late. Not bad for a 2-bit hosting company.
Friday 2 April, 14h20 SAST (day 6): I came in to the office to work on the web site (it's a public holiday in SA) and was greeted with an HTTP error 503. Basically that means the web server is broken. Last night the SQL Server vcnsql82.webhost4life.com was broken or flaky. Today its backup wizard doesn't work. They are obviously having a great time in the server room. I wonder how many days it will take to fix?
Update Saturday (day 7): According to ticket #6902824 the problem was resolved after 22 hours, but the site still isn't working. So now we have had to reopen the ticket. That means the site has been unusable for pretty much an entire week.
I'm not the conspiratorial sort, but I get suspicious about the slow progress in fixing these problems. Especially when they offer a special "Premium Support" option for an additional $29.95 per month. Read between the lines in this sales pitch:

All of your support will be delivered by a team of North American-based Premium Agents.
You'll have access to a customized online Support Console that you can use and route online requests to our Premium agents.
All of your requests will be expedited. So if your Premium Agent needs assistance fulfilling your request, our Tier II agents and Engineers will give it special priority.

Does that mean I'm not getting support from North American-based agents at present? Does it mean that "normal" support requests don't get priority? The usually say that I should expect a response within 24-48 hours, and in the "how did we do?" survey they ask if the problem was resolved within 72 hours.

What Makes a Premium Agent?
The most experienced agents, those with the longest tenure at WebHost4Life;
Agents with the highest internal quality scores, based on audits conducted by our Management Team;
Customer Favorites! Premium Agents consistently receive high customer-satisfaction ratings from customers like you.

In other words if you pay four times the "Advanced Hosting Plan" you get the "A" team who are not nearly as useless as the current "B" team. Spot the ripoff. Now there is talk of a class action lawsuit in the USA, and a community action group of current and past WebHost4Life customers.

Update Sunday (day 8): It is now 8 days since the site was transferred, and the "B" team still insists on trying to stuff up the site. I have finally fixed it myself. Hopefully they don't break anything else.
Wednesday 7th (day 9): The site is down again, and remained flaky the whole day. It either gave out partial pages or refused connections completely. I had to report the problem twice, because the first "help" chat agent refused to log the fault. Ticket #6918854 says they are still "researching" the problem. It took them 5 hours to start working on the issue.
Sunday 11 April (day 10): Site up and down again, refusing connections. Error page not working, stats not working. So far I have been in contact with their abysmal chat line 17 times in 2 weeks. Total time wasted: in excess of 24 working hours. Cost: at least $1200, excluding phone calls and bandwidth costs.

Monday 12 April (day 11): Site still refusing connections some of the time. I wasted 3 hours this morning trying to make a simple change to the site. It seems that there is some kind of delay between uploading the new pages and being able to see them. Stats seem to have been fixed. Error page is still broken. I chatted to a "sales" agent this afternoon and asked her the difference between the "old platform" and the "new platform". She got an error visiting the new platform. Theoretically it should be the same speed or faster than the old, but it isn't. Informal testing shows it to be 30% slower, if it works at all.
Update: one of their "Escalated Support" people said he was getting fast load times (within 1 sec) measured using a Firefox add-in called "lori". Here are my results:

"Old platform": Time to first byte: 5.8 sec, time to completion: 43.8 sec
"New platform": Time to first byte: 17 sec, time to completion: 63.9 sec. Depending on how you do the maths, that's anything from 45% to 193% slower on the "new" platform.

Wed 14 April: The refused connections problem seems to have been solved, thanks to the efforts of a competent technician in the "Escalated Support" department, but the server response times are patchy, from being really fast to painfully slow. At least it works, sort of. Maybe I'll get to keep my job. We are still a long way from their mission statement:

Delight customers and help them succeed by profitably delivering superior solutions and an outstanding customer experience.

Their customers think otherwise.
Thursday 15 April (day 12): last night these bright sparks enhanced my outstanding customer experience by deleting the Mustang.co.za SQL database. I wonder what else they plan to break?

Friday 16 April: It turns out they didn't delete the database, just took it offline for 12 hours. That means the database servers will only be 99.863% reliable this year, even if nothing else goes wrong.

Saturday 17 April: Discovered that the Visitor Stats last updated: 04/13/2010 4:55 PM EDT, which was a day before the SQL servers had their glitch. So there are no visitor stats for 13,14,15,16 and 17 April. Their standard response:

I was able to duplicate your issue. In order to investigate further, I need to escalate the issue to one of our technical specialists. You should be hearing from this specialist within 24-48 hours.

They also changed the stats settings to use AWStats instead of Webalyzer. I wonder why they did that?
Mon 19 April: Today's "outstanding customer experience" is the discovery that the FTP service has been randomly refusing connections all weekend. And another day's worth of server logs is missing. I'm really not looking for these problems: I note them as I bump into them, that's all.

WebHost4Life: why I got so angry | About the Endurance International Group (EIG) | WebHost4Life breaks FishWisePro web site for 12 days | WebHost4Life domain registration fiasco

Please try to break my web site

I'm really proud of this web site. It has taken me hours and hours of work, and it is capable of quite a bit of database editing. Now I want you to try to break it, or at least find bugs.
Until the end of July the site will use the Microsoft Northwind Traders database, a clump of test data used by programmers to develop and test database applications. You should have full editing rights to all records, so screwing up the data doesn't count as a bug, unless it breaks the web site somehow.
I am also looking for feedback on how intuitive the site is, and how easy it is to use the features. I'd also appreciate constructive comments about the layout, styling, and so on. Remember that the site is trying to look plain and simple, because the data is the most important part. You can visit the site at www.fishwisepro.com.
It uses Microsoft IIS7 and ASP.NET 3.5, together with a really cool web control called ASPxGridView from www.devexpress.com. The site is hosted by www.webhost4life.com in the USA. If you manage to break it, please let me know what you did. If you are using a browser other than Internet Explorer 8, Mozilla Firefox 3.5 or Google Chrome 2.0.172.37, then let me know if it does anything weird. Thanks in advance

Update: Testing is over. Thanks for all who helped. I will post another version of this site at www.mustang.co.za at some point.

As the mafia would say: if you breaka my site, I smasha your face!