522 113 hours without electricity!

Trying to get work done at the Wimpy in Cresta

The joys of living in ANC-run Joburg! Extended outrages in between load shedding. Last year in September we had a 66 hour power outrage. During CIDC2023 in September, I had to use a studio flat with solar power (normally R7,500 per month rental) to attend the conference because we were having load shedding several times a day. Fortunately for me, the flat was empty and the owners generously allowed me to use it for free, including the Wi-Fi. Wonderful neighbours.
As I write this, we have just finished an ordeal of 113 hours without power, despite there being no national load shedding at all! The power went off unexpectedly on Wednesday 18 September, around 14h50. I logged a ticket (reference no CPWEB4032200) along with the rest of the suburb, and we waited for news. A cable had shorted out and caught fire at the Windsor substation, so naturally City Power had to wait for Eskom. And that's where the lying, cover-ups, PR speak and obfuscation began. At 01h30 on Thursday  City Power closed the ticket. Very sneaky.
By the time I woke up in the morning (Thursday) it was too late to "escalate" the ticket, so I got a new reference number: CPWEB4032870. I spent several hours (and R150) at Bootleggers in Cresta, using their WiFi and recharging my phone and laptop. It turned out that they didn't have any decaf coffee, so I had to settle for some lemon iced tea and a delicious chocolate brownie. I could catch up on emails and get some work done until 6pm.
On Friday I decided to try the Wild Falcon Spur at Heathway (R300), in the hopes they would have decaf coffee. No such luck with the coffee, but the service was good, and the rooibos tea and hamburger was fine. My waiter had the politeness to leave me alone, and few interruptions.

The power came back at 2pm, and so I went back home to carry on catching up with work. All was fine until 20h10 when the power went off again! This time Eskom and City Power were determined to ruin our entire weekend. CPWEB4033907.
On Saturday I went to the gym for a shower and some sanity. Another neighbour offered anyone in the suburb the chance to use their solar power, so I made use of the opportunity for a few hours while my phone recharged and I could use my laptop. Excuses and various (inaccurate and misleading) ETR promises were made and broken by City Power, who naturally tried to shift the blame to Eskom. We listened to the Rugby World Cup semifinals by candlelight on the radio. Another win for the Bokke by a single point, and SA is in to the finals against the All Blacks.
On Sunday I decided to try to take the day off, and went to the gym again, to get rid of my frustration as much as getting some exercise. Eskom were still busy cutting trees (you can't make this stuff up) and then all of a sudden we were given a 17h00 ETR that came and went, and still no power. By this stage four City of Joburg Wards were being affected by the outrage(s). Later we heard that Eskom had restored power to Windsor substattion, just in time for City Power to switch on the Cresta substation and cause another trip. Then City Power needed to wait until Monday for a testing team to come and see what the problem was at Cresta.
On Monday City Power closed my ticket with "Repairs Completed" at 05h16, and I managed to "escalate" it at 07h19 when I woke up. After sleeping in late and making breakfast on a gas cooker, I decided to try out the decaf coffee at the Cresta Wimpy.

Finally! A decent decaf cappuccino, power, and Wi-Fi (but a bit slow). I could hardly believe my luck when I heard that the power had been restored at 1pm. Let's see how long it lasts.
In the meantime, I have learnt a few things about Eskom: they hadn't done maintenance on an "oil cable" for over a year. The contractors they used dug through a live cable in the dark because they didn't bring any lights with them. A real professional outfit! They only inspect their overhead power lines for trees that cause shorting when a short actually happens, and on the weekend, so they have to pay extra for the trees to be cut back. Eskom has no guards for their urban substations, and they usually take 8 hours to show up when called by City Power.
Things I have learnt about City Power: their infrastructure is crumbling at such a rate that they deal with an average of 4000 tickets per day. They do little or no maintenance, and if they can "back feed" a substation or area instead of fixing it, they will do that. They lurch from one fiasco to the next, and never return to do proper repairs. Likewise, they don't have enough cable testing teams, even though most of their problems involve faulty cables. In our suburb they don't lock the substations, so we provide locks for them. In the neighbouring suburb, they insist on using their own padlocks, but they don't always lock them. Go figure. They don't seem to have any lights for working at night, other than the torch on their cell phones.

The ANC has succeeded in damaging and sabotaging more infrastructure since 1994 than they managed in the decades of "armed struggle" before 1994. They are a disgrace and a national embarassment.

Both City Power and Eskom are "run" by ANC "cadre deployees", who are usually lazy, corrupt and clueless. Most ANC municipalities are bankrupt and dysfunctional, as are all State Owned Enterprises, of which Eskom, Transnet, PRASA and the SABC are prime examples.
The Joburg municipality is being systematically run into the ground by the ANC politicians and municipal employees for the last 3 decades, so none of this should come as a surprise, but still it is shocking to have experience it hitting you in the face for 5 days in a row. So I have decided to delay my rates payments by 5 days every month, in a quiet gesture of reciprocity.

Update Tuesday 24 October: Beeld newspaper published this article:

Update Monday 13 November: Yet another City Power blackout. Started at 13:10 on Monday. Still no power on Tuesday, so I enjoyed the Wi-Fi and coffee at Bootleggers in Cresta from 09:30 to 17:00. It finally came back after 30 hours, at 19:30.

Unplanned outrages in Aldara Park for 2023

excluding load shedding:

1. 12 Jan CPWEB3678171 1 hour
2. 13 Jan CPWEB3682778 7 hours
3. 23 Jan CPWEB3694706 3 hours
4. 27-28 Jan CPWEB3698875 12 hours

5. 04 Feb CPWEB3707256 11 hours
6. 12 Feb CPWEB3718828 2 hours

7. 03 Mar CPWEB3788482 2 hours

8. 05 Apr CPWEB3783338 1 hour
9. 08 Apr CPWEB3785526 2 hours
10. 26 Apr CP2989110 9 hours
11. 28 Apr CPWEB3810728 7 hours

12. 02 May CPWEB3813314 5 hours

13. 22 Jul CP3032305 8 hours

14. 01-02 Sep CPWEB3982795 9 hours
15. 03-04 Sep CPWEB3984580 12 hours
16. 11-12 Sep CPWEB3997968 26 hours
17. 29 Sep CPWEB4018162 13 hours

18. 18-23 Oct CPWEB4032200 CPWEB4032870 CPWEB4033907 113 hours (6 days)

6 Nov: City Power "successfully" takes over load shedding switching duties from Eskom.

19. 09 Nov CPWEB4057606 6 hours
20. 13-14 Nov CPWEB4057606 30 hours
21. 22 Nov CPWEB4076387 8 hours

23 Nov: City Power modifies "successful" load shedding schedule after massive outcry from residents.

22. 23 Nov CPWEB4078336 1 hour
23. 26 Nov CPWEB4081778 8 hours

24. 6-14 Dec CPWEB4097231 204 hours.
25. 18-19 Dec CPWEB4110976 22 hours.

Total 522 hours (21 days, 18 hours). Outrages on 44 days. That's an average of nearly 12 hours per outrage. So my 5 day delay in paying my rates has escalated to a month.

Editor's note: I have deliberately misspelt outage as outrage to make a point. Not that anyone at City Power or the City of Joburg will get it.
Update Friday 24 November: Eskom has declared Stage 6 load shedding for the weekend and City Power can't switch everyone on or off on time because they don't have enough qualified staff to flick the switches without electrocuting themselves. So last night (CPWEB4078336 - cancelled) we came on 50 minutes late, and on Tuesday (CPWEB4075389 - cancelled) it was 25 minutes late off, and 50 minutes late back on.
Update Sunday 26 November: The power "tripped" at 14h50 but since "load shedding" was supposed to be from 14h00 to 16h30 we only reported it at 16h45. It came back on after 8 hours at 22h40.
Update Wednesday 6 December: Power went off at 7am on Wednesday. CPWEB4097231, CPWEB4097063, CPWEB4095858, CPWEB4095833, CPWEB4095843, CPWEB4097214, CP3099926, CPWEB4100022, CPWEB4095809, CPWEB4101046. Power still not back after 8 days, and no sign of anything being done anytime soon.

On Thursday the "test team" disconnected one of the phases, and then left the substation unlocked. We haven't seen them since.

On Saturday afternoon there was some brief excitement when this truck arrived and then left a few minutes later.
On Saturday evening the "test team" arrived, and determined exactly the same as the Thursday test team: the cable is faulty.

They left the cable in this condition: one phase running hot to supply some of the houses, The rest are still without power, and the generator for Carvers Restaurant is still running after being shut down when the restaurant closed on Sunday at 5pm. Finally at 7pm power was temporarily restored to all but one home: CPWEB4101046 and the restaurant CP3191075. Now we wait for a new cable, and pray the faulty one doesn't burn out with too much power on the yellow phase.

On Tuesday a team arrived to start digging. A resident wrote: "They have no work permit. No excavation permit. No lock out permits. No diagram of how the wires are in the ground. ... Not sure what would happen if they hit a live wire." At some point the cable tripped, so Cheyne Road is without power again.

On Wednesday they struck a Vumatel cable, so the restaurant now has no power and no fibre connection. I noticed that they seem to have damaged one of the cables:

It's difficult to tell if the damage was done this time or when the cables were dug up in April 2020. I have sent the following email update to City Power:

From: Donn Edwards
Sent: Wednesday, December 13, 2023 11:19 AM
To: 'Tshililo Nefale'
Cc: 'Sipho Gamede' <sgamede@citypower.co.za>; 'Charles Tlouane' <ctlouane@citypower.co.za>; 'Tshifularo Mashava' <tsmashava@citypower.co.za>; 'Nikki van Dyk' ; 'Beverley Jacobs' ; 'Nthabiseng Moloi' <NthabisengMol@joburg.org.za>; 'Jacob Gaongallwe Mashilwane' <jmashilwane@citypower.co.za>; 'Isaac Mangena' <imangena@citypower.co.za>;
Subject: 7 days without power
Importance: High

Hi all
Ignoring these emails will not make them go away. Now we have people digging outside the substation (and disconnecting the fibre link to 45 Cheyne Road) but no one is willing to explain what is going on or providing an ETR.

Then there is this mess:
CP3099926
CP3191075 45 Cheyne Road
CPWEB4094835 46 Cheyne Cancelled/Closed/Completed
CPWEB4094843 44 Cheyne Canceled
CPWEB4095809 49 Cheyne
CPWEB4095821 46 Cheyne Cancelled/Closed/Completed
CPWEB4095833 44 Cheyne Canceled
CPWEB4095843
CPWEB4095858
CPWEB4097063 46 Cheyne Cancelled/Closed/Completed
CPWEB4097214 47 Cheyne
CPWEB4097231 44 Cheyne Job "Completed"
CPWEB4100022 46 Cheyne Cancelled/Closed/Completed
CPWEB4101046 40 Cheyne Road "Completed"
CPWEB4101064 44 Cheyne Job "Completed"
CPWEB4102459 46 Cheyne Cancelled/Closed/Completed
CPWEB4103655 44 Cheyne Closed
CPWEB4105636 44 Cheyne Allocated
CPWEB4105763 46 Cheyne

Some houses in Doring Close and Cheyne Road have been restored for a while, but a switch has tripped somewhere, and 40 and 45 Cheyne Road have been without power for 172 hours and counting.

Please can we have some feedback?
Thanks in advance
Donn Edwards
(City Power customer and ratepayer)

Update 14 December 2023: It turns out that two people did not ignore my emails. Tshilio Nefale (General Manager) and Beverley Jacobs (Ward 98 Councillor) showed up at the substation and started asking difficult questions. Suddenly things started getting fixed. What's more, Tshililo actually listened to her customers, particularly the ladies at 40 Cheyne Road, who have been having power cable issues for years. She got the testing team to show up for the third time, and diagnose the problems properly. By 7pm on Thursday everyone was reconnected, even if it was a "temporary" fix for some of the houses.

Clarion Fixer (Part 1): Using the App

One of my pet peeves with Clarion is that the applications it generates by default look like they were written for Windows 95, and not for current versions of Windows. In modern versions of Windows, Microsoft has "fixed" the old look by making all the buttons and controls flat, but Clarion insists on using the "Microsoft Sans Serif" font that was introduced in 1997.
It looks old, especially since I have already used and discarded Verdana and Tahoma, for much the same reason. I never liked Arial (1982): it shipped with Windows 3.1 and has been severely over-used. "MS Sans Serif" is a bitmap font from a 1992 version of Windows. I want my programs to look like they were written recently, not 30 years ago.
My first attempt at fixing the fonts in my application didn't work. I was trying to fix things in the wrong place. But I later discovered that I could use a file utility called Search and Replace to fix the Clarion styles and templates. So when I create the app, I don't have to waste time fixing up fonts all over the place. This got me thinking: can't I automate this? Whenever I install new or updated accessories for Clarion, I don't want to have to fix up the fonts yet again.
Introducing clFixer, which is freeware. It's not signed, and some anti-virus engines on VirusTotal think it's a trojan, but I think they are getting false positives. Naturally, it is written with Clarion 11.1, and uses the CapeSoft StringTheory 3 ($97) and WinEvent 5 ($169) accessories. If you have those, you can download and modify the code here. The .rar file doesn't contain any exe or dll files. The zip file contains those. If not, you can read the code in the module files. Extract the zip file into a suitable folder to try it out. The code is also available on GitHub.
The first thing you need to do is make a copy of your Clarion folder, so you can see what changes have been made, and how they work. I have copied mine to "d:\Clarion11.1pe".

Exploring clFixer

Run the clFixer app, and from the main menu choose "Data"  -> "Application Settings".

Click on the ellipse to the right of the "Root Folder" field:

Select the backup folder and click "OK". Note there is a list of "Exclude Files". Some changes, such as the "Courier New" change, will cause trouble in the PDF related files, so you can decide whether a particular action must avoid modifying these files. Click "OK".

From the "Data" menu, select "Replacement Actions". This is a list of actions to be performed on the files. All of these replacement actions are performed on files in the "Root Folder" and subfolders, matching the file name extensions

*.inc;*.clw;*.txa;*.dctx;*.def;*.equ;*.tpl;*.tft;*.tpw;*.red;

If you add an action to modify "readme.txt" it will not be modified because "*.txt" is not one of the file name extensions listed.

General Actions

Here is the first and one of the most generic actions. It applies to all files, except a small number on the "Exclude Files" list.

This is even more general because it applies to the excluded files as well as any other file that still mentions "MS Sans Serif".

Specific Actions

This action is specific to one file, and fixes up a form where the font names wrapped around over 2 lines.

This action is specific to the TFT extension only. It fixes a general replacement from "Arial" to "Segoe UI" intended for forms, and changes the font to "Calibri" for a setting to do with reports. If you use a wild card like this in the file name, be sure to omit the ";" after the wild card, and use only one file name. The software isn't expecting multiple file names in this field.

Line-Based actions

Sometimes a search and replace across an entire file is not what we had in mind, and can cause havoc. Here is an example. The term "#SHEET,HScroll" appears in multiple places, so I only want to change this specific one. Note that I have included the first part of the next line as well, to prevent the same replacement happening again if the file is scanned more than once.

Here is another example of a line-based replacement. You may wish to disable it altogether because your IDE font settings will specify the font anyway if you have them set as follows:

Running the Changes

From the "Data" menu, select "Main Dashboard" to view all the settings. Then click on "Process the Files" to read the file structure and make the changes. Most of the work is now being done by StringTheory and the process is pretty fast. When done, there is a file canned "clFixer.txt" in the root folder, which is automatically opened for you. This lists all the files affected by changes.

Checking the Changes

It's important to check the changes, especially if you have added in new ones. before clFixer saves a changed file it checks to see if a "before" file exists. This is a file with the ".bf" added to the end. It's the original file that was renamed, and will still have the original date on it.

Here is a file comparison using the ExamDiff Pro utility, but there are others you may prefer. The original file is at the bottom, and the new file is at the top. It's important to do this because you don't want to screw up your actual working environment and have to reinstall things.
Once you apply the changes to your working environment, remember that none of your existing apps' forms and reports will have changed. Only new forms, reports and apps, and template-generated items will be affected by these changes.
Why not just pay CapeSoft $97 for their AnyFont product? My main objection is that this product allows the user to make bad font choices, and select something horrible like "Comic Sans". Horrors! Also, fonts have different widths and spacing, and I don't want text in Verdana to be truncated because someone is trying to fit it into the space intended for Arial Narrow. Legibility is more important than style.
In the next part, we will take a look at some of the programming techniques (good and bad) used in this application.

Update 20 October 2023: Following feedback and further insignts from Geoff Robinson and ClarionHub, the download is updated to version 0.2

[ Clarion Fixer (Part 1): Using the App ]
[ Clarion Fixer (Part 2): The Code ]
[ Change Mode vs View Mode ]
[ Clarion Build Events ]
[ Introducing Clarion FileManager (Part 1) ]
[ Introducing Clarion FileManager (Part 2) ]
[ Clarion Resources and Products ]

Logo credits: Toolbox icons created by Roundicons Premium - Flaticon

Clarion Fixer (Part 2): The Code

Let's have a look at the data structure. This is a one-pass program, so you can only process one root folder at once. So there is no relationship between the tables.

We need an alias of the "Action" table to be able to reorder and renumber the Actions. There is a hidden "ReorderNo" field that is used for this purpose, with a reverse order index to assist too. More on that later.

There is only a browse form for the "Action" table, since the "Setting" table has only one record. The "MainDashboard" form doesn't link to any particular table. The values displayed on the form are mostly global variables, so they can be updated from any of the source procedures ("business rules") that have been written for this application.

Since I like to use the CapeSoft WinEvents Global Extension to display the program version number and other information, I added in the System Information form by modifying a demonstration form from WinEvents because I was curious to see how much RAM would be used by the program.

One of the templates that are modified is "versionres.tpl", shown here while it is in use. It's always one of the first Global Extensions I select. The modification was to make the text a bit bigger. As you can see from this screenshot, it is working. The global templates in use are listed here:

• ABC Defaults ($10) is a simple template which allows you to set the default value for some ABC class properties. Set it and forget it.
• dpiAware Fix (download) freeware written by Mike Duglas to correct an error in the generated manifest file.
• StringTheory 3 ($97) is a string object on steroids. Over 200 methods for parsing, manipulating, creating, compressing and encoding strings. Makes you extremely productive.
• WinEvent 5 ($169) is a wrapper around the Windows API. This library provides just about everything: Taskbar, AutoApp Shutdown, GSM, Force windows on desktop, Comms 232 and a host of other functions.
• AJE Backup ($99) is an Automatic Backup Template for Clarion. Configure it for your application, and it will make a numbered backup of key files every time you do a successful build. You don't need it to run the application, but you need it while developing the application.
• UltimateDebug is the ClarionLive debug template, part of the Ultimate Utilities pack. More on this later in the article.
• cwVersion is Clarion's built-in version template.

In WinEvents, this is how you get the EXE version and put it into a global variable.

Global Variables and Properties

A number of global variables are defined in the dictionary. I prefer to do them there than in the application itself, just to keep things organised. In the Global Embeds section, take a look at the "Global Data" section. I have defined two queues:

!// This is a (risky) global queue, partially based on the queue
!   structure needed by DIRECTORY function
clFiles             QUEUE,PRE(CLF),THREAD
name                    STRING(FILE:MAXFILENAME)  
path                    STRING(FILE:MAXFILEPATH)  
uname                   STRING(FILE:MAXFILENAME) ! UPPER(name)
attrib                  BYTE   
scanned                 BYTE   
excluded                BYTE   
                    END

!// This is a (risky) list of excluded file names
clExclude           QUEUE,PRE(CLE),THREAD
name                    STRING(FILE:MAXFILENAME)  
uname                   STRING(FILE:MAXFILENAME) ! UPPER(name)
wild                    BYTE  ! Contains wild cards
                    END

Why risky? In a normal application with multiple things going on, it is possible for the queue buffer to be changed by different threads at the same time. This has been covered by Bruce Johnson in ClarionLive #600. In this application, only one thing can happen at a time, so it made sense to make the queue definitions global, so they could be manipulated from various procedures as needed.
If you go to the Project Options page, here are the settings:

Don't leave the icon field blank. Put the icon for your application there. With websites like flaticon.com there are plenty of icons to choose from.

Some of these options are explained in the recent Clarion Build Events post, and the older Backing up your Clarion projects article.

When you go to the "Global Properties" tab and click on the "Actions" button, don't forget to put your name (or the company name) as the author, and choose the application icon instead of leaving it blank. Leaving the icon blank will default to the "waframe.ico" icon on all forms where no other icon is explicitly set.

Personally I think your application should be professional enough to warrant its own icon, especially when it is so easy to do.

Go to the second tab ("App Settings") and click on the "Application Manifest" button. The settings shown will help with making the user interface more helpful, and disable the unwanted UAC prompt every time you run the app.

The "Set Visual Indicators" window has some useful settings too. You can see these in action when you are running the application, providing cues for adding or modifying fields in change mode. See the "Change Mode vs View Mode" article.

Remember: good layout is invisible, it just draws you in to the important parts of the screen. Bad layout is immediately obvious, and confusing.

The Main Frame

The "Main" frame is the first thing the user sees, so try to make it look professional and pleasing to the eye. Make sure it has a meaningful title, usually the full name of the application. Put the customer's logo in the background, or the application logo, depending on the circumstances. Don't make it too "busy" or too "cute". I am showing the version number and the name of the workstation running it. I changed the "Browse" menu to "Data" and made it bold, indicating its importance.

Select the "Main" frame and click on the "Embeds" button on the right-hand side.

Click on the "Expand Filled Nodes" button and find the "OpenWindow" embed point. Click on the "Filled Source" button to view the code in the Embeditor.

      ! Get the EXE version number and global values      
      ! (c) 2022-2023 Black and White Inc
      !dbg('COMMAND: ' & COMMAND(''))
      glo:dbugoff = FALSE  ! Do not suppress debug messages
      glo:AppVersion = 'Version ' & CLIP(ds_GetFileVersionInfo()) 
      glo:MachineName = clip(ds_GetWorkstationName())
      0{PROP:StatusText,3} = clip(glo:AppVersion)         
      ! Display the version number in the main status bar
      0{PROP:StatusText,2} = clip(glo:MachineName)        
      ! Display the workstation name in the main status bar
      !// Get the setting data
      Access:Setting.Open()
      Access:Setting.UseFile()
      If Records(Setting) < 1
          Access:Setting.PrimeRecord()
          set:GUID = glo:st.MakeGuid()
          Access:Setting.Insert()
      else
          set(set:SettingPK)
          Access:Setting.Next()
      End
      glo:AppDescription = set:stDescription
      glo:RootPath = set:RootPath
      glo:FileExtensions = set:FileExtensions
      glo:ExcludeFiles = set:ExcludeFiles
      !dbg(set:GUID)

Notice anything wrong with this code? No error checking! I broke my own rules!

This code uses WinEvents to set the global values of glo:AppVersion and glo:MachineName, and then displays the results on the main status bar. Next, it inspects the contents of the "Setting" table and creates a record if it doesn't have any. Then it loads four of the fields into corresponding global variables so they can be displayed on the Main Dashboard later.

Click on the "Previous Filled Embed" button to go to the top of the Embeditor code. I always include Dan Read's Principled Programming summary as a reminder to write decent code and professional quality applications. Take a moment to read it, and then click on the green "Save and Close" button.

UltimateDebug and dbg Messages

Find the "dbg" source procedure and open it using the Embeditor. This is a small but very useful procedure during development, because it allows you to send messages to DebugView++ to see what is going on.

!// The purpose of this procedure is to display debug messages
!   This means you can enable/disable the UltimateDebug messages
!   without having to comment out a bunch of ud.debug lines
!   in your code
if glo:dbugoff = false then ! Display messages
    std.setvalue('--- ' & clip(pstrDebugMessage) )
    std.trace() 
end ! Display message

That's all folks! It uses the StringTheory Trace method to send information to the debugger. Why not use "UltimateDebug" instead? Actually, I use both when I'm starting a project, so I can see how the procedures and forms are interacting with one another. See "Adding Extensions to Clarion". At some point in the process, these messages start getting in the way and I just want to see my own messages. I used to write my own messages using "UD.Debug" but if you disable or remove the "UltimateDebug" global extension you then have the tedious task of finding and commenting out all of these statements. So instead of using "UD.Debug" I use "dbg" instead. It displays the messages whether "UltimateDebug" is in use or not.

When "UltimateDebug" is in use, notice the extra

if false then

statement, and its corresponding "end" statement. This stops "UltimateDebug" from adding in a redundant message every time you use "dbg".

To remove the "UltimateDebug" stuff, first open the template and click on "Disable UltimateDebug template". Click "OK" and do a build. You will probably get a compile error or two. Now "Delete" the extension completely and do another build. This time there should be no further errors. If you want to turn off all the messages, just go the the "Main" OpenWindows code and change the value from

glo:dbugoff = FALSE

to

glo:dbugoff = TRUE

UpdateSetting

I don't like the way Clarion makes the primary key fields editable by default. It's just asking for trouble. Ideally, primary key values should be generated before the form opens in insert mode, or at least only be editable during insert mode. After that they should be locked down.
Here is an example. The "GUID" value was generated automatically, so the field ENTRY control is disabled, flattened, made read-only, and is no longer the 1st field in the tabindex order. Give it a high number like 100 and the Window Designer will assign a more realistic number. The gap between "GUID" and "Description" is used to accentuate this.
Also, the "Description" control and prompt have the TabIndex values of 2 and 1 respectively, so the cursor will land in the "Description" field when you open the form. The "Description" field is the same width as the "Root Path" field, even through the former can't accept as many characters as the latter. Be careful not to make the controls much bigger than the amount of data it can support, because the user's input will be truncated without warning.

In the Window Designer you can drag fields onto the form, but when you drag a memo field (in this case set:ExcludeFiles) it comes across as an ENTRY control, not a TEXT control. So you can resize it as required, and then close the Window Designer and go to the Window Properties screen. Here I have changed the highlighted (set:ExcludeFiles) control from "ENTRY(@s255),AT" to "TEXT,AT" to get a more useful input control.

Please take a look at my post "Change Mode vs View Mode" for an explanation of the additional code in the Window.Init embed points of WindowManager, both on this form and on the next one.

UpdateAction

This form has a number of interesting features. There are "Paste" buttons to take the text from the Windows Clipboard and paste it into the "stBefore" and "stAfter" fields. If you look at the "Accepted" embed points for these buttons you will see the code that does it.

The "act:FileName" and "act:DisableYN" fields needed some special attention on this form. When the user sets the "act:DisableYN" check box to TRUE, I wanted to grey out the "act:stDescription" field and change the highlighting of the "act:StepId" field. When the "act:FileName" field is empty, I want to disable and empty the "act:LineNo" field because you can't have a global line number, only one for an individual file. The code to do this is kept in two routines, and they are called from the relevant control's "All Events" embed point, and the "OpenWindow" embed point for when you open an existing record.
The other item of interest is the "Next Step No" button. For this table, I could have had the primary key as a GUID, and "StepId" as an indexed field, but it seemed over-complicated for a simple table with not many records.

So I broke my own rule about not changing the primary key.

I want the step numbers to start at 100 and go up in steps of 10, with the flexibility of being able to add in steps in between other ones. So this is how I calculate the next step, using the alias to the "Action" table, since we are busy inserting/editing an "Action" table record:

CalcNextStep        ROUTINE
    ! Find the largest step number, add 10
    loc:StepId = act:StepId ! Save the current step number
    lngStepIdmax = 0      ! Start
    !// Open an alias of the table so as not to screw things up
    Access:ActionAlias.Open()
    Access:ActionAlias.ClearKey(act1:ActionStepPK)
    SET(act1:ActionStepPK)
    LOOP UNTIL Access:ActionAlias.Next() <> Level:Benign ! Next
        if lngStepIdmax < act1:StepId then ! lngStepIdmax
            lngStepIdmax = act1:StepId   ! The new max
        end ! lngStepIdmax
    END ! Next
    Access:ActionAlias.Close()
    if loc:stepId <> lngStepIdmax then
        ! Add to the most recent step to get the new one
        act:stepId = lngStepIdmax + 10  
        ! unless we already have the largest number
    end

Notice the lack of error checking? What was I thinking?

BrowseAction

Calculating a new primary key value is one thing. Renumbering an entire table is something else entirely. Firstly, it's dangerous and difficult. Secondly, primary key values aren't supposed to be changed without a good reason and careful consideration, if at all. But I did it anyway. Why?
I want to be able to insert steps because the order in which the replacements are done is important. so let's assume we have a StepId sequence like

100, 110, 112, 113, 120, 130, 140, 150

and I would like to add in another step between 112 and 113. I could change 113 to 114 and then insert a new 113, but if I'm going to change the primary key value of one record, I might as well go the whole 9 yards and renumber all of them.
If I start renumbering from the lowest number, it won't be long before I am trying to change 120 to 140, but there is already a step number 140, which will result in an error. What I need is an intermediate field. In this case it is "act1:ReorderNo".

RenumberRecords ROUTINE
    !// First pass: update all the act:ReorderNo fields
    Access:ActionAlias.Open()
    Access:ActionAlias.ClearKey(act1:ActionStepPK)
    SET(act1:ActionStepPK)
    loc:StepId = 90
    LOOP UNTIL Access:ActionAlias.Next() <> Level:Benign ! Next
        loc:StepId += 10
        act1:ReorderNo = 1000000 + loc:StepId
        Access:ActionAlias.Update()            
    END ! Next
    Access:ActionAlias.Close()
    !// Second pass: update all the act1:StepId fields with
    !   big numbers
    Access:ActionAlias.Open()
    Access:ActionAlias.ClearKey(act1:ActionStepPK)
    SET(act1:ActionReorder)
    LOOP UNTIL Access:ActionAlias.Next() <> Level:Benign ! Next
        act1:StepId = act1:ReorderNo
        Access:ActionAlias.Update()          
    END ! Next
    Access:ActionAlias.Close()       
    !// Third pass: update all the act1:StepId fields with
    !   correct step numbers
    Access:ActionAlias.Open()
    Access:ActionAlias.ClearKey(act1:ActionStepPK)
    SET(act1:ActionReorder)
    LOOP UNTIL Access:ActionAlias.Next() <> Level:Benign ! Next
        act1:StepId = act1:ReorderNo - 1000000
        Access:ActionAlias.Update()            
    END ! Next
    Access:ActionAlias.Close()

Again, no error checking! This is just wrong.

In the first pass through the table, using the primary key sort order, I can calculate a new sort value and place it in the "act1:ReorderNo" field. To prevent the renumbering clashes, I am adding one million to each new step number. In the second pass I am using the reverse order of "act1:ReorderNo", so the biggest StepId is replaced first, with a much higher number. In the third pass we return to the primary key sort order, and deduct the million from each StepId value, so we get step numbers starting at 100.

If you inspect the List Box Formatter, you will see the use of a picture

@n~Y~1B

which is a hack to display a "Y" if the value is true, and a blank of the value is false. This is discussed on ClarionHub in great detail, including the "proper" way to do it.

Processing the Files

This is where the file processig gets done. First, we create a queue of file and folder names in the global "clFiles" queue. To achieve this, we scan the root folder and add in all folder names found using "DirectoriesOnly" procedure. Next, we parse the extensions list and get each extension wild card, using "DirFilesMask". Then we run "DirFilesOnly" for each mask. In this way, the queue grows with file and folder names. This is all co-ordinated by "DirAllFilesAndFolders" until all folder names have been scanned. Once all the folders have been scanned, these folder names are removed from the queue, leaving only the file names.
Another procedure creates a smaller queue, "clExclude", with just a list of the excluded files. As each file name gets added to the "clFiles" queue, a boolean field is set if the file is one of the excluded files. The final stage in the process involves looking at each file in the queue and applying all the Actions in the "Action" table, except those which are disabled. This is done by the "ProcessFile" procedure.
If the file is marked as an excluded file, then the excluded actions are also ignored. If the file mask for the action is empty, or the file mask matches the file name in "clFiles", then the Action is performed. Once all the actions have been tried, the new file contents are compared with the original file contents (both stored in StringTheory objects) and if they differ, the old file is renamed with a .bf extension, and the new file is saved under the original name. The file path is also added to a list of changed files, which is eventually saved in the root folder as "clFiles.txt".

The Main Dashboard

Here is the "Main Dashboard". Clicking on the "Process the Files" button will cause the "DirAllFilesAndFolders" procedure to process the files according to the values shown on the form. The line of text near the bottom of the form is based on the "glo:ProgressMessage" global variable, so it can be updated by any of the procedures mentioned above.
I'm pretty sure I have broken all kinds of "good programming" rules by relying so much on global variables, especially the global queues. I also don't have enough knowledge of CLASS definitions to modify the supplied classes, particularly the PRIVATE class elements, to offer any change actions in this version of clFixer. Please leave comments below or contact me directly to help me improve this code and make it more professional.

Feedback

Monday 16 October: Well, this is embarassing. I didn't put any error checking code in the app! Didn't even follow my own advice. So now I'm working on version 0.2 that includes error checking. So much for professional code! Thanks to Geoff Robinson for pointing this out privately, and for several other improvements, espcially in the way I have been using StringTheory.
Friday 20 October 2023: Following feedback and further insignts from Geoff Robinson and ClarionHub, the download is updated to version 0.2

[ Clarion Fixer (Part 1): Using the App ]
[ Clarion Fixer (Part 2): The Code ]
[ Change Mode vs View Mode ]
[ Clarion Build Events ]
[ Introducing Clarion FileManager (Part 1) ]
[ Introducing Clarion FileManager (Part 2) ]
[ Clarion Resources and Products ]

Logo credits: Toolbox icons created by Roundicons Premium - Flaticon