My usual approach to writing boring or repetitive code is to use generative programming, but there are other ways of getting source code generated, and over the last couple of weeks I’ve looked at two of these.
A couple of weeks ago I attended a workshop on Program Synthesis, a technique for generating provably correct software from a specification. I had never come across this before, but checking Wikipedia I find that it’s actually a long established area of computing science research, and indeed, Microsoft Research describe it as ” the holy grail of Computer Science”.
The basic approach seems to be to provide a programme generator with a set of rules that can be used, basically a subset of the eBNF grammar for the target language, and a set of input and output examples. For the toy examples shown this worked well, but it did remind me of a situation I came across several years ago to do with implementations of QTI 1.2. The Specification of qti was provided as a long document, supported by a set of Example questions, i.e inputs, with descriptions of how they should behave, i.e outputs. In practise what happened was that most implementations were able to correctly display and process the example questions, and questions extremely similar to them, however anything that varied even slightly in structure from the examples would usually fail. The reality is that as a problem becomes more complicated, the number of examples needed to correctly define it Increases exponentially.
The alternative to using input and output examples is to define the problem mathematically, but once this is being done the problem definition becomes at least as complex as the solution, so the main advantage becomes familiarity with the language, and possibly a reduction in silly mistakes.
One question asked by an attendee was how AI affects all this, and it turns out that future research will be including AI. It certainly seams reasonable that AI could be used to speed up the iterative process used to find solutions, and the existing checks could be used to ensure that the final AI output is correct.
By chance I also looked at AI tools last week. I usually write in PHP, a rather unfashionable language that is also notorious for the large amount of bad code written in it. That makes it a poor candidate for AI, and I’ve never bothered to opt into the advanced AI tools available in Visual studio. (The PHP tools I use provide some AI code suggestions, but the are almost invariably wrong, and often very bad code.) I was working on a simple C# program, and came to a place where I needed to include threading syntax that I couldn’t remember. With older versions of the Visual Studio IDE the lazy way to do this was run the program, get the crash and look at the error hint: it’s such a common error that the pre-AI system was programmed to show the solution. That hint didn’t appear last week, so my wife flippantly suggested I try AI (following on from a conversation we’d had earlier.) I enabled GitHub co-pilot AI, and immediately got the solution. The program I was working on is a sort of ‘Electron for PHP’, and the part I was working on is a very simple macro language that allows the hosted app to interface with things like file dialogs and the clipboard, which would normally be invisible to PHP and JavaScript. The macro language consists only of string variables and constants, and function calls, and is implemented in a single switch statement with the function names as cases. The next part I added was the ‘value’ function, that replaces a simple assignment in this language; I typed case "value": and got the (correct) suggestion:
if (a.parameters.Count > 0)
{
string value = getStringVal(a.parameters[0], ajaxVars);
setValue(a.returnTo, value, ajaxVars);
}
else
{
MessageBox.Show("No parameters for value action.");
}
break;
That was quite impressive, though also fairly obvious given other parts of the file. The next case was: case "getClipboardImage": , and this time it was less impressive. The generated code did almost the right thing, but using the wrong framework. My app uses Eto.Forms, as I want it to run on Windows, MacOS and Linux. Co-pilot generated Windows Forms code, which has some similarities, but is certainly not compatible.
I’ve now tried giving co-pilot a better prompt: “Can you generate me Eto.Forms code to read an image from the clipboard and save it as a PNG image in a temporary file, returning the file name.” This time the output is almost correct, though it uses an ambiguous Path namespace, which will cause an error. It also uses unnecessary var declarations, a common lazy practice in C# that I detest.
I’m going to experiment more with co-pilot: currently I think it’s looking useful for small bits of code, but the generated code needs careful checking.
Having looked at these two alternatives for generating code, I don’t think either is going to replace generative programming: each has it’s niche, but also there could be major synergies where combining approaches can work. The biggest issue I see with code synthesis is that the definition of the problem is often more complex than the solution. A language based approach, closer to generative programming could possibly solve that: basically generative programming with more ‘intelligent’ code generation. AI could be used to speed the creation of output templates for generative programming – one of the strengths of generative programming is that well tested code is reused, and one of the weaknesses of AI is that the code needs careful testing. So use AI once, test the code, and generate variations. There is also probably an interesting research project to be done looking at Synthetic Programming to help create code generation templates from examples. That is a much more constrained problem, that it might work well for.
I don’t think synthetic programming is at a stage where it is useful for practical work, so I probably will not be look at that further for now. AI is at a stage where it can be time saving, at least in C#, so I’ll continue to use it, and probably will at least use it for generator exemplars.