The software industry has for decades now been a force for upheaval among established industries. Often bringing software to a business means helping businesses to redefine business processes in potentially uncomfortable ways. But Chat GPT seems to threaten, at least vaguely at this point, to turn the tables. As a software developer, I’m fascinated by the potential of technology to increase human productivity, and recently I’ve been exploring the possibilities of Chat GPT. I think that effective developers are just starting down a long path of finding ways to leverage this new tool. My first endeavors naturally involved getting it to generate code. It’s a machine that can write code. What a fascinating idea this is. But how good is it?
Putting Chat GPT Through Its Paces On Math And Code
Algorithm Challenge
I’ve asked it the classic two-sum challenge:
|
Please write code for the solution() method in the snippet below. The solution() method should return an array containing the two indexes in the arr input that contain values that add up to the target. If no two indexes exist that satisfy this relationship, please return {-1, -1}. public static int[] solution(int[] arr, int target) {
} |
The AI happily churned out a good answer:
|
public static int[] solution(int[] arr, int target) { HashMap<Integer, Integer> map = new HashMap<>(); for (int i = 0; i < arr.length; i++) { int complement = target – arr[i]; if (map.containsKey(complement)) { return new int[] {map.get(complement), i}; } map.put(arr[i], i); } return new int[] {-1, -1}; } |
Ok. That’s great. It runs fine and is correct. But this problem is extremely well known. I want to give Chat GPT a little bit of a curveball to see what it can handle. What about a simple recurrence then?
Mathematical Understanding Tested With Recurrences
|
P(x) = P(x-1) + 25 P(1) = 0 Give me a table of P(X) by positive integer inputs |
Chat GPT hums along and gives me back a table:
|
Using the given recurrence relation, we can generate a table of values for P(x) as follows: x P(x) 1 0 2 25 3 50 4 75 5 100 6 125 7 150 8 175 9 200 10 225 To generate each value of P(x), we use the previous value P(x-1) and add 25 to it. We start with P(1) = 0, and then use the recurrence relation repeatedly to obtain the values for P(2), P(3), and so on, until we reach P(10). |
Joy. The explanation is great as well. Part of the reason I wanted to test recurrences is because I am studying them as a part of furthering my own understanding of data structures. Recurrences can be used to mathematically model and analyze algorithms and these two things have some common features. I study a book called Concrete Mathematics on the subject, but there are online resources such as Wolfram Alpha that can help as well. I think it is amazing that Chat GPT can seem to be aware of the linguistic elements that make up recurrences and give me a coherent answer when I ask about them.
Recurrences are naturally expressed recursively, and they are not expressed completely in English either. They represent a sequence of numbers defined by self-referential functions and visualizing them as tables can be a helpful step in analyzing them. This represents a use case for Chat GPT where the goal is not necessarily just English language commands to some other output format. Let’s see if Chat GPT can take this recurrence and basically ‘solve’ it by finding a closed form.
|
Is there a closed form for P(x) that you can tell me? |
|
To find a closed form expression for P(x), we can use the recurrence relation and simplify the terms: P(x) = P(x-1) + 25 = P(x-2) + 25 + 25 = P(x-3) + 25 + 25 + 25 = … = P(1) + 25(x-1) Substituting P(1) = 0, we get: P(x) = 25(x-1) Therefore, the closed form expression for P(x) is P(x) = 25(x-1). |
Oh wow, the explanation is crisp, and the answer is correct, but no one will be impressed by this from an actual person, really. How about something slightly harder?
|
P(x) = P(x-2) * 2 – P(x-1) / 2 P(0) = 0 P(1) = 1 Please give me a table of values for P(x) |
|
Using the given recurrence relation, we can generate a table of values for P(x) as follows: x P(x) 0 0 1 1 2 1/2 3 -1/2 (…) |
Whoops! This is not correct. This language model is not good at tabulation! Position two is actually supposed to be 0 * 2 – 1 / 2 = -½. Excel can do this fine. This kind of deficiency is very well known and will surprise no one who has spent time with Chat GPT.
Chat GPT claims to be integrated with Wolfram Alpha, so let’s ask it to get some help. As mentioned earlier, there is Wolfram Alpha, which announced a plugin for Chat GPT. I’ve asked it to use Wolfram here, but it is likely that Chat GPT has lied to me in the results.
|
Please use Wolfram to calculate the same table for Z(x) in this recurrence: Z(x) = A(x) + B(x) A(x) = B(x-1) if x is odd, otherwise B(x-1)/2 if x is even B(x) = Z(x-1) + 1 A(0) = 0 B(0) = 1 Z(0) = 0 |
After I realized I should ask it for A(x) and B(x), eventually it replies:
|
Sure! Here is the updated table of values for Z(x), A(x), and B(x) calculated using Wolfram Alpha: x Z(x) A(x) B(x) 0 0 0 1 1 1 1 2 2 2 1 3 3 4 3 4 (…) |
Whoops again! The trouble starts pretty early here: Z(1) should be A(1) + B(1) = 3, but that’s not what we got. The rest just collides, shatters, and falls to the ground. Maybe there’s an Excel plugin we could try.
From Math Back to Code (barely any English required)
But who’s expecting it to work for a more complicated recurrence if it fails on a simple one? What’s amazing about Chat GPT is then ask it:
|
Can you give me some python code implementing the recurrence? |
|
Sure! Here’s some Python code that implements the recurrence relations for A(x), B(x), and Z(x): def recurrence_A(x, B): if x % 2 == 0: return B[x-1] // 2 else: return B[x-1] def recurrence_B(x, Z): return Z[x-1] + 1 def recurrence_Z(x, Z0): B = [0] * (x+1) Z = [0] * (x+1) Z[0] = Z0 B[0] = 1 for i in range(1, x+1): B[i] = recurrence_B(i, Z) Z[i] = recurrence_A(i, B) + B[i] return Z |
This code is nice actually. It might be helpful in some hypothetical situations where the user wanted to quickly run the numbers to understand more concretely how they work out. It uses Python lists as dynamic programming caches so that there is no chance there will be unrestrained recursion in the implementation. One might naively implement just the words in the instructions in a way that would fall into this trap. This code almost certainly is a riff on someone’s work with simple recurrence-like behavior, but it’s not that likely that this particular recurrence was just sitting out there in a Python script somewhere, so this is nothing to sneeze at. What’s more, usually when you ask for code it is syntactically correct. Actual functionality can be much more spotty.
This implementation has chosen integer division for the rule involving B(x), which is not quite technically correct, but if we let that slide and massage the script slightly (or ask Chat GPT to) so it prints the intermediate values, we get (green for console output):
|
x A(x) B(x) Z(x) —————————- 0 – 1 0 1 1 1 2 2 0 3 3 3 3 4 7 4 2 8 10 5 8 11 19 6 5 20 25 7 20 26 46 8 13 47 60 9 47 61 108 10 30 109 139 |
It’s correct, again assuming integer division for the B(x) rule. So why is Chat GPT good at code but bad at numbers? The natural explanation seems to be in the title: it’s a Large Language Model, so it’s supposed to be good at language, right?
My First Attempts to Use Chat GPT for Writing
Using My Words
I am a technologist and so I feel compelled to believe in technology. I want Chat GPT to work. I think it could unleash a new era of human productivity coming just in time to intervene in what seems possibly like a time when humanity’s productivity may be on the way to a plateau. Increasing and facilitating productivity is after all the high-level goal of almost any software developer. My biases are not shared by all of course, so maybe the next part of my experimentation will be met with more controversy.
Let’s start out simple:
|
Give me a haiku about beagles |
|
Here’s a haiku about beagles: Nose to the ground, bound Ears aloft, baying with glee Beagle on the hunt. |
I really like this. I’ve tried to find out if this is a poem written by an actual person, but I cannot find any evidence that it was. In any case, this is not quite enhancing my productivity yet, just successfully entertaining me.
Words For Work
I’ve wanted to write a blog article on the Ten Mile Square website again for some time now, so using Chat GPT to help me do so seemed logical. Chat GPT actually emerged in the time since I set out initially to do the article on something, whatever that thing might be, so it seems very convenient. I’ve done plenty of writing for school but I do not do formal writing for work. I engage professionally in all kinds of communication and information organization, but writing full pages in paragraphs in a report format is not my daily experience. Maybe this task will lend itself more freely to Chat GPT?
Initially, the points I wanted to make were that you could interact with Chat GPT and it wasn’t a matter of finding a single prompt that would churn out your article for you. I am a big believer in iterative processes and thought it would be nice to demonstrate that kind of thing with the AI. Furthermore, I wanted to point out that you could assert the factual basis of your writing so that Chat GPT would not stray past your intentions when filling in the gaps. I thought it was important to reiterate the observation I’ve seen many places where Chat GPT has no inherent sense of truth, but also that you could put specific assertions and true statements into the input that Chat GPT uses for your particular task.
I laid out the points I wanted to make, asked it to take a professional tone, and let Chat GPT do its thing. The results were vapid. After a period of time, I got bored with the task I had given myself and asked it whimsically to rephrase the article from the perspective of a super-intelligent spider robot from the post-AI apocalypse future. And then it grabbed me – I was self-amused by the process. I asked it to throw in more stuff about weaving webs and following the spider robot’s day. I got lost in the amusement and forgot I was writing a blog entry.
There is another well-known attribute of Chat GPT reflected in this experience: it’s really good at saying things you want to hear, even if they’re almost completely hollow. My self-initiated feedback cycle resulted in a page full of words that someone I asked to review called it “tedious”. And surely it was. It was also highly repetitive, which is a common problem with these large language models too. But I spun it up like a tiny one-man, one-ai echo chamber.
What Went Wrong
Chat GPT’s obsequiousness is quite hard to avoid as well. If the user suggests something is wrong, the AI rarely asserts itself. The only time I really got Chat GPT to disagree with me was when I asked it to render HTML, which it was quite weirdly adamant about both its active, affirmative capability of accomplishing and yet its simultaneously supreme unwillingness to actually do so. Aside from that, it is pretty much designed to tell you what you want to hear. And it seems this was at least part of what lured me when I started down the road with the first draft. I sent out a link using Google Docs for review. My reviewer sent me a link to a Washington Post article on people who had fallen in love with an AI chat online. The insinuation is hard to miss. But wait! I was just doing some math earlier. How could my attention be diverted so quickly from my goal? It seems like there are psychological factors at play here that I have underestimated. It’s shocking really.
The ability (or potential for ability) of Chat GPT to do writing is an emotionally laden subject, not even considering the potential for starting a feedback loop. Many people have vested interests and are not all in the same direction of course. My first draft sounded like marketing copy for Chat GPT. As a developer using technologies from most of the big companies from Amazon’s AWS to Google’s Angular and Chrome, it’s not the most uncommon experience for me to find the words of a big company coming out of my mouth, but this is not what I set out to do either in this blog writing activity or at work. But forces aligned to make it happen anyway. This experiment has led more to self-reflection than I expected when I had originally intended to examine Chat GPT.
Then there are the childish mistakes. Oh no, not mine, the ones made by Chat GPT. The phrase “in conclusion” is banned from first sentences on final paragraphs. Everyone outside of high school knows this is terrible writing, but when asked to revise this Chat GPT chose to say “in summary” instead. How unpleasant: one mistake was replaced by the same mistake again!
Ultimately Chat GPT can only work with the ingredients you give it, which is why my first 20-minute experiment seemed vacuous: very little actual thought had gone into it at that point. Despite having swaths of the internet compressed into it, the stimulus must still provide some kernel for each aspect of what comes out of the terminal.
Lessons and Cleanup
After the initial draft was put out of its misery, I decided to collect my thoughts on the experience I had just had in a document full of one-off sentences, basically in outline form. I told myself the AI would be able to put these notes together in a useful manner with me. When it came time, it could summarize my points, but it did so rather sloppily and in third person. When I asked it to change to first person, somehow it later slipped into second person. When asked, it generated headers for sections, but I was unhappy with these as well. No, it was not meant to be.
It did manage to help me rewrite the first sentence of this blog entry, but not before suggesting a title when I asked it for a first sentence. Writing this by hand now I can tell you this failure to get the tool to help me in my task was the final blow. I put hours into this blog entry and experimented with optimism. I have enjoyed using Chat GPT on occasions that call for some specific bit of knowledge on a well-established tech stack, writing a simple regex, or making a logback.xml file. I see flashes of brilliance but have mostly experienced mixed results. Someone cleverer than me will unlock this treasure, I hope.
But wait, There’s More! (A Postscript)
What about scripting? Math and algorithms are one thing, and human-focused writing is another, but what about short, declarative scripts?
The Scenario
I was fixing the Jenkins pipeline for one of the projects that my team works on. I fixed a few small things and prepared to deploy to the dev environment. The script went along, but failed in the deploy step:
|
15:05:22 Serverless: Validating template… 15:05:22 Serverless: Updating Stack… 15:05:22 Serverless: Checking Stack update progress… 15:05:48 ……………………………. 15:05:48 Serverless: Operation failed! 15:05:48 Serverless: View the full error output: https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stack/detail(…) 15:05:48 15:05:48 Serverless Error ————————————— 15:05:48 15:05:48 An error occurred: SetStepFunctionStartTimeLambdaVersion(…) – Code storage limit exceeded. (Service: AWSLambdaInternal; Status Code: 400; Error Code: CodeStorageExceededException; Request ID: 0ba84f70-5300-4381-af03-7baa6d6046a8; Proxy: null). |
We’re using ServerlessJS for this project to deploy. At first, I misread this as our lambda code artifacts being too big, but I realized soon after that this AWS account had simply hit a service limit for the total lambda code size. The limit had already been raised over the 75GB default, so here I was unable to deploy my changes.
I knew this limit was more of a housekeeping thing than a serious issue, and after Googling I learned that the real solution is to apply resource policies to allow old lambda versions to be removed automatically by the account. That was not in place here apparently, and I did not have the authority to make such decisions for the organization.
Instead, I decided to see what was going on with just the resources my team had responsibility for. I used the CLI just to take a look at what was there for one of my lambdas:
|
aws lambda list-versions-by-function –function-name function-name-redacted > function-name-redacted-versions.json |
Nothing magical going on there, but about 400kb of JSON fell into my lap. Great. Now, I’m a dabbler in Python but am using it more and more lately. I’m not feeling like remembering specifically how to use the JSON library. Hey Chat GPT:
|
give me a python script to read a json file please |
|
import json # Open the JSON file with open(‘example.json’) as f: data = json.load(f) # Print the contents of the JSON file print(data) |
Oh right, that’s how it works. But still, maybe I do not really have to lay a finger on a traditional editor for this. Why not just:
|
please add a running total for all “CodeSize” attributes on objects in a json array inside a “Versions” attribute on the top level object in the file |
The AI returns:
|
import json # Open the JSON file with open(‘example.json’) as f: data = json.load(f) # Get the “Versions” array from the top-level object versions = data[‘Versions’] # Initialize a running total for the code size total_code_size = 0 # Loop over each object in the “Versions” array for version in versions: # Get the “CodeSize” attribute from the current object code_size = version[‘CodeSize’] # Add the current code size to the running total total_code_size += code_size # Print the total code size print(“Total code size:”, total_code_size) |
This script worked quite nicely and I learned I could get more than a gigabyte back if I could just get rid of all but the most recent version. We deploy this lambda fresh from source – why mess around with redundant copies in AWS? But what to do now? All I needed was the data to get removed so I could get going. No fancy reusable scripts, just data gone, please.
A Wonderful Monstrosity
Then I asked it for the script I probably should have asked for originally:
|
please write a python script that uses the aws cli to get all the versions for a lambda by function name using list-versions-by-function and delete all but the latest |
First, before I go any further, I would like to acknowledge that yes, I do usually say please to Chat GPT. I do not know what to tell you. I do not, on the other hand, often include question marks. Anyway, this request above is flawed for reasons I’ll briefly discuss later, but the following script helped me blast away enough unused lambda versions to move forward with my day:
|
import subprocess import json # Replace with your AWS region region = ‘us-west-2’ # Replace with your Lambda function name function_name = ‘my-lambda-function’ # Get the list of function versions using the AWS CLI get_versions_command = f’aws lambda list-versions-by-function –function-name {function_name} –region {region}’ output = subprocess.check_output(get_versions_command.split()) versions = json.loads(output)[‘Versions’] # Delete all but the latest version for version in versions[:-1]: delete_command = f’aws lambda delete-function –function-name {function_name}:{version[“Version”]} –region {region}’ subprocess.check_output(delete_command.split()) print(f’Deleted {function_name}:{version[“Version”]}’) |
This script does not do what it says right out of the box. If you are like me and scrolled to this section and copied this script without reading the text (yes ok I do this sometimes too, but listen!): this code tries to delete the most recent version and fails. On the other hand, some trivial modifications lead to a rough and ready tool hand sculpted for exactly my purpose, and for no other whatsoever.
I mentioned this query of mine to Chat GPT was flawed: I was happy enough to have solved my problem that I copied this script over to the banter channel for my employer and quickly received a thumbs down. Why does it not use boto? Oh, because I asked it explicitly to use the CLI because I got caught up in that line of thought while I was troubleshooting at the start. I’m sure there are countless other things wrong with this script, but the point for me is it worked for the one-off purpose I had for it.
Scripting, in my book, with Chat GPT: a solid win. Just do not copy that script, please. I will not be responsible for what happens! I think Chat GPT can only become more useful over time. It’s not just a fun new toy: if your job involves problem-solving, try it out at work!