AI as a Coding Tool: A Mixed Experience

Even though AI has been a hot topic in recent times, I had been reluctant to use it in my work as I have always trusted my own research instead of getting an answer directly from someone or something. Now this does not mean I did not use AI at all, I was keenly using it for improving documents, emails or other written content where I found it very helpful in getting ideas and correcting grammar. Although I did not like how long the suggestions were from ChatGPT because I always prefer to get straight to the point, I did however find it useful to pick some words or sentences from the suggestions to make my writings better. So in terms of writing I found the best utilization of the tool was to get suggestions and then write in my own terms taking the suggestion and using what I like instead of copy pasting the whole paragraph from ChatGPT which will contain a thousand adjectives around instead of coming to the point. Now getting back to coding, I was skeptical of using this tool but I decided to experiment with it for one particular task that involved writing some automated tests. What I learned from this experience was to not blindly rely on the information provided by AI systems but to use them as a tool. In short, I went into an approach that was confirmed by ChatGPT to work, trusting that I wasted many hours trying to fix non existent issues when the main issue was the approach. The only way I got to learn that the approach was not possible was when I did my own research. In the end however the AI did recommend some solid alternatives which I ended up implementing and successfully completed my goals. Still this was not a very straightforward process. In the following sections I go into more technical details of the overall experience. The goal of my task was to write some automated tests for an API endpoint that returns a list of products. The tools I was using for this purpose were Jest and Typescript. This would appear to be a quite simple task if anyone has worked on similar things, the idea would be to define a test case in which you call the API endpoint and then loop through all the products and add the tests for each of them. Sounds quite straightforward, but for me it was not quite what I wanted. Adding all this logic in a single test case would cause the test to stop on the first failure it finds and then we would not get a complete picture of all the products as I would have liked. My idea was to define multiple tests that would run for each product and at the end we would get a nice overview of any issues encountered with any individual product and we could then track the issue easily because the test description would mention the product ID. So for comparison in terms of code, below is what would be the simpler approach: describe('GET /api/products', () => { test('should return a list of products', async () => { const response = await request(app).get('/api/products'); expect(response.status).toBe(200); expect(response.body.products).toBeInstanceOf(Array); expect(response.body.length).toBeGreaterThan(0); response.body.forEach((product: any) => { expect(product).toHaveProperty('id'); expect(product).toHaveProperty('name'); expect(product).toHaveProperty('price'); }); }); }); What I wanted to achieve was to write some more organized tests like this: describe('GET /api/products', () => { const products = []; beforeAll(() => { const response = await request(app).get('/api/products'); if (response.status === 200 && response.body.products) { products = response.body.products } else { throw new Error('Something went wrong in the API response'); } }); describe.each(products)('validating product with id %p', (product) => { test('It should have a name', () => { expect(product).toHaveProperty('name'); }); test('It should have a price', () => { expect(product).toHaveProperty('price'); }); }); }) The challenge here was to see if it is possible to define tests in this sort of dynamic data where we first load the result of the API endpoint and then define the tests. Here is where I decided to use ChatGPT. I went and presented my theory of using test.each from Jest on dynamic data that was loaded in a beforeAll block and asked the AI if that would work and it quickly responded that yes it should and gave me a boilerplate to use. I was happy with this and started using parts of the boilerplate to write my test cases. Once I was done I happily hit the run tests command and waited to see the response. The output I got was the always dreaded cannot access property of undefined. Seeing this was not much of a bummer as any engineer working with JavaScript would have encountered this issue millions of times. So I went ahead and searched for where the issue could be in the code. The thing that I could not wrap my head around was that the issue seems to be in the object wh

Mar 30, 2025 - 08:48

Even though AI has been a hot topic in recent times, I had been reluctant to use it in my work as I have always trusted my own research instead of getting an answer directly from someone or something. Now this does not mean I did not use AI at all, I was keenly using it for improving documents, emails or other written content where I found it very helpful in getting ideas and correcting grammar. Although I did not like how long the suggestions were from ChatGPT because I always prefer to get straight to the point, I did however find it useful to pick some words or sentences from the suggestions to make my writings better. So in terms of writing I found the best utilization of the tool was to get suggestions and then write in my own terms taking the suggestion and using what I like instead of copy pasting the whole paragraph from ChatGPT which will contain a thousand adjectives around instead of coming to the point.

Now getting back to coding, I was skeptical of using this tool but I decided to experiment with it for one particular task that involved writing some automated tests. What I learned from this experience was to not blindly rely on the information provided by AI systems but to use them as a tool. In short, I went into an approach that was confirmed by ChatGPT to work, trusting that I wasted many hours trying to fix non existent issues when the main issue was the approach. The only way I got to learn that the approach was not possible was when I did my own research. In the end however the AI did recommend some solid alternatives which I ended up implementing and successfully completed my goals. Still this was not a very straightforward process. In the following sections I go into more technical details of the overall experience.

The goal of my task was to write some automated tests for an API endpoint that returns a list of products. The tools I was using for this purpose were Jest and Typescript. This would appear to be a quite simple task if anyone has worked on similar things, the idea would be to define a test case in which you call the API endpoint and then loop through all the products and add the tests for each of them.

Sounds quite straightforward, but for me it was not quite what I wanted. Adding all this logic in a single test case would cause the test to stop on the first failure it finds and then we would not get a complete picture of all the products as I would have liked. My idea was to define multiple tests that would run for each product and at the end we would get a nice overview of any issues encountered with any individual product and we could then track the issue easily because the test description would mention the product ID.

So for comparison in terms of code, below is what would be the simpler approach:


describe('GET /api/products', () => {
  test('should return a list of products', async () => {
    const response = await request(app).get('/api/products');

    expect(response.status).toBe(200);
    expect(response.body.products).toBeInstanceOf(Array);
    expect(response.body.length).toBeGreaterThan(0);

    response.body.forEach((product: any) => {
      expect(product).toHaveProperty('id');
      expect(product).toHaveProperty('name');
      expect(product).toHaveProperty('price');
    });
  });
});

What I wanted to achieve was to write some more organized tests like this:

describe('GET /api/products', () => {
  const products = [];
  beforeAll(() => {
    const response = await request(app).get('/api/products');
    if (response.status === 200 && response.body.products) {
     products = response.body.products
    } else {
      throw new Error('Something went wrong in the API response');
    }
  });
  describe.each(products)('validating product with id %p',  (product) => {
     test('It should have a name', () => {
       expect(product).toHaveProperty('name');
     });

     test('It should have a price', () => {
       expect(product).toHaveProperty('price');
     });
  });
})

The challenge here was to see if it is possible to define tests in this sort of dynamic data where we first load the result of the API endpoint and then define the tests. Here is where I decided to use ChatGPT. I went and presented my theory of using test.each from Jest on dynamic data that was loaded in a beforeAll block and asked the AI if that would work and it quickly responded that yes it should and gave me a boilerplate to use. I was happy with this and started using parts of the boilerplate to write my test cases. Once I was done I happily hit the run tests command and waited to see the response.

The output I got was the always dreaded cannot access property of undefined. Seeing this was not much of a bummer as any engineer working with JavaScript would have encountered this issue millions of times. So I went ahead and searched for where the issue could be in the code. The thing that I could not wrap my head around was that the issue seems to be in the object which contains the response of the API. This sent me to a detour trying to figure out if the API is working and if I am accessing the correct properties. I tried many things to figure this out but could not find the issue. What clicked at the end was when I tried initializing the variable I was using to store the API result with an empty array and then running the tests. Now Jest returned a different but more confusing message to me stating no tests were defined.

This was quite strange again and since I had trusted the AI I believed this should have worked. I tried a few different iterations of loading the API data in different ways, playing with beforeAll and async/await statements but nothing seems to be working.

After wasting many hours I decided to do some research I would have done originally without the use of AI and lo and behold within the first few results of my Google query I stumbled on a GitHub issue in the Jest repo discussing tests on dynamic data, from which I came to understand that Jest does not support this sort of thing. The way it works is that it will first go over the test file to figure out the number of tests to execute but since I was using an empty array which was supposed to have the list of products after getting the result from the API, Jest assumed there were no tests defined.

The next step I took was to communicate this to the AI that this is not supported after which it politely apologized and recommended alternate approaches to solve the problem, one of which was to put everything in a single test block which I wanted to avoid from the start. But the other solution was quite helpful, which was to preload the API data before executing the tests which is supported by Jest in terms of executing a script before running the test. I quite liked this approach and the AI recommended storing the API response in a JSON file and then loading the data in the tests. But here again ChatGPT made a mistake which I quickly realised this time. The AI recommended in its generated code to read the JSON data from the file in a beforeAll block but since this would be loaded at runtime we would run into the same issue as before where Jest would not know how many tests to run.

At the end I used a simple import statement to load the JSON data at the top of my tests file, in addition to this I had configured Jest to run a custom script which would call the API endpoint and write the response to the JSON file that we import in the tests. This way Jest finally considered the data as static and was able to calculate the number of tests to be run based on the length of the list of products returned by the API.

All in all this was a very mixed experience for me using AI as a coding tool, on one hand I was frustrated and annoyed at being led to believe that an approach would work and then wasting many hours on it but then also when I explained the issue with it’s own suggestions to the AI, it was quickly able to give me an alternate solution that worked and might have taken me some time to figure out otherwise.

The conclusion I got from this is that AI is a powerful tool for helping with coding but I would be as much skeptical of some of the answers it provides for more complex tasks as I am of humans. Maybe it will improve and take over my own job someday, it is a scary thought but that is life, it throws stuff at you and you have to learn to adapt. I guess we should all start preparing for backup careers, I am thinking of becoming a private chef. Otherwise I am pretty hopeful there would be work for me in fixing badly managed codebases written by AI in the future