Fixing Bugs Found on Production

You get a message from the Customer Success team, "can someone please help me identified what is wrong with this?", you find yourself with some time on your hands and decide to help out. "Yes, I can help you out with this. Do you know what are the steps to replicate this issue?". If you are lucky Customer Success says yes and gives you the steps to replicate, on the other hand they will say no. If they said no that means they probably spent a good amount of time trying to replicate the issue but were not able to, now it is on you to guess what what the problem is. Good luck. If you were the one that worked on the feature that is defective you may have some intuition guideline, "this is probably because I did not test this properly when I coded it", in this case you may already know the solution. BUT, if you did not replicate the issue and you just code based on intuition you might be either fixing something else entirely different or even worst introducing a new bug. You might feel tempted to just write some code to fix what you thing is broken, but that's like when a customer at a restaurant complains about a meal being salty, is it the carb that's salty, the protein, or vegetables? If you think you added too much salt to the carb and you just bring back a new meal with just the carb having less salt the customer could actually say that the whole meal is still salty. This is why replicating is crucial. The equivalent of replicating a bug in the restaurant industry is asking the customer to try each of the food items in the meal and tell you which specifically is the one that produces the salty sensation in their palate. In the tech world, Customer Success is the server but in this case the server tries the food to confirm that the bug is present. This way they can say, this specific food item is salty. The Chef would then try the item and confirm, "yes, the problem is there, fixing". Sometimes replicating is very difficult and time consuming. Imagine having the same food item but many different utensils along with different sitting positions that will impact how salty the food item feels. If the customer uses a spoon and sits with the legs crossed the food doesn't feel salty; if the customer uses a tiny fork and sits up straight the food doesn't feel salty; if the customer uses a big fork made out of titanium and eats while standing the food feels salty. I tell you, is difficult and time consuming. After identifying the user experience or user flow that replicates the issue, the dev will then evaluate those steps and recreate them in a development environment, where code can be executed and does not impact any real data that is being used on production. If the food feels salty when using a big fork made of titanium and the customer eats while standing, that means the dev will narrow it down even further. Is the fork the problem or the position of the body? Once this problematic variable is identified then this realization will point directly to the code that is at fault. The dev fixes that part of the code and that's it. Problem is solved. Conclusion Customer Success asks for help or creates a ticket with replicating steps. After this the Engineer goes ahead and replicates on her/his side and identifies the precise location in the code that introduces the bug on production. Once this is identified the problem is fixed and deployed to production as a hot fix.

May 13, 2025 - 14:48

You get a message from the Customer Success team, "can someone please help me identified what is wrong with this?", you find yourself with some time on your hands and decide to help out. "Yes, I can help you out with this. Do you know what are the steps to replicate this issue?". If you are lucky Customer Success says yes and gives you the steps to replicate, on the other hand they will say no. If they said no that means they probably spent a good amount of time trying to replicate the issue but were not able to, now it is on you to guess what what the problem is. Good luck.

If you were the one that worked on the feature that is defective you may have some intuition guideline, "this is probably because I did not test this properly when I coded it", in this case you may already know the solution. BUT, if you did not replicate the issue and you just code based on intuition you might be either fixing something else entirely different or even worst introducing a new bug.

You might feel tempted to just write some code to fix what you thing is broken, but that's like when a customer at a restaurant complains about a meal being salty, is it the carb that's salty, the protein, or vegetables? If you think you added too much salt to the carb and you just bring back a new meal with just the carb having less salt the customer could actually say that the whole meal is still salty. This is why replicating is crucial.

The equivalent of replicating a bug in the restaurant industry is asking the customer to try each of the food items in the meal and tell you which specifically is the one that produces the salty sensation in their palate. In the tech world, Customer Success is the server but in this case the server tries the food to confirm that the bug is present. This way they can say, this specific food item is salty. The Chef would then try the item and confirm, "yes, the problem is there, fixing".

Sometimes replicating is very difficult and time consuming. Imagine having the same food item but many different utensils along with different sitting positions that will impact how salty the food item feels. If the customer uses a spoon and sits with the legs crossed the food doesn't feel salty; if the customer uses a tiny fork and sits up straight the food doesn't feel salty; if the customer uses a big fork made out of titanium and eats while standing the food feels salty. I tell you, is difficult and time consuming.

After identifying the user experience or user flow that replicates the issue, the dev will then evaluate those steps and recreate them in a development environment, where code can be executed and does not impact any real data that is being used on production. If the food feels salty when using a big fork made of titanium and the customer eats while standing, that means the dev will narrow it down even further. Is the fork the problem or the position of the body?
Once this problematic variable is identified then this realization will point directly to the code that is at fault. The dev fixes that part of the code and that's it. Problem is solved.

Conclusion

Customer Success asks for help or creates a ticket with replicating steps. After this the Engineer goes ahead and replicates on her/his side and identifies the precise location in the code that introduces the bug on production. Once this is identified the problem is fixed and deployed to production as a hot fix.