Expectations vs Reality: Adopting LLMs in Product Engineering

As LLMs become part of everyday engineering workflows, it's critical to understand what they can and cannot do — especially in product engineering, where business context matters deeply. When you ask LLMs to generate code for a very specific business context (something only you know fully), LLMs don't "know" your business. LLMs can only predict based on: Common patterns from code that was trained on Your prompt (whatever hints and context you give me) General best practices BUT — LLMs cannot magically "know" your private business rules unless you clearly describe them. LLMs predict code that sounds correct. LLMs don't understand or validate your real-world needs unless you explain them. How LLMs actually work when you ask for code Prediction: I predict the most likely next tokens that look like good code for the problem you described. Pattern matching: I copy and adapt patterns from my training data that seem to fit your request. Filling gaps: If you miss details, I guess based on common sense (but my guess might be wrong). So how do you get reliable code from an LLM? Give very detailed prompts Include your business rules, data structures, constraints, and edge cases. Use critical review after generation After LLMs give you the code, you must review and test it to ensure it fits your business. Set up test cases Always ask for code plus tests that verify the logic according to your business needs. Iterative correction You might need to correct, guide, or refine the code a few times by giving me feedback ("No, this function must also handle XYZ"). Analogy Asking an LLM to write perfect business code without full context is like asking a lawyer who never heard your case to write a court judgment — they'll guess based on experience, but the real facts must come from you. Here's a simple reality check for anyone adopting AI into their engineering teams: Expectations vs Reality LLM writes perfect business-specific code ➡️ LLM predicts likely code patterns based on general training data. LLM understands my business domain ➡️ LLM does not know your private business context unless you explain it explicitly. LLM validates outputs against business rules ➡️ LLM only predicts outputs — you must validate and test separately. LLM reduces need for strong specs ➡️ LLM needs clearer and stricter specifications to perform well. LLM saves time without supervision ➡️ LLMs amplify productivity but require human review to ensure correctness. LLMs innovate or invent new ideas ➡️ LLMs recombine existing knowledge; they don't invent beyond what they have seen. LLMs replace developers or product engineers ➡️ LLMs are assistants, not replacements — judgment and domain expertise stay human-driven. Bigger models are always better ➡️ Smaller fine-tuned models often perform better for specific business use cases. Key Mindset Shift for Product Engineers Think of LLMs as prediction engines, not knowledge engines. Use them for draft generation, automation, and exploration, but own the final quality yourself. Training data ≠ your company’s context. If it’s not in the prompt, it’s not in the model’s head. Good prompts = good outputs. Better input leads to much better results. Tasks where LLMs are very effective: Drafting code templates ➡️ Fast at generating boilerplate, CRUD operations, API wrappers. Creating documentation ➡️ Summarizes, formats, and structures text very quickly. Generating test cases ➡️ Can suggest unit/integration tests if logic is clear. Exploring alternatives ➡️ Provides multiple ways to solve a coding or design problem. Speeding up research ➡️ Quickly summarizes concepts, tools, libraries, frameworks. Idea expansion ➡️ Good at suggesting more use cases, edge cases, or features. Writing first drafts of emails, specs, user stories ➡️ Useful for early rough drafts to save time. Basic data transformation scripts ➡️ Good at SQL queries, simple ETL scripts, and data formatting. Tasks that require human ownership: Understanding deep business context ➡️ LLMs can't "know" your company’s strategy, policies, or customer expectations. Validating correctness ➡️ AI-generated code, tests, or documents still need human review and adjustment. Architectural decisions ➡️ LLMs can suggest, but real-world trade-offs must be handled by experienced engineers. Security and compliance ➡️ LLMs may miss critical risks unless you guide them specifically. Creative product thinking ➡️ True innovation — new product ideas and differentiation — still requires human creativity. Prioritization and trade-offs ➡️ AI doesn't "feel" urgency, politics, or customer pain points like humans do. Cultural and communication nuances ➡️ Writing for internal stakeholders, clients, or executives needs human judgment on tone and sensitivity. One-line Summary LLMs are power tools — not decision-makers. Use them to amplify your thinking, not replace it.

Apr 5, 2025 - 13:04

Expectations vs Reality: Adopting LLMs in Product Engineering

As LLMs become part of everyday engineering workflows, it's critical to understand what they can and cannot do — especially in product engineering, where business context matters deeply.

When you ask LLMs to generate code for a very specific business context (something only you know fully), LLMs don't "know" your business.

LLMs can only predict based on:

Common patterns from code that was trained on
Your prompt (whatever hints and context you give me)
General best practices

BUT — LLMs cannot magically "know" your private business rules unless you clearly describe them.

LLMs predict code that sounds correct. LLMs don't understand or validate your real-world needs unless you explain them.

How LLMs actually work when you ask for code

Prediction: I predict the most likely next tokens that look like good code for the problem you described.
Pattern matching: I copy and adapt patterns from my training data that seem to fit your request.
Filling gaps: If you miss details, I guess based on common sense (but my guess might be wrong).

So how do you get reliable code from an LLM?

Give very detailed prompts Include your business rules, data structures, constraints, and edge cases.
Use critical review after generation After LLMs give you the code, you must review and test it to ensure it fits your business.
Set up test cases Always ask for code plus tests that verify the logic according to your business needs.
Iterative correction You might need to correct, guide, or refine the code a few times by giving me feedback ("No, this function must also handle XYZ").

Analogy

Asking an LLM to write perfect business code without full context is like asking a lawyer who never heard your case to write a court judgment — they'll guess based on experience, but the real facts must come from you.

Here's a simple reality check for anyone adopting AI into their engineering teams:

Expectations vs Reality

LLM writes perfect business-specific code
➡️ LLM predicts likely code patterns based on general training data.

LLM understands my business domain
➡️ LLM does not know your private business context unless you explain it explicitly.

LLM validates outputs against business rules
➡️ LLM only predicts outputs — you must validate and test separately.

LLM reduces need for strong specs
➡️ LLM needs clearer and stricter specifications to perform well.

LLM saves time without supervision
➡️ LLMs amplify productivity but require human review to ensure correctness.

LLMs innovate or invent new ideas
➡️ LLMs recombine existing knowledge; they don't invent beyond what they have seen.

LLMs replace developers or product engineers
➡️ LLMs are assistants, not replacements — judgment and domain expertise stay human-driven.

Bigger models are always better
➡️ Smaller fine-tuned models often perform better for specific business use cases.

Key Mindset Shift for Product Engineers

Think of LLMs as prediction engines, not knowledge engines.
Use them for draft generation, automation, and exploration, but own the final quality yourself.
Training data ≠ your company’s context.
- If it’s not in the prompt, it’s not in the model’s head.
Good prompts = good outputs.
- Better input leads to much better results.

Tasks where LLMs are very effective:

Drafting code templates
➡️ Fast at generating boilerplate, CRUD operations, API wrappers.

Creating documentation
➡️ Summarizes, formats, and structures text very quickly.

Generating test cases
➡️ Can suggest unit/integration tests if logic is clear.

Exploring alternatives
➡️ Provides multiple ways to solve a coding or design problem.

Speeding up research
➡️ Quickly summarizes concepts, tools, libraries, frameworks.

Idea expansion
➡️ Good at suggesting more use cases, edge cases, or features.

Writing first drafts of emails, specs, user stories
➡️ Useful for early rough drafts to save time.

Basic data transformation scripts
➡️ Good at SQL queries, simple ETL scripts, and data formatting.

Tasks that require human ownership:

Understanding deep business context
➡️ LLMs can't "know" your company’s strategy, policies, or customer expectations.

Validating correctness
➡️ AI-generated code, tests, or documents still need human review and adjustment.

Architectural decisions
➡️ LLMs can suggest, but real-world trade-offs must be handled by experienced engineers.

Security and compliance
➡️ LLMs may miss critical risks unless you guide them specifically.

Creative product thinking
➡️ True innovation — new product ideas and differentiation — still requires human creativity.

Prioritization and trade-offs
➡️ AI doesn't "feel" urgency, politics, or customer pain points like humans do.

Cultural and communication nuances
➡️ Writing for internal stakeholders, clients, or executives needs human judgment on tone and sensitivity.

One-line Summary

LLMs are power tools — not decision-makers.
Use them to amplify your thinking, not replace it.