Context Engineering for PMs: RAG vs Fine-Tuning vs Prompting

What is context engineering?

A model is only as good as the context you give it. Context engineering is the craft of giving it the right context to do the job well.

That’s the whole idea. The model has broad general knowledge. Your job is to feed it the specifics (the instructions, the data, the examples) so it produces something useful for your actual product.

It’s becoming the core skill of applied AI PMs because it’s where product decisions live. And it comes down to three tools.

The three tools, and what each is for

Think of these as three dials, not three separate worlds. Most real products use more than one.

• Prompting. How you instruct the model. The fastest, cheapest way to shape behavior. Change the words, change the output.
• RAG (retrieval-augmented generation). Giving the model a trusted, external knowledge base to pull from before it answers. This is how you hand it current or private information it was never trained on, and it cuts down hallucination by grounding answers in real sources.
• Fine-tuning. Changing the model’s base behavior by training it further on curated examples. Powerful, expensive, and slow. It changes who the model is, beyond what it knows right now.

Each solves a different problem. Reaching for the wrong one is where the budget dies.

RAG vs fine-tuning vs prompting

Here’s the same three tools laid out against the questions a PM actually asks:

The decision: which tool, when

Here’s the sequence that saves you months. Work down this list in order, and stop at the first tool that solves your problem.

• Start with prompting. Most of the time, a better prompt is the answer. It’s instant and free to try. Exhaust it before you touch anything heavier. If you want to sharpen this muscle, our guide to ChatGPT for product managers is a good place to practice.
• If it’s a knowledge problem, use RAG. The model gives wrong or outdated answers because it doesn’t have your data. Don’t retrain it. Give it a knowledge base to read from. Many of the AI tools product managers use work this way, answering from sources you provide rather than memory.
• If it’s a behavior problem, consider fine-tuning. The model needs a consistent new personality, format, or skill that prompts can’t hold? Only now do you consider fine-tuning. And even then, prove prompting can’t do it first.

That order is the skill. It’s also exactly what a hiring manager is testing when they ask, “When would you use RAG versus fine-tuning?”

The mistake that costs teams months

The classic rookie move is fine-tuning when a good prompt would have done the job.

It feels sophisticated. It’s usually a waste. Fine-tuning burns time, money, and data you may not have, and half the time, the result is worse than a well-written prompt.

In our experience, the teams that ship fast share one habit: they treat fine-tuning as the last resort, not the first instinct. They squeeze prompting dry, reach for RAG when the problem is missing knowledge, and only fine-tune when they can name exactly why the cheaper tools failed. That discipline is context engineering in practice, and it’s really the same prioritization instinct the core product management frameworks train, applied to AI.

A quick worked example

Say you’re building an AI assistant for your company’s support team.

• It gives generic answers → prompt it with the tone, format, and rules you want.
• It doesn’t know your product’s latest features or your refund policy → add RAG over your help docs and policies.
• You need it to consistently follow a very specific, branded conversational style that is prompting, keeps drifting from → now you have a fine-tuning case.

Notice how far prompting and RAG get you before fine-tuning even comes into play. That’s the point.

Where prompting and RAG go deeper

Context engineering is the map. Two of the three tools, prompting and RAG, are deep enough to be crafts of their own, and each deserves its own study when you’re ready to go past the decision layer. For now, the win is the order: knowing which dial to turn, and in what sequence.

Final Words

Don’t learn all three tools at once. Learn the order.

Next time an AI output is wrong, don’t ask “should we fine-tune this?” Ask the questions in sequence: Can a better prompt fix it? If not, is it missing knowledge we could feed it with RAG? Only if both fail do you reach for fine-tuning. A good way to build the instinct: run a product teardown on an AI product you use and guess which of the three tools each behavior comes from.

When you want to go deep on each tool with real practice, take a look at our AI PM curriculum and free AI PM resources. But the decision is yours to start using today.

Prompt first. Retrieve the second. Fine-tune last.

Frequently asked questions

1. What is context engineering in simple terms?

It’s giving an AI model the right context to do a job well, using three tools: prompting (instructions), RAG (external knowledge), and fine-tuning (changed base behavior). The skill is knowing which one to use for a given problem.

2. Is context engineering the same as prompt engineering?

No. Prompting is one tool inside context engineering. Context engineering is the broader craft that also includes RAG and fine-tuning, plus the judgment of when to use each.

3. When should a PM use RAG instead of fine-tuning?

Use RAG when the problem is missing knowledge, when the model needs current or private information it doesn’t have. Use fine-tuning only when the problem is a base behavior that prompting can’t hold, and only after proving that prompting and RAG fall short.

4. Do PMs need to build these systems themselves?

No. You need to understand what each tool does and when to use it, so you can make the product call and give engineering a clear direction. You don’t need to implement RAG or run a fine-tuning job yourself.

5. Why do teams overuse fine-tuning?

It sounds sophisticated and feels like “real” AI work. In practice, most problems are solved by a better prompt or by adding retrieval. Fine-tuning is expensive and slow, and it’s rarely the first right answer.

6. Does RAG or fine-tuning stop hallucination?

RAG reduces it by grounding answers in real sources you provide, so the model quotes your data instead of guessing. Fine-tuning doesn’t reliably fix hallucination; it changes behavior and format more than it changes truthfulness. If the problem is incorrect facts, reach for retrieval, not retraining.