Context Engineering for PMs: RAG vs Fine-Tuning vs Prompting

What is context engineering?

A model is only as good as the context you give it. Context engineering is the craft of giving it the right context to do the job well.

That is the whole idea. The model has general knowledge. Your job is to feed it the specifics, the instructions, the data, the examples, so it produces something useful for your actual product.

It is becoming the core skill of the applied AI PM because it is where product decisions live. And it comes down to three tools.

The three tools, and what each is for

Think of these as three dials, not three separate worlds. Most products use more than one.

• Prompting. How you instruct the model. The fastest, cheapest way to shape behavior. You change the words, you change the output.
• RAG (retrieval-augmented generation). Giving the model a trusted, external knowledge base to pull from before it answers. This is how you hand it current or private information it was never trained on, and it cuts hallucination.

Fine-tuning. Changing the model’s base behavior by training it further on curated data. Powerful, expensive, and slow. It changes who the model is, not just what it knows right now.

Each solves a different problem. Reaching for the wrong one is where the money goes to die.

The decision: which tool, when

Here is the sequence that saves you months. Always work down this list in order, and stop at the first tool that solves your problem.

1. Start with prompting. Most of the time, a better prompt is the answer. It is instant and free to try. Exhaust it before you touch anything heavier.
2.  If it is a knowledge problem, use RAG. The model gives wrong or outdated answers because it does not have your data? Do not retrain it. Give it a knowledge base to read from. Notion AI and NotebookLM live here, they answer from sources you provide.
3. If it is a behavior problem, consider fine-tuning. The model needs a consistent new personality, format, or skill that prompting cannot hold? Only now do you consider fine-tuning. And even then, prove prompting cannot do it first

That order is the skill. It is also exactly what a hiring manager is testing when they ask “when would you use RAG versus fine-tuning?”

The mistake that costs teams months

The classic rookie move is fine-tuning when a good prompt would have done the job.

It feels sophisticated. It is usually a waste. Fine-tuning burns time, money, and data you may not have, and half the time, the result is worse than a well-written prompt.

Across the teams and PMs we have worked with at HelloPM, the ones who ship fast share one habit: they treat fine-tuning as the last resort, not the first instinct. They squeeze prompting dry, reach for RAG when the problem is missing knowledge, and only fine-tune when they can name exactly why the cheaper tools failed. That discipline is context engineering in practice.

Let’s understand this with an example

Say you are building an AI assistant for your company’s support team.

• It gives generic answers → prompt it with the tone, format, and rules you want.
• It does not know your product’s latest features or your refund policy → add RAG over your help docs and policies.
• You need it to consistently follow a very specific, branded conversational style that is prompting, keeps drifting from → now you have a fine-tuning case.

Notice how far prompting and RAG get you before fine-tuning even comes up. That is the point.

Frequently asked questions

1. What is context engineering in simple terms?

It is giving an AI model the right context to do a job well, using three tools: prompting (instructions), RAG (external knowledge), and fine-tuning (changed base behavior). The skill is knowing which one to use for a given problem.

2. Is context engineering the same as prompt engineering?

No. Prompting is one tool in context engineering. Context engineering is the broader craft that also includes RAG and fine-tuning, and the judgment of when to use each.

3. When should a PM use RAG instead of fine-tuning?

Use RAG when the problem is missing knowledge, the model needs current or private information it does not have. Use fine-tuning only when the problem is a base behavior that prompting cannot hold, and only after proving that prompting and RAG fall short.

4. Do PMs need to build these systems themselves?

No. You need to understand what each tool does and when to use it, so you can make the product call and give engineering a clear direction. You do not need to implement RAG or run a fine-tuning job yourself.

5. Why do teams overuse fine-tuning?

It sounds sophisticated and feels like “real” AI work. In practice, most problems are solved by better prompting or by adding retrieval. Fine-tuning is expensive and slow, and it is rarely the first right answer.