RAG vs fine-tuning: which should we try after improving the prompt?

Improve instructions or tools first when they explain the failure. Use retrieval when the answer needs approved information supplied at runtime. Consider fine-tuning when stable behaviour remains inconsistent despite clear instructions, suitable examples and the right evidence.

RAG vs fine-tuning: what does each approach change?

RAG supplies retrieved evidence when a request runs. Fine-tuning changes model parameters to influence behaviour. Improve the prompt first when clearer instructions, examples or a deterministic tool address the observed failure.

These approaches can be combined. Fine-tuning does not remove the need for current facts or access controls; retrieved text does not guarantee that the model uses it correctly.

Use the editable RAG versus fine-tuning scorecard to record the same held-out tasks, review rubric and lifecycle costs for each candidate. Leave results unfilled until the work has run.

Is the failure missing knowledge, inconsistent behaviour, poor instructions or a missing business tool?

Classify actual failed tasks before choosing an intervention. Start with a reviewed expected result and identify what information or action the system needed.

A missing procedure suggests an evidence problem. An invented live balance suggests a missing authorised lookup. Repeatedly choosing the wrong category despite correct facts may suggest unclear labels or unstable model behaviour. These are diagnostic hypotheses, not automatic classifications.

Try supplying the correct evidence manually while holding the task constant. If that fixes the answer, investigate retrieval or source availability. If it does not, inspect the instructions, examples and model capability. Also check whether reviewers agree on the expected answer; inconsistent labels cannot define a reliable training target.

For a calculation or live system record, propose a permission-limited deterministic tool. Teaching a model to imitate yesterday's balance would not meet a requirement for today's authorised value. Validate tool results and permitted actions independently of the language model.

Mixed failures may need more than one intervention. Keep their contributions separate during evaluation so a better retrieval result is not credited to training.

What can a prompt or retrieval change fix before any training is attempted?

Prompt changes can clarify the task and expected response; retrieval can supply relevant evidence that the model did not receive. Neither guarantees that the answer will be correct.

Hugging Face's prompt-engineering guide describes instructions and examples as ways to guide a model. For this pilot, remove contradictory instructions, name the allowed evidence and specify when to decline. Use a template or validator for a fixed output contract before collecting training examples solely to reproduce formatting.

Retrieval-augmented generation, or RAG, combines retrieved material with generation. The original RAG paper describes combining a generator with external retrieved information. Its historical results do not establish accuracy for your documents, local model or current deployment.

In the proposed document workflow, carry source identifiers, document versions and index timestamps into the evidence record. Apply user permissions before passing passages to the model. If current authorised evidence is missing, return an explicit limitation or send the task for review.

Measure retrieval and answering separately. A system can retrieve the wrong passage, or misuse a correct one. Adding more documents will not necessarily repair either problem. Test obsolete evidence, contradictory documents and instructions embedded in retrieved text before expanding access.

Which stable behaviours might justify an adapter rather than a longer prompt?

Trial an adapter when the task and labels are stable, representative examples exist, and the strongest prompt-based version still fails. Specialised classification, consistent terminology and repeated extraction conventions are plausible candidates.

An adapter is an additional trained component used with a base model. Hugging Face PEFT supports methods that train a limited set of parameters instead of every model weight. The LoRA paper describes freezing pretrained weights and learning low-rank updates.

Those mechanisms support a serious countercase to always extending the prompt: a well-evaluated adapter may encode stable conventions with fewer repeated examples. That is a hypothesis worth testing, not a claim of better local accuracy, speed or lower cost. PEFT's broad performance statements cannot size your hardware or predict the result.

Compare the adapter with the best simpler configuration, including validators and suitable tools. If it produces a useful improvement on unseen work and stays within operating limits, it may be preferable even when prompting remains possible.

Keep changing facts and access decisions outside the training promise. This pilot would not treat company-record training as a dependable fact-update mechanism or permission boundary. The adapter may still need retrieval for current evidence.

Do we have licensed, representative training examples and a genuinely separate evaluation set?

Proceed only after the data owner approves the intended training use and the team can evaluate on examples that did not shape the candidate. Availability of a file does not establish permission to train on it.

Separate software, base-model, adapter and dataset terms. PEFT's licence file uses Apache 2.0 for that software. It does not grant rights in an arbitrary model or your records. Inspect the selected model revision's actual terms, including relevant distribution conditions, before choosing a deployment path.

Hugging Face model cards can document licences, training datasets, limitations and evaluations. Use them to organise the review, while treating their assertions as information to verify rather than independent clearance.

For this pilot, reserve training examples for fitting the adapter, development examples for tuning every candidate, and a locked final test set. Group near-duplicates and related documents together when splitting. Otherwise a rewritten copy of one example can make apparent generalisation misleading.

Have a subject expert resolve disputed labels and identify costly mistakes. Record provenance, permitted purpose and retention for examples, checkpoints, embeddings, logs and backups. Deleting an original file must not be assumed to remove its influence from trained parameters; no unlearning behaviour has been established here.

How do we compare quality, maintenance and data-lifecycle costs across all three approaches?

Compare matched configurations on the same held-out tasks and include the work needed to operate and retire each option. Training time or prompt length alone is too narrow a cost measure.

The proposed pilot covers one low-risk workflow and one pinned base model. First retain the existing prompt as a baseline. Test a revised prompt with examples and validators, then add retrieval as one candidate and a small approved-data adapter as another. Test their combination only if the separate results justify it.

Where a read-only tool is needed, give matched candidates identical access. Record model revision, runtime, quantisation, hardware, decoding settings, context limits, retrieval snapshot and adapter settings. Evaluate the final served configuration, because a training checkpoint is not evidence that the intended deployment works.

Use development tasks to choose settings, then run the locked comparison with repeated stochastic trials. Report correctness, format, evidence support, refusal, latency and review effort separately. Keep human review of expensive errors. Inspect supplies evaluation components; it cannot decide what the business considers acceptable.

A local answer model does not establish a local evaluation path. The Ragas repository includes an external evaluator example and documents usage analytics with an opt-out. Review evaluator providers, telemetry, downloads and cloud fallback before processing sensitive samples. Test the agreed network boundary rather than inferring it from the model location.

The technical lead owns reproducibility and rollback. The data owner approves examples and documents; the security owner approves permissions and dependencies. Inputs are approved examples, a versioned corpus and sanitised tool fixtures. Outputs are draft answers, extracted fields, evidence references and evaluation records, with production writes disabled.

Estimate lifecycle cost over a common period: data preparation + integration + training or indexing + allocated hardware and power + maintenance + evaluation + human review + incident and retirement work. Use dated Australian-dollar inputs and state GST treatment. Measure review time separately; freed time is not automatically cash saved.

What result would justify abandoning the fine-tune rather than adding more training data?

Stop when the adapter fails to beat the strongest simpler baseline by a useful pre-agreed margin, or when its added risk and upkeep outweigh the benefit. More examples are justified only by a specific, correctable training-data gap.

Before execution, the workflow owner must set minimum task quality, acceptable refusal behaviour, review-effort limits and the improvement margin. The technical lead sets a bounded experiment budget and latency ceiling. Any unauthorised disclosure or tool action blocks expansion regardless of average scores.

Abandon or redesign the adapter if gains vanish on unseen task families, required formats regress, deployment compatibility fails or data rights remain unresolved. If the remaining errors come from missing evidence or a broken lookup, return to that component. Repeatedly tuning against the final test set would invalidate its role as unseen evidence.

After a blocking failure, restore the complete known-good configuration, including prompt, model, adapter state and retrieval snapshot. Keep the pilot isolated and use the existing manual workflow until acceptance is recorded.

Choose the intervention that addresses the diagnosed failure and passes the same acceptance criteria. Leave an inconclusive comparison unresolved rather than presenting training as progress merely because a new artefact exists.