With AI and data-driven solutions in focus, the debate between retrieval-augmented generation (RAG) and fine-tuning LLMs remains pivotal. In conversation with AIM, Parita Desai, senior solutions architect at Fractal, offered her insights into these approaches, elucidating their distinct roles, advantages, and practical applications in various industries.
Both RAG and fine-tuning involve intricate processes that require meticulous planning and execution. While they serve different purposes, their workflows share some common elements, especially when it comes to data preparation and deployment. Desai broke down the steps involved for productionising a fine-tuned model and a RAG system.
Productionising a Fine-Tuned Model
- Data Preparation: The process begins with preparing labelled data specific to the task at hand. This labelled dataset forms the foundation for training the model, ensuring it can generate precise outputs.
- Model Training: Once the data is ready, the pre-trained model is fine-tuned by training it on the labelled dataset. This step tailors the model’s capabilities to the specific requirements of the task, such as converting legacy SAS codes into BigQuery, as Desai described in one of her projects.
- Model Deployment: After fine-tuning, the model is deployed into the production environment, ready to serve real-world applications.
- Inference: At runtime, the fine-tuned model generates predictions or outputs based on the data it was trained on, delivering domain-specific responses that meet the user’s needs.
Productionising a RAG System
- Data Preparation: RAG starts by integrating with a large corpus of data, both internal and external. This corpus serves as the base from which the LLM can retrieve information in response to user queries.
- Retrieval Component: A retrieval component, often implemented as a vector database, is then set up to search through this large corpus, enabling the system to find relevant information efficiently.
- LLM Integration: The next step involves integrating the retrieval component with the LLM. The LLM uses the output from the retrieval system, combined with the user query and prompt, to generate a contextual response.
- Deployment: Both the retrieval system and the language model are deployed into the production environment, ensuring seamless operation.
- Inference: During inference, the system retrieves information from the corpus and augments the LLM with this data. The LLM then generates a factually grounded response by combining the user query, prompt, and additional information from the retrieval system.
Practical Use Cases and Integration
Desai shared several real-world applications for both RAG and fine-tuning. In the legal sector, Fractal implemented a legal advisor using RAG to analyse customer documents and provide financial advice. In healthcare, a similar approach was used to create an advisor agent, demonstrating RAG’s versatility.
One practical application of RAG that Desai mentioned was using a query database to provide specific customer data, enabling more accurate responses. “By building a RAG pipeline and feeding context along with the user query, we get precise answers tailored to the customer’s data,” she elaborated.
On the other hand, fine-tuning is about obtaining domain-specific responses. Desai described a project involving code conversion for a banking customer, where legacy SAS codes were converted into BigQuery. “We used fine-tuning by creating a labelled dataset of 500 points with input-output pairs and passing it to the pre-trained LLM,” she said.
Desai emphasised the importance of chunking in handling large datasets, as seen in a project involving code conversion for a banking customer.
Best Practices and Recommendations
When comparing the advantages of RAG and fine-tuning, Desai noted that RAG provides accuracy and dynamic responses by incorporating real-time context, while fine-tuning offers domain-specific responses by training the model on labelled data.
“RAG is less costly and requires fewer resources compared to fine-tuning, which involves more infrastructure and computational power,” she explained.
However, limitations exist. Desai pointed out that RAG may struggle with keywords having multiple meanings in different contexts. “Combining semantic search with keyword search, known as hybrid search, can mitigate this issue,” she suggested.
Desai advises businesses to start with experimentation, assessing the availability of data for the specific use case. Whether the goal is cost-efficiency, dynamic or static data handling, or integration complexity, understanding the business problem is key to choosing between RAG and fine-tuning.
In conclusion, RAG and fine-tuning offer unique advantages and are not mutually exclusive. “It’s not about one being better than the other; it depends on the business problem you’re trying to solve,” Desai asserted. The choice between these approaches should be driven by the specific needs and goals of the organisation.
