RAG (Retrieval-Augmented Generation) has developed rapidly over the past two years. It has moved past the early proof-of-concept stage and into a critical period of large-scale enterprise adoption. Along the way, simple document chunking and basic vector search are no longer enough to meet enterprise expectations for knowledge base Q&A quality, and every stage of the RAG pipeline needs deeper optimization.

In the previous post on RAG optimization, we used the LinkAI agent platform as an example to cover the basic ideas of RAG optimization across knowledge base import, retrieval, generation, and feedback. Through long-term technical practice and product iteration since then, the LinkAI knowledge base module has gone through deeper improvements and upgrades. This post again follows the full lifecycle of knowledge base Q&A to share more advanced RAG optimization ideas and how the product has evolved.

Overall Approach

The ultimate goal of RAG optimization is for the agent to answer user questions accurately based on relevant content in the knowledge base, while minimizing model hallucination. Broken down across the RAG pipeline, this means:

  1. Import: Document parsing should preserve as much of the original information as possible and restore the layout structure; chunking should keep relatively self-contained content within a single chunk
  2. Retrieval: Retrieval should understand the meaning of the user's question, find as much relevant content as possible, and reduce noise from irrelevant content
  3. Generation: Answers should strictly follow the retrieved context, show document citations and suggested questions, and offer a better interaction experience
  4. Evaluation: A complete evaluation system is needed to measure answers, and the knowledge base is continuously improved based on feedback, forming a data loop

Below is the complete RAG pipeline on the LinkAI platform. Every stage has been optimized to improve overall knowledge base Q&A quality:

RAG pipeline

1. Document Import

Document import is where knowledge base retrieval begins. If key information in a document is lost at the very first step of ingestion, it is hard to get accurate results in later retrieval and generation. On top of its existing support for many document types, LinkAI has optimized parsing and chunking for documents with complex formatting, so formatting and paragraph structure are better preserved, and it has also improved and expanded the ways content gets into the knowledge base.

1. Smart Document Parsing

Document parsing reads the content of different file types for later chunking and ingestion. For files with complex elements and layouts, ordinary document parsing libraries or OCR tools struggle to capture the full information.

The platform adds a new Smart Parsing mode built on multimodal LLMs. It does a better job extracting complex elements such as text, images, tables, and formulas, handles complex layouts such as multi-column and mixed-content pages, and also recognizes scanned documents of limited clarity more reliably.

Smart document parsing

With Smart Parsing enabled, PDF, Word, PPT, and other file types are parsed and converted to Markdown. This preserves the paragraph structure of the original document well and gives later chunking a better foundation.

2. Chunking Strategy Optimization

Document chunking is another key part of knowledge base import. Because of model context limits, document content has to be split into chunks, which are embedded and stored; during a conversation, the most similar chunks are retrieved and passed to the LLM. Whether relatively complete content ends up in a single chunk, and whether the connection between adjacent chunks is preserved, therefore directly affects final retrieval quality.

Chunking strategy options

LinkAI offers flexible chunking modes for different scenarios:

  • Smart chunking: The default mode. The user sets a maximum chunk length, and the system picks separators based on the file type so that each chunk stays under the maximum length while keeping sentences intact and adjacent chunks coherent as far as possible
  • Custom chunking: Fully custom separators, with multiple separators applied recursively in order until each chunk is under the maximum length. The overlap between consecutive chunks can also be set to keep adjacent chunks continuous
  • Hierarchical chunking: Chunks along the document hierarchy, which better preserves the document's structure and keeps self-contained content in a single chunk. It applies to Markdown documents, web page imports, and Word, PDF, and PPT files with "Smart Parsing" enabled.

Taking a Markdown file as an example, when the chunking level is set to 3, the content under each third-level heading (usually marked with ###) becomes one chunk, and each chunk keeps the headings of the first two levels, which better restores the document structure:

Hierarchical chunking mode

3. More Import Methods

Beyond parsing and chunking, the way content is imported into the knowledge base also matters. Convenient import paths let users update the knowledge base more promptly, keeping its content current and accurate.

The platform adds two new triggers: Scheduled Update and Workflow Update. Scheduled Update mainly targets knowledge sources such as websites and online documents. When external pages or documents change, the changes are synced to the knowledge base automatically, which removes the need for periodic manual maintenance and keeps knowledge data under unified management. For example, a knowledge base can automatically sync a company's documentation center, and a documentation Q&A agent built on top of it can be integrated into a website through the "Web Embed" feature:

Scheduled knowledge base update

Workflow Update is a more flexible, proactive way to update. Knowledge is written into the knowledge base dynamically through natural-language conversation, and this can be integrated through channel integrations, API calls, and so on, which offers a new model for operating and maintaining a knowledge base. For example, knowledge base maintainers can update the knowledge base by chatting in the office software they already use, and the model automatically extracts the information and writes it into the knowledge base:

Workflow-based knowledge base update

4. Other Optimization Options

There are a few other optimization ideas to consider at the import stage:

  • LLM data cleaning: Use an LLM to clean the raw corpus, for example removing special symbols and irrelevant information, or converting unstructured documents into structured knowledge such as Q&A pairs or tables to increase information density. Keep in mind that LLM processing can occasionally lose information, so the prompt should be written carefully and the output reviewed.
  • Semantic chunking: Use an LLM to understand the document's meaning and chunk it dynamically, putting semantically self-contained content into one chunk. The downside is higher cost, along with the risk of altering the original content.
  • Image understanding and parsing: For images in pdf/word/ppt documents, besides saving them as links, a multimodal model can recognize the image and turn its key information into a descriptive summary that is stored together with the image link. Retrieval can then find the relevant chunk based on what the image contains.
  • GraphRAG: Uses a knowledge graph as the underlying store, so a graph structure can better store and represent the relationships between pieces of knowledge. It suits scenarios such as medical diagnosis, business insights, and legal consultation, where complex links between knowledge entities need to be established. Building a knowledge graph also costs more than a vector database, so the choice should depend on the actual scenario.

2. Knowledge Base Retrieval

The core goal of retrieval optimization is accuracy: the system needs to truly understand the user's question, retrieve the knowledge base content the model actually needs, and reduce retrieval of irrelevant content.

1. Query Rewriting

In real-world use, users have multi-turn conversations with an agent, and their questions often have an unclear subject, missing key information, or vague meaning. If the latest raw question is used directly for knowledge base retrieval, this contextual information can be lost, which hurts retrieval accuracy.

The solution is query rewriting: using the context to complete, refine, and expand the user's question so that more relevant content is retrieved:

Query rewriting example

As shown in the image, the user first asks "Introduce the LinkAI platform" and then asks "What industry solutions are there". Searching with the latest question alone would likely fail to find accurate results because the subject is missing. Query rewriting completes the question as "What industry solutions does the LinkAI platform have", which resolves the reference. Depending on the scenario, it also breaks the original question down into several more specific sub-questions and retrieves them in batch to get more detailed knowledge:

Query rewriting configuration

The platform also adds its own optimization on top of common query rewriting approaches. It does not replace the original question, nor does it change the question that is finally sent to the LLM. Instead, it uses the rewritten question to run an additional knowledge base search and adds the extra knowledge base entries to the context, which reduces the side effects query rewriting can introduce.

2. Hybrid Search and Reranking

Hybrid search combines vector search and keyword search. Vector search understands the meaning of the user's question better and finds similar text, but for short text, rare terms, and exact matching (for example searching for names, model numbers, or proper nouns), traditional keyword search still has advantages nothing else can replace.

The LinkAI platform now uses hybrid search by default, retrieving both vector search results and keyword search results. It also supports enabling a reranking algorithm, which re-sorts and scores both result sets together and places the knowledge base content most relevant to the user's question at the front of the model context.

Reranking configuration

3. Other Optimization Options

A few other optimization ideas for the retrieval stage:

  • Parent-child chunking: The biggest impact chunking has on accuracy is that it splits content that belongs together, so retrieval cannot get the complete content in one pass. The idea behind parent-child chunking is that when a small chunk is retrieved, the consecutive chunks before and after it are combined into a larger chunk and sent to the model together, which improves context continuity.
  • Hierarchical indexing: Hierarchical indexing addresses the case where the knowledge base holds so much content that retrieval cannot find all the relevant information. The idea is to have an LLM summarize several consecutive chunks and build an extra index layer over those summaries. When a summary index is hit, retrieval continues over the smaller chunks under that summary, which improves recall.
  • Metadata filtering: Metadata are tags that can be attached to knowledge records when the knowledge base is built. At retrieval time, records can be filtered by metadata based on the user's intent or identity, which makes features such as knowledge access control and intent routing possible.

3. Content Generation

In the generation stage, the LLM produces the final answer from the knowledge base context retrieved earlier, together with the user's question, the system prompt, conversation history, and so on. The main optimization work in this step is setting how the model responds when the knowledge base has no match, and improving how the output is presented, to improve user experience and the credibility of the results.

1. Refusing Out-of-Scope Questions

Scenarios that demand highly accurate answers, such as knowledge Q&A in healthcare, finance, and government services, need the agent to refuse questions whose answers are not in the knowledge base. The model must never improvise or make things up.

There are two ways to address this. One is to refine the agent's system prompt, spelling out the rules for refusing out-of-scope questions, and set the model temperature to 0. The platform also offers one-click prompt optimization to quickly generate a high-quality prompt.

The other is to define explicit handling for cases where nothing is retrieved from the knowledge base. The platform provides a configurable no-match strategy, with a choice between letting the model answer freely, replying with fixed text, and handing off to a human. Human handoff comes with detailed rule settings: it can be triggered by keywords or by intent, with configurable reply strategy, notification method, and smart summaries, and handoffs are managed in one place under Conversation Management.

Human handoff configuration

2. Knowledge Base Citations

Showing knowledge base citations makes answers more credible and traceable. It also encourages the model to draw on knowledge base content as its source during generation, which reduces irrelevant output. The platform also supports a confidence threshold for displaying citations: a citation is shown only when the passage's retrieval similarity exceeds a set value.

3. Suggested Question Generation

When Question Suggestions is enabled, the agent generates several recommended questions after each reply, guiding the user to dig deeper and to hit the knowledge base more reliably.

The platform has made a series of optimizations to how suggested questions are generated. First, they are relevant to the conversation history and build on the topic step by step, so the user can gradually go deeper and learn more. Second, the suggested questions are all generated from content already in the knowledge base, so they steer the user's next question toward the knowledge base and improve the quality of user questions at the source.

4. Building an Evaluation System

Anything that cannot be measured cannot be optimized scientifically, and it is hard to compare how much an improvement actually helps. That makes building an evaluation system a key step in RAG optimization. A good evaluation framework points the way for optimization and forms an "iterate -> evaluate -> iterate" data loop.

1. Defining Metrics

The first step is to settle on evaluation metrics for the RAG system, that is, which factors determine how well a RAG system performs. The RAG Triad proposed by LlamaIndex and TruLens is a widely used evaluation framework in the industry:

The core idea is to measure RAG quality through the relationships among the user question (Query), the retrieved context (Context), and the generated answer (Response). Following this idea, we can derive more specific metrics for each pair:

  • Context relevance: Measures the accuracy of the retrieval stage, i.e. how relevant the retrieved content is to the user's question, including metrics such as recall and context precision. Ways to improve it include the parsing, chunking, and hybrid search covered above.
  • Groundedness: Measures whether the final answer stays faithful to the retrieved context, i.e. how much the LLM hallucinates. Remedies include optimizing the system prompt, choosing a stronger model, and configuring refusal strategies.
  • Answer relevance: Measures whether the final answer actually addresses the user's question. Besides system scoring, this metric can also include human evaluation. Ways to improve it include prompt engineering and query rewriting.

2. Agent Evaluation Center

In its RAG optimization work, the LinkAI platform currently uses the metrics above as a guide and has built multiple test sets covering different industries and scenarios. After each iteration of core capabilities, the test sets are run automatically in batch to measure how the RAG system's metrics change.

In upcoming iterations, the platform will open the full evaluation capability to customers through an Agent Evaluation Center. Customers will be able to build their own test sets and run them in batch, with a judge model generating the process metrics, and human annotation and scoring will also be supported. Statistical analysis of the evaluation results makes it easier to tune agent configuration and iterate on knowledge base content, enabling end-to-end optimization of agent deployments.

Wrapping Up

Following the full pipeline of a knowledge base system (import, retrieval, generation, evaluation), this post has covered deeper optimizations for RAG. Upgrading each intermediate stage preserves the original knowledge as completely as possible, improves retrieval accuracy, and produces answers that match the user's question and context. Evaluation metrics then quantify the overall effect, forming a RAG optimization system that keeps improving in a loop.

In real enterprise scenarios, building agents and maintaining knowledge bases is always a long-term process. It starts with a cold-start phase with a small amount of knowledge, moves into an AI-assisted human-machine collaboration phase, and then continues with filling in the knowledge base and tuning the agent configuration based on real user questions and feedback. Only through that process does an agent in production gradually deliver better business results.

Tags: LLM, RAG, Agent, agent

Leave a Comment