Building a Retrieval-Augmented Generation (RAG) system often starts very smoothly during the first few interaction turns. However, as the conversation lengthens, a common phenomenon occurs: the AI begins to "forget" what the user said just a few turns prior.
The Problem: Context Window Limits and History Management
The biggest mistake in basic Local RAG scripts is cramming the entire chat history along with the retrieved chunks into a single prompt. When the token count exceeds the local model's limit, crucial information at the beginning or middle of the conversation gets brutally truncated.
Our Approach
Instead of passing the raw history array, we implement a Context Condensation mechanism.
- Standalone Question Generator: Instead of taking the user's latest query directly to the Vector Store, the system uses an auxiliary LLM to rewrite the question based on the chat history.
- Example: If the user asks "How many FPS can it handle?", the system translates it to "How many FPS can the AI Camera process in low-light conditions?".
- Sliding Window with Summary: The chat history only retains the exact verbatim text of the last 3 turns. Older turns are summarized into a short paragraph and prepended to the prompt to maintain background context.
This architecture helps sustain logical reasoning across multiple consecutive questions without overflowing the memory limits of locally deployed systems.
