Struggling with context window limits in long-form RAG pipelines

I’ve been building a legal document analysis tool using LangChain and OpenAI’s GPT-4o, and I’m hitting a wall with context window management. My goal is to process contracts that are often 50-100 pages long, but even with chunking strategies like recursive character splitting, the model tends to lose coherence when I try to synthesize information from chunks that are far apart in the original document. I initially tried simply increasing the chunk size to 2000 tokens, which helped with local context but made the retrieval noisy. I then experimented with a two-stage approach: first using a smaller, cheaper model to summarize each chunk, then feeding those summaries into the final generation step. This improved speed but introduced significant hallucinations because the intermediate summaries sometimes omitted critical legal nuances. Has anyone successfully implemented a hierarchical retrieval system for this specific use case? I’m considering using an embedding model that supports longer context windows natively, but I’m worried about the latency costs. Alternatively, I’ve seen mentions of using a 'map-reduce' pattern where the LLM analyzes each chunk independently and then aggregates the findings. My concern is whether the aggregation step will effectively resolve conflicts between different parts of the contract. I’m open to switching frameworks if there’s a better way to handle this dependency issue. I really need the final output to cite specific page numbers accurately, which the current pipeline struggles with when the relevant clause is split across multiple chunks. Any advice on balancing cost, accuracy, and context retention would be greatly appreciated. I’m currently stuck in a loop of tuning temperature and top_p parameters, which feels like a band-aid solution.