Is local LLM inference actually viable for serious production workloads

I’ve been running into some serious bottlenecks trying to migrate our internal documentation Q&A bot from an API-based setup to a fully local solution. The main driver is cost, since we’re processing millions of tokens monthly, but also data privacy concerns from our legal team who are getting nervous about sending client PII to third-party cloud endpoints. I’m currently testing out a quantized version of Llama 3 70B on a single A100 GPU. The latency is acceptable for non-real-time use cases, hovering around 2-3 seconds per token generation, which is fine for our batch processing jobs. However, the quality drop is noticeable. The model struggles significantly with complex multi-step reasoning and often hallucinates citations that don’t exist in our indexed vector database. I tried adjusting the temperature and top-p parameters, but it feels like a brute-force approach rather than a real solution. I’m wondering if anyone else has successfully deployed models in this size range for production-grade RAG systems without relying on hybrid routing to cloud APIs. Are there specific prompt engineering techniques or fine-tuning strategies that help bridge the gap between cloud model performance and local hardware limitations? I’d appreciate any insights on whether it’s worth investing in a second GPU for tensor parallelism or if I should just accept the performance ceiling of a single node.