AWS Machine Learning Blog detailed a technique for reducing the cost of retrieval-augmented generation (RAG) systems built on Amazon Bedrock. The approach, called query-aware compression, filters retrieved passages against the specific user query before they are sent to the underlying language model, rather than forwarding entire retrieved documents or chunks as-is.
Most RAG systems retrieve a fixed number of document chunks per query and pass all of them into the model's context window, regardless of how much of that text is actually relevant to the question. That inflates the token count per request — and since Bedrock and most hosted LLMs charge per token, it directly inflates the bill. AWS's method scores and trims retrieved content down to the parts that matter for a given query, reducing the tokens sent to the model without requiring a different retrieval index or a different underlying model.
This matters operationally because RAG has become the default architecture for AI-powered support bots, internal knowledge search, and sales enablement assistants at small and mid-sized companies — systems that answer questions by pulling from a company's documents, tickets, or product manuals rather than relying purely on a model's training data. As usage scales from a pilot to full deployment, token costs from oversized context windows are often the single largest line item, exceeding the cost of the model calls themselves.
For a 10-200 person B2B company already running or piloting a RAG-based support or search tool on Bedrock, this technique is directly applicable: it's a pipeline modification, not a re-architecture, and AWS's post includes the mechanics needed to implement it. Companies evaluating vendors for AI support automation should also ask whether the vendor's retrieval pipeline already applies compression or similar filtering — the difference shows up in the monthly invoice, not in the demo.
No pricing figures, latency benchmarks, or adoption numbers were included in AWS's post beyond the architectural description; teams should treat the magnitude of savings as workload-dependent and unconfirmed until tested against their own retrieval corpus.