I ran benchmarks yesterday. The indexer handles 50K documents per second but faceted queries slow down significantly above 1M documents.
That matches what I saw in staging. The bottleneck is the aggregation pipeline. We need to pre-compute facet counts instead of calculating them at query time.
Good call. Should we use materialized views or a separate facet cache? I am leaning toward Redis for the cache layer since we already have it in the stack.
Quick note from the infrastructure side: our Redis cluster is at 78% memory. If we add facet caching, we should provision an additional node before going to production.
I will create the provisioning ticket. Tomas, can you estimate the memory footprint for facet caching across our 12M document corpus?
Sounds like a plan. Let us go with Redis facet caching with 30-second batched invalidation. Ryan, can you prototype the invalidation logic this sprint?