This block + 6,000 more — yours with Pro

Chat Reply Quote

151/200
Docs
EngineeringSearch Indexer10 messages
OH
Olivia Hart11:05 AM
Has anyone tested the new search indexer on the production dataset? I am seeing inconsistent results on faceted queries.
RZ
Ryan Zhao11:08 AM
I ran benchmarks yesterday. The indexer handles 50K documents per second but faceted queries slow down significantly above 1M documents.
NP
Replying to Ryan Zhao

I ran benchmarks yesterday. The indexer handles 50K documents per second but faceted queries slow down significantly above 1M documents.

Nadia Petrov11:11 AM
That matches what I saw in staging. The bottleneck is the aggregation pipeline. We need to pre-compute facet counts instead of calculating them at query time.
OH
Replying to Nadia Petrov

That matches what I saw in staging. The bottleneck is the aggregation pipeline. We need to pre-compute facet counts instead of calculating them at query time.

Olivia Hart11:14 AM
Good call. Should we use materialized views or a separate facet cache? I am leaning toward Redis for the cache layer since we already have it in the stack.
RZ
Replying to Olivia Hart

Good call. Should we use materialized views or a separate facet cache? I am leaning toward Redis for the cache layer since we already have it in the stack.

Ryan Zhao11:18 AM
Redis works but keep in mind cache invalidation. Every document write needs to update the facet counts. We could batch the updates on a 30-second interval to avoid write amplification.
TS
Tomas Silva11:22 AM
Quick note from the infrastructure side: our Redis cluster is at 78% memory. If we add facet caching, we should provision an additional node before going to production.
NP
Replying to Tomas Silva

Quick note from the infrastructure side: our Redis cluster is at 78% memory. If we add facet caching, we should provision an additional node before going to production.

Nadia Petrov11:25 AM
I will create the provisioning ticket. Tomas, can you estimate the memory footprint for facet caching across our 12M document corpus?
TS
Replying to Nadia Petrov

I will create the provisioning ticket. Tomas, can you estimate the memory footprint for facet caching across our 12M document corpus?

Tomas Silva11:30 AM
Rough estimate: each facet key-value pair is about 120 bytes. With 200 facets across 50 categories, that is roughly 1.2 GB. Manageable with an extra node.
OH
Olivia Hart11:33 AM
Sounds like a plan. Let us go with Redis facet caching with 30-second batched invalidation. Ryan, can you prototype the invalidation logic this sprint?
RZ
Replying to Olivia Hart

Sounds like a plan. Let us go with Redis facet caching with 30-second batched invalidation. Ryan, can you prototype the invalidation logic this sprint?

Ryan Zhao11:36 AM
On it. I will have a working prototype by Wednesday and we can load test Thursday. Will share the branch once it is ready for review.