Cross-store product clustering across millions of stores
Shopify groups the same real-world product sold by different merchants under one Universal Product Identifier. Embedding nearest-neighbor search finds plausible pairs across billions of listings, then three scorers of rising cost judge them: a CPU GBDT (L1) over trillions of pairs, a fine-tuned multilingual E5 cross-encoder (L2), and an image-aware reranker (L3) fed from prioritized GPU queues. A bounded union-find splits the graph into single-machine components, and HAC with a learned GBDT merge rule builds high-precision clusters daily, with an LLM judge validated against human labels supplying training data and cached scores reused between builds.
- Published
- 2026-10-20
- Scale
- billions of product listings across millions of stores, trillions of candidate pairs scored by L1, clusters rebuilt daily
- Techniques
- embedding retrieval tiered model cascade bounded union-find graph partitioning hierarchical agglomerative clustering LLM-labeled training data score caching
- Stack
- GBDT Multilingual E5 cross-encoder LLM judge GPUs Vector ANN search