[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"blog-article-en-reranking-with-cross-encoders":3,"blog-related-e37e9cc5-4c9c-4bbe-a538-257b44c32487-en-busca":58},{"post":4,"html":56,"reading_time_minutes":57},{"id":5,"tenant_id":6,"author":7,"status":11,"published_at":12,"reading_time_minutes":13,"view_count":14,"like_count":15,"featured":16,"created_at":17,"updated_at":12,"translations":18,"tags":35},"e37e9cc5-4c9c-4bbe-a538-257b44c32487","3272d9cc-43d0-4e23-8d75-3db7f042b2b3",{"sub":8,"name":9,"email":10},"554c6643-8b1c-4484-b190-4d9c71d0c275","Rodolfo De Bonis","dev@rodolfodebonis.com.br","published","2026-08-07T17:21:16.673906Z",19,17,0,true,"2026-08-07T17:21:16.610092Z",[19,27],{"id":20,"post_id":5,"tenant_id":6,"lang":21,"slug":22,"title":23,"excerpt":24,"content_md":25,"cover_image_url":26,"created_at":17,"updated_at":17},"0de46201-c9b5-417f-8822-e48e7a79bf10","en","reranking-with-cross-encoders","Reranking with Cross-Encoders: the step that separates good search from excellent search","Hybrid search ranks by consensus, and consensus isn't relevance. A cross-encoder reads the query and the document together and reorders your top 100. Third and final post in the series on modern search.","This post closes out a series. The [first one](https://rodolfodebonis.com.br/en/blog/how-semantic-search-works) covered semantic search, the [second](https://rodolfodebonis.com.br/en/blog/hybrid-search-reciprocal-rank-fusion) covered hybrid search with Reciprocal Rank Fusion. If you landed here cold: BM25 nails the literal and misses the meaning, vector search does the opposite, and RRF merges the two rankings using position alone, with no weights to calibrate.\n \nPost 2 ended on an open problem. The query in that article was `running shoes for hot weather`, and the dry-fit tee finished in second place after fusion. It wasn't what the user wanted, but both retrievers found it plausible, and RRF ranks by consensus of position. Consensus isn't relevance.\n \nThe underlying reason is that up to this point in the pipeline, nobody has read the query and the document together.\n \n## What a bi-encoder never actually does\n \nThe mechanism is worth being precise about, because the difference between the two setups is the whole story.\n \nIn vector search, the embedding model processes the document on its own, at index time, and emits a vector. Months later a query shows up, the model processes the query on its own, emits another vector, and the system compares the two with cosine. That arrangement has a name: **bi-encoder**. Two independent encodes, and one late, cheap meeting between two points in space.\n \nThat's what makes vector search viable at all. The heavy work happens offline, once per document, and all that's left on the request path is a geometric comparison. It's also what makes HNSW possible: you can only build a neighborhood index if the document vector exists before the query arrives.\n \nAnd that's precisely where the ceiling is. The vector for \"Dry-fit tee for running in the heat\" was computed without anyone knowing the question would be about shoes. It has to represent the entire document, for every conceivable query, in 1024 fixed numbers. It's lossy compression, and what gets lost is exactly the interaction: which parts of this document matter *for this question*.\n \nA **cross-encoder** tears that arrangement down. It takes the `(query, document)` pair concatenated into a single sequence and runs one forward pass over both at once. Every query token can attend to every document token, and the other way around, across all layers. The output isn't a vector, it's a number: how well this document answers this query.\n \nThe original implementation, from Nogueira and Cho's 2019 paper, is almost embarrassingly plain. The query goes in as sentence A, the passage as sentence B, the `[CLS]` token vector feeds a single dense layer, and out comes the probability that the passage is relevant. The query gets truncated to 64 tokens, the whole thing to 512. No new architecture. Just BERT reading both things at the same time.\n \n## How big the gain actually is\n \nThat same paper has the number that, in my opinion, is the strongest argument for reranking, and it's strong because it isolates the variable.\n \nOn MS MARCO, the candidate set was held fixed: BM25's top 1000 for each query. BM25 on its own, ranking those same thousand documents, gets MRR@10 of 16.7 on the dev set. A BERT Large reordering the exact same list gets 36.5.\n \nNo new documents entered. No index changed. No embeddings were generated. Retrieval was identical in both cases, and the metric more than doubled purely because something read the pairs.\n \nWorth noting what that number doesn't say. It's a 2019 benchmark, in English, on a dataset of Bing questions with roughly one relevant passage per query. Your domain isn't that. But the asymmetry it exposes between \"retrieving\" and \"ordering\" is structural, and it shows up again in every benchmark since. Elastic, measuring their own model across 21 BEIR datasets, reports an average 40% improvement in ranking quality when reranking BM25 results.\n \n## Why you can't just run this on everything\n \nIf cross-encoders are that much better, the obvious question is why not throw out the index entirely and score every document directly.\n \nThe answer lives in the word \"pair\". A bi-encoder does N document encodes once in its life, then one encode per query. A cross-encoder does one forward pass per `(query, document)` pair, every single time, because the score depends on both. There's nothing to precompute. Change the query and all prior work is garbage.\n \nYou can put numbers on this using the published `sentence-transformers` table, which measures throughput for the MS MARCO cross-encoders on a V100 GPU:\n \n| Model | nDCG@10 (TREC DL 19) | MRR@10 (MS MARCO Dev) | Docs/sec |\n| --- | --- | --- | --- |\n| ms-marco-TinyBERT-L2-v2 | 69.84 | 32.56 | 9000 |\n| ms-marco-MiniLM-L4-v2 | 73.04 | 37.70 | 2500 |\n| ms-marco-MiniLM-L6-v2 | 74.30 | 39.01 | 1800 |\n| ms-marco-MiniLM-L12-v2 | 74.31 | 39.02 | 960 |\n| ms-marco-electra-base | 71.99 | 36.41 | 340 |\n \nTake `MiniLM-L6-v2`, the middle of that list. At 1800 documents per second, scoring 100 candidates costs 56 ms. Scoring a million-document catalog costs **nine and a half minutes per query**, on a dedicated GPU, for one user. The `L12`, which buys you half a point of nDCG, takes 17 minutes.\n \nThis isn't something you scale horizontally until it goes away. It's an order-of-magnitude difference that changes what kind of problem you're solving.\n \nLook at the third column against the first, too. Between `L6` and `L12` the quality is essentially identical and the cost doubles. Between `TinyBERT-L2` and `L6` you buy 4.5 points of nDCG at 5× the time. The diminishing-returns curve is already drawn inside the model family itself, before you've even picked a depth.\n \nHence the architecture everyone converges on: cheap retrieval picks a handful of candidates, and the expensive model only looks at that handful.\n \n![image](https://assets.rodolfodebonis.com.br/blog-articles-images/reranking/arhitecture_en.png)\n \nThe first stage exists so nothing relevant gets left out. The second exists to get the order right. Different goals, which is exactly why different models make sense.\n \n## The tee finally drops\n \nBack to the example from post 2. Five documents, the query `running shoes for hot weather`, and the ranking RRF produced:\n \n| Position | Document |\n| --- | --- |\n| 1 | A: Ventus running shoe, high-ventilation mesh |\n| 2 | C: Dry-fit tee for running in the heat |\n| 3 | D: Lightweight shoe for hot, humid weather |\n| 4 | E: Technical running sock for hot days |\n| 5 | B: Rocha 3 trail running shoe, reinforced upper |\n \nNow the cross-encoder gets all five pairs and returns a score for each. The values below are illustrative, but the ordering is what any decent reranker produces here:\n \n| Document | Score | Final position |\n| --- | --- | --- |\n| A: Ventus, mesh | 0.94 | 1st |\n| D: Lightweight, hot weather | 0.91 | 2nd |\n| B: Rocha 3 trail | 0.38 | 3rd |\n| C: Dry-fit tee | 0.07 | 4th |\n| E: Technical sock | 0.04 | 5th |\n \nThe tee dropped from second to fourth, and the reason is mundane once you look at the mechanism: the model processed the query token \"shoes\" in the same forward pass as the document token \"tee\". It isn't comparing two summaries of meaning, it's reading a question about footwear alongside a document about apparel and concluding the category is wrong, even though \"running\" and \"heat\" both match literally.\n \n`B` climbed to third on the complementary logic. It's a running shoe, so it hits the primary intent, but \"reinforced upper\" is the opposite of ventilation, and the model penalizes without discarding. Neither earlier stage can draw that distinction, because neither one read both texts at once.\n \nNotice the gap between 0.91 and 0.38. That isn't scale noise the way RRF scores are. That's the model separating two populations.\n \n## Picking a model\n \nThe ecosystem splits three ways, and the choice is almost never about ranking quality.\n \n**Managed API.** Cohere Rerank is the reference here. The current line is `rerank-v4.0-pro` and `rerank-v4.0-fast`, both multilingual across 100+ languages, with a 32k-token context window and support for semi-structured JSON documents. `rerank-v3.5` is still around with 4k. Voyage and Jina play the same game. You add one HTTP call and you're done: no GPU, no deployment, no reindexing.\n \nThe planning detail is the billing unit. A \"search unit\" is one query with up to 100 documents, and long documents get chunked automatically, with each chunk counting as a separate document. Meaning your 100 candidates can turn into several search units if the texts are large. The AWS Marketplace listing for Rerank v3.5 (Bedrock edition) prices it at $0.002 per search unit. That's roughly $200 for a month with 100k reranked searches, and $2,000 for a month with a million. Prices change, so check before you commit, but the shape holds: API reranking is cheap per search and expensive per traffic.\n \n**Self-hosted open source.** The `cross-encoder/ms-marco-*` family from sentence-transformers is the classic starting point, and the table above already shows the whole trade-off. One warning that matters more than it looks: those models are trained on MS MARCO, which is English. If your corpus is Portuguese, Spanish, or anything else, they will disappoint you. For multilingual work the answer is `BAAI/bge-reranker-v2-m3`, built on top of bge-m3, with a 512-token window and a score you map to 0..1 with a sigmoid. The BGE docs give the obvious and correct advice: test on your real use case and pick on speed/quality balance, not on their table.\n \n**Built into the engine.** Elasticsearch exposes reranking through the `text_similarity_reranker` retriever, which can point at an external inference endpoint (Cohere, for instance) or at Elastic's own model, Elastic Rerank, a 184M-parameter DeBERTa v3. Read the limitations before you get excited: English only, 512-token window, available from Stack 8.17, requires an appropriate subscription level, and still flagged as technical preview. Elastic writes in their own docs that the preview version may be cost prohibitive for high query rates with low-latency requirements, and recommends staying under top-30 for CPU inference.\n \n**The middle option.** ColBERT occupies its own niche. Instead of one score per pair, it stores a vector per document token and performs a late interaction between query tokens and document tokens at query time. The original paper measures effectiveness competitive with the BERT models of the day while running two orders of magnitude faster and using four orders of magnitude fewer FLOPs per query. The price is the index: storing a vector per token inflates storage aggressively, and ColBERTv2 exists largely to shrink that by 5 to 8×. If latency is your constraint and you have disk to spare, it's worth a look.\n \n## Depth is the parameter that matters\n \nHere's the most expensive mistake in this stage, and it runs against intuition.\n \nThe default belief is that reranking more candidates can only help, since a good model would have more material to work with. Elastic measured this systematically, sweeping reranking depth across several models and BEIR datasets, and found three patterns:\n \n- **Fast rise, then saturation**, in 72.6% of cases. What everyone expects.\n- **Rise to a peak, then decay**, in 20.2% of cases.\n- **Steady decay with any amount of reranking**, in 7.1% of cases. The reranker makes BM25 worse at every depth tested.\nThe explanation is mechanical and satisfying. nDCG@10 only moves when the reranker promotes something from below into the top 10. If the promoted document is relevant, the metric goes up. If it's a false positive, it evicts a relevant document and the metric goes down. As you go deeper, the density of relevant documents collapses and the density of irrelevant ones grows. There's a depth at which the model's false-positive rate starts to dominate, and past it you're paying for inference to degrade your own ranking.\n \nThe paper *Drowning in Documents* gets to the same place from another direction, and notes something uncomfortable: rerankers frequently assign high scores to documents with no lexical or semantic overlap with the query at all.\n \nThree practical consequences, in the order you'll need them:\n \nFirst, the number. Applying the rule of picking the depth that reaches 90% of the maximum gain, Elastic landed on roughly **top 100** over BM25, at a third of the compute cost of chasing the maximum. When cost is tight they recommend **top 30** with their model, and still measured over 40% uplift in nDCG@10 on the QA portion of their benchmark.\n \nSecond, which way to tune. The better your first stage, the fewer candidates you need to rerank, because recall saturates earlier. And the better your reranker, the deeper it pays to go, because it makes fewer mistakes on the way down. You already have hybrid search with RRF, so your first stage is good: start shallow.\n \nThird, and this one surprised me: under a latency constraint, reranking deep with a small model tends to beat reranking shallow with a big one. In Elastic's benchmark, `MiniLM-L12-v2` processing 80 candidates beat stronger models capped at 10 or 20 for the same time budget. The relationship flips as you relax the constraint.\n \nOn latency, one concrete number to calibrate against: measuring on two T4 GPUs, Elastic reports 0.0869 seconds to score 10 pairs with Elastic Rerank on HotpotQA. Linearized, top-30 lands around 0.26 s and top-100 around 0.87 s. Those aren't the 50 ms that get thrown around in posts on the topic. Measure on your hardware, with your documents, before you promise an SLA.\n \n## Scores mean something again\n \nOne limitation I raised in post 2 gets resolved here, and it's worth closing the loop.\n \nRRF scores mean nothing. They're crammed into a tiny band, they aren't comparable across queries, and so you can't threshold on them. You can't say \"if the best result is bad, show the empty state\".\n \nA cross-encoder score is a different kind of thing. It's a relevance estimate for that pair, produced by a model trained for exactly that, and it doesn't depend on anyone's position. The sentence-transformers models return logits that sit roughly between -10 and 10, and you apply a sigmoid to get something in 0..1. `bge-reranker-v2-m3` works the same way. Cohere returns a normalized `relevance_score`. Elasticsearch exposes `min_score` right on the retriever precisely so you can cut.\n \nThe trap is confusing \"meaningful\" with \"calibrated\". The scale is specific to the model and the domain. A 0.6 from Cohere is not a 0.6 from BGE, and a 0.6 from BGE on your legal corpus is not the same 0.6 on your support FAQ. The threshold has to come from your data. But it exists, and that's a genuinely new capability in the pipeline.\n \n## What this looks like in code\n \nSelf-hosted, with `sentence-transformers`. The point of the snippet is that reranking is a pure function over the candidate list, with no state and no index:\n \n```python\nfrom sentence_transformers import CrossEncoder\nimport torch\n \n# Sigmoid pins scores to 0..1. Without it you get raw logits.\nreranker = CrossEncoder(\n    \"BAAI/bge-reranker-v2-m3\",   # multilingual; for English, ms-marco-MiniLM-L6-v2\n    activation_fn=torch.nn.Sigmoid(),\n    max_length=512,\n)\n \ndef rerank(query: str, candidates: list[dict], top_k: int = 10) -> list[dict]:\n    \"\"\"Takes the output of RRF, returns the reordered top_k.\"\"\"\n    if not candidates:\n        return []\n \n    pairs = [(query, doc[\"text\"]) for doc in candidates]\n    scores = reranker.predict(pairs, batch_size=32)\n \n    for doc, score in zip(candidates, scores):\n        doc[\"rerank_score\"] = float(score)\n \n    ranked = sorted(candidates, key=lambda d: d[\"rerank_score\"], reverse=True)\n    return ranked[:top_k]\n```\n \nTwo decisions are worth a comment. `batch_size` is what determines whether you saturate the GPU or waste it, and it's the first knob to turn when latency won't close. And `max_length=512` isn't cosmetic: anything longer gets truncated, so the model scores the beginning of the text and ignores the rest. If your documents are long, you need to decide deliberately which chunk gets scored instead of letting the tokenizer decide for you.\n \nSame pipeline through an API, with Cohere:\n \n```python\nimport cohere\n \nco = cohere.ClientV2(api_key=\"...\")\n \nresponse = co.rerank(\n    model=\"rerank-v4.0-fast\",\n    query=\"running shoes for hot weather\",\n    documents=[doc[\"text\"] for doc in candidates],\n    top_n=10,\n)\n \n# results carry index (position in the original list) and relevance_score\ntop10 = [candidates[r.index] for r in response.results]\n```\n \nAnd inside Elasticsearch, chained onto the RRF retriever from the previous post:\n \n```json\nGET /products/_search\n{\n  \"retriever\": {\n    \"text_similarity_reranker\": {\n      \"retriever\": {\n        \"rrf\": {\n          \"retrievers\": [\n            { \"standard\": { \"query\": { \"multi_match\": {\n                \"query\": \"running shoes for hot weather\",\n                \"fields\": [\"title\", \"description\"]\n            }}}},\n            { \"knn\": {\n                \"field\": \"embedding\",\n                \"query_vector\": [0.21, -0.05, \"...\"],\n                \"k\": 100,\n                \"num_candidates\": 300\n            }}\n          ],\n          \"rank_window_size\": 100\n        }\n      },\n      \"field\": \"description\",\n      \"inference_id\": \"my-reranker\",\n      \"inference_text\": \"running shoes for hot weather\",\n      \"rank_window_size\": 100,\n      \"min_score\": 0.4\n    }\n  },\n  \"size\": 10\n}\n```\n \nNote there are two `rank_window_size` values in that query and they mean different things. The inner one is how many candidates fusion considers. The outer one is how many of those go to the model. Matching them is the sane default; setting the outer one higher does nothing, because no candidate exists beyond the fusion window.\n \n## Where reranking doesn't help\n \nA few cases where the honest answer is to leave it off.\n \n**When first-stage recall is the problem.** Reranking retrieves nothing. If the right document isn't among the 100 that arrived, no model is going to invent it. Before investing in the expensive stage, measure Recall@100 on your hybrid search. If it's low, the problem is your embedding model, what you indexed, or your fusion window, and reranking will only do a nicer job of ordering a bad set.\n \n**When value per search is low and volume is high.** The classic long-tail ecommerce situation. Millions of searches, thin margin per transaction, and a relevance gain that may not cover the GPU bill or the API bill. This is a business calculation, not an engineering one.\n \n**When the latency SLA is tight.** If your p95 budget is 100 ms and search already eats 40, you don't have room for top-100. You have room for top-20 with a small model, and maybe not even that.\n \n**When the query is an identifier.** `SKU-A4729` doesn't need semantic understanding, it needs an exact match. Detect the pattern before the pipeline and route around it, same as with hybrid search.\n \n**When the corpus is too homogeneous.** If your 100 candidates are near-identical to each other, there's no correct order to discover and the model will just break ties on noise.\n \n## What to measure, before and after\n \nYou already built the most important piece back in post 2: that set of 50 to 200 real queries with the relevant documents marked by hand. It's what decides this.\n \nMeasure nDCG@10 and MRR for hybrid alone, then hybrid plus reranking, sweeping depth at 10, 20, 30, 50, and 100. What you're looking for is the shape of the curve, not one number. If it saturates at 30, going to 100 is money on fire. If it drops after 50, you just learned the model doesn't suit your domain, and you learned it offline, which beats learning it in production.\n \nAlongside that, not afterwards, record the p95 delta and the cost per query. A relevance gain without a latency number next to it isn't a result, it's half of one.\n \nThen comes the production A/B, because offline gains from reranking almost always look great and users don't always notice. The metrics that answer this are CTR on the top positions, query reformulation rate, and conversion. If top-3 CTR goes up and reformulation goes down, reranking is doing real work. If nothing moves, you paid for precision nobody was waiting on.\n \n## Closing the series\n \nFive things I wish I'd known before starting on any of this:\n \nBM25 isn't legacy. It's the strong baseline you'll spend the rest of the project trying to beat, and on some tasks you won't.\n \nEmbeddings turn text into geometry, and everything else follows from that change of representation. But the vector is lossy compression, and the losses show up in the most annoying place possible: identifiers, negations, short queries.\n \nRRF is the best return on effort in the whole list. No calibration, no training, consistently better than either side alone.\n \nReranking is the only stage that reads both texts together, which is why it wins where the others structurally can't. It's also the only one that precomputes nothing, which is why it runs over dozens of candidates and never over the catalog.\n \nAnd the thing that ties it together: none of this gets decided by reading benchmarks. It gets decided with an evaluation set from your own corpus, which takes half a day of tedious work to build and then answers every question that comes after.\n \nThis series covered what I wish I'd known before starting with semantic search. If you're building something similar, tell me what the context is. I'd love to know.\n \n## References\n \n- Nogueira, R., Cho, K. **Passage Re-ranking with BERT**. arXiv:1901.04085, 2019. [Paper](https://arxiv.org/abs/1901.04085)\n- Khattab, O., Zaharia, M. **ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT**. SIGIR 2020. [Paper](https://arxiv.org/abs/2004.12832)\n- Santhanam, K. et al. **ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction**. arXiv:2112.01488\n- Jacob, M. et al. **Drowning in Documents: Consequences of Scaling Reranker Inference**. arXiv:2411.11767\n- Sentence Transformers, [Pretrained Cross-Encoder models](https://sbert.net/docs/cross_encoder/pretrained_models.html) and [Retrieve & Re-Rank](https://sbert.net/examples/sentence_transformer/applications/retrieve_rerank/README.html)\n- Elastic, docs for [Elastic Rerank](https://www.elastic.co/docs/explore-analyze/machine-learning/nlp/ml-nlp-rerank) and the [text_similarity_reranker retriever](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/retrievers/text-similarity-reranker-retriever)\n- Elastic Search Labs, [Exploring depth in a retrieve-and-rerank pipeline](https://www.elastic.co/search-labs/blog/elastic-semantic-reranker-part-3)\n- Cohere, [Rerank overview](https://docs.cohere.com/docs/rerank-overview), [best practices](https://docs.cohere.com/docs/reranking-best-practices), and the [Rerank 4.0 announcement](https://docs.cohere.com/changelog/rerank-v4.0)\n- [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) and the [BGE Reranker docs](https://bge-model.com/bge/bge_reranker_v2.html)\n---\n \n*Third and final post in the series on modern search. The earlier ones are [Semantic Search](https://rodolfodebonis.com.br/en/blog/how-semantic-search-works) and [Hybrid Search with RRF](https://rodolfodebonis.com.br/en/blog/hybrid-search-reciprocal-rank-fusion).*","https://assets.rodolfodebonis.com.br/blog-covers/bf615840-922f-43d7-bd92-c47130cb9330.png",{"id":28,"post_id":5,"tenant_id":6,"lang":29,"slug":30,"title":31,"excerpt":32,"content_md":33,"cover_image_url":34,"created_at":17,"updated_at":17},"9e8b4420-ae3a-49a2-83d4-68de81aaeb0a","pt-BR","reranking-com-cross-encoder","Reranking com cross-encoder: a cereja que separa busca boa de busca excelente","A busca híbrida ordena por consenso, e consenso não é relevância. Um cross-encoder lê query e documento juntos e reordena o top 100. Terceiro e último artigo da série sobre busca moderna.","Esse post fecha uma série. O [primeiro](https://rodolfodebonis.com.br/blog/como-funciona-busca-semantica) tratou de busca semântica, o [segundo](https://rodolfodebonis.com.br/blog/busca-hibrida-com-rrf) de busca híbrida com Reciprocal Rank Fusion. Se você caiu aqui direto, o resumo é que BM25 acerta o literal e erra o significado, a busca vetorial faz o inverso, e o RRF junta os dois rankings usando só posição, sem calibrar peso nenhum.\n \nO post 2 terminou com um problema em aberto. No exemplo daquele artigo, a query era `tênis pra correr no calor` e a camiseta dry-fit ficou em segundo lugar depois da fusão. Ela não era o que o usuário queria, mas as duas buscas acharam ela razoável, e o RRF ordena por consenso de posição. Consenso não é relevância.\n \nO motivo de fundo é que, até esse ponto do pipeline, ninguém leu a query e o documento juntos.\n \n## O que um bi-encoder nunca chega a fazer\n \nVale ser preciso sobre o mecanismo, porque a diferença entre os dois arranjos é a coisa toda.\n \nNa busca vetorial, o modelo de embedding processa o documento sozinho, no momento da indexação, e cospe um vetor. Meses depois chega uma query, o modelo processa a query sozinha, cospe outro vetor, e o sistema compara os dois com cosseno. Esse arranjo tem nome: **bi-encoder**. Dois encodes independentes, um encontro tardio e barato entre dois pontos no espaço.\n \nIsso é o que torna a busca vetorial viável. O trabalho pesado acontece offline, uma vez por documento, e sobra apenas uma comparação geométrica no caminho da requisição. É também o que torna o HNSW possível: você só consegue construir um índice de vizinhança se o vetor do documento existir antes da query chegar.\n \nE é exatamente aí que mora a limitação. O vetor de \"Camiseta dry-fit para correr no calor\" foi calculado sem que ninguém soubesse que a pergunta seria sobre tênis. Ele precisa representar o documento inteiro, para toda query concebível, em 1024 números fixos. É uma compressão com perda, e a informação que se perde é justamente a interação: quais partes desse documento importam *para essa pergunta*.\n \nUm **cross-encoder** desfaz esse arranjo. Ele recebe o par `(query, documento)` concatenado numa única sequência e roda um forward pass sobre os dois ao mesmo tempo. Cada token da query pode atender a cada token do documento, e vice-versa, em todas as camadas. A saída não é vetor, é um número: o quanto esse documento responde essa query.\n \nA implementação original, no paper de Nogueira e Cho de 2019, é quase constrangedoramente direta. Query entra como sentença A, passagem entra como sentença B, o vetor do token `[CLS]` vai para uma única camada densa, e o que sai é a probabilidade da passagem ser relevante. A query é truncada em 64 tokens, o conjunto todo em 512. Nenhuma arquitetura nova. Só BERT lendo as duas coisas juntas.\n \n## O tamanho do ganho\n \nEsse mesmo paper tem o número que, na minha opinião, é o argumento mais forte a favor de reranking, e ele é forte porque isola a variável.\n \nNo MS MARCO, o conjunto de candidatos foi fixo: o top 1000 do BM25 para cada query. O BM25 sozinho, ordenando esses mesmos mil documentos, entrega MRR@10 de 16.7 no dev set. Um BERT Large reordenando exatamente a mesma lista entrega 36.5.\n \nNenhum documento novo entrou. Nenhum índice mudou. Nenhum embedding foi gerado. A recuperação era idêntica nos dois casos, e a métrica mais que dobrou só porque alguém leu os pares.\n \nVale registrar o que esse número não diz. É um benchmark de 2019, em inglês, num dataset de perguntas do Bing, com uma passagem relevante por query em média. Seu domínio não é esse. Mas a assimetria entre \"recuperar\" e \"ordenar\" que ele expõe é estrutural, e ela aparece de novo em todo benchmark posterior. A Elastic, medindo o próprio modelo em 21 datasets do BEIR, reporta em média 40% de melhoria em qualidade de ranking ao reranquear resultados de BM25.\n \n## Por que você não pode simplesmente usar isso em tudo\n \nSe cross-encoder é tão melhor, a pergunta óbvia é por que não jogar fora o índice inteiro e pontuar todos os documentos direto.\n \nA resposta está na palavra \"par\". O bi-encoder faz N encodes de documento uma vez na vida e depois faz um encode por query. O cross-encoder faz um forward pass por par `(query, documento)`, toda vez, porque o score depende dos dois. Não existe nada para pré-computar. Trocou a query, todo o trabalho anterior virou lixo.\n \nDá pra colocar número nisso com a tabela publicada do `sentence-transformers`, que mede throughput dos cross-encoders treinados em MS MARCO numa GPU V100:\n \n| Modelo | nDCG@10 (TREC DL 19) | MRR@10 (MS MARCO Dev) | Docs/s |\n| --- | --- | --- | --- |\n| ms-marco-TinyBERT-L2-v2 | 69.84 | 32.56 | 9000 |\n| ms-marco-MiniLM-L4-v2 | 73.04 | 37.70 | 2500 |\n| ms-marco-MiniLM-L6-v2 | 74.30 | 39.01 | 1800 |\n| ms-marco-MiniLM-L12-v2 | 74.31 | 39.02 | 960 |\n| ms-marco-electra-base | 71.99 | 36.41 | 340 |\n \nPegue o `MiniLM-L6-v2`, que é o meio de campo dessa lista. A 1800 documentos por segundo, pontuar 100 candidatos custa 56 ms. Pontuar um catálogo de um milhão de documentos custa **9 minutos e meio por query**, numa GPU dedicada, para um usuário. O `L12`, que entrega meio ponto a mais de nDCG, leva 17 minutos.\n \nNão é uma questão de escalar horizontalmente até resolver. É uma diferença de ordem de grandeza que muda a categoria do problema.\n \nRepare também na terceira coluna comparada com a primeira. Entre o `L6` e o `L12` a qualidade é praticamente idêntica e o custo dobra. Entre o `TinyBERT-L2` e o `L6` você ganha 4.5 pontos de nDCG pagando 5× mais tempo. A curva de retorno decrescente já está desenhada dentro da própria família de modelos, antes mesmo de você escolher a profundidade.\n \nDaí a arquitetura que todo mundo converge: a recuperação barata seleciona um punhado de candidatos, e o modelo caro só olha esse punhado.\n \n![image](https://assets.rodolfodebonis.com.br/blog-articles-images/reranking/arhitecture_pt.png)\n \nO primeiro estágio existe pra não deixar nada relevante de fora. O segundo existe pra acertar a ordem. São objetivos diferentes, e é por isso que faz sentido usar modelos diferentes.\n \n## A camiseta finalmente cai\n \nVoltando ao exemplo do post 2. Cinco documentos, a query `tênis pra correr no calor`, e o ranking que o RRF produziu:\n \n| Posição | Documento |\n| --- | --- |\n| 1 | A: Tênis de corrida Ventus, mesh com ventilação alta |\n| 2 | C: Camiseta dry-fit para correr no calor |\n| 3 | D: Tênis leve para clima quente e úmido |\n| 4 | E: Meia técnica de corrida para dias quentes |\n| 5 | B: Tênis de corrida trail Rocha 3, cabedal reforçado |\n \nAgora o cross-encoder recebe os cinco pares e devolve um score por par. Os valores abaixo são ilustrativos, mas a ordem é o que qualquer reranker decente produz aqui:\n \n| Documento | Score | Posição final |\n| --- | --- | --- |\n| A: Tênis Ventus, mesh | 0.94 | 1º |\n| D: Tênis leve, clima quente | 0.91 | 2º |\n| B: Tênis trail Rocha 3 | 0.38 | 3º |\n| C: Camiseta dry-fit | 0.07 | 4º |\n| E: Meia técnica | 0.04 | 5º |\n \nA camiseta desabou da segunda posição para a quarta, e o motivo é banal quando você olha o mecanismo: o modelo processou o token \"tênis\" da query no mesmo forward pass que o token \"camiseta\" do documento. Ele não está comparando dois resumos de significado, ele está lendo uma pergunta sobre calçado e um documento sobre vestuário e concluindo que a categoria está errada, apesar de \"correr\" e \"calor\" baterem literalmente.\n \nO `B` subiu para terceiro por um raciocínio complementar. É um tênis de corrida, então acerta a intenção principal, mas \"cabedal reforçado\" é o oposto de ventilação, e o modelo penaliza sem descartar. Nenhum dos dois estágios anteriores tem como fazer essa distinção, porque nenhum dos dois leu as duas coisas ao mesmo tempo.\n \nRepare no salto entre 0.91 e 0.38. Não é ruído de escala como nos scores do RRF. É o modelo separando duas populações.\n \n## Escolher o modelo\n \nO ecossistema se divide em três caminhos, e a escolha entre eles quase nunca é sobre qualidade de ranking.\n \n**API gerenciada.** O Cohere Rerank é a referência da categoria. A linha atual são o `rerank-v4.0-pro` e o `rerank-v4.0-fast`, ambos multilíngues com mais de 100 idiomas, 32 mil tokens de contexto, e suporte a documentos semiestruturados em JSON. O `rerank-v3.5` continua disponível com 4 mil tokens. Voyage e Jina jogam no mesmo campo. Você adiciona uma chamada HTTP e pronto, sem GPU, sem deploy, sem reindexação.\n \nO detalhe de planejamento é a unidade de cobrança. Uma \"search unit\" é uma query com até 100 documentos, e documentos longos são fatiados automaticamente, com cada pedaço contando como um documento separado. Ou seja, seus 100 candidatos podem virar várias search units se os textos forem grandes. Na listagem do Rerank v3.5 no AWS Marketplace (edição Bedrock) o preço é US$ 0,002 por search unit. Isso dá cerca de US$ 200 num mês com 100 mil buscas reranqueadas, e US$ 2 mil num mês com um milhão. Preço muda, então confira antes de fechar conta, mas a ordem de grandeza é essa: reranking em API é barato por busca e caro por tráfego.\n \n**Open-source auto-hospedado.** A família `cross-encoder/ms-marco-*` do sentence-transformers é o ponto de partida clássico, e a tabela lá em cima já mostra o trade-off inteiro. Um alerta que importa muito pra quem escreve em português: esses modelos são treinados em MS MARCO, que é inglês. Se seu corpus é PT-BR, eles vão te decepcionar. Para multilíngue, o caminho é o `BAAI/bge-reranker-v2-m3`, construído em cima do bge-m3, com 512 tokens de janela e score que vira 0 a 1 com uma sigmoide. A própria documentação do BGE recomenda o óbvio e o correto: teste no seu caso real e escolha pelo equilíbrio velocidade/qualidade, não pela tabela deles.\n \n**Embutido no motor.** O Elasticsearch expõe reranking pelo retriever `text_similarity_reranker`, que pode apontar para um endpoint de inference externo (Cohere, por exemplo) ou para o modelo próprio da Elastic, o Elastic Rerank, um DeBERTa v3 de 184 milhões de parâmetros. Vale ler as limitações antes de se animar: só inglês, janela de 512 tokens, disponível a partir do Stack 8.17, exige nível de assinatura adequado, e ainda está marcado como technical preview. A própria Elastic escreve na documentação que a versão preview pode ser proibitivamente cara para alto volume de queries com requisito de baixa latência, e recomenda não passar de top-30 quando a inferência é em CPU.\n \n**A opção do meio.** O ColBERT ocupa um lugar próprio. Em vez de um score por par, ele guarda um vetor por token do documento e faz uma interação tardia entre os tokens da query e os do documento no momento da busca. O paper original mede efetividade competitiva com os modelos BERT da época executando duas ordens de grandeza mais rápido e com quatro ordens de grandeza menos FLOPs por query. O preço é o índice: guardar um vetor por token infla o armazenamento de forma agressiva, e o ColBERTv2 nasceu justamente pra reduzir isso em 5 a 8 vezes. Se seu problema é latência e você tem disco sobrando, vale olhar.\n \n## A profundidade é o parâmetro que importa\n \nAqui está o erro mais caro dessa etapa, e ele é contraintuitivo.\n \nA crença padrão é que reranquear mais candidatos só pode melhorar, porque o modelo bom teria mais material para trabalhar. A Elastic mediu isso de forma sistemática, variando profundidade de reranking em vários modelos e datasets do BEIR, e encontrou três padrões:\n \n- **Subida rápida e depois saturação**, em 72.6% dos casos. É o comportamento que todo mundo espera.\n- **Subida até um pico e depois queda**, em 20.2% dos casos.\n- **Queda contínua com qualquer reranking**, em 7.1% dos casos. O reranker piora o resultado do BM25 em toda profundidade testada.\nA explicação é mecânica e boa. O nDCG@10 só muda quando o reranker promove alguém de baixo para o top 10. Se o promovido é relevante, a métrica sobe. Se é um falso positivo, ele expulsa um documento relevante e a métrica cai. Conforme você desce na lista, a densidade de documentos relevantes despenca e a de irrelevantes cresce. Existe uma profundidade em que a taxa de falso positivo do modelo passa a dominar, e depois dela você está pagando inferência para piorar o próprio ranking.\n \nO paper *Drowning in Documents* chega ao mesmo lugar por outro caminho, e observa algo desconfortável: rerankers frequentemente atribuem score alto a documentos sem nenhuma sobreposição léxica ou semântica com a query.\n \nAs três consequências práticas, na ordem em que você vai precisar delas:\n \nPrimeiro, o número. Aplicando a regra de pegar a profundidade que atinge 90% do ganho máximo, a Elastic chegou a algo em torno de **top 100** em cima de BM25, com um terço do custo computacional de ir até o máximo. Quando custo aperta, eles recomendam **top 30** com o modelo deles, e mesmo assim mediram uplift acima de 40% em nDCG@10 na parte de QA do benchmark.\n \nSegundo, a direção do ajuste. Quanto melhor o primeiro estágio, menos candidatos você precisa reranquear, porque o recall satura antes. E quanto melhor o reranker, mais fundo compensa ir, porque ele erra menos ao descer. Como você já tem híbrida com RRF, seu primeiro estágio é bom, então comece raso.\n \nTerceiro, e esse me surpreendeu: sob restrição de latência, reranquear fundo com um modelo pequeno costuma ganhar de reranquear raso com um modelo grande. No benchmark da Elastic, o `MiniLM-L12-v2` processando 80 candidatos bateu modelos mais fortes limitados a 10 ou 20 pelo mesmo orçamento de tempo. A relação se inverte conforme você afrouxa a restrição.\n \nSobre latência, um número concreto para calibrar expectativa: medindo em duas GPUs T4, a Elastic reporta 0,0869 segundo para pontuar 10 pares com o Elastic Rerank no HotpotQA. Linearizando, top-30 sai em torno de 0,26 s e top-100 em torno de 0,87 s. Não são os 50 ms que costumam circular em posts sobre o assunto. Meça no seu hardware, com seus documentos, antes de prometer SLA.\n \n## O score volta a significar alguma coisa\n \nUma limitação que eu levantei no post 2 se resolve aqui, e vale fechar o ciclo.\n \nO score do RRF não significa nada. Ele vive espremido numa faixa minúscula, não é comparável entre queries, e por isso não dá pra usar em threshold. Você não consegue dizer \"se o melhor resultado for ruim, mostro a tela de nada encontrado\".\n \nO score de um cross-encoder é diferente em natureza. Ele é uma estimativa de relevância daquele par, produzida por um modelo treinado com esse objetivo, e não depende da posição de nada. Os modelos do sentence-transformers devolvem logits que ficam grosso modo entre -10 e 10, e você aplica uma sigmoide pra ter algo entre 0 e 1. O `bge-reranker-v2-m3` funciona igual. O Cohere devolve `relevance_score` já normalizado. O Elasticsearch expõe `min_score` no próprio retriever justamente pra você cortar.\n \nO cuidado é não confundir \"tem significado\" com \"é calibrado\". A escala é específica do modelo e do domínio. Um 0.6 do Cohere não é um 0.6 do BGE, e o 0.6 do BGE no seu corpus jurídico não é o mesmo 0.6 no seu FAQ de suporte. O threshold precisa sair dos seus dados. Mas ele existe, e isso é uma capacidade nova no pipeline.\n \n## Como isso fica no código\n \nAuto-hospedado, com `sentence-transformers`. O ponto do trecho é que reranking é uma função pura sobre a lista de candidatos, sem estado e sem índice:\n \n```python\nfrom sentence_transformers import CrossEncoder\nimport torch\n \n# Sigmoid força o score pra faixa 0..1. Sem isso, o retorno é logit cru.\nreranker = CrossEncoder(\n    \"BAAI/bge-reranker-v2-m3\",   # multilíngue; para inglês, ms-marco-MiniLM-L6-v2\n    activation_fn=torch.nn.Sigmoid(),\n    max_length=512,\n)\n \ndef rerank(query: str, candidatos: list[dict], top_k: int = 10) -> list[dict]:\n    \"\"\"Recebe a saída do RRF, devolve o top_k reordenado.\"\"\"\n    if not candidatos:\n        return []\n \n    pares = [(query, doc[\"texto\"]) for doc in candidatos]\n    scores = reranker.predict(pares, batch_size=32)\n \n    for doc, score in zip(candidatos, scores):\n        doc[\"rerank_score\"] = float(score)\n \n    ordenado = sorted(candidatos, key=lambda d: d[\"rerank_score\"], reverse=True)\n    return ordenado[:top_k]\n```\n \nDuas decisões valem comentário. O `batch_size` é o que controla se você satura a GPU ou desperdiça ela, e é o primeiro botão a mexer quando a latência não fecha. E `max_length=512` não é cosmético: documento maior que isso é truncado, então o modelo pontua o começo do texto e ignora o resto. Se seus documentos são longos, você precisa decidir conscientemente qual pedaço vai ser pontuado, em vez de deixar o tokenizer decidir por você.\n \nVia API, o mesmo pipeline com Cohere:\n \n```python\nimport cohere\n \nco = cohere.ClientV2(api_key=\"...\")\n \nresposta = co.rerank(\n    model=\"rerank-v4.0-fast\",\n    query=\"tênis pra correr no calor\",\n    documents=[doc[\"texto\"] for doc in candidatos],\n    top_n=10,\n)\n \n# results traz index (posição na lista original) e relevance_score\ntop10 = [candidatos[r.index] for r in resposta.results]\n```\n \nE dentro do Elasticsearch, encadeando no retriever RRF do post anterior:\n \n```json\nGET /produtos/_search\n{\n  \"retriever\": {\n    \"text_similarity_reranker\": {\n      \"retriever\": {\n        \"rrf\": {\n          \"retrievers\": [\n            { \"standard\": { \"query\": { \"multi_match\": {\n                \"query\": \"tênis pra correr no calor\",\n                \"fields\": [\"titulo\", \"descricao\"]\n            }}}},\n            { \"knn\": {\n                \"field\": \"embedding\",\n                \"query_vector\": [0.21, -0.05, \"...\"],\n                \"k\": 100,\n                \"num_candidates\": 300\n            }}\n          ],\n          \"rank_window_size\": 100\n        }\n      },\n      \"field\": \"descricao\",\n      \"inference_id\": \"meu-reranker\",\n      \"inference_text\": \"tênis pra correr no calor\",\n      \"rank_window_size\": 100,\n      \"min_score\": 0.4\n    }\n  },\n  \"size\": 10\n}\n```\n \nRepare que existem dois `rank_window_size` na query, e eles são coisas diferentes. O de dentro é quantos candidatos a fusão considera. O de fora é quantos desses vão para o modelo. Igualar os dois é o default razoável; deixar o de fora maior não faz nada, porque não existe candidato além da janela de fusão.\n \n## Onde reranking não resolve\n \nAlguns casos em que a resposta honesta é não ligar isso.\n \n**Quando o recall do primeiro estágio é o problema.** Reranking não recupera nada. Se o documento certo não está entre os 100 que chegaram, nenhum modelo vai inventá-lo. Antes de investir na etapa cara, meça Recall@100 da sua híbrida. Se estiver baixo, o problema é o embedding, o que você indexou, ou a janela de fusão, e reranking só vai reordenar melhor um conjunto ruim.\n \n**Quando o valor por busca é baixo e o volume é alto.** É a situação típica de e-commerce de cauda longa. Milhões de buscas, margem por transação apertada, e um ganho de relevância que pode não pagar nem a conta de GPU nem a de API. Aqui o cálculo é de negócio, não de engenharia.\n \n**Quando o SLA de latência é apertado.** Se seu orçamento de p95 é 100 ms e a busca já consome 40, você não tem espaço para top-100. Você tem espaço para top-20 com um modelo pequeno, e talvez nem isso.\n \n**Quando a query é um identificador.** `SKU-A4729` não precisa de compreensão semântica, precisa de match exato. Detecte o padrão antes do pipeline e desvie, como já valia para a híbrida.\n \n**Quando o corpus é homogêneo demais.** Se seus 100 candidatos são quase idênticos entre si, não existe ordem certa a descobrir e o modelo vai desempatar ruído.\n \n## O que medir antes e depois\n \nVocê já construiu a peça mais importante no post 2: aquele conjunto de 50 a 200 queries reais com os documentos relevantes marcados à mão. É ele que decide isso.\n \nMeça nDCG@10 e MRR da híbrida sozinha, depois da híbrida com reranking, varrendo a profundidade em 10, 20, 30, 50 e 100. O que você quer enxergar é a forma da curva, não um número isolado. Se ela satura em 30, ir até 100 é dinheiro jogado fora. Se ela cai depois de 50, você acabou de descobrir que o modelo não serve para o seu domínio, e descobriu offline, o que é bem melhor do que descobrir em produção.\n \nJunto disso, e não depois, registre o delta de p95 e o custo por query. Ganho de relevância sem número de latência ao lado não é um resultado, é metade de um resultado.\n \nDepois vem o A/B em produção, porque ganho offline em reranking quase sempre parece ótimo e nem sempre o usuário percebe. As métricas que respondem isso são CTR nas primeiras posições, taxa de reformulação de query e conversão. Se o CTR do top 3 sobe e a reformulação cai, o reranking está funcionando de verdade. Se nada se move, você pagou por precisão que ninguém estava esperando.\n \n## Fechando a série\n \nCinco coisas que eu gostaria de ter sabido antes de começar com tudo isso:\n \nBM25 não é legado. Ele é o baseline forte que você vai passar o resto do projeto tentando bater, e em algumas tarefas não vai conseguir.\n \nEmbedding transforma texto em geometria, e é a partir dessa mudança de representação que todo o resto acontece. Mas o vetor é uma compressão com perda, e as perdas aparecem no lugar mais chato possível: identificadores, negações, queries curtas.\n \nRRF é o melhor retorno sobre esforço da lista. Sem calibração, sem treino, consistentemente melhor que qualquer um dos dois lados sozinho.\n \nReranking é o único estágio que lê os dois textos juntos, e é por isso que ele ganha onde os outros não têm como ganhar. Também é o único que não pré-computa nada, e é por isso que ele só roda em cima de dezenas de candidatos, nunca do catálogo.\n \nE o que amarra tudo: nada disso se decide por leitura de benchmark. Se decide com um conjunto de avaliação do seu próprio corpus, que dá meio dia de trabalho chato para montar e depois responde toda pergunta que vier.\n \nEssa série cobriu o que eu queria ter sabido antes de começar com busca semântica. Se você está implementando algo parecido, me conta em que contexto. Adoraria saber.\n \n## Referências\n \n- Nogueira, R., Cho, K. **Passage Re-ranking with BERT**. arXiv:1901.04085, 2019. [Paper](https://arxiv.org/abs/1901.04085)\n- Khattab, O., Zaharia, M. **ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT**. SIGIR 2020. [Paper](https://arxiv.org/abs/2004.12832)\n- Santhanam, K. et al. **ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction**. arXiv:2112.01488\n- Jacob, M. et al. **Drowning in Documents: Consequences of Scaling Reranker Inference**. arXiv:2411.11767\n- Sentence Transformers, [Pretrained Cross-Encoder models](https://sbert.net/docs/cross_encoder/pretrained_models.html) e [Retrieve & Re-Rank](https://sbert.net/examples/sentence_transformer/applications/retrieve_rerank/README.html)\n- Elastic, documentação do [Elastic Rerank](https://www.elastic.co/docs/explore-analyze/machine-learning/nlp/ml-nlp-rerank) e do [retriever text_similarity_reranker](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/retrievers/text-similarity-reranker-retriever)\n- Elastic Search Labs, [Exploring depth in a retrieve-and-rerank pipeline](https://www.elastic.co/search-labs/blog/elastic-semantic-reranker-part-3)\n- Cohere, [Rerank overview](https://docs.cohere.com/docs/rerank-overview), [best practices](https://docs.cohere.com/docs/reranking-best-practices) e o [anúncio do Rerank 4.0](https://docs.cohere.com/changelog/rerank-v4.0)\n- [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) e a [documentação do BGE Reranker](https://bge-model.com/bge/bge_reranker_v2.html)\n---\n \n*Terceiro e último post da série sobre busca moderna. Os anteriores são [Busca Semântica](https://rodolfodebonis.com.br/blog/como-funciona-busca-semantica) e [Busca Híbrida com RRF](https://rodolfodebonis.com.br/blog/busca-hibrida-com-rrf).*","https://assets.rodolfodebonis.com.br/blog-covers/f77aa6cb-65a3-4d40-87d3-5f704712d1d8.png",[36,40,44,48,52],{"id":37,"tenant_id":6,"slug":38,"created_at":39},"3600c7f5-46a1-44c1-ab4d-14b54bc49300","busca","2026-05-16T05:37:43.100096Z",{"id":41,"tenant_id":6,"slug":42,"created_at":43},"9de48f6e-646d-4d79-9499-a1822a48c0f1","cross-encoder","2026-08-07T17:18:01.99022Z",{"id":45,"tenant_id":6,"slug":46,"created_at":47},"3ec3cd7e-c8e8-4179-af96-1db8e19d55ff","ia","2026-07-28T08:47:55.819755Z",{"id":49,"tenant_id":6,"slug":50,"created_at":51},"43e39379-ec07-4aff-82a6-2946c56fa4f2","ml","2026-05-16T05:39:39.427214Z",{"id":53,"tenant_id":6,"slug":54,"created_at":55},"7886ed4e-ba65-4452-8fd0-e1ea4ce28b07","reranking","2026-08-07T17:18:19.021111Z","\u003Cp>This post closes out a series. The \u003Ca href=\"https://rodolfodebonis.com.br/en/blog/how-semantic-search-works\" target=\"_blank\" rel=\"noopener noreferrer\">first one\u003C/a> covered semantic search, the \u003Ca href=\"https://rodolfodebonis.com.br/en/blog/hybrid-search-reciprocal-rank-fusion\" target=\"_blank\" rel=\"noopener noreferrer\">second\u003C/a> covered hybrid search with Reciprocal Rank Fusion. If you landed here cold: BM25 nails the literal and misses the meaning, vector search does the opposite, and RRF merges the two rankings using position alone, with no weights to calibrate.\u003C/p>\n\u003Cp>Post 2 ended on an open problem. The query in that article was \u003Ccode>running shoes for hot weather\u003C/code>, and the dry-fit tee finished in second place after fusion. It wasn’t what the user wanted, but both retrievers found it plausible, and RRF ranks by consensus of position. Consensus isn’t relevance.\u003C/p>\n\u003Cp>The underlying reason is that up to this point in the pipeline, nobody has read the query and the document together.\u003C/p>\n\u003Ch2>What a bi-encoder never actually does\u003C/h2>\n\u003Cp>The mechanism is worth being precise about, because the difference between the two setups is the whole story.\u003C/p>\n\u003Cp>In vector search, the embedding model processes the document on its own, at index time, and emits a vector. Months later a query shows up, the model processes the query on its own, emits another vector, and the system compares the two with cosine. That arrangement has a name: \u003Cstrong>bi-encoder\u003C/strong>. Two independent encodes, and one late, cheap meeting between two points in space.\u003C/p>\n\u003Cp>That’s what makes vector search viable at all. The heavy work happens offline, once per document, and all that’s left on the request path is a geometric comparison. It’s also what makes HNSW possible: you can only build a neighborhood index if the document vector exists before the query arrives.\u003C/p>\n\u003Cp>And that’s precisely where the ceiling is. The vector for “Dry-fit tee for running in the heat” was computed without anyone knowing the question would be about shoes. It has to represent the entire document, for every conceivable query, in 1024 fixed numbers. It’s lossy compression, and what gets lost is exactly the interaction: which parts of this document matter \u003Cem>for this question\u003C/em>.\u003C/p>\n\u003Cp>A \u003Cstrong>cross-encoder\u003C/strong> tears that arrangement down. It takes the \u003Ccode>(query, document)\u003C/code> pair concatenated into a single sequence and runs one forward pass over both at once. Every query token can attend to every document token, and the other way around, across all layers. The output isn’t a vector, it’s a number: how well this document answers this query.\u003C/p>\n\u003Cp>The original implementation, from Nogueira and Cho’s 2019 paper, is almost embarrassingly plain. The query goes in as sentence A, the passage as sentence B, the \u003Ccode>[CLS]\u003C/code> token vector feeds a single dense layer, and out comes the probability that the passage is relevant. The query gets truncated to 64 tokens, the whole thing to 512. No new architecture. Just BERT reading both things at the same time.\u003C/p>\n\u003Ch2>How big the gain actually is\u003C/h2>\n\u003Cp>That same paper has the number that, in my opinion, is the strongest argument for reranking, and it’s strong because it isolates the variable.\u003C/p>\n\u003Cp>On MS MARCO, the candidate set was held fixed: BM25’s top 1000 for each query. BM25 on its own, ranking those same thousand documents, gets MRR@10 of 16.7 on the dev set. A BERT Large reordering the exact same list gets 36.5.\u003C/p>\n\u003Cp>No new documents entered. No index changed. No embeddings were generated. Retrieval was identical in both cases, and the metric more than doubled purely because something read the pairs.\u003C/p>\n\u003Cp>Worth noting what that number doesn’t say. It’s a 2019 benchmark, in English, on a dataset of Bing questions with roughly one relevant passage per query. Your domain isn’t that. But the asymmetry it exposes between “retrieving” and “ordering” is structural, and it shows up again in every benchmark since. Elastic, measuring their own model across 21 BEIR datasets, reports an average 40% improvement in ranking quality when reranking BM25 results.\u003C/p>\n\u003Ch2>Why you can’t just run this on everything\u003C/h2>\n\u003Cp>If cross-encoders are that much better, the obvious question is why not throw out the index entirely and score every document directly.\u003C/p>\n\u003Cp>The answer lives in the word “pair”. A bi-encoder does N document encodes once in its life, then one encode per query. A cross-encoder does one forward pass per \u003Ccode>(query, document)\u003C/code> pair, every single time, because the score depends on both. There’s nothing to precompute. Change the query and all prior work is garbage.\u003C/p>\n\u003Cp>You can put numbers on this using the published \u003Ccode>sentence-transformers\u003C/code> table, which measures throughput for the MS MARCO cross-encoders on a V100 GPU:\u003C/p>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Model\u003C/th>\n\u003Cth>nDCG@10 (TREC DL 19)\u003C/th>\n\u003Cth>MRR@10 (MS MARCO Dev)\u003C/th>\n\u003Cth>Docs/sec\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>ms-marco-TinyBERT-L2-v2\u003C/td>\n\u003Ctd>69.84\u003C/td>\n\u003Ctd>32.56\u003C/td>\n\u003Ctd>9000\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>ms-marco-MiniLM-L4-v2\u003C/td>\n\u003Ctd>73.04\u003C/td>\n\u003Ctd>37.70\u003C/td>\n\u003Ctd>2500\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>ms-marco-MiniLM-L6-v2\u003C/td>\n\u003Ctd>74.30\u003C/td>\n\u003Ctd>39.01\u003C/td>\n\u003Ctd>1800\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>ms-marco-MiniLM-L12-v2\u003C/td>\n\u003Ctd>74.31\u003C/td>\n\u003Ctd>39.02\u003C/td>\n\u003Ctd>960\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>ms-marco-electra-base\u003C/td>\n\u003Ctd>71.99\u003C/td>\n\u003Ctd>36.41\u003C/td>\n\u003Ctd>340\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\n\u003Cp>Take \u003Ccode>MiniLM-L6-v2\u003C/code>, the middle of that list. At 1800 documents per second, scoring 100 candidates costs 56 ms. Scoring a million-document catalog costs \u003Cstrong>nine and a half minutes per query\u003C/strong>, on a dedicated GPU, for one user. The \u003Ccode>L12\u003C/code>, which buys you half a point of nDCG, takes 17 minutes.\u003C/p>\n\u003Cp>This isn’t something you scale horizontally until it goes away. It’s an order-of-magnitude difference that changes what kind of problem you’re solving.\u003C/p>\n\u003Cp>Look at the third column against the first, too. Between \u003Ccode>L6\u003C/code> and \u003Ccode>L12\u003C/code> the quality is essentially identical and the cost doubles. Between \u003Ccode>TinyBERT-L2\u003C/code> and \u003Ccode>L6\u003C/code> you buy 4.5 points of nDCG at 5× the time. The diminishing-returns curve is already drawn inside the model family itself, before you’ve even picked a depth.\u003C/p>\n\u003Cp>Hence the architecture everyone converges on: cheap retrieval picks a handful of candidates, and the expensive model only looks at that handful.\u003C/p>\n\u003Cp>\u003Cimg src=\"https://assets.rodolfodebonis.com.br/blog-articles-images/reranking/arhitecture_en.png\" alt=\"image\">\u003C/p>\n\u003Cp>The first stage exists so nothing relevant gets left out. The second exists to get the order right. Different goals, which is exactly why different models make sense.\u003C/p>\n\u003Ch2>The tee finally drops\u003C/h2>\n\u003Cp>Back to the example from post 2. Five documents, the query \u003Ccode>running shoes for hot weather\u003C/code>, and the ranking RRF produced:\u003C/p>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Position\u003C/th>\n\u003Cth>Document\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>1\u003C/td>\n\u003Ctd>A: Ventus running shoe, high-ventilation mesh\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>2\u003C/td>\n\u003Ctd>C: Dry-fit tee for running in the heat\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>3\u003C/td>\n\u003Ctd>D: Lightweight shoe for hot, humid weather\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>4\u003C/td>\n\u003Ctd>E: Technical running sock for hot days\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>5\u003C/td>\n\u003Ctd>B: Rocha 3 trail running shoe, reinforced upper\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\n\u003Cp>Now the cross-encoder gets all five pairs and returns a score for each. The values below are illustrative, but the ordering is what any decent reranker produces here:\u003C/p>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Document\u003C/th>\n\u003Cth>Score\u003C/th>\n\u003Cth>Final position\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>A: Ventus, mesh\u003C/td>\n\u003Ctd>0.94\u003C/td>\n\u003Ctd>1st\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>D: Lightweight, hot weather\u003C/td>\n\u003Ctd>0.91\u003C/td>\n\u003Ctd>2nd\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>B: Rocha 3 trail\u003C/td>\n\u003Ctd>0.38\u003C/td>\n\u003Ctd>3rd\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>C: Dry-fit tee\u003C/td>\n\u003Ctd>0.07\u003C/td>\n\u003Ctd>4th\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>E: Technical sock\u003C/td>\n\u003Ctd>0.04\u003C/td>\n\u003Ctd>5th\u003C/td>\n\u003C/tr>\n\u003C/tbody>\n\u003C/table>\n\u003Cp>The tee dropped from second to fourth, and the reason is mundane once you look at the mechanism: the model processed the query token “shoes” in the same forward pass as the document token “tee”. It isn’t comparing two summaries of meaning, it’s reading a question about footwear alongside a document about apparel and concluding the category is wrong, even though “running” and “heat” both match literally.\u003C/p>\n\u003Cp>\u003Ccode>B\u003C/code> climbed to third on the complementary logic. It’s a running shoe, so it hits the primary intent, but “reinforced upper” is the opposite of ventilation, and the model penalizes without discarding. Neither earlier stage can draw that distinction, because neither one read both texts at once.\u003C/p>\n\u003Cp>Notice the gap between 0.91 and 0.38. That isn’t scale noise the way RRF scores are. That’s the model separating two populations.\u003C/p>\n\u003Ch2>Picking a model\u003C/h2>\n\u003Cp>The ecosystem splits three ways, and the choice is almost never about ranking quality.\u003C/p>\n\u003Cp>\u003Cstrong>Managed API.\u003C/strong> Cohere Rerank is the reference here. The current line is \u003Ccode>rerank-v4.0-pro\u003C/code> and \u003Ccode>rerank-v4.0-fast\u003C/code>, both multilingual across 100+ languages, with a 32k-token context window and support for semi-structured JSON documents. \u003Ccode>rerank-v3.5\u003C/code> is still around with 4k. Voyage and Jina play the same game. You add one HTTP call and you’re done: no GPU, no deployment, no reindexing.\u003C/p>\n\u003Cp>The planning detail is the billing unit. A “search unit” is one query with up to 100 documents, and long documents get chunked automatically, with each chunk counting as a separate document. Meaning your 100 candidates can turn into several search units if the texts are large. The AWS Marketplace listing for Rerank v3.5 (Bedrock edition) prices it at $0.002 per search unit. That’s roughly $200 for a month with 100k reranked searches, and $2,000 for a month with a million. Prices change, so check before you commit, but the shape holds: API reranking is cheap per search and expensive per traffic.\u003C/p>\n\u003Cp>\u003Cstrong>Self-hosted open source.\u003C/strong> The \u003Ccode>cross-encoder/ms-marco-*\u003C/code> family from sentence-transformers is the classic starting point, and the table above already shows the whole trade-off. One warning that matters more than it looks: those models are trained on MS MARCO, which is English. If your corpus is Portuguese, Spanish, or anything else, they will disappoint you. For multilingual work the answer is \u003Ccode>BAAI/bge-reranker-v2-m3\u003C/code>, built on top of bge-m3, with a 512-token window and a score you map to 0…1 with a sigmoid. The BGE docs give the obvious and correct advice: test on your real use case and pick on speed/quality balance, not on their table.\u003C/p>\n\u003Cp>\u003Cstrong>Built into the engine.\u003C/strong> Elasticsearch exposes reranking through the \u003Ccode>text_similarity_reranker\u003C/code> retriever, which can point at an external inference endpoint (Cohere, for instance) or at Elastic’s own model, Elastic Rerank, a 184M-parameter DeBERTa v3. Read the limitations before you get excited: English only, 512-token window, available from Stack 8.17, requires an appropriate subscription level, and still flagged as technical preview. Elastic writes in their own docs that the preview version may be cost prohibitive for high query rates with low-latency requirements, and recommends staying under top-30 for CPU inference.\u003C/p>\n\u003Cp>\u003Cstrong>The middle option.\u003C/strong> ColBERT occupies its own niche. Instead of one score per pair, it stores a vector per document token and performs a late interaction between query tokens and document tokens at query time. The original paper measures effectiveness competitive with the BERT models of the day while running two orders of magnitude faster and using four orders of magnitude fewer FLOPs per query. The price is the index: storing a vector per token inflates storage aggressively, and ColBERTv2 exists largely to shrink that by 5 to 8×. If latency is your constraint and you have disk to spare, it’s worth a look.\u003C/p>\n\u003Ch2>Depth is the parameter that matters\u003C/h2>\n\u003Cp>Here’s the most expensive mistake in this stage, and it runs against intuition.\u003C/p>\n\u003Cp>The default belief is that reranking more candidates can only help, since a good model would have more material to work with. Elastic measured this systematically, sweeping reranking depth across several models and BEIR datasets, and found three patterns:\u003C/p>\n\u003Cul>\n\u003Cli>\u003Cstrong>Fast rise, then saturation\u003C/strong>, in 72.6% of cases. What everyone expects.\u003C/li>\n\u003Cli>\u003Cstrong>Rise to a peak, then decay\u003C/strong>, in 20.2% of cases.\u003C/li>\n\u003Cli>\u003Cstrong>Steady decay with any amount of reranking\u003C/strong>, in 7.1% of cases. The reranker makes BM25 worse at every depth tested.\nThe explanation is mechanical and satisfying. nDCG@10 only moves when the reranker promotes something from below into the top 10. If the promoted document is relevant, the metric goes up. If it’s a false positive, it evicts a relevant document and the metric goes down. As you go deeper, the density of relevant documents collapses and the density of irrelevant ones grows. There’s a depth at which the model’s false-positive rate starts to dominate, and past it you’re paying for inference to degrade your own ranking.\u003C/li>\n\u003C/ul>\n\u003Cp>The paper \u003Cem>Drowning in Documents\u003C/em> gets to the same place from another direction, and notes something uncomfortable: rerankers frequently assign high scores to documents with no lexical or semantic overlap with the query at all.\u003C/p>\n\u003Cp>Three practical consequences, in the order you’ll need them:\u003C/p>\n\u003Cp>First, the number. Applying the rule of picking the depth that reaches 90% of the maximum gain, Elastic landed on roughly \u003Cstrong>top 100\u003C/strong> over BM25, at a third of the compute cost of chasing the maximum. When cost is tight they recommend \u003Cstrong>top 30\u003C/strong> with their model, and still measured over 40% uplift in nDCG@10 on the QA portion of their benchmark.\u003C/p>\n\u003Cp>Second, which way to tune. The better your first stage, the fewer candidates you need to rerank, because recall saturates earlier. And the better your reranker, the deeper it pays to go, because it makes fewer mistakes on the way down. You already have hybrid search with RRF, so your first stage is good: start shallow.\u003C/p>\n\u003Cp>Third, and this one surprised me: under a latency constraint, reranking deep with a small model tends to beat reranking shallow with a big one. In Elastic’s benchmark, \u003Ccode>MiniLM-L12-v2\u003C/code> processing 80 candidates beat stronger models capped at 10 or 20 for the same time budget. The relationship flips as you relax the constraint.\u003C/p>\n\u003Cp>On latency, one concrete number to calibrate against: measuring on two T4 GPUs, Elastic reports 0.0869 seconds to score 10 pairs with Elastic Rerank on HotpotQA. Linearized, top-30 lands around 0.26 s and top-100 around 0.87 s. Those aren’t the 50 ms that get thrown around in posts on the topic. Measure on your hardware, with your documents, before you promise an SLA.\u003C/p>\n\u003Ch2>Scores mean something again\u003C/h2>\n\u003Cp>One limitation I raised in post 2 gets resolved here, and it’s worth closing the loop.\u003C/p>\n\u003Cp>RRF scores mean nothing. They’re crammed into a tiny band, they aren’t comparable across queries, and so you can’t threshold on them. You can’t say “if the best result is bad, show the empty state”.\u003C/p>\n\u003Cp>A cross-encoder score is a different kind of thing. It’s a relevance estimate for that pair, produced by a model trained for exactly that, and it doesn’t depend on anyone’s position. The sentence-transformers models return logits that sit roughly between -10 and 10, and you apply a sigmoid to get something in 0…1. \u003Ccode>bge-reranker-v2-m3\u003C/code> works the same way. Cohere returns a normalized \u003Ccode>relevance_score\u003C/code>. Elasticsearch exposes \u003Ccode>min_score\u003C/code> right on the retriever precisely so you can cut.\u003C/p>\n\u003Cp>The trap is confusing “meaningful” with “calibrated”. The scale is specific to the model and the domain. A 0.6 from Cohere is not a 0.6 from BGE, and a 0.6 from BGE on your legal corpus is not the same 0.6 on your support FAQ. The threshold has to come from your data. But it exists, and that’s a genuinely new capability in the pipeline.\u003C/p>\n\u003Ch2>What this looks like in code\u003C/h2>\n\u003Cp>Self-hosted, with \u003Ccode>sentence-transformers\u003C/code>. The point of the snippet is that reranking is a pure function over the candidate list, with no state and no index:\u003C/p>\n\u003Cpre class=\"shiki shiki-themes github-dark github-light\" style=\"--shiki-dark:#e1e4e8;--shiki-light:#24292e;--shiki-dark-bg:#24292e;--shiki-light-bg:#fff\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">from\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> sentence_transformers \u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">import\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> CrossEncoder\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">import\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> torch\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> \u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#6A737D;--shiki-light:#6A737D\"># Sigmoid pins scores to 0..1. Without it you get raw logits.\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">reranker \u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> CrossEncoder(\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#9ECBFF;--shiki-light:#032F62\">    \"BAAI/bge-reranker-v2-m3\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">,   \u003C/span>\u003Cspan style=\"--shiki-dark:#6A737D;--shiki-light:#6A737D\"># multilingual; for English, ms-marco-MiniLM-L6-v2\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#FFAB70;--shiki-light:#E36209\">    activation_fn\u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">torch.nn.Sigmoid(),\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#FFAB70;--shiki-light:#E36209\">    max_length\u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=\u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">512\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">,\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">)\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> \u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">def\u003C/span>\u003Cspan style=\"--shiki-dark:#B392F0;--shiki-light:#6F42C1\"> rerank\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">(query: \u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">str\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">, candidates: list[\u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">dict\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">], top_k: \u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">int\u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\"> =\u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\"> 10\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">) -> list[\u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">dict\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">]:\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#9ECBFF;--shiki-light:#032F62\">    \"\"\"Takes the output of RRF, returns the reordered top_k.\"\"\"\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">    if\u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\"> not\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> candidates:\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">        return\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> []\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> \u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">    pairs \u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> [(query, doc[\u003C/span>\u003Cspan style=\"--shiki-dark:#9ECBFF;--shiki-light:#032F62\">\"text\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">]) \u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">for\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> doc \u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">in\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> candidates]\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">    scores \u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> reranker.predict(pairs, \u003C/span>\u003Cspan style=\"--shiki-dark:#FFAB70;--shiki-light:#E36209\">batch_size\u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=\u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">32\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">)\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> \u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">    for\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> doc, score \u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">in\u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\"> zip\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">(candidates, scores):\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">        doc[\u003C/span>\u003Cspan style=\"--shiki-dark:#9ECBFF;--shiki-light:#032F62\">\"rerank_score\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">] \u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=\u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\"> float\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">(score)\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> \u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">    ranked \u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=\u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\"> sorted\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">(candidates, \u003C/span>\u003Cspan style=\"--shiki-dark:#FFAB70;--shiki-light:#E36209\">key\u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=lambda\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> d: d[\u003C/span>\u003Cspan style=\"--shiki-dark:#9ECBFF;--shiki-light:#032F62\">\"rerank_score\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">], \u003C/span>\u003Cspan style=\"--shiki-dark:#FFAB70;--shiki-light:#E36209\">reverse\u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=\u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">True\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">)\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">    return\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> ranked[:top_k]\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003C/span>\u003C/code>\u003C/pre>\n\u003Cp>Two decisions are worth a comment. \u003Ccode>batch_size\u003C/code> is what determines whether you saturate the GPU or waste it, and it’s the first knob to turn when latency won’t close. And \u003Ccode>max_length=512\u003C/code> isn’t cosmetic: anything longer gets truncated, so the model scores the beginning of the text and ignores the rest. If your documents are long, you need to decide deliberately which chunk gets scored instead of letting the tokenizer decide for you.\u003C/p>\n\u003Cp>Same pipeline through an API, with Cohere:\u003C/p>\n\u003Cpre class=\"shiki shiki-themes github-dark github-light\" style=\"--shiki-dark:#e1e4e8;--shiki-light:#24292e;--shiki-dark-bg:#24292e;--shiki-light-bg:#fff\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">import\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> cohere\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> \u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">co \u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> cohere.ClientV2(\u003C/span>\u003Cspan style=\"--shiki-dark:#FFAB70;--shiki-light:#E36209\">api_key\u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=\u003C/span>\u003Cspan style=\"--shiki-dark:#9ECBFF;--shiki-light:#032F62\">\"...\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">)\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> \u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">response \u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> co.rerank(\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#FFAB70;--shiki-light:#E36209\">    model\u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=\u003C/span>\u003Cspan style=\"--shiki-dark:#9ECBFF;--shiki-light:#032F62\">\"rerank-v4.0-fast\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">,\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#FFAB70;--shiki-light:#E36209\">    query\u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=\u003C/span>\u003Cspan style=\"--shiki-dark:#9ECBFF;--shiki-light:#032F62\">\"running shoes for hot weather\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">,\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#FFAB70;--shiki-light:#E36209\">    documents\u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">[doc[\u003C/span>\u003Cspan style=\"--shiki-dark:#9ECBFF;--shiki-light:#032F62\">\"text\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">] \u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">for\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> doc \u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">in\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> candidates],\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#FFAB70;--shiki-light:#E36209\">    top_n\u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=\u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">10\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">,\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">)\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> \u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#6A737D;--shiki-light:#6A737D\"># results carry index (position in the original list) and relevance_score\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">top10 \u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">=\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> [candidates[r.index] \u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">for\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> r \u003C/span>\u003Cspan style=\"--shiki-dark:#F97583;--shiki-light:#D73A49\">in\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\"> response.results]\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003C/span>\u003C/code>\u003C/pre>\n\u003Cp>And inside Elasticsearch, chained onto the RRF retriever from the previous post:\u003C/p>\n\u003Cpre class=\"shiki shiki-themes github-dark github-light\" style=\"--shiki-dark:#e1e4e8;--shiki-light:#24292e;--shiki-dark-bg:#24292e;--shiki-light-bg:#fff\" tabindex=\"0\">\u003Ccode>\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">GET /products/_search\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">{\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">  \"retriever\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: {\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">    \"text_similarity_reranker\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: {\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">      \"retriever\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: {\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">        \"rrf\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: {\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">          \"retrievers\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: [\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">            { \u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">\"standard\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: { \u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">\"query\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: { \u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">\"multi_match\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: {\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">                \"query\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: \u003C/span>\u003Cspan style=\"--shiki-dark:#9ECBFF;--shiki-light:#032F62\">\"running shoes for hot weather\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">,\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">                \"fields\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: [\u003C/span>\u003Cspan style=\"--shiki-dark:#9ECBFF;--shiki-light:#032F62\">\"title\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">, \u003C/span>\u003Cspan style=\"--shiki-dark:#9ECBFF;--shiki-light:#032F62\">\"description\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">]\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">            }}}},\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">            { \u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">\"knn\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: {\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">                \"field\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: \u003C/span>\u003Cspan style=\"--shiki-dark:#9ECBFF;--shiki-light:#032F62\">\"embedding\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">,\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">                \"query_vector\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: [\u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">0.21\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">, \u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">-0.05\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">, \u003C/span>\u003Cspan style=\"--shiki-dark:#9ECBFF;--shiki-light:#032F62\">\"...\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">],\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">                \"k\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: \u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">100\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">,\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">                \"num_candidates\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: \u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">300\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">            }}\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">          ],\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">          \"rank_window_size\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: \u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">100\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">        }\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">      },\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">      \"field\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: \u003C/span>\u003Cspan style=\"--shiki-dark:#9ECBFF;--shiki-light:#032F62\">\"description\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">,\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">      \"inference_id\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: \u003C/span>\u003Cspan style=\"--shiki-dark:#9ECBFF;--shiki-light:#032F62\">\"my-reranker\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">,\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">      \"inference_text\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: \u003C/span>\u003Cspan style=\"--shiki-dark:#9ECBFF;--shiki-light:#032F62\">\"running shoes for hot weather\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">,\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">      \"rank_window_size\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: \u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">100\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">,\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">      \"min_score\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: \u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">0.4\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">    }\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">  },\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">  \"size\"\u003C/span>\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">: \u003C/span>\u003Cspan style=\"--shiki-dark:#79B8FF;--shiki-light:#005CC5\">10\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003Cspan style=\"--shiki-dark:#E1E4E8;--shiki-light:#24292E\">}\u003C/span>\u003C/span>\n\u003Cspan class=\"line\">\u003C/span>\u003C/code>\u003C/pre>\n\u003Cp>Note there are two \u003Ccode>rank_window_size\u003C/code> values in that query and they mean different things. The inner one is how many candidates fusion considers. The outer one is how many of those go to the model. Matching them is the sane default; setting the outer one higher does nothing, because no candidate exists beyond the fusion window.\u003C/p>\n\u003Ch2>Where reranking doesn’t help\u003C/h2>\n\u003Cp>A few cases where the honest answer is to leave it off.\u003C/p>\n\u003Cp>\u003Cstrong>When first-stage recall is the problem.\u003C/strong> Reranking retrieves nothing. If the right document isn’t among the 100 that arrived, no model is going to invent it. Before investing in the expensive stage, measure Recall@100 on your hybrid search. If it’s low, the problem is your embedding model, what you indexed, or your fusion window, and reranking will only do a nicer job of ordering a bad set.\u003C/p>\n\u003Cp>\u003Cstrong>When value per search is low and volume is high.\u003C/strong> The classic long-tail ecommerce situation. Millions of searches, thin margin per transaction, and a relevance gain that may not cover the GPU bill or the API bill. This is a business calculation, not an engineering one.\u003C/p>\n\u003Cp>\u003Cstrong>When the latency SLA is tight.\u003C/strong> If your p95 budget is 100 ms and search already eats 40, you don’t have room for top-100. You have room for top-20 with a small model, and maybe not even that.\u003C/p>\n\u003Cp>\u003Cstrong>When the query is an identifier.\u003C/strong> \u003Ccode>SKU-A4729\u003C/code> doesn’t need semantic understanding, it needs an exact match. Detect the pattern before the pipeline and route around it, same as with hybrid search.\u003C/p>\n\u003Cp>\u003Cstrong>When the corpus is too homogeneous.\u003C/strong> If your 100 candidates are near-identical to each other, there’s no correct order to discover and the model will just break ties on noise.\u003C/p>\n\u003Ch2>What to measure, before and after\u003C/h2>\n\u003Cp>You already built the most important piece back in post 2: that set of 50 to 200 real queries with the relevant documents marked by hand. It’s what decides this.\u003C/p>\n\u003Cp>Measure nDCG@10 and MRR for hybrid alone, then hybrid plus reranking, sweeping depth at 10, 20, 30, 50, and 100. What you’re looking for is the shape of the curve, not one number. If it saturates at 30, going to 100 is money on fire. If it drops after 50, you just learned the model doesn’t suit your domain, and you learned it offline, which beats learning it in production.\u003C/p>\n\u003Cp>Alongside that, not afterwards, record the p95 delta and the cost per query. A relevance gain without a latency number next to it isn’t a result, it’s half of one.\u003C/p>\n\u003Cp>Then comes the production A/B, because offline gains from reranking almost always look great and users don’t always notice. The metrics that answer this are CTR on the top positions, query reformulation rate, and conversion. If top-3 CTR goes up and reformulation goes down, reranking is doing real work. If nothing moves, you paid for precision nobody was waiting on.\u003C/p>\n\u003Ch2>Closing the series\u003C/h2>\n\u003Cp>Five things I wish I’d known before starting on any of this:\u003C/p>\n\u003Cp>BM25 isn’t legacy. It’s the strong baseline you’ll spend the rest of the project trying to beat, and on some tasks you won’t.\u003C/p>\n\u003Cp>Embeddings turn text into geometry, and everything else follows from that change of representation. But the vector is lossy compression, and the losses show up in the most annoying place possible: identifiers, negations, short queries.\u003C/p>\n\u003Cp>RRF is the best return on effort in the whole list. No calibration, no training, consistently better than either side alone.\u003C/p>\n\u003Cp>Reranking is the only stage that reads both texts together, which is why it wins where the others structurally can’t. It’s also the only one that precomputes nothing, which is why it runs over dozens of candidates and never over the catalog.\u003C/p>\n\u003Cp>And the thing that ties it together: none of this gets decided by reading benchmarks. It gets decided with an evaluation set from your own corpus, which takes half a day of tedious work to build and then answers every question that comes after.\u003C/p>\n\u003Cp>This series covered what I wish I’d known before starting with semantic search. If you’re building something similar, tell me what the context is. I’d love to know.\u003C/p>\n\u003Ch2>References\u003C/h2>\n\u003Cul>\n\u003Cli>Nogueira, R., Cho, K. \u003Cstrong>Passage Re-ranking with BERT\u003C/strong>. arXiv:1901.04085, 2019. \u003Ca href=\"https://arxiv.org/abs/1901.04085\" target=\"_blank\" rel=\"noopener noreferrer\">Paper\u003C/a>\u003C/li>\n\u003Cli>Khattab, O., Zaharia, M. \u003Cstrong>ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT\u003C/strong>. SIGIR 2020. \u003Ca href=\"https://arxiv.org/abs/2004.12832\" target=\"_blank\" rel=\"noopener noreferrer\">Paper\u003C/a>\u003C/li>\n\u003Cli>Santhanam, K. et al. \u003Cstrong>ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction\u003C/strong>. arXiv:2112.01488\u003C/li>\n\u003Cli>Jacob, M. et al. \u003Cstrong>Drowning in Documents: Consequences of Scaling Reranker Inference\u003C/strong>. arXiv:2411.11767\u003C/li>\n\u003Cli>Sentence Transformers, \u003Ca href=\"https://sbert.net/docs/cross_encoder/pretrained_models.html\" target=\"_blank\" rel=\"noopener noreferrer\">Pretrained Cross-Encoder models\u003C/a> and \u003Ca href=\"https://sbert.net/examples/sentence_transformer/applications/retrieve_rerank/README.html\" target=\"_blank\" rel=\"noopener noreferrer\">Retrieve & Re-Rank\u003C/a>\u003C/li>\n\u003Cli>Elastic, docs for \u003Ca href=\"https://www.elastic.co/docs/explore-analyze/machine-learning/nlp/ml-nlp-rerank\" target=\"_blank\" rel=\"noopener noreferrer\">Elastic Rerank\u003C/a> and the \u003Ca href=\"https://www.elastic.co/docs/reference/elasticsearch/rest-apis/retrievers/text-similarity-reranker-retriever\" target=\"_blank\" rel=\"noopener noreferrer\">text_similarity_reranker retriever\u003C/a>\u003C/li>\n\u003Cli>Elastic Search Labs, \u003Ca href=\"https://www.elastic.co/search-labs/blog/elastic-semantic-reranker-part-3\" target=\"_blank\" rel=\"noopener noreferrer\">Exploring depth in a retrieve-and-rerank pipeline\u003C/a>\u003C/li>\n\u003Cli>Cohere, \u003Ca href=\"https://docs.cohere.com/docs/rerank-overview\" target=\"_blank\" rel=\"noopener noreferrer\">Rerank overview\u003C/a>, \u003Ca href=\"https://docs.cohere.com/docs/reranking-best-practices\" target=\"_blank\" rel=\"noopener noreferrer\">best practices\u003C/a>, and the \u003Ca href=\"https://docs.cohere.com/changelog/rerank-v4.0\" target=\"_blank\" rel=\"noopener noreferrer\">Rerank 4.0 announcement\u003C/a>\u003C/li>\n\u003Cli>\u003Ca href=\"https://huggingface.co/BAAI/bge-reranker-v2-m3\" target=\"_blank\" rel=\"noopener noreferrer\">BAAI/bge-reranker-v2-m3\u003C/a> and the \u003Ca href=\"https://bge-model.com/bge/bge_reranker_v2.html\" target=\"_blank\" rel=\"noopener noreferrer\">BGE Reranker docs\u003C/a>\u003C/li>\n\u003C/ul>\n\u003Chr>\n\u003Cp>\u003Cem>Third and final post in the series on modern search. The earlier ones are \u003Ca href=\"https://rodolfodebonis.com.br/en/blog/how-semantic-search-works\" target=\"_blank\" rel=\"noopener noreferrer\">Semantic Search\u003C/a> and \u003Ca href=\"https://rodolfodebonis.com.br/en/blog/hybrid-search-reciprocal-rank-fusion\" target=\"_blank\" rel=\"noopener noreferrer\">Hybrid Search with RRF\u003C/a>.\u003C/em>\u003C/p>\n",18,[59,69,103],{"id":5,"tenant_id":6,"author":60,"status":11,"published_at":12,"reading_time_minutes":13,"view_count":14,"like_count":15,"featured":16,"created_at":17,"updated_at":12,"translations":61,"tags":63},{"sub":8,"name":9,"email":10},[62],{"id":20,"post_id":5,"tenant_id":6,"lang":21,"slug":22,"title":23,"excerpt":24,"content_md":25,"cover_image_url":26,"created_at":17,"updated_at":17},[64,65,66,67,68],{"id":37,"tenant_id":6,"slug":38,"created_at":39},{"id":41,"tenant_id":6,"slug":42,"created_at":43},{"id":45,"tenant_id":6,"slug":46,"created_at":47},{"id":49,"tenant_id":6,"slug":50,"created_at":51},{"id":53,"tenant_id":6,"slug":54,"created_at":55},{"id":70,"tenant_id":6,"author":71,"status":11,"published_at":72,"scheduled_at":73,"reading_time_minutes":74,"view_count":75,"like_count":76,"featured":16,"created_at":77,"updated_at":78,"translations":79,"tags":88},"924b3ca2-7bd5-43e3-8b62-a0ea64171948",{"sub":8,"name":9,"email":10},"2026-07-28T08:48:49.010736Z","2026-07-28T12:30:00Z",9,28,1,"2026-07-28T08:48:48.773791Z","2026-07-30T18:18:43.476223Z",[80],{"id":81,"post_id":70,"tenant_id":6,"lang":21,"slug":82,"title":83,"excerpt":84,"content_md":85,"cover_image_url":86,"created_at":77,"updated_at":87},"dbf1003d-cafa-4630-897f-37293c6ecb45","hybrid-search-reciprocal-rank-fusion","Hybrid Search with RRF: why the best systems combine BM25 and vectors","BM25 and vector search fail in different places. Reciprocal Rank Fusion merges both rankings without you calibrating a single weight. Second post in a series on modern search.","This post assumes you've read [Semantic Search: teaching machines to understand intent](/en/blog/how-semantic-search-works). Two-line recap: BM25 nails the surface of the text and misses the meaning; vector search nails the meaning and misses the literal. The blind spots are complementary, so the obvious move is to use both.\n\nThe catch is that \"use both\" hides a much nastier decision than it sounds.\n\n## Two lists on your screen, one results page\n\nSay you've run both searches. BM25 returned ten ranked documents. Vector search returned another ten, with some overlap. Now you have to hand the user a single list.\n\nWhich document goes in position one?\n\nThat's genuinely the only question hybrid search has to answer. It's also where most homegrown implementations start falling apart, because the intuitive answer is the wrong one.\n\n## Adding the scores looks obvious and almost always breaks\n\nEveryone's first attempt is a linear combination:\n\n```\nfinal_score = α × bm25_score + (1 - α) × cosine_score\n```\n\nNeat on paper. Turn the α dial, favor one side or the other, done. Except the two numbers don't live on the same ruler.\n\nCosine similarity is bounded: with text, in practice, it sits between 0 and 1. BM25 has no ceiling. Its score depends on term IDF, document length, query length, and corpus size. A single rare term against a million-document index can produce a score dozens of times larger than the maximum possible cosine. Add them raw and BM25 swallows vector search whole. You didn't configure weights, you configured a scale.\n\nThe obvious fix is to normalize first. Per-query min-max, for instance: the best result in each list becomes 1.0, the worst becomes 0.0. And that's where the trap lives.\n\nMin-max is relative to the query. If vector search found nothing good, the best of the bad results still becomes 1.0. Normalization erased precisely the information that mattered: this list has nothing useful in it. You just handed maximum weight to a terrible result because of an arithmetic rescale.\n\nYou can work around it with distribution-based normalization, z-scores, per-domain calibration. All of that works, up to a point. But look at what happened: you showed up to solve ranking and ended up doing score statistics. That α is a hyperparameter that shifts when you swap the embedding model, when the corpus grows, when query mix changes. It's permanent maintenance.\n\n## Rank is universal, score isn't\n\nReciprocal Rank Fusion sidesteps all of it by throwing the scores away.\n\nThe idea, published by Cormack, Clarke and Büttcher at SIGIR 2009, fits on one line:\n\n```\nRRF(d) = Σ  1 / (k + rank_r(d))\n        r∈R\n```\n\nFor each ranking `r` where document `d` shows up, add the inverse of its position, shifted by a constant `k`. Sort by the sum. That's the whole algorithm.\n\nNo normalization. No weights to calibrate. No training. Both lists come in as sequences of positions, and a position is a unit every ranker on earth produces identically. BM25's first place and the vector search's first place are the same thing: first place. That's what makes fusion possible without anyone having to answer what a score of 14.7 is worth.\n\nThe most important consequence comes for free. Since each list contributes a small, similar increment, the document that rises is the one both searches agree is decent. Not either one's absolute favorite. RRF rewards consensus.\n\n## Five documents, two lists, one winner\n\nWorth watching this happen with actual numbers. Query: `running shoes for hot weather`, on a sporting goods catalog.\n\nBM25 matches literal tokens and returns:\n\n| Rank | Document                                          |\n| ---- | ------------------------------------------------- |\n| 1    | C: Dry-fit tee for running in hot weather         |\n| 2    | A: Ventus running shoe, high-ventilation mesh     |\n| 3    | B: Rocha 3 trail running shoe, reinforced upper   |\n| 4    | E: Technical running sock for warm days           |\n| 5    | D: Lightweight shoe for humid, warm climates      |\n\nThe t-shirt takes first place because it matches two discriminative terms, \"running\" and \"hot weather\". It's useless for the user's intent, but BM25 has no way to know that. Meanwhile `D`, which is nearly a perfect paraphrase of the query, sinks: \"warm\" isn't \"hot\" to an inverted index.\n\nVector search returns a different order:\n\n| Rank | Document                          |\n| ---- | --------------------------------- |\n| 1    | D: Lightweight shoe, warm climate |\n| 2    | A: Ventus running shoe, mesh      |\n| 3    | E: Technical sock for warm days   |\n| 4    | C: Dry-fit tee                    |\n| 5    | B: Rocha 3 trail shoe             |\n\nNow `D` climbs to the top, as expected. But so does the sock, because \"running\" and \"heat\" dominate the query vector and the product category carries less weight than it should.\n\nEach list has a wrong first place, for opposite reasons. Now RRF with `k = 60`:\n\n| Document        | BM25 rank | Vector rank | RRF                     | Final |\n| --------------- | --------- | ----------- | ----------------------- | ----- |\n| A: Ventus shoe  | 2         | 2           | 1/62 + 1/62 = 0.032258  | 1st   |\n| C: Dry-fit tee  | 1         | 4           | 1/61 + 1/64 = 0.032018  | 2nd   |\n| D: Lightweight  | 5         | 1           | 1/65 + 1/61 = 0.031778  | 3rd   |\n| E: Running sock | 4         | 3           | 1/64 + 1/63 = 0.031498  | 4th   |\n| B: Trail shoe   | 3         | 5           | 1/63 + 1/65 = 0.031258  | 5th   |\n\n`A` wins without having been anybody's first choice. It's the only document both searches put near the top for different reasons: BM25 saw \"running\" and \"shoe\", the embedding model saw \"ventilation mesh\" and understood heat. That's consensus doing its job.\n\nNotice what RRF **didn't** do: the t-shirt is still second. Rank fusion isn't a relevance filter. If a wrong document shows up decently in both lists, it stays decent afterward. Hold that thought, it comes back later.\n\n## What k actually does\n\n`k = 60` gets passed around as a magic number. The origin is far less mystical than that: the paper says the value was fixed during a pilot experiment and never changed in the validation runs that followed.\n\nWhat the published data does support is the shape of the curve. In Table 1 of the paper, MAP for `k` between 20 and 100 stays between 0.2134 and 0.2147. Under 1% of variation. The nominal peak is at `k = 80`, not 60. Only the extremes move the needle: `k = 0` drops to 0.2072 and `k = 500` to 0.2098.\n\nTranslated to your system: `k` controls how much a single list's first place counts against consensus.\n\nTake the same shoe example with `k = 0`, meaning the score is just `1/rank`:\n\n| Document        | RRF with k = 0    | Final |\n| --------------- | ----------------- | ----- |\n| C: Dry-fit tee  | 1/1 + 1/4 = 1.25  | 1st   |\n| D: Lightweight  | 1/5 + 1/1 = 1.20  | 2nd   |\n| A: Ventus shoe  | 1/2 + 1/2 = 1.00  | 3rd   |\n\nThe order flipped. With a small `k`, the gap between first and second place is huge, so each ranker essentially drags its own favorite onto the podium and consensus loses. With a large `k`, positions become nearly indistinguishable and what's left is vote counting.\n\n`k = 60` is a reasonable balance point on that spectrum, not a global optimum for your domain. If you have an evaluation set, sweep the range. If you don't, tweaking `k` by intuition is the last thing that'll improve your results.\n\n## What this looks like in the architecture\n\nThe flow is simpler than most people expect:\n![image](https://assets.rodolfodebonis.com.br/blog-articles-images/how-semantic-search-works/architecture_en.png)\n\nThree things here change your results in practice.\n\n**Both searches run in parallel, so latency is the max of the two, not the sum.** If BM25 takes 12 ms and vector takes 25 ms, the search costs 25 ms plus fusion, and fusion is noise: sorting a few dozen items in memory won't show up in your p99.\n\n**The real bottleneck is usually generating the query embedding.** If you're calling an external API, every search becomes a network round trip before you even touch the index, and that typically dominates total time. Two ways out: cache query embeddings, which works remarkably well because query distribution is heavily concentrated in the short head (the same phrases repeat all day long), or run the model yourself. A small embedding model on CPU already changes that math.\n\n**Fusion window size matters more than `k`.** If each side returns only 10 candidates, a document sitting at position 11 in both lists simply doesn't exist as far as RRF is concerned. Your hybrid recall is capped by the union of the two candidate sets. In Elasticsearch that parameter is `rank_window_size` and it defaults to 10, which is far too small for almost any real use case. Pulling 50 or 100 per side and truncating afterward is often the difference between \"works\" and \"why doesn't this obvious result show up?\".\n\n## Fifteen lines of Python, or one line of JSON\n\nWriting RRF from scratch is almost embarrassingly simple:\n\n```python\nfrom collections import defaultdict\n\ndef rrf(rankings: list[list[str]], k: int = 60) -> list[tuple[str, float]]:\n    \"\"\"Fuses ranked ID lists. Each list arrives sorted from most to least relevant.\"\"\"\n    scores = defaultdict(float)\n    for ranking in rankings:\n        for position, doc_id in enumerate(ranking, start=1):\n            scores[doc_id] += 1 / (k + position)\n    return sorted(scores.items(), key=lambda item: item[1], reverse=True)\n\n\nbm25   = [\"C\", \"A\", \"B\", \"E\", \"D\"]\nvector = [\"D\", \"A\", \"E\", \"C\", \"B\"]\n\nrrf([bm25, vector])\n# [('A', 0.032258), ('C', 0.032018), ('D', 0.031778), ('E', 0.031498), ('B', 0.031258)]\n```\n\nNotice the function knows nothing about search. It takes lists of identifiers. That's a property, not an accident: you can fuse three, four, five rankers, including business rules and popularity ordering, without changing a line.\n\nIn practice you probably won't write it, because the engines ship RRF already:\n\n```json\nGET /products/_search\n{\n  \"retriever\": {\n    \"rrf\": {\n      \"retrievers\": [\n        { \"standard\": { \"query\": { \"multi_match\": {\n            \"query\": \"running shoes for hot weather\",\n            \"fields\": [\"title\", \"description\"]\n        }}}},\n        { \"knn\": {\n            \"field\": \"embedding\",\n            \"query_vector\": [0.21, -0.05, \"...\"],\n            \"k\": 50,\n            \"num_candidates\": 200\n        }}\n      ],\n      \"rank_constant\": 60,\n      \"rank_window_size\": 100\n    }\n  },\n  \"size\": 20\n}\n```\n\nIn Elasticsearch, `rank_constant` already defaults to 60 and `rank_window_size` defaults to 10, so both fields above are spelled out on purpose. One planning detail that catches teams off guard: RRF in Elasticsearch is a paid feature, listed by Elastic itself among Platinum and Enterprise capabilities. On a self-hosted cluster with a Basic license, the query comes back with a license error.\n\nOpenSearch added rank-based fusion in 2.19 through the `score-ranker-processor` with the `rrf` technique, also defaulting to a rank constant of 60, and it's open source. Qdrant exposes `Fusion.RRF` directly in the Query API. Weaviate offers `rankedFusion`, which is RRF, and `relativeScoreFusion`, which is score normalization, with the latter as the default since 1.24. If you're on Postgres with pgvector there's nothing native: it's two CTEs with `row_number()` and a `full outer join`, which works fine too.\n\n## Where hybrid wins with no effort at all\n\nTwo query patterns show up in basically every domain.\n\n**Intent descriptions, in e-commerce.** Someone types \"dress for a beach wedding\". BM25 chases \"dress\", \"beach\" and \"wedding\" and brings back a heavy formal gown, a beach cover-up, and maybe a flower girl dress. Vector search understands the context (light, flowing, breathable fabric, long cut, outdoor formal event) and returns the right category, then falls apart the moment the shopper adds \"Zara\" or a specific model number. Fused, vector search defines the right candidate set and BM25 keeps the literal term anchoring the top.\n\n**Technical support and FAQs.** Someone searches \"401 error\". One article in your knowledge base explains that authentication fails when the token expires, talks about `Bearer`, JWT and refresh tokens, and never writes the number 401 anywhere. Vector search finds it. At the same time you have a troubleshooting doc with `HTTP 401` literally in the first line, and that's probably the one the user wants first. Vector alone buries the second, BM25 alone never finds the first. RRF puts both on page one.\n\nWhat both cases have in common is exactly what makes hybrid good: the errors are independent. When one ranker fails, the other fails differently.\n\n## Where RRF breaks\n\nThis is where I part ways with the internet's default enthusiasm for the technique.\n\n**RRF is blind to magnitude, and that cuts both ways.** Ignoring scores is precisely what solves the scale problem, and it's also what prevents the algorithm from knowing the entire vector list is garbage. If the best similarity in that list was 0.31, RRF still grants it full voting power, because all it sees is \"first place\". Score-based methods like Weaviate's `relativeScoreFusion` or Qdrant's DBSF exist for exactly this reason. If one of your retrievers frequently has nothing good to offer in your domain, measure both approaches.\n\n**The output scores mean nothing.** Look at the table: 0.032258 against 0.031258. Every result lives squeezed into a tiny band, and the value isn't comparable across queries. That kills anything that depends on a threshold. You can't say \"only show results above X\", and you can't say \"if the score is low, render the empty state\". If your product needs that decision, you'll need a different signal, usually a raw ranker score captured before fusion.\n\n**A wrong document with consensus stays on top.** That's the dry-fit tee holding second place. RRF orders, it doesn't judge relevance.\n\n**Exact identifiers get diluted.** If the user types `SKU-A4729`, the desired behavior isn't fusion, it's exact match dominating everything. Hybrid can push up a mediocre document that shows up decently in both lists. The fix isn't tuning `k`, it's detecting the query pattern and routing around the hybrid pipeline before it starts.\n\n**If both rankers agree too much, you're paying for two searches and getting one.** Fusion only adds value when the lists disagree. Measure the overlap between the two top 20s on real traffic. If it's very high, vector search is probably just reproducing BM25, and the problem is your embedding model or what you chose to index.\n\n**The operational cost is real.** Two indexes over the same corpus, two write paths that need to stay consistent, and one detail that's easy to forget: filters (category, stock, permissions) have to be applied on both sides, with identical criteria, before fusion. Filtering after fusion wrecks pagination and result counts. Switching embedding models means reindexing everything, and while the reindex runs you have two generations of vectors in the same index.\n\n## How to start, and what to measure\n\nThe sequence that avoids rework:\n\n1. Turn on hybrid with equal weights and `k = 60`. Don't touch anything else yet.\n2. Build a small, honest evaluation set. Somewhere between 50 and 200 real queries from your logs, with relevant documents labeled by hand. It's half a day of tedious work and it's the only way to know whether any later change helped.\n3. Measure three things against baselines: nDCG@10, Recall@50 and MRR. Compare hybrid against BM25 alone and against vectors alone.\n4. If hybrid doesn't beat both, stop and investigate. It's usually a fusion window that's too small, an embedding model that doesn't fit your vocabulary, or the wrong field indexed. It isn't `k`.\n5. Only after that should you touch `rank_window_size`, then per-ranker weights, and `k` last.\n\nThe order matters. Weights and `k` are the most visible knobs and the ones that pay off least.\n\nThere are libraries for the evaluation side, like [ranx](https://github.com/AmenRa/ranx), which implements RRF alongside other fusion methods and all the standard metrics. Using it for offline comparison is faster than writing your own nDCG.\n\n## The last 10%\n\nHybrid search with RRF is, in my opinion, the best return on effort in search today. You switch it on, you calibrate nothing, and the result is consistently better than either side alone. For most products, this is where you can stop.\n\nBut look at the limit that surfaced along the way: RRF orders by positional consensus, and consensus isn't the same thing as relevance. The t-shirt stayed in second place because both searches found it plausible, and neither ranker ever read the query and the document together to decide whether one answered the other.\n\nThat's what a cross-encoder does. It takes the 50 or 100 candidates hybrid retrieved, processes query and document in the same forward pass, and reorders them with an understanding no first-stage retriever can have. It's far too expensive to run over the full catalog, which is exactly why it only makes sense on top of good retrieval.\n\nWhen you want a perfect top 10 rather than a decent top 100, that's the next step. Saving it for the third post.\n\nBefore then, if you already have BM25 and vector search running separately, today's experiment is short: grab twenty real queries, save both rankings, fuse them with the fifteen lines of Python above, and eyeball what changed in the top 5. You'll know in an afternoon whether your domain needs this, and you'll probably find out where each of your two searches is failing badly.\n\n## References\n\n- Cormack, G. V., Clarke, C. L. A., Büttcher, S. **Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods**. SIGIR 2009. [PDF](https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf)\n- Elasticsearch docs: [RRF retriever](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/retrievers/rrf-retriever) and [reciprocal rank fusion](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion)\n- Elastic, [Pricing FAQ](https://www.elastic.co/pricing/faq), on RRF availability per subscription tier\n- OpenSearch, [score-ranker-processor](https://docs.opensearch.org/latest/search-plugins/search-pipelines/score-ranker-processor/) and the [2.19 RRF announcement](https://opensearch.org/blog/introducing-reciprocal-rank-fusion-hybrid-search/)\n- Qdrant, [Hybrid Queries](https://qdrant.tech/documentation/search/hybrid-queries/)\n- [ranx](https://github.com/AmenRa/ranx), a fusion and ranking evaluation library\n\n---\n\n*Second post in a series on modern search. The first one is [Semantic Search](/en/blog/how-semantic-search-works). The next closes it out with cross-encoder reranking. If you're building hybrid search somewhere, tell me how it's going.*","https://assets.rodolfodebonis.com.br/blog-covers/2937beef-df6e-4450-8800-b7dffa5e7ade.png","2026-07-28T09:23:27.367288Z",[89,90,94,98,99],{"id":37,"tenant_id":6,"slug":38,"created_at":39},{"id":91,"tenant_id":6,"slug":92,"created_at":93},"e017c671-1f93-407e-9334-d439622653c5","elasticsearch","2026-07-28T08:47:33.948155Z",{"id":95,"tenant_id":6,"slug":96,"created_at":97},"7bfa9291-72ca-46cf-b0e0-4ff8d29a4f78","embeddings","2026-05-16T05:38:28.835806Z",{"id":45,"tenant_id":6,"slug":46,"created_at":47},{"id":100,"tenant_id":6,"slug":101,"created_at":102},"af8ebc67-e047-40f3-9fac-b498bd570c92","rrf","2026-07-28T08:47:12.831063Z",{"id":104,"tenant_id":6,"author":105,"status":11,"published_at":106,"cover_image_url":107,"reading_time_minutes":108,"view_count":109,"like_count":110,"featured":16,"created_at":111,"updated_at":112,"translations":113,"tags":122},"43c0f013-6a6b-4286-8e74-bbb0bd8eaf93",{"sub":8,"name":9,"email":10},"2026-05-16T05:52:44.530779Z","https://assets.rodolfodebonis.com.br/blog-covers/198d42d7-9779-4d52-85bf-37d93abd1233.png",13,347,4,"2026-05-16T05:52:44.463814Z","2026-07-30T18:26:52.464187Z",[114],{"id":115,"post_id":104,"tenant_id":6,"lang":21,"slug":116,"title":117,"excerpt":118,"content_md":119,"cover_image_url":120,"created_at":111,"updated_at":121},"4ee0c693-b676-4ea5-ba02-d600111df7b7","how-semantic-search-works","Semantic Search: teaching machines to understand intent, not just words","BM25 still works, but it has four serious blind spots. Embeddings turn text into geometry — and that's the foundation of everything that's hyped in AI today. First in a series on modern search.","A few years ago, if you asked me how search works in a serious system, I'd answer in three words: inverted index, BM25, done. That was the state of the art, that was what ran everywhere, and that was what I knew well enough to teach.\n \nToday, after putting semantic search into production on top of nearly a million documents, I'd change my answer. Not because BM25 got worse — quite the opposite, it's still the foundation of almost every search system in the world. I'd change it because BM25 alone is leaving a lot of value on the table. And what fills that gap is an idea that looks like magic, until you understand what's actually happening underneath.\n \nThis is the first post in a series on modern search. Here we cover **semantic search** — what it is, why it works, how to use it. The next post adds BM25 and vector search together as **hybrid search**. The third closes with **reranking**, the cherry that separates good search from excellent search. But first we need to understand the underlying problem.\n \n## The problem most people don't notice\n \nImagine you're building a movie catalog. Could be a Letterboxd, an IMDB, an internal app. A user shows up and types:\n \n> \"movie about a guy stuck in the same day\"\n \nYou know what they want. I know what they want. Anyone who's seen *Groundhog Day* knows what they want. The problem is that your database doesn't know.\n \nIf you're using what 99% of systems use — text search based on an inverted index — your database will take that query, look for the literal words (\"guy\", \"stuck\", \"same\", \"day\"), and return:\n \n- *Stuck on You* (it has \"stuck\" in the title)\n- *Day After Tomorrow* (it has \"day\" in the title)\n- Any documentary about prison life (because \"stuck\" shows up in summaries about confinement)\nThe result: the user doesn't find *Groundhog Day*, closes your app, opens Google, types exactly the same phrase. And Google finds it. Why? **Because Google isn't running `LIKE '%stuck%'`.**\n \nThe difference between \"search that works\" and \"search that frustrates the user\" doesn't live in which database you picked, or how many servers you threw at it. It lives in understanding that **words are not meanings**, and that there are tools to bridge that gap.\n \n## How the machine sees text: BM25 and friends\n \nBefore we talk about the pretty stuff, let's understand what's running almost everywhere today.\n \nWhen you send `\"comfortable white sneakers\"` to Elasticsearch (or OpenSearch, or Solr — all the same family), three things happen in sequence. First, **tokenization**: the string gets broken into individual words. Second, **stemming**: suffixes get cut to reduce morphological variations. \"comfortable\", \"white\", \"sneakers\" become \"comfort\", \"white\", \"sneak\". Third, that gets matched against the **inverted index**: a structure that, for each token, stores the list of documents where that token appears.\n \n```\nsneak    -> [12, 47, 89, 156, ...]\nwhite    -> [12, 102, 230, ...]\ncomfort  -> [47, 88, 102, ...]\n```\n \nDocument 12 shows up in two lists. Document 47 also. Document 88, in just one. The more lists a document appears in, the more likely it's relevant. But that's just the start.\n \nWhat ranks the results is **BM25** — Best Match 25, the twenty-fifth iteration of a family of algorithms that started in the 70s. It's the default in Elasticsearch, OpenSearch, and Solr. If you use text search anywhere, BM25 is what's scoring.\n \nThe formula looks intimidating, but it has only three ideas:\n \n**Term Frequency (TF)**: how many times the token appears in the document. More times, more relevant. But not linear — if it appears 50 times, it's not 50× better than appearing once. The formula saturates.\n \n**Inverse Document Frequency (IDF)**: rare terms are worth more. The word \"sneakers\" appears in 2% of your fashion catalog — high weight. The word \"the\" appears in 100% of documents — weight nearly zero. Makes sense: if you searched for \"white sneakers\", matching \"sneakers\" tells me a lot more about relevance than matching \"white\".\n \n**Length normalization**: a short document containing the term is more relevant than a long document containing the term, because the chance that it's actually about that thing is higher. Without this, a 5000-word technical manual would beat a short description just by volume.\n \nBM25 mixes these three things and produces a score. Higher score, higher in the results. It's elegant, it scales well, and it works reasonably in a closed domain. But it has four serious blind spots.\n \n## Where BM25 breaks\n \n**Synonyms.** User searches for \"cellphone\", catalog has \"smartphone\". Zero match. You can solve this with a manual synonym dictionary, but you're going to maintain that for English, Portuguese, regional slang, and every niche's jargon? Good luck.\n \n**Vocabulary.** User searches for \"lightweight clothes for hot weather\". Catalog has \"short-sleeve linen blouse\". Same intent, zero words in common. BM25 returns nothing.\n \n**Intent.** User searches for \"good movie to watch with my girlfriend\". What does that mean? BM25 will match on \"movie\", \"good\", and \"girlfriend\" and return random results.\n \n**Multilingual.** You indexed in English, the user searches in Spanish. Same content, different languages, BM25 has no way to know they're the same thing.\n \nThe problem, at the core, is the same: **BM25 looks at the surface of the text, not at the meaning.** It's a tool for token coincidence, not for understanding.\n \nThat's where the interesting part comes in.\n \n## Embeddings: text becomes geometry\n \nThe simplest and most powerful definition that exists:\n \n> **An embedding is a dense vector of N numbers that represents the meaning of a piece of text.**\n \nYou send the word \"pizza\" to the embedding model. It returns a list of — say — 1024 numbers between -1 and 1:\n \n```\n[0.21, -0.05, 0.78, 0.13, -0.42, ..., 0.09]\n```\n \nYou send \"lasagna\". It returns another list of 1024 numbers:\n \n```\n[0.19, -0.08, 0.81, 0.10, -0.40, ..., 0.07]\n```\n \nThe magic is that these two lists will be **nearly identical**. Not because the model saw \"pizza\" and \"lasagna\" together — though that helped during training — but because it learned that both live in the same semantic neighborhood: Italian food, main dish, pasta or dough base, dinner context.\n \nNow send \"Python\". The vector will be very different. Because Python is a programming language, it's tech, it's an entirely different context. The vector for \"Java\" will be similar to the one for \"Python\", because both are languages. Pizza and lasagna sit together in one corner, Python and Java sit together in another corner, cat and dog sit together in a third corner.\n \n**The distance between two vectors becomes a measure of semantic similarity.** That's the trick. You've converted text — which is symbolic, discrete, hard to compare — into geometry. And geometry, we know how to measure.\n \nIn practice, generating an embedding looks like this:\n \n```python\nfrom openai import OpenAI\n \nclient = OpenAI()\nresponse = client.embeddings.create(\n    input=\"pizza margherita\",\n    model=\"text-embedding-3-small\"\n)\nvector = response.data[0].embedding  # list of 1536 floats\n```\n \nText goes in, geometry comes out.\n \n## Algebra with meanings\n \nTo make this less abstract: because these vectors actually carry meaning, you can do math with them. Real math. The classic experiment, from Mikolov's 2013 paper:\n \n```\nvector(\"king\") - vector(\"man\") + vector(\"woman\") ≈ vector(\"queen\")\n```\n \nThe model learned, without anyone explicitly teaching it, that there's a masculinity-femininity axis in vector space. Another one:\n \n```\nvector(\"Paris\") - vector(\"France\") + vector(\"Italy\") ≈ vector(\"Rome\")\n```\n \nThe concept of \"capital of a country\" became a direction in space. You can subtract \"France\" to remove the \"specific country\" component, then add \"Italy\" to put it back. The result lands near the Italian capital.\n \nThis isn't pretty theoretical math. It's literally what's happening inside the model. That's why semantic search works: the model learned structure about the world, and you're doing geometry on top of that structure.\n \n## Where these numbers come from\n \nEmbedding models are neural networks trained on absurd amounts of text to learn these representations. The main families today:\n \n**Commercial APIs.** OpenAI (`text-embedding-3-small`, `text-embedding-3-large`), Cohere (`embed-v3`, strong on multilingual), Voyage AI. Expensive, but quality near the top of the leaderboard, no infra to maintain on your side.\n \n**State-of-the-art open-source.** BGE (from BAAI), E5 (from Microsoft), GTE (from Alibaba). You run it on your GPU, zero API cost, zero vendor lock-in. BGE-M3 and BGE-large multilingual compete well with OpenAI in many benchmarks.\n \n**The classic base.** Sentence-Transformers — the library that popularized all of this. Smaller, simpler models, great for prototyping.\n \n**How to choose?** Go to [MTEB](https://huggingface.co/spaces/mteb/leaderboard) — the Massive Text Embedding Benchmark, the public reference leaderboard. Pick a model in your cost and size range, and **test it on your domain**. A model that's good at English may be bad at Portuguese. A model that's good at short text may be bad at long documents. A model that's good at general domain may be bad at legal, medical, or technical vocabulary. Always measure.\n \n## How to compare two vectors\n \nYou have the query vector, you have the document vectors. How do you compare them? Three options, in the order you'll probably use them.\n \n**Cosine similarity** measures the angle between vectors, ignoring magnitude. It ranges from -1 to 1 (in practice, with text, it falls between 0 and 1). It's the default in the overwhelming majority of cases, because meaning lives in direction, not in the size of the vector.\n \n**Dot product** is the scalar product. It cares about direction *and* magnitude. If your vectors are normalized — and most modern models return normalized vectors — dot product is mathematically equivalent to cosine and cheaper to compute. Use it for optimization.\n \n**Euclidean distance** is the straight line between two points. It works, but it's less common with text. It shows up more with image embeddings.\n \nRule of thumb: start with cosine, switch to dot product when optimizing.\n \n## kNN, ANN, and the scale problem\n \nHow do you find the documents most similar to the query? The conceptual algorithm is **k-Nearest Neighbors (kNN)**. Four steps: take the query, generate its embedding, measure the distance to every document in the catalog, sort, and return the top K.\n \nIt works perfectly on a thousand documents. On a million, computing distance against every single one is O(n), infeasible in real time.\n \nThe solution is **ANN — Approximate Nearest Neighbors**. You give up a bit of precision to gain orders of magnitude in speed. Instead of comparing against everything, you compare against an intelligently chosen subset.\n \nThe most popular algorithm today is **HNSW — Hierarchical Navigable Small World**. It works like a world map with multiple zoom levels: you start at the highest level, navigate quickly until you're close to the answer, then descend into more detailed levels and refine. From O(n) you drop to O(log n). It's what Elasticsearch uses, what pgvector uses, what practically every vector database uses underneath.\n \nIn production, recall of 95-98% is easy to hit with a well-configured HNSW, and latency lands in the milliseconds.\n \n## Where to run this\n \nThe ecosystem has exploded in the last few years. I split it into two families.\n \n**Dedicated vector databases**: Pinecone (SaaS), Weaviate, Qdrant, Milvus, Chroma. They were born for this. They generally deliver better performance and more mature features for vector search. The downside is it's one more database to maintain.\n \n**Databases that added vector support**: Elasticsearch and OpenSearch (already had mature text search, got vector); PostgreSQL with the pgvector extension; Redis; MongoDB. The advantage here is reusing a stack you already have. The downside is that features and performance sometimes aren't as polished as in the dedicated ones.\n \nHow to decide? If you already have Elasticsearch in production, **start by adding vector search to it**. Don't switch databases over a new feature. If you're starting from zero and want simplicity, Qdrant and Weaviate are great. If you're Postgres-first and your volume is moderate, pgvector handles it. There's no single answer — there's tradeoff.\n \n## The movie example, now with semantics\n \nRemember the query from the beginning? \"movie about a guy stuck in the same day\". Before, with BM25, it returned *Stuck on You* and *Day After Tomorrow* because of the literal words. Now, with vector search:\n \n| Position | Movie | Score |\n|---|---|---|\n| 1 | *Groundhog Day* | 0.89 |\n| 2 | *Edge of Tomorrow* | 0.85 |\n| 3 | *Palm Springs* | 0.81 |\n \nNotice the important detail: the synopsis of *Groundhog Day* says \"a man relives the same day repeatedly\". The word \"stuck\" never appears. But the model understood that being in a time loop is a form of being stuck in time. *Palm Springs* is an indie film many people have never heard of — but the model knows it, because it trained on descriptions from the entire internet.\n \nSame query. Completely different results. **No synonym dictionary. No manual rules.** The model just did geometry.\n \n## Where vector search also breaks\n \nBefore you walk away from here with stars in your eyes — vector search has serious blind spots too.\n \n**Exact identifiers.** User searches for `SKU-A4729`. Vector search will return things semantically similar, which is not what they want. For SKUs, product codes, IDs, order numbers — you need exact match, not similarity.\n \n**Negations.** \"Shoe without laces\" might return shoes with laces, because the concept \"laces\" is strongly represented in the query vector. Modern models handle this better, but it remains fragile.\n \n**Very short or ambiguous queries.** \"java\" — is it the language, the island, or the coffee? BM25 also suffers, but vector search doesn't magically solve it.\n \n**Cost.** Generating an embedding per document costs. Vector storage costs (1024 dimensions × 4 bytes × N documents). Reindexing when you switch models costs. The latency of generating an embedding for every query also costs.\n \nWhen you stack the blind spots — exactness, short queries, cost — it becomes clear: **replacing BM25 with vector search is trading one set of problems for another**.\n \n## The answer is to combine, not replace\n \nThe good news is that these two worlds have complementary blind spots. BM25 is strong exactly where vector search is weak: exact match, SKUs, specific keywords. Vector search is strong exactly where BM25 is weak: meaning, intent, divergent vocabulary.\n \nWhen you combine them, what one misses the other catches.\n \nThat's what the next post is about: **hybrid search**. How to run BM25 and vector search in parallel, how to fuse the rankings without falling into the obvious trap of weighting score against score, and why this is the architecture that delivers the best results in production from day zero, with no manual weight tuning.\n \nUntil then, if you've never played with embeddings, **open a notebook**. Grab the OpenAI API (or run BGE locally), embed a few dozen sentences from your domain, compute cosine between them, and see what clusters together. It's the best way to internalize the idea: to see with your own eyes that the model actually understands.\n \n---\n \n*This is the first post in a series on modern search. Next: hybrid search with Reciprocal Rank Fusion. If you're applying this in some context, drop me a message — I'd love to know where.*","https://assets.rodolfodebonis.com.br/blog-covers/d4f08aa1-1801-4afe-bfad-c156788359f1.png","2026-07-28T09:29:52.268081Z",[123,124,125,129],{"id":37,"tenant_id":6,"slug":38,"created_at":39},{"id":95,"tenant_id":6,"slug":96,"created_at":97},{"id":126,"tenant_id":6,"slug":127,"created_at":128},"c0801447-701e-4cf6-8859-e4cf89f9d8a0","machine-learning","2026-05-16T05:38:07.501314Z",{"id":49,"tenant_id":6,"slug":50,"created_at":51}]