Re-Ranking: The Second Look

A bi-encoder embeds query and document separately — which is why it can index a million documents offline and look them up in milliseconds. The cost: query and doc never interact inside the model; similarity is measured after the fact.

A cross-encoder reads the pair jointly — query and candidate concatenated through the model, attention flowing both directions — producing a far sharper relevance judgment. But it can't index: there are no precomputed vectors, so it must run per candidate. That's the architecture of the pipeline: bi-encoder retrieves top-100 cheaply, cross-encoder re-ranks them into a top-5 the LLM actually sees.