Indexing: What the Model Gets to See
Chunking: The Unglamorous Lever
Chunk size is a bet about the questions
Small chunks (100–300 tokens): precise retrieval, clean embeddings, fragmented context — the model gets 'the answer sentence' but not the paragraph that makes it meaningful. Large chunks (1,000+ tokens): complete context, diluted embeddings — a 2,000-token chunk about five topics is kind of about all five and really about none.
An embedding averages the meaning of everything in the chunk. Ask 'what questions will retrieve this?' — a chunk should be sized so its embedding represents one thing and its content answers one kind of question.
insight
The embedding is a centroid. Every extra topic in the chunk pulls it toward an average that matches nothing — big chunks don't retrieve poorly, they retrieve *vaguely*.