What chunk size should a RAG system use when nobody on the team has measured anything yet? A reasonable starting point is 256 to 512 tokens per chunk with 10 to 20% overlap for questions that look up a fact, and 1,024 tokens or a whole page for questions that need a full argument or table. Two limits come before any rule of thumb: a chunk must fit inside your embedding model’s input limit, and overlap multiplies the size of your index.

The rest of this piece shows what the popular tools do by default, what those defaults cost on a real-sized document, and how to pick a number you can defend.

Last checked November 16, 2026 against the documentation and source code linked in each row. Tool defaults change between releases, so recheck the version you run.

The defaults you inherit

Most teams never choose a chunk size. They get one from whatever library or managed service they started with, and those defaults disagree by a factor of five. Some are measured in tokens and some in characters, which makes them harder to compare than they look.

Tool Default chunk size Default overlap Unit Source
OpenAI file search (vector stores) 800 400 tokens API reference
LlamaIndex SentenceSplitter 1,024 200 tokens sentence.py, constants.py
Amazon Bedrock Knowledge Bases, default chunking about 300 not stated tokens Bedrock user guide
LangChain TextSplitter base class 4,000 200 characters base.py
Open WebUI 1,024 100 characters config.py
Azure AI Search Text Split skill 5,000 0 characters skill reference

A few details from those sources matter in practice. OpenAI calls 800/400 its “auto” strategy and lets you set anything from 100 to 4,096 tokens, as long as the overlap does not exceed half the chunk size. LangChain measures length with Python’s len unless you pass another function, so its 4,000 is characters, not tokens. Open WebUI uses LangChain’s recursive character splitter unless you switch it to tokens, and a comment in its config explains the 1,024 figure: it approximates a 256-token embedding window at four characters per token. Bedrock’s default chunking keeps whole sentences together, so its 300 tokens is approximate. Azure’s skill counts characters by default and can switch to Azure OpenAI tokens.

What those defaults cost on one document

Take a hypothetical 60,000-token policy manual, about the length of a 120-page PDF at 500 tokens a page. Each new chunk starts one “stride” after the last, where the stride is the chunk size minus the overlap. That gives an upper-bound chunk count of (60,000 minus overlap) divided by the stride, rounded up. Character defaults are converted at four characters per token, the same approximation Open WebUI’s config uses, so treat those rows as rough.

Setting Chunk / overlap (tokens) Chunks Tokens stored and embedded Index overhead Context sent for top 5
OpenAI 800/400 800 / 400 149 119,200 1.99x 4,000
LlamaIndex 1,024/200 1,024 / 200 73 74,400 1.24x 5,120
Bedrock default about 300 / 0 200 60,000 1.00x 1,500
LangChain 4,000/200 chars about 1,000 / 50 64 63,150 1.05x 5,000
Open WebUI 1,024/100 chars about 256 / 25 260 66,475 1.11x 1,280
Azure 5,000 chars about 1,250 / 0 48 60,000 1.00x 6,250

Two things stand out. A 50% overlap, as in OpenAI’s default, nearly doubles the tokens you embed and store, because almost every sentence lives in two chunks. And the context column shows why chunk size is also a cost decision at question time: five 1,250-token chunks send four times as much text to the model as five 300-token chunks, much of it unrelated to the question.

Real splitters cut at paragraph and sentence boundaries, so actual counts land a little higher or lower. The ratios hold.

The limit that matters more than any rule of thumb

An embedding model reads a fixed maximum number of tokens. Text past that limit is not an error you see; it is simply dropped before the vector is computed. The model card for the widely used all-MiniLM-L6-v2 says it plainly: “By default, input text longer than 256 word pieces is truncated.”

Pair that model with LangChain’s base default of 4,000 characters, roughly 1,000 tokens, and about three quarters of each chunk never reaches the vector. The chunk is still stored and can still be sent to the model when it is retrieved, but it can only be found by its first 256 word pieces. A question about a clause near the end of a long chunk will miss it even though the text sits in the index.

So the first check is mechanical: look up your embedding model’s maximum input and keep every chunk, overlap included, below it. If you change embedding models later, recheck this before re-indexing; our guide to embedding model migration covers the rest of that cutover.

What the published tests found

Two public evaluations are worth reading before picking a number, with the caveat that each was run by a vendor on its own pipeline.

Chroma’s chunking evaluation of July 3, 2024 measured retrieval at the token level with five retrieved chunks and OpenAI’s text-embedding-3-large. It found that a recursive character splitter at 200 tokens with no overlap was “consistently high performing across all evaluation metrics,” that reducing overlap improved token efficiency, and that the 800/400 file search default, assuming a plain token splitter, gave slightly below-average recall and the lowest scores on its other metrics.

NVIDIA’s chunking study of June 18, 2025 measured end-to-end answer accuracy on five document sets. Page-level chunking had the best average accuracy, 0.648, and the lowest spread; fixed token sizes ranged from 0.603 to 0.645. Factoid questions favored 256 to 512 tokens, analytical questions on financial filings favored 1,024 tokens or whole pages, and the extremes of 128 and 2,048 tokens generally did worse. On one dataset at 1,024 tokens, 15% overlap beat 10% and 20%.

Read together, the studies agree on the shape rather than the number: medium chunks beat tiny and huge ones, heavy overlap costs more than it returns, and the best size depends on the kind of question.

How to choose and defend a number

  1. Write down the questions first. Collect 30 to 50 real questions with the passage that answers each one. If they are mostly lookups (“What is the refund window?”), start at 256 to 512 tokens. If they need a whole section or table (“How did margins change and why?”), start at 1,024 tokens or page boundaries.
  2. Check the embedding limit. Chunk size plus overlap must stay under the model’s maximum input.
  3. Keep overlap modest. Start at 10 to 15%. Raise it only if answers keep breaking across chunk boundaries in your tests.
  4. Respect structure. Split on headings and paragraphs before falling back to fixed sizes, and keep tables whole. NVIDIA’s test kept tables and charts as single units rather than splitting them.
  5. Test two or three sizes, not twenty. Run the same question set against each and compare whether the right passage appears in the top results. Freeze that question set so later comparisons stay fair; our guide to AI evaluation set versioning explains how.
  6. Record the choice. Store the chunk size, overlap, splitter and its version next to each chunk. When an answer cites a passage, that record is part of RAG data provenance, and permission filters described in RAG document access control have to survive re-chunking too.

Reporting & Sources

Checked November 16, 2026: OpenAI vector store file API reference and retrieval guide; LlamaIndex sentence.py and constants.py; LangChain base.py; Open WebUI config.py; Amazon Bedrock Knowledge Bases chunking; Azure AI Search Text Split skill; all-MiniLM-L6-v2 model card; Chroma, Evaluating Chunking Strategies for Retrieval (July 3, 2024); NVIDIA, Finding the Best Chunking Strategy for Accurate AI Responses (June 18, 2025). The 60,000-token document is hypothetical; chunk counts, overhead and context sizes are ASYA’s arithmetic from the published defaults, with character settings converted at four characters per token.