One chunking engine.
Python, JavaScript, Rust.
A fast, high-fidelity document chunking engine for RAG. 36 formats, one Rust core — with byte-identical bindings for Python, JavaScript, and Rust.
pip install py-chunkspy-chunks · js-chunks · rs-chunks
from py_chunks import get_chunks
# One API. Every format. Rust underneath.
chunks = get_chunks("report.pdf", mode="semantic")
for chunk in chunks:
print(chunk["content_type"], "->", chunk["content"][:48])
# heading -> Q3 Financial Summary
# semantic -> Revenue grew 18% quarter-over-quarter, driven...
# table -> | Region | Revenue | YoY |Performance
Milliseconds, not seconds
A Rust core does the parsing and segmentation. Measured over 446 real files against its peers, Docling and Unstructured — see the full benchmark.
Documents per second (pooled)
files / sec · higher is betterTime per document (pooled)
ms / file · lower is betterMeasured over 446 files on Apple M1 Max (py-chunks 0.5.0). See the benchmarks page for full methodology.
Why chunking
Chunking is the hidden lever in retrieval quality
Retrieval is only as good as its chunks. Bad boundaries bury the answer; slow, dependency-heavy splitters bottleneck ingestion. chunk-engine fixes both.
Naive fixed-size splitting
Sentences cut mid-word, tables severed, headings orphaned — the embedding model never sees a coherent unit, so retrieval misses.
chunk-engine — structure-aware
Each chunk is a complete semantic unit with a typed content_type — headings, tables, and lists stay intact, so retrieval lands on the answer.
Formats
36 formats, one engine
Office, PDF, web, plain-text, data, email, eBooks, and notebooks — grouped by family, each with an honest availability badge.
Word
5OOXML + legacy binary Word
PowerPoint
6OOXML + legacy binary PowerPoint
Spreadsheets
7Excel + OpenDocument sheets
Text + page-scoped images
Web & Markup
4Markup and rich text
OpenDocument
2LibreOffice / OpenOffice documents
Plain & Data
6Text and structured data
Outlook + MIME email
eBooks & Notebooks
2EPUB and Jupyter
New formats live in the library source and latest builds; the current published wheel documents a subset. Upgrade with pip install -U py-chunks to pick up the newest formats.
See it
Watch a document become chunks
A sample document on the left; colored chunk blocks on the right, keyed by content type. Toggle the mode and watch the blocks regroup.
Q3 Financial Summary
Revenue grew 18%…
Operating margin…
Regional Breakdown
Region / Revenue table
Growth bullet list
Ingest Example
get_chunks(…) call
One chunk per element — each heading, paragraph, table, and list stands alone.
One engine, three languages
Same chunks, everywhere
Pick the SDK that fits your stack — Python, JavaScript, or Rust. They wrap the same engine and produce byte-identical output.
Byte-identical, verified
All three SDKs wrap the same Rust engine, so they emit exactly the same chunks. Parity is checked over every fixture × every mode — the only differences trace to the reference engine's own non-determinism.
pip install py-chunks- 2204 / 2214 chunk comparisons byte-identical (99.5%)
- 1056 / 1056 image extractions identical
- 273 / 273 markdown conversions identical
Modes
Seven ways to chunk a document
default, structural, section, semantic, sliding_window, sentence, page_aware — plus row / table / sheet for spreadsheets.
defaultElement-level chunks — one per paragraph or heading.
Best for: Fine-grained, element-level control.
structuralIndex every paragraph and heading individually.
Best for: Fine-grained retrieval over document structure.
sectionrecommendedKeep all content under a heading together in one chunk.
Best for: Section-level search / document indexes.
semanticrecommendedGroup semantically coherent passages together.
Best for: Feeding an LLM or embedding model.
sliding_windowOverlapping windows with configurable size and overlap.
Best for: Dense retrieval / sliding-context inference.
sentenceEnforce a fixed number of sentences per chunk.
Best for: Tight token budgets.
page_awarePreserve the document's original page layout.
Best for: Page-referenced citations.
Spreadsheets (XLSX / XLS / ODS / CSV) add dedicated modes:
Quick start
Chunk your first document
One call. Switch the language tab to see the exact API for your SDK.
from py_chunks import get_chunks, stream_chunks, get_markdown
# Batch — works for every supported format
chunks = get_chunks("document.pdf")
chunks = get_chunks("notes.md", mode="semantic")
chunks = get_chunks("deck.pptx", mode="sliding_window", window_size=3, overlap=1)
for chunk in chunks:
print(chunk["content"], chunk["content_type"], chunk["metadata"])
# Streaming — constant memory over huge files
for chunk in stream_chunks("large.pdf", mode="section"):
handle(chunk)
# Markdown conversion
md = get_markdown("report.docx")Start chunking in one line
Install for your language, point it at a file, get clean chunks back. No services, no config, no heavy dependency tree.
pip install py-chunks