What if the smartest AI coding assistant still gave you broken code? The problem might not be the model — it's how your code gets sliced up before the AI ever sees it. Most code intelligence systems rely on a surprisingly crude approach: they chop code into fixed-size chunks the same way they would slice through a novel. But code isn't natural language. When a function gets split in half or a class loses its context, the AI receives puzzle pieces that simply don't fit together. The researchers discovered that this preprocessing bottleneck silently degrades performancee across every major coding benchmark. Their solution? Stop counting lines and start respecting structure.
The Hidden Flaw in AI Code Assistants
When an AI assistant searches a codebase, it needs retrievable segments. The standard approach is brutally simple: pick a line count, cut, and repeat. This works for plain text, but code follows rigid rules. Serve a function mid-statement, and you hand the AI misleading information.
Consider compute_stats, a function returning three values: distinct counts, mean, and variance. If chunking splits it midway, an AI asked to use compute_stats will never know what it returns. It might guess "total" or "count" — generating code that crashes. The model isn't at fault. It never saw the complete picture. The researchers confirmed this experimentally across multiple benchmarks.
CAST: Respecting Code's Natural Boundaries
The team solved this with CAST — Chunking via Abstract Syntax Trees. Instead of counting lines, CAST parses each file into an Abstract Syntax Tree, a hierarchical structure where every node represents a real language construct: modules, classes, functions, loops, and imports. Using the open-source tree-sitter library, CAST works across virtually any programming language without embedding language-specific rules.
The algorithm applies a split-then-merge strategy. Starting from the root, it packs the largest complete constructs into each chunk. If a node exceeds the size limit, it recursively breaks into its children. A final merging step combines small adjacent siblings — like import statements — into dense, information-rich chunks. Size is measured by non-whitespace characters, a more reliable density indicator across coding styles.
The design follows four principles: syntactic integrity (chunks never split constructs); high information density (packing each chunk to its limit); language invariance (no language-specific tricks); and plug-and-play compatibility (concatenating chunks reproduces the original file). That last property means CAST replaces any chunking method without touching the rest of the pipeline
What the Numbers Reveal
To put CAST to the test, the team ran it through three demanding benchmarks. RepoEval measures code completion within large single files. CrossCodeEval tests multi-language cross-file reasoning across Python, Java, C#, and TypeScript. SWE-bench involves resolving real-world GitHub issues by generating code patches.
On RepoEval, retrieval precision improved by up to 3.3 points and recall by 4.3 points. On the SWE-bench, Claude 3.7 Sonnet's Pass@1 score climbed from 13.7 to 16.3. Across four languages in CrossCodeEval, exact match scores improved by an average of 2.9 points, with TypeScript showing the largest gains — its dense syntax makes structural boundaries especially critical. The merging step proved essential: a split-only variant bloated indexes with tiny import-level chunks and degraded performance. Optimal chunk size falls between 2,000 and 2,500 non-whitespace characters, with a 4,000-character query window striking the best balance.
A Universal Solution for Polyglot Codebases
Techniques optimized for Python often stumble on Java's verbosity or TypeScript's annotations. Fixed-size chunking compounds this: a line limit calibrated for Python over-segments Java and under-segments TypeScript. CAST sidesteps this entirely because it never asks what language it handles — only what tree structure it sees. For teams building multi-language assistants, this language-agnostic consistency is a significant advantage.
Conclusion
Retrieval-augmented generation has fundamentally changed how AI assists with code — giving language models the ability to search and draw from entire repositories rather than relying on memory alone. Yet the preprocessing step that feeds these systems has been treated as a settled matter, defaulting to line-based heuristics borrowed from natural language pipelines. The mismatch between that assumption and the deeply structured nature of code has quietly placed a ceiling on what these systems can achieve.
The need is clear: code intelligence systems deserve a chunking strategy that actually understands code. Feeding a language model structurally broken fragments is a problem no amount of model capability can fully compensate for. CAST demonstrates, through rigorous experimentation, that respecting syntactic boundaries at the preprocessing stage produces consistent, measurable gains across retrieval quality, generation accuracy, and cross-language robustness.
For any team running a code RAG pipeline today, the actionable recommendation is straightforward: swap in AST-based chunking. Because CAST concatenates into a perfect reconstruction of the original file and requires no changes downstream, adoption carries minimal risk and no pipeline redesign. It is one of the simplest improvements available with some of the most reliable returns. Future directions include incorporating multi-level AST context awareness, integrating execution traces for per-query adaptiveness, and exploring multiple structural views of the same code snippet to further reduce dependence on any single representation.
If your AI coding assistant has been underperforming, the fix might not be a bigger model. It might be smarter slicing. Try CAST, and see the difference structure makes.