Off-the-shelf OCR engines are trained mostly on English and Latin scripts. Point them at a scanned Vietnamese invoice and they will happily drop the diacritics - turning 'phân bón' into 'phan bon' - which is worse than useless for a financial pipeline. Getting Vietnamese right takes deliberate choices at every stage.
Layout before text
The mistake most teams make is running OCR on the whole page at once. Documents have structure - titles, paragraphs, tables - and a table read as flowing text is unrecoverable. We run a YOLOv7 layout detector first to find those regions, then OCR each region with the right strategy. Tables get cell-aware reconstruction; paragraphs get line-by-line recognition.
# 1. Detect layout regions
regions = layout_model(page_image) # YOLOv7 -> [title, paragraph, table, ...]
# 2. Route each region to the right reader
results = []
for r in sorted(regions, key=lambda r: (r.y, r.x)):
crop = page_image[r.y1:r.y2, r.x1:r.x2]
if r.kind == "table":
results.append(reconstruct_table(crop))
else:
results.append(viet_ocr.recognize(crop)) # VietOCR / TrOCRPreprocessing earns its keep
Real documents are photographed at an angle, under bad light, on crumpled paper. Deskewing and adaptive thresholding before recognition consistently gave us the biggest single accuracy jump - larger than swapping the recognition model. Diacritics live in a few pixels above each character, so anything that sharpens those pixels pays off directly.
- Deskew and dewarp before anything else
- Adaptive thresholding for uneven lighting
- Upscale small text so diacritics survive
- Keep a color copy - some stamps and text are color-coded
Validate with the language, not just the pixels
Recognition confidence alone lies. A model can be very confident about a wrong-but-plausible word. Because Vietnamese has strict syllable structure, a lightweight language check catches outputs that are not valid Vietnamese syllables and flags them for review - or lets an LLM propose a correction using surrounding context.
For financial documents, a wrong digit is a disaster. We would rather flag 5% of fields for a human than silently ship a confident mistake.
What we would tell our past selves
- Invest in layout analysis early - it multiplies the value of every downstream step.
- Measure on YOUR documents, not a public benchmark; domain shift is brutal.
- Always keep confidence scores and a human-in-the-loop review path.
- Combine OCR with an LLM only after the basics are solid - it is a polish step, not a rescue.
Our Vietnamese OCR and Document Layout Analysis projects are open source, and the Document AI solution page has an interactive demo of the layout-then-OCR pipeline described here.
