Picking a document parser off a leaderboard is how you get it wrong

Contest guidelines usually arrive as PDFs or HWP files. I once started from a summary announcement and missed a scoring table that existed only in the attached original, so I needed a pipeline that parses and reads every attachment. HWP is the document format of Hancom Office, the word processor that Korean government agencies and contest organizers use by default.

That led me to look for the "best PDF parser." I checked leaderboards. I found the top-ranked entry on OmniDocBench and thought that would settle it.

That was wrong. After reviewing 137 tools, the conclusion was not which model ranked first. It was that my search method was structurally incapable of seeing the answer I needed.

The short version

The key is not inside the table but in its first column. born-digital and scanned documents are different problems. My first bad assumption was that one strongest tool could handle both.

A PDF with a live text layer does not need OCR. A rule-based parser can read its structure on CPU in 0.015 seconds per page. Putting a heavy vision model on top makes it slower, more expensive, and less accurate. Send a scanned file to a rule-based parser instead, and it produces nothing.

So the final design is routing behind a single entry point. It checks for a text layer first, sends files with one to the rule-based path, and sends the rest to Korean OCR. The caller gives it a file and does not need to care where it went.

The three things leaderboards cannot show

This is the core of the post. Why does a leaderboard alone not get me to the table above?

1. Origin bias

Nearly every tool at the top of the leaderboard comes from China: Baidu, OpenDataLab, DeepSeek, Tencent, Alibaba. OmniDocBench itself is a benchmark made by China's OpenDataLab.

The measured languages center on Chinese and English; Korean is partly buried inside multilingual categories. In other words, overall rank #1 does not guarantee rank #1 for Korean. A benchmark measures only what it chose to measure.

Hancom's parser is not on this leaderboard at all. The answer I was looking for was absent from the table I was reading.

2. Library-vs-model bias

Leaderboards line up only end-to-end models. In practice, though, the things I install and use directly are usually libraries and frameworks, such as Hancom, Marker, Docling, and Unstructured.

They are absent from leaderboards. They are not listed because they are in a different category, not because they are worse. A leaderboard-only view does not even tell me they exist.

There is another layer here: the self-benchmark trap.

• #1 on opendataloader-bench = Hancom (a benchmark made by Hancom)
• #1 on ParseBench = LlamaParse (a benchmark made by LlamaParse)
• #1 on SCORE-Bench = Unstructured (a benchmark made by Unstructured)

All of this is true, and every one is its own benchmark. When I see the phrase "best overall," I need to look at who measured it before I look at the score.

3. Recency bias

The top spot in this field changes every few weeks. Rankings reshuffle whenever a version is released.

Chasing only the latest work misses the other side. Stable requirements such as CPU determinism, Korean handwriting, permissive licensing, and on-premises deployment are often better served by mature tools. The latest SOTA is usually less tested along those axes.

What I needed was not the weekly #1 but a tool that gives the same output for the same input months from now.

So I changed how I searched

One frame, "best PDF parser," cannot get past these three biases. I split the search across axes instead.

• Language axis: Korean-specific tools and Korean benchmarks
• Origin axis: Korea, China, the United States, and Europe separately
• Category axis: models / libraries / commercial APIs / on-premises solutions
• Recency axis: current SOTA and mature standards separately
• Completeness axis: queries such as "list," "full comparison," and "alternatives"

That is how the list reached 137. With one search frame, I would probably have looked at around twenty, and the answer would not have been among them.

This is not limited to document parsers. The same trap appears every time I ask for "the best X." What rises to the top of search results is not ordered by ability but by what was measured. What was not measured is not merely unranked; it is missing from the list.

The honest limits that remain

• I verified Hancom's parser locally on born-digital Hangul. I did not test it on scanned documents. That is not what it is for in the first place.
• Accuracy figures for commercial APIs come from their providers; I did not reproduce them. I left numbers such as handwriting accuracy as stated and gave them little weight in my decision.
• I ran only a small fraction of the 137 tools. I filtered the rest by license, language support, and input format. "Researched" and "verified" are different words; mixing them is how I mislead myself later.

The question that stays

The slowest part of this research was not comparing tools. It was deciding what belonged in the comparison table. Start with the wrong candidates and no amount of precision in the comparison that follows will help.

And candidate gathering usually receives no review. I show people a comparison table, but nobody asks, "How did you choose these twenty?"

The question I need to ask before choosing a tool is this: Do the queries that produced my candidate list structurally exclude the answer I am looking for?