HN Debrief

A spectre is haunting Unicode

  • Unicode
  • Infrastructure
  • Programming
  • Internationalization

The post is a tour of “ghost characters” in Unicode and earlier East Asian standards. These are code points for characters that slipped into official registries through mistakes like manuscript copying, dictionary carryovers, or scan errors, then became hard to remove once standards adopted them. The piece uses examples from Japanese and Chinese text history to show how a non-character can become real enough for software to carry forever.

If your product handles Chinese, Japanese, or Korean text, treat Unicode code points as an encoding layer, not proof that a character is meaningful or interchangeable across locales. Budget for font, normalization, search, and OCR edge cases instead of assuming Unicode solved them for you.

Discussion mood

Interested and amused, with a strong undercurrent of frustration about how messy CJK encoding still is in practice. People liked the historical oddity, but the most engaged comments turned it into a reminder that Unicode, fonts, normalization, and OCR still break down badly on East Asian text.

Key insights

  1. 01

    One ghost likely came from OCR noise

    A specific case in the post appears to be even more accidental than it first sounds. The character 彁 was likely introduced by a bad digitization of a Taisho newspaper printing of 彊, with a better source copy showing the original was never there. That sharpens the article’s premise from “copying mistakes happen” to “digitization pipelines can mint durable fake characters that later look historical.”

    Be careful when treating digitized corpora as authority, especially if they feed training data, search indexes, or standards work. For archival text, keep image provenance and higher-quality rescans around so you can challenge suspicious forms later.

      Attribution:
    • erjiang #1
    • lidavidm #1
  2. 02

    Unicode inherited older CJK baggage

    The mess predates Unicode. Large source works like the Kangxi Dictionary already carried questionable characters, and standards like CCCII had been grappling with Han character unification before Unicode adopted the problem. That makes these ghost entries less like a modern standards failure and more like technical debt from centuries of lexicography and decades of incompatible encoding efforts.

    If you work on multilingual text systems, expect legacy data assumptions to survive every rewrite. Migration plans need historical compatibility analysis, not just a switch to modern Unicode libraries.

      Attribution:
    • hnfong #1
    • gsnedders #1
  3. 03

    Han unification is really a rendering problem

    The practical failure mode is not just that some characters were merged and others were split. The same code point can legitimately need different shapes across Chinese and Japanese contexts, while nearby forms that users think of as equivalent can still refuse to match in search. The issue shows up outside CJK too, but commenters stressed that East Asian scripts make the gap between abstract character and displayed glyph impossible to ignore.

    Audit any user-facing CJK workflow for font selection, locale tagging, and search behavior. If your app just stores code points and hopes the platform renders the right thing, you will ship culturally wrong text.

      Attribution:
    • hnfong #1
    • wodenokoto #1
    • yorwba #1
    • wl #1
  4. 04

    CJK OCR breaks on structure, not just rarity

    OCR is especially fragile here because many characters are built from reusable components called radicals. A small structural mistake can produce a plausible-looking but nonexistent character, and older OCR systems were too unreliable to cleanly disambiguate that across thousands of forms. That explains how fake characters can emerge from scanning even when the source page itself was merely degraded, not totally unreadable.

    For CJK document pipelines, do not trust OCR output at character level without validation against lexicons or human review. Component-aware error checking can catch failures that ordinary spellcheck or string matching will miss.

      Attribution:
    • gucci-on-fleek #1
    • Izkata #1
    • Kye #1

Against the grain

  1. 01

    Extra code points may be the right tradeoff

    Keeping bogus or obsolete characters in the standard may be better than omitting real ones. Once a script has centuries of source material and multiple conflicting registries, over-inclusion preserves round-tripping and lets archivists represent what appears in historical data, even if some entries later turn out to be mistakes.

    When designing internal schemas or interchange formats, bias toward preserving source distinctions first and cleaning them up in higher layers. Lossy normalization is much harder to undo later.

      Attribution:
    • sedatk #1
  2. 02

    Unification was partly a reasonable engineering choice

    CJK unification was not pure malpractice. Commenters argued that collapsing tens of thousands of ideographs looked very different from keeping Latin and Cyrillic separate, especially when early systems were fixated on 16-bit character spaces and compatibility with older encodings. Some of Unicode’s ugly leftovers make more sense as constrained engineering decisions than as arbitrary disrespect for Asian languages.

    When you inherit an odd standard, look for the original system constraints before rewriting around it. Some awkward compatibility rules still exist because they solved real deployment limits that modern teams no longer remember.

      Attribution:
    • esrauch #1
    • gucci-on-fleek #1

In plain english

CCCII
Chinese Character Code for Information Interchange, an older encoding standard for Chinese characters developed before Unicode.
CJK
Chinese, Japanese, and Korean, often used when discussing shared character sets and text rendering.
glyph
The specific visual shape used to draw a character on screen or on paper.
Han unification
A Unicode design choice that encodes many Chinese, Japanese, and Korean ideographs with the same code point when they are considered the same underlying character despite regional shape differences.
Kangxi Dictionary
A major historical Chinese dictionary whose character listings influenced later standards work.
OCR
Optical Character Recognition, software that tries to read text from images or pixels.
radicals
Recurring graphical components used to build and classify many Chinese characters.
Taisho
A period in Japanese history from 1912 to 1926, used here to date a newspaper source.
Unicode
The dominant global text standard that assigns code points to characters across many languages and scripts.

Reference links

Background on Japanese NLP

  • fugashi
    A Python wrapper for MeCab mentioned while discussing the article author's broader work in Japanese language processing.
  • Japanese NLP book
    A book recommendation for English speakers who want practical background on Japanese natural language processing.
  • spaCy
    Named as another project the article author worked on, giving readers context on his NLP background.

CJK encoding history and standards

  • Ideographic Research Group
    Referenced to clarify that Han unification work involved Asian standards groups and predates Unicode-only decisions.

Art and cultural references

  • A Book from the Sky
    An artwork made entirely of invented Chinese-like characters, relevant as a cultural parallel to ghost characters.

Typography and character semantics

  • Ö
    Used in a side discussion to show that search, sorting, and equivalence problems are not unique to CJK scripts.