The Name That Matched Twice: Unicode Normalization, Grapheme Clusters, and Getting Bengali Text Right

Picture a clinic’s patient registration system in Dhaka. A receptionist creates a record for সৈয়দ রহমান on the front-desk desktop. Three weeks later the same patient registers themselves through the mobile app before a follow-up visit. The duplicate-detection job compares names, finds no match, and creates a second record. Now there are two medical histories, two sets of lab results, and a doctor looking at half of each.

Nothing was misspelled. On screen the two names are pixel-identical. But one input path produced the letter য় as a single code point, U+09DF BENGALI LETTER YYA, and the other produced it as two: U+09AF BENGALI LETTER YA followed by U+09BC BENGALI SIGN NUKTA. The database compared bytes. Python’s == compares code points. JavaScript’s === compares UTF-16 code units. All three agree that these are different strings:

"\u09B8\u09C8\u09DF\u09A6" === "\u09B8\u09C8\u09AF\u09BC\u09A6"  // false; both render as সৈয়দ

The Unicode Standard disagrees. It says these two sequences are canonically equivalent, the same abstract text that should look and behave identically¹. The gap between “the standard says they are equal” and “my code says they are different” is where duplicate records, account takeovers, failed searches and silently broken uniqueness constraints come from.

This post is about closing that gap, with Bengali as the running example because that is the text AiCE-Lab’s systems actually handle, and because Bengali hits nearly every hard case Unicode has: composition exclusions, two-part vowel signs, conjuncts, a historically re-encoded letter, a second set of decimal digits, and joiners that carry meaning. The lessons apply to any script. If you only ever store ASCII, you can stop reading; nobody in Bangladesh only stores ASCII.

We cover what equivalence means, what each normalization form does, the Bengali characters that NFC will not put back together, the legacy encoding NFC cannot fix at all, grapheme clusters and the Unicode 15.1 conjunct rule, digits, confusables, the PRECIS framework for identifiers and its idempotence rule, tested code in Python and JavaScript, and what PostgreSQL can enforce for you. A separate post covered Unicode in passwords, where the rules are different on purpose. This one is about everything else: names, usernames, search keys, and anything you put a unique index on.

Two kinds of “the same”

Unicode Standard Annex #15, Unicode Normalization Forms, defines two equivalence relations. The current revision is 58, published 2026-08-12 for Unicode 18.0, edited by Ken Whistler¹.

Canonical equivalence is the strong one:

“Canonical equivalence is a fundamental equivalency between characters or sequences of characters which represent the same abstract character, and which when correctly displayed should always have the same visual appearance and behavior.”¹

The Latin example everyone knows is é as U+00E9 versus e + U+0301 COMBINING ACUTE ACCENT. The Bengali examples are more interesting and come up below.

Compatibility equivalence is the weak one:

“Compatibility equivalence is a weaker type of equivalence between characters or sequences of characters which represent the same abstract character (or sequence of abstract characters), but which may have distinct visual appearances or behaviors.”¹

Examples are the fi ligature versus fi, fullwidth Ａ versus A, superscript ² versus 2. These differ visibly, but a search engine probably wants to treat them as the same.

The four normalization forms are the four combinations of two choices: decompose or compose, and canonical only or canonical plus compatibility.

Two strings are canonically equivalent if and only if their NFC forms are identical (equivalently, their NFD forms). That gives you a mechanical test: normalize both sides to the same form, then compare bytes.

The rule most teams should start with is this. Store NFC, compare NFC, and use NFKC only for derived keys such as search terms, never for the text you store and display. NFKC throws away distinctions that sometimes matter; ² and 2 are not the same thing in a chemical formula. NFC throws away only differences the user could not see.

The stability promise

The reason NFC is safe to use as a storage form is a promise in UAX #15:

“if a string that does not have any unassigned characters is normalized under one version of Unicode, it must remain normalized under all future versions of Unicode.”¹

Normalization results for assigned characters do not change between versions. A key you computed in 2019 with Unicode 12 is still the right key under Unicode 18. That is not true of everything in Unicode: confusable mappings and segmentation rules change from version to version, as we will see. Normalization does not, and that is why it can sit under a unique index.

Note the qualifier “does not have any unassigned characters”. A code point that is unassigned in your runtime’s Unicode version passes through normalization unchanged, and a later runtime that knows the character might normalize it differently. Python 3.13’s unicodedata reports version 15.1.0; Node 22 with its bundled ICU 78.3 reports Unicode 17.0. If your identifiers can contain code points assigned after 15.1, two services in the same system can disagree. The defensive answer, which we use below, is to reject code points outside a deliberate allowlist.

Bengali under NFC: what gets composed and what does not

Here is what Python 3.13 does to the Bengali characters that have canonical decompositions. Every row was produced by unicodedata.normalize on the machine this post was written on:

The first two rows behave as you would expect. The vowel sign ো is written in two parts, one on each side of the consonant. The Unicode core specification describes U+09CB and U+09CC as two-part vowel signs, and U+09D7 BENGALI AU LENGTH MARK as the right-hand piece of ৌ². A user who types ে then া gets the decomposed sequence, and NFC recomposes it into the single code point. কে + া becomes কো. Good.

The last three rows are the surprise. NFC does not compose ড়, ঢ়, and য় into their single code points. It decomposes them. The NFC form of U+09DF is the two-code-point sequence য + ◌়, and it stays that way.

The reason is the composition exclusion list. UAX #15 section 5 says that “no composition exclusion character can occur in any normalized form of Unicode text: NFD, NFC, NFKD, or NFKC”¹. One category of exclusion is “script-specific”: “certain canonically decomposable characters whose decomposition includes one of a small set of combining marks for particular Indian scripts, for Tibetan, or for Hebrew”¹. The nukta is one of those marks. The list was “defined at the time of the composition version in 2001 and unchanged since that version”¹. The current CompositionExclusions.txt still lists all three under “(1) Script Specifics”³:

09DC    #  BENGALI LETTER RRA
09DD    #  BENGALI LETTER RHA
09DF    #  BENGALI LETTER YYA

So in a correctly normalized Bengali database, there is no U+09DF anywhere. Three practical consequences follow.

Allowlists written as character lists break. If your validation says “a Bengali name may contain these 50 characters” and the list includes য় typed as one code point, then normalizing the input first means U+09DF never arrives, and the decomposed form is not on the list. If you validate before normalizing, you accept both and store them differently. Validate on the normalized form, and write the allowlist as code point ranges plus combining marks, not as a string of letters.

Byte-level sorting moves. In code point order, U+09DF sorts after হ (U+09B9), near the end of the consonants. Its NFC form begins with U+09AF and sorts right after য. I checked this in Node: sorting ["য়া", "হ", "য"] by code units puts য়া last when it is precomposed and in the middle when it is decomposed. An ICU collator (new Intl.Collator("bn")) compares the two forms as equal, which is the behaviour you want. Never use code point order as a user-visible sort for Bengali.

Your test fixtures may be lying to you. If a developer types test data with a keyboard that emits U+09DF, and the code under test normalizes, the assertion assert stored == "সৈয়দ" fails. Normalize the expected value too, or write fixtures with explicit escapes.

The khanda ta that no normalization form will fix

ৎ (khanda ta) is the dead form of ত, used in words like উৎস (source) and হঠাৎ (suddenly). Its encoding history is a trap. From section 12.2.1 of the Unicode core specification:

“Versions of the Unicode Standard prior to Version 4.1 recommended that khanda ta be represented as the sequence <U+09A4 ত BENGALI LETTER TA, U+09CD ◌্ BENGALI SIGN VIRAMA, U+200D ZERO WIDTH JOINER> in all circumstances.”²

“U+09CE ৎ BENGALI LETTER KHANDA TA should instead be used explicitly in newly generated text, but users are cautioned that instances of the older representation may exist.”²

U+09CE has no decomposition, and the old three-code-point sequence is not canonically equivalent to it. NFC leaves both exactly as they are, and an ICU collator compares them as different. Any Bengali text that went through software older than Unicode 4.1, or through a modern input method that still emits the old sequence, can contain a khanda ta that will never match a modern one under any standard normalization.

This is the general lesson that Bengali teaches early: normalization is the standard’s definition of equivalence, not yours. Where your users consider two encodings the same and Unicode does not, you need an explicit, documented, application-level mapping. Ours maps U+09A4 U+09CD U+200D to U+09CE in identifier keys. We do not apply it to stored display text, because rewriting what a user typed is a separate decision with its own consequences.

Joiners: the invisible characters that mean something

ZERO WIDTH JOINER (U+200D) and ZERO WIDTH NON-JOINER (U+200C) are invisible, and in Latin-script identifiers they are almost always an attack: two usernames that render identically but differ by an invisible character. The tempting fix is to strip them.

In Bengali they carry meaning. The legacy khanda ta above is one case, and joiners also select between conjunct and non-conjunct renderings of the same consonant cluster. Stripping them can change how a word displays.

UTS #39 handles this in its confusable skeleton by removing default-ignorable code points, which include both joiners⁴. That is the right move for a comparison key used to detect lookalikes. It is the wrong move for stored text.

For identifiers, our policy is narrower than “allow” or “strip”. A joiner is allowed only immediately after a virama (U+09CD), the one position where it has a defined effect in Bengali, and rejected anywhere else. That blocks ক‍খ (a ZWJ wedged between two consonants, which renders the same as কখ) while still allowing the joiner sequences that actually do something.

Grapheme clusters: what a “character” is to the person typing it

Ask a developer how long the word স্বাস্থ্য (health) is:

"স্বাস্থ্য".length                   // 9  (UTF-16 code units)
[..."স্বাস্থ্য"].length              // 9  (code points)
new TextEncoder().encode("স্বাস্থ্য").length  // 27 (UTF-8 bytes)

A Bengali reader sees two units, স্বা and স্থ্য. UAX #29, Unicode Text Segmentation, calls these extended grapheme clusters and is explicit about why they matter: “In implementations, the notion of user-perceived characters corresponds to the concept of grapheme clusters.”⁵

Every place your code counts, cuts, or steps through text should be asking what a grapheme is, not what a code unit is:

• Truncation. "ক্ষমা".slice(0, 2) returns ক্, a consonant with a dangling virama. The user’s name now ends in a half-letter in every email subject and notification preview.
• Length limits. A 20-character limit on a display name, counted in code units, allows 20 Latin letters but far fewer Bengali syllables: বাংলাদেশ is 8 code units for 4 clusters, and স্বাস্থ্য is 9 for 2. Users experience that as unfairness, not as a technical limit.
• Cursor movement and deletion in custom text inputs.
• Reversing, masking, or highlighting substrings.

Unicode 15.1 changed the answer for conjuncts

For most of Unicode’s history, the default grapheme rules split Indic conjuncts. ক্ষ (ka + virama + ssa) came out as two clusters, ক্ and ষ, because nothing in the default rules said a virama joins the following consonant. Text editors that followed the defaults put the cursor in the middle of a conjunct.

Unicode 15.1, released 2023-09-12, fixed this: “There is a new grapheme cluster segmentation rule GB9c in UAX #29 which refers to a new enumerated property Indic_Conjunct_Break.”⁶ The rule, as written in the current UAX #29 (revision 49, 2026-09-01), is:

GB9c:  \p{InCB=Linker} \p{InCB=Extend}* × \p{InCB=Consonant}

In words: do not break between a linker (a virama), any intervening extending marks, and a following consonant⁵.

The initial InCB=Linker set in Unicode 15.1 is six viramas, and Bengali is one of them⁷:

094D  ; InCB; Linker  # DEVANAGARI SIGN VIRAMA
09CD  ; InCB; Linker  # BENGALI SIGN VIRAMA
0ACD  ; InCB; Linker  # GUJARATI SIGN VIRAMA
0B4D  ; InCB; Linker  # ORIYA SIGN VIRAMA
0C4D  ; InCB; Linker  # TELUGU SIGN VIRAMA
0D4D  ; InCB; Linker  # MALAYALAM SIGN VIRAMA

The 15.1 release notes say the “list of scripts affected by this rule is expected to expand in subsequent versions”⁶. That sentence is your warning. Grapheme segmentation is not stable across Unicode versions, unlike normalization. A runtime on Unicode 15.0 or earlier counts ক্ষমা as three clusters; one on 15.1 or later counts it as two. Two services in the same system can count the same string differently.

Here is what Node 22 (ICU 78.3, Unicode 17.0) returns:

const seg = new Intl.Segmenter("bn", { granularity: "grapheme" });
[...seg.segment("ক্ষমা")].map(s => s.segment)      // ["ক্ষ", "মা"]
[...seg.segment("স্বাস্থ্য")].map(s => s.segment)  // ["স্বা", "স্থ্য"]
[...seg.segment("সৈয়দ")].map(s => s.segment)      // ["সৈ", "য়", "দ"]
[...seg.segment("বাংলাদেশ")].map(s => s.segment)   // ["বাং", "লা", "দে", "শ"]

So never store a grapheme count as a hard constraint. Enforce byte length in storage, because bytes are what the column holds and bytes do not change with Unicode versions. Use grapheme count only for UX, such as counters, truncation, and soft limits, and compute it at the edge where it is shown.

Intl.Segmenter is the tool in JavaScript; MDN reports it as available across browsers since April 2024⁸. Python’s standard library has no grapheme segmenter; use ICU bindings (PyICU) or the third-party regex module’s \X, and check which Unicode version the one you pick implements.

Digits: the validator that disagrees with itself

Bengali has its own decimal digits, ০ through ৯, at U+09E6 to U+09EF. Bangladeshi users type them, often without thinking about it, into phone number, NID and amount fields. Look at what two mainstream runtimes do with them:

>>> int("১২৩")
123
>>> import re
>>> bool(re.fullmatch(r"\d+", "১২৩"))
True
>>> bool(re.fullmatch(r"\d+", "১২৩", re.ASCII))
False

Number("১২৩")            // NaN
parseInt("১২৩")          // NaN
/^\d+$/.test("১২৩")      // false
/^\p{Nd}+$/u.test("১২৩") // true

Python’s \d on str patterns and int() accept every Unicode decimal digit. JavaScript’s \d and Number() accept only ASCII. A Python service validates a phone number with \d{11}, passes it on, and a Node service downstream gets NaN. Or the reverse: a front end rejects input the backend would have accepted. Worse, a Python validator that accepts \d is accepting Devanagari, Arabic-Indic, fullwidth, and dozens of other digit sets as well, which is rarely what anyone intended.

Decide at the boundary. For numeric fields we accept exactly two digit sets, ASCII and Bengali, convert to ASCII immediately, and reject everything else rather than guessing. The ascii_digits function in the module below does exactly this. The stored value is always ASCII, so every downstream service agrees on it.

Confusables: when different strings look the same

Normalization answers “do these encode the same text?” A different question is “would a human tell these apart?” paypal with a Cyrillic а (U+0430) is not canonically or compatibly equivalent to the Latin version. NFKC leaves it alone. It is still an impersonation.

UTS #39, Unicode Security Mechanisms (revision 34, 2026-08-27, for Unicode 18.0), defines a confusable test based on a skeleton function⁴. Roughly, it converts to NFD, removes default-ignorable code points, maps each character to a prototype from the confusables.txt data, and reapplies NFD. Then:

“The strings X and Y are then defined to be confusable if and only if skeleton(X) = skeleton(Y).”⁴

Two warnings in the spec matter for anyone building on this. Skeletons “are not intended for display. Further, they are not stable across versions of Unicode”⁴. And “A skeleton is intended only for internal use for testing confusability of strings”⁴. Do not store a skeleton as the user’s identifier. If you store skeletons in an index for lookalike detection, record the Unicode version and plan to rebuild the index when you upgrade.

UTS #39 also defines restriction levels for identifiers, from strictest to loosest: ASCII-Only, Single Script, Highly Restrictive, Moderately Restrictive, Minimally Restrictive, and Unrestricted⁴. Highly Restrictive allows a single script, or a few specific combinations that are normal in East Asian writing, such as Latin with Han and Kana. Most real-world spoofing is mixed-script, so a cheap policy catches most of it: one letter script per identifier. A username can be all Latin or all Bengali, with ASCII digits and a little punctuation, but not hasibর.

This does not catch whole-script confusables, a name written entirely in one script that looks like a name in another. For high-value namespaces such as organisation slugs, payment handles, and anything users rely on to identify each other, run a skeleton check against existing names at registration time. ICU’s SpoofChecker implements UTS #39.

PRECIS: the IETF’s answer for identifiers

If you need a standard to point at in a design review, it is PRECIS. RFC 8264, PRECIS Framework: Preparation, Enforcement, and Comparison of Internationalized Strings in Application Protocols, by Peter Saint-Andre and Marc Blanchet (October 2017, obsoleting RFC 7564), defines two base string classes: IdentifierClass for things like usernames, and FreeformClass for things like passwords and display names⁹. RFC 8265, by Saint-Andre and Alexey Melnikov (October 2017, obsoleting RFC 7613), builds username and password profiles on top¹⁰.

The UsernameCaseMapped profile in RFC 8265 section 3.3.1 applies, in order¹⁰:

1. Width mapping: “Map fullwidth and halfwidth code points to their decomposition mappings.”
2. No additional mapping.
3. Case mapping: “Map uppercase and titlecase code points to their lowercase equivalents, preferably using the Unicode toLowerCase() operation.”
4. Normalization: “Apply Unicode Normalization Form C (NFC) to all strings.”
5. Directionality: apply the Bidi Rule from RFC 5893.

Two things about this list are worth noticing. The normalization form is NFC, not NFKC; only width differences are folded, not every compatibility distinction. And there is a case-mapping step even though Bengali has no case. A username system that accepts both Latin and Bengali still has to fold Hasib and hasib together.

Idempotence, and the 2013 Spotify bug

The most important sentence in RFC 8264 is easy to miss. Section 7 says that “applying the rules for any given PRECIS profile is not necessarily an idempotent procedure”, so “an implementation SHOULD apply the rules repeatedly until the output string is stable”, and it fails the string if it has not stabilised after “reapplying the rules three (3) additional times after the first application”⁹.

Why would canonicalisation not be idempotent? Spotify’s engineering blog has the best explanation anyone has written. In June 2013 Mikael Goldmann published “Creative usernames and Spotify account hijacking”¹¹. Spotify canonicalised usernames with nodeprep.prepare from Twisted’s XMPP stringprep module. The username ᴮᴵᴳᴮᴵᴿᴰ, built from modifier letters, canonicalised to BIGBIRD, and canonicalising that produced bigbird. The function “was not idempotent”¹¹.

The attack, which a user reported on Spotify’s support forum, followed from that. Register ᴮᴵᴳᴮᴵᴿᴰ. Its canonical form BIGBIRD does not collide with the existing bigbird, so registration succeeds. Request a password reset: the name is canonicalised once, so the email goes to the attacker’s own address. Then use the link. The reset code canonicalised the name again, which yields bigbird, so “the new password was instead set for the ‘bigbird’ account”¹¹. The root cause was that “the code points making up ᴮᴵᴳᴮᴵᴿᴰ are not part of unicode 3.2” — the version stringprep was defined against — and the bug “came into being when upgrading from python 2.4 to python 2.5”, whose unicodedata stopped raising on those inputs¹¹. Spotify’s fix rejected any name where old_prepare(old_prepare(name)) != old_prepare(name)¹¹.

Take two rules from this:

1. Your canonical function must be a fixed point on its own output. Test it: canon(canon(x)) == canon(x) for every stored key, and assert it at write time.
2. Canonicalise in exactly one place, exactly once per comparison. If registration, login, reset, and the admin search each canonicalise separately, even a small difference between them is an identity bug.

UTS #46, the IDNA compatibility processing used for domain names, states the property directly: “The processing is idempotent—reapplying the processing to the output will make no further changes.”¹² That is the property to require of your own canonicaliser.

A canonicalisation module you can defend in review

Below is a small, dependency-free Python module that implements the policy described above for identifiers that may be Latin or Bengali. It is deliberately narrow: an allowlist of what we expect, rather than a general Unicode identifier engine. It is not a conforming PRECIS implementation. It borrows PRECIS’s ordering, NFC, and stabilise-then-assert rule, and adds the Bengali-specific decisions.

"""Canonical forms for identifiers that may contain Bengali text.

Policy, not magic: every rule below is a decision you should be able to
defend in review. Standard library only (Python 3.9+).
"""
import unicodedata

BENGALI = range(0x0980, 0x0A00)          # Bengali block
BN_DIGITS = range(0x09E6, 0x09F0)        # ০..৯
ZWJ, ZWNJ, VIRAMA = "\u200D", "\u200C", "\u09CD"
LEGACY_KHANDA_TA = "\u09A4\u09CD\u200D"  # ত + ্ + ZWJ (pre-Unicode 4.1)
KHANDA_TA = "\u09CE"                     # ৎ


class Rejected(ValueError):
    pass


def ascii_digits(s: str) -> str:
    """For numeric fields (phone, NID, amount): accept ASCII or Bengali
    digits, return ASCII. Any other digit script is rejected, not guessed."""
    out = []
    for ch in s:
        if "0" <= ch <= "9":
            out.append(ch)
        elif ord(ch) in BN_DIGITS:
            out.append(chr(ord(ch) - 0x09E6 + 0x30))
        else:
            raise Rejected(f"not a digit we accept: U+{ord(ch):04X}")
    return "".join(out)


def _script_of(ch: str) -> str:
    cp = ord(ch)
    if cp < 0x80:
        return "digit" if ch.isdigit() else "latin" if ch.isalpha() else "ascii-other"
    if cp in BENGALI:
        return "digit" if cp in BN_DIGITS else "bengali"
    if ch in (ZWJ, ZWNJ):
        return "joiner"
    return "other"


def _one_pass(s: str) -> str:
    s = unicodedata.normalize("NFC", s)
    s = s.replace(LEGACY_KHANDA_TA, KHANDA_TA)   # application policy, not NFC
    s = s.casefold()                             # Bengali has no case; Latin does
    return unicodedata.normalize("NFC", s)


def canonical_username(raw: str) -> str:
    s = raw.strip()
    # Apply until stable: RFC 8264 section 7 warns profiles are not
    # necessarily idempotent. Give up after three extra passes.
    for _ in range(4):
        nxt = _one_pass(s)
        if nxt == s:
            break
        s = nxt
    else:
        raise Rejected("canonical form did not stabilise")

    if not s or len(s.encode("utf-8")) > 64:
        raise Rejected("empty or longer than 64 bytes")
    scripts = set()
    prev = ""
    for ch in s:
        kind = _script_of(ch)
        if kind in ("other", "ascii-other") and ch not in "._-":
            raise Rejected(f"code point not allowed: U+{ord(ch):04X}")
        if kind == "joiner" and prev != VIRAMA:
            raise Rejected("ZWJ/ZWNJ only allowed after a virama")
        if kind in ("latin", "bengali"):
            scripts.add(kind)
        prev = ch
    if len(scripts) > 1:
        raise Rejected("mixing Latin and Bengali letters in one name")
    assert _one_pass(s) == s   # the stored key must be a fixed point
    return s

Some decisions here deserve a sentence each:

• casefold() rather than lower(). Python’s casefold is Unicode default case folding, which is more aggressive than RFC 8265’s toLowerCase (for example, ß folds to ss). For a Latin-or-Bengali allowlist the difference rarely shows, and folding more is the safer direction for an identifier. Pick one and write it down; what matters is that every code path uses the same one.
• Fullwidth and everything outside the allowlist is rejected, not mapped. RFC 8265 maps fullwidth forms. We reject them because none of our users type them on purpose, and a rejection is easier to reason about than a mapping.
• The 64-byte limit is on UTF-8 bytes, for the reasons given in the grapheme section.
• NFC runs twice per pass, because case folding can produce strings that need recomposing.
• The final assert is the Spotify rule turned into code.

The tests, which pass on Python 3.13 (Unicode 15.1):

import unittest
from bntext import canonical_username, ascii_digits, Rejected

class T(unittest.TestCase):
    def test_nukta_forms_converge(self):
        a = "\u09B8\u09C8\u09DF\u09A6"          # সৈয়দ with precomposed য় (U+09DF)
        b = "\u09B8\u09C8\u09AF\u09BC\u09A6"    # সৈয়দ with য + nukta
        self.assertNotEqual(a, b)
        self.assertEqual(canonical_username(a), canonical_username(b))
    def test_two_part_vowel(self):
        self.assertEqual(canonical_username("\u0995\u09C7\u09BE"),   # ক + ে + া
                         "\u0995\u09CB")                     # কো
    def test_legacy_khanda_ta(self):
        self.assertEqual(canonical_username("\u0989\u09A4\u09CD\u200D\u09B8"),   # legacy উৎস
                         canonical_username("\u0989\u09CE\u09B8"))
    def test_case_and_trim(self):
        self.assertEqual(canonical_username("  Hasib.R "), "hasib.r")
    def test_mixed_script_rejected(self):
        with self.assertRaises(Rejected):
            canonical_username("hasibর")
    def test_cyrillic_rejected(self):
        with self.assertRaises(Rejected):
            canonical_username("p\u0430ypal")       # Cyrillic а
    def test_stray_zwj_rejected(self):
        with self.assertRaises(Rejected):
            canonical_username("\u0995\u200D\u0996")   # ক + ZWJ + খ
    def test_idempotent(self):
        for s in ["\u09B8\u09C8\u09DF\u09A6", "Hasib", "\u0995\u09CD\u09B7\u09AE\u09BE"]:
            c = canonical_username(s)
            self.assertEqual(canonical_username(c), c)
    def test_digits(self):
        self.assertEqual(ascii_digits("০১৭১১২২৩৩৪৪"), "01711223344")
        self.assertEqual(ascii_digits("01711"), "01711")
        with self.assertRaises(Rejected):
            ascii_digits("\u0967\u0968")      # Devanagari digits
        with self.assertRaises(Rejected):
            ascii_digits("\uFF11")            # fullwidth 1

unittest.main()

Write test data with \u escapes, as above. It is the only way to be sure which code points a test contains, and it survives editors, terminals, and copy-paste that silently normalize.

Grapheme-safe truncation in JavaScript

The front end needs the other half: never cut text inside a cluster. This helper respects both a grapheme budget, for what the user sees, and a byte budget, for what the column holds:

const seg = new Intl.Segmenter("bn", { granularity: "grapheme" });

export function graphemeLength(s) {
  let n = 0;
  for (const _ of seg.segment(s)) n++;
  return n;
}

// Cut to at most `max` user-perceived characters AND `maxBytes` UTF-8 bytes,
// never splitting a cluster (no dangling ্, no orphaned vowel sign).
export function truncate(s, max, maxBytes = Infinity) {
  const enc = new TextEncoder();
  let out = "", n = 0, bytes = 0;
  for (const { segment } of seg.segment(s.normalize("NFC"))) {
    const b = enc.encode(segment).length;
    if (n + 1 > max || bytes + b > maxBytes) break;
    out += segment; n++; bytes += b;
  }
  return out;
}

On Node 22:

"ক্ষমা".slice(0, 2)            // "ক্"   — broken
truncate("ক্ষমা", 1)            // "ক্ষ"  — whole conjunct
truncate("বাংলাদেশ", 3)         // "বাংলাদে"
truncate("বাংলাদেশ", 10, 10)    // "বাং"  — byte budget wins (বাংলাদেশ is 24 bytes)

Truncate on the server too, with the same rule, if any server-side code shortens names for SMS, push notifications, or PDF headers. Those are exactly the places where nobody checks the output by eye.

Holding the line in PostgreSQL

Application code is the first line of defence; the database should be the last. PostgreSQL has had the tools since version 13, whose release notes added “SQL functions NORMALIZE() to normalize Unicode strings, and IS NORMALIZED to check for normalization”¹³. Both “can only be used when the server encoding is UTF8”¹⁴.

A constraint that rejects un-normalized text means a code path that forgot to normalize fails loudly at write time instead of creating a duplicate quietly:

CREATE TABLE patient (
    id            uuid PRIMARY KEY,
    display_name  text NOT NULL CHECK (display_name IS NFC NORMALIZED),
    name_key      text NOT NULL,   -- canonical_username()-style key, set by the app
    phone         text CHECK (phone ~ '^[0-9]{11}$')   -- ASCII only, after ascii_digits()
);
CREATE UNIQUE INDEX patient_name_key ON patient (name_key);

The documentation notes that “checking for normalization using this expression is often faster than normalizing possibly already normalized strings”¹⁴, so a CHECK costs little. Note that the regex uses an explicit [0-9] class, not \d.

For case- or accent-insensitive comparison without a separate key column, PostgreSQL offers nondeterministic ICU collations. The docs give this example¹⁵:

CREATE COLLATION case_insensitive (
    provider = icu, locale = 'und-u-ks-level2', deterministic = false
);

A nondeterministic collation “may determine strings to be equal even if they consist of different bytes”, including canonically equivalent ones. It comes with costs that the documentation states plainly: “their use leads to a performance penalty”, B-tree indexes “cannot use deduplication”, and “certain operations are not possible with nondeterministic collations, such as some pattern matching operations”¹⁵. The same page suggests the alternative we prefer: use normalize and is normalized “to preprocess or check the strings, instead of using nondeterministic collations”¹⁵.

Our recommendation is an explicit key column computed by one application function, plus a CHECK on the display column. It is boring and visible, and it keeps the policy in code you can test, rather than in a collation that behaves differently from your application’s comparisons. (The SQL above follows the PostgreSQL 18 documentation. Run it against your own server version before you rely on it.)

A checklist for Monday

1. Pick NFC as the storage form for all user text, normalize at the input boundary, and add CHECK (... IS NFC NORMALIZED) to the columns that matter.
2. Compute identifier keys in one function, apply it until stable, assert canon(canon(x)) == canon(x), and use it in every path: registration, login, reset, search, admin tools, imports.
3. Write allowlists as code point ranges against normalized text. Remember that U+09DC, U+09DD and U+09DF never appear in NFC.
4. Map legacy khanda ta (U+09A4 U+09CD U+200D → U+09CE) in keys and search, and document the mapping as your policy, not Unicode’s.
5. Convert digits at the boundary: ASCII and Bengali in, ASCII out, everything else rejected. Grep your code for \d and int( on user input.
6. Enforce byte limits in storage and grapheme limits in UX. Never slice user text on code units.
7. One letter script per identifier, plus a UTS #39 skeleton check for namespaces people rely on to recognise each other.
8. Pin and record your Unicode versions. Know which version each runtime in your system implements, and test that they agree on the strings that matter to you.
9. Write test fixtures with escapes, and include both forms of every Bengali character that has two.

Closing

There is a comfortable belief that Unicode is solved: use UTF-8 everywhere and move on. UTF-8 solves encoding. It does nothing about identity, the question of when two strings are the same, and that question has three different answers in the standard (canonical, compatibility, confusable) and a fourth in your product requirements.

Bengali does not make the problem harder so much as it makes it impossible to ignore. A Latin-only team can ship for years before a precomposed é collides with a decomposed one. A team handling Bengali names will meet a decomposed য়, a legacy khanda ta, and a phone number in Bengali digits in its first month. Those teams have the advantage, if they take it, of building the discipline early.

That discipline is small. Normalize to NFC at the boundary. Canonicalise identifiers in one function that is idempotent and tested. Add the mappings the standard does not make, and write them down as policy. Make the database refuse what the application forgot. Count graphemes for people and bytes for machines.

The clinic’s duplicate record was not a Unicode bug. Unicode said clearly that the two names were the same. The bug was a system that never asked.

References