When “é” Is Not Equal to “é”: Canonical Equivalence Explained
It seems impossible, but two strings that look exactly the same on screen can be considered different by computers. This phenomenon occurs because Unicode defines multiple valid ways to represent the same visual character through a concept known as canonical equivalence.
When two sequences of code points are canonically equivalent, they must be treated as identical for comparison, searching, and sorting operations. However, they are not necessarily identical at the byte level, which causes problems in systems that perform direct string matching.
Real Examples in Everyday Use
A user on a Mac types their name “René” and saves a document. The operating system stores it in decomposed form. Later, a Windows-based web application receives the same text in composed form. A simple equality check between the two versions returns false, even though both display identically to humans.
Similar issues appear in URL handling, form validation, and authentication systems. A password entered with decomposed characters on one device may be rejected when compared against a stored composed version, frustrating users and creating support tickets.
The Root Cause
- Combining marks can be applied in different orders
- Precomposed characters exist alongside decomposed sequences
- Different platforms prefer different default forms
- Legacy text may use compatibility characters instead of standard ones
How to Solve It
The Unicode Standard provides clear rules for normalizing text to eliminate these differences. By converting all input to a single consistent form, applications can ensure reliable comparisons regardless of where or how the text was created.
Client-side tools that perform instant normalization help developers catch these issues early, before data enters databases or file systems. This preventive approach is far more effective than trying to handle mixed forms later in the pipeline.
Canonical equivalence is one of the most common sources of subtle text-processing bugs, but it is also one of the easiest to prevent with proper normalization.