GUIDE · UNICODE
Unicode, emoji and why identical-looking text is not identical
Two lines can look identical on screen and still be different text. That single fact explains most of the strange behaviour people run into with sorting, duplicate detection and character counts on non-English input. This guide explains the three levels at which text is measured, why an emoji is not always one character, and the normalization trap that catches even experienced developers.
Characters, code points and bytes
Three different things get called “a character”, and confusing them is the root of most Unicode surprises.
- A byte is the storage unit. In UTF-8, an ASCII letter takes one byte, most Latin accented letters take two, most CJK characters take three, and emoji usually take four.
- A code point is a single Unicode value, written as U+ followed by four to six hex digits. This is the level the counters on this site use, because it is the level that corresponds to one visible symbol in the common case.
- A grapheme cluster is what a reader perceives as one character. It can be built from several code points, which is exactly how emoji with skin tones and flags work.
So “café” is four grapheme clusters, but it might be four code points or five, depending on how the accent was typed — and the byte count will differ again. None of those numbers is wrong; they are answers to different questions.
Why one emoji can be several characters
An emoji that looks like a single symbol is often a sequence. A few common patterns:
| Emoji | How it is built | Code points |
|---|---|---|
| 😀 | A single code point | 1 |
| 😀️ | Base emoji plus a variation selector | 2 |
| 👍🏻 | Base emoji plus a skin-tone modifier | 2 |
| 👨👩👦 | Three emoji joined by zero-width joiners | 5 |
| 🇹🇼 | Two regional indicator symbols forming a flag | 2 |
This matters for anything that counts or compares. A character count reports the code point total, so a family emoji that looks like one symbol can count as five characters. A deduplication pass treats two visually identical emoji as different text if one carries a skin-tone modifier and the other does not. Both behaviours are correct at the level of the text, and both surprise people who are looking at the screen rather than the code points.
What each action does with non-ASCII text
| Action | Behaviour with Unicode |
|---|---|
| Clean spacing | Trims each line and collapses runs of spaces and tabs. An internal non-breaking space is not collapsed, because it is a different character. |
| Remove blank lines | Removes lines whose trimmed content is empty. Format characters such as a zero-width space survive. |
| Remove duplicate lines | Compares code points exactly. No normalization is applied, so visually identical but differently-encoded text is not treated as a duplicate. |
| Sort lines A–Z | Uses locale-aware comparison with case and accent ignored, so é and e compare as equal. |
| Case actions | Unicode-aware: they affect letters that have case and leave scripts without case unchanged. |
| Counters | Characters are counted as code points; words are split on whitespace. |
The normalization trap
Unicode allows the same visual character to be encoded in more than one way. The letter é can be a single code point (U+00E9) or the letter e followed by a combining acute accent (U+0065 U+0301). Both render identically. Neither is wrong. But a code-point comparison sees two different strings, so:
- Deduplication does not catch the pair, and you get what looks like a duplicate that will not go away.
- Sorting may place the two forms in different positions, because their code points differ.
- A character count differs by one, because one form is longer.
- An exact-match search for one form fails to find the other.
The standard fix is to normalize text to one form before comparing it — usually the composed form, in which a base letter and its accent become a single code point. This tool does not normalize, deliberately: silent normalization would change the bytes of your text without telling you. The honest workaround is to recognize the symptom. If two lines look identical and refuse to deduplicate, or a character count is one higher than you expect, an encoding difference is the likely cause. Re-typing the accented character, or normalizing the file at its source, resolves it.
Case, collation and script-specific behaviour
Not every script behaves like Latin text, and the differences are worth knowing before you conclude that something is broken. Case actions have nothing to change in Chinese, Japanese, Korean, Arabic, Hebrew or Thai, because those scripts do not distinguish upper and lower case; the action runs and the text comes back identical. Sorting is locale-aware, so the same list of names can come out in a different order depending on the language the browser is set to, and a language whose collation rules you do not know may place characters in a sequence that looks arbitrary but follows a published standard. Character counts, by contrast, are consistent everywhere, which is one reason they are the most portable number for non-Latin text.
Right-to-left scripts add a further wrinkle that is a display matter rather than a text matter. Arabic and Hebrew pasted into a left-to-right editor can appear to have its punctuation in the wrong place, because punctuation is a neutral character being resolved by the surrounding direction. The underlying code points are correct; only the rendering order changes. If the output of an action looks scrambled in the editor while the character count is unchanged, direction resolution is the likely explanation rather than data loss.
Testing Unicode handling in your own text
A five-line test tells you most of what you need to know about how any tool handles your text. Put one line of plain Latin text, one line with an accented character, one line with a CJK character, one line containing an emoji, and one line containing a non-breaking space. Paste the block, note the character count, then run each action in turn and watch what moves.
If the accented and CJK lines survive every action unchanged, the tool is safe for your script. If the emoji line changes length under a case action, the tool is working at the code-point level rather than the byte level, which is what you want. And if a line containing only the non-breaking space refuses to disappear when you remove blank lines, you have found the boundary described in the invisible characters guide — useful to know before you trust the same tool on a real file.
Practical advice for mixed-script text
- Trust the counters over your eyes. If the character count seems high for what you see, there are invisible or multi-code-point characters present.
- For CJK text, count characters, not words. Word counting splits on whitespace, and CJK text has no spaces, so a whole sentence counts as one word.
- Expect emoji to break exact-match logic. Two emoji that look the same may carry different modifiers. Deduplicate on a plain-text version if the emoji are decoration.
- Normalize at the source when you can. A database import, an export script or a form field is the right place to fix encoding differences, because it is a one-time fix rather than a repeated manual step.
- Do not assume a case conversion will do anything. Scripts without case are left unchanged, which is correct but can look like the button failed.
Frequently asked questions
Why is my emoji counted as five characters?
Because the counter reports code points, and some emoji are sequences — a family emoji built from three base emoji joined by zero-width joiners is five code points. It looks like one symbol and counts as five characters.
Why does deduplication not remove two lines that look identical?
Most often because they differ in encoding rather than in appearance. An accented letter can be one code point or two, and the comparison is exact. Re-typing the character or normalizing the file at its source resolves it.
Does sorting work for languages other than English?
Yes. The comparison is locale-aware, so accented letters sort according to the browser's locale rules. Two browsers set to different languages can order the same list slightly differently, and both results are correct.
Can the tool convert between Unicode normalization forms?
No. It leaves your text exactly as it is, because silent normalization would change characters without telling you. Normalize in the system that produces the data.