GUIDE · UNICODE
Invisible characters: the whitespace that is not a space
Some of the most frustrating text problems come from characters you cannot see. A value that will not match, a line that will not delete, a field that is one character over its limit — all three are usually an invisible character doing something ordinary. This guide names the characters involved, states exactly which ones the tool removes, and gives a way to find them without special software.
The characters you will actually meet
The list is short, and each entry behaves differently. The last column states what happens when you run Clean spacing followed by Remove blank lines.
| Name | Code point | What it does | Cleaning result |
|---|---|---|---|
| Space | U+0020 | The ordinary space | Collapsed and trimmed |
| Tab | U+0009 | Alignment in plain text | Collapsed to one space, trimmed at the edges |
| Non-breaking space | U+00A0 | A space that stops a line from wrapping | Trimmed at the edges; kept inside the line |
| Narrow no-break space | U+202F | A narrower non-breaking space, common in French and in some number formats | Trimmed at the edges; kept inside the line |
| En space, em space | U+2002, U+2003 | Fixed-width spaces | Trimmed at the edges; kept inside the line |
| Zero-width space | U+200B | A break opportunity with no width | Not removed |
| Zero-width joiner | U+200D | Joins characters into one glyph, used in emoji | Not removed |
| Soft hyphen | U+00AD | A hidden hyphenation point | Not removed |
| Byte order mark | U+FEFF | A legacy encoding marker; sometimes appears mid-file | Trimmed at the edges only |
The pattern is that trimming handles the Unicode space separators — the ones the Unicode standard classifies as whitespace — while the format characters such as the zero-width space and the soft hyphen are not whitespace at all and survive untouched.
Why they cause so much trouble
- Duplicate detection fails. Two lines that look the same differ by a code point, so the exact comparison keeps both. This is the most common symptom.
- Lookups and joins fail. A database value with a trailing non-breaking space does not match the same value without it, even though the screen shows no difference.
- Sorting looks wrong. A leading zero-width space changes the first character of a line, so it sorts to an unexpected position.
- Character limits are exceeded. A field that reads as 99 characters fails a 100-character check because a byte order mark was counted.
- A blank line will not delete. The line is not empty; it contains a format character.
How to spot them without special software
You do not need a hex editor. Three techniques cover almost every case:
- Use the counters. Paste the text and compare the character count against what you can see. A count that is one or two higher than the visible length means an invisible character is present. This is the fastest signal available.
- Compare two lines that should match. Put them on consecutive lines, paste into the input panel, and read the character count. If the counts differ, they are not the same text, and the difference is the invisible character.
- Re-type the value by hand. If re-typing makes the problem disappear, the original contained something you did not type — which confirms the diagnosis even when you never identify the exact code point.
What this tool removes, and what it does not
- Removed: leading and trailing spaces and tabs, on every line.
- Collapsed: runs of spaces and tabs inside a line, to a single space.
- Removed at the edges: non-breaking spaces, narrow no-break spaces, en and em spaces, and a byte order mark, because trimming treats them as whitespace.
- Not removed: non-breaking and fixed-width spaces inside a line, because the collapsing step only recognizes ordinary spaces and tabs.
- Not removed: zero-width spaces, zero-width joiners, soft hyphens and other format characters, anywhere.
That boundary is deliberate. An internal non-breaking space may be load-bearing — it can be the only thing preventing a number from being split across a line break — and a zero-width joiner is structurally part of the emoji or script it sits in. Silently deleting either would change the text in ways the author did not ask for.
A cleaning sequence for stubborn text
- Run Clean spacing. This deals with everything at the edges and collapses ordinary runs.
- Run Remove blank lines. Lines that were only spaces are now empty and disappear. Lines containing a zero-width character will remain — that is the diagnostic.
- Run Remove duplicate lines. If a pair that looks identical survives, the difference is a code point, not a display artefact.
- Check the character count against the visible length to confirm how many invisible characters remain.
- Fix the remainder at the source. If the text came from a form, an export or a spreadsheet, correct the producing system rather than the output — otherwise the character comes back with the next import.
Preventing them at the source
- Do not paste from formatted documents into structured fields. Paste through a plain-text step first.
- Trim on the way in. A database that trims values on insert prevents the trailing non-breaking space from ever being stored.
- Normalize encodings at import. A one-time fix in the import script beats a repeated manual cleanup.
- Be suspicious of any value that will not match. An exact-match failure on text that looks identical is almost always an invisible character, and treating it as one saves an hour of confusion.
Frequently asked questions
Why will a blank line not delete?
Because it is not empty. It contains a character that is not whitespace — most often a zero-width space — so trimming leaves it in place. Use the character count to confirm, then delete the character itself.
Does clean spacing remove non-breaking spaces?
At the edges, yes, because trimming treats them as whitespace. Inside a line, no, because the collapsing step only recognizes ordinary spaces and tabs.
What is a soft hyphen and should I remove it?
A soft hyphen (U+00AD) marks a possible hyphenation point and is invisible unless the text wraps there. Remove it when the text will be re-flowed, because it can surface as a stray hyphen in the middle of a word.
Why does my field fail a character limit when it looks short enough?
Invisible characters still occupy positions. A byte order mark or a zero-width space counts toward the limit even though it has no visible width. Check the counter, then remove the character.