GUIDE · LISTS

Removing duplicate lines from a list

Duplicate entries are the most common defect in lists that were built by several people or assembled from several files. They inflate counts, send twice the email, and make a checklist look unfinished when one item is ticked twice. Removing them is mechanical once you know what the tool is comparing.

Where duplicates come from

Merged files. Two exports each contain the same core entries plus their own additions; concatenating them doubles the shared rows.
Copy-paste loops. Selecting a list, pasting it, then pasting again lower down because you forgot it was already there.
Re-sent forms. A respondent submits twice, or a form rewrites a row on each save.
Casing and spacing variants. The same entry written slightly differently is not the same text — see below.

What counts as a duplicate

Line-based deduplication compares the trimmed text of each line, exactly. Two lines match only when their letters, digits, punctuation and internal spacing are identical after leading and trailing whitespace is removed. So these two are duplicates:

  apple
apple

And these three are not duplicates, even though a person would say they are the same thing:

apple
Apple
apple

The first pair matches because trimming removes the spaces. The second set does not match because capital A is a different character from lowercase a, and the trailing-space variant only looks identical in a font that hides it. Keep this distinction in mind: automatic deduplication is deliberately literal so that it never deletes something you did not intend to lose.

The recommended sequence

One item per line. If entries contain commas or are wrapped in quotes, split them first — one row per line is what the comparison needs.
Use Clean spacing. Trims each line so padding does not protect a duplicate from detection.
Use Remove duplicate lines. The first occurrence is kept in its original position; later copies are dropped.
Use Sort lines A–Z only if you want them ordered. Sorting is a separate step — deduplication preserves input order by design.

When duplicates are intentional

Some lists legitimately repeat. A rota or schedule may need the same name on several days; a lyrics sheet or a refrain repeats on purpose. Because line deduplication is literal, it cannot tell a meaningful repeat from an accidental one. If duplicates carry meaning, keep the original list and deduplicate a copy instead.

← All guides