Cleaning a pasted list is a sequence, and the sequence matters more than the individual operations. The working order is: normalise line endings, trim each line, drop empty lines, decide what counts as a duplicate, deduplicate, sort, then number. Every failure people report with these tools — "it didn't remove the duplicates", "the sort scrambled my data", "the header row ended up in the middle" — comes from doing one of those steps before another.
The reason is simple: each step changes the input the next step sees. Deduplicating before trimming compares strings that still carry leading spaces, so apple and apple survive as two rows. Sorting before dropping empty lines moves blanks into a block at the top or bottom, where they are easy to delete but equally easy to overlook.
A worked example
Take this paste, which is typical of a column copied out of a spreadsheet:
apple
apple
Apple
banana
banana
apple
There are seven lines on screen and, by exact comparison, zero duplicates: no two strings are identical. After trimming, the padded rows collapse into their plain forms and the set is apple, apple, Apple, banana, banana, apple — still three distinct strings. After dropping the blank, the count is six. After deduplication, case-sensitive, you are left with apple, Apple, banana. Only if you decide case does not matter — usually the right call for a list of fruit, rarely the right call for a list of product codes — do you end with apple, banana.
That progression explains the most common complaint about deduplication tools: they are correct, and the user expected a different definition of "duplicate".
The six steps, in order
- Normalise line endings. Files produced on Windows, macOS and the web mix
\r\n,\nand sometimes a lone\r. Convert everything to\nfirst, or the last token on each line keeps an invisible carriage return that defeats every comparison after it. - Trim each line. Remove leading and trailing spaces and tabs. Cell padding from a spreadsheet is invisible on screen and decisive in a string comparison.
- Drop empty lines. A line with only spaces is not empty to a computer. Decide whether you also want to drop lines that contain only punctuation, such as a stray
-used as a placeholder. - Decide the duplicate definition. Exact match is the default. Case-insensitive and whitespace-insensitive are safer for human-entered data. Near-duplicate matching is a different problem and needs a threshold, not a checkbox.
- Deduplicate. Keep the first occurrence unless you have a reason to keep the last — for example, when merging an old list into a newer one where the newest value is the corrected one.
- Sort, then number. Sorting last means the output reads as a finished list. Numbering before sorting leaves numbers attached to the wrong lines; number after and the sequence matches what the reader sees.
If you are working with a file rather than a paste, the same order applies, with one addition: check the encoding first. A file saved as Windows-1252 and opened as UTF-8 produces replacement characters in accented names, and those names will then fail to deduplicate against their correct forms.
Four meanings of "duplicate"
| Rule | Merges | Typical use | Risk |
|---|---|---|---|
| Exact match | apple and apple | IDs, hashes, machine-generated data | Misses human variants |
| Ignore surrounding whitespace | apple and apple | Anything pasted from a spreadsheet | Almost none |
| Ignore case | apple and Apple | Names, tags, categories | Collapses codes that differ only by case |
| Ignore internal whitespace | New York and New York | Free-text addresses | Nothing, usually |
There is a fifth, fuzzy meaning — "these two are almost the same" — which requires edit distance or a similarity score. It is useful for finding typos in a list of company names and dangerous everywhere else, because the threshold is always a guess. If you need it, review every merged pair by hand rather than accepting the output.
Sorting: lexicographic, natural and locale-aware
A plain alphabetical sort compares character by character using code points. That produces two well-known surprises. First, item10 sorts before item2, because 1 comes before 2 at the first position where they differ — the digits are never read as a number. Second, uppercase letters sort before lowercase in plain code-point order, so Zebra can land before apple, which is not how a dictionary works.
Natural sort fixes the first problem by recognising runs of digits as numbers and comparing them numerically; case-insensitive collation fixes the second by comparing letters by their base form. TextBench sorts with a locale-aware, numeric, case-insensitive collator, so item1, item2, item10 and apple, Banana, cherry both come out the way a person expects. The trade-off is that a locale-aware sort is not byte-stable: two machines with different locale settings can produce different orders for accented characters.
Sorting by length is a different tool with different uses: finding titles that will be truncated in a fixed-width column, ordering tags by prominence, or spotting entries that are suspiciously short because a field was cut off during export. Ties are broken alphabetically so the output is deterministic.
Pasting from a spreadsheet: the traps
- Excel pastes tabs, not commas. A copied range arrives with tab separators and a trailing tab on every row, which produces an empty final column. If you are treating the paste as CSV, split on tabs instead.
- The header row is data. A sort that includes "Email" alongside addresses puts the header somewhere in the middle. Remove it, clean the list, then prepend it again.
- Quoted fields contain newlines. A cell with a line break inside it splits into two lines when pasted, and both halves look like valid data. This is the single most destructive paste problem with addresses and product descriptions.
- Leading apostrophes survive. Excel writes
'01234to keep a postal code as text; the apostrophe comes along for the ride and breaks exact matching against01234. - Numbers lose precision. Spreadsheets store 15 significant digits, so long IDs become rounded and two distinct IDs collapse into one duplicate. Export IDs as text before cleaning.
- Blank rows between blocks. Merged cells and grouped data leave gaps. Removing empty lines first is usually what you want; occasionally the gaps carry meaning (a section break), in which case sort within each block, not across the file.
Verifying the result
Keep the before and after counts. If a 1,200-line list comes back as 1,180 lines, "20 removed" is checkable: the duplicates should be findable, and the difference should equal the number of repeats. A large drop is a signal that the comparison was too aggressive — case-insensitive matching on a list of chemical formulas, for instance, will merge entries that are genuinely different.
Two habits catch most mistakes. Sort the output and scan the first and last twenty lines, where anomalies concentrate. And keep the original paste until the cleaned version has been used successfully once; re-pasting from a spreadsheet is cheap, reconstructing a damaged list is not.
FAQ
Does the order of deduplicate and sort change the result?
For an exact-match deduplication, the set of surviving lines is the same either way, but which copy survives differs: deduplicate-then-sort keeps the first occurrence in the original order, while sort-then-deduplicate keeps the alphabetically first spelling. If the values are interchangeable, it does not matter; if one copy carries corrected data, it does.
Why are two lines that look identical not being merged?
Invisible differences. The usual suspects are a trailing space, a non-breaking space ( ) instead of a normal space, a carriage return left over from a Windows paste, or two Unicode forms of the same accented letter — é as one code point versus e plus a combining accent. Trimming first and normalising Unicode solves almost all of it.
Can I deduplicate only part of each line?
Not with a line-based tool, and that is a deliberate limit. If lines are id,email,name, you need a column-aware tool that understands the delimiter and the quote rules; splitting on commas with a text tool will break on any field that contains one. Export the column you want to deduplicate, clean it, then join it back by key.
Run these steps in order with the TextBench line tools — remove duplicates, remove empty lines, trim, sort A→Z, sort by length, add line numbers and reverse. For the JSON equivalent of this problem, see the JSON syntax checklist.
Written by the TextBench team. Last updated November 19, 2026.