Skip to content
ToolBoxGenie

Remove Duplicate Lines

Text Tools ยท Added 9 July 2026

Paste a list and get back one copy of each line. Useful for email lists, keyword sets, log extracts and anything else that arrived with repeats. You can match case-insensitively, ignore leading and trailing spaces, and invert the operation to see only the duplicates.

One item per line.

Matching options

How to use the remove duplicate lines

  1. 1Paste your list โ€” one item per line.
  2. 2Choose your matching options: case sensitivity, trimming, blank-line handling.
  3. 3Press Remove duplicates.
  4. 4The summary tells you how many lines were removed; copy the cleaned list.

Examples

Cleaning an email list

Input
540 addresses exported from two overlapping sources
Result
487 unique addresses ยท 53 duplicates removed

Finding the repeats instead

Input
Same list with 'Show only duplicates' enabled
Result
The 53 repeated addresses, listed once each

About the remove duplicate lines

Why duplicates appear in the first place

Almost every duplicate has the same origin: two sources merged without a shared key. Export a contact list from a CRM and a mailing platform, concatenate them, and you get every person who exists in both.

Line-level deduplication solves the easy half of the problem โ€” records that are byte-identical. It cannot catch 'J. Smith' versus 'John Smith', or the same address with a different capitalisation and a stray space. Normalising first, by trimming and lowercasing, catches far more than deduplicating raw text.

Deduplicate, then sort โ€” not the other way round

If original order carries meaning โ€” chronological logs, ranked keywords, a priority list โ€” deduplicate first and leave it alone. Sorting first destroys that information permanently.

If order does not matter, sorting afterwards makes the result far easier to scan and to diff against another list later.

Frequently asked questions

Does it keep the original order?
Yes. The first occurrence of each line stays exactly where it was and later copies are dropped. If you want alphabetical output, run the result through the Text Sorter afterwards.
Are 'Apple' and 'apple' the same line?
Only if you turn off case sensitivity. By default the comparison is exact, so those are two distinct lines. For email addresses and domain names, case-insensitive matching is usually what you want.
What about lines that differ only by a trailing space?
Enable 'trim whitespace' and they will be treated as identical. Invisible trailing spaces are the single most common reason a deduplication appears not to have worked.

Related tools