Duplicate Line Remover

Dedupe, sort, trim and clean lists of lines in bulk.

—

What "duplicate" means depends on the comparison

The most common surprise is whitespace and case. The lines apple, apple and Apple are three different strings to a computer, but one item to a person reading the list. Trimming whitespace and comparing case-insensitively removes far more duplicates in real data than a strict comparison does. This tool gives you both modes because the right choice depends on the data: for email addresses, case-insensitive is correct; for passwords or identifiers, case-sensitive is essential. Always check which one you used before trusting a dedupe result.

Sorting: alphabetical versus numeric

String comparison is character by character, so 10 sorts before 9 — the first character "1" comes before "9". This is the classic bug behind "my sorted list is wrong" reports. Numeric sorting compares values instead and puts 9 before 10. It also matters for version strings: 2.10 is newer than 2.9 but sorts before it as text. If you are sorting anything containing numbers — versions, IDs, sizes, prices — use numeric mode or a natural sort that handles embedded numbers properly.

Stability and which duplicate survives

Keeping the first occurrence preserves the original order of your data, which matters when the list has meaning beyond its contents — a log sequence, a ranked list, a playlist. Keeping the last occurrence instead changes which line survives when trailing whitespace or capitalisation differs between copies. For dirty data, trim first, then dedupe: that way the version you keep is the normalised one rather than whichever copy happened to appear first.

Typical uses

Scale and privacy

Processing happens entirely in your browser, so lists containing personal data, credentials or internal identifiers never leave your machine — which is the main reason to prefer this over an online service for anything sensitive. Because the work is done in the page, extremely large lists will be slow. Above roughly a hundred thousand lines, command-line tools are more appropriate: sort -u dedupes and sorts in one step, uniq -c counts occurrences, and both handle files far larger than a browser tab comfortably. For pattern-based cleaning before or after, use the regex tester.