How to remove duplicates from a list online
Quick answer
Paste the list into DO101 Remove Duplicate Lines. Every repeated line disappears, the first occurrence is kept and the original order is preserved, and the tool tells you how many duplicates it removed. Turn on "ignore surrounding spaces" and turn off "case sensitive" to catch near-duplicates such as "Anna@mail.com " and "anna@mail.com".
Open the Remove Duplicate Lines — freeStep by step
- 1Open Remove Duplicate Lines and paste your list, one item per line.
- 2Choose whether capitals and surrounding spaces should count as differences.
- 3Copy the cleaned list.
Why duplicates slip through
Lists copied from spreadsheets and emails pick up trailing spaces and inconsistent capitals, so two lines that look identical are not identical to a computer. That is the usual reason another tool "missed" a duplicate. Ignoring case and surrounding spaces fixes it.
Doing it in Excel or Google Sheets instead
Excel has Data › Remove Duplicates, and Google Sheets has Data › Data cleanup › Remove duplicates. Both work on whole rows, and both treat "Anna" and "Anna " as different. For a single column pasted from anywhere, pasting it here is faster, and the list never leaves your browser.
Common jobs it saves time on
Merging two email lists before a mail-out, so nobody gets the message twice. Cleaning a keyword list gathered from several research tools. Checking a column of order numbers, student IDs or SKUs pasted from two exports. Tidying a list of links or file names. In each case the useful number is the count of removed lines — if you expected ten repeats and it removed three hundred, something upstream was exported twice.
Before you paste
Put one item per line. If your list is comma-separated, use Find and Replace to turn each comma into a new line first. Blank lines count as items too — Remove Empty Lines clears them out.
Frequently asked questions
Which copy is kept?
The first one. Later repeats are removed, so the order stays as it was.
How long can the list be?
Tens of thousands of lines process instantly. The only ceiling is your browser's memory.
Can I sort the list as well?
Yes — paste the result into the Text Sorter.
The tool for the job
Strip repeated lines out of a list.