Remove Duplicate Lines

Remove Duplicate Lines takes a list — email addresses, product codes, names, URLs, anything with one item per line — and removes every line that has already appeared, while the rest stay in the order you gave them. You decide what counts as “the same”: capitals can be ignored so that Priya@Example.in matches priya@example.in, stray spaces at the ends of a line can be ignored (they are by default), and so can extra spaces inside it. You can keep the first copy of each line, the last copy, or only the lines that never repeat at all. Under the result, a short list shows which lines repeated and how many times, so you can see what was wrong with the data rather than only fixing it.

1. Paste your list into the box, one item per line, or press Try an example.
2. Choose which copy to keep: the first, the last, or only lines that appear once.
3. Turn on Ignore capitals or Ignore extra spaces inside if your list was typed by hand.
4. Copy or download the result. The bar under it says how many lines were removed, and Repeated lines lists the worst offenders.

• Lines are compared whole. Two lines that say the same thing in different words, or the same address with a different name in front, are different lines.
• Ignore capitals uses Unicode’s rules, so it works for accented letters and other scripts that have capitals; scripts without capitals, such as Devanagari, are unaffected.
• A letter written with a separate accent mark and the same letter written as one character count as the same line.
• Blank lines are left where they are by default, so paragraphs stay apart. Turn off Leave blank lines in place to treat them as duplicates of each other.
• The result keeps your line endings, Windows or Unix, and a final newline if the list had one.
• Everything happens in your browser. Your list isn’t uploaded or kept, which matters when it’s a list of customers’ email addresses or phone numbers.

No. Lines stay in the order you pasted them; only the repeats are taken out. If you also want the list sorted, use Sort Lines, which can remove duplicates while it sorts.

First copy keeps each line where it first appears. Last copy keeps it where it last appears — useful when later entries are more up to date. Only unrepeated removes every line that appears more than once, leaving only the ones that were unique to begin with.

Paste the list and read Repeated lines under the result: it shows each repeated line and how many times it appears, most repeated first. Your original list in the box is untouched.

Large ones. Each line is looked up in a set rather than compared with every other line, so 100,000 lines took about a third of a second on a laptop in our test, and longer on a phone. While a long list is being worked through, you can keep typing; the result catches up.