Duplicate lines remover

Clean repeated lines out of any list

Merged two mailing lists? Exported keywords from several tools? Copied log entries that repeat? The duplicate lines remover goes through your text line by line and keeps only the first copy of each, so every line appears once.

When it's done it shows how many lines you started with, how many new lines remain, and how many removed lines were duplicates. The original order of the remaining lines is kept.

It's the quickest way to dedupe data without opening a spreadsheet or writing a script.

How to use the duplicate lines remover

1. Paste your list, with one item per line.
2. Submit to remove duplicates.
3. Check how many lines were removed.
4. Copy the cleaned list.

Examples

A messy fruit list

Input:

apple
banana
apple
Apple
banana 
cherry
apple

Output:

apple
banana
Apple
banana 
cherry

7 lines in, 5 out, 2 removed. Only exact copies go: Apple (capital A) and banana (trailing space) are kept as different lines.

Merging two email lists

Input:

[email protected]
[email protected]
[email protected]

Output:

[email protected]
[email protected]

The first copy keeps its position and later copies disappear.

Keywords from two tools

Input:

free password generator
strong password generator
free password generator
Free password generator

Output:

free password generator
strong password generator
Free password generator

The capitalized copy survives because it isn't an exact match. Lowercase everything first to merge it.

Common uses

• Deduplicating email lists before importing them into a newsletter tool.
• Cleaning keyword lists for SEO and ad campaigns.
• Removing repeated entries from logs, URLs or product SKUs.
• Tidying lists of names, tags or hashtags.
• Merging lists from different sources into one unique list.

Tips and things to know

• Lines must match exactly to count as duplicates. Differences in capital letters or extra spaces make lines unique; normalize case first with the case converter.
• Sort the cleaned list with the list alphabetizer, or shuffle it with the list randomizer.
• If your list is separated by semicolons, spaces or pipes, convert it to one item per line first with the text separator.

Edge cases and limitations

• Trailing spaces and tabs make lines different. If the counts look too low, trim the lines first.
• Matching is case-sensitive. Convert to lowercase with the case converter before deduping emails or tags.
• Line endings are standardized to Windows style (CRLF) in the output, which every editor and spreadsheet reads fine.
• Repeated blank lines count as duplicates, so only the first blank line is kept.

Frequently asked questions

Does it keep the original order?

Yes. The first occurrence of each line stays where it was, and later repeats are removed.

Is it case-sensitive?

Lines need to match exactly, so "Apple" and "apple" are treated as different. Convert everything to lowercase first if you want them merged.

Does it remove empty lines?

Repeated empty lines count as duplicates, so only one empty line would remain.

Can I remove duplicate words instead of lines?

This tool works on whole lines. Put each word on its own line first to dedupe words.

Related tools

• List alphabetizer: Order text lines in alphabetical order (A-Z or Z-A) with ease.
• Text separator: Separate text back and forth by new lines, commas, dots, etc.
• List randomizer: Easily convert a list of given text into a randomized list.
• Email extractor: Extract email addresses from any kind of text content.
• Reverse list: Reverse a list of given text lines.