Remove Duplicate Lines
About this tool
Paste a list of lines, such as email addresses, URLs, log entries, IDs or keywords, and remove the repeats. You can also keep only the lines that appear once, list only the repeated ones, or count how often each line occurs.
Everything runs in your browser. Your list is never sent to a server.
How it works
- Lines are compared exactly, character by character, unless you turn on Ignore case or Trim. With Ignore case on, the line that is kept is shown the way it was first (or last) written.
- Trim removes spaces and tabs at the start and end of each line before comparing, and the result shows the trimmed lines.
- Only lines that appear once drops every line that has any duplicate, including the first copy. Only lines that are duplicated lists each repeated line once.
- Count shows
3 × linefor each distinct line, most frequent first (ties keep the original order). Ordering options do not apply to this mode. - Sorting compares text the way a person would read it, with numbers inside lines in numeric order (
item2beforeitem10) and accents and case handled by your browser's locale rules. - Windows (CRLF) and Unix (LF) line endings are both handled.
FAQ
Q. Does it handle very large lists?
A. It works in your browser's memory, so lists of hundreds of thousands of lines are fine on a normal computer. There is no server-side limit because nothing is uploaded.
Q. Two lines look the same but are not merged.
A. Check for trailing spaces, tabs, different capitalization, or look-alike characters such as a non-breaking space. Turn on Trim and Ignore case for the first three.