Text utilities·Free, no signup
Remove Duplicate Lines
Deduplicate a list, tidy the whitespace and sort it, with a count of everything removed.
Treats Apple and apple as the same line.
- chars
- 31
- words
- 5
- lines
- 5
- size
- 31 B
apple banana cherry Banana date
8 lines in, 5 out — 2 duplicates removed.
Nothing is uploaded
Everything runs in your browser. Your text never reaches a server.
Up to 100,000 repeats
Ten times the usual ceiling, and it stays responsive at the top end.
Any language, any emoji
Full Unicode support, with emoji counted as single characters.
Copy or download
One click to the clipboard, or save the whole result as a .txt file.
Paste a list with one item per line and this tool keeps the first occurrence of each line while dropping the rest. Along the way it can trim leading and trailing spaces, collapse runs of double spaces, remove empty lines, ignore differences in capitalisation and sort the result A to Z or Z to A.
It is the kind of clean-up that happens whenever two lists are merged: exported email addresses, product codes copied from a spreadsheet, tags pasted from three documents, URLs collected over a week. The counter tells you how many duplicates were removed, which is a quick sanity check that the sources overlapped as much as you expected.
Three steps
How to use the remove duplicate lines
- 1
Paste one item per line
The line break is the delimiter. If your data is comma-separated, replace the commas with line breaks first, otherwise the whole thing is treated as a single line and nothing can be compared.
- 2
Choose how strict to be
Trimming whitespace and ignoring case both widen what counts as a duplicate. Switch them on when the same value arrived from different systems in different shapes, and off when the exact string matters.
- 3
Sort and copy
Leave the order untouched to keep the original sequence, or sort A to Z to make the result easy to scan. The summary shows how many lines went in, how many came out and how many were dropped.
Exact matching versus forgiving matching
By default two lines are duplicates only if they are identical, character for character. That is the right behaviour when a trailing space or a capital letter carries meaning. It is the wrong behaviour for most real lists, where the same value has arrived from a form, a spreadsheet and a CRM export in three different shapes.
Trim whitespace removes spaces and tabs from both ends of each line, so a value indented by a copy-paste matches the same value without the indent. Collapse double spaces fixes gaps inside a line. Ignore case treats London and london as one entry, and the version that survives is the first one in your list.
Typical clean-up jobs
Deduplication is rarely the whole task. Most lists arrive with a mixture of problems, and the order in which the options are applied matters here: whitespace is trimmed and double spaces collapsed before anything is compared, so entries that differ only by padding fall together instead of surviving as separate rows. Empty lines go at the same stage.
- Merging two mailing list exports and finding out how many subscribers overlapped.
- Cleaning a keyword list before uploading it to an advertising platform that rejects repeats.
- Reducing a log file to the set of distinct error messages it actually contains.
- Turning a messy column of city or supplier names into a tidy reference list.
- Building a unique list of URLs collected across several browser sessions.
What the tool will not do
It compares whole lines, not fields. A CSV row where only the timestamp differs is not a duplicate here, because the lines genuinely differ. To deduplicate on one column you need a spreadsheet or a script that can split the row first. Sorting is alphabetical by character order, so numbers sort as text and 10 comes before 9.
Processing happens in your browser, which keeps the data private but means very large files are limited by your device's memory rather than by any server. Tens of thousands of lines are comfortable on a normal laptop; a multi-million-line export is a job for a command line tool such as sort with its unique flag.
Frequently asked questions
Which copy of a duplicate is kept?
The first one, reading from top to bottom. Later repeats are discarded, so surviving entries stay in the order they originally appeared unless you also switch sorting on. When case-insensitive matching is enabled, the first version's capitalisation is the one that survives into the output.
Can it keep only the lines that appear more than once?
No. The tool produces a unique set, so every distinct line appears exactly once in the output. Finding only the values that occurred repeatedly is a different operation, and a spreadsheet pivot or a COUNTIF column handles it better than a text tool would.
Does it handle different line endings?
Yes. Text copied from Windows uses a carriage return and a line feed, while text from macOS and Linux uses a line feed alone. Both are recognised as line breaks, so a list pasted from a Windows editor deduplicates correctly against one pasted from a terminal.
Will it change the text inside my lines?
Only if you ask it to. Trimming removes spaces from the ends of a line, and collapsing reduces runs of two or more spaces to one. With both options switched off, every surviving line is returned exactly as you pasted it, character for character.
How does sorting treat accents and capitals?
Sorting compares characters using your browser's locale rules, so accented letters generally file next to their unaccented equivalents rather than at the end of the alphabet. Uppercase and lowercase versions of the same word sort together, and digits sort before letters.
Is my list uploaded anywhere?
No. The comparison runs entirely inside the page, so a customer list or an internal export never leaves your computer. There is no account and no history, and reloading the page clears the box, so keep your source file if you will need it again.
Keep going
Related tools
Built on the same engine, tuned for a different job.
Word Counter
Track a draft against an essay limit or an article brief without leaving the page.
Open toolCharacter Counter
See characters, characters without spaces, bytes and reading time update on every keystroke.
Open toolCase Converter
Switch text between eleven capitalisation styles, from Sentence case to CONSTANT_CASE.
Open toolBlank Text Generator
Build empty-looking text from four kinds of invisible character, then copy or download it.
Open toolThere are more where this came from
Every tool on the site runs the same way: in your browser, with no account, no limits and nothing sent anywhere.
Browse all toolsLast updated 23 August 2026