String Functions
Duplicate Lines Remover
Remove duplicate lines from a block of text. The first occurrence of each line is kept; later repeats are stripped.
About removing duplicate lines
Duplicate Lines Remover takes a block of text and deletes every line that already appeared earlier in it, keeping only each line's first occurrence. Paste apple, banana, apple, cherry, banana and you get back apple, banana, cherry — the second apple and second banana are gone, and the order of the remaining lines is unchanged. This is a global dedupe, not a uniq-style pass that only catches back-to-back repeats: two identical lines with other lines in between are still caught and removed, no matter how far apart they sit in the pasted text.
The text is split into lines on \n, \r\n, or \r, then each line is checked against a JavaScript Set of lines already seen. The first time a line appears it's kept and added to the set; every later exact match is dropped. Comparison is case-sensitive and whitespace-sensitive — Apple and apple count as different lines, and a trailing space makes two otherwise-identical lines distinct. Surviving lines are rejoined with \n, which also normalizes any \r\n line endings to \n in the output. The whole transform runs in your browser; no text is sent to a server.
This mirrors PHP's array_unique() applied to an array of lines, which is exactly what the legacy version of this tool did server-side — first occurrence wins, order is preserved. Reach for it when cleaning up a pasted log dump with repeated status lines, deduplicating a list of emails, URLs, or CSV rows collected from different sources, or tidying a copy-pasted tag or keyword list before feeding it downstream. Because matching is exact, normalize whitespace or casing first if your near-duplicates differ only in trailing spaces or letter case — otherwise they'll survive as distinct lines.
Examples
Input:
apple
banana
apple
cherry
banana
Output:
apple
banana
cherryThe second apple and second banana are dropped; apple, banana, and cherry each keep the position of their first occurrence.
const lines = text.split(/\r?\n|\r/);
const deduped = [...new Set(lines)].join('\n');A JS Set preserves insertion order and silently ignores values it has already seen, so wrapping the split lines in a Set reproduces the same first-occurrence dedupe this tool runs.
Input:
INFO connected
INFO connected
ERROR timeout
INFO connected
INFO ready
Output:
INFO connected
ERROR timeout
INFO readyThe third INFO connected line is dropped even though an ERROR line sits between the repeats — this catches exact repeats anywhere in the text, not just back-to-back ones like Unix uniq.
Frequently asked questions
Does this remove all duplicate lines, or only consecutive ones?
All of them, anywhere in the text, not just adjacent ones. The first time a line appears it's kept; every later exact repeat of it is removed, even with unrelated lines sitting between the repeats. That's different from a uniq-style tool that only folds back-to-back duplicates.
Is the comparison case-sensitive?
Yes. Apple, apple, and APPLE are treated as three different lines and all three survive. Convert the text to a single case first if you want case-insensitive deduplication.
Does it remove blank lines too?
Only duplicate ones. The first blank line in the input is kept like any other line, and any later blank line is treated as a repeat of it and removed. To strip every blank line regardless of position, use the Empty Lines Remover instead.
Does deduplicating change the order of my lines?
No. Nothing is sorted — surviving lines stay in the order they first appeared in the input. The only thing removed is a line that repeats one already seen earlier.
Is my text sent to a server?
No. The deduplication is a plain JavaScript function that runs in your browser; nothing is uploaded, logged, or stored anywhere. Refreshing or closing the tab clears everything.