Plain Uniq
If this is the file test.txt:
00
00
01
01
00
00
02
02
This is the output on passing it through the uniq command, either via pipes or as input via STDIN:
Command: uniq < test.txt
00
01
00
02
The first two lines of the original file are the same, . The next two lines are which are followed by two repetitions of again and two repetitions of . The uniq command replaces the consecutive repetitions with only one line in each case.
Uniq with counts
uniq -c < test.txt
This example indicates the count of repetitions for each of the lines it collapses.
If this is the test file, testCounts.txt:
00
00
01
01
00
00
02
02
03
aa
aa
aa
Command: uniq -c < input00.txt
2 00
2 01
2 00
2 02
1 03
3 aa
The first number is the count of the repeated occurrences in the original file.
Printing only duplicate lines
The -d option only prints those lines that are followed by one or more repetitions immediately after them:
uniq -d < testCounts.txt
OR
cat testCounts.txt | uniq -d
OR
uniq -d testCounts.txt
Printing only unique lines
The -u option only prints those lines that are succeeded and preceded by different lines:
uniq -u < testCounts.txt
OR
cat testCounts.txt | uniq -u
OR
uniq -u testCounts.txt
Other possible options:
- Limit comparison only to the first characters (using the -w option).
- Avoid comparing the first characters (using the -s option).
- Ignore variations in case between lines (using the -i option).
- Avoid comparing the first fields (using the -f option).
(This may be useful while processing TSV files when you'd like to ignore the first column if it has serial numbers.)
You might find these examples interesting and useful.