How to diff large files on Linux

Question

I'm getting a diff: memory exhausted error when trying to diff two 27 GB files that are largely similar on a Linux box with CentOS 5 and 4 GB of RAM. This is a known problem, it seems.

I would expect there to be an alternative for such an essential utility, but I can't find one. I imagine the solution would have to use temporary files rather than memory to store the information it needs.

• I tried to use rdiff and xdelta, but they are better for showing the changes between two files, like a patch, and are not that useful for inspecting the differences between two files.
• Tried VBinDiff, but it is a visual tool which is better for comparing binary files. I need something that can pipe the differences to STDOUT like regular diff.
• There are a lot of other utilities such as vimdiff that only work with smaller files.
• I've also read about Solaris bdiff but I could not find a port for Linux.

Any ideas besides splitting the file into smaller pieces? I have 40 of these files so trying to avoid the work of breaking them up.

Answer 1

cmp does things byte-by-byte, so it probably won't run out of memory (just tested it on two 7 GB files) -- but you might be looking for more detail than a list of "files X and Y differ at byte x, line y". If the similarities of your files are offset (e.g., file Y has an identical block of text, but not at the same location), you can pass offsets to cmp; you could probably turn it into a resynchronizing compare with a small script.

Aside: In case anyone else lands here when looking for a way to confirm that two directory structures (containing very large files) are identical: diff --recursive --brief (or diff -r -q for short, or maybe even diff -rq) will work and not run out of memory.

Answer 2

I found this link

diff -H might help, or you can try installing the textproc/2bsd-diff port which apparently doesn't try to load the files into RAM, so it can work on large files more easily.

I'm not sure if you tried those two options or if they might work for you. Good luck.

Answer 3

If the files are identical (same length) except for a few byte values, you can use a script like following (w is the number of bytes per line to hexdump, adjust to your display width):

w=12;
while read -ru7 x && read -ru8 y;
do
  [ ".$x" = ".$y" ] || echo "$x | $y";
done 7< <(od -vw$w -tx1z FILE1) 8< <(od -vw$w -tx1z FILE2) > DIFF-FILE1-FILE2 &

less DIFF-FILE1-FILE2

It's not very fast, but does the job.

Answer 4

If the files have the same number of lines and differ by the content of a few of them, use the following command. Substitute \a (alert) with any other character that does not occur within the files.

paste -d $'\a' file1 file2 | awk -F$'\a' '$1 != $2'

This works by pairing the lines of the two files and then comparing each pair.

Answer 5

This may not work for all types of files, but if your files have a regular structure to them you may be able to split them into smaller chunks and diff the chunks individually.

For example:

csplit large-file.txt '/separator pattern/' '{*}'

Caveat: this only works if your file has something you can use a separator without producing hundreds of small files and where the smaller chunks are still comparable.