Da página man do diff:
If both from-file and to-file are directories, diff compares corresponding files in both directories, in alphabetical order; this comparison is not recursive unless the -r or --recursive option is given. diff never compares the actual contents of a directory as if it were a file. The file that is fully specified may not be standard input, because standard input is nameless and the notion of ‘‘file with the same name’’ does not apply.
Então, para comparar diretórios: diff --brief -r dir1 dir2
Para comparar arquivos lado a lado: diff --side-by-side file1 file2