Seu comando deve funcionar, talvez haja um bug.
Use tshark (pacote wireshark) em seu lugar:
tshark -i eth0 -b duration:3600 -b filesize:102400 -s 65535 -w trace.pcap
The created filenames are based on the filename given with the -w option, the number of the file and on the creation date and time, e.g. outfile_00001_20050604120117.pcap, outfile_00002_20050604120523.pcap, ...