Você pode usar wget
( para janelas ) (ou via cgywin ) para baixar o site recursivamente,
$ wget -c \
--recursive \
--no-clobber \
--page-requisites \
--html-extension \
--convert-links \
--restrict-file-names=windows \
--domains vault.centos.org \
--no-parent \
http://vault.centos.org/5.4/os/i386/
As opções são:
--recursive: download the entire Web site.
--domains vault.centos.org: don't follow links outside vault.centos.org.
--no-parent: don't follow links outside the directory tutorials/html/.
--page-requisites: get all the elements that compose the page (images, CSS and so on).
--html-extension: save files with the .html extension.
--convert-links: convert links so that they work locally, off-line.
--restrict-file-names=windows: modify filenames so that they will work in Windows as well.
--no-clobber: don't overwrite any existing files (used in case the download is interrupted and resumed).