Você precisa pip install filechunkio e, em seguida, use a opção --multipart de s3put .
Eu posso carregar arquivos pequenos usando os mesmos comandos, no entanto, sempre que tento enviar um arquivo de 5 GB para o S3, o script s3 falha:
[backup]$ s3put --bucket john-murdoch-bucket --reduced --debug 2 --callback 100 data.tgz
/usr/bin/s3put:45: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
err.message[len('No module named '):] + \
send: 'GET /?&max-keys=0 HTTP/1.1\r\nHost: john-murdoch-bucket.s3.amazonaws.com\r\nAccept-Encoding: identity\r\nDate: Tue, 11 Jun 2013 07:00:45 GMT\r\nContent-Length: 0\r\nAuthorization: AWS <snip>=\r\nUser-Agent: Boto/2.9.4 (linux2)\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: x-amz-id-2: <snip>
header: x-amz-request-id: 1E11AD8B3C93E6B5
header: Date: Tue, 11 Jun 2013 07:00:46 GMT
header: Content-Type: application/xml
header: Transfer-Encoding: chunked
header: Server: AmazonS3
Copying /backup/data.tgz to john-murdoch-bucket/backup/data.tgz
send: 'PUT /backup/data.tgz HTTP/1.1\r\nHost: john-murdoch-bucket.s3.amazonaws.com\r\nAccept-Encoding: identity\r\nContent-Length: 5522978068\r\nExpect: 100-Continue\r\nx-amz-storage-class: REDUCED_REDUNDANCY\r\nDate: Tue, 11 Jun 2013 07:02:15 GMT\r\nContent-MD5: idp0nQ+9y5kshw0AjL9zMw==\r\nContent-Type: application/x-tar\r\nAuthorization: AWS <snip>=\r\nUser-Agent: Boto/2.9.4 (linux2)\r\n \r\n'
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
0 bytes transferred / 5522978068 bytes total
Traceback (most recent call last):
File "/usr/bin/s3put", line 374, in <module>
main()
File "/usr/bin/s3put", line 371, in main
headers=headers)
File "/usr/bin/s3put", line 216, in singlepart_upload
k.set_contents_from_filename(fullpath, *kargs, **kwargs)
File "/usr/lib/python2.6/site-packages/boto/s3/key.py", line 1241, in set_contents_from_filename
encrypt_key=encrypt_key)
File "/usr/lib/python2.6/site-packages/boto/s3/key.py", line 1172, in set_contents_from_file
chunked_transfer=chunked_transfer, size=size)
File "/usr/lib/python2.6/site-packages/boto/s3/key.py", line 710, in send_file
chunked_transfer=chunked_transfer, size=size)
File "/usr/lib/python2.6/site-packages/boto/s3/key.py", line 882, in _send_file_internal
query_args=query_args)
File "/usr/lib/python2.6/site-packages/boto/s3/connection.py", line 544, in make_request
override_num_retries=override_num_retries)
File "/usr/lib/python2.6/site-packages/boto/connection.py", line 939, in make_request
return self._mexe(http_request, sender, override_num_retries)
File "/usr/lib/python2.6/site-packages/boto/connection.py", line 901, in _mexe
raise e
socket.error: [Errno 104] Connection reset by peer
Estou faltando algumas outras ferramentas ou opções de linha de comando para o s3put, o que faria várias tentativas, ou o upload em vários pedaços, ou assim por diante?
P.S. Eu acabei de fazer "sudo yum upgrade python-boto" mas isso não mudou nada.
Atualização - tentei s3cmd 1.0.0:
[ec2-user@ip-10-212-234-60 backup]$ s3cmd put data.tgz s3://john-murdoch-bucket/data.tgz
data.tgz -> s3://john-murdoch-bucket/data.tgz [1 of 1]
36864 of 5522978068 0% in 1s 20.31 kB/s failed
WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.00)
WARNING: Waiting 3 sec...
data.tgz -> s3://john-murdoch-bucket/data.tgz [1 of 1]
233472 of 5522978068 0% in 1s 144.89 kB/s failed
WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.01)
WARNING: Waiting 6 sec...
data.tgz -> s3://john-murdoch-bucket/data.tgz [1 of 1]
45056 of 5522978068 0% in 1s 27.88 kB/s failed
WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.05)
WARNING: Waiting 9 sec...
data.tgz -> s3://john-murdoch-bucket/data.tgz [1 of 1]
36864 of 5522978068 0% in 0s 45.52 kB/s failed
WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=0.25)
WARNING: Waiting 12 sec...
data.tgz -> s3://john-murdoch-bucket/data.tgz [1 of 1]
20480 of 5522978068 0% in 1s 19.90 kB/s failed
WARNING: Upload failed: /data.tgz ([Errno 32] Broken pipe)
WARNING: Retrying on lower speed (throttle=1.25)
WARNING: Waiting 15 sec...
data.tgz -> s3://john-murdoch-bucket/data.tgz [1 of 1]
12288 of 5522978068 0% in 2s 4.63 kB/s failed
ERROR: Upload of 'data.tgz' failed too many times. Skipping that file.
Você precisa pip install filechunkio e, em seguida, use a opção --multipart de s3put .