Estou ficando preso a um problema por alguns dias e espero que alguém aqui possa ajudar ou dar algumas dicas.
Então, aqui está o problema: quando estou executando algumas imagens docker que executam grandes operações de E / S, o contêiner torna-se somente leitura no meio da execução. Meu contêiner docker está em execução no AWS ECS no AWS EC2. A imagem que estou executando é baseada no link .
Atualizei o tamanho do armazenamento da base de encaixe para 200G e, quando o contêiner está em execução, vejo que ele definitivamente tem espaço suficiente para o arquivo que está sendo gerado, para que não pareça um problema de espaço em disco. Eu também fiz alguma pesquisa no google e parece que a máquina host está encontrando erro XOS ENOSPC - mas não tenho idéia do que isso significa e como resolvê-lo.
Quando verifico os logs do dmesg na instância EC2 da máquina host, vejo os erros abaixo:
[Tue Oct 16 06:16:14 2018] device-mapper: thin: 253:2: switching pool to out-of-data-space (queue IO) mode
[Tue Oct 16 06:17:15 2018] device-mapper: thin: 253:2: switching pool to out-of-data-space (error IO) mode
[Tue Oct 16 06:17:15 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 788529152 size 8388608 starting block 5588992)
[Tue Oct 16 06:17:15 2018] buffer_io_error: 585671 callbacks suppressed
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588992
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588993
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588994
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588995
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588996
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588997
[Tue Oct 16 06:17:16 2018] Buffer I/O error on device dm-4, logical block 5588998
[Tue Oct 16 06:17:16 2018] Buffer I/O error on device dm-4, logical block 5588999
[Tue Oct 16 06:17:16 2018] Buffer I/O error on device dm-4, logical block 5589000
[Tue Oct 16 06:17:16 2018] Buffer I/O error on device dm-4, logical block 5589001
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 788529152 size 8388608 starting block 5589504)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 788529152 size 8388608 starting block 5590016)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 788529152 size 8388608 starting block 5590528)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 796917760 size 8388608 starting block 5591040)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 796917760 size 8388608 starting block 5591552)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 796917760 size 8388608 starting block 5592064)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 796917760 size 8388608 starting block 5592576)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 805306368 size 8388608 starting block 5593088)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 805306368 size 8388608 starting block 5593600)
[Tue Oct 16 06:17:16 2018] JBD2: Detected IO errors while flushing file data on dm-4-8
[Tue Oct 16 06:17:16 2018] JBD2: Detected IO errors while flushing file data on dm-4-8
[Tue Oct 16 06:17:16 2018] Aborting journal on device dm-4-8.
[Tue Oct 16 06:17:16 2018] EXT4-fs error (device dm-4): ext4_journal_check_start:61: Detected aborted journal
[Tue Oct 16 06:17:16 2018] EXT4-fs error (device dm-4): ext4_journal_check_start:61: Detected aborted journal
[Tue Oct 16 06:17:16 2018] EXT4-fs (dm-4): Remounting filesystem read-only
Abaixo está a especificação do EC2:
SO: Linux ip-10-119-5-115 4.14.59-64.43.amzn1.x86_64 # 1 SMP Qui 2 de Agosto 21:29:33 UTC 2018 x86_64 x86_64 x86_64 GNU / Linux
Informações do Docker:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 3
Server Version: 18.06.1-ce
Storage Driver: devicemapper
Pool Name: docker-docker--pool
Pool Blocksize: 524.3kB
Base Device Size: 214.7GB
Backing Filesystem: ext4
Udev Sync Supported: true
Data Space Used: 802.2MB
Data Space Total: 23.33GB
Data Space Available: 22.53GB
Metadata Space Used: 606.2kB
Metadata Space Total: 25.17MB
Metadata Space Available: 24.56MB
Thin Pool Minimum Free Space: 2.333GB
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.14.59-64.43.amzn1.x86_64
Operating System: Amazon Linux AMI 2018.03
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.1GiB
Name: ip-10-119-5-115
ID: NJ4J:RBYM:3A25:QOZJ:HRSH:6BPD:3UO3:FW3X:4ZMI:5NWV:6IHV:OOHF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 58
Goroutines: 86
System Time: 2018-10-16T06:19:26.381414812Z
EventsListeners: 2
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Versão do Docker:
Client:
Version: 18.06.1-ce
API version: 1.38
Go version: go1.10.3
Git commit: e68fc7a215d7133c34aa18e3b72b4a21fd0c6136
Built: Tue Oct 2 18:06:45 2018
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 18.06.1-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: e68fc7a/18.06.1-ce
Built: Tue Oct 2 18:08:26 2018
OS/Arch: linux/amd64
Experimental: false