O armazenamento do Docker é montado novamente em modo somente leitura quando o contêiner está gerando muitos arquivos grandes (20G +)

1

Estou ficando preso a um problema por alguns dias e espero que alguém aqui possa ajudar ou dar algumas dicas.

Então, aqui está o problema: quando estou executando algumas imagens docker que executam grandes operações de E / S, o contêiner torna-se somente leitura no meio da execução. Meu contêiner docker está em execução no AWS ECS no AWS EC2. A imagem que estou executando é baseada no link .

Atualizei o tamanho do armazenamento da base de encaixe para 200G e, quando o contêiner está em execução, vejo que ele definitivamente tem espaço suficiente para o arquivo que está sendo gerado, para que não pareça um problema de espaço em disco. Eu também fiz alguma pesquisa no google e parece que a máquina host está encontrando erro XOS ENOSPC - mas não tenho idéia do que isso significa e como resolvê-lo.

Quando verifico os logs do dmesg na instância EC2 da máquina host, vejo os erros abaixo:

[Tue Oct 16 06:16:14 2018] device-mapper: thin: 253:2: switching pool to out-of-data-space (queue IO) mode
[Tue Oct 16 06:17:15 2018] device-mapper: thin: 253:2: switching pool to out-of-data-space (error IO) mode
[Tue Oct 16 06:17:15 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 788529152 size 8388608 starting block 5588992)
[Tue Oct 16 06:17:15 2018] buffer_io_error: 585671 callbacks suppressed
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588992
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588993
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588994
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588995
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588996
[Tue Oct 16 06:17:15 2018] Buffer I/O error on device dm-4, logical block 5588997
[Tue Oct 16 06:17:16 2018] Buffer I/O error on device dm-4, logical block 5588998
[Tue Oct 16 06:17:16 2018] Buffer I/O error on device dm-4, logical block 5588999
[Tue Oct 16 06:17:16 2018] Buffer I/O error on device dm-4, logical block 5589000
[Tue Oct 16 06:17:16 2018] Buffer I/O error on device dm-4, logical block 5589001
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 788529152 size 8388608 starting block 5589504)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 788529152 size 8388608 starting block 5590016)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 788529152 size 8388608 starting block 5590528)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 796917760 size 8388608 starting block 5591040)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 796917760 size 8388608 starting block 5591552)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 796917760 size 8388608 starting block 5592064)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 796917760 size 8388608 starting block 5592576)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 805306368 size 8388608 starting block 5593088)
[Tue Oct 16 06:17:16 2018] EXT4-fs warning (device dm-4): ext4_end_bio:323: I/O error 3 writing to inode 10356694 (offset 805306368 size 8388608 starting block 5593600)
[Tue Oct 16 06:17:16 2018] JBD2: Detected IO errors while flushing file data on dm-4-8
[Tue Oct 16 06:17:16 2018] JBD2: Detected IO errors while flushing file data on dm-4-8
[Tue Oct 16 06:17:16 2018] Aborting journal on device dm-4-8.
[Tue Oct 16 06:17:16 2018] EXT4-fs error (device dm-4): ext4_journal_check_start:61: Detected aborted journal
[Tue Oct 16 06:17:16 2018] EXT4-fs error (device dm-4): ext4_journal_check_start:61: Detected aborted journal
[Tue Oct 16 06:17:16 2018] EXT4-fs (dm-4): Remounting filesystem read-only

Abaixo está a especificação do EC2:

SO: Linux ip-10-119-5-115 4.14.59-64.43.amzn1.x86_64 # 1 SMP Qui 2 de Agosto 21:29:33 UTC 2018 x86_64 x86_64 x86_64 GNU / Linux

Informações do Docker:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 18.06.1-ce
Storage Driver: devicemapper
 Pool Name: docker-docker--pool
 Pool Blocksize: 524.3kB
 Base Device Size: 214.7GB
 Backing Filesystem: ext4
 Udev Sync Supported: true
 Data Space Used: 802.2MB
 Data Space Total: 23.33GB
 Data Space Available: 22.53GB
 Metadata Space Used: 606.2kB
 Metadata Space Total: 25.17MB
 Metadata Space Available: 24.56MB
 Thin Pool Minimum Free Space: 2.333GB
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 468a545b9edcd5932818eb9de8e72413e616e86e
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.14.59-64.43.amzn1.x86_64
Operating System: Amazon Linux AMI 2018.03
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.1GiB
Name: ip-10-119-5-115
ID: NJ4J:RBYM:3A25:QOZJ:HRSH:6BPD:3UO3:FW3X:4ZMI:5NWV:6IHV:OOHF
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 58
 Goroutines: 86
 System Time: 2018-10-16T06:19:26.381414812Z
 EventsListeners: 2
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Versão do Docker:

Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a215d7133c34aa18e3b72b4a21fd0c6136
 Built:             Tue Oct  2 18:06:45 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a/18.06.1-ce
  Built:            Tue Oct  2 18:08:26 2018
  OS/Arch:          linux/amd64
  Experimental:     false
    
por UTS-AnQ 16.10.2018 / 08:41

0 respostas