Como eu maximizo o IO para o meu programa

2

Eu tenho cerca de 130000% de arquivos.tif no meu diretório. Eu preciso extrair o texto deles usando o tesseract. Eu criei uma unidade tmpfs ram e movi todos os arquivos para lá. Abaixo está o meu um forro:

printf "%03d\n" {1..163} | parallel find . -name 'k-{}\*' -print0 | parallel -j100 -0 'tesseract {/} /mnt/ramdisk/output/{/.} > /dev/null 2>&1'

Minha máquina é uma VM em nuvem que executa o Ubuntu 16.04, com 4 núcleos de vCPU e 16 GB de RAM. Embora eu tenha mais de 100 trabalhos em paralelo, não consigo obter mais de 25 a 30 arquivos por minuto.

A entrada / saída do disco é o gargalo? Eu tenho quase nulo conhecimento sobre este tópico. Como posso executar mais arquivos por minuto?

Abaixo está a saída ps -e :

21002 pts/0    00:00:06 tesseract <defunct>
21010 pts/0    00:00:05 tesseract <defunct>
21023 pts/0    00:00:06 tesseract <defunct>
21035 pts/0    00:00:04 tesseract <defunct>
21039 pts/0    00:00:06 tesseract <defunct>
21040 pts/0    00:00:04 tesseract <defunct>
21044 pts/0    00:00:00 tesseract <defunct>
21056 pts/0    00:00:04 tesseract <defunct>
21065 pts/0    00:00:05 tesseract <defunct>
21067 pts/0    00:00:06 tesseract <defunct>
21068 pts/0    00:00:06 tesseract <defunct>
21080 pts/0    00:00:06 tesseract <defunct>
21081 pts/0    00:00:06 tesseract <defunct>
21082 pts/0    00:00:05 tesseract <defunct>
21083 pts/0    00:00:06 tesseract <defunct>
21084 pts/0    00:00:06 tesseract <defunct>
21085 pts/0    00:00:00 tesseract <defunct>
21087 pts/0    00:00:06 tesseract <defunct>
21090 pts/0    00:00:05 tesseract <defunct>
21094 pts/0    00:00:00 tesseract <defunct>
21099 pts/0    00:00:05 tesseract <defunct>
21100 pts/0    00:00:02 tesseract <defunct>
21102 pts/0    00:00:06 tesseract <defunct>
21103 pts/0    00:00:05 tesseract <defunct>
21104 pts/0    00:00:06 tesseract <defunct>
21106 pts/0    00:00:08 tesseract <defunct>
21107 pts/0    00:00:06 tesseract <defunct>
21108 pts/0    00:00:04 tesseract <defunct>
21110 pts/0    00:00:06 tesseract <defunct>
21111 pts/0    00:00:08 tesseract <defunct>
21112 pts/0    00:00:06 tesseract <defunct>
21113 pts/0    00:00:04 tesseract <defunct>
21114 pts/0    00:00:06 tesseract <defunct>
21115 pts/0    00:00:06 tesseract <defunct>
21117 pts/0    00:00:01 tesseract <defunct>
21118 pts/0    00:00:00 tesseract <defunct>
21119 pts/0    00:00:04 tesseract <defunct>
21128 pts/0    00:00:05 tesseract <defunct>
21129 pts/0    00:00:00 tesseract <defunct>
21134 pts/0    00:00:06 tesseract <defunct>
21135 pts/0    00:00:04 tesseract <defunct>
21137 pts/0    00:00:06 tesseract <defunct>
21138 pts/0    00:00:00 tesseract <defunct>
21141 pts/0    00:00:06 tesseract <defunct>
21144 pts/0    00:00:05 tesseract <defunct>
21145 pts/0    00:00:06 tesseract <defunct>
21150 pts/0    00:00:06 tesseract <defunct>
21151 pts/0    00:00:02 tesseract <defunct>
21152 pts/0    00:00:06 tesseract <defunct>
21154 pts/0    00:00:00 tesseract <defunct>
21155 pts/0    00:00:05 tesseract <defunct>
21158 pts/0    00:00:05 tesseract <defunct>
21159 pts/0    00:00:06 tesseract <defunct>
21160 pts/0    00:00:06 tesseract <defunct>
21165 pts/0    00:00:06 tesseract <defunct>
21167 pts/0    00:00:05 tesseract <defunct>
21173 pts/0    00:00:00 tesseract <defunct>
21175 pts/0    00:00:00 tesseract <defunct>
21181 pts/0    00:00:06 tesseract <defunct>
21184 pts/0    00:00:05 tesseract <defunct>
21185 pts/0    00:00:06 tesseract <defunct>
21187 pts/0    00:00:05 tesseract <defunct>
21191 pts/0    00:00:06 tesseract <defunct>
21196 pts/0    00:00:00 tesseract <defunct>
21197 pts/0    00:00:05 tesseract <defunct>
21198 pts/0    00:00:06 tesseract <defunct>
21199 pts/0    00:00:08 tesseract
21200 pts/0    00:00:00 tesseract <defunct>
21201 pts/0    00:00:06 tesseract <defunct>
21202 pts/0    00:00:03 tesseract <defunct>
21204 pts/0    00:00:08 tesseract
21207 pts/0    00:00:06 tesseract <defunct>
21210 pts/0    00:00:00 tesseract <defunct>
21214 pts/0    00:00:00 tesseract <defunct>
21215 pts/0    00:00:06 tesseract <defunct>
21217 pts/0    00:00:05 tesseract <defunct>
21218 pts/0    00:00:03 tesseract <defunct>
21221 pts/0    00:00:06 tesseract <defunct>
21224 ?        00:00:00 kworker/0:1
21229 pts/0    00:00:06 tesseract <defunct>
21239 pts/0    00:00:06 tesseract <defunct>
21240 pts/0    00:00:04 tesseract <defunct>
21241 pts/0    00:00:06 tesseract <defunct>
21242 pts/0    00:00:02 tesseract <defunct>
21245 pts/0    00:00:06 tesseract <defunct>
21246 pts/0    00:00:05 tesseract <defunct>
21247 pts/0    00:00:04 tesseract <defunct>
21248 pts/0    00:00:00 tesseract <defunct>
21250 pts/0    00:00:06 tesseract <defunct>
21251 pts/0    00:00:05 tesseract <defunct>
21252 pts/0    00:00:01 tesseract <defunct>
21253 pts/0    00:00:06 tesseract <defunct>
21265 pts/0    00:00:06 tesseract <defunct>
21268 pts/0    00:00:06 tesseract <defunct>
21269 pts/0    00:00:06 tesseract <defunct>
21270 pts/0    00:00:01 tesseract <defunct>
21277 pts/0    00:00:06 tesseract <defunct>
21285 pts/0    00:00:05 tesseract
21289 pts/0    00:00:07 tesseract <defunct>
21291 pts/0    00:00:07 tesseract
21292 pts/0    00:00:06 tesseract
21293 pts/0    00:00:06 tesseract
21297 pts/0    00:00:06 tesseract
21299 pts/0    00:00:05 tesseract
21300 pts/0    00:00:06 tesseract <defunct>
21302 pts/0    00:00:05 tesseract
21303 pts/0    00:00:06 tesseract <defunct>
21308 pts/0    00:00:05 tesseract <defunct>
21320 pts/0    00:00:05 tesseract
21322 pts/0    00:00:00 tesseract <defunct>
21324 pts/0    00:00:05 tesseract <defunct>
21329 pts/0    00:00:05 tesseract <defunct>
21330 pts/0    00:00:05 tesseract <defunct>
21333 pts/0    00:00:05 tesseract
21334 pts/0    00:00:05 tesseract
21335 pts/0    00:00:06 tesseract
21337 pts/0    00:00:00 tesseract <defunct>
21338 pts/0    00:00:04 tesseract <defunct>
21340 pts/0    00:00:04 tesseract
21341 pts/0    00:00:05 tesseract
21344 pts/0    00:00:05 tesseract
21353 pts/0    00:00:06 tesseract <defunct>
21362 pts/0    00:00:06 tesseract <defunct>
21363 pts/0    00:00:00 tesseract <defunct>
21366 pts/0    00:00:03 tesseract
21372 pts/0    00:00:04 tesseract
21373 pts/0    00:00:04 tesseract
21377 pts/0    00:00:00 tesseract <defunct>
21378 pts/0    00:00:05 tesseract
21382 pts/0    00:00:05 tesseract <defunct>
21384 pts/0    00:00:04 tesseract
21385 pts/0    00:00:05 tesseract
21391 pts/0    00:00:05 tesseract <defunct>
21393 pts/0    00:00:01 tesseract <defunct>
21394 pts/0    00:00:04 tesseract
21396 pts/0    00:00:01 tesseract <defunct>
21397 pts/0    00:00:02 tesseract <defunct>
21399 pts/0    00:00:05 tesseract
21401 pts/0    00:00:06 tesseract <defunct>
21403 pts/0    00:00:00 tesseract <defunct>
21404 pts/0    00:00:01 tesseract <defunct>
21405 pts/0    00:00:00 tesseract <defunct>
21407 pts/0    00:00:04 tesseract
21408 pts/0    00:00:00 tesseract <defunct>
21409 pts/0    00:00:05 tesseract
21413 pts/0    00:00:02 tesseract <defunct>
21414 pts/0    00:00:04 tesseract
21415 pts/0    00:00:05 tesseract
21416 pts/0    00:00:04 tesseract
21417 pts/0    00:00:00 tesseract <defunct>
21418 pts/0    00:00:03 tesseract
21419 pts/0    00:00:03 tesseract
21422 pts/0    00:00:03 tesseract
21424 pts/0    00:00:04 tesseract
21426 pts/0    00:00:03 tesseract
21427 pts/0    00:00:04 tesseract
21429 pts/0    00:00:05 tesseract
21434 pts/0    00:00:01 tesseract <defunct>
21437 pts/0    00:00:04 tesseract
21438 pts/0    00:00:00 nest.sh
21440 pts/0    00:00:02 tesseract
21445 pts/0    00:00:03 tesseract
21446 pts/0    00:00:04 tesseract
21447 pts/0    00:00:04 tesseract
21448 pts/0    00:00:02 tesseract
21449 pts/0    00:00:03 tesseract
21450 pts/0    00:00:04 tesseract
21451 pts/0    00:00:04 tesseract
21452 pts/0    00:00:02 tesseract <defunct>
21454 pts/0    00:00:03 tesseract
21455 pts/0    00:00:03 tesseract
21457 pts/0    00:00:00 perl
21467 ?        00:00:00 kworker/2:2
21471 pts/0    00:00:03 tesseract
21472 pts/0    00:00:03 tesseract
21475 pts/0    00:00:00 tesseract <defunct>
21480 ?        00:00:00 kworker/u8:2
21485 pts/0    00:00:02 tesseract
21487 pts/0    00:00:03 tesseract
21488 pts/0    00:00:02 tesseract
21489 pts/0    00:00:03 tesseract
21493 pts/0    00:00:02 tesseract
21496 pts/0    00:00:03 tesseract
21498 pts/0    00:00:03 tesseract
21501 pts/0    00:00:03 tesseract
21508 pts/0    00:00:02 tesseract <defunct>
21510 pts/0    00:00:03 tesseract
21516 pts/0    00:00:00 tesseract <defunct>
21519 pts/0    00:00:02 tesseract
21521 pts/0    00:00:03 tesseract
21523 pts/0    00:00:02 tesseract
21531 pts/0    00:00:02 tesseract
21532 pts/0    00:00:02 tesseract
21536 pts/0    00:00:00 tesseract <defunct>
21537 pts/0    00:00:02 tesseract
21543 pts/0    00:00:01 tesseract
21545 pts/0    00:00:02 tesseract
21549 pts/0    00:00:02 tesseract
21553 pts/0    00:00:02 tesseract
21556 pts/0    00:00:00 tesseract <defunct>
21560 pts/0    00:00:00 tesseract
21561 pts/0    00:00:01 tesseract
21562 pts/0    00:00:01 tesseract
21566 pts/0    00:00:01 tesseract
21569 pts/0    00:00:01 tesseract
21570 pts/0    00:00:01 tesseract
21575 pts/0    00:00:00 tesseract <defunct>
21582 pts/0    00:00:02 tesseract
21583 pts/0    00:00:00 tesseract
21585 pts/0    00:00:01 tesseract
21586 pts/0    00:00:01 tesseract
21590 pts/0    00:00:01 tesseract
21595 pts/0    00:00:00 tesseract <defunct>
21596 pts/0    00:00:00 tesseract
21597 pts/0    00:00:01 tesseract
21598 pts/0    00:00:01 tesseract
21599 pts/0    00:00:00 tesseract
21600 pts/0    00:00:00 tesseract <defunct>
21606 pts/0    00:00:00 tesseract
21610 pts/0    00:00:00 tesseract
21611 pts/0    00:00:01 tesseract
21612 pts/0    00:00:00 tesseract
21615 pts/0    00:00:01 tesseract
21620 pts/0    00:00:01 tesseract
21621 pts/0    00:00:00 tesseract
21622 pts/0    00:00:00 tesseract
21624 pts/0    00:00:00 tesseract
21625 pts/0    00:00:00 tesseract
21626 pts/0    00:00:00 tesseract
21639 pts/0    00:00:01 tesseract
21641 pts/0    00:00:01 tesseract
21646 ?        00:00:00 mysql-systemd-s
21648 pts/0    00:00:00 tesseract
21653 pts/0    00:00:00 tesseract
21654 pts/0    00:00:00 tesseract
21655 pts/0    00:00:00 tesseract
21660 pts/0    00:00:00 tesseract
21665 pts/0    00:00:00 tesseract
21666 pts/0    00:00:00 tesseract
21670 pts/0    00:00:00 tesseract
21671 pts/0    00:00:00 tesseract
21674 pts/0    00:00:00 tesseract
21675 pts/0    00:00:00 tesseract
21677 pts/0    00:00:00 tesseract
21682 pts/0    00:00:00 tesseract
21683 pts/0    00:00:00 tesseract
21684 pts/0    00:00:00 tesseract
21688 pts/0    00:00:00 tesseract
21690 pts/0    00:00:00 tesseract
21691 pts/0    00:00:00 tesseract
21692 pts/0    00:00:00 tesseract
21693 pts/0    00:00:00 tesseract
21696 pts/0    00:00:00 tesseract
21697 pts/0    00:00:00 tesseract
21699 pts/0    00:00:00 tesseract
21703 pts/0    00:00:00 tesseract
21705 pts/0    00:00:00 tesseract
21708 pts/0    00:00:00 tesseract
21709 pts/0    00:00:00 tesseract
21710 pts/0    00:00:00 tesseract
21711 pts/0    00:00:00 tesseract
21712 pts/0    00:00:00 tesseract
21713 pts/0    00:00:00 tesseract
21714 pts/0    00:00:00 tesseract
21716 pts/0    00:00:00 tesseract
21717 pts/0    00:00:00 tesseract
21720 pts/0    00:00:00 tesseract
21724 pts/0    00:00:00 tesseract
21727 pts/0    00:00:00 tesseract
21730 pts/0    00:00:00 tesseract
21732 pts/0    00:00:00 tesseract
21733 pts/0    00:00:00 tesseract
21736 pts/0    00:00:00 tesseract
21739 pts/0    00:00:00 tesseract
21742 pts/0    00:00:00 tesseract
21745 pts/0    00:00:00 tesseract
21752 pts/0    00:00:00 tesseract
21755 pts/0    00:00:00 tesseract
21756 pts/0    00:00:00 tesseract
21758 pts/0    00:00:00 tesseract
21760 pts/0    00:00:00 tesseract
21762 pts/0    00:00:00 tesseract
21764 pts/0    00:00:00 tesseract
21765 pts/0    00:00:00 tesseract
21766 pts/0    00:00:00 tesseract
21769 pts/0    00:00:00 tesseract
21773 pts/0    00:00:00 tesseract
21778 pts/0    00:00:00 tesseract
21779 pts/0    00:00:00 tesseract
21780 pts/0    00:00:00 tesseract
21782 pts/0    00:00:00 tesseract
21784 pts/0    00:00:00 sleep
21785 pts/0    00:00:00 sleep
21786 pts/0    00:00:00 sleep
21787 pts/0    00:00:00 tesseract
21788 pts/0    00:00:00 bash
21789 pts/0    00:00:00 sleep
21790 pts/0    00:00:00 sleep
21791 pts/0    00:00:00 sleep
21792 pts/0    00:00:00 sleep
21794 pts/0    00:00:00 sleep
21795 pts/0    00:00:00 tesseract
21796 pts/0    00:00:00 tesseract
21801 pts/0    00:00:00 tesseract
21803 pts/0    00:00:00 bash
21804 pts/0    00:00:00 bash
21807 pts/0    00:00:00 sleep
21808 pts/0    00:00:00 tesseract
21809 pts/0    00:00:00 tesseract
21810 pts/0    00:00:00 bash
21811 pts/0    00:00:00 sleep
21812 pts/0    00:00:00 sleep
21813 pts/0    00:00:00 perl
21814 ?        00:00:00 sshd
21815 pts/0    00:00:00 perl
21816 ?        00:00:00 sleep
21817 pts/1    00:00:00 ps
    
por Naveed 29.12.2017 / 23:22

2 respostas

3

Se você já está no tmpfs, seu IO não deve ser o gargalo. Claramente errado em seu script é que você está tentando executar muitas tarefas ao mesmo tempo. Como sua tarefa não deve estar esperando nada devido ao tmpfs, você deve tentar usar a quantidade de trabalhos paralelos igual à quantidade de núcleos que você tem, pode ser +1. Também reduza seu conjunto de testes para digamos 200 arquivos para que você possa testá-lo mais rápido.

    
por 30.12.2017 / 03:56
0

Você deve tentar determinar o que causa essa E / S. Ler e escrever de ou para tmpfs não causa I / O:

:> cd /run/user/1000
:> LC_ALL=C df -h .
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           2.4G  128M  2.3G   6% /run/user/1000
:> dd if=/dev/urandom of=bigfile bs=1M count=100
:> ls -lh bigfile
-rw-r--r-- 1 hl hauke 100M 30. Dez 16:07 bigfile
:> while true; do cat <bigfile >bigfilecopy || break; echo -n .; done

Isso lê e grava dados a uma taxa de mais de 1 GiB / s, mas top mostra 0,0% de espera de E / S.

Iniciar os processos não deve ser um problema, porque isso acontece apenas 25 vezes por segundo e deve ser coberto pelo cache de páginas de qualquer maneira.

Se você não puder encontrar (e remover) a causa da E / S, então deverá comparar a espera de E / S por diferentes números pequenos de processos paralelos. Como você já foi informado, geralmente não faz sentido ter mais processos do que núcleos de CPU. Mas talvez você tenha uma espera de E / S muito baixa com dois ou três processos e uma carga muito maior com quatro, de modo que possa ser mais rápido ficar com apenas três processos paralelos.

    
por 30.12.2017 / 16:23

Tags