Método Potencial 1 - F_DROP_CACHES
Eu encontrei um método a partir de 2012 que discute um patch proposto para o kernel Linux neste tópico de e-mail intitulado: [RFC Patch] fs: implemente caches de lançamento por arquivo .
trechoCong> This is a draft patch of implementing per-file drop caches.
Interesting. So can I do this from outside a process? I'm a SysAdmin, so my POV is from noticing, finding and fixing performance problems when the system is under pressure.
Cong> It introduces a new fcntl command F_DROP_CACHES to drop Cong> file caches of a specific file. The reason is that currently Cong> we only have a system-wide drop caches interface, it could Cong> cause system-wide performance down if we drop all page caches Cong> when we actually want to drop the caches of some huge file.How can I tell how much cache is used by a file? And what is the performance impact of this when run on a busy system? And what does this patch buy us since I figure the VM should already be dropping caches once the system comes under mem pressure...
Cong> Below is small test case for this patch:
O encadeamento inclui um testcase e o patch real para vários arquivos dentro do kernel do Linux, o que adiciona uma função adicional ao fs/drop_caches.c chamado drop_pagecache_file(struct file *filp) . Esta função é então acessível através da ferramenta frontend, fnctl.c através do comando F_DROP_CACHES . Este caso chama esta função:
file_drop_caches(filp, arg);
Que lida com a queda de todos os caches associados ao arquivo fornecido. Do arquivo include/linux/mm.h :
void file_drop_caches(struct file *filp, unsigned long which);
Não encontrei nenhuma evidência de que esse patch tenha entrado no repositório principal de código do kernel do Linux, portanto, esta opção parece estar disponível apenas se você estiver disposto a recompilar o kernel do Linux.
Método Potencial 2 - Usando o dd
Nesse mesmo segmento, outro usuário menciona uma metodologia completamente diferente que faz uso de dd .
TesteThis is useful functionality. Though isn't it already provided with
Here are the examples from that patch:POSIX_FADV_DONTNEED? This functionality was added to GNU dd (8.11) a year ago.
Advise to drop cache for whole file
$ dd if=ifile iflag=nocache count=0Ensure drop cache for the whole file
$ dd of=ofile oflag=nocache conv=notrunc,fdatasync count=0Drop cache for part of file
$ dd if=ifile iflag=nocache skip=10 count=10 of=/dev/nullStream data using just the read-ahead cache
$ dd if=ifile of=ofile iflag=nocache oflag=nocache
Eu não tinha 100% de certeza sobre como testar isso, mas desenvolvi a seguinte abordagem.
-
crie um arquivo de 100MB
$ dd if=/dev/urandom of=sample.txt bs=100M count=1 -
acessa arquivos de rastreio usando
fatrace$ sudo fatrace | grep sample.txt -
execute
toppara que possamos monitorar o uso da memória, valor da nota livre.$ top -
abra o arquivo, anote a quantidade de memória livre agora. Observe o
fatracedo arquivosample.txt.$ cat sample.txt > /dev/null -
elimine o arquivo da memória, observe a quantidade de memória livre agora. Observe a saída de
fatrace.$ sudo dd of=/home/saml/tst/162600/sample.txt \ oflag=nocache conv=notrunc,fdatasync count=0
Exemplo
No terminal 1:$ dd if=/dev/urandom of=sample.txt bs=100M count=1
1+0 records in
1+0 records out
104857600 bytes (105 MB) copied, 7.37996 s, 14.2 MB/s
$ ls -l sample.txt
-rw-rw-r--. 1 saml saml 104857600 Oct 17 22:54 sample.txt
$ top
...
KiB Mem: 7968336 total, 6900956 used, 1067380 free, 267080 buffers
...
$ sudo fatrace | grep sample.txt
sample.txt e anote a quantidade de RAM. No terminal 1.
$ cat sample.txt > /dev/null
KiB Mem: 7968336 total, 7011896 used, 956440 free, 267336 buffers
fatrace no terminal # 3:
cat(25940): R /home/saml/tst/162600/sample.txt
cat(25940): R /home/saml/tst/162600/sample.txt
cat(25940): RC /home/saml/tst/162600/sample.txt
$ sudo dd of=/home/saml/tst/162600/sample.txt \
oflag=nocache conv=notrunc,fdatasync count=0
fatrace no terminal # 2:
dd(26229): O /home/saml/tst/162600/sample.txt
dd(26229): CW /home/saml/tst/162600/sample.txt
KiB Mem: 7968336 total, 6908364 used, 1059972 free, 267364 buffers
Assim, parece que tudo o que foi consumido pelo arquivo na RAM é liberado.
Método Potencial nº 3 - python-fadvise
Graças a um comentário de @frostchutz, há outra ferramenta, um script Python, chamado [pyadvise][4] , que fornece uma interface muito mais simples do que os métodos dd acima. Este script faz uso da mesma interface posix_fadvise(2) .
$ sudo pyadvise --help
Usage:
pyadvise [options] [FILE]..
Options:
-h, --help show this help message and exit
-w, --willneed The specified files will be accessed in the near future
-s, --sequential The application expects to access the specified files
sequentially (with lower offsets read before higher ones)
-d, --dontneed The specified files will not be accessed in the near
future
-r, --random The specified files will be accessed in random order
-o, --noreuse The specified files will be accessed only once. Under
Linux, this operation is a no-op; see contrib/copyfileobj-
fadvise.py in the python-fadvise source tree for an
example on how to achieve approximately the same effect
-n, --normal Indicates that the application has no advice to give about
its access pattern for the specified files. If no advice
is given for an open file, this is the default assumption
-v, --verbose Explain what is being done
E se repetirmos o teste acima e usar pyadvise no lugar de dd :
$ pyadvise -d /home/saml/tst/162600/sample.txt
Eu notei uma queda idêntica na RAM sendo consumida como antes, quando usei dd .