O caso curioso de alta média de carga de 5 min

Question

Procurando por alguns conselhos de especialistas aqui. Eu sou um administrador do sistema pela primeira vez no meu próprio servidor e não consigo entender o gargalo no meu servidor.

Tenho recebido toneladas de alerta médio de carga de 5 min, variando entre 8 e 80 do CSF

Então eu fui em frente e instalei o sqltuner no meu servidor e deixei ele rodar por 3 dias

Os resultados não mostram nada fora do comum, mas eu ainda estou recebendo alta de 5 minutos de carga média diária

Estou tentando encontrar o gargalo da garrafa (CPU, carga causada por problemas de falta de memória ou carga vinculada a E / S)

Ficaria feliz se alguém pudesse compartilhar alguma ideia ...

(Eu incluí o relatório do sqltizer e a saída de e-mail de alta carga abaixo)

-------- Security Recommendations  -------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[!!] There is no basic password file list!
-------- Performance Metrics -------------------------------------------------
[--] Up for: 120d 18h 27m 20s (227M q [21.795 qps], 51M conn, TX: 907B, RX: 26B)
[--] Reads / Writes: 38% / 62%
[--] Binary logging is disabled
[--] Total buffers: 15.4G global + 4.1M per thread (600 max threads)
[OK] Maximum reached memory usage: 16.2G (51.75% of installed RAM)
[OK] Maximum possible memory usage: 17.8G (56.91% of installed RAM)
[OK] Slow queries: 0% (11/227M)
[OK] Highest usage of available connections: 33% (199/600)
[OK] Aborted connections: 0.56%  (284327/51183230)
[OK] Query cache efficiency: 83.0% (78M cached / 94M selects)
[!!] Query cache prunes per day: 10288
[OK] Sorts requiring temporary tables: 0% (392 temp sorts / 1M sorts)
[OK] Temporary tables created on disk: 4% (65K on disk / 1M total)
[OK] Thread cache hit rate: 99% (199 created / 51M connections)
[OK] Table cache hit rate: 22% (425 open / 1K opened)
[OK] Open file limit used: 0% (433/50K)
[OK] Table locks acquired immediately: 99% (41M immediate / 41M locks)
-------- MyISAM Metrics -----------------------------------------------------
[!!] Key buffer used: 20.2% (108M used / 536M cache)
[OK] Key buffer size / total MyISAM indexes: 512.0M/14.7M
[OK] Read Key buffer hit rate: 99.8% (51M cached / 121K reads)
[!!] Write Key buffer hit rate: 40.8% (4M cached / 2M writes)
-------- InnoDB Metrics -----------------------------------------------------
[--] InnoDB is enabled.
[OK] InnoDB buffer pool / data size: 14.6G/140.9M
[!!] InnoDB buffer pool instances: 1
[!!] InnoDB Used buffer: 3.39% (32546 used/ 959999 total)
[OK] InnoDB Read buffer efficiency: 100.00% (5437258684 hits/ 5437259670 total)
[!!] InnoDB Write buffer efficiency: 0.00% (0 hits/ 1 total)
[OK] InnoDB log waits: 0.00% (0 waits / 24069213 writes)
-------- AriaDB Metrics -----------------------------------------------------
[--] AriaDB is disabled.
-------- Replication Metrics -------------------------------------------------
[--] No replication slave(s) for this server.
[--] This is a standalone server..
-------- Recommendations -----------------------------------------------------
General recommendations:
    Run OPTIMIZE TABLE to defragment tables for better performance
    Increasing the query_cache size over 128M may reduce performance
Variables to adjust:
    query_cache_size (> 128M) [see warning above]
    innodb_buffer_pool_instances(=14)
----------------------
(The only change I've made is to reduce InnoDB size and add multiple pool instances)

O email de carregamento diário alto:

Time:                    Sun Dec  6 05:43:53 2015 -0500
1 Min Load Avg:          80.26
5 Min Load Avg:          21.19
15 Min Load Avg:         7.46
Running/Total Processes: 221/875
ps.txt
O
Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process
Srv    PID    Acc    M    CPU     SS    Req    Conn    Child    Slot    Client    VHost    Request
0-1424    16243    0/149/2713129    W     1.28    14    0    0.0    2.34    65107.59     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
1-1424    17057    0/18/2701770    W     2.15    4    0    0.0    0.30    62402.50     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
2-1424    17064    0/24/2685073    W     2.11    13    0    0.0    0.32    62668.14     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
3-1424    15319    0/215/2657841    W     3.50    4    0    0.0    3.88    61950.21     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
4-1424    11567    0/204/2651294    W     7.10    7    0    0.0    3.00    63562.61     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
5-1424    16512    0/37/2640191    W     2.19    5    0    0.0    0.60    63637.48     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
6-1424    17735    0/8/2630311    W     0.62    19    0    0.0    0.06    65036.68     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
7-1424    16521    0/31/2613938    W     2.20    19    0    0.0    0.36    62385.07     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
8-1424    16081    0/33/2611913    W     2.46    5    0    0.0    0.42    60535.12     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
9-1424    14711    0/120/2603042    W     1.89    18    0    0.0    2.11    59868.26     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
10-1424    16838    0/21/2592501    W     1.77    15    0    0.0    0.24    62195.33     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
11-1424    16531    0/42/2584776    W     2.45    11    0    0.0    0.39    62253.11     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
12-1424    17065    0/20/2570161    W     1.29    12    0    0.0    0.18    60474.65     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
13-1424    17770    0/13/2564128    W     1.27    2    0    0.0    0.63    59748.24     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
14-1424    17771    0/14/2542936    W     1.30    2    0    0.0    0.17    60513.73     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
15-1424    15736    0/64/2536855    W     2.91    7    0    0.0    1.16    61453.61     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
16-1424    17077    0/19/2522131    W     2.76    15    0    0.0    0.35    59307.60     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
17-1424    14723    0/93/2521068    W     3.38    6    0    0.0    1.77    60437.40     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
18-1424    16279    0/62/2509938    W     1.81    15    0    0.0    1.07    61401.24     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
19-1424    15333    0/116/2498356    W     3.24    19    0    0.0    1.69    57911.45     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
20-1424    16297    1/35/2494463    W     0.98    53    62    16.1    0.47    59474.66     58.174.24.65    suspensionrevolution.com:80    GET /new/wp-content/themes/optimizePressTheme/lib/assets/defaul
21-1424    16298    0/40/2473943    W     3.83    3    0    0.0    0.54    57987.71     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
22-1424    18054    0/6/2469193    W     1.23    1    0    0.0    0.05    59122.65     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
23-1424    12894    0/162/2458774    W     5.90    17    0    0.0    2.42    56404.92     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
24-1424    18088    0/4/2452422    W     0.90    11    0    0.0    0.00    58405.08     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
25-1424    18089    0/6/2446570    W     1.22    1    0    0.0    0.03    57036.34     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
26-1424    17079    0/30/2439491    W     1.88    0    0    0.0    0.43    54697.67     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
27-1424    16101    0/64/2416961    W     1.53    18    0    0.0    1.69    57160.43     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
28-1424    18140    0/9/2403931    W     0.62    18    0    0.0    0.02    55901.03     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
29-982    1505    1/18/1548355    G     0.14    2914733    450294    2.8    0.29    34947.04     96.47.70.4    suspensionrevolution.com:80    POST /dap/dap-clickbank-6.0.php HTTP/1.1
30-1424    15338    0/100/2384316    W     2.20    7    0    0.0    1.24    53919.77     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
31-1424    16300    2/56/2379897    K     3.47    3    1365    2.4    1.01    55195.75     89.166.18.35    appcoiner.com:80    GET /favicon.ico HTTP/1.1
32-1424    15749    0/108/2369131    W     3.26    17    0    0.0    2.09    55452.02     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
33-1424    17100    0/17/2359616    W     1.70    11    0    0.0    0.16    52564.23     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
34-1424    16310    0/162/2356424    W     3.95    15    0    0.0    2.68    55800.32     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
35-1424    16543    0/63/2326471    W     1.29    4    0    0.0    0.75    55028.80     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
36-1424    17101    0/18/2331624    W     2.05    14    0    0.0    0.21    53656.66     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
37-1424    17102    0/20/2314444    W     1.51    19    0    0.0    0.29    55684.29     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
38-1424    19665    0/1/2295814    W     0.00    3    0    0.0    0.00    52187.61     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
39-984    8727    1/69/1464486    G     0.68    2906097    450284    2.8    0.83    33844.50     74.63.153.4    suspensionrevolution.com:80    POST /dap/dap-clickbank-6.0.php HTTP/1.1
40-1424    19720    0/1/2277467    W     0.00    2    0    0.0    0.00    55864.93     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
41-1424    18141    0/6/2270838    W     0.62    14    0    0.0    0.02    54059.95     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
42-983    18177    1/49/1440183    G     0.63    2910665    450302    2.8    0.57    31224.74     74.63.153.4    suspensionrevolution.com:80    POST /dap/dap-clickbank-6.0.php HTTP/1.1
43-1424    16104    2/57/2242969    W     2.62    5    0    8.8    0.83    56170.39     54.202.7.147    appcoiner.com:80    GET /start-2/?utm_expid=111102625-1.-ThtNpCTSByWcbkMGdBOow.1&ho
44-1424    16547    0/28/2247277    W     3.84    5    0    0.0    0.31    53028.08     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
45-1424    15797    0/80/2225028    W     3.24    5    0    0.0    1.63    51333.94     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
46-1424    19721    0/1/2205346    W     0.00    2    0    0.0    0.00    52025.79     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
47-1424    18142    0/11/2207016    W     0.94    1    0    0.0    0.07    51355.07     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
48-1424    17104    0/137/2172322    W     1.28    7    0    0.0    2.32    49665.11     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
49-1424    16314    0/63/2168481    W     4.14    5    0    0.0    1.16    49191.04     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
50-1424    19763    0/0/2141243    W     2.41    12    0    0.0    0.00    49538.97     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
51-1424    17106    0/20/2137681    W     1.24    7    0    0.0    0.29    49973.70     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
52-1424    16549    0/34/2125106    W     1.99    7    0    0.0    0.50    50442.63     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
53-1424    17107    0/20/2109740    W     2.18    2    0    0.0    0.31    48074.92     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
54-1424    18143    0/4/2087977    W     1.21    8    0    0.0    0.00    49243.64     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
55-1424    17114    0/30/2062106    W     0.34    17    0    0.0    0.46    48605.36     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
56-1424    17115    0/19/2064562    W     2.07    0    0    0.0    0.30    47600.46     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
57-1424    16569    0/41/2051051    W     3.50    8    0    0.0    0.54    47547.57     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
58-1424    17116    0/28/2023150    W     1.28    4    0    0.0    0.42    49170.14     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
59-1424    17117    0/30/2010767    W     1.41    1    0    0.0    0.51    47681.03     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
60-1424    17118    0/17/1999913    R     0.05    53    5    0.0    0.26    46914.84     65.30.135.196       
61-1424    16572    0/35/1978028    W     2.73    16    0    0.0    0.45    45848.82     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
62-1424    16573    0/27/1957566    W     1.60    0    0    0.0    0.45    46768.01     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
63-1424    16574    0/43/1936669    W     2.37    3    0    0.0    0.54    43520.07     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
64-1424    16575    0/28/1922381    W     1.54    1    0    0.0    0.33    45007.49     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
65-1424    16576    0/32/1903916    W     2.15    13    0    0.0    0.73    45117.83     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
66-1424    17119    0/28/1878566    W     1.18    6    0    0.0    0.45    44448.25     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1
67-1424    16578    1/39/1869043    K     2.01    0    0    1.2    0.53    44966.25     114.79.47.51    suspensionrevolution.com:80    GET /favicon.ico HTTP/1.1
68-1424    16579    0/37/1841958    W     2.59    5    0    0.0    0.48    44262.26     72.5.231.11    appcoiner.com:80    GET /?hopc2s=nakt123 HTTP/1.1

Esta é a minha maior saída:

root@ns513521 [~]# top
top - 08:21:31 up 153 days,  3:51,  1 user,  load average: 0.15, 0.27, 0.51
Tasks: 230 total,  2 running, 227 sleeping,  0 stopped,  1 zombie
Cpu(s):  0.4%us,  0.3%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32855908k total, 25102984k used,  7752924k free,  886004k buffers
Swap:  1569780k total,  63984k used,  1505796k free, 21254784k cached

Esta é minha saída ioStat:

Device:  tps  Blk_read/s  Blk_wrtn/s  Blk_read  Blk_wrtn
sda  21.99  1476.57  549.50 19540299168 7271856568
sdb  19.53  982.85  549.49 13006647390 7271718616
sdc  19.46  978.26  549.49 12945853934 7271718616
md2  9.78  492.22  399.94 6513868322 5292584264
md1  20.05  27.80  140.15  367920858 1854711136

    

Answer 1

Se isso acontecer sempre entre 4h e 6h da manhã, o culpado provavelmente será alguns trabalhos diários do cron. updatedb em particular pode retardar o acesso ao disco, o que pode fazer com que um banco de dados muito usado cruze um limite que aumentará a carga.

No caso geral, top mostrará o suficiente para diagnosticar qualquer problema de carga alta, enquanto o sqltuner e seu e-mail, que parece fornecer conexões http, só serão úteis se o problema estiver em seu banco de dados ou suas conexões http. No entanto, sua saída principal é tirada várias horas depois, o problema desapareceu (e você precisaria do resto da página).

    

Answer 2

Tente executar atsar em segundo plano por um dia ou dois. Isso irá capturar dados de desempenho em um arquivo para análise posterior. Examine seus dados para os períodos em que você tem uma média de carga alta. Se você não tem certeza de quando isso está ocorrendo, você pode obter os valores médios de carga de sar .

Se você também ativar a contabilização de processos, poderá identificar um programa que está causando o problema. Existem alguns programas que podem minimizar o impacto dos trabalhos em segundo plano. nice pode limitar o programa a usar ciclos de CPU sobressalentes. ionice pode reprogramar IO para que tenha menos impacto em processos de prioridade mais alta. No entanto, qualquer um deles pode aumentar significativamente o tempo de execução do programa problemático.

    

Answer 3

O CSF também faz um md5sum nos arquivos para compará-los com o md5sum do dia anterior. Eu vi isso realmente causar problemas em ambientes de produção (servidor compartilhado para hospedagem de empresa que hospeda 300 clientes). Se o seu servidor tiver muitos arquivos, isso pode ser um dos culpados. Cronjobs podem ser definitivamente culpados se ocorrer em uma determinada hora do dia. Qual é a saída do seguinte comando?

for u in 'cat /etc/passwd | cut -d":" -f1';do crontab -l -u $u;done