Por que o 'mpstat' e o 'pidstat' podem não concordar com os níveis de carga da CPU?

Question

Estou monitorando um processo mysql no Ubuntu 12.04 usando mpstat e pidstat , simultaneamente. Ao depurar seu uso da CPU, notei algo incomum. Embora ambas as ferramentas concordem que a carga da CPU está ocorrendo, elas discordam sobre o tipo de carga que é. Aqui está uma fatia de tempo exemplar:

$ pidstat -p 2763 1
01:36:36 PM       PID    %usr %system  %guest    %CPU   CPU  Command
01:36:36 PM      2763    0.00    0.00    0.00    0.00     0  mysqld
01:36:37 PM      2763    0.00    1.00    0.00    1.00     0  mysqld
01:36:38 PM      2763    0.00    5.00    0.00    5.00     0  mysqld
01:36:39 PM      2763    0.00   11.00    0.00   11.00     0  mysqld
01:36:40 PM      2763    0.00   12.00    0.00   12.00     0  mysqld
01:36:41 PM      2763    0.00    0.00    0.00    0.00     0  mysqld
01:36:42 PM      2763    0.00    0.00    0.00    0.00     0  mysqld
01:36:43 PM      2763    0.00    5.00    0.00    5.00     0  mysqld
01:36:44 PM      2763    0.00  170.00    0.00  170.00     0  mysqld
01:36:45 PM      2763    0.00  150.00    0.00  150.00     0  mysqld
01:36:46 PM      2763    0.00    4.00    0.00    4.00     0  mysqld
01:36:47 PM      2763    0.00    7.00    0.00    7.00     0  mysqld
01:36:48 PM      2763    0.00    7.00    0.00    7.00     0  mysqld
01:36:49 PM      2763    0.00   59.00    0.00   59.00     0  mysqld
01:36:50 PM      2763    0.00   85.00    0.00   85.00     0  mysqld

Observe que pidstat está reportando o uso pesado no nível do sistema em vários pontos no tempo. Aqui está mpstat para praticamente o mesmo período de tempo:

$ mpstat -P ALL 1
01:36:37 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
01:36:38 PM  all    0.38    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.62
01:36:38 PM    0    2.02    0.00    0.00    0.00    0.00    0.00    0.00    0.00   97.98
01:36:38 PM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:38 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:38 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:38 PM    4    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:38 PM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:38 PM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:38 PM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

01:36:38 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
01:36:39 PM  all    0.88    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.12
01:36:39 PM    0    4.08    0.00    0.00    0.00    0.00    0.00    0.00    0.00   95.92
01:36:39 PM    1    2.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   98.00
01:36:39 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:39 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:39 PM    4    0.99    0.00    0.99    0.00    0.00    0.00    0.00    0.00   98.02
01:36:39 PM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:39 PM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:39 PM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

01:36:39 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
01:36:40 PM  all    0.62    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.38
01:36:40 PM    0    5.05    0.00    0.00    0.00    0.00    0.00    0.00    0.00   94.95
01:36:40 PM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:40 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:40 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:40 PM    4    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:40 PM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:40 PM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:40 PM    7    0.99    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.01

01:36:40 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
01:36:41 PM  all    1.25    0.00    0.25    0.00    0.00    0.12    0.00    0.00   98.38
01:36:41 PM    0   10.00    0.00    1.00    0.00    0.00    1.00    0.00    0.00   88.00
01:36:41 PM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:41 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:41 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:41 PM    4    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:41 PM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:41 PM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:41 PM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

01:36:41 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
01:36:42 PM  all    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:42 PM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:42 PM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:42 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:42 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:42 PM    4    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:42 PM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:42 PM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:42 PM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

01:36:42 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
01:36:43 PM  all    0.12    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.88
01:36:43 PM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:43 PM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:43 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:43 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:43 PM    4    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:43 PM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:43 PM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:43 PM    7    1.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.00

01:36:43 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
01:36:44 PM  all    2.88    0.00    0.00    0.00    0.00    0.00    0.00    0.00   97.12
01:36:44 PM    0   22.77    0.00    0.99    0.00    0.00    0.00    0.00    0.00   76.24
01:36:44 PM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:44 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:44 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:44 PM    4    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:44 PM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:44 PM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:44 PM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

01:36:44 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
01:36:45 PM  all   23.72    0.00    0.12    0.00    0.00    0.12    0.00    0.00   76.03
01:36:45 PM    0   99.00    0.00    0.00    0.00    0.00    1.00    0.00    0.00    0.00
01:36:45 PM    1   89.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   11.00
01:36:45 PM    2    1.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.00
01:36:45 PM    3    0.00    0.00    0.99    0.00    0.00    0.00    0.00    0.00   99.01
01:36:45 PM    4    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:45 PM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:45 PM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:45 PM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

01:36:45 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
01:36:46 PM  all   13.89    0.00    0.00    0.00    0.00    0.00    0.13    0.00   85.98
01:36:46 PM    0   34.34    0.00    0.00    0.00    0.00    1.01    0.00    0.00   64.65
01:36:46 PM    1   69.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   31.00
01:36:46 PM    2    5.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   95.00
01:36:46 PM    3    3.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   97.00
01:36:46 PM    4    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:46 PM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:46 PM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:46 PM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

01:36:46 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
01:36:47 PM  all    0.50    0.00    0.00    0.00    0.00    0.13    0.00    0.00   99.37
01:36:47 PM    0    3.96    0.00    0.00    0.00    0.00    0.00    0.00    0.00   96.04
01:36:47 PM    1    1.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.00
01:36:47 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:47 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:47 PM    4    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:47 PM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:47 PM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:47 PM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

01:36:47 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
01:36:48 PM  all    0.63    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.37
01:36:48 PM    0    5.10    0.00    0.00    0.00    0.00    0.00    0.00    0.00   94.90
01:36:48 PM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:48 PM    2    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:48 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:48 PM    4    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:48 PM    5    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:48 PM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
01:36:48 PM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00

Não estou surpreso em vê-los discordar das porcentagens gerais, mas o que não consigo descobrir é por que pidstat e mpstat não concordam com o tipo de carga da CPU. mpstat indica que a carga está acontecendo no nível do aplicativo. pidstat indica que está acontecendo no nível do kernel. O que está acontecendo aqui? Presumi que ambos eram pontos de vista em /proc , mas não sou versado o suficiente em seus internos para saber onde eles estão recebendo sinais diferentes.

EDIT: Melhor eu posso dizer que isso resulta de discrepâncias entre o mecanismo de cálculo /proc usado por cada programa.

pidstat está usando /proc/<pid>/stat : link

mpstat está usando /proc/uptime e /proc/stat : link

    

Answer 1

Então, eu tenho uma causa, mas não é realmente uma causa raiz:

mpstat depende de /proc/stat para suas métricas de CPU. pidstat depende de /proc/<pid>/stat . Esses arquivos às vezes discordam sobre o número de jiffies de kernel observados em um determinado período de tempo, o que, suponho, faz sentido, já que estamos falando sobre esses incrementos de resolução de tempo:

17:06:41 /proc/stat 158257197
17:06:46 /proc/stat 158257230
17:06:51 /proc/stat 158257236
17:06:56 /proc/stat 158257279
# t3 - t0 = 82 jiffies

versus ...

17:06:41 /proc/2763/stat 208862696
17:06:46 /proc/2763/stat 208862971
17:06:51 /proc/2763/stat 208863114
17:06:56 /proc/2763/stat 208863143
# t3 - t0 = 447 jiffies

Cada diferença está, obviamente, sujeita a cálculo derivativo dependendo do código-fonte de mpstat e pidstat , massageando em porcentagens, etc.

Se você aprofundar em /proc e soma /proc/<pid>/task/**/stat , o resultado faz mais sentido. Esse número cresce mais lentamente que /proc/stat como um todo e significativamente mais lentamente que /proc/<pid>/stat .

Parece que se eu vou levar isso adiante, eu preciso entender melhor como o kernel está escrevendo esses arquivos, para minha versão particular. É claro que /proc/<pid>/stat não concorda com /proc/<pid>/task/**/stat .