Como eu usaria o bash para contar o número total de certas linhas em um arquivo?

Question

Eu quero que o bash passe por um arquivo de log de erros do apache2 e liste todas as suas linhas (semelhante a cat ), mas de uma maneira que conte quantas vezes cada linha está contida no arquivo e liste apenas DISTINCT linhas com uma contagem total delas.

Então, por exemplo, se o log de erros se parece com isso:

Error 1: file failed
Error 2: client failed
Error 3: server failed
Error 1: file failed

Eu gostaria de ter isso ecoado:

[2] Error 1: file failed
[1] Error 2: client failed
[1] Error 3: server failed

    

Answer 1

Use sort e uniq :

sort file | uniq -c

Se você deseja que a saída seja classificada pelo número de ocorrências, adicione | sort -n no final.

    

Answer 2

Usando o awk:

$ awk '{a[$0]++}END{for (i in a)print "["a[i]"]",i;}' file
[1] Error 3: server failed
[2] Error 1: file failed
[1] Error 2: client failed

Conte o não. de ocorrências na matriz a. Na etiqueta END, imprima todos os elementos de um.

    

Answer 3

Para que qualquer um dos exemplos mencionados seja realmente significativo, você precisará remover as datas de cada linha. Assumindo que uma linha no seu log se parece com isso:

[Wed Oct 24 10:17:17 2012] [notice] caught SIGTERM, shutting down

então você poderia fazer algo como:

tail -n 5000 /var/log/apache2/error_log |\
  cut -d] -f2- |\
  awk '{a[$0]++} END \
       {for (i in a) {printf "%s\t%s\n", a[i], i;} }' |\
  sort -n  

Isso produzirá uma saída semelhante a:

13   [error] [client 127.0.0.1] column: address_phonerename: phone, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: address_streetrename: street, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: address_zipcoderename: zipcode, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: user_activerename: active, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: user_idrename: id, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: user_nameFirstrename: nameFirst, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: user_nameLastrename: nameLast, referer: http://l27/symfony/web/varelager/adminpanel
13   [error] [client 127.0.0.1] column: user_usernamerename: username, referer: http://l27/symfony/web/varelager/adminpanel
56   [error] [client 127.0.0.1] PHP Parse error:  syntax error, unexpected '}' in /var/www/localhost/htdocs/tmp/php/log/index.php on line 6, referer: http://l27/symfony/web/varelager/adminpanel
56   [error] [client 127.0.0.1] PHP Parse error:  syntax error, unexpected ';' in /var/www/localhost/htdocs/tmp/php/log/index.php on line 9, referer: http://l27/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: address_cityrename: city, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: address_idrename: address_id, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: address_namerename: addr_name, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: address_streetrename: street, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: address_zipcoderename: zipcode, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: user_activerename: active, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: user_idrename: id, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: user_nameFirstrename: nameFirst, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: user_nameLastrename: nameLast, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
300  [error] [client 192.168.0.105] column: user_usernamerename: username, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
301  [error] [client 192.168.0.105] column: address_commentrename: comment, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
301  [error] [client 192.168.0.105] column: address_countryrename: country, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
301  [error] [client 192.168.0.105] column: address_emailrename: email, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
301  [error] [client 192.168.0.105] column: address_phonerename: phone, referer: http://192.168.0.100/symfony/web/varelager/adminpanel
451  [error] [client 192.168.0.103] File does not exist: /var/www/localhost/htdocs/WebServices