Estou tentando imprimir uma lista das visitas mais frequentes dos Agentes do usuário no log de acesso do Nginx.
Eu até agora tenho essa ideia:
cat /var/log/nginx/access.log | awk -F\" '{print $6}' | sort | uniq -c | sort -nr | head -20
E isso de fato me dá os 20 principais Agentes do Usuário classificados por quantas vezes eles visitaram o site.
No entanto, gostaria de modificar isso para contar apenas as visitas nos últimos 30 dias e estou com dificuldades para obter um comando em funcionamento.
cat from=$(date -d'now-30 days' +[%d/%b:%H:%M:%S) to==$(date +[%d/%b:%H:%M:%S) /var/log/nginx/access.log | awk -F\" '{print $6}' | sort | uniq -c | sort -nr | head -20
O objetivo é identificar os indexadores / bots de mecanismos de pesquisa que visitam o site com muita frequência, então, se você puder recomendar algo mais para isso, entre em contato.
Eu incluo algumas linhas de exemplo do arquivo de log abaixo: -
157.55.39.46 - - [11/Jun/2018:13:23:20 +0100] "GET /accessories/antique-gold--pine--wall-mounted?width=1308%2C1334 HTTP/1.1 "200 27726 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"[RT:3.041] [C:995725]
157.55.39.46 - - [11/Jun/2018:13:23:24 +0100] "GET /accessories/antique-gold--chrome--pine--wall-mounted?limit=25&mode=list HTTP/1.1 "200 34206 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" "-"[RT:3.412] [C:995725]
54.36.148.11 - - [11/Jun/2018:13:23:29 +0100] "GET /bathroom-mirrors/mirrors/700 HTTP/1.1 "404 10570 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)" "-"[RT:0.393] [C:995737]
Atualmente, o comando imprime o seguinte: -
8603 Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
4051 Mozilla/5.0 (compatible; SemrushBot/2~bl; +http://www.semrush.com/bot.html)
1707 Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
1585 Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.79 Safari/537.36
1519 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
1368 Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
1185 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36
993 Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)
903 Mozilla/5.0 (iPhone; CPU iPhone OS 11_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.0 Mobile/15E148 Safari/604.1
658 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
648 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1 Safari/605.1.15
572 Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0
435 Mozilla/5.0 (iPhone; CPU iPhone OS 11_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.0 Mobile/15E148 Safari/604.1
336 Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
324 Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Firefox/24.0
324 Mozilla/5.0 (iPad; CPU OS 10_3_3 like Mac OS X) AppleWebKit/603.1.30 (KHTML, like Gecko) CriOS/67.0.3396.69 Mobile/14G60 Safari/602.1
319 Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0
309 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
295 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.79 Safari/537.36
280 Mozilla/5.0 (iPad; CPU OS 11_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.0 Mobile/15E148 Safari/604.1