Adiciona quebra de linha entre registros de data e hora no log com regex

Question

Eu tenho alguns dados de registro:

2017-12-03 01:35:58 [Notice] syslog: local  IP address 
2017-12-03 01:35:58 [Notice] syslog: remote IP address 
2017-12-03 01:35:58 [Notice] syslog: primary   DNS address 
2017-12-03 01:35:58 [Notice] syslog: secondary DNS address 
2017-12-03 01:35:59 [Warning] kernel: Link State: PVC_8_0 logistic interface up.
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: started, version 2.52 cachesize 150
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP
2017-12-03 01:35:59 [Warning] dnsmasq[10463]: ignoring nameserver 127.0.0.1 - local interface
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: using nameserver 87.216.1.66#53(via ppp80)
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: using nameserver 87.216.1.65#53(via ppp80)
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: read /etc/hosts - 6 addresses
2017-12-03 01:36:00 [Informational] dnsmasq[10532]: started, version 2.52 cachesize 150
2017-12-03 01:36:00 [Informational] dnsmasq[10532]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP
2017-12-03 01:36:00 [Informational] dnsmasq[10532]: using nameserver 87.216.1.66#53(via ppp80)
2017-12-03 01:36:01 [Warning] kernel: ^M
2017-12-03 01:36:01 [Warning] kernel: Send DNS Query : domain=ntp2.jazztel.com qType=A dnsServer=87.216.1.65
2017-12-03 01:36:01 [Warning] kernel: domain: ntp2.jazztel.com , IP: 87.216.1.241
2017-12-03 01:36:01 [Warning] kernel: sntp server=ntp2.jazztel.com: 0x5 ntpServerIP=87.216.1.241

Eu quero adicionar uma quebra de linha toda vez que o registro de data e hora for alterado para que fique assim:

2017-12-03 01:35:58 [Notice] syslog: local  IP address 
2017-12-03 01:35:58 [Notice] syslog: remote IP address 
2017-12-03 01:35:58 [Notice] syslog: primary   DNS address 
2017-12-03 01:35:58 [Notice] syslog: secondary DNS address 

2017-12-03 01:35:59 [Warning] kernel: Link State: PVC_8_0 logistic interface up.
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: started, version 2.52 cachesize 150
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP
2017-12-03 01:35:59 [Warning] dnsmasq[10463]: ignoring nameserver 127.0.0.1 - local interface
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: using nameserver 87.216.1.66#53(via ppp80)
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: using nameserver 87.216.1.65#53(via ppp80)
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: read /etc/hosts - 6 addresses

2017-12-03 01:36:00 [Informational] dnsmasq[10532]: started, version 2.52 cachesize 150
2017-12-03 01:36:00 [Informational] dnsmasq[10532]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP
2017-12-03 01:36:00 [Informational] dnsmasq[10532]: using nameserver 87.216.1.66#53(via ppp80)

2017-12-03 01:36:01 [Warning] kernel: ^M
2017-12-03 01:36:01 [Warning] kernel: Send DNS Query : domain=ntp2.jazztel.com qType=A dnsServer=87.216.1.65
2017-12-03 01:36:01 [Warning] kernel: domain: ntp2.jazztel.com , IP: 87.216.1.241
2017-12-03 01:36:01 [Warning] kernel: sntp server=ntp2.jazztel.com: 0x5 ntpServerIP=87.216.1.241

Funciona no link :

s/(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} ).*\n(?!)/$0\n/g

Mas quando eu tento no Terminal (OSX) ele não faz nada:

curl -s http://192.168.1.1/cgi-bin/status_log2.cgi | grep 2017 | tail -n 30 | perl -pe 's/(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} ).*\n(?!)/$0\n/g'

Eu também tentei gsed e sed sem sucesso.

(Bônus se houver uma maneira de remover completamente todos os carimbos de hora redundantes!)

    

Answer 1

Para colocar uma quebra de linha antes de cada nova hora:

awk '!a[$1,$2]++ && NR>1{print ""} 1'

Como funciona: No awk, $1 e $2 são os primeiros campos, neste caso, a data e a hora. a[$1,$2] é uma matriz associativa que conta quantas vezes vimos esses dois campos. Se já vimos essa data e hora antes, !a[$1,$2] e não estamos na primeira linha, NR>1 , então imprimimos uma linha em branco para separação, print "" . O% final 1 é apenas uma abreviação para imprimir na linha atual.

Exemplo

Com seus logs de amostra no arquivo logfile :

$ awk '!a[$1,$2]++ && NR>1{print ""} 1' logfile
2017-12-03 01:35:58 [Notice] syslog: local  IP address 
2017-12-03 01:35:58 [Notice] syslog: remote IP address 
2017-12-03 01:35:58 [Notice] syslog: primary   DNS address 
2017-12-03 01:35:58 [Notice] syslog: secondary DNS address 

2017-12-03 01:35:59 [Warning] kernel: Link State: PVC_8_0 logistic interface up.
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: started, version 2.52 cachesize 150
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP
2017-12-03 01:35:59 [Warning] dnsmasq[10463]: ignoring nameserver 127.0.0.1 - local interface
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: using nameserver 87.216.1.66#53(via ppp80)
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: using nameserver 87.216.1.65#53(via ppp80)
2017-12-03 01:35:59 [Informational] dnsmasq[10463]: read /etc/hosts - 6 addresses

2017-12-03 01:36:00 [Informational] dnsmasq[10532]: started, version 2.52 cachesize 150
2017-12-03 01:36:00 [Informational] dnsmasq[10532]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP
2017-12-03 01:36:00 [Informational] dnsmasq[10532]: using nameserver 87.216.1.66#53(via ppp80)

2017-12-03 01:36:01 [Warning] kernel: ^M
2017-12-03 01:36:01 [Warning] kernel: Send DNS Query : domain=ntp2.jazztel.com qType=A dnsServer=87.216.1.65
2017-12-03 01:36:01 [Warning] kernel: domain: ntp2.jazztel.com , IP: 87.216.1.241
2017-12-03 01:36:01 [Warning] kernel: sntp server=ntp2.jazztel.com: 0x5 ntpServerIP=87.216.1.241

Removendo timestamps duplicados

$ awk '{if(a[$1,$2]++){gsub(/./," ",$1); gsub(/./," ",$2)} else if (NR>1) print""} 1' logfile
2017-12-03 01:35:58 [Notice] syslog: local  IP address 
                    [Notice] syslog: remote IP address
                    [Notice] syslog: primary DNS address
                    [Notice] syslog: secondary DNS address

2017-12-03 01:35:59 [Warning] kernel: Link State: PVC_8_0 logistic interface up.
                    [Informational] dnsmasq[10463]: started, version 2.52 cachesize 150
                    [Informational] dnsmasq[10463]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP
                    [Warning] dnsmasq[10463]: ignoring nameserver 127.0.0.1 - local interface
                    [Informational] dnsmasq[10463]: using nameserver 87.216.1.66#53(via ppp80)
                    [Informational] dnsmasq[10463]: using nameserver 87.216.1.65#53(via ppp80)
                    [Informational] dnsmasq[10463]: read /etc/hosts - 6 addresses

2017-12-03 01:36:00 [Informational] dnsmasq[10532]: started, version 2.52 cachesize 150
                    [Informational] dnsmasq[10532]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP
                    [Informational] dnsmasq[10532]: using nameserver 87.216.1.66#53(via ppp80)

2017-12-03 01:36:01 [Warning] kernel: ^M
                    [Warning] kernel: Send DNS Query : domain=ntp2.jazztel.com qType=A dnsServer=87.216.1.65
                    [Warning] kernel: domain: ntp2.jazztel.com , IP: 87.216.1.241
                    [Warning] kernel: sntp server=ntp2.jazztel.com: 0x5 ntpServerIP=87.216.1.241

Nesse caso, se tivermos visto a data, $1 e hora, $2 , antes, substituiremos o conteúdo por espaços em branco, gsub(/./," ",$1); gsub(/./," ",$2) . Se não, e se não estivermos na primeira linha, imprimiremos uma linha em branco para separação.

    

Answer 2

Você precisa dizer ao Perl para carregar a coisa toda de uma só vez, como normalmente lê a entrada linha por linha. Use -0777 para isso.

Além disso, $0 em Perl é o nome do script ( -e para um one-liner). Capture toda a linha e faça referência como $1 , use para a data:

perl -0777 -pe 's/((\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} ).*\n)(?!)/$1\n/g'

    

Answer 3

Eu gostaria de postar uma solução sed (GNU).

sed -nr 'h;s/^([^[]+) \[.*//;x;p;:a;g;N;s/^([^\n]+)\n(.*)//;Tb;:c;s/[0-9:-]([0-9 :-]+\[)/ /;tc;p;ba;:b;s/^[^\n]+//;P;D' logfile
2017-12-03 01:35:58 [Notice] syslog: local  IP address                                                                                      
                    [Notice] syslog: remote IP address                                                                                      
                    [Notice] syslog: primary   DNS address                                                                                  
                    [Notice] syslog: secondary DNS address                                                                                  

2017-12-03 01:35:59 [Warning] kernel: Link State: PVC_8_0 logistic interface up.                                                            
                    [Informational] dnsmasq[10463]: started, version 2.52 cachesize 150                                                     
                    [Informational] dnsmasq[10463]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP         
                    [Warning] dnsmasq[10463]: ignoring nameserver 127.0.0.1 - local interface                                               
                    [Informational] dnsmasq[10463]: using nameserver 87.216.1.66#53(via ppp80)                                              
                    [Informational] dnsmasq[10463]: using nameserver 87.216.1.65#53(via ppp80)                                              
                    [Informational] dnsmasq[10463]: read /etc/hosts - 6 addresses                                                           

2017-12-03 01:36:00 [Informational] dnsmasq[10532]: started, version 2.52 cachesize 150                                                     
                    [Informational] dnsmasq[10532]: compile time options: no-IPv6 GNU-getopt no-RTC no-DBus no-I18N no-DHCP no-TFTP         
                    [Informational] dnsmasq[10532]: using nameserver 87.216.1.66#53(via ppp80)                                              

2017-12-03 01:36:01 [Warning] kernel: ^M                                                                                                    
                    [Warning] kernel: Send DNS Query : domain=ntp2.jazztel.com qType=A dnsServer=87.216.1.65
                    [Warning] kernel: domain: ntp2.jazztel.com , IP: 87.216.1.241
                    [Warning] kernel: sntp server=ntp2.jazztel.com: 0x5 ntpServerIP=87.216.1.241