Depurando tráfego de entrada alto

3

Um dos meus servidores de desenvolvimento (CentOS 6) viu de repente um enorme aumento no tráfego de rede de entrada

Está atrasando o rastreamento, o SSH leva > 10 segundos para fazer login e há um atraso na digitação, os sites estão acabando, o Nagios está chateado porque as verificações do NRPE mantêm o tempo limite (esse é meu host Nagios) então parece haver uma tempestade repentina de tráfego de nework, mas não consigo descobrir de onde ele vem. O servidor tem um IP público, portanto está diretamente acessível, ele executa um conjunto de regras IPTables muito restritivas (permitindo apenas 80, 443 e algumas outras portas de utilitário para coisas como Jenkins). Eu tentei usar ferramentas como iftop , mas elas não estão mostrando nada fora do comum. Não tenho certeza se isso ocorre porque o IPTables está bloqueando as conexões para que elas não sejam exibidas, mas como não tenho certeza se esses dispositivos externos estão tentando se conectar ao meu servidor ou qualquer outra coisa. Parece estranho que ele esteja tornando o SSH lento e outros serviços não responsivos, mas o tráfego de rede começou ao mesmo tempo e comecei a receber problemas. Onde procuro descobrir de onde vem esse tráfego e como pará-lo? Eu não tenho acesso direto a nenhum dos roteadores, mas posso fazer o que quiser no servidor. Eu olhei em / var / log / messages e há um monte de mensagens estranhas sobre o DNS que eu nunca vi antes, mas elas não parecem ser erros, apenas logging exageradamente detalhado (veja abaixo).

Material útil padrão;

[sr@ns309372 ~]$ sudo uptime
 23:51:41 up  6:30,  3 users,  load average: 0.03, 0.12, 0.11
[sr@ns309372 ~]$ sudo free -m
             total       used       free     shared    buffers     cached
Mem:          3920       2197       1722          0        103       1060
-/+ buffers/cache:       1032       2887
Swap:         1019          0       1019
[sr@ns309372 ~]$ sudo tail -n 30 /var/log/messages
Apr 18 23:11:08 ns309372 named[2451]: success resolving 'ftp.halifax.rwth-aachen.de/A' (in 'rwth-aachen.de'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:11:10 ns309372 named[2451]: success resolving 'deneb.dfn.de/AAAA' (in 'dfn.de'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:11:10 ns309372 named[2451]: success resolving 'ns1.leaseweb.nl/AAAA' (in 'leaseweb.nl'?) after disabling EDNS
Apr 18 23:11:15 ns309372 named[2451]: success resolving 'ns4.leaseweb.net/AAAA' (in 'leaseweb.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:11:22 ns309372 named[2451]: success resolving 'pkg.jenkins-ci.org/A' (in 'jenkins-ci.org'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:11:30 ns309372 named[2451]: success resolving 'mirror.ovh.net/A' (in 'ovh.net'?) after disabling EDNS
Apr 18 23:11:30 ns309372 named[2451]: success resolving 'mirror.ovh.net/AAAA' (in 'ovh.net'?) after disabling EDNS
Apr 18 23:33:54 ns309372 named[2451]: success resolving 'vs1.nagios.org/A' (in 'nagios.org'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:36 ns309372 named[2451]: success resolving 'ns.ripe.net/A' (in 'ripe.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:36 ns309372 named[2451]: success resolving 'dns1.ntli.net/AAAA' (in 'ntli.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:37 ns309372 named[2451]: success resolving 'dns2.ntli.net/AAAA' (in 'ntli.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:37 ns309372 named[2451]: success resolving 'dns2.ntli.net/A' (in 'ntli.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:38 ns309372 named[2451]: success resolving 'sec1.apnic.net/AAAA' (in 'apnic.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:34:39 ns309372 named[2451]: success resolving 'sec3.apnic.net/AAAA' (in 'apnic.net'?) after disabling EDNS
Apr 18 23:34:40 ns309372 named[2451]: success resolving 'sec3.apnic.net/A' (in 'apnic.net'?) after disabling EDNS
Apr 18 23:34:40 ns309372 named[2451]: success resolving 'dns2.ntli.net/AAAA' (in 'ntli.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:35:02 ns309372 named[2451]: success resolving 'urlatron.com/AAAA' (in 'urlatron.com'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:35:03 ns309372 named[2451]: success resolving 'urlatron.com/A' (in 'urlatron.com'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:35:56 ns309372 named[2451]: success resolving 'bitbucket.org/A' (in 'bitbucket.org'?) after disabling EDNS
Apr 18 23:48:26 ns309372 named[2451]: success resolving '113.155.23.94.in-addr.arpa/PTR' (in '155.23.94.in-addr.arpa'?) after disabling EDNS
Apr 18 23:48:29 ns309372 named[2451]: success resolving '8.137.145.217.in-addr.arpa/PTR' (in '217.in-addr.arpa'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:48:29 ns309372 named[2451]: success resolving '10.169.216.196.in-addr.arpa/PTR' (in '169.216.196.in-addr.arpa'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:48:29 ns309372 named[2451]: success resolving 'ns2.lacnic.net/AAAA' (in 'lacnic.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:48:30 ns309372 named[2451]: success resolving 'ns2.dns.br/AAAA' (in 'br'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:48:30 ns309372 named[2451]: success resolving 'ns2.dns.br/A' (in 'br'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:48:34 ns309372 named[2451]: success resolving 'ns2.afrinic.net/A' (in 'afrinic.net'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:50:03 ns309372 named[2451]: success resolving 'urlatron.com/A' (in 'urlatron.com'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:50:04 ns309372 named[2451]: success resolving 'ns2.ecogeek.org/A' (in 'ecogeek.org'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:50:05 ns309372 named[2451]: success resolving 'ns1.ecogeek.org/AAAA' (in 'ecogeek.org'?) after reducing the advertised EDNS UDP packet size to 512 octets
Apr 18 23:50:05 ns309372 named[2451]: success resolving 'urlatron.com/AAAA' (in 'urlatron.com'?) after reducing the advertised EDNS UDP packet size to 512 octets
[sr@ns309372 ~]$ sudo ifconfig
eth0      Link encap:Ethernet  HWaddr 00:27:0E:0B:86:51  
          inet addr:188.165.192.119  Bcast:188.165.192.255  Mask:255.255.255.0
          inet6 addr: fe80::227:eff:fe0b:8651/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:456082 errors:0 dropped:91 overruns:0 frame:0
          TX packets:821015 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:59793427 (57.0 MiB)  TX bytes:1008283171 (961.5 MiB)
          Interrupt:43 Base address:0xc000 

eth0:0    Link encap:Ethernet  HWaddr 00:27:0E:0B:86:51  
          inet addr:94.23.155.32  Bcast:94.23.155.32  Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:43 Base address:0xc000 

eth0:1    Link encap:Ethernet  HWaddr 00:27:0E:0B:86:51  
          inet addr:94.23.155.113  Bcast:94.23.155.113  Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:43 Base address:0xc000 

eth0:2    Link encap:Ethernet  HWaddr 00:27:0E:0B:86:51  
          inet addr:178.32.48.78  Bcast:178.32.48.78  Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Interrupt:43 Base address:0xc000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:169675 errors:0 dropped:0 overruns:0 frame:0
          TX packets:169675 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:172646550 (164.6 MiB)  TX bytes:172646550 (164.6 MiB)
[sr@ns309372 ~]$ sudo sar -n DEV 1 3
Linux 2.6.38.2-grsec-xxxx-grs-ipv6-64 (ns309372.ovh.net)    18/04/12    _x86_64_    (2 CPU)

23:57:35        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
23:57:36           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:36       dummy0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:36         eth0     13.00      8.00      1.11      5.08      0.00      0.00      0.00
23:57:36        tunl0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:36         sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:36      ip6tnl0      0.00      0.00      0.00      0.00      0.00      0.00      0.00

23:57:36        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
23:57:37           lo     10.00     10.00      2.92      2.92      0.00      0.00      0.00
23:57:37       dummy0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:37         eth0     11.00      8.00      0.91      3.47      0.00      0.00      0.00
23:57:37        tunl0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:37         sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:37      ip6tnl0      0.00      0.00      0.00      0.00      0.00      0.00      0.00

23:57:37        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
23:57:38           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:38       dummy0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:38         eth0      7.00      9.00      7.54      1.33      0.00      0.00      0.00
23:57:38        tunl0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:38         sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
23:57:38      ip6tnl0      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Average:        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s
Average:           lo      3.33      3.33      0.97      0.97      0.00      0.00      0.00
Average:       dummy0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth0     10.33      8.33      3.19      3.30      0.00      0.00      0.00
Average:        tunl0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:      ip6tnl0      0.00      0.00      0.00      0.00      0.00      0.00      0.00

Eu tenho um número de interfaces 'virtuais' para suportar vários IPs, apenas uma interface física

    
por Smudge 19.04.2012 / 01:58

1 resposta

2

Eu faria duas coisas:

  1. Registre o tráfego no iptables. Você cria uma regra e usa os destinos LOG (usa o sistema de log do kernel) e ULOG (direciona logs para sockets ao invés do sistema de kernel) . Então, você provavelmente estaria interessado em todos os -A INPUT pacotes sendo enviados para -j LOG com talvez --log-prefix "incoming packets " e --log-level 6 (segue os níveis do syslog)

  2. Monitore seus processos para ver se algum deles é o destinatário da vasta largura de banda. Sugiro o uso de nethogs . Pode ser que o tráfego esteja passando pelo iptables, mas não sendo gravado no disco, mas sim descartado imediatamente por algum processo excêntrico.

por 19.04.2012 / 02:13