Parece que o problema realmente foi por causa dos pacotes chegando através de um e saindo através de outra interface.
Obrigado a @NickW por dica!
O que eu fiz foi criar duas tabelas de roteamento (uma para cada ISP),
# ip route show table isp1
default via 1.1.1.254 dev eth0
10.10.10.0/24 dev eth2 scope link src 10.10.10.10
127.0.0.0/8 dev lo scope link
1.1.1.0/24 dev eth0 scope link src 1.1.1.1
# ip route show table isp2
default via 192.168.2.254 dev eth1
10.10.10.0/24 dev eth2 scope link src 10.10.10.10
127.0.0.0/8 dev lo scope link
192.168.2.0/24 dev eth1 scope link src 192.168.2.1
Criamos duas regras para que os pacotes recebidos saibam o que fazer a seguir
# ip rule
0: from all lookup local
32764: from 192.168.2.1 lookup isp2
32765: from 1.1.1.1 lookup isp1
32766: from all lookup main
32767: from all lookup default
E configure a manobra de rota padrão entre os dois roteadores ISP:
# ip route
default
nexthop via 1.1.1.254 dev eth0 weight 1
nexthop via 192.168.2.254 dev eth1 weight 1
10.10.10.0/24 dev eth2 proto kernel scope link src 10.10.10.10
127.0.0.0/8 dev lo scope link
192.168.2.0/24 dev eth1 proto kernel scope link src 192.168.2.1
1.1.1.32/27 dev eth0 proto kernel scope link src 1.1.1.1
Saída de salvamento do iptables do firewall:
# Generated by iptables-save v1.4.12.1 on Thu May 22 11:27:18 2014
*nat
:PREROUTING ACCEPT [3549:255693]
:INPUT ACCEPT [1284:97835]
:OUTPUT ACCEPT [2116:146271]
:POSTROUTING ACCEPT [168:12371]
-A POSTROUTING -o eth0 -j MASQUERADE
-A POSTROUTING -o eth1 -j MASQUERADE
COMMIT
# Completed on Thu May 22 11:27:18 2014
# Generated by iptables-save v1.4.12.1 on Thu May 22 11:27:18 2014
*mangle
:PREROUTING ACCEPT [105533:57605385]
:INPUT ACCEPT [21841:6013942]
:FORWARD ACCEPT [82551:51514621]
:OUTPUT ACCEPT [24003:30467247]
:POSTROUTING ACCEPT [106672:81997112]
COMMIT
# Completed on Thu May 22 11:27:18 2014
# Generated by iptables-save v1.4.12.1 on Thu May 22 11:27:18 2014
*filter
:INPUT DROP [135:8424]
:FORWARD DROP [20:1360]
:OUTPUT DROP [0:0]
:LOCAL_DROP - [0:0]
:LOGGING - [0:0]
:LOG_DROP - [0:0]
:SMTP_LOG - [0:0]
-A INPUT -d 10.10.10.10/32 -i eth2 -p tcp -j ACCEPT
-A INPUT -s 192.168.2.0/24 -d 10.10.10.10/32 -i eth2 -p tcp -j ACCEPT
-A INPUT -d 1.1.1.1/32 -i eth2 -p tcp -j ACCEPT
-A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -s 1.1.1.1/32 -d 192.168.2.1/32 -i eth0 -p tcp -j ACCEPT
-A INPUT -s 192.168.2.1/32 -d 1.1.1.1/32 -i eth1 -p tcp -j ACCEPT
-A INPUT -s 10.10.10.11/32 -i eth2 -j ACCEPT
-A INPUT -s 192.168.2.1/32 -i eth1 -j ACCEPT
-A INPUT -s 1.1.1.1/32 -p tcp -j ACCEPT
-A INPUT -s 1.1.1.1/32 -p udp -j ACCEPT
-A INPUT -s 192.168.2.1/32 -p tcp -j ACCEPT
-A INPUT -s 192.168.2.1/32 -p udp -j ACCEPT
-A INPUT -s 10.10.10.0/24 -i eth0 -j DROP
-A INPUT -s 127.0.0.0/8 -i eth0 -j DROP
-A INPUT -s 10.10.10.0/24 -i eth1 -j DROP
-A INPUT -s 127.0.0.0/8 -i eth1 -j DROP
-A INPUT -p tcp -m tcp --dport 25 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A INPUT -p tcp -m tcp --dport 110 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A INPUT -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A INPUT -i eth2 -p tcp -m tcp --sport 68 --dport 67 -j ACCEPT
-A INPUT -i eth2 -p udp -m udp --sport 68 --dport 67 -j ACCEPT
-A INPUT -p udp -m udp --sport 53 -j ACCEPT
-A INPUT -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -p tcp -m tcp --sport 53 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -s 10.10.10.0/24 -j LOCAL_DROP
-A INPUT -p udp -j DROP
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -j DROP
-A FORWARD -i eth2 -o eth1 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth1 -o eth2 -m state --state NEW,RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth0 -o eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth1 -o eth2 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth2 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth2 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i eth1 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 10.10.10.0/24 -d 10.10.10.0/24 -i eth2 -p tcp -j ACCEPT
-A FORWARD -s 10.10.10.0/24 -d 10.10.10.0/24 -i eth2 -p udp -j ACCEPT
-A FORWARD -s 192.168.2.0/24 -d 192.168.2.0/24 -i eth1 -j ACCEPT
-A FORWARD -s 10.10.10.0/24 -i eth2 -p tcp -m tcp --dport 22 -j ACCEPT
-A FORWARD -s 10.10.10.0/24 -i eth2 -p tcp -m tcp --dport 25 -j ACCEPT
-A FORWARD -s 10.10.10.0/24 -i eth2 -p tcp -m tcp --dport 80 -j ACCEPT
-A FORWARD -s 10.10.10.0/24 -i eth2 -p tcp -m tcp --dport 110 -j ACCEPT
-A FORWARD -s 10.10.10.0/24 -i eth2 -p tcp -m tcp --dport 993 -j ACCEPT
-A FORWARD -s 10.10.10.0/24 -i eth2 -p tcp -m tcp --dport 443 -j ACCEPT
-A FORWARD -s 10.10.10.0/24 -i eth2 -p udp -m udp --dport 53 -j ACCEPT
-A FORWARD -s 10.10.10.0/24 -i eth2 -p tcp -m tcp --dport 53 -j ACCEPT
-A FORWARD -s 10.10.10.0/24 -i eth2 -p udp -m udp --sport 137 --dport 137 -j ACCEPT
-A FORWARD -s 10.10.10.0/24 -i eth2 -p tcp -m tcp --dport 143 -j ACCEPT
-A FORWARD -s 10.10.10.11/32 -i eth2 -p tcp -j ACCEPT
-A FORWARD -s 10.10.10.11/32 -i eth2 -p udp -j ACCEPT
-A FORWARD -s 10.10.10.18/32 -i eth2 -p tcp -j ACCEPT
-A FORWARD -s 10.10.10.18/32 -i eth2 -p udp -j ACCEPT
-A FORWARD -s 10.10.10.1/32 -i eth2 -p tcp -j ACCEPT
-A FORWARD -s 10.10.10.1/32 -i eth2 -p udp -j ACCEPT
-A OUTPUT -s 1.1.1.1/32 -o eth0 -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -s 10.10.10.0/24 -d 10.10.10.0/24 -o eth2 -j ACCEPT
-A OUTPUT -s 1.1.1.1/32 -o eth2 -j ACCEPT
-A OUTPUT -s 192.168.2.0/24 -d 192.168.2.0/24 -o eth1 -j ACCEPT
-A OUTPUT -s 1.1.1.1/32 -o eth1 -j ACCEPT
-A OUTPUT -o eth1 -j ACCEPT
-A OUTPUT -o eth0 -j ACCEPT
-A OUTPUT -s 192.168.2.1/32 -o eth2 -j ACCEPT
-A OUTPUT -s 1.1.1.1/32 -o eth2 -j ACCEPT
-A OUTPUT -s 10.10.10.0/24 -d 1.1.1.1/32 -o eth2 -j ACCEPT
-A OUTPUT -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A LOCAL_DROP -p tcp -j ULOG --ulog-prefix "[---LOCAL DROP---]: "
-A LOCAL_DROP -p udp -j ULOG --ulog-prefix "[---LOCAL DROP---]: "
-A LOCAL_DROP -j DROP
-A LOGGING -j ULOG --ulog-prefix "LOGGING: "
-A LOGGING -j ACCEPT
-A LOG_DROP -p tcp -j ULOG --ulog-prefix "[---TCP LOGDROP---]: "
-A LOG_DROP -p udp -j ULOG --ulog-prefix "[---UDP LOGDROP---]: "
-A LOG_DROP -j DROP
-A SMTP_LOG -j ACCEPT
COMMIT
# Completed on Thu May 22 11:27:18 2014
A configuração do firewall provavelmente está longe de ser perfeita, mas funciona e agora tenho medo de tocá-la: D