[BBLISA] Troubleshooting iptables/netfilter problems

[John Miller]

On Fri, Feb 28, 2014 at 12:14 PM, John P. Rouillard <rouilj at cs.umb.edu>wrote:

>
> In message <5310B66A.5090101 at brandeis.edu>,
> John Miller writes:
> >We're running into an issue with our LDAP servers where legitimate
> >packets are being intermittently (say 1/10000) rejected by iptables.
> >They match one of our ACCEPT rules, yet still get rejected (we're seeing
> >the return ICMP traffic).
>
> Have you captured the inbound and outbound packets? Are you sure that
> the first ldap packet is being rejected and not a packet that should
> be allowed through by a:
>
>    --state established -j ACCEPT
>
> rule somewhere in your rule stack? This would also point to a state
> table issue, but might provide some more insight.
>
>
Yep--have done captures on the outbound packets (when sending from a test
host); have not approached from the incoming point of view.  The packets
are being logged by the next-to-last rule (last is a reject-all rule), so I
don't think there's anything interfering in the routing path.


> >I don't yet have any evidence to support this, but I'm thinking that
> >we're bumping up against some sort of internal netfilter limit--perhaps
> >with connection tracking or stateful matching.
> >
> >The problem seems to have cropped up in moving from RHEL 5 to RHEL 6.
> >
> >The rules in question:
> >
> >What should be hit:
> >-A RH-Firewall-1-INPUT -s 129.64.0.0/255.255.0.0 -p tcp -m state --state
> >NEW -m tcp --dport 636 -j ACCEPT
>
> I have not seen this, but you could simplify the rule and remove
>
>      "-m state --state NEW"
>
> for testing to see if the problem goes away. That should eliminate any
> issues with the state setup and allow all ldap traffic to pass
> through.
>
>
Have tried this; results were inconclusive, so going to test it out again
this afternoon.


> Or what you could do is put a rule:
>
> -A RH-Firewall-1-INPUT -s 129.64.0.0/255.255.0.0 -p tcp -m state -m tcp
> --dport 636 -j RECORD_LDAP
>
> after the rule you expect to get hit. If any packet hits this new
> RECORD_LDAP rule, it should be an error and in the RECORD_LDAP chain
> you can get a count or possibly full packet dump for a better idea of
> what's happening.
>
> Also there was a bash script that automtically instrumented an
> existing iptbles rule. Try googling for Tony Clayton iptables-trace.
>
>
Thanks--was not aware of


> Also you can try looking for:
>
>   conntrack-tools
>
> to see if that provides any info.
>
>
Also saw that in my Googling; if the problem turns out to be connection
tracking, I'll certainly post to that effect.  I'd have expected, however,
to see something along the lines of
*nf_conntrack*: *table* *full*, *dropping* *packet*.

rather than the packet traversing the entire ruleset.  If I understand
correctly, packets hit the conntrack module fairly early on in the
netfilter process, so you'd think they would be dropped at that step.


> >What is actually being hit:
> >-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
>
> I assume this is the last rule in RH-Firewall-1-INPUT after the ldap rule.
>
>
Correct--we allow a limited set of traffic, then reject everything else.

The help is much appreciated, John.  Always helps to get a fresh
perspective on something.

John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.bblisa.org/pipermail/bblisa/attachments/20140228/cb0ec928/attachment.html>