[BBLISA] Troubleshooting iptables/netfilter problems

[John Miller]

On Fri, Feb 28, 2014 at 2:16 PM, John Miller <johnmill at brandeis.edu> wrote:

> On Fri, Feb 28, 2014 at 12:14 PM, John P. Rouillard <rouilj at cs.umb.edu>wrote:
>
>>
>> In message <5310B66A.5090101 at brandeis.edu>,
>> John Miller writes:
>> >We're running into an issue with our LDAP servers where legitimate
>> >packets are being intermittently (say 1/10000) rejected by iptables.
>> >They match one of our ACCEPT rules, yet still get rejected (we're seeing
>> >the return ICMP traffic).
>>
>> Have you captured the inbound and outbound packets? Are you sure that
>> the first ldap packet is being rejected and not a packet that should
>> be allowed through by a:
>>
>>    --state established -j ACCEPT
>>
>> rule somewhere in your rule stack? This would also point to a state
>> table issue, but might provide some more insight.
>>
>>
> Yep--have done captures on the outbound packets (when sending from a test
> host); have not approached from the incoming point of view.  The packets
> are being logged by the next-to-last rule (last is a reject-all rule), so I
> don't think there's anything interfering in the routing path.
>
>
>> >I don't yet have any evidence to support this, but I'm thinking that
>> >we're bumping up against some sort of internal netfilter limit--perhaps
>> >with connection tracking or stateful matching.
>> >
>> >The problem seems to have cropped up in moving from RHEL 5 to RHEL 6.
>> >
>> >The rules in question:
>> >
>> >What should be hit:
>> >-A RH-Firewall-1-INPUT -s 129.64.0.0/255.255.0.0 -p tcp -m state --state
>> >NEW -m tcp --dport 636 -j ACCEPT
>>
>> I have not seen this, but you could simplify the rule and remove
>>
>>      "-m state --state NEW"
>>
>> for testing to see if the problem goes away. That should eliminate any
>> issues with the state setup and allow all ldap traffic to pass
>> through.
>>
>>
> Have tried this; results were inconclusive, so going to test it out again
> this afternoon.
>
>
>> Or what you could do is put a rule:
>>
>> -A RH-Firewall-1-INPUT -s 129.64.0.0/255.255.0.0 -p tcp -m state -m tcp
>> --dport 636 -j RECORD_LDAP
>>
>> after the rule you expect to get hit. If any packet hits this new
>> RECORD_LDAP rule, it should be an error and in the RECORD_LDAP chain
>> you can get a count or possibly full packet dump for a better idea of
>> what's happening.
>>
>> Also there was a bash script that automtically instrumented an
>> existing iptbles rule. Try googling for Tony Clayton iptables-trace.
>>
>>
> Thanks--was not aware of
>
>
>> Also you can try looking for:
>>
>>   conntrack-tools
>>
>> to see if that provides any info.
>>
>>
> Also saw that in my Googling; if the problem turns out to be connection
> tracking, I'll certainly post to that effect.  I'd have expected, however,
> to see something along the lines of
> *nf_conntrack*: *table* *full*, *dropping* *packet*.
>
> rather than the packet traversing the entire ruleset.  If I understand
> correctly, packets hit the conntrack module fairly early on in the
> netfilter process, so you'd think they would be dropped at that step.
>
>
>> >What is actually being hit:
>> >-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
>>
>> I assume this is the last rule in RH-Firewall-1-INPUT after the ldap rule.
>>
>>
> Correct--we allow a limited set of traffic, then reject everything else.
>
> The help is much appreciated, John.  Always helps to get a fresh
> perspective on something.
>
> John
>


I've just disabled connection tracking for all of the incoming LDAP traffic
on one of our servers.  Will see what sort of difference, if any, this
makes to our packet reject logs.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.bblisa.org/pipermail/bblisa/attachments/20140228/2e2e0df5/attachment-0001.html>