[BBLISA] Jim Gettys on Internet buffer bloat: smoke ping

[Bill Bogstad]

On Fri, Nov 16, 2012 at 2:16 PM, Tom Metro <tmetro+bblisa at gmail.com> wrote:
>
> Eventually the problem was traced back to an incorrectly configured QoS
> setting where the bandwidth limit was set in excess of the WAN speed.
> This was apparently filling up the router's buffers, and spilling over
> to its ability to pass packets between the WLAN and the LAN.

Sounds like a head of line blocking problem:

http://en.wikipedia.org/wiki/Head-of-line_blocking

I think Jim mentioned that in passing when I asked about Ethernet flow
control as a solution to bufferbloat.   (Although he admitted that he
was concentrating on "last mile" rather then data center/core network
equipment.)  The wikipedia article on
the subject (http://en.wikipedia.org/wiki/Ethernet_flow_control) has
the following to say:

"For example, a flow can come into a switch on a higher speed link
than the one it goes out, or several flows can come in over two or
more links that total more than an output link's bandwidth. These will
eventually exhaust any amount of buffering in the switch. However,
blocking the sending link will cause all flows over that link to be
delayed, even those that are not causing any congestion. This
situation is a case of head-of-line blocking, and can happen more
often in core network switches due to the large numbers of flows
generally being aggregated. Many switches use a technique called
Virtual Output Queues to eliminate the HOL blocking internally, so
will never send pause frames.[3]"


As I understand it, the issue is that Ethernet PAUSE frames are too
coarse a mechanism.   It works on a physical link rather then flow
level.   If you have a typical situation where you aggregate multiple
links to a single uplink, it's easy to fill up the uplink.   When that
happens, the link that has been paused has trouble communicating with
other (non-uplink) ports because it keeps getting PAUSEd due to the
uplink port being permanently full.  So an upload to the Internet
might cause you to have trouble accessing a local file server or
network printer even when theoretically you have plenty of local
bandwidth available to you.

Bill Bogstad