Announcement
The bufferbloat project [1] is pleased to announce the availability
of the debloat-testing Linux kernel git tree:
git://git.infradead.org/debloat-testing.git
The purpose of this tree is to provide a reasonably stable base for
the development and testing of new algorithms, miscellaneous fixes,
and maybe a few hacks intended to advance the cause of eliminating
or at least mitigating bufferbloat in the Linux world.
Introduction
Bufferbloat is a term coined by Jim Gettys to describe the increasing
prevalence of large and (particularly) unmanaged network buffers along
the network links that comprise the Internet [2]. If you are not aware
of the problems with network latency under load that the Internet is
already encountering, we encourage you to visit Jim Gettys' blog [3].
There Jim has begun to fit together enough puzzle pieces to at least
frame the issue.
Jim has also made available slides and an audio recording (edited
for time) from a presentation on this topic:
http://mirrors.bufferbloat.net/Talks/BellLabs01192011/
Kernel Bits
The debloat-testing tree is intended to track full and -rc releases
from linux-2.6, with interesting patches cherry-picked from net-next
and various experimental bits added on top. The current stable of
such patches includes the following:
Eric Dumazet (based on original work by Juliusz Chroboczek):
net_sched: SFB flow scheduler
stephen hemminger:
sched: CHOKe flow scheduler
John Fastabend:
net: implement mechanism for HW based QOS
net_sched: implement a root container qdisc sch_mqprio
John W. Linville:
mac80211: implement eBDP algorithm to fight bufferbloat
Nathaniel J. Smith:
iwlwifi: Simplify tx queue management
iwlwifi: Convert the tx queue high_mark to an atomic_t
iwlwifi: Invert the sense of the queue high_mark
iwlwifi: auto-tune tx queue size to minimize latency
iwlwifi: make current tx queue sizes visible in debugfs
Dave Taht:
Bufferbloat reduction for the e1000 driver that started it all
Reduce bufferbloated default for e1000e, increase dynamic range
Smash bufferbloat in the ath9k driver
Userland Bits
Patches for the userspace tc utility incorporating support for both the
CHOKe AQM and the Stochastic Fair Blue scheduler (SFB) are available:
https://github.com/dtaht/iproute2bufferbloat
Contributions
Please send any experimental or research-oriented patches related to
bufferbloat to the [email protected] list. Reminders
of more mainstream patches that may be relevant and/or interesting
for cherry-picking into debloat-testing are welcome there as well.
Obviously, patches that are ready for normal merge consideration
should continue to be sent to netdev, linux-wireless, linux-kernel,
or whatever other existing list is appropriate for them.
Thanks
Finally, we want to offer a huge thanks to the 130+ new members of
the bloat mailing list [4] for leaping into the fray, and to David
Woodhouse for hosting the debloat-testing tree at infradead.
Please help us beat the bloat. Good luck, and happy debloating!
Notes
[1] http://bufferbloat.net
[2] http://gettys.wordpress.com/what-is-bufferbloat-anyway/
[3] http://en.wordpress.com/tag/bufferbloat/
[4] https://lists.bufferbloat.net
--
John W. Linville Someday the world will need a hero, and you
[email protected] might be all we have. Be ready.
On Sun, Feb 27, 2011 at 4:56 PM, Dave Täht <[email protected]> wrote:
> Sedat Dilek <[email protected]> writes:
>
>> On Sun, Feb 27, 2011 at 4:31 PM, Dave Täht <[email protected]> wrote:
>>>
>>> Sedat Dilek <[email protected]> writes:
>>>
>>>> On Fri, Feb 25, 2011 at 11:22 PM, John W. Linville
>>>> <[email protected]> wrote:
>>>>> Announcement
>>>>>
>>>>> The bufferbloat project [1] is pleased to announce the availability
>>>>> of the debloat-testing Linux kernel git tree:
>>>>>
>>>>> git://git.infradead.org/debloat-testing.git
>>>
>>> ----snip----
>
>>> Excellent. At moment I would recommend building "low latency preempt
>>> desktop" kernels with a high HZ value (400 or 1000), enabling highres
>>> timers, and compiling in SFB as a module. (I'd like the default for SFB
>>> to be "m" rather than "n", too)
>>>
>
>> These "debloat guys" are fast :-). I was just preparing my
>> build-system (which I normally use to debianize linux-next kernels).
>> Any other recommendation for kernel-config options? For example:
>> linux-next has already CONFIG_NET_SCH_CHOKE (but I have unset it).
>
> Enable CHOKe.
>
> The HZ value change is due to my worry that we've smashed latency so
> much in the driver/mac layer that it's interacting with the higher
> layers somewhat badly... So we need to add more hooks to the servo loops
> involved in order to have a normal HZ.
>
>> Which commits are in debloat-testing GIT but not in linux-next tree?
>
> The current list was in the release announcement. More on the way
> (mostly embedded drivers at this point) git pull early and often!
>
>> Are you planning debloat feature for 2.6.39?
>
> Depends on how many testers we get and what the results are.
>
> I feel the eBDP stuff will not be ready during this release cycle. SFB
> and CHOKe are in net-next, so, probably. Various driver patches -
> particularly those that increase the available dynamic range via
> ethtool, (e.g lowering the bottommost TX queue limit to, like, 4,
> especially for home gateways) may make it out if people look harder into
> the issue.
>
>>
>> - Sedat -
>
> --
> Dave Taht
> http://nex-6.taht.net
>
OK, thanks for the explanations.
Concerning "more drivers":
What would I have to do to modify ath5k?
I looked into the ath9k patch in debloat-testing GIT and it was to mod
some (TX/BUF) values only.
Not sure if ath9k is/was "well" prepared or only a good choice by the
testers/committers as they own such a device.
- Sedat -
On Fri, Feb 25, 2011 at 11:22 PM, John W. Linville
<[email protected]> wrote:
> Announcement
>
> The bufferbloat project [1] is pleased to announce the availability
> of the debloat-testing Linux kernel git tree:
>
> git://git.infradead.org/debloat-testing.git
>
> The purpose of this tree is to provide a reasonably stable base for
> the development and testing of new algorithms, miscellaneous fixes,
> and maybe a few hacks intended to advance the cause of eliminating
> or at least mitigating bufferbloat in the Linux world.
>
> Introduction
>
> Bufferbloat is a term coined by Jim Gettys to describe the increasing
> prevalence of large and (particularly) unmanaged network buffers along
> the network links that comprise the Internet [2]. If you are not aware
> of the problems with network latency under load that the Internet is
> already encountering, we encourage you to visit Jim Gettys' blog [3].
> There Jim has begun to fit together enough puzzle pieces to at least
> frame the issue.
>
> Jim has also made available slides and an audio recording (edited
> for time) from a presentation on this topic:
>
> http://mirrors.bufferbloat.net/Talks/BellLabs01192011/
>
> Kernel Bits
>
> The debloat-testing tree is intended to track full and -rc releases
> from linux-2.6, with interesting patches cherry-picked from net-next
> and various experimental bits added on top. The current stable of
> such patches includes the following:
>
> Eric Dumazet (based on original work by Juliusz Chroboczek):
> net_sched: SFB flow scheduler
>
> stephen hemminger:
> sched: CHOKe flow scheduler
>
> John Fastabend:
> net: implement mechanism for HW based QOS
> net_sched: implement a root container qdisc sch_mqprio
>
> John W. Linville:
> mac80211: implement eBDP algorithm to fight bufferbloat
>
> Nathaniel J. Smith:
> iwlwifi: Simplify tx queue management
> iwlwifi: Convert the tx queue high_mark to an atomic_t
> iwlwifi: Invert the sense of the queue high_mark
> iwlwifi: auto-tune tx queue size to minimize latency
> iwlwifi: make current tx queue sizes visible in debugfs
>
> Dave Taht:
> Bufferbloat reduction for the e1000 driver that started it all
> Reduce bufferbloated default for e1000e, increase dynamic range
> Smash bufferbloat in the ath9k driver
>
> Userland Bits
>
> Patches for the userspace tc utility incorporating support for both the
> CHOKe AQM and the Stochastic Fair Blue scheduler (SFB) are available:
>
> https://github.com/dtaht/iproute2bufferbloat
>
> Contributions
>
> Please send any experimental or research-oriented patches related to
> bufferbloat to the [email protected] list. Reminders
> of more mainstream patches that may be relevant and/or interesting
> for cherry-picking into debloat-testing are welcome there as well.
>
> Obviously, patches that are ready for normal merge consideration
> should continue to be sent to netdev, linux-wireless, linux-kernel,
> or whatever other existing list is appropriate for them.
>
> Thanks
>
> Finally, we want to offer a huge thanks to the 130+ new members of
> the bloat mailing list [4] for leaping into the fray, and to David
> Woodhouse for hosting the debloat-testing tree at infradead.
>
> Please help us beat the bloat. Good luck, and happy debloating!
>
> Notes
>
> [1] http://bufferbloat.net
> [2] http://gettys.wordpress.com/what-is-bufferbloat-anyway/
> [3] http://en.wordpress.com/tag/bufferbloat/
> [4] https://lists.bufferbloat.net
> --
> John W. Linville Someday the world will need a hero, and you
> [email protected] might be all we have. Be ready.
>
Hi,
it should be "localversion-debloat" in the commit-subject in [1] (not
"localversion-wireless") :-).
"-db" as suffix is IMHO not very meaningful... Why not add simply a
suffix called "-debloat"?
(Anyway, I will revert this patch because I don't want to have any
suffix added automatically.)
I have several other questions, but I start compiling first and test
this debloat kernel.
Regards,
- Sedat -
[1] "Add localversion-wireless to identify builds from this tree."
http://git.infradead.org/debloat-testing.git/commit/3f9bdb4f44b076feda72d353d8ad717831416f36
Sedat Dilek <[email protected]> writes:
> On Sun, Feb 27, 2011 at 4:56 PM, Dave Täht <[email protected]> wrote:
>> Sedat Dilek <[email protected]> writes:
>>
>>> On Sun, Feb 27, 2011 at 4:31 PM, Dave Täht <[email protected]> wrote:
>>>>
>>>> Sedat Dilek <[email protected]> writes:
>>
>>> Are you planning debloat feature for 2.6.39?
>>
>> Depends on how many testers we get and what the results are.
>>
>> I feel the eBDP stuff will not be ready during this release cycle. SFB
>> and CHOKe are in net-next, so, probably. Various driver patches -
>> particularly those that increase the available dynamic range via
>> ethtool, (e.g lowering the bottommost TX queue limit to, like, 4,
>> especially for home gateways) may make it out if people look harder into
>> the issue.
>
> OK, thanks for the explanations.
>
> Concerning "more drivers":
> What would I have to do to modify ath5k?
> I looked into the ath9k patch in debloat-testing GIT and it was to mod
> some (TX/BUF) values only.
Yes, reducing your TX buffer size greatly is the first, best, and
easiest patch.
For wireless routers and cable home gateways especially, this research
shows that the total un-managed buffers in your system should be less
than 32.
http://www.cs.clemson.edu/~jmarty/papers/PID1154937.pdf
I found their data convincing, and there are dozens of other papers that
we are sorting out on the bufferbloat.net web site.
(PLEASE Note the key word there is un-managed)
0 would be the best value. :/
In the case of wireless, you also have retries to take into account.
I'd argue in those cases, that what I say above is that the number
should be FAR less than 32.
Now, whether there is a good compromise between throughput and latency
in that range in a DMA TX queue + TXQUEUE, remains to be seen.
> Not sure if ath9k is/was "well" prepared or only a good choice by the
> testers/committers as they own such a device.
My test network is mostly ath9k - the nanostation M5s and the WNDR5700
router described here:
http://www.bufferbloat.net/projects/bloat/wiki/Experiment_-_Bloated_LAGN_vs_debloated_WNDR5700
There are people looking into the ath6kl, but you're the first to step
up with the ath5k. :) Maybe the folk over at #ath6kl on irc can help.
The ath9k patch improves latency under load enormously - I can run voip
over it AND do big transfers and stream audio via samba... Which I
couldn't before - and DNS, ND, NTP, babel, etc behave much better, but
the currently hard coded nature of the TX queue limit does put an upper
limit on packet aggregation that the eBDP folk are trying to resolve
more generically.
In practice, at "normal" 180Mbit rates, with the new queue depth of 3, I
get most of the benefits of packet aggregation without the lag.
I do see higher packet loss than I would like, at present.
>
> - Sedat -
--
Dave Taht
http://nex-6.taht.net
On Sun, Feb 27, 2011 at 4:31 PM, Dave Täht <[email protected]> wrote:
>
> Sedat Dilek <[email protected]> writes:
>
>> On Fri, Feb 25, 2011 at 11:22 PM, John W. Linville
>> <[email protected]> wrote:
>>> Announcement
>>>
>>> The bufferbloat project [1] is pleased to announce the availability
>>> of the debloat-testing Linux kernel git tree:
>>>
>>> git://git.infradead.org/debloat-testing.git
>
> ----snip----
>
>> Hi,
>>
>> it should be "localversion-debloat" in the commit-subject in [1] (not
>> "localversion-wireless") :-). "-db" as suffix is IMHO not very
>> meaningful... Why not add simply a suffix called "-debloat"? (Anyway,
>> I will revert this patch because I don't want to have any suffix added
>> automatically.)
>>
>> I have several other questions, but I start compiling first and test
>> this debloat kernel.
>
> Excellent. At moment I would recommend building "low latency preempt
> desktop" kernels with a high HZ value (400 or 1000), enabling highres
> timers, and compiling in SFB as a module. (I'd like the default for SFB
> to be "m" rather than "n", too)
>
>>
>> Regards,
>> - Sedat -
>>
>> [1] "Add localversion-wireless to identify builds from this tree."
>> http://git.infradead.org/debloat-testing.git/commit/3f9bdb4f44b076feda72d353d8ad717831416f36
>> _______________________________________________
>> Bloat-devel mailing list
>> [email protected]
>> https://lists.bufferbloat.net/listinfo/bloat-devel
>
> --
> Dave Taht
> http://nex-6.taht.net
>
These "debloat guys" are fast :-).
I was just preparing my build-system (which I normally use to
debianize linux-next kernels).
Any other recommendation for kernel-config options?
For example:
linux-next has already CONFIG_NET_SCH_CHOKE (but I have unset it).
Which commits are in debloat-testing GIT but not in linux-next tree?
Are you planning debloat feature for 2.6.39?
- Sedat -
Sedat Dilek <[email protected]> writes:
> On Sun, Feb 27, 2011 at 4:31 PM, Dave Täht <[email protected]> wrote:
>>
>> Sedat Dilek <[email protected]> writes:
>>
>>> On Fri, Feb 25, 2011 at 11:22 PM, John W. Linville
>>> <[email protected]> wrote:
>>>> Announcement
>>>>
>>>> The bufferbloat project [1] is pleased to announce the availability
>>>> of the debloat-testing Linux kernel git tree:
>>>>
>>>> git://git.infradead.org/debloat-testing.git
>>
>> ----snip----
>> Excellent. At moment I would recommend building "low latency preempt
>> desktop" kernels with a high HZ value (400 or 1000), enabling highres
>> timers, and compiling in SFB as a module. (I'd like the default for SFB
>> to be "m" rather than "n", too)
>>
> These "debloat guys" are fast :-). I was just preparing my
> build-system (which I normally use to debianize linux-next kernels).
> Any other recommendation for kernel-config options? For example:
> linux-next has already CONFIG_NET_SCH_CHOKE (but I have unset it).
Enable CHOKe.
The HZ value change is due to my worry that we've smashed latency so
much in the driver/mac layer that it's interacting with the higher
layers somewhat badly... So we need to add more hooks to the servo loops
involved in order to have a normal HZ.
> Which commits are in debloat-testing GIT but not in linux-next tree?
The current list was in the release announcement. More on the way
(mostly embedded drivers at this point) git pull early and often!
> Are you planning debloat feature for 2.6.39?
Depends on how many testers we get and what the results are.
I feel the eBDP stuff will not be ready during this release cycle. SFB
and CHOKe are in net-next, so, probably. Various driver patches -
particularly those that increase the available dynamic range via
ethtool, (e.g lowering the bottommost TX queue limit to, like, 4,
especially for home gateways) may make it out if people look harder into
the issue.
>
> - Sedat -
--
Dave Taht
http://nex-6.taht.net
Sedat Dilek <[email protected]> writes:
> On Fri, Feb 25, 2011 at 11:22 PM, John W. Linville
> <[email protected]> wrote:
>> Announcement
>>
>> The bufferbloat project [1] is pleased to announce the availability
>> of the debloat-testing Linux kernel git tree:
>>
>> git://git.infradead.org/debloat-testing.git
----snip----
> Hi,
>
> it should be "localversion-debloat" in the commit-subject in [1] (not
> "localversion-wireless") :-). "-db" as suffix is IMHO not very
> meaningful... Why not add simply a suffix called "-debloat"? (Anyway,
> I will revert this patch because I don't want to have any suffix added
> automatically.)
>
> I have several other questions, but I start compiling first and test
> this debloat kernel.
Excellent. At moment I would recommend building "low latency preempt
desktop" kernels with a high HZ value (400 or 1000), enabling highres
timers, and compiling in SFB as a module. (I'd like the default for SFB
to be "m" rather than "n", too)
>
> Regards,
> - Sedat -
>
> [1] "Add localversion-wireless to identify builds from this tree."
> http://git.infradead.org/debloat-testing.git/commit/3f9bdb4f44b076feda72d353d8ad717831416f36
> _______________________________________________
> Bloat-devel mailing list
> [email protected]
> https://lists.bufferbloat.net/listinfo/bloat-devel
--
Dave Taht
http://nex-6.taht.net
Hi!
> > Bandwidth varies greatly with time. On cellphone, you can go from
> > 5KB/sec GPRS to 300KB/sec HSDPA and back...
>
> How quickly/often and does that trigger an event in the host that would
> allow adjustment of the queue?
I'd expect it to get fast when you leave the building, and back to
slow when you re-enter. It all depends how good 3G coverage is, but
seeing it change more than once a minute is not that uncommon.
Event... not sure. Linux can ask for current signal strength, and I
assume connection type works in similar way...
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
On 03/03/2011 12:16 PM, Rick Jones wrote:
>> For wireless routers and cable home gateways especially, this research
>> shows that the total un-managed buffers in your system should be less
>> than 32.
>
> Would it be a good thing to start describing these queues not so much in
> terms of packets but in terms of delay (or bandwidth X delay)?
>
The unit of bandwidth is something like Mbps, that of delay can be
second, so bandwidth X delay --> Mb, which is the unit of packet size.
So both packets and delay should have the same effect for sizing buffers.
Tianji
> Constants tend to live longer than they should, no matter how noble in
> birth.
>
> rick jones
>
> _______________________________________________
> Bloat-devel mailing list
> [email protected]
> https://lists.bufferbloat.net/listinfo/bloat-devel
>
On Thu 2011-03-03 13:45:24, Tianji Li wrote:
>
>
> On 03/03/2011 12:16 PM, Rick Jones wrote:
> >>For wireless routers and cable home gateways especially, this research
> >>shows that the total un-managed buffers in your system should be less
> >>than 32.
> >
> >Would it be a good thing to start describing these queues not so much in
> >terms of packets but in terms of delay (or bandwidth X delay)?
> >
>
> The unit of bandwidth is something like Mbps, that of delay can be
> second, so bandwidth X delay --> Mb, which is the unit of packet
> size. So both packets and delay should have the same effect for
> sizing buffers.
Bandwidth varies greatly with time. On cellphone, you can go from
5KB/sec GPRS to 300KB/sec HSDPA and back...
Pavel
--
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
On Tue, 2011-03-08 at 07:58 +0100, Pavel Machek wrote:
> Bandwidth varies greatly with time. On cellphone, you can go from
> 5KB/sec GPRS to 300KB/sec HSDPA and back...
How quickly/often and does that trigger an event in the host that would
allow adjustment of the queue?
rick jones
On Thu, Mar 03, 2011 at 10:16:08AM -0800, Rick Jones wrote:
> > For wireless routers and cable home gateways especially, this research
> > shows that the total un-managed buffers in your system should be less
> > than 32.
>
> Would it be a good thing to start describing these queues not so much in
> terms of packets but in terms of delay (or bandwidth X delay)?
>
> Constants tend to live longer than they should, no matter how noble in
> birth.
Yes, it probably would -- that would be more meaningful between
different technologies and/or for technologies that might have variable
latencies due to shared media or whatnot.
Now, the question becomes what patches to propose to make that
happen... :-)
John
--
John W. Linville Someday the world will need a hero, and you
[email protected] might be all we have. Be ready.
On Thu, 2011-03-03 at 13:45 -0600, Tianji Li wrote:
>
> On 03/03/2011 12:16 PM, Rick Jones wrote:
> >> For wireless routers and cable home gateways especially, this research
> >> shows that the total un-managed buffers in your system should be less
> >> than 32.
> >
> > Would it be a good thing to start describing these queues not so much in
> > terms of packets but in terms of delay (or bandwidth X delay)?
> >
>
> The unit of bandwidth is something like Mbps, that of delay can be
> second, so bandwidth X delay --> Mb, which is the unit of packet size.
> So both packets and delay should have the same effect for sizing buffers.
Yet, not all packets are not created equal - in size. So, 42, 150 byte
packets queued is not the same as 42, 1500 byte packets. However, 10,
1500 byte packets is the same as 100, 150 byte packets.
Had you said something like MB and delay should have the same effect I
would have agreed with you :)
rick jones
> For wireless routers and cable home gateways especially, this research
> shows that the total un-managed buffers in your system should be less
> than 32.
Would it be a good thing to start describing these queues not so much in
terms of packets but in terms of delay (or bandwidth X delay)?
Constants tend to live longer than they should, no matter how noble in
birth.
rick jones