Kan Yan <[email protected]> writes:
>> In theory, this ought to produce a histogram of sojourn times (in
>> microseconds):
>> bpftrace -e 'kretprobe:codel_skb_time_func { @sojourn = lhist((nsecs -
> (retval << 10))/1000, 0, 100000, 1000); }'
>
> Thanks for the tips!
>
>> Can't get the CoDel drop mechanism to trigger on my system at all,
>> though (a laptop running on iwl). I guess because there's queue
>> backpressure to userspace first?
>
> What's the tcp_congestion_control in your system? Maybe it is BBR that
> prevents bufferbloat.
It's not BBR, just plain old CUBIC. I've seen the issue before that it's
almost impossible to build a queue in the mac80211 layer when the TCP
session is originated on the local machine, though...
>> It would be interesting to see if it works for you, assuming you can get
>> bpftrace to work on your test system :)
>
> I can enable required kernel configuration easily, but cross-compile
> bpftrace for an ARM64 platform may take some time and effort.
Yeah, bpftrace can be a bit of a pain to get running; but it may be
worth the investment longer term as well. It really is quite useful! :)
Some links:
Install guide:
https://github.com/iovisor/bpftrace/blob/master/INSTALL.md
Tutorial by one-liners:
https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md
Reference guide:
https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md#5-tracepoint-static-tracing-kernel-level
-Toke
> Yeah, bpftrace can be a bit of a pain to get running; but it may be
> worth the investment longer term as well. It really is quite useful! :)
My attempt to build bpftrace didn't work out, so I just got the
sojourn time using old fashioned trace event.
The raw trace, parsed data in csv format and plots can be found here:
https://drive.google.com/open?id=1Mg_wHu7elYAdkXz4u--42qGCVE1nrILV
All tests are done with 2 TCP download sessions that oversubscribed
the link bandwidth.
With AQL on, the mean sojourn time about ~20000us, matches the default
codel "target".
With AQL off, the mean sojourn time is less than 4us even the latency
is off the charts, just as we expected that fd_codel with mac80211
alone is not effective for drivers with deep firmware/hardware queues.
> Any chance you could turn ecn on and off and give it a go
> again in your next test run?
ECN on shows very similar results as with ECN off. "aqm" stats from
debugfs shows it is doing ECN marking instead of dropping packets as
expected. Flent test data also is in the same link.
On Fri, Nov 22, 2019 at 2:45 AM Toke Høiland-Jørgensen <[email protected]> wrote:
>
> Kan Yan <[email protected]> writes:
>
> >> In theory, this ought to produce a histogram of sojourn times (in
> >> microseconds):
> >> bpftrace -e 'kretprobe:codel_skb_time_func { @sojourn = lhist((nsecs -
> > (retval << 10))/1000, 0, 100000, 1000); }'
> >
> > Thanks for the tips!
> >
> >> Can't get the CoDel drop mechanism to trigger on my system at all,
> >> though (a laptop running on iwl). I guess because there's queue
> >> backpressure to userspace first?
> >
> > What's the tcp_congestion_control in your system? Maybe it is BBR that
> > prevents bufferbloat.
>
> It's not BBR, just plain old CUBIC. I've seen the issue before that it's
> almost impossible to build a queue in the mac80211 layer when the TCP
> session is originated on the local machine, though...
>
> >> It would be interesting to see if it works for you, assuming you can get
> >> bpftrace to work on your test system :)
> >
> > I can enable required kernel configuration easily, but cross-compile
> > bpftrace for an ARM64 platform may take some time and effort.
>
> Yeah, bpftrace can be a bit of a pain to get running; but it may be
> worth the investment longer term as well. It really is quite useful! :)
>
> Some links:
>
> Install guide:
> https://github.com/iovisor/bpftrace/blob/master/INSTALL.md
>
> Tutorial by one-liners:
> https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md
>
> Reference guide:
> https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md#5-tracepoint-static-tracing-kernel-level
>
> -Toke
>
Kan Yan <[email protected]> writes:
>> Yeah, bpftrace can be a bit of a pain to get running; but it may be
>> worth the investment longer term as well. It really is quite useful! :)
>
> My attempt to build bpftrace didn't work out, so I just got the
> sojourn time using old fashioned trace event.
> The raw trace, parsed data in csv format and plots can be found here:
> https://drive.google.com/open?id=1Mg_wHu7elYAdkXz4u--42qGCVE1nrILV
>
> All tests are done with 2 TCP download sessions that oversubscribed
> the link bandwidth.
> With AQL on, the mean sojourn time about ~20000us, matches the default
> codel "target".
Yeah, since CoDel is trying to control the latency to 20ms, it makes
sense that the value is clustered around that. That means that the
algorithm is working as they're supposed to :)
While you're running tests, could you do one with the target changed to
10ms, just to see what it looks like? Both sojourn time values and
throughput would be interesting here, of course.
> With AQL off, the mean sojourn time is less than 4us even the latency
> is off the charts, just as we expected that fd_codel with mac80211
> alone is not effective for drivers with deep firmware/hardware queues.
Yup, also kinda expected; but another good way to visualise the impact.
Nice!
-Toke
Toke Høiland-Jørgensen <[email protected]> writes:
> Kan Yan <[email protected]> writes:
>
>>> Yeah, bpftrace can be a bit of a pain to get running; but it may be
>>> worth the investment longer term as well. It really is quite useful! :)
>>
>> My attempt to build bpftrace didn't work out, so I just got the
>> sojourn time using old fashioned trace event.
>> The raw trace, parsed data in csv format and plots can be found here:
>> https://drive.google.com/open?id=1Mg_wHu7elYAdkXz4u--42qGCVE1nrILV
>>
>> All tests are done with 2 TCP download sessions that oversubscribed
>> the link bandwidth.
>> With AQL on, the mean sojourn time about ~20000us, matches the default
>> codel "target".
>
> Yeah, since CoDel is trying to control the latency to 20ms, it makes
> sense that the value is clustered around that. That means that the
> algorithm is working as they're supposed to :)
>
> While you're running tests, could you do one with the target changed to
> 10ms, just to see what it looks like? Both sojourn time values and
> throughput would be interesting here, of course.
>
>> With AQL off, the mean sojourn time is less than 4us even the latency
>> is off the charts, just as we expected that fd_codel with mac80211
>> alone is not effective for drivers with deep firmware/hardware queues
I hope to take a close look at the iwl ax200 chips soon. Unless
someone beats me to it. Can we get these sort of stats out of it?
Has anyone looked at the marvell chips of late?
>
> Yup, also kinda expected; but another good way to visualise the impact.
> Nice!
>
> -Toke
>
> _______________________________________________
> Make-wifi-fast mailing list
> [email protected]
> https://lists.bufferbloat.net/listinfo/make-wifi-fast
Dave Taht <[email protected]> writes:
> I hope to take a close look at the iwl ax200 chips soon. Unless
> someone beats me to it. Can we get these sort of stats out of it?
Here is a patch for the trace event I used to get the sojourn time:
https://drive.google.com/open?id=1Mq8BO_kcneXBqf3m5Rz5xhEMj9jNbcJv
Toke Høiland-Jørgensen <[email protected]> writes:
> While you're running tests, could you do one with the target changed to
> 10ms, just to see what it looks like? Both sojourn time values and
> throughput would be interesting here, of course.
Apologize for the late reply. Here is the test results with target set to 10ms.
The trace for the sojourn time:
https://drive.google.com/open?id=1MEy_wbKKdl22yF17hZaGzpv3uOz6orTi
Flent test for 20 ms target time vs 10 ms target time:
https://drive.google.com/open?id=1leIWe0-L0XE78eFvlmRJlNmYgbpoH8xZ
The sojourn time measured during throughput test with a relative good
5G connection has mean value around 11 ms, pretty close to the 10 ms
target.
A smaller CoDel "target" time could help reduce latency, but it may
drop packets too aggressively for stations with low data rate and
hurts throughput, as shown in one of the tests with 2.4 GHz client.
Overall, I think AQL and fq_codel works well, at least with ath10k.
The current target value of 20 ms is a reasonable default. It is
relatively conservative that helps stations with weak signal to
maintain stable throughput. Although, a debugfs entry that allows
runtime adjustment of target value could be useful.
On Tue, Nov 26, 2019 at 6:13 PM Dave Taht <[email protected]> wrote:
>
> Toke Høiland-Jørgensen <[email protected]> writes:
>
> > Kan Yan <[email protected]> writes:
> >
> >>> Yeah, bpftrace can be a bit of a pain to get running; but it may be
> >>> worth the investment longer term as well. It really is quite useful! :)
> >>
> >> My attempt to build bpftrace didn't work out, so I just got the
> >> sojourn time using old fashioned trace event.
> >> The raw trace, parsed data in csv format and plots can be found here:
> >> https://drive.google.com/open?id=1Mg_wHu7elYAdkXz4u--42qGCVE1nrILV
> >>
> >> All tests are done with 2 TCP download sessions that oversubscribed
> >> the link bandwidth.
> >> With AQL on, the mean sojourn time about ~20000us, matches the default
> >> codel "target".
> >
> > Yeah, since CoDel is trying to control the latency to 20ms, it makes
> > sense that the value is clustered around that. That means that the
> > algorithm is working as they're supposed to :)
> >
> > While you're running tests, could you do one with the target changed to
> > 10ms, just to see what it looks like? Both sojourn time values and
> > throughput would be interesting here, of course.
> >
> >> With AQL off, the mean sojourn time is less than 4us even the latency
> >> is off the charts, just as we expected that fd_codel with mac80211
> >> alone is not effective for drivers with deep firmware/hardware queues
>
> I hope to take a close look at the iwl ax200 chips soon. Unless
> someone beats me to it. Can we get these sort of stats out of it?
>
> Has anyone looked at the marvell chips of late?
>
> >
> > Yup, also kinda expected; but another good way to visualise the impact.
> > Nice!
> >
> > -Toke
> >
> > _______________________________________________
> > Make-wifi-fast mailing list
> > [email protected]
> > https://lists.bufferbloat.net/listinfo/make-wifi-fast
Kan Yan <[email protected]> writes:
> Dave Taht <[email protected]> writes:
>
>> I hope to take a close look at the iwl ax200 chips soon. Unless
>> someone beats me to it. Can we get these sort of stats out of it?
>
> Here is a patch for the trace event I used to get the sojourn time:
> https://drive.google.com/open?id=1Mq8BO_kcneXBqf3m5Rz5xhEMj9jNbcJv
>
> Toke Høiland-Jørgensen <[email protected]> writes:
>
>> While you're running tests, could you do one with the target changed to
>> 10ms, just to see what it looks like? Both sojourn time values and
>> throughput would be interesting here, of course.
>
> Apologize for the late reply. Here is the test results with target set to 10ms.
> The trace for the sojourn time:
> https://drive.google.com/open?id=1MEy_wbKKdl22yF17hZaGzpv3uOz6orTi
>
> Flent test for 20 ms target time vs 10 ms target time:
> https://drive.google.com/open?id=1leIWe0-L0XE78eFvlmRJlNmYgbpoH8xZ
>
> The sojourn time measured during throughput test with a relative good
> 5G connection has mean value around 11 ms, pretty close to the 10 ms
> target.
>
> A smaller CoDel "target" time could help reduce latency, but it may
> drop packets too aggressively for stations with low data rate and
> hurts throughput, as shown in one of the tests with 2.4 GHz client.
>
> Overall, I think AQL and fq_codel works well, at least with ath10k.
> The current target value of 20 ms is a reasonable default. It is
> relatively conservative that helps stations with weak signal to
> maintain stable throughput. Although, a debugfs entry that allows
> runtime adjustment of target value could be useful.
Why not make it configurable via nl80211? We should use debugfs only for
testing and debugging, not in production builds, and to me the use case
for this value sounds like more than just testing.
--
https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches