by Alexander Wetzel

[permalink] [raw]

Subject: Re: [PATCH 04/11] wil6210: fix PTK re-key race

>> Hi Denis,
>>
>>>> I don't know anything about the driver here but in mac80211 the idea
>>>> to avoid the race is to simply flush the queues prior deleting the
>>>> outgoing key.
>>>
>>> Maybe a silly question, but what does flushing the queue mean in this
>>> context? Is it waiting for all the packets to be sent or dropping
>>> them on the floor?
>>>
>>
>> It's stopping them to make sure nothing can be added and then sends
>> out all MPDUs in the queues.
>>
>>>>
>>>> Now wpa_supplicant is not yet bypassing qdisks, but adding the
>>>> socket parameter PACKET_QDISC_BYPASS is basically a one-liner in
>>>> wpa_supplicant and should allow a generic way for drivers to avoid
>>>> the race with a simple queue flush...
>>>
>>> Can you expand on this actually? What would the sequence of events be?
>>>
>>
>> 1) wpa_supplicant hands over eapol #4 to the kernel.
>>     When bypassing the QDISC the frame is directly added to a driver
>>     queue or directly send out. When the send call returns the driver
>>     has eapol 4 either in the queuem already send it or the send command
>>     has failed.
>>
>> 2) wpa_supplicant deletes the old key (NL80211_CMD_DEL_KEY)
>>
>> 3) The driver stops all hw queues and sends out all MPDUs queued up to
>>     that time
>>
>> 4) Driver makes sure no traffic can be send with no/wrong key or PN to
>>     STA
>>
>> 5) the driver really removes the key from the HW/installs the new and
>>     resumes normal operation
>>
>> I've just posted my hostpad patch to use PACKET_QDISC_BYPASS for eapol
>> frames; It's probably too optimistic and need more code to retry a
>> transmit to compensate for the missing QDISC buffers.
>>
>>> Also, how would this be made to work with CONTROL_PORT over NL80211 ?
>>>
>>
>> Control port is an optional feature drivers can provide.
>> wpa_supplicant should just use it when available or fall back to the
>> "traditional" path when not. Now the driver don't has to flush all
>> queues when using control port, as long as it makes sure the control
>> port frame will be send out prior to deleting the key.
>>
>> But then the driver must know that eapol frames will really be handed
>> over via control port; So I guess flushing all queues is still the
>> simpler solution. So I guess it will change next to nothing...
>
> Well, in the steps you describe (maybe its just how you describe it) it
> relies on how the driver is handling it all. I mean step 4) seems more
> the goal of the whole approach.
>
Well, if you do not take care there are plenty of pitfalls a driver can
fall into when trying to rekey, especially when having ongoing traffic.
Most drivers will need some code to make sure they can safely delete the
old key for a STA and install a new one without a full disassociation.
Just what exactly is driver/hw depended. (I've detailed knowledge for
iwlfifi and ath9k and good guess how ath10 is handling it. All other
cards: No idea...)

I've tested around ten different cards (Android, iPhone, notebooks, usb
dongles) and found only two handling it correctly. The chances that
someone has both an AP and a device handling that correctly is therefore
not very good, but then my sample is still too small to be representative.
Known broken devices are e.g. Samsung galaxy S5, Nexus 5x, HTC 10, my
Samsumg Smart TV, iwlwifi cards (both windows and linux, just different)
and for sure any device using ath9k driver with a kernel < 4.19.

The only "good" devices I found were an iPhone (forgot the model) and a
Microsoft Surface Pro (also forgot the exact model)

I was focusing on cards I'm using: iwlwifi, ath9k and ath10k. Of those
cards ath10k was ok, iwlwif was working around 50% of the rekeys and
ath9k 100% broken (pretty sure it compromised even the security by
sending out the some frames two times: With encryption and without.

The details of that are best documented here, which fixed it for many -
but probably not all - mac80211 cards:
https://patchwork.kernel.org/patch/10583675/

The core idea here is, to tell hostpd/wpa_supplicant when a driver
believes it can rekey correctly and without that confirmation refuses to
reky but disconnect/reconects fast. But that is work in progress,
delayed by first implementing the "ideal" rekey solution added in IEEE
802.11 - 2102 "Extended key ID".

Problem is of course, that all card/drivers are handling things a bit
different and what works for one may well be broken for another. ath10k
is a good sample for that: Doing basically everything in HW it worked
quite well, bypassing the pitfalls.

The generic risks are:
- PN out of sync with the key (ath9k's main fault)
Especially risky are drivers using HW crypto but generating the PNs in
SW.

- A-MPDU sessions across rekeys. (Holding back MPDUs till all belonging
to the session are received. And then bump the PN for the new key to
the value the old key used. And then dropping all MPDUs for the new
key as "replay")

- Not stopping/blocking Tx depending on the outgoing key

- repeating lost frames originally send with the old with the new key

> Basically, we now have two bypass methods dealing with the same/similar
> issue:
>
> 1) bypass the QDISC.
> 2) bypass network stack entirely with CONTROL_PORT.
>
> How does option 1) work for drivers that skip the QDISC for all traffic

Which drivers skip QDISC, and how? I'm not aware of a way "normal"
network traffic can do that.
A "normal" linux wlan driver will register as a network card and short
of setting PACKET_QDISC_BYPASS on the socket or providing a non-standard
API all network traffic will pass trough QDISC. (But that's mostly new
area for me, just stitched the path together end2end some days ago.)

Now assuming you start a load generator using PACKET_QDISC_BYPASS and
try to rekey the connection: the NIC driver still will have the eapol#4
in one of it's queue. So stopping to add new skb's for the queues and
send out everything which is in the driver queues would send out tons of
other MPDUs, but one of them would be the eapol #4 one.

The only "trick" here is, that the the sendto() call from wpa_supplicant
up to the *driver* queues is atomic. With PACKET_QDISC_BYPASS set it -
at least for my understanding after investigating the issue some hours.

Once the sendto call returns and code execution in wpa_supplicant
continues - heading for the key deletion - the eapol #4 MPDU is
accessible for the driver and can be send out.

> and rely on mac80211 to schedule the packets? Guess mac80211 can control
> that, right?
>

Not sure i understand that part.. mac80211 is like the top half of a
wlan driver: It handles some parts but the full driver consists of
(mac80211 + low level driver).

Alexander

2019-09-17 18:14:09

by Denis Kenzior

[permalink] [raw]

Subject: Re: [PATCH 04/11] wil6210: fix PTK re-key race

Hi Alexander,

> And that the intend of it is to replace the "old" path.
Correct.

>
> So the best way forward here would be to
>
> 1) implement the patch here to work around the problem without
> control_port or the theoretical QDISC bypass
> 2) start implementing control port for the future.
>
> correct?
>

I don't know what the right answer is, but it seems strange to me that
we developed a 'better way', upstreamed it several years ago, but are
still trying to kludge around adding special flags to what is now
considered a legacy approach. Also disconcerting that not a single
fullmac driver has added support for this 'better way' yet.

>
>> CONTROL_PORT was added specifically to take care of the re-keying
>> races and can be extended with additional attributes over time, as
>> needed (perhaps for extended key id, etc). Also note that in our
>> testing CONTROL_PORT is _way_ faster than PAE socket...
>>
>
> Extended Key ID is pretty robust when rekeying and the driver/card only
> has to take care to not mix KeyIDs within one A-MPDU. It's no problem
> encrypting eapol#4 with the new key. You can even encrypt it at the
> initial connect and it will work. Basically all races the "classical"
> rekey has to work around go away.
>
> For "normal" rekeys it's working pretty well with ath9k and iwlwifi even
> without control_port and just learned some weeks ago that QDISC could
> still cause issues...

Okay, if control port doesn't need to handle extended keys then even better.

By the way, thanks for your earlier explanation (upthread).

Regards,
-Denis