2012-02-04 22:42:34

by Andrew Lutomirski

[permalink] [raw]
Subject: 6300agn: queue stuck and driver doesn't recover

I've recently started to notice wireless failures -- after being
connected for a few minutes, the connection dies. Other devices on
the same network continue to work. 'iw dev wlan0 disconnect' will fix
it.

I'm not at all sure, but I think this is a 3.2 regression. My kernel
is 3.2.2-1.fc16.x86_64.

My device reports as:

[ 9.971111] iwlwifi 0000:03:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
[ 9.971121] iwlwifi 0000:03:00.0: setting latency timer to 64
[ 9.971155] iwlwifi 0000:03:00.0: pci_resource_len = 0x00002000
[ 9.971156] iwlwifi 0000:03:00.0: pci_resource_base = ffffc900050dc000
[ 9.971158] iwlwifi 0000:03:00.0: HW Revision ID = 0x35
[ 9.971256] iwlwifi 0000:03:00.0: irq 50 for MSI/MSI-X
[ 9.971307] iwlwifi 0000:03:00.0: CONFIG_IWLWIFI_DEBUG disabled
[ 9.971309] iwlwifi 0000:03:00.0: CONFIG_IWLWIFI_DEBUGFS disabled
[ 9.971310] iwlwifi 0000:03:00.0: CONFIG_IWLWIFI_DEVICE_TRACING disabled
[ 9.971312] iwlwifi 0000:03:00.0: CONFIG_IWLWIFI_DEVICE_TESTMODE disabled
[ 9.971313] iwlwifi 0000:03:00.0: CONFIG_IWLWIFI_P2P disabled
[ 9.971319] iwlwifi 0000:03:00.0: Detected Intel(R) Centrino(R)
Ultimate-N 6300 AGN, REV=0x74
[ 9.971373] iwlwifi 0000:03:00.0: L1 Disabled; Enabling L0S
[ 9.981946] iwlwifi 0000:03:00.0: device EEPROM VER=0x436, CALIB=0x6
[ 9.981949] iwlwifi 0000:03:00.0: Device SKU: 0x1F0
[ 9.981951] iwlwifi 0000:03:00.0: Valid Tx ant: 0x7, Valid Rx ant: 0x7
[ 9.981982] iwlwifi 0000:03:00.0: Tunable channels: 13 802.11bg, 24
802.11a channels
[ 10.119812] iwlwifi 0000:03:00.0: loaded firmware version 9.221.4.1
build 25532

and the failure and subsequent forced reconnect looks like:

[ 659.272849] cfg80211: Calling CRDA for country: US
[ 659.280937] cfg80211: Regulatory domain changed to country: US
[ 659.280945] cfg80211: (start_freq - end_freq @ bandwidth),
(max_antenna_gain, max_eirp)
[ 659.280952] cfg80211: (2402000 KHz - 2472000 KHz @ 40000 KHz),
(300 mBi, 2700 mBm)
[ 659.280957] cfg80211: (5170000 KHz - 5250000 KHz @ 40000 KHz),
(300 mBi, 1700 mBm)
[ 659.280962] cfg80211: (5250000 KHz - 5330000 KHz @ 40000 KHz),
(300 mBi, 2000 mBm)
[ 659.280968] cfg80211: (5490000 KHz - 5600000 KHz @ 40000 KHz),
(300 mBi, 2000 mBm)
[ 659.280973] cfg80211: (5650000 KHz - 5710000 KHz @ 40000 KHz),
(300 mBi, 2000 mBm)
[ 659.280977] cfg80211: (5735000 KHz - 5835000 KHz @ 40000 KHz),
(300 mBi, 3000 mBm)
[ 659.372989] wlan0: moving STA xx:xx:xx:xx:xx:xx to state 3
[ 682.786577] iwlwifi 0000:03:00.0: Tx aggregation enabled on ra =
xx:xx:xx:xx:xx:xx tid = 0
[ 686.067870] iwlwifi 0000:03:00.0: Queue 12 stuck for 2000 ms.
[ 686.067874] iwlwifi 0000:03:00.0: Current SW read_ptr 6 write_ptr 99
[ 686.067907] iwlwifi 0000:03:00.0: Current HW read_ptr 6 write_ptr 99
[ 686.067918] iwlwifi 0000:03:00.0: On demand firmware reload
[ 686.068318] ieee80211 phy0: Hardware restart was requested
[ 686.068385] iwlwifi 0000:03:00.0: L1 Disabled; Enabling L0S
[ 686.068521] iwlwifi 0000:03:00.0: Radio type=0x0-0x3-0x1

Most of the time, though, the connection dies with nothing printed to dmesg.


2012-02-05 18:27:01

by Alexander Schnaidt

[permalink] [raw]
Subject: Re: [Ilw] Re: 6300agn: queue stuck and driver doesn't recover

On Sun, Feb 5, 2012 at 4:31 PM, Guy, Wey-Yi <[email protected]> wrote:
> Hi,
>
> On Sun, 2012-02-05 at 16:45 +0100, Alexander Schnaidt wrote:
>> On Sat, Feb 4, 2012 at 11:42 PM, Andrew Lutomirski <[email protected]> wrote:
>> >
>> > I've recently started to notice wireless failures -- after being
>> > connected for a few minutes, the connection dies.  Other devices on
>> > the same network continue to work.  'iw dev wlan0 disconnect' will fix
>> > it.
>> >
>> > I'm not at all sure, but I think this is a 3.2 regression.  My kernel
>> >  is 3.2.2-1.fc16.x86_64.
>> Hi!
>>
>> I am experiencing a similar regression on every kernel >=3.2.
>> Connected to a wpa protected AP, every application
>> loses it's connection after a period of time. If I let it be, it
>> eventually reconnects
>> and continues for a while until the cycle repeats itself.
>>
>> 03:00.0 Network controller: Intel Corporation Ultimate N WiFi Link 5300
>>         Subsystem: Intel Corporation Device 1011
>>         Flags: bus master, fast devsel, latency 0, IRQ 48
>>         Memory at f2500000 (64-bit, non-prefetchable) [size=8K]
>>         Capabilities: <access denied>
>>         Kernel driver in use: iwlwifi
>>
>> Networkmanager, netcfg or wicd do not report any problems. The logs
>> are clean and only ever mention the reconnection-event itself.
>>
>> Now, running wpa_supplicant interactively spits out some info:
>>
>> [alex@lx200s ~]$ sudo wpa_supplicant -D wext -i wlan0 -c wpstest.conf
>> Password:
>> Trying to associate with 00:1f:3f:13:47:1d (SSID='MyFancyAP' freq=2417 MHz)
>> Associated with 00:1f:3f:13:47:1d
>> WPA: Key negotiation completed with 00:1f:3f:13:47:1d [PTK=TKIP GTK=TKIP]
>> CTRL-EVENT-CONNECTED - Connection to 00:1f:3f:13:47:1d
>> completed (auth) [id=0 id_str=]
>> WPA: Group rekeying completed with 00:1f:3f:13:47:1d [GTK=TKIP]
>> WPA: Group rekeying completed with 00:1f:3f:13:47:1d [GTK=TKIP]
>> CTRL-EVENT-DISCONNECTED bssid=00:1f:3f:13:47:1d reason=0
>> Trying to associate with 00:1f:3f:13:47:1d (SSID='MyFancyAP' freq=2417 MHz)
>> Associated with 00:1f:3f:13:47:1d
>> WPA: Key negotiation completed with 00:1f:3f:13:47:1d [PTK=TKIP GTK=TKIP]
>> CTRL-EVENT-CONNECTED - Connection to 00:1f:3f:13:47:1d
>> completed (reauth) [id=0 id_str=]
>> ^CCTRL-EVENT-TERMINATING - signal 2 received
>>
>> The connection *always* stalls at the second group rekeying event.
>> When the third group rekeying happens wpa_supplicant(?) re-associates the
>> connection and the cycle repeats.
>>
>> Here's the the dmesg output during the time frame:
>>
>> [ 126.172145] iwlwifi 0000:03:00.0: L1 Disabled; Enabling L0S
>> [ 126.172530] iwlwifi 0000:03:00.0: Radio type=0x0-0x2-0x0
>> [ 126.322917] iwlwifi 0000:03:00.0: L1 Disabled; Enabling L0S
>> [ 126.323315] iwlwifi 0000:03:00.0: Radio type=0x0-0x2-0x0
>> [ 129.644682] wlan0: authenticate with 00:1f:3f:13:47:1d (try 1)
>> [ 129.647687] wlan0: authenticated
>> [ 129.649798] wlan0: associate with 00:1f:3f:13:47:1d (try 1)
>> [ 129.653886] wlan0: RX AssocResp from 00:1f:3f:13:47:1d
>> (capab=0x31 status=0 aid=2)
>> [ 129.653895] wlan0: associated
>> [ 1506.536175] wlan0: deauthenticated from 00:1f:3f:13:47:1d (Reason: 2)
>> [ 1506.600035] cfg80211: Calling CRDA to update world regulatory domain
>> [ 1509.857439] wlan0: authenticate with 00:1f:3f:13:47:1d (try 1)
>> [ 1509.860511] wlan0: authenticated
>> [ 1509.862438] wlan0: associate with 00:1f:3f:13:47:1d (try 1)
>> [ 1509.866443] wlan0: RX AssocResp from 00:1f:3f:13:47:1d
>> (capab=0x31 status=0 aid=2)
>> [ 1509.866451] wlan0: associated
>>
>> At 1506.5 the re-association happens.
>>
>> I can't influence the interval of the wpa rekeying of my wlan-router, so I'm
>> not sure if this is related.
>> The amount of network traffic doesn't seem to influence this behavior, either.
>>
>> I tried to bisect it but ended up at:
>>
>> commit 3c607d27c818cf4a5d28f2c73b18a88f8fbdfa33
>> Author: Don Fry <[email protected]>
>> Date:   Fri Sep 30 11:40:20 2011 -0700
>>
>>     iwlagn: rename iwlagn module iwlwifi and alias to iwlagn.
>>
>>     Rename the iwlagn module as iwlwifi in preparation for future
>>     changes.  Add an alias to iwlagn for backward compatibility.
>>
>>     Signed-off-by: Don Fry <[email protected]>
>>     Signed-off-by: Wey-Yi Guy <[email protected]>
>>     Signed-off-by: John W. Linville <[email protected]>
>>
>> That doesn't make a lot of sense to me but I wanted to mention it
>>
>> I'll gladly provide more info.
>
> I agree that does not make much sense, but we will take a look into it.
> btw, could you try the attach patch and see if it help?
>
> thanks
> Wey
>>
>> _______________________________________________
>> ilw mailing list
>> [email protected]
>> http://linux.intel.com/mailman/listinfo/ilw
>

Applied the patch against 3.2-rc2.
Observing the same behavior; dmesg gained some extended output.
Here is a complete cycle from loading the module to re-association.

[ 368.927325] iwlwifi 0000:03:00.0: L1 Disabled; Enabling L0S
[ 368.927707] iwlwifi 0000:03:00.0: Radio type=0x0-0x2-0x0
[ 369.074907] iwlwifi 0000:03:00.0: L1 Disabled; Enabling L0S
[ 369.075332] iwlwifi 0000:03:00.0: Radio type=0x0-0x2-0x0
[ 369.119379] ADDRCONF(NETDEV_UP): wlan0: link is not ready
[ 372.475697] wlan0: authenticate with 00:1f:3f:13:47:1d (try 1)
[ 372.478739] wlan0: authenticated
[ 372.480701] wlan0: associate with 00:1f:3f:13:47:1d (try 1)
[ 372.484821] wlan0: RX AssocResp from 00:1f:3f:13:47:1d
(capab=0x31 status=0 aid=2)
[ 372.484830] wlan0: associated
[ 372.484838] wlan0: moving STA 00:1f:3f:13:47:1d to state 1
[ 372.484843] wlan0: moving STA 00:1f:3f:13:47:1d to state 2
[ 372.484849] wlan0: moving STA 00:1f:3f:13:47:1d to state 3
[ 372.494324] ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[ 382.786111] wlan0: no IPv6 routers present
[ 1644.937825] wlan0: deauthenticated from 00:1f:3f:13:47:1d (Reason: 2)
[ 1644.948276] wlan0: moving STA 00:1f:3f:13:47:1d to state 2
[ 1644.948284] wlan0: moving STA 00:1f:3f:13:47:1d to state 1
[ 1644.948289] wlan0: moving STA 00:1f:3f:13:47:1d to state 0
[ 1644.963173] cfg80211: Calling CRDA to update world regulatory domain
[ 1648.242734] wlan0: authenticate with 00:1f:3f:13:47:1d (try 1)
[ 1648.245786] wlan0: authenticated
[ 1648.247750] wlan0: associate with 00:1f:3f:13:47:1d (try 1)
[ 1648.251843] wlan0: RX AssocResp from 00:1f:3f:13:47:1d (capab=0x31
status=0 aid=2)
[ 1648.251852] wlan0: associated
[ 1648.251859] wlan0: moving STA 00:1f:3f:13:47:1d to state 1
[ 1648.251865] wlan0: moving STA 00:1f:3f:13:47:1d to state 2
[ 1648.251871] wlan0: moving STA 00:1f:3f:13:47:1d to state 3

Thanks

alex

2012-02-05 15:45:43

by Alexander Schnaidt

[permalink] [raw]
Subject: Re: 6300agn: queue stuck and driver doesn't recover

On Sat, Feb 4, 2012 at 11:42 PM, Andrew Lutomirski <[email protected]> wrote:
>
> I've recently started to notice wireless failures -- after being
> connected for a few minutes, the connection dies.  Other devices on
> the same network continue to work.  'iw dev wlan0 disconnect' will fix
> it.
>
> I'm not at all sure, but I think this is a 3.2 regression.  My kernel
>  is 3.2.2-1.fc16.x86_64.
Hi!

I am experiencing a similar regression on every kernel >=3.2.
Connected to a wpa protected AP, every application
loses it's connection after a period of time. If I let it be, it
eventually reconnects
and continues for a while until the cycle repeats itself.

03:00.0 Network controller: Intel Corporation Ultimate N WiFi Link 5300
Subsystem: Intel Corporation Device 1011
Flags: bus master, fast devsel, latency 0, IRQ 48
Memory at f2500000 (64-bit, non-prefetchable) [size=8K]
Capabilities: <access denied>
Kernel driver in use: iwlwifi

Networkmanager, netcfg or wicd do not report any problems. The logs
are clean and only ever mention the reconnection-event itself.

Now, running wpa_supplicant interactively spits out some info:

[alex@lx200s ~]$ sudo wpa_supplicant -D wext -i wlan0 -c wpstest.conf
Password:
Trying to associate with 00:1f:3f:13:47:1d (SSID='MyFancyAP' freq=2417 MHz)
Associated with 00:1f:3f:13:47:1d
WPA: Key negotiation completed with 00:1f:3f:13:47:1d [PTK=TKIP GTK=TKIP]
CTRL-EVENT-CONNECTED - Connection to 00:1f:3f:13:47:1d
completed (auth) [id=0 id_str=]
WPA: Group rekeying completed with 00:1f:3f:13:47:1d [GTK=TKIP]
WPA: Group rekeying completed with 00:1f:3f:13:47:1d [GTK=TKIP]
CTRL-EVENT-DISCONNECTED bssid=00:1f:3f:13:47:1d reason=0
Trying to associate with 00:1f:3f:13:47:1d (SSID='MyFancyAP' freq=2417 MHz)
Associated with 00:1f:3f:13:47:1d
WPA: Key negotiation completed with 00:1f:3f:13:47:1d [PTK=TKIP GTK=TKIP]
CTRL-EVENT-CONNECTED - Connection to 00:1f:3f:13:47:1d
completed (reauth) [id=0 id_str=]
^CCTRL-EVENT-TERMINATING - signal 2 received

The connection *always* stalls at the second group rekeying event.
When the third group rekeying happens wpa_supplicant(?) re-associates the
connection and the cycle repeats.

Here's the the dmesg output during the time frame:

[ 126.172145] iwlwifi 0000:03:00.0: L1 Disabled; Enabling L0S
[ 126.172530] iwlwifi 0000:03:00.0: Radio type=0x0-0x2-0x0
[ 126.322917] iwlwifi 0000:03:00.0: L1 Disabled; Enabling L0S
[ 126.323315] iwlwifi 0000:03:00.0: Radio type=0x0-0x2-0x0
[ 129.644682] wlan0: authenticate with 00:1f:3f:13:47:1d (try 1)
[ 129.647687] wlan0: authenticated
[ 129.649798] wlan0: associate with 00:1f:3f:13:47:1d (try 1)
[ 129.653886] wlan0: RX AssocResp from 00:1f:3f:13:47:1d
(capab=0x31 status=0 aid=2)
[ 129.653895] wlan0: associated
[ 1506.536175] wlan0: deauthenticated from 00:1f:3f:13:47:1d (Reason: 2)
[ 1506.600035] cfg80211: Calling CRDA to update world regulatory domain
[ 1509.857439] wlan0: authenticate with 00:1f:3f:13:47:1d (try 1)
[ 1509.860511] wlan0: authenticated
[ 1509.862438] wlan0: associate with 00:1f:3f:13:47:1d (try 1)
[ 1509.866443] wlan0: RX AssocResp from 00:1f:3f:13:47:1d
(capab=0x31 status=0 aid=2)
[ 1509.866451] wlan0: associated

At 1506.5 the re-association happens.

I can't influence the interval of the wpa rekeying of my wlan-router, so I'm
not sure if this is related.
The amount of network traffic doesn't seem to influence this behavior, either.

I tried to bisect it but ended up at:

commit 3c607d27c818cf4a5d28f2c73b18a88f8fbdfa33
Author: Don Fry <[email protected]>
Date: Fri Sep 30 11:40:20 2011 -0700

iwlagn: rename iwlagn module iwlwifi and alias to iwlagn.

Rename the iwlagn module as iwlwifi in preparation for future
changes. Add an alias to iwlagn for backward compatibility.

Signed-off-by: Don Fry <[email protected]>
Signed-off-by: Wey-Yi Guy <[email protected]>
Signed-off-by: John W. Linville <[email protected]>

That doesn't make a lot of sense to me but I wanted to mention it

I'll gladly provide more info.

alex

2012-02-05 16:39:07

by Wey-Yi Guy

[permalink] [raw]
Subject: Re: [Ilw] Re: 6300agn: queue stuck and driver doesn't recover

Hi,

On Sun, 2012-02-05 at 16:45 +0100, Alexander Schnaidt wrote:
> On Sat, Feb 4, 2012 at 11:42 PM, Andrew Lutomirski <[email protected]> wrote:
> >
> > I've recently started to notice wireless failures -- after being
> > connected for a few minutes, the connection dies. Other devices on
> > the same network continue to work. 'iw dev wlan0 disconnect' will fix
> > it.
> >
> > I'm not at all sure, but I think this is a 3.2 regression. My kernel
> > is 3.2.2-1.fc16.x86_64.
> Hi!
>
> I am experiencing a similar regression on every kernel >=3.2.
> Connected to a wpa protected AP, every application
> loses it's connection after a period of time. If I let it be, it
> eventually reconnects
> and continues for a while until the cycle repeats itself.
>
> 03:00.0 Network controller: Intel Corporation Ultimate N WiFi Link 5300
> Subsystem: Intel Corporation Device 1011
> Flags: bus master, fast devsel, latency 0, IRQ 48
> Memory at f2500000 (64-bit, non-prefetchable) [size=8K]
> Capabilities: <access denied>
> Kernel driver in use: iwlwifi
>
> Networkmanager, netcfg or wicd do not report any problems. The logs
> are clean and only ever mention the reconnection-event itself.
>
> Now, running wpa_supplicant interactively spits out some info:
>
> [alex@lx200s ~]$ sudo wpa_supplicant -D wext -i wlan0 -c wpstest.conf
> Password:
> Trying to associate with 00:1f:3f:13:47:1d (SSID='MyFancyAP' freq=2417 MHz)
> Associated with 00:1f:3f:13:47:1d
> WPA: Key negotiation completed with 00:1f:3f:13:47:1d [PTK=TKIP GTK=TKIP]
> CTRL-EVENT-CONNECTED - Connection to 00:1f:3f:13:47:1d
> completed (auth) [id=0 id_str=]
> WPA: Group rekeying completed with 00:1f:3f:13:47:1d [GTK=TKIP]
> WPA: Group rekeying completed with 00:1f:3f:13:47:1d [GTK=TKIP]
> CTRL-EVENT-DISCONNECTED bssid=00:1f:3f:13:47:1d reason=0
> Trying to associate with 00:1f:3f:13:47:1d (SSID='MyFancyAP' freq=2417 MHz)
> Associated with 00:1f:3f:13:47:1d
> WPA: Key negotiation completed with 00:1f:3f:13:47:1d [PTK=TKIP GTK=TKIP]
> CTRL-EVENT-CONNECTED - Connection to 00:1f:3f:13:47:1d
> completed (reauth) [id=0 id_str=]
> ^CCTRL-EVENT-TERMINATING - signal 2 received
>
> The connection *always* stalls at the second group rekeying event.
> When the third group rekeying happens wpa_supplicant(?) re-associates the
> connection and the cycle repeats.
>
> Here's the the dmesg output during the time frame:
>
> [ 126.172145] iwlwifi 0000:03:00.0: L1 Disabled; Enabling L0S
> [ 126.172530] iwlwifi 0000:03:00.0: Radio type=0x0-0x2-0x0
> [ 126.322917] iwlwifi 0000:03:00.0: L1 Disabled; Enabling L0S
> [ 126.323315] iwlwifi 0000:03:00.0: Radio type=0x0-0x2-0x0
> [ 129.644682] wlan0: authenticate with 00:1f:3f:13:47:1d (try 1)
> [ 129.647687] wlan0: authenticated
> [ 129.649798] wlan0: associate with 00:1f:3f:13:47:1d (try 1)
> [ 129.653886] wlan0: RX AssocResp from 00:1f:3f:13:47:1d
> (capab=0x31 status=0 aid=2)
> [ 129.653895] wlan0: associated
> [ 1506.536175] wlan0: deauthenticated from 00:1f:3f:13:47:1d (Reason: 2)
> [ 1506.600035] cfg80211: Calling CRDA to update world regulatory domain
> [ 1509.857439] wlan0: authenticate with 00:1f:3f:13:47:1d (try 1)
> [ 1509.860511] wlan0: authenticated
> [ 1509.862438] wlan0: associate with 00:1f:3f:13:47:1d (try 1)
> [ 1509.866443] wlan0: RX AssocResp from 00:1f:3f:13:47:1d
> (capab=0x31 status=0 aid=2)
> [ 1509.866451] wlan0: associated
>
> At 1506.5 the re-association happens.
>
> I can't influence the interval of the wpa rekeying of my wlan-router, so I'm
> not sure if this is related.
> The amount of network traffic doesn't seem to influence this behavior, either.
>
> I tried to bisect it but ended up at:
>
> commit 3c607d27c818cf4a5d28f2c73b18a88f8fbdfa33
> Author: Don Fry <[email protected]>
> Date: Fri Sep 30 11:40:20 2011 -0700
>
> iwlagn: rename iwlagn module iwlwifi and alias to iwlagn.
>
> Rename the iwlagn module as iwlwifi in preparation for future
> changes. Add an alias to iwlagn for backward compatibility.
>
> Signed-off-by: Don Fry <[email protected]>
> Signed-off-by: Wey-Yi Guy <[email protected]>
> Signed-off-by: John W. Linville <[email protected]>
>
> That doesn't make a lot of sense to me but I wanted to mention it
>
> I'll gladly provide more info.

I agree that does not make much sense, but we will take a look into it.
btw, could you try the attach patch and see if it help?

thanks
Wey
>
> _______________________________________________
> ilw mailing list
> [email protected]
> http://linux.intel.com/mailman/listinfo/ilw


Attachments:
tid.patch (1.95 kB)

2012-02-06 21:27:44

by Alexander Schnaidt

[permalink] [raw]
Subject: Re: [Ilw] Re: 6300agn: queue stuck and driver doesn't recover

On Mon, Feb 6, 2012 at 8:39 AM, Berg, Johannes <[email protected]> wrote:
>> > I tried to bisect it but ended up at:
>> >
>> > commit 3c607d27c818cf4a5d28f2c73b18a88f8fbdfa33
>> > Author: Don Fry <[email protected]>
>> > Date:   Fri Sep 30 11:40:20 2011 -0700
>> >
>> >     iwlagn: rename iwlagn module iwlwifi and alias to iwlagn.
>
>> I agree that does not make much sense, but we will take a look into it.
>
> I think you probably had something like "options iwlagn 11n_disable=1" or so in /etc/modprobe.d/ somewhere. This caused the problem to happen only with the driver called "iwlwifi", since "iwlagn" would get 11n disabled.
>
> johannes

Thanks for the hint!

You're almost spot on!, somewhere along the bisect I put swcrypto=1 into
modprobe.conf... so much for scientific reproducibility. Sorry for that.

I redid the whole bisect with no options in modprobe.conf which defaults
to swcrypto=0 on >=3.0 if I'm not mistaken.

Heres the bisect log:
git bisect start
# bad: [c3b92c8787367a8bb53d57d9789b558f1295cc96] Linux 3.1
git bisect bad c3b92c8787367a8bb53d57d9789b558f1295cc96
# good: [02f8c6aee8df3cdc935e9bdd4f2d020306035dbe] Linux 3.0
git bisect good 02f8c6aee8df3cdc935e9bdd4f2d020306035dbe
# bad: [138051659902da7e6a09d379fee5dade2a80fcfd]
Merge branch 'staging-next' of
git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging-2.6
git bisect bad 138051659902da7e6a09d379fee5dade2a80fcfd
# good: [9d1c02135516866cbbb2f80e20cfb65c63a3ce40]
Merge branch 'for-linus' of
git://git.kernel.org/pub/scm/linux/kernel/git/ericvh/v9fs
git bisect good 9d1c02135516866cbbb2f80e20cfb65c63a3ce40
# bad: [ae4c42e4e4d76d003f8ca551fe1aef93ff9a4b21]
Merge branch 'next/cleanup' of
git://git.kernel.org/pub/scm/linux/kernel/git/arm/linux-arm-soc
git bisect bad ae4c42e4e4d76d003f8ca551fe1aef93ff9a4b21
# good: [ff0c4ad2c3a75ccfe6adca916e50804eb45bb2d9]
Merge branch 'for-upstream' of git://openrisc.net/jonas/linux
git bisect good ff0c4ad2c3a75ccfe6adca916e50804eb45bb2d9
# good: [933b44732caad0c3b65224453c54846c75d97936]
gma500: udlay(20000) is too large
git bisect good 933b44732caad0c3b65224453c54846c75d97936
# bad: [096a705bbc080a4041636d07514560da8d78acbe]
Merge branch 'for-3.1/core' of git://git.kernel.dk/linux-block
git bisect bad 096a705bbc080a4041636d07514560da8d78acbe
# bad: [e1b1c0875daa8ef396593270b5d3ec0b8483c1ea]
iwlagn: rename iwlagn_set_dynamic_key
git bisect bad e1b1c0875daa8ef396593270b5d3ec0b8483c1ea
# good: [b473bc176702cb22529632b5c4315bda27e0d979]
b43: HT-PHY: fix typo in 0x2059 radio init
git bisect good b473bc176702cb22529632b5c4315bda27e0d979
# good: [3db4f989384c90f5f6be14e88c19732bfb0ac331]
libertas: mesh: misc cleanup
git bisect good 3db4f989384c90f5f6be14e88c19732bfb0ac331
# good: [41c50542669cd7aec45ad708f5120ff8fdaa1194]
iwlagn: transport layer receives struct iwl_trans*
git bisect good 41c50542669cd7aec45ad708f5120ff8fdaa1194
# good: [898ed67be047d0762cc7592f67bf1313dff53ca9]
iwlagn: remove un-necessary "_agn"
git bisect good 898ed67be047d0762cc7592f67bf1313dff53ca9
# bad: [c8ac61cf6e53fefb3b439fc58390fb65d2730e63]
iwlagn: implement WoWLAN
git bisect bad c8ac61cf6e53fefb3b439fc58390fb65d2730e63
# bad: [5a3d9882b84edf5fa8e8ca33a5d6df25e2e727a5]
iwlagn: rewrite HW crypto
git bisect bad 5a3d9882b84edf5fa8e8ca33a5d6df25e2e727a5
# good: [0bfe9895d402faf03d5ccd30cc0e6bbad09e5bbc]
iwlagn: remove forgotten debugfs function
git bisect good 0bfe9895d402faf03d5ccd30cc0e6bbad09e5bbc

If my interpretation is right the culprit would be:

commit 5a3d9882b84edf5fa8e8ca33a5d6df25e2e727a5
Author: Johannes Berg <[email protected]>
Date: Fri Jul 15 13:03:12 2011 -0700

iwlagn: rewrite HW crypto

As I just discovered while doing WoWLAN, HW crypto
is done wrong for GTKs: they should be programmed
for the AP station ID (in the managed mode case)
and the HW can actually deal with multiple group
keys per station as well (which is useful in IBSS
RSN but that I've chosen not to use this).

To fix all this, modify the way keys are sent to
the device and key offsets are allocated. After
these changes, key offsets are stored into the
hw_key_idx which we can then track for the key
lifetime, not relying on our sta_cmd array. WEP
default keys get special treatment, of course.

Additionally, since I had the API for it, we can
now pre-fill TKIP phase 1 keys for RX now that we
can obtain the P1K from mac80211, a capability I
had added for WoWLAN initially.

Finally, some keys simply don't need to be added
into the device's key cache -- a key that won't
be used for RX is only needed in the TX header,
so "pretend" to have accepted any key without
adding it into the device -- no need to use up
key space there for it.

Signed-off-by: Johannes Berg <[email protected]>
Signed-off-by: Wey-Yi Guy <[email protected]>

To me it makes sense considering that swcrypto=1 fixes
my problems all the way up till kernel 3.3.0-rc2.

ps.: 11n_disable=1 as suggested by Wey-Yi does not
change the faulty behavior.

Again I apologize for my blunder and thanks for the effort.

alex

2012-02-10 17:29:07

by Andrew Lutomirski

[permalink] [raw]
Subject: Re: [Ilw] Re: 6300agn: queue stuck and driver doesn't recover

On Mon, Feb 6, 2012 at 1:33 PM, Johannes Berg <[email protected]> wrote:
> On Mon, 2012-02-06 at 22:27 +0100, Alexander Schnaidt wrote:
>
>> > I think you probably had something like "options iwlagn
>> 11n_disable=1" or so in /etc/modprobe.d/ somewhere. This caused the
>> problem to happen only with the driver called "iwlwifi", since
>> "iwlagn" would get 11n disabled.
>
>> You're almost spot on!, somewhere along the bisect I put swcrypto=1 into
>> modprobe.conf... so much for scientific reproducibility. Sorry for that.
>>
>> I redid the whole bisect with no options in modprobe.conf which defaults
>> to swcrypto=0 on >=3.0 if I'm not mistaken.
>
> ...
>
>> If my interpretation is right the culprit would be:
>>
>> commit 5a3d9882b84edf5fa8e8ca33a5d6df25e2e727a5
>> Author: Johannes Berg <[email protected]>
>> Date: ? Fri Jul 15 13:03:12 2011 -0700
>>
>> ? ? iwlagn: rewrite HW crypto
>
>> To me it makes sense considering that swcrypto=1 fixes
>> my problems all the way up till kernel 3.3.0-rc2.
>>
>> ps.: 11n_disable=1 as suggested by Wey-Yi does not
>> change the faulty behavior.
>>
>> Again I apologize for my blunder and thanks for the effort.
>
> Ok this makes sense. A few other people also recently reported this. I'm
> currently (still) away from home so I can't reproduce this issue, and I
> didn't find anything by just looking at the code. Traces for mac80211
> and iwlwifi subsystems would be useful to look at in this case, I think,
> if you want to record the failure case see:
> http://wireless.kernel.org/en/developers/Documentation/mac80211/tracing
> (-e mac80211 -e iwlwifi). If you do record it, please don't send the
> traces to the list, bzip2 it and send it to me directly.

I just reproduced the problem (or at least the similar problem
affecting me) with swcrypto=1. I'll try to get a trace.

--Andy

2012-02-06 07:40:49

by Berg, Johannes

[permalink] [raw]
Subject: RE: [Ilw] Re: 6300agn: queue stuck and driver doesn't recover

PiA+IEkgdHJpZWQgdG8gYmlzZWN0IGl0IGJ1dCBlbmRlZCB1cCBhdDoNCj4gPg0KPiA+IGNvbW1p
dCAzYzYwN2QyN2M4MThjZjRhNWQyOGYyYzczYjE4YTg4ZjhmYmRmYTMzDQo+ID4gQXV0aG9yOiBE
b24gRnJ5IDxkb25hbGQuaC5mcnlAaW50ZWwuY29tPg0KPiA+IERhdGU6ICAgRnJpIFNlcCAzMCAx
MTo0MDoyMCAyMDExIC0wNzAwDQo+ID4NCj4gPiAgICAgaXdsYWduOiByZW5hbWUgaXdsYWduIG1v
ZHVsZSBpd2x3aWZpIGFuZCBhbGlhcyB0byBpd2xhZ24uDQoNCj4gSSBhZ3JlZSB0aGF0IGRvZXMg
bm90IG1ha2UgbXVjaCBzZW5zZSwgYnV0IHdlIHdpbGwgdGFrZSBhIGxvb2sgaW50byBpdC4NCg0K
SSB0aGluayB5b3UgcHJvYmFibHkgaGFkIHNvbWV0aGluZyBsaWtlICJvcHRpb25zIGl3bGFnbiAx
MW5fZGlzYWJsZT0xIiBvciBzbyBpbiAvZXRjL21vZHByb2JlLmQvIHNvbWV3aGVyZS4gVGhpcyBj
YXVzZWQgdGhlIHByb2JsZW0gdG8gaGFwcGVuIG9ubHkgd2l0aCB0aGUgZHJpdmVyIGNhbGxlZCAi
aXdsd2lmaSIsIHNpbmNlICJpd2xhZ24iIHdvdWxkIGdldCAxMW4gZGlzYWJsZWQuDQoNCmpvaGFu
bmVzDQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQpJbnRlbCBHbWJICkRvcm5hY2hlciBT
dHJhc3NlIDEKODU2MjIgRmVsZGtpcmNoZW4vTXVlbmNoZW4sIERldXRzY2hsYW5kIApTaXR6IGRl
ciBHZXNlbGxzY2hhZnQ6IEZlbGRraXJjaGVuIGJlaSBNdWVuY2hlbgpHZXNjaGFlZnRzZnVlaHJl
cjogRG91Z2xhcyBMdXNrLCBQZXRlciBHbGVpc3NuZXIsIEhhbm5lcyBTY2h3YWRlcmVyClJlZ2lz
dGVyZ2VyaWNodDogTXVlbmNoZW4gSFJCIDQ3NDU2IApVc3QuLUlkTnIuL1ZBVCBSZWdpc3RyYXRp
b24gTm8uOiBERTEyOTM4NTg5NQpDaXRpYmFuayBGcmFua2Z1cnQgYS5NLiAoQkxaIDUwMiAxMDkg
MDApIDYwMDExOTA1Mgo=


2012-02-10 17:57:07

by Alexander Schnaidt

[permalink] [raw]
Subject: Re: [Ilw] Re: 6300agn: queue stuck and driver doesn't recover

>> Ok this makes sense. A few other people also recently reported this. I'm
>> currently (still) away from home so I can't reproduce this issue, and I
>> didn't find anything by just looking at the code. Traces for mac80211
>> and iwlwifi subsystems would be useful to look at in this case, I think,
>> if you want to record the failure case see:
>> http://wireless.kernel.org/en/developers/Documentation/mac80211/tracing
>> (-e mac80211 -e iwlwifi). If you do record it, please don't send the
>> traces to the list, bzip2 it and send it to me directly.
>
> I just reproduced the problem (or at least the similar problem
> affecting me) with swcrypto=1.  I'll try to get a trace.
>
> --Andy

Hey!

Before you get started; Johannes Berg already came up with a patch for this.
Unfortunately the discussion drifted off the list so i reattach the patch here.
I hope that's fine with Johannes.

Applying it against 3.3.0-rc2 fixes my problem, you might want to give it a try.

alex

Here's Johannes' original message:

>From: Berg, Johannes <[email protected]>
>Date: Tue, Feb 7, 2012 at 11:50 PM
>
>Great, thanks. I attached a more complete patch, if you want to give it a try -- I'll be putting that into the tree.
>
>johannes


Attachments:
crypto-fix.patch (1.97 kB)

2012-02-06 02:02:37

by Wey-Yi Guy

[permalink] [raw]
Subject: Re: [Ilw] Re: 6300agn: queue stuck and driver doesn't recover

hi,

how about use "swcrypto=1"

Wey


On Sun, 2012-02-05 at 19:27 +0100, Alexander Schnaidt wrote:
> On Sun, Feb 5, 2012 at 4:31 PM, Guy, Wey-Yi <[email protected]> wrote:
> > Hi,
> >
> > On Sun, 2012-02-05 at 16:45 +0100, Alexander Schnaidt wrote:
> >> On Sat, Feb 4, 2012 at 11:42 PM, Andrew Lutomirski <[email protected]> wrote:
> >> >
> >> > I've recently started to notice wireless failures -- after being
> >> > connected for a few minutes, the connection dies. Other devices on
> >> > the same network continue to work. 'iw dev wlan0 disconnect' will fix
> >> > it.
> >> >
> >> > I'm not at all sure, but I think this is a 3.2 regression. My kernel
> >> > is 3.2.2-1.fc16.x86_64.
> >> Hi!
> >>
> >> I am experiencing a similar regression on every kernel >=3.2.
> >> Connected to a wpa protected AP, every application
> >> loses it's connection after a period of time. If I let it be, it
> >> eventually reconnects
> >> and continues for a while until the cycle repeats itself.
> >>
> >> 03:00.0 Network controller: Intel Corporation Ultimate N WiFi Link 5300
> >> Subsystem: Intel Corporation Device 1011
> >> Flags: bus master, fast devsel, latency 0, IRQ 48
> >> Memory at f2500000 (64-bit, non-prefetchable) [size=8K]
> >> Capabilities: <access denied>
> >> Kernel driver in use: iwlwifi
> >>
> >> Networkmanager, netcfg or wicd do not report any problems. The logs
> >> are clean and only ever mention the reconnection-event itself.
> >>
> >> Now, running wpa_supplicant interactively spits out some info:
> >>
> >> [alex@lx200s ~]$ sudo wpa_supplicant -D wext -i wlan0 -c wpstest.conf
> >> Password:
> >> Trying to associate with 00:1f:3f:13:47:1d (SSID='MyFancyAP' freq=2417 MHz)
> >> Associated with 00:1f:3f:13:47:1d
> >> WPA: Key negotiation completed with 00:1f:3f:13:47:1d [PTK=TKIP GTK=TKIP]
> >> CTRL-EVENT-CONNECTED - Connection to 00:1f:3f:13:47:1d
> >> completed (auth) [id=0 id_str=]
> >> WPA: Group rekeying completed with 00:1f:3f:13:47:1d [GTK=TKIP]
> >> WPA: Group rekeying completed with 00:1f:3f:13:47:1d [GTK=TKIP]
> >> CTRL-EVENT-DISCONNECTED bssid=00:1f:3f:13:47:1d reason=0
> >> Trying to associate with 00:1f:3f:13:47:1d (SSID='MyFancyAP' freq=2417 MHz)
> >> Associated with 00:1f:3f:13:47:1d
> >> WPA: Key negotiation completed with 00:1f:3f:13:47:1d [PTK=TKIP GTK=TKIP]
> >> CTRL-EVENT-CONNECTED - Connection to 00:1f:3f:13:47:1d
> >> completed (reauth) [id=0 id_str=]
> >> ^CCTRL-EVENT-TERMINATING - signal 2 received
> >>
> >> The connection *always* stalls at the second group rekeying event.
> >> When the third group rekeying happens wpa_supplicant(?) re-associates the
> >> connection and the cycle repeats.
> >>
> >> Here's the the dmesg output during the time frame:
> >>
> >> [ 126.172145] iwlwifi 0000:03:00.0: L1 Disabled; Enabling L0S
> >> [ 126.172530] iwlwifi 0000:03:00.0: Radio type=0x0-0x2-0x0
> >> [ 126.322917] iwlwifi 0000:03:00.0: L1 Disabled; Enabling L0S
> >> [ 126.323315] iwlwifi 0000:03:00.0: Radio type=0x0-0x2-0x0
> >> [ 129.644682] wlan0: authenticate with 00:1f:3f:13:47:1d (try 1)
> >> [ 129.647687] wlan0: authenticated
> >> [ 129.649798] wlan0: associate with 00:1f:3f:13:47:1d (try 1)
> >> [ 129.653886] wlan0: RX AssocResp from 00:1f:3f:13:47:1d
> >> (capab=0x31 status=0 aid=2)
> >> [ 129.653895] wlan0: associated
> >> [ 1506.536175] wlan0: deauthenticated from 00:1f:3f:13:47:1d (Reason: 2)
> >> [ 1506.600035] cfg80211: Calling CRDA to update world regulatory domain
> >> [ 1509.857439] wlan0: authenticate with 00:1f:3f:13:47:1d (try 1)
> >> [ 1509.860511] wlan0: authenticated
> >> [ 1509.862438] wlan0: associate with 00:1f:3f:13:47:1d (try 1)
> >> [ 1509.866443] wlan0: RX AssocResp from 00:1f:3f:13:47:1d
> >> (capab=0x31 status=0 aid=2)
> >> [ 1509.866451] wlan0: associated
> >>
> >> At 1506.5 the re-association happens.
> >>
> >> I can't influence the interval of the wpa rekeying of my wlan-router, so I'm
> >> not sure if this is related.
> >> The amount of network traffic doesn't seem to influence this behavior, either.
> >>
> >> I tried to bisect it but ended up at:
> >>
> >> commit 3c607d27c818cf4a5d28f2c73b18a88f8fbdfa33
> >> Author: Don Fry <[email protected]>
> >> Date: Fri Sep 30 11:40:20 2011 -0700
> >>
> >> iwlagn: rename iwlagn module iwlwifi and alias to iwlagn.
> >>
> >> Rename the iwlagn module as iwlwifi in preparation for future
> >> changes. Add an alias to iwlagn for backward compatibility.
> >>
> >> Signed-off-by: Don Fry <[email protected]>
> >> Signed-off-by: Wey-Yi Guy <[email protected]>
> >> Signed-off-by: John W. Linville <[email protected]>
> >>
> >> That doesn't make a lot of sense to me but I wanted to mention it
> >>
> >> I'll gladly provide more info.
> >
> > I agree that does not make much sense, but we will take a look into it.
> > btw, could you try the attach patch and see if it help?
> >
> > thanks
> > Wey
> >>
> >> _______________________________________________
> >> ilw mailing list
> >> [email protected]
> >> http://linux.intel.com/mailman/listinfo/ilw
> >
>
> Applied the patch against 3.2-rc2.
> Observing the same behavior; dmesg gained some extended output.
> Here is a complete cycle from loading the module to re-association.
>
> [ 368.927325] iwlwifi 0000:03:00.0: L1 Disabled; Enabling L0S
> [ 368.927707] iwlwifi 0000:03:00.0: Radio type=0x0-0x2-0x0
> [ 369.074907] iwlwifi 0000:03:00.0: L1 Disabled; Enabling L0S
> [ 369.075332] iwlwifi 0000:03:00.0: Radio type=0x0-0x2-0x0
> [ 369.119379] ADDRCONF(NETDEV_UP): wlan0: link is not ready
> [ 372.475697] wlan0: authenticate with 00:1f:3f:13:47:1d (try 1)
> [ 372.478739] wlan0: authenticated
> [ 372.480701] wlan0: associate with 00:1f:3f:13:47:1d (try 1)
> [ 372.484821] wlan0: RX AssocResp from 00:1f:3f:13:47:1d
> (capab=0x31 status=0 aid=2)
> [ 372.484830] wlan0: associated
> [ 372.484838] wlan0: moving STA 00:1f:3f:13:47:1d to state 1
> [ 372.484843] wlan0: moving STA 00:1f:3f:13:47:1d to state 2
> [ 372.484849] wlan0: moving STA 00:1f:3f:13:47:1d to state 3
> [ 372.494324] ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
> [ 382.786111] wlan0: no IPv6 routers present
> [ 1644.937825] wlan0: deauthenticated from 00:1f:3f:13:47:1d (Reason: 2)
> [ 1644.948276] wlan0: moving STA 00:1f:3f:13:47:1d to state 2
> [ 1644.948284] wlan0: moving STA 00:1f:3f:13:47:1d to state 1
> [ 1644.948289] wlan0: moving STA 00:1f:3f:13:47:1d to state 0
> [ 1644.963173] cfg80211: Calling CRDA to update world regulatory domain
> [ 1648.242734] wlan0: authenticate with 00:1f:3f:13:47:1d (try 1)
> [ 1648.245786] wlan0: authenticated
> [ 1648.247750] wlan0: associate with 00:1f:3f:13:47:1d (try 1)
> [ 1648.251843] wlan0: RX AssocResp from 00:1f:3f:13:47:1d (capab=0x31
> status=0 aid=2)
> [ 1648.251852] wlan0: associated
> [ 1648.251859] wlan0: moving STA 00:1f:3f:13:47:1d to state 1
> [ 1648.251865] wlan0: moving STA 00:1f:3f:13:47:1d to state 2
> [ 1648.251871] wlan0: moving STA 00:1f:3f:13:47:1d to state 3
>
> Thanks
>
> alex



2012-02-06 21:33:08

by Berg, Johannes

[permalink] [raw]
Subject: Re: [Ilw] Re: 6300agn: queue stuck and driver doesn't recover

T24gTW9uLCAyMDEyLTAyLTA2IGF0IDIyOjI3ICswMTAwLCBBbGV4YW5kZXIgU2NobmFpZHQgd3Jv
dGU6Cgo+ID4gSSB0aGluayB5b3UgcHJvYmFibHkgaGFkIHNvbWV0aGluZyBsaWtlICJvcHRpb25z
IGl3bGFnbgo+IDExbl9kaXNhYmxlPTEiIG9yIHNvIGluIC9ldGMvbW9kcHJvYmUuZC8gc29tZXdo
ZXJlLiBUaGlzIGNhdXNlZCB0aGUKPiBwcm9ibGVtIHRvIGhhcHBlbiBvbmx5IHdpdGggdGhlIGRy
aXZlciBjYWxsZWQgIml3bHdpZmkiLCBzaW5jZQo+ICJpd2xhZ24iIHdvdWxkIGdldCAxMW4gZGlz
YWJsZWQuCgo+IFlvdSdyZSBhbG1vc3Qgc3BvdCBvbiEsIHNvbWV3aGVyZSBhbG9uZyB0aGUgYmlz
ZWN0IEkgcHV0IHN3Y3J5cHRvPTEgaW50bwo+IG1vZHByb2JlLmNvbmYuLi4gc28gbXVjaCBmb3Ig
c2NpZW50aWZpYyByZXByb2R1Y2liaWxpdHkuIFNvcnJ5IGZvciB0aGF0Lgo+IAo+IEkgcmVkaWQg
dGhlIHdob2xlIGJpc2VjdCB3aXRoIG5vIG9wdGlvbnMgaW4gbW9kcHJvYmUuY29uZiB3aGljaCBk
ZWZhdWx0cwo+IHRvIHN3Y3J5cHRvPTAgb24gPj0zLjAgaWYgSSdtIG5vdCBtaXN0YWtlbi4KCi4u
LgoKPiBJZiBteSBpbnRlcnByZXRhdGlvbiBpcyByaWdodCB0aGUgY3VscHJpdCB3b3VsZCBiZToK
PiAKPiBjb21taXQgNWEzZDk4ODJiODRlZGY1ZmE4ZThjYTMzYTVkNmRmMjVlMmU3MjdhNQo+IEF1
dGhvcjogSm9oYW5uZXMgQmVyZyA8am9oYW5uZXMuYmVyZ0BpbnRlbC5jb20+Cj4gRGF0ZTogICBG
cmkgSnVsIDE1IDEzOjAzOjEyIDIwMTEgLTA3MDAKPiAKPiAgICAgaXdsYWduOiByZXdyaXRlIEhX
IGNyeXB0bwoKPiBUbyBtZSBpdCBtYWtlcyBzZW5zZSBjb25zaWRlcmluZyB0aGF0IHN3Y3J5cHRv
PTEgZml4ZXMKPiBteSBwcm9ibGVtcyBhbGwgdGhlIHdheSB1cCB0aWxsIGtlcm5lbCAzLjMuMC1y
YzIuCj4gCj4gcHMuOiAxMW5fZGlzYWJsZT0xIGFzIHN1Z2dlc3RlZCBieSBXZXktWWkgZG9lcyBu
b3QKPiBjaGFuZ2UgdGhlIGZhdWx0eSBiZWhhdmlvci4KPiAKPiBBZ2FpbiBJIGFwb2xvZ2l6ZSBm
b3IgbXkgYmx1bmRlciBhbmQgdGhhbmtzIGZvciB0aGUgZWZmb3J0LgoKT2sgdGhpcyBtYWtlcyBz
ZW5zZS4gQSBmZXcgb3RoZXIgcGVvcGxlIGFsc28gcmVjZW50bHkgcmVwb3J0ZWQgdGhpcy4gSSdt
CmN1cnJlbnRseSAoc3RpbGwpIGF3YXkgZnJvbSBob21lIHNvIEkgY2FuJ3QgcmVwcm9kdWNlIHRo
aXMgaXNzdWUsIGFuZCBJCmRpZG4ndCBmaW5kIGFueXRoaW5nIGJ5IGp1c3QgbG9va2luZyBhdCB0
aGUgY29kZS4gVHJhY2VzIGZvciBtYWM4MDIxMQphbmQgaXdsd2lmaSBzdWJzeXN0ZW1zIHdvdWxk
IGJlIHVzZWZ1bCB0byBsb29rIGF0IGluIHRoaXMgY2FzZSwgSSB0aGluaywKaWYgeW91IHdhbnQg
dG8gcmVjb3JkIHRoZSBmYWlsdXJlIGNhc2Ugc2VlOgpodHRwOi8vd2lyZWxlc3Mua2VybmVsLm9y
Zy9lbi9kZXZlbG9wZXJzL0RvY3VtZW50YXRpb24vbWFjODAyMTEvdHJhY2luZwooLWUgbWFjODAy
MTEgLWUgaXdsd2lmaSkuIElmIHlvdSBkbyByZWNvcmQgaXQsIHBsZWFzZSBkb24ndCBzZW5kIHRo
ZQp0cmFjZXMgdG8gdGhlIGxpc3QsIGJ6aXAyIGl0IGFuZCBzZW5kIGl0IHRvIG1lIGRpcmVjdGx5
LgoKSSBkbyBleHBlY3QgdG8gYmUgYWJsZSB0byByZXByb2R1Y2UgdGhpcyBlYXNpbHkgdGhvdWdo
LCBzbyBpZiB5b3UgZG9uJ3QKaGF2ZSB0cmFjaW5nIHNldHVwIGV0Yy4gSSdsbCBqdXN0IHdvcmsg
b24gaXQgZWFybHkgbmV4dCB3ZWVrLgoKam9oYW5uZXMKCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tCkludGVsIEdtYkgKRG9ybmFjaGVyIFN0cmFzc2UgMQo4NTYyMiBGZWxka2lyY2hlbi9N
dWVuY2hlbiwgRGV1dHNjaGxhbmQgClNpdHogZGVyIEdlc2VsbHNjaGFmdDogRmVsZGtpcmNoZW4g
YmVpIE11ZW5jaGVuCkdlc2NoYWVmdHNmdWVocmVyOiBEb3VnbGFzIEx1c2ssIFBldGVyIEdsZWlz
c25lciwgSGFubmVzIFNjaHdhZGVyZXIKUmVnaXN0ZXJnZXJpY2h0OiBNdWVuY2hlbiBIUkIgNDc0
NTYgClVzdC4tSWROci4vVkFUIFJlZ2lzdHJhdGlvbiBOby46IERFMTI5Mzg1ODk1CkNpdGliYW5r
IEZyYW5rZnVydCBhLk0uIChCTFogNTAyIDEwOSAwMCkgNjAwMTE5MDUyCg==