2022-01-07 10:59:14

by Thorsten Leemhuis

[permalink] [raw]
Subject: Special regressions report for the pending 5.16 release

Hi Linus,



a quick brief manual regressions report, as I assume you'll likely
release 5.16 soon and thus might find this helpful. Below is a list of
remaining regressions in 5.16-rc I'm currently aware of.


regressions where a fix exists

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~



* This fix afaics is not yet mainlined:

[PATCH] md/raid1: fix missing bitmap update w/o WriteMostly devices

https://lore.kernel.org/all/[email protected]/



But I'm pretty sure Jens (CCed) will send it onwards soon:

https://lore.kernel.org/all/[email protected]/



* There is an open regression "Applications that need amdgpu doesn't run
after waking up from suspend":

https://lore.kernel.org/all/[email protected]/



Wolfram (CCed) plans to revert a i2c commit to fix it, but I'm not sure
if he plans to send in onwards for 5.16 (or if that would be a good idea
at all):

https://lore.kernel.org/lkml/[email protected]/



* There are suspend and resume problems related to amdgpu:

https://bugzilla.kernel.org/show_bug.cgi?id=215436

Mario (CCed) recently found the root cause and came up with a fix, but
it likely needs a little more time to bake:

https://lore.kernel.org/all/BL1PR12MB5157F21C23A020052FF5C13BE24C9@BL1PR12MB5157.namprd12.prod.outlook.com/


https://lore.kernel.org/all/[email protected]/

no fix in sight

~~~~~~~~~~~~~~~



* screen contents do get restored after some input events (so it
doesn't stay blank).

https://bugzilla.kernel.org/show_bug.cgi?id=215203

(related: https://gitlab.freedesktop.org/drm/amd/-/issues/1840 )



Alex (CCed) is trying hard to find a fix for, but afaics needs more time.



Need more time to analyse

~~~~~~~~~~~~~~~~~~~~~~~~~



I'm aware of two more reports that afaics need (a lot?) more time to
analyse:



* suspend issues for a user that faces another regression that might
interfer:

https://lore.kernel.org/linux-pm/256689953.114854578.1640622738334.JavaMail.root@zimbra40-e7.priv.proxad.net/

https://bugzilla.kernel.org/show_bug.cgi?id=215427



* 5-10% increase in IO latencies with nohz balance patch

https://lore.kernel.org/lkml/YaUH5GFFoLiS4%2F3%[email protected]/



Closing words

~~~~~~~~~~~~~



HTH, ciao, Thorsten


2022-01-07 16:24:38

by Deucher, Alexander

[permalink] [raw]
Subject: RE: Special regressions report for the pending 5.16 release

[Public]

> -----Original Message-----
> From: Thorsten Leemhuis <[email protected]>
> Sent: Friday, January 7, 2022 5:59 AM
> To: Linus Torvalds <[email protected]>;
> [email protected]
> Cc: Song Liu <[email protected]>; Jens Axboe <[email protected]>;
> [email protected]; Limonciello, Mario <[email protected]>;
> Deucher, Alexander <[email protected]>; Koenig, Christian
> <[email protected]>; Pan, Xinhui <[email protected]>; Linux
> Kernel Mailing List <[email protected]>
> Subject: Special regressions report for the pending 5.16 release
>
> Hi Linus,
>
>
>
> a quick brief manual regressions report, as I assume you'll likely release 5.16
> soon and thus might find this helpful. Below is a list of remaining regressions
> in 5.16-rc I'm currently aware of.
>
>
> regressions where a fix exists
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>
>
> * This fix afaics is not yet mainlined:
>
> [PATCH] md/raid1: fix missing bitmap update w/o WriteMostly devices
>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> kernel.org%2Fall%2F20220103230401.180704-1-
> song%40kernel.org%2F&amp;data=04%7C01%7Calexander.deucher%40amd.
> com%7Ce1087ce64824412faec908d9d1ccbaf1%7C3dd8961fe4884e608e11a82
> d994e183d%7C0%7C0%7C637771500622271889%7CUnknown%7CTWFpbGZsb
> 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> %3D%7C3000&amp;sdata=RlZm%2B2vPqhVO0Vuy1eNtAh%2FKuVjuPIIQ2Fo
> 5RHXsp6U%3D&amp;reserved=0
>
>
>
> But I'm pretty sure Jens (CCed) will send it onwards soon:
>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> kernel.org%2Fall%2F499b185d-ff9a-934e-7768-
> ec796244fa1a%40kernel.dk%2F&amp;data=04%7C01%7Calexander.deucher
> %40amd.com%7Ce1087ce64824412faec908d9d1ccbaf1%7C3dd8961fe4884e60
> 8e11a82d994e183d%7C0%7C0%7C637771500622271889%7CUnknown%7CTW
> FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJX
> VCI6Mn0%3D%7C3000&amp;sdata=rdgKJ4GDQsnmb5RF5ZXEn48Ra32mcuFW
> n3esC1sDRr0%3D&amp;reserved=0
>
>
>
> * There is an open regression "Applications that need amdgpu doesn't run
> after waking up from suspend":
>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> kernel.org%2Fall%2F1295184560.182511.1639075777725%40mail.yahoo.com%
> 2F&amp;data=04%7C01%7Calexander.deucher%40amd.com%7Ce1087ce648
> 24412faec908d9d1ccbaf1%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C
> 0%7C637771500622271889%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj
> AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;
> sdata=Jn6vjdwISjMdm8tn0Pw0JSI1%2FD0JmUSiMCMX2FFD%2B70%3D&amp
> ;reserved=0
>
>
>
> Wolfram (CCed) plans to revert a i2c commit to fix it, but I'm not sure if he
> plans to send in onwards for 5.16 (or if that would be a good idea at all):
>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> kernel.org%2Flkml%2F20220106122452.18719-1-
> wsa%40kernel.org%2F&amp;data=04%7C01%7Calexander.deucher%40amd.
> com%7Ce1087ce64824412faec908d9d1ccbaf1%7C3dd8961fe4884e608e11a82
> d994e183d%7C0%7C0%7C637771500622271889%7CUnknown%7CTWFpbGZsb
> 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> %3D%7C3000&amp;sdata=%2FibaOVHm2hqKUOf%2BGnQThVw2lsF3znB5v2
> Q41ja%2BDtM%3D&amp;reserved=0
>
>
>
> * There are suspend and resume problems related to amdgpu:
>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugz
> illa.kernel.org%2Fshow_bug.cgi%3Fid%3D215436&amp;data=04%7C01%7Cal
> exander.deucher%40amd.com%7Ce1087ce64824412faec908d9d1ccbaf1%7C3
> dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637771500622271889%7C
> Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=vUyTuQkw3Z5bpIUkwf
> IwHyc5kTKaG3TRr34Tl%2FXHGLo%3D&amp;reserved=0
>
> Mario (CCed) recently found the root cause and came up with a fix, but it
> likely needs a little more time to bake:
>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> kernel.org%2Fall%2FBL1PR12MB5157F21C23A020052FF5C13BE24C9%40BL1PR
> 12MB5157.namprd12.prod.outlook.com%2F&amp;data=04%7C01%7Calexan
> der.deucher%40amd.com%7Ce1087ce64824412faec908d9d1ccbaf1%7C3dd89
> 61fe4884e608e11a82d994e183d%7C0%7C0%7C637771500622271889%7CUnkn
> own%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik
> 1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=z5%2FJB2I9ciDhwdvwxjPKZ
> O1t49kwO3Br%2BhXdvUQgyoY%3D&amp;reserved=0
>
>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> kernel.org%2Fall%2F20220106163054.13781-3-
> mario.limonciello%40amd.com%2F&amp;data=04%7C01%7Calexander.deuch
> er%40amd.com%7Ce1087ce64824412faec908d9d1ccbaf1%7C3dd8961fe4884e
> 608e11a82d994e183d%7C0%7C0%7C637771500622271889%7CUnknown%7CT
> WFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> JXVCI6Mn0%3D%7C3000&amp;sdata=TGrjrWX22DBOshfAiD9mh8diCq%2FkI
> Mwg6xMdU03aLHc%3D&amp;reserved=0
>
> no fix in sight
>
> ~~~~~~~~~~~~~~~
>
>
>
> * screen contents do get restored after some input events (so it doesn't stay
> blank).
>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugz
> illa.kernel.org%2Fshow_bug.cgi%3Fid%3D215203&amp;data=04%7C01%7Cal
> exander.deucher%40amd.com%7Ce1087ce64824412faec908d9d1ccbaf1%7C3
> dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637771500622271889%7C
> Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=%2FpAS9qTCWx%2Br%
> 2FS%2B9WRGC%2FcSFdgw5hWTZIRKHjp189Lg%3D&amp;reserved=0
>
> (related:
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitla
> b.freedesktop.org%2Fdrm%2Famd%2F-
> %2Fissues%2F1840&amp;data=04%7C01%7Calexander.deucher%40amd.com
> %7Ce1087ce64824412faec908d9d1ccbaf1%7C3dd8961fe4884e608e11a82d994
> e183d%7C0%7C0%7C637771500622271889%7CUnknown%7CTWFpbGZsb3d8
> eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
> D%7C3000&amp;sdata=hrNAXNori6%2BtgP8X6YEJT5V2Hsnm1mTiZNZI0y7XKL
> A%3D&amp;reserved=0 )
>
>
>
> Alex (CCed) is trying hard to find a fix for, but afaics needs more time.

I already sent a workaround to restore the previous behavoir. It's on its way to Linus in Dave's last -fixes PR. We are working on a proper fix hopefully for 5.17.

Alex

>
>
>
> Need more time to analyse
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~
>
>
>
> I'm aware of two more reports that afaics need (a lot?) more time to
> analyse:
>
>
>
> * suspend issues for a user that faces another regression that might
> interfer:
>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> kernel.org%2Flinux-
> pm%2F256689953.114854578.1640622738334.JavaMail.root%40zimbra40-
> e7.priv.proxad.net%2F&amp;data=04%7C01%7Calexander.deucher%40amd.
> com%7Ce1087ce64824412faec908d9d1ccbaf1%7C3dd8961fe4884e608e11a82
> d994e183d%7C0%7C0%7C637771500622271889%7CUnknown%7CTWFpbGZsb
> 3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0
> %3D%7C3000&amp;sdata=4ycXrg1yKHneHJisDmUcSHnXTArFMDP2EZacmtIG
> %2BqM%3D&amp;reserved=0
>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugz
> illa.kernel.org%2Fshow_bug.cgi%3Fid%3D215427&amp;data=04%7C01%7Cal
> exander.deucher%40amd.com%7Ce1087ce64824412faec908d9d1ccbaf1%7C3
> dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637771500622271889%7C
> Unknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=xw15%2BdfZMQr%2BU
> 5S9mexnYJyRW1UahrrBLUJRu15oDQ8%3D&amp;reserved=0
>
>
>
> * 5-10% increase in IO latencies with nohz balance patch
>
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flore.
> kernel.org%2Flkml%2FYaUH5GFFoLiS4%252F3%252F%40localhost.localdomai
> n%2F&amp;data=04%7C01%7Calexander.deucher%40amd.com%7Ce1087ce6
> 4824412faec908d9d1ccbaf1%7C3dd8961fe4884e608e11a82d994e183d%7C0%
> 7C0%7C637771500622271889%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4
> wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&a
> mp;sdata=Rt%2F06JdscRKj5nQKoXiM75l3JPjwwMQ8f9dx%2FsXSUDw%3D&a
> mp;reserved=0
>
>
>
> Closing words
>
> ~~~~~~~~~~~~~
>
>
>
> HTH, ciao, Thorsten

2022-01-07 16:30:14

by Jens Axboe

[permalink] [raw]
Subject: Re: Special regressions report for the pending 5.16 release

On 1/7/22 3:59 AM, Thorsten Leemhuis wrote:
> * This fix afaics is not yet mainlined:
>
> [PATCH] md/raid1: fix missing bitmap update w/o WriteMostly devices
>
> https://lore.kernel.org/all/[email protected]/
>
>
>
> But I'm pretty sure Jens (CCed) will send it onwards soon:
>
> https://lore.kernel.org/all/[email protected]/

Going upstream today.

--
Jens Axboe


2022-01-07 20:31:44

by Wolfram Sang

[permalink] [raw]
Subject: Re: Special regressions report for the pending 5.16 release


> * There is an open regression "Applications that need amdgpu doesn't run
> after waking up from suspend":
>
> https://lore.kernel.org/all/[email protected]/
>
> Wolfram (CCed) plans to revert a i2c commit to fix it, but I'm not sure
> if he plans to send in onwards for 5.16 (or if that would be a good idea
> at all):
>
> https://lore.kernel.org/lkml/[email protected]/

I'll send a pull request tomorrow.


Attachments:
(No filename) (463.00 B)
signature.asc (833.00 B)
Download all attachments

2022-01-07 21:33:59

by Linus Torvalds

[permalink] [raw]
Subject: Re: Special regressions report for the pending 5.16 release

On Fri, Jan 7, 2022 at 2:59 AM Thorsten Leemhuis <[email protected]> wrote:
>
> [PATCH] md/raid1: fix missing bitmap update w/o WriteMostly devices

Merged.

> Wolfram (CCed) plans to revert a i2c commit to fix it, but I'm not sure
> if he plans to send in onwards for 5.16 (or if that would be a good idea
> at all):

So apparently I'm getting the pull tomorrow.

> * There are suspend and resume problems related to amdgpu:

Fix merged (and tested at least on my system).

Linus