Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <3b7b3462-0f15-58a7-b49c-eb563d20a8ec@oracle.com>
Date:   Wed, 11 Jan 2023 03:31:29 -0800
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0)
 Gecko/20100101 Thunderbird/102.6.1
Subject: Re: [PATCH 1/1] NFSD: fix WARN_ON_ONCE in __queue_delayed_work
Content-Language: en-US
To:     Mike Galbraith <efault@gmx.de>, Jeff Layton <jlayton@kernel.org>,
        Chuck Lever III <chuck.lever@oracle.com>
Cc:     Linux NFS Mailing List <linux-nfs@vger.kernel.org>
References: <1673333310-24837-1-git-send-email-dai.ngo@oracle.com>
 <57dc06d57b4b643b4bf04daf28acca202c9f7a85.camel@kernel.org>
 <71672c07-5e53-31e6-14b1-e067fd56df57@oracle.com>
 <8C3345FB-6EDF-411A-B942-5AFA03A89BA2@oracle.com>
 <5e34288720627d2a09ae53986780b2d293a54eea.camel@kernel.org>
 <42876697-ba42-c38f-219d-f760b94e5fed@oracle.com>
 <f0f56b451287d17426defe77aee1b1240d2a1b31.camel@kernel.org>
 <8e0cb925-9f73-720d-b402-a7204659ff7f@oracle.com>
 <37c80eaf2f6d8a5d318e2b10e737a1c351b27427.camel@gmx.de>
 <ce3724b88bb2987ac773057f523aa0ed2abacaed.camel@kernel.org>
 <2067b4b4ce029ab5be982820b81241cd457ff475.camel@kernel.org>
 <ec6593bce96f8a6a7928394f19419fb8a4725413.camel@gmx.de>
From:   dai.ngo@oracle.com
In-Reply-To: <ec6593bce96f8a6a7928394f19419fb8a4725413.camel@gmx.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?Nm42SkUyMkdPVFF6ejBNeGZQYUhiZGJrVDFwUFdxTStEV2lJZkY0alh4OUV6?=
 =?utf-8?B?b2dYTldyOUV4K3Fyemo5bzdkNkhRUGErOEU4bXhhQnpFSFRMaFZrRlAzVXgy?=
 =?utf-8?B?dWU0eFltQXRPdjFTQ084TjZpSXpVV3hwK3FmQnlmbE9mTGpETzYrajZiMlFk?=
 =?utf-8?B?L29BaVk0N1RRNFo0R3ZhazhsRWRla2NlU01JT3lLKzNvN2prZWFBQXBuV0Zy?=
 =?utf-8?B?NXZWQWNuTWIrMGsydnNtb01rQkhsV0JBSlUvN2x3WnU4VGVlSkkyTm9PaGty?=
 =?utf-8?B?SjFBQ3J1bzAyTWNEQTZhclorT0w3K2NGOHB2bTNJVFNWYWFBLzB3ajdsY3Q3?=
 =?utf-8?B?STFnMWNDSDZFb2d0SytHczZ4SzVIUXVkSDBwN2d6VDVYRG5EZitUMitZeko5?=
 =?utf-8?B?OTVrN1R0VmV1eFArK2pHR0hvWWJRelBaUjRiRUhhR1NFa3VhUFNZNTYwZHIy?=
 =?utf-8?B?R2N5bEhuL0NvOG14dURxTU9xYkFCMGtBdVZEYStnUDVUQlRuMlcwMytrNzAy?=
 =?utf-8?B?bFljTEs5YTA3Z3NheGRUZXRiUHo1eUF6c2ZNMkZrU0JGVmpCRDNWNTlIRTNK?=
 =?utf-8?B?bVZMK1hINm54WEkreU15ekZzNHcyQTR6RFpTaVlXQXM2S0ZvdVV4RzZ1UWxl?=
 =?utf-8?B?cThDY2lqNG51RWtlSTdJU1Ixa3NLc2hoNWFjRXExM0l1U1pWMFBaMnp3U2xO?=
 =?utf-8?B?eEM5YmNBWFUwN3k1V2k1bUZHZnFmY1ZnUk1MRmQ5ZXl5d0ZyZStpR1d1eS9w?=
 =?utf-8?B?Uy9aN3VlZ21kV05XNXpwZTVaRi9aSktWQlhUTjRFaXdmUmRnZjZGem5Oc2E4?=
 =?utf-8?B?a1Vub3BPQi90WlVNRTNpcjRUWGZTMTNzWDR0VmJEUlRmTlFRa09yamdtaW55?=
 =?utf-8?B?eXdySkYyak8zOFk0UmY0L0oxeWYyL0ZqMVhycU15T3ZhbzhydnloVHRhK2h6?=
 =?utf-8?B?MTBlbUpDek8vRkxsQzJGWFNvQ20xejJpNWJpa2t2cVJlZFJiY2pyWnhmOGlw?=
 =?utf-8?B?WW16TmhDRmVTbzJvWlFicVE4QitZS2ZKN3lZR3FQbGc4eGZBaXI2alZJbC9W?=
 =?utf-8?B?YmUzaEQwYlM0dEtMeEVHQXdMejBLcjQ3UEVtM0g5Z1IreGU5dExZU1pNeWlL?=
 =?utf-8?B?UGtBZ3BqSFhpTm50U052NWx6RGVoSHN3a2FQZE5zMURrbVdzakp6aFhGNnpQ?=
 =?utf-8?B?MlBTMW0xRjA0T1dWRlp4eUZ2V2NqS3pQMkcvVDh3QzErZkFEcWJjWm92Mkgx?=
 =?utf-8?B?K1o5b2hXdUlQV2pGL2pxb0MxVk5Jb0ViZytOMGdFUU0reitPVXI5c294UVdB?=
 =?utf-8?B?Q3NRazJ3RGpkVkFheGNEb0oxSmszSjljQ1M5V1J1Z1BtSmhKYnRvcDhDVTBZ?=
 =?utf-8?B?Z0VrdE04T1NRZHhNeUxEenRvbkpNRTVrSG9sMHBSeklHUUZhblFXQXhCYW1I?=
 =?utf-8?B?Uk5RUWRVQVpkVU1kSEFIV0I5cXA2eU02WDJPQ2xYbEdTQndUS1N0bFFLclhu?=
 =?utf-8?B?eDdGRTdzcTB1YWZucVdOYm1wT1g5Q0JGZXNoOXo2ejhVZXZQQjB4RVRxVnZk?=
 =?utf-8?B?WTJRN3BqdHhXeXJ2bzkrQjFINW5QTWJVRlcrVFo0L3JncU9FVmFKR1M5N2NK?=
 =?utf-8?B?bHFnOGtUYmE0UWk0WWFCU25DNmNRcGptaEtsbHorS1QvVFZ5ZUdWRFEyT1p3?=
 =?utf-8?B?Y0d2alA2Z2trd1F5TFl5c0w1VmZzWk40UHhrWTBla3dvR015WFNrbHhKRjZZ?=
 =?utf-8?B?NUN6UENudDNhWkZyYzJ1OWc1R1hsV1VLYmVFR2ZDeit3R3pvS1JPODMxY0FB?=
 =?utf-8?B?OUNZRmZacDdYc2VVdC83eGt6UTNrTnRLSXNyY3A3bm1SUWt6elp0ekRYYjJB?=
 =?utf-8?B?V21yYVpLb1hIQTNaWmNuZnFseHo4V0ttRHJBMXpnY0RxNDlOQjFNbnRyb01W?=
 =?utf-8?B?ZUhNN0JWaXJRem9YQjdYYVcvbUE4UXpHcERoeGVubHhWRm9ZOC9hbnRab2dT?=
 =?utf-8?B?Z0F6ZGZNUnQrcC9Md2dId0JTUldra1BTYlNKVm9YMGd2bUVGak9qYytvcXFL?=
 =?utf-8?B?ZHZ5eWtOSWpoTEpxV1JHbDJsQnFxam84N1Z3S3hKNTB4ZDNVUFlCZi96dUFa?=
 =?utf-8?Q?9UANs3K7JL1RWMqIcl1YRkE5G?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: LIhP4rPhsx4+roBHFemJET/XU4C8VGd2yoXwNiJKRtx173oPpYrqvEz6K8Qsr7fLOJkqGtmQqtMjipBDoY23pkP1G8z90t1znxEZi4YenWtSsRU8wUdzeL2gfiPHzGaezmnIeetIsk+pMORmXz/JtAgL4pHGE6/wO33UMkfhJZK8edW2X5KOdnzw3kKGqwX48KlqPn/+KvJmw12MP6Hw/VDhK2bzJl1EImP4gyNaEHPemsF1lBP+lsjWv+KuoiuWbQk/BkUw/YX/5YfgRIgIku3ZRygl05jiwvlUYXd2KQqt275DNvte/bj3KqYQHllq2aWyL+pFBFdy5t/GgB7lKWlyyjt17o3FNVHQX186JocpKqemQa3FlDQi3kWdU79iOkUfHLE1Rxwo5MrkC+VbpZslHSF6/QGrjWAqyevpdbSMCm1OBHPC2UGFu4lNJjCHMJmqYKFBz4bCsoes7TZBSf8YfqvYfXWbHLDpJUZ0AKiPSZG9OFVh0rTPRgz+OYKslVcBy+C8sFsnjtJAFBUsRRGP/vDCQxmno5I+fVgA24W2BNXRiDLtKeylD2OivPVX+8pucrGsGTq6aD+9siAVQzjEou+hbaTga8KjQdbAfQ2plnLlb1FcN4Apce3LCc9O5lVn1ZwFsGMMmaXwsJKn2dx5sVlbj0SlmYxHFtbv3i1m48llGlDQc8e/aW/36hMOWVHxfX2tAzjmrR+eMFZm//qRxUl3ewiylQGCukvzoOJIrx7p/lZGlWuImoYQaOAxWFiTPyDMeyVrPm+zi+nkbFc8yoKMnBYHDab3nnOsPk6KaWpnf3r8Qma53h3adzk/
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 95ac87b2-895b-4e01-6f61-08daf3c76397
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Jan 2023 11:31:32.2818
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: IqfwtsGiXpph/jGJoHAuc+E6O1yarusuYK7w2W/P9AxldDS/Jv0Y1/PgLXuBAieRgSa25VOzIreyb4RbBJljYQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR10MB4203
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.219,Aquarius:18.0.923,Hydra:6.0.545,FMLib:17.11.122.1
 definitions=2023-01-11_05,2023-01-11_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=999
 malwarescore=0 mlxscore=0 suspectscore=0 phishscore=0 adultscore=0
 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2212070000 definitions=main-2301110085
X-Proofpoint-ORIG-GUID: 6Qy8Tmrp8E1dnpMw1PpSuvPimyle3E9D
X-Proofpoint-GUID: 6Qy8Tmrp8E1dnpMw1PpSuvPimyle3E9D
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


On 1/11/23 3:19 AM, Mike Galbraith wrote:
> On Wed, 2023-01-11 at 05:55 -0500, Jeff Layton wrote:
>>>> crash> delayed_work ffff8881601fab48
>>>> struct delayed_work {
>>>>    work = {
>>>>      data = {
>>>>        counter = 1
>>>>      },
>>>>      entry = {
>>>>        next = 0x0,
>>>>        prev = 0x0
>>>>      },
>>>>      func = 0x0
>>>>    },
>>>>    timer = {
>>>>      entry = {
>>>>        next = 0x0,
>>>>        pprev = 0x0
>>>>      },
>>>>      expires = 0,
>>>>      function = 0x0,
>>>>      flags = 0
>>>>    },
>>>>    wq = 0x0,
>>>>    cpu = 0
>>>> }
>>> That looks more like a memory scribble or UAF. Merely having multiple
>>> tasks calling queue_work at the same time wouldn't be enough to trigger
>>> this, IMO. It's more likely that the extra locking is changing the
>>> timing of your reproducer somehow.
>>>
>>> It might be interesting to turn up KASAN if you're able.
> I can try that.
>
>> If you still have this vmcore, it might be interesting to do the pointer
>> math and find the nfsd_net structure that contains the above
>> delayed_work. Does the rest of it also seem to be corrupt? My guess is
>> that the corrupted structure extends beyond just the delayed_work above.
>>
>> Also, it might be helpful to do this:
>>
>>       kmem -s ffff8881601fab48
>>
>> ...which should tell us whether and what part of the slab this object is
>> now a part of. That said, net-namespace object allocations are somewhat
>> weird, and I'm not 100% sure they come out of the slab.
> I tossed the vmcore, but can generate another.  I had done kmem sans -s
> previously, still have that.
>
> crash> kmem ffff8881601fab48
> CACHE             OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE  NAME
> kmem: kmalloc-1k: partial list slab: ffffea0005b20c08 invalid page.inuse: -1
> ffff888100041840     1024       2329      2432     76    32k  kmalloc-1k
>    SLAB              MEMORY            NODE  TOTAL  ALLOCATED  FREE
>    ffffea0005807e00  ffff8881601f8000     0     32         32     0
>    FREE / [ALLOCATED]
>    [ffff8881601fa800]
>
>        PAGE        PHYSICAL      MAPPING       INDEX CNT FLAGS
> ffffea0005807e80 1601fa000 dead000000000400        0  0 200000000000000
> crash

Can you try:

crash7latest> nfsd_net_id
nfsd_net_id = $2 = 9                <<===
crash7latest> struct net.gen  init_net
   gen = 0xffff97fc17d07d80
crash7latest> x /10g 0xffff97fc17d07d80
0xffff97fc17d07d80:	0x000000000000000d	0x0000000000000000
0xffff97fc17d07d90:	0x0000000000000000	0xffff97fc0ac40060
0xffff97fc17d07da0:	0xffff994e7bf87600	0xffff98f731172a20
0xffff97fc17d07db0:	0xffff9844b05d9c00	0xffff9832a6a0add0
0xffff97fc17d07dc0:	0xffff984a4470d740	0xffff984a93eb0600    <<=== entry for nfsd_net_id
crash7latest> nfsd_net 0xffff984a93eb0600

Thanks,
-Dai

>