Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <48da33f3-327d-7144-f967-84f063417b0a@oracle.com>
Date:   Mon, 1 Aug 2022 11:55:03 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.11.0
Subject: Re: [PATCH] NFSD: fix use-after-free on source server when doing
 inter-server copy
Content-Language: en-US
To:     Chuck Lever III <chuck.lever@oracle.com>
Cc:     Olga Kornievskaia <olga.kornievskaia@gmail.com>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
References: <1659298792-5735-1-git-send-email-dai.ngo@oracle.com>
 <37585D43-78F3-4132-8ADF-D11BD11DDCD4@oracle.com>
From:   dai.ngo@oracle.com
In-Reply-To: <37585D43-78F3-4132-8ADF-D11BD11DDCD4@oracle.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?TTdkWFJwc1IvRW5KVVVRdHlPa3Q3V1o4ay9wWWZzVlFHczY2UGFnZUwzaHd1?=
 =?utf-8?B?MmJEUkZwcEtNUUJLemNVRG9aQ2QxZEZ2Zit0NHgzQUFSWEt1bWJvZUUrT1Iw?=
 =?utf-8?B?K01CbkFjUmlMNnFHeXVOVWR6UWlvZTVCRVhDclM3NHM0NC84Rmd2YnJFcEpN?=
 =?utf-8?B?YTVFLzlXYy9OcGdJamNGNldYMCtkWFBIcmE1UW00SnNrZkxRbUVNTDdEVlhu?=
 =?utf-8?B?cmNKMXg1N2lzbzluc0V1WkplS2YvYTljYWpTTENzN1lmQ0YvYXZqZUg1aEJR?=
 =?utf-8?B?ZFZjNnlRSER0QUE3THFnOU5yeFpCbnJGbzZoMysyMzFtZGpCU3h4bEtaU1Vn?=
 =?utf-8?B?UndXcHg2M2hJYTVZd0w2VFlhTjliTk95MWN1MmZPVW1Rc0NCc25YTGJNb1N0?=
 =?utf-8?B?Yy9RL0ZpZW1TT1pXV0dLaHVyVmRkOE16Tit0NmQ4czZpVmNWem80TE51SHVk?=
 =?utf-8?B?a05KRTY3cE0zbE5uU2ZHOU15VGpIQ2tRMnNFUUlSRm1lL0xVdy9oYkUzZk5K?=
 =?utf-8?B?Y1NTdmRydm1mRkgxK1V6Q0N2SFgxSHF3R0RTMWIwVmZIYmlVMksxTlZOV0xC?=
 =?utf-8?B?SENvWW9pNFEvK1Q3dzR2K1hzR21teFV1V2I4V1A1NzJOSEZrU2YzdHFMeXZy?=
 =?utf-8?B?TzdzNVR0VUZkWVRaQWZpT3lWbTNXc2VtUFNEaU43cWVlQW1aMmJ6RHIrUUJK?=
 =?utf-8?B?RjJrUk02OExFUEtQRkNGVnNHYnVuNkpzbGl2WGc4TzF4aDEwblRNVTBtUzdy?=
 =?utf-8?B?UzJJREhiNHAvOTd6cDVVcUJuYmh6N3drMDJxRG9ad2lUcmlHUkExaWJkUW85?=
 =?utf-8?B?aHV5WlZaSzBKc3o5N0JaMFUrejRMNVYrNnovZTJJb0ZoRFZxdVV1UG5MeEFz?=
 =?utf-8?B?cG1ZOVpSVndoUWV2dHV6VUdxNmFQSEcrOXh6bzUzV3NIZE1pUzFIRy9hajhE?=
 =?utf-8?B?K3ZDYkNoQmM2dWc3elV4MVpOWU1UdjNnd0tzY0pQbkE2NWJnQUFUWE9ENTRX?=
 =?utf-8?B?elVnU01sWlhTWmRUVml1ZE1nd3RDMVEvL0lRR2FKbDJYREI1WEswWGdnL1c2?=
 =?utf-8?B?UjlHa2xGNEFxWlNKcWE1dE1tYW1yeXAraFFLeWw2N1QybDR3U2NPM2c0QXZ1?=
 =?utf-8?B?K2VqaTA1ZkJvK0IxYzRKWXorelBVbGxrQkVWaTRlWmpEVVFRb2o3OExFMHgw?=
 =?utf-8?B?MXBZb2NBRXUza25wR29KL3hlMENMdUlESFQrZXZZR1huQmpxTTBEQTZnbXd3?=
 =?utf-8?B?dHJZbFZnRXpidmMyN0xUN2VuQ2hObUxwc1BGSWg0bGlXcFFqWVFuNHpuaGJP?=
 =?utf-8?B?aERGc0R1TTh6M2ZiZllJbHdkeDZjZ2xpZWFqcHZRaFFudGZaRHpnWnlSSlc2?=
 =?utf-8?B?QXc3cmVNdURHUnZhakQxc0J4NFBlaHdlb2M1REMzbWdHVDNSK3NkM3Z5aXdn?=
 =?utf-8?B?emgvbDlFTXZ3UDdmZ2x6anpFSGVkdysxZ2p3bmtpMlRXRG9Pc0RtWmZTbnkv?=
 =?utf-8?B?R1BRcW1KNjNpTUtzUmVIT3JXRGN4Qzd5TTNJVnkvR0lHdVZiUGtkMXdLQ2sv?=
 =?utf-8?B?RkpOQm9Fb0tpK3dKVkg2ek9ITDNHSmNVZU16amU1Wlo4N2REUWNLYXhMZTZw?=
 =?utf-8?B?VW9vMzRKWDZyVndrVSsvekJWeUpWU1VGMGpGZlRwL05GY3pzbDVwTXZ0eXM0?=
 =?utf-8?B?bUVDS0FERUdQb3hJWmJuWFBuYmxjcFJQaHc0bGxQSkpZS1BqZTcrSWVRU1pZ?=
 =?utf-8?B?bWg0UG1VUHdVczdPSTV3dlVVQnhBSVRoNnVPN003VEdlQWswL0NsK0ZKbXY5?=
 =?utf-8?B?ak5YNE5MRisxYnk3U3JKcXNySEJzS1hyZHE4ejR6b0pGTkE1VnpLU211UVlZ?=
 =?utf-8?B?T3hTSGJsOHFpamdtbWdGQXY0ejRGOFZTQkYvWHd4eUtvWThZRkZWbEtlMjdC?=
 =?utf-8?B?bzBaLzRIQ2FOR1A2QUlVSUo3YS9ISFh1c3NJQU5mZ2h6VzFlM0l2b1FGMjFq?=
 =?utf-8?B?R1ZuYjRwL1NLZndCZlF5bHJmbVdPZC83VHFGOUdlVDdiQzhUemZ5bHliQ0Ey?=
 =?utf-8?B?N2xnT0lKSUc1RDdNVnREK0k4SHU2Z2RjMUxYMkhKekJsTm1RYmIzUWp5aVVD?=
 =?utf-8?Q?llHjM1/qgYpDYeoT9+iWSuzbj?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 01085af2-3be1-4a4d-7044-08da73ef5987
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Aug 2022 18:55:09.7647
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: ahFAeXf50uIteqCJcfzCCgix6b+gfpdP2jbqXVHdpwpbsQTrq+5cXfpf1BtqXHjneI/yyAL07AWojAuWlUHBWg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR10MB1746
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1
 definitions=2022-08-01_08,2022-08-01_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxlogscore=999
 malwarescore=0 bulkscore=0 spamscore=0 phishscore=0 mlxscore=0
 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2206140000 definitions=main-2208010094
X-Proofpoint-ORIG-GUID: ucZRNH5x1O-zfN2FGADhPaQ2SaqiRMs0
X-Proofpoint-GUID: ucZRNH5x1O-zfN2FGADhPaQ2SaqiRMs0
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


On 8/1/22 9:29 AM, Chuck Lever III wrote:
>
>> On Jul 31, 2022, at 4:19 PM, Dai Ngo <dai.ngo@oracle.com> wrote:
>>
>> Use-after-free occurred when the laundromat tried to free expired
>> cpntf_state entry on the s2s_cp_stateids list after inter-server
>> copy completed. The sc_cp_list that the expired copy state was
>> inserted on was already freed.
>>
>> When COPY completes, the Linux client normally sends LOCKU(lock_state x),
>> FREE_STATEID(lock_state x) and CLOSE(open_state y) to the source server.
>> The nfs4_put_stid call from nfsd4_free_stateid cleans up the copy state
>> from the s2s_cp_stateids list before freeing the lock state's stid.
>>
>> However, sometimes the CLOSE was sent before the FREE_STATEID request.
>> When this happens, the nfsd4_close_open_stateid call from nfsd4_close
>> frees all lock states on its st_locks list without cleaning up the copy
>> state on the sc_cp_list list. When the time the FREE_STATEID arrives the
>> server returns BAD_STATEID since the lock state was freed. This causes
>> the use-after-free error to occur when the laundromat tries to free
>> the expired cpntf_state.
>>
>> This patch adds a call to nfs4_free_cpntf_statelist in
>> nfsd4_close_open_stateid to clean up the copy state before calling
>> free_ol_stateid_reaplist to free the lock state's stid on the reaplist.
>>
>> Signed-off-by: Dai Ngo <dai.ngo@oracle.com>
> I'm interested in Olga's comments as well, so I'm going to
> wait a bit before applying this one.
>
> Also, did you figure out where this crash started to occur?
> I'd like to have a precise sense of whether this should be
> backported.

I think this started with the commit 624322f1adc58 to introduce
COPY_NOTIFY operation.

-Dai

>
>
>> ---
>> fs/nfsd/nfs4state.c | 3 +++
>> 1 file changed, 3 insertions(+)
>>
>> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
>> index 9409a0dc1b76..749f51dff5c7 100644
>> --- a/fs/nfsd/nfs4state.c
>> +++ b/fs/nfsd/nfs4state.c
>> @@ -6608,6 +6608,7 @@ static void nfsd4_close_open_stateid(struct nfs4_ol_stateid *s)
>> 	struct nfs4_client *clp = s->st_stid.sc_client;
>> 	bool unhashed;
>> 	LIST_HEAD(reaplist);
>> +	struct nfs4_ol_stateid *stp;
>>
>> 	spin_lock(&clp->cl_lock);
>> 	unhashed = unhash_open_stateid(s, &reaplist);
>> @@ -6616,6 +6617,8 @@ static void nfsd4_close_open_stateid(struct nfs4_ol_stateid *s)
>> 		if (unhashed)
>> 			put_ol_stateid_locked(s, &reaplist);
>> 		spin_unlock(&clp->cl_lock);
>> +		list_for_each_entry(stp, &reaplist, st_locks)
>> +			nfs4_free_cpntf_statelist(clp->net, &stp->st_stid);
>> 		free_ol_stateid_reaplist(&reaplist);
>> 	} else {
>> 		spin_unlock(&clp->cl_lock);
>> -- 
>> 2.9.5
>>
> --
> Chuck Lever
>
>
>