Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <c969d149-36be-a161-2b2a-469dbf7b9bcf@oracle.com>
Date:   Mon, 1 Aug 2022 11:52:33 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.11.0
Subject: Re: [PATCH] NFSD: fix use-after-free on source server when doing
 inter-server copy
Content-Language: en-US
To:     Jeff Layton <jlayton@kernel.org>, chuck.lever@oracle.com,
        olga.kornievskaia@gmail.com
Cc:     linux-nfs@vger.kernel.org
References: <1659298792-5735-1-git-send-email-dai.ngo@oracle.com>
 <ea6d0556198dd6b77f1ab711401ca65bc39e9912.camel@kernel.org>
From:   dai.ngo@oracle.com
In-Reply-To: <ea6d0556198dd6b77f1ab711401ca65bc39e9912.camel@kernel.org>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?OVVKaytoVzV1K0IvUmxvVW5vN2xpL3BseGNZUEZtZk9sY0tIN3JRNkZpT3VI?=
 =?utf-8?B?VC9QU2FrUXBDSXRDRDNMR3BmeVB1aDN1Qm1iRllXUDRMdEtPM3Zxb1hYM2dx?=
 =?utf-8?B?aDlwb3RMN0dIRzIxR081bGY4Y3VLc1dWS3MrTkRvYW1LeVFzVENZQ0I0MXhB?=
 =?utf-8?B?YlZ4VWNYUFdiOXgxQWVzMjFqYjhjWEpsVFRsT1BaVWNVS1o3eUpEN2NKSEpV?=
 =?utf-8?B?b1BQUkJLZ2FESEFQbmxOVTdRa2dOcUE5YkdDUFNoc2JZaUJtSDcyYTM5ZmRx?=
 =?utf-8?B?YkxPOU0zMnlic2oxRi9mWTlFUGI1ak11Y2l3UEtEUWRYNWhXSjdSSFFoL3Jq?=
 =?utf-8?B?MTRrRkp4WllnNFVNcGZoRXI4Y0U5c1ZZY0tWeVFWN1FGNVpGRWRNZ1VVZXM1?=
 =?utf-8?B?c3FOM0o4V1p6RWF2aWRkZFBTeEl0UVJKcmM4bGNqMGRodW90QSswZy9MZTFN?=
 =?utf-8?B?U0ZCTFVCWS9UUjEyeVNKSmpzZ3F2ZW9IZVJYdDZXZDFJYTlqT1pyV1pJZEJL?=
 =?utf-8?B?Yi8rN0NuaUxtWS8ra3l1RWpqMjRWYmx4WlhJR2owSHRzNEhkVDdCZlVLKzh6?=
 =?utf-8?B?dlVlUitVWlBwdUE2Nm1aTWZvd1dBOWpJWm1WZXFnN3VpV0p6UnJGSWlaeWw2?=
 =?utf-8?B?bUJGMjJNSTNaY0NONFY5cHZUcjJxWVNqNjlsTkZKcFZQa0xlb3dmTzNDcnls?=
 =?utf-8?B?RUEzbkE0MXJBYm1qQlhxa2p3QXVmSGtGNlRWUzFkRkFxaUJqc0xXUiszOVhy?=
 =?utf-8?B?Zyt3bWcreUt6N1dRZEFZYkRDV003VWRRUHBEUTNHNEFxRGN2c1IrbHFFSlZH?=
 =?utf-8?B?NmF6aEk2bnB4cW52S2NBOGlCbFdwanl0cXJTbVpSWFdocERxMUJBa1lvQWUv?=
 =?utf-8?B?b2Ztd2Zla0tmcjNLMXFSOEY0OGpaMGkzQnJmTFgrcEtTVjJhMDY2RnE3VEF0?=
 =?utf-8?B?RDM2U3ZnenpyNzVsNVNBNUdXUjRCZ0xqRUNUdGFYcEFXRFdUbjRWZTczd2Zj?=
 =?utf-8?B?V0J1WWRUbzVtM3VsbHp5MTA2SnlPMVZBODlMeG1WY1l5NHpFTHp2Nm82clJa?=
 =?utf-8?B?V1pLejhac0loOUx6d2VkTGVYWUxVNDFjK3NPdHJkQ0hxSmNCRU1Ka1pmczdr?=
 =?utf-8?B?T3phUGsrbk9iVS84M2VqaFA5RklYYmY3M0ZzUnJJNVF6TEYyUTFSSzNnNHdH?=
 =?utf-8?B?K0M1WUh1c0wyaWM3dWIzNjlFUlZrTXozS21KR2tIU1FPUHhLK2Z1NzlZN0Nk?=
 =?utf-8?B?S050MlBqUUJ6cTQ1KzNucWs0dzBXeWdNN05QdUdieXkxQmpCMERGZVNmbDdm?=
 =?utf-8?B?bmMyZzl5ckcyZ20rYndLZHBEQU9iSHo5enFjNlRVekJQSjdiSFZHZXM5bENF?=
 =?utf-8?B?eFNqRWRCbGZjc0ZYTFJGNEhSYzQvUHRnM2RBeGM3SlVhRkgrVnN3dTJZMDU5?=
 =?utf-8?B?TGFlZGc4VUVJL3RNTUx2YUppeGRVTEs4c2tJZGdUVkdtNnNFclZtNWpZcEJG?=
 =?utf-8?B?UE1zL3hVRWxuT0c4WkRxY3drUzU5N2RJeGN4b0ZNUDdSUFZrVkhPMVMxcGFT?=
 =?utf-8?B?MnNwMVpjWnRIUDE5VUJRaHQzYTNEZStlNjhJNnNkalJYNWJQcmE0TDJVQ09t?=
 =?utf-8?B?YnVYbFZRTEljR1htUEV3WHJZMTFLemJYQ3B2azAyUU45M3RraDQ5QUNKY0Yw?=
 =?utf-8?B?bVdLMklJcjZBZkN3T2orMVhGQm5qcmpzd0g5UW1zY1F5S2toWUlSdmJKdWli?=
 =?utf-8?B?U2pRQktFT0NZZTlUZkwzZjIreFZaNnFwOHcxREFQZzQreGprQnk1TkdSN0t0?=
 =?utf-8?B?Ni93TStxNW42bmk3TlQzdjZKalhVT3o3djJoQmlNdWNLeEFjT3cvQS9NOHBL?=
 =?utf-8?B?VjRVZkRPamlWMVBmNXVuRlpTb2ZXeHprdzZ4NEIvNStZeTZaOG11UTFWQ3Ez?=
 =?utf-8?B?dXh2YTNzaFlUcTI0SHpubjVmcXVMQm9IVjVYMWNvM3NMYnZjOUF3U0tNUGF4?=
 =?utf-8?B?SFR3MnBXYVBCVVF0d09xb2laeS9yS3g3bXlNKzl4NHFtK1N0OHo0RHJ4cnY5?=
 =?utf-8?B?MTFpaDIrWERhTDg4QUoxSGp4QjAxN2lGaVFJUm84QUNTdndwSG1JYStPdDBB?=
 =?utf-8?Q?l6Tbb1vJwB5rHFSmuSBbCu1P2?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 142e186b-d7a0-4827-e1e0-08da73eefffb
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Aug 2022 18:52:36.5453
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: 7zkvUQvFQbPYW3L9yxOZSbWYxYfOyxCyrr0RFEy1yYN/J+OD/W+TtWhoXg0yRP5AGgKw2QrIX3i2QMT3Yy8Ynw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR10MB4808
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.883,Hydra:6.0.517,FMLib:17.11.122.1
 definitions=2022-08-01_08,2022-08-01_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 suspectscore=0
 adultscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 phishscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2206140000
 definitions=main-2208010094
X-Proofpoint-GUID: nBEunUvbDqhV1vptliDENjwxDHPSTgt1
X-Proofpoint-ORIG-GUID: nBEunUvbDqhV1vptliDENjwxDHPSTgt1
X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org


On 8/1/22 4:47 AM, Jeff Layton wrote:
> On Sun, 2022-07-31 at 13:19 -0700, Dai Ngo wrote:
>> Use-after-free occurred when the laundromat tried to free expired
>> cpntf_state entry on the s2s_cp_stateids list after inter-server
>> copy completed. The sc_cp_list that the expired copy state was
>> inserted on was already freed.
>>
>> When COPY completes, the Linux client normally sends LOCKU(lock_state x),
>> FREE_STATEID(lock_state x) and CLOSE(open_state y) to the source server.
>> The nfs4_put_stid call from nfsd4_free_stateid cleans up the copy state
>> from the s2s_cp_stateids list before freeing the lock state's stid.
>>
>> However, sometimes the CLOSE was sent before the FREE_STATEID request.
>> When this happens, the nfsd4_close_open_stateid call from nfsd4_close
>> frees all lock states on its st_locks list without cleaning up the copy
>> state on the sc_cp_list list. When the time the FREE_STATEID arrives the
>> server returns BAD_STATEID since the lock state was freed. This causes
>> the use-after-free error to occur when the laundromat tries to free
>> the expired cpntf_state.
>>
>> This patch adds a call to nfs4_free_cpntf_statelist in
>> nfsd4_close_open_stateid to clean up the copy state before calling
>> free_ol_stateid_reaplist to free the lock state's stid on the reaplist.
>>
>> Signed-off-by: Dai Ngo <dai.ngo@oracle.com>
>> ---
>>   fs/nfsd/nfs4state.c | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
>> index 9409a0dc1b76..749f51dff5c7 100644
>> --- a/fs/nfsd/nfs4state.c
>> +++ b/fs/nfsd/nfs4state.c
>> @@ -6608,6 +6608,7 @@ static void nfsd4_close_open_stateid(struct nfs4_ol_stateid *s)
>>   	struct nfs4_client *clp = s->st_stid.sc_client;
>>   	bool unhashed;
>>   	LIST_HEAD(reaplist);
>> +	struct nfs4_ol_stateid *stp;
>>   
>>   	spin_lock(&clp->cl_lock);
>>   	unhashed = unhash_open_stateid(s, &reaplist);
>> @@ -6616,6 +6617,8 @@ static void nfsd4_close_open_stateid(struct nfs4_ol_stateid *s)
>>   		if (unhashed)
>>   			put_ol_stateid_locked(s, &reaplist);
>>   		spin_unlock(&clp->cl_lock);
>> +		list_for_each_entry(stp, &reaplist, st_locks)
>> +			nfs4_free_cpntf_statelist(clp->net, &stp->st_stid);
>>   		free_ol_stateid_reaplist(&reaplist);
>>   	} else {
>>   		spin_unlock(&clp->cl_lock);
> Nice catch.
>
> There are a number of places that call free_ol_stateid_reaplist. Is it
> really only in nfsd4_close_open_stateid that we need to do this? I
> wonder if it would be better to do this inside free_ol_stateid_reaplist
> instead so that all of those callers clean up the copy states as well?

Yes, we can do this in free_ol_stateid_reaplist too, I tested it.

The linux client uses either delegation state or lock state to send with
the COPY_NOTIFY to the source server. If the server grants the delegation
in the OPEN then the client uses the delegation state, otherwise it sends
the LOCK to the source and uses the lock state for the COPY_NOTIFY. This
problem happens only when the lock state is used *and* the client sends
the CLOSE and FREE_STATEID out of order.

free_ol_stateid_reaplist is called from release_open_stateid, release_openowner,
nfsd4_close_open_stateid and nfsd4_release_lockowner. Among these functions,
only nfsd4_close_open_stateid deals with lock state that may have cpntf_state
associated with it and only for the minorversion > 1 case.

nfsd4_release_lockowner will free the lock states but if the client has
not send LOCKU yet then put_ol_stateid_locked would fail to add the lock
state on the reaplist.

I'm ok to move it to free_ol_stateid_reaplist if you still think we should
and don't mind a little overhead on the unneeded cases.

-Dai