Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <1557e8bb-1739-29a4-a04c-c62732cfc7c4@oracle.com>
Date:   Fri, 9 Sep 2022 11:12:22 -0700
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0)
 Gecko/20100101 Thunderbird/91.13.0
Subject: Re: [PATCH v2] NFSD: fix use-after-free on source server when doing
 inter-server copy
Content-Language: en-US
To:     Chuck Lever III <chuck.lever@oracle.com>
Cc:     Olga Kornievskaia <olga.kornievskaia@gmail.com>,
        Jeff Layton <jlayton@kernel.org>,
        Linux NFS Mailing List <linux-nfs@vger.kernel.org>
References: <1659405154-21910-1-git-send-email-dai.ngo@oracle.com>
 <E146112C-33EE-4143-92F7-7515638301D7@oracle.com>
From:   dai.ngo@oracle.com
In-Reply-To: <E146112C-33EE-4143-92F7-7515638301D7@oracle.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?ZEYyTFZZc0FHV1dTQ2FFY3A0U2lCR1ZibGlYVEhUYzlJQW1hcFQyQ1VTMEM3?=
 =?utf-8?B?aUxIYlM0S0VZd0lpV0h3RElLSUw0dERvQk9ucWlPRkE0SGZGRk41czgxK3BJ?=
 =?utf-8?B?YjJqa0hLSFRyQmw3Uk1jZG5naXBlRWZUVWVMVlJsS3hFSHBYTVJDbEQ1dXpQ?=
 =?utf-8?B?UjA5TXpSYmtHQlZDaHdmUTlGWUdMU20rSUhUakRhVmYrNHdicXpCamFXSm5T?=
 =?utf-8?B?cDV4bFVsc2JHNFpNd1Y1NGYxV1NnMlVtUEVmL1hxTWkwU0pZTkRKTnVhcTVm?=
 =?utf-8?B?dXBjd0htVTVRTFRNUERKN0ZLaHl5TzhBNEhLSHJqMHVxVUsxaVlhNXNFSzRC?=
 =?utf-8?B?dDFzWGNOQU54Vmp2QzAxWjlqTThCSG4rdWdJRUZ0TzVTQ0NTYzRubEpQaU1r?=
 =?utf-8?B?a3dUNVdTOVQreDgxbUVkeHRTVVMxY0k3ZWlGekVQcm14R3pEZy95ZysxaXpm?=
 =?utf-8?B?RmlXSzNITW5FZkFqK1FZbEFZSDVoclZ4TzlyVDFGOEMxUlNSYTlTeThhUFJQ?=
 =?utf-8?B?KzZkRkdOaVRiNTloSk9ST2FKRDVJUWEzK2dNOGdqaVR2bFNIL1R6M3RudURE?=
 =?utf-8?B?YlMwZFVYaVAyUS92U29xNjVXMHEzSzY1ZFFUQ1VSclpOYk4vb0ZBblBvUUh1?=
 =?utf-8?B?eDVockR5S0NnR3hrUzhMclRMeC9zeVMyWDJ1VmtOcUgvUm5mT0l2SDRvNzlu?=
 =?utf-8?B?RktZakNnZzJjcFdPRElvVURVcXdJenozWU5LZ0wvZHhEUTFQV0oyMEYxSkZj?=
 =?utf-8?B?Mkl5UU9RNTBKYkVRUGxOaHVoNzlqRzlpUlVDeU1UODJQa3pRdVdLb0RGUVl4?=
 =?utf-8?B?a25WSURoM244K2haMitaNlBKVlJSa1ovS2E3cTdoQklHSytTRlVVMmM4Vy91?=
 =?utf-8?B?dE1Jb09sVVBtaWtOZ21OWldHQVQ5WGVFZ3M3U09ORHp1OFNRa0xqWW4rZjJG?=
 =?utf-8?B?akY5Y0pkYjR3T0FFakNqeXRWNUwvZjVLandONjhZTDNqQ0FFREVPV3RGUFEv?=
 =?utf-8?B?WlB1V1htUjN0YXJkYlB1b3FWNVRSTU4waUZLcTZ5SzFrSHBWclpQbWxDdFM5?=
 =?utf-8?B?aGVZMTJXclhibTV0cXRSQjczbUkyOGhLVG14MGJGd2VmRWxSdk9aV3lEeGxC?=
 =?utf-8?B?VVpPRnl4WjcrdUsrQWM0R2FvT3I2MkdjM3VkQkhRSzlqbzJjY0JmRjN5aFV2?=
 =?utf-8?B?clRvZER1LzdFZ3ZhbElLQ2h5SGROZFJYc0FlQlBxamNBcHRuQm9JcmR1QzMz?=
 =?utf-8?B?VG5kelJBdExzNHI5bVAybjlNQmw4MGZGMjNOc0VEUTVwMis0ajJPZ253OHNi?=
 =?utf-8?B?TnUrdzJDaGREZ3Bhd3ZxVFJlYmlySVRYb2NrWlBGSUl6K01YeG9IR3RvRW9L?=
 =?utf-8?B?WnFXcjhDTHpoY1RIZzNNWktad1hyaXBHYWtNYTExMU56YXRNRlZwZGJtNTNZ?=
 =?utf-8?B?OU42dDhETG5NdTJZeS9iL01SVGo0ZFJ5Z0x3VFJ1UnZwSFJwWTg0ZzlKcElI?=
 =?utf-8?B?SE5HTDVHY0F5NVArQ3hWVlQvUnR5b0xhK1pJVVFtUGV0S1hMUFEwQldUb1B6?=
 =?utf-8?B?ang0SGwyREdTMHpqY292S1lIZC9lVmF0UWdta3kvM29RR200YXNhWkdCdmxR?=
 =?utf-8?B?aHdJd2xRTmJmMzNYaDBtL0UwUkg4VVFmRnJiV2VXTkV1aGxFQk5nNDR4Skdo?=
 =?utf-8?B?Yk53UjVjYVBaVlVMZ2liZzBMM29xUjRsVUFBWjdQaTM2K29taU1XZmtmVmdS?=
 =?utf-8?B?cTNuZHBKNCtoMjRLZ3NtamtHUWdGaGFKNlZhTkZMQUhzYSt1ODBla2t2RjQ2?=
 =?utf-8?B?RWdSQWVXcDlXVzNEemJHWnBHaUxaZnFqVTd2TmQyelFNM2dwOE5PVzlyOWdQ?=
 =?utf-8?B?dng0UkVRWkNlSXNVOFQ1dVVUWHg1OGRTdEM2K0dnMVkxazM0a0I1ZU5UVkow?=
 =?utf-8?B?YU5aVjJaZm9MTFdMTGU1UXIva0lvVTY3TExrMExudHYwWC9JWCtUZUxhL3ZW?=
 =?utf-8?B?Q0lxeVhzRG5oMStOUTBSZHZFbDhyeTBhNkxQWFNpejQyWk5ydTVlZnBhYWtQ?=
 =?utf-8?B?djdONVFLeThxYS9hOGJtS3VxVm9hZGt2bTNHL0pubERSWkhZcDI5T1ZUTmhR?=
 =?utf-8?Q?U9eTN7xsUTXcN0kxBUV9DFkt/?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 27815bfb-73c8-43bc-64cd-08da928ed958
X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4257.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Sep 2022 18:12:25.8618
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: pNUmW+S7mSN3oOd58R3g5x3CTQX4rhqjfWGtwhWRU46hXNdTuqsezYNyVR3zO1bcfbSv53mgdYfC+1lbSnZGGQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR10MB5292
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1
 definitions=2022-09-09_08,2022-09-09_01,2022-06-22_01
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 malwarescore=0 mlxscore=0
 suspectscore=0 mlxlogscore=999 phishscore=0 adultscore=0 bulkscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2207270000
 definitions=main-2209090064
X-Proofpoint-GUID: u-uS5yG77Gyx6o8GaMBLE2-c_6huQa81
X-Proofpoint-ORIG-GUID: u-uS5yG77Gyx6o8GaMBLE2-c_6huQa81
X-Spam-Status: No, score=-4.9 required=5.0 tests=BAYES_00,DKIM_SIGNED,
        DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,NICE_REPLY_A,RCVD_IN_DNSWL_LOW,
        RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE
        autolearn=ham autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-nfs.vger.kernel.org>
X-Mailing-List: linux-nfs@vger.kernel.org

Hi Chuck,

Is there anything else you want me to do with this patch?

Thanks,
-Dai

On 8/2/22 7:35 AM, Chuck Lever III wrote:
>
>> On Aug 1, 2022, at 9:52 PM, Dai Ngo <dai.ngo@oracle.com> wrote:
>>
>> Use-after-free occurred when the laundromat tried to free expired
>> cpntf_state entry on the s2s_cp_stateids list after inter-server
>> copy completed. The sc_cp_list that the expired copy state was
>> inserted on was already freed.
>>
>> When COPY completes, the Linux client normally sends LOCKU(lock_state x),
>> FREE_STATEID(lock_state x) and CLOSE(open_state y) to the source server.
>> The nfs4_put_stid call from nfsd4_free_stateid cleans up the copy state
>> from the s2s_cp_stateids list before freeing the lock state's stid.
>>
>> However, sometimes the CLOSE was sent before the FREE_STATEID request.
>> When this happens, the nfsd4_close_open_stateid call from nfsd4_close
>> frees all lock states on its st_locks list without cleaning up the copy
>> state on the sc_cp_list list. When the time the FREE_STATEID arrives the
>> server returns BAD_STATEID since the lock state was freed. This causes
>> the use-after-free error to occur when the laundromat tries to free
>> the expired cpntf_state.
>>
>> This patch adds a call to nfs4_free_cpntf_statelist in
>> nfsd4_close_open_stateid to clean up the copy state before calling
>> free_ol_stateid_reaplist to free the lock state's stid on the reaplist.
>>
>> Signed-off-by: Dai Ngo <dai.ngo@oracle.com>
> Applied to my private tree for testing.
> I added "Cc: <stable@vger.kernel.org> # 5.6+".
>
>
>> ---
>> fs/nfsd/nfs4state.c | 7 +++++++
>> 1 file changed, 7 insertions(+)
>>
>> diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
>> index 9409a0dc1b76..b99c545f93e4 100644
>> --- a/fs/nfsd/nfs4state.c
>> +++ b/fs/nfsd/nfs4state.c
>> @@ -1049,6 +1049,9 @@ static struct nfs4_ol_stateid * nfs4_alloc_open_stateid(struct nfs4_client *clp)
>>
>> static void nfs4_free_deleg(struct nfs4_stid *stid)
>> {
>> +	struct nfs4_ol_stateid *stp = openlockstateid(stid);
>> +
>> +	WARN_ON(!list_empty(&stp->st_stid.sc_cp_list));
>> 	kmem_cache_free(deleg_slab, stid);
>> 	atomic_long_dec(&num_delegations);
>> }
>> @@ -1463,6 +1466,7 @@ static void nfs4_free_ol_stateid(struct nfs4_stid *stid)
>> 	release_all_access(stp);
>> 	if (stp->st_stateowner)
>> 		nfs4_put_stateowner(stp->st_stateowner);
>> +	WARN_ON(!list_empty(&stp->st_stid.sc_cp_list));
>> 	kmem_cache_free(stateid_slab, stid);
>> }
>>
>> @@ -6608,6 +6612,7 @@ static void nfsd4_close_open_stateid(struct nfs4_ol_stateid *s)
>> 	struct nfs4_client *clp = s->st_stid.sc_client;
>> 	bool unhashed;
>> 	LIST_HEAD(reaplist);
>> +	struct nfs4_ol_stateid *stp;
>>
>> 	spin_lock(&clp->cl_lock);
>> 	unhashed = unhash_open_stateid(s, &reaplist);
>> @@ -6616,6 +6621,8 @@ static void nfsd4_close_open_stateid(struct nfs4_ol_stateid *s)
>> 		if (unhashed)
>> 			put_ol_stateid_locked(s, &reaplist);
>> 		spin_unlock(&clp->cl_lock);
>> +		list_for_each_entry(stp, &reaplist, st_locks)
>> +			nfs4_free_cpntf_statelist(clp->net, &stp->st_stid);
>> 		free_ol_stateid_reaplist(&reaplist);
>> 	} else {
>> 		spin_unlock(&clp->cl_lock);
>> -- 
>> 2.9.5
>>
> --
> Chuck Lever
>
>
>