Received: by 2002:a05:6a10:1d13:0:0:0:0 with SMTP id pp19csp1551077pxb; Tue, 17 Aug 2021 14:52:24 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyAt+5EfD81EbWxDpSflG6vZ/uv/R36KWaHVnYBCxp7/SXUKmNPPjtxvrbK+lSLFZcE5yUm X-Received: by 2002:a05:6e02:1c27:: with SMTP id m7mr3932825ilh.190.1629237143768; Tue, 17 Aug 2021 14:52:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1629237143; cv=none; d=google.com; s=arc-20160816; b=AREb+qBFse+jDMJgtR9JYS4bf5x2EEFNlu7YaQw/1sfp0sNF/3JHN17QJf8f/rGTby Tnq8IW6BSn/EYY3CelMdcxLYYl8tMMng+cHxQ7BZSf+uMv0jSu3cHZ9+0QKndLzcEjMj CZPj+Re9xbPo5C3gpweAeLGVulul+P4wtjwkxoYzmQbgk0Za/4vq5rREUB4ROhqQxYZm 838wBWSJx8qtPzHF4gxzr0nJ2THcbJ0v9ztH0Db2a3kGjYZmCWH3Qp1qPkXYw1nieqjf JHmxziLnBFInxSx8fVNmDeuctotCeyY2BXY0yDANrivTTUPrxXopqy/7y5CiU2GR/we9 kfRA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:mime-version:user-agent:date :message-id:from:references:cc:to:subject:dkim-signature; bh=wLW7blM5n2kW36fyMTYaD5wvUUW3O2T7OO/lcOi9dJg=; b=qE0umbbN/nL85ILff2JojLH0II9FHvvuQf5xoSnBQF3C1TkCBrHSBVHS+r99UEWRJQ NCj5m9RIK+x8a68S0afMAbgifARp8070r3l3n9FfCuN/qH3BRoUpmcF7sA9CE6zfvLIH 1u7DMMQdgCVgNXOKjkReu0eLyzuDFNyIdUNC6E3ivh7+GEyLvhI5PrFVZagkmw1FobPu QMdqBMgPX2H07keHyde4gKmScg3/FQIdN+awuMPO6kqpc5kq9LBaMJRjhg+4vWRkvRtx SyPg66tx6lae+l4RdQskKED3+sw7o6dm29r+/d0grqqBuz4GEKohIbXu9K4sRukGyN8y bInQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@rothenpieler.org header.s=mail header.b=MwJQDmGN; spf=pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id d4si3704165jak.4.2021.08.17.14.52.01; Tue, 17 Aug 2021 14:52:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@rothenpieler.org header.s=mail header.b=MwJQDmGN; spf=pass (google.com: domain of linux-nfs-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-nfs-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=rothenpieler.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232589AbhHQVwd (ORCPT + 99 others); Tue, 17 Aug 2021 17:52:33 -0400 Received: from btbn.de ([136.243.74.85]:38230 "EHLO btbn.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232564AbhHQVwd (ORCPT ); Tue, 17 Aug 2021 17:52:33 -0400 Received: from [authenticated] by btbn.de (Postfix) with ESMTPSA id C9DEF343955; Tue, 17 Aug 2021 23:51:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rothenpieler.org; s=mail; t=1629237118; bh=wLW7blM5n2kW36fyMTYaD5wvUUW3O2T7OO/lcOi9dJg=; h=Subject:To:Cc:References:From:Date:In-Reply-To; b=MwJQDmGNPX6rBXLrjrlbq+dZs1HvMfrOpuIqX367MyabiAGBag8JkrUS8agALG1ta /xv8/KS2ZYqqAnEYoef2tGWxqgX9LO3EuKBznHimoIpAqKK4ZkqW8klTSzx1EE56Dt Uu87nsFpCRXry8KEZeLO3CBhMpvok1X11tKO1b8k/1L704cs+xhR9Js/UrEoRevByv QoUR7YC7SUNR4SmuJITE/nZBPd61OInYtKO9hhHdJMrMMJd22lmywePEhGqV0vEHT0 Tf30p1Gzgljm+63xQlRkuGaxVnsjXcIDXcn6SG2aZjoud8/I6SF0Q0c65nBQwXpKLf xNOzp8YV7lLmw== Subject: Re: Spurious instability with NFSoRDMA under moderate load To: Chuck Lever III Cc: Linux NFS Mailing List , Olga Kornievskaia , Dai Ngo References: <4da3b074-a6be-d83f-ccd4-b151557066aa@rothenpieler.org> <72ECF9E1-1F6E-44AF-850C-536BED898DDD@oracle.com> <9355de20-921c-69e0-e5a4-733b64e125e1@rothenpieler.org> <4BA2A532-9063-4893-AF53-E1DAB06095CC@oracle.com> <141fdf51-2aa1-6614-fe4e-96f168cbe6cf@rothenpieler.org> <99DFF0B0-FE0F-4416-B3F6-1F9535884F39@oracle.com> <64F9A492-44B9-4057-ABA5-C8202828A8DD@oracle.com> <1b8a24a9-5dba-3faf-8b0a-16e728a6051c@rothenpieler.org> <5DD80ADC-0A4B-4D95-8CF7-29096439DE9D@oracle.com> <0444ca5c-e8b6-1d80-d8a5-8469daa74970@rothenpieler.org> <3AF4F6CA-8B17-4AE9-82E2-21A2B9AA0774@oracle.com> <4caff277-8e53-3c75-70c1-8938b2a26933@rothenpieler.org> <7B44D7FD-9D0F-4A0D-ABE9-E295072D953F@oracle.com> From: Timo Rothenpieler Message-ID: <97fe445e-6c4f-b7cf-fa39-fd0c4222a89e@rothenpieler.org> Date: Tue, 17 Aug 2021 23:51:57 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 MIME-Version: 1.0 In-Reply-To: <7B44D7FD-9D0F-4A0D-ABE9-E295072D953F@oracle.com> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="------------ms040601040403010605080000" Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org This is a cryptographically signed message in MIME format. --------------ms040601040403010605080000 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 17.08.2021 23:08, Chuck Lever III wrote: > I tried reproducing this using your 'xfs_io -fc "copy_range > testfile" testfile.copy' reproducer, but couldn't. >=20 > A network capture shows that the client tries CLONE first. The > server reports that's not supported, so the client tries COPY. > The COPY works, and the reply shows that the COPY was synchro- > nous. Thus there's no need for a callback, and I'm not tripping > over backchannel misbehavior. Make sure the testfile is of sufficient size. I'm not sure what the=20 threshold is, but if it's too small, it'll just do a synchronous copy=20 for me as well. I'm using a 50MB file in my tests. > The export I'm using is an xfs filesystem. Did you already > report the filesystem type you're testing against? I can't > find it in the thread. >=20 > If there's a way to force an offload-style COPY, let me know. >=20 > Oh. Also I looked at what might have been pulled into the > linux-5.12.y kernel between .12 and .19, and I don't see > anything that's especially relevant to either COPY_OFFLOAD > or backchannel. I'm observing this with both an ext4 and zfs filesystem. Can easily test xfs as well if desired. Are you testing this on a normal network, or with RDMA? With normal tcp, = I also can't observe this issue(it doesn't time out the backchannel in=20 the first place), it only happens in RDMA mode. I'm using Mellanox ConnectX-4 cards in IB mode for my tests. --------------ms040601040403010605080000 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCC DVkwggXkMIIDzKADAgECAhAI/yx7V5dPIG8WuMetnzcsMA0GCSqGSIb3DQEBCwUAMIGBMQsw CQYDVQQGEwJJVDEQMA4GA1UECAwHQmVyZ2FtbzEZMBcGA1UEBwwQUG9udGUgU2FuIFBpZXRy bzEXMBUGA1UECgwOQWN0YWxpcyBTLnAuQS4xLDAqBgNVBAMMI0FjdGFsaXMgQ2xpZW50IEF1 dGhlbnRpY2F0aW9uIENBIEczMB4XDTIxMDIxNDE5MTM0N1oXDTIyMDIxNDE5MTM0N1owIDEe MBwGA1UEAwwVdGltb0Byb3RoZW5waWVsZXIub3JnMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A MIIBCgKCAQEA0WP2SBuRIpVw5O7QPakKoJjg7B4UNAKTyky1XMsievLNGnR4Nxe6kKU+1oW0 oF5FqMVH9NkT9zhWYJzr5sNwJMKb9t5k8kYC7GXzOM9PxVx3bkLF5bWZrbfelUUwcdiyEYoh d29C+PxiNLHvmayWb3NtxpWiax9A4x7dRhhtqB/0BkPix+ZsIFn8vxpCvIChE2YlQWK3i8UX uBtqm26zBl3BIjj+bpd+7ePVt60vRx/R3LFHtF6kL/gQvgRcm8CFc8Nj3dCUeR2lfG+DzoTY ED6yAi838kRh5JHbqIl/Fo9YRwOYUaq2TFT/fGue87d7duLbckX1aVot+OqE0aeV2QIDAQAB o4IBtjCCAbIwDAYDVR0TAQH/BAIwADAfBgNVHSMEGDAWgBS+l6mqhL+AvxBTfQky+eEuMhvP dzB+BggrBgEFBQcBAQRyMHAwOwYIKwYBBQUHMAKGL2h0dHA6Ly9jYWNlcnQuYWN0YWxpcy5p dC9jZXJ0cy9hY3RhbGlzLWF1dGNsaWczMDEGCCsGAQUFBzABhiVodHRwOi8vb2NzcDA5LmFj dGFsaXMuaXQvVkEvQVVUSENMLUczMCAGA1UdEQQZMBeBFXRpbW9Acm90aGVucGllbGVyLm9y ZzBHBgNVHSAEQDA+MDwGBiuBHwEYATAyMDAGCCsGAQUFBwIBFiRodHRwczovL3d3dy5hY3Rh bGlzLml0L2FyZWEtZG93bmxvYWQwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMEgG A1UdHwRBMD8wPaA7oDmGN2h0dHA6Ly9jcmwwOS5hY3RhbGlzLml0L1JlcG9zaXRvcnkvQVVU SENMLUczL2dldExhc3RDUkwwHQYDVR0OBBYEFK/aNb0BTZd0BqHgSJnmTftGSlabMA4GA1Ud DwEB/wQEAwIFoDANBgkqhkiG9w0BAQsFAAOCAgEAT3W2bBaISi7Utg/WA3U+bBhiouolnROR AB0vW4m3igjMcWx5GrPb8CSWNcq0/+BG+bhj6s+q7D1E9h1HO9CZUCfD7ujXj/VT/h7oMAqX w3Tf6H92bvHmZCvZmb2HKEnAAa4URjeZyNI1uwsMirF/gC5zYX5pm2ydVGxGYusWq8VRZzgc m1a0f3SPtX2dmmqjCzfINsQPs3N7BQo6FO/PfCbCzt22e+9Zm0Lra0Wt2URFTYCKSTjsK2xC SkysTfVIrBZCOb83oTMsgYE9dBmK7Tmob/HzHKs0NUOu4TfEpCgFgoXozMqTLFQac7aW26YK O8ClFDaauyOC71A+kjrth/gkUNEK+Cd3W52hK2FWvxbG/8LQLDMYviZFKxv/LAHU0fb6omva R4dzu9Sagi1z5uI5KHs5SR85lH4Up0dYs+I2xyFb8wZVYa+VuvsJ4W/pL2OaMm0tez+aNprg XURytCSPfAlz3JQdEYIiKPlJrz7O6eL2j7RwxMcKFLQl117mhImjdauIjaaS60w92P7v+F7+ 7INJ8g0PFN2vHVCB9e1g4iSYIgiydDLcbs73Jp1yVp97plWZI9oirxvH1/vI05FUJ3gw9qg2 WfbttAr0AEakAUo3Dv8jB7aQor/5fu8NMOvWjFV7P7GTAgrwil8u6fXa8ae/kWzG/850vgqq GM0wggdtMIIFVaADAgECAhAXED7ePYoctcoGUZPnykNrMA0GCSqGSIb3DQEBCwUAMGsxCzAJ BgNVBAYTAklUMQ4wDAYDVQQHDAVNaWxhbjEjMCEGA1UECgwaQWN0YWxpcyBTLnAuQS4vMDMz NTg1MjA5NjcxJzAlBgNVBAMMHkFjdGFsaXMgQXV0aGVudGljYXRpb24gUm9vdCBDQTAeFw0y MDA3MDYwODQ1NDdaFw0zMDA5MjIxMTIyMDJaMIGBMQswCQYDVQQGEwJJVDEQMA4GA1UECAwH QmVyZ2FtbzEZMBcGA1UEBwwQUG9udGUgU2FuIFBpZXRybzEXMBUGA1UECgwOQWN0YWxpcyBT LnAuQS4xLDAqBgNVBAMMI0FjdGFsaXMgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIENBIEczMIIC IjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA7eaHlqHBpLbtwkJV9z8PDyJgXxPgpkOI hkmReRwbLxpQD9xGAe72ujqGzFFh78QPgAhxKVqtGHzYeq0VJVCzhnCKRBbVX+JwIhL3ULYh UAZrViUp952qDB6qTL5sGeJS9F69VPSR5k6pFNw7mHDTTt0voWFg2aVkG3khomzVXoieJGOi Q4dH76paCtQbLkt59joAKz2BnwGLQ4wr09nfumJt5AKx2YxHK2XgSPslVZ4z8G00gimsfA7U tjT/wiekY6Z0b7ksLrEcvODncHQe9VSrNRA149SE3AlkWaZM/joVei/GYfj9K5jkiReinR4m qM353FEceLOeBhSTURpMdQ5wsXLi9DSTGBuNv4aw2Dozb/qBlkhGTvwk92mi0jAecE22Sn3A 9UfrU2p1w/uRs+TIteQ0xO0B/J2mY2caqocsS9SsriIGlQ8b0LT0o6Ob07KGtPa5/lIvMmx5 72Dv2v+vDiECByxm1Hdgjp8JtE4mdyYP6GBscJyT71NZw1zXHnFkyCbxReag9qaSR9x4CVVX j1BDmNROCqd5NAfIXUXYTFeZ/jukQigkxXGWhEhfLBC4Ha6pwizz9fq1+wwPKcWaF9P/SZOu BDrG30MiyCZa66G9mEtF5ZLuh4rGfKqxy4Z5Mxecuzt+MZmrSKfKGeXOeED/iuX5Z02M1o7i MS8CAwEAAaOCAfQwggHwMA8GA1UdEwEB/wQFMAMBAf8wHwYDVR0jBBgwFoAUUtiIOsifeGbt ifN7OHCUyQICNtAwQQYIKwYBBQUHAQEENTAzMDEGCCsGAQUFBzABhiVodHRwOi8vb2NzcDA1 LmFjdGFsaXMuaXQvVkEvQVVUSC1ST09UMEUGA1UdIAQ+MDwwOgYEVR0gADAyMDAGCCsGAQUF BwIBFiRodHRwczovL3d3dy5hY3RhbGlzLml0L2FyZWEtZG93bmxvYWQwHQYDVR0lBBYwFAYI KwYBBQUHAwIGCCsGAQUFBwMEMIHjBgNVHR8EgdswgdgwgZaggZOggZCGgY1sZGFwOi8vbGRh cDA1LmFjdGFsaXMuaXQvY24lM2RBY3RhbGlzJTIwQXV0aGVudGljYXRpb24lMjBSb290JTIw Q0EsbyUzZEFjdGFsaXMlMjBTLnAuQS4lMmYwMzM1ODUyMDk2NyxjJTNkSVQ/Y2VydGlmaWNh dGVSZXZvY2F0aW9uTGlzdDtiaW5hcnkwPaA7oDmGN2h0dHA6Ly9jcmwwNS5hY3RhbGlzLml0 L1JlcG9zaXRvcnkvQVVUSC1ST09UL2dldExhc3RDUkwwHQYDVR0OBBYEFL6XqaqEv4C/EFN9 CTL54S4yG893MA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAgEAJpvnG1kNdLMS A+nnVfeEgIXNQsM7YRxXx6bmEt9IIrFlH1qYKeNw4NV8xtop91Rle168wghmYeCTP10FqfuK MZsleNkI8/b3PBkZLIKOl9p2Dmz2Gc0I3WvcMbAgd/IuBtx998PJX/bBb5dMZuGV2drNmxfz 3ar6ytGYLxedfjKCD55Yv8CQcN6e9sW5OUm9TJ3kjt7Wdvd1hcw5s+7bhlND38rWFJBuzump 5xqm1NSOggOkFSlKnhSz6HUjgwBaid6Ypig9L1/TLrkmtEIpx+wpIj7WTA9JqcMMyLJ0rN6j jpetLSGUDk3NCOpQntSy4a8+0O+SepzS/Tec1cGdSN6Ni2/A7ewQNd1Rbmb2SM2qVBlfN0e6 ZklWo9QYpNZyf0d/d3upsKabE9eNCg1S4eDnp8sJqdlaQQ7hI/UYCAgDtLIm7/J9+/S2zuwE WtJMPcvaYIBczdjwF9uW+8NJ/Zu/JKb98971uua7OsJexPFRBzX7/PnJ2/NXcTdwudShJc/p d9c3IRU7qw+RxRKchIczv3zEuQJMHkSSM8KM8TbOzi/0v0lU6SSyS9bpGdZZxx19Hd8Qs0cv +R6nyt7ohttizwefkYzQ6GzwIwM9gSjH5Bf/r9Kc5/JqqpKKUGicxAGy2zKYEGB0Qo761Mcc IyclBW9mfuNFDbTBeDEyu80xggPzMIID7wIBATCBljCBgTELMAkGA1UEBhMCSVQxEDAOBgNV BAgMB0JlcmdhbW8xGTAXBgNVBAcMEFBvbnRlIFNhbiBQaWV0cm8xFzAVBgNVBAoMDkFjdGFs aXMgUy5wLkEuMSwwKgYDVQQDDCNBY3RhbGlzIENsaWVudCBBdXRoZW50aWNhdGlvbiBDQSBH MwIQCP8se1eXTyBvFrjHrZ83LDANBglghkgBZQMEAgEFAKCCAi0wGAYJKoZIhvcNAQkDMQsG CSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMjEwODE3MjE1MTU4WjAvBgkqhkiG9w0BCQQx IgQgelsr5LzeBSoLeBiqK//7ONjKP+kJvbyws1oL8Ubg++YwbAYJKoZIhvcNAQkPMV8wXTAL BglghkgBZQMEASowCwYJYIZIAWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDAN BggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDCBpwYJKwYBBAGCNxAEMYGZ MIGWMIGBMQswCQYDVQQGEwJJVDEQMA4GA1UECAwHQmVyZ2FtbzEZMBcGA1UEBwwQUG9udGUg U2FuIFBpZXRybzEXMBUGA1UECgwOQWN0YWxpcyBTLnAuQS4xLDAqBgNVBAMMI0FjdGFsaXMg Q2xpZW50IEF1dGhlbnRpY2F0aW9uIENBIEczAhAI/yx7V5dPIG8WuMetnzcsMIGpBgsqhkiG 9w0BCRACCzGBmaCBljCBgTELMAkGA1UEBhMCSVQxEDAOBgNVBAgMB0JlcmdhbW8xGTAXBgNV BAcMEFBvbnRlIFNhbiBQaWV0cm8xFzAVBgNVBAoMDkFjdGFsaXMgUy5wLkEuMSwwKgYDVQQD DCNBY3RhbGlzIENsaWVudCBBdXRoZW50aWNhdGlvbiBDQSBHMwIQCP8se1eXTyBvFrjHrZ83 LDANBgkqhkiG9w0BAQEFAASCAQCfmE9GOqtG0oLTcnA05Qv1yh1FrwCUJa/c1exvp1UAKGVC mX0PIbkF9XDd6kvLDO8+QCSQyTX0WL/5vOvt/56KQ3Umv41ZwlfQz2pcK476z5vag0JC5K+H VIB4Z2ConE6Yuk5WIzVnzQpuEtbyAnLVScEaRLjdaqbPAVM6hTai0TtzS9u180nGRZDNRHj6 Cdt68eu9OTBn8wDeS9sBWnZQZkS0Yv8FrzGG2DHQpAgyDqbGKU6SWhYIj8PxJIFrxx1uBFXH HljTVromLU+OQwqnFDeQEH9oQpD9nBcbFTk+hdlyXD5/66bup7daGQ4PnTOjpZ7SG2acYGxY ddWCcSJGAAAAAAAA --------------ms040601040403010605080000--