Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Jan Glauber <jglauber@marvell.com>
To:     "catalin.marinas@arm.com" <catalin.marinas@arm.com>,
        "will.deacon@arm.com" <will.deacon@arm.com>
CC:     "linux-arm-kernel@lists.infradead.org" 
        <linux-arm-kernel@lists.infradead.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Jayachandran Chandrasekharan Nair <jnair@marvell.com>
Subject: [RFC] Disable lockref on arm64
Thread-Topic: [RFC] Disable lockref on arm64
Thread-Index: AQHU/psgIPX1WiXk3kG5688bmgHV9A==
Date:   Mon, 29 Apr 2019 14:52:11 +0000
Message-ID: <20190429145159.GA29076@hc>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-messagesentrepresentingtype: 1
x-originating-ip: [46.223.67.200]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: 9e1e54d5-1717-4bf1-9c83-08d6ccb2428f
x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600141)(711020)(4605104)(2017052603328)(7193020);SRVR:DM5PR18MB2311;
x-ms-traffictypediagnostic: DM5PR18MB2311:
x-microsoft-antispam-prvs: <DM5PR18MB2311FDFCB23C12E5622BB7D3D8390@DM5PR18MB2311.namprd18.prod.outlook.com>
x-ms-oob-tlc-oobclassifiers: OLM:9508;
x-forefront-prvs: 0022134A87
x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(7916004)(366004)(39860400002)(396003)(136003)(376002)(346002)(189003)(199004)(305945005)(97736004)(7736002)(73956011)(102836004)(6512007)(9686003)(6506007)(386003)(256004)(14444005)(8676002)(26005)(81156014)(81166006)(2501003)(8936002)(33716001)(486006)(476003)(6486002)(66066001)(186003)(6436002)(33656002)(71200400001)(71190400001)(107886003)(66476007)(66556008)(68736007)(66946007)(64756008)(66446008)(4326008)(25786009)(54906003)(316002)(110136005)(52116002)(99286004)(14454004)(5660300002)(2906002)(3846002)(6116002)(478600001)(1076003)(86362001)(53936002);DIR:OUT;SFP:1101;SCL:1;SRVR:DM5PR18MB2311;H:DM5PR18MB1578.namprd18.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1;
received-spf: None (protection.outlook.com: marvell.com does not designate
 permitted sender hosts)
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam-message-info: 7jZ2YdGBMhjQ0u0cCoJMeAKLiNPYcxnFcXE+mTC1Ft+7j/i5PXNKemabX1FmWmEqTd1A63YDQHw2D2SoyPhxjxnVfgebVmIw1MJYdCh41k+5WXnzR7siSRWDnlrWmbCCjig/tc0J0KTcUuMQNbz0+YWa0CHu8Za1D6fSACA5HDPH3VHjU52DGIi+h6y6bhwplqPZodyogNYazhKltE1reRBwqrV/HnxQ2Il8Nwqnths9aEBJA1dTShcc/4Yu7cc3i9oaZH2RQLeBwrk1wOltfw9BTGyorEhOodCSz6GxTvOXqc3MzS86XkaSCuiU0heVdP7JOdrw74VdfSvBFpRofqkQ35tGYWvg1HN++S6+jVH1NMpkJSy5I25ai/7HRjUENBthsdH+4nGMRAf56Y3lo01avTPR33Az3Pt9EQ6foqc=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <E532FD5EC62C24439F37BEADDF53CAA4@namprd18.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: 9e1e54d5-1717-4bf1-9c83-08d6ccb2428f
X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Apr 2019 14:52:11.6610
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 70e1fb47-1155-421d-87fc-2e58f638b6e0
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR18MB2311
X-OriginatorOrg: marvell.com
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-29_08:,,
 signatures=0
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Catalin & Will,

I've been looking into performance issues that were reported for several
test-cases, for instance an nginx benchmark.

It turned out the issue we have on ThunderX2 is the file open-close sequenc=
e
with small read sizes. If the used files are opened read-only the
lockref code (enabled by ARCH_USE_CMPXCHG_LOCKREF) is used.

The lockref CMPXCHG_LOOP uses an unbound (as long as the associated
spinlock isn't taken) while loop to change the lock count. This behaves
badly under heavy contention (~25x retries for one cmpxchg to succeed
with 28 threads operating on the same file). In case of a NUMA system
it also behaves badly as the access from the other socket is much slower.

The fact that on ThunderX2 cpu_relax() turns only into one NOP
instruction doesn't help either. On Intel pause seems to block the thread
much longer, avoiding the heavy contention thereby.

With the queued spinlocks implementation I can see a major improvement
when I disable lockref. A trivial open-close test-case improves by
factor 2 while system time is decreasing also 2x. Looking at kernel compile
and dbench numbers didn't show any regression with lockref disabled.

Can we simply disable lockref? Is anyone else seeing this issue? Is there
an arm64 platform that actually implements yield?

Thanks,
Jan