Received: by 2002:ac0:aa62:0:0:0:0:0 with SMTP id w31-v6csp833919ima; Wed, 24 Oct 2018 09:53:40 -0700 (PDT) X-Google-Smtp-Source: AJdET5fTkoZl9c3Mx5NLeikdMPS5v+8yCMl8krrhrh32+9Zig/aWl6XIgz4FQqDX3zJuZvS+/T6X X-Received: by 2002:a63:36c8:: with SMTP id d191-v6mr3192431pga.404.1540400020686; Wed, 24 Oct 2018 09:53:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1540400020; cv=none; d=google.com; s=arc-20160816; b=dnxZfzrhH4KcBED2z0Re8ST2828jryTsLOGskA9hfZ/HutR3iv7wU6a/p4OCP96+bK +85ylOJU624jAvmEUYrZ2y/qiOpGLJ54Vyt9i2Fq1BCtYZ8AScy/93fOyhfmaL+f3RuB h7d2zTvnnDBnRLiskJphucBqHj5KvVhImFtz35NyZka9ZliojyOzGOS0qceiF2f5FffY fBiXkzsSLFK51exOvEmnT/xq0WQB+rc4Ad0O9P2EnRga4z5ehs2fsQuW4t4ndM3sqcLj k02gKx2N5+0cQRWNcbxjQRk8zXvQG/I8/V6hOqniGzkDLv5hJ9To0u3A1jDQhZzUUD37 SGaw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:content-transfer-encoding :spamdiagnosticmetadata:spamdiagnosticoutput:msip_labels :content-language:accept-language:in-reply-to:references:message-id :date:thread-index:thread-topic:subject:cc:to:from:dkim-signature; bh=5y9tsG6vVEdADsuuIW1pEkjV0hWxrOIFWajMDy/cYsk=; b=g1fWSeBhgEtYiOerac+OGkuTqy95PqWJ/SODtrW/Xp9AR5XY2xY+ZEwQyFgZZ2/rWs niMhfExmapNnYy6nU7ZrsMeDaKP3KR1uEPR0Z+LMSQogrFrnn7RrPe+LogfJ1g92Y/Ob wcw78NbFQGUisYYfaEwSs7QS7xFecevcQSU4E26TGBhaQWbUEfjGB107oUswFUHOSFt2 NAcE4QCgFDRGvb0p8Fs5853F7uFCNJdxbv28FI54cHtIWPHsqKvRCB+p2u8Nbg45odwz joig7s1OGTtBzar9+Z8xOTl/9M2CrHspfi2R2txT/HXHm1yy9I7fX4lHzhTwH/iT8iQ8 61yw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@microsoft.com header.s=selector1 header.b=BJBThkHv; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=microsoft.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e9-v6si5366559pln.265.2018.10.24.09.53.24; Wed, 24 Oct 2018 09:53:40 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@microsoft.com header.s=selector1 header.b=BJBThkHv; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=microsoft.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727407AbeJYBVw (ORCPT + 99 others); Wed, 24 Oct 2018 21:21:52 -0400 Received: from mail-sn1nam01on0137.outbound.protection.outlook.com ([104.47.32.137]:27136 "EHLO NAM01-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727369AbeJYBVw (ORCPT ); Wed, 24 Oct 2018 21:21:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5y9tsG6vVEdADsuuIW1pEkjV0hWxrOIFWajMDy/cYsk=; b=BJBThkHvhGmBr+sJmWDLsndUsYPvKHrgbcqZKgeR/uputrnHOqwzXbh3nbiRh9MDTDSAUYgqpi1jb15GXS9LoWfuM3tCmlNQQ6ygoChFo7qa6IK4SnShC0Va0tjs8s8etVUe4j++Jf6RSc3VaMQ8zqUEKnLsrQdD+35qILBa7Sw= Received: from CY4PR21MB0773.namprd21.prod.outlook.com (10.173.192.19) by CY4PR21MB0118.namprd21.prod.outlook.com (10.173.189.12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1294.8; Wed, 24 Oct 2018 16:53:00 +0000 Received: from CY4PR21MB0773.namprd21.prod.outlook.com ([fe80::6d0b:b90f:7a73:99fb]) by CY4PR21MB0773.namprd21.prod.outlook.com ([fe80::6d0b:b90f:7a73:99fb%8]) with mapi id 15.20.1294.001; Wed, 24 Oct 2018 16:53:00 +0000 From: Michael Kelley To: Yi Sun , "linux-kernel@vger.kernel.org" CC: "x86@kernel.org" , "tglx@linutronix.de" , "jgross@suse.com" , "chao.p.peng@intel.com" , "chao.gao@intel.com" , "isaku.yamahata@intel.com" , Tianyu Lan , KY Srinivasan , Haiyang Zhang , Stephen Hemminger Subject: RE: [PATCH v1 2/2] x86/hyperv: make HvNotifyLongSpinWait hypercall Thread-Topic: [PATCH v1 2/2] x86/hyperv: make HvNotifyLongSpinWait hypercall Thread-Index: AQHUZ7E6J7MVriGzlECNPmkOKu+8U6Uul5eQ Date: Wed, 24 Oct 2018 16:53:00 +0000 Message-ID: References: <1539954835-34035-1-git-send-email-yi.y.sun@linux.intel.com> <1539954835-34035-3-git-send-email-yi.y.sun@linux.intel.com> In-Reply-To: <1539954835-34035-3-git-send-email-yi.y.sun@linux.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Owner=mikelley@ntdev.microsoft.com; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2018-10-24T16:52:55.3507908Z; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=General; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Application=Microsoft Azure Information Protection; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Extended_MSFT_Method=Automatic; Sensitivity=General x-originating-ip: [12.130.117.98] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;CY4PR21MB0118;6:8heH+PbX596/d0ArLlq0m3Cw1DGLIUXvTYSuW+ABp0DuhY43yqYSSTy3gokk7sfzEtjVto4jz+aUfJTL0lgj8/tEjJLZKjWPDMVeF0a7DUavqCfrGFWKLsieC/Hfb7qUDJNbJkphvTgYl7pQ7R3iI72te+7mkRiK9BQiZU2rrjjezsiBlNTYURYiv/0RO9sLqALaAevX1jRHXpqbla2ILqT7mzsK33ODNKN/CRO5CNyO5mdnGAcKy5hdSPr5qPfy0ZejYgz3Lv1byoNEagaog46c/NOcCS4fWDYH/XrmIyS80l1Va3a+O9uHnUnP9C93uV6wK+boe9U85Ss+P80Jvtv97OVYovhNtsRVsNIrlNeBy1aaabfBBWbblWXDEjkNx3rmL4oNt4SVyhXJFHSFUJ7qY3GxbvGkh6Z5LjYwvz0q6Gld671qPr+MaYKvfgBCRmumnjSVTWMIqbZRuEH59w==;5:iKemLWLB1f9KHfuyqUNbhb565km43fQooGJMrecRjida4XiuwIywz6yXJNETP7U7IjB6jGXYeVIJISdSE//FiBIxHa7hE4Sis7+/eEqrnx9gOaICueO1z0inpCw5/y+KbtBs95nTRb7M6AItdXnMrdrAOy/9hVRupi9vp9kY7J0=;7:IXOdcdV0SvQWmSjoSy8CHiYN4cHTX6lXdJd1XRI/CyeNq8vb2idHNANUNQFs9zACnB8ePKRim3tkOtVz42EMcZYw/UPQ9rJMKMhyHNGcpg/iu+uU6ZqjWr5iz0tpmWJH3H0ArEkpBECgq/iKrL+9Qw== x-ms-office365-filtering-correlation-id: e3acf948-1d1b-481f-1418-08d639d1285a x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0;PCL:0;RULEID:(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600074)(711020)(4618075)(2017052603328)(7193020);SRVR:CY4PR21MB0118; x-ms-traffictypediagnostic: CY4PR21MB0118: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(228905959029699); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(8211001083)(6040522)(8220027)(2401047)(5005006)(8121501046)(93006095)(93001095)(10201501046)(3231355)(944501410)(52105095)(2018427008)(3002001)(6055026)(148016)(149066)(150057)(6041310)(20161123558120)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123564045)(201708071742011)(7699051)(76991095);SRVR:CY4PR21MB0118;BCL:0;PCL:0;RULEID:;SRVR:CY4PR21MB0118; x-forefront-prvs: 083526BF8A x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6029001)(346002)(376002)(396003)(366004)(136003)(39860400002)(199004)(189003)(97736004)(102836004)(6346003)(74316002)(66066001)(68736007)(7736002)(2906002)(8990500004)(6436002)(305945005)(186003)(6116002)(5660300001)(107886003)(10090500001)(110136005)(256004)(33656002)(14444005)(54906003)(26005)(2900100001)(2501003)(8936002)(81156014)(3846002)(81166006)(6246003)(486006)(316002)(10290500003)(86612001)(55016002)(4326008)(5250100002)(478600001)(6506007)(476003)(446003)(11346002)(8676002)(86362001)(7696005)(14454004)(76176011)(25786009)(9686003)(106356001)(53936002)(71200400001)(71190400001)(229853002)(105586002)(99286004)(22452003);DIR:OUT;SFP:1102;SCL:1;SRVR:CY4PR21MB0118;H:CY4PR21MB0773.namprd21.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: microsoft.com does not designate permitted sender hosts) authentication-results: spf=none (sender IP is ) smtp.mailfrom=mikelley@microsoft.com; x-microsoft-antispam-message-info: B7I1GtVVysHcB26GKfAUhBq/VQ1DAQkAb4xTY+Tcx6p4pKwF5NSeukx1dA5pa4GkHaR6csjDQPzcdC9zvbhl94vX4d9GkRk7PK7mo1OBt4aeIP48v2jkhKF7cH6KUKN8ZL90ZherMrFNaE1WetF4ZAPYlgXnn1IGz9zJP9ws55YUtpGf1MoDFWgl9pIkgbIwfJZ+VEmv3P7UwN2/5cudsSNxmJfNVPBMnhh2onuH+zW/QntQHMsngo07zgU7f3Ob0qGO8LZh8r0DFZwRkolkxhdRPTMVHgzXR/Myr646ZAMnYqFRqqDxEmcb7cz0ADElTGYoiAXnqRzrAwnJGxdoRO7i6xXsjcNnQXbueyUwXis= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-Network-Message-Id: e3acf948-1d1b-481f-1418-08d639d1285a X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Oct 2018 16:53:00.5432 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR21MB0118 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Yi Sun Sent: Friday, October 19, 2018 6:1= 4 AM >=20 > The HvNotifyLongSpinWait hypercall (HVCALL_NOTIFY_LONG_SPIN_WAIT) > is used by a guest OS to notify the hypervisor that the calling > virtual processor is attempting to acquire a resource that is > potentially held by another virtual processor within the same > Virtual Machine. This scheduling hint improves the scalability of > VMs with more than one virtual processor on Hyper-V. >=20 > Per MSFT TLFS, the retry number (SpinWaitInfo) is sent to hypervisor > only when the retry number exceeds the recommended number. If > recommended number is 0xFFFFFFFF, never retry. The HvNotifyLongSpinWait hypercall should be understood to be advisory only. As you noted, it is a scheduling hint to the hypervisor that some virtual CPU in the VM holds a spin lock. Even though Linux knows which vCPU holds the spin lock, the hypercall does not provide a way to give that information to Hyper-V. The hypercall always returns immediately. The "retry number" is a bit mis-named in the Hyper-V Top Level Functional Spec (TLFS). It is essentially a threshold value. Hyper-V is saying "don't bother to advise me about the spin lock until you have done a certain number of spins." This threshold prevents over-notifying Hyper-V such that the notification becomes somewhat meaningless. It's not immediately clear to me why the hypercall passes that value as an input, but maybe it lets the Hyper-V scheduler prioritize among vCPUs based on how many times they have spun for a lock. I think we were told that current Hyper-V implementations ignore this input value anyway. I believe the description of the sentinel value 0xFFFFFFFF in the Hyper-V TLFS is incorrect. Because it is the max possible threshold value, that value in the EBX register just means to not ever bother to notify. The description should be "0xFFFFFFFF indicates never to notify." The value does *not* indicate anything about retrying to obtain the spin lock. > static bool __initdata hv_pvspin =3D true; >=20 > +bool hv_notify_long_spin_wait(int retry_num) retry_num should probably be declared as unsigned int. You don't want it to be treated as a negative number if the high order bit is set. > +{ > + /* > + * Per MSFT TLFS, the SpinWaitInfo is sent to hypervisor only when > + * the retry number exceeds the recommended number. > + * > + * If recommended number is 0xFFFFFFFF, never retry. > + */ > + if (ms_hyperv.num_spin_retry =3D=3D HYPERV_SPINLOCK_RETRY_NEVER) > + return false; > + > + if ((0 =3D=3D retry_num % ms_hyperv.num_spin_retry) && retry_num) I don't know if the "%" function is right here. Your implementation will notify Hyper-V on every multiple of num_spin_retry. The alternative is to notify once when the threshold is exceeded, and never again for this particular attempt to obtain a spin lock. We should check with the Hyper-V team for which approach they expect to be used. > + hv_do_fast_hypercall8(HVCALL_NOTIFY_LONG_SPIN_WAIT, > + retry_num); The Hyper-V TLFS seems to be inconsistent on whether the input parameter is 32-bits or 64-bits. In one place it is typed as UINT64, but in another= place it is shown as only 4 bytes. Need to clear this up with the Hyper-V team a= s well. > + > + return true; I don't see a need for this function to return true vs. false. Any calling= code should not change its behavior based on num_spin_retry. This function wil= l either notify Hyper-V or not notify Hyper-V, depending on whether the numbe= r of attempts to obtain the spinlock meets the threshold. But calling code w= ill do the same thing regardless of whether such a notification is made.=20 Michael