Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Dexuan Cui <decui@microsoft.com>
To:     Bjorn Helgaas <bhelgaas@google.com>,
        "linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
        Jake Oshins <jakeo@microsoft.com>,
        KY Srinivasan <kys@microsoft.com>,
        Stephen Hemminger <sthemmin@microsoft.com>
CC:     "devel@linuxdriverproject.org" <devel@linuxdriverproject.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Haiyang Zhang <haiyangz@microsoft.com>,
        Jork Loeser <Jork.Loeser@microsoft.com>,
        "Chris Valean (Cloudbase Solutions SRL)" <v-chvale@microsoft.com>,
        "Adrian Suhov (Cloudbase Solutions SRL)" <v-adsuho@microsoft.com>,
        Simon Xiao <sixiao@microsoft.com>,
        'Eyal Mizrachi' <eyalmi@mellanox.com>,
        "Jack Morgenstein" <jackm@mellanox.com>,
        Armen Guezalian <armeng@mellanox.com>,
        Firas Mahameed <firas@mellanox.com>,
        Tziporet Koren <tziporet@mellanox.com>,
        Daniel Jurgens <danielj@mellanox.com>
Subject: [PATCH] PCI: hv: use effective affinity mask
Thread-Topic: [PATCH] PCI: hv: use effective affinity mask
Thread-Index: AdNTSQFbm8/mhC8xRuOwOLkoIvvyZg==
Date:   Wed, 1 Nov 2017 20:30:53 +0000
Message-ID: <PS1P15301MB00112EE79DE499E8B2BA338BBF5F0@PS1P15301MB0011.APCP153.PROD.OUTLOOK.COM>
Accept-Language: en-US
Content-Language: en-US
msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True;
 MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47;
 MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Ref=https://api.informationprotection.azure.com/api/72f988bf-86f1-41af-91ab-2d7cd011db47;
 MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Owner=decui@microsoft.com;
 MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2017-11-01T13:30:50.3431552-07:00;
 MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=General;
 MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Application=Microsoft Azure
 Information Protection;
 MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Extended_MSFT_Method=Automatic;
 Sensitivity=General
authentication-results: spf=none (sender IP is )
 smtp.mailfrom=decui@microsoft.com; 
x-originating-ip: [2001:4898:80e8:2::7ae]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1;PS1P15301MB0009;6:RCLn/1nrp9rsMPmxsqxVW3UivINanEGapziGXwO6K0dHKjEtrvGbMPkMkkBkcgmWalLlTDAjGAXHC0VtM1fGafcJxKxHL90iYK3936BFwHEvOBbJoqGHqhAx5vAKp3LHNdFkbIebK9NBjoZoEKNXyAAY2umJojSOlVSLmV+J3PNfUPkIqooPfYRO9P5MGAkSOayDURXWJffdZB5wcx2Te4uar85jZWiIznWyoB0ue5ErMctMOW6J6wX07mjlMdD/T/uAtQ4vHsuOuvMu688suQPUMF0eN+MuRrxTKTg5NvK+jo3WYbcfiwJ769DzQytbuLkZC3/LPj0tHnG9vgVv2xZTR9dWkSwG3oJZv1QcHzU=;5:uzhv0M5bYFHi4ZsjUi9D0HGji1GiUOU1H0Ic/Ungg/HDR+/YPIXE37Ek+hYF35dccqCFcDlMDZ9uk6nsKDMcicii+4f/SV3mxA9KYhmaSOncO4Lbk0D9PtT4f33wTWH4JaxlLwmqxNgCddejI7fjcGiO8Ta8Y0VXz/VDGvouTkU=;24:paw8gvNXDcfgSsrYWQ5QD8vzBcP7fIX4xBS0sZ+MTa25mtYxzPd66+NrAi3FBnrlz6e3Qg19tOoxJ8sUMJUCwfNgYHirKSNrePy0xOcp6sM=;7:M5iHiGZ59Z5kdTpM7OuApPLRzcQKa4t5NLGEhOnzFH8bKHvNCKnM68mUR5DVlrYNiOJx3MbEqViOt+NfGhj02nwEw3QCFIeGOxlijVFZJaPDNJ3Mp9xfp0K5tsdEMT7ywcXbY8sywjRagphbFUgSsqUkQVNVLrkos/Rll4PeWyEutRvh5GoEcGmcZonl1ipzx7PdA6UL7l3+1Dub4UvS6PpiUOjOR6r6qN7WOgKVtUeZR6tznqvv8j59nT8NKZD3
x-ms-exchange-antispam-srfa-diagnostics: SSOS;
x-ms-office365-filtering-correlation-id: f5cfcc9a-74ea-4fd9-7e45-08d5216772ec
x-ms-office365-filtering-ht: Tenant
x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(4534020)(4602075)(48565401081)(2017052603199);SRVR:PS1P15301MB0009;
x-ms-traffictypediagnostic: PS1P15301MB0009:
x-ld-processed: 72f988bf-86f1-41af-91ab-2d7cd011db47,ExtAddr
x-exchange-antispam-report-test: UriScan:(89211679590171);
x-microsoft-antispam-prvs: <PS1P15301MB0009BED07D30DE4CF33420D1BF5F0@PS1P15301MB0009.APCP153.PROD.OUTLOOK.COM>
x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(61425038)(6040450)(2401047)(5005006)(8121501046)(10201501046)(93006095)(93001095)(100000703101)(100105400095)(3231020)(3002001)(6055026)(61426038)(61427038)(6041248)(20161123562025)(20161123560025)(20161123558100)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123564025)(20161123555025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:PS1P15301MB0009;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:PS1P15301MB0009;
x-forefront-prvs: 0478C23FE0
x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(376002)(346002)(39860400002)(47760400005)(199003)(189002)(110136005)(105586002)(1511001)(2501003)(189998001)(55016002)(5250100002)(53936002)(2421001)(9686003)(2900100001)(6636002)(4326008)(2561002)(7416002)(7696004)(3660700001)(5660300001)(97736004)(101416001)(106356001)(68736007)(6116002)(102836003)(2906002)(3280700002)(50986999)(8990500004)(10090500001)(14454004)(6506006)(74316002)(6436002)(54356999)(25786009)(7736002)(305945005)(478600001)(8936002)(8676002)(81166006)(10290500003)(86362001)(81156014)(22452003)(86612001)(54906003)(33656002)(316002)(99286004);DIR:OUT;SFP:1102;SCL:1;SRVR:PS1P15301MB0009;H:PS1P15301MB0011.APCP153.PROD.OUTLOOK.COM;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en;
received-spf: None (protection.outlook.com: microsoft.com does not designate
 permitted sender hosts)
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: microsoft.com
X-MS-Exchange-CrossTenant-Network-Message-Id: f5cfcc9a-74ea-4fd9-7e45-08d5216772ec
X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Nov 2017 20:30:53.2336
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PS1P15301MB0009
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org


The effective_affinity_mask is always set when an interrupt is assigned in
__assign_irq_vector() -> apic->cpu_mask_to_apicid(), e.g. for struct apic
apic_physflat: -> default_cpu_mask_to_apicid() ->
irq_data_update_effective_affinity(), but it looks d->common->affinity
remains all-1's before the user space or the kernel changes it later.

In the early allocation/initialization phase of an irq, we should use the
effective_affinity_mask, otherwise Hyper-V may not deliver the interrupt
to the expected cpu. Without the patch, if we assign 7 Mellanox ConnectX-3
VFs to a 32-vCPU VM, one of the VFs may fail to receive interrupts.

Signed-off-by: Dexuan Cui <decui@microsoft.com>
Cc: Jake Oshins <jakeo@microsoft.com>
Cc: Jork Loeser <jloeser@microsoft.com>
Cc: Stephen Hemminger <sthemmin@microsoft.com>
Cc: K. Y. Srinivasan <kys@microsoft.com>
---

Please consider this for v4.14, if it's not too late.

 drivers/pci/host/pci-hyperv.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/host/pci-hyperv.c b/drivers/pci/host/pci-hyperv.c
index 5ccb47d..8b5f66d 100644
--- a/drivers/pci/host/pci-hyperv.c
+++ b/drivers/pci/host/pci-hyperv.c
@@ -879,7 +879,7 @@ static void hv_irq_unmask(struct irq_data *data)
 	int cpu;
 	u64 res;
=20
-	dest =3D irq_data_get_affinity_mask(data);
+	dest =3D irq_data_get_effective_affinity_mask(data);
 	pdev =3D msi_desc_to_pci_dev(msi_desc);
 	pbus =3D pdev->bus;
 	hbus =3D container_of(pbus->sysdata, struct hv_pcibus_device, sysdata);
@@ -1042,6 +1042,7 @@ static void hv_compose_msi_msg(struct irq_data *data,=
 struct msi_msg *msg)
 	struct hv_pci_dev *hpdev;
 	struct pci_bus *pbus;
 	struct pci_dev *pdev;
+	struct cpumask *dest;
 	struct compose_comp_ctxt comp;
 	struct tran_int_desc *int_desc;
 	struct {
@@ -1056,6 +1057,7 @@ static void hv_compose_msi_msg(struct irq_data *data,=
 struct msi_msg *msg)
 	int ret;
=20
 	pdev =3D msi_desc_to_pci_dev(irq_data_get_msi_desc(data));
+	dest =3D irq_data_get_effective_affinity_mask(data);
 	pbus =3D pdev->bus;
 	hbus =3D container_of(pbus->sysdata, struct hv_pcibus_device, sysdata);
 	hpdev =3D get_pcichild_wslot(hbus, devfn_to_wslot(pdev->devfn));
@@ -1081,14 +1083,14 @@ static void hv_compose_msi_msg(struct irq_data *dat=
a, struct msi_msg *msg)
 	switch (pci_protocol_version) {
 	case PCI_PROTOCOL_VERSION_1_1:
 		size =3D hv_compose_msi_req_v1(&ctxt.int_pkts.v1,
-					irq_data_get_affinity_mask(data),
+					dest,
 					hpdev->desc.win_slot.slot,
 					cfg->vector);
 		break;
=20
 	case PCI_PROTOCOL_VERSION_1_2:
 		size =3D hv_compose_msi_req_v2(&ctxt.int_pkts.v2,
-					irq_data_get_affinity_mask(data),
+					dest,
 					hpdev->desc.win_slot.slot,
 					cfg->vector);
 		break;
--=20
2.7.4


From 1583708833967349521@xxx Fri Nov 10 19:33:59 +0000 2017
X-GM-THRID: 1583708833967349521
X-Gmail-Labels: Inbox,Category Forums,HistoricalUnread