Received: by 2002:a05:7412:8d10:b0:f3:1519:9f41 with SMTP id bj16csp4485660rdb; Tue, 12 Dec 2023 00:09:17 -0800 (PST) X-Google-Smtp-Source: AGHT+IEf1efwE5NpWpG5BZ9vNMlzyYCpXxBvnlFpF0YJ4NL7uhzWWWdADIKEWJtQ+Pxxpd11aHAx X-Received: by 2002:a05:6358:cc27:b0:16e:2876:5eed with SMTP id gx39-20020a056358cc2700b0016e28765eedmr3264270rwb.15.1702368557427; Tue, 12 Dec 2023 00:09:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702368557; cv=none; d=google.com; s=arc-20160816; b=0RGJp6+JrU2wCdlg6oIC1jayDyNLAOQm6iTIo3Xo7fkchS33BUo4XaXbs8vwWDXBhw 7sCSDKhkPIaD+MiYbElDasU+LzbvMNeysn+QxXlMUcEodVOVd1NojfCPQzI6oNZqQRY9 LfpB30CnWUpM8+AWlnoKwnv0sie2UEG5X+iw3pJqbsTVNtzoqAVk6joVt3cCB3jkUJdx m09y2dNLCw6qR5VEPz5+eFWoQb7INJf2qjlrie1PpuLGevx5VUWNKX4N0wlA79NsPVnA XrRKtzIX22lFU6smw8v3WdUmcvsHicAuIs620ExL1d00r7KI/7unlFspNZUb9UCS+Ba7 FsQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=ZZ9ea9ok6ZP/daUsmJ5Nh5VRT1Vhh0QOAkcU3YgmvrE=; fh=NBmREaXSAZFpoQahUCeDwuDc4C2pm3/orU1mjt7tsXY=; b=Tfv3NyEnE9YQDegdHTE/hf3s+1mr4lxvDPR/8nvU0pCKu55HqIxA8snyAPOeblfjbo 2HuXczWgNmVc8P3raLOYLOQ1IeThpiDT4gixTJJ+/U+Ekf5KvPHO4hw8mxk5CQrmdQAp 4f3YkgoakLYtIrOUSA4WD5edmynoidtQWGKtGvRf+rojIvXXIzejWVPqcoo+jo/5fmnJ 1a9LYZPCN6vzH4eAYHHKCvwESzzSGxBedjk1EYymAFL1FmXOjV7UVE3hlxmDjWQLViO0 UNxFlL+6hrhv9YEvLx9u6B33VFO6yrR+toeQ94LQ06qz2qGHjats7Y+hl8pEJaa//mzU /oeQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@citrix.com header.s=google header.b=FtT1NeLI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=citrix.com Return-Path: Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id z11-20020a17090ab10b00b00286f945f4f0si7515938pjq.131.2023.12.12.00.09.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Dec 2023 00:09:17 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@citrix.com header.s=google header.b=FtT1NeLI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=citrix.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 542C7804DD98; Tue, 12 Dec 2023 00:09:16 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345945AbjLLIJG (ORCPT + 99 others); Tue, 12 Dec 2023 03:09:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47720 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235059AbjLLIIy (ORCPT ); Tue, 12 Dec 2023 03:08:54 -0500 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12B71EB for ; Tue, 12 Dec 2023 00:09:00 -0800 (PST) Received: by mail-wm1-x332.google.com with SMTP id 5b1f17b1804b1-40c39e936b4so30776055e9.1 for ; Tue, 12 Dec 2023 00:08:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.com; s=google; t=1702368538; x=1702973338; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=ZZ9ea9ok6ZP/daUsmJ5Nh5VRT1Vhh0QOAkcU3YgmvrE=; b=FtT1NeLIlgYig+jqCcb4c0GjeuBJQU0XdUUsgHK7SGUmoY/fzr6Y/qUfPhLbn/lJY9 3eEWFlS4EmyNMBpm2zGPal1sX0b5LVDFTwjzEEATxYCucL9vW4lH54I/FL4WJJV9423C +HHlBgSM18/lJZ0XKsV07jUcKMUdIESxVfdQ8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1702368538; x=1702973338; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=ZZ9ea9ok6ZP/daUsmJ5Nh5VRT1Vhh0QOAkcU3YgmvrE=; b=aLY9lM9hcpjozu9LY76ZD/SQDc50oAFdWDrGgCyv8/SXNa2srTGSDmUROC/afbtqxl wtnVZXDNO7rX56b3WP3Hk6xrkqvkiOGt4Bsc1AU1Dt2JamkYCPVUB5aRywbMKWRQATni WWv9krvOyNh4YoPnlU7GmBSC1BycmUMLwwL3vnBnvRoVZ0tksvD9UKEjTNr+1e5wNGT1 0qVNDpkGtQliCf2DlqJ2wMQxd3L/kI2225hPQE6+aoj4iwh82pqQZbfmSnnKtf7YVuv6 Ev7/9Dw6y7HAwGeWO+ZajacfKglo0L21XfFghiTD6DF3itEZ+BNJWG8q2D/CpAkLgW2D 3EeA== X-Gm-Message-State: AOJu0Yx0R8vjKCvtrq7eO3n6WfV5ljGWHy2H9RcdglqZjYSwgFifA8wg W7/LWCSSxYRzxdbD62swGRzFzg== X-Received: by 2002:a05:600c:28e:b0:40c:53d4:391 with SMTP id 14-20020a05600c028e00b0040c53d40391mr34795wmk.24.1702368538240; Tue, 12 Dec 2023 00:08:58 -0800 (PST) Received: from localhost ([213.195.113.99]) by smtp.gmail.com with ESMTPSA id n10-20020a05600c500a00b004094e565e71sm15611143wmr.23.2023.12.12.00.08.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Dec 2023 00:08:58 -0800 (PST) Date: Tue, 12 Dec 2023 09:08:57 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Jiqian Chen Cc: Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , "Rafael J . Wysocki" , Len Brown , Bjorn Helgaas , Thomas Gleixner , Boris Ostrovsky , linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-acpi@vger.kernel.org, xen-devel@lists.xenproject.org, Alex Deucher , Christian Koenig , Stewart Hildebrand , Xenia Ragiadakou , Honglei Huang , Julia Zhang , Huang Rui Subject: Re: [RFC KERNEL PATCH v3 1/3] xen/pci: Add xen_reset_device_state function Message-ID: References: <20231210161519.1550860-1-Jiqian.Chen@amd.com> <20231210161519.1550860-2-Jiqian.Chen@amd.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20231210161519.1550860-2-Jiqian.Chen@amd.com> X-Spam-Status: No, score=-1.8 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Tue, 12 Dec 2023 00:09:16 -0800 (PST) On Mon, Dec 11, 2023 at 12:15:17AM +0800, Jiqian Chen wrote: > When device on dom0 side has been reset, the vpci on Xen side > won't get notification, so that the cached state in vpci is > all out of date with the real device state. > To solve that problem, add a new function to clear all vpci > device state when device is reset on dom0 side. > > And call that function in pcistub_init_device. Because when > using "pci-assignable-add" to assign a passthrough device in > Xen, it will reset passthrough device and the vpci state will > out of date, and then device will fail to restore bar state. > > Co-developed-by: Huang Rui > Signed-off-by: Jiqian Chen > --- > drivers/xen/pci.c | 12 ++++++++++++ > drivers/xen/xen-pciback/pci_stub.c | 4 ++++ > include/xen/interface/physdev.h | 8 ++++++++ > include/xen/pci.h | 6 ++++++ > 4 files changed, 30 insertions(+) > > diff --git a/drivers/xen/pci.c b/drivers/xen/pci.c > index 72d4e3f193af..e9b30bc09139 100644 > --- a/drivers/xen/pci.c > +++ b/drivers/xen/pci.c > @@ -177,6 +177,18 @@ static int xen_remove_device(struct device *dev) > return r; > } > > +int xen_reset_device_state(const struct pci_dev *dev) > +{ > + struct physdev_pci_device device = { > + .seg = pci_domain_nr(dev->bus), > + .bus = dev->bus->number, > + .devfn = dev->devfn > + }; > + > + return HYPERVISOR_physdev_op(PHYSDEVOP_pci_device_state_reset, &device); > +} > +EXPORT_SYMBOL_GPL(xen_reset_device_state); > + > static int xen_pci_notifier(struct notifier_block *nb, > unsigned long action, void *data) > { > diff --git a/drivers/xen/xen-pciback/pci_stub.c b/drivers/xen/xen-pciback/pci_stub.c > index e34b623e4b41..24f599eaec14 100644 > --- a/drivers/xen/xen-pciback/pci_stub.c > +++ b/drivers/xen/xen-pciback/pci_stub.c > @@ -421,6 +421,10 @@ static int pcistub_init_device(struct pci_dev *dev) > else { > dev_dbg(&dev->dev, "resetting (FLR, D3, etc) the device\n"); > __pci_reset_function_locked(dev); > + if (!xen_pv_domain()) > + err = xen_reset_device_state(dev); > + if (err) > + goto config_release; I think you are missing other instances where __pci_reset_function_locked() is called in pci_stub.c? See pcistub_device_release() and pcistub_put_pci_dev(). Overall I'm not sure why the hypercall wrapper needs to live in xen/pci.c. I think it would be better if you could create a static wrapper in pci_stub.c that does the call to __pci_reset_function_locked() plus PHYSDEVOP_pci_device_state_reset. That would make it less likely that new callers of __pci_reset_function_locked() are introduced without noticing the need to also call PHYSDEVOP_pci_device_state_reset. Thanks, Roger.