Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Date:   Mon, 22 Aug 2022 13:38:05 -0700
From:   Ira Weiny <ira.weiny@intel.com>
To:     Dan Williams <dan.j.williams@intel.com>
CC:     Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
        Bjorn Helgaas <bhelgaas@google.com>,
        Jonathan Cameron <Jonathan.Cameron@huawei.com>,
        Alison Schofield <alison.schofield@intel.com>,
        Vishal Verma <vishal.l.verma@intel.com>,
        Ben Widawsky <bwidawsk@kernel.org>,
        <linux-cxl@vger.kernel.org>, <linux-kernel@vger.kernel.org>,
        <linux-pci@vger.kernel.org>
Subject: Re: [PATCH 1/2] PCI: Allow drivers to request exclusive config
 regions
Message-ID: <YwPpLWgG37ObTMAP@iweiny-desk3>
References: <20220822005237.540039-1-ira.weiny@intel.com>
 <20220822005237.540039-2-ira.weiny@intel.com>
 <YwMktMqN0oFgCeZn@kroah.com>
 <6303dcb63e3e1_1b32294d9@dwillia2-xfh.jf.intel.com.notmuch>
Content-Type: text/plain; charset="us-ascii"
Content-Disposition: inline
In-Reply-To: <6303dcb63e3e1_1b32294d9@dwillia2-xfh.jf.intel.com.notmuch>
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?LmMoZ8FTGD8b/0dspk2Jm5QmSLGqGsd3hyqqfRhiK7xKigD61oSlDM7VOqA7?=
 =?us-ascii?Q?yO+Dw+cWcVU9gc+ZPOq1cDSO4mULQh6ER8mzMC4olP4rtKpuSyShfdP2SnIQ?=
 =?us-ascii?Q?HlTo1mOV6zEOxrvs35P3nNivYgWqpHnX7YsWbA+w9g6J8+Q/dwILCH0CWhoN?=
 =?us-ascii?Q?OVS3fe6KsSbTWg12A6rZU0ODbbKalScTOFEzYmyZ/PTUJWmfNnls3eXNfdjq?=
 =?us-ascii?Q?ws9GWk3Eg0C89MAAEUzYz2bYt6mfB+GsP7to1KtvmMUrvU4QXzVbH3m6sCDC?=
 =?us-ascii?Q?rMtE9YsEznSgCROzMzyS1Omr6bW4zCnuhd2hTsZ50HvefHKxMmcXbnShQOs5?=
 =?us-ascii?Q?qgjdDfT3lMKzqFJ9l3zL67+xN0LWUImMPrYFeHdXi+js4jdyEYhaQ3x7vvrB?=
 =?us-ascii?Q?ntdJ7S2BK/kQ6hcmzK7SVEFY+ccdUOE7hAp0lor2s9wf7WAfDWOlo+fzyutd?=
 =?us-ascii?Q?MVoFxhQuwgPM+fcPIa4QHdVmBIvfmLVumATXw2kPJXIiMUx82n44KBa+Iyo8?=
 =?us-ascii?Q?2PAkRaZxRq980dydYY2w1OyL23gFjm9FLZCYzijUER7EZT6WDg/7ZV7JO/Rm?=
 =?us-ascii?Q?96SAtj0Ns8NqiZvZD4sIQVgFPsw2KrhiczLADer2NLPk+HucyhZMTbUEPiv5?=
 =?us-ascii?Q?NZP3NRnvPvJXsbAtzuPlLMXRcvhTwDEsKaXDKhNTd3DcqB8cOV1A6nsjQwn9?=
 =?us-ascii?Q?6IRhUFpZIJdEI0qCu7gIJu8xOB2zZASBU0CfexpZc8nImzdnQjmPs4g6Hl7x?=
 =?us-ascii?Q?8rKWTis/L3Ohbp9/R8BtfEmxQYp2ZN4wq9eBeFOaZOR1xxrXEQf4NTj9aGgZ?=
 =?us-ascii?Q?wqizfYFDmKXXajBaAmeU5WSo5ShRo0ubYPaNKjel+cxwJKhXYAME641/WabH?=
 =?us-ascii?Q?ORWFxBP7Jx7APK8Esmtlhs9DUtoLjL9DQzyybB65B1Wd7cLa4oUIfxhb277a?=
 =?us-ascii?Q?z2RFUR+bVZongR3RaqRWEb8eLe6nVcJBD3n5AnADkoEkXl2OMbUXADzZqJtQ?=
 =?us-ascii?Q?ubwZA6HTa4xRzH9ijxkJ6RHeUIHbXQyL3uKXNG8IMJVdgKBn3e5BLZO+6iDl?=
 =?us-ascii?Q?dTjQ9I2BFNpD8or8MxI/nZhlSW2IxyjMonJzTy5Bd3MKtWmDkb0H+fzgJxLZ?=
 =?us-ascii?Q?rsY9s9HxOt0MnHyNCC7BlX25mi4l63Qx/s00eBgRuLIJkmUFkqXC45CZv0ik?=
 =?us-ascii?Q?mOFJMzZEMlh0OZj4Nl3uhUAEYYK6+15JXjy7rBoI2I4KLUgCKxl0ncl1JHLS?=
 =?us-ascii?Q?/7OS66uo0RFCmEH2M6gjQt+YGMy+JOQW4mszjDo9aaBJsken8/ZaNRmApSl3?=
 =?us-ascii?Q?NJ8DvnVHB37Nsw8Zu/jTBAaXzw9P63O8u9ZuoBUHVB4JYJxKWfqrhjJZI5Oy?=
 =?us-ascii?Q?4lZeZZqm+4pW80L86t3wpxDzzPh3nSs91Lpa8nVch2darbSBtVkWSGNaTvYG?=
 =?us-ascii?Q?Lgnb6Q7G+M0ZkO/64ns5CRo7zb9IT6lxdgFzWSv+GFqE7DjUv4NMehwXPxoE?=
 =?us-ascii?Q?Eu5NEXOF6HVkEjmenS5pBQQtrCDkDe1iqnczZ653ktLIbls2R11gZ2Rgr0xz?=
 =?us-ascii?Q?LdnYbRtAeF3TnK3KVD90TaN1qTyNJTxwBWC/ovC5?=
X-MS-Exchange-CrossTenant-Network-Message-Id: 9613723a-d7c7-4bd4-8a9c-08da847e3a44
X-MS-Exchange-CrossTenant-AuthSource: SA1PR11MB6733.namprd11.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Aug 2022 20:38:10.6049
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 46c98d88-e344-4ed4-8496-4ed7712e255d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: RJk4mBN2Av0waI4XRfhgrXqy8vhEq36TbLF8RflUzjwbrNW5hbkTyi5Ru24sft+cThDJYwIFnRZWSNi9KTbRlQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR11MB2930
X-OriginatorOrg: intel.com
X-Spam-Status: No, score=-7.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH,
        DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_HI,
        SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham
        autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
        lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

On Mon, Aug 22, 2022 at 12:44:54PM -0700, Dan Williams wrote:
> Greg Kroah-Hartman wrote:
> > On Sun, Aug 21, 2022 at 08:52:36PM -0400, ira.weiny@intel.com wrote:
> > > From: Ira Weiny <ira.weiny@intel.com>
> > > 
> > > PCI config space access from user space has traditionally been
> > > unrestricted with writes being an understood risk for device operation.
> > > 
> > > Unfortunately, device breakage or odd behavior from config writes lacks
> > > indicators that can leave driver writers confused when evaluating
> > > failures.  This is especially true with the new PCIe Data Object
> > > Exchange (DOE) mailbox protocol where backdoor shenanigans from user
> > > space through things such as vendor defined protocols may affect device
> > > operation without complete breakage.
> > 

I was about to respond when I saw Dan's message so I will respond to both of
you.

> > What userspace tools are out there messing with PCI config space through
> > userspace on these devices today?  How is this the kernel's fault if
> > someone runs such a thing?

It is not the kernels fault but multi-party actors on a DOE mailbox can result
in the kernel getting incorrect responses for it's valid query.  After my
conversation with Jonathan during DOE development[1] I'm more convinced that
this tainting of the kernel will be valuable.

[1] https://lore.kernel.org/linux-cxl/20220704074508.00000cac@Huawei.com/

> > 
> > > A prior proposal restricted read and writes completely.[1]  Greg and
> > > Bjorn pointed out that proposal is flawed for a couple of reasons.
> > > First, lspci should always be allowed and should not interfere with any
> > > device operation.  Second, setpci is a valuable tool that is sometimes
> > > necessary and it should not be completely restricted.[2]  Finally
> > > methods exist for full lock of device access if required.
> > > 
> > > Even though access should not be restricted it would be nice for driver
> > > writers to be able to flag critical parts of the config space such that
> > > interference from user space can be detected.
> > > 
> > > Introduce pci_request_config_region_exclusive() to mark exclusive config
> > > regions.  Such regions trigger a warning and kernel taint if accessed
> > > via user space.
> > > 
> > > [1] https://lore.kernel.org/all/161663543465.1867664.5674061943008380442.stgit@dwillia2-desk3.amr.corp.intel.com/
> > > [2] https://lore.kernel.org/all/YF8NGeGv9vYcMfTV@kroah.com/
> > > 
> > > Cc: Bjorn Helgaas <bhelgaas@google.com>
> > > Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
> > > Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > > Suggested-by: Dan Williams <dan.j.williams@intel.com>
> > > Signed-off-by: Ira Weiny <ira.weiny@intel.com>
> > > 
> > > ---
> > > Changes from[1]:
> > > 	Change name to pci_request_config_region_exclusive()
> > > 	Don't flag reads at all.
> > > 	Allow writes with a warn and taint of the kernel.
> > > 	Update commit message
> > > 	Forward port to latest tree.
> > > ---
> > >  drivers/pci/pci-sysfs.c |  6 ++++++
> > >  drivers/pci/probe.c     |  6 ++++++
> > >  include/linux/ioport.h  |  2 ++
> > >  include/linux/pci.h     | 16 ++++++++++++++++
> > >  kernel/resource.c       | 13 ++++++++-----
> > >  5 files changed, 38 insertions(+), 5 deletions(-)
> > > 
> > > diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
> > > index fc804e08e3cb..de41d761bdf5 100644
> > > --- a/drivers/pci/pci-sysfs.c
> > > +++ b/drivers/pci/pci-sysfs.c
> > > @@ -755,6 +755,12 @@ static ssize_t pci_write_config(struct file *filp, struct kobject *kobj,
> > >  	if (ret)
> > >  		return ret;
> > >  
> > > +	if (resource_is_exclusive(&dev->config_resource, off,
> > > +				  count)) {
> > > +		pci_warn(dev, "Write to restricted range %llx detected", off);
> 
> Note to Ira, I would expect a message like:
> 
> "Unexpected user write to kernel-exclusive config offset %#llx\n"

Done.

> 
> ...this probaly also wants current->comm similar to the
> lockdown_is_locked_down() warning.

Done.

> 
> > Will this allow any user to spam the kernel log from userspace?  You
> > might want to rate-limit it, right?
> 
> It should be a once-only message. You only get one chance to trample on
> a configuration address range that the kernel cares about and then
> tainted.

Yes my bad.  I should have thought of this.  Changed to pci_WARN_ONCE().

> 
> > > +		add_taint(TAINT_USER, LOCKDEP_STILL_OK);
> > > +	}
> > > +
> > >  	if (off > dev->cfg_size)
> > >  		return 0;
> > >  	if (off + count > dev->cfg_size) {
> > > diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
> > > index 6280e780a48c..d81d7457058b 100644
> > > --- a/drivers/pci/probe.c
> > > +++ b/drivers/pci/probe.c
> > > @@ -2303,6 +2303,12 @@ struct pci_dev *pci_alloc_dev(struct pci_bus *bus)
> > >  	INIT_LIST_HEAD(&dev->bus_list);
> > >  	dev->dev.type = &pci_dev_type;
> > >  	dev->bus = pci_bus_get(bus);
> > > +	dev->config_resource = (struct resource) {
> > > +		.name = "PCI Config",
> > > +		.start = 0,
> > > +		.end = -1,
> > > +	};
> > > +
> > >  #ifdef CONFIG_PCI_MSI
> > >  	raw_spin_lock_init(&dev->msi_lock);
> > >  #endif
> > > diff --git a/include/linux/ioport.h b/include/linux/ioport.h
> > > index 616b683563a9..cf1de55d14da 100644
> > > --- a/include/linux/ioport.h
> > > +++ b/include/linux/ioport.h
> > > @@ -312,6 +312,8 @@ extern void __devm_release_region(struct device *dev, struct resource *parent,
> > >  				  resource_size_t start, resource_size_t n);
> > >  extern int iomem_map_sanity_check(resource_size_t addr, unsigned long size);
> > >  extern bool iomem_is_exclusive(u64 addr);
> > > +extern bool resource_is_exclusive(struct resource *resource, u64 addr,
> > > +				  resource_size_t size);
> > >  
> > >  extern int
> > >  walk_system_ram_range(unsigned long start_pfn, unsigned long nr_pages,
> > > diff --git a/include/linux/pci.h b/include/linux/pci.h
> > > index 81a57b498f22..dde37bfa0ca5 100644
> > > --- a/include/linux/pci.h
> > > +++ b/include/linux/pci.h
> > > @@ -409,6 +409,7 @@ struct pci_dev {
> > >  	 */
> > >  	unsigned int	irq;
> > >  	struct resource resource[DEVICE_COUNT_RESOURCE]; /* I/O and memory regions + expansion ROMs */
> > > +	struct resource config_resource;		 /* driver exclusive config ranges */
> > 
> > So the driver only gets 1 range to mark this way?
> 
> No, config_resource is just the base of the resource tree to walk
> similar to how iomem_resource is consumed in the current
> iomem_is_exclusive() implementation.

I had to double check but yea, Dan is correct here.

> 
> > What are the ranges for typical devices that have this problem?
> 
> Unfortunately DOE is an extended capability that can pop any number of
> instances in that per-device list.
> 
> Although I also think there is potential to use this in something like
> pci_iomap() to trigger a future warning if userspace mucks with the BARs
> that the driver is using.
> 
> > This still feels very odd to me, how far do we have to go to protect
> > userspace from doing bad things on systems when they have the
> > permissions and access to do those bad things?
> 
> Right, this mechanism isn't about protection as much as it is reserving
> the space for kernel implementations of DOE protocols. Outright
> 'protection' is already there today in the form CONFIG_LOCK_DOWN_KERNEL
> that prevents userspace config writes. There just are not many distro
> kernels that turn that protection on.

I agree that there is no real protection.  This is just a way of us knowing
that bug reports have outside influence which can have real bad consequences.

> 
> > What are you trying to protect yourself from, bogus bug reports by
> > people doing bad things and then blaming you?  That's easy to handle,
> > just ignore them :)

I think the problem is that the bug report could be very difficult to know that
something bad was done.  I'm ok with ignoring them.  But this makes it clear
that the bug report was potentially user doing bad things.

I see bug reports being something like:

"Kernel issued query for CDAT.  Device returned garbage."

Vs

"Tainted kernel (error message seen) issued query for CDAT.  Device returned
garbage."

The first could be a malfunctioning device.  Where the second can clearly be
pushed back to the user to stop doing bad things.  I can't in good conscience
ignore the first report.

> 
> I asked Ira to push on this to protect the kernel from people like me,
> :). So, there is this massively complicated specification for device
> attestation and link integrity / encryption protection (SPDM and IDE)
> that has applications to both PCIe and CXL. I do not see a path in the
> near term to land that support in the kernel.
> 
> DOE being user accessible though, lends itself to pure userspace
> implementations of SPDM and IDE infrastructure. I want to develop that
> infrastructure, but also have the kernel reserve the space / right to
> obviate that implementation with kernel control of the DOE mailbox, SPDM
> sessions, and IDE keys in the future.
> 
> The warning helps keeps proof-of-concept and/or proprietary DOE code out
> of production release streams on the observation that end users tend to
> demand warn-free and taint-free kernel deployments.
> 
> Similar to CONFIG_IO_STRICT_DEVMEM, just turn this warning off if you
> are a distribution kernel vendor who does not care about DOE vendor
> tools affecting system state, and turn on CONFIG_LOCK_DOWN_KERNEL if you
> are on the other end of the spectrum and want protection from such
> things. The warn is middle-road option that I expect most distros would
> use.

FWIW I did not add any Kconfig for the feature.  Should I?

Ira