Received: by 2002:a05:6a10:f3d0:0:0:0:0 with SMTP id a16csp600634pxv; Thu, 24 Jun 2021 15:24:02 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz50lyV7Rn59vocXm7f2HcbImC6pxupSBhljULEEshSrLXqIqnp/MVPU741XpZDJECdiEtW X-Received: by 2002:a17:906:35c7:: with SMTP id p7mr7360782ejb.277.1624573442713; Thu, 24 Jun 2021 15:24:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1624573442; cv=none; d=google.com; s=arc-20160816; b=ahQ4SB55JgKUSbqcyYLHGSXgoMd5PUyl2cgUSRgDFbrkgDbUkrzEvKQb2amUuUjsvC BQPCZfiiXOCMQ3OUvG+RbNaPqn8GSuXc/oijA3aXzI3cehtal+S084bM9bezEHp7QSdD OITEGULZtjfYxi0fr2LhqFstUBcGDWKldJf9gcXsdVVqMiQnOmJo0u/GdbxD6IOzzSHX dVbZvZ72zDnDAGY4bdBLuNgfXnXkcz6KXq9Fy8ALaa9Ai4JEBvWpNAnUDS8elgtL3K8X TJrAJnhJ6vII1vl5KMNobade6yiPiCpW1ZiGB7ht5oU4CKkW0g93BkoBxv2SFUdvzvSx G7EA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:subject:cc:to:from:date :dkim-signature; bh=g477a7vYcPKjBKrDFYuYGrViI13ze8NFDaxSg09H1cQ=; b=B1HGb5008saLSYNCHhBO3WtcdrzcEZiP7ZhyC536bIRRMBBo6BHSJUFor2RsokLin8 Wmjz7hgGBQETPQiZq49C6vH+DUPY9b7iq0eeaYsidBp+gOJX4uf9XAlPnxQ1F7vrUKB8 CI5TTXKSm6cIEhIobwCjO3ISjPx65l8evI0oEZbYZjEncUUDHmT8AfPudtTCTt8VSwIe D+eyCgdeQ+/ox9pkCxKbCWgYKs8gC3hi15Yla4x4UwUpEPdADLlVtMWlf8Hnn2e5HAN1 S7QA4NDc4IplkMrrfoPjkk51+nu0DFnCmyOdfBJyYhAKgmli4OXDBZDRfz156cr2HG6n 3Ytg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=SZvzVfWZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id i14si4847175eje.489.2021.06.24.15.23.38; Thu, 24 Jun 2021 15:24:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=SZvzVfWZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232664AbhFXWZB (ORCPT + 99 others); Thu, 24 Jun 2021 18:25:01 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:54277 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229712AbhFXWY7 (ORCPT ); Thu, 24 Jun 2021 18:24:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1624573359; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g477a7vYcPKjBKrDFYuYGrViI13ze8NFDaxSg09H1cQ=; b=SZvzVfWZDgxUs16lzZoL+yzcVQ0ndXdSGGjxUDZST62VFoLqV+ODwHOArDRESz18CUdW5z +Joy9dL62IZKbo7bWjbDkNCEF+toc7ivsVBDjuU8jaApx7MjjCTpLlrtVK/N12hQzugECa jiFgt5q+qYxdrF9jw9t8cQDV5vTNyNw= Received: from mail-oi1-f199.google.com (mail-oi1-f199.google.com [209.85.167.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-485-qXCJhEJ-Psq7XK7GtZSgxg-1; Thu, 24 Jun 2021 18:22:37 -0400 X-MC-Unique: qXCJhEJ-Psq7XK7GtZSgxg-1 Received: by mail-oi1-f199.google.com with SMTP id n144-20020acad6960000b029023d8bff0429so896481oig.21 for ; Thu, 24 Jun 2021 15:22:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=g477a7vYcPKjBKrDFYuYGrViI13ze8NFDaxSg09H1cQ=; b=OazHg+AYyP4gghtTGwqliIgqlA2YTB8M/fxTn4jOOHCZvbCWQ8QubmPYn9CeJ3iAol c/SZ77tk16FCeQcX3ct4J2G18ssWq7/uXi2i/sOkaIRYK5imIUCccyH4LI/NwsVCNKq0 ICCTmzvlU6kR4M+nF8CDg5tody2YNNr6BrMta0rpM11uEo8cbwwSXquaH2kLlNI9pykb bkso1Gm5SJfHhMDnZ2xB33+jpaXThYXZlMk3punNl3anltwivqSszCfdQwmjRc44SbIE MUVoNHFBF43KNwl2ODoAEOOLiKrTtvf2PZCpyLrxrP1CYDgdNDNqtOxCPRvwBBrkJ4vn H2Mw== X-Gm-Message-State: AOAM533nLeTs+5jZHo4pjRI0jwtqX/gDx8DhFPdHevGOFiL4HY+hjaju rvktniiDl8YWlEaSk2m7NRjZa4J3Drfvp0BT6f2tOAz4EYEWr3andjCcme/y0kb2Hwc73thyhjl 5UQl3aQso5JWErQQ0SW7B3s+6 X-Received: by 2002:aca:3047:: with SMTP id w68mr1880002oiw.6.1624573357104; Thu, 24 Jun 2021 15:22:37 -0700 (PDT) X-Received: by 2002:aca:3047:: with SMTP id w68mr1879989oiw.6.1624573356916; Thu, 24 Jun 2021 15:22:36 -0700 (PDT) Received: from redhat.com ([198.99.80.109]) by smtp.gmail.com with ESMTPSA id s4sm502047oou.43.2021.06.24.15.22.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 24 Jun 2021 15:22:36 -0700 (PDT) Date: Thu, 24 Jun 2021 16:22:34 -0600 From: Alex Williamson To: Thomas Gleixner Cc: "Tian\, Kevin" , Jason Gunthorpe , "Dey\, Megha" , "Raj\, Ashok" , "Pan\, Jacob jun" , "Jiang\, Dave" , "Liu\, Yi L" , "Lu\, Baolu" , "Williams\, Dan J" , "Luck\, Tony" , "Kumar\, Sanjay K" , LKML , KVM , Kirti Wankhede , Peter Zijlstra , Marc Zyngier , Bjorn Helgaas Subject: Re: [PATCH] vfio/pci: Document the MSI[X] resize side effects properly Message-ID: <20210624162234.6476712c.alex.williamson@redhat.com> In-Reply-To: <87im23bh72.ffs@nanos.tec.linutronix.de> References: <20210622131217.76b28f6f.alex.williamson@redhat.com> <87o8bxcuxv.ffs@nanos.tec.linutronix.de> <20210623091935.3ab3e378.alex.williamson@redhat.com> <87mtrgatqo.ffs@nanos.tec.linutronix.de> <20210623204828.2bc7e6dc.alex.williamson@redhat.com> <87im23bh72.ffs@nanos.tec.linutronix.de> X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 24 Jun 2021 14:06:09 +0200 Thomas Gleixner wrote: > The documentation of VFIO_IRQ_INFO_NORESIZE is inaccurate as it suggests > that it is safe to dynamically add new MSI-X vectors even when > previously allocated vectors are already in use and enabled. > > Enabling additional vectors is possible according the MSI-X specification, > but the kernel does not have any mechanisms today to do that safely. > > The only available mechanism is to teardown the already active vectors > and to setup the full vector set afterwards. > > This requires to temporarily disable MSI-X which redirects any interrupt > raised by the device during this time to the legacy PCI/INTX which is > not handled and the interrupt is therefore lost. > > Update the documentation of VFIO_IRQ_INFO_NORESIZE accordingly. > > Signed-off-by: Thomas Gleixner > --- > include/uapi/linux/vfio.h | 17 +++++++++++++---- > 1 file changed, 13 insertions(+), 4 deletions(-) > --- a/include/uapi/linux/vfio.h > +++ b/include/uapi/linux/vfio.h > @@ -699,10 +699,19 @@ struct vfio_region_info_cap_nvlink2_lnks > * disabling the entire index. This is used for interrupts like PCI MSI > * and MSI-X where the driver may only use a subset of the available > * indexes, but VFIO needs to enable a specific number of vectors > - * upfront. In the case of MSI-X, where the user can enable MSI-X and > - * then add and unmask vectors, it's up to userspace to make the decision > - * whether to allocate the maximum supported number of vectors or tear > - * down setup and incrementally increase the vectors as each is enabled. > + * upfront. > + * > + * MSI cannot be resized safely when interrupts are in use already because > + * resizing requires temporary disablement of MSI for updating the relevant > + * PCI config space entries. Disabling MSI redirects an interrupt raised by > + * the device during this time to the unhandled legacy PCI/INTX, which > + * means the interrupt is lost. > + * > + * Enabling additional vectors for MSI-X is possible at least from the > + * perspective of the MSI-X specification, but not supported by the > + * exisiting PCI/MSI-X mechanisms in the kernel. The kernel provides > + * currently only a full teardown/setup cycle which requires to disable > + * MSI-X temporarily with the same side effects as for MSI. > */ > struct vfio_irq_info { > __u32 argsz; > There's good information here, but as per my other reply I think NORESIZE might be only a host implementation issue for both MSI and MSI/X. I'd also rather not focus on that existing implementation in this header, which is essentially the uAPI spec, because that implementation can change and we're unlikely to remember to update the description here. We might even be describing a device that emulates MSI/X in some way that it's not bound by this limitation. For example maybe Intel's emulation of MSI-X backed by IMS wouldn't need this flag and we could update QEMU to finally have a branch that avoids the teardown/setup. We have a flag to indicate this behavior, consequences should be relative to the presence of that flag. Finally a nit, I don't really see a strong case that the existing text is actually inaccurate or implying some safety against lost interrupts. It's actually making note of the issue here already, though the more explicit description is welcome. Thanks, Alex