Received: by 2002:a25:e74b:0:0:0:0:0 with SMTP id e72csp1160945ybh; Thu, 23 Jul 2020 01:52:26 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyQUPy/W8MX+7POy8plQAqi7ovz0aTdVDSvgw2PJdglWHydJQNKx0XtFpTwAk5hJ/tcyjt5 X-Received: by 2002:a17:906:edb3:: with SMTP id sa19mr3272886ejb.21.1595494345924; Thu, 23 Jul 2020 01:52:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1595494345; cv=none; d=google.com; s=arc-20160816; b=TWGjF7cPSh8DBwPT7DQnLB5/lFgGiistAmN902kahl4+qh6I2tPSOELvqwSHxeoj2f xKFFPc1AOtCsJ7ZWndgw8yvD9chMWtNVpipcJro7tQNXUyvG1y9kUafEYTlSEHkd5nQ3 Oark3/W3EzQEpf39T3gHOORhJESB/irE/A5mYycjuL3vCfq2aTRu9CkNrc/+T4l/R0Z1 B40hWlXZVb083fXMXbE0k3yZZb28ZLSmCPlzUCSrKcMZvXMVsAMJcJjUGvBUcint9Xjs 1pjt24Zo0JnAdaTP6Su5sYkwImgc69d8Ruvpk1MmYoNS4b51Nq9SNa+KfR9HzYiylMnl XSQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:message-id:user-agent:references :in-reply-to:subject:cc:to:from:date:content-transfer-encoding :mime-version:dkim-signature; bh=52YeuqC1ikzs4HIfZdiVIryWLV4T4w6LMauHcDumagc=; b=zet1/9L09rM85FpC9gtWgouunAzwj1wDUbja564dObw0PZxUZjoRx7hLltzoUavR4Q hfFzaZUdxYnvjMznmdiTg8OERlqY1L1I+fPvIkQUx+PUwsROlo6QmNcdu5WD09ZqpjGR 5Mofj/MvJY3g6MrzQPkUpHFvcWV15PdpEa+B9cbxKa5sg1a1l0AamVHTfMv8vDOs6bzV HtUMHb8i/BMMldlstOLqNB6Q5lJcfQ83CjkzArEEbsDNqfhC3abMIJu8wTeHuvMM4toY e2JM9HVXPtbkPU0e6Qp5OHL9+Xxt/zbYJNQhublnEHHOA2jdiZwRzHgIP9eCEs9NPeeg aOKQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=JoLsejGO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id t2si812821edw.603.2020.07.23.01.52.03; Thu, 23 Jul 2020 01:52:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=JoLsejGO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727120AbgGWIv4 (ORCPT + 99 others); Thu, 23 Jul 2020 04:51:56 -0400 Received: from mail.kernel.org ([198.145.29.99]:46020 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725858AbgGWIvz (ORCPT ); Thu, 23 Jul 2020 04:51:55 -0400 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5B3B72071A; Thu, 23 Jul 2020 08:51:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1595494314; bh=IRF2/hQ81/fJN2jLe+emwgKLQGhbq9l8YIf8OxkkSik=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=JoLsejGOUeUv83nQAMznfUoFwoaNSxjbwr27sVlLosQjoq8SM2k7dgTUgxUvKR+8r zYhfXQFkgR18bjjIH+98j5xi2EK4WyKJq63OkADl9v+HxRRQj9Pgwx7OxMUDW18eS2 5B4NMpR85PDvoLQsxH3EtQR8ml/ZXD0/LjkdFNnk= Received: from disco-boy.misterjones.org ([51.254.78.96] helo=www.loen.fr) by disco-boy.misterjones.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1jyWxE-00EDOw-Q3; Thu, 23 Jul 2020 09:51:52 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Thu, 23 Jul 2020 09:51:52 +0100 From: Marc Zyngier To: Jason Gunthorpe Cc: Dave Jiang , vkoul@kernel.org, megha.dey@intel.com, bhelgaas@google.com, rafael@kernel.org, gregkh@linuxfoundation.org, tglx@linutronix.de, hpa@zytor.com, alex.williamson@redhat.com, jacob.jun.pan@intel.com, ashok.raj@intel.com, yi.l.liu@intel.com, baolu.lu@intel.com, kevin.tian@intel.com, sanjay.k.kumar@intel.com, tony.luck@intel.com, jing.lin@intel.com, dan.j.williams@intel.com, kwankhede@nvidia.com, eric.auger@redhat.com, parav@mellanox.com, dave.hansen@intel.com, netanelg@mellanox.com, shahafs@mellanox.com, yan.y.zhao@linux.intel.com, pbonzini@redhat.com, samuel.ortiz@intel.com, mona.hossain@intel.com, dmaengine@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org Subject: Re: [PATCH RFC v2 02/18] irq/dev-msi: Add support for a new DEV_MSI irq domain In-Reply-To: <20200722195928.GN2021248@mellanox.com> References: <159534667974.28840.2045034360240786644.stgit@djiang5-desk3.ch.intel.com> <159534734833.28840.10067945890695808535.stgit@djiang5-desk3.ch.intel.com> <878sfbxtzi.wl-maz@kernel.org> <20200722195928.GN2021248@mellanox.com> User-Agent: Roundcube Webmail/1.4.5 Message-ID: X-Sender: maz@kernel.org X-SA-Exim-Connect-IP: 51.254.78.96 X-SA-Exim-Rcpt-To: jgg@mellanox.com, dave.jiang@intel.com, vkoul@kernel.org, megha.dey@intel.com, bhelgaas@google.com, rafael@kernel.org, gregkh@linuxfoundation.org, tglx@linutronix.de, hpa@zytor.com, alex.williamson@redhat.com, jacob.jun.pan@intel.com, ashok.raj@intel.com, yi.l.liu@intel.com, baolu.lu@intel.com, kevin.tian@intel.com, sanjay.k.kumar@intel.com, tony.luck@intel.com, jing.lin@intel.com, dan.j.williams@intel.com, kwankhede@nvidia.com, eric.auger@redhat.com, parav@mellanox.com, dave.hansen@intel.com, netanelg@mellanox.com, shahafs@mellanox.com, yan.y.zhao@linux.intel.com, pbonzini@redhat.com, samuel.ortiz@intel.com, mona.hossain@intel.com, dmaengine@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2020-07-22 20:59, Jason Gunthorpe wrote: > On Wed, Jul 22, 2020 at 07:52:33PM +0100, Marc Zyngier wrote: > >> Which is exactly what platform-MSI already does. Why do we need >> something else? > > It looks to me like all the code is around managing the > dev->msi_domain of the devices. > > The intended use would have PCI drivers create children devices using > mdev or virtbus and those devices wouldn't have a msi_domain from the > platform. Looks like platform_msi_alloc_priv_data() fails immediately > because dev->msi_domain will be NULL for these kinds of devices. > > Maybe that issue should be handled directly instead of wrappering > platform_msi_*? > > For instance a trivial addition to the platform_msi API: > > platform_msi_assign_domain(struct_device > *newly_created_virtual_device, > struct device *physical_device); > > Which could set the msi_domain of new device using the topology of > physical_device to deduce the correct domain? That would seem like a sensible course of action, as losing the topology information will likely result in problems down the line. > Then the question is how to properly create a domain within the > hardware topology of physical_device with the correct parameters for > the platform. > > Why do we need a dummy msi_domain anyhow? Can this just use > physical_device->msi_domain directly? (I'm at my limit here of how > much of this I remember, sorry) The parent device would be a PCI device, if I follow you correctly. It would thus expect to be able to program the MSI the PCI way, which wouldn't work. So we end-up with this custom MSI domain that knows about *this* particular family of devices. > If you solve that it should solve the remapping problem too, as the > physical_device is already assigned by the platform to a remapping irq > domain if that is what the platform wants. > >>> + parent = irq_get_default_host(); >> Really? How is it going to work once you have devices sending their >> MSIs to two different downstream blocks? This looks rather >> short-sighted. > > .. and fix this too, the parent domain should be derived from the > topology of the physical_device which is originating the interrupt > messages. > >> On the other hand, masking an interrupt is an irqchip operation, and >> only concerns the irqchip level. Here, you seem to be making it an >> end-point operation, which doesn't really make sense to me. Or is this >> device its own interrupt controller as well? That would be extremely >> surprising, and I'd expect some block downstream of the device to be >> able to control the masking of the interrupt. > > These are message interrupts so they originate directly from the > device and generally travel directly to the CPU APIC. On the wire > there is no difference between a MSI, MSI-X and a device using the > dev-msi approach. I understand that. > IIRC on Intel/AMD at least once a MSI is launched it is not maskable. Really? So you can't shut a device with a screaming interrupt, for example, should it become otherwise unresponsive? > So the model for MSI is always "mask at source". The closest mapping > to the Linux IRQ model is to say the end device has a irqchip that > encapsulates the ability of the device to generate the MSI in the > first place. This is an x86'ism, I'm afraid. Systems I deal with can mask any interrupt at the interrupt controller level, MSI or not. > It looks like existing platform_msi drivers deal with "masking" > implicitly by halting the device interrupt generation before releasing > the interrupt and have no way for the generic irqchip layer to mask > the interrupt. No. As I said above, the interrupt controller is perfectly capable of masking interrupts on its own, without touching the device. > I suppose the motivation to make it explicit is related to vfio using > the generic mask/unmask functionality? > > Explicit seems better, IMHO. If masking at the source is the only way to shut the device up, and assuming that the device provides the expected semantics (a MSI raised by the device while the interrupt is masked isn't lost and gets sent when unmasked), that's fair enough. It's just ugly. Thanks, M. -- Jazz is not dead. It just smells funny...