Received: by 2002:a05:6a10:9848:0:0:0:0 with SMTP id x8csp1116651pxf; Fri, 26 Mar 2021 01:09:36 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwf5O7hM3Yely5jBSR4qsWeHbBd6tNgXfa4hm5Z2Qwl6EV6ItS9QeOdcxvpo2WIVDSrvwnR X-Received: by 2002:a17:907:75fc:: with SMTP id jz28mr14025230ejc.490.1616746176308; Fri, 26 Mar 2021 01:09:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1616746176; cv=none; d=google.com; s=arc-20160816; b=fkrAE0CVAlz1xa+jPw29JKcyx1wq/45w4bIuSFLcd7/BLipiprLtnL6BEQVUuSZlv3 v+zcZZpbAExw8cLDtQUxXupOZGFWyQd2SvM2JM1T5RI/NqPfasabSaSbDtOMEOrBPcEP bjnjSYGKDDikTQPPEHIdsou46WvJ8VWid3OKg1yju01PNcJ4JFB8qJpwztFjXM2XRxMW Vm7VNuw2KYa5XK3PttYHTM9bg/Kb3ExRgwXWNOf1hRv4DLiXS0JlFogSOavPZno4ntk2 5Cdjg0t++TMCxMTefe6sBB+BoaUCu1yq/a8K6Fs4aTa19CpRi4qFzZ/utYmLBa5QDWbT ZjOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-disposition:mime-version :references:message-id:subject:cc:to:from:date:dkim-signature; bh=NacPEJvwU26BHEcF1r3DKOPrdoZ0K2lnEiVQ/oV37r4=; b=eY9DKDsZBqw0R7Rxo99geLAhIv4woJBqtQDir3CKvqMFdxKBaV2c/5StXxjDP9DEMW HADh9byW1agxiFpJcqD49VG8M8j1i0+W5srgRhK0L9rO/gib4wm7aKyy0+WJQZLO82eh d3EGXXbJfeAt1N7LxP9DI74iJwOochmFycs+jwI2Eec+rsUQ29T1cL3n2hgokxXz7CHT dIag9Sd7IkFtalnQDkqVK1hVizWQGvJk+Oj4FI5of4R5r7r8dbyqBJQAB8XV56k4F3V2 hLb5b6pBXvz0zCwCTvx/45G5Bv5SZvODhrHTGMI6tWcqrJGLAdS1NqGx7h91XU9G2yEc Y+gA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=kus60mso; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id h26si6369895edv.198.2021.03.26.01.09.10; Fri, 26 Mar 2021 01:09:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=kus60mso; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229933AbhCZIHL (ORCPT + 99 others); Fri, 26 Mar 2021 04:07:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53460 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229812AbhCZIHC (ORCPT ); Fri, 26 Mar 2021 04:07:02 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6029CC0613AA for ; Fri, 26 Mar 2021 01:07:02 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id z2so4721922wrl.5 for ; Fri, 26 Mar 2021 01:07:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=NacPEJvwU26BHEcF1r3DKOPrdoZ0K2lnEiVQ/oV37r4=; b=kus60msoSJlwyU1GomSA58McYHQudRjt8lZoUOVyLCyxExYPuQiC/FgDUjEDEdiFJH OtPkVS983YcgkVOZ3NbX19SKHfGojdxWiV+mZsVTfFS3PH2ZKq07YDEns8DQHizqSqjj kAlYqYai0ZCtnRvwuakJexNRFQxOeUsmhfllUZTGd3aMTne5GjehFbclbRF9zPTDvxfr 7mUrcCG9VaWh1DWLhfC1B0KnAzLxhwQqcmOk5cN/v/L72Qlm6lRW3AGNrgKkaY+vlaHX i+2MikP0ctvy5L5KCWHU8Q958BKQKlSB4DTgMhSfZv+A2P690PsjqM55+5z0YJQ2NIsx 6Bpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=NacPEJvwU26BHEcF1r3DKOPrdoZ0K2lnEiVQ/oV37r4=; b=W3ZooOr3j7200zg+dlIv0rwhJYf/g82a+YvE6AEPvPY/wt32SgL0CMDN3+LH4K90bk dMdaixb5Q6fHz/QOje/EL82++X/QAQWnA+odm9YwLtEzzo5zMpnokqpIYeznw1T1/52b BkXFQZw6KHzqBpKr9sbQ9KnHemdRsv119oNBF5j97qOKxMyeWZcdMNyYejfmr3+Kk0N5 4SQq5QZe+PxPbIWJp7KsKGfgOOAntiSvSUIEjNZnMF2vhs3Pdb/XjfPEK9NNa9yQUILh uq0nPFAKKaIyI5xRK9Zu4PvEPw6rXY7YeDh62GsnLLOIP60X8psOFOwhjRvoEcTjwxz1 HjQA== X-Gm-Message-State: AOAM530loWydpCVlCtMx3iAS0D/yQ6QIxwUTSDdd0zqcWxEVklMdwQxL p1C+33vX5JYgjti4LB0SJQ+5bjKzxGMc+A== X-Received: by 2002:a5d:591a:: with SMTP id v26mr12982847wrd.172.1616746021071; Fri, 26 Mar 2021 01:07:01 -0700 (PDT) Received: from myrica ([2001:1715:4e26:a7e0:116c:c27a:3e7f:5eaf]) by smtp.gmail.com with ESMTPSA id 7sm9654955wmk.8.2021.03.26.01.06.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 26 Mar 2021 01:07:00 -0700 (PDT) Date: Fri, 26 Mar 2021 09:06:42 +0100 From: Jean-Philippe Brucker To: Jason Gunthorpe Cc: Jacob Pan , LKML , Joerg Roedel , Lu Baolu , David Woodhouse , iommu@lists.linux-foundation.org, cgroups@vger.kernel.org, Tejun Heo , Li Zefan , Johannes Weiner , Jean-Philippe Brucker , Alex Williamson , Eric Auger , Jonathan Corbet , Raj Ashok , "Tian, Kevin" , Yi Liu , Wu Hao , Dave Jiang Subject: Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs Message-ID: References: <20210319135432.GT2356281@nvidia.com> <20210319112221.5123b984@jacob-builder> <20210324100246.4e6b8aa1@jacob-builder> <20210324170338.GM2356281@nvidia.com> <20210324151230.466fd47a@jacob-builder> <20210325100236.17241a1c@jacob-builder> <20210325171645.GF2356281@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20210325171645.GF2356281@nvidia.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Mar 25, 2021 at 02:16:45PM -0300, Jason Gunthorpe wrote: > On Thu, Mar 25, 2021 at 10:02:36AM -0700, Jacob Pan wrote: > > Hi Jean-Philippe, > > > > On Thu, 25 Mar 2021 11:21:40 +0100, Jean-Philippe Brucker > > wrote: > > > > > On Wed, Mar 24, 2021 at 03:12:30PM -0700, Jacob Pan wrote: > > > > Hi Jason, > > > > > > > > On Wed, 24 Mar 2021 14:03:38 -0300, Jason Gunthorpe > > > > wrote: > > > > > On Wed, Mar 24, 2021 at 10:02:46AM -0700, Jacob Pan wrote: > > > > > > > Also wondering about device driver allocating auxiliary domains > > > > > > > for their private use, to do iommu_map/unmap on private PASIDs (a > > > > > > > clean replacement to super SVA, for example). Would that go > > > > > > > through the same path as /dev/ioasid and use the cgroup of > > > > > > > current task? > > > > > > > > > > > > For the in-kernel private use, I don't think we should restrict > > > > > > based on cgroup, since there is no affinity to user processes. I > > > > > > also think the PASID allocation should just use kernel API instead > > > > > > of /dev/ioasid. Why would user space need to know the actual PASID > > > > > > # for device private domains? Maybe I missed your idea? > > > > > > > > > > There is not much in the kernel that isn't triggered by a process, I > > > > > would be careful about the idea that there is a class of users that > > > > > can consume a cgroup controlled resource without being inside the > > > > > cgroup. > > > > > > > > > > We've got into trouble before overlooking this and with something > > > > > greenfield like PASID it would be best built in to the API to prevent > > > > > a mistake. eg accepting a cgroup or process input to the allocator. > > > > > > > > > Make sense. But I think we only allow charging the current cgroup, how > > > > about I add the following to ioasid_alloc(): > > > > > > > > misc_cg = get_current_misc_cg(); > > > > ret = misc_cg_try_charge(MISC_CG_RES_IOASID, misc_cg, 1); > > > > if (ret) { > > > > put_misc_cg(misc_cg); > > > > return ret; > > > > } > > > > > > Does that allow PASID allocation during driver probe, in kernel_init or > > > modprobe context? > > > > > Good point. Yes, you can get cgroup subsystem state in kernel_init for > > charging/uncharging. I would think module_init should work also since it is > > after kernel_init. I have tried the following: > > static int __ref kernel_init(void *unused) > > { > > int ret; > > + struct cgroup_subsys_state *css; > > + css = task_get_css(current, pids_cgrp_id); > > > > But that would imply: > > 1. IOASID has to be built-in, not as module If IOASID is a module, the device driver will probe once the IOMMU module is available, which I think always happens in probe deferral kworker. > > 2. IOASIDs charged on PID1/init would not subject to cgroup limit since it > > will be in the root cgroup and we don't support migration nor will migrate. > > > > Then it comes back to the question of why do we try to limit in-kernel > > users per cgroup if we can't enforce these cases. It may be better to explicitly pass a cgroup during allocation as Jason suggested. That way anyone using the API will have to be aware of this and pass the root cgroup if that's what they want. > Are these real use cases? Why would a driver binding to a device > create a single kernel pasid at bind time? Why wouldn't it use > untagged DMA? It's not inconceivable to have a control queue doing DMA tagged with PASID. The devices I know either use untagged DMA, or have a choice to use a PASID. We're not outright forbidding PASID allocation at boot (I don't think we can or should) and we won't be able to check every use of the API, so I'm trying to figure out whether it will always default to root cgroup, or crash in some corner case. Thanks, Jean