Received: by 2002:a05:7412:419a:b0:f3:1519:9f41 with SMTP id i26csp4186741rdh; Tue, 28 Nov 2023 14:32:40 -0800 (PST) X-Google-Smtp-Source: AGHT+IG60uJ45y+Z/dIjLuPEgrm0Xa/kvD9lnh5GUKijVKLLxOVIhaJLHgcWTqLyvCx8fOwFNvJq X-Received: by 2002:a05:6a00:23cc:b0:6cd:8a19:c324 with SMTP id g12-20020a056a0023cc00b006cd8a19c324mr11767144pfc.3.1701210760645; Tue, 28 Nov 2023 14:32:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701210760; cv=none; d=google.com; s=arc-20160816; b=mJB4hT61UG4jSPjsCBdE3CBi96ia9jJz6e61lIIEg784FRo/D6/Avs5LjRpMgPT8/F K5ys2w8soyBXiNjPj6b7dZ5IuWm77P7wFFLNwkWa5fJ090lEA4HnxF+fpQ34SQ0N6R6u nOLZHwj58ZylVXPdtWO4uXZanAdIy3cLsR32XkP8+RhXjlJCb0u1k8GtM3WIi0FSlOjI QQhQHWRFA9lSnkdoMZYECPzj+bvw6y+lAb1lhahS+RuLPhtYIAREe5bdEVPOPo1h7ykg AIvil4GsX0RYg3/R6p5PnmmyPYIWOW7MTVkcxwi51I1YkIfMCqEkY5IYCWBI+wOkVe9F 34oA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:cc:to:subject :message-id:date:from:in-reply-to:references:mime-version :dkim-signature; bh=zXmxZJr3Rk1RKmtjvCLdJSQvKQY1rkK5dlcrmMuZJj4=; fh=YH4gGHNWntA9iCM+7nXsk9GsZliLYvjJdNUz8c+LV/U=; b=qvZKNvsmtoeDGMsW99aaz1tYbjh8UqmqxE5WorkKlx4Fbn1Envim9qNlW1kMCDjoDI UpxMJmI74hja6Hh7l6fbmm8v9Z9U6dAFsGkclDUgseGXAa9+uogmWgUkDSE4v87EKleZ 70meCdQxfYcfZM0POWmhvg6yHzkOL9oLwKjxjKVJ5q6MUSIv3h1+7bdqTsnVwCqeZkcn oTnQrsd2PmA2PWUD6vXBdHNsGdauFS+HUJlCK75mtAw2rnuW8pw5RMekBsbS2ZpVajlu r8NoRbPpNz2Ddg20nzCnEwObxXpbfa0SSEfWi76D2cj+2ICiY4aaEGS7J7QNiTnKxPAO FD2g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@soleen.com header.s=google header.b=hNWTXija; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id bm6-20020a656e86000000b00584e65212d4si12828606pgb.609.2023.11.28.14.32.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Nov 2023 14:32:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@soleen.com header.s=google header.b=hNWTXija; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 5D84380BC44F; Tue, 28 Nov 2023 14:32:39 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229927AbjK1Wc3 (ORCPT + 99 others); Tue, 28 Nov 2023 17:32:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53110 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229727AbjK1Wc2 (ORCPT ); Tue, 28 Nov 2023 17:32:28 -0500 Received: from mail-qt1-x836.google.com (mail-qt1-x836.google.com [IPv6:2607:f8b0:4864:20::836]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9BA02D63 for ; Tue, 28 Nov 2023 14:32:32 -0800 (PST) Received: by mail-qt1-x836.google.com with SMTP id d75a77b69052e-41cba6e8e65so31401721cf.2 for ; Tue, 28 Nov 2023 14:32:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; t=1701210752; x=1701815552; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=zXmxZJr3Rk1RKmtjvCLdJSQvKQY1rkK5dlcrmMuZJj4=; b=hNWTXijaF5/80f98cfTTnZ6ba6x+wEh/fdlFcwentxd4sua8os6P5dQ+5lDh2JuOVi fo99gFkEM7suv6QillI1aIsoJ9RZOdg85ps38EfUX9GFJhacXnbcJn1SbeG8HVVZGGgi 38G5Rx8UvoBp0E0sMqEtmYlcJL8f/12m3CVRkWg1pz7Axi+u0wVdTH9Tn8BEIcoj9DlI RrSu/LjDWxpieVNLwyqWcRXA64/2tv0Nq6QaON0bDyn0Eu9LiN6qFO6a5YuGTvNqvqyp d02JpkE+A2fY8H6Uo0+kbZm132ILbWyfEidOp3OEGoqFW1B8P6zgWLCLJ4EMeRT9JW3O x3PQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701210752; x=1701815552; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zXmxZJr3Rk1RKmtjvCLdJSQvKQY1rkK5dlcrmMuZJj4=; b=USqhnur3il7/e6Ua19IE4+sYXs7yf5YSUv/JrgN09TXBw7i9eLkGFd9wPHRPIflgAs DI/lGPo1wtvPOMar1gslTx6yub3fEnL9BOfnA0/64WZ/k/bCpoGYwpNwPo0thsdbTeOI GqJf1sfIsvKmQ3FVTj0DqWjWdh7uCrH5TUBZQb5PgtYG9uMtN0wviIYhVE/8U92tFuIE kE+9f1KvsWBjj1LzOo2S+C7TCrisUYxivoDKONOA7enNXpat9iCgKsH7GhtM6Ma5nV2V bHBSpWxVevIwLAC21i6LUe01R67bldbzVanpZ7p3WhbTZKve5awdx2VUDv7PjVFsmyAY bzLQ== X-Gm-Message-State: AOJu0YywcgV9GG9X9SVAkiDJWu+YEScgrvfmu2fF90VGuhWtHTfcSu3C 6hspBePcT1egvpSaNau2ciL7ddDDUnBoe95bd0SqcQ== X-Received: by 2002:a05:622a:5da6:b0:423:a0e1:c58f with SMTP id fu38-20020a05622a5da600b00423a0e1c58fmr10804025qtb.59.1701210751688; Tue, 28 Nov 2023 14:32:31 -0800 (PST) MIME-Version: 1.0 References: <20231128204938.1453583-1-pasha.tatashin@soleen.com> In-Reply-To: From: Pasha Tatashin Date: Tue, 28 Nov 2023 17:31:54 -0500 Message-ID: Subject: Re: [PATCH 00/16] IOMMU memory observability To: Yosry Ahmed Cc: akpm@linux-foundation.org, alex.williamson@redhat.com, alim.akhtar@samsung.com, alyssa@rosenzweig.io, asahi@lists.linux.dev, baolu.lu@linux.intel.com, bhelgaas@google.com, cgroups@vger.kernel.org, corbet@lwn.net, david@redhat.com, dwmw2@infradead.org, hannes@cmpxchg.org, heiko@sntech.de, iommu@lists.linux.dev, jasowang@redhat.com, jernej.skrabec@gmail.com, jgg@ziepe.ca, jonathanh@nvidia.com, joro@8bytes.org, kevin.tian@intel.com, krzysztof.kozlowski@linaro.org, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-rockchip@lists.infradead.org, linux-samsung-soc@vger.kernel.org, linux-sunxi@lists.linux.dev, linux-tegra@vger.kernel.org, lizefan.x@bytedance.com, marcan@marcan.st, mhiramat@kernel.org, mst@redhat.com, m.szyprowski@samsung.com, netdev@vger.kernel.org, paulmck@kernel.org, rdunlap@infradead.org, robin.murphy@arm.com, samuel@sholland.org, suravee.suthikulpanit@amd.com, sven@svenpeter.dev, thierry.reding@gmail.com, tj@kernel.org, tomas.mudrunka@gmail.com, vdumpa@nvidia.com, virtualization@lists.linux.dev, wens@csie.org, will@kernel.org, yu-cheng.yu@intel.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Tue, 28 Nov 2023 14:32:39 -0800 (PST) On Tue, Nov 28, 2023 at 4:34=E2=80=AFPM Yosry Ahmed = wrote: > > On Tue, Nov 28, 2023 at 12:49=E2=80=AFPM Pasha Tatashin > wrote: > > > > From: Pasha Tatashin > > > > IOMMU subsystem may contain state that is in gigabytes. Majority of tha= t > > state is iommu page tables. Yet, there is currently, no way to observe > > how much memory is actually used by the iommu subsystem. > > > > This patch series solves this problem by adding both observability to > > all pages that are allocated by IOMMU, and also accountability, so > > admins can limit the amount if via cgroups. > > > > The system-wide observability is using /proc/meminfo: > > SecPageTables: 438176 kB > > > > Contains IOMMU and KVM memory. > > > > Per-node observability: > > /sys/devices/system/node/nodeN/meminfo > > Node N SecPageTables: 422204 kB > > > > Contains IOMMU and KVM memory memory in the given NUMA node. > > > > Per-node IOMMU only observability: > > /sys/devices/system/node/nodeN/vmstat > > nr_iommu_pages 105555 > > > > Contains number of pages IOMMU allocated in the given node. > > Does it make sense to have a KVM-only entry there as well? > > In that case, if SecPageTables in /proc/meminfo is found to be > suspiciously high, it should be easy to tell which component is > contributing most usage through vmstat. I understand that users can do > the subtraction, but we wouldn't want userspace depending on that, in > case a third class of "secondary" page tables emerges that we want to > add to SecPageTables. The in-kernel implementation can do the > subtraction for now if it makes sense though. Hi Yosry, Yes, another counter for KVM could be added. On the other hand KVM only can be computed by subtracting one from another as there are only two types of secondary page tables, KVM and IOMMU: /sys/devices/system/node/node0/meminfo Node 0 SecPageTables: 422204 kB /sys/devices/system/node/nodeN/vmstat nr_iommu_pages 105555 KVM only =3D SecPageTables - nr_iommu_pages * PAGE_SIZE / 1024 Pasha > > > > > Accountability: using sec_pagetables cgroup-v2 memory.stat entry. > > > > With the change, iova_stress[1] stops as limit is reached: > > > > # ./iova_stress > > iova space: 0T free memory: 497G > > iova space: 1T free memory: 495G > > iova space: 2T free memory: 493G > > iova space: 3T free memory: 491G > > > > stops as limit is reached. > > > > This series encorporates suggestions that came from the discussion > > at LPC [2]. > > > > [1] https://github.com/soleen/iova_stress > > [2] https://lpc.events/event/17/contributions/1466 > > > > Pasha Tatashin (16): > > iommu/vt-d: add wrapper functions for page allocations > > iommu/amd: use page allocation function provided by iommu-pages.h > > iommu/io-pgtable-arm: use page allocation function provided by > > iommu-pages.h > > iommu/io-pgtable-dart: use page allocation function provided by > > iommu-pages.h > > iommu/io-pgtable-arm-v7s: use page allocation function provided by > > iommu-pages.h > > iommu/dma: use page allocation function provided by iommu-pages.h > > iommu/exynos: use page allocation function provided by iommu-pages.h > > iommu/fsl: use page allocation function provided by iommu-pages.h > > iommu/iommufd: use page allocation function provided by iommu-pages.h > > iommu/rockchip: use page allocation function provided by iommu-pages.= h > > iommu/sun50i: use page allocation function provided by iommu-pages.h > > iommu/tegra-smmu: use page allocation function provided by > > iommu-pages.h > > iommu: observability of the IOMMU allocations > > iommu: account IOMMU allocated memory > > vhost-vdpa: account iommu allocations > > vfio: account iommu allocations > > > > Documentation/admin-guide/cgroup-v2.rst | 2 +- > > Documentation/filesystems/proc.rst | 4 +- > > drivers/iommu/amd/amd_iommu.h | 8 - > > drivers/iommu/amd/init.c | 91 +++++----- > > drivers/iommu/amd/io_pgtable.c | 13 +- > > drivers/iommu/amd/io_pgtable_v2.c | 20 +- > > drivers/iommu/amd/iommu.c | 13 +- > > drivers/iommu/dma-iommu.c | 8 +- > > drivers/iommu/exynos-iommu.c | 14 +- > > drivers/iommu/fsl_pamu.c | 5 +- > > drivers/iommu/intel/dmar.c | 10 +- > > drivers/iommu/intel/iommu.c | 47 ++--- > > drivers/iommu/intel/iommu.h | 2 - > > drivers/iommu/intel/irq_remapping.c | 10 +- > > drivers/iommu/intel/pasid.c | 12 +- > > drivers/iommu/intel/svm.c | 7 +- > > drivers/iommu/io-pgtable-arm-v7s.c | 9 +- > > drivers/iommu/io-pgtable-arm.c | 7 +- > > drivers/iommu/io-pgtable-dart.c | 37 ++-- > > drivers/iommu/iommu-pages.h | 231 ++++++++++++++++++++++++ > > drivers/iommu/iommufd/iova_bitmap.c | 6 +- > > drivers/iommu/rockchip-iommu.c | 14 +- > > drivers/iommu/sun50i-iommu.c | 7 +- > > drivers/iommu/tegra-smmu.c | 18 +- > > drivers/vfio/vfio_iommu_type1.c | 8 +- > > drivers/vhost/vdpa.c | 3 +- > > include/linux/mmzone.h | 5 +- > > mm/vmstat.c | 3 + > > 28 files changed, 415 insertions(+), 199 deletions(-) > > create mode 100644 drivers/iommu/iommu-pages.h > > > > -- > > 2.43.0.rc2.451.g8631bc7472-goog > > > >