Received-SPF: pass (google.com: domain of linux-kernel+bounces-50325-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1;
Message-ID: <bcddef81-3979-4ad0-b3f6-771871a44433@arm.com>
Date: Fri, 2 Feb 2024 18:27:50 +0000
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH] iommu/iova: use named kmem_cache for iova magazines
Content-Language: en-GB
To: Pasha Tatashin <pasha.tatashin@soleen.com>
Cc: joro@8bytes.org, will@kernel.org, iommu@lists.linux.dev,
 linux-kernel@vger.kernel.org, linux-mm@kvack.org, rientjes@google.com
References: <20240201193014.2785570-1-tatashin@google.com>
 <02610629-05ef-4956-a122-36b6ac98fbc2@arm.com>
 <CA+CK2bD_q3pnThtLVSzFCjyevBEaG6Ad+1o2=1tVZsYg35UMmg@mail.gmail.com>
 <84c7e816-f749-48d8-a429-8b0ef799cdbb@arm.com>
 <CA+CK2bCFbeRHT172KUci5+8WJNUeo-2A1M=UXsSqqOy64w1LJw@mail.gmail.com>
From: Robin Murphy <robin.murphy@arm.com>
In-Reply-To: <CA+CK2bCFbeRHT172KUci5+8WJNUeo-2A1M=UXsSqqOy64w1LJw@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 02/02/2024 6:04 pm, Pasha Tatashin wrote:
>> Hmm, I did misspeak slightly (it's late and I really should have left
>> this for tomorrow...) - that's 12KB per CPU *per domain*, but still that
>> would seem to imply well over 100 domains if you have 242MB of magazine
>> allocations while the iommu_iova cache isn't even on the charts... what
>> the heck is that driver doing?
>>
>> (I don't necessarily disagree with the spirit of the patch BTW, I just
>> really want to understand the situation that prompted it, and make sure
>> we don't actually have a subtle leak somewhere.)
> 
> Hi Robin,
> 
> The following tracing is without Google TPU, simply upstream kernel:
> 
> The iova_domain_init_rcaches is called 159 with the following stack:
> 
>   iova_domain_init_rcaches
>   iommu_setup_dma_ops
>   amd_iommu_probe_finalize
>   bus_iommu_probe
>   iommu_device_register
>   iommu_init_pci
>   amd_iommu_init_pci
>   state_next
>   iommu_go_to_state
>   amd_iommu_init
>   pci_iommu_init
>   do_one_initcall
> 
> Each time 1536K is allocated: in total 159 * 1536K = 238.5M

Yikes, so it really does just have that many IOMMU groups? OK, fair 
enough, call me convinced :)

On balance though, I think I'd prefer to just stick the lifecycle 
management into iova_cache_{get,put} for simplicity - spending ~256 
bytes on another kmem_cache we might not use can hardly be significantly 
more than the extra code and static data necessary to track its usage 
separately anyway.

Thanks,
Robin.

> The allocation happens like this:
> for (IOVA_RANGE_CACHE_MAX_SIZE)
>      for_each_possible_cpu()
>          iova_magazine_alloc
>          iova_magazine_alloc
> 
> IOVA_RANGE_CACHE_MAX_SIZE = 6
> ncpu = 128
> sizeof (struct iova_magazine) = 1K
> 
> 6 * 128 * (1K + 1K) = 1536K
> 
> Pasha