Received-SPF: pass (google.com: domain of linux-kernel+bounces-54781-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161;
Message-ID: <0d1f81a8-999e-459d-9e5c-7efff665646e@arm.com>
Date: Tue, 6 Feb 2024 11:32:17 +0000
Precedence: bulk
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v3 3/3] iommu/iova: use named kmem_cache for iova
 magazines
Content-Language: en-GB
To: John Garry <john.g.garry@oracle.com>, joro@8bytes.org
Cc: will@kernel.org, pasha.tatashin@soleen.com, iommu@lists.linux.dev,
 linux-kernel@vger.kernel.org, linux-mm@kvack.org, rientjes@google.com,
 yosryahmed@google.com
References: <cover.1707144953.git.robin.murphy@arm.com>
 <dc5c51aaba50906a92b9ba1a5137ed462484a7be.1707144953.git.robin.murphy@arm.com>
 <83a0aafc-8a9c-48ce-a203-96b901489d41@oracle.com>
From: Robin Murphy <robin.murphy@arm.com>
In-Reply-To: <83a0aafc-8a9c-48ce-a203-96b901489d41@oracle.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

On 06/02/2024 11:24 am, John Garry wrote:
> On 05/02/2024 15:32, Robin Murphy wrote:
>> From: Pasha Tatashin <pasha.tatashin@soleen.com>
>>
>> The magazine buffers can take gigabytes of kmem memory, dominating all
>> other allocations. For observability purpose create named slab cache so
>> the iova magazine memory overhead can be clearly observed.
>>
>> With this change:
>>
>>> slabtop -o | head
>>   Active / Total Objects (% used)    : 869731 / 952904 (91.3%)
>>   Active / Total Slabs (% used)      : 103411 / 103974 (99.5%)
>>   Active / Total Caches (% used)     : 135 / 211 (64.0%)
>>   Active / Total Size (% used)       : 395389.68K / 411430.20K (96.1%)
>>   Minimum / Average / Maximum Object : 0.02K / 0.43K / 8.00K
>>
>> OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
>> 244412 244239 99%    1.00K  61103       4    244412K iommu_iova_magazine
>>   91636  88343 96%    0.03K    739     124      2956K kmalloc-32
>>   75744  74844 98%    0.12K   2367      32      9468K kernfs_node_cache
>>
>> On this machine it is now clear that magazine use 242M of kmem memory.
> 
> Those caches could do with a trimming ...

See the discussion on v1 for more details:

https://lore.kernel.org/linux-iommu/20240201193014.2785570-1-tatashin@google.com/

but it seems this really is the idle baseline for lots of CPUs * lots of 
domains - if all those devices get going in anger it's likely that 
combined iova + iova_magazine usage truly will blow up into gigabytes.

Cheers,
Robin.

> 
>>
>> Acked-by: David Rientjes <rientjes@google.com>
>> Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
>> [ rm: adjust to rework of iova_cache_{get,put} ]
>> Signed-off-by: Robin Murphy <robin.murphy@arm.com>
>> ---
> 
> FWIW:
> Reviewed-by: John Garry <john.g.garry@oracle.com>