Received-SPF: pass (google.com: domain of linux-kernel+bounces-54762-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1;
Message-ID: <83a0aafc-8a9c-48ce-a203-96b901489d41@oracle.com>
Date: Tue, 6 Feb 2024 11:24:05 +0000
User-Agent: Mozilla Thunderbird
Subject: Re: [PATCH v3 3/3] iommu/iova: use named kmem_cache for iova
 magazines
Content-Language: en-US
To: Robin Murphy <robin.murphy@arm.com>, joro@8bytes.org
Cc: will@kernel.org, pasha.tatashin@soleen.com, iommu@lists.linux.dev,
        linux-kernel@vger.kernel.org, linux-mm@kvack.org, rientjes@google.com,
        yosryahmed@google.com
References: <cover.1707144953.git.robin.murphy@arm.com>
 <dc5c51aaba50906a92b9ba1a5137ed462484a7be.1707144953.git.robin.murphy@arm.com>
From: John Garry <john.g.garry@oracle.com>
Organization: Oracle Corporation
In-Reply-To: <dc5c51aaba50906a92b9ba1a5137ed462484a7be.1707144953.git.robin.murphy@arm.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Precedence: bulk
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: 
	=?utf-8?B?OEk2d09uRk8vN0I2U0RmZnFlWWk4aUMxTHJZS0tXdlNBTDZxRG0zYlNZSVNj?=
 =?utf-8?B?ZUlkMGVoMU5QbkFNSUVoaGhiUGpYTmY1bjdrVHdoNEZYbSs4Mi9QSnVoYnlz?=
 =?utf-8?B?TlNOYjZzWWRzTDJlWXMyeWtlRDdQK2haSTV4S1J2S0k3cTNtc3kreU9ucFU5?=
 =?utf-8?B?ZzI4bXZ0VHkwSHpqdUY4TkhrdUQvYjgvd2JKV0JVOEhyMXpGSUIvUDBDQVVl?=
 =?utf-8?B?WTRsMzAxQ0FMczN1ZFB5d2pPSUVhclcydlhKb1BJaHdZMC81OW82TFVCd0hj?=
 =?utf-8?B?RHhyTDQrSWJPYnlod2hIcWU0Mktwc0lGNTdScWZ3ZTVnc1g3OURQOTlTVkJQ?=
 =?utf-8?B?MUZnODlIVUgyV2Vzc292UCt5TXlQeVZ2Z3l2NkY1aVFoMitaSVUxWVYxdmxo?=
 =?utf-8?B?R0dLeHlxL0kzeFRXVm9obmNkMHNKS05XTWxUTlI0TSt1ZWk5U3hlaVZnd2pq?=
 =?utf-8?B?TzdxeDJJdzREOW1GMnpXTVFraEd5M2dQeHhBVUJKQUdicllxa0NNR2ZqbURu?=
 =?utf-8?B?dEdFWS9TTnZQRUp0bUpkUnhsM0R1Zmw0Y2FVREdmTkE1dEZmQ1JkWUVBVWxL?=
 =?utf-8?B?RWI1amVscnpid2hFWTBJeHg4bEVJQ3FiUTZNUzgyRko3UU1ia1BXdGZEbVg5?=
 =?utf-8?B?UWI1eU5mcU5kRjJvM2lBN3Bkb3dqVTA0WStQWFNIUFFGb0R2NmlKY2xGS2Ez?=
 =?utf-8?B?TzZ6c21hZkd4c2c3aHVzWEw5L2tSSFpBOG5Zc3JUemZzbGVHRTdYMi9NL1RU?=
 =?utf-8?B?WXI5OHNhcm1nY2x6VGthTHlrT2Y3aXEwOXBXM2FGRGM4cm96RUw2U1ZuYXZa?=
 =?utf-8?B?aDU4Y1NEYndRaFBFcjVOa1R0U1J2UFNHTVY3ZjI3RmJUa1ErMmNIaGVXcVh6?=
 =?utf-8?B?cHpxaExHdG1BcklpaVFIbHQzNUdldVdacE9BQVNRdWNWaGJGNENxSWVjdk1N?=
 =?utf-8?B?cVZoWk5rZUprOGJUeUx1VUtCK1ZGaEZ3d2duU1gwMjkvenFiR2hFU1Y1Tm9V?=
 =?utf-8?B?TysxOWlCQ1lFM2xhdWpEdTl2YW9aejVEQTBteWlNMzNhT0RDTkhaNy80ZjJC?=
 =?utf-8?B?NU82UzFOOS91QmkrdkFlN0pRUnRCbDE2NDVtejl2N2NaVzBsSnZ4cG9jSGpw?=
 =?utf-8?B?ckM0akRJeGZwYnppT2EvZWNxY0dhR01yTzNReVR5aXk0ZnNkM1VySEFLZlRr?=
 =?utf-8?B?T25NVVYvc1l2YTIwMUFZQWRCeGRnOHNFV3FwNWF2TlJndzZXL0ZuWktTME4r?=
 =?utf-8?B?M3hZNmhDYlI2c2d1bUNtazFLcXFWY0xLRFVTb0svMUhHRXpSVW9Vc2RqY0xp?=
 =?utf-8?B?dEE5YUN3cTFyb1lsTWJ1Q3A2YnBIenFtbUtVZXQrMWwvVkFYSmtkaExaNVQ1?=
 =?utf-8?B?d0JSV0NLSXJXOUxsWVl2akd1bnZVY0hoNklMV3FpMVNwa2d3T1UrRWw5aFlS?=
 =?utf-8?B?V25mejVYdzEvYnV3WkZidHhFUGpHTGRaK1E1a1pVMm1EZ3JxMjdiaEZLWXBx?=
 =?utf-8?B?RjkxcHZ4anhHSFNseGNiaW9ZMEZqWDdBVWNqdDBkZ1RET1hDRUxTcWNqaXg4?=
 =?utf-8?B?eVZWaHp5azNBYUlFaWc4Y3lVK0ZRVUxkVERtTGVoUTN1UXZJVTBnU3VuOVQy?=
 =?utf-8?B?QmgrV3pZVHdVUGFjWHFvemthVTRUT0R6RkJ2VVkwOHkrQUlFNTdxZ2NEZHBE?=
 =?utf-8?B?MUFCUXJLWDBSdm1BamtQSTluNnVlcnRNYTEvdVJZa21QK1YwSlA1QWZXK0dk?=
 =?utf-8?B?cmNwMjJhWEFpZnVqMkJFNlYzYmpUUGZxQ1NOdkpacUtIa25WVkhRV2wybStp?=
 =?utf-8?B?Y1hlSmxldzlieXI5YWVPa0FqckZvSnpwMlJsSEkwRkFHaER6QUQvY0d0RGN4?=
 =?utf-8?B?Mi9PSVlzUXZneHN5TkR0Z0gvK1o0Y3FRY1RnQ1FaeS80RDdMRlpLU2ZjZHR1?=
 =?utf-8?B?YlZrUElpazYwdXdjQTJJSDljODd3bDUvdDNIVnB3NDB3MEdpOHR5SUd6bkhY?=
 =?utf-8?B?azFLamRlSmFUYzFLMkxLdmpkS2QrdlhHRlRaRURiZXhXRUF3SEdJMmgwZ1Yz?=
 =?utf-8?B?RENGVEZ4TitGZWtlL3pCQW56R29MN3hyRU1vcnNEYnJCTVhPbDhyMlJNZU82?=
 =?utf-8?B?QStmVEFPM0RvcWhoRFdGemlmME9UVHNhdGpvd1RGWWhtNm85RGlOd055cXk3?=
 =?utf-8?B?MXc9PQ==?=
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 
	IipLDUtuOHWZSZLQwcPBcCRjnHAVkxyFgCAAc5qYHNB4RtRjG8PWrkiUgB0q1e58SesQ/yVN1ZKjXZ6ayyJGHQabfeuGXDtaSi3YAnDNqrqxtiW/u01lnm1jwQ5apZMFcSoPdQ3Vcsb84cHCcHYET2PLwO6DcRSgPBGqfM/xdPzCKBWv8lJ/QmKQRb5q/C6ax21M+2+GIiVyoVCMMLHD274YUa2UzkrVJwLSJKLfuK5Ya4cAbxbPG9OfbrNXqDq2zHjU8+6ZQeMJLla8oT3lekY4d/iTJ9DGJssUO75sY3NMhbqGV4fzBxMFECCK36o3TZV8tv5qQuBPGopA8KSQoRphd28DxZJRY/WgQR+Fmc8uQ00uECji+rOCE613ufXxAxyxb/OvGdpUeljE2HqX0wHEJMn3rQyqjB3B6vDubHE0m/es+x3M7ephF6bLdK5XDSTg1Gtwockj2YwDfuDVYNEzROSOXqor5p57MVfnD4gbpr87yjLDuu/5dVfUVMcFijPomaJOpuqGXKAWlos4sG22yUzLpchvlK83r3lLScOjuq5AnO0DFUwXXH0hr1olN/qZnZV+IvnaSzejdJa8KtWOQqAxk7/rblVnWiL+kpM=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 9ec9e48c-e381-4028-ef1b-08dc27062390
X-MS-Exchange-CrossTenant-AuthSource: DM6PR10MB4313.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2024 11:24:10.1432
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: L6couULElPHeD2kAre8LnS4bYyk2xvncYy2mC/H1UTjGG6ChtHBwgOdVohZCT/i32XRRKhdBlXtu7ZHCFlXyDQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CYYPR10MB7626
X-Proofpoint-Virus-Version: vendor=baseguard
 engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26
 definitions=2024-02-06_04,2024-01-31_01,2023-05-22_02
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 spamscore=0
 mlxlogscore=535 bulkscore=0 mlxscore=0 phishscore=0 adultscore=0
 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1
 engine=8.12.0-2311290000 definitions=main-2402060080
X-Proofpoint-ORIG-GUID: 5Q9kVVkjQ5pzNL6WFjaHVtqtaQ2FEpjJ
X-Proofpoint-GUID: 5Q9kVVkjQ5pzNL6WFjaHVtqtaQ2FEpjJ

On 05/02/2024 15:32, Robin Murphy wrote:
> From: Pasha Tatashin <pasha.tatashin@soleen.com>
> 
> The magazine buffers can take gigabytes of kmem memory, dominating all
> other allocations. For observability purpose create named slab cache so
> the iova magazine memory overhead can be clearly observed.
> 
> With this change:
> 
>> slabtop -o | head
>   Active / Total Objects (% used)    : 869731 / 952904 (91.3%)
>   Active / Total Slabs (% used)      : 103411 / 103974 (99.5%)
>   Active / Total Caches (% used)     : 135 / 211 (64.0%)
>   Active / Total Size (% used)       : 395389.68K / 411430.20K (96.1%)
>   Minimum / Average / Maximum Object : 0.02K / 0.43K / 8.00K
> 
> OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
> 244412 244239 99%    1.00K  61103       4    244412K iommu_iova_magazine
>   91636  88343 96%    0.03K    739     124      2956K kmalloc-32
>   75744  74844 98%    0.12K   2367      32      9468K kernfs_node_cache
> 
> On this machine it is now clear that magazine use 242M of kmem memory.

Those caches could do with a trimming ...

> 
> Acked-by: David Rientjes <rientjes@google.com>
> Signed-off-by: Pasha Tatashin <pasha.tatashin@soleen.com>
> [ rm: adjust to rework of iova_cache_{get,put} ]
> Signed-off-by: Robin Murphy <robin.murphy@arm.com>
> ---

FWIW:
Reviewed-by: John Garry <john.g.garry@oracle.com>