From:   Ofir Gal <ofir.gal@volumez.com>
To:     ming.lei@redhat.com
Cc:     axboe@kernel.dk, cgroups@vger.kernel.org,
        linux-block@vger.kernel.org, linux-kernel@vger.kernel.org,
        tj@kernel.org, yi.zhang@huawei.com, yukuai3@huawei.com,
        Ofir Gal <ofir@gal.software>
Subject: [PATCH -next] blk-throttle: enable io throttle for root in cgroup v2
Date:   Sun,  5 Feb 2023 17:55:41 +0200
Message-Id: <20230205155541.1320485-1-ofir.gal@volumez.com>
In-Reply-To: <YgMxjyVjMjmkMQU5@T590>
References: <YgMxjyVjMjmkMQU5@T590>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?GkVbIK4SbV6AGf/V2qWVJ28qDvh/dCk1UbU1gzbG20rzqfJpTaLWKlAWcbAX?=
 =?us-ascii?Q?j7fZ+3xpIRrsIgaMPJE/ftDumtYZdaC2V/to4I8u80DbIOErsO5EqStENMfz?=
 =?us-ascii?Q?15VC7U4rBxWKPlQ7KB/MnwuaV53O1y54PX9JU1jW2rFE3DnRe1T/w/oK9sLt?=
 =?us-ascii?Q?JzEfk9mIikePwaCcWu8WnZfpPmc3vv0MVGmoAmciHiamn9g3o0cAwDWZolW1?=
 =?us-ascii?Q?7hL6sY217aiCL8mKbZz9HBp03oeGGFkvnrse5hAOK3vNpOzhj2Y0l/SmmuXO?=
 =?us-ascii?Q?H5SXJ+rnRuFaDB+wmZDoaI2Dek6nFwi0D7DAH2re6iO0cMyU4lal+N2mufZi?=
 =?us-ascii?Q?Z/RT0Kv8VVO5kpUds9eJT+tdGHHMYeswqM78UThPPqup0GS6nbHjEn4Mbov3?=
 =?us-ascii?Q?SXCSDfyXCrge5RSHVs7Y/iJPTYnjPqHGbI/OXPi8+w/M0JMeGeMwonSt82nv?=
 =?us-ascii?Q?uMCR+w0d1A9VJVjxPK+FvwlCSXPxkcpfJwhb8mPPJZG0mFUMq+JM7SeBE68e?=
 =?us-ascii?Q?+FPCSh6eG0vJhbw8OBRCFeaS8lGwSLQkJNm4kGT2oDc86IB6sCzccfUiSGf3?=
 =?us-ascii?Q?tn2kD54reJFsM0g65C6A22viqd3TRpTuIaC8o79cGI3G7PGcn5HhZ/cXpAeN?=
 =?us-ascii?Q?yMPIS7oC/vmVfjc0Vbq/wA4kADOmjIfPC3ziFGwETL0jGVhsKeUYThWPbZXy?=
 =?us-ascii?Q?LkRNKn7z5PcgsBm02LoVlu/b8jV9ioUnLtj4UmyzsgIXvoVdcW7LrQg+qbrK?=
 =?us-ascii?Q?L02UsBYKo37atw/4n1yweA/OSBU46Oodv5DDM2qhLpPteFjkFoHD9UpX5Kc0?=
 =?us-ascii?Q?SUQWNNpdp+Y59sZb4CM2wSh62Bfm8gJpj2AoWCIERUNqczDiWe8tYPrY8CBX?=
 =?us-ascii?Q?5I2umzsbWw2aynshMGrkJXO3ZPRCTsM++9fXbPOUa/HSY67OibV20gXMCTlR?=
 =?us-ascii?Q?PRp2L3lKZLNaZjPZHVV5DPqJFVDEAktj0lbr4k6dtaZ8/p1q82ABCwnHnqwm?=
 =?us-ascii?Q?RGwTM73wmO3ObkHLVg4kbQcbWw5+yF7hBazeDYRogX2mBzbA215oMjCSdhME?=
 =?us-ascii?Q?Wy+T2JtKiSL9myHa94ZIQyWDcI5H74wOvule92uAe65JRx74f90+Yi+XWq6M?=
 =?us-ascii?Q?chqGNWj+gOsuACF5vdeke7FmkblxARPbh/lB1sVWUuChk/oJEOvuTxia5xEF?=
 =?us-ascii?Q?dQNEMWVisbHVnlnz0WwgnfmWE0IMOJp4/9eRbTyixr3eBuDqIdwF5XR+PHk9?=
 =?us-ascii?Q?fkp866xriyZk/hGG1h+cOQBDspbHbENVF3Ubveh55pY0xsFFbfngwGEsjLsj?=
 =?us-ascii?Q?hBw4copsgGLdkg7Bsh9ZbRT9sTivkEbp8EuqcCQqzP6Ruwo18XMHQuppejlk?=
 =?us-ascii?Q?9GptM9+gw/hIggENXBLtN8TOVUbrEQWyPVL8kLJcbcdH7ud9dmIY1+C6rgCY?=
 =?us-ascii?Q?AFs89JDE5163z7I0HrSahq2Tv9yG3JSqDKCd8VuS6NBMFViNyQ5961frOl+N?=
 =?us-ascii?Q?1dQKz553pvZ3OXleZGHwGX1TYb5IJehUIt9sW1ypX86+VVnTmTlkFrYzlsYt?=
 =?us-ascii?Q?yT8QB2Pjyk08HnDqTP8KyJXXCmdBBapAA3VwlfYI?=
X-OriginatorOrg: volumez.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 67cf0c36-a75e-4a73-5de6-08db07918366
X-MS-Exchange-CrossTenant-AuthSource: AS1PR04MB9335.eurprd04.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Feb 2023 15:56:16.1435
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: b1841924-914b-4377-bb23-9f1fac784a1d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: IPK4M5ALwEXfWP9JcH2I1NQdSpcpPyi8SvOa6aHkHcXOKgqRhwMU9r5x2l+2QgCkF1ZhDqOMCzGOk3WPDBCHiw==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS1PR04MB9408
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

From: Ofir Gal <ofir@gal.software>

Hello Ming Lei,

I am trying to use cgroups v2 to throttle a media disk that is controlled by an NVME target.
Unfortunately, it cannot be done without setting the limit in the root cgroup.
It can be done via cgroups v1. Yu Kuai's patch allows this to be accomplished.

My setup consist from 3 servers.
Server #1:
    a. SSD media disk (needs to be throttled to 100K IOPs)
    b. NVME target controlling the SSD (1.a)

Server #2:
    a. NVME initiator is connected to Server #1 NVME target (1.b)

Server #3:
    a. NVME initiator is connected to Server #1 NVME target (1.b)

My setup accesses this media from multiple servers using NVMe over TCP,
but the initiator servers' workloads are unknown and can be changed dynamically. I need to limit the media disk to 100K IOPS on the target side.

I have tried to limit the SSD on Server #1, but it seems that the NVME target kworkers are not affected unless I use Yu Kuai's patch.

Can you elaborate on the issues with this patch or how the scenario described above can be done with cgroups v2?

Best regards, Ofir Gal.