Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
Message-ID: <e01e469f-488a-8f2d-008f-8427289d2ff3@kernel.dk>
Date:   Mon, 16 Jan 2023 06:55:20 -0700
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.0
Subject: Re: [PATCH 2/2] io_uring: Split io_issue_def struct
Content-Language: en-US
To:     Breno Leitao <leitao@debian.org>,
        Gabriel Krisman Bertazi <krisman@suse.de>
Cc:     asml.silence@gmail.com, dylany@meta.com, io-uring@vger.kernel.org,
        leit@fb.com, linux-kernel@vger.kernel.org
References: <20230112144411.2624698-1-leitao@debian.org>
 <20230112144411.2624698-2-leitao@debian.org> <87v8lbcwz9.fsf@suse.de>
 <Y8UseW5sTqu72M2U@gmail.com>
From:   Jens Axboe <axboe@kernel.dk>
In-Reply-To: <Y8UseW5sTqu72M2U@gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Precedence: bulk

On 1/16/23 3:52 AM, Breno Leitao wrote:
> On Thu, Jan 12, 2023 at 05:35:22PM -0300, Gabriel Krisman Bertazi wrote:
>> Breno Leitao <leitao@debian.org> writes:
>>
>>> This patch removes some "cold" fields from `struct io_issue_def`.
>>>
>>> The plan is to keep only highly used fields into `struct io_issue_def`, so,
>>> it may be hot in the cache. The hot fields are basically all the bitfields
>>> and the callback functions for .issue and .prep.
>>>
>>> The other less frequently used fields are now located in a secondary and
>>> cold struct, called `io_cold_def`.
>>>
>>> This is the size for the structs:
>>>
>>> Before: io_issue_def = 56 bytes
>>> After: io_issue_def = 24 bytes; io_cold_def = 40 bytes
>>
>> Does this change have an observable impact in run time? Did it show
>> a significant decrease of dcache misses?
> 
> I haven't tested it. I expect it might be hard to came up with such test.
> 
> A possible test might be running io_uring heavy tests, while adding
> enough memory pressure. Doing this in two different instant (A/B test),
> might be a unpredicable and the error deviation might hide the benefit.

I think what you'd want is two (or more) io_uring ops being really
busy and measuring dcache pressure while running that test. I don't
think this is very feasible to accurately measure, and I also don't
think that is an issue. The split into hot/cold parts of the op
definitions is obviously a good idea. For ideal setups, we'll never
be using the cold part at all, and having a smaller op definition
for the fast path is always going to be helpful.

-- 
Jens Axboe