Hi,
I started to see buffer registration randomly failing with ENOMEM on
5.13. Registering buffer or two often succeeds, but more than that
rarely. Running the same program as root succeeds - but the user has a high
rlimit.
The issue is that io_sqe_buffer_register() doesn't initialize
imu. io_buffer_account_pin() does imu->acct_pages++, before calling
io_account_mem(ctx, imu->acct_pages);
Which means that a random amount of memory is being accounted for. On the first
few allocations this sometimes fails to fail because the memory is zero, but
after a bit of reuse...
It only doesn't fail as root because the rlimit doesn't apply.
This is caused by
commit 41edf1a5ec967bf4bddedb83c48e02dfea8315b4
Author: Pavel Begunkov <[email protected]>
Date: 2021-04-25 14:32:23 +0100
io_uring: keep table of pointers to ubufs
Greetings,
Andres Freund
On 5/29/21 1:33 AM, Andres Freund wrote:
> Hi,
>
> I started to see buffer registration randomly failing with ENOMEM on
> 5.13. Registering buffer or two often succeeds, but more than that
> rarely. Running the same program as root succeeds - but the user has a high
> rlimit.
>
> The issue is that io_sqe_buffer_register() doesn't initialize
> imu. io_buffer_account_pin() does imu->acct_pages++, before calling
> io_account_mem(ctx, imu->acct_pages);
>
> Which means that a random amount of memory is being accounted for. On the first
> few allocations this sometimes fails to fail because the memory is zero, but
> after a bit of reuse...
Makes sense, thanks for digging in. I've just sent a patch, would
be great if you can test it or send your own.
> It only doesn't fail as root because the rlimit doesn't apply.
>
> This is caused by
>
> commit 41edf1a5ec967bf4bddedb83c48e02dfea8315b4
> Author: Pavel Begunkov <[email protected]>
> Date: 2021-04-25 14:32:23 +0100
>
> io_uring: keep table of pointers to ubufs
>
> Greetings,
>
> Andres Freund
>
--
Pavel Begunkov
Hi,
On 2021-05-29 12:03:12 +0100, Pavel Begunkov wrote:
> On 5/29/21 1:33 AM, Andres Freund wrote:
> > Hi,
> >
> > I started to see buffer registration randomly failing with ENOMEM on
> > 5.13. Registering buffer or two often succeeds, but more than that
> > rarely. Running the same program as root succeeds - but the user has a high
> > rlimit.
> >
> > The issue is that io_sqe_buffer_register() doesn't initialize
> > imu. io_buffer_account_pin() does imu->acct_pages++, before calling
> > io_account_mem(ctx, imu->acct_pages);
> >
> > Which means that a random amount of memory is being accounted for. On the first
> > few allocations this sometimes fails to fail because the memory is zero, but
> > after a bit of reuse...
>
> Makes sense, thanks for digging in. I've just sent a patch, would
> be great if you can test it or send your own.
Sorry for the slow response, I'm off this week. I did just get around to
test and unsurprisingly: The patch does fix the issue.
Greetings,
Andres Freund
On 6/3/21 12:06 PM, Andres Freund wrote:
> Hi,
>
> On 2021-05-29 12:03:12 +0100, Pavel Begunkov wrote:
>> On 5/29/21 1:33 AM, Andres Freund wrote:
>>> Hi,
>>>
>>> I started to see buffer registration randomly failing with ENOMEM on
>>> 5.13. Registering buffer or two often succeeds, but more than that
>>> rarely. Running the same program as root succeeds - but the user has a high
>>> rlimit.
>>>
>>> The issue is that io_sqe_buffer_register() doesn't initialize
>>> imu. io_buffer_account_pin() does imu->acct_pages++, before calling
>>> io_account_mem(ctx, imu->acct_pages);
>>>
>>> Which means that a random amount of memory is being accounted for. On the first
>>> few allocations this sometimes fails to fail because the memory is zero, but
>>> after a bit of reuse...
>>
>> Makes sense, thanks for digging in. I've just sent a patch, would
>> be great if you can test it or send your own.
>
> Sorry for the slow response, I'm off this week. I did just get around to
> test and unsurprisingly: The patch does fix the issue.
OK good, thanks for confirming, I did ship it out earlier today so
should be in the next -rc.
--
Jens Axboe
On 6/3/21 7:06 PM, Andres Freund wrote:
> Hi,
>
> On 2021-05-29 12:03:12 +0100, Pavel Begunkov wrote:
>> On 5/29/21 1:33 AM, Andres Freund wrote:
>>> Hi,
>>>
>>> I started to see buffer registration randomly failing with ENOMEM on
>>> 5.13. Registering buffer or two often succeeds, but more than that
>>> rarely. Running the same program as root succeeds - but the user has a high
>>> rlimit.
>>>
>>> The issue is that io_sqe_buffer_register() doesn't initialize
>>> imu. io_buffer_account_pin() does imu->acct_pages++, before calling
>>> io_account_mem(ctx, imu->acct_pages);
>>>
>>> Which means that a random amount of memory is being accounted for. On the first
>>> few allocations this sometimes fails to fail because the memory is zero, but
>>> after a bit of reuse...
>>
>> Makes sense, thanks for digging in. I've just sent a patch, would
>> be great if you can test it or send your own.
>
> Sorry for the slow response, I'm off this week. I did just get around to
> test and unsurprisingly: The patch does fix the issue.
Yep, since you already narrowed it down. Thanks for testing
--
Pavel Begunkov