On 2/3/22 11:26 AM, Usama Arif wrote:
> Hmm, maybe i didn't understand you and Pavel correctly. Are you
> suggesting to do the below diff over patch 3? I dont think that would be
> correct, as it is possible that just after checking if ctx->io_ev_fd is
> present unregister can be called by another thread and set ctx->io_ev_fd
> to NULL that would cause a NULL pointer exception later? In the current
> patch, the check of whether ev_fd exists happens as the first thing
> after rcu_read_lock and the rcu_read_lock are extremely cheap i believe.
They are cheap, but they are still noticeable at high requests/sec
rates. So would be best to avoid them.
And yes it's obviously racy, there's the potential to miss an eventfd
notification if it races with registering an eventfd descriptor. But
that's not really a concern, as if you register with inflight IO
pending, then that always exists just depending on timing. The only
thing I care about here is that it's always _safe_. Hence something ala
what you did below is totally fine, as we're re-evaluating under rcu
protection.
> diff --git a/fs/io_uring.c b/fs/io_uring.c
> index 25ed86533910..0cf282fba14d 100644
> --- a/fs/io_uring.c
> +++ b/fs/io_uring.c
> @@ -1736,12 +1736,13 @@ static void io_eventfd_signal(struct io_ring_ctx
> *ctx)
> {
> struct io_ev_fd *ev_fd;
>
> + if (likely(!ctx->io_ev_fd))
> + return;
> +
> rcu_read_lock();
> /* rcu_dereference ctx->io_ev_fd once and use it for both for
> checking and eventfd_signal */
> ev_fd = rcu_dereference(ctx->io_ev_fd);
>
> - if (likely(!ev_fd))
> - goto out;
> if (READ_ONCE(ctx->rings->cq_flags) & IORING_CQ_EVENTFD_DISABLED)
> goto out;
>
>
>> synchronize_rcu() can take a long time, and I think this is in the wrong
>> spot. It should be on the register side, IFF we need to expedite the
>> completion of a previous event fd unregistration. If we do it that way,
>> at least it'll only happen if it's necessary. What do you think?
>>
>
>
> How about the approach in v4? so switching back to call_rcu as in v2 and
> if ctx->io_ev_fd is NULL then we call rcu_barrier to make sure all rcu
> callbacks are finished and check for NULL again.
I'll check, haven't looked at v4 yet!
--
Jens Axboe
On 2/3/22 18:29, Jens Axboe wrote:
> On 2/3/22 11:26 AM, Usama Arif wrote:
>> Hmm, maybe i didn't understand you and Pavel correctly. Are you
>> suggesting to do the below diff over patch 3? I dont think that would be
>> correct, as it is possible that just after checking if ctx->io_ev_fd is
>> present unregister can be called by another thread and set ctx->io_ev_fd
>> to NULL that would cause a NULL pointer exception later? In the current
>> patch, the check of whether ev_fd exists happens as the first thing
>> after rcu_read_lock and the rcu_read_lock are extremely cheap i believe.
>
> They are cheap, but they are still noticeable at high requests/sec
> rates. So would be best to avoid them.
>
> And yes it's obviously racy, there's the potential to miss an eventfd
> notification if it races with registering an eventfd descriptor. But
> that's not really a concern, as if you register with inflight IO
> pending, then that always exists just depending on timing. The only
> thing I care about here is that it's always _safe_. Hence something ala
> what you did below is totally fine, as we're re-evaluating under rcu
> protection.
Indeed, the patch doesn't have any formal guarantees for propagation
to already inflight requests, so this extra unsynchronised check
doesn't change anything.
I'm still more сurious why we need RCU and extra complexity when
apparently there is no use case for that. If it's only about
initial initialisation, then as I described there is a much
simpler approach.
--
Pavel Begunkov