Received: by 2002:a25:d7c1:0:0:0:0:0 with SMTP id o184csp2785417ybg; Mon, 28 Oct 2019 02:19:23 -0700 (PDT) X-Google-Smtp-Source: APXvYqxguzUIeknrJI1LkI+XdDm6CpSWKSVPkJkbu+fyt5KUYqrkd97LdOCaXPxpROJthGxwAYft X-Received: by 2002:a17:906:85da:: with SMTP id i26mr4422830ejy.186.1572254362910; Mon, 28 Oct 2019 02:19:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1572254362; cv=none; d=google.com; s=arc-20160816; b=hgpg34JkpXa2HQKF/o+4K4AKOuynfEZx45sL8BYAFxU58HZtf+Xt1HZBRk/XghrKno alrOg45+CK5FZ42s372TFmPBP+DvOiPAsxS8xrgFETwBNDNyJRcz5GEL4HXI4rKQrWa3 kEmgsgvuwovsL8b105B+BGE7n6gRceXcjtfsiQPD6uwOHDBGlB6kKstrCHi2CYJBpvrE 4cKn1vWenU4UUrqsB1Xy+Ywnh1nbwNgfJRgRM24IQ+kGQNjtc7NwGD/qKkY+iVD9iWjA 1bOdcW92KdEp0Sou0k7SDPQD6slTt0aHSWa4HwsfXO+QmtnqxGufzI020+s69oDeaAjt 62Pg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=D+Q2AC56xhKpVizkCWjsQZdEQdeYozAJok6r6C+tDys=; b=vZIBU8YBcbjgUae41RohOVu+GF+5CxZHvGLwNcWQ++Cm7ohGBFMwaSB3C57q1IApuS WJyfovDTi3qogg4jQW/HTSIC9Fm7ZlXUIajj5JsRGkWT7PovPgZWAIhi9Yahck4fS98a yCvTLjHwBWnWsaYpNAl7SeNoZ91GsJweekEXjihR5hNIBpMLCiv1V4E8GO2pBM5DSQ+T 9eu9sqh8fHN+ix4QqtGbcEf/CIy54X5KkxU8S8+/mzNaqJkjoLp0DW4juNvAIcM2eGNR T8lmKKmElJI/npg6kKksICB5YcgB9w4fpzWU3/mHaUtPwcEA7esO8VtDey67u6TmI2h/ 90wA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=PuABERHg; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id w1si6256541eda.433.2019.10.28.02.18.56; Mon, 28 Oct 2019 02:19:22 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=PuABERHg; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732348AbfJ0VXZ (ORCPT + 99 others); Sun, 27 Oct 2019 17:23:25 -0400 Received: from mail.kernel.org ([198.145.29.99]:44528 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732334AbfJ0VXU (ORCPT ); Sun, 27 Oct 2019 17:23:20 -0400 Received: from localhost (100.50.158.77.rev.sfr.net [77.158.50.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5F0922064A; Sun, 27 Oct 2019 21:23:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1572211400; bh=97MBzUNNqyU+uABWapxiR2K5S7vS+qfj9+fyGXsu65o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PuABERHg+VG27/suFJiaossoJbTwdj6Qp1GPWQFk2JmfFk9KOZGE8YAmOO61WBSAd Vsg2obyGC/7cN6lXBsRNbQlRdHuDeBskSQ8080A6Qy7gpm4HjkIOAyodurFIN0tvMW G0DMljHyXHeOXoaOc7t6nINC1XtbaTdAhf1cAVGc= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Pavel Begunkov , Jens Axboe Subject: [PATCH 5.3 094/197] io_uring: fix bad inflight accounting for SETUP_IOPOLL|SETUP_SQTHREAD Date: Sun, 27 Oct 2019 22:00:12 +0100 Message-Id: <20191027203356.845912397@linuxfoundation.org> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191027203351.684916567@linuxfoundation.org> References: <20191027203351.684916567@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Jens Axboe commit 2b2ed9750fc9d040b9f6d076afcef6f00b6f1f7c upstream. We currently assume that submissions from the sqthread are successful, and if IO polling is enabled, we use that value for knowing how many completions to look for. But if we overflowed the CQ ring or some requests simply got errored and already completed, they won't be available for polling. For the case of IO polling and SQTHREAD usage, look at the pending poll list. If it ever hits empty then we know that we don't have anymore pollable requests inflight. For that case, simply reset the inflight count to zero. Reported-by: Pavel Begunkov Reviewed-by: Pavel Begunkov Signed-off-by: Jens Axboe Signed-off-by: Greg Kroah-Hartman --- fs/io_uring.c | 44 ++++++++++++++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 12 deletions(-) --- a/fs/io_uring.c +++ b/fs/io_uring.c @@ -816,19 +816,11 @@ static void io_iopoll_reap_events(struct mutex_unlock(&ctx->uring_lock); } -static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned *nr_events, - long min) +static int __io_iopoll_check(struct io_ring_ctx *ctx, unsigned *nr_events, + long min) { - int iters, ret = 0; + int iters = 0, ret = 0; - /* - * We disallow the app entering submit/complete with polling, but we - * still need to lock the ring to prevent racing with polled issue - * that got punted to a workqueue. - */ - mutex_lock(&ctx->uring_lock); - - iters = 0; do { int tmin = 0; @@ -864,6 +856,21 @@ static int io_iopoll_check(struct io_rin ret = 0; } while (min && !*nr_events && !need_resched()); + return ret; +} + +static int io_iopoll_check(struct io_ring_ctx *ctx, unsigned *nr_events, + long min) +{ + int ret; + + /* + * We disallow the app entering submit/complete with polling, but we + * still need to lock the ring to prevent racing with polled issue + * that got punted to a workqueue. + */ + mutex_lock(&ctx->uring_lock); + ret = __io_iopoll_check(ctx, nr_events, min); mutex_unlock(&ctx->uring_lock); return ret; } @@ -2327,7 +2334,20 @@ static int io_sq_thread(void *data) unsigned nr_events = 0; if (ctx->flags & IORING_SETUP_IOPOLL) { - io_iopoll_check(ctx, &nr_events, 0); + /* + * inflight is the count of the maximum possible + * entries we submitted, but it can be smaller + * if we dropped some of them. If we don't have + * poll entries available, then we know that we + * have nothing left to poll for. Reset the + * inflight count to zero in that case. + */ + mutex_lock(&ctx->uring_lock); + if (!list_empty(&ctx->poll_list)) + __io_iopoll_check(ctx, &nr_events, 0); + else + inflight = 0; + mutex_unlock(&ctx->uring_lock); } else { /* * Normal IO, just pretend everything completed.