Received: by 2002:a05:7208:9594:b0:7e:5202:c8b4 with SMTP id gs20csp1599365rbb; Mon, 26 Feb 2024 15:08:27 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVq3gN2fLrYVIBvj/Ys2zWYY16NPxiW7v6mikc4CmYTiDb7U2RnUad0uAatxA6lvY47QBcUifmqo7UkB7hyYcmkiqDBgEqF0McniOUYWA== X-Google-Smtp-Source: AGHT+IElYy0CwHj319UZTbcZHjvxe84bTDJfMoqikl42HSveV7iIhi0LvVpv7bFtD55GPdfpcxRP X-Received: by 2002:a05:6a21:8ccc:b0:1a0:60b2:45b with SMTP id ta12-20020a056a218ccc00b001a060b2045bmr678553pzb.6.1708988907662; Mon, 26 Feb 2024 15:08:27 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1708988907; cv=pass; d=google.com; s=arc-20160816; b=bKRHNSR4xF4jlBuiqcrMoiheJOqA7FjumyKZxvopO02OfGBnmHBN5ZMzaPgFYLEM9u DMLjgjZAtiZ8pp6xHXgFpkRUryairIKoTflYvJWnVCjKOi5BVN/ncPOZDqEtEA1vHxQb H6bcNVYtBdgiYrmNQNmjaJcT+4I7h1TBwsMlZh0//t3gTDUIdSffx7oqSVCln8sEus/7 YCws8q0WsoxRpW6cRsT12aiwqgQRN1QbDrPYZzl1tvNuE5cM61Rm0/4XsmFyEDnZYbdI RyLHrn991oajjZlbVUEC8ssFD3PpjArk2Fol8KIHRJz9vVHUsVgChUCZ6U4jctr5NGkc 7fCg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=in-reply-to:content-disposition:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:message-id:subject:cc :to:from:date:dkim-signature; bh=jfUpsl1oMGN9xb/gKqon3kPZ1VzdIKEfk1TUwZlYS0c=; fh=pLqVG82dKEKtwR79WearxyXqSZsjaM8zMdYHPaxw/bE=; b=SM/at/XZyLPl7QWj7Kzr2pK4g+9c/VYNc/Dkn8TT/M9rx+cGRrpVyux1z1+z4BZaez mFRPPyrMKvzhJVm03pFlcgOETIDofyrtvRO3aB1G6C2FQCTsPeEe9VCQ6YajTQ5vvwgf BJ0VJL3AxgGY94RpvBgiIGBlzu6KWVWnxFvyB8XuqPgG5ip4PQQTVUs5XeIcn82RqXuy xrsl1eR1xoExJB7Y4xXM1SptgyWFnKKwdtsT+DHEYaa+E+E+Uoqk6Y2BkF9E9lqfGqcu BDhPVN4vNgpcnmJl/I6HRBzeYC1X3ev8pc4VaNVK7fxxkC8BWLhQ6o3Pj4LW8ptLdVF6 kmdg==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=oEfcwq7i; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-82433-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82433-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id p11-20020a17090a2d8b00b0029909da2167si6264194pjd.32.2024.02.26.15.08.27 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Feb 2024 15:08:27 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-82433-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=oEfcwq7i; arc=pass (i=1 dkim=pass dkdomain=kernel.org); spf=pass (google.com: domain of linux-kernel+bounces-82433-linux.lists.archive=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-82433-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 8B5F5B2377F for ; Mon, 26 Feb 2024 23:07:43 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2A54113541E; Mon, 26 Feb 2024 23:07:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oEfcwq7i" Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50B6B12F394; Mon, 26 Feb 2024 23:07:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708988855; cv=none; b=VehVgsi4BQgKFalRayC+T1DKdXhE/LcCrlSciwwsm5OW+qivnL2vfA9VZUJlxk4vnrp/Pkr+Ehk7dqdgDKa1EqQBptny6L8BRf4TEHN4xJkrVnguSl+NjS9AIvPJ2wQO6mcw/lXke9Xggfwg44bUMKkGabxARc2QKrx0DD0So+Q= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708988855; c=relaxed/simple; bh=adz6SAgNBJ7AGaeqHuV2mXfoAvBXu435vI2nwYExYEI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=gLg3tQoNyUvXxgzCPlNWV/cDjk3Ob6lA7bjKqBT7dzd7e+PiO2fTxtVOG2q4IDzatOa/VdZMIH9Ze/IBhxLDgOLvF8vNPZC6U67gzzy+gC/oDTEbly+h3HQ2x7oVDJqkBG/ZVjnWncYTRk1xJ4oMPGV1e7jquWY7IEKIU87sGOQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oEfcwq7i; arc=none smtp.client-ip=10.30.226.201 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 68D39C433F1; Mon, 26 Feb 2024 23:07:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1708988854; bh=adz6SAgNBJ7AGaeqHuV2mXfoAvBXu435vI2nwYExYEI=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=oEfcwq7i9f+T6i2I8ee6x7nirzDO3u+1IU5tyQt02L4dTvhh2wLwjl5Vugkduxzru wDwFty/4I4t4rUqmPx4dXLXYbpcuTYxj2j2/nQtI3pbHfTBGDT3ZztHi6PDhn505An 2jbi0GIVxhrOAr/jZJDZl+QF9zEk2dcanErt3WoJ9zxNzn6w8gLIfqzgdw0sJgCTk1 JZo/NUP+E++QAHJvNA1NeBzZLplrw91QPoOvDOyVi3509yLdRnLGC2w2BJTzYjYMHW 6TOEo9Fj/fhgmW/MkTT/Tad5mguVW0Pce7F4MC3BOfDDUkfEFKKPgomLeZO93WKfPa IzeyEZm37RdCA== Date: Tue, 27 Feb 2024 00:07:32 +0100 From: Frederic Weisbecker To: "Uladzislau Rezki (Sony)" Cc: "Paul E . McKenney" , RCU , Neeraj upadhyay , Boqun Feng , Hillf Danton , Joel Fernandes , LKML , Oleksiy Avramchenko Subject: Re: [PATCH v5 2/4] rcu: Reduce synchronize_rcu() latency Message-ID: References: <20240220183115.74124-1-urezki@gmail.com> <20240220183115.74124-3-urezki@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240220183115.74124-3-urezki@gmail.com> On Tue, Feb 20, 2024 at 07:31:13PM +0100, Uladzislau Rezki (Sony) wrote: > +static void rcu_sr_normal_gp_cleanup_work(struct work_struct *work) > +{ > + struct llist_node *done, *rcu, *next, *head; > + > + /* > + * This work execution can potentially execute > + * while a new done tail is being updated by > + * grace period kthread in rcu_sr_normal_gp_cleanup(). > + * So, read and updates of done tail need to > + * follow acq-rel semantics. > + * > + * Given that wq semantics guarantees that a single work > + * cannot execute concurrently by multiple kworkers, > + * the done tail list manipulations are protected here. > + */ > + done = smp_load_acquire(&rcu_state.srs_done_tail); > + if (!done) > + return; > + > + WARN_ON_ONCE(!rcu_sr_is_wait_head(done)); > + head = done->next; > + done->next = NULL; Can the following race happen? CPU 0 CPU 1 ----- ----- // wait_tail == HEAD1 rcu_sr_normal_gp_cleanup() { // has passed SR_MAX_USERS_WAKE_FROM_GP wait_tail->next = next; // done_tail = HEAD1 smp_store_release(&rcu_state.srs_done_tail, wait_tail); queue_work() { test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work) __queue_work() } } set_work_pool_and_clear_pending() rcu_sr_normal_gp_cleanup_work() { // new GP, wait_tail == HEAD2 rcu_sr_normal_gp_cleanup() { // executes all completion, but stop at HEAD1 wait_tail->next = HEAD1; // done_tail = HEAD2 smp_store_release(&rcu_state.srs_done_tail, wait_tail); queue_work() { test_and_set_bit(WORK_STRUCT_PENDING_BIT, work_data_bits(work) __queue_work() } } // done = HEAD2 done = smp_load_acquire(&rcu_state.srs_done_tail); // head = HEAD1 head = done->next; done->next = NULL; llist_for_each_safe() { // completes all callbacks, release HEAD1 } } // Process second queue set_work_pool_and_clear_pending() rcu_sr_normal_gp_cleanup_work() { // done = HEAD2 done = smp_load_acquire(&rcu_state.srs_done_tail); // new GP, wait_tail == HEAD3 rcu_sr_normal_gp_cleanup() { // Finds HEAD2 with ->next == NULL at the end rcu_sr_put_wait_head(HEAD2) ... // A few more GPs later rcu_sr_normal_gp_init() { HEAD2 = rcu_sr_get_wait_head(); llist_add(HEAD2, &rcu_state.srs_next); // head == rcu_state.srs_next head = done->next; done->next = NULL; llist_for_each_safe() { // EXECUTE CALLBACKS TOO EARLY!!! } }