Received: by 2002:a05:6a10:9e8c:0:0:0:0 with SMTP id y12csp270292pxx; Wed, 28 Oct 2020 04:37:38 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzLfR/Wvm1rWcTC/8mn4VSTB4bYc/aWtvnT+VKfTTnZjygpasF9aMtISPe9CkjuZJcYHyQe X-Received: by 2002:a17:907:40eb:: with SMTP id nn19mr7340403ejb.240.1603885057920; Wed, 28 Oct 2020 04:37:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1603885057; cv=none; d=google.com; s=arc-20160816; b=EHYiJejrzV5dKFiDwASWtz72thsc5zCYkWW4BpH8Q5i/Vf+nRyT+Z9BbyHmWQUFxyQ G+0ZShkLxZGg//jf/Bwidu5XKoIPfJ0bOQMieWgkhS/pqrYtMDWfdaGInCy8QEsUaH3P m0l7XQF6Z8/Ryjz2GROv5fyUPNEyPAa1nnT9Z507Yx/jGAhC1pt6xak12zg2T8YTgKhh L7+YI2O4pEtxgeQ/ao5yWCzjb6s2Q07ImTY1LiikcXatAz/rVbDO2HkEaZ8XY7IOuJ3n CMkZypUVKXgWE7k1XMWckNhgxaFClRHhpHS/qpc4kCFyTWhuMtcBR/gfyTUoiSdnTtZt d3bg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :user-agent:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=3kGdBbBzykMFpb1WXtdG4KxOp0rYA5ZEUpis7mR5u8w=; b=FY7AnrRRyJ307kam+jnB1kzU4eu8brq2wZwMowsWtfkHPMQNvLfbY7vFd1e1uG+/QJ t2XZ2dG5y7XFLk/s11cORH/ZIxUsiALwXa98AcaaV/VRlonHdZeJAzn1R1/lRs68jDU3 k9+spzRE3QNXaC8I37msKhKmENaWR5wQ6n3FOPjamfytYsOl6MVnprS0HiOuaoo5E7wJ Y6csSs48jfJzCCW4w9A6mFtiOvdXasW4XgZq46HdyZZUIMAaWZLYSyc1hLd9LForsdzj LkKulYa8kml9dUpmgvNrGeG/0YrdZbiwExzu1TjWwivVBGDT9+qOOMBk4qjdWr3x9PHn 9BDA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=RfVqtbnm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id o10si2705925ejr.482.2020.10.28.04.37.15; Wed, 28 Oct 2020 04:37:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=default header.b=RfVqtbnm; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=linuxfoundation.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1794012AbgJ0PJm (ORCPT + 99 others); Tue, 27 Oct 2020 11:09:42 -0400 Received: from mail.kernel.org ([198.145.29.99]:39522 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1790944AbgJ0PEx (ORCPT ); Tue, 27 Oct 2020 11:04:53 -0400 Received: from localhost (83-86-74-64.cable.dynamic.v4.ziggo.nl [83.86.74.64]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 2EF92206E5; Tue, 27 Oct 2020 15:04:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1603811092; bh=5wfmlOIqJHNnDmiU31IwiV6Wt0RSsM/VminhRtvnA9c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RfVqtbnmsaIA7IAHav4jsz6eH+JIrLG4EeWLraPiJeAemNUi/GODJ734P+bLA04CY ausAe9ir4dSpaxE9AZZRaZF2i4VudfEKD3i/a2qz07ynbJsCiZzf6V6rDj421tO5Or TMpVxsh/vr/Yx43neb1sC10sQLJBDWq0U2biLcB0= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Nicholas Piggin , Michael Ellerman , Sasha Levin Subject: [PATCH 5.8 370/633] powerpc/64s/radix: Fix mm_cpumask trimming race vs kthread_use_mm Date: Tue, 27 Oct 2020 14:51:53 +0100 Message-Id: <20201027135540.059566103@linuxfoundation.org> X-Mailer: git-send-email 2.29.1 In-Reply-To: <20201027135522.655719020@linuxfoundation.org> References: <20201027135522.655719020@linuxfoundation.org> User-Agent: quilt/0.66 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Nicholas Piggin [ Upstream commit a665eec0a22e11cdde708c1c256a465ebe768047 ] Commit 0cef77c7798a7 ("powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask") added a mechanism to trim the mm_cpumask of a process under certain conditions. One of the assumptions is that mm_users would not be incremented via a reference outside the process context with mmget_not_zero() then go on to kthread_use_mm() via that reference. That invariant was broken by io_uring code (see previous sparc64 fix), but I'll point Fixes: to the original powerpc commit because we are changing that assumption going forward, so this will make backports match up. Fix this by no longer relying on that assumption, but by having each CPU check the mm is not being used, and clearing their own bit from the mask only if it hasn't been switched-to by the time the IPI is processed. This relies on commit 38cf307c1f20 ("mm: fix kthread_use_mm() vs TLB invalidate") and ARCH_WANT_IRQS_OFF_ACTIVATE_MM to disable irqs over mm switch sequences. Fixes: 0cef77c7798a7 ("powerpc/64s/radix: flush remote CPUs out of single-threaded mm_cpumask") Signed-off-by: Nicholas Piggin Reviewed-by: Michael Ellerman Depends-on: 38cf307c1f20 ("mm: fix kthread_use_mm() vs TLB invalidate") Signed-off-by: Michael Ellerman Link: https://lore.kernel.org/r/20200914045219.3736466-5-npiggin@gmail.com Signed-off-by: Sasha Levin --- arch/powerpc/include/asm/tlb.h | 13 ------------- arch/powerpc/mm/book3s64/radix_tlb.c | 23 ++++++++++++++++------- 2 files changed, 16 insertions(+), 20 deletions(-) diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index 862985cf51804..cf87bbdcfdcb2 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -67,19 +67,6 @@ static inline int mm_is_thread_local(struct mm_struct *mm) return false; return cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm)); } -static inline void mm_reset_thread_local(struct mm_struct *mm) -{ - WARN_ON(atomic_read(&mm->context.copros) > 0); - /* - * It's possible for mm_access to take a reference on mm_users to - * access the remote mm from another thread, but it's not allowed - * to set mm_cpumask, so mm_users may be > 1 here. - */ - WARN_ON(current->mm != mm); - atomic_set(&mm->context.active_cpus, 1); - cpumask_clear(mm_cpumask(mm)); - cpumask_set_cpu(smp_processor_id(), mm_cpumask(mm)); -} #else /* CONFIG_PPC_BOOK3S_64 */ static inline int mm_is_thread_local(struct mm_struct *mm) { diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index b5cc9b23cf024..277a07772e7d6 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -644,19 +644,29 @@ static void do_exit_flush_lazy_tlb(void *arg) struct mm_struct *mm = arg; unsigned long pid = mm->context.id; + /* + * A kthread could have done a mmget_not_zero() after the flushing CPU + * checked mm_is_singlethreaded, and be in the process of + * kthread_use_mm when interrupted here. In that case, current->mm will + * be set to mm, because kthread_use_mm() setting ->mm and switching to + * the mm is done with interrupts off. + */ if (current->mm == mm) - return; /* Local CPU */ + goto out_flush; if (current->active_mm == mm) { - /* - * Must be a kernel thread because sender is single-threaded. - */ - BUG_ON(current->mm); + WARN_ON_ONCE(current->mm != NULL); + /* Is a kernel thread and is using mm as the lazy tlb */ mmgrab(&init_mm); - switch_mm(mm, &init_mm, current); current->active_mm = &init_mm; + switch_mm_irqs_off(mm, &init_mm, current); mmdrop(mm); } + + atomic_dec(&mm->context.active_cpus); + cpumask_clear_cpu(smp_processor_id(), mm_cpumask(mm)); + +out_flush: _tlbiel_pid(pid, RIC_FLUSH_ALL); } @@ -671,7 +681,6 @@ static void exit_flush_lazy_tlbs(struct mm_struct *mm) */ smp_call_function_many(mm_cpumask(mm), do_exit_flush_lazy_tlb, (void *)mm, 1); - mm_reset_thread_local(mm); } void radix__flush_tlb_mm(struct mm_struct *mm) -- 2.25.1