Received: by 2002:a05:6902:102b:0:0:0:0 with SMTP id x11csp520303ybt; Mon, 6 Jul 2020 15:27:22 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxRJq8IibtgsPkkuTPrFC5rtxKjSOd6XaEfAFjwNQoXNJNyvMCgJ0GJps2Dwmn2M0pNkuRK X-Received: by 2002:a17:906:2505:: with SMTP id i5mr44159421ejb.545.1594074442785; Mon, 06 Jul 2020 15:27:22 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1594074442; cv=none; d=google.com; s=arc-20160816; b=dFuWUjwpb1OM8xBhpQMvraFFluv8vTDLxSUsNWGVIdYBqQtbG96wtR09CE3ksy7rDm VWkDv2xNUydaevzpHZCCaCHlD91O87kjymjZWV11ogQZ1Rr+HJU2tUjaoClibBVktfoI Tx6UiuQzhztyGsT7gXH3Z876s0R0Nl3/igeTeJO42T3DY6vAsMeEXXks3rQvzbW+NUu8 RlLd01Z372AtwE9O2trzyG2c1Ea86uxXEJiNRyR9A7qiL6pSCBQ+CHxUZXXErCaiijv9 O/7bMqWP8iTkQwm+nBgLCHD7IlROUJfabnb9TsRxjeP/ZQOYn/fdCZhcnalxziPvnYiu I14g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:dkim-signature:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from; bh=T3rNe1ZTEVG4rl8pgs5SeLiYvOfEdwAB9yHLkYyf/KY=; b=hVEwtUbZ1qFrD3ffy5D0yhfYkCPyuUrhbwOloD/41KEdV8kTz9+gWFR664LuMjl6rS 4h4lWieLz4dQnYt2qvbulwRQhpgxEafNENi9y85qx+5hJIXlUF1iPZIRDvkWBpZHzguP 2+0m3SCkg0+05pSYO5p/HmbP91XfK680/WnyzC2vDy2LKHp8Dc0fIqQ4uFTWnmBdtIkE 5ak/tt9g8uRK61x4HFUIxsHMZ0ECg/8O6KTMQMyoB+4hsPwPWdEyZWOCbIw7NxIfRsh/ IDKFs90cGzO7DtXAN/glkKZT0iObWw2FSiioo6wCT7n7RT0NFO0QAih48s4Gd+nIxbdE O5Hg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@nvidia.com header.s=n1 header.b=lACYXImj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [23.128.96.18]) by mx.google.com with ESMTP id w4si13366478ejj.300.2020.07.06.15.26.59; Mon, 06 Jul 2020 15:27:22 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) client-ip=23.128.96.18; Authentication-Results: mx.google.com; dkim=pass header.i=@nvidia.com header.s=n1 header.b=lACYXImj; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.18 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727800AbgGFWYI (ORCPT + 99 others); Mon, 6 Jul 2020 18:24:08 -0400 Received: from hqnvemgate25.nvidia.com ([216.228.121.64]:1741 "EHLO hqnvemgate25.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727055AbgGFWYF (ORCPT ); Mon, 6 Jul 2020 18:24:05 -0400 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate25.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Mon, 06 Jul 2020 15:23:11 -0700 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Mon, 06 Jul 2020 15:24:04 -0700 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Mon, 06 Jul 2020 15:24:04 -0700 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL101.nvidia.com (172.20.187.10) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Mon, 6 Jul 2020 22:23:58 +0000 Received: from hqnvemgw03.nvidia.com (10.124.88.68) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1473.3 via Frontend Transport; Mon, 6 Jul 2020 22:23:58 +0000 Received: from rcampbell-dev.nvidia.com (Not Verified[10.110.48.66]) by hqnvemgw03.nvidia.com with Trustwave SEG (v7,5,8,10121) id ; Mon, 06 Jul 2020 15:23:58 -0700 From: Ralph Campbell To: , , , , , CC: Jerome Glisse , John Hubbard , Christoph Hellwig , Jason Gunthorpe , "Andrew Morton" , Shuah Khan , "Ben Skeggs" , Bharata B Rao , "Ralph Campbell" Subject: [PATCH 3/5] mm/notifier: add migration invalidation type Date: Mon, 6 Jul 2020 15:23:45 -0700 Message-ID: <20200706222347.32290-4-rcampbell@nvidia.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200706222347.32290-1-rcampbell@nvidia.com> References: <20200706222347.32290-1-rcampbell@nvidia.com> MIME-Version: 1.0 X-NVConfidentiality: public Content-Transfer-Encoding: quoted-printable Content-Type: text/plain DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1594074191; bh=T3rNe1ZTEVG4rl8pgs5SeLiYvOfEdwAB9yHLkYyf/KY=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:MIME-Version:X-NVConfidentiality: Content-Transfer-Encoding:Content-Type; b=lACYXImjh2MKxWWjgEAbn5Mb2PV8VPVQapX5510HmfjELKQmCjzJUHwb4sknDDLpI IBK0RUbMz/85An/pTVvFvqQvh0MY00fVMhgsCtmJSFJyXR2fleDBkkD9pqj+T3viaR hU4/zpU3284WwuwBqRmWGGwYCZd5ImIArCPpnt8Co+ZJQwUTklkxabolcER2//9kqn /PdtaZr3c4ket4bY6e+9NUJYVI0Zfv+ZRhdh2M0k9AYvPpVZjlFNtc92vS1OUs/pSi +xhN7+H3646QSrlE6O9t4dOM9PIRnQgEgZcPq90AK3zw+L/sRdsg/Io6V5iFh5iPJV yaJzaRnKwunsQ== Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Currently migrate_vma_setup() calls mmu_notifier_invalidate_range_start() which flushes all device private page mappings whether or not a page is being migrated to/from device private memory. In order to not disrupt device mappings that are not being migrated, shift the responsibility for clearing device private mappings to the device driver and leave CPU page table unmapping handled by migrate_vma_setup(). To support this, the caller of migrate_vma_setup() should always set struct migrate_vma::src_owner to a non NULL value that matches the device private page->pgmap->owner. This value is then passed to the struct mmu_notifier_range with a new event type which the driver's invalidation function can use to avoid device MMU invalidations. Signed-off-by: Ralph Campbell --- include/linux/mmu_notifier.h | 7 +++++++ mm/migrate.c | 8 +++++++- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index fc68f3570e19..bd0b34dbe4de 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -38,6 +38,10 @@ struct mmu_interval_notifier; * * @MMU_NOTIFY_RELEASE: used during mmu_interval_notifier invalidate to si= gnal * that the mm refcount is zero and the range is no longer accessible. + * + * @MMU_NOTIFY_MIGRATE: used during migrate_vma_collect() invalidate to si= gnal + * a device driver to possibly ignore the invalidation if the src_own + * field matches. */ enum mmu_notifier_event { MMU_NOTIFY_UNMAP =3D 0, @@ -46,6 +50,7 @@ enum mmu_notifier_event { MMU_NOTIFY_PROTECTION_PAGE, MMU_NOTIFY_SOFT_DIRTY, MMU_NOTIFY_RELEASE, + MMU_NOTIFY_MIGRATE, }; =20 #define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) @@ -264,6 +269,7 @@ struct mmu_notifier_range { unsigned long end; unsigned flags; enum mmu_notifier_event event; + void *data; }; =20 static inline int mm_has_notifiers(struct mm_struct *mm) @@ -513,6 +519,7 @@ static inline void mmu_notifier_range_init(struct mmu_n= otifier_range *range, range->start =3D start; range->end =3D end; range->flags =3D flags; + range->data =3D NULL; } =20 #define ptep_clear_flush_young_notify(__vma, __address, __ptep) \ diff --git a/mm/migrate.c b/mm/migrate.c index 2bbc5c4c672e..62270e6727b0 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -2391,8 +2391,14 @@ static void migrate_vma_collect(struct migrate_vma *= migrate) { struct mmu_notifier_range range; =20 - mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, + /* + * Note that the src_owner is passed to the mmu notifier callback so + * that the registered device driver can skip invalidating device + * private page mappings that won't be migrated. + */ + mmu_notifier_range_init(&range, MMU_NOTIFY_MIGRATE, 0, migrate->vma, migrate->vma->vm_mm, migrate->start, migrate->end); + range.data =3D migrate->src_owner; mmu_notifier_invalidate_range_start(&range); =20 walk_page_range(migrate->vma->vm_mm, migrate->start, migrate->end, --=20 2.20.1