Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20;
From:   Liam Howlett <liam.howlett@oracle.com>
To:     Michel Lespinasse <michel@lespinasse.org>
CC:     Linux-MM <linux-mm@kvack.org>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        Andrew Morton <akpm@linux-foundation.org>,
        "kernel-team@fb.com" <kernel-team@fb.com>,
        Laurent Dufour <ldufour@linux.ibm.com>,
        Jerome Glisse <jglisse@google.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Michal Hocko <mhocko@suse.com>,
        Vlastimil Babka <vbabka@suse.cz>,
        Davidlohr Bueso <dave@stgolabs.net>,
        Matthew Wilcox <willy@infradead.org>,
        Rik van Riel <riel@surriel.com>,
        Paul McKenney <paulmck@kernel.org>,
        Song Liu <songliubraving@fb.com>,
        Suren Baghdasaryan <surenb@google.com>,
        Minchan Kim <minchan@google.com>,
        Joel Fernandes <joelaf@google.com>,
        David Rientjes <rientjes@google.com>,
        Axel Rasmussen <axelrasmussen@google.com>,
        Andy Lutomirski <luto@kernel.org>
Subject: Re: [PATCH v2 13/35] x86/mm: attempt speculative mm faults first
Thread-Topic: [PATCH v2 13/35] x86/mm: attempt speculative mm faults first
Thread-Index: AQHYFEhl4v7PtaX0EUi6FfnWr7s+M6x+9mSA
Date:   Tue, 1 Feb 2022 17:16:43 +0000
Message-ID: <20220201171617.z5zgafitgolyv5np@revolver>
References: <20220128131006.67712-1-michel@lespinasse.org>
 <20220128131006.67712-14-michel@lespinasse.org>
In-Reply-To: <20220128131006.67712-14-michel@lespinasse.org>
Accept-Language: en-US
Content-Language: en-US
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?a2aumMr9JZsx++3cA2/u1ew8o/Keiy7+QJpSrNqDP7ydSF0aBpxI8GJzKIl1?=
 =?us-ascii?Q?D0nQHE7BTdpdCbh/NOyix7+8Ia74SALbPuSrdo1y4QS/rBOjRVf7QNM7XKN3?=
 =?us-ascii?Q?t6sZkOTWAmbu4iPVIvZFQUu0UH1ZMGV8lSKh2ijZ4PZEhQ/FosyTRzrfhFne?=
 =?us-ascii?Q?lErabUUXmUwKHo4Gjo1U2zgTMH9ySPEg3T9NBr95XbzBz4MFdIZIa1ymPHw+?=
 =?us-ascii?Q?/XAcE3NetEnAa8jneyop5XVjkD0RbBg9BuEVG4AWRfUOnnhq+rcEOfHtlxzY?=
 =?us-ascii?Q?fqNGuafA/fVMGhcbqHKE/xfDvlJZt35TqK1MdCBvm9/5v1/542B5FDMX1/Cm?=
 =?us-ascii?Q?YvdYBlQ2jBPQC8HtcA0m2F/L0wybQOHsEurhOQb7bCx0K6jU7kQKSKZsThXw?=
 =?us-ascii?Q?HvHmJjtEg6dvOhfZUVHPQZi1UC8cWHWoBbu06J8FhavfvWfY8ry4WQ2P1w7y?=
 =?us-ascii?Q?aXN7m27u2J6mowrW6P0fOvOExDWbdyIjEqoW4cNz64j3hDkKATWer5FRcfkH?=
 =?us-ascii?Q?FE0WWJfX8TLN/SB6W2XmBWv4ovs1mYUhkc59geElywluX/0G6u/1uDo+GLNl?=
 =?us-ascii?Q?NaDgfQTIA5J+PnatuapeO3/VtIfNgVTtZdhETwBJZGwphnH+QThKWNfJwp9q?=
 =?us-ascii?Q?4qFOvK5LoAc0uynav4e+Qca3vY4uOpXp/9I6/rLK7shfcs9l59uTkEq73s0f?=
 =?us-ascii?Q?JVDrzSStqVpOvmdVd3hsRFbsoB/8FJlSd7bl1FnFqdmp3iotIWRc6LYJtJ/J?=
 =?us-ascii?Q?GbdtnUnMeZwYHKhe/w7Yp2Dzrd6D0gcnLQIU5U8holz90xxNVsbmcOyB9aqg?=
 =?us-ascii?Q?ZkUsoiCg7mba/SypHbdlySDG3vPdS9hG9oW7ehTL73aw6iWVqrMyN2oiesHj?=
 =?us-ascii?Q?EhW84FEaTimPoPB0wL+/A1QqAYakKmn5+jKd2ypbzJfXgIP9M9DqFwn+6yqA?=
 =?us-ascii?Q?KebDgZ9/35cOHnIaDiE1h7GKNCnCIvQOkvXACcFTUt8xuv9HdZ8UG1Nexwqi?=
 =?us-ascii?Q?vQ/0XfARgoeJnRu7pAPwqsTwtcGhEzQi+2BlqwRNqM3A/0HV3FpTaOlWknuW?=
 =?us-ascii?Q?brxZxBe8oTWDHHkldinLzO1Gxag5WUvfVpC41+MxAM/SC8LXhRk7VCC8UiYc?=
 =?us-ascii?Q?UssqDRWnV1pG84AUZsndOjz6k61n6o3BXBb0t5kbfgzZEqe2bZwsA1ZuhzAc?=
 =?us-ascii?Q?QTAyl6lhJgTmZUOfpLGLBPRPPj0NF5bsKY7fPVl1rIwVmc84Hoi7FFb/einQ?=
 =?us-ascii?Q?HsTRZzJ+8OJrID+1yxitoD9Sz7Vym+A5wtxqipvjMEZuLlm4BqX1yMsr6bIY?=
 =?us-ascii?Q?I0fnqRPUuYH9gmKxnt/zW7UkJZWVPukIjBnsUx2FvEEdWrU+aMwHrYVUrU1j?=
 =?us-ascii?Q?/Bnduei+4009NTQNid9yLaYMzZA9cwDptpwxQkOl44gIqxZdF/DXvCpk8d6v?=
 =?us-ascii?Q?av8OhVPjrE8k6dR441xChegy5OsG75md9AA2pFg6o+VpFzSvSsMDxjM61SI1?=
 =?us-ascii?Q?Mg70CD5HDLnFseaFF8m+h/7szdI0J89nvawp/evfQIZy/I6it+cdicb2rhMW?=
 =?us-ascii?Q?+ANrql2UGdYG7B7NdNqFUE+4QdxWzgQxQAfG2iHYO3G0xgSICPVl97KHUiGy?=
 =?us-ascii?Q?MegHSocGhMogSbAjUaYHa8E=3D?=
Content-Type: text/plain; charset="us-ascii"
Content-ID: <360D3B7B2E89FD4EA43623E00C209378@namprd10.prod.outlook.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: SN6PR10MB3022.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b5377fe9-7cfd-461c-09a8-08d9e5a69e7b
X-MS-Exchange-CrossTenant-originalarrivaltime: 01 Feb 2022 17:16:43.4893
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: i5PaVO9Sk7Q4sRw6C1EP8ewqdro9Jxbi1txPtqcbc2FIuy2M/6LIWPhnfNiIxHytDlRG7zS2kNnPh6CG8mY70A==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR10MB5194
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10245 signatures=673430
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxscore=0 spamscore=0
 bulkscore=0 adultscore=0 mlxlogscore=691 phishscore=0 malwarescore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2201110000
 definitions=main-2202010097
X-Proofpoint-GUID: qKxjSgxONCIBdl7CzjdOYqGxXeE2eKJ2
X-Proofpoint-ORIG-GUID: qKxjSgxONCIBdl7CzjdOYqGxXeE2eKJ2
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

* Michel Lespinasse <michel@lespinasse.org> [220128 08:10]:
> Attempt speculative mm fault handling first, and fall back to the
> existing (non-speculative) code if that fails.
>=20
> The speculative handling closely mirrors the non-speculative logic.
> This includes some x86 specific bits such as the access_error() call.
> This is why we chose to implement the speculative handling in arch/x86
> rather than in common code.
>=20
> The vma is first looked up and copied, under protection of the rcu
> read lock. The mmap lock sequence count is used to verify the
> integrity of the copied vma, and passed to do_handle_mm_fault() to
> allow checking against races with mmap writers when finalizing the fault.
>=20
> Signed-off-by: Michel Lespinasse <michel@lespinasse.org>
> ---
>  arch/x86/mm/fault.c           | 44 +++++++++++++++++++++++++++++++++++
>  include/linux/mm_types.h      |  5 ++++
>  include/linux/vm_event_item.h |  4 ++++
>  mm/vmstat.c                   |  4 ++++
>  4 files changed, 57 insertions(+)
>=20
> diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
> index d0074c6ed31a..99b0a358154e 100644
> --- a/arch/x86/mm/fault.c
> +++ b/arch/x86/mm/fault.c
> @@ -1226,6 +1226,10 @@ void do_user_addr_fault(struct pt_regs *regs,
>  	struct mm_struct *mm;
>  	vm_fault_t fault;
>  	unsigned int flags =3D FAULT_FLAG_DEFAULT;
> +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT
> +	struct vm_area_struct pvma;
> +	unsigned long seq;
> +#endif
> =20
>  	tsk =3D current;
>  	mm =3D tsk->mm;
> @@ -1323,6 +1327,43 @@ void do_user_addr_fault(struct pt_regs *regs,
>  	}
>  #endif
> =20
> +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT
> +	count_vm_event(SPF_ATTEMPT);
> +	seq =3D mmap_seq_read_start(mm);
> +	if (seq & 1)
> +		goto spf_abort;
> +	rcu_read_lock();
> +	vma =3D __find_vma(mm, address);
> +	if (!vma || vma->vm_start > address) {

This fits the vma_lookup() pattern - although you will have to work
around the locking issue still.  This is the same for the other
platforms too; they fit the pattern also.

> +		rcu_read_unlock();
> +		goto spf_abort;
> +	}
> +	pvma =3D *vma;
> +	rcu_read_unlock();
> +	if (!mmap_seq_read_check(mm, seq))
> +		goto spf_abort;
> +	vma =3D &pvma;
> +	if (unlikely(access_error(error_code, vma)))
> +		goto spf_abort;
> +	fault =3D do_handle_mm_fault(vma, address,
> +				   flags | FAULT_FLAG_SPECULATIVE, seq, regs);
> +
> +	if (!(fault & VM_FAULT_RETRY))
> +		goto done;
> +
> +	/* Quick path to respond to signals */
> +	if (fault_signal_pending(fault, regs)) {
> +		if (!user_mode(regs))
> +			kernelmode_fixup_or_oops(regs, error_code, address,
> +						 SIGBUS, BUS_ADRERR,
> +						 ARCH_DEFAULT_PKEY);
> +		return;
> +	}
> +
> +spf_abort:
> +	count_vm_event(SPF_ABORT);
> +#endif
> +
>  	/*
>  	 * Kernel-mode access to the user address space should only occur
>  	 * on well-defined single instructions listed in the exception
> @@ -1419,6 +1460,9 @@ void do_user_addr_fault(struct pt_regs *regs,
>  	}
> =20
>  	mmap_read_unlock(mm);
> +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT
> +done:
> +#endif
>  	if (likely(!(fault & VM_FAULT_ERROR)))
>  		return;
> =20
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index b6678578a729..305f05d2a4bc 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -370,6 +370,11 @@ struct anon_vma_name {
>   * per VM-area/task. A VM area is any part of the process virtual memory
>   * space that has a special rule for the page-fault handlers (ie a share=
d
>   * library, the executable area etc).
> + *
> + * Note that speculative page faults make an on-stack copy of the VMA,
> + * so the structure size matters.
> + * (TODO - it would be preferable to copy only the required vma attribut=
es
> + *  rather than the entire vma).
>   */
>  struct vm_area_struct {
>  	/* The first cache line has the info for VMA tree walking. */
> diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.=
h
> index 7b2363388bfa..f00b3e36ff39 100644
> --- a/include/linux/vm_event_item.h
> +++ b/include/linux/vm_event_item.h
> @@ -133,6 +133,10 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOU=
T,
>  #ifdef CONFIG_X86
>  		DIRECT_MAP_LEVEL2_SPLIT,
>  		DIRECT_MAP_LEVEL3_SPLIT,
> +#endif
> +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT
> +		SPF_ATTEMPT,
> +		SPF_ABORT,
>  #endif
>  		NR_VM_EVENT_ITEMS
>  };
> diff --git a/mm/vmstat.c b/mm/vmstat.c
> index 4057372745d0..dbb0160e5558 100644
> --- a/mm/vmstat.c
> +++ b/mm/vmstat.c
> @@ -1390,6 +1390,10 @@ const char * const vmstat_text[] =3D {
>  	"direct_map_level2_splits",
>  	"direct_map_level3_splits",
>  #endif
> +#ifdef CONFIG_SPECULATIVE_PAGE_FAULT
> +	"spf_attempt",
> +	"spf_abort",
> +#endif
>  #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */
>  };
>  #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG =
*/
> --=20
> 2.20.1
> =