Add a couple of arch_prctl() handles:
- ARCH_ENABLE_TAGGED_ADDR enabled LAM. The argument is required number
of tag bits. It is rounded up to the nearest LAM mode that can
provide it. For now only LAM_U57 is supported, with 6 tag bits.
- ARCH_GET_UNTAG_MASK returns untag mask. It can indicates where tag
bits located in the address.
Signed-off-by: Kirill A. Shutemov <[email protected]>
---
arch/x86/include/uapi/asm/prctl.h | 3 ++
arch/x86/kernel/process_64.c | 55 ++++++++++++++++++++++++++++++-
2 files changed, 57 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h
index 500b96e71f18..38164a05c23c 100644
--- a/arch/x86/include/uapi/asm/prctl.h
+++ b/arch/x86/include/uapi/asm/prctl.h
@@ -20,4 +20,7 @@
#define ARCH_MAP_VDSO_32 0x2002
#define ARCH_MAP_VDSO_64 0x2003
+#define ARCH_GET_UNTAG_MASK 0x4001
+#define ARCH_ENABLE_TAGGED_ADDR 0x4002
+
#endif /* _ASM_X86_PRCTL_H */
diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c
index 1962008fe743..e328b91d1492 100644
--- a/arch/x86/kernel/process_64.c
+++ b/arch/x86/kernel/process_64.c
@@ -742,6 +742,55 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr)
}
#endif
+static void enable_lam_func(void *mm)
+{
+ struct mm_struct *loaded_mm = this_cpu_read(cpu_tlbstate.loaded_mm);
+
+ if (loaded_mm != mm)
+ return;
+
+ /* Counterpart of smp_wmb() in prctl_enable_tagged_addr() */
+ smp_rmb();
+
+ /* Update CR3 to get LAM active on the CPU */
+ switch_mm(loaded_mm, loaded_mm, current);
+}
+
+static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits)
+{
+ int ret = 0;
+
+ if (!cpu_feature_enabled(X86_FEATURE_LAM))
+ return -ENODEV;
+
+ mutex_lock(&mm->context.lock);
+
+ /* Already enabled? */
+ if (mm->context.lam_cr3_mask) {
+ ret = -EBUSY;
+ goto out;
+ }
+
+ if (!nr_bits) {
+ ret = -EINVAL;
+ goto out;
+ } else if (nr_bits <= 6) {
+ mm->context.lam_cr3_mask = X86_CR3_LAM_U57;
+ mm->context.untag_mask = ~GENMASK(62, 57);
+ } else {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ /* Make lam_cr3_mask and untag_mask visible on other CPUs */
+ smp_wmb();
+
+ on_each_cpu_mask(mm_cpumask(mm), enable_lam_func, mm, true);
+out:
+ mutex_unlock(&mm->context.lock);
+ return ret;
+}
+
long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)
{
int ret = 0;
@@ -829,7 +878,11 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2)
case ARCH_MAP_VDSO_64:
return prctl_map_vdso(&vdso_image_64, arg2);
#endif
-
+ case ARCH_GET_UNTAG_MASK:
+ return put_user(task->mm->context.untag_mask,
+ (unsigned long __user *)arg2);
+ case ARCH_ENABLE_TAGGED_ADDR:
+ return prctl_enable_tagged_addr(task->mm, arg2);
default:
ret = -EINVAL;
break;
--
2.35.1
On Wed, Jun 22, 2022 at 6:22 PM Kirill A. Shutemov
<[email protected]> wrote:
>
> Add a couple of arch_prctl() handles:
>
> - ARCH_ENABLE_TAGGED_ADDR enabled LAM. The argument is required number
> of tag bits. It is rounded up to the nearest LAM mode that can
> provide it. For now only LAM_U57 is supported, with 6 tag bits.
>
> - ARCH_GET_UNTAG_MASK returns untag mask. It can indicates where tag
> bits located in the address.
>
Am I right that the desired way to detect the presence of LAM without
enabling it is to check that arch_prctl(ARCH_GET_UNTAG_MASK, ...)
returns zero?
Overall, I think these new arch_prctls should be documented following
the spirit of PR_SET_TAGGED_ADDR_CTRL/PR_GET_TAGGED_ADDR_CTRL
somewhere.
> +
> +static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits)
> +{
> + int ret = 0;
> +
> + if (!cpu_feature_enabled(X86_FEATURE_LAM))
> + return -ENODEV;
> +
> + mutex_lock(&mm->context.lock);
> +
> + /* Already enabled? */
> + if (mm->context.lam_cr3_mask) {
> + ret = -EBUSY;
> + goto out;
> + }
> +
> + if (!nr_bits) {
> + ret = -EINVAL;
One would expect that `arch_prctl(ARCH_ENABLE_TAGGED_ADDR, 0)`
disables tagging for the current process.
Shouldn't this workflow be supported as well?
On Tue, Jul 12, 2022 at 03:12:01PM +0200, Alexander Potapenko wrote:
> On Wed, Jun 22, 2022 at 6:22 PM Kirill A. Shutemov
> <[email protected]> wrote:
> >
> > Add a couple of arch_prctl() handles:
> >
> > - ARCH_ENABLE_TAGGED_ADDR enabled LAM. The argument is required number
> > of tag bits. It is rounded up to the nearest LAM mode that can
> > provide it. For now only LAM_U57 is supported, with 6 tag bits.
> >
> > - ARCH_GET_UNTAG_MASK returns untag mask. It can indicates where tag
> > bits located in the address.
> >
> Am I right that the desired way to detect the presence of LAM without
> enabling it is to check that arch_prctl(ARCH_GET_UNTAG_MASK, ...)
> returns zero?
Returns -1UL, but yes.
> Overall, I think these new arch_prctls should be documented following
> the spirit of PR_SET_TAGGED_ADDR_CTRL/PR_GET_TAGGED_ADDR_CTRL
> somewhere.
The plan is to update man page for the syscall once the interface is
upstream.
> > +
> > +static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits)
> > +{
> > + int ret = 0;
> > +
> > + if (!cpu_feature_enabled(X86_FEATURE_LAM))
> > + return -ENODEV;
> > +
> > + mutex_lock(&mm->context.lock);
> > +
> > + /* Already enabled? */
> > + if (mm->context.lam_cr3_mask) {
> > + ret = -EBUSY;
> > + goto out;
> > + }
> > +
> > + if (!nr_bits) {
> > + ret = -EINVAL;
>
> One would expect that `arch_prctl(ARCH_ENABLE_TAGGED_ADDR, 0)`
> disables tagging for the current process.
> Shouldn't this workflow be supported as well?
Is there an use-case for it?
I would rather keep the interface minimal. We can always add this in the
future if an use-case comes.
--
Kirill A. Shutemov
On Tue, Jul 12, 2022 at 7:14 PM Kirill A. Shutemov
<[email protected]> wrote:
>
> On Tue, Jul 12, 2022 at 03:12:01PM +0200, Alexander Potapenko wrote:
> > On Wed, Jun 22, 2022 at 6:22 PM Kirill A. Shutemov
> > <[email protected]> wrote:
> > >
> > > Add a couple of arch_prctl() handles:
> > >
> > > - ARCH_ENABLE_TAGGED_ADDR enabled LAM. The argument is required number
> > > of tag bits. It is rounded up to the nearest LAM mode that can
> > > provide it. For now only LAM_U57 is supported, with 6 tag bits.
> > >
> > > - ARCH_GET_UNTAG_MASK returns untag mask. It can indicates where tag
> > > bits located in the address.
> > >
> > Am I right that the desired way to detect the presence of LAM without
> > enabling it is to check that arch_prctl(ARCH_GET_UNTAG_MASK, ...)
> > returns zero?
>
> Returns -1UL, but yes.
No, I meant the return value of arch_prctl(), but in fact neither
seems to be true.
Right now e.g. for the 5.17 kernel arch_prctl(ARCH_GET_UNTAG_MASK,
&bits) returns -EINVAL regardless of the underlying hardware.
A new kernel with your patches will return 0 and set bits=-1UL on both
non-LAM and LAM-enabled machines. How can we distinguish those?
> >
> > One would expect that `arch_prctl(ARCH_ENABLE_TAGGED_ADDR, 0)`
> > disables tagging for the current process.
> > Shouldn't this workflow be supported as well?
>
> Is there an use-case for it?
>
> I would rather keep the interface minimal. We can always add this in the
> future if an use-case comes.
As discussed offline, we don't have a use-case for this yet, so I don't insist.
> --
> Kirill A. Shutemov
--
Alexander Potapenko
Software Engineer
Google Germany GmbH
Erika-Mann-Straße, 33
80636 München
Geschäftsführer: Paul Manicle, Liana Sebastian
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
On Thu, Jul 14, 2022 at 04:28:36PM +0200, Alexander Potapenko wrote:
> On Tue, Jul 12, 2022 at 7:14 PM Kirill A. Shutemov
> <[email protected]> wrote:
> >
> > On Tue, Jul 12, 2022 at 03:12:01PM +0200, Alexander Potapenko wrote:
> > > On Wed, Jun 22, 2022 at 6:22 PM Kirill A. Shutemov
> > > <[email protected]> wrote:
> > > >
> > > > Add a couple of arch_prctl() handles:
> > > >
> > > > - ARCH_ENABLE_TAGGED_ADDR enabled LAM. The argument is required number
> > > > of tag bits. It is rounded up to the nearest LAM mode that can
> > > > provide it. For now only LAM_U57 is supported, with 6 tag bits.
> > > >
> > > > - ARCH_GET_UNTAG_MASK returns untag mask. It can indicates where tag
> > > > bits located in the address.
> > > >
> > > Am I right that the desired way to detect the presence of LAM without
> > > enabling it is to check that arch_prctl(ARCH_GET_UNTAG_MASK, ...)
> > > returns zero?
> >
> > Returns -1UL, but yes.
>
> No, I meant the return value of arch_prctl(), but in fact neither
> seems to be true.
>
> Right now e.g. for the 5.17 kernel arch_prctl(ARCH_GET_UNTAG_MASK,
> &bits) returns -EINVAL regardless of the underlying hardware.
> A new kernel with your patches will return 0 and set bits=-1UL on both
> non-LAM and LAM-enabled machines. How can we distinguish those?
With CPUID?
--
Kirill A. Shutemov