While stressing compaction, one run oopsed on NULL capc->cc in
__free_one_page()'s task_capc(zone): compact_zone_order() had been
interrupted, and a page was being freed in the return from interrupt.
Though you would not expect it from the source, both gccs I was using
(a 4.8.1 and a 7.5.0) had chosen to compile compact_zone_order() with
the ".cc = &cc" implemented by mov %rbx,-0xb0(%rbp) immediately before
callq compact_zone - long after the "current->capture_control = &capc".
An interrupt in between those finds capc->cc NULL (zeroed by an earlier
rep stos).
This could presumably be fixed by a barrier() before setting
current->capture_control in compact_zone_order(); but would also need
more care on return from compact_zone(), in order not to risk leaking
a page captured by interrupt just before capture_control is reset.
Maybe that is the preferable fix, but I felt safer for task_capc() to
exclude the rather surprising possibility of capture at interrupt time.
Fixes: 5e1f0f098b46 ("mm, compaction: capture a page under direct compaction")
Cc: [email protected] # 5.1+
Signed-off-by: Hugh Dickins <[email protected]>
---
mm/page_alloc.c | 1 +
1 file changed, 1 insertion(+)
--- 5.8-rc0/mm/page_alloc.c 2020-06-08 14:38:47.298625588 -0700
+++ linux/mm/page_alloc.c 2020-06-10 12:12:34.982950441 -0700
@@ -814,6 +814,7 @@ static inline struct capture_control *ta
struct capture_control *capc = current->capture_control;
return capc &&
+ in_task() &&
!(current->flags & PF_KTHREAD) &&
!capc->page &&
capc->cc->zone == zone &&
On Wed, Jun 10, 2020 at 01:48:59PM -0700, Hugh Dickins wrote:
> While stressing compaction, one run oopsed on NULL capc->cc in
> __free_one_page()'s task_capc(zone): compact_zone_order() had been
> interrupted, and a page was being freed in the return from interrupt.
>
> Though you would not expect it from the source, both gccs I was using
> (a 4.8.1 and a 7.5.0) had chosen to compile compact_zone_order() with
> the ".cc = &cc" implemented by mov %rbx,-0xb0(%rbp) immediately before
> callq compact_zone - long after the "current->capture_control = &capc".
> An interrupt in between those finds capc->cc NULL (zeroed by an earlier
> rep stos).
>
> This could presumably be fixed by a barrier() before setting
> current->capture_control in compact_zone_order(); but would also need
> more care on return from compact_zone(), in order not to risk leaking
> a page captured by interrupt just before capture_control is reset.
>
> Maybe that is the preferable fix, but I felt safer for task_capc() to
> exclude the rather surprising possibility of capture at interrupt time.
>
> Fixes: 5e1f0f098b46 ("mm, compaction: capture a page under direct compaction")
> Cc: [email protected] # 5.1+
> Signed-off-by: Hugh Dickins <[email protected]>
Acked-by: Mel Gorman <[email protected]>
--
Mel Gorman
SUSE Labs
On 6/10/20 10:48 PM, Hugh Dickins wrote:
> While stressing compaction, one run oopsed on NULL capc->cc in
> __free_one_page()'s task_capc(zone): compact_zone_order() had been
> interrupted, and a page was being freed in the return from interrupt.
>
> Though you would not expect it from the source, both gccs I was using
> (a 4.8.1 and a 7.5.0) had chosen to compile compact_zone_order() with
> the ".cc = &cc" implemented by mov %rbx,-0xb0(%rbp) immediately before
> callq compact_zone - long after the "current->capture_control = &capc".
> An interrupt in between those finds capc->cc NULL (zeroed by an earlier
> rep stos).
Ugh, nasty. Same here with gcc 10.
> This could presumably be fixed by a barrier() before setting
> current->capture_control in compact_zone_order(); but would also need
> more care on return from compact_zone(), in order not to risk leaking
> a page captured by interrupt just before capture_control is reset.
I was hoping a WRITE_ONCE(current->capture_control) would be enough,
but apparently it's not (I tried).
> Maybe that is the preferable fix, but I felt safer for task_capc() to
> exclude the rather surprising possibility of capture at interrupt time.
> Fixes: 5e1f0f098b46 ("mm, compaction: capture a page under direct compaction")
> Cc: [email protected] # 5.1+
> Signed-off-by: Hugh Dickins <[email protected]>
Acked-by: Vlastimil Babka <[email protected]>
But perhaps I would also make sure that we don't expose the half initialized
capture_control and run into this problem again later. It's not like this is a
fast path where barriers hurt. Something like this then? (with added comments)
diff --git a/mm/compaction.c b/mm/compaction.c
index fd988b7e5f2b..c89e26817278 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -2316,15 +2316,17 @@ static enum compact_result compact_zone_order(struct zone *zone, int order,
.page = NULL,
};
- current->capture_control = &capc;
+ barrier();
+
+ WRITE_ONCE(current->capture_control, &capc);
ret = compact_zone(&cc, &capc);
VM_BUG_ON(!list_empty(&cc.freepages));
VM_BUG_ON(!list_empty(&cc.migratepages));
- *capture = capc.page;
- current->capture_control = NULL;
+ WRITE_ONCE(current->capture_control, NULL);
+ *capture = READ_ONCE(capc.page);
return ret;
}
On Fri, 12 Jun 2020, Vlastimil Babka wrote:
> On 6/10/20 10:48 PM, Hugh Dickins wrote:
> > While stressing compaction, one run oopsed on NULL capc->cc in
> > __free_one_page()'s task_capc(zone): compact_zone_order() had been
> > interrupted, and a page was being freed in the return from interrupt.
> >
> > Though you would not expect it from the source, both gccs I was using
> > (a 4.8.1 and a 7.5.0) had chosen to compile compact_zone_order() with
> > the ".cc = &cc" implemented by mov %rbx,-0xb0(%rbp) immediately before
> > callq compact_zone - long after the "current->capture_control = &capc".
> > An interrupt in between those finds capc->cc NULL (zeroed by an earlier
> > rep stos).
>
> Ugh, nasty. Same here with gcc 10.
Thanks for checking, nice to know that I'm in good company :)
>
> > This could presumably be fixed by a barrier() before setting
> > current->capture_control in compact_zone_order(); but would also need
> > more care on return from compact_zone(), in order not to risk leaking
> > a page captured by interrupt just before capture_control is reset.
>
> I was hoping a WRITE_ONCE(current->capture_control) would be enough,
> but apparently it's not (I tried).
Right, I don't think volatiles themselves actually constitute barriers;
but I'd better keep quiet, I notice the READ_ONCE/WRITE_ONCE/data_race
industry has been busy recently, and I'm likely out-of-date and mistaken.
>
> > Maybe that is the preferable fix, but I felt safer for task_capc() to
> > exclude the rather surprising possibility of capture at interrupt time.
>
> > Fixes: 5e1f0f098b46 ("mm, compaction: capture a page under direct compaction")
> > Cc: [email protected] # 5.1+
> > Signed-off-by: Hugh Dickins <[email protected]>
>
> Acked-by: Vlastimil Babka <[email protected]>
Thanks, and to Mel for his.
>
> But perhaps I would also make sure that we don't expose the half initialized
> capture_control and run into this problem again later. It's not like this is a
> fast path where barriers hurt. Something like this then? (with added comments)
Would it be very rude if I leave that to you and to Mel? to add, or
to replace mine if you wish - go ahead. I can easily see that more
sophistication at the compact_zone_order() end may be preferable to
another test and branch inside __free_one_page() (and would task_capc()
be better with an "unlikely" in it?).
But it seems unnecessary to have a fix at both ends, and I'm rather too
wound up in other things at the moment, to want to read up on the current
state of such barriers, and sign off on the Vlastipatch below myself (but
I do notice that READ_ONCE seems to have more in it today than I remember,
which probably accounts for why you did not put the barrier() I expected
to see on the way out).
Hugh
>
> diff --git a/mm/compaction.c b/mm/compaction.c
> index fd988b7e5f2b..c89e26817278 100644
> --- a/mm/compaction.c
> +++ b/mm/compaction.c
> @@ -2316,15 +2316,17 @@ static enum compact_result compact_zone_order(struct zone *zone, int order,
> .page = NULL,
> };
>
> - current->capture_control = &capc;
> + barrier();
> +
> + WRITE_ONCE(current->capture_control, &capc);
>
> ret = compact_zone(&cc, &capc);
>
> VM_BUG_ON(!list_empty(&cc.freepages));
> VM_BUG_ON(!list_empty(&cc.migratepages));
>
> - *capture = capc.page;
> - current->capture_control = NULL;
> + WRITE_ONCE(current->capture_control, NULL);
> + *capture = READ_ONCE(capc.page);
>
> return ret;
> }
On 6/15/20 11:03 PM, Hugh Dickins wrote:
> On Fri, 12 Jun 2020, Vlastimil Babka wrote:
>> > This could presumably be fixed by a barrier() before setting
>> > current->capture_control in compact_zone_order(); but would also need
>> > more care on return from compact_zone(), in order not to risk leaking
>> > a page captured by interrupt just before capture_control is reset.
>>
>> I was hoping a WRITE_ONCE(current->capture_control) would be enough,
>> but apparently it's not (I tried).
>
> Right, I don't think volatiles themselves actually constitute barriers;
> but I'd better keep quiet, I notice the READ_ONCE/WRITE_ONCE/data_race
> industry has been busy recently, and I'm likely out-of-date and mistaken.
Same here, but from what I've read, volatiles should enforce order against other
volatiles, but not non-volatiles (which is the struct initialization). So
barrier() is indeed necessary, and WRITE_ONCE just to prevent (very
hypothetical, hopefully) store tearing.
>>
>> > Maybe that is the preferable fix, but I felt safer for task_capc() to
>> > exclude the rather surprising possibility of capture at interrupt time.
>>
>> > Fixes: 5e1f0f098b46 ("mm, compaction: capture a page under direct compaction")
>> > Cc: [email protected] # 5.1+
>> > Signed-off-by: Hugh Dickins <[email protected]>
>>
>> Acked-by: Vlastimil Babka <[email protected]>
>
> Thanks, and to Mel for his.
>
>>
>> But perhaps I would also make sure that we don't expose the half initialized
>> capture_control and run into this problem again later. It's not like this is a
>> fast path where barriers hurt. Something like this then? (with added comments)
>
> Would it be very rude if I leave that to you and to Mel? to add, or
No problem.
> to replace mine if you wish - go ahead. I can easily see that more
> sophistication at the compact_zone_order() end may be preferable to
> another test and branch inside __free_one_page()
Right, I think so, and will also generally sleep better if we don't put pointers
to unitialized structures to current.
> (and would task_capc()
> be better with an "unlikely" in it?).
I'll try and see if it generates better code. We should be also able to remove
the "capc->cc->direct_compaction" check, as the only place where we set capc is
compact_zone_order() which sets direct_compaction true unconditionally.
> But it seems unnecessary to have a fix at both ends, and I'm rather too
> wound up in other things at the moment, to want to read up on the current
> state of such barriers, and sign off on the Vlastipatch below myself (but
> I do notice that READ_ONCE seems to have more in it today than I remember,
> which probably accounts for why you did not put the barrier() I expected
> to see on the way out).
Right, minimally it's a volatile cast (I've checked 5.1 too, for stable reasons)
which should be enough.
So I'll send the proper patch.
Thanks!
Vlastimil
> Hugh
>
>>
>> diff --git a/mm/compaction.c b/mm/compaction.c
>> index fd988b7e5f2b..c89e26817278 100644
>> --- a/mm/compaction.c
>> +++ b/mm/compaction.c
>> @@ -2316,15 +2316,17 @@ static enum compact_result compact_zone_order(struct zone *zone, int order,
>> .page = NULL,
>> };
>>
>> - current->capture_control = &capc;
>> + barrier();
>> +
>> + WRITE_ONCE(current->capture_control, &capc);
>>
>> ret = compact_zone(&cc, &capc);
>>
>> VM_BUG_ON(!list_empty(&cc.freepages));
>> VM_BUG_ON(!list_empty(&cc.migratepages));
>>
>> - *capture = capc.page;
>> - current->capture_control = NULL;
>> + WRITE_ONCE(current->capture_control, NULL);
>> + *capture = READ_ONCE(capc.page);
>>
>> return ret;
>> }
>
Hugh reports:
=====
While stressing compaction, one run oopsed on NULL capc->cc in
__free_one_page()'s task_capc(zone): compact_zone_order() had been
interrupted, and a page was being freed in the return from interrupt.
Though you would not expect it from the source, both gccs I was using
(a 4.8.1 and a 7.5.0) had chosen to compile compact_zone_order() with
the ".cc = &cc" implemented by mov %rbx,-0xb0(%rbp) immediately before
callq compact_zone - long after the "current->capture_control = &capc".
An interrupt in between those finds capc->cc NULL (zeroed by an earlier
rep stos).
This could presumably be fixed by a barrier() before setting
current->capture_control in compact_zone_order(); but would also need
more care on return from compact_zone(), in order not to risk leaking
a page captured by interrupt just before capture_control is reset.
Maybe that is the preferable fix, but I felt safer for task_capc() to
exclude the rather surprising possibility of capture at interrupt time.
=====
I have checked that gcc10 also behaves the same.
The advantage of fix in compact_zone_order() is that we don't add another
test in the page freeing hot path, and that it might prevent future problems
if we stop exposing pointers to unitialized structures in current task.
So this patch implements the suggestion for compact_zone_order() with barrier()
(and WRITE_ONCE() to prevent store tearing) for setting
current->capture_control, and prevents page leaking with WRITE_ONCE/READ_ONCE
in the proper order.
Fixes: 5e1f0f098b46 ("mm, compaction: capture a page under direct compaction")
Cc: [email protected] # 5.1+
Reported-by: Hugh Dickins <[email protected]>
Suggested-by: Hugh Dickins <[email protected]>
Signed-off-by: Vlastimil Babka <[email protected]>
---
mm/compaction.c | 17 ++++++++++++++---
1 file changed, 14 insertions(+), 3 deletions(-)
diff --git a/mm/compaction.c b/mm/compaction.c
index fd988b7e5f2b..86375605faa9 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -2316,15 +2316,26 @@ static enum compact_result compact_zone_order(struct zone *zone, int order,
.page = NULL,
};
- current->capture_control = &capc;
+ /*
+ * Make sure the structs are really initialized before we expose the
+ * capture control, in case we are interrupted and the interrupt handler
+ * frees a page.
+ */
+ barrier();
+ WRITE_ONCE(current->capture_control, &capc);
ret = compact_zone(&cc, &capc);
VM_BUG_ON(!list_empty(&cc.freepages));
VM_BUG_ON(!list_empty(&cc.migratepages));
- *capture = capc.page;
- current->capture_control = NULL;
+ /*
+ * Make sure we hide capture control first before we read the captured
+ * page pointer, otherwise an interrupt could free and capture a page
+ * and we would leak it.
+ */
+ WRITE_ONCE(current->capture_control, NULL);
+ *capture = READ_ONCE(capc.page);
return ret;
}
--
2.27.0
On Tue, 16 Jun 2020, Vlastimil Babka wrote:
> Hugh reports:
>
> =====
> While stressing compaction, one run oopsed on NULL capc->cc in
> __free_one_page()'s task_capc(zone): compact_zone_order() had been
> interrupted, and a page was being freed in the return from interrupt.
>
> Though you would not expect it from the source, both gccs I was using
> (a 4.8.1 and a 7.5.0) had chosen to compile compact_zone_order() with
> the ".cc = &cc" implemented by mov %rbx,-0xb0(%rbp) immediately before
> callq compact_zone - long after the "current->capture_control = &capc".
> An interrupt in between those finds capc->cc NULL (zeroed by an earlier
> rep stos).
>
> This could presumably be fixed by a barrier() before setting
> current->capture_control in compact_zone_order(); but would also need
> more care on return from compact_zone(), in order not to risk leaking
> a page captured by interrupt just before capture_control is reset.
>
> Maybe that is the preferable fix, but I felt safer for task_capc() to
> exclude the rather surprising possibility of capture at interrupt time.
> =====
>
> I have checked that gcc10 also behaves the same.
>
> The advantage of fix in compact_zone_order() is that we don't add another
> test in the page freeing hot path, and that it might prevent future problems
> if we stop exposing pointers to unitialized structures in current task.
>
> So this patch implements the suggestion for compact_zone_order() with barrier()
> (and WRITE_ONCE() to prevent store tearing) for setting
> current->capture_control, and prevents page leaking with WRITE_ONCE/READ_ONCE
> in the proper order.
>
> Fixes: 5e1f0f098b46 ("mm, compaction: capture a page under direct compaction")
> Cc: [email protected] # 5.1+
> Reported-by: Hugh Dickins <[email protected]>
> Suggested-by: Hugh Dickins <[email protected]>
> Signed-off-by: Vlastimil Babka <[email protected]>
Acked-by: Hugh Dickins <[email protected]>
> ---
> mm/compaction.c | 17 ++++++++++++++---
> 1 file changed, 14 insertions(+), 3 deletions(-)
>
> diff --git a/mm/compaction.c b/mm/compaction.c
> index fd988b7e5f2b..86375605faa9 100644
> --- a/mm/compaction.c
> +++ b/mm/compaction.c
> @@ -2316,15 +2316,26 @@ static enum compact_result compact_zone_order(struct zone *zone, int order,
> .page = NULL,
> };
>
> - current->capture_control = &capc;
> + /*
> + * Make sure the structs are really initialized before we expose the
> + * capture control, in case we are interrupted and the interrupt handler
> + * frees a page.
> + */
> + barrier();
> + WRITE_ONCE(current->capture_control, &capc);
>
> ret = compact_zone(&cc, &capc);
>
> VM_BUG_ON(!list_empty(&cc.freepages));
> VM_BUG_ON(!list_empty(&cc.migratepages));
>
> - *capture = capc.page;
> - current->capture_control = NULL;
> + /*
> + * Make sure we hide capture control first before we read the captured
> + * page pointer, otherwise an interrupt could free and capture a page
> + * and we would leak it.
> + */
> + WRITE_ONCE(current->capture_control, NULL);
> + *capture = READ_ONCE(capc.page);
>
> return ret;
> }
> --
> 2.27.0