From: Takao Indoh <[email protected]>
I found a performance issue related on the implementation of Linux's TLB
flush for arm64.
When I run a single-threaded test program on moderate environment, it
usually takes 39ms to finish its work. However, when I put a small
apprication, which just calls mprotest() continuously, on one of sibling
cores and run it simultaneously, the test program slows down significantly.
It becomes 49ms(125%) on ThunderX2. I also detected the same problem on
ThunderX1 and Fujitsu A64FX.
I suppose the root cause of this issue is the implementation of Linux's TLB
flush for arm64, especially use of TLBI-is instruction which is a broadcast
to all processor core on the system. In case of the above situation,
TLBI-is is called by mprotect().
This is not a problem for small environment, but this causes a significant
performance noise for large-scale HPC environment, which has more than
thousand nodes with low latency interconnect.
To fix this problem, this patch adds new boot parameter
'disable_tlbflush_is'. In the case of flush_tlb_mm() *without* this
parameter, TLB entry is invalidated by __tlbi(aside1is, asid). By this
instruction, all CPUs within the same inner shareable domain check if there
are TLB entries which have this ASID, this causes performance noise. OTOH,
when this new parameter is specified, TLB entry is invalidated by
__tlbi(aside1, asid) only on the CPUs specified by mm_cpumask(mm).
Therefore TLB flush is done on minimal CPUs and performance problem does
not occur. Actually I confirm the performance problem is fixed by this
patch.
Takao Indoh (2):
arm64: mm: Restore mm_cpumask (revert commit 38d96287504a ("arm64: mm:
kill mm_cpumask usage"))
arm64: tlb: Add boot parameter to disable TLB flush within the same
inner shareable domain
.../admin-guide/kernel-parameters.txt | 4 +
arch/arm64/include/asm/mmu_context.h | 7 +-
arch/arm64/include/asm/tlbflush.h | 61 ++-----
arch/arm64/kernel/Makefile | 2 +-
arch/arm64/kernel/smp.c | 6 +
arch/arm64/kernel/tlbflush.c | 155 ++++++++++++++++++
arch/arm64/mm/context.c | 2 +
7 files changed, 186 insertions(+), 51 deletions(-)
create mode 100644 arch/arm64/kernel/tlbflush.c
--
2.20.1
Hi Takao,
[+Peter Z]
On Mon, Jun 17, 2019 at 11:32:53PM +0900, Takao Indoh wrote:
> From: Takao Indoh <[email protected]>
>
> I found a performance issue related on the implementation of Linux's TLB
> flush for arm64.
>
> When I run a single-threaded test program on moderate environment, it
> usually takes 39ms to finish its work. However, when I put a small
> apprication, which just calls mprotest() continuously, on one of sibling
> cores and run it simultaneously, the test program slows down significantly.
> It becomes 49ms(125%) on ThunderX2. I also detected the same problem on
> ThunderX1 and Fujitsu A64FX.
This is a problem for any applications that share hardware resources with
each other, so I don't think it's something we should be too concerned about
addressing unless there is a practical DoS scenario, which there doesn't
appear to be in this case. It may be that the real answer is "don't call
mprotect() in a loop".
> I suppose the root cause of this issue is the implementation of Linux's TLB
> flush for arm64, especially use of TLBI-is instruction which is a broadcast
> to all processor core on the system. In case of the above situation,
> TLBI-is is called by mprotect().
On the flip side, Linux is providing the hardware with enough information
not to broadcast to cores for which the remote TLBs don't have entries
allocated for the ASID being invalidated. I would say that the root cause
of the issue is that this filtering is not taking place.
> This is not a problem for small environment, but this causes a significant
> performance noise for large-scale HPC environment, which has more than
> thousand nodes with low latency interconnect.
If you have a system with over a thousand nodes, without snoop filtering
for DVM messages and you expect performance to scale in the face of tight
mprotect() loops then I think you have a problem irrespective of this patch.
What happens if somebody runs I-cache invalidation in a loop?
> To fix this problem, this patch adds new boot parameter
> 'disable_tlbflush_is'. In the case of flush_tlb_mm() *without* this
> parameter, TLB entry is invalidated by __tlbi(aside1is, asid). By this
> instruction, all CPUs within the same inner shareable domain check if there
> are TLB entries which have this ASID, this causes performance noise. OTOH,
> when this new parameter is specified, TLB entry is invalidated by
> __tlbi(aside1, asid) only on the CPUs specified by mm_cpumask(mm).
> Therefore TLB flush is done on minimal CPUs and performance problem does
> not occur. Actually I confirm the performance problem is fixed by this
> patch.
Other than my comments above, my overall concern with this patch is that
it introduces divergent behaviour for our TLB invalidation flow, which is
undesirable from both maintainability and usability perspectives. If you
wish to change the code, please don't put it behind a command-line option,
but instead improve the code that is already there. However, I suspect that
blowing away the local TLB on every context-switch may have hidden costs
which are only apparent with workloads different from the contrived case
that you're seeking to improve. You also haven't taken into account the
effects of virtualisation, where it's likely that the hypervisor will
upgrade non-shareable operations to inner-shareable ones anyway.
Thanks,
Will
Hi Will,
I am Takao's colleague, thank you very much for your reply.
On 6/18/19 2:03 AM, Will Deacon wrote:
> Hi Takao,
>
> [+Peter Z]
>
> On Mon, Jun 17, 2019 at 11:32:53PM +0900, Takao Indoh wrote:
>> From: Takao Indoh <[email protected]>
>>
>> I found a performance issue related on the implementation of Linux's TLB
>> flush for arm64.
>>
>> When I run a single-threaded test program on moderate environment, it
>> usually takes 39ms to finish its work. However, when I put a small
>> apprication, which just calls mprotest() continuously, on one of sibling
>> cores and run it simultaneously, the test program slows down significantly.
>> It becomes 49ms(125%) on ThunderX2. I also detected the same problem on
>> ThunderX1 and Fujitsu A64FX.
> This is a problem for any applications that share hardware resources with
> each other, so I don't think it's something we should be too concerned about
> addressing unless there is a practical DoS scenario, which there doesn't
> appear to be in this case. It may be that the real answer is "don't call
> mprotect() in a loop".
I think there has been a misunderstanding, please let me explain.
This application is just an example using for reproducing the
performance issue we found.
Our original purpose is reducing OS jitter by this series.
The OS jitter on massively parallel processing systems have been known
and studied for many years.
The 2.5% OS jitter can result in over a factor of 20 slowdown for the
same application [1].
Though it may be an extreme example, reducing the OS jitter has been an
issue in HPC environment.
[1] Ferreira, Kurt B., Patrick Bridges, and Ron Brightwell.
"Characterizing application sensitivity to OS interference using
kernel-level noise injection." Proceedings of the 2008 ACM/IEEE
conference on Supercomputing. IEEE Press, 2008.
>> I suppose the root cause of this issue is the implementation of Linux's TLB
>> flush for arm64, especially use of TLBI-is instruction which is a broadcast
>> to all processor core on the system. In case of the above situation,
>> TLBI-is is called by mprotect().
> On the flip side, Linux is providing the hardware with enough information
> not to broadcast to cores for which the remote TLBs don't have entries
> allocated for the ASID being invalidated. I would say that the root cause
> of the issue is that this filtering is not taking place.
Do you mean that the filter should be implemented in hardware?
Thanks,
Qi Fuli
>> This is not a problem for small environment, but this causes a significant
>> performance noise for large-scale HPC environment, which has more than
>> thousand nodes with low latency interconnect.
> If you have a system with over a thousand nodes, without snoop filtering
> for DVM messages and you expect performance to scale in the face of tight
> mprotect() loops then I think you have a problem irrespective of this patch.
> What happens if somebody runs I-cache invalidation in a loop?
>
>> To fix this problem, this patch adds new boot parameter
>> 'disable_tlbflush_is'. In the case of flush_tlb_mm() *without* this
>> parameter, TLB entry is invalidated by __tlbi(aside1is, asid). By this
>> instruction, all CPUs within the same inner shareable domain check if there
>> are TLB entries which have this ASID, this causes performance noise. OTOH,
>> when this new parameter is specified, TLB entry is invalidated by
>> __tlbi(aside1, asid) only on the CPUs specified by mm_cpumask(mm).
>> Therefore TLB flush is done on minimal CPUs and performance problem does
>> not occur. Actually I confirm the performance problem is fixed by this
>> patch.
> Other than my comments above, my overall concern with this patch is that
> it introduces divergent behaviour for our TLB invalidation flow, which is
> undesirable from both maintainability and usability perspectives. If you
> wish to change the code, please don't put it behind a command-line option,
> but instead improve the code that is already there. However, I suspect that
> blowing away the local TLB on every context-switch may have hidden costs
> which are only apparent with workloads different from the contrived case
> that you're seeking to improve. You also haven't taken into account the
> effects of virtualisation, where it's likely that the hypervisor will
> upgrade non-shareable operations to inner-shareable ones anyway.
>
> Thanks,
>
> Will
On Mon, Jun 24, 2019 at 10:34:02AM +0000, [email protected] wrote:
> On 6/18/19 2:03 AM, Will Deacon wrote:
> > On Mon, Jun 17, 2019 at 11:32:53PM +0900, Takao Indoh wrote:
> >> From: Takao Indoh <[email protected]>
> >>
> >> I found a performance issue related on the implementation of Linux's TLB
> >> flush for arm64.
> >>
> >> When I run a single-threaded test program on moderate environment, it
> >> usually takes 39ms to finish its work. However, when I put a small
> >> apprication, which just calls mprotest() continuously, on one of sibling
> >> cores and run it simultaneously, the test program slows down significantly.
> >> It becomes 49ms(125%) on ThunderX2. I also detected the same problem on
> >> ThunderX1 and Fujitsu A64FX.
> > This is a problem for any applications that share hardware resources with
> > each other, so I don't think it's something we should be too concerned about
> > addressing unless there is a practical DoS scenario, which there doesn't
> > appear to be in this case. It may be that the real answer is "don't call
> > mprotect() in a loop".
> I think there has been a misunderstanding, please let me explain.
> This application is just an example using for reproducing the
> performance issue we found.
> Our original purpose is reducing OS jitter by this series.
> The OS jitter on massively parallel processing systems have been known
> and studied for many years.
> The 2.5% OS jitter can result in over a factor of 20 slowdown for the
> same application [1].
I think it's worth pointing out that the system in question was neither
ARM-based nor running Linux, so I'd be cautious in applying the conclusions
of that paper directly to our TLB invalidation code. Furthermore, the noise
being generated in their experiments uses a timer interrupt, which has a
/vastly/ different profile to a DVM message in terms of both system impact
and frequency.
> Though it may be an extreme example, reducing the OS jitter has been an
> issue in HPC environment.
>
> [1] Ferreira, Kurt B., Patrick Bridges, and Ron Brightwell.
> "Characterizing application sensitivity to OS interference using
> kernel-level noise injection." Proceedings of the 2008 ACM/IEEE
> conference on Supercomputing. IEEE Press, 2008.
>
> >> I suppose the root cause of this issue is the implementation of Linux's TLB
> >> flush for arm64, especially use of TLBI-is instruction which is a broadcast
> >> to all processor core on the system. In case of the above situation,
> >> TLBI-is is called by mprotect().
> > On the flip side, Linux is providing the hardware with enough information
> > not to broadcast to cores for which the remote TLBs don't have entries
> > allocated for the ASID being invalidated. I would say that the root cause
> > of the issue is that this filtering is not taking place.
>
> Do you mean that the filter should be implemented in hardware?
Yes. If you're building a large system and you care about "jitter", then
you either need to partition it in such a way that sources of noise are
contained, or you need to introduce filters to limit their scope. Rewriting
the low-level memory-management parts of the operating system is a red
herring and imposes a needless burden on everybody else without solving
the real problem, which is that contended use of shared resources doesn't
scale.
Will
Hi Will,
Thanks for your comments.
On 6/27/19 7:27 PM, Will Deacon wrote:
> On Mon, Jun 24, 2019 at 10:34:02AM +0000, [email protected] wrote:
>> On 6/18/19 2:03 AM, Will Deacon wrote:
>>> On Mon, Jun 17, 2019 at 11:32:53PM +0900, Takao Indoh wrote:
>>>> From: Takao Indoh <[email protected]>
>>>>
>>>> I found a performance issue related on the implementation of Linux's TLB
>>>> flush for arm64.
>>>>
>>>> When I run a single-threaded test program on moderate environment, it
>>>> usually takes 39ms to finish its work. However, when I put a small
>>>> apprication, which just calls mprotest() continuously, on one of sibling
>>>> cores and run it simultaneously, the test program slows down significantly.
>>>> It becomes 49ms(125%) on ThunderX2. I also detected the same problem on
>>>> ThunderX1 and Fujitsu A64FX.
>>> This is a problem for any applications that share hardware resources with
>>> each other, so I don't think it's something we should be too concerned about
>>> addressing unless there is a practical DoS scenario, which there doesn't
>>> appear to be in this case. It may be that the real answer is "don't call
>>> mprotect() in a loop".
>> I think there has been a misunderstanding, please let me explain.
>> This application is just an example using for reproducing the
>> performance issue we found.
>> Our original purpose is reducing OS jitter by this series.
>> The OS jitter on massively parallel processing systems have been known
>> and studied for many years.
>> The 2.5% OS jitter can result in over a factor of 20 slowdown for the
>> same application [1].
> I think it's worth pointing out that the system in question was neither
> ARM-based nor running Linux, so I'd be cautious in applying the conclusions
> of that paper directly to our TLB invalidation code. Furthermore, the noise
> being generated in their experiments uses a timer interrupt, which has a
> /vastly/ different profile to a DVM message in terms of both system impact
> and frequency.
My original purpose was to explain that the OS jitter is a vital issue for
large-scale HPC environment by referencing this paper.
Please allow me to introduce the issue that had occurred to our HPC
environment.
We used FWQ [1] to do an experiment on 1 node of our HPC environment,
we expected it would be tens of microseconds of maximum OS jitter, but
it was
hundreds of microseconds, which didn't meet our requirement. We tried to
find
out the cause by using ftrace, but we cannot find any processes which would
cause noise and only knew the extension of processing time. Then we
confirmed
the CPU instruction count through CPU PMU, we also didn't find any changes.
However, we found that with the increase of that the TLB flash was called,
the noise was also increasing. Here we understood that the cause of this
issue
is the implementation of Linux's TLB flush for arm64, especially use of
TLBI-is
instruction which is a broadcast to all processor core on the system.
Therefore,
we made this patch set to fix this issue. After testing for several
times, the
noise was reduced and our original goal was achieved, so we do think
this patch
makes sense.
As I mentioned, the OS jitter is a vital issue for large-scale HPC
environment.
We tried a lot of things to reduce the OS jitter. One of them is task
separation
between the CPUs which are used for computing and the CPUs which are
used for
maintenance. All of the daemon processes and I/O interrupts are bounden
to the
maintenance CPUs. Further more, we used nohz_full to avoid the noise
caused by
computing CPU interruption, but all of the CPUs were affected by TLBI-is
instruction, the task separation of CPUs didn't work. Therefore, we
would like
to implement that TLB flush is done on minimal CPUs to reducing the OS
jitter
by using this patch set.
[1] https://asc.llnl.gov/sequoia/benchmarks/FTQ_summary_v1.1.pdf
Thanks,
QI Fuli
>> Though it may be an extreme example, reducing the OS jitter has been an
>> issue in HPC environment.
>>
>> [1] Ferreira, Kurt B., Patrick Bridges, and Ron Brightwell.
>> "Characterizing application sensitivity to OS interference using
>> kernel-level noise injection." Proceedings of the 2008 ACM/IEEE
>> conference on Supercomputing. IEEE Press, 2008.
>>
>>>> I suppose the root cause of this issue is the implementation of Linux's TLB
>>>> flush for arm64, especially use of TLBI-is instruction which is a broadcast
>>>> to all processor core on the system. In case of the above situation,
>>>> TLBI-is is called by mprotect().
>>> On the flip side, Linux is providing the hardware with enough information
>>> not to broadcast to cores for which the remote TLBs don't have entries
>>> allocated for the ASID being invalidated. I would say that the root cause
>>> of the issue is that this filtering is not taking place.
>> Do you mean that the filter should be implemented in hardware?
> Yes. If you're building a large system and you care about "jitter", then
> you either need to partition it in such a way that sources of noise are
> contained, or you need to introduce filters to limit their scope. Rewriting
> the low-level memory-management parts of the operating system is a red
> herring and imposes a needless burden on everybody else without solving
> the real problem, which is that contended use of shared resources doesn't
> scale.
>
> Will
On 7/8/19 8:25 PM, Jon Masters wrote:
> On 7/2/19 10:45 PM, [email protected] wrote:
>
>> However, we found that with the increase of that the TLB flash was called,
>> the noise was also increasing. Here we understood that the cause of this
>> issue is the implementation of Linux's TLB flush for arm64, especially use of
>> TLBI-is instruction which is a broadcast to all processor core on the system.
>
> Are you saying that for a microbenchmark in which very large numbers of
> threads are created and destroyed rapidly there are a large number of
> associated tlb range flushes which always use broadcast TLBIs?
>
> If that's the case, and the hardware doesn't do any ASID filtering and
> each TLBI results in a DVM to every PE, would it make sense to look at
> whether there are ways to improve batching/switch to an IPI approach
> rather than relying on broadcasts, as a more generic solution?
What I meant was a heuristic to do this automatically, rather than via a
command line.
Jon.
On 7/2/19 10:45 PM, [email protected] wrote:
> However, we found that with the increase of that the TLB flash was called,
> the noise was also increasing. Here we understood that the cause of this
> issue is the implementation of Linux's TLB flush for arm64, especially use of
> TLBI-is instruction which is a broadcast to all processor core on the system.
Are you saying that for a microbenchmark in which very large numbers of
threads are created and destroyed rapidly there are a large number of
associated tlb range flushes which always use broadcast TLBIs?
If that's the case, and the hardware doesn't do any ASID filtering and
each TLBI results in a DVM to every PE, would it make sense to look at
whether there are ways to improve batching/switch to an IPI approach
rather than relying on broadcasts, as a more generic solution?
Jon.
On Mon, Jul 08, 2019 at 08:29:26PM -0400, Jon Masters wrote:
> On 7/8/19 8:25 PM, Jon Masters wrote:
> > On 7/2/19 10:45 PM, [email protected] wrote:
> >
> >> However, we found that with the increase of that the TLB flash was called,
> >> the noise was also increasing. Here we understood that the cause of this
> >> issue is the implementation of Linux's TLB flush for arm64, especially use of
> >> TLBI-is instruction which is a broadcast to all processor core on the system.
> >
> > Are you saying that for a microbenchmark in which very large numbers of
> > threads are created and destroyed rapidly there are a large number of
> > associated tlb range flushes which always use broadcast TLBIs?
> >
> > If that's the case, and the hardware doesn't do any ASID filtering and
> > each TLBI results in a DVM to every PE, would it make sense to look at
> > whether there are ways to improve batching/switch to an IPI approach
> > rather than relying on broadcasts, as a more generic solution?
>
> What I meant was a heuristic to do this automatically, rather than via a
> command line.
One of my main initial objections to this patch [1] still applies to that
approach, though, which is that I don't want the maintenance headache of
maintaining two very different TLB invalidation schemes in the kernel.
Dynamically switching between them is arguably worse. If "jitter" is such a
big deal, then I don't think changing our TLBI mechanism even helps on a
system that has broadcast cache maintenance (including for the I-side) as
well as shared levels of cache further from the CPUs -- it just happens to
solve the case of a spinning mprotect(), well yeah, maybe don't do that if
your hardware can't handle it gracefully.
What I would be interested in seeing is an evaluation of a real workload
that suffers due to our mmu_gather/tlb_flush implementation on arm64 so that
we can understand where the problem lies and whether or not we can do
something to address it. But "jitter is bad, use IPIs" isn't helpful at all.
Will
[1] https://lkml.kernel.org/r/[email protected]
On Wed, Jul 03, 2019 at 02:45:43AM +0000, [email protected] wrote:
> We used FWQ [1] to do an experiment on 1 node of our HPC environment,
> we expected it would be tens of microseconds of maximum OS jitter, but
> it was
> hundreds of microseconds, which didn't meet our requirement. We tried to
> find
> out the cause by using ftrace, but we cannot find any processes which would
> cause noise and only knew the extension of processing time. Then we
> confirmed
> the CPU instruction count through CPU PMU, we also didn't find any changes.
> However, we found that with the increase of that the TLB flash was called,
> the noise was also increasing. Here we understood that the cause of this
> issue
> is the implementation of Linux's TLB flush for arm64, especially use of
> TLBI-is
> instruction which is a broadcast to all processor core on the system.
> Therefore,
> we made this patch set to fix this issue. After testing for several
> times, the
> noise was reduced and our original goal was achieved, so we do think
> this patch
> makes sense.
>
> As I mentioned, the OS jitter is a vital issue for large-scale HPC
> environment.
> We tried a lot of things to reduce the OS jitter. One of them is task
> separation
> between the CPUs which are used for computing and the CPUs which are
> used for
> maintenance. All of the daemon processes and I/O interrupts are bounden
> to the
> maintenance CPUs. Further more, we used nohz_full to avoid the noise
> caused by
> computing CPU interruption, but all of the CPUs were affected by TLBI-is
> instruction, the task separation of CPUs didn't work. Therefore, we
> would like
> to implement that TLB flush is done on minimal CPUs to reducing the OS
> jitter
> by using this patch set.
So have you confirmed that this is due to TLBI traffic and not, for example,
stores sitting in remote store buffers that get flushed by the IPI or
something else like that? It feels like you're inferring things about the
underlying behaviour, whereas you should be in a position to simulate this
if nothing else.
If it *is* because of TLBI, then where are they coming from? Is FWQ calling
munmap/mprotect all the time? Why?
Will
Hi,
First of all thanks for the comments for the patch.
I'm still struggling with this problem to find out the solution.
As a result of an investigation on this problem, after all, I think it
is necessary to improve TLB flush mechanism of the kernel to fix this
problem completely.
So, I'd like to restart a discussion. At first, I summarize this problem
to recall what was the problem and then I want to discuss how to fix it.
Summary of the problem:
A few months ago I proposed patches to solve a performance problem due
to TLB flush.[1]
A problem is that TLB flush on a core affects all other cores even if
all other cores do not need actual flush, and it causes performance
degradation.
In this thread, I explained that:
* I found a performance problem which is caused by TLBI-is instruction.
* The problem occurs like this:
1) On a core, OS tries to flush TLB using TLBI-is instruction
2) TLBI-is instruction causes a broadcast to all other cores, and
each core received hard-wired signal
3) Each core check if there are TLB entries which have the specified
ASID/VA
4) This check causes performance degradation
* We ran FWQ[2] and detected OS jitter due to this problem, this noise
is serious for HPC usage.
The noise means here a difference between maximum time and minimum time
which the same work takes.
How to fix:
I think the cause is TLB flush by TLBI-is because the instruction
affects cores that are not related to its flush.
So the previous patch I posted is
* Use mm_cpumask in mm_struct to find appropriate CPUs for TLB flush
* Exec TLBI instead of TLBI-is only to CPUs specified by mm_cpumask
(This is the same behavior as arm32 and x86)
And after the discussion about this patch, I got the following comments.
1) This patch switches the behavior (original flush by TLBI-is and new
flush by TLBI) by boot parameter, this implementation is not acceptable
due to bad maintainability.
2) Even if this patch fixes this problem, it may cause another
performance problem.
I'd like to start over the implementation by considering these points.
For the second comment above, I will run a benchmark test to analyze the
impact on performance.
Please let me know if there are other points I should take into
consideration.
[1] https://lkml.org/lkml/2019/6/17/703
[2] https://asc.llnl.gov/sequoia/benchmarks/FTQ_summary_v1.1.pdf
Thanks,
QI Fuli
On 6/17/19 11:32 PM, Takao Indoh wrote:
> From: Takao Indoh <[email protected]>
>
> I found a performance issue related on the implementation of Linux's TLB
> flush for arm64.
>
> When I run a single-threaded test program on moderate environment, it
> usually takes 39ms to finish its work. However, when I put a small
> apprication, which just calls mprotest() continuously, on one of sibling
> cores and run it simultaneously, the test program slows down significantly.
> It becomes 49ms(125%) on ThunderX2. I also detected the same problem on
> ThunderX1 and Fujitsu A64FX.
>
> I suppose the root cause of this issue is the implementation of Linux's TLB
> flush for arm64, especially use of TLBI-is instruction which is a broadcast
> to all processor core on the system. In case of the above situation,
> TLBI-is is called by mprotect().
>
> This is not a problem for small environment, but this causes a significant
> performance noise for large-scale HPC environment, which has more than
> thousand nodes with low latency interconnect.
>
> To fix this problem, this patch adds new boot parameter
> 'disable_tlbflush_is'. In the case of flush_tlb_mm() *without* this
> parameter, TLB entry is invalidated by __tlbi(aside1is, asid). By this
> instruction, all CPUs within the same inner shareable domain check if there
> are TLB entries which have this ASID, this causes performance noise. OTOH,
> when this new parameter is specified, TLB entry is invalidated by
> __tlbi(aside1, asid) only on the CPUs specified by mm_cpumask(mm).
> Therefore TLB flush is done on minimal CPUs and performance problem does
> not occur. Actually I confirm the performance problem is fixed by this
> patch.
>
> Takao Indoh (2):
> arm64: mm: Restore mm_cpumask (revert commit 38d96287504a ("arm64: mm:
> kill mm_cpumask usage"))
> arm64: tlb: Add boot parameter to disable TLB flush within the same
> inner shareable domain
>
> .../admin-guide/kernel-parameters.txt | 4 +
> arch/arm64/include/asm/mmu_context.h | 7 +-
> arch/arm64/include/asm/tlbflush.h | 61 ++-----
> arch/arm64/kernel/Makefile | 2 +-
> arch/arm64/kernel/smp.c | 6 +
> arch/arm64/kernel/tlbflush.c | 155 ++++++++++++++++++
> arch/arm64/mm/context.c | 2 +
> 7 files changed, 186 insertions(+), 51 deletions(-)
> create mode 100644 arch/arm64/kernel/tlbflush.c
>
Hi,
[please note that my email address has changed and the old one doesn't work
any more]
On Fri, Nov 01, 2019 at 09:56:05AM +0000, [email protected] wrote:
> First of all thanks for the comments for the patch.
>
> I'm still struggling with this problem to find out the solution.
> As a result of an investigation on this problem, after all, I think it
> is necessary to improve TLB flush mechanism of the kernel to fix this
> problem completely.
>
> So, I'd like to restart a discussion. At first, I summarize this problem
> to recall what was the problem and then I want to discuss how to fix it.
>
> Summary of the problem:
> A few months ago I proposed patches to solve a performance problem due
> to TLB flush.[1]
>
> A problem is that TLB flush on a core affects all other cores even if
> all other cores do not need actual flush, and it causes performance
> degradation.
>
> In this thread, I explained that:
> * I found a performance problem which is caused by TLBI-is instruction.
> * The problem occurs like this:
> 1) On a core, OS tries to flush TLB using TLBI-is instruction
> 2) TLBI-is instruction causes a broadcast to all other cores, and
> each core received hard-wired signal
> 3) Each core check if there are TLB entries which have the specified
> ASID/VA
For those following along at home, my understanding is that this "check"
effectively stalls the pipeline as though it is being performed in software.
Some questions:
Does this mean a malicious virtual machine can effectively DoS the system?
What about a malicious application calling mprotect()?
Do all broadcast TLBI instructions cause this expensive check, or are
some significantly slower than others?
> 4) This check causes performance degradation
> * We ran FWQ[2] and detected OS jitter due to this problem, this noise
> is serious for HPC usage.
>
> The noise means here a difference between maximum time and minimum time
> which the same work takes.
>
> How to fix:
> I think the cause is TLB flush by TLBI-is because the instruction
> affects cores that are not related to its flush.
Does broadcast I-cache maintenance cause the same problem?
> So the previous patch I posted is
> * Use mm_cpumask in mm_struct to find appropriate CPUs for TLB flush
> * Exec TLBI instead of TLBI-is only to CPUs specified by mm_cpumask
> (This is the same behavior as arm32 and x86)
>
> And after the discussion about this patch, I got the following comments.
> 1) This patch switches the behavior (original flush by TLBI-is and new
> flush by TLBI) by boot parameter, this implementation is not acceptable
> due to bad maintainability.
> 2) Even if this patch fixes this problem, it may cause another
> performance problem.
>
> I'd like to start over the implementation by considering these points.
> For the second comment above, I will run a benchmark test to analyze the
> impact on performance.
> Please let me know if there are other points I should take into
> consideration.
I think it's worth bearing in mind that I have little sympathy for the
problem that you are seeing. As far as I can tell, you've done the
following:
1. You designed a CPU micro-architecture that stalls whenever it receives
a TLB invalidation request.
2. You integrated said CPU design into a system where broadcast TLB
invalidation is not filtered and therefore stalls every CPU every
time that /any/ TLB invalidation is broadcast.
3. You deployed a mixture of Linux and jitter-sensitive software on
this system, and now you're failing to meet your performance
requirements.
Have I got that right?
If so, given that your CPU design isn't widely available, nobody else
appears to have made this mistake and jitter hasn't been reported as an
issue for any other systems, it's very unlikely that we're going to make
invasive upstream kernel changes to support you. I'm sorry, but all I can
suggest is that you check that your micro-architecture and performance
requirements are aligned with the design of Linux *before* building another
machine like this in future.
I hate to be blunt, but I also don't want to waste your time.
Thanks,
Will
On 01/11/2019 18:28, Will Deacon wrote:
> Hi,
>
> [please note that my email address has changed and the old one doesn't work
> any more]
>
> On Fri, Nov 01, 2019 at 09:56:05AM +0000, [email protected] wrote:
>> First of all thanks for the comments for the patch.
>>
>> I'm still struggling with this problem to find out the solution.
>> As a result of an investigation on this problem, after all, I think it
>> is necessary to improve TLB flush mechanism of the kernel to fix this
>> problem completely.
>>
>> So, I'd like to restart a discussion. At first, I summarize this problem
>> to recall what was the problem and then I want to discuss how to fix it.
>>
>> Summary of the problem:
>> A few months ago I proposed patches to solve a performance problem due
>> to TLB flush.[1]
>>
>> A problem is that TLB flush on a core affects all other cores even if
>> all other cores do not need actual flush, and it causes performance
>> degradation.
>>
>> In this thread, I explained that:
>> * I found a performance problem which is caused by TLBI-is instruction.
>> * The problem occurs like this:
>> 1) On a core, OS tries to flush TLB using TLBI-is instruction
>> 2) TLBI-is instruction causes a broadcast to all other cores, and
>> each core received hard-wired signal
>> 3) Each core check if there are TLB entries which have the specified
>> ASID/VA
>
> For those following along at home, my understanding is that this "check"
> effectively stalls the pipeline as though it is being performed in software.
>
> Some questions:
>
> Does this mean a malicious virtual machine can effectively DoS the system?
> What about a malicious application calling mprotect()?
>
> Do all broadcast TLBI instructions cause this expensive check, or are
> some significantly slower than others?
>
>> 4) This check causes performance degradation
>> * We ran FWQ[2] and detected OS jitter due to this problem, this noise
>> is serious for HPC usage.
>>
>> The noise means here a difference between maximum time and minimum time
>> which the same work takes.
>>
>> How to fix:
>> I think the cause is TLB flush by TLBI-is because the instruction
>> affects cores that are not related to its flush.
>
> Does broadcast I-cache maintenance cause the same problem?
>
>> So the previous patch I posted is
>> * Use mm_cpumask in mm_struct to find appropriate CPUs for TLB flush
>> * Exec TLBI instead of TLBI-is only to CPUs specified by mm_cpumask
>> (This is the same behavior as arm32 and x86)
>>
>> And after the discussion about this patch, I got the following comments.
>> 1) This patch switches the behavior (original flush by TLBI-is and new
>> flush by TLBI) by boot parameter, this implementation is not acceptable
>> due to bad maintainability.
>> 2) Even if this patch fixes this problem, it may cause another
>> performance problem.
>>
>> I'd like to start over the implementation by considering these points.
>> For the second comment above, I will run a benchmark test to analyze the
>> impact on performance.
>> Please let me know if there are other points I should take into
>> consideration.
>
> I think it's worth bearing in mind that I have little sympathy for the
> problem that you are seeing. As far as I can tell, you've done the
> following:
>
> 1. You designed a CPU micro-architecture that stalls whenever it receives
> a TLB invalidation request.
>
> 2. You integrated said CPU design into a system where broadcast TLB
> invalidation is not filtered and therefore stalls every CPU every
> time that /any/ TLB invalidation is broadcast.
>
> 3. You deployed a mixture of Linux and jitter-sensitive software on
> this system, and now you're failing to meet your performance
> requirements.
>
> Have I got that right?
>
> If so, given that your CPU design isn't widely available, nobody else
> appears to have made this mistake and jitter hasn't been reported as an
> issue for any other systems, it's very unlikely that we're going to make
> invasive upstream kernel changes to support you. I'm sorry, but all I can
> suggest is that you check that your micro-architecture and performance
> requirements are aligned with the design of Linux *before* building another
> machine like this in future.
>
I just wanted to note that the cover letter states that they have also seen this
on Thunderx1 and Thunderx2.
Not sure about other machines, like the Huawei TaiShan 200 series.
What I want to say, it seems not to be something that only affects Fujitsu but
also other vendors. So maybe we should consider adding an erratum like the one
for the repeated TLBI on Qualcomm SoCs.
Regards,
Matthias
> I hate to be blunt, but I also don't want to waste your time.
>
> Thanks,
>
> Will
>
> _______________________________________________
> linux-arm-kernel mailing list
> [email protected]
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>
On Tue, Nov 26, 2019 at 03:26:48PM +0100, Matthias Brugger wrote:
> On 01/11/2019 18:28, Will Deacon wrote:
> > On Fri, Nov 01, 2019 at 09:56:05AM +0000, [email protected] wrote:
> >> First of all thanks for the comments for the patch.
> >>
> >> I'm still struggling with this problem to find out the solution.
> >> As a result of an investigation on this problem, after all, I think it
> >> is necessary to improve TLB flush mechanism of the kernel to fix this
> >> problem completely.
> >>
> >> So, I'd like to restart a discussion. At first, I summarize this problem
> >> to recall what was the problem and then I want to discuss how to fix it.
> >>
> >> Summary of the problem:
> >> A few months ago I proposed patches to solve a performance problem due
> >> to TLB flush.[1]
> >>
> >> A problem is that TLB flush on a core affects all other cores even if
> >> all other cores do not need actual flush, and it causes performance
> >> degradation.
> >>
> >> In this thread, I explained that:
> >> * I found a performance problem which is caused by TLBI-is instruction.
> >> * The problem occurs like this:
> >> 1) On a core, OS tries to flush TLB using TLBI-is instruction
> >> 2) TLBI-is instruction causes a broadcast to all other cores, and
> >> each core received hard-wired signal
> >> 3) Each core check if there are TLB entries which have the specified
> >> ASID/VA
> >
> > For those following along at home, my understanding is that this "check"
> > effectively stalls the pipeline as though it is being performed in software.
> >
> > Some questions:
> >
> > Does this mean a malicious virtual machine can effectively DoS the system?
> > What about a malicious application calling mprotect()?
> >
> > Do all broadcast TLBI instructions cause this expensive check, or are
> > some significantly slower than others?
> >
> >> 4) This check causes performance degradation
> >> * We ran FWQ[2] and detected OS jitter due to this problem, this noise
> >> is serious for HPC usage.
> >>
> >> The noise means here a difference between maximum time and minimum time
> >> which the same work takes.
> >>
> >> How to fix:
> >> I think the cause is TLB flush by TLBI-is because the instruction
> >> affects cores that are not related to its flush.
> >
> > Does broadcast I-cache maintenance cause the same problem?
> >
> >> So the previous patch I posted is
> >> * Use mm_cpumask in mm_struct to find appropriate CPUs for TLB flush
> >> * Exec TLBI instead of TLBI-is only to CPUs specified by mm_cpumask
> >> (This is the same behavior as arm32 and x86)
> >>
> >> And after the discussion about this patch, I got the following comments.
> >> 1) This patch switches the behavior (original flush by TLBI-is and new
> >> flush by TLBI) by boot parameter, this implementation is not acceptable
> >> due to bad maintainability.
> >> 2) Even if this patch fixes this problem, it may cause another
> >> performance problem.
> >>
> >> I'd like to start over the implementation by considering these points.
> >> For the second comment above, I will run a benchmark test to analyze the
> >> impact on performance.
> >> Please let me know if there are other points I should take into
> >> consideration.
> >
> > I think it's worth bearing in mind that I have little sympathy for the
> > problem that you are seeing. As far as I can tell, you've done the
> > following:
> >
> > 1. You designed a CPU micro-architecture that stalls whenever it receives
> > a TLB invalidation request.
> >
> > 2. You integrated said CPU design into a system where broadcast TLB
> > invalidation is not filtered and therefore stalls every CPU every
> > time that /any/ TLB invalidation is broadcast.
> >
> > 3. You deployed a mixture of Linux and jitter-sensitive software on
> > this system, and now you're failing to meet your performance
> > requirements.
> >
> > Have I got that right?
> >
> > If so, given that your CPU design isn't widely available, nobody else
> > appears to have made this mistake and jitter hasn't been reported as an
> > issue for any other systems, it's very unlikely that we're going to make
> > invasive upstream kernel changes to support you. I'm sorry, but all I can
> > suggest is that you check that your micro-architecture and performance
> > requirements are aligned with the design of Linux *before* building another
> > machine like this in future.
> >
>
> I just wanted to note that the cover letter states that they have also seen this
> on Thunderx1 and Thunderx2.
>
> Not sure about other machines, like the Huawei TaiShan 200 series.
>
> What I want to say, it seems not to be something that only affects Fujitsu but
> also other vendors. So maybe we should consider adding an erratum like the one
> for the repeated TLBI on Qualcomm SoCs.
Careful here -- we're talking about a reported performance issue, not a
correctness one. The "repeated TLBI" sequence is very much a workaround for
the latter.
In the case of TX1/TX2, I can imagine the "let's sit in a loop of mprotect()
calls" scaling poorly, which is what the cover letter is referring to, but
that's not really a workload that we need to optimise for. However, the case
that Fujitsu are reporting seems to go beyond that because of the design of
their CPU micro-architecture, where even just a single TLB invalidation
message stalls all of the other CPUs in the system. I don't have any reason
to believe that particular problem affects other CPU designs.
Thanks,
Will
On 11/1/19 1:28 PM, Will Deacon wrote:
> On Fri, Nov 01, 2019 at 09:56:05AM +0000, [email protected] wrote:
>> In this thread, I explained that:
>> * I found a performance problem which is caused by TLBI-is instruction.
>> * The problem occurs like this:
>> 1) On a core, OS tries to flush TLB using TLBI-is instruction
>> 2) TLBI-is instruction causes a broadcast to all other cores, and
>> each core received hard-wired signal
>> 3) Each core check if there are TLB entries which have the specified
>> ASID/VA
(the above confuses implementation with architecture)
<snip>
> I think it's worth bearing in mind that I have little sympathy for the
> problem that you are seeing. As far as I can tell, you've done the
> following:
>
> 1. You designed a CPU micro-architecture that stalls whenever it receives
> a TLB invalidation request.
s/SPARC/Arm/ && wire in DVM
> 2. You integrated said CPU design into a system where broadcast TLB
> invalidation is not filtered and therefore stalls every CPU every
> time that /any/ TLB invalidation is broadcast.
>
> 3. You deployed a mixture of Linux and jitter-sensitive software on
> this system, and now you're failing to meet your performance
> requirements.
>
> Have I got that right?
>
> If so, given that your CPU design isn't widely available, nobody else
> appears to have made this mistake and jitter hasn't been reported as an
> issue for any other systems, it's very unlikely that we're going to make
> invasive upstream kernel changes to support you. I'm sorry, but all I can
> suggest is that you check that your micro-architecture and performance
> requirements are aligned with the design of Linux *before* building another
> machine like this in future.
>
> I hate to be blunt, but I also don't want to waste your time.
I always tried to ask nicely for the above to be heeded. There's a
difference between "hi, our implementation doesn't scale, and here's
why" vs "there's a problem with all TLBIs...". There isn't. The problem
is the implementation and that should have been called out first thing.
Jon.
--
Computer Architect