Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Subject: Re: question on lazy tlb flush
To:     Rik van Riel <riel@surriel.com>,
        LKML <linux-kernel@vger.kernel.org>
References: <cd421c2c-8507-6652-2ef7-a6f3b20efcd2@oracle.com>
 <5af5f7aa8ac48d2c8a2d1809e3c7932824958a76.camel@surriel.com>
From:   Zhenzhong Duan <zhenzhong.duan@oracle.com>
Organization: Oracle Corporation
Message-ID: <d11f26c9-5a5b-9e3c-5a19-ef931f16814c@oracle.com>
Date:   Thu, 30 May 2019 10:09:58 +0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101
 Thunderbird/60.7.0
MIME-Version: 1.0
In-Reply-To: <5af5f7aa8ac48d2c8a2d1809e3c7932824958a76.camel@surriel.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Content-Language: en-US
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk

On 2019/5/29 22:17, Rik van Riel wrote:
> On Wed, 2019-05-29 at 12:54 +0800, Zhenzhong Duan wrote:
>> Hi Maintainers,
>>
>> A question raised when I learned below code.  Appreciate any help me
>> understand the code.
>>
>> void native_flush_tlb_others(const struct cpumask *cpumask,
>>                                const struct flush_tlb_info *info)
>>
>> {
>>
>> ...
>>
>>           /*
>>            * If no page tables were freed, we can skip sending IPIs to
>>            * CPUs in lazy TLB mode. They will flush the CPU themselves
>>            * at the next context switch.
>>            *
>>            * However, if page tables are getting freed, we need to
>> send the
>>            * IPI everywhere, to prevent CPUs in lazy TLB mode from
>> tripping
>>            * up on the new contents of what used to be page tables,
>> while
>>            * doing a speculative memory access.
>>            */
>>           if (info->freed_tables)
>>                   smp_call_function_many(cpumask,
>> flush_tlb_func_remote,
>>                                  (void *)info, 1);
>>           else
>>                   on_each_cpu_cond_mask(tlb_is_not_lazy,
>> flush_tlb_func_remote,
>>                                   (void *)info, 1, GFP_ATOMIC,
>> cpumask);
>>
>> }
>>
>> I just didn't understand how a kernel thread could trip up on the
>> new
>> contents of what used to be page tables. I presume the freed page
>> tables
>> are user mapping?
> That is correct, and you ask a very good question.
>
> The software does indeed not access userspace memory
> addresses from kernel threads.
>
> However, the CPU itself can do speculative loads from
> userspace memory addresses, even when the software thread
> is running exclusively in kernel mode.
>
> Add to that the fact that the page table pages can get
> reused for something else after they have been freed, and
> that the CPU can cache intermediate translations (eg. PUD
> and PMD level things get cached in the TLB), and you might
> end up with a speculative load poking at a PCI register,
> or something else that trips up the machine.
>
> For that reason, when page table pages get freed, we need
> to flush the TLBs of all users, inluding lazy TLB kernel
> threads.

Understood. Thanks for your detailed explanation :)

Zhenzhong