On 2020-05-25 12:23, Keqian Zhu wrote:
> This patch series add support for stage2 hardware DBM, and it is only
> used for dirty log for now.
>
> It works well under some migration test cases, including VM with 4K
> pages or 2M THP. I checked the SHA256 hash digest of all memory and
> they keep same for source VM and destination VM, which means no dirty
> pages is missed under hardware DBM.
>
> However, there are some known issues not solved.
>
> 1. Some mechanisms that rely on "write permission fault" become
> invalid,
> such as kvm_set_pfn_dirty and "mmap page sharing".
>
> kvm_set_pfn_dirty is called in user_mem_abort when guest issues
> write
> fault. This guarantees physical page will not be dropped directly
> when
> host kernel recycle memory. After using hardware dirty management,
> we
> have no chance to call kvm_set_pfn_dirty.
Then you will end-up with memory corruption under memory pressure.
This also breaks things like CoW, which we depend on.
>
> For "mmap page sharing" mechanism, host kernel will allocate a new
> physical page when guest writes a page that is shared with other
> page
> table entries. After using hardware dirty management, we have no
> chance
> to do this too.
>
> I need to do some survey on how stage1 hardware DBM solve these
> problems.
> It helps if anyone can figure it out.
>
> 2. Page Table Modification Races: Though I have found and solved some
> data
> races when kernel changes page table entries, I still doubt that
> there
> are data races I am not aware of. It's great if anyone can figure
> them out.
>
> 3. Performance: Under Kunpeng 920 platform, for every 64GB memory, KVM
> consumes about 40ms to traverse all PTEs to collect dirty log. It
> will
> cause unbearable downtime for migration if memory size is too big. I
> will
> try to solve this problem in Patch v1.
This, in my opinion, is why Stage-2 DBM is fairly useless.
From a performance perspective, this is the worse possible
situation. You end up continuously scanning page tables, at
an arbitrary rate, without a way to evaluate the fault rate.
One thing S2-DBM would be useful for is SVA, where a device
write would mark the S2 PTs dirty as they are shared between
CPU and SMMU. Another thing is SPE, which is essentially a DMA
agent using the CPU's PTs.
But on its own, and just to log the dirty pages, S2-DBM is
pretty rubbish. I wish arm64 had something like Intel's PML,
which looks far more interesting for the purpose of tracking
accesses.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
Hi Marc,
On 2020/5/25 23:44, Marc Zyngier wrote:
> On 2020-05-25 12:23, Keqian Zhu wrote:
>> This patch series add support for stage2 hardware DBM, and it is only
>> used for dirty log for now.
>>
>> It works well under some migration test cases, including VM with 4K
>> pages or 2M THP. I checked the SHA256 hash digest of all memory and
>> they keep same for source VM and destination VM, which means no dirty
>> pages is missed under hardware DBM.
>>
>> However, there are some known issues not solved.
>>
>> 1. Some mechanisms that rely on "write permission fault" become invalid,
>> such as kvm_set_pfn_dirty and "mmap page sharing".
>>
>> kvm_set_pfn_dirty is called in user_mem_abort when guest issues write
>> fault. This guarantees physical page will not be dropped directly when
>> host kernel recycle memory. After using hardware dirty management, we
>> have no chance to call kvm_set_pfn_dirty.
>
> Then you will end-up with memory corruption under memory pressure.
> This also breaks things like CoW, which we depend on.
>
Yes, these problems looks knotty. But I think x86 PML support will face these
problems too. I believe there must be some methods to solve them.
>>
>> For "mmap page sharing" mechanism, host kernel will allocate a new
>> physical page when guest writes a page that is shared with other page
>> table entries. After using hardware dirty management, we have no chance
>> to do this too.
>>
>> I need to do some survey on how stage1 hardware DBM solve these problems.
>> It helps if anyone can figure it out.
>>
>> 2. Page Table Modification Races: Though I have found and solved some data
>> races when kernel changes page table entries, I still doubt that there
>> are data races I am not aware of. It's great if anyone can figure them out.
>>
>> 3. Performance: Under Kunpeng 920 platform, for every 64GB memory, KVM
>> consumes about 40ms to traverse all PTEs to collect dirty log. It will
>> cause unbearable downtime for migration if memory size is too big. I will
>> try to solve this problem in Patch v1.
>
> This, in my opinion, is why Stage-2 DBM is fairly useless.
> From a performance perspective, this is the worse possible
> situation. You end up continuously scanning page tables, at
> an arbitrary rate, without a way to evaluate the fault rate.
>
> One thing S2-DBM would be useful for is SVA, where a device
> write would mark the S2 PTs dirty as they are shared between
> CPU and SMMU. Another thing is SPE, which is essentially a DMA
> agent using the CPU's PTs.
>
> But on its own, and just to log the dirty pages, S2-DBM is
> pretty rubbish. I wish arm64 had something like Intel's PML,
> which looks far more interesting for the purpose of tracking
> accesses.
Sure, PML is a better solution on hardware management of dirty state.
However, compared to optimizing hardware, optimizing software is with
shorter cycle time.
Here I have an optimization in mind to solve it. Scanning page tables
can be done parallel, which can greatly reduce time consumption. For there
is no communication between parallel CPUs, we can achieve high speedup
ratio.
>
> Thanks,
>
> M.
Thanks,
Keqian