Subject: Re: [PART1 RFC v2 05/10] KVM: x86: Detect and Initialize AVIC support
To: Paolo Bonzini <pbonzini@redhat.com>, <rkrcmar@redhat.com>,
        <joro@8bytes.org>, <bp@alien8.de>, <gleb@kernel.org>,
        <alex.williamson@redhat.com>
References: <1457124368-2025-1-git-send-email-Suravee.Suthikulpanit@amd.com>
 <1457124368-2025-6-git-send-email-Suravee.Suthikulpanit@amd.com>
 <56DDAF2D.3060703@redhat.com> <56E841CC.4090806@amd.com>
 <56E844E2.6090507@redhat.com>
CC: <kvm@vger.kernel.org>, <linux-kernel@vger.kernel.org>, <wei@redhat.com>,
        <sherry.hurwitz@amd.com>
From: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Message-ID: <56E8FB8B.7050700@amd.com>
Date: Wed, 16 Mar 2016 13:22:03 +0700
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.5.1
MIME-Version: 1.0
In-Reply-To: <56E844E2.6090507@redhat.com>
Content-Type: text/plain; charset="windows-1252"; format=flowed
Content-Transfer-Encoding: 7bit
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Mar 2016 06:22:21.0567
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR12MB0434
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 2876
Lines: 77

Hi,

On 03/16/2016 12:22 AM, Paolo Bonzini wrote:
>>> Why is this necessary?  The APIC access page is a peculiarity of Intel
>>> >>processors (and the special memslot for only needs to map 0xfee00000 to
>>> >>0xfee00fff; after that there is the MSI area).
>> >
>> >The current lapic regs page is allocated using get_zeroed_page(), which
>> >can be paged out. If I use these pages for AVIC backing pages, it seems
>> >to cause VM to slow down quite a bit due to a lot of page faults.
> What causes the lapic regs page to be paged out?
>

This is mainly causing a large number of VMEXIT due to NPF.  In my test 
running hackbench in the guest. The following are perf result profiling 
for 10 seconds in the host in two cases:

CASE1: Using x86_set_memory_region() for AVIC backing page

# ./perf-vmexit.sh 10
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 2.813 MB perf.data.guest (30356 samples) ]


Analyze events for all VMs, all VCPUs:

              VM-EXIT    Samples  Samples%     Time%    Min Time    Max 
Time         Avg time

            interrupt      10042    66.30%    81.33%      0.43us 
202.50us      7.43us ( +-   1.20% )
                  msr       5004    33.04%    15.76%      0.73us 
12.21us      2.89us ( +-   0.43% )
                pause         58     0.38%     0.18%      0.56us 
5.88us      2.92us ( +-   6.43% )
                  npf         35     0.23%     2.01%      6.41us 
207.78us     52.70us ( +-  23.67% )
                  nmi          4     0.03%     0.02%      2.31us 
4.67us      3.49us ( +-  14.26% )
                   io          3     0.02%     0.70%     82.75us 
360.90us    214.28us ( +-  37.64% )
      avic_incomp_ipi          1     0.01%     0.00%      2.17us 
2.17us      2.17us ( +-   0.00% )

Total Samples:15147, Total events handled time:91715.78us.


CASE2: Using the lapic regs page for AVIC backing page.

# ./perf-vmexit.sh 10
[ perf record: Woken up 255 times to write data ]
[ perf record: Captured and wrote 509.202 MB perf.data.guest (5718856 
samples) ]


Analyze events for all VMs, all VCPUs:

              VM-EXIT    Samples  Samples%     Time%    Min Time    Max 
Time         Avg time

                  npf    1897710    99.33%    98.08%      1.09us 
243.22us      1.67us ( +-   0.04% )
            interrupt       7818     0.41%     1.44%      0.44us 
216.55us      5.97us ( +-   1.92% )
                  msr       5001     0.26%     0.45%      0.68us 
12.58us      2.89us ( +-   0.50% )
                pause         25     0.00%     0.00%      0.71us 
4.23us      2.03us ( +-  10.76% )
                   io          4     0.00%     0.03%     73.91us 
337.29us    206.74us ( +-  26.38% )
                  nmi          1     0.00%     0.00%      5.92us 
5.92us      5.92us ( +-   0.00% )

Total Samples:1910559, Total events handled time:3229214.64us.

Thanks,
Suravee