Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
From:   Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Subject: Re: [PATCH v4 13/17] kvm: i8254: Deactivate APICv when using
 in-kernel PIT re-injection mode.
To:     Paolo Bonzini <pbonzini@redhat.com>,
        "rkagan@virtuozzo.com" <rkagan@virtuozzo.com>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "kvm@vger.kernel.org" <kvm@vger.kernel.org>,
        "rkrcmar@redhat.com" <rkrcmar@redhat.com>,
        "joro@8bytes.org" <joro@8bytes.org>,
        "vkuznets@redhat.com" <vkuznets@redhat.com>,
        "graf@amazon.com" <graf@amazon.com>,
        "jschoenh@amazon.de" <jschoenh@amazon.de>,
        "karahmed@amazon.de" <karahmed@amazon.de>,
        "rimasluk@amazon.com" <rimasluk@amazon.com>,
        "Grimm, Jon" <Jon.Grimm@amd.com>
References: <1572648072-84536-1-git-send-email-suravee.suthikulpanit@amd.com>
 <1572648072-84536-14-git-send-email-suravee.suthikulpanit@amd.com>
 <70fb2b49-2198-bde4-a38b-f37bc8bc9847@redhat.com>
 <20191104231712.GD23545@rkaganb.lan>
 <ac4313a6-df96-2223-bed3-33c3a8555c98@redhat.com>
Message-ID: <9361adbc-77e8-4964-c859-8956e1fbb182@amd.com>
Date:   Mon, 11 Nov 2019 11:37:53 -0600
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101
 Thunderbird/60.9.0
In-Reply-To: <ac4313a6-df96-2223-bed3-33c3a8555c98@redhat.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Received-SPF: None (protection.outlook.com: amd.com does not designate
 permitted sender hosts)
X-MS-Exchange-CrossTenant-Network-Message-Id: 414015f7-b405-4949-a65a-08d766cdf4ba
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Nov 2019 17:38:25.8645
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: SFPhlcBxwjFgfaFyfwfXFbvSlba8KePFUOLCduk76EczLVqChwLpyxi20SKaD+Y3Hs/nhe18+NiCGxVGM8c2hQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB3931
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Roman/Paolo

On 11/5/2019 4:47 PM, Paolo Bonzini wrote:
> On 05/11/19 00:17, Roman Kagan wrote:
>>> This is not too nice for Intel which does support (through the EOI exit
>>> mask) APICv even if PIT reinjection active.
>> Hmm, it's tempting to just make svm_load_eoi_exitmap() disable AVIC when
>> given a non-empty eoi_exit_bitmap, and enable it back on a clear
>> eoi_exit_bitmap.  This may remove the need to add special treatment to
>> PIT etc.
> 
> That is a very nice idea---we can make that a single disable reason,
> like APICV_DEACTIVATE_REASON_EOI, and Intel can simply never use it.

I took at look at the svm_load_eoi_exitmap() and it is called via:
     kvm_make_scan_ioapic_request() ->
         KVM_REQ_SCAN_IOAPIC -> vcpu_scan_ioapic() ->
             KVM_REQ_LOAD_EOI_EXITMAP -> vcpu_load_eoi_exitmap()

The kvm_make_scan_ioapic_request() is called from multiple places:

arch/x86/kvm/irq_comm.c:
     * kvm_arch_post_irq_routing_update() : Called from kvm_set_irq_routing()

arch/x86/kvm/ioapic.c:
     * kvm_arch_post_irq_ack_notifier_list_update() : (Un)registering irq ack notifier
     * kvm_set_ioapic() : Setting ioapic irqchip
     * ioapic_mmio_write() -> ioapic_write_indirect()

arch/x86/kvm/lapic.c:
     * recalculate_apic_map()

Most calls would be from ioapic_mmio_write()->ioapic_write_indirect().

In case of AMD AVIC, the svm_load_e::vsoi_exitmap() is called several times, and requesting
APICV (de)activate from here when the eoi_exit_bitmap is set/clear would introduce
large overhead especially with SMP machine. So, for now, let's just disable APICv
when in-kernel PIT is in reinject (delay) mode.

I'll also add the logic to avoid unnecessary overhead for Intel.

Thanks,
Suravee