Subject: Re: [RFC/RFT PATCH v3 0/4] KVM: x86: full virtualization of guest
 MTRR
To: Paolo Bonzini <pbonzini@redhat.com>, <linux-kernel@vger.kernel.org>,
        <kvm@vger.kernel.org>
References: <1436368710-5452-1-git-send-email-pbonzini@redhat.com>
CC: <guangrong.xiao@linux.intel.com>, <jroedel@suse.de>,
        <alex.williamson@redhat.com>, <amirv@mellanox.com>,
        Eugene Voronov <eugene@mellanox.com>, Mike Dubman <miked@mellanox.com>,
        "Michael S. Tsirkin" <mst@redhat.com>
From: Or Gerlitz <ogerlitz@mellanox.com>
Message-ID: <55B8DAB9.9080204@mellanox.com>
Date: Wed, 29 Jul 2015 16:52:57 +0300
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101
 Thunderbird/38.1.0
MIME-Version: 1.0
In-Reply-To: <1436368710-5452-1-git-send-email-pbonzini@redhat.com>
Content-Type: text/plain; charset="windows-1252"; format=flowed
Content-Transfer-Encoding: 7bit
AM3PR05MB354: X-MS-Exchange-Organization-RulesExecuted
SpamDiagnosticOutput: 1:23
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Jul 2015 13:53:28.1700
 (UTC)
X-MS-Exchange-CrossTenant-Id: a652971c-7d2e-4d9b-a6a4-d149256f461b
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=a652971c-7d2e-4d9b-a6a4-d149256f461b;Ip=[193.47.165.134];Helo=[mtlcas13.mtl.com]
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM3PR05MB354
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
Content-Length: 3021
Lines: 82

On 7/8/2015 6:18 PM, Paolo Bonzini wrote:
> This part of the MTRR patches was dropped by Xiao.  Bring SVM on feature
> parity with VMX, and then do guest MTRR virtualization for both VMX and SVM.
>
> The IPAT bit of VMX extended page tables is emulated by mangling the guest
> PAT value.
>
> I do not have any AMD machines that support an IOMMU, so I would like
> some help testing these patches.  Thanks,
>
>

Hi Paolo,

We (finally) have results showing that the patches work well and provide 
benefit.

For getting better latency with ConnectX RDMA devices, we write send 
descriptors
to a write-combining (WC) mapped buffer instead of ringing a doorbell 
and having
the HW fetch the descriptor from system memory. In the mlx4 jargon, this 
optimization
is called Blue-Flame (BF).

To test the patches, we booted two hosts with 4.2-rc, patched with
this series, over which a legacy (RHEL 6.x) guests arerunning.

Under SRIOV, the mlx4 VF driver queries the mlx4 PF driver on the
host if BF is availablefor them to use.

We did two runs:

In [1] the guest VF driver was told by the host PF driver
that BF isn't supportedand hence they didn't use WC.

In [2] the VFs were told by the host PF driver that BF is supported
and hence used WC.

The results are provided in micro-seconds and account for half RTT of
nativeRDMA latency test, the WC advantage is notable, so +1for this series!

Or.


[1] guests not-using Blue-Flame / Write-Combining

root@host-194-168-80-68 ~]# ib_send_lat -a
  #bytes #iterations    t_min[usec]    t_max[usec] t_typical[usec]
  2       1000          1.13           11.53        1.16
  4       1000          1.13           6.33         1.16
  8       1000          1.13           5.17         1.17
  16      1000          1.14           4.37         1.17
  32      1000          1.15           5.01         1.18
  64      1000          1.19           7.96         1.22
  128     1000          1.28           5.44         1.31
  256     1000          1.62           6.90         1.65
  512     1000          1.78           5.65         1.82


[2] guests using Blue-Flame when the host allows Write-Combining mapping 
by VMs

root@host-194-168-80-68 ~]# ib_send_lat -a
#bytes #iterations    t_min[usec]    t_max[usec] t_typical[usec]
  2       1000          0.86           16.97        0.89
  4       1000          0.87           4.81         0.90
  8       1000          0.87           4.89         0.90
  16      1000          0.87           6.46         0.90
  32      1000          0.88           4.22         0.91
  64      1000          0.94           4.03         0.97
  128     1000          1.03           6.50         1.06
  256     1000          1.36           7.71         1.39
  512     1000          1.49           5.92         1.52


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/