Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:18 as permitted sender) client-ip=2620:137:e000::1:18;
From:   Liam Merwick <liam.merwick@oracle.com>
To:     kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
        stable@vger.kernel.org, x86@kernel.org
Cc:     pbonzini@redhat.com, bp@alien8.de, thomas.lendacky@amd.com,
        brijesh.singh@amd.com, krish.sadhukhan@oracle.com,
        liam.merwick@oracle.com
Subject: [PATCH 5.4 0/4] Backport fixes to avoid SEV guest with 380GB+ memory causing host cpu softhang
Date:   Fri, 11 Mar 2022 11:29:23 +0000
Message-Id: <20220311112927.8400-1-liam.merwick@oracle.com>
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?A+ch/Mnz+c6CTZbB4n5Ew36SiEPB22ka8KOmg041z3xWc7G2u62VlVoqiBR0?=
 =?us-ascii?Q?3Nggui9y3jiVwNAluL4x3LoC8XHP7dU1badZS4vijz7fMqMd5T316346K7V7?=
 =?us-ascii?Q?9afxwMVTK9UAB/EpwOziKO5eB6OXTQhujJVR0gIDWVwUlczdXnyw4LoAyPFZ?=
 =?us-ascii?Q?6cEp9jp2LAccVavJ5+B7AD9HoNkljL2JX/1rsKhn2f4NOtbFeHJ/BYtoxSBl?=
 =?us-ascii?Q?lQKei+GINNOGSjjgdTjEDoxBKPyrH7S1jn9DQv8LcYupQLhVNDKy7AzySaAH?=
 =?us-ascii?Q?nH0alRTlkMaZyhJqTsyTXhQN7WqDArTXRplhvm8xPq2aA/EAcIiBQjnX3DBF?=
 =?us-ascii?Q?J5Yag3D9fhTr4xbxy1dUrFXwj73rnPRIMkUBRN59Sm69TWcQrk5DdsVepRt+?=
 =?us-ascii?Q?LjgNNHYF4OCYGKjLLnnaJP3T8HcM/s4Z3NkvA+vT6W10ZyRR3JPYdJ0g7jnI?=
 =?us-ascii?Q?HuN5XAnhKgTecJUX7u7ToMGFxuWBJbsGPTKmPELDLpeWETxXgr8LJ/YyYnl0?=
 =?us-ascii?Q?edmErb5dS/cbknuz6XCtokNaoGkkbADeQyD0BxZ3R2b1YEtYa4yhzim1H8Gz?=
 =?us-ascii?Q?RZW4NNTj+qsCG0KeogSk2mTVrhmXPXo+L4StP8tVWOY0EBcqYXBplZ4BF1Rb?=
 =?us-ascii?Q?M03jPuigM+MYkZ/YH5huC1PPtWNKI1D5czBIsIeEBY2vgT4S3RLUsdv5Iysg?=
 =?us-ascii?Q?5e9aAku/ytyCT81og7qnPNU8Ctl/i+4VcOvEyvAzYjKVUNj9pXm+Z5WvXtMk?=
 =?us-ascii?Q?+rjuaKWHIqAXn/BTk3Uow8gMCQ/pQ569OagNaCbpCEnH1EykQjaA87GBLeDO?=
 =?us-ascii?Q?NPsvUWFrv05yBbgWp2wvQxvEUcB9FsZsZL/8AsY/ZhPSVDEzKCRt+Afy7ujz?=
 =?us-ascii?Q?2/Ovl2V3FEy+eUDnl7GeZhKF0fr1nz9i9sITL9zcxZM0KXegb9pnvgn2bQKr?=
 =?us-ascii?Q?I7Pq0Em1QaOKMHv3/3rVXPqcquah7LbYPo8gl/NbmbqNKWBCT8F6oUJ9iqhL?=
 =?us-ascii?Q?rJ6DPs7q3sk0pvW1KCC5zMiXhgHZtGFiemd4iwKlIPd/LzEw5+Gw3WaebXTo?=
 =?us-ascii?Q?j6lCnlciNB2+QRDahL0Ec3o3uX557gwnoYFI6pcOKJXfWaRLUOb9TYn0jxO1?=
 =?us-ascii?Q?EGL42AAzUGr/l+vxGMOzPJPXdbKWL71HQ8xCmiI7E7u3kYW2EO0ba8oD2rek?=
 =?us-ascii?Q?bic8lfkbdHTRni98s/ZYoWJ05xnmBefr4k0swFoA6OsrI6+qrEj9QfH0xB09?=
 =?us-ascii?Q?eO5dSZMdSlk209LDAAlG61kQWeXcx8qsBB8vhYMvrjS9SLtq2PKTT1pxKKUC?=
 =?us-ascii?Q?dQp1K/UHiH3JhtSjTUWoT2zRJMpsRHkbIe4p1WlchLBTA436pQgBfeb9/8Qc?=
 =?us-ascii?Q?lW9FOu2GYr+S4Q90cHCoZLRPHHq1oonL67rHzojHDPQTePp+JFPHYZY51GVe?=
 =?us-ascii?Q?Fap7Ue/fNBcjBSs7iQ8/3ZRLR2LNEo4uIiLqqCmhGc8n0fvoKA98arXkxFls?=
 =?us-ascii?Q?wIbPpg+g0MlnJqjB1SbEd4VNXzU2ppQCQEBAXtZ7bKi0WItsDbLPbPugrc7r?=
 =?us-ascii?Q?tTXMv4bpBzYJYv9ffSvW8KcYW9DB5ddAb2CCvKFw71J2YKtyyrOkG/lC3t53?=
 =?us-ascii?Q?PvHmVmM4rkwRdbSZVTxykZ0=3D?=
X-OriginatorOrg: oracle.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 358b3b9d-546b-481b-b2f9-08da03528115
X-MS-Exchange-CrossTenant-AuthSource: BN0PR10MB5030.namprd10.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Mar 2022 11:30:11.7059
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: /fj/k7m4ktdgEWmjaXMzALslicxwqKloOxXRoK9evAboovRksbz4BouSBIKR+iWth0n73Z+EMPsmM9Oja7heoQ==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR10MB1430
X-Proofpoint-Virus-Version: vendor=nai engine=6300 definitions=10282 signatures=692556
X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxlogscore=704 bulkscore=0
 phishscore=0 malwarescore=0 suspectscore=0 mlxscore=0 adultscore=0
 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000
 definitions=main-2203110055
X-Proofpoint-ORIG-GUID: VJ43iHY2riC7DVRiJTYLKZqjzrHGNEmU
X-Proofpoint-GUID: VJ43iHY2riC7DVRiJTYLKZqjzrHGNEmU
X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIM_SIGNED,
	DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,
	MAILING_LIST_MULTI,RDNS_NONE,SPF_HELO_NONE,T_SCC_BODY_TEXT_LINE
	autolearn=no autolearn_force=no version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
	lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

[ patch series targeting linux-5.4.y stable branch. ]

Creating a SEV-enabled guest with 380GB or more of memory causes a
cpu soft-hang in the host running 5.4 with the following stacktrace:

kernel: watchdog: BUG: soft lockup - CPU#214 stuck for 22s! [qemu-kvm:6424]
...
kernel: CPU: 214 PID: 6424 Comm: qemu-kvm Not tainted 5.4.183.stable #1
kernel: Hardware name: Oracle Corporation ORACLE SERVER E4-2c/Asm,MB
Tray,2U,E4-2c, BIOS 78014000 01/05/2022
kernel: RIP: 0010:clflush_cache_range+0x35/0x40
kernel: Code: f0 0f b7 15 63 53 99 01 89 f6 48 89 d0 48 f7 d8 48 21 f8 48 01 f7
48 39 f8 73 0c 66 0f ae 38 48 01 d0 48 39 c7 77 f4 0f ae f0 <5d> c3 66 0f 1f 84
00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 0f ae
kernel: RSP: 0018:ffffacba5e98fc30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
kernel: RAX: ffffa1193cfbc000 RBX: ffffacbba4701000 RCX: 0000000000000000
kernel: RDX: 0000000000000040 RSI: 0000000000001000 RDI: ffffa1193cfbc000
kernel: RBP: ffffacba5e98fc30 R08: ffffacba5f44aca0 R09: ffffacbac3701000
kernel: R10: 0000000000000080 R11: ffff9f8500000af0 R12: ffffa18074a22f80
kernel: R13: ffffacbaf6889dd8 R14: ffffacba5f41d960 R15: ffffacba5f44aca0
kernel: FS:  00007fbe04321f00(0000) GS:ffffa1814ed80000(0000)
knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 00007dfbebd7d000 CR3: 000801fb93d68002 CR4: 0000000000760ee0
kernel: PKRU: 55555554
kernel: Call Trace:
kernel: sev_clflush_pages.part.56+0x50/0x70 [kvm_amd]
kernel: svm_register_enc_region+0xe2/0x120 [kvm_amd]
kernel: kvm_arch_vm_ioctl+0x524/0xbd0 [kvm]
kernel: ? release_pages+0x212/0x430
kernel: ? __pagevec_lru_add_fn+0x192/0x2f0
kernel: kvm_vm_ioctl+0x9c/0x9d0 [kvm]
kernel: ? __lru_cache_add+0x59/0x70
kernel: ? lru_cache_add_active_or_unevictable+0x39/0xb0
kernel: ? __handle_mm_fault+0xa74/0xfd0
kernel: ? __switch_to_asm+0x34/0x70
kernel: do_vfs_ioctl+0xa9/0x640
kernel: ? __audit_syscall_entry+0xdd/0x130
kernel: ksys_ioctl+0x67/0x90
kernel: __x64_sys_ioctl+0x1a/0x20
kernel: do_syscall_64+0x60/0x1d0
kernel: entry_SYSCALL_64_after_hwframe+0x44/0xa9
kernel: RIP: 0033:0x7fbe0086563b
kernel: Code: 0f 1e fa 48 8b 05 4d b8 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff
ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff
73 01 c3 48 8b 0d 1d b8 2c 00 f7 d8 64 89 01 48
kernel: RSP: 002b:00007ffedf577418 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
kernel: RAX: ffffffffffffffda RBX: 00007dfbebe00000 RCX: 00007fbe0086563b
kernel: RDX: 00007ffedf577490 RSI: ffffffff8010aebb RDI: 000000000000000d
kernel: RBP: 000001c200000000 R08: ffffffffffffffff R09: ffffffffffffffff
kernel: R10: ffffffffffffffff R11: 0000000000000246 R12: 000001c200000000
kernel: R13: 00007dfbebe00000 R14: 0000000000000000 R15: 0000000000000000

The problem is the time spent flushing the caches when pinning memory for
SEV but it's unnecessary as it turns out - it is resolved by backporting
the following commits from Linux 5.10

e1ebb2b49048 KVM: SVM: Don't flush cache if hardware enforces cache coherency
across encryption domains
(conflict due to the function it fixed being moved in a refactoring in 5.7).

along with 3 other commits needed as dependencies.
(fbd5969d1ff2 avoids a conflict in 5866e9205b47)

fbd5969d1ff2 x86/cpufeatures: Mark two free bits in word 3
5866e9205b47 x86/cpu: Add hardware-enforced cache coherency as a CPUID feature
75d1cc0e05af x86/mm/pat: Don't flush cache if hardware enforces cache coherency across encryption domnains                                                                                                        

Tested by creating various sized guests up to 1.8TB, with and without SEV enabled,
running a few benchmarks and passing kvm-unit-tests.


Borislav Petkov (1):
  x86/cpufeatures: Mark two free bits in word 3

Krish Sadhukhan (3):
  x86/cpu: Add hardware-enforced cache coherency as a CPUID feature
  x86/mm/pat: Don't flush cache if hardware enforces cache coherency
    across encryption domnains
  KVM: SVM: Don't flush cache if hardware enforces cache coherency
    across encryption domains

 arch/x86/include/asm/cpufeatures.h | 2 ++
 arch/x86/kernel/cpu/scattered.c    | 1 +
 arch/x86/kvm/svm.c                 | 3 ++-
 arch/x86/mm/pageattr.c             | 2 +-
 4 files changed, 6 insertions(+), 2 deletions(-)

-- 
2.27.0