Received: by 10.223.185.116 with SMTP id b49csp1120111wrg; Fri, 23 Feb 2018 12:16:29 -0800 (PST) X-Google-Smtp-Source: AH8x224yoADwo9Fr4kPLWOqM+pkokJ2QN8+wbfn8LVTuZcMx/bzjXdOk3PzRvIIEk9BOqMwh+eK3 X-Received: by 10.167.129.195 with SMTP id c3mr2889928pfn.14.1519416989221; Fri, 23 Feb 2018 12:16:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1519416989; cv=none; d=google.com; s=arc-20160816; b=hq5Y17h+PADmLK/FsG9XOzQpoUX26UZSx4kc07g9uKSTFIUStSHoy8Kc09DbUSY1m7 wFfWpLEUaRUEnmq5yTu3IckVy+5piUNa3TVY+QIX+WgvOYugGQCsn7TOMB9RETdRNFIt K9c2GQOnLpnO+UZGEXQZTT2RJ8XGCG70PB2eQliQoBM1oZExxTwlk4FDBfLCAW12JOIR AQqFmnKMLiw3HUFhKyRpH62cEBvQFTBS3mk9Ku3pcixwyM5rK01RIqrK7msGQt9pA2G4 YgcFsd4ZNP3hGORt0QrVSN6mSqgkYhhyySQodTZzMxM26U/rsEUaBjxgKScIYKBDrO3o iNKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:mime-version:user-agent:references :in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=THJ/Ylt2miY5hnOh3A644fBb7p/wCwK7zJqBp03xUMQ=; b=lU6HvLpVTz8KIf3WRxrwMFDdr76GnB4ibKUnBWZ6XqOJu8nPrtITRUg9oygFfTiy+X xbKmQa/hkvI3Zo68+07/BoA+tgYbL90CxypLU+af8YVmbt1KAT22JzjSAEA0X5V1G/3V kJWajh9yWwAJWGHurEoROkGf71lzW5c2f46rAEW64qKfCvCQouIPFui8bOa+fBY3pl49 21iRDUGlXBv2Ufjixmq9vcG5gtM5JDkjqdCD7zkOEpVBFEW1nmxnMJAtiHMfFMDY8BGh eeMbHGRVOu1d68oJG1mEYIbtYRwDMpw1d9BjKtJ6Lwk9+E2OS2GKI2qy8YFY5vBROmyz j2rw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id b13-v6si2233368plz.218.2018.02.23.12.16.15; Fri, 23 Feb 2018 12:16:29 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755208AbeBWUPK (ORCPT + 99 others); Fri, 23 Feb 2018 15:15:10 -0500 Received: from mail.linuxfoundation.org ([140.211.169.12]:40308 "EHLO mail.linuxfoundation.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932705AbeBWSky (ORCPT ); Fri, 23 Feb 2018 13:40:54 -0500 Received: from localhost (LFbn-1-12258-90.w90-92.abo.wanadoo.fr [90.92.71.90]) by mail.linuxfoundation.org (Postfix) with ESMTPSA id 72EC911E9; Fri, 23 Feb 2018 18:40:53 +0000 (UTC) From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Cyril Novikov , Dan Williams , Thomas Gleixner , linux-arch@vger.kernel.org, kernel-hardening@lists.openwall.com, Peter Zijlstra , Catalin Marinas , Will Deacon , Russell King , torvalds@linux-foundation.org, alan@linux.intel.com, David Woodhouse , Jack Wang Subject: [PATCH 4.4 174/193] array_index_nospec: Sanitize speculative array de-references Date: Fri, 23 Feb 2018 19:26:47 +0100 Message-Id: <20180223170353.313139741@linuxfoundation.org> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180223170325.997716448@linuxfoundation.org> References: <20180223170325.997716448@linuxfoundation.org> User-Agent: quilt/0.65 X-stable: review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 4.4-stable review patch. If anyone has any objections, please let me know. ------------------ From: Dan Williams (cherry picked from commit f3804203306e098dae9ca51540fcd5eb700d7f40) array_index_nospec() is proposed as a generic mechanism to mitigate against Spectre-variant-1 attacks, i.e. an attack that bypasses boundary checks via speculative execution. The array_index_nospec() implementation is expected to be safe for current generation CPUs across multiple architectures (ARM, x86). Based on an original implementation by Linus Torvalds, tweaked to remove speculative flows by Alexei Starovoitov, and tweaked again by Linus to introduce an x86 assembly implementation for the mask generation. Co-developed-by: Linus Torvalds Co-developed-by: Alexei Starovoitov Suggested-by: Cyril Novikov Signed-off-by: Dan Williams Signed-off-by: Thomas Gleixner Cc: linux-arch@vger.kernel.org Cc: kernel-hardening@lists.openwall.com Cc: Peter Zijlstra Cc: Catalin Marinas Cc: Will Deacon Cc: Russell King Cc: gregkh@linuxfoundation.org Cc: torvalds@linux-foundation.org Cc: alan@linux.intel.com Link: https://lkml.kernel.org/r/151727414229.33451.18411580953862676575.stgit@dwillia2-desk3.amr.corp.intel.com Signed-off-by: David Woodhouse [jwang: cherry pick to 4.4] Signed-off-by: Jack Wang Signed-off-by: Greg Kroah-Hartman --- include/linux/nospec.h | 72 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) create mode 100644 include/linux/nospec.h --- /dev/null +++ b/include/linux/nospec.h @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: GPL-2.0 +// Copyright(c) 2018 Linus Torvalds. All rights reserved. +// Copyright(c) 2018 Alexei Starovoitov. All rights reserved. +// Copyright(c) 2018 Intel Corporation. All rights reserved. + +#ifndef _LINUX_NOSPEC_H +#define _LINUX_NOSPEC_H + +/** + * array_index_mask_nospec() - generate a ~0 mask when index < size, 0 otherwise + * @index: array element index + * @size: number of elements in array + * + * When @index is out of bounds (@index >= @size), the sign bit will be + * set. Extend the sign bit to all bits and invert, giving a result of + * zero for an out of bounds index, or ~0 if within bounds [0, @size). + */ +#ifndef array_index_mask_nospec +static inline unsigned long array_index_mask_nospec(unsigned long index, + unsigned long size) +{ + /* + * Warn developers about inappropriate array_index_nospec() usage. + * + * Even if the CPU speculates past the WARN_ONCE branch, the + * sign bit of @index is taken into account when generating the + * mask. + * + * This warning is compiled out when the compiler can infer that + * @index and @size are less than LONG_MAX. + */ + if (WARN_ONCE(index > LONG_MAX || size > LONG_MAX, + "array_index_nospec() limited to range of [0, LONG_MAX]\n")) + return 0; + + /* + * Always calculate and emit the mask even if the compiler + * thinks the mask is not needed. The compiler does not take + * into account the value of @index under speculation. + */ + OPTIMIZER_HIDE_VAR(index); + return ~(long)(index | (size - 1UL - index)) >> (BITS_PER_LONG - 1); +} +#endif + +/* + * array_index_nospec - sanitize an array index after a bounds check + * + * For a code sequence like: + * + * if (index < size) { + * index = array_index_nospec(index, size); + * val = array[index]; + * } + * + * ...if the CPU speculates past the bounds check then + * array_index_nospec() will clamp the index within the range of [0, + * size). + */ +#define array_index_nospec(index, size) \ +({ \ + typeof(index) _i = (index); \ + typeof(size) _s = (size); \ + unsigned long _mask = array_index_mask_nospec(_i, _s); \ + \ + BUILD_BUG_ON(sizeof(_i) > sizeof(long)); \ + BUILD_BUG_ON(sizeof(_s) > sizeof(long)); \ + \ + _i &= _mask; \ + _i; \ +}) +#endif /* _LINUX_NOSPEC_H */