Received: by 2002:ad5:474a:0:0:0:0:0 with SMTP id i10csp546004imu; Wed, 12 Dec 2018 23:38:53 -0800 (PST) X-Google-Smtp-Source: AFSGD/WZkde5b9975nz//FG7aY1poWqG2AJL/hNcVPFtQaC+wxqL5pxgIVwh3QC6UQYTF2wfAKkq X-Received: by 2002:aa7:8354:: with SMTP id z20mr22736048pfm.81.1544686733128; Wed, 12 Dec 2018 23:38:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1544686733; cv=none; d=google.com; s=arc-20160816; b=pm+I5SZRfHcDDaQ6T/mkkTQRE/ZVDcnn7QdRbVruPpeWBFHn312Iy+q1j0d+OxKBRj Ah2Nbad43zlXDeTJvS3DFRK+D0qig7D0bPQV/7sQEIfm4CRWNBHfwxX+RW1n2+pjyz6x 7SO23SRBna1lWJv1K6nDQnRhuTZVRAokp224hyWWMa8yxHJIoxlgmbbQVGRO9Y2Kr1Wq FvuvoSz6ounQ6iv1+bGcgGGRgu2RkkJlZoSXbyuTyhLgUk/OCAlPZmgQbMS6qhP9TA73 p0zy+qpL5UUXrm4fyCiUQ8hZCOzHoay6QE3C2quRV1Ti+gFCGrBohLc4ijrePIuA2UtK Kvgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-transfer-encoding:content-disposition:mime-version :references:message-id:subject:cc:to:from:date; bh=kLdZs6dR8rC1zBh0ZrCHjYfG527R0ZCJNd2Gpbd+prk=; b=fUn557B+XgAKlSkOLrSzGz8KU/cmI7JD8qQFrUp9/uMTjg1rtB4VRihvHMvSlXNsvp ChGAS3QeIAhHin/4teas+5STSPQzzKvuYA57O58iesY6Yg/tbXz634rbVGuHbCi1f2SJ 6QVyOTlv6UwBUroxij0UJkDyzs3NCWQzUT+K6cIV82k4rKzXc1N+MTnRZZZrIEfBINsN qC7eDSMhD26InClwtDVvKm/CZE/DG1c1lOG8CBoYIQkLyjyDRP7i9eKRAVd+Sn+wmG1F P+xYzfPwSswyA6uGi6uu8J0RbNK0tPtmCXIgx7g/8ZejgwVu1jc6H1ztrih0As3Jy461 LcNg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id c15si935243pgg.446.2018.12.12.23.38.38; Wed, 12 Dec 2018 23:38:53 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727072AbeLMHhS (ORCPT + 99 others); Thu, 13 Dec 2018 02:37:18 -0500 Received: from metis.ext.pengutronix.de ([85.220.165.71]:50665 "EHLO metis.ext.pengutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726564AbeLMHhR (ORCPT ); Thu, 13 Dec 2018 02:37:17 -0500 Received: from ptx.hi.pengutronix.de ([2001:67c:670:100:1d::c0]) by metis.ext.pengutronix.de with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1gXLYU-0001kV-Hr; Thu, 13 Dec 2018 08:37:10 +0100 Received: from ukl by ptx.hi.pengutronix.de with local (Exim 4.89) (envelope-from ) id 1gXLYS-0008KV-Ir; Thu, 13 Dec 2018 08:37:08 +0100 Date: Thu, 13 Dec 2018 08:37:08 +0100 From: Uwe =?iso-8859-1?Q?Kleine-K=F6nig?= To: Jeremy Cline Cc: Thierry Reding , Andrew Morton , Thomas Gleixner , Jonathan Corbet , Joe Perches , linux-kernel@vger.kernel.org Subject: Re: [PATCH 1/2] scripts/spdxcheck.py: Always open files in binary mode Message-ID: <20181213073708.nwj4nmnccuugvrhc@pengutronix.de> References: <20181212131210.28024-1-thierry.reding@gmail.com> <20181212181410.GC2352@laptop.jcline.org> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20181212181410.GC2352@laptop.jcline.org> User-Agent: NeoMutt/20170113 (1.7.2) X-SA-Exim-Connect-IP: 2001:67c:670:100:1d::c0 X-SA-Exim-Mail-From: ukl@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: linux-kernel@vger.kernel.org Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Dec 12, 2018 at 01:14:10PM -0500, Jeremy Cline wrote: > Hi, > > On Wed, Dec 12, 2018 at 02:12:09PM +0100, Thierry Reding wrote: > > From: Thierry Reding > > > > The spdxcheck script currently falls over when confronted with a binary > > file (such as Documentation/logo.gif). To avoid that, always open files > > in binary mode and decode line-by-line, ignoring encoding errors. I suggest pointing out that the breakage only happens with python3 and results in a UnicodeDecodeError. > > One tricky case is when piping data into the script and reading it from > > standard input. By default, standard input will be opened in text mode, > > so we need to reopen it in binary mode. > > > > Signed-off-by: Thierry Reding > > --- > > scripts/spdxcheck.py | 6 ++++-- > > 1 file changed, 4 insertions(+), 2 deletions(-) > > > > diff --git a/scripts/spdxcheck.py b/scripts/spdxcheck.py > > index 5056fb3b897d..e559c6294c39 100755 > > --- a/scripts/spdxcheck.py > > +++ b/scripts/spdxcheck.py > > @@ -168,6 +168,7 @@ class id_parser(object): > > self.curline = 0 > > try: > > for line in fd: > > + line = line.decode(locale.getpreferredencoding(False), errors='ignore') > > self.curline += 1 > > if self.curline > maxlines: > > break > > @@ -249,12 +250,13 @@ if __name__ == '__main__': > > > > try: > > if len(args.path) and args.path[0] == '-': > > - parser.parse_lines(sys.stdin, args.maxlines, '-') > > + stdin = os.fdopen(sys.stdin.fileno(), 'rb') > > + parser.parse_lines(stdin, args.maxlines, '-') > > else: > > if args.path: > > for p in args.path: > > if os.path.isfile(p): > > - parser.parse_lines(open(p), args.maxlines, p) > > + parser.parse_lines(open(p, 'rb'), args.maxlines, p) > > elif os.path.isdir(p): > > scan_git_subtree(repo.head.reference.commit.tree, p) > > else: > > -- > > 2.19.1 > > > > It might be worth noting this fixes commit 6f4d29df66ac > ("scripts/spdxcheck.py: make python3 compliant") and also Cc this for > stable since 6f4d29df66ac got backported to v4.19. While that commit > did indeed make the script work with Python 3 for piping data, it broke > Python 2 and made its way to stable. It didn't break for me. Can you provide details about how and when it broke for you? Best regards Uwe -- Pengutronix e.K. | Uwe Kleine-K?nig | Industrial Linux Solutions | http://www.pengutronix.de/ |