Received: by 2002:a25:b323:0:0:0:0:0 with SMTP id l35csp334944ybj; Thu, 19 Sep 2019 15:16:21 -0700 (PDT) X-Google-Smtp-Source: APXvYqzoKuDCdU2jChg7+syCurYJj2N01hfwM48AZyiSzF4VVzTc6M6H6TDNrFZnGmh7zTGMFDY0 X-Received: by 2002:a17:906:9381:: with SMTP id l1mr10992621ejx.93.1568931381327; Thu, 19 Sep 2019 15:16:21 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1568931381; cv=none; d=google.com; s=arc-20160816; b=LIPdxAmfnlbJwFU+jEasnunS9kGbsAn6uproFfo0BBAV1/cNmiAYBWGZcqrSDDre3q MAzrWsCb1RdHeoUe1l9Ubb0UIY89R3yBDvPsgVLCdX+8O1ICEyQAZ8ERBrp9So0UDag3 H61SmwmtuylCD8CbkGklQTAJXh0BM5qdTVdg0DzNp1skKNb5xxHHa6US8RbOcsM+6YGw 7M7gaEJv/CVHPuMMO5TG80wWPD2y14sjDdXqtbinTNFL+30f2ogiJLowhS69AZX8606n ICBkqwsWCBfnjKj5/rPdx2VL0UY8NC6dBww2qO/sGg30rOZE/YFP1uIttYwfAFu8ZUbM BtWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:content-transfer-encoding:mime-version :references:in-reply-to:message-id:subject:cc:to:from:date :dkim-signature; bh=3vJe3OMoI9/fq2WHf+U73uPJ4vyM059pSbCzmST6qPQ=; b=tqfnN7GNLcm9i7oSZOx7UJvqqcjcWdEXgBHH2IUqr6jBO5mF1nlKEuK6wy00YwZ4L0 zeyKixRwnrAfzGYLWITuq17eaGtAJH0Qusa1kF+QTLUAJNV/WtZhJCEqFVmvWzxPYleR tbR7rhPpRPMcZLcSvdkY6ZI4ZNUpV4M6gY/PzYysB3baMn3UXAZYc4D8U4fcbZMbMw7A nj9XzMbtlT1f9MAoCM+5ITQ8B5YuF4UB/DYj4GOr/u5vGjXxGRChj1q2d1kPLGO1vtaf ry+rBH1AqJpFbetcdiDF3hKGpR9sDf3K8tL3KpObP71HOqIpNSTdEE+Zt1Wib04+p26q WFsg== ARC-Authentication-Results: i=1; mx.google.com; dkim=fail header.i=@infradead.org header.s=bombadil.20170209 header.b=BGQVcMar; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id u17si71837edy.145.2019.09.19.15.15.58; Thu, 19 Sep 2019 15:16:21 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=fail header.i=@infradead.org header.s=bombadil.20170209 header.b=BGQVcMar; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2404429AbfISTZ4 (ORCPT + 99 others); Thu, 19 Sep 2019 15:25:56 -0400 Received: from bombadil.infradead.org ([198.137.202.133]:52612 "EHLO bombadil.infradead.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2404419AbfISTZ4 (ORCPT ); Thu, 19 Sep 2019 15:25:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Subject:Cc:To: From:Date:Sender:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=3vJe3OMoI9/fq2WHf+U73uPJ4vyM059pSbCzmST6qPQ=; b=BGQVcMartnwZd1GiAMWMsufYs DHfSQ1lrTCooCqwFpGEeHsN2xJiMAFvx0hsUO4aG5GZAO9hloi0XaJG5GH07mARrm+bj8rApUEYVb 5VIK2ILOq49RUIBvPcuQR2pJ4ELd4F+gShfI2J3tNi9cGSMusJn5uxnRMgPtgQG5BLQMZMQe7t9RL dn3JHXYHbXl1k60kSFeoiMgInXOxZny6DbA2GHHppvcGLoiHke3/zqQDzu/u4S5Gt+oJYfXRIn8j5 OE/TwRRgvU2NDIbQUAl9Rp3BysfnUc+YY2PN+tN1DJASSemKkt7aBSvzaMgCe+xJ6NSaXhIEKYhha CQOJ1nYlw==; Received: from 201.86.135.82.dynamic.adsl.gvt.net.br ([201.86.135.82] helo=coco.lan) by bombadil.infradead.org with esmtpsa (Exim 4.92.2 #3 (Red Hat Linux)) id 1iB23u-0000Gh-8N; Thu, 19 Sep 2019 19:25:54 +0000 Date: Thu, 19 Sep 2019 16:25:49 -0300 From: Mauro Carvalho Chehab To: Kees Cook Cc: Jonathan Corbet , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH] docs: Use make invocation's -j argument for parallelism Message-ID: <20190919162549.5ccdab62@coco.lan> In-Reply-To: <201909191116.192A096C68@keescook> References: <201909191116.192A096C68@keescook> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; x86_64-redhat-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Em Thu, 19 Sep 2019 11:18:52 -0700 Kees Cook escreveu: > While sphinx 1.7 and later supports "-jauto" for parallelism, this > effectively ignores the "-j" flag used in the "make" invocation, which > may cause confusion for build systems. Instead, extract the available > parallelism from "make"'s job server (since it is not exposed in any > special variables) and use that for the "sphinx-build" run. Now things > work correctly for builds where -j is specified at the top-level: > > make -j16 htmldocs > > Signed-off-by: Kees Cook > --- > Documentation/Makefile | 3 +-- > scripts/jobserver-count | 46 +++++++++++++++++++++++++++++++++++++++++ > 2 files changed, 47 insertions(+), 2 deletions(-) > create mode 100755 scripts/jobserver-count > > diff --git a/Documentation/Makefile b/Documentation/Makefile > index e145e4db508b..4408eeaf2891 100644 > --- a/Documentation/Makefile > +++ b/Documentation/Makefile > @@ -33,8 +33,6 @@ ifeq ($(HAVE_SPHINX),0) > > else # HAVE_SPHINX > > -export SPHINXOPTS = $(shell perl -e 'open IN,"sphinx-build --version 2>&1 |"; while () { if (m/([\d\.]+)/) { print "-jauto" if ($$1 >= "1.7") } ;} close IN') > - > # User-friendly check for pdflatex and latexmk > HAVE_PDFLATEX := $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo 1; else echo 0; fi) > HAVE_LATEXMK := $(shell if which latexmk >/dev/null 2>&1; then echo 1; else echo 0; fi) > @@ -68,6 +66,7 @@ quiet_cmd_sphinx = SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4) > PYTHONDONTWRITEBYTECODE=1 \ > BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(srctree)/$(src)/$5/$(SPHINX_CONF)) \ > $(SPHINXBUILD) \ > + -j $(shell $(srctree)/scripts/jobserver-count) \ > -b $2 \ > -c $(abspath $(srctree)/$(src)) \ > -d $(abspath $(BUILDDIR)/.doctrees/$3) \ > diff --git a/scripts/jobserver-count b/scripts/jobserver-count > new file mode 100755 > index 000000000000..5fc9d2fd5254 > --- /dev/null > +++ b/scripts/jobserver-count > @@ -0,0 +1,46 @@ > +#!/usr/bin/env python3 > +# SPDX-License-Identifier: GPL-2.0-or-later > +# > +# This determines how many parallel tasks "make" is expecting, as it is > +# not exposed via an special variables. > +# https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#POSIX-Jobserver > +import os, sys, fcntl > + > +# Set non-blocking for a given file descriptor. > +def nonblock(fd): > + flags = fcntl.fcntl(fd, fcntl.F_GETFL) > + fcntl.fcntl(fd, fcntl.F_SETFL, flags | os.O_NONBLOCK) > + return fd > + > +# Fetch the make environment options. > +flags = os.environ.get('MAKEFLAGS', None) > +if flags == None: > + print("1") > + sys.exit(0) > + > +# Look for "--jobserver=R,W" > +opts = [x for x in flags.split(" ") if x.startswith("--jobserver")] > +if len(opts) != 1: > + print("1") > + sys.exit(0) Using "1" as default if -j is not specified doesn't sound a good idea. I mean, Sphinx is very slow without any parallelism. I would keep "auto" as default here, if version >= 1.7 (and if there's no explicit SPHINXOPTS). > + > +# Parse out R,W file descriptor numbers and set them nonblocking. > +fds = opts[0].split("=", 1)[1] > +reader, writer = [nonblock(int(x)) for x in fds.split(",", 1)] > + > +# Read out as many jobserver slots as possible. > +jobs = b"" > +while True: > + try: > + slot = os.read(reader, 1) > + jobs += slot > + except: > + break > +# Return all the reserved slots. > +os.write(writer, jobs) > + > +# Report available slots (with a bump for our caller's reserveration). > +if len(jobs) < 1: > + print("1") > +else: > + print(len(jobs) + 1) Thanks, Mauro