Received: by 2002:ad5:474a:0:0:0:0:0 with SMTP id i10csp3201251imu; Mon, 17 Dec 2018 15:23:05 -0800 (PST) X-Google-Smtp-Source: AFSGD/U37feLRArh0cRuGw4xmYHLyZxpf177cObvJh/JwvEmjyw6fwkSTr0z/AwroJ3PJg+u6Mo2 X-Received: by 2002:a17:902:541:: with SMTP id 59mr14698273plf.88.1545088985770; Mon, 17 Dec 2018 15:23:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1545088985; cv=none; d=google.com; s=arc-20160816; b=KYy5BBG09MVWqe9z9OaJLBf4sjYptTzNPwquLAdtvPjpFVVw3Bg9saFgWRCC8gFvza FoaKBlYaRAmBrBH0+ozLiju7b/Tp2jBGFRRAoZsGTUguxRIzjbX86Gclhm/j8gpe0tHx N0gLySOBClbJk6UNsl5FMttdoWXcwkkzSipqRk72D9LoqpQwWS+JQz/dkTn8Kj7Bf+cn kDnhHWhRZZA+w3S/oJtzrGcFcOjj82lAGDxVpJ77KNxd4IshwXYGRJuGLQH4yD0E9tVm mCNWoJYUZQTTy9n1q2uVn4mMfCCIIQkJrRqBnlJi7Xamu+fAhkIUIrywOCObh57HFpes GYJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date:dkim-signature; bh=tRqDX3mvbAKAnKDtvep2dsCs/MxLTiDqqvg91a+H7gw=; b=kS/JvQ1iNz7Dnc56gM7i5A4W9nE7SwiSntm5ZUovIvdU+FlogREh7sgqxZlv4lHEaP rpLSkexpbLwOPOwdxgN1YEALKVPhRutKQKrnrtFG6ywfigjCcDzfzThmN44SfCLQAyf6 SIqMuVWYvYnLSeBnM+QVebjgfkjzC1h0BKSGwBEfHCRmAadNFNgAaucnsPRrVuuqa309 Lg29eG9FcO1McI1446iyeQ9pymylW4fg+tNWm3BEYHVL1pmUu5fwuepXJ78Yihy3l2xI 3culscTTIyd+zopTe59zwqX4XSsU7drUwDkOn4QkcMI8glHPu8F9NJGOQNdQm5F0251R WKGg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=jX2P+Zu1; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id e2si11775161pgs.94.2018.12.17.15.22.50; Mon, 17 Dec 2018 15:23:05 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=jX2P+Zu1; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728328AbeLQVk2 (ORCPT + 99 others); Mon, 17 Dec 2018 16:40:28 -0500 Received: from mail-pf1-f193.google.com ([209.85.210.193]:37563 "EHLO mail-pf1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727594AbeLQVk2 (ORCPT ); Mon, 17 Dec 2018 16:40:28 -0500 Received: by mail-pf1-f193.google.com with SMTP id y126so7014024pfb.4 for ; Mon, 17 Dec 2018 13:40:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=tRqDX3mvbAKAnKDtvep2dsCs/MxLTiDqqvg91a+H7gw=; b=jX2P+Zu1EKzHOg06kJmhb2Ih+p+ePF5hoi7VPhnoGFKyPMMMvwpDALriktE6bq2Y9e 786y9XQQ7UYYqYgOwzKioPG1/UTjvqYCSkuFzcSVmcz2jb1EUCrDc4L75r0GWpEJd+P8 u3wWoBypPdPyiXOZ0VeI8zKmeW1vF7KgdRF/t6cxuAsz1uVSWLo9vn629w572nu93CZ3 NOU3dI5LzFOAcM++iJhbZ6VGepBnJAbYPNc12PNJR0INLEVKTrBBORHqYxQKyNc8N+YN +ZMxcMqUpPRazwzS9BvXJwHsAopR3dQz4mvlKgARCw8P7DLkoARDhOdT/58YRPg1/BB2 qEkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=tRqDX3mvbAKAnKDtvep2dsCs/MxLTiDqqvg91a+H7gw=; b=eM3sF2YPm/Q2747x3ds0Zjod8v5yMal7cUENaAP2PgB7+/i2svL+UqlFu9r0YujOqc Qi1xChFi6RqIIXF+bJsO82+n7X2fmouGUPRQbmq87600C/SnLJGzc8DHIiioKLwxprjG z2OxXzPufd5Ujax6vxxQ+AnHVZgPowuxLm4S0MtY7Z1uiL6ftkGspZR6bW/khL4iVnTr LnMf4RPqQRDV9kxSZrfoRUCMN9ZlFGvnsR9jqer10mEHTXWlH0ytsYbduR19UdOMczEc bFSlTNLcvaH0csu4fdwokE95aM3jlGHtU8bPvNLmtx14ZcmyDr243hSNUqoDwyEMKuD/ OdYQ== X-Gm-Message-State: AA+aEWYmoMb6k+EeLy671qC7fEzljqPtIgLsbzUg6eyiHzeYSckuz8Lt vhz2EWOv5PWIzgwIAV6gwzIktP4xc38R5Q== X-Received: by 2002:a62:c101:: with SMTP id i1mr14269517pfg.80.1545082826824; Mon, 17 Dec 2018 13:40:26 -0800 (PST) Received: from google.com ([2620:15c:17:4:f0b1:8ff5:16a0:5f15]) by smtp.gmail.com with ESMTPSA id k14sm19189290pgs.52.2018.12.17.13.40.25 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 17 Dec 2018 13:40:25 -0800 (PST) Date: Mon, 17 Dec 2018 13:40:22 -0800 From: Tom Roeder To: Masahiro Yamada Cc: Michal Marek , Linux Kbuild mailing list , Linux Kernel Mailing List Subject: Re: [PATCH] scripts: add a tool to produce a compile_commands.json file Message-ID: <20181217214022.GA38778@google.com> References: <20181206222318.218157-1-tmroeder@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Sat, Dec 15, 2018 at 06:37:49PM +0900, Masahiro Yamada wrote: > On Fri, Dec 7, 2018 at 7:24 AM Tom Roeder wrote: > > > > The LLVM/Clang project provides many tools for analyzing C source code. > > Many of these tools are based on LibTooling > > (https://clang.llvm.org/docs/LibTooling.html), which depends on a > > database of compiler flags. The standard container for this database is > > compile_commands.json, which consists of a list of JSON objects, each > > with "directory", "file", and "command" fields. > > > > Some build systems, like cmake or bazel, produce this compilation > > information directly. Naturally, Makefiles don't. However, the kernel > > makefiles already create ..o.cmd files that contain all the > > information needed to build a compile_commands.json file. > > > > So, this commit adds scripts/gen_compile_commands.py, which recursively > > searches through a directory for ..o.cmd files and extracts > > appropriate compile commands from them. It writes a > > compile_commands.json file that LibTooling-based tools can use. > > > > By default, gen_compile_commands.py starts its search in its working > > directory and (over)writes compile_commands.json in the working > > directory. However, it also supports --output and --directory flags for > > out-of-tree use. > > > > Note that while gen_compile_commands.py enables the use of clang-based > > tools, it does not require the kernel to be compiled with clang. E.g., > > the following sequence of commands produces a compile_commands.json file > > that works correctly with LibTooling. > > > > make defconfig > > make > > scripts/gen_compile_commands.py > > > > Also note that this script is written to work correctly in both Python 2 > > and Python 3, so it does not specify the Python version in its first > > line. > > > > For an example of the utility of this script: after running > > gen_compile_commands.json on the latest kernel version, I was able to > > use Vim + the YouCompleteMe pluging + clangd to automatically jump to > > definitions and declarations. Obviously, cscope and ctags provide some > > of this functionality; the advantage of supporting LibTooling is that it > > opens the door to many other clang-based tools that understand the code > > directly and do not rely on regular expressions and heuristics. > > > > Tested: Built several recent kernel versions and ran the script against > > them, testing tools like clangd (for editor/LSP support) and clang-check > > (for static analysis). Also extracted some test .cmd files from a kernel > > build and wrote a test script to check that the script behaved correctly > > with all permutations of the --output and --directory flags. > > > > Signed-off-by: Tom Roeder > > > I am fine with this, > but I have one question. > > The generated compile_commands.json > contains $(pound) To make sure we're talking about the same thing: the instances that I've seen of "#" occur in macro definitions in the "command" field in some of the JSON objects. For example, I see things like -D\"KBUILD_STR(s)=\\#s\". > > How is it handled? The Python json module takes care of escaping the output to make a valid JSON string for the "command" field. The gen_compile_commands.py script doesn't take any special action for that or any other character in its output. > Should it be replaced with '\#' ? I don't think it needs to be changed, given my experience with this script and its testing so far: the output seems to work for me. However, are you running into problems due to the presence of this character or inadequate escaping? Please let me know, and I'd be happy to look into it. Tom