Received: by 2002:ac0:a594:0:0:0:0:0 with SMTP id m20-v6csp1552675imm; Tue, 22 May 2018 05:59:08 -0700 (PDT) X-Google-Smtp-Source: AB8JxZpaoxnYXevSuCRazX5bvNtarGZ9+kWJBzSOWPK1PqWXoqnJ11I3vF9I2WCzGS1cu+DycKrS X-Received: by 2002:aa7:82c7:: with SMTP id f7-v6mr1318263pfn.218.1526993948038; Tue, 22 May 2018 05:59:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1526993948; cv=none; d=google.com; s=arc-20160816; b=w+3BpvYlNBkOrysNWwmsIX9h41oHeb7h7DioJ0jTf7XwaQHVxeOjlUNd3OOSLzNcbu LLTMPZdVJHED1Jcoo+nfTMPGtgiOJm0pvgFmgVrBYl5nKH1NYIAkLXAThAHWalbVS0Kx WocXwQW1y3C/v9fj4ASsy14B/dLPF7oUOKr9EfEdWhMk+j1Z32qBOP8rQwHdSgm3vGBs oz/x05d+A0dGxn4DSvmy57K+qMwYNid2wl3rhtb2O5mnyQcG0g/dfFg5r6ewGDTkN9nk 0hZwngwb7LLobfWtDjRuFBlTCABRl7A7rSp9EKKAiA/wc95E6cy+PEeiv74CJ25JtmX5 HRLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:user-agent:in-reply-to :content-disposition:mime-version:references:message-id:subject:cc :to:from:date:arc-authentication-results; bh=anuKMUFu5macAjKaoeuRUGmfqR4HNTrt/70o3tv9Z60=; b=gFbZaTxM5x+o+z8B/x1taXjh9afwG07T4Vqa6yKR+UsZn9hXoRlOmRc1L0MpYKAa9Q W99sPpDTih3BbZoZRYn86LBbIePMEEZGzTlGlZ6YQSG176FE+Ao0z45gEUTEV/8Vmki5 SMJHdBgY3Nsm3iM97udqmoEC8jP6K5gsnA5h6axEJjHT0xN0QDzZwjRTrIRYwPLwZmoy XDSGIgodSFCEimkPIYaeWN3C5DnX+Bg0/F7U3zau2XhegkUq4cWvSWg+N447uIcTpA4D zs8QqOUNmsNFpoEv8m9JnmqNsKDW0oDCz63gTJe6+eHriBTcvobkfWbzL86moT0rRFDF SSSw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id d15-v6si17392427pln.533.2018.05.22.05.58.53; Tue, 22 May 2018 05:59:08 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751456AbeEVM57 (ORCPT + 99 others); Tue, 22 May 2018 08:57:59 -0400 Received: from mail-wm0-f67.google.com ([74.125.82.67]:54440 "EHLO mail-wm0-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751189AbeEVM5z (ORCPT ); Tue, 22 May 2018 08:57:55 -0400 Received: by mail-wm0-f67.google.com with SMTP id f6-v6so31638207wmc.4 for ; Tue, 22 May 2018 05:57:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=anuKMUFu5macAjKaoeuRUGmfqR4HNTrt/70o3tv9Z60=; b=jxDfGyW6RYS/XS18xrUMJf75iT3nINh8yiocUY0BjHBPLqyHYHK0V3T+++Dlfo1mDQ lnQ6ALOSjzwl4qkqkjZnCtA7n3Jy8qjpOwYNfHZ5Nl1rvfhD8tPqJZy6DOEeQqRS4mph OodYLF2lC33SJO/HGS5AsToLYJJ7EKl8/kIKs4EtELAP4gWy4+VR1WkmZ27Fi5pYTG2F OV6djWuONGZgSNi08B3xnQmzEtnf0l+rqcBg/xbE2Fjx79iLI3HAhJPK28c8VcuIab9O 9sWt/Nmv1rpbVD7X0UHw5xIUWNcJiq/zV/LbrSyLmIvo6wAson2fKItj3g75Kmn2tsqQ CYww== X-Gm-Message-State: ALKqPwerPrlrYiA3LoibwdjVkEcKeA9LCbjsH2KDRdjaXeHQO78gF3lf u+O0riiXhLRSW69dSqIPtvZrM3DMHBs= X-Received: by 2002:a1c:2d93:: with SMTP id t141-v6mr1227847wmt.94.1526993873865; Tue, 22 May 2018 05:57:53 -0700 (PDT) Received: from localhost.localdomain ([151.15.207.242]) by smtp.gmail.com with ESMTPSA id n11-v6sm5534799wrh.10.2018.05.22.05.57.52 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 22 May 2018 05:57:53 -0700 (PDT) Date: Tue, 22 May 2018 14:57:50 +0200 From: Juri Lelli To: Waiman Long Cc: Tejun Heo , Li Zefan , Johannes Weiner , Peter Zijlstra , Ingo Molnar , cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kernel-team@fb.com, pjt@google.com, luto@amacapital.net, Mike Galbraith , torvalds@linux-foundation.org, Roman Gushchin Subject: Re: [PATCH v8 2/6] cpuset: Add new v2 cpuset.sched.domain flag Message-ID: <20180522125750.GA31040@localhost.localdomain> References: <1526590545-3350-1-git-send-email-longman@redhat.com> <1526590545-3350-3-git-send-email-longman@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1526590545-3350-3-git-send-email-longman@redhat.com> User-Agent: Mutt/1.9.2 (2017-12-15) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, On 17/05/18 16:55, Waiman Long wrote: [...] > /** > + * update_isolated_cpumask - update the isolated_cpus mask of parent cpuset > + * @cpuset: The cpuset that requests CPU isolation > + * @oldmask: The old isolated cpumask to be removed from the parent > + * @newmask: The new isolated cpumask to be added to the parent > + * Return: 0 if successful, an error code otherwise > + * > + * Changes to the isolated CPUs are not allowed if any of CPUs changing > + * state are in any of the child cpusets of the parent except the requesting > + * child. > + * > + * If the sched_domain flag changes, either the oldmask (0=>1) or the > + * newmask (1=>0) will be NULL. > + * > + * Called with cpuset_mutex held. > + */ > +static int update_isolated_cpumask(struct cpuset *cpuset, > + struct cpumask *oldmask, struct cpumask *newmask) > +{ > + int retval; > + int adding, deleting; > + cpumask_var_t addmask, delmask; > + struct cpuset *parent = parent_cs(cpuset); > + struct cpuset *sibling; > + struct cgroup_subsys_state *pos_css; > + int old_count = parent->isolation_count; > + bool dying = cpuset->css.flags & CSS_DYING; > + > + /* > + * Parent must be a scheduling domain with non-empty cpus_allowed. > + */ > + if (!is_sched_domain(parent) || cpumask_empty(parent->cpus_allowed)) > + return -EINVAL; > + > + /* > + * The oldmask, if present, must be a subset of parent's isolated > + * CPUs. > + */ > + if (oldmask && !cpumask_empty(oldmask) && (!parent->isolation_count || > + !cpumask_subset(oldmask, parent->isolated_cpus))) { > + WARN_ON_ONCE(1); > + return -EINVAL; > + } > + > + /* > + * A sched_domain state change is not allowed if there are > + * online children and the cpuset is not dying. > + */ > + if (!dying && (!oldmask || !newmask) && > + css_has_online_children(&cpuset->css)) > + return -EBUSY; > + > + if (!zalloc_cpumask_var(&addmask, GFP_KERNEL)) > + return -ENOMEM; > + if (!zalloc_cpumask_var(&delmask, GFP_KERNEL)) { > + free_cpumask_var(addmask); > + return -ENOMEM; > + } > + > + if (!old_count) { > + if (!zalloc_cpumask_var(&parent->isolated_cpus, GFP_KERNEL)) { > + retval = -ENOMEM; > + goto out; > + } > + old_count = 1; > + } > + > + retval = -EBUSY; > + adding = deleting = false; > + if (newmask) > + cpumask_copy(addmask, newmask); > + if (oldmask) > + deleting = cpumask_andnot(delmask, oldmask, addmask); > + if (newmask) > + adding = cpumask_andnot(addmask, newmask, delmask); > + > + if (!adding && !deleting) > + goto out_ok; > + > + /* > + * The cpus to be added must be in the parent's effective_cpus mask > + * but not in the isolated_cpus mask. > + */ > + if (!cpumask_subset(addmask, parent->effective_cpus)) > + goto out; > + if (parent->isolation_count && > + cpumask_intersects(parent->isolated_cpus, addmask)) > + goto out; > + > + /* > + * Check if any CPUs in addmask or delmask are in a sibling cpuset. > + * An empty sibling cpus_allowed means it is the same as parent's > + * effective_cpus. This checking is skipped if the cpuset is dying. > + */ > + if (dying) > + goto updated_isolated_cpus; > + > + cpuset_for_each_child(sibling, pos_css, parent) { > + if ((sibling == cpuset) || !(sibling->css.flags & CSS_ONLINE)) > + continue; > + if (cpumask_empty(sibling->cpus_allowed)) > + goto out; > + if (adding && > + cpumask_intersects(sibling->cpus_allowed, addmask)) > + goto out; > + if (deleting && > + cpumask_intersects(sibling->cpus_allowed, delmask)) > + goto out; > + } Just got the below by echoing 1 into cpuset.sched.domain of a sibling with "isolated" cpuset.cpus. Guess you are missing proper locking about here above. --->8--- [ 7509.905005] ============================= [ 7509.905009] WARNING: suspicious RCU usage [ 7509.905014] 4.17.0-rc5+ #11 Not tainted [ 7509.905017] ----------------------------- [ 7509.905023] /home/juri/work/kernel/linux/kernel/cgroup/cgroup.c:3826 cgroup_mutex or RCU read lock required! [ 7509.905026] other info that might help us debug this: [ 7509.905031] rcu_scheduler_active = 2, debug_locks = 1 [ 7509.905036] 4 locks held by bash/1480: [ 7509.905039] #0: 00000000bf288709 (sb_writers#6){.+.+}, at: vfs_write+0x18a/0x1b0 [ 7509.905072] #1: 00000000ebf23fc9 (&of->mutex){+.+.}, at: kernfs_fop_write+0xe2/0x1a0 [ 7509.905098] #2: 00000000de7c626e (kn->count#302){.+.+}, at: kernfs_fop_write+0xeb/0x1a0 [ 7509.905124] #3: 00000000a6a2bd9f (cpuset_mutex){+.+.}, at: cpuset_write_u64+0x23/0x140 [ 7509.905149] stack backtrace: [ 7509.905156] CPU: 6 PID: 1480 Comm: bash Not tainted 4.17.0-rc5+ #11 [ 7509.905160] Hardware name: LENOVO 30B6S2F900/1030, BIOS S01KT56A 01/15/2018 [ 7509.905164] Call Trace: [ 7509.905176] dump_stack+0x85/0xcb [ 7509.905187] css_next_child+0x90/0xd0 [ 7509.905195] update_isolated_cpumask+0x18f/0x2e0 [ 7509.905208] update_flag+0x1f3/0x210 [ 7509.905220] cpuset_write_u64+0xff/0x140 [ 7509.905230] cgroup_file_write+0x178/0x230 [ 7509.905244] kernfs_fop_write+0x113/0x1a0 [ 7509.905254] __vfs_write+0x36/0x180 [ 7509.905264] ? rcu_read_lock_sched_held+0x6b/0x80 [ 7509.905270] ? rcu_sync_lockdep_assert+0x2e/0x60 [ 7509.905278] ? __sb_start_write+0x13e/0x1a0 [ 7509.905283] ? vfs_write+0x18a/0x1b0 [ 7509.905293] vfs_write+0xc1/0x1b0 [ 7509.905302] ksys_write+0x55/0xc0 [ 7509.905317] do_syscall_64+0x60/0x200 [ 7509.905327] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 7509.905333] RIP: 0033:0x7fee4fdfe414 [ 7509.905338] RSP: 002b:00007fff364a80a8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001 [ 7509.905346] RAX: ffffffffffffffda RBX: 0000000000000002 RCX: 00007fee4fdfe414 [ 7509.905350] RDX: 0000000000000002 RSI: 000055eb12f93740 RDI: 0000000000000001 [ 7509.905354] RBP: 000055eb12f93740 R08: 000000000000000a R09: 00007fff364a7c30 [ 7509.905358] R10: 000000000000000a R11: 0000000000000246 R12: 00007fee500cd760 [ 7509.905361] R13: 0000000000000002 R14: 00007fee500c8760 R15: 0000000000000002 --->8--- Best, - Juri