Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1031981AbdDUAir (ORCPT ); Thu, 20 Apr 2017 20:38:47 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:39213 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S948247AbdDUAio (ORCPT ); Thu, 20 Apr 2017 20:38:44 -0400 Date: Thu, 20 Apr 2017 17:38:34 -0700 From: "Paul E. McKenney" To: Paolo Bonzini Cc: Michael Ellerman , linux-kernel@vger.kernel.org, "paulus@samba.org" , kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, "linuxppc-dev@lists.ozlabs.org" , mingo@kernel.org, jiangshanlai@gmail.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@efficios.com, josh@joshtriplett.org, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com, fweisbec@gmail.com, oleg@redhat.com, bobby.prani@gmail.com Subject: Re: powerpc KVM build break in linux-next (was Re: [PATCH tip/core/rcu 40/40] srcu: Parallelize callback handling) Reply-To: paulmck@linux.vnet.ibm.com References: <20170412174003.GA23207@linux.vnet.ibm.com> <1492018825-25634-40-git-send-email-paulmck@linux.vnet.ibm.com> <871ssn692a.fsf@concordia.ellerman.id.au> <24c85d26-bb4b-892d-5837-6d02af570e0e@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <24c85d26-bb4b-892d-5837-6d02af570e0e@redhat.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 x-cbid: 17042100-0044-0000-0000-0000030ED96C X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006946; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000208; SDB=6.00850246; UDB=6.00419963; IPR=6.00628945; BA=6.00005305; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00015114; XFM=3.00000013; UTC=2017-04-21 00:38:41 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17042100-0045-0000-0000-0000073CDFDD Message-Id: <20170421003834.GA14351@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-04-20_22:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1703280000 definitions=main-1704210010 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 3519 Lines: 86 On Thu, Apr 20, 2017 at 05:28:32PM +0200, Paolo Bonzini wrote: > > > On 20/04/2017 05:40, Michael Ellerman wrote: > > "Paul E. McKenney" writes: > > > >> diff --git a/include/linux/srcutree.h b/include/linux/srcutree.h > >> index f2b3bd6c6bc2..0400e211aa44 100644 > >> --- a/include/linux/srcutree.h > >> +++ b/include/linux/srcutree.h > >> @@ -24,25 +24,75 @@ > > ... > >> +/* > >> + * Per-SRCU-domain structure, similar in function to rcu_state. > >> + */ > >> struct srcu_struct { > >> - unsigned long completed; > >> - unsigned long srcu_gp_seq; > >> - atomic_t srcu_exp_cnt; > >> - struct srcu_array __percpu *per_cpu_ref; > >> - spinlock_t queue_lock; /* protect ->srcu_cblist */ > >> - struct rcu_segcblist srcu_cblist; > >> + struct srcu_node node[NUM_RCU_NODES]; /* Combining tree. */ > >> + struct srcu_node *level[RCU_NUM_LVLS + 1]; > >> + /* First node at each level. */ > >> + struct mutex srcu_cb_mutex; /* Serialize CB preparation. */ > >> + spinlock_t gp_lock; /* protect ->srcu_cblist */ > >> + struct mutex srcu_gp_mutex; /* Serialize GP work. */ > >> + unsigned int srcu_idx; /* Current rdr array element. */ > >> + unsigned long srcu_gp_seq; /* Grace-period seq #. */ > >> + unsigned long srcu_gp_seq_needed; /* Latest gp_seq needed. */ > >> + atomic_t srcu_exp_cnt; /* # ongoing expedited GPs. */ > >> + struct srcu_data __percpu *sda; /* Per-CPU srcu_data array. */ > >> + unsigned long srcu_barrier_seq; /* srcu_barrier seq #. */ > >> + struct mutex srcu_barrier_mutex; /* Serialize barrier ops. */ > >> + struct completion srcu_barrier_completion; > >> + /* Awaken barrier rq at end. */ > >> + atomic_t srcu_barrier_cpu_cnt; /* # CPUs not yet posting a */ > >> + /* callback for the barrier */ > >> + /* operation. */ > > > > This change seems to have had the non-obvious effect of breaking the > > powerpc KVM build. > > > > Because struct kvm contains two srcu_structs which are located > > before the kvm_arch struct, the increase in size of srcu_struct has > > caused the offset from the start of struct kvm to kvm_arch to be too big > > for some of our asm. > > > > struct kvm { > > spinlock_t mmu_lock; > > struct mutex slots_lock; > > struct mm_struct *mm; /* userspace tied to this vm */ > > struct kvm_memslots *memslots[KVM_ADDRESS_SPACE_NUM]; > > struct srcu_struct srcu; > > struct srcu_struct irq_srcu; > > ... > > struct kvm_arch arch; > > > > > > Example error: > > arch/powerpc/kvm/book3s_hv_rmhandlers.S:617: Error: operand out of range (0x000000000000b328 is not between 0xffffffffffff8000 and 0x0000000000007fff) > > > > Where line 617 is: > > lwz r7,KVM_LPID(r9) > > > > And the KVM_LPID constant comes from asm-offsets.s. The diff of old vs > > new is: > > > > -->KVM_LPID 17752 offsetof(struct kvm, arch.lpid) # > > +->KVM_LPID 45864 offsetof(struct kvm, arch.lpid) # > > > > > > We can probably just fix it by changing the asm to keep the address of > > kvm_arch in a register, and then offset from that. > > Michael, if you want to move the two srcu structs at the end of struct > kvm, that would be fine by me. Please send a patch yourself so you can > test it on PPC. Thanks, On the off-chance that it is at all helpful, I have added the commit shown below to -rcu. If it helps, I am happy to push this, but am just as happy to drop it in favor of some other fix. If at all possible, I would like to get this into the upcoming merge window. Thanx, Paul