Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932153AbbLDXvF (ORCPT ); Fri, 4 Dec 2015 18:51:05 -0500 Received: from e38.co.us.ibm.com ([32.97.110.159]:36361 "EHLO e38.co.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753962AbbLDXtu (ORCPT ); Fri, 4 Dec 2015 18:49:50 -0500 X-IBM-Helo: d03dlp03.boulder.ibm.com X-IBM-MailFrom: paulmck@linux.vnet.ibm.com X-IBM-RcptTo: linux-kernel@vger.kernel.org From: "Paul E. McKenney" To: linux-kernel@vger.kernel.org Cc: mingo@kernel.org, jiangshanlai@gmail.com, dipankar@in.ibm.com, akpm@linux-foundation.org, mathieu.desnoyers@efficios.com, josh@joshtriplett.org, tglx@linutronix.de, peterz@infradead.org, rostedt@goodmis.org, dhowells@redhat.com, edumazet@google.com, dvhart@linux.intel.com, fweisbec@gmail.com, oleg@redhat.com, bobby.prani@gmail.com, "Paul E. McKenney" Subject: [PATCH tip/core/rcu 1/8] documentation: Record RCU requirements Date: Fri, 4 Dec 2015 15:50:19 -0800 Message-Id: <1449273026-23633-1-git-send-email-paulmck@linux.vnet.ibm.com> X-Mailer: git-send-email 2.5.2 In-Reply-To: <20151204234952.GA22805@linux.vnet.ibm.com> References: <20151204234952.GA22805@linux.vnet.ibm.com> X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 15120423-0029-0000-0000-00000E9D2E55 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 399245 Lines: 8671 This commit adds RCU requirements as published in a 2015 LWN series. Bringing these requirements in-tree allows them to be updated as changes are discovered. Signed-off-by: Paul E. McKenney --- .../RCU/Design/Requirements/2013-08-is-it-dead.png | Bin 0 -> 100825 bytes .../Design/Requirements/GPpartitionReaders1.svg | 374 +++ .../RCU/Design/Requirements/RCUApplicability.svg | 237 ++ .../Design/Requirements/ReadersPartitionGP1.svg | 639 +++++ .../RCU/Design/Requirements/Requirements.html | 2799 ++++++++++++++++++++ .../RCU/Design/Requirements/Requirements.htmlx | 2643 ++++++++++++++++++ Documentation/RCU/Design/htmlqqz.sh | 108 + 7 files changed, 6800 insertions(+) create mode 100644 Documentation/RCU/Design/Requirements/2013-08-is-it-dead.png create mode 100644 Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg create mode 100644 Documentation/RCU/Design/Requirements/RCUApplicability.svg create mode 100644 Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg create mode 100644 Documentation/RCU/Design/Requirements/Requirements.html create mode 100644 Documentation/RCU/Design/Requirements/Requirements.htmlx create mode 100755 Documentation/RCU/Design/htmlqqz.sh diff --git a/Documentation/RCU/Design/Requirements/2013-08-is-it-dead.png b/Documentation/RCU/Design/Requirements/2013-08-is-it-dead.png new file mode 100644 index 0000000000000000000000000000000000000000..7496a55e4e7b41becdb658a2bd34765fbe80013f GIT binary patch literal 100825 zcmaI8c|2C@8#TTWl}t}j$kbCx3K5Zr>Mlnb z5(*hAnadoRertEW@8|b<|9Ly7Pv__v_P+1yzOHqxwXS{jsE!6ZD?cki5bOsvX~zhH z$%G&n9aga5CvPs*`Qi@-w__UniR||R1Na{nvqKs*Vu}1Gu{<>jKUwLlY3PQxx01gZ z_KU@N;fKrJ4<1%q{*{%ohGTi>PcaUH5FrlI_UU=G4SzlFclgiJ^3m2e#`{@TYrXrd zCi=iA#kMd-Ho7gN&_g6+I_)pN#K@6M>l2Up)zT$SjzqR}Zcwik%8!cc`13nLA@xt) z;Y}2aviOJ^|6<4YehF?)@3$qmsR%REXmrLzMZ$@gVkl;GX6c@+t=zSphv)bJVMfpx zLUzr_^H+HgTNv)K+<1A6LeceHcbcX*Z@^^D-pxaFh9-Gk)jW87jggU2l3w~TXIIyY zj~`2H(|EPhJ0*4F=FKd+y1F-S-@e`a`@JmDC(M!;MR+n6Hw;oJld-v{j|M;=Y*Vp&t^;)`4>H&9mSu-;;E+L`o zPpleVbae?YE-n_k53K&!)YN=%ikN0FSZTVEu5dCUBxIdAO|PJwFilGmQ>Oh;Bb;4a zB7c2-w#}if;6iHZmX5Dq4MmguZrr{dveW8)cYkBDLAj6I;=-SMcT;b44*y}&qa?0uX$B#F8s6KU)#$_Qq+fJJH z*2NT`{~iA~UzpxVywBfH!>^z?X8MNEq{|FkTefUjkni%_h(*~?YtZoVDIM-8InFQd zxSd$WC+M}E`DnbbA&vHG`KgMK(AZl<7W2`jx;m}BbHh6=JBnSzPGoRzJD{hUb?Pa{ z_t)2|udSC)6w|sH63#rTyzKx@Z}QHQWi&N)eKp-8M(5v+$;)>+v^^|aTAa(xJNtW- z#(eO;qQ?%4TCU>a;_jKT?`z+@dE>;8#%j7;#z%V-HHjK3VL}T^;G+)^uiHLhN1o_&r!djG7+yY6q|w++XnH&RSWWF0{!UqTRrs^+X%-PL z<5qhA(J#@J&R@{Q=>(yL5icD%RU$wU0dLEXx!qwTiYOpzzM@^S7-M(!r*_VIMun9k^b;GX;`5)!)%1!dxg*CH z=Kko{MQ^RP<>EPO5-^+c*PShMA5BkobPsl=-q6i$;#T&VXTjnpGIIH8Y8~sH$}TW1 z=OC);2_1YgEoeQ0hLv+Y;jv!MR&V>ITgEw3(_09;qx0?~Z72K(U)gd|r2Qz3Pmc4@ z4W|a1S9Ju7ii)27l5cs$VHT0#)f=Oq8@ROCSDFwI5D;?ps<-5j&cTEGnjRg_vuY5k zSQARQ^vBoZ~zhByHG~kb^Tu2CSMM(db zbPnDH(Xy&y|G8a+h=hbh>x86X{xXEo%lG&9Jh}H10Z9;h_B`t=ium*E>j57h#b2G} zalh-0i|qOMWv%ItPd#1R&;Q|xex8|{!KqVrzXPfN;?$~Ikf@8|{+@MReiQH3_*)Iy1a#RVPBo7lsd;eV ziqI*h<;$02F`A@ocJ7RpIJGu3sTli#bHjW{RnJ-q-D_Vx_46UT8K`*W_?z_evp7A*EWk(cFT9m%a73 z9<&afJsbHaqO9-hIM4d^aeagq5)gTQXb&GgBnTX`c&*Wph={Uc4~J4W%PZ|7Iswyigb zR|!ejM@U?ER}@T2N(wh##m?URUZ09PxP~+m?bNeP{_%eE*Wu4EE$963#Oeqe?+zj| zGV->JC9779I5SdVwD<{DK}C1^ojZn)P9|KwyliH6_AHL~=gL48Y(`Ii{|YB3r?=P( z%erW3qt-W3TXdf~p85Rc%Z3db7(9krf>KgM7iar~$$WTA^349DN4b#K$9_Hg{cWn% zq@h46Jw2UB^qhMju^JcS#c;(MeorI@H+OOp1?od!ve9_dj#;xytP+VNx8xOpKtUswo5?+6K zV%0M+a5Y}pH_777%_Dw7_jbFnWS)HdqTKgSV$;BfVi$Aw;m^|TJ(T?i4v!+HlYqrp6FHu;CTB`*A&DbQ$mMy8;O&JmSmi1bGeq}}3I7X~Q zI=(+7R4uYq>PN^qae~<8)H15vs(YLxmszn1c#cr7Ee@5-)R&LPmf{x(7+L}qKQgo)^(SZ=~~sO7Q1;hff? z!oGKR<<<}~jvWfgHlNnAl^aVDn-mp|_Abr4F^Cu&8|V8>*&aN6xcO0~`@qeXq{hm? zz~q*##J8V6AC;f8vDtbeLoc#HW$E;TrH|q`}*4YoeuTLc(2~QJ51Exm9uj#3O}l z#jnoD#N?*zgatj!!WpBJx`s$<8n8e~%eQS7Pwr0A!UjuzW_Mzr2l{2E7e9Mum$O?@ zk^k$jTg?v9!TT(K)JA^9s*a6~MV_q=zIy4>CBLa=eW3c8ms7rTBTt#>;k?q+dK*I1 z%q+@%6~o;bAcikdS9(qs@(_L=0O zMB?b^5}Fe~-yP20C@FdUDRNHZ&Q~1rA^mSuG=(}HZ{NzxMJO}Ox5msRBSoCGvAK$@ zkbEpf3yA;9n}Mvyk9&UnxZK}w-BIds$oI_G&DGa=hKGl_)~^q{eVaF`-qp?R0FAb6 z|AD1vo!25FG&D6=;61vy2B0b)z%bdW+b5aIjU&iB(An9Uo10tp=y06&`}GtG1EJtMy&ZKyx5)lWjpNhg`8 z&LZ>;ik;6E{kvCp_kwA`gy9$KM!TwBMI7@zq*uA+QE<*sYYw?+yn8&i)0P1r7@!V6 z65^WJLm!;-30blE@ZHP80W&>$aw}G?YO>|x8P<*c>O>1|P1VpL0e*Az zJiI_G?;EFo$ZBVN?acf-YNf$$ZDV88HvVLlgYEJygVSj>o3>1()$2$2pY#y@4_@xq z5IntG>Qd|8U$J|tbH|FidxnRz95l4FSP?FT4l?!rqFE+j3x6W&+LAVY`vgYu8>OtQ&LBR%=hpPc*2sxvgKnUe@;03c`8p8%iCms<~ON z-W$a>e3AFYEfrQS3PqWH*fu^S{`TLmeAU@66E=CF!DKoR%(hQYPcIvbTy?Dap3k$p zJt8k`Ri)>3Zf%${JHsAR75Fqe{kg+v_w$*`r8yO%>bl^b9rmAdm0BD0>h(=cMS?j5 zLqkGVh=nEvJ*p{OR5oHlO_xO=x8|7CWEnmgpDNCjFbRI@y>}4p_)PaDPJ%$={L(I` zoQ0Eh(EI>+kDvqQn`8Zpy~e22jT`gM|28gg>Jqwo_3DnDJF9POI6%y|=G*X=T_p>u zYf_b`Tvx`4Z44slgX^ctq@|^gW$0~X5J6vRbpG4nyRwm}=@iQ2xyeGi)^pFYuqk#E zTSP^#hG^m)%ya;iGjWc?pYs@10ag!7_BHF9AeRF*Sm%8C@?|9f_Y(FUwGiX{on0)4$=W1bL zC+>{XXU_EE9=}MYYD5UN$$WZ#WjPz~4I6lSh2&Q7bJ=`1FJ3x&GuMo%3bxZ{!UCEmY(A0b#<>eN-y zCc|czcHK86So3(qcyE3E72$x}-`(CeG#nu`N7@Q^O4Pr9uY)zScooXCT2SzQFRx=) z;8OY8ismcH$;ofys*oVx-O?7lZqNB&(SQi`h@Zn8iNn8tFMYO5+gA^;|IGQw;lnW~s}`j#kDz=i{kB1oo*qV51N`LzVWaUm6JOG zmQ3tak$oiEsDrk)gXVn)H8pPpE-mPz`cYf6&(yop`*3HPXgjOdk-NxqZS=%ZG$2}t zMN5ztuZmazL&nkJ)rInGi~n1SPfRORQuZ39ZxJEiqRjH_?v^BKGVSW>&-s>z^0F_K zmn-jb>YRT&_*hF*^FwL;7AzVXZEBqc(tp~ir;^8x9gDZVVe`KqC)pbq8JPi|pb(75LB!BYXrT_nb{=Z(xvzP4r0vG)v+jFoG z>z+Dw{bbk-$ZO~Pt7G@ti~q4Av%E@PJBoa-qhT&CJszMfw z1RIhUK$aRqdEw7$NBnGSSD!$E%rD9yoBo zW#FSIQ3dV^%!Fk;+cL^Nd`G9 zo#u~h-x_~QO!wlJ>YaLA4a;Nh7`}c-RlG0RIx*<3NH)PD2FuPeZ(ccD?w@b(TmqTQ zQ#%^ddh}pP-|ycM`X>H7kBtgn6gjlBc1+)ii|bSLyRYoKcGcSLRT(ET!yWbA-Q8(% zyBtP89+G+v0EI){R*gTeU%w8Da5iOIxlm(cV@P;7O;0a++R55lOy{1$S{P-Afvgz}S%nCk}V*mP0lq_|yaE%2v$3LYX*EnT-mRndj0v&x1YSYZz z?@AXGh+t~Iwr;c`b)1PFR9mbT`7dz9R9@&?{0MA4>20$CeLJ^S?Dnft6*EbyBhPvQ zr2UlljI=+k`SB*IvBa&9aUs4|(hsaGAL#8@3v+zudyp&GkfurMWV?h!*xNi1ovP>~ z8cRLwh$Fu8thwydp4vqI2|_srSmQ zG2$m)pb-SCV~{v~=Bt#Oo7?BlpGPgc)IoObu&m?$gl1B;a*K9Q4)DWde>Jbjf4~xog)hPa=RRmzBP;!STOY6zjSAALf@@RL z(OlnF^bprc*!#F_e!QOea6mVDC!ypWOreYw9iaNqfb5czlOylm4FzZNqN77#>(;Hg zMMdvr9*TvUeVXz_3WJ(pLZY$Uyu1rxVXHa==0<8*O6^*+&o0i-0+tAXtcr|}zuw(# z`tsGQ2j|}1HS|<<{`2e1Jw*>zz$J}i$5w+~6A%>a`S$H1_75}&O9|_-!Jt3M7mlSi z^?-yx>QF+z(*5h#b<|Lc5&uxtER(^-XLJN+%E6H)$}EJIr6Rg$pSww$>DjW#}fe08$38T(g;Mq|&p*>WnEi zq2Z2$B%Wa{`%Tm?F3k1X@GCec5;xFRgoS9@+GY@F*w(waxkVvy)&U)>(Jau`FsdTu z&}bF_UgMpIQ3$OOAHp3@nameI*>9zCWd zLg^#=u)HTv>I;UU_8dpEBO_ssE?T{vQx(DfvZ?7fvQO;DU-Y}ZKvE<@9? z`}e^W>Vj_=ceih9YU%|!a1%89)v&NI;sz+cu$-J6w5s*iZIw%lr{2DO`?9ms7}tNR zR|X8I_|Km|KLS%~tL$;^rn~>@_+l-L@JRtB25KutyXiONCZj@IQNU!h&w_hAR}kr- zm*!{t622h}^%_V3HtFO4$p3V%>4mb{pK)z_W49hx2N)=vczZ)9H3~GRZN1;vt98Av z)|to}9YLw;g;4aZr)LB4_Gzcjp_VMejUfACcf0+pMZOnRP%xm;DrUZ| z;@j=|aM|+Z>(A)@8EQR(Lbd_it}Aj-`q2j$TNqQC1`0e*WEtoZwMcUXy$m;Sh8G?n84UN<1`s6?!K&^7d^p2v9)lIfc3QT@`0mT}D)^?^Dfb zwd-M`&&O%p;3fDfBI8{`;N~%m`X|8Lj;OM*V7`Jkt zsbdXSgXR?gSJko(t@%G*06yJBbSMA zNBQFizTjJNfzTBhRedJ=$=x+9+U(K)Vc&Iu-ANKnU0o{e{x)p|=`9y;u0fL&lMuLQ zcwJaU66Zz6|F1g|Prd9JprHbX9aC<{zkU1m5SfHgb@P$!t`xs&(qbkJX%yMKtIR+7Ax@V2^DBt;DMGcFcAL#Phcb1jtR>=e0 ziX=PZTq#Eda(jIuvjp)2I3eF_%xt9WYk|$0H76v&*J^@LnhZ^P$dL9X!ksF9ede(# zRp#J`>VceA{2MpyvsYPJ5O=KL_LGfN@{y14oQ{RiGd+*%O+!thhhOI)tOad8x3w`6 zNYAELR<~?+v-8Um#4;8Zeh-GF;_FJEdi&f*J0~aI!I5ti6I+3f<`VMAn*lbHN}nll zq@;%nOCTNH!8&oRKHOi{pv4Y_y|*Qo#2niDqS3~ zp!l;PAp&8|NUYs%Kt~eimq-ObfQ~}?9 z<}Uh_xADsBr>Cb^u(7>$H4N^lcP74Pz0k+C6xBmHJpjJ5pjGO0aU$~*4fN&TQ1;b( zB)0SnJ!#l`0A=Zk;HZ4&0ttJ6zP}&W>xVP)8Z4Q#*~=9pa!^*Y&OBR7N;&A3T>v+= zCk)|X!3~r2&RSDp=%PKKk@~wTRdAQLM%)Lb!AAtItsttfyx<_J!CQ|(lQ9J~M@p6~ z?EKL-y9aMOpoOXLP6;kRIU+T}EjssVW&G0H&l+@Tq&#{=-LZoU?Un{U!L2Of8o%5N zoP0VMT#|XjW!JlQ;#lyW|J6b2CL}wJgEMh}%GnvX)bC^)P*anFN&sm3th<^QGEka}n)rgHJSw48T=$;%$v`N24zB>0^Xc)abp#iv z(hnbwZrZd-1m`AI=e~@QG4ktZ-L@Q3+9y#q*);O;U7eZnQUZd+34I0dk_YBXJ8l>I z_nL^n;R>=1MHPk9;5j%Xw}K_y@&%;TF{xFnR*8i&s_EuJN`%nz?nl|6wD#e{{=+_S z1QqPC81y3&*+-ha`W=rkeJ`;&Kx_n8wXrlQX{98sn+@q zBb~BQy*o^*g5(vAitMlM_5Yg)IS=COW~9ShpdEl$qqxe`CHJOWA9tF~7LVO!ld4*? zW=&!47OM5<+zli$cdbAgThAk|_hGu`=}=akW>q~r7`5Zd2_T)|}|1B)*oOQ6jND0y+k#>OHOEmskV zuL5LRsiC29_1ZQ6xMq>0Z?D5eMUpx$=lu9#@z_#46&P*K5KGwRHsz~iI~3fudc6BN z6SAs>(RH5<*~Ske_~lsl?gcYB(br&(lazKMb1i9L=p9GB=IBSp@O{Dt6n*jfi;8;F{A5@|n&i zImXKOl)P8>^z@XpX`DZQ{&7k^7&-+gl8<@=Ay<(Z5E%UfXi_i}qfFmc8WXbAn6oOpbLp9-qeXasRG#~7#9wi{J_pO#kWS{gh)jTgRFN?bzgi(R@gRj%Kn@+uQ<*#5nWvz!+_hOP* zv2zb&r{BcksQZ3tQ#(BDI>|GGB4A&wj@Sd;eYKYi@EhnE(fMqYm6fd@pe?&4WwOiG z@-Kp76f2MjfM#U1cGWsb8o&YRZxd4X;bpdz^HXly7c8!zz>Cw@Y7~J>xUkOHi$PVr zKE7u*k!4KdQubCb4K35snXye-(|K4TGWWsvLHgXz0^*R)sl~-PCqS?&sG)I1h|qcG z+U38?Ca(XDeoQKkdLqhA*+122q-pB$oMYiL7XmnH&DHDIW2Mbs|43W)hh+QW&I6B7 zsjlD=wZArWRR)^)jsOU>~863w9i9V8o`&?H_7DKRlk zbWF?W(NZSHq^Isn?%7NCR055W)<$0k1 zQLMf00s8VrmcRFr1Bn~45~t2Ry9d*w^~905T@p?C$w^5CtE@d#f8e!u1a|Pzk>llS z<`6n!YuB!2R3-gY$=-Wse?)12KYS-Ccpoti-`zP@Cg>b7_%x_b*L2y@&wC()@DO)i zfBoBg%jD(PugdjY^eas3Gd

BOI+6w9lx6S1n|?%q|T1_QrBfA?tl=g3IP~p)R(* zo>O%Ff!=Vq{pk@R#ihy9PlZJIk#ry zd-C@?=?wNr_s4MT929r%-02O?@gdBl_`!hHbmysZACFj%z{z=XNqRvZ%;tB3gZu)y z<9Jskbc({)%V!#O5WLK~M+h0K_xG&+t{X&k3_~-nm!WqMPd9ZAS$FK$EcFpDiz%V}NDZru(N#e1}S-=kjC##-3kUV-rU9YF^ogDhwG)bW0! ztKPHozgr|X3Mvc5ecw!-0(-)d@U_}duL}~UVF^fpub=@MPV4?+Oua*UQtmj zVB#TQoFCNyM81dxi6`*;dx1^ULqz_a>BgEG z4M0p0dB=`gPdo7e`w zZ&R;pYV2x$B}eS_8Jsgr(c;nN77Z01q3PYr&Gb%s@tvQD5>!xfu-(Gy-HWnFLIDsC z8W2twxM5O`hAVQVNOFzReVtT~tPgOQ`Q;qKcDbP00vKXP+`exv3%-9t ziSv=-D(~o&l-@P>w~>yl`)S-S{ok%EWQwqoE4tNopbxc!>?oqQ=}Xr4EdL}q!R|Zt zc3ZXa=EGCF?sfr4JL3Y5&0B9t`gR$7Esu~iadBZ`L2X1dcSl}f;VW27BY)RFMmfiB zT>=x&Kec%4=FKGL97vmIK{_9?th{&sej_q`{C$?Qs_YKI)vVm-2E8Y<7@sH>RKv&+ z2^SMwDi&W3rSMk>$!8nsptf2o`|S-Ioyv`8?UEyxs5dOlc~(?X((zl8T3C2#rQ|QI zXt4|E=GLy+tX&1ut21;s7(<>nH}jxXropmf_|OW>mk_d!+@w#r?w_S-H`{G3C=*AY zoI4D`$P8s?bB)LGrS2QT5+U%svWmXwuTNT^8l$t={~Do6*4_c@@K)yh6rtnv;h^h>{FsPg;w zrq%y+>wRMq!~)diA6Ts{7F~+@%z^~+9SLxVQIM!`M7*>%nws;N9lI~pU8y4vn=EOS z6{}_^j>9>^BP^U?HU7$r>NDL_kB$&>GC-y5Pt1e@nj>M3H62lT-rBkj z3JJZkQUwylVVxY$g0oIe4NUuJliZ_6PSJv*i2mh?f5FTQ>I2B4fq3`S(-p2cx_pi> zB4(z*fsIa1MuSngbLrt~p1jfKLOXpNMh*f^!%ciP(ruw7na*)}8}7 zLNx`6ILXt+bkC8t2j~q+aRKuNwJ%zeB}|A>2vgM%I$*wq^{9T*4N?O@48hm6W@2=7 z$3lJATebxMX+C;*OLnqGglz-g{{poyo6(C?bth4Sj|~F6`VP{o28%-x6nw`@;V~Wj z?EmORE7=8~@AO_yZAqNz+PWhDYQa8}y zy}k~nM`l7yKU=`w!C?$la8#9xqRcFSX@LIr(NSGNRj2zptwR8KS@Q_f{kq(M?^X5n zt2#S7N5{waW9?1N&8ssg%6+4j8Hk;>B8M^>5kNoKj5XE!hnKpp=+)UkrT}cJH=n zaBy%ie8rdzh61YfuQgJAAOgrUwTjic>w&mfPyO4sY)mX{Re&8XV8oI~nF*3zK&O!B zKj(>}{{W&E_|@3hU~qNTOH!wbwiAf}aZ23)KeQu}Jj4;=0I{9HnV;`#>lQ|?y{xTd zJ9kr9J)HNs&)#ufS?e20t6-|45E#_C(<_VGNSXrhrCd0<3?e{DRH_Wr9EZ3#SsSX0 zi_4v#3Cg}jyAI<*3R;^!e*9r*(@~sP%8nTq7MyD_9{N&sO>!1Pw2W8oKxTJ7ORfV* zv!9sm3HTPoH+f@Lh<@A5xZRcr2sr0dTT|f2K#z-CPjV3;?aTgj`Wob0Tx}VbgPma> z;YoHVB$oxm-veER1{M|6z%>l7m=AnRY0JlHUq%#ziW_Ws90t} zC4ozeYo2Ps)qj#(?W);924ae+MNc)aZFCuXyN!5=!}&$$DfnQLH;3(MAE5~bd-@4b z45=;&x8Lm>8p?d<26GC7NZ48lpUravy}hEuq*8fEn!;J9D>Rx~4F_>Vsn`?BCbIVi zs+Hj}LRE|a39k|r1GDPNs=%)5IxW;VQuSLe?^yNzJ;%9oOAA`iMijG8ms|u^+pim~ zJpm-bk7NE4Qh7nEvEP{5+XNM1VgJd-onIP&aCdZ*J_X&gQ!>OXY*z7UYNV&#w!ZfK z_N{~}07_xN!rz96YGM)+hhJY8+y<4*cenrsgjo2)>kz}3w!mr4N500;N+@aUgyOLU zpmn_LOMCke!|5Lbq_>O~B)93Mx0jbvquLlaxaQI31kvUWcNZ5*oiJQUgS(3&QrA*= zwv>(B2ubVbqO2_QpDRkcFZCm6pVj)Zkx`X>7z+pq{dUH6wg)1?VABrFn2<#-Ew?lV zXyfwb%StvgTFaeQ?sR7)YWCJ|@%{7bM8YBlW&W{-JZ*?{5bbqMZ{Zw+iI3d9v#Q*s`bT_PJCdf&Prnev=n&R`L@64m-TnA&6xB7@FHcL}mTd(S* z6gg`GPtItGqr%VQj=4{+*-M<@77c(H>Du>R@Z?5LD1_23-)JJqsnw!cEB^cNvyX@! zyK^0N?kQ(lN*4K9dYaH)Z{D;?4BF#axp7l~zhb{x8BBw9lVk+Br@ILHcEkKlt>#i- z6t%9d76rRa%MUk{VR9CrXBicSGI{p8+UKzHi1!2PFdq#{T}|O>HNlvWp8JoI-l5T+ z>X5Lo<1~vc^|wWp=?vh;5jPrUA1oCu^<~QeTPCYO*z!v$rH)L6G+@Z{QnqaoCVOj z7%tS$k$<^_(-dvPIMgN#DkwFz93Y#hWDQO^uOOzhx~3oiiIC8nYeJJWwY0RdC8MD{ z*Hmg&B)IyFEBmCE_=eSF88wG8nerzh)(P zk?l;zyVXChD$|#kZg8lgpQNI0Zufk!O+-YMG}`Co<{rJ&8bR25`TW}#T{gl;N-F2{ znH^g;Z6a9#q2UXXDrKzQAFI1J;Oinm_W>qfZ6|DEB`kBjW=?5{BA<&0>re$&J-%rGiQKf`%|m)ud^NKbA$tO-9e zc#`)Au_lRWF%ovMqv;teiSjZP%crD-ib8v*C|q1O`eHNTC*II7gu}y8)>tVMKRY!1L+W{tmM{XyE~@a?>1i0%EWM9z2p(ZX;TV8Zs$q)kkpt5&b& ze@<{D>^aY&pMCmG6C;~Zg@B>Xf37p1x*j`dHkk1^Mzdg-aon2x!d#Gm6tU~b&i3~9 z%p+g6ZTPoTj@c;;4_nfYZ_$4mUue^$UOts2eaN7{m_aiKWKq21o(_l-KEqad(KK2E z2Wn{uYo$Je!Ous{2u_sC0K%ESYGrKhhm{P@{lJx^iZb<@x+B+o;1n4OK9?k4cL8c*-RpndhplPAWy(FPVB z`IY zUY5QLS60{bbsIR!{jRS07sF3B$i~971W45Hz4b!)KF9l!jqK<5RVw?65#$&O`hL~x z=Wg$Id-V5{FsGB@!~Jj86BsYK_ni+(B?yU_p^AyHVjUY>N4V7Xr|e|qW7@y!$V#es zKR3cZ^uah$CS+XZ!;l6e@a_wN(MkI^!>jA*8Sz^>0B!K-*w_V3Oom!&F6Wxq5zeeK zU;Sb$#-2O64foTSnWbbUR6<}` zF2AZsg8dcKJ5z{oeeT}llss41AW)^Z`Tw0H z=cqAmOUSVBn&7=Bvrk`*lXqg|dwe#N2w(1eh38+$CuJEj@MehBZKAk{pD8WSj#n}S zi6a6-nV}%H?jyRRtKccVA$uL=7FDBFEFV>|aPR!jOK6eENuDAq<9@|i6^8444w5pOLr}4* zt}ZlYLy`bE!b`aHU^c#_kQE%ld+l??w48_y#G++Th?q)VZ&(OxvMzDX5SG|?2`VOh z@Lr;l&6=Ug_rn1Muhv##EiuEU%8#LI93?%01;W$-Ax>IBBOegK%}8GmIoOP8lX{=B3^A8@@1GJwI5EzspyZA-;ZUdL zT4#&Zbz=15&g|UWwi6lCJiK^jz<2trbMAFm3I3g~}XEXW=R&a(;^$)Fy{7GET6Q@P%lfaDi($X4At5d~n@N*3J zrjz={pRPcCZnZo;ZcCoe$7LzX3;}d5-F^D2s{TOsJ_<9NJaAh4ZrC`||IcB|YLGJhB9- zNqvJL#bGOBCO)0R7%_SC{G5>f^s-Oii4jTadykQF0P71$VnSuX|6)@77`NJ3nn4~F z?lS{6syG)IRuSqhLOh@UU%aaY#f2#7r+V;G0gA7XT=8nhx-;@=4J{BaPNL={*VS7j%1=rIwEBSD9)kWharz_T9b zfTZ@ML6vV$)g3zCVHd&zbc5aJ@szz%***0g9ZEG*;*N6sd;JuUg06rymiFt7 z0&pgu39;1B>3>Dd<#8af^gh4pqpwub3avt7wQpFtU3lIrgQVBi+7!3L;ahFZwo9u= zd0pc#*EPQ;&yT-cHhG0vjhV9V;mV7XrMYv9iw2&O+YS|8s7UDhqH*v3?wu3whlk62 z-`^w0$!jfSDsD)>R+u#l&wCW@m-6soB7VJ>lhI09jgbVi@e@anid6ie?%I=e=v`|V z|3-g*fAE7(-`G1H*Q_8!cI>zzWg4E>H?b$?KozgAaY=Eps(8%Bkin_Z07u8{nXEU} zZ`QT?HQ)1Fd6C$-QD`wFO6oPQd7JaQb>_pf_FAY}>X9 zBGieyyHZk9!vgmxDq?%p=L368-(jB@VYFZ3&Jrv11U16X#VY6Z2%3}c9!$XSZH2~dH}!W z=~DNVe0+THVz50twiQ5`1*U-8ad9L81?K&RR*Km6?aK&WcnsucSc4cLi@w0A!gn(; zF)^XAc0=;Gl$>mS?cYNTFv7>p%1lmNojG&n+MgILQF3hdqf=9l z^N+}Z19^@ipBrFVU||u9(JHWRWJJ;3j|w|sM%Bu(uXQOFoe8*YDf?vtY zlZLgU=lgdiJG&vpSF+d&&~WPTndD*6lavn7+pEL3jATa+S4*Xq;sD230)I)t^R*crQh|952cs$>ws~{#$t;6?P%_t~ zo(LEi9Q=qLtgpme0+@n(F534{)y?q<}l&|xZT?3=E6FkfWHH&;=NAo zMb|nyI%I$Dl$L$~O6(X`vD~~BBq^gRh<_1k68=`aoiv$KskkN>OuM^PA&+veUcDbj z+{oPG+&Oh*q4fNU@o`t95?9eLZEd!1o$Txmk=9~3YHi-;NCsjE%);LObg#m_J?C}I z14>ToASy)>s@J>)1q4u-DiX)X$9-!0QAR}U?Ccc1$1P$!;DoVs_dlM5A4AjD`nsV%*y1M!@`dSGi zT5xbM|HR|-pX;7kw41Yg5202$_~K7alIV|Qk>rJ31& zq}t@vUhwZHa(op`9`TunOTT`jFA6b1gpf6@BCZWUf_sDmWK`xQZE0blchF3{OT_6* zeh?~~HAF58E2~tO?^f2p;g-22(vs|KGA z&#gOhc54|Dv2AMzS6A00zyK5N5DOVTa6McUJSl;K!L=jX>tDRMI6pt11`+&Q0qiA5 z(QcQUrB;*#j1;zJ9Ioi}8Iqt<@9sG->Fwk55vlbcyldt{SoOgV4=$-wKf)b$%>$|K zDArMKbv788DA=JDNcUfKLV}cJ6CO8njdyl-wh^w;+V=K1;s#96UiR<~9zc6zW{4Sk zZZ59G$bJ}G!b3wt2YEII1_m;!!Z+|(VFdiuiJNW^YsM!hH^KB_i^t6zj#nx%9xB1$ z+#3rS+pnc^TRVS+eSJ{q5vc} z$EyIfeH$M%C(Vb9su0anP_6MWfX6EcIj7Eh`T6-TXDb(`Pkn4|PJzt@g_^e9b2YK#$vjj7NJRvypfWUq8?5Tc7#$JWQy=PAo2DG%aQjYma$;jM<;<$CQrfrx1pZmq^c(7115~FnF z@h;iGiFfie8i*lb3V})JWFQEe;d)pP)8XA+M#p0Z^m0B21qUCqu-N3#Q5?y}##YM%uM?5VU`^QN(vB^CwM2{~qZR&#ycwPZ0B|OAHKP%nc!y^SsQ4lUtsQ&O$_0_Y6 zhJ60GQM3F#Za?1|)zAD6TslPbV^kqAV(;G&4gm#K^u0!X{s7vAj$iK*A5hS)hFF7D zTt+v>1h`R!Ux|VfX5VkKDWCp6GwO-z)$1yVi>1~pRV@p+i*-5J}#WUc#+Z7 z)3ZfWeei&PwIedkh!Q*`4>7t+j)ct4%zQu>SMXlm=Hn)5bxcL@zf(F@V7(8mo6)Uj zzJDb7czJK=iKn1sQmKDiD;L!WYe;A(em1;&mx3}C`tI{l-Dd_RKeO^t@glRcC2>Sc zG243I{QJ<*A;KC%ll(R81a8peNa5y3Am<9-`(jGeWv)POA?sI780w{+-!32#W6Mrh z0&OFn?HUwj7IIuYkBLbd|);X9OpmjE;?RGYsrB@RkZ zh`Rdv2bq~+D1RH2l*-x|JVr)FUR6S>V<491Cphtaqc8(iH8xUkE{$%De9qfQItXzP z*!X2v0M-z6nAoog4H75*`~(5bLXeX{t|bC!l6NDaXGm1S{|3D98sj4$uxdW9K79DV zMbD@X8WiJZjgE~q1Azy9!?sX)lcIux^~*HWE`@-3g-xqhR?LqdB0iLIK6>;Bo?=y; zSW=DyA&F5IR>B#I(c=ls0B{n{FdFpt_jkkOqweip(lYK~Z;#Tw6UOBA$ko-*q|c;& zPV;@GC!CD@23VJrmBj{^)^j8`Q)}xL8#ivmC`UD(W&(6@z}eZk{Q7gW8+p(vfu}^I zq&R(id_FWa0iJCnpVxx`{0=*RVeIjesEE7QQ;m=S@F2Z6Z-}GjsITwjO26K!?=>j2 z;*nwv#@UCBPPVyqK z2f{(T>W+3W+(N|C`YqZIj+z3o5p>wO5?_0yr`N1}2G6Up?}(g$f-`|Zi);_yHLF(> zs@fI!TvghM)jJA7jhI0OAZ19HRF3wIC(y~so z5p4W@5!bK3#>u}9FQw{(qe?Bj%J=+AN=r$;gqu%48 znc*i5qnPTk^Z$*d4+#z3ucO1UcX7rX?);X|Q5L;FG9}Nv%k&j^TVKyaj9s~c@Ipj3 zXWp7I^ff?c!oriI<)CXyUf%{*S#gn|X536m$AR7;ElmM!?bm1??lS(0*Sz5-N*n1A zK`GlujiiTkcX!hf87nT*Nn<*0f*cOnfkrQ{rS9w3axFu-Yu+bz=N(Jd0Jla` zaWOA2a)!sLW;}5ra7tf{`=}Uqvu98M{yxt$OR`t~;UPy4{VCeAp#8e$Y_{vy{C{M9 zcRbha9{0~IiHIU2qe93kD>HlVkTOCNDkBZMDIv-VDKjfOWm7guGAqeUMx-)Rp7*!= zJpVk;>vhh3?sM+bkKgxuU7zdod9O?Fco_ChWPCi0w6yd>Q>93cxZr^Uc^)-*M{qPJ zV-tD{T4b9^$*h|2n-gFRR{ZGL*uBWepxIeB=AAo7(c;K>PcuW&bpr=aM&F+Uf`WvE zot(0C?LlcBE%!?t#vk2rwWVv<(evj;Bub%C6?b@)}#|i{1Me zo>iFT7%0gQ9!3WVbfZMiM}GhuL;!$-{q3i602G?%_iY zXn5?fx7nJ(V$-2(-%L+`OOH<#TriHIYPNP$ep{35X9kkCIR0O(?8@Po`LNy0yL$-w ziBYJtdoOo_V*r$iCk5f~(S!w;&}{5Mv>1|2!>%pmby!phH;7qwsQTACT~hMG>t6hF zlJ`p>M?ra^Km$xl_IcCL+N!-LCF?BrP}_X2t3ywxlRqATeX;T9@!Tiu*50c`Kuc1> ztfbn+kp!WBVX*O3fWv&<#MhAHFjHzhF)`t-L)kMSYW3`EQM1OU^?XH&W3<+xoVp#?srK9!N3i_cMUy)U&**$*LaD0uPu4aK&Xr;>_&u@_x4q z;&{BTKG($3Mzf^kPql%yO`W>l9bRbOiLu+at+gNhJy;H6Dg2u5)pd9TK$rX)H=z_V zFfhO%x#Rr!v{kNYxw&4o8!|Nj?FB=~62>l_|AAgST4x_X%P=(lsi2?SZ1%FG zOH23Q++67l{Ciz#W4**cQUKkOy@XM@hA1`-q%sZYIUDZ^?v4dt3TagT{=aA=sTfDl z*fw3GqHTB*6qD^5>f(AWJAYLW(=+-ABFAR^^TwaYeOG`h-X0ybhiyQAP|5o>w5F~n zI#E(iLPUD1ThcEUR$l1cB$!Uwe!Dkhxs;4d`T%HBln6V zN<~{sOENT0cdk<_xgoZL7-!KtM>RdrCVqRFWY4o>VkxQZGx#m;bQhc`=e^rVOr2=8 zlkp1Bk?j7nT)UBhq!>>y+;N|P$`O+yDKb1|{MP+D6nz87NKVI*|P3Pz_P#Ar#X*3MtnT{Bw`@_ zD8zu>^jkUU>4o)qd;)T&#X#vHHVk=W2n6rW@M=6m#D)(9vkgZcK<+xjya5SLa9CJa z!(yr1`}_D%4VW%zFc94(4iA_2#rrsw0;&u?B3Qz3k46|b2}6nZw+}H0|JejkV&@Bje5o~G4rN& zXBwKK2)UlHKuZqnK-XK58vWoe$eZ86FwsC3-dj*yEU{(lR_5lA-2FTzqC^@P6p4z6 z7dZ1t5yp~=%L}P}9v_@uKQNyBb~iLM4Y-0SRPkCsAB1ZGNk>OTsge#+`+$4P_Y44w z@tE`$o2E2B4d)_W8D@ER9Nn%xW%YDT1GGfq41qEs!{9m~u)LojefJzZ$OK030R-4G z3V@x^zYuwLt|v4nCiQm*ETtoWsO;i5wx#*cgbWO(WLg4th1WeioDQc(5Ny05`dzx(09C8Tn~N+Kfk=y&Z`MXP^phcD6C)vIV4y4;h{2@pIKfEOZ#jZi0z z{)U)P{o+L_APRYG?OlC`OwaxpZ@s)a5r0Nv5^obG+#+eLl<>=3=zy`eQ zfk-hRX91>`J@WD#r%s&`lawS9CjkUe8ya@{a8iIsf-0X0bI=Ww>;?ef?l|l5{WyVI z5b1aDmb(ZOVR=LgPE=9EEWB!I*^Cv?j88!@*DY*;SHKUk9_w-b!(^9Sw2t3p^ zk&%&y&tHOws1RLZ(JlvIn1mY(l^C#>Jzf=CCpmgM7Nx%{d$3XY42~B&(rl%pBjk#C zGx(OOaD$;aZfB7rYG4ud<)@OihVAhAwG0eA0PL^%R1jDgD#so`GXSl=1Rat90+DVc zAj5FtXCtexO4%1QC%e-C0dWB@IE*MOSyn3o$A#HY#mR}UIg8PyMaCVQbNa5YoMTu~ zb6C`tyNv%(Di-3e___aD#D9NF3|9YOjsKdSe}7A0-v9lt1~oF2|NGmf*8l$x|2sSX zedNE6>Hqn||NcZ#(|%OI>u4VS`=PViZEbBTIHtsruR>4lJBzs%aZYFbx_=NTk*5FNQ8XBHrtPcNvb>9H!CpLrMdj*wYPSPaS z6N!Y~NYCm!OEi%fkx|2gufP27`wjdgq=S^Cz5M(Za)ufj)TFwBft8zgI)G=r1;`0v z=10g6O3b;WAeEG1JF%t&;r+ou;!T!kkwD^-q?zPb?JQ{vyMP9QAONP2@R}5lJz~Jn zPs%Eq68hiQeA*hN<#Jx&9$Mky#@y8eSudcEykg+X6^$Qp=i>qYdj+#Mjp7mt*4O<= zqBb4o^miXHL-q6yCT-}bi z!>jN5X2s;>*fHO0X=`6SArk2JaS;I|p_1oFXeL7-!q_7!8nNwzmk)j4#0kg$W~`?n z52Vhr5ydL=N(4Z5$j%6%*XT0MMT?KMP0+0k5I$iFl|}& ziv!;w4ti8cwr;e4MTF-LrZa?PvWnYNq8nNtaG?gkk4kbCFVF$@qT6@x-inK3!GV1j zP25Rf4VU3ey_b|kM1_q$-t|PE9a($SAfGHtoWlElh$6w2Xy@>Uf{-7E zlPnx8hqS;sbYmNTmuq1)HO`wq?dWI+y&$WC&mov8-r|Q699R^e{wq`wwhJc+cQ77P z3Xbu@l9KC5Nu1C;-iDBV7>YbPK`$V?tcreRD}y%rm>RnP0iq6`;Rpp40Oj*0L{}j$ zO9P^~@8~U{%hYz2#YFk_;|ozF;-oG9fqN4g>}^C0goE|!`hR;c!qAm;2+f_(2Mj?P zBmo|tl)fvP;cV)puNcq&p}Y*FJW(LzTafiD;DED|G6=v(Hcr~HYDkc7w?9cw2S$AI zBoBs|2(02~`Dv)$#2l-6w;r%rqr^5&1bV9}0aT4yBp}t>w;{gOW}r+!Vi2-jt3R-2 z*%I*4jF7N!I=Za;IvG}z^FT^YIXT_=85|tUL~{A~`~Y%$EaGhiW8T5h+qnAk=g%qF z@pwYiQcKq%UNb@)U2T5>As-&AAW1ydQ17SEZ}L%4Tz6Rxe4BiP?yv}Y(SqlCA!;h< zaI~x2Ik1w*k_qs*?+Vx?9g;I9oY7|h9W5&LCE~*Zur38BvN|)c6kM!3H*xtXlbj)1 zQ&VGRW>zB|x;s_!9b<0J^4Wht{bD2MIkBbzRh$7tqzi03JMekA9yW_X1u>H!tF9a* zns`)UBJdnT^#G7(&?5XqZAV`U5GQAaHUMeJCmkJ$h#-R1Jzu>TYuSgfm5dPhxJQ>j zvGYGIz=aExB&2YX!&MSoQ=>>oVuFH5;K8_oW*~Cw2;`=RFU^JF#e>9>lA*_0O)Lk< zFgzyOb`XORNsCQ^%jEiMv=5RCD!U`RmJ@d}U#kGVlvVv*y&65K4tRWcHwEx=Jo z^F$KZm5hWRe8#%7z5~!E&QbtQtyi2RZK0v33_=$hYBgC} zSjdf@6_Y3cO(bd*J=d{5o(`=10y!bpdG}xl!4%}mTz=oLY8;5q%@z8D#<%S-sF^IN zj{osXzXt3jfLD<#qZ4_0aW~o$M4ct$@OcRCwEEA|M}Y2`;)Wg`MV5OA-o;}LGvQ_u zXksHQ&zNFpC9;9KQYxo8H3Pl}Sor6A{kC*7!Nw`<@ib)d-2PaJr%kR5_YI7{>4Fo; z1qwowhb;h--TSUCwEAUA(;u4Pn`0so%m-3VGcOJ~IXW7FfOLJTpYsN5orZ*z#j3t5 zDEj&&0SIOLuAIlfg5(9tAhh-c3pRM(n@Jd`j?EOKZ1R(yq11=7O@I+54l>DOL|@{t z4HFPT3E+%-cyVd=7_b~(G^kk7q@b!Qc~D;Uh~G_P3PN%(0XQO$nUtKFxd|r+MiUMN z-zCi7{O-_bhkL(Ckfddoa#Kb%$NuIbhGbyT<^DQUEe8Q&C`@&?s~Z%yAW4m+eC*h<(w_ig0UXwU`osyG*vj>gqeQ$V`(*(6 z9)vrl;SY$YD=`ybX^2zo!c3q3pePt?tbHCd+-PI%JD5nwj~I)y4d@cEwXk4819QEk z#4F|p(q9*_ksRuR$OueQab=i7E2A_}-6eOmxTuJX1bSRG+l&`eCGZH~tG>dMr>|U* zB}jcJZMc<`xc-N`mXsXi(|Zuq($XS0X`%XN6-IBu9)OJmWSAin+EsA1#4Ju=W`ty@ z`~A9CHI){cqcZjQW<>G`V?5)#`xMT1-csS-`YfEVsYc+nQ_ zj3yC`>0NNv)L6k}q;DMsH5-3@5pcMTAXVCJDlY6NjM306>p1uzXQbiDHT+ct6}S+1 zJNR>0vXj9!841SVNY}u8K)4x*T$q`|z>CPT5^5Wpo9jP*{GGkewxD{6jpzZu1z5;E z#wIN;DS?<=YZC=$0vN{Wbt*q$iw zk|si6emFF(qUtL4+7hHYQLw>ZV2!BzVbJ2jbcC7X2%@wI_c&gz4`(nQv>k{cwvd^6 zZ8MyMfO<)QEGHk4;Q{i*=n(KD)+q3k(N{NV3DBqrHV@))1mz?y16ssgDYunWIOHyF zMK_>=8$$YuaRlGQpYa2G#?}qOjEqEtcgC4s8h+J5zw3X`8#pd4N*g>EEgc;ue-RcqLZHpObwY$NF@X(pN1P6DdxD%i+9!uX2Bj{M zOiQ%LMKE!a>K=$YaBn7uBOJw;1R2@W6XnGQzlk7d%lo3<-&D`G_G4o~UO3Oz4p?0i z+96OPJaO%K2ZU|{;5s6Hc6bko;_)psRXxd{0679AhfZ4hg^RCpk9mMhx6Csp<1&HvF0kr@cMh|s3^CpJ;j_CcSurlfs`>wNd8@?E#P>HJ{ zK=XLu@WsL%pO_%VLpySDrz`uM@}NR2JwZ~H*bjR2Npop^qsNoNivbY`vAw^`CjAx z;-0<}1)c{>UcFL@j)~#n=idaEWB$*8Y$hG=x>@er!)$seCSeVRG+`1v@k zkz^FgYUxT~hDiwiBvDxv6?YRki~x5CqDmm9&Baw*NDj_0ya%iX7@A6@W}@3CzV_y_ zAPp%u&p*!qAq^1v-h_=u1^a{m&e6W#i;B9AxGg*MG)Ld8;&J1d>aL5w>gjo=Q;z+K zhE@jlmQ?Xiz(H~D)Z+jrQt3Q=pg8~m6N*+()zHvTPY)k&mH8wY(;_h_N{SO80v@f$ zm)E-pT@q;+VPs2!%SZ{mQ#+pfIM|rq44l0rAGzGc*|&!y^!Vzb?IK(XL zCzFgM>AZl@C4S9L^jFBcVOIU8bf0ahq<8M1+taM9%}8HIPkrufKz3qk?;(viC~m=- z5ML9idkwU#9`ki8YSS2(2t%DpD~dReO6ME zyCl0GMS*Bx)Xe5zKD9-rm)rYdDhOJRKoWiVDaH{3!ADMts{CZHr0w+C9|!}wdi6-@ z!RnFTsug@8Q5C`*m>8gmtpdvse0l?l$_k%wlpPv`Kn3R{6ri#`Dxqm_uQc`?3Rgk<&VBgCFq?YdPC}4z| zz{oOyeu44^Z{1o~7X+7QzIU}w#(hYev_#@eAsflrss;pJ53Ux(PO@NA= z@e~}PWdc0=Nahn-n!)EdvV9uIuzKy6r=XN&tYqJ$jNoGo{OH#qeF@y~4?k56!|P}< z_*1C&5jRF3J%}s;czLsn2SN9q0SWocnTU!`~HoXjz&qNY3U5E-`K~<=nt2i)#uM}=3J%_*-TDFP5lm-&>6IB_T=dS*qOMRjF7r00J~rd zQMS%Le@m#=2H6>cLPBsy<4KS!qaV#GdWb>f_z6WM>Nz|usF;iM2@}H!O0JBxhjG7tm5MXw;xbo>vP3&->44b|1vdvFN*#ObOLVT5S&8UD=JZV<8s8)CW!P;QMh*te{vFnQy&J_OgNgt&Ou03T)Nq#>F|HjQ$SLS%mk|dO@uS--xt91{x_v#Z>Qr$CDb5$(j^X&Ey>V1yh9dQHiic9 zuIgCsPY@>(iI%vuU0qYt6s1VioUiFlke71p3j=L3 zrpYE|nQdl*W&;D=9YR5fx4$>46`uF{`s-lMP3I*Lex;2qOGq1Q5%M8ponrJB?3gpU z|3ld6E^&}RFtf4g5S$6teH40l*n(~W^qTtmwMRC8noEiF7Tde=Y6&~V|1?9b{Y)lM z4ogkuJWWr(fs7Q>dCaIk1jNOQ`q*0`^2(d(ea;E@R3=)HF3?o_`S~?x{D@wcEE#%s z1rvFUnO}k4pgyNB`k(D++x;ANP=sK_#zLgGq4hfh*y({ZJ}`yo6*P#Cg1f9gL99>h z(}iLwh%Ta&lB_ewJ{Vz9p%2OE5-zL(p_K*5XyzB3or0w{rTc;Ozz$~$k}@ey)^mI_ z8MmrxJvFB-@ioF@>UiwJBu$)1o@haCVa<9S+D*pz5~f)6c?fZ0aFlp|@beRFtu&xz zc+Y#dl3C2q-C#VUMVSGs@Uo$S5@O;NAV5Uth-ugX41y$m*dO64OvlqF_f=5H)N9`V z*H6r@mMPP`^2cD42-tr3!5?hqU8PL7pi!zbaI!<>sN#g&$L7^nV-`wFfrmt7$J;eVs7> zJT<7I_ZkIpB=Bz>zxezbHz73oiV=)wr0!iHqcj?_vx&HB1$$c%=IntG5m_ zYu*%J5-5AmT8Du!;fXCZ5LzOlc`u@shV#wY*5>%NTE@XgmnqFk^KfeHgg z76cKS<}L9hE$mT@fkAMqZv)EXNirop9c>Z&2dr1O%#jaJB{yUBaso@FfX@;pn1ns` ztgexOr%z&(TtVl+vsp!uagoCP2L6HX!fXT>)_cBtE%IdoCe9jGiaz^~kG&G)x`zthvm8JLwx~h0LAGj+&cH*|O1RLSLXy zl}wmon2H4G9sD9W+~>4V>MAnr@_jimIq5*$0SC4lx17;rn9v+ym}BhXB`HxDxkvv) z7y5pck*^f;P8A1)P2@yLPzC)a{Vf>1H9{G5-zPea!ftO+1nF7!7pvr_E6Zd-bflVG z%hL{-6m}dztt1~317K;S>TnSR0!~h11inkJ^rLOy=iS@xIEG-V8-vycenV1BbaYn1 zr8qJ-szOF(Mor*%*ogX}ly#J($0X-f=KkELm#xVP>FbV=j`hCI-DF1=w2kzj=irnn zNyerqFfP$(yF6K)8p-}V2M`&EmAO%jcn}PVlkXOoM3^e=TwK~tOE~E3VaCCn7a0?C z6Vu@^umK`76G8<6+|2f)+F(o2SyI5&oE>EUNFPfky^5wIi?8qfIXJY_+qALLsMq}4 zwHzqq%SZ#;j&?U|8q*I;lMa$ON~er5Z%)3{SbLQ;_^e}ZjLyk(=L}H2uYqu(yeb-{ z--zTFX}9pU_K9DT+hJrNEM#3jH#s?pjQyKHbXLoxm)Kd42ymwgXSuAE z2>tQhmh)jXlyu*9e-o|pCeouFt>!{mRw6BIh1zX*78@<9Mg{+H(s4hIU;{naCw7a} zXYTj@9qqc{_31cA2gQ8@2cJ=q&?t1LAr>NLZP2Fqr2n~6vcfW6o}8sMzI$3)nwr7D zJZY{Ojp-^B+`VhyJrP5(dDC~vpuc_{jpfmcZvDV*P=kOa>O-%rV*(i?X# zJUvVb;sGgNSXz44od<9M5)7#@1Me{E4cP1M{5W@JNoa@YrEqqhBSGAoxp^^+5%VzI z0EBc=kMFu3UvB9tI$*UkXR=oWMMN|pBN+FTg~Dg=_7E8^#uVONm89YMI~J_ZF0GA_ zj-Gue7o?KqvN%VkMP*bM98>f}vyI2@sf~}aBy|{L@Mz$&h>_&SO0wRAM@KleFGU|C z!4}uJXGbwqE|_g75;}E?pXBWcDj;3xk9F71FexhGB#znwHQsx041t^0a<%7+Yd5yC zx`P@DotlbBobJBu+L{?A@9*7Kyf%HT==+^GfAv=rAqk0hSj_g?PmE-7$YLrS!N*HM=j3jkK#|x#- z{3XSbSbw;`%E-;7Mb`#tnKECLLR2_}+yVjuSr5Y?DKyL9)vYz#Bk85KcMcQaQNhN) zs=JkY1;;E%8_Hf=wg15^lKTtD2k7$;<+zi8T*!@Jnt#bpb2+38sK-qN=JJCh~s!5Mg+mKAnVm{BRS5_UfR5-Fu!K(WG@Lw}EyiGXky{o}Jwqq9#F} zBi{6S^^iTGKX@eRpH(p}IbtNg(X-qmi&)wXdXDWB@(oP~F${5Aj|@UWfl%b6M(AsGEK z_D*U>aPyvR%r-1&T8N_!IA@+3ZwU3<5JMrNrGQk?~Qu~^UZpc{}yOPKwsRNAs79rvMJFh5a3Bhav_*I65ikLkFlEm@Edy0aWl4uwnz5l~R#OG(LfZB3 z3*82Ff2p{+>OAQ;`Kr-G-1mcYH8jvufc;x_n+Qn?pAJDgMpXTCnmW3=jr2B9!lo2# zxM_|+ZVIs~l*JVgh%@SByxXBH^4K#>L6+!IzwZd5=eTv+Q?-=2PMDs`SM3FfTSDT8 zag=EOs>{O~>bJ*|gb4ip}iJ42F3-%q1pV+kj-$_L68u(Y52{Eo0*1k z?;!O2*$Vp$)s>e=XufyDSk)n#LvUNI(hjyD=FY3k@m zPC+^4DUd0Ej>@%8hKwIl&dBx&JfcXg8-gzs)N~lK+j7?lYugW?1Co3eU{LWd{B-;k zh-YNGuCa~zt%rjrR#tt|e(G`-dbCTGRY+Y2OqoVs26Ev)56|2Fav%n$5TMlf~;%b3NWr zxUZ?!0>RR`bM#O1gvB@hy#Lylu}5BE_xaqecbHp#{)miL#+bqH2MBKlFV@d0a>LP=*>qDA> zT+-k66uZTB#ZX{D-n#nwJ0S0xv92Qme1JNWSSjv3IEQ@(O#^{1yb8x=QW-BCsB zV+ySMW7>vLhM=W0t!6pb_ms%7rX<}ZJ@=G(9M3><=;>F0KL1}P|DDiKB3@%ip>lLE zyZ*cL`?BCCAC^BWMMghNZ4+4h#l2a7Qu;g;*N%>kWlUY(Syp$%nszt;;Rp_@4(g+L z&^5qxAVe()qWY@iFI1X`l~Ik9Z4G0br)DhHb2B#qsn^OdX=wcXQt9M~jNdQAmiD&D z$F^^KB1~n2u8{h^?{A-OAyc*opqmbsjjo~#fjV-)b||uZRH_?z{HJG04HROOX4ULC zBUJ=rQo$(z>QyP2vvyy9`-rdTp!kb(4*f%)J%%IF${0B*aH8Oj1j+b>Lc)y=%*HI+ z%!*84ygQ0;uH=d;krJfv`#rt;C1%mvk3NivO&EpOKfi5gZLay76gF7oun9BR@%fuj=iC4pN|EDK zut(;h;U^0tx;pw0HF5|Ypd|^q?GE*LnWWdtMcvcfF;dlkUu~fc{oa08!LoRkJ-TfI z%8YZ{wJo6}h1AYK5O2@VM}iod7QH!?A2$_?udU@c38ZP#)z8M8zU3+W<8WZZK!vx) zww6ZePZb?X3KiAO#Sc=PoSbD9G7ie0hd{7iE05omZo>I=HGtxrTBCfP*nW~5NsK+{ zzTru8T2Yi+bO+j4bd>01eGcfU!Hm#^!HLwn;*}_yZWvAzwAbvHJDB7V*g1|cL`Fy7 zSYMY|dhc%3QKF*BEzdSTy`*@f{HE;Fyt8bJOR+IABT!mF0^wiRv27PI?)tE9 z9kz2AfPj$mvQL@AP-ThOmzWW3M8DLTsP>70$_?%ss|8M=Y@zqxnFui75&Ve;8+8UT zk@Y>&O%9`FUCv!q@qH438)E_*5s7P-XmC7h$%oAw#lyx-<9qGtf-VuMQhrPCPcPcY zpl}1GPJ=$ogm(+Mdy?v!*$t8p zb7N?Q>!BjkosSDN0-~b9ckj{*A9yVuFe`M~;g#$6<)8Vc$9BbUd)v0L&++u6VqQm_ z>QUOI1Ny&EAJMP()^F~DY7zCkGxxI zyhnh2p)-WGcALW<)xAv`@|WbStpuB2%l>p9j&N{yzi>m3&o%mEwwWNV5TLoK(nVKy zOwsz^OGscPm!aSuK6W+;pJSvf$01in>*Vw#e46%XwLe`w|OafzRh(JzQ40mX;PZkP+bE1(qBNn7!b5S@28w-tbkSC-Wr>up3wsW`dGq z*OzKJ+R09iGiD)iYkZT&i4_kL<41H>J{P_y@lE(7Faswhex;Y^NJ870m~Mg@(RV%g z@dFY=*nl;utKV0OQC(9j=;n#x$8DNqBq$h0hV7P5@Y|5SudRp=)Vk6lB7^-~BoVjt zwyVp7S}eiv7}452J#0I+Ysu2d*)C|_+&`osPVsZn;`!@wb=ti4N^IZArmVgOl=;0dw~YV=Ld^j7Wl{EhzU# z`g=F0S?gZ@N&i!C+<0gDdG_tgFqyN3?S?I!kKyZ{4)e|0H8%ui&4+J2OzL)#gsQF3 zzx(`P!0~yP9ltc__N;o<7p$zV*+g`ljdc>y;if2~8>m9?Ah^tSuz2}rsKCUY25}|` z9wA&OV@*UUg=r#<eczAlY&@`QU5fzi9fAJ#yvG=l| zeehqx>qn9uem}o3{~;u?;!=ulzrl^ZE#=cSZtp9uAE}>ja(iypaQqYbuh=glIv#!x zFftFBF*m+|u#LE`3}aVngAgHML~O2^9$yr>9(7G@VXPHHWKiyFdCaeeT>D=C0Tv+m zsq*4*rTXKe?ZboqpAS9%HaIDwXcFJh&;Z6w(bhP6*Sj#{TbZ3-9?m&;`O_6&_JOi5 zniUtO-K}5vZcs!NdHgiLZ))<%VrXzf#X8XSCZ(-}^}Bp|Sy_#xS-F-spM2k!`*KxY zoNxa8IWDonyVbDeaQBm=q4u3`L^c*!_ze76OUefeg2KrIBJo}JbF8qkKIxu{#&r!x z7C}wNu52`bU1b%s3j)}bM(P}4W3{|jXj8K3>b95Oy}1w$Q&O&fI*Hti%U`OOl{ALL z%dDs;U%&Ty_U#?P=bBaI+U^X!3ASP!K|$y1_Uj+R8dqCP-(*1E@4t2X^_O+UjgQ40 z-+t@=qyJdu3L6v>E;lL1zMakb{%!xASFNU}2g-eI|JGJNJzn~-@a<^l8%60!Z^f0% zN3L8vB2RlLma8&V{{H0{oyB{p0z5!F&57m3cF5T%q=J{YPXNA90qE=mhfmw_M(*RO z=JUO3K-FsTGP!M-?fUo2X3YC1BjEcsJxg(Dspib=KsK!sftdI zJj=GWhDfOGg=DDgL)~jA35jees8Lc(vfl{ak=QOH%yZygY@MpSZ$gjp5u&=Sy#KK+!8ka2;C5vn&+jtn#IdDJbm|~wfFSfyVW22wSVbyONl%( z=REh)St90?A$uQ5N%ky6un0&u3*nuiTUMAa zQ?^98@6@Guxak$?nbiTDSl+eHrNs4w`(&lIh>CW)a6LVEy0fe1V;-pEba?~2m|RdI zLQW-DWWI+Ua^=y{$oHc$)`3L1YGplY`|a&H7Sh)&Y<;NDpPzr1Z?E({NzKHp&}=vm zd8Xu-NHg@i<+G5;69Z14AbD_AjW)Q*dpB`0NGJF+Ga`yCXjj>-O6NTz!sfNt8gw&E ztUY^$U<8>m<DXc~itQt2|hQk0J5@f&#P3eNkgNb&}{M-%rp zWgh)tC&NK6QBtgJ*vYN1=itV9>O16zlXfN<1REr7SY-bGEt|A6&pgy17qNhG-#?t} zmv8y8d5h)B9&a!%AWi_!XCV*HrVOcKQ}^HvOKQ z{$i8ZBqgHoJC%4-)3@;NyRp|c)yE}lCLnZVRVlhH36Xmt;+mUbtWg)gzufmi6gY*T)m@2)J5^6wZ-YTSx5GUvuV65Gp0a*xWVMz zZ#wcK@01p+&a%o~ny)|i!uD*R{IHZ=?jcuWnk!^4Mk{!wShHpVUbgF=>B+h`jhTJ! zCr_PR9S8Rdwq`f#?=)rkKD|gb)jHY=Q|tG>K-ws_hcf{$^i}L}^!*o|k8LJRh?aJ=T;-Tl30JEV@~)i_R;ai%(l_+N-E}_k%;)FcAum|G*Y`>+;lsZD9@H>P6eNyFBcB)bJ&@>w|37y%G3mzJGrdQd?%K-#Uva#Z}&FnV~)S_b#jU_uN8(l z4n@kK@o2MkKY!h4Zd&g8U*wnfvP++($baA`8T~})Z==X?&f8-~83!4ht@W8iNz3}h zsNJYN%e2U0d(h$ZL_hZAI}uzBCw^I3>Ei5=ZFa?OTvlx+7{iuz-E}h=Ps0wacq!wg zfglOn{9;St3)626QdhT?s$A=u(Y5*B=Jwp^+Tq=jdy7-IXm5IM*1bGOF=)NAsq&?r zd+h@ZrW$D@Zx~f|tFJ~(2DYZ2uPJzUghSDd`(kC@yKiwe1r`*X-;gX1JmsFoe|{i||fbs1^qpJxm=@LKn;aIL@#Jt5*$GSrCs=rN@F?_dQeEJae>(^Fh^=vb){D_DMFy@Qs zYO$A9m?zba?WEuOJ#KR>Mag>=nqtv#HJ#gnDJ^9jFx<{H0~JEKd!kpBsmJd8`MG4} zi~N&#{9lJ$xzAVqIjJp}vvZ+;Bu#$3!AtFsTRn@de|^J(!Y23NuD3=J{@L!dZOm`LykSZ%Uq7jLll1H&ifDU_PhTpOi$&S>vkKtu-o6o z&<)()T+p#;{`JI{#ERINrr&cN zU}3>)N&kAMuJ&AWOQaT!@xBHYc}UK2eG7fb`><4jEJ>H9yKEkR0#>p%J}LR*U07FM z0Qs0%{4I`YbL5kMYiZro^9;+cA3yd!Grz4?$N_csNZDFUfn`WE!_ilL*ETto#N%?z zhXxh}qlXzsdUAynw&XXyJ@Ti>DlcYcQ-)REFk8F9W^{lI>d`(7THzY1bb(4CcCxr& zZZ2_7XtZRzikYAxsXoZ?p~0jmRY;t9?#Z5(liQPAFSr&Y7FoWIae0jXCA#Q%EwP(; zS}hoHw=qXrkx5kuUX8|B_yOsM(GOoe{&;cwCB+)ERtuVnjRq~R9jR?AJ057Kyu4gZ zH}q?&SC!W*-0FqP-B0J5opR-0Se!Y;w!=tOk&`E1Z_7DU|NN_;ocIqO{9Q5U=48I} z=Hq9)^gU1Ik5)02ef~b9!#( zO!9FZCKjT!|J|SejP}cWH%F&aF~8~Gf4fM}Dce;3K*q%6otCyfaRxpAHlS~BA^{3$ ze4O>uvevrkuX4-3^VJcn!(&wP%fHg(e~#Dhn|f!#+cjyw+S$G_mT4QHn%iqH5g$4m z+&H?HufW38?91ifv@W`H#9f9}>M}`GabvH3;vqrB)pt`yw&m$wuwAH_$x5?4a_&i1 zq`tn}{it%a>!735MmeXbq3F{q=4{d~bq5UH4w1W1{QpGcMG>Dq{>4{X(raSRYztcBfYSh$)y1ayyf+Vz%FUi{Z2L{nYN>Q#B4f zm!HoMs-DY_89b5uTSoS5ACvsAfs)q&KTbLgDv{ZKE~XQyI@2yyadFC0pUXGrR``P?N$vC&y`4F2~mXVRW498s5Q7Zmk;n)+;tmsarlsXS2 zD_+{VtL3&FHRDdJ+@9ZERmYevkz4MnFv1WknNL=Fli13}H+E=I=^Q<%_|4uYTdKlw zYMcD}ikIcjrw=+}xAx!>}>ob?;JncWp!EPLn+wM=)UPV?CwPKq5KP9TM`$%fIfZ#cHTo5(T~ zthhMW;$^6GMDR|DT}?6E>DPLimIc2q(#bT$3j=Mfq|U==oU)gO{P>G+ z_J1`uEJiusw6*a~H|(e9l5JWR3OM=u%_jtX-%s*i^mx7{f4Kbq-u^#sr_&q;id;n_ zI{$79XP30YAvGxLC$KovTmHpvX|^l8zS&?q^96^{tzSL|KBX>B4Y|U|zFw$U=d!5~+Gnez&9oy`Cui}}kBK8j!=D{~1gwo5|5cG8muGb-Q-5aX+~o2% z%TA&fNHeSsYIVH0yd6cWe&&2tvmBS74z>u}-+*UsU17$1f;~3UeFvKpUeWJ9tfHmW z!E5o*er)vEi<3<6Wo73?`a-l@XO-MN`)Km6VW&Sn$@0q(6b2R;p{O{7bq|Y^(LsU8;bCr< zFOOfoe)EQ>*?I4tJyEKfcL`XC{;xN4)cYGSrt?LW5Bx5~W2e6Uh?vyYhZG;YAVp2h z7diO(RkgO4penwddbV{skbT*(A}+w+*Lv~Ga(KT&fTr$U{!vP#{;Np>f-Ebtf0UdjF0XVTr?H`P!CF8r7|zwl*{ zWtY-9vwKnUSP_|*I#V;gJr48`QCj~+Ymk}i^AzY^L9MclO>4zXmJ4|`zQ42{`MB-0 z%8MQ2H}E(#O$m5*e0-|pgqBu=qlAOlR}UGS;qO_!`rU2f4<_=3v)h=8{r*)EpZUml z^;TdOUldKLmA{eazp9iv7=^24U27+G^UO{8c^~3{|5Y*Z#MLGM9S(7J;*r>krxv{f0U&M2C%4&6OZFpj;)nQww-SqfE&(^id zb>FmG_qK8l*0!vdmz$5iX-@0+HI@^8*3)~2T{27!r(LPlr{%a$n_=(1Q$J!U(sBHD zuBrBBHOcYt+>a--zg{@}ZqzW^cOs{ad%r9dR@->6s`~rF1w)PSL_H_-=NySQvJ{7? zcZ)u7?k$$pTsm9+eD(m(y+<$T##B^Bk5+ChR+Y=)p7{%gpA{XMZ4azOemJQ<9U$g0 zt?E~6pW*v8Ymk+cK+?+T!cw?8_=h6t^NbE}~u*>(HE48X5 z--N2+amodV(JhylQzxs3Lc33=hsC{jaY+6U5S(?Y^e>Osy#F#3$ArG&dM}b?;3x zxRl;f_?`d9KBTsm%hqpgH(TD50m0y*A?jP(I2+JB33RPhRntBFv*`Ou#Wkhh*LtDs zl-O6y7x4{Ae?x=VBrmT~fA;E~iA}`5dZ72j)Ao|~+(_G-HyUYOSwV6cS7<%FJv-yF z?jFXsk9z-lfhn`#;~OjCaz{s}oxabPd=C5bB}5J-u!%cXDui3=OzEKQXfvX@LRJF zIjZQW*(BZSdRmubbLvdad{wBcxTl=KrWDBZZemE-q*Kzdb4YdTRbChNI*EnS_a%6V54wL=KAJCsHZjH<2H<&2? z+;vF@Lt4xt*>_4KiaNLF-*M(FOy07+^ZLSX*}2|pKHhss zmsgcaJ~cl=74^J4?}Rh%48yT`iDr(7+?`(q7_jXsHMUOVGpZKIe8v;>YL2`*`=La1 zW8jyi;L*$0Qc*y>EeKyUxbGsVVLHmbJNqWiF*{lF6Qb zm2!J^oXn-T{MV?An+=P;^q+d>+C6=1)44ZZhVC;#_r6}Rg7)p}Kn-2&;0Y%MiNuhx zW(kLBH85-k7S`nDFIZIm*mol@A6W>pT62=$K_BP!^HQ~^?mimL4KfYe?z7jZedTaGh8~V8qmMK)Se-B@HdP1k; z#v6MTACos2?^>jN|_SP?fwX|pbrEhUf$7~Q}TVo#Tvdws!?xu8~;PrK!eXEl8m zdUAqQ3f65XXy`~CKg`UYxLhUySQr3$GoL%Fy+hK+#a8><1dilSW4g%sKL>%BGRi^o zN!%T{fK2uX2nfI&?1oVpX~|d@a$NB5&i3;9}1;!eR0Qb|VH9;@=?Yat+I{#S@Yv7u`NFihW=2LWjCnqxd!}L!p zQzaOQl$0$kiOVfnT+CNXPEZ$%Kc{Bvwe1LCjtqF*mdS{T1b)_!H2<2|!r^sxye<~r zcE{Wv0o^7b({QuXKr#0Oifrg_)<*nw&NHSJvt*vwM)~?;yv$7fZN(Ik{p0Z*sbRLL zibFk0Di;0MBtdycNp!75En0dyqL_Z*8V**{y}yFC-T{UxqhS~lNY4lS^pQ`slax0s zK^Yeq6x7AJX#rh3^!95n=W<^^270&RpJO0M1Ez$MV7eKMu5?Fo-)?|OkHfxj3Ou>D zH5Qalr&a1|v(+vx$f`S4Xlh2b6DBpV$Xu^c%HD}hp4vyUT*s7=J-MgB6%D3vFO!7j zVPYRloVK=nZkjxaLrs${Aw)t90nYUZe1!G`0Qv^YlznM57h|6qH#JXCO>j&J_lxuvKWZuUj)mmms=hDy$wH727GF6-?{~5+0MxZ(eD1?#{~*cVm@bf ziirxAetyt*6;4oWwZpS7H+^S$@Ym9}VTj-#G#o+q*QvAh1$BxW9GSUOcL%vRS*w~rKnQuf zYagRlK@|ozS)V|x9aLwBH_i4|G5_}8ZR4l7uW~lT!hrmERj;M#Xl=Kt4%I%gqUPk} z%9<{HSm)|WjExRU@-`+WQ*rLFQR}S5-_!O7{D%X6HWtjtgIDoz|{{|uqoyrsWCT!5La!o*FAY;uHa&6Zk{6E?>81gIN% zR2x{3<@w&LM=FsTv)lP2fd1U45GaK+{m0-c(n|m)Igyg=z}7E=tF-+5vET;41yin& zO!RxeoUr8cDUFufX_fBNrR4nwy|2p`a+G~-;R?!@Qo;nfTE8dWVfZ5kHgQ8Uj!4t* zqHyRUv{_LEtxt#@o(KSL_3>Ru^Xvl_mWbY^T9*!E6dogXt8rQwR`w4HN(1mbsw+G! z45bj;LaKyt?$5#>s0)n{AvwcFX9smLPx90?Y&_gaub}hCMi&~)oNV7dWsl_+ZM zInFXnKdZJpF6I@ZCHFoIxkDT}urs>$$n7lbLOP$>rQ15-tv2N&nY!5T ztH1NdEz$Nmx(XV#!|U4P-32gE6iLg%R7%fsuP-6K$xW)ZN1(yw7tvX-f8HZR&TH?d z*RHU8q89=ah|-cU_is#jZ??>E@+S&Fa34;eVyz4%PZ2!T?=G2JRXPfApLRZcgys{x zco4&7?{p6Kbwn@-`vOL9qF^T#5?Xf zi>f?jLt#5RJu7xx9O!f%yI-z1s`c-Bg-c&9;aQT_)2NBBNER&a{O;n(F8&r<8O*;Y zfE$CRZ43kq^ne=86frtWX@{(FTZj;Lsr4D ziK4^12aZCSUpH#fekwRP9M+rbm5e!j?AhQfkX5=r$3|+;0g9wIi$6cfZrK09O*hrbH2ZLrvt8HDNr;zyqWuOBrSf zoh~nIH^&q}UlkZHe{nJ&Qc&5&kmTHiG>Cbb35F>ZlShkj+pqgdr60n50~oRE)uNrQ zuCQf&9VA@f&z(8w z5QUUnNN%0t4a>9lVgAtbEl(F#f1~C2Z!!Uls*vHYa`b9=1KGn77%FDN*#y9a2_F4T z4XXcjQMYY0j7iT&H+n|-huZG8@cuu;+S6+$Y2-ri|t*xu{f$mF#-V{UBZ0UX$M z_S?DyY{R|IKM`Fq`&5+C{9%vqw;xrw+`gT^`mzpDcd_AOM#UKm#ZgpSFlm3T0~aWA zp20X|Q76PUc!3vgA;dJ5o1H9X>!o|F{~7g7a_(t4ZzEjY`X*v4r-_8I4Vy73LF~qE zKVwQz&5_y)dg0>FR%?wNqF0jMI7^74{@RnaNo{`en&{rL=GZ@=Pq z3+M5kC}_b3shqz>f+`$Vp+07a+n4rSFE4pp(qt75n?LCDoJN5d&_>T+*cb za=iMbfSox1LW`M@E8&Xc;KlO+)t!V|| zr^E-P<`c0;rs`Z~T?sWiUHHytr)th6);Aj)pUcJAou$agbaya%p z734G~1=#Nr183HPA0;Kd3RCIhlxVU#6+BQp4X&S}Powp?@f4e{wW)Z0n;WY|p5T*= z%>?Yj>>AfuzCeJBrqX@-$?FFR{N?kP*G^NKb8aI+9l%KU!0+I-)mf)g!&Dv|FzEY1 z4IM~ozX1lJtnz;qVtxx&!-pp&F70NL!PbY4r>CRS9bH{Z6-jjxIm4VLzY$vUGWsK_ zy@i(DAx}TNn5Z!AhCuL2vk5Z(6I)8Bvzwrk`8n%$O2x>c&b`=dchxCiUi0e6X-Q?q zf)_UCmYFZWzzwU`@7x~_Xn6lD_PH;pm@|w``M03yDX|M{Bs|osLk|Hm3eT50zvgzI zLahLece=~ONNe5ImT%HG5SlDbyIOWcM~Gn@f0`Yg-kbk39c{5qe7p}W@tie36ygxK zkn$>Pg!AoW+Eeoi)I)7DE@Pb05!)@9*NkY5p@m_*jG@xj|jy4__ zcke3eL#?vG|MJrZ*o5t`4BL>ra?5Zon|)~P(J_lecPzIp&emJrDq@>u_Bze-l<^y% zx-^^L9yRxlnmRc+Ji$gsgOYBl7c4qqPiTHe5<^#;vs+@D7QCx{`SRXoI~sE2HtXUV zpCgv1r3#Ll^h=1W1iK${P@~`^M1qD@tNgJ&^dkJW2C^S6J4dhcc6ap#BTY5r zdMkwi_wcEBuDwuIw*EAkQ6`96l{!1O&HY8A*VOC}rRLw5Q!}ev{RDNv>*@N6xUkm2 zgbg}=$V7YyCQerU{c@Fd+~>BrnXjdGK1OZ-O=daAA*Xr5dR4qHM=kp7yU~wMPvY`^ zZOOXbON5#B0OU%KfD!~h#I(=3K+w^DCBY``GxIor- z=2+=l(Hr4dX**nHFCH|D8gra?_*Q+I3)1!Fe$hmAf!(L-(8i1l6}0a53_Wo3&Zq+K!Qo=1;eP++{g< zbXg?_ac5(i2x@rOOH)m6^G>sh`T2F3I%tC&))O2+{+aii7>MDIl->+AfM9PP$Cb%y>8c+rBthqy?YqhU$n18!#Q2pLt20yR|5`+i7@(DIHell{*5_?@&* z_ubqEm$g}E52KQDc8WeR%}4DaAH`KoZ@$>xVnciTu*6cmd7ZXJ*(QZy3S55@G8?cP zRae@eqmPAF?$P?3u9Ifj@omt(U)|P=XD_>o0gH3cc9Q`0*LEP{yuJc8b>vLP%mtiEp?WSqUG0+I+zC-`5)5m3b@n)&J zpwcNjZ$uET;Y7;Bnu{}aUW&W!4CxM-O10nE5?-n&lc^V5^F8#r^X(^E%-gp?aqOPQ z%PwuP+L5!%%T67Fa57C!e!BJK*Pu{7(K8d1+!V$?0tK6|L0sgEUeez*BU2OTYK z%cb+{*MyR(9GXhs`39W-qZkI>+>J+PkLj*C+!BNbA)!w!Y>iuRy**___5(UQ#d<9Y zS4~Zhd4slu5RiHmf139A!ZrPQSk`ZC)_T^zgdp14rF!OZ#!E`Jdl;;=v>d>6oXl)i z*=uIxET43gNcH|sfeun4F(f$HhBu+7$X%7@(SQ5T2meI7(Mu*>_mV0uuxsaHqn#Ju z3oM(!Wh3cu( zjrrX&QYGcRKfSy=W|eJv+zh1QhvqgGSPX2(HtlT*FH%PeZ7=EF=?2KXe?AFN*#k|Y z<=#Qep28D=>0<^Nl(-lbXkC)@Qy`N zKM0th9;?=PT4;053Nn*!VAYxWSv@{!*++n*u^^ksQmoAXRd;(M;pJ32)Rd-J{Z(&AUtjO?L- zM4i`8Pgz+Lb2pgpOPsz;VHPi&uiJ)h?{)rJt;XxvsHpbe9O0FEzY$xc94aZ!1aL$- zG4!VCck6JI$`|cN%h9_};j#SpdIbNv^B|tY1)E3Zh5B7^onm6n=sKP+Yv-qg#LxN!>{P`&8FRrEv6-=owe5@ zST3(#)Sax6xn5e{cC3vOCn6TqB^Nrwem|Oc*;*(3@hla58MVrP`%IZJ(9^eNG5!g3 z2Q8X7U%$*PuLl%rZbxxJ^!S}_8XZAd__R!?Ks?FHZJd7ATn5_M;x7eapRP$%MO_5t z5|bVdSxLY-0O!#dlqolxZk<=?oeh~*Q+!POCMCAOJSow7eIU#dKO-Ev@TeY&mGP>F zpc8_(@_>sg&hc7J;QyBy97RrA2{uyb6wtKT)0&u+O<9}p9R8QApUvqv~ z5SkQnym1^^R!daH`1bwml&Ek$q=03#^vlKu`N(zA%?KISo(tvYF_)VH$1~}p_@KfyFd$BT z|Gt%8DkCm+edE+7BcXb`U0?G~octXrR!6 zsP3JHLrw|W2G9Q(Ai{yoneT-vOASqWc>R^=K``N)GL9bqnAf}b*eUbTZft(z`!v}}^~lo(bWCfF&R4ERej z6lQC29v>a`gPQo&tHA)l$0ADuhWPYU>K#<|XWRxC%JpKaP= z03M_Xq znKXDQpxstAfxu=Q$1~lkJivzW&Y( znsOyTmJO;u_13BCi>=2(1g{R_F{_^1EpNHNpjXx3(Rbwre@2tsTCNhvc!}^|Ot99D zmFwB1-`e=9>9o6ByH_MYO^uQ>>?o!5{b~9mOyr3@8pn6-`;4W5%07R_8-#xF8{a;% zJL{Ph0};5Q5sXFRLvbnp{~4II^*Yp4WQey2)ZlDjW2Dgs1B18v@e= zL@}0^RO-KfcP(VK>k~K#!@(9edfDYse|NB2S>ft@w8$@QC6h5P$h-ahD z-@ux=gNW?i zZ2-9}4ScnP4>2FWPn~WT-VRBu3(%Rf>=NrjeVC#(bqgJb3Y?O0TpXvIo4=ebA8AT# z9fKncov?5s#CCyTzGK#+2Bn6nTfuMrMH^@p|5#V%`xv4Clq${fVt0Q(uT&r&uIaY! ziNCO2LADBNw`4fZK!NNF!lqxxlsdh2Xd7crp~$QrK2yBoe?osWBTteBSTpcnW%6YN z7Ofm=rT;4!{FGuik^YoWc8l+a2R93~I7c~(5GH*rk7?I#eL0#1EB)6Vp|h~hSG-1t z!vb3^ZLNb6>NZ{Lo2jhKVSG>c-E3YgYfLP!{NWoBsq z()wy`MdA0uOqAuvhgjDjfK)~5!9xUj^9HsobekZU%i1471}MmGN+VDDl$rCoeV>TaMCzK8P748YdnB5&z?_3SwLpn_UQHW5MIPWSg9&ty2OC z>P=mKm_)sSJ^~UBgP4E6`x;?MoT~k;ba-5GRJT}-@fq%G7>m9CvPKRVaKf1%++5UV zIWCwsG?}Uk zyqC&QY&*=!Z)c*tcBm~Sl_s5I1{mDK8Q)3u2VRVc47fN$H~I{?p(~e*sdoDG@neZ? zGd-YDuPR>h0@PJ7vFo(-P+>4z!b<;TlJyiC-*UwD?(nzm-Q$AiDk|RrU@b@cDvghi zWFeVKu%2x;bwm{u;5?3Wl?L%*_GBCnsXX)9lG9SRw$#h7VIVOf>~Tv%>cR#c&8av8 z;4*(-A@C287^Y9>Rr#gDodJs;c-&&;R9WZN9L_59Kg?%`o82dtsrl#SP6s4%GVjpA z$oOS=2e-P?_v`SC|AYi8`v2VTJ)oia0xap#)>I$}d1ROGnw-z8m5ygmSlP`l<$;v& z`)UADW!qu3`5to+eK^4_RJnmtFI=@gQeF*qn2GA_VaYMnzDGu+;(>A_kt%mtQ%|;Y7oF} zi*7G^_Zx`ky*7_%kwupOCr>Th^uRqf5}5=+zQM#}@X`WfPDoEahq zAD(}4M7P(bq48owVc1b?^y)XzUCyt?i!d%+iTk*6=m;b36nA~~>cqwW`^2H9rMVSST|Yab~)ZNdoSFXi#O`bEEb+ znmYTuH$>w9rmi$_RWAMvm%fB##&jl4T;i6}_`>?ZD&V>SX^|~8r+}OAca*%_-P@Bx z*}x;9X2N+?N!_w&islGE1!+P2=1ugucV#87S^cH$%S7wN`yj8NqB0-3E{|tT;8!_s zxI^M8g%O^pM&Of%1=X(%u2V-=e z?(U>FhISZW9PZfa-#;HFvQkgquNGEG)Nl6p09t)aVO}lI5!03D&S>~bROK$`?_%7j zQDP0%R7@}vfMDQ{kj#NhIEO$CI3@0PSSrJZkO7=}v<9yDBL;|OaN_L-Dh zhtt!EFFsI%x%)(_(3^@muUWt&Dj)%}vN*uM6R5{!*Ovvyy0aHJ!~#b4-mV>5pw|)= z#rxklPXnm30CWRKETl>WHaAkGV(+9}U`D}jV03%NPVPMjz>j<>E0d0=U2D4@m%ZEH3onuz;NtZ651xNBB%Sgyo_6iz5s-R)Ki|`` z#PX!O{2IZosbzzs_2)q&sdf95$z)ALwavr7^?GRVu>e8-c$sP&(5jn-5g$eWzrFvO z#Um8~mY>UDycLzuuV2FPzUf9JtL8gelyjp--=MnvuXv_iW))U*3nU=R`(A&SVzHq4 z#mRb~Qm%u;jhpDW_lczx6%jl}CnZLQ{Ge@ygU|mKPUp3H9RIyCScjl!7601r2OJ#O z0)Q$E5KeMIYp)*-2vE+x$o#qq^ezj+5_%Hq zAQt_2g>kn>f}6^Tq?J`(yr7NGPke^4lH&*Ud2d5nzjeZmL<}R8XGg34jXw%hJRE^9 zxL97io>Ph67%$F{D&}49A7j2aoA_8&HBvN?_8$)RCmXAf><93Q;u4xwziLt#1g=Tzb#8zI#q`z$BrHLj%>+6w`GR3i0R6Wi6O3xHgBFv&4 z>O^pbmApuzpKJ7PS1*13@XzYsexhTq%bWv~+sdBc99>U*y%*Q=Rm;?QvW4OJ4Nd-X zvE{E@>|8vj88M}fmxNvFmDo@@{H^-P;)6q`vT{cOd2X8I2IIT(Pq@7$rr7#((>=3L1ui#r2hcWoRa3Jnnh& zXhRj*7(6`rL;eC5E5!5)ae=|cQOR!8iHojMG@>W_M3q!$xW`)}Ezx0OO6|qvUAyA- z0TM1wYT%xP+Mt3kenSA96+sKivoKFK;UwP|c4$fylN)jLOgubyqVhG`W2~YquI32d zyo;V68Rm*e{2CvxflaZC)f0b?4>%z)=WjG8J2R%;cb(`j@9)wKP&5#{&LhUAuOkl1 zmSRABLiqvpQl;x<^$^OZu4oUCrpyeYG_TH|hkcL#BCWjqLoKA|O|`{cFVW}Tc)TA$ zQ0PSIVTFQGZzy^1UVNNg!)4zK+`v{G#NVQ|W*zngcT#t2XYRWh$<$5xdFy?dpGKL& zpFytA@1JTBthc7yTAORFv}=NbL|?ytGG6+*IZN-nzMA6m;Tg-DLi^U>gM%Q#+;<$r z(}YXQfsKqYH0mv)$%+~7V1C~YijKgmq?j1*M9OCN_D_$#o^3bz49h-duAvlm{BnsA zqC1hHIxUzwVmj{Q*4#u*7-)r3S(1(XR^ouKdS7?^{@nuNkL-~LGSXQC-lNkM+sSpc zWQpD{V78nrye*{3TiI7u$oEO1Hrv8-peAAnWybqU0>wEBHu{gwY*Nq079!8xyhpEI z(1b}XHGm_1`%3@KPf2 z2=IdZk)wPk z^Xqn0;0i>-G-uny6@5IXj2A2>>X9dP8^p#7Y+V_i#(z6DhS+)PPLVs#PK?cmk2$SSUUKB9fNQ3mc>4Yt{ST$%%?ID6b2R3hVa^aP{u$Vuqc0iy)!*5G1`+fSmPPQiE&FHTQ0@%2?usM76~n>y zU~N;=`ij4Dp_Nf^o9&ZImwi!hZz4r@M**?GX8bE^+U(oU^($}`ch0nqFtxU%kby>f4G5>d6LjvL%sDQ zqpfx4z2{EvB^mEpy)6A~1k`_g!m|C7wE~XEo4)necOzx&QQRSG*h@5--(52I4+pWF zGY9cr?*2nw@|W02y~k@~C7GW=%Z$u{v9USCLt9y2e+cQNkWz3xR{Vj!E{J3Jg88EF1in`dt5o6v3=%r&}iF2#IX$cnis9pzz<3Ck%2*KUd65_$$ydyN3 zhAhlSteU(bSebX9%Ll9Hd(qW6;TWr{!FX-5YKnAYDSCy}Eqk)UcDX2puH zH^B!FOSr82z0xnj7F`Y+Ey+1}aN;$7ErC%=w{E+5`9*~pyqU(l8*KXi?F6|EvvH~! z1y(fiC}R|FS8s27^zA{*69WV+>%KY{CuG9s>E5EjcX?HrxFSwO0@b3m4`ddv`zi{6@|+jlP_QiS+w1t#QdfP6>01zd+qjgWxVUQ zGDC=1MbGBPtt5z0I-(Nvxf!U|RQ!^H`EBw2yLSofjK2XACoeBA4OV#^?iI+bf{iXl zD2^DhsG4;4c=inPRMNozgDJ4t7biS9nVLnTG8BSB0 zOUd>6?D7g<>zjYky{anSa9L#r)MunL_YD2K7B+bjQ<}v-eC6WWC77yq^cl(1kkgaA zn?Axa^}FS--wjmZ!;KODyQij^K{)-dS|d)L#K-ape0$cwfPh>YG042&HX34tctIlR zX0Yq~JvF7-HNX2A(9&Ks1GMs~m`0uNI5voTh@U$T@i149y0I3wd zvdH6M-jrBXef^p~Ew>EPGZ>NK?6d{u7h~1u09CnZS^NYkTC>8IYuEN@Rr&CkzU{P=s1vX z?d`3%!#QH9J~+wgfkfY=7}MVA^BA5l@qCXtk=a7?Sg5=OO>`89HC_;w8TbGCo0+a9 zU_PP(UlDXcl8E2ncn`w{etrgEz0Ym4rV^vIL?!I}1FlOz<5cUOJ@`Gde;S8)QE}l5ls>__h2|Nf-yyA4s5JGn3%|cSeeDf zCRZUz7|JXO!m{p10(|8fZ2ex*?AfMflIv4{Rdfh_htV!;e^nlIv8zHZ?VxMSCB&1C z&-qvgi0I)k9p~1lw!RIiI?rJyK(_V@CSV}Xlk8}7(hHQc9r3>b|0*?_MoSQ=Zq7EN z;I=FcA9D*7ToJI~-7q0b@vsrnsIx^Qx~}p0xk3yT=0gYvL_EN|&0#a}CSK6x9uB1- zGv@hRD;Co08}db3L0SoMD?u=m^FzDd^`};C^^hT?#6f?<4>n(-kP;U0Gb`&dJU+gL zvOyZrkt@SR9Q11CZUfb`uww_x5DYXI&A($$z9i<~Jvg*Z&da;aA}?=XXecVHjCSVe zSol*@R{s;!5I|hVJ+CNoywumIzXvX-&hQS=5OT((dTVKx#KI<_a+2I|``}ufkzZVd z%pOghgUu0!4>m~vydL-FtQzVG@cNovSb)Ugp$^1I$y(cgZRUL%)IzX@eyO6Ws+!25 zhn}I3bfsFdoAep{jIk;Bkx5Q%u(Rq5|Eb{th}N9tK40lL>(zBTLW*`}Y8s!Pf4I=9 zbKE@EFsk6+vuTf;BVHi-3ydis@`rV0djq5tSl40E-uCcg`1Qpu%lnfBk7SBoO~*}!B7LK&H9j}jToArxTv3( zsGoQfTaZpmE5n0130hU&^SMguJPpTmsE_oY)_i|~LP$Y1*6DX>AuF9k z?%Qn(H6)xT+wbgfK#&o;{SNNIN&frKoc|onOX&N<)i&WAW5b>Q&Ngo%xv29G6M#=yuZ042KE zs7>cLPWfKz38mp}onKP4#igaCNVh{Bb8|M;28gc)#F7}Wvv%0D$*HLiRV!ggZGI)J zxzhejv8wzw^h6{WcPZP0J2M~L$IKR|KO))Tf|rOIB0SaKlt-cw`@{p93)|qTh=(r| zgdy#f^#1uTfB)Oyv#W@_iRWJ`D6~_#U7*0CySO}_hSL)t{Vw?}J3G6Wi_rAU%z1F- zlX$EPZ&n%%QDhKsa}{Sa!1E&Mtwdu-(Yf`E%AM7H91-u1It}qIMiS~doi_M)F=)bW z!KI_9q~t*wh4qo(qlKz!I1{!oX`z}doUzw@A}sR<%s&9RBu5#YYhned+RzAhlo;Wsca0;@E( zPGR+^BFu_28ghpeIzfd>{fo6o(2pNYgnp`OLA)>ukS9PW@S&W#+ z=sw)KD|*`NI5WiVIJrjubq-g*ZXuY@4m6@*T6$vzTa{(20U}>dle3ghDRh>Wmgd$D)p;G($XT`Th&TN@ z{@v5#0mn^gS(&1|I@WNup_sZX78BDSKbK4N0$r_#^hZ2is3=CU#6*4^_^Bz|G|9N( z8QsrM=4aS$D(m7+3LLk#NGr2L4i}rNIhde>w#*v>-a8J+A&mpPR@pBuFL$@51P->R zsUUYM)N!#Gra&B?;=G5V zal+Vgg_!V$4M2ST$Pvw+4fobhs2*M=t<%szWB|ljA!B$`;fnZE+AGAv6Y;LXOyctF zh`C~Q?)wRmNIW$^o+0Dtx&HVsmG6hQtkejy(B@F?;EG`7o z?JxH-fSYl)dIc*0dhW2MK7NImYcI(VN_mZ&Xy&l^3+^?4htVUxK55I*f4PCG|4gxf9&9Z$^SRLR-1< z%v<#-9PlTuMFlVUHGa1SOG0_D$4ALiX$T*li(xqzEU(L&O46Kmmtd+xKx_zgAtNf0Mzi ziNbpl+I$AjwD*pWGpPt5%BQ%n@CO9DfsH~27=0?k-GV&e#c7n1ab*#OBsV{8M|ZvT^abaX7$ZN&r#kTbBH--g+6wuXcNj|Xen zAv`XSYcZoP^3fzzwBN#0h008)Y9?P3d^`;z;rreeyhrQXX1lPm@(kdp2&N>-AmZoH z0!E!7J}>lo8p4$E!Nb7{^JzBQ zFfcY7+?x6YeT%=nAm#40xx~}k?$hm`-wWJ>9xF9qfTTek8Y~E?OaZ_kMhhESfxmOzH)lN4;8AKknqP1QWiuHh#|n*2OoUfg^WXMvYTElP?$O&u zjVOJ4_>rAB$Xr1lL39HZWf;gxR;?Pwec4%v;rw7=AZ4}Gl?PUmxCy=(k(gYa9X}s3yX_-O)m!aWd5ut~X7&Dxha-(EL+1+q1Mqm*&kP{F@ zXb2}Mz)6$|iNG;hw?GsC4E=*_<;=SvS&cexV55Of z-0Axa`CG>O-=N-$iH#kAl+zX9De3ZqGSkz;f`fzQF6CT|AODecQXW!c-HVY1U(n0< zt!hx27(n5&h#40dDGA>sgH^K{H>CX`;0|DzaO2sRFA=iaA>oDpRlg7NP{grrpc;)8 z)73hkzFN9_30AYn_7h^LLmeecXh9#@F_J<3(bTh$J8pg*SS9fY9=-b*jm&YK>&JbG5Y3FCS`1oS5F=!yhGg~PYT|S1>8+@nL zH)3IAJS;2Aki(LYghUz)M#FF2_>valY1$Uz&523A-GjQrP^Td+jWUDw%=4pax*m?s zV|SJB7G?l3g+h7ZU~eB zxj>c%OjD8UKlCiLVN&9;v9SQMXfU>zCF@AYCGnuasTfK%j1s@Ff0~w=sm3~33kG!D zwi|D<-eNhPS3vW;DF# zB|7{1KC#pL3%h=FcG8B`kh!(ai2EMyrQAGH%3_}4H!?m_KfjK|t`c>#Oqux%cE0q(pIY6E&_kgDC`-eDSqlo=+w7B}Mh;#em8A z9>{eDa%wP|L+>Y~CZ)m7gM!!#6)PeNVUODPq1zKmz&|2`8|k^Z@vZZ_NR;4vSeO!P z(TDf%pTmr9Iy85f>LVpZhT=9XEKIaIaA9&{F`5_b4&|G6nlsgGO;$^xpE=Ur$6SH0 z$GJEa%VUg8(N?u>uz({imfwbn0!A>cZEPMn9mX8xWv(Bs)c-kMa-~Nd91QW^?A8VWJuxAYlAGvlh#RwXP zdRq$v0x)+^oxy+m-s$P7((Bi+tIofioSbl8Iz@;pf6CM)#HlcSCy^={d@UA3* zqJPC^kKxQkwyt{z2goG?II_1oz7^sIVg*B=Ck!4(S_X;xvS@N>d_ux1czGe{lrU!tfZ5xnc7=ldnk*gpx2_0FApw{N1ZO*ST!pXX=zG1$pWk!}o>-iGI8{xW<+l5C#p}mABdp|;rZVf%22feD+;CrEZ#RJgcVj8cCkDzcuKEP6RziR`2++y zzPpL}U{i~T)H4;)Nz&rE*brDlm(FcCK$|s>g^dlb@q12gu3$9J^fsO`qFvF#HO)Yg zZDwCuQ$84s$fZ$U9Ca+Xv2AM74(Aw4Lx`{M+oL}>u@IR%V{hTX-hoB0?ESh*rm3St zTEBu${J-$^jnOKJL!ITwGH&OSUXh@kSOGg$9>}|H5|Cq(fdLl^5!ZNr+l`Ydv1L01 zA)-%D9+(z(1RX*TV#Wp>1Sw%ZmTPc*f~8sdp#u*J^6)==8O%9!P#OII3XzbnS7U)o zcDCv9qbd^Zwl54S6&HQS~E;i{~HRkJ?2*e4F;-3feUf6eD5 zujM+UuIX(k-@SSBy;C%u<_+M2YHAh8j|PB-vz>&9=mnfs4uE7O=1Qv>g3wIdWBW%+ z#C^(;7v1p`NF@BTadV)HewZSopYB^NGaommuim26z%na={7m%B8!mr1os_ERuXOeF zI{N#=Qc_Z&HH~z%UKdw?lAo63N_vn)s09%VaLkGW-v7#vIFbRF>da}^^I$-~F)Qd~ z5b0Ysrj&MU~Se*mG|g~r?;IEfu^};U3OL-&EcL=)EnpJsxT%4@4a1!LW<+RB1^72N*)cVdai~pi!wPq!cs;0f$xKk|E!g=c6ePPE6Pe!v9><_YWy zLd1v)n0)%ga3-+hT+}%1IWhV(!~fpl@rv~9XFP%L5zp$)r=K0Jy?Z5dftA%v4lKs% z+FEFD{Oad2zvzhjVj8R!e;pz@QU#^8wUg`x6ICYU0BZqs*->?AF)k@d0p5=G$@%R& zcj%m)&M{jl?%x*&1_0}s&`4Jfb{it!jj0+4nc<3jS+_(XdS!Lq8aE7^)uFGYrQEBT z9xf|TQ5b-B<%5xt9DZByMat%7oeP@3c+-v7H#j*eDxWP40=7UfLIpyL(6TZfxHU>9 zC$@5PbL$G*oKE*XLWgzD_rH<|9uFukip$G05(MlbJMk=4XF!8Gl&>iO{|L@uEKP|n zRK;K(F<4pJ$tdpU@LG@tKXzM_kL)t5KJ`)}L4?*oBDFsO0h!C}u%})8(TXEF36Uog zOZ@Wxec`~LNOdW!r0Uf=6Fuk$>P<2+6SlMEXiKcV`TnCTy->YieDHU9USP6my? zNl=SaQ+=siDk@M9Mj0SzV4K~T$`65c#a{xcw#(DKLURJgj=5pFymp*noA8pf!2^D4 z%l3f9iP^X>_OB~aNtBd9#m=661&*5(-*Es&0X;V5+T2OhPGM4g zds5bd;RJ6Ow(4&TRX`|YjG(F}Pao|s--}F&n7vG>9G-Fbf{FJitp}xDn`0FA-2+Ak zY*<@wxPZ7~{Z4vtd^qhJ%T5hd+89cSlYdZYMw#D)p@&s$aQYYP+d@@@2%?P?yb>I|;KEV@(NKW4H3*y1^v&IcHzQZd%UD z2UN9gWZT%u-HmseFLnpD5oo5pD9S!wrk-CNJf6VEFxmAr0=LVRqty^yC8&k=BBT3a z)wKtAHfy(Cwa?c*EMn<_=h;kzc>RP9kiRHBF+7dBB6hlnq@DgZF>x2!oh?`hh>8Na ziOBei^5&x9@@EShw<&EaMRpp}uy+9K=*T%5F8TO#dCCSN+!^qiq%#Z>^0WcS8g*dd$%Pj zyHn<@v{<|F>~QIYi#F05&!l!mul}(kWifYEO@%}fW%cx!nzHRhSLgm^(3AxYGDqiL z*SL4zQr_OefL-ynnOZM zGZ(7wkKmE2A6V%}JlLI8YaT2oEe@Ux~;)q8y zCMhXN8DknRY_$OEMZC-HM5cNaCCLFYx3WBUrD0;Pj3F-fj`lC>i8D-ftlLJiE9SU( zxt|*r6VIPIu#aoQS<|DvsmP#6v-3(M=S!9gC-zOTzP|MKr+1ttGvk}kOY8wZ$BcnW zkDw~Go??N|>Urfie5{8LA0C4T7*2FSdGLE|=K8ea&DCZH!y=+0CPNCPeRQTG9d(R= zKZ7A$jeq`p1pU#7^06N({wUJl0o+BX8cgA!qvC_rv`ByyfZ*z(Ga}fx+*l1J0ah3*B%Kwtt{x zH?CWGKiq0sX5YJ$77n>PpFeC+8v)38#$vusv%Ae3jE`2*M^#{ou>i#D@mvXBfwcU3 zS&K>E#@ADf$LizEL$5F30}=;vJ&_bNfRgok!6hmAo{yh@P+%JAOa(@0$6OP=l3>l3{p^6Ekf>-pE>Scv_*3rn zmE0Ni?Hy15^%5Rtd4>*xFYT`F2hX`kGD|wR0cc|7ib0U$z~o?+ufFfzjS)&YZt=S<(GN-X>%5otw9x2)=r7Lq%DG zHnZ?VS)nQw7xLMolMW?^Itp35UlpfSt{?fa?N`a22Fik$KAw;g>V3WM@!FI#%Azx` zXU03;budxulR^CDgqhZ}XZt!<=I{-PUd6@EYYZr9aHD9YTE7bm>mRHRA-I7<``cR8 z!Xf~_7Tm$Xs2g(NX6wB6qm)B_=8PawXZLnu&3lmp#LbrWkBNzWhUS2R&bmz7*KFB| zeLHR_bF9a*wz9ILdh8qH7ru42{ku-$$%M9ny4{QVmfSPuug5J32?@+EH0dk%!i=O6 zXkq0$jJoq#C@0Yu@5z@}ykdz!gTML0qSE18~dtg!MMbZe*w|;lGx||i zpA;5`>4|Vdx`{l1nSNl34x8&@_y#l6z4GjwoY5fcF?d@Bv*MV~x$Jo`kyzZR7ynMA zt5|LOfvS17yHj0CAJ+;VANGW|O5gmrq*Gi6BF==4?sjUvkRH z%aha5(Fux(=wc{|Ouqh!&inmGC3c!G4?5SwIG@8R`*od4Q+BH7>;8|QV3cjlW__bV zb5GJY!O+l9e0i>gzSvmy_u_Vx5LBgp*^XSX4+Ao-Y;DhjbR$Fo;(^!=R;@s(%YOXW z31F@Se-C2`GIyD2RgVf}Nj+0j{`e&%W{-@F%}Bgbc<=G%!X=HbGa);#X1pXZVj6M2|~)4zWWcRF=@ z;G{?A9Y_ifj*1 zlN}=`VxB!~KgB?YbE^C=lvkxJ89k z|JL4q81z6X5}e+#F}+(~SIFElGf(3up5mOL$}|RbgkgcwSm_s#8+vqxyhukTjS;aQ zFi|q+@!aiDyEK5wX6;{7c>I5u(%vOCFKGFB5Q2b9oP@2WwL)X^wbRd@KmS%+yBF8N zS-4jQyO^0LbQ`+660)FeH@D;ykV=6A^mj5WLMS=$JEC_PlJU8lw>>?W9X zt+(GlF=(BuzJOg+YGrNB!}JAse`#6SHn`?lD=S-A2;g+4+m5P#vFv6R7Z*P$<#@)h z*ySZy1~x?7ozzEUpgj2yt)3++aM#@26p6XR>O40@($BD9r8PBsj$iaG0jDm6|wTMzx*Dcm2%HrL8>Qzb$E3ySwNYRohFP7Ueot$1i)Gj%_^o;5+&J zSi>PeB-I5XLSyuLb|W6$Z_-LoG5xeF22b(fjYphj%OA_XyL#&)Sh=|q{&t^+dU18Z zya{)U(wpF~Uqt|-y?Xoh)0G(t0y8u7eoBw0x=T@3e3ZqPE|E)wJqOacA?5hZrjo$n z?&cay&*-M6T?pO_`&x6X0@J&n#mCb$llJ?K*i&}KJrKSo1k$M6g zt_K90diQY%)zu;-OObtK7?+aIPqZ`s`SYj#&!6WQ8i8@2q*k%>3JU!|g>*r_K^DBA zTK87#8JOi>9COl=*VNWF#H+E8HZf`mq#(&xI^xwLFXC8ZuX~r2$RI&c{y2@7_!mM; zFaVbcKa*F4c(y$n^0~_3L$D%I{}3YOUbu%b(-4P7Au6G(K^7vSL|aZMLQ1}>CS}|z zQ|h34>^8@kcB6yclHKLfH??hReJv|p>a`K%C1@qcAx`Yr%yrGIF+Y7evp%!?WX;=C zv`{E;>9kecIAwVLeALH;P}}e)2;FUMy|^(D+SD4lsb1<<-l)oGtgiZd5aY96`1r^# zjCr?H4E6Or0V5aV?`$SL&PjK%!LSvs6-{Rn-bqM_CL4O69u_!s>*h^rvh=B)f|=2U zhF}*kk(_6tZ*6JHw5KGoaKvi(7qZ!o9Nq38gVq^1#PEhD184p12?Z4tdYE5Lh4_q% zl@U-p24};wmj^=p*KgmxzzX&uerx1%)5h@wj{qbqgJZr>2UxA{5S3 z>1J17Idgx|=PJ`LLLSs%H4#$Rhf*wVk#Z4I5>L7@u`PKvdcptJ$jK=x?p8hWulMvF zCg#pUziLLm-qyKclGYpwmpG@_-BcBC_VOO136}B^BmTR2=5t2HKviJ%#`|IAr%+uV zBPgDi)x>s_wTof8!d>dv+hW%Amu$F>vD@+BfdgbVHgh}%CSpgZ)z#IZruy*l#8XP9yRW2c!O+BuGS?m86)Ex~kL<;{!ubxKH3oGB zJKLwJm$N7^>*^_W__k`o|S7PjWVvGyPM@@ z*x72j@863~{oylQ?(`BwkJ-{xH?M8A2`f^nLG<+`F&^kNG4SZC2noGb%M*SDI^|xm zd(qSc1HzPRj}R}75puCOi(smK`xH>r4N7O|TvehmLaZ0mxN=rPg7MIx@g)W1rx|7q zY{;mO9Xm!qgeAS$Q}_R#L?)2;);$dqWPMrAG2{GTQ9M!Zb~HBNI%-NYGTK00l<8%5 zfcMaIxf9298BP|-nG;ed$jOLAVv*HfUNyk<4Ikups)P!~H8Kl`i!*V4zi1t*sH3fY z(e(OZ&EEl$Tc1hSRj4#03Q`R4aLnxoku}|3%#zYpJ=D~e{6-8Zy(iXwN{Q%&nRRqV z^^J{VKl3O(iXv+$w=}vF{k~aVIuD0iexn<~LB?iLf@H?)PeZQ1UgoLpECUV@yLkBE zp+jEaj<_UeIId9_*bGzCYEfRiaDf7oN=tDP575zVhfeg)ojdyZkM_#AOOOLqMOICT z$qv@^h8{%yjdy#$u2$eGnR)qE`>ewj&JVpyCaISYbq^-usl-X@-@M6wFEsQghX7ub zfz!UI>mk(E%H7G2m3#WeoxJW&tTcJ;7OK_)jY?$W9XdH}>SkXP>6 zyB8=2mB(3uHzz!Jjl3nNv|b>WK#gIZnJV|N-^}2YEM^ik{NH z^xoPQ651C3Bp06&XUOOCdl98*FDf4IG~37r6288^&wxAQmtXTqW=B$&0Sym^P?sQ^ zz+8*BV1RHTpTsQVPe0Q=3>i8)IKe} zU3e>>d6{=vz@m#%K<9`nMR$c~dF)Ms>sHCFj~ZuFuV`pI_3*fmz!(pgOg-VzuP%{i zACe8DuA7pu$R_3E zFzBhi`ECgbyA@?_iU=af1!bBT2Ru5Her7bmOfrg!3EK}lV;Z|9QOd3l}42&?YZ%JI`aiHY|@ zzbRe2b}g~$E9Qw3KGCr}jZ7nYJbM;2n&kW8AOoK+?2gWL6g;NAJxKJgQp&dIguc#q zy6UmU4m|7g=&NKWj}Wjaes+kkS>mLogU5qthr%Z<BJ1So-81{1$E$S#s?&bLy{ zps;-nW>LardNy$}A)QB=2&_MWGw|~Jv+e*=oC9|{A}vWyNx7z+d=tUP6Ody6>cV*N zpGd|(RfxI|LH)i-S0`!BKk?;lJ4v@|yY<322S0AQ=U|KdSk9#y&u|o) z@>STC{=~Ab1mQ_fX+9l&Wu4?7^P^o?v^Qr9((nVMV2)^O^6ziPNJ|>aWOwmaa}M^t zx92|QC_k;0oz%_d2SLK@1mJd1)IvWvtpFXbK$C*4ho}f>nP8=(G`N=i!C^)#Jw|Pl z?^@B<%F5@!ZT>yDWi#+^*p;sR(WI)fs3{87Ev)V8e#E%&9Z|3)`gL>9|y&4%@SI&9<(#~vz&X5c!SKJ z3Wxx1%H+R`;h#Qnt{u%$cb_{&k5pgddZ@~it7y=R!ETYjCCA-Zk zS6_rb^{h9Yj*IDQ+y4{phrLL~B^8SvVqr9nG zK7tv#`a?f;Am%-A@ZgK}^|Uo^SG;Ik*|=&ll^Bh1hmx{gpcfTNUuo8+ak-k}omkn} zta08&Y9^PWm!(N&dz0fEg7gQodt_$Sy;d-nZ&BaKRyMw_FKz-`fz>uA->{{$;n9+gh`F`x^PLvu7o*FGnND1P^0Oo@+NdC z#JG=fRZ_E=!-DdP2~jE5C8aqHjn^MJZw&E^+f51n!oZVlJuKRq1ZlCk;E)iVp`Rhx zf6%P&KYDbd*&I9955R9wl3EH^UF10hTqepLUL{xdAcEsoc`LEh-c~R@KR>X`Lrzee zjZ{Kge;L*p#)Ivj^r)|!-Z1K~<8?oH`nK==y}F9K zA$~Y-Y>8RAy=U2--W#r|>?-Wm+V~<8Szoe|@ zy0_;P??*Q8H+k09)=_bBTDRTs!0Ks9|K4YoVopo4>ya59W@AU@LN8;MKgEE*^rOpi|P{<1A1;+#A?O|S$t zki>tT`AhvW9^148vgI->vQv#pV-*7lbh-lh)`OI!!;ZP!eaHs2K9dg*56f$4WU5c0 z53!1@{+S&MCF{y>_t1h|=M%9bYhSn;-43_fpqW+ow^FG*;<(b*e)4umha4PMAyTHTU4TKV(m zo9pE`zlT_KenU1Mb8+D_qhO3tMfAhWuOv~qek#w!y)5KT`4{y+T(LUnb_$2pQX=R! z8@HI)A-2~eNzTncIU7>0i+=j_NwaMmv1qoUTxSrYAly-#+yF4A#i@!MY-oCAojci< z)K^P);SS-a+=u-reG+MQkNwR!4up!Hy+|cd<-29~geNG9Z+~^AO+}$#85sOebuZRB zdpH@M=6~DT`p_Ionm~+sFniJIctp&4I|O5vKsrGdr~QiNrG!q;%*^bJslkgI6^tR( z7p#@MJJ>EOs(Lj32tE|UbSv_}9@kBBKY_^?{;n{t5I|p{Zx$~@Z`$~9PeZ;$$dCS` z#uW=&!Gd?c9^vHVeB&ta4QD8}XJseg0j+Zn=Su{9r`HqA)ce3Ym5_B1 z4zBkMYJC~jdG}};n<-82yswW>8E*96NyA*z?r$k==bwSci@{yKWja;RACuT9y%}!U zNTqm*5O^iZ`Yk7az80YK;YP?GY9GkA3VBt zKf5ePCH$z+`G(%@&@_P2j6&GnWA)fC2eI8CmE4~x|upOo8o(rNMjN3{@b zq7TpqH1~ZVo1~Xw-`}U)bMH&ZHqsg35Yc9u^IV(E_s1V{4T*}C6VubLO?8vCLpU?3 zckd?4@0@Uol#`Ql7z(|kn{9CvOs(X~Kp=5-G#)rp1c@F4KQ6u8vn~5X&7)t9%E~ZR zXYQU0hR75eu7lKSs_y0wQ<*CVt$+SBI6O1mALsz zxNEqVF*6!r3@|#f>Z4?zI>wLf?+^o^@Wbp(BZS;l!4qC5B3|9iYBTaNq}esYVI}PB zcFaSr^cAZ-yT&KXR)g1Fwfrx}2=NP_&+{2oyXw}KxIfi*tZ6G`IF*s1Uu4@bVfVtz zixs`4=)r|BytT+#0-rVzs3zSTtYb(*yqND+lUO6Bmp>|WAWvpxWfeMhZ4ZaQaO2Cs zzUmNzFeP3Ak>{2}zvR+>&-|)!tuI`XSsWg;S*|=Ky)j&@c)#^EMYq6r<4@l$vT2VT zDNl`qvCA_;mgVL$n`+j*{5)c4@2Z|3x3kN&+) zcYOQ;gHas$(?$eOp}Sg>r+I&aGPzDPo$9e3$EcUksfOqo>&vE$e@F_@2bXy$Mcd{F zYg}ioH%rTOScs9`=&3<#F{QWwZT{RR>oc`?VsZBr(57 z=RsQEl_?rI<*qz*H+uJW8S%4=PIe>pvg#BzS>q_l2N-PLJ?RFh+ENM{zn8l;bg% z)&F}Y(8EINYPPS_O3R(o++wc2uC?zJBfF&P?`GlQ5??l#e?JptWaRnXF!NDAe=*Bx zYwY+^+HGGV=d*0Q^Ey0TH^z-ZB_9i+1;@^1NHwHPuUhBo)z{T*^^`6V*rrOH-)8G( zFzN;aB`$h*(d1C}KH_>o!9*UgC!rnUgkO>8>X2n_CLOK@X6p6gMHDVv+ljVdaJcPx z1>hn5zVDt1_|!g>B9Svy#GD#QHoEY4;t{D!ptoxEacQWz3Rf@&TfMB)uO#&mr?og$ zOq@J*>VSt_gYKRa7PLlp2>=WiDt8Mr3CpuESJr#(?3xX)y?vZJV|VdmyYKwn3x&le zI0L!K6e<@22;(Q>t3N2k<@G0JTj_t@#Vo-}6=-i5=`JqTsc6BYJzKUW zV-(Og2G*Y4sb+D8=~|vGw`JkNo7nMstT`ZoC(x9NljmPZqnu#Rv+g$rs0J) z?_SxATkQg99-W-+|1Q^6}++OZcfSEA}aXNfPkhs0A9lCPreU8V^HpU8_ z!n8XB)?(q}U;6vvc;EA#wfk+am-5{ASNP46D8j!J@yqI5boP8j1d==sK@Bn1gNjiI zG;x~XIvqXz?reERlHu^?=2CI@Vm=U`-x;O~Q2TPNII2MW^zrwoA*~b->ADe`p{Ju` zzs8tp)V3sy!#X@ZJ#7W)eKm&J)q-I3^6@dP3g94Fjh(UVdY|3x66Q+Wf@#wL&Z7E7xOn=FYGguE8&wSe!$Z08}T#=~(&Zgjr+k z?&juZo>L$5D9k6Q!fJWE$c2?&4CivwKHiF{vskgHsr>M{{?)79a>Qo-PY`pax)s);jj$IGs80d=onF~rz!Y^-=3aP&8ds&JfuUymk9@sVLVz@Rzb>@ZNYMtA= zt{u5@T{O$g%XGTT$2RD!W84z_mw5yQhXH9xPM2&mOo6jCu$SlcS4?Uig)BqLdIR_- zab++>`l>(NfC65Do5{Vu_+N!33O{@S*8NH5$f%~pO4|GCcN|NM^13)pwGLHgQvX_KtD_ywRd)%eTd zNYUoJ2dR#+vdZrGLND*VRvr>h^Qe3GOPapXrVmHkzfR?i&b{5S_heY@)wBBhinoz- z$R=q#Xq{LT_emQYyQiJ3OG*bi@}?J*D{l&snjWe@_?(3>?d9R_yrUu*Fo_oENSYjt zk1DUx3slVGsDW3RMDCMsmCWe`U#4QlMuLl{v2^3X@4sW0j7?~lJK_P9HhhiJ+5QAI_6rU}(wq@AIES#?vrS=zMu-a>v!@wT4Fhizgm z6$MHfZ5^V0X?BJwE&oUtR}miGoFoBR>a_3Xj~pHts0UjWnOF_pJ6)I0fGlg|n)`mE zC##Lr3Kk5Co^&na3+KowP~a?axQd1 zur1qiN?X}6ibxrqLye6MnA)0>(}=&SLzkL!Z|P3L96 zqZd{8wRJe2mFg*)dJ)pAH+W95ZvR84zdsn>M@4n~^yxqlV-c>Kk4rKH(Lo-GbDUxN zgBl|8Z{94MZkT%TkBpK!TC4i@&90VUblEn3wPR|3^sg^%ig?~)lG8t{^KXV)y2KQ=SlHAo+fM&wAfa~nbU3PM&rB^JTK!8;#_T-WIdU|9^*c+lKDQEw&1 zW{$SGs|lKhDd$rLl@l$Ni@u|-!N#gpjWj593`S{#35>lq3#7@3p!T<}8yZUM%%?^b z?WzvhZ-hjlYDfEW!gAj0!v@sXv+fPf1S-Lbr-@ku9 zKq*d~ayiDMiMwcwX&wqHN}r0ptPaC-aRmkXx<&e2w_P1wlkpZ&7u6*>t<qD;<75 ziWKM1Fvu1jcq7PK1Yeq8QgTIXMZOo0`!EU?Nw4g_0UkVn+gx3p2AzX&-Dp~AMR#9o zwj~SFpj%LC@WHib-?{buNXT#nh={zwU3Thb<*P84VM;Vz*EW=S7T!38?;96eemXZ4 z)&vEsC)HF`$}w5p$E3o!PZ>G;YK<68UtAdDh~8c9L3%B|>tFwU-0(y<)d@jCe^?7V zgK&{(1ha@^#=+;j(_o*_H0rtIDH-M!TU&o-FPFP@ivt4NzF#(ESq;>d#Kq<4Y7>5- zpioj(XFq5cIr#R3qwbZoHMujgCDB8xW9DFB#KV8Zm$!fb<`%U7mzl1YvWG=lzj4qX z9n(mI3YPnP_zd!bz%{8G8vcU*X1Skb>k=}cR+y4ORbY$}$pTXL$X|L*O!T?Lhi4oS zxx>MTxa~J00kXy?&C2Whq-SnrwsJ|@={0(UuO@BXLT&*srmk57TiY|F$cc|iL@mVP zerMw_zjgR7tzt2icZy&Db3;Lb`&UaegXlY8SovRGAlovIch_8MtAp zHuuUmoZBEYk&1VBk66FO-ler$<`xh4LnsXa!1YeBX!!}!E~MNYLSzj)DNKYyB@KU| z%-GrU#+ZAuq*hkaxVl$z0x3BY(QBNnbv}nJcufmyP=GnwP?FtU^Me?8AY0o%!@QAj&2HFGvA*2@!)b-hRoGkiPPsw z?`0`P8()m^@?<&uBep7VJZtHT>(=J|*3XazR%*tnKc>61Si1Qzkv~h%;Lc2z>y``F zuD~L9Q!SFIv(NolIU^kdLqJf_z`j1J@XMkq^Xa7h8(#4uL6iCaX#ql%;%0YOlcqQE zA(`R?MPik6rlFn|GtXA`TN^9 zxnXuhL@$?S(gNbP&mH{hSjZe!`@Hr3qC+8K5~4b6v$*kNwYRC{dcF!%_UKU0+%j>V zWN@kcHRHf){bz9YTXk9{afLF(Ka$ep38_pcWo9D_tKDS&D~)rwX}W^+S1-SMH>kY5 z-3sLaAKyTP)8x$IkYAq6j=N$S{!;E=IZ9f;$@>wzs9U6r9!f}}89|kH0!vbLKhM4@ z`E&G>o*v`uqecmujwyx6ep`!Oi`!4As#{5X(dFxt{b%^5XHSud@%ryN4TA^sf2%Te z(oAcX6ZrJLyai5~J0U4pyeR3NpA3yj8o4{A-i9e0n1Kc;sk-W=kO6pIlf%3zL8ETI%J${_4 zTprSV@4|;G9pVz+ULOV$?b|Tzq;!3{Y@dacrzW2fXP_eMbH?nsBUdP{T)SFjetL7@ zbzthUiLlz`P5SCL&T4>LI$esuh3oM#RO=bh7%e<(4Gd`i&|o$nN}*`f^Zj9Nbo4j= z24Z0qIxcRBhg*1$+$Du{2?nFvnXM<*+w0zoJ*~QFT?AncjxMR4AFv0m`#lsC6a_Qi z55qL)350m7^P^={f(0d@{4rc+?oxSSKD|-uE=!nLsc338@kfgKnpb8e=@hDqhF>18 zsU528Q%-9+ZnNC9t3_UP-|mtrvOUes#M##S)GT5{e`p0ZHg|GLoem$Y)*%5B**f*N zGn)m@K3H_$%;6P%#GQL|wed-;;FW9FM)frT|IbUzEMQq;u5dB$m*Ph?KkZ{ZPP1w1?sQGa&vM#`K7z*UKGk>cGh z(k3caOyD^cKk^m%Rop1G$nK^)SbbPFvO_UUBt=pDN-pMz*ydQ^&QJnvLn7}861 zEHJ173mQ6z=JPnlOz@wBFdZcAP9x_9dfe3LObHUR8LBUKbxqaJ{gyvfq>L^VFQi4d z9aX|aUb=41MUo75pO*H_tm6KrJ#gS*%a4S_#Jq`tQr4n^%wcnEyZhZmohh=SvFXE9 z?=5t^ZoGXcq1_{xMLX0iHqOl@IWZeZTv%ZHShy^K^8q`SydON-CS^O4wa5L77F81J zBAD^k+U|>F$F1)=6Qsn!^peDd6QHK81n;w+8Ue28P4(dHiSzzc&z5P{Mc*gQ7JbP; zb2gn%g`AcaJ#6;2A=Y<1Y?99+eSkQmi2Z<_Bp}-8gY&}?Y2sTQBj;e`LY`>R#oTi_ z5KU%qhwztLJZM?8!rAS-bQG<^76w_o+}yb-`B6zP%nvFkX4rRNR3JaN2U1X-IC4Al z8co8^)OTIvz5>4*lwoo(=#+IJ$^ib{Z1Lfqsfj7O5S&*yaBKL{d|iG$;cKV zt9-5G?n0y48N18re8Mxgvh+Nbty$}&*Z8Oy_y{tbfejGCKukSkPyXBN3@3q2_I;1v z6>rX6m!4R>-@itKuGgB#bKNUZk!S7(v_F%BDiZaIFO%@kmB)bXfkNRz&#>&4-rSf% z?HH-1sY!=hzHrnCZ+^U~2r^Wgq{E-o6Uy(M!~^j$P$|W2?GtOEeZl#{{tF5PMXTtBvXT-TH+Pj_=TQxHbuKP0t0GIZ zqvL4Z#JOqzkg?To`~5C=KUulM%$ zqFzAP+vqxJI0HI`;<>J;80Xy-3%bX^Ix=s0-KD?e> zu$vUy3YKSuMbXUli_DKm(sKm}n4>`g8u{#hiV~o>angs>Tvr2s$5ok~xslnr-ZtBe zOs`mfW}W+g2`Jb0h@lHTtWmEU%=!}IBkxoMq`Ax}9JbCrIhY$_783Cb0M9r*PiHWC zX99tTh;?V*Yxag#0(g-`E*gXQrax*{=)hMeydtF2;>S%P#qWc^?u$5G*!6SJnn~0` z1<@7DJ^X;;UP?vsVmHFgdpXtk-<^NLPXR^525?%q){O^6Ceu%6L{j13&RlsvTrRU~^Q!u`x_--gU*QWB2%kfEbr!OdhC+MqYj5+$fqT&GjAP`k8S5g6}Lx3)pauaDCQi0zq z(uk7acyd$jqobyGp~dJ9Y9SaI!qZg~?8U}qq7-T5%3|xCeQ17t@tRasmz*naK>`?l z3x#iDok&sbZIVunL;)>OR0S7-tgo~IQVmgln&y|v@-tjyA*FHlyKw$CJ}dtxq*cni|xF) zrNqhC;@FE)PaI6kTnP13gxuRQm~W}|C#A1p zn!S$(Rq(@_5g!xvvjkY-!&L~cay4@LhLt|?8F zJr86|xxLa>eI?YrZFgO|w-p*c{j7Z$nWPF}GGI=yYI_MtXe&&&YH)tMMU7hWIjO$w zIkEhH&>T9PU!|&X>d5(tUuPf)CC!lhQJ18hl%dOKnBs&S7h7O%Jd9Ke?@-?@-=oCi zrsg)~rr$%bqE`5{qFkchBPz2`hCi_C2-*FcLTB>!qtAsTSw^N(JixQqs4Un3E=j^~ zySVuuJ`Cv2(n}FhBX0@SAU2=S^=j&%-e{QRptvb?$X{}H7s=ZLhAD{rE4QQ+UNS*Q zxO}sr|H;;KD+#*3>JyauJPAnhAuE(dC&1V9cw@)8r&7!^mZY18M4zDtXdkWb1u^uYYpg#RUZX zP@8wWCQ)~A7r7x4X34wDx4GF~^LzQgM{J$4LAxm!Fr#^EsOm(d?Mg-*d*yL9Hl&$K zLRKn!LuZ8AJgMetu!bJDsLe)zaI8iTFJnn-%hjYK~Z zMEmjMPb_`=PF-$C18l=6hDb``bBLw!>2Hyfs8OW>Mg+g20;+-})`5DHGzbJYh8RUi ze!wlr+0>n-WETV*@(jeOZoBr*3PC(4wiLH=8do2LseFoigaz(sO;90fKYlE>Pip>^ znl?azS!i7v(T?f3lNoYMQ{);zi?ewJ1Pt$Hx3kR8#~H54rD(4ew0|fy-j%aq*cC0( z_o3KJ&<)KZRS;Od1HwVNoR<`Hjzvu$st#$rKZKb2NI%(H3~a#C=yT-Z5|FlF<8(QW z^8<1a!wiRC&P$yN$WS{u3rb9c&xB6aP_doi3xbo^!}zAa@xc1;(H#1}WP7+|&~*Sy z;0!hGZx-37(XPhB%NvR|P=ljL>>^+U15CG(Wx1nF>#kMIa-#W@TM1r*34=%G?yL)s z>^Lhoj^b$Tf*`3lT8>>Nd1dD09JFFTaamDu2Hy=Szx}vzHf|7pj_fj6x4Jo;YRd5_ zZeFy<3x(^%bS|Yzq;xn|=rdX;o5u3%lL zk35dIoXznzbumzyJk7EDt$IpfSec4B6F>#r>t=JAAFK^ww7h`a;k$UsR+8?DpZnj?goCT zbms_$U*k~D{xx+EBaa1_ft(`pi*^&qu*g>2fREh%%C-VBzls=y>kRW2fk>`pl>Vr* zs38_xP`b3POd4Xh)aA4f2(wsyHv~~PGL}6s64@w%m7l-%&`dqcA!*`Sh^9=>tC8M8 z$JxgZ3qB>hzR!xZeT2>vKWkYiVx#PZ(DtEa9GKxl3Zc+{eudP{O#BQ!t^9rZDcYv? znAL(?A|@v0Di_`BT^$vC#x@hr9>e83AiJ*^zhiyE($y4<_5x*iWGJH%@hSry z-4}ARDef7h(&NHk6}hWErxPFhA+Fsimv@at{r6VoqeH5~Hu_!oW^XNt#!A6$9foMa zl$@(ME!{nWi<5)9%86_mdo0I#rSa%S3Wh~q^Lc}Id({s91v`YB-%|31kR{PKMlkR zGLu8$mw!OsAjPEZ1oJsGfoUZD9-X!79?RDt!cbwgpyz;vXGBHhQwsu`n%<#_CEA&NXYFly&Onl{K(WIyi>`uDw4&y1NFp86FL0Q7Q+L$D(|&UDL`ZdKpZ^vT zdhu-uIW#yPf0|&@mlKxlMw?~YN~&&3Q+5 zP*9K<>NVoikf5L|`uhJ4REIPnncs`ZJvT0nHm0i@8pjz;liN>`Ym^=cx&t!wS4P=& zC`=3;dMRAy!n$E_T?^lYLT!fs1$Kn5mqAK{)S~+SmuIxP3zOBUgav~?Z}(oFy;F^w z>5TI6iQr25VM3%?NxI+A~an<mjw!+jD_sNoDKayok7f=Qde;qQF7@vJCkNeMO z7Cli=vhe&iJ1B;lNz8iS#dY$NiXJE;s!)8o@H<{7rIvSBgu-eAc=acKW?_~#Zao)P zU27W~MOD>>1R+m_i((hj1+VC2E7{Z1(2yUqc7FVvF?Z5M*TA6G)L&|-!JO0*1&aYU zQi3&>l=d01Bl}rEnjpFtAoNE=%PW@>Mghq$ze9xB*uD){gy4L=!TCZnz`95uYe~9E z8DW(m)X)zQ_t`Re?46a*t@nuw%{p#nV}q(8M|D1<#)rfEh{mm=;w^c=#FhMPRY#qw zfZxY{d+F)nLB{Ug`~A2knJHOP#SIva_#^n_ZNvglRC#~)HCjMVS@p<}+PzL(4BBGgCDmY^xh(N)8G%S1kHfY1BUx# z>bJYc|J|{k`$%pJtCKe;!aqiQ4*0>4LXf1kHRsHD8lZoZOeC)RW?OC@}a<$&JY z+#L2HE8NTFqZO6y?6PjQ$3VvT-**xlisdXy-FfTjr#MpWvpk8@+vU3D4537PiCg~Jpx@_XHF?G#oCd=$UpvZlHzO!d^;3u~KAkIrR{R#CEOC%WB>d>6~ zD*GpC@^Ekz!7hTVZ@+`%{lipmABY^j^Xq-@Z?$uUN!wtG`kKU1qMkTuTtc9uq<~Mt zz+BzA6VhFxSqngu97f|pZpzBaR=ZK#^A_PB=snnIY=b_HwP?L_wi)uNc|RXXRT9FR zR&pW0!V~h0IIv|_x(ZMX4luqf#14bM*ignC8on@z!mfIl1X*fn-^kJ{?Be7J{yDf- z6w4-He)n%WE5f<<9#Q7u{Vg#hH8N?>tAl$PR$K-|S|Ptr>O9~#z%4!D_FcUg?Xo`O z?;4BdSLyCca1s*#zm*4nQ=QZz=Zu}`8^PX32XnT|mCL&!KK|6=X+9nO1M7%L<@?CN z)B^MXU-8}hY8*P76=4PfCnVXP%zkc^pgnzW@1y)$egR*kHy=T94fZK3wb-DvMTu~6 zkEmhq-l!u-Qkx+?{NuV|3i==RQAWfW3KC+AqgfL!5rRN)Ra3JZ5mlDtvKm<4u2u6+ z*e62D>7K24Cs|Z#FXfaTvGu-4&25_pn3DV=ZB=i2`@-@4Y|Ms?Mkt@I{`XCH&kVl# zWV6r<`w@~Af|YK6EYqpvR5azIV_0tZSK}_**k-eYQ=P5S=ScAKkITvWxs8K^Ik=Ns zv&=(`eA3|Qpvj{Zd(fgl?$mUA+wKKV78eJN8&kk>GLTV`LN3Y07D!HKo?SosMC~sw z6Nkg?QKNq`%Kw1m>J8sDz@GO$U5pXi)pUVE?C?d-z~-dz*rB02LZQ!=XUW~ur;?B! z6MA3l%K0zS1b8pQRESJFi5{pKI_E*VE%-d|f_zQwgpQw=`N(YHwH_>GqO)$-Hpj3aoM%c&&hXhZr#O z5csvbm(6c~y2UL5lB%6_H#M2@(z4?%Nfx-CL+SBtuqC32+=5ig zWVc`Vu03V4-7r_6=pVozzH695_8nd4Es04<5^q96%rW?x?f*3O-tk!X{rk94RwXOS zrbs192_+<>NTRaJC?nZqXH_IbJ4GmxBqOqCAxR~~sbpto@2v0f>b^d|$M3rTxvwjo z&i8md$MHOl<2kl*%&}wF7)wU0L+(iR{C=0;422$}?j`n*(?CY-~UsvkH$ zxA&$87}(EliOd|1C;eJ4F>FeoN=4xneu zF6a;vLp8q0mMzaE$1II2xK2opizFTircz7L$>Z}r>N8*1B826O#h~^%2Q;q{|A&r_ z>hf40es*FPHdP@<`0veZ)PQk`_NZyt&4{kSMz%OsQ{7A-q=$3+q8~66h42}yzu(j; zlUPPuR>HHo8KziEC2wHibQGH`yo>#K$9iKzB@0_6elf3}CYGeYAs2ir4uj)JyIx+f z>JIZ^ayg!*v7w{nk^)Ld`0(2cABAZtqr?s2i{KY>UO!aPhB`lUciJG*^LeYPL8x zjvKGdRV;z$pK-e)HXY*i6Gns0^zHC?PtWVm4}*b(u94ApIT?-ovuId+)0(W096N4> zGE-uwI!Zt}I3tUQq-DK7S~ub5TWV^)`}dr@s%*SyEUGdnJtDxY$e;R)?Fqg!o1i58 z-tStlq*O7y4Bf*s-Q+=$lIQ110Y_8^Qg2jG-jJM~KrMY<>TwipU3_2A9NQ}9<28}T zL??~X(v2d^Aq0u_>fKao(*f~rJ*)xK4}K&n0I3}(CiS*K2UeBXb5e|haFDK#Z6*NBj;PyX%X*Ge=SVh+DUkoIuj^KGy!zE@#7{53|HJC#K*vUDsg_5r{6zd#az1ni$tjw=Yd%APY1C+@s(*K;M5d z4aDuQM1R-w75#!Z6k^8HCIX*qP~yqG0fyL4k%9Rg3RSJ0(5S0Um4G%t#ctY`=|e<6 z&|*_zYstX4)1MK%($W@|UR$H=P_R~2zGX4ZO#=JsqebRMfiqvUiX?LAXngN$Sl&J> zq5kXy3;~V-P#|I^MD~4A-NqpYuDvS*T7Hs=iOJ6&^b-A=nwnvo?7vE25C)gAbwpMC z?ECFTozrF^B`ps^G|b8De!dMs8Y(R={Vz*k^fip=g@n>F!4tQ6`i;+~wRBkFBC@$8 zu&1W*yXYC_RpX#SN+QL}e&hVeUMz{WW93b!)22IKnm&!aGfa$(<55SPqcgLO zq}0%0f`JcJC^+4?pr50N5#Oo$Wm8;SR4uE&t1ZW$E>q+%%JO6yG31+>eRTLW{Y7 zQ(E8JBA%Ma`!hZ$jz>W``GlKC^P&WVbPGU|v`%>*XDT<4rf}s=W2k_3`b?uyF*G7b zc(a9#9}qF_-rw@2ziS=0%1ING-rO$D&^qP~OhMoG(OOcK(x^e@m#F;gDc(Q~V(Vd) zepK}`2G$S}M-byb2qHjQ8_5Q!!jI8lCh!euT!FdSh7;Y1!u+fL(%6QpnOneN9*Z9~ zZ{1pzo+^f$auU__S5H;Hd2P zB94bM(q*z{TQg3l>fHha{t5gEta_}(J^&0=A3ra*8=Ceeyb74PYyb>;a(V>&;hUSA zMGz5{QyL{`uTu+|i2d~Ay(6HBN!X7<56Sfkb5Ad6r<8(ftwg+sOwwn=tbvOsPX?_QqHdXNDI)I^l9^r+^ZffBZ z4ikUz=v+V!I-@`taD8Mm*z5kNEqPB^q&RC=N-TlcJR2gnZS5eDIGu#>0!xP~ zP@z9!bmS}5taaoF1JXQvHZ|ulYfZj9Jtz$juG2P1*%{Nwo*IixQ(CQq2G0-5&U}6V z9w_-P_^Tow2MeM?*tgz|j;?{vpX%Gg?~G4E*@^m;`eYvfbx`kg=O6QW)2yiE@zII? zni z4x(UE1J2WfiPYsbhum5geI+wNaa7=6dB_dTfZtcy9gErIq}YkV>(lNombo{i^Bvc+ z>#og1yLbq8v_+Ac}K zQVBNuebJNI6ncY1HYkzmtki>iwRP>uAf&%v8Zhc7TZjgsE2@ zOPBPdD2b<79D=XLv01aOr^bLh?5xBs8eJz9oegsfKB{{UnT1JmZ8=(O?y&Z4r(=Gx z4IJdwfm+0-|0wcyS4sL%h=+NrPZYhAO50`RTie6R%EVj_Ho1)0<#Isz@ZkZh|HB9& zvK075C<2ePZA;E@+*>325~EZJJu!|^YGdc{2QGYLyUgH0NmjnVxsCpmAT_v{?}txt ze=p#pV*?iwXj;u%qf{~})9k2{ZDl?8{gA<1rI<2%2+7YIj;vZa;%o|Dg6U3@$Tp!u*Is>_B$F-Bw*XEEhcbUy$-me zb;1jD*_gv$7T9dEja-CEu=2p}&-KF>s22k2xWnm+O=^M7z)L#Ul-|=xI(sB!z5Eyz zxE1<1rKc&VVp%~lYTw2Y$`r1aMZHmTX(nVTVDnfA2oIqc-AOrjsnqpT?f?}6ycY*} z55)n^_O5fO{dV=wLzr07_5d1MH1-*A-GA$e6L>EHn^oSH77%Zn(^Ayy0G`A1XK`no zI*?bx&eAT;qy{Fy0vSzsq~Kcs z+-mtTxX)P_YFLJ=UE26aPkr~&!Cg!l;I2M*d-WEEgHjs? zAOpF(y}|9!h{pIdeTTi1L0UsY!=_zk0nIHvbO>BHsMjvVHE+HpU%y>idfGVK$QO0< zNZGl@0`}>Qjh?(2K#-&y+vK4Qnh$U|vrWW+t|}IYkQm?i2J3|qD&m=2ZDF&%;zkOA zjRn5y9Nbn?@^^5D-98@FAef$N4qh43cz*lr{PCAqT&Iz|1ppDE&XE!a?_v{|hv^j+ zDz5uuQf7RL*A*?F>=YUMd{qr~5@Z>-mqu5X)8ROOXm+2?ck2?3#OJ-n>cL=KisyhMgPqulR3f*Al$Q zQ8wg~UzBc?>TWoLu)xo>ca{2<9Mn#^dR@T5P`3(3)}Uyc)g%lz5TTv|6}u}|i_(xf zcuiyY#&9)sgDThZ3i6X47#CjGOGs`l{hc`^tf0`A=%5Zn5N^(Dx;vV|`~}mw+AgBVaJ)$Xe1n2Q)4Ohc=&=Jn^kU0 zciEeEaro2s_xA_b%4Gfc4BhS#4UM}hUZG)OcWQbh__|k3m;Y}pPS%k)x#43tI5^C5 zTWV?C+`BQJHfTK(zkNHUg`vz$1mn2KOZ8Hp zv{T!+VhdRSz*Ax|Sht<#5Y@!{HIiePM^iC}=_CIU?XmSn#*@azTBw|e+Bx~uL}B|< z1c@N5qAU!*KEX`{qxVP_Gn~P}x~(fG#5lC!^N|IZ00&Pcg<%!-N7LmAF8CmJa+qME zz{M#kknr;54Llkkpug7uAf(a0;yy%|S zW+@2<5N+AYeB26kQJ;-9xb54vb)rA`x&oXAYj4V|vd(22QL8(|q9j4?!y2h=xF2F1 zOm!hLPW|76Am#4|8_j>d#PO zF@#5n47^_X?F}WCfW?LO6&Q5Hf+yys;zo~TXvx7v&!c6(I97{)i; zSd<)##4X0v9)>(wH4N{&8hj8y4#2co<|w#^1|Xm$#-tlF5EmVLYmW zEDY4W7e1=8dp_y22_g!-pfNTEn%e|mm>yJx=`Un0xUyKkzZXlnOsK#cQp6QPsIuQ| zVFfp(2tGcDl_BUMCcdB4*1?;HI-T%80lrI;AICs{gG?B%Y#UAhd8;yE2v6y8nIBaR z^W=Y%Maeowtx!KSok;}#5De4_(;N!G$1y_+RE7o{5>XPMaL?J`$;vVf6bM{zvIIHc zSn|2O+;EH029Ak43K8B>96=1KEkO66B4``x1jL*}Xa@88VlP*OZ^q+OgWwR%xI0(` z`!urJA~Q^oH&)f4M80ZO2Ya!JTlp~KgAK_vpdV8s{m z6+$xtkmN>h(psTh`GM`aixQ!al?3N3OXj-P#Cmoy-c(E2bAJ&C;fd`Js41KfnxIU2 z<2KY3@M;#(|06c(0h7mc@7w%5sw2dquit4vBcdaL0z|S~_>E*Wx-&a$zC4&RDC9lu zk-SN_oY*k%yWMAU9a1>QzxPSNcZ#+4;4H6uAs^(kK+Xn$x=I;#e2}+k$=Qkc56TO@ ziZ4cH5W^?~KR~fw#_DQbInhHngyKedn830ijkz?$=7E++x>sNr2+FK75NkQ*6LZoS zadC0G>m;Dh)83-K*|6;&N7X8 z%ns;>32_s>8Wp*xt=1L`i}UwQ3qSmI8zP4ng$gDkEB==~`1RdlMx7Vk^kjp3GDe?@ zUlqJj_~Ksv?Wmo45f5duUSIs4({jo@tM=DphcCyz>^!3uU4LwiDxX%vS?Bg|_bXPP zIHaVbbX%*l%+POk{q(U_BX(VV^PW>TvX;J%&K_(R2+nP9pPh6o%0VdzchfNx1g&*J z8i6c36OeB0+KLQm9tEbiv~4biSQSM zM3*2gbKS5;v;+@n?(bhVKE8DU@nZMkU6FN5L1WaR^~jlwfScFoF?D@cSs4VQHM)eW zL`tT^L2%XqS!D)BGq#e7Tk;IzQXAJs!2#|G0)S>1gL5$Ay;=S9qXwyZr?zi^RuZw_ zl)IO{zLN%605vRa$hWW3cF|CQonQ zKu$7X)Ym4(M*DBEDB~(6ovVGzN>Q}Un6CWgB2=!oH&{HL;$J8zWvIB+i$#w;a#H7{ zvyzdK z(KUMjzgQ9YPE)d21E(-kYYT$GS{)|Hu#P~p?(d_#gL7A<=ffB~tzh*mb3t`GzYKM~t0 zT){U8kRn1y5+@g?C2ehO^}9UG=6+_Jqs2`5jUUAlqUH}Z7Vp1QCf3>V^D8{LS|pDP zL*~N7{%zeQc-RfJS^@63M~8ooUJN|fTI$-OH8Pvj|O2oqr; z85v4loSztq6@+^8HwMqy8HJ4Y1{fRBA0wy6{h2^U6WJn;v^+T(w{jLK@W3wwM_$Dp zp~@;MfqOqmzt>minSYg*W(7AF><5wYs=in^&`|XE`$?_qlBSZ91yvP*u8%?qP6eDY zJQlNA_~;=nud5HiA>EFpkgR2}?OC;p(bWEhOh%j8t~}x|(-*{L^6!{>!2DKyz!mos z)LNaC9?G9aY}(M(*QfmM|2q7#q=eWD#s5Q^lwvNpC!%qrmF&uF)PSoL8P5#?@tXuEs?7VX8)_QE{gnkW) zuw`l^cBC1Cmx(n;pr37u6zz2-o1#w!1$}??MgP~={&&}$y=RRv#)i)Z*teP0eU+7! zyFvZ6v9;a3ajgptlaP@}z&gJDQ<6Xln(cB{9_ZZ}Yvg7NIF(S>KOSbfcooM8u|)_U zfYL1E(A*$sU+xq2N&r@KE+(wX%-BesH`P zI-w32*;iYARw3nxzzAoT~TPhkd9gf60F>J@-!STVVxQ~WbjxmUZt!#^ro9vr1j0gIsyTT+V zTG~IoOkwGYPcQlV>5a()lk^K1(`%WyG#Q~0RE3@kFp;DF@F^XgLm+5CtcIY{{pSzS zE~0yZQFq7iux~?{4s5h_Pn=LfZ3Ui*#r#AM`qI~WJK+9^_}?`&6joSx0EP+xBk-A| z<=F2Qin-Qjt37Zrn2=oc#C48imgXe8J`R z^(n5%5g`(JIjNfM_AP^9BI_{ZTCi`yyxAc)3lK5hG%tAc@oiSLPtD1BhyxA&(qU9z zU~-CLM~jEYtkvc9_o`Ktm9Hk82m&7$T@-n4qoe+ZCy4?KB30*s-kudoV`Jx^KQJ>~ z1MfxL0TT}%Si*Gl^nN%<(V?E?x+gkb8??sR%|ks=7u@4t7IQ8h9x$WOKCUdJ61^$| zl}?eNv2iNi*4)I?TYRLGUTVIpal~q@qUndLe1t0e#SJq9k;gi^UQ(bJ`1$pLP7{I6E+rEuXztf0ym=@=Hgbm?({p8RzVlMh;Z@a@pAO#ii6) zyPoC$!GUk1KE|bdBZt8ZYE)gIzDwz93G>-+KW0~dq%;OmpSh`KJ_fSg2NVdBJB0(S zVWp69p?2Kim(~Aq0r;!zuf6;B>OminwQ>+SuN_Y^$1T<}iE3)Xn}{F;@B!n3i;y{L zVY-RBrlPeq1~3=g8II*P8wkh8qpTae-MH{~df?hm>n*u?8x`+`DEcq%p43m<@4 zaR)AlP0lU6F}Eu%a)?aYs;w~1L3iy6hThA?mTNAAhVG(J?qzyQJC;nbX*Q*ztv256 zV=Q4L9Ef=0u%7$kRngTZOeUAP!tq1Yx3-A9Su9^au!;S! zg_)q6w6ssQ=Wcaoffq$Zyg^|bmiNCOwJ5&F4pBd{o01Y$-R|8Wc93;9de2{Ub$7li z;5&duLM-$>hA=$#=4b@~bnV6-5V=i3EcTzOs)FhHJ(aS}-d#P%8J2il=l*d0_VAFvp8GI?t3w2ud2?G+RqkCX~yCvxe@iBH0WA= zWiVCaOH+g>!gW6%F}^Xk+1?p*;L_mKMdYT?{aP9i07j7`Q>9p@2F=ou8rjL%Q5E!aj`@ z%4D6oKxcP%IRcAgO}iFKARho5B!Li=0Bsr-ia%JnS)xCNrE~z&3L-IQM{BL(g{JpG zi+(@%jtIy<8q-YQwA)E=Gaa)<>(?0hUH06#=?Km4FZ~h$m!Lm%a^9H^gK5mXdA|5! zDh$I8C}$j7lq3TC^Qi4`uN{q}Xhk=pTIL>Gg*n;vK+LFNMd)zWI!eNQ#)$cX7ww_x%H5uwgXqo|~oF+gP)eP<3U1hG@Jf{f@2@|`+NexF+7=+Sl zuquZ8bryP@-y0h*GUFX4rwE8q_vmQ*NQ7xp!yD|00>L>sJv}KS<2LHw6FNFqOwy~` z+v8yTvd6Xm1U8MObMrUaNMn5_DEimn68qL@Cx-3`FLF2`SkFC^n>k%L`ZCwRU~D+| zbxU!y^R0tTos#Rw`no5+DoagX=J6d9Wgclh_0l!=D(p1zPK)@yP== za~fbJuY{`X79Z+7>Y=rChua}(heOAN+KTXwKo;@8af`2w3h*&Jrpd zCVVO;bBzuy@HoDWJG%!17;sJwp_)Nq29eTL)N*iGa`22+OSpp|0=On1E=~Z7iVnMr z+vbzBv-~I$CAFX8GhZ=DI*z(;(uhWh!0znqOzawP6ad#$qq--S2Z%}6-YH>f0Plm$ zv)Mv{)}z#)UyFT;IFNsEEL8nX5yH`|I5xEYAm4R`kid^7tzE1mKlci%~;-@~f2UQReG$L&q zl=#TuBC8r3H&7v6VkbB-w+BydE8lQW#yuefjvlOr%TY`4I{cmzQ^oMw)!m(+LqMSR z)#((AI~qbNS%K-u*j6d>`X^2V98c;-x4W+QVYWJ`4M`YbtzN_V)?oS_=s!ez^x>Ak zEfk#I#U+)LQ?}Z;uUGU^dQc&SH!Th}rSbqQc*w5H;9R&WV}Pw>gqN;^XY1Ca%b+kr zzU)@GqHigrval?5aZK}cYA;gPm&{@KAa$JYDFj56J>-u4UGS@mEh{g-2K$H;Cr*45 z9!=0qMsZRO5A2jiZ}lL_aJ6eVKDb%oy`oi>m7svAf&TyGKBBi!z|!>bb!Hw$b1pO` z?k+CeoP2y=AU?yhyR+vIgL4C!i-PLV?QU#LVzB)~`@(_(bVbl5l0i5^%Tr(fXD=Ix zYy&93O3!9V|Kl5YvMnQ zA>IAZdhy1g>CG4i!Rw>A+uKVhFmHpl{J6@s7R<@MH#P0koXcz(F00HgA!)u*#?(>_~enV%aJ&;+Us>cZ`R zcYSq+sm9!(r)$~yViU#vK+8d-KnIuDw?-(c5_MA3(@#%~QhUL-$}KX+{vz0)cISZN zhgq&U>LX;$>`_Us>?m~gNFJY1bT-4N3<9#^zITfaYVq}hgMr!ymC7mQJE1_uXE zdE)KEqd5n|zKH1(4DNuJVQ<*%L)x48mUycog!y5viee`V%zK;|MkXd3%A_6|Aq52m zQubiCAMn(2aYJ{@N?&0QJy1>E=nj+Uflz`m8|Nw|YCdMTDrBuBKM z_1Pxei zRGK!O{KuV`JpdjA&R*AY-nj?o2M+#ZITl8FXs_NH4QtsCA$6laO1XX1=mGbmq=o^}RjXEIz@v_!M!{=oOdja)&4C3& z>2VVKWW@r9;lc_sId&3#AEAiveS8eeTo!)*uNa_$7`+n`x3aRblrumkw=u7PLM{9{3hcub1590*UW6GOR9oZY3q9LIK80ND9Djv2pIw z?~knbbTwC3*Lw^0*+xLa)zT?@0I%=(df7$;;|{!F=w6l!)S4ZFARwQtjJxM2!FTD( zrpvGRh8$B3;xf+1*r|FQekb1ydFln(j9x0%XRH)sHyIfjE*Nnb7&Jd9T|Bj&Y>0tU z2)(<@c_twT3YGh_1iFj}1E<5Q7c~>=h%pPh~5-x+A_tbK6$R00I-uMHc- zTSkBfqdkCu;1eVzq7jFZquuO2+#%#X)%(wN1|u>9z5je>yHg!E9`r`YQt_hoKn#+Y zk&#h%3B^D#h9}tn7*Qvzssj_18tEOHe!#(i`0h30Ny*<9`u=XCrdX)t`-#M!u9%hC zgsm3?+RO|uKI_un;In__WL3{|3cFtV1-^99B_2u`Z{PZyr7MS>hv#xWLsdH@7`A-l z8xM4Sq$o;rf8oHIY;t3roHC{?gfJ{+WF(vXFwl@&-iYnSNkEo?inLD5l)rxuhEgBm zlo_B(q6n2~^St)+rx83Z`xy)))!##xD(AN-i^WtRfYV{qle_Vg%@}bsg$LEWJ-+ zSWCV<&BMEQT^zB?#>*@+Fnz}Mx>2fM9Tr&wfNphukP*mYpROIJ0I6c(JCQ4ZPJoUj z8Rm=*4r)MZjHaY$y6FrW%!y#d#VgxRzBr+;UxkW`Z}O`IJxBDH571*Z!v_=OQS9gJ z93F0!Sfh6~Ht8)H(vSSA#&|k8Elm{M9OM}w?8BaEFv#4#9rkMnvS4(C(iIcWORGMk z9>yZ3Z`&v15JlHSNGS-c=&0(TT78Q1 zq3;t5JO~{VSzeGC`(O6roW$>04oXVeiXTZtc&NV=#FF|IJ2f#h1@n)07Bzc!Pfw%W zC2a&_?6n01cn@{RWR>Fb9rVs7e)E>gb5)9LV1L192j)|MYX#;b)yj#J7XT!pMF(gI zms5Fpc?WCe)0(h^vqByK=mxqX{Jemm5rAJ>?rMK;uNnsAc}|@)Scom{0+OYcpWQ78&M~d!$ zHWHv`%NCX68bTHJd}W*xXFVN#eSL!&&G;<8Qd(d;Bz>>C0+V69v61{c(6mFGua{n8e~umi93J)4vZHAQ1#@=lQg5w6RCQH%1sI_l;OlgN7id%3$D-pfp43? zo#J#YWu=c3Sunh<=VALR>B3bWzQHsKVB=_S2T0C$r`Ii?; zS9p-RV!a2qp=^3AcS#gJV!Mz(y2&{Pb#R?p9*63)v@~YC6xc!VNXC($csavh2?c%5 zMINAw*@vFp9x_g-YiPKKCK5gZ=q)awOIasTZnsS}DXb&7a`ge~d(>P{ISqXCHwU;E zaF%z|i7MC`4W!)Z}|laQrr2kV(bQ>p`{(bRxN;K6ReouLifY7}9ssAzU=8#>n z{3}z^v+6n=sN-dQ=63DSdo2h@Z796v3M5i+>o1@|({w@|;1#hctJZhGx zmN_<+M*Bs%MU6z=;$egvAh#G*)b9ym873~BoA$ekp&@gQL+1jQ75m23PFSf&9EnjH z5p=|J%nZ#1S(#0g^jHIkTk^J=GC1vxq6B&9mm9@6b$_rm7>$6#=~Q*{VEUVGTLB;K zdl*!cZB^w%kan}Qw{~P@L|>`ig)ECtX53) zIeJ#M_+-aY0z8b3jmZs0PgT!@tF1>`^cv1y5kHMOetc}q?C0ByJ4;4Rog^*!*RP`J zI>FW>#6!F!>bPD+6+X>};++^kg_nw-+K%2Fcq+MGY)Ic{VE$SJ7knLaS!gvt=IBZ* z94Y91vxLEp1=@gcPT?G*RN`HV+)J!L|ES6$=0 z(O$^Nz+gR*4ZI8c{chkWgFSHrDkpNugWm^Uq--de{CWTYZpX+7fnCXfGX2bi)UI+I zOz=K(G&bxYorB`SkJ=;0kGubB8gK=3XS~_`R`!4&qVT=QNOg>`(9L7b1H`~b5X&KG zCOj4UT{a#+dg?Non+?$8emftT^99;QFd#d+ES~_kC>)%mq@D?Gb-3{TrH;N2JRKN; z(7GZDtIp&Dx40I(Uf1wUNbtLBu?M~PZF-7X%lt^;sH|&03qU4R!0ZA7{*?jln}P5P z_>3Kju5Cf&(PJ=6-xfL%8<_rZ&$_uJz~gwq;!R8a>K6i8hE444cjB26lb&@npV5e< z%SAOr!N^V)*&mz)40qU2)Fa@?p>Mx7e=ah)(FWB!jvm<$YAAVBeEatEsQDwt`UVF- zL5_r*a5hc!$Ir(tXJ^dbt>TnJm2Z~|M1ojoUMo6=~=cZxhxD~3BYTwl4WGtek!;B0A zpw`x6<5XjJLwE`T+a`t*5GU?in89BLwGnUNi%PZGK)eqIRqIW)X)oc6+MxOP1+sg6 zOCU>b47w})V1r;0%6oK$#8K$ybO|Oyq~4X1l0tj}@`L$Qt5_h$(dW;f_w2Yr@8s@Y z(m9))lCoJur1A}?;nAZZ-Mi5V!fNXS32Nfv&(ZXwoIu5e!5Uf+(ak63I`eb;@3 zK9oiI?3b-^KJT#xU~psIm9T{UM6>!VnBOKcL0W6|8z2t%(9Giv!RxgyLOrE1zF~l9 zK5s-sl#C2>a&g_mLJLGx)KypoNQ^iinbRS>lPCt03{NkwbGtU;3LU;X;1h_mAGizH zrlEW@6cb8H+MMH#CvAq6G$Xs{+XXpI#rq)oqZ)!cUztO$dDahZ-ji@C#p_)- z_|pnqKG2stAS5BVgDF+iy*u0J3u%C6XRBIT7GAs`Ahja2;Anx-tQx!VgvQINH6lqN ziaE)zOvHbe*%MDchWm@fMK{zQY>eK);txQeX!-9DYG~u*QbH9myk#b*IeL586TsBI1$DC$G<|SANgLKHZ$jyA|dB)7?Bk70=}W_`nE23IkfoH<%NV z)(A|4+F!2s_>WaE$`Y{s_~H zV9;TeI&x@ObvM;(F$Fn*c!Ypew^6|- z&%AB`MQs!i0xB-a18d>NyldCk!&xm5^x>xoGzGl7v)5|qwf>E5;B%y_r*}PG3}zN( ztR;Azfk1q6+iB#3Jxst34d9Z7(2c}FmUqHV!r9&ZLv8Ko!&|C494G=S?}}*lLydsu zE^nYlD2OFK!8f-J&IQB!0!nAK@^z}smVTf;6+)UyZ$_FJPH^je>xru1N>f`EeI*i&4y{#q&6N{J%0vDS|i3`u~4Q)jo3 zjcy9hJI!t)7(ha z%D-3*_N>Qd#hq6>JDLf z`NdZWUN}`&2(J+>+1c6P$ff_7*@2{n7oU}nuhru6t5e(Y3v@Vw@Lq%43w?SI-ZoST z(%Wz7Vu%RH!4QuKS#fbHNl^r_;HOU-&{usXe}EP40BKOLgE{PZGT{_zW&*Fc4(MV= z;sx$3_BE&7^kfuLVF$SUvxmXp^8AJGmjfE%JUs^Y=-n6Vw+!r7zHvYoq{<&R0M7^bcnr`i6sXS&3bI~unbJ3CGJ!(aqPjXrO{B0w*ERjKo*(LDPg@RZ|;BkL*73(+)6><^pA9*llz+ zXG66(PBjx6FDCG-y3>{_13M)02nqqhfU0*eaiV33@V$e)v7{qCo z?=-7F?|Ukc_WNO_2g{RTn-y#|35wgcUDT0^ilO?b%Ar$n0&xJ;-AZ4;of^MJR^zMf zUT$cwYQR^GvWz1nB&<+!3qNIC(^Ef<(+ZKXGma#^d_9mj>9!OVU`i8oC&Dz^cd()o25PQ997Rv%bv6Br?8J3{XU*>wOZ&NB8lC4{Te(4y61C9g#h7z#Ay4_Wdsm! z#}|x3;ZH^m!!6XZsNl*0j!H&yF$t**t;$%@U9qC2yNBw3G%ufZe zbaA2<$%H-;UV)^exL2DmFZ@D`31e_!xE#W9$SPXVO&Tz_|6;cQIObz7I5n2PIQo@? zQqnH?6bKcekYniT=y@5~0Ks5A+Ps1hJ_^1l2B^7@%ixE;^t^abC&!93x-A%%}a&eJD;dC)msR|DG%&I!5T164lF&qJH zo&i1Mj9W#?(##nUCu$-kO9#egfCfQ2-MixBZ2U^X6U1jMpNbnQ5eLwE3pKG6Xxe}P z1^Lt%gti1^dsEq_jy0}sw%W_R6bC%1&2Pbe{w-2(5E3TI$%v`IRS1BdH~32wlE>po zAbo%Zj@!N&qI+=X$zJA(iGnmaEyaiMc!KK%YRh*G4U;c!5tRrEpiCTUH6b<$iRS}D zLz-BHhf;PO(@FdasLY9pH<2dcHY2reYm~rU(V{W14NSkG`EEx)%Aj8C(R!e5$RK2- z1CS87kzI4f75r?~Uqk&RmlMfg$N#T@1mFq|05)r2#qIm{_La~pn0}E04Dd5vhCNX| z6D7dfh#L-`bNbwqngv^_#~|h5uLDkm@g@p7(##Pod7GIKzKISM9JWc0GsVl3&?4vK zR>D!FOepLa3<$-gu_(WNgH{=h6p+#mw5#Y8;8a~bpV69&3?0m?YlA5MVxe{!gs)FGm`QN0<9(n zNu~;~R?o=|7~H-pD*DA*ArYL3!4<6RrOP*Vyo~qsqVBp7>vbIjoQ@X~`bGRe?sygb zp5QyeWb+H6G0*E~Jt%rn&l9>3HnLru$+tsev!A%!yTFDg$C}DEd2E9ZXh(`z=s;H0 z?wnaIj+W{aJze=tzjP)?vLA3}}XSgPG!wz1{;GqBgSo~(Z0cJ?l zxx8f#B0l*vLoqQi^q4*pBZ{MjrvLx1B=K+FUx;ZiDXhMJ{W=ly?>CV%DQ!L{MU{f; z`TB>dDl)%9tny*|?{W`p3n+dn_UZG(wWeRoWDpgP5;lNoDL)q%*ZivgE{m-!YPXLZ zwdoKHrcg;>l-SkVE4owie}8ZQ1r>awc+E-gD2IE8@?+^0^)cUY`-=eV^>VCUuW{O%J;u^nS zO4~b;NT&cn3_D^B;zSrYO#l1XysH8?1C~yv>S-6ZjV013G}>fq|6N!hV^|+w-$d#E z`!r6PM+G4r(kC%h>jrNKEenF?ZN2|{d(NyiQ;eOQz609BJ2btR)?gO=bMQcl*ryGS zrUQM^zOO}c>h47JxVL=x@^&ev{zd8=-;0iN>wn9WhrYZSs3Bz90B~Ly$I{M}k4^KB zF$G{@Z>nBi+|+b-vyJ literal 0 HcmV?d00001 diff --git a/Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg b/Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg new file mode 100644 index 000000000000..4b4014fda770 --- /dev/null +++ b/Documentation/RCU/Design/Requirements/GPpartitionReaders1.svg @@ -0,0 +1,374 @@ + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + synchronize_rcu() + + + + + + + WRITE_ONCE(a, 1); + WRITE_ONCE(b, 1); + r1 = READ_ONCE(a); + WRITE_ONCE(c, 1); + r2 = READ_ONCE(b); + r3 = READ_ONCE(c); + thread0() + thread1() + thread2() + + + + rcu_read_lock(); + rcu_read_lock(); + rcu_read_unlock(); + rcu_read_unlock(); + + QS + + QS + + + QS + + diff --git a/Documentation/RCU/Design/Requirements/RCUApplicability.svg b/Documentation/RCU/Design/Requirements/RCUApplicability.svg new file mode 100644 index 000000000000..ebcbeee391ed --- /dev/null +++ b/Documentation/RCU/Design/Requirements/RCUApplicability.svg @@ -0,0 +1,237 @@ + + + + + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + Read-Mostly, Stale & + + Inconsistent Data OK + + (RCU Works Great!!!) + + (RCU Works Well) + + Read-Mostly, Need Consistent Data + + Read-Write, Need Consistent Data + + Update-Mostly, Need Consistent Data + + (RCU Might Be OK...) + + (1) Provide Existence Guarantees For Update-Friendly Mechanisms + + (2) Provide Wait-Free Read-Side Primitives for Real-Time Use) + + (RCU is Very Unlikely to be the Right Tool For The Job, But it Can: + + diff --git a/Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg b/Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg new file mode 100644 index 000000000000..48cd1623d4d4 --- /dev/null +++ b/Documentation/RCU/Design/Requirements/ReadersPartitionGP1.svg @@ -0,0 +1,639 @@ + + + + + + + + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + + synchronize_rcu() + + + + + + + WRITE_ONCE(a, 1); + WRITE_ONCE(b, 1); + r1 = READ_ONCE(a); + WRITE_ONCE(c, 1); + WRITE_ONCE(d, 1); + r2 = READ_ONCE(c); + thread0() + thread1() + thread2() + + + + rcu_read_lock(); + rcu_read_lock(); + rcu_read_unlock(); + rcu_read_unlock(); + + QS + + QS + + + QS + + + + synchronize_rcu() + + + + + + + r3 = READ_ONCE(d); + WRITE_ONCE(e, 1); + + QS + r4 = READ_ONCE(b); + r5 = READ_ONCE(e); + rcu_read_lock(); + rcu_read_unlock(); + QS + + QS + + QS + + thread3() + thread4() + + diff --git a/Documentation/RCU/Design/Requirements/Requirements.html b/Documentation/RCU/Design/Requirements/Requirements.html new file mode 100644 index 000000000000..95ef7b5de951 --- /dev/null +++ b/Documentation/RCU/Design/Requirements/Requirements.html @@ -0,0 +1,2799 @@ + + + + + A Tour Through RCU's Requirements [LWN.net] + + +

A Tour Through RCU's Requirements

+ +

Copyright IBM Corporation, 2015

+

Author: Paul E. McKenney

+

The initial version of this document appeared in the +LWN articles +here, +here, and +here.

+ +

Introduction

+ +

+Read-copy update (RCU) is a synchronization mechanism that is often +used as a replacement for reader-writer locking. +RCU is unusual in that updaters do not block readers, +which means that RCU's read-side primitives can be exceedingly fast +and scalable. +In addition, updaters can make useful forward progress concurrently +with readers. +However, all this concurrency between RCU readers and updaters does raise +the question of exactly what RCU readers are doing, which in turn +raises the question of exactly what RCU's requirements are. + +

+This document therefore summarizes RCU's requirements, and can be thought +of as an informal, high-level specification for RCU. +It is important to understand that RCU's specification is primarily +empirical in nature; +in fact, I learned about many of these requirements the hard way. +This situation might cause some consternation, however, not only +has this learning process been a lot of fun, but it has also been +a great privilege to work with so many people willing to apply +technologies in interesting new ways. + +

+All that aside, here are the categories of currently known RCU requirements: +

+ +
    +
  1. + Fundamental Requirements +
  2. Fundamental Non-Requirements +
  3. + Parallelism Facts of Life +
  4. + Quality-of-Implementation Requirements +
  5. + Linux Kernel Complications +
  6. + Software-Engineering Requirements +
  7. + Other RCU Flavors +
  8. + Possible Future Changes +
+ +

+This is followed by a summary, +which is in turn followed by the inevitable +answers to the quick quizzes. + +

Fundamental Requirements

+ +

+RCU's fundamental requirements are the closest thing RCU has to hard +mathematical requirements. +These are: + +

    +
  1. + Grace-Period Guarantee +
  2. + Publish-Subscribe Guarantee +
  3. + RCU Primitives Guaranteed to Execute Unconditionally +
  4. + Guaranteed Read-to-Write Upgrade +
+ +

Grace-Period Guarantee

+ +

+RCU's grace-period guarantee is unusual in being premeditated: +Jack Slingwine and I had this guarantee firmly in mind when we started +work on RCU (then called “rclock”) in the early 1990s. +That said, the past two decades of experience with RCU have produced +a much more detailed understanding of this guarantee. + +

+RCU's grace-period guarantee allows updaters to wait for the completion +of all pre-existing RCU read-side critical sections. +An RCU read-side critical section +begins with the marker rcu_read_lock() and ends with +the marker rcu_read_unlock(). +These markers may be nested, and RCU treats a nested set as one +big RCU read-side critical section. +Production-quality implementations of rcu_read_lock() and +rcu_read_unlock() are extremely lightweight, and in +fact have exactly zero overhead in Linux kernels built for production +use with CONFIG_PREEMPT=n. + +

+This guarantee allows ordering to be enforced with extremely low +overhead to readers, for example: + +

+
+ 1 int x, y;
+ 2
+ 3 void thread0(void)
+ 4 {
+ 5   rcu_read_lock();
+ 6   r1 = READ_ONCE(x);
+ 7   r2 = READ_ONCE(y);
+ 8   rcu_read_unlock();
+ 9 }
+10
+11 void thread1(void)
+12 {
+13   WRITE_ONCE(x, 1);
+14   synchronize_rcu();
+15   WRITE_ONCE(y, 1);
+16 }
+
+
+ +

+Because the synchronize_rcu() on line 14 waits for +all pre-existing readers, any instance of thread0() that +loads a value of zero from x must complete before +thread1() stores to y, so that instance must +also load a value of zero from y. +Similarly, any instance of thread0() that loads a value of +one from y must have started after the +synchronize_rcu() started, and must therefore also load +a value of one from x. +Therefore, the outcome: +

+
+(r1 == 0 && r2 == 1)
+
+
+cannot happen. + +

Quick Quiz 1: +Wait a minute! +You said that updaters can make useful forward progress concurrently +with readers, but pre-existing readers will block +synchronize_rcu()!!! +Just who are you trying to fool??? +
Answer + +

+This scenario resembles one of the first uses of RCU in +DYNIX/ptx, +which managed a distributed lock manager's transition into +a state suitable for handling recovery from node failure, +more or less as follows: + +

+
+ 1 #define STATE_NORMAL        0
+ 2 #define STATE_WANT_RECOVERY 1
+ 3 #define STATE_RECOVERING    2
+ 4 #define STATE_WANT_NORMAL   3
+ 5
+ 6 int state = STATE_NORMAL;
+ 7
+ 8 void do_something_dlm(void)
+ 9 {
+10   int state_snap;
+11
+12   rcu_read_lock();
+13   state_snap = READ_ONCE(state);
+14   if (state_snap == STATE_NORMAL)
+15     do_something();
+16   else
+17     do_something_carefully();
+18   rcu_read_unlock();
+19 }
+20
+21 void start_recovery(void)
+22 {
+23   WRITE_ONCE(state, STATE_WANT_RECOVERY);
+24   synchronize_rcu();
+25   WRITE_ONCE(state, STATE_RECOVERING);
+26   recovery();
+27   WRITE_ONCE(state, STATE_WANT_NORMAL);
+28   synchronize_rcu();
+29   WRITE_ONCE(state, STATE_NORMAL);
+30 }
+
+
+ +

+The RCU read-side critical section in do_something_dlm() +works with the synchronize_rcu() in start_recovery() +to guarantee that do_something() never runs concurrently +with recovery(), but with little or no synchronization +overhead in do_something_dlm(). + +

Quick Quiz 2: +Why is the synchronize_rcu() on line 28 needed? +
Answer + +

+In order to avoid fatal problems such as deadlocks, +an RCU read-side critical section must not contain calls to +synchronize_rcu(). +Similarly, an RCU read-side critical section must not +contain anything that waits, directly or indirectly, on completion of +an invocation of synchronize_rcu(). + +

+Although RCU's grace-period guarantee is useful in and of itself, with +quite a few use cases, +it would be good to be able to use RCU to coordinate read-side +access to linked data structures. +For this, the grace-period guarantee is not sufficient, as can +be seen in function add_gp_buggy() below. +We will look at the reader's code later, but in the meantime, just think of +the reader as locklessly picking up the gp pointer, +and, if the value loaded is non-NULL, locklessly accessing the +->a and ->b fields. + +

+
+ 1 bool add_gp_buggy(int a, int b)
+ 2 {
+ 3   p = kmalloc(sizeof(*p), GFP_KERNEL);
+ 4   if (!p)
+ 5     return -ENOMEM;
+ 6   spin_lock(&gp_lock);
+ 7   if (rcu_access_pointer(gp)) {
+ 8     spin_unlock(&gp_lock);
+ 9     return false;
+10   }
+11   p->a = a;
+12   p->b = a;
+13   gp = p; /* ORDERING BUG */
+14   spin_unlock(&gp_lock);
+15   return true;
+16 }
+
+
+ +

+The problem is that both the compiler and weakly ordered CPUs are within +their rights to reorder this code as follows: + +

+
+ 1 bool add_gp_buggy_optimized(int a, int b)
+ 2 {
+ 3   p = kmalloc(sizeof(*p), GFP_KERNEL);
+ 4   if (!p)
+ 5     return -ENOMEM;
+ 6   spin_lock(&gp_lock);
+ 7   if (rcu_access_pointer(gp)) {
+ 8     spin_unlock(&gp_lock);
+ 9     return false;
+10   }
+11   gp = p; /* ORDERING BUG */
+12   p->a = a;
+13   p->b = a;
+14   spin_unlock(&gp_lock);
+15   return true;
+16 }
+
+
+ +

+If an RCU reader fetches gp just after +add_gp_buggy_optimized executes line 11, +it will see garbage in the ->a and ->b +fields. +And this is but one of many ways in which compiler and hardware optimizations +could cause trouble. +Therefore, we clearly need some way to prevent the compiler and the CPU from +reordering in this manner, which brings us to the publish-subscribe +guarantee discussed in the next section. + +

Publish/Subscribe Guarantee

+ +

+RCU's publish-subscribe guarantee allows data to be inserted +into a linked data structure without disrupting RCU readers. +The updater uses rcu_assign_pointer() to insert the +new data, and readers use rcu_dereference() to +access data, whether new or old. +The following shows an example of insertion: + +

+
+ 1 bool add_gp(int a, int b)
+ 2 {
+ 3   p = kmalloc(sizeof(*p), GFP_KERNEL);
+ 4   if (!p)
+ 5     return -ENOMEM;
+ 6   spin_lock(&gp_lock);
+ 7   if (rcu_access_pointer(gp)) {
+ 8     spin_unlock(&gp_lock);
+ 9     return false;
+10   }
+11   p->a = a;
+12   p->b = a;
+13   rcu_assign_pointer(gp, p);
+14   spin_unlock(&gp_lock);
+15   return true;
+16 }
+
+
+ +

+The rcu_assign_pointer() on line 13 is conceptually +equivalent to a simple assignment statement, but also guarantees +that its assignment will +happen after the two assignments in lines 11 and 12, +similar to the C11 memory_order_release store operation. +It also prevents any number of “interesting” compiler +optimizations, for example, the use of gp as a scratch +location immediately preceding the assignment. + +

Quick Quiz 3: +But rcu_assign_pointer() does nothing to prevent the +two assignments to p->a and p->b +from being reordered. +Can't that also cause problems? +
Answer + +

+It is tempting to assume that the reader need not do anything special +to control its accesses to the RCU-protected data, +as shown in do_something_gp_buggy() below: + +

+
+ 1 bool do_something_gp_buggy(void)
+ 2 {
+ 3   rcu_read_lock();
+ 4   p = gp;  /* OPTIMIZATIONS GALORE!!! */
+ 5   if (p) {
+ 6     do_something(p->a, p->b);
+ 7     rcu_read_unlock();
+ 8     return true;
+ 9   }
+10   rcu_read_unlock();
+11   return false;
+12 }
+
+
+ +

+However, this temptation must be resisted because there are a +surprisingly large number of ways that the compiler +(to say nothing of +DEC Alpha CPUs) +can trip this code up. +For but one example, if the compiler were short of registers, it +might choose to refetch from gp rather than keeping +a separate copy in p as follows: + +

+
+ 1 bool do_something_gp_buggy_optimized(void)
+ 2 {
+ 3   rcu_read_lock();
+ 4   if (gp) { /* OPTIMIZATIONS GALORE!!! */
+ 5     do_something(gp->a, gp->b);
+ 6     rcu_read_unlock();
+ 7     return true;
+ 8   }
+ 9   rcu_read_unlock();
+10   return false;
+11 }
+
+
+ +

+If this function ran concurrently with a series of updates that +replaced the current structure with a new one, +the fetches of gp->a +and gp->b might well come from two different structures, +which could cause serious confusion. +To prevent this (and much else besides), do_something_gp() uses +rcu_dereference() to fetch from gp: + +

+
+ 1 bool do_something_gp(void)
+ 2 {
+ 3   rcu_read_lock();
+ 4   p = rcu_dereference(gp);
+ 5   if (p) {
+ 6     do_something(p->a, p->b);
+ 7     rcu_read_unlock();
+ 8     return true;
+ 9   }
+10   rcu_read_unlock();
+11   return false;
+12 }
+
+
+ +

+The rcu_dereference() uses volatile casts and (for DEC Alpha) +memory barriers in the Linux kernel. +Should a +high-quality implementation of C11 memory_order_consume [PDF] +ever appear, then rcu_dereference() could be implemented +as a memory_order_consume load. +Regardless of the exact implementation, a pointer fetched by +rcu_dereference() may not be used outside of the +outermost RCU read-side critical section containing that +rcu_dereference(), unless protection of +the corresponding data element has been passed from RCU to some +other synchronization mechanism, most commonly locking or +reference counting. + +

+In short, updaters use rcu_assign_pointer() and readers +use rcu_dereference(), and these two RCU API elements +work together to ensure that readers have a consistent view of +newly added data elements. + +

+Of course, it is also necessary to remove elements from RCU-protected +data structures, for example, using the following process: + +

    +
  1. Remove the data element from the enclosing structure. +
  2. Wait for all pre-existing RCU read-side critical sections + to complete (because only pre-existing readers can possibly have + a reference to the newly removed data element). +
  3. At this point, only the updater has a reference to the + newly removed data element, so it can safely reclaim + the data element, for example, by passing it to kfree(). +
+ +This process is implemented by remove_gp_synchronous(): + +
+
+ 1 bool remove_gp_synchronous(void)
+ 2 {
+ 3   struct foo *p;
+ 4
+ 5   spin_lock(&gp_lock);
+ 6   p = rcu_access_pointer(gp);
+ 7   if (!p) {
+ 8     spin_unlock(&gp_lock);
+ 9     return false;
+10   }
+11   rcu_assign_pointer(gp, NULL);
+12   spin_unlock(&gp_lock);
+13   synchronize_rcu();
+14   kfree(p);
+15   return true;
+16 }
+
+
+ +

+This function is straightforward, with line 13 waiting for a grace +period before line 14 frees the old data element. +This waiting ensures that readers will reach line 7 of +do_something_gp() before the data element referenced by +p is freed. +The rcu_access_pointer() on line 6 is similar to +rcu_dereference(), except that: + +

    +
  1. The value returned by rcu_access_pointer() + cannot be dereferenced. + If you want to access the value pointed to as well as + the pointer itself, use rcu_dereference() + instead of rcu_access_pointer(). +
  2. The call to rcu_access_pointer() need not be + protected. + In contrast, rcu_dereference() must either be + within an RCU read-side critical section or in a code + segment where the pointer cannot change, for example, in + code protected by the corresponding update-side lock. +
+ +

Quick Quiz 4: +Without the rcu_dereference() or the +rcu_access_pointer(), what destructive optimizations +might the compiler make use of? +
Answer + +

+This simple linked-data-structure scenario clearly demonstrates the need +for RCU's stringent memory-ordering guarantees on systems with more than +one CPU: + +

    +
  1. Each CPU that has an RCU read-side critical section that + begins before synchronize_rcu() starts is + guaranteed to execute a full memory barrier between the time + that the RCU read-side critical section ends and the time that + synchronize_rcu() returns. + Without this guarantee, a pre-existing RCU read-side critical section + might hold a reference to the newly removed struct foo + after the kfree() on line 14 of + remove_gp_synchronous(). +
  2. Each CPU that has an RCU read-side critical section that ends + after synchronize_rcu() returns is guaranteed + to execute a full memory barrier between the time that + synchronize_rcu() begins and the time that the RCU + read-side critical section begins. + Without this guarantee, a later RCU read-side critical section + running after the kfree() on line 14 of + remove_gp_synchronous() might + later run do_something_gp() and find the + newly deleted struct foo. +
  3. If the task invoking synchronize_rcu() remains + on a given CPU, then that CPU is guaranteed to execute a full + memory barrier sometime during the execution of + synchronize_rcu(). + This guarantee ensures that the kfree() on + line 14 of remove_gp_synchronous() really does + execute after the removal on line 11. +
  4. If the task invoking synchronize_rcu() migrates + among a group of CPUs during that invocation, then each of the + CPUs in that group is guaranteed to execute a full memory barrier + sometime during the execution of synchronize_rcu(). + This guarantee also ensures that the kfree() on + line 14 of remove_gp_synchronous() really does + execute after the removal on + line 11, but also in the case where the thread executing the + synchronize_rcu() migrates in the meantime. +
+ +

Quick Quiz 5: +Given that multiple CPUs can start RCU read-side critical sections +at any time without any ordering whatsoever, how can RCU possibly tell whether +or not a given RCU read-side critical section starts before a +given instance of synchronize_rcu()? +
Answer + +

Quick Quiz 6: +The first and second guarantees require unbelievably strict ordering! +Are all these memory barriers really required? +
Answer + +

+In short, RCU's publish-subscribe guarantee is provided by the combination +of rcu_assign_pointer() and rcu_dereference(). +This guarantee allows data elements to be safely added to RCU-protected +linked data structures without disrupting RCU readers. +This guarantee can be used in combination with the grace-period +guarantee to also allow data elements to be removed from RCU-protected +linked data structures, again without disrupting RCU readers. + +

+This guarantee was only partially premeditated. +DYNIX/ptx used an explicit memory barrier for publication, but had nothing +resembling rcu_dereference() for subscription, nor did it +have anything resembling the smp_read_barrier_depends() +that was later subsumed into rcu_dereference(). +The need for these operations made itself known quite suddenly at a +late-1990s meeting with the DEC Alpha architects, back in the days when +DEC was still a free-standing company. +It took the Alpha architects a good hour to convince me that any sort +of barrier would ever be needed, and it then took me a good two hours +to convince them that their documentation did not make this point clear. +More recent work with the C and C++ standards committees have provided +much education on tricks and traps from the compiler. +In short, compilers were much less tricky in the early 1990s, but in +2015, don't even think about omitting rcu_dereference()! + +

RCU Primitives Guaranteed to Execute Unconditionally

+ +

+The common-case RCU primitives are unconditional. +They are invoked, they do their job, and they return, with no possibility +of error, and no need to retry. +This is a key RCU design philosophy. + +

+However, this philosophy is pragmatic rather than pigheaded. +If someone comes up with a good justification for a particular conditional +RCU primitive, it might well be implemented and added. +After all, this guarantee was reverse-engineered, not premeditated. +The unconditional nature of the RCU primitives was initially an +accident of implementation, and later experience with synchronization +primitives with conditional primitives caused me to elevate this +accident to a guarantee. +Therefore, the justification for adding a conditional primitive to +RCU would need to be based on detailed and compelling use cases. + +

Guaranteed Read-to-Write Upgrade

+ +

+As far as RCU is concerned, it is always possible to carry out an +update within an RCU read-side critical section. +For example, that RCU read-side critical section might search for +a given data element, and then might acquire the update-side +spinlock in order to update that element, all while remaining +in that RCU read-side critical section. +Of course, it is necessary to exit the RCU read-side critical section +before invoking synchronize_rcu(), however, this +inconvenience can be avoided through use of the +call_rcu() and kfree_rcu() API members +described later in this document. + +

Quick Quiz 7: +But how does the upgrade-to-write operation exclude other readers? +
Answer + +

+This guarantee allows lookup code to be shared between read-side +and update-side code, and was premeditated, appearing in the earliest +DYNIX/ptx RCU documentation. + +

Fundamental Non-Requirements

+ +

+RCU provides extremely lightweight readers, and its read-side guarantees, +though quite useful, are correspondingly lightweight. +It is therefore all too easy to assume that RCU is guaranteeing more +than it really is. +Of course, the list of things that RCU does not guarantee is infinitely +long, however, the following sections list a few non-guarantees that +have caused confusion. +Except where otherwise noted, these non-guarantees were premeditated. + +

    +
  1. + Readers Impose Minimal Ordering +
  2. + Readers Do Not Exclude Updaters +
  3. + Updaters Only Wait For Old Readers +
  4. + Grace Periods Don't Partition Read-Side Critical Sections +
  5. + Read-Side Critical Sections Don't Partition Grace Periods +
  6. + Disabling Preemption Does Not Block Grace Periods +
+ +

Readers Impose Minimal Ordering

+ +

+Reader-side markers such as rcu_read_lock() and +rcu_read_unlock() provide absolutely no ordering guarantees +except through their interaction with the grace-period APIs such as +synchronize_rcu(). +To see this, consider the following pair of threads: + +

+
+ 1 void thread0(void)
+ 2 {
+ 3   rcu_read_lock();
+ 4   WRITE_ONCE(x, 1);
+ 5   rcu_read_unlock();
+ 6   rcu_read_lock();
+ 7   WRITE_ONCE(y, 1);
+ 8   rcu_read_unlock();
+ 9 }
+10
+11 void thread1(void)
+12 {
+13   rcu_read_lock();
+14   r1 = READ_ONCE(y);
+15   rcu_read_unlock();
+16   rcu_read_lock();
+17   r2 = READ_ONCE(x);
+18   rcu_read_unlock();
+19 }
+
+
+ +

+After thread0() and thread1() execute +concurrently, it is quite possible to have + +

+
+(r1 == 1 && r2 == 0)
+
+
+ +(that is, y appears to have been assigned before x), +which would not be possible if rcu_read_lock() and +rcu_read_unlock() had much in the way of ordering +properties. +But they do not, so the CPU is within its rights +to do significant reordering. +This is by design: Any significant ordering constraints would slow down +these fast-path APIs. + +

Quick Quiz 8: +Can't the compiler also reorder this code? +
Answer + +

Readers Do Not Exclude Updaters

+ +

+Neither rcu_read_lock() nor rcu_read_unlock() +exclude updates. +All they do is to prevent grace periods from ending. +The following example illustrates this: + +

+
+ 1 void thread0(void)
+ 2 {
+ 3   rcu_read_lock();
+ 4   r1 = READ_ONCE(y);
+ 5   if (r1) {
+ 6     do_something_with_nonzero_x();
+ 7     r2 = READ_ONCE(x);
+ 8     WARN_ON(!r2); /* BUG!!! */
+ 9   }
+10   rcu_read_unlock();
+11 }
+12
+13 void thread1(void)
+14 {
+15   spin_lock(&my_lock);
+16   WRITE_ONCE(x, 1);
+17   WRITE_ONCE(y, 1);
+18   spin_unlock(&my_lock);
+19 }
+
+
+ +

+If the thread0() function's rcu_read_lock() +excluded the thread1() function's update, +the WARN_ON() could never fire. +But the fact is that rcu_read_lock() does not exclude +much of anything aside from subsequent grace periods, of which +thread1() has none, so the +WARN_ON() can and does fire. + +

Updaters Only Wait For Old Readers

+ +

+It might be tempting to assume that after synchronize_rcu() +completes, there are no readers executing. +This temptation must be avoided because +new readers can start immediately after synchronize_rcu() +starts, and synchronize_rcu() is under no +obligation to wait for these new readers. + +

Quick Quiz 9: +Suppose that synchronize_rcu() did wait until all readers had completed. +Would the updater be able to rely on this? +
Answer + +

+Grace Periods Don't Partition Read-Side Critical Sections

+ +

+It is tempting to assume that if any part of one RCU read-side critical +section precedes a given grace period, and if any part of another RCU +read-side critical section follows that same grace period, then all of +the first RCU read-side critical section must precede all of the second. +However, this just isn't the case: A single grace period does not +partition the set of RCU read-side critical sections. +An example of this situation can be illustrated as follows, where +x, y, and z are initially all zero: + +

+
+ 1 void thread0(void)
+ 2 {
+ 3   rcu_read_lock();
+ 4   WRITE_ONCE(a, 1);
+ 5   WRITE_ONCE(b, 1);
+ 6   rcu_read_unlock();
+ 7 }
+ 8
+ 9 void thread1(void)
+10 {
+11   r1 = READ_ONCE(a);
+12   synchronize_rcu();
+13   WRITE_ONCE(c, 1);
+14 }
+15
+16 void thread2(void)
+17 {
+18   rcu_read_lock();
+19   r2 = READ_ONCE(b);
+20   r3 = READ_ONCE(c);
+21   rcu_read_unlock();
+22 }
+
+
+ +

+It turns out that the outcome: + +

+
+(r1 == 1 && r2 == 0 && r3 == 1)
+
+
+ +is entirely possible. +The following figure show how this can happen, with each circled +QS indicating the point at which RCU recorded a +quiescent state for each thread, that is, a state in which +RCU knows that the thread cannot be in the midst of an RCU read-side +critical section that started before the current grace period: + +

GPpartitionReaders1.svg

+ +

+If it is necessary to partition RCU read-side critical sections in this +manner, it is necessary to use two grace periods, where the first +grace period is known to end before the second grace period starts: + +

+
+ 1 void thread0(void)
+ 2 {
+ 3   rcu_read_lock();
+ 4   WRITE_ONCE(a, 1);
+ 5   WRITE_ONCE(b, 1);
+ 6   rcu_read_unlock();
+ 7 }
+ 8
+ 9 void thread1(void)
+10 {
+11   r1 = READ_ONCE(a);
+12   synchronize_rcu();
+13   WRITE_ONCE(c, 1);
+14 }
+15
+16 void thread2(void)
+17 {
+18   r2 = READ_ONCE(c);
+19   synchronize_rcu();
+20   WRITE_ONCE(d, 1);
+21 }
+22
+23 void thread3(void)
+24 {
+25   rcu_read_lock();
+26   r3 = READ_ONCE(b);
+27   r4 = READ_ONCE(d);
+28   rcu_read_unlock();
+29 }
+
+
+ +

+Here, if (r1 == 1), then +thread0()'s write to b must happen +before the end of thread1()'s grace period. +If in addition (r4 == 1), then +thread3()'s read from b must happen +after the beginning of thread2()'s grace period. +If it is also the case that (r2 == 1), then the +end of thread1()'s grace period must precede the +beginning of thread2()'s grace period. +This mean that the two RCU read-side critical sections cannot overlap, +guaranteeing that (r3 == 1). +As a result, the outcome: + +

+
+(r1 == 1 && r2 == 1 && r3 == 0 && r4 == 1)
+
+
+ +cannot happen. + +

+This non-requirement was also non-premeditated, but became apparent +when studying RCU's interaction with memory ordering. + +

+Read-Side Critical Sections Don't Partition Grace Periods

+ +

+It is also tempting to assume that if an RCU read-side critical section +happens between a pair of grace periods, then those grace periods cannot +overlap. +However, this temptation leads nowhere good, as can be illustrated by +the following, with all variables initially zero: + +

+
+ 1 void thread0(void)
+ 2 {
+ 3   rcu_read_lock();
+ 4   WRITE_ONCE(a, 1);
+ 5   WRITE_ONCE(b, 1);
+ 6   rcu_read_unlock();
+ 7 }
+ 8
+ 9 void thread1(void)
+10 {
+11   r1 = READ_ONCE(a);
+12   synchronize_rcu();
+13   WRITE_ONCE(c, 1);
+14 }
+15
+16 void thread2(void)
+17 {
+18   rcu_read_lock();
+19   WRITE_ONCE(d, 1);
+20   r2 = READ_ONCE(c);
+21   rcu_read_unlock();
+22 }
+23
+24 void thread3(void)
+25 {
+26   r3 = READ_ONCE(d);
+27   synchronize_rcu();
+28   WRITE_ONCE(e, 1);
+29 }
+30
+31 void thread4(void)
+32 {
+33   rcu_read_lock();
+34   r4 = READ_ONCE(b);
+35   r5 = READ_ONCE(e);
+36   rcu_read_unlock();
+37 }
+
+
+ +

+In this case, the outcome: + +

+
+(r1 == 1 && r2 == 1 && r3 == 1 && r4 == 0 && r5 == 1)
+
+
+ +is entirely possible, as illustrated below: + +

ReadersPartitionGP1.svg

+ +

+Again, an RCU read-side critical section can overlap almost all of a +given grace period, just so long as it does not overlap the entire +grace period. +As a result, an RCU read-side critical section cannot partition a pair +of RCU grace periods. + +

Quick Quiz 10: +How long a sequence of grace periods, each separated by an RCU read-side +critical section, would be required to partition the RCU read-side +critical sections at the beginning and end of the chain? +
Answer + +

+Disabling Preemption Does Not Block Grace Periods

+ +

+There was a time when disabling preemption on any given CPU would block +subsequent grace periods. +However, this was an accident of implementation and is not a requirement. +And in the current Linux-kernel implementation, disabling preemption +on a given CPU in fact does not block grace periods, as Oleg Nesterov +demonstrated. + +

+If you need a preempt-disable region to block grace periods, you need to add +rcu_read_lock() and rcu_read_unlock(), for example +as follows: + +

+
+ 1 preempt_disable();
+ 2 rcu_read_lock();
+ 3 do_something();
+ 4 rcu_read_unlock();
+ 5 preempt_enable();
+ 6
+ 7 /* Spinlocks implicitly disable preemption. */
+ 8 spin_lock(&mylock);
+ 9 rcu_read_lock();
+10 do_something();
+11 rcu_read_unlock();
+12 spin_unlock(&mylock);
+
+
+ +

+In theory, you could enter the RCU read-side critical section first, +but it is more efficient to keep the entire RCU read-side critical +section contained in the preempt-disable region as shown above. +Of course, RCU read-side critical sections that extend outside of +preempt-disable regions will work correctly, but such critical sections +can be preempted, which forces rcu_read_unlock() to do +more work. +And no, this is not an invitation to enclose all of your RCU +read-side critical sections within preempt-disable regions, because +doing so would degrade real-time response. + +

+This non-requirement appeared with preemptible RCU. +If you need a grace period that waits on non-preemptible code regions, use +RCU-sched. + +

Parallelism Facts of Life

+ +

+These parallelism facts of life are by no means specific to RCU, but +the RCU implementation must abide by them. +They therefore bear repeating: + +

    +
  1. Any CPU or task may be delayed at any time, + and any attempts to avoid these delays by disabling + preemption, interrupts, or whatever are completely futile. + This is most obvious in preemptible user-level + environments and in virtualized environments (where + a given guest OS's VCPUs can be preempted at any time by + the underlying hypervisor), but can also happen in bare-metal + environments due to ECC errors, NMIs, and other hardware + events. + Although a delay of more than about 20 seconds can result + in splats, the RCU implementation is obligated to use + algorithms that can tolerate extremely long delays, but where + “extremely long” is not long enough to allow + wrap-around when incrementing a 64-bit counter. +
  2. Both the compiler and the CPU can reorder memory accesses. + Where it matters, RCU must use compiler directives and + memory-barrier instructions to preserve ordering. +
  3. Conflicting writes to memory locations in any given cache line + will result in expensive cache misses. + Greater numbers of concurrent writes and more-frequent + concurrent writes will result in more dramatic slowdowns. + RCU is therefore obligated to use algorithms that have + sufficient locality to avoid significant performance and + scalability problems. +
  4. As a rough rule of thumb, only one CPU's worth of processing + may be carried out under the protection of any given exclusive + lock. + RCU must therefore use scalable locking designs. +
  5. Counters are finite, especially on 32-bit systems. + RCU's use of counters must therefore tolerate counter wrap, + or be designed such that counter wrap would take way more + time than a single system is likely to run. + An uptime of ten years is quite possible, a runtime + of a century much less so. + As an example of the latter, RCU's dyntick-idle nesting counter + allows 54 bits for interrupt nesting level (this counter + is 64 bits even on a 32-bit system). + Overflowing this counter requires 254 + half-interrupts on a given CPU without that CPU ever going idle. + If a half-interrupt happened every microsecond, it would take + 570 years of runtime to overflow this counter, which is currently + believed to be an acceptably long time. +
  6. Linux systems can have thousands of CPUs running a single + Linux kernel in a single shared-memory environment. + RCU must therefore pay close attention to high-end scalability. +
+ +

+This last parallelism fact of life means that RCU must pay special +attention to the preceding facts of life. +The idea that Linux might scale to systems with thousands of CPUs would +have been met with some skepticism in the 1990s, but these requirements +would have otherwise have been unsurprising, even in the early 1990s. + +

Quality-of-Implementation Requirements

+ +

+These sections list quality-of-implementation requirements. +Although an RCU implementation that ignores these requirements could +still be used, it would likely be subject to limitations that would +make it inappropriate for industrial-strength production use. +Classes of quality-of-implementation requirements are as follows: + +

    +
  1. Specialization +
  2. Performance and Scalability +
  3. Composability +
  4. Corner Cases +
+ +

+These classes is covered in the following sections. + +

Specialization

+ +

+RCU is and always has been intended primarily for read-mostly situations, as +illustrated by the following figure. +This means that RCU's read-side primitives are optimized, often at the +expense of its update-side primitives. + +

RCUApplicability.svg

+ +

+This focus on read-mostly situations means that RCU must interoperate +with other synchronization primitives. +For example, the add_gp() and remove_gp_synchronous() +examples discussed earlier use RCU to protect readers and locking to +coordinate updaters. +However, the need extends much farther, requiring that a variety of +synchronization primitives be legal within RCU read-side critical sections, +including spinlocks, sequence locks, atomic operations, reference +counters, and memory barriers. + +

Quick Quiz 11: +What about sleeping locks? +
Answer + +

+It often comes as a surprise that many algorithms do not require a +consistent view of data, but many can function in that mode, +with network routing being the poster child. +Internet routing algorithms take significant time to propagate +updates, so that by the time an update arrives at a given system, +that system has been sending network traffic the wrong way for +a considerable length of time. +Having a few threads continue to send traffic the wrong way for a +few more milliseconds is clearly not a problem: In the worst case, +TCP retransmissions will eventually get the data where it needs to go. +In general, when tracking the state of the universe outside of the +computer, some level of inconsistency must be tolerated due to +speed-of-light delays if nothing else. + +

+Furthermore, uncertainty about external state is inherent in many cases. +For example, a pair of veternarians might use heartbeat to determine +whether or not a given cat was alive. +But how long should they wait after the last heartbeat to decide that +the cat is in fact dead? +Waiting less than 400 milliseconds makes no sense because this would +mean that a relaxed cat would be considered to cycle between death +and life more than 100 times per minute. +Moreover, just as with human beings, a cat's heart might stop for +some period of time, so the exact wait period is a judgment call. +One of our pair of veternarians might wait 30 seconds before pronouncing +the cat dead, while the other might insist on waiting a full minute. +The two veternarians would then disagree on the state of the cat during +the final 30 seconds of the minute following the last heartbeat, as +fancifully illustrated below: + +

2013-08-is-it-dead.png

+ +

+Interestingly enough, this same situation applies to hardware. +When push comes to shove, how do we tell whether or not some +external server has failed? +We send messages to it periodically, and declare it failed if we +don't receive a response within a given period of time. +Policy decisions can usually tolerate short +periods of inconsistency. +The policy was decided some time ago, and is only now being put into +effect, so a few milliseconds of delay is normally inconsequential. + +

+However, there are algorithms that absolutely must see consistent data. +For example, the translation between a user-level SystemV semaphore +ID to the corresponding in-kernel data structure is protected by RCU, +but it is absolutely forbidden to update a semaphore that has just been +removed. +In the Linux kernel, this need for consistency is accommodated by acquiring +spinlocks located in the in-kernel data structure from within +the RCU read-side critical section, and this is indicated by the +green box in the figure above. +Many other techniques may be used, and are in fact used within the +Linux kernel. + +

+In short, RCU is not required to maintain consistency, and other +mechanisms may be used in concert with RCU when consistency is required. +RCU's specialization allows it to do its job extremely well, and its +ability to interoperate with other synchronization mechanisms allows +the right mix of synchronization tools to be used for a given job. + +

Performance and Scalability

+ +

+Energy efficiency is a critical component of performance today, +and Linux-kernel RCU implementations must therefore avoid unnecessarily +awakening idle CPUs. +I cannot claim that this requirement was premeditated. +In fact, I learned of it during a telephone conversation in which I +was given “frank and open” feedback on the importance +of energy efficiency in battery-powered systems and on specific +energy-efficiency shortcomings of the Linux-kernel RCU implementation. +In my experience, the battery-powered embedded community will consider +any unnecessary wakeups to be extremely unfriendly acts. +So much so that mere Linux-kernel-mailing-list posts are +insufficient to vent their ire. + +

+Memory consumption is not particularly important for in most +situations, and has become decreasingly +so as memory sizes have expanded and memory +costs have plummeted. +However, as I learned from Matt Mackall's +bloatwatch +efforts, memory footprint is critically important on single-CPU systems with +non-preemptible (CONFIG_PREEMPT=n) kernels, and thus +tiny RCU +was born. +Josh Triplett has since taken over the small-memory banner with his +Linux kernel tinification +project, which resulted in +SRCU +becoming optional for those kernels not needing it. + +

+The remaining performance requirements are, for the most part, +unsurprising. +For example, in keeping with RCU's read-side specialization, +rcu_dereference() should have negligible overhead (for +example, suppression of a few minor compiler optimizations). +Similarly, in non-preemptible environments, rcu_read_lock() and +rcu_read_unlock() should have exactly zero overhead. + +

+In preemptible environments, in the case where the RCU read-side +critical section was not preempted (as will be the case for the +highest-priority real-time process), rcu_read_lock() and +rcu_read_unlock() should have minimal overhead. +In particular, they should not contain atomic read-modify-write +operations, memory-barrier instructions, preemption disabling, +interrupt disabling, or backwards branches. +However, in the case where the RCU read-side critical section was preempted, +rcu_read_unlock() may acquire spinlocks and disable interrupts. +This is why it is better to nest an RCU read-side critical section +within a preempt-disable region than vice versa, at least in cases +where that critical section is short enough to avoid unduly degrading +real-time latencies. + +

+The synchronize_rcu() grace-period-wait primitive is +optimized for throughput. +It may therefore incur several milliseconds of latency in addition to +the duration of the longest RCU read-side critical section. +On the other hand, multiple concurrent invocations of +synchronize_rcu() are required to use batching optimizations +so that they can be satisfied by a single underlying grace-period-wait +operation. +For example, in the Linux kernel, it is not unusual for a single +grace-period-wait operation to serve more than +1,000 separate invocations +of synchronize_rcu(), thus amortizing the per-invocation +overhead down to nearly zero. +However, the grace-period optimization is also required to avoid +measurable degradation of real-time scheduling and interrupt latencies. + +

+In some cases, the multi-millisecond synchronize_rcu() +latencies are unacceptable. +In these cases, synchronize_rcu_expedited() may be used +instead, reducing the grace-period latency down to a few tens of +microseconds on small systems, at least in cases where the RCU read-side +critical sections are short. +There are currently no special latency requirements for +synchronize_rcu_expedited() on large systems, but, +consistent with the empirical nature of the RCU specification, +that is subject to change. +However, there most definitely are scalability requirements: +A storm of synchronize_rcu_expedited() invocations on 4096 +CPUs should at least make reasonable forward progress. +In return for its shorter latencies, synchronize_rcu_expedited() +is permitted to impose modest degradation of real-time latency +on non-idle online CPUs. +That said, it will likely be necessary to take further steps to reduce this +degradation, hopefully to roughly that of a scheduling-clock interrupt. + +

+There are a number of situations where even +synchronize_rcu_expedited()'s reduced grace-period +latency is unacceptable. +In these situations, the asynchronous call_rcu() can be +used in place of synchronize_rcu() as follows: + +

+
+ 1 struct foo {
+ 2   int a;
+ 3   int b;
+ 4   struct rcu_head rh;
+ 5 };
+ 6
+ 7 static void remove_gp_cb(struct rcu_head *rhp)
+ 8 {
+ 9   struct foo *p = container_of(rhp, struct foo, rh);
+10
+11   kfree(p);
+12 }
+13
+14 bool remove_gp_asynchronous(void)
+15 {
+16   struct foo *p;
+17
+18   spin_lock(&gp_lock);
+19   p = rcu_dereference(gp);
+20   if (!p) {
+21     spin_unlock(&gp_lock);
+22     return false;
+23   }
+24   rcu_assign_pointer(gp, NULL);
+25   call_rcu(&p->rh, remove_gp_cb);
+26   spin_unlock(&gp_lock);
+27   return true;
+28 }
+
+
+ +

+A definition of struct foo is finally needed, and appears +on lines 1-5. +The function remove_gp_cb() is passed to call_rcu() +on line 25, and will be invoked after the end of a subsequent +grace period. +This gets the same effect as remove_gp_synchronous(), +but without forcing the updater to wait for a grace period to elapse. +The call_rcu() function may be used in a number of +situations where neither synchronize_rcu() nor +synchronize_rcu_expedited() would be legal, +including within preempt-disable code, local_bh_disable() code, +interrupt-disable code, and interrupt handlers. +However, even call_rcu() is illegal within NMI handlers. +The callback function (remove_gp_cb() in this case) will be +executed within softirq (software interrupt) environment within the +Linux kernel, +either within a real softirq handler or under the protection +of local_bh_disable(). +In both the Linux kernel and in userspace, it is bad practice to +write an RCU callback function that takes too long. +Long-running operations should be relegated to separate threads or +(in the Linux kernel) workqueues. + +

Quick Quiz 12: +Why does line 19 use rcu_access_pointer()? +After all, call_rcu() on line 25 stores into the +structure, which would interact badly with concurrent insertions. +Doesn't this mean that rcu_dereference() is required? +
Answer + +

+However, all that remove_gp_cb() is doing is +invoking kfree() on the data element. +This is a common idiom, and is supported by kfree_rcu(), +which allows “fire and forget” operation as shown below: + +

+
+ 1 struct foo {
+ 2   int a;
+ 3   int b;
+ 4   struct rcu_head rh;
+ 5 };
+ 6
+ 7 bool remove_gp_faf(void)
+ 8 {
+ 9   struct foo *p;
+10
+11   spin_lock(&gp_lock);
+12   p = rcu_dereference(gp);
+13   if (!p) {
+14     spin_unlock(&gp_lock);
+15     return false;
+16   }
+17   rcu_assign_pointer(gp, NULL);
+18   kfree_rcu(p, rh);
+19   spin_unlock(&gp_lock);
+20   return true;
+21 }
+
+
+ +

+Note that remove_gp_faf() simply invokes +kfree_rcu() and proceeds, without any need to pay any +further attention to the subsequent grace period and kfree(). +It is permissible to invoke kfree_rcu() from the same +environments as for call_rcu(). +Interestingly enough, DYNIX/ptx had the equivalents of +call_rcu() and kfree_rcu(), but not +synchronize_rcu(). +This was due to the fact that RCU was not heavily used within DYNIX/ptx, +so the very few places that needed something like +synchronize_rcu() simply open-coded it. + +

Quick Quiz 13: +Earlier it was claimed that call_rcu() and +kfree_rcu() allowed updaters to avoid being blocked +by readers. +But how can that be correct, given that the invocation of the callback +and the freeing of the memory (respectively) must still wait for +a grace period to elapse? +
Answer + +

+But what if the updater must wait for the completion of code to be +executed after the end of the grace period, but has other tasks +that can be carried out in the meantime? +The polling-style get_state_synchronize_rcu() and +cond_synchronize_rcu() functions may be used for this +purpose, as shown below: + +

+
+ 1 bool remove_gp_poll(void)
+ 2 {
+ 3   struct foo *p;
+ 4   unsigned long s;
+ 5
+ 6   spin_lock(&gp_lock);
+ 7   p = rcu_access_pointer(gp);
+ 8   if (!p) {
+ 9     spin_unlock(&gp_lock);
+10     return false;
+11   }
+12   rcu_assign_pointer(gp, NULL);
+13   spin_unlock(&gp_lock);
+14   s = get_state_synchronize_rcu();
+15   do_something_while_waiting();
+16   cond_synchronize_rcu(s);
+17   kfree(p);
+18   return true;
+19 }
+
+
+ +

+On line 14, get_state_synchronize_rcu() obtains a +“cookie” from RCU, +then line 15 carries out other tasks, +and finally, line 16 returns immediately if a grace period has +elapsed in the meantime, but otherwise waits as required. +The need for get_state_synchronize_rcu and +cond_synchronize_rcu() has appeared quite recently, +so it is too early to tell whether they will stand the test of time. + +

+RCU thus provides a range of tools to allow updaters to strike the +required tradeoff between latency, flexibility and CPU overhead. + +

Composability

+ +

+Composability has received much attention in recent years, perhaps in part +due to the collision of multicore hardware with object-oriented techniques +designed in single-threaded environments for single-threaded use. +And in theory, RCU read-side critical sections may be composed, and in +fact may be nested arbitrarily deeply. +In practice, as with all real-world implementations of composable +constructs, there are limitations. + +

+Implementations of RCU for which rcu_read_lock() +and rcu_read_unlock() generate no code, such as +Linux-kernel RCU when CONFIG_PREEMPT=n, can be +nested arbitrarily deeply. +After all, there is no overhead. +Except that if all these instances of rcu_read_lock() +and rcu_read_unlock() are visible to the compiler, +compilation will eventually fail due to exhausting memory, +mass storage, or user patience, whichever comes first. +If the nesting is not visible to the compiler, as is the case with +mutually recursive functions each in its own translation unit, +stack overflow will result. +If the nesting takes the form of loops, either the control variable +will overflow or (in the Linux kernel) you will get an RCU CPU stall warning. +Nevertheless, this class of RCU implementations is one +of the most composable constructs in existence. + +

+RCU implementations that explicitly track nesting depth +are limited by the nesting-depth counter. +For example, the Linux kernel's preemptible RCU limits nesting to +INT_MAX. +This should suffice for almost all practical purposes. +That said, a consecutive pair of RCU read-side critical sections +between which there is an operation that waits for a grace period +cannot be enclosed in another RCU read-side critical section. +This is because it is not legal to wait for a grace period within +an RCU read-side critical section: To do so would result either +in deadlock or +in RCU implicitly splitting the enclosing RCU read-side critical +section, neither of which is conducive to a long-lived and prosperous +kernel. + +

+In short, although RCU read-side critical sections are highly composable, +care is required in some situations, just as is the case for any other +composable synchronization mechanism. + +

Corner Cases

+ +

+A given RCU workload might have an endless and intense stream of +RCU read-side critical sections, perhaps even so intense that there +was never a point in time during which there was not at least one +RCU read-side critical section in flight. +RCU cannot allow this situation to block grace periods: As long as +all the RCU read-side critical sections are finite, grace periods +must also be finite. + +

+That said, preemptible RCU implementations could potentially result +in RCU read-side critical sections being preempted for long durations, +which has the effect of creating a long-duration RCU read-side +critical section. +This situation can arise only in heavily loaded systems, but systems using +real-time priorities are of course more vulnerable. +Therefore, RCU priority boosting is provided to help deal with this +case. +That said, the exact requirements on RCU priority boosting will likely +evolve as more experience accumulates. + +

+Other workloads might have very high update rates. +Although one can argue that such workloads should instead use +something other than RCU, the fact remains that RCU must +handle such workloads gracefully. +This requirement is another factor driving batching of grace periods, +but it is also the driving force behind the checks for large numbers +of queued RCU callbacks in the call_rcu() code path. +Finally, high update rates should not delay RCU read-side critical +sections, although some read-side delays can occur when using +synchronize_rcu_expedited(), courtesy of this function's use +of try_stop_cpus(). +(In the future, synchronize_rcu_expedited() will be +converted to use lighter-weight inter-processor interrupts (IPIs), +but this will still disturb readers, though to a much smaller degree.) + +

+Although all three of these corner cases were understood in the early +1990s, a simple user-level test consisting of close(open(path)) +in a tight loop +in the early 2000s suddenly provided a much deeper appreciation of the +high-update-rate corner case. +This test also motivated addition of some RCU code to react to high update +rates, for example, if a given CPU finds itself with more than 10,000 +RCU callbacks queued, it will cause RCU to take evasive action by +more aggressively starting grace periods and more aggressively forcing +completion of grace-period processing. +This evasive action causes the grace period to complete more quickly, +but at the cost of restricting RCU's batching optimizations, thus +increasing the CPU overhead incurred by that grace period. + +

+Software-Engineering Requirements

+ +

+Between Murphy's Law and “To err is human”, it is necessary to +guard against mishaps and misuse: + +

    +
  1. It is all too easy to forget to use rcu_read_lock() + everywhere that it is needed, so kernels built with + CONFIG_PROVE_RCU=y will spat if + rcu_dereference() is used outside of an + RCU read-side critical section. + Update-side code can use rcu_dereference_protected(), + which takes a + lockdep expression + to indicate what is providing the protection. + If the indicated protection is not provided, a lockdep splat + is emitted. + +

    + Code shared between readers and updaters can use + rcu_dereference_check(), which also takes a + lockdep expression, and emits a lockdep splat if neither + rcu_read_lock() nor the indicated protection + is in place. + In addition, rcu_dereference_raw() is used in those + (hopefully rare) cases where the required protection cannot + be easily described. + Finally, rcu_read_lock_held() is provided to + allow a function to verify that it has been invoked within + an RCU read-side critical section. + I was made aware of this set of requirements shortly after Thomas + Gleixner audited a number of RCU uses. +

  2. A given function might wish to check for RCU-related preconditions + upon entry, before using any other RCU API. + The rcu_lockdep_assert() does this job, + asserting the expression in kernels having lockdep enabled + and doing nothing otherwise. +
  3. It is also easy to forget to use rcu_assign_pointer() + and rcu_dereference(), perhaps (incorrectly) + substituting a simple assignment. + To catch this sort of error, a given RCU-protected pointer may be + tagged with __rcu, after which running sparse + with CONFIG_SPARSE_RCU_POINTER=y will complain + about simple-assignment accesses to that pointer. + Arnd Bergmann made me aware of this requirement, and also + supplied the needed + patch series. +
  4. Kernels built with CONFIG_DEBUG_OBJECTS_RCU_HEAD=y + will splat if a data element is passed to call_rcu() + twice in a row, without a grace period in between. + (This error is similar to a double free.) + The corresponding rcu_head structures that are + dynamically allocated are automatically tracked, but + rcu_head structures allocated on the stack + must be initialized with init_rcu_head_on_stack() + and cleaned up with destroy_rcu_head_on_stack(). + Similarly, statically allocated non-stack rcu_head + structures must be initialized with init_rcu_head() + and cleaned up with destroy_rcu_head(). + Mathieu Desnoyers made me aware of this requirement, and also + supplied the needed + patch. +
  5. An infinite loop in an RCU read-side critical section will + eventually trigger an RCU CPU stall warning splat. + However, RCU is not obligated to produce this splat + unless there is a grace period waiting on that particular + RCU read-side critical section. + This requirement made itself known in the early 1990s, pretty + much the first time that it was necessary to debug a CPU stall. +
  6. Although it would be very good to detect pointers leaking out + of RCU read-side critical sections, there is currently no + good way of doing this. + One complication is the need to distinguish between pointers + leaking and pointers that have been handed off from RCU to + some other synchronization mechanism, for example, reference + counting. +
  7. In kernels built with CONFIG_RCU_TRACE=y, RCU-related + information is provided via both debugfs and event tracing. +
  8. Open-coded use of rcu_assign_pointer() and + rcu_dereference() to create typical linked + data structures can be surprisingly error-prone. + Therefore, RCU-protected + linked lists + and, more recently, RCU-protected + hash tables + are available. + Many other special-purpose RCU-protected data structures are + available in the Linux kernel and the userspace RCU library. +
  9. Some linked structures are created at compile time, but still + require __rcu checking. + The RCU_POINTER_INITIALIZER() macro serves this + purpose. +
  10. It is not necessary to use rcu_assign_pointer() + when creating linked structures that are to be published via + a single external pointer. + The RCU_INIT_POINTER() macro is provided for + this task and also for assigning NULL pointers + at runtime. +
+ +

+This not a hard-and-fast list: RCU's diagnostic capabilities will +continue to be guided by the number and type of usage bugs found +in real-world RCU usage. + +

Linux Kernel Complications

+ +

+The Linux kernel provides an interesting environment for all kinds of +software, including RCU. +Some of the relevant points of interest are as follows: + +

    +
  1. Configuration. +
  2. Firmware Interface. +
  3. Early Boot. +
  4. + Interrupts and non-maskable interrupts (NMIs). +
  5. Loadable Modules. +
  6. Hotplug CPU. +
  7. Scheduler and RCU. +
  8. Tracing and RCU. +
  9. Energy Efficiency. +
  10. + Performance, Scalability, Response Time, and Reliability. +
+ +

+This list is probably incomplete, but it does give a feel for the +most notable Linux-kernel complications. +Each of the following sections covers one of the above topics. + +

Configuration

+ +

+RCU's goal is automatic configuration, so that almost nobody +needs to worry about RCU's Kconfig options. +And for almost all users, RCU does in fact work well +“out of the box.” + +

+However, there are specialized use cases that are handled by +kernel boot parameters and Kconfig options. +Unfortunately, the Kconfig system will explicitly ask users +about new Kconfig options, which requires almost all of them +be hidden behind a CONFIG_RCU_EXPERT Kconfig option. + +

+This all should be quite obvious, but the fact remains that +Linus Torvalds recently had to +remind +me of this requirement. + +

Firmware Interface

+ +

+In many cases, kernel obtains information about the system from the +firmware, and sometimes things are lost in translation. +Or the translation is accurate, but the original message is bogus. + +

+For example, some systems' firmware overreports the number of CPUs, +sometimes by a large factor. +If RCU naively believed the firmware, as it used to do, +it would create too many per-CPU kthreads. +Although the resulting system will still run correctly, the extra +kthreads needlessly consume memory and can cause confusion +when they show up in ps listings. + +

+RCU must therefore wait for a given CPU to actually come online before +it can allow itself to believe that the CPU actually exists. +The resulting “ghost CPUs” (which are never going to +come online) cause a number of +interesting complications. + +

Early Boot

+ +

+The Linux kernel's boot sequence is an interesting process, +and RCU is used early, even before rcu_init() +is invoked. +In fact, a number of RCU's primitives can be used as soon as the +initial task's task_struct is available and the +boot CPU's per-CPU variables are set up. +The read-side primitives (rcu_read_lock(), +rcu_read_unlock(), rcu_dereference(), +and rcu_access_pointer()) will operate normally very early on, +as will rcu_assign_pointer(). + +

+Although call_rcu() may be invoked at any +time during boot, callbacks are not guaranteed to be invoked until after +the scheduler is fully up and running. +This delay in callback invocation is due to the fact that RCU does not +invoke callbacks until it is fully initialized, and this full initialization +cannot occur until after the scheduler has initialized itself to the +point where RCU can spawn and run its kthreads. +In theory, it would be possible to invoke callbacks earlier, +however, this is not a panacea because there would be severe restrictions +on what operations those callbacks could invoke. + +

+Perhaps surprisingly, synchronize_rcu(), +synchronize_rcu_bh() +(discussed below), +and +synchronize_sched() +will all operate normally +during very early boot, the reason being that there is only one CPU +and preemption is disabled. +This means that the call synchronize_rcu() (or friends) +itself is a quiescent +state and thus a grace period, so the early-boot implementation can +be a no-op. + +

+Both synchronize_rcu_bh() and synchronize_sched() +continue to operate normally through the remainder of boot, courtesy +of the fact that preemption is disabled across their RCU read-side +critical sections and also courtesy of the fact that there is still +only one CPU. +However, once the scheduler starts initializing, preemption is enabled. +There is still only a single CPU, but the fact that preemption is enabled +means that the no-op implementation of synchronize_rcu() no +longer works in CONFIG_PREEMPT=y kernels. +Therefore, as soon as the scheduler starts initializing, the early-boot +fastpath is disabled. +This means that synchronize_rcu() switches to its runtime +mode of operation where it posts callbacks, which in turn means that +any call to synchronize_rcu() will block until the corresponding +callback is invoked. +Unfortunately, the callback cannot be invoked until RCU's runtime +grace-period machinery is up and running, which cannot happen until +the scheduler has initialized itself sufficiently to allow RCU's +kthreads to be spawned. +Therefore, invoking synchronize_rcu() during scheduler +initialization can result in deadlock. + +

Quick Quiz 14: +So what happens with synchronize_rcu() during +scheduler initialization for CONFIG_PREEMPT=n +kernels? +
Answer + +

+I learned of these boot-time requirements as a result of a series of +system hangs. + +

Interrupts and NMIs

+ +

+The Linux kernel has interrupts, and RCU read-side critical sections are +legal within interrupt handlers and within interrupt-disabled regions +of code, as are invocations of call_rcu(). + +

+Some Linux-kernel architectures can enter an interrupt handler from +non-idle process context, and then just never leave it, instead stealthily +transitioning back to process context. +This trick is sometimes used to invoke system calls from inside the kernel. +These “half-interrupts” mean that RCU has to be very careful +about how it counts interrupt nesting levels. +I learned of this requirement the hard way during a rewrite +of RCU's dyntick-idle code. + +

+The Linux kernel has non-maskable interrupts (NMIs), and +RCU read-side critical sections are legal within NMI handlers. +Thankfully, RCU update-side primitives, including +call_rcu(), are prohibited within NMI handlers. + +

+The name notwithstanding, some Linux-kernel architectures +can have nested NMIs, which RCU must handle correctly. +Andy Lutomirski +surprised me +with this requirement; +he also kindly surprised me with +an algorithm +that meets this requirement. + +

Loadable Modules

+ +

+The Linux kernel has loadable modules, and these modules can +also be unloaded. +After a given module has been unloaded, any attempt to call +one of its functions results in a segmentation fault. +The module-unload functions must therefore cancel any +delayed calls to loadable-module functions, for example, +any outstanding mod_timer() must be dealt with +via del_timer_sync() or similar. + +

+Unfortunately, there is no way to cancel an RCU callback; +once you invoke call_rcu(), the callback function is +going to eventually be invoked, unless the system goes down first. +Because it is normally considered socially irresponsible to crash the system +in response to a module unload request, we need some other way +to deal with in-flight RCU callbacks. + +

+RCU therefore provides +rcu_barrier(), +which waits until all in-flight RCU callbacks have been invoked. +If a module uses call_rcu(), its exit function should therefore +prevent any future invocation of call_rcu(), then invoke +rcu_barrier(). +In theory, the underlying module-unload code could invoke +rcu_barrier() unconditionally, but in practice this would +incur unacceptable latencies. + +

+Nikita Danilov noted this requirement for an analogous filesystem-unmount +situation, and Dipankar Sarma incorporated rcu_barrier() into RCU. +The need for rcu_barrier() for module unloading became +apparent later. + +

Hotplug CPU

+ +

+The Linux kernel supports CPU hotplug, which means that CPUs +can come and go. +It is of course illegal to use any RCU API member from an offline CPU. +This requirement was present from day one in DYNIX/ptx, but +on the other hand, the Linux kernel's CPU-hotplug implementation +is “interesting.” + +

+The Linux-kernel CPU-hotplug implementation has notifiers that +are used to allow the various kernel subsystems (including RCU) +to respond appropriately to a given CPU-hotplug operation. +Most RCU operations may be invoked from CPU-hotplug notifiers, +including even normal synchronous grace-period operations +such as synchronize_rcu(). +However, expedited grace-period operations such as +synchronize_rcu_expedited() are not supported, +due to the fact that current implementations block CPU-hotplug +operations, which could result in deadlock. + +

+In addition, all-callback-wait operations such as +rcu_barrier() are also not supported, due to the +fact that there are phases of CPU-hotplug operations where +the outgoing CPU's callbacks will not be invoked until after +the CPU-hotplug operation ends, which could also result in deadlock. + +

Scheduler and RCU

+ +

+RCU depends on the scheduler, and the scheduler uses RCU to +protect some of its data structures. +This means the scheduler is forbidden from acquiring +the runqueue locks and the priority-inheritance locks +in the middle of an outermost RCU read-side critical section unless +it also releases them before exiting that same +RCU read-side critical section. +This same prohibition also applies to any lock that is acquired +while holding any lock to which this prohibition applies. +Violating this rule results in deadlock. + +

+For RCU's part, the preemptible-RCU rcu_read_unlock() +implementation must be written carefully to avoid similar deadlocks. +In particular, rcu_read_unlock() must tolerate an +interrupt where the interrupt handler invokes both +rcu_read_lock() and rcu_read_unlock(). +This possibility requires rcu_read_unlock() to use +negative nesting levels to avoid destructive recursion via +interrupt handler's use of RCU. + +

+This pair of mutual scheduler-RCU requirements came as a +complete surprise. + +

+As noted above, RCU makes use of kthreads, and it is necessary to +avoid excessive CPU-time accumulation by these kthreads. +This requirement was no surprise, but RCU's violation of it +when running context-switch-heavy workloads when built with +CONFIG_NO_HZ_FULL=y +did come as a surprise [PDF]. +RCU has made good progress towards meeting this requirement, even +for context-switch-have CONFIG_NO_HZ_FULL=y workloads, +but there is room for further improvement. + +

Tracing and RCU

+ +

+It is possible to use tracing on RCU code, but tracing itself +uses RCU. +For this reason, rcu_dereference_raw_notrace() +is provided for use by tracing, which avoids the destructive +recursion that could otherwise ensue. +This API is also used by virtualization in some architectures, +where RCU readers execute in environments in which tracing +cannot be used. +The tracing folks both located the requirement and provided the +needed fix, so this surprise requirement was relatively painless. + +

Energy Efficiency

+ +

+Interrupting idle CPUs is considered socially unacceptable, +especially by people with battery-powered embedded systems. +RCU therefore conserves energy by detecting which CPUs are +idle, including tracking CPUs that have been interrupted from idle. +This is a large part of the energy-efficiency requirement, +so I learned of this via an irate phone call. + +

+Because RCU avoids interrupting idle CPUs, it is illegal to +execute an RCU read-side critical section on an idle CPU. +(Kernels built with CONFIG_PROVE_RCU=y will splat +if you try it.) +The RCU_NONIDLE() macro and _rcuidle +event tracing is provided to work around this restriction. +In addition, rcu_is_watching() may be used to +test whether or not it is currently legal to run RCU read-side +critical sections on this CPU. +I learned of the need for diagnostics on the one hand +and RCU_NONIDLE() on the other while inspecting +idle-loop code. +Steven Rostedt supplied _rcuidle event tracing, +which is used quite heavily in the idle loop. + +

+It is similarly socially unacceptable to interrupt an +nohz_full CPU running in userspace. +RCU must therefore track nohz_full userspace +execution. +And in +CONFIG_NO_HZ_FULL_SYSIDLE=y +kernels, RCU must separately track idle CPUs on the one hand and +CPUs that are either idle or executing in userspace on the other. +In both cases, RCU must be able to sample state at two points in +time, and be able to determine whether or not some other CPU spent +any time idle and/or executing in userspace. + +

+These energy-efficiency requirements have proven quite difficult to +understand and to meet, for example, there have been more than five +clean-sheet rewrites of RCU's energy-efficiency code, the last of +which was finally able to demonstrate +real energy savings running on real hardware [PDF]. +As noted earlier, +I learned of many of these requirements via angry phone calls: +Flaming me on the Linux-kernel mailing list was apparently not +sufficient to fully vent their ire at RCU's energy-efficiency bugs! + +

+Performance, Scalability, Response Time, and Reliability

+ +

+Expanding on the +earlier discussion, +RCU is used heavily by hot code paths in performance-critical +portions of the Linux kernel's networking, security, virtualization, +and scheduling code paths. +RCU must therefore use efficient implementations, especially in its +read-side primitives. +To that end, it would be good if preemptible RCU's implementation +of rcu_read_lock() could be inlined, however, doing +this requires resolving #include issues with the +task_struct structure. + +

+The Linux kernel supports hardware configurations with up to +4096 CPUs, which means that RCU must be extremely scalable. +Algorithms that involve frequent acquisitions of global locks or +frequent atomic operations on global variables simply cannot be +tolerated within the RCU implementation. +RCU therefore makes heavy use of a combining tree based on the +rcu_node structure. +RCU is required to tolerate all CPUs continuously invoking any +combination of RCU's runtime primitives with minimal per-operation +overhead. +In fact, in many cases, increasing load must decrease the +per-operation overhead, witness the batching optimizations for +synchronize_rcu(), call_rcu(), +synchronize_rcu_expedited(), and rcu_barrier(). +As a general rule, RCU must cheerfully accept whatever the +rest of the Linux kernel decides to throw at it. + +

+The Linux kernel is used for real-time workloads, especially +in conjunction with the +-rt patchset. +The real-time-latency response requirements are such that the +traditional approach of disabling preemption across RCU +read-side critical sections is inappropriate. +Kernels built with CONFIG_PREEMPT=y therefore +use an RCU implementation that allows RCU read-side critical +sections to be preempted. +This requirement made its presence known after users made it +clear that an earlier +real-time patch +did not meet their needs, in conjunction with some +RCU issues +encountered by a very early version of the -rt patchset. + +

+In addition, RCU must make do with a sub-100-microsecond real-time latency +budget. +In fact, on smaller systems with the -rt patchset, the Linux kernel +provides sub-20-microsecond real-time latencies for the whole kernel, +including RCU. +RCU's scalability and latency must therefore be sufficient for +these sorts of configurations. +To my surprise, the sub-100-microsecond real-time latency budget + +applies to even the largest systems [PDF], +up to and including systems with 4096 CPUs. +This real-time requirement motivated the grace-period kthread, which +also simplified handling of a number of race conditions. + +

+Finally, RCU's status as a synchronization primitive means that +any RCU failure can result in arbitrary memory corruption that can be +extremely difficult to debug. +This means that RCU must be extremely reliable, which in +practice also means that RCU must have an aggressive stress-test +suite. +This stress-test suite is called rcutorture. + +

+Although the need for rcutorture was no surprise, +the current immense popularity of the Linux kernel is posing +interesting—and perhaps unprecedented—validation +challenges. +To see this, keep in mind that there are well over one billion +instances of the Linux kernel running today, given Android +smartphones, Linux-powered televisions, and servers. +This number can be expected to increase sharply with the advent of +the celebrated Internet of Things. + +

+Suppose that RCU contains a race condition that manifests on average +once per million years of runtime. +This bug will be occurring about three times per day across +the installed base. +RCU could simply hide behind hardware error rates, given that no one +should really expect their smartphone to last for a million years. +However, anyone taking too much comfort from this thought should +consider the fact that in most jurisdictions, a successful multi-year +test of a given mechanism, which might include a Linux kernel, +suffices for a number of types of safety-critical certifications. +In fact, rumor has it that the Linux kernel is already being used +in production for safety-critical applications. +I don't know about you, but I would feel quite bad if a bug in RCU +killed someone. +Which might explain my recent focus on validation and verification. + +

Other RCU Flavors

+ +

+One of the more surprising things about RCU is that there are now +no fewer than five flavors, or API families. +In addition, the primary flavor that has been the sole focus up to +this point has two different implementations, non-preemptible and +preemptible. +The other four flavors are listed below, with requirements for each +described in a separate section. + +

    +
  1. Bottom-Half Flavor +
  2. Sched Flavor +
  3. Sleepable RCU +
  4. Tasks RCU +
+ +

Bottom-Half Flavor

+ +

+The softirq-disable (AKA “bottom-half”, +hence the “_bh” abbreviations) +flavor of RCU, or RCU-bh, was developed by +Dipankar Sarma to provide a flavor of RCU that could withstand the +network-based denial-of-service attacks researched by Robert +Olsson. +These attacks placed so much networking load on the system +that some of the CPUs never exited softirq execution, +which in turn prevented those CPUs from ever executing a context switch, +which, in the RCU implementation of that time, prevented grace periods +from ever ending. +The result was an out-of-memory condition and a system hang. + +

+The solution was the creation of RCU-bh, which does +local_bh_disable() +across its read-side critical sections, and which uses the transition +from one type of softirq processing to another as a quiescent state +in addition to context switch, idle, user mode, and offline. +This means that RCU-bh grace periods can complete even when some of +the CPUs execute in softirq indefinitely, thus allowing algorithms +based on RCU-bh to withstand network-based denial-of-service attacks. + +

+Because +rcu_read_lock_bh() and rcu_read_unlock_bh() +disable and re-enable softirq handlers, any attempt to start a softirq +handlers during the +RCU-bh read-side critical section will be deferred. +In this case, rcu_read_unlock_bh() +will invoke softirq processing, which can take considerable time. +One can of course argue that this softirq overhead should be associated +with the code following the RCU-bh read-side critical section rather +than rcu_read_unlock_bh(), but the fact +is that most profiling tools cannot be expected to make this sort +of fine distinction. +For example, suppose that a three-millisecond-long RCU-bh read-side +critical section executes during a time of heavy networking load. +There will very likely be an attempt to invoke at least one softirq +handler during that three milliseconds, but any such invocation will +be delayed until the time of the rcu_read_unlock_bh(). +This can of course make it appear at first glance as if +rcu_read_unlock_bh() was executing very slowly. + +

+The +RCU-bh API +includes +rcu_read_lock_bh(), +rcu_read_unlock_bh(), +rcu_dereference_bh(), +rcu_dereference_bh_check(), +synchronize_rcu_bh(), +synchronize_rcu_bh_expedited(), +call_rcu_bh(), +rcu_barrier_bh(), and +rcu_read_lock_bh_held(). + +

Sched Flavor

+ +

+Before preemptible RCU, waiting for an RCU grace period had the +side effect of also waiting for all pre-existing interrupt +and NMI handlers. +However, there are legitimate preemptible-RCU implementations that +do not have this property, given that any point in the code outside +of an RCU read-side critical section can be a quiescent state. +Therefore, RCU-sched was created, which follows “classic” +RCU in that an RCU-sched grace period waits for for pre-existing +interrupt and NMI handlers. +In kernels built with CONFIG_PREEMPT=n, the RCU and RCU-sched +APIs have identical implementations, while kernels built with +CONFIG_PREEMPT=y provide a separate implementation for each. + +

+Note well that in CONFIG_PREEMPT=y kernels, +rcu_read_lock_sched() and rcu_read_unlock_sched() +disable and re-enable preemption, respectively. +This means that if there was a preemption attempt during the +RCU-sched read-side critical section, rcu_read_unlock_sched() +will enter the scheduler, with all the latency and overhead entailed. +Just as with rcu_read_unlock_bh(), this can make it look +as if rcu_read_unlock_sched() was executing very slowly. +However, the highest-priority task won't be preempted, so that task +will enjoy low-overhead rcu_read_unlock_sched() invocations. + +

+The +RCU-sched API +includes +rcu_read_lock_sched(), +rcu_read_unlock_sched(), +rcu_read_lock_sched_notrace(), +rcu_read_unlock_sched_notrace(), +rcu_dereference_sched(), +rcu_dereference_sched_check(), +synchronize_sched(), +synchronize_rcu_sched_expedited(), +call_rcu_sched(), +rcu_barrier_sched(), and +rcu_read_lock_sched_held(). +However, anything that disables preemption also marks an RCU-sched +read-side critical section, including +preempt_disable() and preempt_enable(), +local_irq_save() and local_irq_restore(), +and so on. + +

Sleepable RCU

+ +

+For well over a decade, someone saying “I need to block within +an RCU read-side critical section” was a reliable indication +that this someone did not understand RCU. +After all, if you are always blocking in an RCU read-side critical +section, you can probably afford to use a higher-overhead synchronization +mechanism. +However, that changed with the advent of the Linux kernel's notifiers, +whose RCU read-side critical +sections almost never sleep, but sometimes need to. +This resulted in the introduction of +sleepable RCU, +or SRCU. + +

+SRCU allows different domains to be defined, with each such domain +defined by an instance of an srcu_struct structure. +A pointer to this structure must be passed in to each SRCU function, +for example, synchronize_srcu(&ss), where +ss is the srcu_struct structure. +The key benefit of these domains is that a slow SRCU reader in one +domain does not delay an SRCU grace period in some other domain. +That said, one consequence of these domains is that read-side code +must pass a “cookie” from srcu_read_lock() +to srcu_read_unlock(), for example, as follows: + +

+
+ 1 int idx;
+ 2
+ 3 idx = srcu_read_lock(&ss);
+ 4 do_something();
+ 5 srcu_read_unlock(&ss, idx);
+
+
+ +

+As noted above, it is legal to block within SRCU read-side critical sections, +however, with great power comes great responsibility. +If you block forever in one of a given domain's SRCU read-side critical +sections, then that domain's grace periods will also be blocked forever. +Of course, one good way to block forever is to deadlock, which can +happen if any operation in a given domain's SRCU read-side critical +section can block waiting, either directly or indirectly, for that domain's +grace period to elapse. +For example, this results in a self-deadlock: + +

+
+ 1 int idx;
+ 2
+ 3 idx = srcu_read_lock(&ss);
+ 4 do_something();
+ 5 synchronize_srcu(&ss);
+ 6 srcu_read_unlock(&ss, idx);
+
+
+ +

+However, if line 5 acquired a mutex that was held across +a synchronize_srcu() for domain ss, +deadlock would still be possible. +Furthermore, if line 5 acquired a mutex that was held across +a synchronize_srcu() for some other domain ss1, +and if an ss1-domain SRCU read-side critical section +acquired another mutex that was held across as ss-domain +synchronize_srcu(), +deadlock would again be possible. +Such a deadlock cycle could extend across an arbitrarily large number +of different SRCU domains. +Again, with great power comes great responsibility. + +

+Unlike the other RCU flavors, SRCU read-side critical sections can +run on idle and even offline CPUs. +This ability requires that srcu_read_lock() and +srcu_read_unlock() contain memory barriers, which means +that SRCU readers will run a bit slower than would RCU readers. +It also motivates the smp_mb__after_srcu_read_unlock() +API, which, in combination with srcu_read_unlock(), +guarantees a full memory barrier. + +

+The +SRCU API +includes +srcu_read_lock(), +srcu_read_unlock(), +srcu_dereference(), +srcu_dereference_check(), +synchronize_srcu(), +synchronize_srcu_expedited(), +call_srcu(), +srcu_barrier(), and +srcu_read_lock_held(). +It also includes +DEFINE_SRCU(), +DEFINE_STATIC_SRCU(), and +init_srcu_struct() +APIs for defining and initializing srcu_struct structures. + +

Tasks RCU

+ +

+Some forms of tracing use “tramopolines” to handle the +binary rewriting required to install different types of probes. +It would be good to be able to free old trampolines, which sounds +like a job for some form of RCU. +However, because it is necessary to be able to install a trace +anywhere in the code, it is not possible to use read-side markers +such as rcu_read_lock() and rcu_read_unlock(). +In addition, it does not work to have these markers in the trampoline +itself, because there would need to be instructions following +rcu_read_unlock(). +Although synchronize_rcu() would guarantee that execution +reached the rcu_read_unlock(), it would not be able to +guarantee that execution had completely left the trampoline. + +

+The solution, in the form of +Tasks RCU, +is to have implicit +read-side critical sections that are delimited by voluntary context +switches, that is, calls to schedule(), +cond_resched_rcu_qs(), and +synchronize_rcu_tasks(). +In addition, transitions to and from userspace execution also delimit +tasks-RCU read-side critical sections. + +

+The tasks-RCU API is quite compact, consisting only of +call_rcu_tasks(), +synchronize_rcu_tasks(), and +rcu_barrier_tasks(). + +

Possible Future Changes

+ +

+One of the tricks that RCU uses to attain update-side scalability is +to increase grace-period latency with increasing numbers of CPUs. +If this becomes a serious problem, it will be necessary to rework the +grace-period state machine so as to avoid the need for the additional +latency. + +

+Expedited grace periods scan the CPUs, so their latency and overhead +increases with increasing numbers of CPUs. +If this becomes a serious problem on large systems, it will be necessary +to do some redesign to avoid this scalability problem. + +

+RCU disables CPU hotplug in a few places, perhaps most notably in the +expedited grace-period and rcu_barrier() operations. +If there is a strong reason to use expedited grace periods in CPU-hotplug +notifiers, it will be necessary to avoid disabling CPU hotplug. +This would introduce some complexity, so there had better be a very +good reason. + +

+The tradeoff between grace-period latency on the one hand and interruptions +of other CPUs on the other hand may need to be re-examined. +The desire is of course for zero grace-period latency as well as zero +interprocessor interrupts undertaken during an expedited grace period +operation. +While this ideal is unlikely to be achievable, it is quite possible that +further improvements can be made. + +

+The multiprocessor implementations of RCU use a combining tree that +groups CPUs so as to reduce lock contention and increase cache locality. +However, this combining tree does not spread its memory across NUMA +nodes nor does it align the CPU groups with hardware features such +as sockets or cores. +Such spreading and alignment is currently believed to be unnecessary +because the hotpath read-side primitives do not access the combining +tree, nor does call_rcu() in the common case. +If you believe that your architecture needs such spreading and alignment, +then your architecture should also benefit from the +rcutree.rcu_fanout_leaf boot parameter, which can be set +to the number of CPUs in a socket, NUMA node, or whatever. +If the number of CPUs is too large, use a fraction of the number of +CPUs. +If the number of CPUs is a large prime number, well, that certainly +is an “interesting” architectural choice! +More flexible arrangements might be considered, but only if +rcutree.rcu_fanout_leaf has proven inadequate, and only +if the inadequacy has been demonstrated by a carefully run and +realistic system-level workload. + +

+Please note that arrangements that require RCU to remap CPU numbers will +require extremely good demonstration of need and full exploration of +alternatives. + +

+There is an embarrassingly large number of flavors of RCU, and this +number has been increasing over time. +Perhaps it will be possible to combine some at some future date. + +

+RCU's various kthreads are reasonably recent additions. +It is quite likely that adjustments will be required to more gracefully +handle extreme loads. +It might also be necessary to be able to relate CPU utilization by +RCU's kthreads and softirq handlers to the code that instigated this +CPU utilization. +For example, RCU callback overhead might be charged back to the +originating call_rcu() instance, though probably not +in production kernels. + +

Summary

+ +

+This document has presented more than two decade's worth of RCU +requirements. +Given that the requirements keep changing, this will not be the last +word on this subject, but at least it serves to get an important +subset of the requirements set forth. + +

Acknowledgments

+ +I am grateful to Steven Rostedt, Lai Jiangshan, Ingo Molnar, +Oleg Nesterov, Borislav Petkov, Peter Zijlstra, Boqun Feng, and +Andy Lutomirski for their help in rendering +this article human readable, and to Michelle Rankin for her support +of this effort. +Other contributions are acknowledged in the Linux kernel's git archive. +The cartoon is copyright (c) 2013 by Melissa Broussard, +and is provided +under the terms of the Creative Commons Attribution-Share Alike 3.0 +United States license. + +

+Answers to Quick Quizzes

+ + +

Quick Quiz 1: +Wait a minute! +You said that updaters can make useful forward progress concurrently +with readers, but pre-existing readers will block +synchronize_rcu()!!! +Just who are you trying to fool??? + + +

Answer: +First, if updaters do not wish to be blocked by readers, they can use +call_rcu() or kfree_rcu(), which will +be discussed later. +Second, even when using synchronize_rcu(), the other +update-side code does run concurrently with readers, whether pre-existing +or not. + + +

Back to Quick Quiz 1. + + +

Quick Quiz 2: +Why is the synchronize_rcu() on line 28 needed? + + +

Answer: +Without that extra grace period, memory reordering could result in +do_something_dlm() executing do_something() +concurrently with the last bits of recovery(). + + +

Back to Quick Quiz 2. + + +

Quick Quiz 3: +But rcu_assign_pointer() does nothing to prevent the +two assignments to p->a and p->b +from being reordered. +Can't that also cause problems? + + +

Answer: +No, it cannot. +The readers cannot see either of these two fields until +the assignment to gp, by which time both fields are +fully initialized. +So reordering the assignments +to p->a and p->b cannot possibly +cause any problems. + + +

Back to Quick Quiz 3. + + +

Quick Quiz 4: +Without the rcu_dereference() or the +rcu_access_pointer(), what destructive optimizations +might the compiler make use of? + + +

Answer: +Let's start with what happens to do_something_gp() +if it fails to use rcu_dereference(). +It could reuse a value formerly fetched from this same pointer. +It could also fetch the pointer from gp in a byte-at-a-time +manner, resulting in load tearing, in turn resulting a bytewise +mash-up of two distince pointer values. +It might even use value-speculation optimizations, where it makes a wrong +guess, but by the time it gets around to checking the value, an update +has changed the pointer to match the wrong guess. +Too bad about any dereferences that returned pre-initialization garbage +in the meantime! + +

+For remove_gp_synchronous(), as long as all modifications +to gp are carried out while holding gp_lock, +the above optimizations are harmless. +However, +with CONFIG_SPARSE_RCU_POINTER=y, +sparse will complain if you +define gp with __rcu and then +access it without using +either rcu_access_pointer() or rcu_dereference(). + + +

Back to Quick Quiz 4. + + +

Quick Quiz 5: +Given that multiple CPUs can start RCU read-side critical sections +at any time without any ordering whatsoever, how can RCU possibly tell whether +or not a given RCU read-side critical section starts before a +given instance of synchronize_rcu()? + + +

Answer: +If RCU cannot tell whether or not a given +RCU read-side critical section starts before a +given instance of synchronize_rcu(), +then it must assume that the RCU read-side critical section +started first. +In other words, a given instance of synchronize_rcu() +can avoid waiting on a given RCU read-side critical section only +if it can prove that synchronize_rcu() started first. + + +

Back to Quick Quiz 5. + + +

Quick Quiz 6: +The first and second guarantees require unbelievably strict ordering! +Are all these memory barriers really required? + + +

Answer: +Yes, they really are required. +To see why the first guarantee is required, consider the following +sequence of events: + +

    +
  1. CPU 1: rcu_read_lock() +
  2. CPU 1: q = rcu_dereference(gp); + /* Very likely to return p. */ +
  3. CPU 0: list_del_rcu(p); +
  4. CPU 0: synchronize_rcu() starts. +
  5. CPU 1: do_something_with(q->a); + /* No smp_mb(), so might happen after kfree(). */ +
  6. CPU 1: rcu_read_unlock() +
  7. CPU 0: synchronize_rcu() returns. +
  8. CPU 0: kfree(p); +
+ +

+Therefore, there absolutely must be a full memory barrier between the +end of the RCU read-side critical section and the end of the +grace period. + +

+The sequence of events demonstrating the necessity of the second rule +is roughly similar: + +

    +
  1. CPU 0: list_del_rcu(p); +
  2. CPU 0: synchronize_rcu() starts. +
  3. CPU 1: rcu_read_lock() +
  4. CPU 1: q = rcu_dereference(gp); + /* Might return p if no memory barrier. */ +
  5. CPU 0: synchronize_rcu() returns. +
  6. CPU 0: kfree(p); +
  7. CPU 1: do_something_with(q->a); /* Boom!!! */ +
  8. CPU 1: rcu_read_unlock() +
+ +

+And similarly, without a memory barrier between the beginning of the +grace period and the beginning of the RCU read-side critical section, +CPU 1 might end up accessing the freelist. + +

+The “as if” rule of course applies, so that any implementation +that acts as if the appropriate memory barriers were in place is a +correct implementation. +That said, it is much easier to fool yourself into believing that you have +adhered to the as-if rule than it is to actually adhere to it! + + +

Back to Quick Quiz 6. + + +

Quick Quiz 7: +But how does the upgrade-to-write operation exclude other readers? + + +

Answer: +It doesn't, just like normal RCU updates, which also do not exclude +RCU readers. + + +

Back to Quick Quiz 7. + + +

Quick Quiz 8: +Can't the compiler also reorder this code? + + +

Answer: +No, the volatile casts in READ_ONCE() and +WRITE_ONCE() prevent the compiler from reordering in +this particular case. + + +

Back to Quick Quiz 8. + + +

Quick Quiz 9: +Suppose that synchronize_rcu() did wait until all readers had completed. +Would the updater be able to rely on this? + + +

Answer: +No. +Even if synchronize_rcu() were to wait until +all readers had completed, a new reader might start immediately after +synchronize_rcu() completed. +Therefore, the code following +synchronize_rcu() cannot rely on there being no readers +in any case. + + +

Back to Quick Quiz 9. + + +

Quick Quiz 10: +How long a sequence of grace periods, each separated by an RCU read-side +critical section, would be required to partition the RCU read-side +critical sections at the beginning and end of the chain? + + +

Answer: +In theory, an infinite number. +In practice, an unknown number that is sensitive to both implementation +details and timing considerations. +Therefore, even in practice, RCU users must abide by the theoretical rather +than the practical answer. + + +

Back to Quick Quiz 10. + + +

Quick Quiz 11: +What about sleeping locks? + + +

Answer: +These are forbidden within Linux-kernel RCU read-side critical sections +because it is not legal to place a quiescent state (in this case, +voluntary context switch) within an RCU read-side critical section. +However, sleeping locks may be used within userspace RCU read-side critical +sections, and also within Linux-kernel sleepable RCU +(SRCU) +read-side critical sections. +In addition, the -rt patchset turns spinlocks into a sleeping locks so +that the corresponding critical sections can be preempted, which +also means that these sleeplockified spinlocks (but not other sleeping locks!) +may be acquire within -rt-Linux-kernel RCU read-side critical sections. + +

+Note that it is legal for a normal RCU read-side critical section +to conditionally acquire a sleeping locks (as in mutex_trylock()), +but only as long as it does not loop indefinitely attempting to +conditionally acquire that sleeping locks. +The key point is that things like mutex_trylock() +either return with the mutex held, or return an error indication if +the mutex was not immediately available. +Either way, mutex_trylock() returns immediately without sleeping. + + +

Back to Quick Quiz 11. + + +

Quick Quiz 12: +Why does line 19 use rcu_access_pointer()? +After all, call_rcu() on line 25 stores into the +structure, which would interact badly with concurrent insertions. +Doesn't this mean that rcu_dereference() is required? + + +

Answer: +Presumably the ->gp_lock acquired on line 18 excludes +any changes, including any insertions that rcu_dereference() +would protect against. +Therefore, any insertions will be delayed until after ->gp_lock +is released on line 25, which in turn means that +rcu_access_pointer() suffices. + + +

Back to Quick Quiz 12. + + +

Quick Quiz 13: +Earlier it was claimed that call_rcu() and +kfree_rcu() allowed updaters to avoid being blocked +by readers. +But how can that be correct, given that the invocation of the callback +and the freeing of the memory (respectively) must still wait for +a grace period to elapse? + + +

Answer: +We could define things this way, but keep in mind that this sort of +definition would say that updates in garbage-collected languages +cannot complete until the next time the garbage collector runs, +which does not seem at all reasonable. +The key point is that in most cases, an updater using either +call_rcu() or kfree_rcu() can proceed to the +next update as soon as it has invoked call_rcu() or +kfree_rcu(), without having to wait for a subsequent +grace period. + + +

Back to Quick Quiz 13. + + +

Quick Quiz 14: +So what happens with synchronize_rcu() during +scheduler initialization for CONFIG_PREEMPT=n +kernels? + + +

Answer: +In CONFIG_PREEMPT=n kernel, synchronize_rcu() +maps directly to synchronize_sched(). +Therefore, synchronize_rcu() works normally throughout +boot in CONFIG_PREEMPT=n kernels. +However, your code must also work in CONFIG_PREEMPT=y kernels, +so it is still necessary to avoid invoking synchronize_rcu() +during scheduler initialization. + + +

Back to Quick Quiz 14. + + + diff --git a/Documentation/RCU/Design/Requirements/Requirements.htmlx b/Documentation/RCU/Design/Requirements/Requirements.htmlx new file mode 100644 index 000000000000..c5bb5bddd7ee --- /dev/null +++ b/Documentation/RCU/Design/Requirements/Requirements.htmlx @@ -0,0 +1,2643 @@ + + + A Tour Through RCU's Requirements [LWN.net] + + +

A Tour Through RCU's Requirements

+ +

Copyright IBM Corporation, 2015

+

Author: Paul E. McKenney

+

The initial version of this document appeared in the +LWN articles +here, +here, and +here.

+ +

Introduction

+ +

+Read-copy update (RCU) is a synchronization mechanism that is often +used as a replacement for reader-writer locking. +RCU is unusual in that updaters do not block readers, +which means that RCU's read-side primitives can be exceedingly fast +and scalable. +In addition, updaters can make useful forward progress concurrently +with readers. +However, all this concurrency between RCU readers and updaters does raise +the question of exactly what RCU readers are doing, which in turn +raises the question of exactly what RCU's requirements are. + +

+This document therefore summarizes RCU's requirements, and can be thought +of as an informal, high-level specification for RCU. +It is important to understand that RCU's specification is primarily +empirical in nature; +in fact, I learned about many of these requirements the hard way. +This situation might cause some consternation, however, not only +has this learning process been a lot of fun, but it has also been +a great privilege to work with so many people willing to apply +technologies in interesting new ways. + +

+All that aside, here are the categories of currently known RCU requirements: +

+ +
    +
  1. + Fundamental Requirements +
  2. Fundamental Non-Requirements +
  3. + Parallelism Facts of Life +
  4. + Quality-of-Implementation Requirements +
  5. + Linux Kernel Complications +
  6. + Software-Engineering Requirements +
  7. + Other RCU Flavors +
  8. + Possible Future Changes +
+ +

+This is followed by a summary, +which is in turn followed by the inevitable +answers to the quick quizzes. + +

Fundamental Requirements

+ +

+RCU's fundamental requirements are the closest thing RCU has to hard +mathematical requirements. +These are: + +

    +
  1. + Grace-Period Guarantee +
  2. + Publish-Subscribe Guarantee +
  3. + RCU Primitives Guaranteed to Execute Unconditionally +
  4. + Guaranteed Read-to-Write Upgrade +
+ +

Grace-Period Guarantee

+ +

+RCU's grace-period guarantee is unusual in being premeditated: +Jack Slingwine and I had this guarantee firmly in mind when we started +work on RCU (then called “rclock”) in the early 1990s. +That said, the past two decades of experience with RCU have produced +a much more detailed understanding of this guarantee. + +

+RCU's grace-period guarantee allows updaters to wait for the completion +of all pre-existing RCU read-side critical sections. +An RCU read-side critical section +begins with the marker rcu_read_lock() and ends with +the marker rcu_read_unlock(). +These markers may be nested, and RCU treats a nested set as one +big RCU read-side critical section. +Production-quality implementations of rcu_read_lock() and +rcu_read_unlock() are extremely lightweight, and in +fact have exactly zero overhead in Linux kernels built for production +use with CONFIG_PREEMPT=n. + +

+This guarantee allows ordering to be enforced with extremely low +overhead to readers, for example: + +

+
+ 1 int x, y;
+ 2
+ 3 void thread0(void)
+ 4 {
+ 5   rcu_read_lock();
+ 6   r1 = READ_ONCE(x);
+ 7   r2 = READ_ONCE(y);
+ 8   rcu_read_unlock();
+ 9 }
+10
+11 void thread1(void)
+12 {
+13   WRITE_ONCE(x, 1);
+14   synchronize_rcu();
+15   WRITE_ONCE(y, 1);
+16 }
+
+
+ +

+Because the synchronize_rcu() on line 14 waits for +all pre-existing readers, any instance of thread0() that +loads a value of zero from x must complete before +thread1() stores to y, so that instance must +also load a value of zero from y. +Similarly, any instance of thread0() that loads a value of +one from y must have started after the +synchronize_rcu() started, and must therefore also load +a value of one from x. +Therefore, the outcome: +

+
+(r1 == 0 && r2 == 1)
+
+
+cannot happen. + +

@@QQ@@ +Wait a minute! +You said that updaters can make useful forward progress concurrently +with readers, but pre-existing readers will block +synchronize_rcu()!!! +Just who are you trying to fool??? +

@@QQA@@ +First, if updaters do not wish to be blocked by readers, they can use +call_rcu() or kfree_rcu(), which will +be discussed later. +Second, even when using synchronize_rcu(), the other +update-side code does run concurrently with readers, whether pre-existing +or not. +

@@QQE@@ + +

+This scenario resembles one of the first uses of RCU in +DYNIX/ptx, +which managed a distributed lock manager's transition into +a state suitable for handling recovery from node failure, +more or less as follows: + +

+
+ 1 #define STATE_NORMAL        0
+ 2 #define STATE_WANT_RECOVERY 1
+ 3 #define STATE_RECOVERING    2
+ 4 #define STATE_WANT_NORMAL   3
+ 5
+ 6 int state = STATE_NORMAL;
+ 7
+ 8 void do_something_dlm(void)
+ 9 {
+10   int state_snap;
+11
+12   rcu_read_lock();
+13   state_snap = READ_ONCE(state);
+14   if (state_snap == STATE_NORMAL)
+15     do_something();
+16   else
+17     do_something_carefully();
+18   rcu_read_unlock();
+19 }
+20
+21 void start_recovery(void)
+22 {
+23   WRITE_ONCE(state, STATE_WANT_RECOVERY);
+24   synchronize_rcu();
+25   WRITE_ONCE(state, STATE_RECOVERING);
+26   recovery();
+27   WRITE_ONCE(state, STATE_WANT_NORMAL);
+28   synchronize_rcu();
+29   WRITE_ONCE(state, STATE_NORMAL);
+30 }
+
+
+ +

+The RCU read-side critical section in do_something_dlm() +works with the synchronize_rcu() in start_recovery() +to guarantee that do_something() never runs concurrently +with recovery(), but with little or no synchronization +overhead in do_something_dlm(). + +

@@QQ@@ +Why is the synchronize_rcu() on line 28 needed? +

@@QQA@@ +Without that extra grace period, memory reordering could result in +do_something_dlm() executing do_something() +concurrently with the last bits of recovery(). +

@@QQE@@ + +

+In order to avoid fatal problems such as deadlocks, +an RCU read-side critical section must not contain calls to +synchronize_rcu(). +Similarly, an RCU read-side critical section must not +contain anything that waits, directly or indirectly, on completion of +an invocation of synchronize_rcu(). + +

+Although RCU's grace-period guarantee is useful in and of itself, with +quite a few use cases, +it would be good to be able to use RCU to coordinate read-side +access to linked data structures. +For this, the grace-period guarantee is not sufficient, as can +be seen in function add_gp_buggy() below. +We will look at the reader's code later, but in the meantime, just think of +the reader as locklessly picking up the gp pointer, +and, if the value loaded is non-NULL, locklessly accessing the +->a and ->b fields. + +

+
+ 1 bool add_gp_buggy(int a, int b)
+ 2 {
+ 3   p = kmalloc(sizeof(*p), GFP_KERNEL);
+ 4   if (!p)
+ 5     return -ENOMEM;
+ 6   spin_lock(&gp_lock);
+ 7   if (rcu_access_pointer(gp)) {
+ 8     spin_unlock(&gp_lock);
+ 9     return false;
+10   }
+11   p->a = a;
+12   p->b = a;
+13   gp = p; /* ORDERING BUG */
+14   spin_unlock(&gp_lock);
+15   return true;
+16 }
+
+
+ +

+The problem is that both the compiler and weakly ordered CPUs are within +their rights to reorder this code as follows: + +

+
+ 1 bool add_gp_buggy_optimized(int a, int b)
+ 2 {
+ 3   p = kmalloc(sizeof(*p), GFP_KERNEL);
+ 4   if (!p)
+ 5     return -ENOMEM;
+ 6   spin_lock(&gp_lock);
+ 7   if (rcu_access_pointer(gp)) {
+ 8     spin_unlock(&gp_lock);
+ 9     return false;
+10   }
+11   gp = p; /* ORDERING BUG */
+12   p->a = a;
+13   p->b = a;
+14   spin_unlock(&gp_lock);
+15   return true;
+16 }
+
+
+ +

+If an RCU reader fetches gp just after +add_gp_buggy_optimized executes line 11, +it will see garbage in the ->a and ->b +fields. +And this is but one of many ways in which compiler and hardware optimizations +could cause trouble. +Therefore, we clearly need some way to prevent the compiler and the CPU from +reordering in this manner, which brings us to the publish-subscribe +guarantee discussed in the next section. + +

Publish/Subscribe Guarantee

+ +

+RCU's publish-subscribe guarantee allows data to be inserted +into a linked data structure without disrupting RCU readers. +The updater uses rcu_assign_pointer() to insert the +new data, and readers use rcu_dereference() to +access data, whether new or old. +The following shows an example of insertion: + +

+
+ 1 bool add_gp(int a, int b)
+ 2 {
+ 3   p = kmalloc(sizeof(*p), GFP_KERNEL);
+ 4   if (!p)
+ 5     return -ENOMEM;
+ 6   spin_lock(&gp_lock);
+ 7   if (rcu_access_pointer(gp)) {
+ 8     spin_unlock(&gp_lock);
+ 9     return false;
+10   }
+11   p->a = a;
+12   p->b = a;
+13   rcu_assign_pointer(gp, p);
+14   spin_unlock(&gp_lock);
+15   return true;
+16 }
+
+
+ +

+The rcu_assign_pointer() on line 13 is conceptually +equivalent to a simple assignment statement, but also guarantees +that its assignment will +happen after the two assignments in lines 11 and 12, +similar to the C11 memory_order_release store operation. +It also prevents any number of “interesting” compiler +optimizations, for example, the use of gp as a scratch +location immediately preceding the assignment. + +

@@QQ@@ +But rcu_assign_pointer() does nothing to prevent the +two assignments to p->a and p->b +from being reordered. +Can't that also cause problems? +

@@QQA@@ +No, it cannot. +The readers cannot see either of these two fields until +the assignment to gp, by which time both fields are +fully initialized. +So reordering the assignments +to p->a and p->b cannot possibly +cause any problems. +

@@QQE@@ + +

+It is tempting to assume that the reader need not do anything special +to control its accesses to the RCU-protected data, +as shown in do_something_gp_buggy() below: + +

+
+ 1 bool do_something_gp_buggy(void)
+ 2 {
+ 3   rcu_read_lock();
+ 4   p = gp;  /* OPTIMIZATIONS GALORE!!! */
+ 5   if (p) {
+ 6     do_something(p->a, p->b);
+ 7     rcu_read_unlock();
+ 8     return true;
+ 9   }
+10   rcu_read_unlock();
+11   return false;
+12 }
+
+
+ +

+However, this temptation must be resisted because there are a +surprisingly large number of ways that the compiler +(to say nothing of +DEC Alpha CPUs) +can trip this code up. +For but one example, if the compiler were short of registers, it +might choose to refetch from gp rather than keeping +a separate copy in p as follows: + +

+
+ 1 bool do_something_gp_buggy_optimized(void)
+ 2 {
+ 3   rcu_read_lock();
+ 4   if (gp) { /* OPTIMIZATIONS GALORE!!! */
+ 5     do_something(gp->a, gp->b);
+ 6     rcu_read_unlock();
+ 7     return true;
+ 8   }
+ 9   rcu_read_unlock();
+10   return false;
+11 }
+
+
+ +

+If this function ran concurrently with a series of updates that +replaced the current structure with a new one, +the fetches of gp->a +and gp->b might well come from two different structures, +which could cause serious confusion. +To prevent this (and much else besides), do_something_gp() uses +rcu_dereference() to fetch from gp: + +

+
+ 1 bool do_something_gp(void)
+ 2 {
+ 3   rcu_read_lock();
+ 4   p = rcu_dereference(gp);
+ 5   if (p) {
+ 6     do_something(p->a, p->b);
+ 7     rcu_read_unlock();
+ 8     return true;
+ 9   }
+10   rcu_read_unlock();
+11   return false;
+12 }
+
+
+ +

+The rcu_dereference() uses volatile casts and (for DEC Alpha) +memory barriers in the Linux kernel. +Should a +high-quality implementation of C11 memory_order_consume [PDF] +ever appear, then rcu_dereference() could be implemented +as a memory_order_consume load. +Regardless of the exact implementation, a pointer fetched by +rcu_dereference() may not be used outside of the +outermost RCU read-side critical section containing that +rcu_dereference(), unless protection of +the corresponding data element has been passed from RCU to some +other synchronization mechanism, most commonly locking or +reference counting. + +

+In short, updaters use rcu_assign_pointer() and readers +use rcu_dereference(), and these two RCU API elements +work together to ensure that readers have a consistent view of +newly added data elements. + +

+Of course, it is also necessary to remove elements from RCU-protected +data structures, for example, using the following process: + +

    +
  1. Remove the data element from the enclosing structure. +
  2. Wait for all pre-existing RCU read-side critical sections + to complete (because only pre-existing readers can possibly have + a reference to the newly removed data element). +
  3. At this point, only the updater has a reference to the + newly removed data element, so it can safely reclaim + the data element, for example, by passing it to kfree(). +
+ +This process is implemented by remove_gp_synchronous(): + +
+
+ 1 bool remove_gp_synchronous(void)
+ 2 {
+ 3   struct foo *p;
+ 4
+ 5   spin_lock(&gp_lock);
+ 6   p = rcu_access_pointer(gp);
+ 7   if (!p) {
+ 8     spin_unlock(&gp_lock);
+ 9     return false;
+10   }
+11   rcu_assign_pointer(gp, NULL);
+12   spin_unlock(&gp_lock);
+13   synchronize_rcu();
+14   kfree(p);
+15   return true;
+16 }
+
+
+ +

+This function is straightforward, with line 13 waiting for a grace +period before line 14 frees the old data element. +This waiting ensures that readers will reach line 7 of +do_something_gp() before the data element referenced by +p is freed. +The rcu_access_pointer() on line 6 is similar to +rcu_dereference(), except that: + +

    +
  1. The value returned by rcu_access_pointer() + cannot be dereferenced. + If you want to access the value pointed to as well as + the pointer itself, use rcu_dereference() + instead of rcu_access_pointer(). +
  2. The call to rcu_access_pointer() need not be + protected. + In contrast, rcu_dereference() must either be + within an RCU read-side critical section or in a code + segment where the pointer cannot change, for example, in + code protected by the corresponding update-side lock. +
+ +

@@QQ@@ +Without the rcu_dereference() or the +rcu_access_pointer(), what destructive optimizations +might the compiler make use of? +

@@QQA@@ +Let's start with what happens to do_something_gp() +if it fails to use rcu_dereference(). +It could reuse a value formerly fetched from this same pointer. +It could also fetch the pointer from gp in a byte-at-a-time +manner, resulting in load tearing, in turn resulting a bytewise +mash-up of two distince pointer values. +It might even use value-speculation optimizations, where it makes a wrong +guess, but by the time it gets around to checking the value, an update +has changed the pointer to match the wrong guess. +Too bad about any dereferences that returned pre-initialization garbage +in the meantime! + +

+For remove_gp_synchronous(), as long as all modifications +to gp are carried out while holding gp_lock, +the above optimizations are harmless. +However, +with CONFIG_SPARSE_RCU_POINTER=y, +sparse will complain if you +define gp with __rcu and then +access it without using +either rcu_access_pointer() or rcu_dereference(). +

@@QQE@@ + +

+This simple linked-data-structure scenario clearly demonstrates the need +for RCU's stringent memory-ordering guarantees on systems with more than +one CPU: + +

    +
  1. Each CPU that has an RCU read-side critical section that + begins before synchronize_rcu() starts is + guaranteed to execute a full memory barrier between the time + that the RCU read-side critical section ends and the time that + synchronize_rcu() returns. + Without this guarantee, a pre-existing RCU read-side critical section + might hold a reference to the newly removed struct foo + after the kfree() on line 14 of + remove_gp_synchronous(). +
  2. Each CPU that has an RCU read-side critical section that ends + after synchronize_rcu() returns is guaranteed + to execute a full memory barrier between the time that + synchronize_rcu() begins and the time that the RCU + read-side critical section begins. + Without this guarantee, a later RCU read-side critical section + running after the kfree() on line 14 of + remove_gp_synchronous() might + later run do_something_gp() and find the + newly deleted struct foo. +
  3. If the task invoking synchronize_rcu() remains + on a given CPU, then that CPU is guaranteed to execute a full + memory barrier sometime during the execution of + synchronize_rcu(). + This guarantee ensures that the kfree() on + line 14 of remove_gp_synchronous() really does + execute after the removal on line 11. +
  4. If the task invoking synchronize_rcu() migrates + among a group of CPUs during that invocation, then each of the + CPUs in that group is guaranteed to execute a full memory barrier + sometime during the execution of synchronize_rcu(). + This guarantee also ensures that the kfree() on + line 14 of remove_gp_synchronous() really does + execute after the removal on + line 11, but also in the case where the thread executing the + synchronize_rcu() migrates in the meantime. +
+ +

@@QQ@@ +Given that multiple CPUs can start RCU read-side critical sections +at any time without any ordering whatsoever, how can RCU possibly tell whether +or not a given RCU read-side critical section starts before a +given instance of synchronize_rcu()? +

@@QQA@@ +If RCU cannot tell whether or not a given +RCU read-side critical section starts before a +given instance of synchronize_rcu(), +then it must assume that the RCU read-side critical section +started first. +In other words, a given instance of synchronize_rcu() +can avoid waiting on a given RCU read-side critical section only +if it can prove that synchronize_rcu() started first. +

@@QQE@@ + +

@@QQ@@ +The first and second guarantees require unbelievably strict ordering! +Are all these memory barriers really required? +

@@QQA@@ +Yes, they really are required. +To see why the first guarantee is required, consider the following +sequence of events: + +

    +
  1. CPU 1: rcu_read_lock() +
  2. CPU 1: q = rcu_dereference(gp); + /* Very likely to return p. */ +
  3. CPU 0: list_del_rcu(p); +
  4. CPU 0: synchronize_rcu() starts. +
  5. CPU 1: do_something_with(q->a); + /* No smp_mb(), so might happen after kfree(). */ +
  6. CPU 1: rcu_read_unlock() +
  7. CPU 0: synchronize_rcu() returns. +
  8. CPU 0: kfree(p); +
+ +

+Therefore, there absolutely must be a full memory barrier between the +end of the RCU read-side critical section and the end of the +grace period. + +

+The sequence of events demonstrating the necessity of the second rule +is roughly similar: + +

    +
  1. CPU 0: list_del_rcu(p); +
  2. CPU 0: synchronize_rcu() starts. +
  3. CPU 1: rcu_read_lock() +
  4. CPU 1: q = rcu_dereference(gp); + /* Might return p if no memory barrier. */ +
  5. CPU 0: synchronize_rcu() returns. +
  6. CPU 0: kfree(p); +
  7. CPU 1: do_something_with(q->a); /* Boom!!! */ +
  8. CPU 1: rcu_read_unlock() +
+ +

+And similarly, without a memory barrier between the beginning of the +grace period and the beginning of the RCU read-side critical section, +CPU 1 might end up accessing the freelist. + +

+The “as if” rule of course applies, so that any implementation +that acts as if the appropriate memory barriers were in place is a +correct implementation. +That said, it is much easier to fool yourself into believing that you have +adhered to the as-if rule than it is to actually adhere to it! +

@@QQE@@ + +

+In short, RCU's publish-subscribe guarantee is provided by the combination +of rcu_assign_pointer() and rcu_dereference(). +This guarantee allows data elements to be safely added to RCU-protected +linked data structures without disrupting RCU readers. +This guarantee can be used in combination with the grace-period +guarantee to also allow data elements to be removed from RCU-protected +linked data structures, again without disrupting RCU readers. + +

+This guarantee was only partially premeditated. +DYNIX/ptx used an explicit memory barrier for publication, but had nothing +resembling rcu_dereference() for subscription, nor did it +have anything resembling the smp_read_barrier_depends() +that was later subsumed into rcu_dereference(). +The need for these operations made itself known quite suddenly at a +late-1990s meeting with the DEC Alpha architects, back in the days when +DEC was still a free-standing company. +It took the Alpha architects a good hour to convince me that any sort +of barrier would ever be needed, and it then took me a good two hours +to convince them that their documentation did not make this point clear. +More recent work with the C and C++ standards committees have provided +much education on tricks and traps from the compiler. +In short, compilers were much less tricky in the early 1990s, but in +2015, don't even think about omitting rcu_dereference()! + +

RCU Primitives Guaranteed to Execute Unconditionally

+ +

+The common-case RCU primitives are unconditional. +They are invoked, they do their job, and they return, with no possibility +of error, and no need to retry. +This is a key RCU design philosophy. + +

+However, this philosophy is pragmatic rather than pigheaded. +If someone comes up with a good justification for a particular conditional +RCU primitive, it might well be implemented and added. +After all, this guarantee was reverse-engineered, not premeditated. +The unconditional nature of the RCU primitives was initially an +accident of implementation, and later experience with synchronization +primitives with conditional primitives caused me to elevate this +accident to a guarantee. +Therefore, the justification for adding a conditional primitive to +RCU would need to be based on detailed and compelling use cases. + +

Guaranteed Read-to-Write Upgrade

+ +

+As far as RCU is concerned, it is always possible to carry out an +update within an RCU read-side critical section. +For example, that RCU read-side critical section might search for +a given data element, and then might acquire the update-side +spinlock in order to update that element, all while remaining +in that RCU read-side critical section. +Of course, it is necessary to exit the RCU read-side critical section +before invoking synchronize_rcu(), however, this +inconvenience can be avoided through use of the +call_rcu() and kfree_rcu() API members +described later in this document. + +

@@QQ@@ +But how does the upgrade-to-write operation exclude other readers? +

@@QQA@@ +It doesn't, just like normal RCU updates, which also do not exclude +RCU readers. +

@@QQE@@ + +

+This guarantee allows lookup code to be shared between read-side +and update-side code, and was premeditated, appearing in the earliest +DYNIX/ptx RCU documentation. + +

Fundamental Non-Requirements

+ +

+RCU provides extremely lightweight readers, and its read-side guarantees, +though quite useful, are correspondingly lightweight. +It is therefore all too easy to assume that RCU is guaranteeing more +than it really is. +Of course, the list of things that RCU does not guarantee is infinitely +long, however, the following sections list a few non-guarantees that +have caused confusion. +Except where otherwise noted, these non-guarantees were premeditated. + +

    +
  1. + Readers Impose Minimal Ordering +
  2. + Readers Do Not Exclude Updaters +
  3. + Updaters Only Wait For Old Readers +
  4. + Grace Periods Don't Partition Read-Side Critical Sections +
  5. + Read-Side Critical Sections Don't Partition Grace Periods +
  6. + Disabling Preemption Does Not Block Grace Periods +
+ +

Readers Impose Minimal Ordering

+ +

+Reader-side markers such as rcu_read_lock() and +rcu_read_unlock() provide absolutely no ordering guarantees +except through their interaction with the grace-period APIs such as +synchronize_rcu(). +To see this, consider the following pair of threads: + +

+
+ 1 void thread0(void)
+ 2 {
+ 3   rcu_read_lock();
+ 4   WRITE_ONCE(x, 1);
+ 5   rcu_read_unlock();
+ 6   rcu_read_lock();
+ 7   WRITE_ONCE(y, 1);
+ 8   rcu_read_unlock();
+ 9 }
+10
+11 void thread1(void)
+12 {
+13   rcu_read_lock();
+14   r1 = READ_ONCE(y);
+15   rcu_read_unlock();
+16   rcu_read_lock();
+17   r2 = READ_ONCE(x);
+18   rcu_read_unlock();
+19 }
+
+
+ +

+After thread0() and thread1() execute +concurrently, it is quite possible to have + +

+
+(r1 == 1 && r2 == 0)
+
+
+ +(that is, y appears to have been assigned before x), +which would not be possible if rcu_read_lock() and +rcu_read_unlock() had much in the way of ordering +properties. +But they do not, so the CPU is within its rights +to do significant reordering. +This is by design: Any significant ordering constraints would slow down +these fast-path APIs. + +

@@QQ@@ +Can't the compiler also reorder this code? +

@@QQA@@ +No, the volatile casts in READ_ONCE() and +WRITE_ONCE() prevent the compiler from reordering in +this particular case. +

@@QQE@@ + +

Readers Do Not Exclude Updaters

+ +

+Neither rcu_read_lock() nor rcu_read_unlock() +exclude updates. +All they do is to prevent grace periods from ending. +The following example illustrates this: + +

+
+ 1 void thread0(void)
+ 2 {
+ 3   rcu_read_lock();
+ 4   r1 = READ_ONCE(y);
+ 5   if (r1) {
+ 6     do_something_with_nonzero_x();
+ 7     r2 = READ_ONCE(x);
+ 8     WARN_ON(!r2); /* BUG!!! */
+ 9   }
+10   rcu_read_unlock();
+11 }
+12
+13 void thread1(void)
+14 {
+15   spin_lock(&my_lock);
+16   WRITE_ONCE(x, 1);
+17   WRITE_ONCE(y, 1);
+18   spin_unlock(&my_lock);
+19 }
+
+
+ +

+If the thread0() function's rcu_read_lock() +excluded the thread1() function's update, +the WARN_ON() could never fire. +But the fact is that rcu_read_lock() does not exclude +much of anything aside from subsequent grace periods, of which +thread1() has none, so the +WARN_ON() can and does fire. + +

Updaters Only Wait For Old Readers

+ +

+It might be tempting to assume that after synchronize_rcu() +completes, there are no readers executing. +This temptation must be avoided because +new readers can start immediately after synchronize_rcu() +starts, and synchronize_rcu() is under no +obligation to wait for these new readers. + +

@@QQ@@ +Suppose that synchronize_rcu() did wait until all readers had completed. +Would the updater be able to rely on this? +

@@QQA@@ +No. +Even if synchronize_rcu() were to wait until +all readers had completed, a new reader might start immediately after +synchronize_rcu() completed. +Therefore, the code following +synchronize_rcu() cannot rely on there being no readers +in any case. +

@@QQE@@ + +

+Grace Periods Don't Partition Read-Side Critical Sections

+ +

+It is tempting to assume that if any part of one RCU read-side critical +section precedes a given grace period, and if any part of another RCU +read-side critical section follows that same grace period, then all of +the first RCU read-side critical section must precede all of the second. +However, this just isn't the case: A single grace period does not +partition the set of RCU read-side critical sections. +An example of this situation can be illustrated as follows, where +x, y, and z are initially all zero: + +

+
+ 1 void thread0(void)
+ 2 {
+ 3   rcu_read_lock();
+ 4   WRITE_ONCE(a, 1);
+ 5   WRITE_ONCE(b, 1);
+ 6   rcu_read_unlock();
+ 7 }
+ 8
+ 9 void thread1(void)
+10 {
+11   r1 = READ_ONCE(a);
+12   synchronize_rcu();
+13   WRITE_ONCE(c, 1);
+14 }
+15
+16 void thread2(void)
+17 {
+18   rcu_read_lock();
+19   r2 = READ_ONCE(b);
+20   r3 = READ_ONCE(c);
+21   rcu_read_unlock();
+22 }
+
+
+ +

+It turns out that the outcome: + +

+
+(r1 == 1 && r2 == 0 && r3 == 1)
+
+
+ +is entirely possible. +The following figure show how this can happen, with each circled +QS indicating the point at which RCU recorded a +quiescent state for each thread, that is, a state in which +RCU knows that the thread cannot be in the midst of an RCU read-side +critical section that started before the current grace period: + +

GPpartitionReaders1.svg

+ +

+If it is necessary to partition RCU read-side critical sections in this +manner, it is necessary to use two grace periods, where the first +grace period is known to end before the second grace period starts: + +

+
+ 1 void thread0(void)
+ 2 {
+ 3   rcu_read_lock();
+ 4   WRITE_ONCE(a, 1);
+ 5   WRITE_ONCE(b, 1);
+ 6   rcu_read_unlock();
+ 7 }
+ 8
+ 9 void thread1(void)
+10 {
+11   r1 = READ_ONCE(a);
+12   synchronize_rcu();
+13   WRITE_ONCE(c, 1);
+14 }
+15
+16 void thread2(void)
+17 {
+18   r2 = READ_ONCE(c);
+19   synchronize_rcu();
+20   WRITE_ONCE(d, 1);
+21 }
+22
+23 void thread3(void)
+24 {
+25   rcu_read_lock();
+26   r3 = READ_ONCE(b);
+27   r4 = READ_ONCE(d);
+28   rcu_read_unlock();
+29 }
+
+
+ +

+Here, if (r1 == 1), then +thread0()'s write to b must happen +before the end of thread1()'s grace period. +If in addition (r4 == 1), then +thread3()'s read from b must happen +after the beginning of thread2()'s grace period. +If it is also the case that (r2 == 1), then the +end of thread1()'s grace period must precede the +beginning of thread2()'s grace period. +This mean that the two RCU read-side critical sections cannot overlap, +guaranteeing that (r3 == 1). +As a result, the outcome: + +

+
+(r1 == 1 && r2 == 1 && r3 == 0 && r4 == 1)
+
+
+ +cannot happen. + +

+This non-requirement was also non-premeditated, but became apparent +when studying RCU's interaction with memory ordering. + +

+Read-Side Critical Sections Don't Partition Grace Periods

+ +

+It is also tempting to assume that if an RCU read-side critical section +happens between a pair of grace periods, then those grace periods cannot +overlap. +However, this temptation leads nowhere good, as can be illustrated by +the following, with all variables initially zero: + +

+
+ 1 void thread0(void)
+ 2 {
+ 3   rcu_read_lock();
+ 4   WRITE_ONCE(a, 1);
+ 5   WRITE_ONCE(b, 1);
+ 6   rcu_read_unlock();
+ 7 }
+ 8
+ 9 void thread1(void)
+10 {
+11   r1 = READ_ONCE(a);
+12   synchronize_rcu();
+13   WRITE_ONCE(c, 1);
+14 }
+15
+16 void thread2(void)
+17 {
+18   rcu_read_lock();
+19   WRITE_ONCE(d, 1);
+20   r2 = READ_ONCE(c);
+21   rcu_read_unlock();
+22 }
+23
+24 void thread3(void)
+25 {
+26   r3 = READ_ONCE(d);
+27   synchronize_rcu();
+28   WRITE_ONCE(e, 1);
+29 }
+30
+31 void thread4(void)
+32 {
+33   rcu_read_lock();
+34   r4 = READ_ONCE(b);
+35   r5 = READ_ONCE(e);
+36   rcu_read_unlock();
+37 }
+
+
+ +

+In this case, the outcome: + +

+
+(r1 == 1 && r2 == 1 && r3 == 1 && r4 == 0 && r5 == 1)
+
+
+ +is entirely possible, as illustrated below: + +

ReadersPartitionGP1.svg

+ +

+Again, an RCU read-side critical section can overlap almost all of a +given grace period, just so long as it does not overlap the entire +grace period. +As a result, an RCU read-side critical section cannot partition a pair +of RCU grace periods. + +

@@QQ@@ +How long a sequence of grace periods, each separated by an RCU read-side +critical section, would be required to partition the RCU read-side +critical sections at the beginning and end of the chain? +

@@QQA@@ +In theory, an infinite number. +In practice, an unknown number that is sensitive to both implementation +details and timing considerations. +Therefore, even in practice, RCU users must abide by the theoretical rather +than the practical answer. +

@@QQE@@ + +

+Disabling Preemption Does Not Block Grace Periods

+ +

+There was a time when disabling preemption on any given CPU would block +subsequent grace periods. +However, this was an accident of implementation and is not a requirement. +And in the current Linux-kernel implementation, disabling preemption +on a given CPU in fact does not block grace periods, as Oleg Nesterov +demonstrated. + +

+If you need a preempt-disable region to block grace periods, you need to add +rcu_read_lock() and rcu_read_unlock(), for example +as follows: + +

+
+ 1 preempt_disable();
+ 2 rcu_read_lock();
+ 3 do_something();
+ 4 rcu_read_unlock();
+ 5 preempt_enable();
+ 6
+ 7 /* Spinlocks implicitly disable preemption. */
+ 8 spin_lock(&mylock);
+ 9 rcu_read_lock();
+10 do_something();
+11 rcu_read_unlock();
+12 spin_unlock(&mylock);
+
+
+ +

+In theory, you could enter the RCU read-side critical section first, +but it is more efficient to keep the entire RCU read-side critical +section contained in the preempt-disable region as shown above. +Of course, RCU read-side critical sections that extend outside of +preempt-disable regions will work correctly, but such critical sections +can be preempted, which forces rcu_read_unlock() to do +more work. +And no, this is not an invitation to enclose all of your RCU +read-side critical sections within preempt-disable regions, because +doing so would degrade real-time response. + +

+This non-requirement appeared with preemptible RCU. +If you need a grace period that waits on non-preemptible code regions, use +RCU-sched. + +

Parallelism Facts of Life

+ +

+These parallelism facts of life are by no means specific to RCU, but +the RCU implementation must abide by them. +They therefore bear repeating: + +

    +
  1. Any CPU or task may be delayed at any time, + and any attempts to avoid these delays by disabling + preemption, interrupts, or whatever are completely futile. + This is most obvious in preemptible user-level + environments and in virtualized environments (where + a given guest OS's VCPUs can be preempted at any time by + the underlying hypervisor), but can also happen in bare-metal + environments due to ECC errors, NMIs, and other hardware + events. + Although a delay of more than about 20 seconds can result + in splats, the RCU implementation is obligated to use + algorithms that can tolerate extremely long delays, but where + “extremely long” is not long enough to allow + wrap-around when incrementing a 64-bit counter. +
  2. Both the compiler and the CPU can reorder memory accesses. + Where it matters, RCU must use compiler directives and + memory-barrier instructions to preserve ordering. +
  3. Conflicting writes to memory locations in any given cache line + will result in expensive cache misses. + Greater numbers of concurrent writes and more-frequent + concurrent writes will result in more dramatic slowdowns. + RCU is therefore obligated to use algorithms that have + sufficient locality to avoid significant performance and + scalability problems. +
  4. As a rough rule of thumb, only one CPU's worth of processing + may be carried out under the protection of any given exclusive + lock. + RCU must therefore use scalable locking designs. +
  5. Counters are finite, especially on 32-bit systems. + RCU's use of counters must therefore tolerate counter wrap, + or be designed such that counter wrap would take way more + time than a single system is likely to run. + An uptime of ten years is quite possible, a runtime + of a century much less so. + As an example of the latter, RCU's dyntick-idle nesting counter + allows 54 bits for interrupt nesting level (this counter + is 64 bits even on a 32-bit system). + Overflowing this counter requires 254 + half-interrupts on a given CPU without that CPU ever going idle. + If a half-interrupt happened every microsecond, it would take + 570 years of runtime to overflow this counter, which is currently + believed to be an acceptably long time. +
  6. Linux systems can have thousands of CPUs running a single + Linux kernel in a single shared-memory environment. + RCU must therefore pay close attention to high-end scalability. +
+ +

+This last parallelism fact of life means that RCU must pay special +attention to the preceding facts of life. +The idea that Linux might scale to systems with thousands of CPUs would +have been met with some skepticism in the 1990s, but these requirements +would have otherwise have been unsurprising, even in the early 1990s. + +

Quality-of-Implementation Requirements

+ +

+These sections list quality-of-implementation requirements. +Although an RCU implementation that ignores these requirements could +still be used, it would likely be subject to limitations that would +make it inappropriate for industrial-strength production use. +Classes of quality-of-implementation requirements are as follows: + +

    +
  1. Specialization +
  2. Performance and Scalability +
  3. Composability +
  4. Corner Cases +
+ +

+These classes is covered in the following sections. + +

Specialization

+ +

+RCU is and always has been intended primarily for read-mostly situations, as +illustrated by the following figure. +This means that RCU's read-side primitives are optimized, often at the +expense of its update-side primitives. + +

RCUApplicability.svg

+ +

+This focus on read-mostly situations means that RCU must interoperate +with other synchronization primitives. +For example, the add_gp() and remove_gp_synchronous() +examples discussed earlier use RCU to protect readers and locking to +coordinate updaters. +However, the need extends much farther, requiring that a variety of +synchronization primitives be legal within RCU read-side critical sections, +including spinlocks, sequence locks, atomic operations, reference +counters, and memory barriers. + +

@@QQ@@ +What about sleeping locks? +

@@QQA@@ +These are forbidden within Linux-kernel RCU read-side critical sections +because it is not legal to place a quiescent state (in this case, +voluntary context switch) within an RCU read-side critical section. +However, sleeping locks may be used within userspace RCU read-side critical +sections, and also within Linux-kernel sleepable RCU +(SRCU) +read-side critical sections. +In addition, the -rt patchset turns spinlocks into a sleeping locks so +that the corresponding critical sections can be preempted, which +also means that these sleeplockified spinlocks (but not other sleeping locks!) +may be acquire within -rt-Linux-kernel RCU read-side critical sections. + +

+Note that it is legal for a normal RCU read-side critical section +to conditionally acquire a sleeping locks (as in mutex_trylock()), +but only as long as it does not loop indefinitely attempting to +conditionally acquire that sleeping locks. +The key point is that things like mutex_trylock() +either return with the mutex held, or return an error indication if +the mutex was not immediately available. +Either way, mutex_trylock() returns immediately without sleeping. +

@@QQE@@ + +

+It often comes as a surprise that many algorithms do not require a +consistent view of data, but many can function in that mode, +with network routing being the poster child. +Internet routing algorithms take significant time to propagate +updates, so that by the time an update arrives at a given system, +that system has been sending network traffic the wrong way for +a considerable length of time. +Having a few threads continue to send traffic the wrong way for a +few more milliseconds is clearly not a problem: In the worst case, +TCP retransmissions will eventually get the data where it needs to go. +In general, when tracking the state of the universe outside of the +computer, some level of inconsistency must be tolerated due to +speed-of-light delays if nothing else. + +

+Furthermore, uncertainty about external state is inherent in many cases. +For example, a pair of veternarians might use heartbeat to determine +whether or not a given cat was alive. +But how long should they wait after the last heartbeat to decide that +the cat is in fact dead? +Waiting less than 400 milliseconds makes no sense because this would +mean that a relaxed cat would be considered to cycle between death +and life more than 100 times per minute. +Moreover, just as with human beings, a cat's heart might stop for +some period of time, so the exact wait period is a judgment call. +One of our pair of veternarians might wait 30 seconds before pronouncing +the cat dead, while the other might insist on waiting a full minute. +The two veternarians would then disagree on the state of the cat during +the final 30 seconds of the minute following the last heartbeat, as +fancifully illustrated below: + +

2013-08-is-it-dead.png

+ +

+Interestingly enough, this same situation applies to hardware. +When push comes to shove, how do we tell whether or not some +external server has failed? +We send messages to it periodically, and declare it failed if we +don't receive a response within a given period of time. +Policy decisions can usually tolerate short +periods of inconsistency. +The policy was decided some time ago, and is only now being put into +effect, so a few milliseconds of delay is normally inconsequential. + +

+However, there are algorithms that absolutely must see consistent data. +For example, the translation between a user-level SystemV semaphore +ID to the corresponding in-kernel data structure is protected by RCU, +but it is absolutely forbidden to update a semaphore that has just been +removed. +In the Linux kernel, this need for consistency is accommodated by acquiring +spinlocks located in the in-kernel data structure from within +the RCU read-side critical section, and this is indicated by the +green box in the figure above. +Many other techniques may be used, and are in fact used within the +Linux kernel. + +

+In short, RCU is not required to maintain consistency, and other +mechanisms may be used in concert with RCU when consistency is required. +RCU's specialization allows it to do its job extremely well, and its +ability to interoperate with other synchronization mechanisms allows +the right mix of synchronization tools to be used for a given job. + +

Performance and Scalability

+ +

+Energy efficiency is a critical component of performance today, +and Linux-kernel RCU implementations must therefore avoid unnecessarily +awakening idle CPUs. +I cannot claim that this requirement was premeditated. +In fact, I learned of it during a telephone conversation in which I +was given “frank and open” feedback on the importance +of energy efficiency in battery-powered systems and on specific +energy-efficiency shortcomings of the Linux-kernel RCU implementation. +In my experience, the battery-powered embedded community will consider +any unnecessary wakeups to be extremely unfriendly acts. +So much so that mere Linux-kernel-mailing-list posts are +insufficient to vent their ire. + +

+Memory consumption is not particularly important for in most +situations, and has become decreasingly +so as memory sizes have expanded and memory +costs have plummeted. +However, as I learned from Matt Mackall's +bloatwatch +efforts, memory footprint is critically important on single-CPU systems with +non-preemptible (CONFIG_PREEMPT=n) kernels, and thus +tiny RCU +was born. +Josh Triplett has since taken over the small-memory banner with his +Linux kernel tinification +project, which resulted in +SRCU +becoming optional for those kernels not needing it. + +

+The remaining performance requirements are, for the most part, +unsurprising. +For example, in keeping with RCU's read-side specialization, +rcu_dereference() should have negligible overhead (for +example, suppression of a few minor compiler optimizations). +Similarly, in non-preemptible environments, rcu_read_lock() and +rcu_read_unlock() should have exactly zero overhead. + +

+In preemptible environments, in the case where the RCU read-side +critical section was not preempted (as will be the case for the +highest-priority real-time process), rcu_read_lock() and +rcu_read_unlock() should have minimal overhead. +In particular, they should not contain atomic read-modify-write +operations, memory-barrier instructions, preemption disabling, +interrupt disabling, or backwards branches. +However, in the case where the RCU read-side critical section was preempted, +rcu_read_unlock() may acquire spinlocks and disable interrupts. +This is why it is better to nest an RCU read-side critical section +within a preempt-disable region than vice versa, at least in cases +where that critical section is short enough to avoid unduly degrading +real-time latencies. + +

+The synchronize_rcu() grace-period-wait primitive is +optimized for throughput. +It may therefore incur several milliseconds of latency in addition to +the duration of the longest RCU read-side critical section. +On the other hand, multiple concurrent invocations of +synchronize_rcu() are required to use batching optimizations +so that they can be satisfied by a single underlying grace-period-wait +operation. +For example, in the Linux kernel, it is not unusual for a single +grace-period-wait operation to serve more than +1,000 separate invocations +of synchronize_rcu(), thus amortizing the per-invocation +overhead down to nearly zero. +However, the grace-period optimization is also required to avoid +measurable degradation of real-time scheduling and interrupt latencies. + +

+In some cases, the multi-millisecond synchronize_rcu() +latencies are unacceptable. +In these cases, synchronize_rcu_expedited() may be used +instead, reducing the grace-period latency down to a few tens of +microseconds on small systems, at least in cases where the RCU read-side +critical sections are short. +There are currently no special latency requirements for +synchronize_rcu_expedited() on large systems, but, +consistent with the empirical nature of the RCU specification, +that is subject to change. +However, there most definitely are scalability requirements: +A storm of synchronize_rcu_expedited() invocations on 4096 +CPUs should at least make reasonable forward progress. +In return for its shorter latencies, synchronize_rcu_expedited() +is permitted to impose modest degradation of real-time latency +on non-idle online CPUs. +That said, it will likely be necessary to take further steps to reduce this +degradation, hopefully to roughly that of a scheduling-clock interrupt. + +

+There are a number of situations where even +synchronize_rcu_expedited()'s reduced grace-period +latency is unacceptable. +In these situations, the asynchronous call_rcu() can be +used in place of synchronize_rcu() as follows: + +

+
+ 1 struct foo {
+ 2   int a;
+ 3   int b;
+ 4   struct rcu_head rh;
+ 5 };
+ 6
+ 7 static void remove_gp_cb(struct rcu_head *rhp)
+ 8 {
+ 9   struct foo *p = container_of(rhp, struct foo, rh);
+10
+11   kfree(p);
+12 }
+13
+14 bool remove_gp_asynchronous(void)
+15 {
+16   struct foo *p;
+17
+18   spin_lock(&gp_lock);
+19   p = rcu_dereference(gp);
+20   if (!p) {
+21     spin_unlock(&gp_lock);
+22     return false;
+23   }
+24   rcu_assign_pointer(gp, NULL);
+25   call_rcu(&p->rh, remove_gp_cb);
+26   spin_unlock(&gp_lock);
+27   return true;
+28 }
+
+
+ +

+A definition of struct foo is finally needed, and appears +on lines 1-5. +The function remove_gp_cb() is passed to call_rcu() +on line 25, and will be invoked after the end of a subsequent +grace period. +This gets the same effect as remove_gp_synchronous(), +but without forcing the updater to wait for a grace period to elapse. +The call_rcu() function may be used in a number of +situations where neither synchronize_rcu() nor +synchronize_rcu_expedited() would be legal, +including within preempt-disable code, local_bh_disable() code, +interrupt-disable code, and interrupt handlers. +However, even call_rcu() is illegal within NMI handlers. +The callback function (remove_gp_cb() in this case) will be +executed within softirq (software interrupt) environment within the +Linux kernel, +either within a real softirq handler or under the protection +of local_bh_disable(). +In both the Linux kernel and in userspace, it is bad practice to +write an RCU callback function that takes too long. +Long-running operations should be relegated to separate threads or +(in the Linux kernel) workqueues. + +

@@QQ@@ +Why does line 19 use rcu_access_pointer()? +After all, call_rcu() on line 25 stores into the +structure, which would interact badly with concurrent insertions. +Doesn't this mean that rcu_dereference() is required? +

@@QQA@@ +Presumably the ->gp_lock acquired on line 18 excludes +any changes, including any insertions that rcu_dereference() +would protect against. +Therefore, any insertions will be delayed until after ->gp_lock +is released on line 25, which in turn means that +rcu_access_pointer() suffices. +

@@QQE@@ + +

+However, all that remove_gp_cb() is doing is +invoking kfree() on the data element. +This is a common idiom, and is supported by kfree_rcu(), +which allows “fire and forget” operation as shown below: + +

+
+ 1 struct foo {
+ 2   int a;
+ 3   int b;
+ 4   struct rcu_head rh;
+ 5 };
+ 6
+ 7 bool remove_gp_faf(void)
+ 8 {
+ 9   struct foo *p;
+10
+11   spin_lock(&gp_lock);
+12   p = rcu_dereference(gp);
+13   if (!p) {
+14     spin_unlock(&gp_lock);
+15     return false;
+16   }
+17   rcu_assign_pointer(gp, NULL);
+18   kfree_rcu(p, rh);
+19   spin_unlock(&gp_lock);
+20   return true;
+21 }
+
+
+ +

+Note that remove_gp_faf() simply invokes +kfree_rcu() and proceeds, without any need to pay any +further attention to the subsequent grace period and kfree(). +It is permissible to invoke kfree_rcu() from the same +environments as for call_rcu(). +Interestingly enough, DYNIX/ptx had the equivalents of +call_rcu() and kfree_rcu(), but not +synchronize_rcu(). +This was due to the fact that RCU was not heavily used within DYNIX/ptx, +so the very few places that needed something like +synchronize_rcu() simply open-coded it. + +

@@QQ@@ +Earlier it was claimed that call_rcu() and +kfree_rcu() allowed updaters to avoid being blocked +by readers. +But how can that be correct, given that the invocation of the callback +and the freeing of the memory (respectively) must still wait for +a grace period to elapse? +

@@QQA@@ +We could define things this way, but keep in mind that this sort of +definition would say that updates in garbage-collected languages +cannot complete until the next time the garbage collector runs, +which does not seem at all reasonable. +The key point is that in most cases, an updater using either +call_rcu() or kfree_rcu() can proceed to the +next update as soon as it has invoked call_rcu() or +kfree_rcu(), without having to wait for a subsequent +grace period. +

@@QQE@@ + +

+But what if the updater must wait for the completion of code to be +executed after the end of the grace period, but has other tasks +that can be carried out in the meantime? +The polling-style get_state_synchronize_rcu() and +cond_synchronize_rcu() functions may be used for this +purpose, as shown below: + +

+
+ 1 bool remove_gp_poll(void)
+ 2 {
+ 3   struct foo *p;
+ 4   unsigned long s;
+ 5
+ 6   spin_lock(&gp_lock);
+ 7   p = rcu_access_pointer(gp);
+ 8   if (!p) {
+ 9     spin_unlock(&gp_lock);
+10     return false;
+11   }
+12   rcu_assign_pointer(gp, NULL);
+13   spin_unlock(&gp_lock);
+14   s = get_state_synchronize_rcu();
+15   do_something_while_waiting();
+16   cond_synchronize_rcu(s);
+17   kfree(p);
+18   return true;
+19 }
+
+
+ +

+On line 14, get_state_synchronize_rcu() obtains a +“cookie” from RCU, +then line 15 carries out other tasks, +and finally, line 16 returns immediately if a grace period has +elapsed in the meantime, but otherwise waits as required. +The need for get_state_synchronize_rcu and +cond_synchronize_rcu() has appeared quite recently, +so it is too early to tell whether they will stand the test of time. + +

+RCU thus provides a range of tools to allow updaters to strike the +required tradeoff between latency, flexibility and CPU overhead. + +

Composability

+ +

+Composability has received much attention in recent years, perhaps in part +due to the collision of multicore hardware with object-oriented techniques +designed in single-threaded environments for single-threaded use. +And in theory, RCU read-side critical sections may be composed, and in +fact may be nested arbitrarily deeply. +In practice, as with all real-world implementations of composable +constructs, there are limitations. + +

+Implementations of RCU for which rcu_read_lock() +and rcu_read_unlock() generate no code, such as +Linux-kernel RCU when CONFIG_PREEMPT=n, can be +nested arbitrarily deeply. +After all, there is no overhead. +Except that if all these instances of rcu_read_lock() +and rcu_read_unlock() are visible to the compiler, +compilation will eventually fail due to exhausting memory, +mass storage, or user patience, whichever comes first. +If the nesting is not visible to the compiler, as is the case with +mutually recursive functions each in its own translation unit, +stack overflow will result. +If the nesting takes the form of loops, either the control variable +will overflow or (in the Linux kernel) you will get an RCU CPU stall warning. +Nevertheless, this class of RCU implementations is one +of the most composable constructs in existence. + +

+RCU implementations that explicitly track nesting depth +are limited by the nesting-depth counter. +For example, the Linux kernel's preemptible RCU limits nesting to +INT_MAX. +This should suffice for almost all practical purposes. +That said, a consecutive pair of RCU read-side critical sections +between which there is an operation that waits for a grace period +cannot be enclosed in another RCU read-side critical section. +This is because it is not legal to wait for a grace period within +an RCU read-side critical section: To do so would result either +in deadlock or +in RCU implicitly splitting the enclosing RCU read-side critical +section, neither of which is conducive to a long-lived and prosperous +kernel. + +

+In short, although RCU read-side critical sections are highly composable, +care is required in some situations, just as is the case for any other +composable synchronization mechanism. + +

Corner Cases

+ +

+A given RCU workload might have an endless and intense stream of +RCU read-side critical sections, perhaps even so intense that there +was never a point in time during which there was not at least one +RCU read-side critical section in flight. +RCU cannot allow this situation to block grace periods: As long as +all the RCU read-side critical sections are finite, grace periods +must also be finite. + +

+That said, preemptible RCU implementations could potentially result +in RCU read-side critical sections being preempted for long durations, +which has the effect of creating a long-duration RCU read-side +critical section. +This situation can arise only in heavily loaded systems, but systems using +real-time priorities are of course more vulnerable. +Therefore, RCU priority boosting is provided to help deal with this +case. +That said, the exact requirements on RCU priority boosting will likely +evolve as more experience accumulates. + +

+Other workloads might have very high update rates. +Although one can argue that such workloads should instead use +something other than RCU, the fact remains that RCU must +handle such workloads gracefully. +This requirement is another factor driving batching of grace periods, +but it is also the driving force behind the checks for large numbers +of queued RCU callbacks in the call_rcu() code path. +Finally, high update rates should not delay RCU read-side critical +sections, although some read-side delays can occur when using +synchronize_rcu_expedited(), courtesy of this function's use +of try_stop_cpus(). +(In the future, synchronize_rcu_expedited() will be +converted to use lighter-weight inter-processor interrupts (IPIs), +but this will still disturb readers, though to a much smaller degree.) + +

+Although all three of these corner cases were understood in the early +1990s, a simple user-level test consisting of close(open(path)) +in a tight loop +in the early 2000s suddenly provided a much deeper appreciation of the +high-update-rate corner case. +This test also motivated addition of some RCU code to react to high update +rates, for example, if a given CPU finds itself with more than 10,000 +RCU callbacks queued, it will cause RCU to take evasive action by +more aggressively starting grace periods and more aggressively forcing +completion of grace-period processing. +This evasive action causes the grace period to complete more quickly, +but at the cost of restricting RCU's batching optimizations, thus +increasing the CPU overhead incurred by that grace period. + +

+Software-Engineering Requirements

+ +

+Between Murphy's Law and “To err is human”, it is necessary to +guard against mishaps and misuse: + +

    +
  1. It is all too easy to forget to use rcu_read_lock() + everywhere that it is needed, so kernels built with + CONFIG_PROVE_RCU=y will spat if + rcu_dereference() is used outside of an + RCU read-side critical section. + Update-side code can use rcu_dereference_protected(), + which takes a + lockdep expression + to indicate what is providing the protection. + If the indicated protection is not provided, a lockdep splat + is emitted. + +

    + Code shared between readers and updaters can use + rcu_dereference_check(), which also takes a + lockdep expression, and emits a lockdep splat if neither + rcu_read_lock() nor the indicated protection + is in place. + In addition, rcu_dereference_raw() is used in those + (hopefully rare) cases where the required protection cannot + be easily described. + Finally, rcu_read_lock_held() is provided to + allow a function to verify that it has been invoked within + an RCU read-side critical section. + I was made aware of this set of requirements shortly after Thomas + Gleixner audited a number of RCU uses. +

  2. A given function might wish to check for RCU-related preconditions + upon entry, before using any other RCU API. + The rcu_lockdep_assert() does this job, + asserting the expression in kernels having lockdep enabled + and doing nothing otherwise. +
  3. It is also easy to forget to use rcu_assign_pointer() + and rcu_dereference(), perhaps (incorrectly) + substituting a simple assignment. + To catch this sort of error, a given RCU-protected pointer may be + tagged with __rcu, after which running sparse + with CONFIG_SPARSE_RCU_POINTER=y will complain + about simple-assignment accesses to that pointer. + Arnd Bergmann made me aware of this requirement, and also + supplied the needed + patch series. +
  4. Kernels built with CONFIG_DEBUG_OBJECTS_RCU_HEAD=y + will splat if a data element is passed to call_rcu() + twice in a row, without a grace period in between. + (This error is similar to a double free.) + The corresponding rcu_head structures that are + dynamically allocated are automatically tracked, but + rcu_head structures allocated on the stack + must be initialized with init_rcu_head_on_stack() + and cleaned up with destroy_rcu_head_on_stack(). + Similarly, statically allocated non-stack rcu_head + structures must be initialized with init_rcu_head() + and cleaned up with destroy_rcu_head(). + Mathieu Desnoyers made me aware of this requirement, and also + supplied the needed + patch. +
  5. An infinite loop in an RCU read-side critical section will + eventually trigger an RCU CPU stall warning splat. + However, RCU is not obligated to produce this splat + unless there is a grace period waiting on that particular + RCU read-side critical section. + This requirement made itself known in the early 1990s, pretty + much the first time that it was necessary to debug a CPU stall. +
  6. Although it would be very good to detect pointers leaking out + of RCU read-side critical sections, there is currently no + good way of doing this. + One complication is the need to distinguish between pointers + leaking and pointers that have been handed off from RCU to + some other synchronization mechanism, for example, reference + counting. +
  7. In kernels built with CONFIG_RCU_TRACE=y, RCU-related + information is provided via both debugfs and event tracing. +
  8. Open-coded use of rcu_assign_pointer() and + rcu_dereference() to create typical linked + data structures can be surprisingly error-prone. + Therefore, RCU-protected + linked lists + and, more recently, RCU-protected + hash tables + are available. + Many other special-purpose RCU-protected data structures are + available in the Linux kernel and the userspace RCU library. +
  9. Some linked structures are created at compile time, but still + require __rcu checking. + The RCU_POINTER_INITIALIZER() macro serves this + purpose. +
  10. It is not necessary to use rcu_assign_pointer() + when creating linked structures that are to be published via + a single external pointer. + The RCU_INIT_POINTER() macro is provided for + this task and also for assigning NULL pointers + at runtime. +
+ +

+This not a hard-and-fast list: RCU's diagnostic capabilities will +continue to be guided by the number and type of usage bugs found +in real-world RCU usage. + +

Linux Kernel Complications

+ +

+The Linux kernel provides an interesting environment for all kinds of +software, including RCU. +Some of the relevant points of interest are as follows: + +

    +
  1. Configuration. +
  2. Firmware Interface. +
  3. Early Boot. +
  4. + Interrupts and non-maskable interrupts (NMIs). +
  5. Loadable Modules. +
  6. Hotplug CPU. +
  7. Scheduler and RCU. +
  8. Tracing and RCU. +
  9. Energy Efficiency. +
  10. + Performance, Scalability, Response Time, and Reliability. +
+ +

+This list is probably incomplete, but it does give a feel for the +most notable Linux-kernel complications. +Each of the following sections covers one of the above topics. + +

Configuration

+ +

+RCU's goal is automatic configuration, so that almost nobody +needs to worry about RCU's Kconfig options. +And for almost all users, RCU does in fact work well +“out of the box.” + +

+However, there are specialized use cases that are handled by +kernel boot parameters and Kconfig options. +Unfortunately, the Kconfig system will explicitly ask users +about new Kconfig options, which requires almost all of them +be hidden behind a CONFIG_RCU_EXPERT Kconfig option. + +

+This all should be quite obvious, but the fact remains that +Linus Torvalds recently had to +remind +me of this requirement. + +

Firmware Interface

+ +

+In many cases, kernel obtains information about the system from the +firmware, and sometimes things are lost in translation. +Or the translation is accurate, but the original message is bogus. + +

+For example, some systems' firmware overreports the number of CPUs, +sometimes by a large factor. +If RCU naively believed the firmware, as it used to do, +it would create too many per-CPU kthreads. +Although the resulting system will still run correctly, the extra +kthreads needlessly consume memory and can cause confusion +when they show up in ps listings. + +

+RCU must therefore wait for a given CPU to actually come online before +it can allow itself to believe that the CPU actually exists. +The resulting “ghost CPUs” (which are never going to +come online) cause a number of +interesting complications. + +

Early Boot

+ +

+The Linux kernel's boot sequence is an interesting process, +and RCU is used early, even before rcu_init() +is invoked. +In fact, a number of RCU's primitives can be used as soon as the +initial task's task_struct is available and the +boot CPU's per-CPU variables are set up. +The read-side primitives (rcu_read_lock(), +rcu_read_unlock(), rcu_dereference(), +and rcu_access_pointer()) will operate normally very early on, +as will rcu_assign_pointer(). + +

+Although call_rcu() may be invoked at any +time during boot, callbacks are not guaranteed to be invoked until after +the scheduler is fully up and running. +This delay in callback invocation is due to the fact that RCU does not +invoke callbacks until it is fully initialized, and this full initialization +cannot occur until after the scheduler has initialized itself to the +point where RCU can spawn and run its kthreads. +In theory, it would be possible to invoke callbacks earlier, +however, this is not a panacea because there would be severe restrictions +on what operations those callbacks could invoke. + +

+Perhaps surprisingly, synchronize_rcu(), +synchronize_rcu_bh() +(discussed below), +and +synchronize_sched() +will all operate normally +during very early boot, the reason being that there is only one CPU +and preemption is disabled. +This means that the call synchronize_rcu() (or friends) +itself is a quiescent +state and thus a grace period, so the early-boot implementation can +be a no-op. + +

+Both synchronize_rcu_bh() and synchronize_sched() +continue to operate normally through the remainder of boot, courtesy +of the fact that preemption is disabled across their RCU read-side +critical sections and also courtesy of the fact that there is still +only one CPU. +However, once the scheduler starts initializing, preemption is enabled. +There is still only a single CPU, but the fact that preemption is enabled +means that the no-op implementation of synchronize_rcu() no +longer works in CONFIG_PREEMPT=y kernels. +Therefore, as soon as the scheduler starts initializing, the early-boot +fastpath is disabled. +This means that synchronize_rcu() switches to its runtime +mode of operation where it posts callbacks, which in turn means that +any call to synchronize_rcu() will block until the corresponding +callback is invoked. +Unfortunately, the callback cannot be invoked until RCU's runtime +grace-period machinery is up and running, which cannot happen until +the scheduler has initialized itself sufficiently to allow RCU's +kthreads to be spawned. +Therefore, invoking synchronize_rcu() during scheduler +initialization can result in deadlock. + +

@@QQ@@ +So what happens with synchronize_rcu() during +scheduler initialization for CONFIG_PREEMPT=n +kernels? +

@@QQA@@ +In CONFIG_PREEMPT=n kernel, synchronize_rcu() +maps directly to synchronize_sched(). +Therefore, synchronize_rcu() works normally throughout +boot in CONFIG_PREEMPT=n kernels. +However, your code must also work in CONFIG_PREEMPT=y kernels, +so it is still necessary to avoid invoking synchronize_rcu() +during scheduler initialization. +

@@QQE@@ + +

+I learned of these boot-time requirements as a result of a series of +system hangs. + +

Interrupts and NMIs

+ +

+The Linux kernel has interrupts, and RCU read-side critical sections are +legal within interrupt handlers and within interrupt-disabled regions +of code, as are invocations of call_rcu(). + +

+Some Linux-kernel architectures can enter an interrupt handler from +non-idle process context, and then just never leave it, instead stealthily +transitioning back to process context. +This trick is sometimes used to invoke system calls from inside the kernel. +These “half-interrupts” mean that RCU has to be very careful +about how it counts interrupt nesting levels. +I learned of this requirement the hard way during a rewrite +of RCU's dyntick-idle code. + +

+The Linux kernel has non-maskable interrupts (NMIs), and +RCU read-side critical sections are legal within NMI handlers. +Thankfully, RCU update-side primitives, including +call_rcu(), are prohibited within NMI handlers. + +

+The name notwithstanding, some Linux-kernel architectures +can have nested NMIs, which RCU must handle correctly. +Andy Lutomirski +surprised me +with this requirement; +he also kindly surprised me with +an algorithm +that meets this requirement. + +

Loadable Modules

+ +

+The Linux kernel has loadable modules, and these modules can +also be unloaded. +After a given module has been unloaded, any attempt to call +one of its functions results in a segmentation fault. +The module-unload functions must therefore cancel any +delayed calls to loadable-module functions, for example, +any outstanding mod_timer() must be dealt with +via del_timer_sync() or similar. + +

+Unfortunately, there is no way to cancel an RCU callback; +once you invoke call_rcu(), the callback function is +going to eventually be invoked, unless the system goes down first. +Because it is normally considered socially irresponsible to crash the system +in response to a module unload request, we need some other way +to deal with in-flight RCU callbacks. + +

+RCU therefore provides +rcu_barrier(), +which waits until all in-flight RCU callbacks have been invoked. +If a module uses call_rcu(), its exit function should therefore +prevent any future invocation of call_rcu(), then invoke +rcu_barrier(). +In theory, the underlying module-unload code could invoke +rcu_barrier() unconditionally, but in practice this would +incur unacceptable latencies. + +

+Nikita Danilov noted this requirement for an analogous filesystem-unmount +situation, and Dipankar Sarma incorporated rcu_barrier() into RCU. +The need for rcu_barrier() for module unloading became +apparent later. + +

Hotplug CPU

+ +

+The Linux kernel supports CPU hotplug, which means that CPUs +can come and go. +It is of course illegal to use any RCU API member from an offline CPU. +This requirement was present from day one in DYNIX/ptx, but +on the other hand, the Linux kernel's CPU-hotplug implementation +is “interesting.” + +

+The Linux-kernel CPU-hotplug implementation has notifiers that +are used to allow the various kernel subsystems (including RCU) +to respond appropriately to a given CPU-hotplug operation. +Most RCU operations may be invoked from CPU-hotplug notifiers, +including even normal synchronous grace-period operations +such as synchronize_rcu(). +However, expedited grace-period operations such as +synchronize_rcu_expedited() are not supported, +due to the fact that current implementations block CPU-hotplug +operations, which could result in deadlock. + +

+In addition, all-callback-wait operations such as +rcu_barrier() are also not supported, due to the +fact that there are phases of CPU-hotplug operations where +the outgoing CPU's callbacks will not be invoked until after +the CPU-hotplug operation ends, which could also result in deadlock. + +

Scheduler and RCU

+ +

+RCU depends on the scheduler, and the scheduler uses RCU to +protect some of its data structures. +This means the scheduler is forbidden from acquiring +the runqueue locks and the priority-inheritance locks +in the middle of an outermost RCU read-side critical section unless +it also releases them before exiting that same +RCU read-side critical section. +This same prohibition also applies to any lock that is acquired +while holding any lock to which this prohibition applies. +Violating this rule results in deadlock. + +

+For RCU's part, the preemptible-RCU rcu_read_unlock() +implementation must be written carefully to avoid similar deadlocks. +In particular, rcu_read_unlock() must tolerate an +interrupt where the interrupt handler invokes both +rcu_read_lock() and rcu_read_unlock(). +This possibility requires rcu_read_unlock() to use +negative nesting levels to avoid destructive recursion via +interrupt handler's use of RCU. + +

+This pair of mutual scheduler-RCU requirements came as a +complete surprise. + +

+As noted above, RCU makes use of kthreads, and it is necessary to +avoid excessive CPU-time accumulation by these kthreads. +This requirement was no surprise, but RCU's violation of it +when running context-switch-heavy workloads when built with +CONFIG_NO_HZ_FULL=y +did come as a surprise [PDF]. +RCU has made good progress towards meeting this requirement, even +for context-switch-have CONFIG_NO_HZ_FULL=y workloads, +but there is room for further improvement. + +

Tracing and RCU

+ +

+It is possible to use tracing on RCU code, but tracing itself +uses RCU. +For this reason, rcu_dereference_raw_notrace() +is provided for use by tracing, which avoids the destructive +recursion that could otherwise ensue. +This API is also used by virtualization in some architectures, +where RCU readers execute in environments in which tracing +cannot be used. +The tracing folks both located the requirement and provided the +needed fix, so this surprise requirement was relatively painless. + +

Energy Efficiency

+ +

+Interrupting idle CPUs is considered socially unacceptable, +especially by people with battery-powered embedded systems. +RCU therefore conserves energy by detecting which CPUs are +idle, including tracking CPUs that have been interrupted from idle. +This is a large part of the energy-efficiency requirement, +so I learned of this via an irate phone call. + +

+Because RCU avoids interrupting idle CPUs, it is illegal to +execute an RCU read-side critical section on an idle CPU. +(Kernels built with CONFIG_PROVE_RCU=y will splat +if you try it.) +The RCU_NONIDLE() macro and _rcuidle +event tracing is provided to work around this restriction. +In addition, rcu_is_watching() may be used to +test whether or not it is currently legal to run RCU read-side +critical sections on this CPU. +I learned of the need for diagnostics on the one hand +and RCU_NONIDLE() on the other while inspecting +idle-loop code. +Steven Rostedt supplied _rcuidle event tracing, +which is used quite heavily in the idle loop. + +

+It is similarly socially unacceptable to interrupt an +nohz_full CPU running in userspace. +RCU must therefore track nohz_full userspace +execution. +And in +CONFIG_NO_HZ_FULL_SYSIDLE=y +kernels, RCU must separately track idle CPUs on the one hand and +CPUs that are either idle or executing in userspace on the other. +In both cases, RCU must be able to sample state at two points in +time, and be able to determine whether or not some other CPU spent +any time idle and/or executing in userspace. + +

+These energy-efficiency requirements have proven quite difficult to +understand and to meet, for example, there have been more than five +clean-sheet rewrites of RCU's energy-efficiency code, the last of +which was finally able to demonstrate +real energy savings running on real hardware [PDF]. +As noted earlier, +I learned of many of these requirements via angry phone calls: +Flaming me on the Linux-kernel mailing list was apparently not +sufficient to fully vent their ire at RCU's energy-efficiency bugs! + +

+Performance, Scalability, Response Time, and Reliability

+ +

+Expanding on the +earlier discussion, +RCU is used heavily by hot code paths in performance-critical +portions of the Linux kernel's networking, security, virtualization, +and scheduling code paths. +RCU must therefore use efficient implementations, especially in its +read-side primitives. +To that end, it would be good if preemptible RCU's implementation +of rcu_read_lock() could be inlined, however, doing +this requires resolving #include issues with the +task_struct structure. + +

+The Linux kernel supports hardware configurations with up to +4096 CPUs, which means that RCU must be extremely scalable. +Algorithms that involve frequent acquisitions of global locks or +frequent atomic operations on global variables simply cannot be +tolerated within the RCU implementation. +RCU therefore makes heavy use of a combining tree based on the +rcu_node structure. +RCU is required to tolerate all CPUs continuously invoking any +combination of RCU's runtime primitives with minimal per-operation +overhead. +In fact, in many cases, increasing load must decrease the +per-operation overhead, witness the batching optimizations for +synchronize_rcu(), call_rcu(), +synchronize_rcu_expedited(), and rcu_barrier(). +As a general rule, RCU must cheerfully accept whatever the +rest of the Linux kernel decides to throw at it. + +

+The Linux kernel is used for real-time workloads, especially +in conjunction with the +-rt patchset. +The real-time-latency response requirements are such that the +traditional approach of disabling preemption across RCU +read-side critical sections is inappropriate. +Kernels built with CONFIG_PREEMPT=y therefore +use an RCU implementation that allows RCU read-side critical +sections to be preempted. +This requirement made its presence known after users made it +clear that an earlier +real-time patch +did not meet their needs, in conjunction with some +RCU issues +encountered by a very early version of the -rt patchset. + +

+In addition, RCU must make do with a sub-100-microsecond real-time latency +budget. +In fact, on smaller systems with the -rt patchset, the Linux kernel +provides sub-20-microsecond real-time latencies for the whole kernel, +including RCU. +RCU's scalability and latency must therefore be sufficient for +these sorts of configurations. +To my surprise, the sub-100-microsecond real-time latency budget + +applies to even the largest systems [PDF], +up to and including systems with 4096 CPUs. +This real-time requirement motivated the grace-period kthread, which +also simplified handling of a number of race conditions. + +

+Finally, RCU's status as a synchronization primitive means that +any RCU failure can result in arbitrary memory corruption that can be +extremely difficult to debug. +This means that RCU must be extremely reliable, which in +practice also means that RCU must have an aggressive stress-test +suite. +This stress-test suite is called rcutorture. + +

+Although the need for rcutorture was no surprise, +the current immense popularity of the Linux kernel is posing +interesting—and perhaps unprecedented—validation +challenges. +To see this, keep in mind that there are well over one billion +instances of the Linux kernel running today, given Android +smartphones, Linux-powered televisions, and servers. +This number can be expected to increase sharply with the advent of +the celebrated Internet of Things. + +

+Suppose that RCU contains a race condition that manifests on average +once per million years of runtime. +This bug will be occurring about three times per day across +the installed base. +RCU could simply hide behind hardware error rates, given that no one +should really expect their smartphone to last for a million years. +However, anyone taking too much comfort from this thought should +consider the fact that in most jurisdictions, a successful multi-year +test of a given mechanism, which might include a Linux kernel, +suffices for a number of types of safety-critical certifications. +In fact, rumor has it that the Linux kernel is already being used +in production for safety-critical applications. +I don't know about you, but I would feel quite bad if a bug in RCU +killed someone. +Which might explain my recent focus on validation and verification. + +

Other RCU Flavors

+ +

+One of the more surprising things about RCU is that there are now +no fewer than five flavors, or API families. +In addition, the primary flavor that has been the sole focus up to +this point has two different implementations, non-preemptible and +preemptible. +The other four flavors are listed below, with requirements for each +described in a separate section. + +

    +
  1. Bottom-Half Flavor +
  2. Sched Flavor +
  3. Sleepable RCU +
  4. Tasks RCU +
+ +

Bottom-Half Flavor

+ +

+The softirq-disable (AKA “bottom-half”, +hence the “_bh” abbreviations) +flavor of RCU, or RCU-bh, was developed by +Dipankar Sarma to provide a flavor of RCU that could withstand the +network-based denial-of-service attacks researched by Robert +Olsson. +These attacks placed so much networking load on the system +that some of the CPUs never exited softirq execution, +which in turn prevented those CPUs from ever executing a context switch, +which, in the RCU implementation of that time, prevented grace periods +from ever ending. +The result was an out-of-memory condition and a system hang. + +

+The solution was the creation of RCU-bh, which does +local_bh_disable() +across its read-side critical sections, and which uses the transition +from one type of softirq processing to another as a quiescent state +in addition to context switch, idle, user mode, and offline. +This means that RCU-bh grace periods can complete even when some of +the CPUs execute in softirq indefinitely, thus allowing algorithms +based on RCU-bh to withstand network-based denial-of-service attacks. + +

+Because +rcu_read_lock_bh() and rcu_read_unlock_bh() +disable and re-enable softirq handlers, any attempt to start a softirq +handlers during the +RCU-bh read-side critical section will be deferred. +In this case, rcu_read_unlock_bh() +will invoke softirq processing, which can take considerable time. +One can of course argue that this softirq overhead should be associated +with the code following the RCU-bh read-side critical section rather +than rcu_read_unlock_bh(), but the fact +is that most profiling tools cannot be expected to make this sort +of fine distinction. +For example, suppose that a three-millisecond-long RCU-bh read-side +critical section executes during a time of heavy networking load. +There will very likely be an attempt to invoke at least one softirq +handler during that three milliseconds, but any such invocation will +be delayed until the time of the rcu_read_unlock_bh(). +This can of course make it appear at first glance as if +rcu_read_unlock_bh() was executing very slowly. + +

+The +RCU-bh API +includes +rcu_read_lock_bh(), +rcu_read_unlock_bh(), +rcu_dereference_bh(), +rcu_dereference_bh_check(), +synchronize_rcu_bh(), +synchronize_rcu_bh_expedited(), +call_rcu_bh(), +rcu_barrier_bh(), and +rcu_read_lock_bh_held(). + +

Sched Flavor

+ +

+Before preemptible RCU, waiting for an RCU grace period had the +side effect of also waiting for all pre-existing interrupt +and NMI handlers. +However, there are legitimate preemptible-RCU implementations that +do not have this property, given that any point in the code outside +of an RCU read-side critical section can be a quiescent state. +Therefore, RCU-sched was created, which follows “classic” +RCU in that an RCU-sched grace period waits for for pre-existing +interrupt and NMI handlers. +In kernels built with CONFIG_PREEMPT=n, the RCU and RCU-sched +APIs have identical implementations, while kernels built with +CONFIG_PREEMPT=y provide a separate implementation for each. + +

+Note well that in CONFIG_PREEMPT=y kernels, +rcu_read_lock_sched() and rcu_read_unlock_sched() +disable and re-enable preemption, respectively. +This means that if there was a preemption attempt during the +RCU-sched read-side critical section, rcu_read_unlock_sched() +will enter the scheduler, with all the latency and overhead entailed. +Just as with rcu_read_unlock_bh(), this can make it look +as if rcu_read_unlock_sched() was executing very slowly. +However, the highest-priority task won't be preempted, so that task +will enjoy low-overhead rcu_read_unlock_sched() invocations. + +

+The +RCU-sched API +includes +rcu_read_lock_sched(), +rcu_read_unlock_sched(), +rcu_read_lock_sched_notrace(), +rcu_read_unlock_sched_notrace(), +rcu_dereference_sched(), +rcu_dereference_sched_check(), +synchronize_sched(), +synchronize_rcu_sched_expedited(), +call_rcu_sched(), +rcu_barrier_sched(), and +rcu_read_lock_sched_held(). +However, anything that disables preemption also marks an RCU-sched +read-side critical section, including +preempt_disable() and preempt_enable(), +local_irq_save() and local_irq_restore(), +and so on. + +

Sleepable RCU

+ +

+For well over a decade, someone saying “I need to block within +an RCU read-side critical section” was a reliable indication +that this someone did not understand RCU. +After all, if you are always blocking in an RCU read-side critical +section, you can probably afford to use a higher-overhead synchronization +mechanism. +However, that changed with the advent of the Linux kernel's notifiers, +whose RCU read-side critical +sections almost never sleep, but sometimes need to. +This resulted in the introduction of +sleepable RCU, +or SRCU. + +

+SRCU allows different domains to be defined, with each such domain +defined by an instance of an srcu_struct structure. +A pointer to this structure must be passed in to each SRCU function, +for example, synchronize_srcu(&ss), where +ss is the srcu_struct structure. +The key benefit of these domains is that a slow SRCU reader in one +domain does not delay an SRCU grace period in some other domain. +That said, one consequence of these domains is that read-side code +must pass a “cookie” from srcu_read_lock() +to srcu_read_unlock(), for example, as follows: + +

+
+ 1 int idx;
+ 2
+ 3 idx = srcu_read_lock(&ss);
+ 4 do_something();
+ 5 srcu_read_unlock(&ss, idx);
+
+
+ +

+As noted above, it is legal to block within SRCU read-side critical sections, +however, with great power comes great responsibility. +If you block forever in one of a given domain's SRCU read-side critical +sections, then that domain's grace periods will also be blocked forever. +Of course, one good way to block forever is to deadlock, which can +happen if any operation in a given domain's SRCU read-side critical +section can block waiting, either directly or indirectly, for that domain's +grace period to elapse. +For example, this results in a self-deadlock: + +

+
+ 1 int idx;
+ 2
+ 3 idx = srcu_read_lock(&ss);
+ 4 do_something();
+ 5 synchronize_srcu(&ss);
+ 6 srcu_read_unlock(&ss, idx);
+
+
+ +

+However, if line 5 acquired a mutex that was held across +a synchronize_srcu() for domain ss, +deadlock would still be possible. +Furthermore, if line 5 acquired a mutex that was held across +a synchronize_srcu() for some other domain ss1, +and if an ss1-domain SRCU read-side critical section +acquired another mutex that was held across as ss-domain +synchronize_srcu(), +deadlock would again be possible. +Such a deadlock cycle could extend across an arbitrarily large number +of different SRCU domains. +Again, with great power comes great responsibility. + +

+Unlike the other RCU flavors, SRCU read-side critical sections can +run on idle and even offline CPUs. +This ability requires that srcu_read_lock() and +srcu_read_unlock() contain memory barriers, which means +that SRCU readers will run a bit slower than would RCU readers. +It also motivates the smp_mb__after_srcu_read_unlock() +API, which, in combination with srcu_read_unlock(), +guarantees a full memory barrier. + +

+The +SRCU API +includes +srcu_read_lock(), +srcu_read_unlock(), +srcu_dereference(), +srcu_dereference_check(), +synchronize_srcu(), +synchronize_srcu_expedited(), +call_srcu(), +srcu_barrier(), and +srcu_read_lock_held(). +It also includes +DEFINE_SRCU(), +DEFINE_STATIC_SRCU(), and +init_srcu_struct() +APIs for defining and initializing srcu_struct structures. + +

Tasks RCU

+ +

+Some forms of tracing use “tramopolines” to handle the +binary rewriting required to install different types of probes. +It would be good to be able to free old trampolines, which sounds +like a job for some form of RCU. +However, because it is necessary to be able to install a trace +anywhere in the code, it is not possible to use read-side markers +such as rcu_read_lock() and rcu_read_unlock(). +In addition, it does not work to have these markers in the trampoline +itself, because there would need to be instructions following +rcu_read_unlock(). +Although synchronize_rcu() would guarantee that execution +reached the rcu_read_unlock(), it would not be able to +guarantee that execution had completely left the trampoline. + +

+The solution, in the form of +Tasks RCU, +is to have implicit +read-side critical sections that are delimited by voluntary context +switches, that is, calls to schedule(), +cond_resched_rcu_qs(), and +synchronize_rcu_tasks(). +In addition, transitions to and from userspace execution also delimit +tasks-RCU read-side critical sections. + +

+The tasks-RCU API is quite compact, consisting only of +call_rcu_tasks(), +synchronize_rcu_tasks(), and +rcu_barrier_tasks(). + +

Possible Future Changes

+ +

+One of the tricks that RCU uses to attain update-side scalability is +to increase grace-period latency with increasing numbers of CPUs. +If this becomes a serious problem, it will be necessary to rework the +grace-period state machine so as to avoid the need for the additional +latency. + +

+Expedited grace periods scan the CPUs, so their latency and overhead +increases with increasing numbers of CPUs. +If this becomes a serious problem on large systems, it will be necessary +to do some redesign to avoid this scalability problem. + +

+RCU disables CPU hotplug in a few places, perhaps most notably in the +expedited grace-period and rcu_barrier() operations. +If there is a strong reason to use expedited grace periods in CPU-hotplug +notifiers, it will be necessary to avoid disabling CPU hotplug. +This would introduce some complexity, so there had better be a very +good reason. + +

+The tradeoff between grace-period latency on the one hand and interruptions +of other CPUs on the other hand may need to be re-examined. +The desire is of course for zero grace-period latency as well as zero +interprocessor interrupts undertaken during an expedited grace period +operation. +While this ideal is unlikely to be achievable, it is quite possible that +further improvements can be made. + +

+The multiprocessor implementations of RCU use a combining tree that +groups CPUs so as to reduce lock contention and increase cache locality. +However, this combining tree does not spread its memory across NUMA +nodes nor does it align the CPU groups with hardware features such +as sockets or cores. +Such spreading and alignment is currently believed to be unnecessary +because the hotpath read-side primitives do not access the combining +tree, nor does call_rcu() in the common case. +If you believe that your architecture needs such spreading and alignment, +then your architecture should also benefit from the +rcutree.rcu_fanout_leaf boot parameter, which can be set +to the number of CPUs in a socket, NUMA node, or whatever. +If the number of CPUs is too large, use a fraction of the number of +CPUs. +If the number of CPUs is a large prime number, well, that certainly +is an “interesting” architectural choice! +More flexible arrangements might be considered, but only if +rcutree.rcu_fanout_leaf has proven inadequate, and only +if the inadequacy has been demonstrated by a carefully run and +realistic system-level workload. + +

+Please note that arrangements that require RCU to remap CPU numbers will +require extremely good demonstration of need and full exploration of +alternatives. + +

+There is an embarrassingly large number of flavors of RCU, and this +number has been increasing over time. +Perhaps it will be possible to combine some at some future date. + +

+RCU's various kthreads are reasonably recent additions. +It is quite likely that adjustments will be required to more gracefully +handle extreme loads. +It might also be necessary to be able to relate CPU utilization by +RCU's kthreads and softirq handlers to the code that instigated this +CPU utilization. +For example, RCU callback overhead might be charged back to the +originating call_rcu() instance, though probably not +in production kernels. + +

Summary

+ +

+This document has presented more than two decade's worth of RCU +requirements. +Given that the requirements keep changing, this will not be the last +word on this subject, but at least it serves to get an important +subset of the requirements set forth. + +

Acknowledgments

+ +I am grateful to Steven Rostedt, Lai Jiangshan, Ingo Molnar, +Oleg Nesterov, Borislav Petkov, Peter Zijlstra, Boqun Feng, and +Andy Lutomirski for their help in rendering +this article human readable, and to Michelle Rankin for her support +of this effort. +Other contributions are acknowledged in the Linux kernel's git archive. +The cartoon is copyright (c) 2013 by Melissa Broussard, +and is provided +under the terms of the Creative Commons Attribution-Share Alike 3.0 +United States license. + +

@@QQAL@@ + + diff --git a/Documentation/RCU/Design/htmlqqz.sh b/Documentation/RCU/Design/htmlqqz.sh new file mode 100755 index 000000000000..d354f069559b --- /dev/null +++ b/Documentation/RCU/Design/htmlqqz.sh @@ -0,0 +1,108 @@ +#!/bin/sh +# +# Usage: sh htmlqqz.sh file +# +# Extracts and converts quick quizzes in a proto-HTML document file.htmlx. +# Commands, all of which must be on a line by themselves: +# +# "

@@QQ@@": Start of a quick quiz. +# "

@@QQA@@": Start of a quick-quiz answer. +# "

@@QQE@@": End of a quick-quiz answer, and thus of the quick quiz. +# "

@@QQAL@@": Place to put quick-quiz answer list. +# +# Places the result in file.html. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, you can access it online at +# http://www.gnu.org/licenses/gpl-2.0.html. +# +# Copyright (c) 2013 Paul E. McKenney, IBM Corporation. + +fn=$1 +if test ! -r $fn.htmlx +then + echo "Error: $fn.htmlx unreadable." + exit 1 +fi + +echo "" > $fn.html +echo "" >> $fn.html +awk < $fn.htmlx >> $fn.html ' + +state == "" && $1 != "

@@QQ@@" && $1 != "

@@QQAL@@" { + print $0; + if ($0 ~ /^

@@QQ/) + print "Bad Quick Quiz command: " NR " (expected

@@QQ@@ or

@@QQAL@@)." > "/dev/stderr" + next; +} + +state == "" && $1 == "

@@QQ@@" { + qqn++; + qqlineno = NR; + haveqq = 1; + state = "qq"; + print "

Quick Quiz " qqn ":" + next; +} + +state == "qq" && $1 != "

@@QQA@@" { + qq[qqn] = qq[qqn] $0 "\n"; + print $0 + if ($0 ~ /^

@@QQ/) + print "Bad Quick Quiz command: " NR ". (expected

@@QQA@@)" > "/dev/stderr" + next; +} + +state == "qq" && $1 == "

@@QQA@@" { + state = "qqa"; + print "
Answer" + next; +} + +state == "qqa" && $1 != "

@@QQE@@" { + qqa[qqn] = qqa[qqn] $0 "\n"; + if ($0 ~ /^

@@QQ/) + print "Bad Quick Quiz command: " NR " (expected

@@QQE@@)." > "/dev/stderr" + next; +} + +state == "qqa" && $1 == "

@@QQE@@" { + state = ""; + next; +} + +state == "" && $1 == "

@@QQAL@@" { + haveqq = ""; + print "

" + print "Answers to Quick Quizzes

" + print ""; + for (i = 1; i <= qqn; i++) { + print "" + print "

Quick Quiz " i ":" + print qq[i]; + print ""; + print "

Answer:" + print qqa[i]; + print ""; + print "

Back to Quick Quiz " i "." + print ""; + } + next; +} + +END { + if (state != "") + print "Unterminated Quick Quiz: " qqlineno "." > "/dev/stderr" + else if (haveqq) + print "Missing \"

@@QQAL@@\", no Quick Quiz." > "/dev/stderr" +}' -- 2.5.2 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/