Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932708AbbELJn7 (ORCPT ); Tue, 12 May 2015 05:43:59 -0400 Received: from mail-wi0-f175.google.com ([209.85.212.175]:33613 "EHLO mail-wi0-f175.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752844AbbELJnz (ORCPT ); Tue, 12 May 2015 05:43:55 -0400 Date: Tue, 12 May 2015 11:43:49 +0200 From: Ingo Molnar To: Len Brown Cc: Jan =?iso-8859-1?Q?H=2E_Sch=F6nherr?= , Thomas Gleixner , X86 ML , "linux-kernel@vger.kernel.org" , Anthony Liguori , Ingo Molnar , "H. Peter Anvin" , Tim Deegan , Gang Wei , Linus Torvalds Subject: Re: [PATCH] x86: skip delays during SMP initialization similar to Xen Message-ID: <20150512094348.GA11477@gmail.com> References: <1430732554-7294-1-git-send-email-jschoenh@amazon.de> <20150506082759.GA30019@gmail.com> <20150507102351.GA14347@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 4690 Lines: 88 * Len Brown wrote: > On Thu, May 7, 2015 at 6:23 AM, Ingo Molnar wrote: > > These numbers do not look right. > > > Here's what the boot time looks like on a 120 CPUs system, with the > > patch applied: > > > > [ 0.558947] x86: Booting SMP configuration: > > [ 0.563375] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 > > [ 0.644851] .... node #1, CPUs: #15 #16 #17 #18 #19 #20 #21 #22 #23 #24 #25 #26 #27 #28 #29 > > [ 0.830474] .... node #2, CPUs: #30 #31 #32 #33 #34 #35 #36 #37 #38 #39 #40 #41 #42 #43 #44 > > [ 1.016357] .... node #3, CPUs: #45 #46 #47 #48 #49 #50 #51 #52 #53 #54 #55 #56 #57 #58 #59 > > [ 1.202342] .... node #0, CPUs: #60 #61 #62 #63 #64 #65 #66 #67 #68 #69 #70 #71 #72 #73 #74 > > [ 1.283864] .... node #1, CPUs: #75 #76 #77 #78 #79 #80 #81 #82 #83 #84 #85 #86 #87 #88 #89 > > [ 1.397131] .... node #2, CPUs: #90 #91 #92 #93 #94 #95 #96 #97 #98 #99 #100 #101 #102 #103 #104 > > [ 1.510417] .... node #3, CPUs: #105 #106 #107 #108 #109 #110 #111 #112 #113 #114 #115 #116 #117 #118 #119 > > [ 1.620967] x86: Booted up 4 nodes, 120 CPUs > > [ 1.625928] smpboot: Total of 120 processors activated (672866.16 BogoMIPS) > > > > 1.1 seconds to boot 120 CPUs, 10.8 seconds to hit init, that's an > > entirely reasonable runtime I think. > > > > It was 20+ seconds before that, 10+ seconds for the SMP bootup > > sequence. > > (1.625928-0.558947) = 1.07 seconds to online 119 additional cpus. > /119 = .0089662268 each, lets call it 9ms. > > Here is my ivb-ex running stock fedora 21's Linux-3.19 (no patch applied): > > [ 0.404369] x86: Booting SMP configuration: > [ 0.409492] .... node #0, CPUs: #1 > [ 0.439900] NMI watchdog: enabled on all CPUs, permanently consumes > one hw-PMU counter. > [ 0.450533] #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 > [ 0.652409] .... node #1, CPUs: #15 #16 #17 #18 #19 #20 > #21 #22 #23 #24 #25 #26 #27 #28 #29 > [ 0.999645] .... node #2, CPUs: #30 #31 #32 #33 #34 #35 > #36 #37 #38 #39 #40 #41 #42 #43 #44 > [ 1.346991] .... node #3, CPUs: #45 #46 #47 #48 #49 #50 > #51 #52 #53 #54 #55 #56 #57 #58 #59 > [ 1.694171] .... node #0, CPUs: #60 #61 #62 #63 #64 #65 > #66 #67 #68 #69 #70 #71 #72 #73 #74 > [ 1.928248] .... node #1, CPUs: #75 #76 #77 #78 #79 #80 > #81 #82 #83 #84 #85 #86 #87 #88 #89 > [ 2.198370] .... node #2, CPUs: #90 #91 #92 #93 #94 #95 > #96 #97 #98 #99 #100 #101 #102 #103 #104 > [ 2.468574] .... node #3, CPUs: #105 #106 #107 #108 #109 #110 > #111 #112 #113 #114 #115 #116 #117 #118 #119 > [ 2.737884] x86: Booted up 4 nodes, 120 CPUs > [ 2.743758] smpboot: Total of 120 processors activated (671097.18 BogoMIPS) > > (2.743758-0.404369) = 2.339389 for all 119 processors > /119 = .01965873109243697478 - lets call it 19ms each > > so this baseline case is 19ms/processor -- which matches above, where > we delete 10ms/processor. > But even at 19ms each, this is totals only 2.3 seconds for all 119 > processors. So I don't understand your reference to 10+ seconds, > above. Hm, so what I was referring to is: [ 0.205318] x86: Booting SMP configuration: [ 0.206005] .... node #0, CPUs: #1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 #23 #24 #25 #26 #27 #28 #29 #30 #31 #32 #33 #34 #35 #36 #37 #38 #39 #40 #41 #42 #43 #44 #45 #46 #47 #48 #49 #50 #51 #52 #53 #54 #55 #56 #57 #58 #59 #60 #61 #62 #63 #64 #65 #66 #67 #68 #69 #70 #71 #72 #73 #74 #75 #76 #77 #78 #79 #80 #81 #82 #83 #84 #85 #86 #87 #88 #89 #90 #91 #92 #93 #94 #95 #96 #97 #98 #99 #100 #101 #102 #103 #104 #105 #106 #107 #108 #109 #110 #111 #112 #113 #114 #115 #116 #117 #118 #119 [ 9.021191] x86: Booted up 1 node, 120 CPUs [ 9.021803] smpboot: Total of 120 processors activated (672102.75 BogoMIPS) [ 9.027282] devtmpfs: initialized So it's 8.9 seconds - but still much more than you'd expect. The delay is evenly distributed amongst CPUs. > ps. I wish the BIOS boot time on this box were as fast as Linux:-) LOL! :-) Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/