Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752543Ab3GSXvw (ORCPT ); Fri, 19 Jul 2013 19:51:52 -0400 Received: from mail-ie0-f170.google.com ([209.85.223.170]:60277 "EHLO mail-ie0-f170.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751368Ab3GSXvu (ORCPT ); Fri, 19 Jul 2013 19:51:50 -0400 MIME-Version: 1.0 In-Reply-To: <20130717093051.GK3421@sgi.com> References: <1373594635-131067-1-git-send-email-holt@sgi.com> <51E628F8.6030303@gmail.com> <20130717093051.GK3421@sgi.com> Date: Fri, 19 Jul 2013 16:51:49 -0700 X-Google-Sender-Auth: 7YLsUeEd4CRNK_BJr_oC3ZQN78o Message-ID: Subject: Re: [RFC 0/4] Transparent on-demand struct page initialization embedded in the buddy allocator From: Yinghai Lu To: Robin Holt Cc: Sam Ben , "H. Peter Anvin" , Ingo Molnar , Nate Zimmer , Linux Kernel , Linux MM , Rob Landley , Mike Travis , Daniel J Blueman , Andrew Morton , Greg KH , Mel Gorman Content-Type: multipart/mixed; boundary=90e6ba6e901ca9348804e1e602df Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Length: 7099 Lines: 118 --90e6ba6e901ca9348804e1e602df Content-Type: text/plain; charset=ISO-8859-1 On Wed, Jul 17, 2013 at 2:30 AM, Robin Holt wrote: > On Wed, Jul 17, 2013 at 01:17:44PM +0800, Sam Ben wrote: >> >With this patch, we did boot a 16TiB machine. Without the patches, >> >the v3.10 kernel with the same configuration took 407 seconds for >> >free_all_bootmem. With the patches and operating on 2MiB pages instead >> >of 1GiB, it took 26 seconds so performance was improved. I have no feel >> >for how the 1GiB chunk size will perform. >> >> How to test how much time spend on free_all_bootmem? > > We had put a pr_emerg at the beginning and end of free_all_bootmem and > then used a modified version of script which record the time in uSecs > at the beginning of each line of output. used two patches, found 3TiB system will take 100s before slub is ready. about three portions: 1. sparse vmemap buf allocation, it is with bootmem wrapper, so clear those struct page area take about 30s. 2. memmap_init_zone: take about 25s 3. mem_init/free_all_bootmem about 30s. so still wonder why 16TiB will need hours. also your patches looks like only address 2 and 3. Yinghai --90e6ba6e901ca9348804e1e602df Content-Type: application/octet-stream; name="printk_time_tsc_0.patch" Content-Disposition: attachment; filename="printk_time_tsc_0.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hjc1alqv0 U3ViamVjdDogW1BBVENIXSBwcmludGtfdGltZTogcHJlcGFyZSBzdHViIGZvciB1c2luZyBvdGhl ciB0aGFuIGNwdV9jbG9jawoKLXYyOiByZWZyZXNoIHRvIHYzLjEwCgpTaWduZWQtb2ZmLWJ5OiBZ aW5naGFpIEx1IDx5aGx1Lmtlcm5lbEBnbWFpbC5jb20+CgotLS0KIGluY2x1ZGUvbGludXgvcHJp bnRrLmggfCAgICAzICsrKwogaW5pdC9tYWluLmMgICAgICAgICAgICB8ICAgIDEgKwoga2VybmVs L3ByaW50ay5jICAgICAgICB8ICAgMTcgKysrKysrKysrKysrKysrLS0KIDMgZmlsZXMgY2hhbmdl ZCwgMTkgaW5zZXJ0aW9ucygrKSwgMiBkZWxldGlvbnMoLSkKCkluZGV4OiBsaW51eC0yLjYvaW5p dC9tYWluLmMKPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PQotLS0gbGludXgtMi42Lm9yaWcvaW5pdC9tYWluLmMKKysrIGxp bnV4LTIuNi9pbml0L21haW4uYwpAQCAtNjAxLDYgKzYwMSw3IEBAIGFzbWxpbmthZ2Ugdm9pZCBf X2luaXQgc3RhcnRfa2VybmVsKHZvaWQKIAkJbGF0ZV90aW1lX2luaXQoKTsKIAlzY2hlZF9jbG9j a19pbml0KCk7CiAJY2FsaWJyYXRlX2RlbGF5KCk7CisJc2V0X3ByaW50a190aW1lX2Nsb2NrKGxv Y2FsX2Nsb2NrKTsKIAlwaWRtYXBfaW5pdCgpOwogCWFub25fdm1hX2luaXQoKTsKICNpZmRlZiBD T05GSUdfWDg2CkluZGV4OiBsaW51eC0yLjYva2VybmVsL3ByaW50ay5jCj09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0t IGxpbnV4LTIuNi5vcmlnL2tlcm5lbC9wcmludGsuYworKysgbGludXgtMi42L2tlcm5lbC9wcmlu dGsuYwpAQCAtMjE3LDYgKzIxNywxOSBAQCBzdHJ1Y3QgbG9nIHsKIHN0YXRpYyBERUZJTkVfUkFX X1NQSU5MT0NLKGxvZ2J1Zl9sb2NrKTsKIAogI2lmZGVmIENPTkZJR19QUklOVEsKKworc3RhdGlj IHU2NCBkZWZhdWx0X3ByaW50a190aW1lX2Nsb2NrKHZvaWQpCit7CisJcmV0dXJuIDA7Cit9CisK K3N0YXRpYyB1NjQgKCpwcmludGtfdGltZV9jbG9jaykodm9pZCkgPSBkZWZhdWx0X3ByaW50a190 aW1lX2Nsb2NrOworCit2b2lkIHNldF9wcmludGtfdGltZV9jbG9jayh1NjQgKCpmbikodm9pZCkp Cit7CisJcHJpbnRrX3RpbWVfY2xvY2sgPSBmbjsKK30KKwogREVDTEFSRV9XQUlUX1FVRVVFX0hF QUQobG9nX3dhaXQpOwogLyogdGhlIG5leHQgcHJpbnRrIHJlY29yZCB0byByZWFkIGJ5IHN5c2xv ZyhSRUFEKSBvciAvcHJvYy9rbXNnICovCiBzdGF0aWMgdTY0IHN5c2xvZ19zZXE7CkBAIC0zNTQs NyArMzY3LDcgQEAgc3RhdGljIHZvaWQgbG9nX3N0b3JlKGludCBmYWNpbGl0eSwgaW50CiAJaWYg KHRzX25zZWMgPiAwKQogCQltc2ctPnRzX25zZWMgPSB0c19uc2VjOwogCWVsc2UKLQkJbXNnLT50 c19uc2VjID0gbG9jYWxfY2xvY2soKTsKKwkJbXNnLT50c19uc2VjID0gcHJpbnRrX3RpbWVfY2xv Y2soKTsKIAltZW1zZXQobG9nX2RpY3QobXNnKSArIGRpY3RfbGVuLCAwLCBwYWRfbGVuKTsKIAlt c2ctPmxlbiA9IHNpemVvZihzdHJ1Y3QgbG9nKSArIHRleHRfbGVuICsgZGljdF9sZW4gKyBwYWRf bGVuOwogCkBAIC0xNDU3LDcgKzE0NzAsNyBAQCBzdGF0aWMgYm9vbCBjb250X2FkZChpbnQgZmFj aWxpdHksIGludCBsCiAJCWNvbnQuZmFjaWxpdHkgPSBmYWNpbGl0eTsKIAkJY29udC5sZXZlbCA9 IGxldmVsOwogCQljb250Lm93bmVyID0gY3VycmVudDsKLQkJY29udC50c19uc2VjID0gbG9jYWxf Y2xvY2soKTsKKwkJY29udC50c19uc2VjID0gcHJpbnRrX3RpbWVfY2xvY2soKTsKIAkJY29udC5m bGFncyA9IDA7CiAJCWNvbnQuY29ucyA9IDA7CiAJCWNvbnQuZmx1c2hlZCA9IGZhbHNlOwpJbmRl eDogbGludXgtMi42L2luY2x1ZGUvbGludXgvcHJpbnRrLmgKPT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQotLS0gbGludXgt Mi42Lm9yaWcvaW5jbHVkZS9saW51eC9wcmludGsuaAorKysgbGludXgtMi42L2luY2x1ZGUvbGlu dXgvcHJpbnRrLmgKQEAgLTEwNyw2ICsxMDcsNyBAQCB2b2lkIGVhcmx5X3ByaW50ayhjb25zdCBj aGFyICpzLCAuLi4pIHsKICNlbmRpZgogCiAjaWZkZWYgQ09ORklHX1BSSU5USworZXh0ZXJuIHZv aWQgc2V0X3ByaW50a190aW1lX2Nsb2NrKHU2NCAoKmZuKSh2b2lkKSk7CiBhc21saW5rYWdlIF9f cHJpbnRmKDUsIDApCiBpbnQgdnByaW50a19lbWl0KGludCBmYWNpbGl0eSwgaW50IGxldmVsLAog CQkgY29uc3QgY2hhciAqZGljdCwgc2l6ZV90IGRpY3RsZW4sCkBAIC0xNTAsNiArMTUxLDggQEAg dm9pZCBkdW1wX3N0YWNrX3NldF9hcmNoX2Rlc2MoY29uc3QgY2hhcgogdm9pZCBkdW1wX3N0YWNr X3ByaW50X2luZm8oY29uc3QgY2hhciAqbG9nX2x2bCk7CiB2b2lkIHNob3dfcmVnc19wcmludF9p bmZvKGNvbnN0IGNoYXIgKmxvZ19sdmwpOwogI2Vsc2UKK3N0YXRpYyBpbmxpbmUgdm9pZCBzZXRf cHJpbnRrX3RpbWVfY2xvY2sodTY0ICgqZm4pKHZvaWQpKSB7IH0KKwogc3RhdGljIGlubGluZSBf X3ByaW50ZigxLCAwKQogaW50IHZwcmludGsoY29uc3QgY2hhciAqcywgdmFfbGlzdCBhcmdzKQog ewo= --90e6ba6e901ca9348804e1e602df Content-Type: application/octet-stream; name="printk_time_tsc_1.patch" Content-Disposition: attachment; filename="printk_time_tsc_1.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hjc1au5d1 U3ViamVjdDogW1BBVENIXSB4ODY6IHByaW50a190aW1lIHRvIHVzZSB0c2MgYmVmb3JlIGNwdV9j bG9jayBpcyByZWFkeQoKc28gY2FuIGdldCB0c2MgdmFsdWUgb24gcHJpbnRrCgpuZWVkIHRvIGFw cGx5IGFmdGVyCglbUEFUQ0hdIHByaW50a190aW1lOiBwcmVwYXJlIHN0dWIgZm9yIHVzaW5nIG90 aGVyIHRoYW4gY3B1X2Nsb2NrCgotdjI6IHJlZnJlc2ggb24gdjMuMTAKClNpZ25lZC1vZmYtYnk6 IFlpbmdoYWkgTHUgPHlpbmdoYWlAa2VybmVsLm9yZz4KCi0tLQogYXJjaC94ODYva2VybmVsL2Nw dS9jb21tb24uYyB8ICAgMjEgKysrKysrKysrKysrKysrKysrKysrCiAxIGZpbGUgY2hhbmdlZCwg MjEgaW5zZXJ0aW9ucygrKQoKSW5kZXg6IGxpbnV4LTIuNi9hcmNoL3g4Ni9rZXJuZWwvY3B1L2Nv bW1vbi5jCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT0KLS0tIGxpbnV4LTIuNi5vcmlnL2FyY2gveDg2L2tlcm5lbC9jcHUv Y29tbW9uLmMKKysrIGxpbnV4LTIuNi9hcmNoL3g4Ni9rZXJuZWwvY3B1L2NvbW1vbi5jCkBAIC03 MzcsNiArNzM3LDI1IEBAIHN0YXRpYyB2b2lkIF9faW5pdCBlYXJseV9pZGVudGlmeV9jcHUoc3QK IAlzZXR1cF9mb3JjZV9jcHVfY2FwKFg4Nl9GRUFUVVJFX0FMV0FZUyk7CiB9CiAKK3N0YXRpYyB1 NjQgX19pbml0ZGF0YSB0c2NfY2xvY2tfYmFzZTsKKworc3RhdGljIF9faW5pdCB1NjQgdHNjX2Ns b2NrX29mZnNldCh2b2lkKQoreworCXVuc2lnbmVkIGxvbmcgbG9uZyB0OworCisJcmR0c2NsbCh0 KTsKKworCXJldHVybiB0IC0gdHNjX2Nsb2NrX2Jhc2U7Cit9CisKK3N0YXRpYyBfX2luaXQgdm9p ZCBlYXJseV9wcmludGtfdGltZV9pbml0KHZvaWQpCit7CisJaWYgKGNwdV9oYXNfdHNjKSB7CisJ CXJkdHNjbGwodHNjX2Nsb2NrX2Jhc2UpOworCQlzZXRfcHJpbnRrX3RpbWVfY2xvY2sodHNjX2Ns b2NrX29mZnNldCk7CisJfQorfQorCiB2b2lkIF9faW5pdCBlYXJseV9jcHVfaW5pdCh2b2lkKQog ewogCWNvbnN0IHN0cnVjdCBjcHVfZGV2ICpjb25zdCAqY2RldjsKQEAgLTc2MSw2ICs3ODAsOCBA QCB2b2lkIF9faW5pdCBlYXJseV9jcHVfaW5pdCh2b2lkKQogCX0KIAogCWVhcmx5X2lkZW50aWZ5 X2NwdSgmYm9vdF9jcHVfZGF0YSk7CisKKwllYXJseV9wcmludGtfdGltZV9pbml0KCk7CiB9CiAK IC8qCg== --90e6ba6e901ca9348804e1e602df-- -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/