Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id ; Wed, 20 Feb 2002 16:37:48 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id ; Wed, 20 Feb 2002 16:37:38 -0500 Received: from vger.timpanogas.org ([207.109.151.240]:36839 "EHLO vger.timpanogas.org") by vger.kernel.org with ESMTP id ; Wed, 20 Feb 2002 16:37:23 -0500 Date: Wed, 20 Feb 2002 14:54:49 -0700 From: "Jeff V. Merkey" To: linux-kernel@vger.kernel.org Cc: jmerkey@timpanogas.org Subject: Re: ioremap()/PCI sickness in 2.4.18-rc2 (FIXED ALMOST) Message-ID: <20020220145449.A1102@vger.timpanogas.org> In-Reply-To: <20020220103320.A32211@vger.timpanogas.org> <20020220103539.B32211@vger.timpanogas.org> <3C73DC34.E83CCD35@mandrakesoft.com> <20020220.093034.112623671.davem@redhat.com> <20020220110004.A32431@vger.timpanogas.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5i In-Reply-To: <20020220110004.A32431@vger.timpanogas.org>; from jmerkey@vger.timpanogas.org on Wed, Feb 20, 2002 at 11:00:04AM -0700 Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org The following information is submitted regarding this problem: I have corrected the prefetch allocation problem by reducing the prefetch address space size from 512 MB down to 32 MB. The failing code is in get_vm_area(). This bug effectively reduces the ability of Linux over other OS's to run scalable SCI based applications with large numbers of nodes if there are large numbers of nodes in a cluster and these nodes have more than 1 GB of memory in each system. My RAID/FS application does not need a huge prefetch address space so I can get around this problem easily, but some of the SCI applications do, and this problem will relegate Linux to a back seat status for supercomputer applications that use this technology if Linux in unable to map larger regions of memory. I would propose that the maintainer of vmalloc.c look at using 48 bit PTE entries or some other solution as a way to alloc larger virtual address frames when the system has a lot of physical memory. It's seems pretty lame to me for a machine with 2 GB of physical memory not to have at lest 256 MB of address space left over for address mapping. Offending code attached. Please advise as to what a proposed solution could be if any is possible with this problem. I am happy to adjust the Dolphin IRM driver behavior to accomodate Linux but I think some larger clusters (i.e. > 1000 nodes) may not work with Linux is there's not enough address space to map remotely across the cluster. Jeff struct vm_struct * get_vm_area(unsigned long size, unsigned long flags) { unsigned long addr; struct vm_struct **p, *tmp, *area; area = (struct vm_struct *) kmalloc(sizeof(*area), GFP_KERNEL); if (!area) return NULL; size += PAGE_SIZE; addr = VMALLOC_START; write_lock(&vmlist_lock); for (p = &vmlist; (tmp = *p) ; p = &tmp->next) { ===============> we barf here since the size + addr wraps if ((size + addr) < addr) goto out; if (size + addr <= (unsigned long) tmp->addr) break; addr = tmp->size + (unsigned long) tmp->addr; if (addr > VMALLOC_END-size) goto out; } area->flags = flags; area->addr = (void *)addr; area->size = size; area->next = *p; *p = area; write_unlock(&vmlist_lock); return area; out: write_unlock(&vmlist_lock); kfree(area); return NULL; } On Wed, Feb 20, 2002 at 11:00:04AM -0700, Jeff V. Merkey wrote: > > > David, > > Someone had a thought that perhaps the Serverworks chipset is mapping > addresses above the 4GB boundry. Any thoughts on how to get around > this problem? > > Jeff > > > On Wed, Feb 20, 2002 at 09:30:34AM -0800, David S. Miller wrote: > > From: Jeff Garzik > > Date: Wed, 20 Feb 2002 12:26:12 -0500 > > > > type abuse aside, and alpha bugs aside, this looks ok... what is the > > value of as->msize? > > > > Jeff and Jeff, the problem is one of two things: > > > > 1) when you have ~2GB of memory the vmalloc pool is very small > > and this it the same place ioremap allocations come from > > > > 2) the BIOS or Linus is not assigning resources of the device > > properly, or it simple can't because the available PCI MEM space > > with this much memory is too small > > > > I note that one of the resources of the card is 16MB or so. > > - > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > Please read the FAQ at http://www.tux.org/lkml/ > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/