From: Ashutosh Dixit <ashutosh.dixit@intel.com>
To: Toshi Kani <toshi.kani@hp.com>
Cc: "linux-kernel\@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "Dutt\, Sudeep" <sudeep.dutt@intel.com>,
        "Rao\, Nikhil" <nikhil.rao@intel.com>,
        "Williams\, Dan J" <dan.j.williams@intel.com>
Subject: Re: Regression in v4.2-rc1: vmalloc_to_page with ioremap
References: <6DC2528F945B4149AB6566DFB5F22ED39BC5540B@ORSMSX115.amr.corp.intel.com>
	<1437407695.3214.156.camel@hp.com> <1437407942.3214.159.camel@hp.com>
	<ndhwbqwpxuznkx.fsf@rasscmlmus001.pdx.intel.com>
	<1437420078.3214.185.camel@hp.com>
Date: Tue, 21 Jul 2015 08:17:02 -0700
Message-ID: <ndhwbqd1zla6c1.fsf@rasscmlmus001.pdx.intel.com>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.1 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Sender: linux-kernel-owner@vger.kernel.org
Content-Length: 3224
Lines: 70

On Mon, Jul 20 2015 at 12:21:18 PM, Toshi Kani <toshi.kani@hp.com> wrote:
>> > Can you send me outputs of the following files?  If the driver
>> > fails to load in v4.2-rc1, you can obtain the info in v4.1.
>> >
>> >   /proc/mtrr
>> >   /proc/iomem
>> >   /proc/vmallocinfo
>> >   /sys/kernel/debug/kernel_page_tables (need CONFIG_X86_PTDUMP set)
>>
>> Since the outputs are large I have sent you the outputs in a separate
>> mail outside the mailing list.
>
> Did you collect the info with your driver loaded?  I did not see ioremap
> requests from your driver in the vmallocinfo output.

Sorry, yes the driver was loaded. The ioremap itself is not done by scif.ko
but by mic_host.ko and then the addresses are passed to scif.ko. So I
think what you are looking for is this:

0xffffc90040000000-0xffffc90240001000 8589938688 mic_probe+0x281/0x5c0 \
[mic_host] phys=3c7e00000000 ioremap
0xffffc90280000000-0xffffc90480001000 8589938688 mic_probe+0x281/0x5c0 \
[mic_host] phys=3c7c00000000 ioremap

>> > Also, does the driver map a regular memory range with ioremap?  If not,
>> > how does 'struct page' get allocated for the range (since
>> > vmalloc_to_page returns a page pointer)?
>>
>> No the driver does not map regular memory with ioremap, only device
>> memory. vmalloc_to_page was returning a valid 'struct page' in this case
>> too. It appears it can do this correctly using pte_page as long as all
>>  four
>> page table levels (pgd, pud, pmd, pte) are present and the problem seems 
>> to
>> be happening because in the case of huge pages they are not. For us the 
>> bar
>> size is 8 G so we think that with the new ioremap maps the bars using 1 G
>> pages.
>
> Well, it is probably not a valid `struct page` pointer you got from
> vmalloc_to_page...  pte_page(pte) simply adds a pfn to vmemmap.  So, yes,
> you get a pointer, which can be put back to the pfn by subtracting vmemmap.
>  It may look fine, but this pointer does not point to a struct page unless
> it is part of the regular memory (or you somehow allocated struct page for
> the range).  Can you check to see if the struct page pointer from
> vmalloc_to_page actually points a struct page entry?
>
>   /* memmap is virtually contiguous.  */
>   #define __pfn_to_page(pfn)      (vmemmap + (pfn))
>   #define __page_to_pfn(page)     (unsigned long)((page) - vmemmap)

Yes, you are correct, the 'struct page' pointer returned by
vmalloc_to_page does not point to a "real" struct page entry. Neither
have we allocated a struct page for the range. However, because we pass
the returned pointer to streaming DMA mapping API's
(dma_map_ops->dma_map_sg or dma_map_ops->dma_map_page) and all those
functions do is call page_to_phys, they only care about the physical
address, it used to work.

Would it be possible to have a different API which can do this or can
vmalloc_to_page be updated to handle huge ioremaps without crashing?  Or
would you have a suggestion for doing this differently?

Thanks,
Ashutosh
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/