Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Mon, 20 Jan 2020 22:40:46 +0100
From:   Pali =?utf-8?B?Um9ow6Fy?= <pali.rohar@gmail.com>
To:     OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>
Cc:     linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org,
        "Theodore Y. Ts'o" <tytso@mit.edu>,
        Namjae Jeon <linkinjeon@gmail.com>,
        Gabriel Krisman Bertazi <krisman@collabora.com>
Subject: Re: vfat: Broken case-insensitive support for UTF-8
Message-ID: <20200120214046.f6uq7rlih7diqahz@pali>
References: <20200119221455.bac7dc55g56q2l4r@pali>
 <87sgkan57p.fsf@mail.parknet.co.jp>
 <20200120110438.ak7jpyy66clx5v6x@pali>
 <875zh6pc0f.fsf@mail.parknet.co.jp>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
        protocol="application/pgp-signature"; boundary="uh7dvwiyhyuai6mk"
Content-Disposition: inline
In-Reply-To: <875zh6pc0f.fsf@mail.parknet.co.jp>
User-Agent: NeoMutt/20180716
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


--uh7dvwiyhyuai6mk
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Monday 20 January 2020 21:07:12 OGAWA Hirofumi wrote:
> Pali Roh=C3=A1r <pali.rohar@gmail.com> writes:
>=20
> >> To be perfect, the table would have to emulate what Windows use. It can
> >> be unicode standard, or something other.
> >
> > Windows FAT32 implementation (fastfat.sys) is opensource. So it should
> > be possible to inspect code and figure out how it is working.
> >
> > I will try to look at it.
>=20
> I don't think the conversion library is not in fs driver though,
> checking implement itself would be good.

Ok, I did some research. It took me it longer as I thought as lot of
stuff is undocumented and hard to find all relevant information.

So... fastfat.sys is using ntos function RtlUpcaseUnicodeString() which
takes UTF-16 string and returns upper case UTF-16 string. There is no
mapping table in fastfat.sys driver itself.

RtlUpcaseUnicodeString() is a ntos kernel function and after my research
it seems that this function is using only conversion table stored in
file l_intl.nls (from c:\windows\system32).

Project wine describe this file as "unicode casing tables" and seems
that it can parse this file format. Even more it distributes its own
version of this file which looks like to be generated from official
Unicode UnicodeData.txt via Perl script make_unicode (part of wine).

So question is... how much is MS changing l_intl.nls file in their
released Windows versions?

I would try to decode what is format of that file l_intl.nls and try to
compare data in it from some Windows versions.

Can we reuse upper case mapping table from that file?

--=20
Pali Roh=C3=A1r
pali.rohar@gmail.com

--uh7dvwiyhyuai6mk
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iF0EARECAB0WIQS4VrIQdKium2krgIWL8Mk9A+RDUgUCXiYeWQAKCRCL8Mk9A+RD
Uh0bAJ9oXybx7i8wTkLtZEpeDpnDUmCgNwCgys3n6EWFgeEQxy3My+sXPUnxXXQ=
=nzVm
-----END PGP SIGNATURE-----

--uh7dvwiyhyuai6mk--