Received-SPF: pass (google.com: best guess record for domain of linux-kernel-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67;
Date:   Mon, 20 Jan 2020 17:27:01 +0100
From:   Pali =?utf-8?B?Um9ow6Fy?= <pali.rohar@gmail.com>
To:     David Laight <David.Laight@ACULAB.COM>
Cc:     OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>,
        "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
        "linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
        "Theodore Y. Ts'o" <tytso@mit.edu>,
        Namjae Jeon <linkinjeon@gmail.com>,
        Gabriel Krisman Bertazi <krisman@collabora.com>
Subject: Re: vfat: Broken case-insensitive support for UTF-8
Message-ID: <20200120162701.guxcrmqysejaqw6y@pali>
References: <20200119221455.bac7dc55g56q2l4r@pali>
 <87sgkan57p.fsf@mail.parknet.co.jp>
 <20200120110438.ak7jpyy66clx5v6x@pali>
 <89eba9906011446f8441090f496278d2@AcuMS.aculab.com>
 <20200120152009.5vbemgmvhke4qupq@pali>
 <1a4c545dc7f14e33b7e59321a0aab868@AcuMS.aculab.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
        protocol="application/pgp-signature"; boundary="rb3yd7yh2odisgq3"
Content-Disposition: inline
In-Reply-To: <1a4c545dc7f14e33b7e59321a0aab868@AcuMS.aculab.com>
User-Agent: NeoMutt/20180716
Sender: linux-kernel-owner@vger.kernel.org
Precedence: bulk


--rb3yd7yh2odisgq3
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Monday 20 January 2020 15:47:22 David Laight wrote:
> From: Pali Roh=C3=A1r
> > Sent: 20 January 2020 15:20
> ...
> > This is not possible. There is 1:1 mapping between UTF-8 sequence and
> > Unicode code point. wchar_t in kernel represent either one Unicode code
> > point (limited up to U+FFFF in NLS framework functions) or 2bytes in
> > UTF-16 sequence (only in utf8s_to_utf16s() and utf16s_to_utf8s()
> > functions).
>=20
> Unfortunately there is neither a 1:1 mapping of all possible byte sequenc=
es
> to wchar_t (or unicode code points),

I was talking about valid UTF-8 sequence (invalid, illformed is out of
game and for sure would always cause problems).

> nor a 1:1 mapping of all possible wchar_t values to UTF-8.

This is not truth. There is exactly only one way how to convert sequence
of Unicode code points to UTF-8. UTF is Unicode Transformation Format
and has exact definition how is Unicode Transformed.

If you have valid UTF-8 sequence then it describe one exact sequence of
Unicode code points. And if you have sequence (ordinals) of Unicode code
points there is exactly one and only one its representation in UTF-8.

I would suggest you to read Unicode standard, section 2.5 Encoding Forms.

> Really both need to be defined - even for otherwise 'invalid' sequences.
>=20
> Even the 16-bit values above 0xd000 can appear on their own in
> windows filesystems (according to wikipedia).

If you are talking about UTF-16 (which is _not_ 16-bit as you wrote),
look at my previous email:

"MS FAT32 implementations allows half of UTF-16 surrogate pair stored in FS=
=2E"

> It is all to easy to get sequences of values that cannot be converted
> to/from UTF-8.
>=20
> 	David
>=20
> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1=
 1PT, UK
> Registration No: 1397386 (Wales)

--=20
Pali Roh=C3=A1r
pali.rohar@gmail.com

--rb3yd7yh2odisgq3
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iF0EARECAB0WIQS4VrIQdKium2krgIWL8Mk9A+RDUgUCXiXU0wAKCRCL8Mk9A+RD
UlffAJ0elMMTIUY0wtAoDO7B5Dqo/pfzcwCdFon3xWqzyaeLu9BTsknYE0wNmjE=
=ZCx6
-----END PGP SIGNATURE-----

--rb3yd7yh2odisgq3--