LinuxLists.cc - [PATCH] Convert files to UTF-8 and some cleanups

2007-05-14 00:01:10

Subject: [PATCH] Convert files to UTF-8 and some cleanups

Convert files to UTF-8 encoding, remove useless NBSP and fix "table"
drawing.

Signed-off-by: Marcin Garski <[email protected]>

--
Marcin Garski

Attachments:

linux-2.6.22-rc1-utf-8-strings.patch.bz2 (19.82 kB)

2007-05-14 00:08:18

by Marcin Garski

[permalink] [raw]

Subject: Re: [PATCH] Convert files to UTF-8 and some cleanups

Marcin Garski wrote:
> Convert files to UTF-8 encoding, remove useless NBSP and fix "table"
> drawing.
>
> Signed-off-by: Marcin Garski <[email protected]>

By mistake I sent wrong patch that won't apply cleanly, here's the good
one. Sorry.

--
Marcin Garski

Attachments:

linux-2.6.22-rc1-utf-8-strings.patch.bz2 (19.73 kB)

2007-05-14 14:15:24

by Stefan Richter

[permalink] [raw]

Subject: Re: [PATCH] Convert files to UTF-8 and some cleanups

Marcin Garski wrote:
> Content-Type: application/octet-stream;
> name="linux-2.6.22-rc1-utf-8-strings.patch.bz2"
> Content-Transfer-Encoding: base64
> Content-Disposition: attachment;
> filename="linux-2.6.22-rc1-utf-8-strings.patch.bz2"
>
> QlpoOTFBWSZTWSc/F00Ahxp/v///itd///////////////8QQO6yN0QYAABArrIlRGBofng9
> APLeS9999e3t9HBeePnl0PJwG73R777qguxoH3sAPfYaKB0BwABdbe4C9nRevqU6lwXTg8De

It's customary to post in plain text. Best is inline plain text. Thanks,
--
Stefan Richter
-=====-=-=== -=-= -===-
http://arcgraph.de/sr/

2007-05-14 15:28:41

by John Anthony Kazos Jr.

[permalink] [raw]

Subject: Re: [PATCH] Convert files to UTF-8 and some cleanups

> Marcin Garski wrote:
> > Content-Type: application/octet-stream;
> > name="linux-2.6.22-rc1-utf-8-strings.patch.bz2"
> > Content-Transfer-Encoding: base64
> > Content-Disposition: attachment;
> > filename="linux-2.6.22-rc1-utf-8-strings.patch.bz2"
> >
> > QlpoOTFBWSZTWSc/F00Ahxp/v///itd///////////////8QQO6yN0QYAABArrIlRGBofng9
> > APLeS9999e3t9HBeePnl0PJwG73R777qguxoH3sAPfYaKB0BwABdbe4C9nRevqU6lwXTg8De
>
> It's customary to post in plain text. Best is inline plain text. Thanks,

You can't post a patch to UTF-8 in plain text, because it isn't plain
text. MTAs and MUAs mangle the hell out of them. Look back in the archives
for my postings. Once you've got something in UTF-8, -*THEN*- you can post
in plain inline text.

2007-05-14 16:26:28

by Stefan Richter

[permalink] [raw]

Subject: Re: [PATCH] Convert files to UTF-8 and some cleanups

John Anthony Kazos Jr. wrote:
> You can't post a patch to UTF-8 in plain text, because it isn't plain
> text.

Of course it is, you said so...

> MTAs and MUAs mangle the hell out of them. Look back in the archives
> for my postings. Once you've got something in UTF-8, -*THEN*- you can post
> in plain inline text.

...yourself. Just send as

Content-type: text/plain; charset="utf-8"

Of course a patch like that will usually contain several character sets,
but that's no problem as long as the byte codes are preserved and we can
easily see how the end result looks like. The important thing is that
the submitter's and committer's MUAs do it right and reviewers' MUAs do
it sort of right. Also, nobody cares if there is still some pre-WWI
7bit-only MTA operated in some lone desert town. And if some web
archive of LKML isn't compatible to internet mail, then that's tough luck.
--
Stefan Richter
-=====-=-=== -=-= -===-
http://arcgraph.de/sr/

2007-05-14 16:43:19

by Stefan Richter

[permalink] [raw]

Subject: Re: [PATCH] Convert files to UTF-8 and some cleanups

> The important thing is that the submitter's and committer's MUAs do it
> right and reviewers' MUAs do it sort of right.

PS: If the contributor really wants to use a dysfunctional MUA, he
should at least include the diffstat in plaintext. That might motivate
some more people to unroll an attachment just to have a look at it.
--
Stefan Richter
-=====-=-=== -=-= -===-
http://arcgraph.de/sr/

2007-05-14 16:46:17

by Adrian Bunk

[permalink] [raw]

Subject: Re: [PATCH] Convert files to UTF-8 and some cleanups

On Mon, May 14, 2007 at 06:25:14PM +0200, Stefan Richter wrote:
> John Anthony Kazos Jr. wrote:
> > You can't post a patch to UTF-8 in plain text, because it isn't plain
> > text.
>
> Of course it is, you said so...
>
> > MTAs and MUAs mangle the hell out of them. Look back in the archives
> > for my postings. Once you've got something in UTF-8, -*THEN*- you can post
> > in plain inline text.
>
> ...yourself. Just send as
>
> Content-type: text/plain; charset="utf-8"
>
> Of course a patch like that will usually contain several character sets,
> but that's no problem as long as the byte codes are preserved and we can
> easily see how the end result looks like. The important thing is that
> the submitter's and committer's MUAs do it right and reviewers' MUAs do
> it sort of right. Also, nobody cares if there is still some pre-WWI
> 7bit-only MTA operated in some lone desert town. And if some web
> archive of LKML isn't compatible to internet mail, then that's tough luck.

The problem is that the unconverted characters are always invalid UTF-8 [1],
and it's 100% correct for a MUA to convert these invalid UTF-8
characters to replacement characters - if a MUA claims to send valid
charset="utf-8", it is simply not allowed to preserve the byte codes.

> Stefan Richter

cu
Adrian

[1] except for some cases with two or more non 7bit ASCII
characters in a row

--

"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

2007-05-14 17:19:27

by Stefan Richter

[permalink] [raw]

Subject: Re: [PATCH] Convert files to UTF-8 and some cleanups

Adrian Bunk wrote:
> The problem is that the unconverted characters are always invalid UTF-8 [1],
> and it's 100% correct for a MUA to convert these invalid UTF-8
> characters to replacement characters - if a MUA claims to send valid
> charset="utf-8", it is simply not allowed to preserve the byte codes.
...
> [1] except for some cases with two or more non 7bit ASCII
> characters in a row

OK, I missed that, sorry. Then

Content-type: text/plain; charset="unknown-8bit"

can be put to good use.
But as I said, at least a diffstat would be nice.
--
Stefan Richter
-=====-=-=== -=-= -===-
http://arcgraph.de/sr/

2007-05-14 17:51:20

by Jan Engelhardt

[permalink] [raw]

Subject: Re: [PATCH] Convert files to UTF-8 and some cleanups

On May 14 2007 16:14, Stefan Richter wrote:
>Marcin Garski wrote:
>> Content-Type: application/octet-stream;
>> name="linux-2.6.22-rc1-utf-8-strings.patch.bz2"
>> Content-Transfer-Encoding: base64
>> Content-Disposition: attachment;
>> filename="linux-2.6.22-rc1-utf-8-strings.patch.bz2"
>>
>> QlpoOTFBWSZTWSc/F00Ahxp/v///itd///////////////8QQO6yN0QYAABArrIlRGBofng9
>> APLeS9999e3t9HBeePnl0PJwG73R777qguxoH3sAPfYaKB0BwABdbe4C9nRevqU6lwXTg8De

I've had a quick look at it...

* Copyright (C) 2004 Nokia Corporation
* Written by Tony Lindgren <[email protected]>
- * Major cleanups by Juha YrjXlX <[email protected]>
+ * Major cleanups by Juha Yrjölä <[email protected]>

I thought I already fixed that part with the patch from
http://lkml.org/lkml/2007/1/8/222

but of course I can't tell without getting at that .bz2 thing I
sent back them. Anyone know of a site that stores these too?

Jan
--

2007-05-14 18:06:42

by Stefan Richter

[permalink] [raw]

Subject: Re: [PATCH] Convert files to UTF-8 and some cleanups

On 14 May, Jan Engelhardt wrote:
> I've had a quick look at it...
>
> * Copyright (C) 2004 Nokia Corporation
> * Written by Tony Lindgren <[email protected]>
> - * Major cleanups by Juha YrjXlX <[email protected]>
> + * Major cleanups by Juha Yrj?l? <[email protected]>
>
>
> I thought I already fixed that part with the patch from
> http://lkml.org/lkml/2007/1/8/222
>
> but of course I can't tell without getting at that .bz2 thing I
> sent back them. Anyone know of a site that stores these too?

http://marc.info/?l=linux-kernel&m=116828781409931
--
Stefan Richter
-=====-=-=== -=-= -===-
http://arcgraph.de/sr/

2007-05-14 20:11:33

by Jan Engelhardt

[permalink] [raw]

Subject: Re: [PATCH] Convert files to UTF-8 and some cleanups

On May 14 2007 20:06, Stefan Richter wrote:
>On 14 May, Jan Engelhardt wrote:
>> I've had a quick look at it...
>>
>> * Copyright (C) 2004 Nokia Corporation
>> * Written by Tony Lindgren <[email protected]>
>> - * Major cleanups by Juha YrjXlX <[email protected]>
>> + * Major cleanups by Juha Yrjölä <[email protected]>
>>
>>
>> I thought I already fixed that part with the patch from
>> http://lkml.org/lkml/2007/1/8/222
>>
>> but of course I can't tell without getting at that .bz2 thing I
>> sent back them. Anyone know of a site that stores these too?
>
>http://marc.info/?l=linux-kernel&m=116828781409931

I checked.
Now that kinda sucks, everyone does the same job over and over,
I did a ton of utf8 fixes and they did not get applied. (Granted,
I did not cc Andrew, Linus or trivial).

>--
>Stefan Richter
>-=====-=-=== -=-= -===-
>http://arcgraph.de/sr/
>
>

Jan
--

2007-05-14 20:25:43

by Adrian Bunk

[permalink] [raw]

Subject: Re: [PATCH] Convert files to UTF-8 and some cleanups

On Mon, May 14, 2007 at 10:10:20PM +0200, Jan Engelhardt wrote:
>
> On May 14 2007 20:06, Stefan Richter wrote:
> >On 14 May, Jan Engelhardt wrote:
> >> I've had a quick look at it...
> >>
> >> * Copyright (C) 2004 Nokia Corporation
> >> * Written by Tony Lindgren <[email protected]>
> >> - * Major cleanups by Juha YrjXlX <[email protected]>
> >> + * Major cleanups by Juha Yrjölä <[email protected]>
> >>
> >>
> >> I thought I already fixed that part with the patch from
> >> http://lkml.org/lkml/2007/1/8/222
> >>
> >> but of course I can't tell without getting at that .bz2 thing I
> >> sent back them. Anyone know of a site that stores these too?
> >
> >http://marc.info/?l=linux-kernel&m=116828781409931
>
> I checked.
> Now that kinda sucks, everyone does the same job over and over,
> I did a ton of utf8 fixes and they did not get applied. (Granted,
> I did not cc Andrew, Linus or trivial).

If you had Cc'ed trivial they would now be in Linus' tree...

> Jan

cu
Adrian

--

"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

2007-05-14 20:38:22

by Jan Engelhardt

[permalink] [raw]

Subject: Re: [PATCH] Convert files to UTF-8 and some cleanups

On May 14 2007 22:25, Adrian Bunk wrote:
>> >
>> >http://marc.info/?l=linux-kernel&m=116828781409931
>>
>> I checked.
>> Now that kinda sucks, everyone does the same job over and over,
>> I did a ton of utf8 fixes and they did not get applied. (Granted,
>> I did not cc Andrew, Linus or trivial).
>
>If you had Cc'ed trivial they would now be in Linus' tree...

I figured that by now, but does it still matter now?

Jan
--

2007-05-14 20:41:17

by Adrian Bunk

[permalink] [raw]

Subject: Re: [PATCH] Convert files to UTF-8 and some cleanups

On Mon, May 14, 2007 at 10:36:00PM +0200, Jan Engelhardt wrote:
>
> On May 14 2007 22:25, Adrian Bunk wrote:
> >> >
> >> >http://marc.info/?l=linux-kernel&m=116828781409931
> >>
> >> I checked.
> >> Now that kinda sucks, everyone does the same job over and over,
> >> I did a ton of utf8 fixes and they did not get applied. (Granted,
> >> I did not cc Andrew, Linus or trivial).
> >
> >If you had Cc'ed trivial they would now be in Linus' tree...
>
> I figured that by now, but does it still matter now?

Depends on whether you want your patches included in 2.6.23... ;-)

> Jan

cu
Adrian

--

"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

2007-05-14 21:15:21

by Jan Engelhardt

[permalink] [raw]

Subject: [PATCH] Convert files to UTF-8 and some cleanups

On May 14 2007 22:41, Adrian Bunk wrote:
>> >> >
>> >> >http://marc.info/?l=linux-kernel&m=116828781409931
>> >>
>> >> I did a ton of utf8 fixes and they did not get applied. (Granted,
>> >> I did not cc Andrew, Linus or trivial).
>> >
>> >If you had Cc'ed trivial they would now be in Linus' tree...
>>
>> I figured that by now, but does it still matter now?
>
>Depends on whether you want your patches included in 2.6.23... ;-)

Meh :p
Rebased to tonight's trivial.git and attached (yes, it's better than
inline), please consider to apply.

Jan
--

Attachments:

utf8spellos.diff.bz2 (29.14 kB)

2007-05-14 22:22:23

by Stefan Richter

[permalink] [raw]

Subject: Re: [PATCH] Convert files to UTF-8 and some cleanups

2007-05-15 08:08:23

by Jan Engelhardt

[permalink] [raw]

Subject: Re: [PATCH] Convert files to UTF-8 and some cleanups

On May 15 2007 00:20, Stefan Richter wrote:
>
>You forgot this in the plain-text part of your mail:

Well thanks for pasting :)

>
>[quote]
>
> * Convert files to UTF-8.
>
> * Also correct some people's names
> (one example is Eißfeldt, which was found in a source file.
> Given that the author used an ß at all in a source file
> indicates that the real name has in fact a 'ß' and not an 'ss',
> which is commonly used as a substitute for 'ß' when limited to
> 7bit.)
>
> * Correct town names (Goettingen -> Göttingen)
>
> * Update Eberhard Mönkeberg's address (http://lkml.org/lkml/2007/1/8/313)
>
>[...]

Jan
--

2007-05-15 17:04:38

by Pekka Pietikäinen

[permalink] [raw]

Subject: Re: [PATCH] Pekka Pietikainen goes UTF-8

Since everyone else is doing it, why not me as well. Looks like I had
a nice retro 7-bit a-umlaut in the tree from ten years ago too!
(That was back when ISO 8859-1 didn't work universally...).
I hope my mailer is setup correctly for this,
http://www.ee.oulu.fi/~pp/pp-goes-utf8.patch has the same in
case something in the middle munges up the utf-8.

Signed-off-by: Pekka Pietikäinen <[email protected]>

--- linux-2.6.21/arch/m68k/sun3/config.c~ 2007-05-15 19:01:45.000000000 +0300
+++ linux-2.6.21/arch/m68k/sun3/config.c 2007-05-15 19:01:45.000000000 +0300
@@ -1,7 +1,7 @@
/*
* linux/arch/m68k/sun3/config.c
*
- * Copyright (C) 1996,1997 Pekka Pietik{inen
+ * Copyright (C) 1996,1997 Pekka Pietikäinen
*
* This file is subject to the terms and conditions of the GNU General Public
* License. See the file COPYING in the main directory of this archive
--- linux-2.6.21/drivers/media/video/msp3400-driver.c~ 2007-05-15 19:04:10.000000000 +0300
+++ linux-2.6.21/drivers/media/video/msp3400-driver.c 2007-05-15 19:04:10.000000000 +0300
@@ -22,7 +22,7 @@
*
* NICAM (B/G, L , used in UK, Scandinavia, Spain and France)
* should work, with autodetect. Support for NICAM was added by
- * Pekka Pietikainen <[email protected]>
+ * Pekka Pietikäinen <[email protected]>
*
* TODO:
* - better SAT support
--- linux-2.6.21/drivers/net/wireless/bcm43xx/bcm43xx_dma.c~ 2007-05-15 19:05:36.000000000 +0300
+++ linux-2.6.21/drivers/net/wireless/bcm43xx/bcm43xx_dma.c 2007-05-15 19:05:36.000000000 +0300
@@ -8,7 +8,7 @@

Some code in this file is derived from the b44.c driver
Copyright (C) 2002 David S. Miller
- Copyright (C) Pekka Pietikainen
+ Copyright (C) Pekka Pietikäinen

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
--- linux-2.6.21/drivers/net/b44.c~ 2007-05-15 19:18:39.000000000 +0300
+++ linux-2.6.21/drivers/net/b44.c 2007-05-15 19:18:24.000000000 +0300
@@ -1,7 +1,7 @@
/* b44.c: Broadcom 4400 device driver.
*
* Copyright (C) 2002 David S. Miller ([email protected])
- * Fixed by Pekka Pietikainen ([email protected])
+ * Copyright (C) 2004 Pekka Pietikäinen ([email protected])
* Copyright (C) 2006 Broadcom Corporation.
*
* Distribute under GPL.
@@ -86,7 +86,7 @@
static char version[] __devinitdata =
DRV_MODULE_NAME ".c:v" DRV_MODULE_VERSION " (" DRV_MODULE_RELDATE ")\n";

-MODULE_AUTHOR("Florian Schirmer, Pekka Pietikainen, David S. Miller");
+MODULE_AUTHOR("Florian Schirmer, Pekka Pietikäinen, David S. Miller");
MODULE_DESCRIPTION("Broadcom 4400 10/100 PCI ethernet driver");
MODULE_LICENSE("GPL");
MODULE_VERSION(DRV_MODULE_VERSION);

2007-05-16 06:07:16

by H. Peter Anvin

[permalink] [raw]

Subject: Re: [PATCH] Pekka Pietikainen goes UTF-8

Pekka Pietikainen wrote:
> Since everyone else is doing it, why not me as well. Looks like I had
> a nice retro 7-bit a-umlaut in the tree from ten years ago too!
> (That was back when ISO 8859-1 didn't work universally...).
> I hope my mailer is setup correctly for this,
> http://www.ee.oulu.fi/~pp/pp-goes-utf8.patch has the same in
> case something in the middle munges up the utf-8.

Your message came through tagged as iso-8859-1.

-hpa

2007-05-16 07:00:09

by Jan Engelhardt

[permalink] [raw]

Subject: Re: [PATCH] Pekka Pietikainen goes UTF-8

On May 15 2007 23:05, H. Peter Anvin wrote:
>Pekka Pietikainen wrote:
>> Since everyone else is doing it, why not me as well. Looks like I had
>> a nice retro 7-bit a-umlaut in the tree from ten years ago too!
>> (That was back when ISO 8859-1 didn't work universally...).
>> I hope my mailer is setup correctly for this,
>> http://www.ee.oulu.fi/~pp/pp-goes-utf8.patch has the same in
>> case something in the middle munges up the utf-8.
>
>Your message came through tagged as iso-8859-1.

Which perhaps is best, because UTF-8 is 'valid' ISO-8859-1 afaict,
while the reverse is not true.

Jan
--

2007-05-16 15:14:31

by H. Peter Anvin

[permalink] [raw]

Subject: Re: [PATCH] Pekka Pietikainen goes UTF-8

Jan Engelhardt wrote:
> On May 15 2007 23:05, H. Peter Anvin wrote:
>> Pekka Pietikainen wrote:
>>> Since everyone else is doing it, why not me as well. Looks like I had
>>> a nice retro 7-bit a-umlaut in the tree from ten years ago too!
>>> (That was back when ISO 8859-1 didn't work universally...).
>>> I hope my mailer is setup correctly for this,
>>> http://www.ee.oulu.fi/~pp/pp-goes-utf8.patch has the same in
>>> case something in the middle munges up the utf-8.
>> Your message came through tagged as iso-8859-1.
>
> Which perhaps is best, because UTF-8 is 'valid' ISO-8859-1 afaict,
> while the reverse is not true.

NAK. The vast majority of UTF-8 is *not* valid ISO-8859-1. These kinds
of patches really should be sent as binary attachments (although not
necessarily *compressed* such, and 8-bit encoding would be better than
base64, but that seems to be a lot to ask from an MUA.)

-hpa

2007-05-16 16:47:24

by Pekka Pietikäinen

[permalink] [raw]

Subject: Re: [PATCH] Pekka Pietikainen goes UTF-8 (try #2)

On Tue, May 15, 2007 at 11:05:34PM -0700, H. Peter Anvin wrote:
> Pekka Pietikainen wrote:
> > Since everyone else is doing it, why not me as well. Looks like I had
> > a nice retro 7-bit a-umlaut in the tree from ten years ago too!
> > (That was back when ISO 8859-1 didn't work universally...).
> > I hope my mailer is setup correctly for this,
> > http://www.ee.oulu.fi/~pp/pp-goes-utf8.patch has the same in
> > case something in the middle munges up the utf-8.
>
> Your message came through tagged as iso-8859-1.
I noticed. Reconfigured mutt, tested the change, then sent out the email
from an unrestarted mutt. Shame on me :-( .

I wonder how much hassle these kinds of patches will cause when the UTF-8
chars they introduce start occasionally showing up in the diff -u context.
Somehow I think the world might not be ready yet (maybe in 5 years or so
I'll dare start using umlauts in email headers :D ). Oh well :-)

Since I'm polluting the list already, might as well take the opportunity
to resubmit :)

Signed-off-by: Pekka Pietikäinen <[email protected]>

Attachments:

(No filename) (1.07 kB)
pp-goes-utf8.patch (2.26 kB)
Download all attachments

2007-05-16 17:25:26

by Pekka Pietikäinen

[permalink] [raw]

Subject: Re: [PATCH] Pekka Pietikainen goes UTF-8 (try #2)

On Wed, May 16, 2007 at 07:38:18PM +0300, Pekka Pietikainen wrote:
> Since I'm polluting the list already, might as well take the opportunity
> to resubmit :)
Blah, I think it now thought the UTF-8 in the patch was actually
latin-1 and then helpfully re-encoded into UTF-8.

I give up :-(

2007-05-16 17:53:36

by Björn Steinbrink

[permalink] [raw]

Subject: Re: [PATCH] Pekka Pietikainen goes UTF-8 (try #2)

On 2007.05.16 19:38:18 +0300, Pekka Pietikainen wrote:
> On Tue, May 15, 2007 at 11:05:34PM -0700, H. Peter Anvin wrote:
> > Pekka Pietikainen wrote:
> > > Since everyone else is doing it, why not me as well. Looks like I had
> > > a nice retro 7-bit a-umlaut in the tree from ten years ago too!
> > > (That was back when ISO 8859-1 didn't work universally...).
> > > I hope my mailer is setup correctly for this,
> > > http://www.ee.oulu.fi/~pp/pp-goes-utf8.patch has the same in
> > > case something in the middle munges up the utf-8.
> >
> > Your message came through tagged as iso-8859-1.
> I noticed. Reconfigured mutt, tested the change, then sent out the email
> >from an unrestarted mutt. Shame on me :-( .
>
> I wonder how much hassle these kinds of patches will cause when the UTF-8
> chars they introduce start occasionally showing up in the diff -u context.
> Somehow I think the world might not be ready yet (maybe in 5 years or so
> I'll dare start using umlauts in email headers :D ). Oh well :-)
>
> Since I'm polluting the list already, might as well take the opportunity
> to resubmit :)

Broken again :/

Since you're using mutt, try manually choosing the mime type and
transfer encoding.
After adding the attachment, select it, press Ctrl+t to change the type
to e.g. application/octet-stream, then press Ctrl+e to change the
transfer encoding back to 8bit. Mutt is quite friendly in that regard :)

HTH
Bj?rn