From:   "Weber, Olaf (HPC Data Management & Storage)" <olaf.weber@hpe.com>
To:     Theodore Ts'o <tytso@mit.edu>,
        Gabriel Krisman Bertazi <krisman@collabora.com>
CC:     "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>,
        "sfrench@samba.org" <sfrench@samba.org>,
        "darrick.wong@oracle.com" <darrick.wong@oracle.com>,
        "jlayton@kernel.org" <jlayton@kernel.org>,
        "bfields@fieldses.org" <bfields@fieldses.org>,
        "paulus@samba.org" <paulus@samba.org>,
        "linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
        Olaf Weber <olaf@sgi.com>,
        "Gabriel Krisman Bertazi" <krisman@collabora.co.uk>
Subject: RE: [PATCH RFC v6 04/11] unicode: reduce the size of utf8data[]
Thread-Topic: [PATCH RFC v6 04/11] unicode: reduce the size of utf8data[]
Thread-Index: AQHU3ckozABztBO3pUm6I27ECEeW/aYvqUwAgAKevwA=
Date:   Mon, 8 Apr 2019 12:02:49 +0000
Message-ID: <DF4PR8401MB13058AE34765190922714B2F852C0@DF4PR8401MB1305.NAMPRD84.PROD.OUTLOOK.COM>
References: <20190318202745.5200-1-krisman@collabora.com>
 <20190318202745.5200-5-krisman@collabora.com>
 <20190406195342.GA18897@mit.edu>
In-Reply-To: <20190406195342.GA18897@mit.edu>
Accept-Language: en-US
Content-Language: en-US
received-spf: None (protection.outlook.com: hpe.com does not designate
 permitted sender hosts)
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 8BIT
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-Network-Message-Id: c23b5529-5bf5-4256-3cce-08d6bc1a1ef5
X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Apr 2019 12:02:49.3073
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 105b2061-b669-4b31-92ac-24d304d195dc
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DF4PR8401MB1161
X-OriginatorOrg: hpe.com
X-HPE-SCL: -1
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2019-04-08_04:,,
 signatures=0
X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501
 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0
 clxscore=1011 lowpriorityscore=0 mlxscore=0 impostorscore=0
 mlxlogscore=866 adultscore=0 classifier=spam adjust=0 reason=mlx
 scancount=1 engine=8.0.1-1810050000 definitions=main-1904080104
Sender: linux-ext4-owner@vger.kernel.org
Precedence: bulk
List-ID: <linux-ext4.vger.kernel.org>
X-Mailing-List: linux-ext4@vger.kernel.org

From: Theodore Ts'o
> On Mon, Mar 18, 2019 at 04:27:38PM -0400, Gabriel Krisman Bertazi wrote:
> > From: Olaf Weber <olaf@sgi.com>
> >
> > Remove the Hangul decompositions from the utf8data trie, and do
> > algorithmic decomposition to calculate them on the fly. To store
> > the decomposition the caller of utf8lookup()/utf8nlookup() must
> > provide a 12-byte buffer, which is used to synthesize a leaf with
> > the decomposition. Trie size is reduced from 245kB to 90kB.
> 
> I'm seeing sizes much smaller; the actual utf8data[] array is 63,584.
> And size utf8-norm.o reports:
> 
>    text	   data	    bss	    dec	    hex	filename
>   68752	     96	      0	  68848	  10cf0	fs/unicode/utf8-norm.o
> 
> Were you measuring the size of the utf8-norm.o file?  That will vary
> in size depending on whether debugging symbols are enabled, etc.
> 
>    		     	     	       	       - Ted

These numbers came from the size of the array reported in utf8data.h,
and were correct for the NFKDI + NFKDICF normalizations for Unicode 9.
The switch to NFDI + NFDICF reduced the size, and it looks like the commit
message was not updated to account for this.

Olaf