From: "=?UTF-8?Q?Alfredo_Fern=c3=a1ndez_D=c3=adaz?=" <lswitcher-dev@2rosenthals.com>
Subject: Re: [lswitcher-dev] lSwitcher-2-93-0-RC_6.wpi
To: lSwitcher Developers Mailing List <lswitcher-dev@2rosenthals.com>
References: <list-2273425@2rosenthals.com> <list-2273485@2rosenthals.com>
 <list-2273573@2rosenthals.com> <list-2273894@2rosenthals.com>
 <list-2274130@2rosenthals.com> <list-2280228@2rosenthals.com>
 <list-2280344@2rosenthals.com>
Message-ID: <6119AC52.8000709@yahoo.es>
Date: Mon, 16 Aug 2021 01:07:46 +0100
User-Agent: Mozilla/5.0 (OS/2; Warp 4.5; rv:38.0) Gecko/20100101 Firefox/38.0
 SeaMonkey/2.35
MIME-Version: 1.0
In-Reply-To: <list-2280344@2rosenthals.com>
Content-Type: multipart/mixed;
 boundary="------------030401050502020209060208"
Content-Length: 6897

This is a multi-part message in MIME format.
--------------030401050502020209060208
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi Gregg,

On 2021/08/15 21:52, Gregg Young wrote:
> Hi Lewis
>
> This is a warpin bug. REQUIRES="Ulrich Möller\XWorkplace\Kernel\1\0\1" isn't meant
> to be user  readable it is an internal check.

It's both since WarpIN has a 'database' mode that can show its contents to 
users (that should match the REQUIRES). However, I don't think users not being 
able to read a few characters here and there is a problem. The database not 
being built consistently is.

>
> I think you will also see problems on a Russian system with PACKAGEID="Ulrich Möller\XWorkplace\Kernel\1\0\1"
> installed post 1.0.24. If this is done with codepage 1208 or no codepage. the database will contain
> "Ulrich M?ller\XWorkplace\Kernel\1\0\1". If you have a REQUIRES="Ulrich Möller\XWorkplace\Kernel\1\0\1"
> codepage 1208 it will probably work but if you have a wis with this that is codepage 850 it will fail since
> the ö will be present.
> The ö isn't present in codepage 866. Only ASCII character (0-127) are (probably) guaranteed  between codepages.

Yes, 0-127 are English + minimal typesetting characters common to all codepages.

Regarding database contents...

My main system database contains "55 6C 72 69 63 68 20 4D C3 B6 6C 6C 65 72" 
which is valid UTF-8 for "Ulrich Möller", which means all code worked 
correctly to build it.

On the Russian system, the database contains "55 6C 72 69 63 68 20 4D D0 A4 6C 
6C 65 72", which is not valid UTF-8. (D0A4 seems to be a Hangul character in 
UTF8, BTW.)

So apparently the problem lies in translation to Unicode on non-CP850 systems 
(because the same WPIs are processed on all systems but we have different 
results), and this badly translated stuff being stored afterwards in the database.

> What is needed is for warpin to convert these "internal use" strings to codepage 850 use them and then
> convert the rest to codepage 866.

Not really. A REQUIRES string must be converted to UTF8 if CODEPAGE != 1208, 
and then it can be compared directly to whatever is in the database, because 
that is supposed to be UTF8. The problem is, it is mangled on the Russian system.

> The other problem is with wises with no codepage (most if not all of which are codepage 850).
> These fail for REQUIRES="Ulrich Möller\XWorkplace\Kernel\1\0\1" on Russian systems because they are now
> read out as codepage 866 (process default).

Are we sure of this? (Reminder: docs say when no CODEPAGE= is found in the 
WIS, 850 is assumed.)

> This case requires that the "internal use" be read first in
> codepage 850 and used before the codepage 866 (default) read. This can also be fixed by assuming they are
> codepage 850 not the process codepage.
>
> OK some questions about this. What version of iconv are you using I have found several?
> What are the exact steps to build the wis? I assume you need to reconvert any time you edit the file unless
> you use a UTF-8 enabled editor (are there any).

IIRC iconv has always had problems. I have put together a little REXX script 
to use Alex Taylor's ULS to convert text files between codepages, find it 
attached. He also has an UTF8 capable, QT-based editor somewhere, but I can't 
recall the name right now.

Thank you,
AFD.


--------------030401050502020209060208
Content-Type: text/plain;
 name="RxTxtCnv.cmd"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
 filename="RxTxtCnv.cmd"

LyogQ29udmVydCB0ZXh0IGZpbGVzIGJldHdlZW4gdHdvIGNvZGVwYWdlcywgbGluZSBieSBs
aW5lIC0+IFRoeCBBbGV4IFRheWxvciEhICovDQoNCmlmIFJ4RnVuY1F1ZXJ5KCdTeXNMb2Fk
RnVuY3MnKSB0aGVuIGRvDQogIGNhbGwgUnhGdW5jQWRkICdTeXNMb2FkRnVuY3MnLCAnUmV4
eFV0aWwnLCAnU3lzTG9hZEZ1bmNzJw0KICBjYWxsIFN5c0xvYWRGdW5jcw0KIGVuZA0KaWYg
UnhGdW5jUXVlcnkoJ1VMU0xvYWRGdW5jcycpIHRoZW4gZG8NCiAgY2FsbCBSeEZ1bmNBZGQg
J1VMU0xvYWRGdW5jcycsICdSWFVMUycsICdVTFNMb2FkRnVuY3MnDQogIGNhbGwgVUxTTG9h
ZEZ1bmNzDQogIGlmIChSeEZ1bmNRdWVyeSgnVUxTQ29udmVydENvZGVwYWdlJykgPD4gMCkg
dGhlbiBkbw0KICAgIGNhbGwgbGluZW91dCBTdGRFcnIsJ0ZhdGFsIGVycm9yOiBSeFVMUyBm
dW5jdGlvbnMgbm90IGxvYWRlZCcNCiAgICBleGl0IDENCiAgIGVuZA0KIGVuZA0KDQpsb2Nh
bENQID0gU3lzUXVlcnlQcm9jZXNzQ29kZVBhZ2UoKQ0KDQpwYXJzZSBhcmcgbXlBcmdzDQpt
eUFyZ3MgPSBzdHJpcChteUFyZ3MpDQppZiBteUFyZ3MgPSAnJyB0aGVuIGRvDQogIGNhbGwg
bGluZW91dCBTdGRFcnIsJ1JFWFggVHh0IENvbnZlcnRlciBUb29sIHYwLjEgYnkgJ3x8VUxT
Q29udmVydENvZGVwYWdlKCJBbGZyZWRvIEZlcm6gbmRleiBEoWF6Iiw4NTAsbG9jYWxDUCl8
fCcgLSBVc2FnZTonDQogIGNhbGwgbGluZW91dCBTdGRFcnIsJycNCiAgY2FsbCBsaW5lb3V0
IFN0ZEVyciwnUnhUeHRDbnYgPGZpbGU+IFtvdXRwdXQgW0lucHV0IGNvZGVwYWdlIG5vLiBb
T3V0cHV0IGNvZGVwYWdlIG5vLl1dXScNCiAgY2FsbCBsaW5lb3V0IFN0ZEVyciwnJw0KICBj
YWxsIGxpbmVvdXQgU3RkRXJyLCc8PiA9IG1hbmRhdG9yeSBwYXJhbWV0ZXIsIHN1YnN0aXR1
dGUgdmFsdWUnDQogIGNhbGwgbGluZW91dCBTdGRFcnIsJ1tdID0gb3B0aW9uYWwgcGFyYW1l
dGVyJw0KICBjYWxsIGxpbmVvdXQgU3RkRXJyLCdEZWZhdWx0czogJw0KICBjYWxsIGxpbmVv
dXQgU3RkRXJyLCcgIG91dHB1dCA9IHN0ZG91dCcNCiAgY2FsbCBsaW5lb3V0IFN0ZEVyciwn
ICBpbnB1dCBDUCA9IDEyNTInDQogIGNhbGwgbGluZW91dCBTdGRFcnIsJyAgb3V0cHV0IENQ
ID0gYWN0aXZlIGNvZGVwYWdlICgnfHxsb2NhbENQfHwnIGhlcmUgYW5kIG5vdyknDQogIGNh
bGwgbGluZW91dCBTdGRFcnIsJycNCiAgY2FsbCBsaW5lb3V0IFN0ZEVyciwnUmVtYXJrczog
cGFyYW1ldGVycyBhcmUgcmVjb2duaXplZCBieSBvcmRlciAtPiBzdGRvdXQgY2FuIGJlIHNw
ZWNpZmllZCcNCiAgY2FsbCBsaW5lb3V0IFN0ZEVyciwnYXMgIi0iIHRvIGFsbG93IGNoYW5n
aW5nIGRlZmF1bHQgaW5wdXQvb3V0cHV0IGNvZGVwYWdlIG51bWJlcnMnDQogIGNhbGwgbGlu
ZW91dCBTdGRFcnIsJycNCiAgY2FsbCBsaW5lb3V0IFN0ZEVyciwnVXN1YWxseSB1c2VmdWwg
Y29kZXBhZ2UgbnVtYmVyczonDQogIGNhbGwgbGluZW91dCBTdGRFcnIsJ1dpbmRvd3MgV2Vz
dCBFdXJvcGU6IDEyNTInDQogIGNhbGwgbGluZW91dCBTdGRFcnIsJ0lCTSBmb3IgV2VzdCBF
dXJvcGU6ICA4NTAnDQogIGNhbGwgbGluZW91dCBTdGRFcnIsJ1VURi04OiAgICAgICAgICAg
ICAgIDEyMDgnDQogIGNhbGwgbGluZW91dCBTdGRFcnIsJycNCiAgZXhpdCAxDQogZW5kDQpp
ZiBsZWZ0KG15QXJncywxKSA9ICciJyB0aGVuDQogIHBhcnNlIHZhciBteUFyZ3MgJyInIGlu
RmlsZSAnIicgTXlBcmdzDQplbHNlDQogIHBhcnNlIHZhciBteUFyZ3MgaW5GaWxlIE15QXJn
cw0KTXlBcmdzID0gc3RyaXAoTXlBcmdzKQ0KaWYgbGVmdChteUFyZ3MsMSkgPSAnIicgdGhl
bg0KICBwYXJzZSB2YXIgbXlBcmdzICciJyBvdXRGaWxlICciJyBGcm9tQ1AgVG9DUA0KZWxz
ZQ0KICBwYXJzZSB2YXIgbXlBcmdzIG91dEZpbGUgRnJvbUNQIFRvQ1ANCmlmIChvdXRGaWxl
ID0gJycpIHwgKG91dEZpbGUgPSAnLScpIHRoZW4NCiAgb3V0RmlsZSA9IHN0ZG91dA0KaWYg
RnJvbUNQID0gJycgdGhlbg0KICBGcm9tQ1AgPSAnMTI1MicNCmlmIFRvQ1AgPSAnJyB0aGVu
DQogIFRvQ1AgPSBsb2NhbENQDQoNCmlmIEZyb21DUCA9IFRvQ1AgdGhlbiBkbw0KICBjYWxs
IGxpbmVvdXQgU3RkRXJyLCdFcnJvcjogaW5wdXQgYW5kIG91dHB1dCBDUCBhcmUgdGhlIHNh
bWUnDQogIGV4aXQgMQ0KIGVuZA0KDQpkbyB3aGlsZSBsaW5lcyhpbkZpbGUpPiAwDQogIG5l
dyA9IGxpbmVpbihpbkZpbGUpDQogIGlmIG5ldyA8PiAnJyB0aGVuIGRvDQogICAgbmV3ID0g
VUxTQ29udmVydENvZGVwYWdlKG5ldyxGcm9tQ1AsVG9DUCkNCiAgICBjYWxsIGxpbmVvdXQg
b3V0RmlsZSxuZXcNCiAgIGVuZA0KICBlbHNlDQogICAgY2FsbCBsaW5lb3V0IG91dEZpbGUs
IiINCiBlbmQNCmNhbGwgbGluZW91dCBpbkZpbGUNCmNhbGwgbGluZW91dCBvdXRGaWxlDQoN
CmV4aXQgMA0K
--------------030401050502020209060208--