UTF-8 and QWK
From Digital Man
@1:103/705 to All
on Sunday, August 18, 2019 16:11:00
The QWK specifications, including QWKE, provide no mechanism to specify the character set used in the message body text or the header header fields (e.g. to/from/subject). It's always been an unstated assumption that the message header fields and message body text contained either per US-ASCII or CP437 (IBM
extended-ASCII) characters. If/how-well CP437 is actually supported in offline
readers, especially in header fields, is not really well known or documented. CP437 is common enough in BBS message body texts that I suspect support is pretty universal.
The Synchronet QWK HEADERS.DAT file includes header fields that can indicate a UTF-8 encoded message body text (e.g. FTN-CHRS: UTF-8 4), but until very recently, no method to indicate the use of UTF-8 in header fields.
Also until very recently, Synchronet would include unconverted/transcoded UTF-8
characters in messages placed in QWK/REP packets. Recognizing that this is likely problematic for all existing offline readers, I added a user/QWKnet node
option and a QWKnet hub option (in SCFG) to enable the inclusion of UTF-8 characters. If the user/node/hub does *not* supported UTF-8 (this is the default), then UTF-8 character sequences are converted to ASCII/CP437 equivalents when the packets are created.
So... if you're running a recent build of v3.17c, you'll want to set SCFG->Networks->QWK->Hubs->VERT->Include UTF-8 Characters to "Yes". Also, login
with your QWKnet account (e.g. over telnet) and enable the "Include UTF-8" option at vert.synchro.net.
If/when offline readers support UTF-8 in QWK packets, you'll be able to toggle that option to include UTF-8 in your user-downloaded packets as well.
This isn't that big of a deal right now, but as the prevelence/use of UTF-8 enabled BBS terminals increases and hence the existence of UTF-8 encoded messages becomes more common, it'll potentially be more of an issue. So best to
prepare for the future now. :-)
Synchronet "Real Fact" #14:
SBBSecho was originally written by Allen Christiansen (King Drafus) in 1994. Norco, CA WX: 84.4øF, 46.0% humidity, 15 mph NE wind, 0.00 inches rain/24hrs --- SBBSecho 3.08-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)