UTF(6)                                                     UTF(6)

     NAME
          UTF, Unicode, ASCII - character set and format

     DESCRIPTION
          The Inferno character set and representation are based on
          the Unicode Standard and on the ISO multibyte UTF-8 encoding
          (Universal Character Set Transformation Format, 8 bits
          wide).  The Unicode Standard represents its characters in 16
          bits; UTF-8 represents such values in an 8-bit byte stream.
          Throughout this manual, UTF-8 is shortened to UTF.

          Internally, programs store individual Unicode characters as
          16-bit values.  However, any external manifestation of tex-
          tual information, in files or at the interface between pro-
          grams, uses the machine-independent, byte-stream encoding
          called UTF.

          UTF is designed so the 7-bit ASCII set (values hexadecimal
          00 to 7F), appear only as themselves in the encoding.  Char-
          acters with values above 7F appear as sequences of two or
          more bytes with values only from 80 to FF.

          The UTF encoding of the Unicode Standard is backward compat-
          ible with ASCII: Inferno programs handle ASCII text, as well
          as uninterpreted byte streams, without special arrangement.
          However, programs that perform semantic processing on char-
          acters must convert from UTF to Unicode in order to work
          properly with non-ASCII input.  Normally, all necessary con-
          versions are done by the Limbo compiler and execution envi-
          ronment, but sometimes more is necessary, such as when a
          program receives UTF input one byte at a time; see sys-
          byte2char(2) for routines to handle such processing.

          Letting numbers be binary, a Unicode character x is con-
          verted to a multibyte UTF sequence as follows:
          01.   x in [00000000.0bbbbbbb] -> 0bbbbbbb
          10.   x in [00000bbb.bbbbbbbb] -> 110bbbbb, 10bbbbbb
          11.   x in [bbbbbbbb.bbbbbbbb] -> 1110bbbb, 10bbbbbb, 10bbbbbb

          Conversion 01 provides a one-byte sequence that spans the
          ASCII character set in a compatible way.  Conversions 10 and
          11 represent higher-valued characters as sequences of two or
          three bytes with the high bit set.  Inferno does not support
          the 4-, 5-, and 6-byte sequences proposed by X-Open.  When
          there are multiple ways to encode a value, for example rune
          0, the shortest encoding is used.

          In the inverse mapping, any sequence except those described
          above is incorrect and is converted to the Unicode value of
          hexadecimal 0080.

     Page 1                       Plan 9            (printed 11/18/24)

     UTF(6)                                                     UTF(6)

     SEE ALSO
          sys-byte2char(2), The Unicode Standard.

     Page 2                       Plan 9            (printed 11/18/24)