Description

Internally, programs store individual Unicode characters as 16-bit values. However, any external manifestation of textual information, in files or at the interface between programs, uses the machine-independent, byte-stream encoding called UTF.

UTF is designed so the 7-bit ASCII set (values hexadecimal 00 to 7F), appear only as themselves in the encoding. Characters with values above 7F appear as sequences of two or more bytes with values only from 80 to FF.

The UTF encoding of the Unicode Standard is backward compatible with ASCII: Inferno programs handle ASCII text, as well as uninterpreted byte streams, without special arrangement. However, programs that perform semantic processing on characters must convert from UTF to Unicode in order to work properly with non-ASCII input. Normally, all necessary conversions are done by the Limbo compiler and execution environment, but sometimes more is necessary, such as when a program receives UTF input one byte at a time; see byte2char for routines to handle such processing.

A Unicode character x (in binary format) is converted to a multibyte UTF sequence as follows:

  01. x in [00000000.0bbbbbbb] \(-> 0bbbbbbb
  10. x in [00000bbb.bbbbbbbb] \(-> 110bbbbb, 10bbbbbb
  11. x in [bbbbbbbb.bbbbbbbb] \(-> 1110bbbb, 10bbbbbb, 10bbbbbb

In the inverse mapping, any sequence except those described above is incorrect and is converted to the Unicode value of hexadecimal 0080.

See Also