UTF-8

int unittest_string_utf8( void )

Test <escape_char>.

char32_t maxchar_utf8( void )

Returns the maximum character value (unicode code point) which can be encoded into utf-8.  The minumum unicode code point is 0.  The returned value is 0x10FFFF.

uint8_t maxsize_utf8( void )

Returns the maximum size in bytes of an utf-8 encoded multibyte sequence.

bool isvalidfirstbyte_utf8( const  uint8_t  firstbyte )

Returns true if the byte is a legal first byte of an utf8 encoded multibyte sequence.

bool isfirstbyte_utf8( const  uint8_t  firstbyte )

Returns true if this byte is a possible first (start) byte of an utf-8 encoded multibyte sequence.  This function assumes correct encoding therefore it is possible that isfirstbyte_utf8 returns true and isvalidfirstbyte_utf8 returns false.

uint8_t sizefromfirstbyte_utf8( const  uint8_t  firstbyte )

Returns the size in bytes of a correct encoded mb-sequence by means of the value of its first byte.  The number of bytes is calculated from firstbyte - the first byte of the encoded byte sequence.  The returned values are in the range 0..4 (0..<maxsize_utf8>).  A return value between 1 and 4 describes a valid first byte.  A value of 0 indicates that firstbyte is not a valid first byte of an utf8 encoded byte sequence.

uint8_t sizechar_utf8( char32_t  uchar )

Returns the size in bytes of uchar as encoded mb-sequence.  The returned values are in the range 1..<maxchar_utf8>.  If uchar is bigger than maxchar_utf8 no error is reported and the function returns maxsize_utf8.

size_t length_utf8( const  uint8_t  * strstart, const  uint8_t  * strend )

Returns number of UTF-8 characters encoded in string buffer.  The first byte of a multibyte sequence determines its size.  This function assumes that utf8 encodings are correct and does not check the encoding of bytes following the first.  Illegal encoded start bytes are not counted but skipped.  The last multibyte sequence is counted as one character even if one or more bytes are missing.

Parameter

uint8_t decodechar_utf8(     const  uint8_t  strstart[/*maxsize_utf8() or big enough*/],     /*out*/char32_t  * uchar )

Decodes utf-8 encoded bytes beginning from strstart and returns character in uchar.  The string must be big enough but needs never larger as maxsize_utf8.  Use sizefromfirstbyte_utf8 to determine the size if strstart contains less then maxsize_utf8 bytes.

The number of decoded bytes is returned.

A return value of 0 indicates an invalid first byte of the multibyte sequence (EILSEQ).  The function assumes that all other bytes except the first are encoded correctly.  Use utf8validator_t to make sure that a string contains only a valid encoded utf8 string.

Example

uint8_t * str    = &strbuffer[0] ;
uint8_t * strend = strbuffer + sizeof(strbuffer) ;
while (str < strend) {
   if (strend-str < maxsize_utf8()) {
      if (sizefromfirstbyte_utf8(str[0]) > (strend-str)) {
         ...not enough data for last character...
         break ;
      }
   }
   char32_t uchar ;
   uint8_t  len = decodechar_utf8(str, &uchar) ;
   str += len ;
   ... do something with uchar ...
}

uint8_t encodechar_utf8( size_t  strsize, /*out*/uint8_t  strstart[strsize], char32_t  uchar )

Encodes uchar into UTF-8 enocoded string of size strsize starting at strstart.  The number of written bytes are returned.  The maximum return value is maxsize_utf8.  A return value of 0 indicates an error.  Either uchar is greater then maxchar_utf8 or strsize is not big enough.

uint8_t skipchar_utf8( const  uint8_t  strstart[/*maxsize_utf8() or big enough*/] )

Skips the next utf-8 encoded character.  The encoded byte sequence is not checked for correctness.  The number of skipped bytes is returned.  The maximum return value is maxsize_utf8.  A return value of 0 indicates an error, i.e. the first byte of the multibyte sequence is invalid (EILSEQ).

void init_utf8validator( /*out*/utf8validator_t  * utf8validator )

Same as assigning utf8validator_INIT.

int free_utf8validator( utf8validator_t  * utf8validator )

Clear data members and checks that there is no internal prefix stored.

Returns

uint8_t sizeprefix_utf8validator( const  utf8validator_t  * utf8validator )

Returns a value != 0 if the last multibyte sequence was not fully contained in the last validated buffer.

int validate_utf8validator( utf8validator_t  * utf8validator, size_t  size, const  uint8_t  data[size], /*err*/size_t  * erroffset )

Validates a data block of length size in bytes.  If the last multibyte sequence is not fully contained in the data block but a valid prefix it is stored internally as prefix.  If this function is called another time the internal prefix is prepended to the data block.  If an error occurs EILSEQ is returned the parameter offset is set to the offset of the byte which is not encoded correctly.

int nextutf8_stringstream( struct  stringstream_t  * strstream, /*out*/char32_t  * uchar )

Reads next utf-8 encoded character from strstream.  The character is returned as unicode character (codepoint) in uchar.  The next pointer of strstream is incremented with the number of decoded bytes.

Returns

int peekutf8_stringstream( const struct  stringstream_t  * strstream, /*out*/char32_t  * uchar )

Same as nextutf8_stringstream except the strstream is not changed.  Calling this function more than once returns always the same value in uchar.

int skiputf8_stringstream( struct  stringstream_t  * strstream )

Skips next utf-8 encoded character from strstream.  The next pointer of strstream is incremented with the size of the next character.

Returns

void skipillegalutf8_strstream( struct  stringstream_t  * strstream )

Skips bytes until end of stream or the begin of a valid utf-8 encoding is found.

const uint8_t * findutf8_stringstream( const struct  stringstream_t  * strstream, char32_t  uchar )

Searches for unicode character in utf8 encoded stringstream.  The returned value points to the start addr of the multibyte sequence in the unread buffer.  A return value of 0 inidcates that strstream does not contain the multibyte sequence or that uchar is bigger than maxchar_utf8 and therefore invalid.

Implements utf8.maxchar_utf8.

Implements utf8.isfirstbyte_utf8.

Implements utf8.isvalidfirstbyte_utf8.

Implements utf8.maxsize_utf8.

Implements utf8.sizefromfirstbyte_utf8.

Implements utf8.sizechar_utf8.

Implements utf8.skipchar_utf8.