Decoding UTF-8. Part VI: Simplistic Non-Validating Decoder

In parts 2-5 we spent some time on determining length of a UTF-8 sequence. Now, we move on to actual decoding. We’ll start not just simple, but simplistic - without validation.

Validation is important. Not every stream of bytes is valid UTF-8, and assuming it is, leads not only to bugs in functionality, but also security problems. A UTF-8 decoder needs to operate under assumption that the input stream may not be valid UTF-8 and ready to abort decoding gracefully and report an error when it detects invalid encoding.

That said, there are some scenarios in which it is OK to skip validation and save some CPU cycles:

The input has already been validated. It is OK to do validation once and then non-validated decoding if the input bytes are immutable.

The input is produced by a trusted source. If you are processing strings that you generate internally - i.e. string literals and various resource files that you know are valid UTF-8, it is OK to skip validation.

Here is a minimal C function that performs UTF‑8 decoding without any validation:

/ A sequence length function we covered in previous posts / int utf8_sequence_length(unsigned char lead_byte);

/ Returns the code point and advances p by the number of bytes consumed. */ unsigned int utf8_decode(const unsigned char **p) { const unsigned char s = p; unsigned int cp;

switch(utf8_sequence_length(s[0])) { case 1: cp = s[0]; • p += 1; break; case 2: cp = ((s[0] & 0x1F) << 6) | (s[1] & 0x3F); • p += 2; break; case 3: cp = ((s[0] & 0x0F) << 12) | ((s[1] & 0x3F) << 6) | (s[2] & 0x3F); • p += 3; break; default: cp = ((s[0] & 0x07) << 18) | ((s[1] & 0x3F) << 12) | ((s[2] & 0x3F) << 6) | (s[3] & 0x3F); • p += 4; break; }

return cp; }

The function does what we described in: Decoding UTF-8. Part I: Manual Decoding.

A one-byte sequence is decoded by taking the low 7 bits from the byte (simply casting the unsigned char to unsigned int).

A two-byte sequence is decoded by laying out the five low bits from the first byte (…