Decoding UTF-8. Part VII: Validation

In part VI of the UTF-8 decoding series we saw how a simple non-validating decoder might look like, but we emphasized the importance of validation. It is important to remember that in general we cannot assume that a stream of bytes represents valid UTF-8 encoded text. We need to be able to recognize any invalid UTF-8 sequence and deal with it.

To validate a UTF-8 string we have to perform the following three kinds of checks:

The byte sequence is well-formed.

Decoding results in a valid Unicode code point

The code point is encoded in the minimal number of bytes (no overlong encoding).

Let’s look at them in some detail.

Validity of Bytes in the Sequence

As we have seen in the Part I, a valid UTF-8 encoding of a Unicode code point consists of a leading byte followed by a number of continuation bytes.

We start the check for validity of a byte sequence by looking at the lead byte. A valid lead byte has to start with one of the following bit fields:

0, in which case the lead byte alone encodes a code point.

110, which starts a two-byte sequence.

1110, which starts a three-byte sequence

11110, which starts a four-byte sequence.

Lead bytes that start with any other bit combination are invalid.

Once we establish the validity of the lead byte and compute the expected sequence length 1, we can perform the next important check: ensure the expected number of valid continuation bytes. A valid continuation byte starts with bits 10 and therefore falls in the range 0x80-0xBF.

Depending on the value of the lead byte, we expect between 0 and 3 continuation bytes to follow. If there are fewer, it means we have an incomplete sequence; if there are more, we have detected unexpected continuation bytes.

Validity of Decoded Code Points

Some well-formed UTF-8 sequences are decoded into invalid Unicode code points:

Values greater than U+10FFFF.

UTF-16 surrogates.

Unicode 2.0 and later support code points in the range U+0000-U+10FFFF. A well-formed UTF-8 sequence can be us…