Fixing a bug with byte order marks

Recently I’ve been tidying up the subtitles in my local media library. There are two popular file formats for subtitles: SRT (SubRip Subtitle ) and WebVTT ( Web Video Text Tracks ).

I’ve been standardising on WebVTT because it works with the HTML5 element, and I play all my videos through the element embedded in static websites. However, lots of subtitles are only available as SRT, so I wrotea Python function to convert SRT files to WebVTT. The formats looked simple and the conversion seemed straightforward. Famous last words!

When I spot checked the converted subtitles, I noticed a bug in my handling of byte order marks (BOM), and it took several steps to fix.

Failing to look for the UTF‑8 BOM

A byte order mark is a special use of the zero width no-break space character U+FEFF at the beginning of a text file, which tella a program reading the file about how the text is encoded. It depends on the exact sequence of bytes used to encode the character. Here are a few examples:

EF BB BF – the file is UTF‑8 text. UTF‑8 always has the same byte order, so it’s just telling us about the encoding.

FE FF – the file is UTF‑16 text, with big-endian byte order (UTF‑16BE).

FF FE – the file is UTF‑16 text, with little-endian byte order (UTF‑16LE).

00 00 FE FF – the file is UTF‑32 text, with big-endian byte order.

All of my SRT input files were UTF‑8 encoded, and some of them had the UTF‑8 byte order mark, and I wasn’t handling it correctly. For example, suppose I had this input SRT file:

1 00:00:01,001 --> 00:00:10,010 You have grown, Keyne.

2 00:02:00,002 --> 00:20:00,020 Soon you’ll be needing another name.

When I convert to WebVTT, I want to add the WEBVTT header, remove the sequence numbers, and change the timestamp format.

To remove sequence numbers, I was checking if a line was all digits. Because the BOM is on the same line as the first sequence number, the line isn’t all digits, so I didn’t remove it. Instead, I copied the entire line into the middle of the…