Twenty Years of my Open Source Project

In the fall of 2005, my employer was acquired by another company, and I was transferred to the team that developed a centralized translation memory system. The group had a strong not-invented-here culture, which I did not appreciate at the time1. One example of functionality that, in my opinion, should have been reused rather than rewritten was the handling of UTF‑8 encoded text. I looked around to find some portable C++ open-source libraries to prove my point, and to my astonishment, there was almost nothing. There was only ICU, which started as a Java library and then was ported to some very non-idiomatic C++. I wanted something light, portable, and unintrusive that plays well with the C++ Standard Library.

Beginnings

In the summer of 2006, while my family was away for a few weeks2, I got busy. The primary design goals I came up with were:

• Portability. I wanted it to run on any reasonably C++98 compliant compiler and be OS agnostic.
• Flexibility. It was meant to be easy to use with as many existing C++ string classes as possible and play well with STL and Boost.
• Ease of use. It was going to be easy to install, easy to learn, and surprise-free.
• Safety. It had to catch malformed data by default.

The first version was coded quickly. I was lucky to find Markus Kuhn's UTF-8 decoder capability and stress test file early, and testing against it gave me confidence that I covered the edge cases.

Initially, I had envisioned the library as a Code Project3 article, so the documentation was almost as important to me as the code itself. The article helped me get the first users and the first round of feedback.

Naming the library was confusing. Initially, I called it UTF-8 C++, which landed as utfcpp in most URLs. Then it became more common to refer it as utfcpp, and after I extended it to support UTF-16 related operations, it actually became a better name. In any case, I should have thought about the name harder. The title of the initial Code Project article was excellent, though: UTF-8 with C++ in a Portable Way. People would often search for something like “C++ utf-8 portable” and it would bring them directly to utfcpp.

Soon, I realized I had a little more than an article and decided to set up an open-source project. The choice of the hosting platform was easy - SourceForge was pretty much the only option back then, and it supported Subversion, my source control system of choice at the time4. SourceForge was pretty rough around the edges: setting up builds, tests, releases and the home page for the project was not a pleasant experience. That said, once I did set it up and linked it from my Code Project article, the experience was much better.

Choosing the license did not give me any headaches either. I’ve always disliked restrictive “copyleft” ones such as GPL. On the other hand, I was a fan of The Boost C++ Libraries which I used and advocated at work at the time5, so I picked Boost Software License - Version 1.0 which I found simple and straightforward, while still offering legal protection to the author.

I only vaguely remember the original build and test framework. There was a lot of Perl, which I used mostly as a portable shell to run builds, pack releases and run tests. Unit-tests were present from the very beginning, and I even think I started the project using Test Driven Development6 with simple asserts instead of any “proper” unit-test frameworks.

Growth

Encouraged by the initial reception of the library at Code Project, I decided to submit it to Boost. To my surprise, the process was informal and somewhat confusing. I was supposed to post a proposal to the Boost mailing lists and hope that someone important enough picks it. So I posted the proposal in July 2006, got a few generally favorable reactions, but then nothing but silence. Then, in December, I tried again. This time there was some technical discussion about appropriateness of exceptions as a mechanism for reporting invalid UTF-8 sequences, but when the discussion was over, I was left hanging again. No one rejected my proposal; it was simply ignored. So, I gave up and decided to run the library as a standalone project.

Perhaps it was for the best. I kept chugging along slowly, adding an occasional feature now and then and fixing bugs when they were reported. Sometimes, multiple months would pass without me doing any work on utfcpp, and it never felt bad as long as the library was in good shape and there were no outstanding bugs.

In the beginning, I expected the C++ Standard would eventually include a good way of dealing with Unicode encodings and my library would be retired. Interestingly, it did not happen. C++11 standard included a mechanism called codecvt that at least covered conversion between the encodings, but even that was deprecated in C++17 and removed in C++26. Against all odds, utfcpp remained relevant and relatively popular for two decades.

For the first few years, the library was hosted at SourceForge. Then in the early 2010s two things happened that made me move it to GitHub. First, SourceForge started wrapping adware into installers, which made me worried my work would be used for spreading malware. Second, around that time I started using Git at work and soon was ready to use it for my open-source project as well. In August 2015, I moved utfcpp to GitHub where it is still hosted.

Soon after the move to GitHub, I replaced my Make+Perl build/test system with CMake. In April 2019, I started using GoogleTest as a testing framework. After it became too complicated to use with C++98, I wrote a small unit-test framework and called it FTest. Still happily running it, and apparently some other developers are too.

utfcpp continued growing slowly, but steadily, and at the time of this writing it has over 2000 stars and over two hundred forks. It is also being distributed by quite a few Linux distributions.

At GitHub, I started accepting code and documentation contributions - currently there are 22 contributors to the project. Most of their help was in the areas I needed it most, such as fixing CMake scripts and improving documentation. I am thankful for it.

Interestingly, I have never attempted to create a community around the project. Not sure why really.

Good Decisions, Bad Decisions

In hindsight, there are a few things I would have done differently.

Technically speaking, there were a couple of blunders, such as the broken previous() function7 that I replaced early by a corrected version that I called prior(). Trying to support UTF-32 encoded strings was a mistake, not because the implementation is incorrect, but because it is wasteful to encode entire strings as UTF-32 and I shouldn’t have supported that scenario at all.

Switching to CMake was a mixed blessing. I like it as a replacement for autotools, but it started being used as a poor man’s package manager, and I unfortunately accepted contributions that added that kind of functionality to my project. I dislike package managers in general8 and never understood why anybody needed that mechanism to install a simple, header-only library like utfcpp.

As for good decisions, there were some as well: the most important one was probably the commitment - maintaining a small open-source project for two decades requires dedication even if realistically it does not take much time. It helped that I was focused and kept the scope limited; I rejected a few proposals that would have probably increased the usage of the library but would make it grow in scope; one such example is support for CESU-8 encoding. I do not regret passing on these opportunities.

Being conservative was also a good decision. I resisted the urge to break backward compatibility to improve the API9. As the new versions of the C++ standard appeared, I kept adding separate layers on top of the existing ones, without breaking them. Nowadays, a developer can use the original C++98 compatible version or opt for something more modern - it is just a matter of preference or compiler availability.

Use of Artificial Intelligence

Recently, LLM based coding tools became good enough to be used beyond simple prototyping. For the time being, at least, I choose to keep manually editing the core code and documentation.

That said, I did use GitHub Copilot to build some very useful performance tools and refactor a part of the test framework. If I ever need to do anything with CMake or GitHub workflows again, it will be all Copilot.

Conclusion

In retrospect, I am glad I have worked on the utfcpp library all these years. It did not bring me any material benefits: even when hunting for a new job, I found that prospective employers were not aware of my small project (StackOverflow reputation and programming articles were much more valuable in this regard). It did make me learn more about Unicode, which is a skill I regularly use at work.

Most importantly, it has been fun.

1

I was wrong, but that’s a different story.

2

Any similarity with Ken Thompson’s development of UNIX is purely coincidental.

3

I enjoyed being a part of The Code Project community for many years. Unfortunately, the business behind it collapsed in 2024 and the articles are gone now.

4

I liked Subversion. It was less powerful than Git, but much simpler.

5

Ten years later I removed the Boost dependency from my employer’s code base. We used only small parts of it and suffered increased compile times.

6

I laugh at some of the views on software development I held at the time.

7

As far as I remember, the reason it was broken was that it took the “one before the first” argument which is undefined behavior, unlike “one past the last”.

8

Ginger Bill, the creator of Odin programming language, wrote a nice article on the topic: Package Managers are Evil. I could probably add a few points, but not here…

9

I started a separate project, utfcpp20, to explore how the library could look like if it was written with C++20 from the beginning. It is still far from being production ready, and I am not sure it will ever be.