Taming Intel’s “Brain Damaged” Chip

The first generation of Intel’s 16-bit microprocessors turned out to be a surprising success after IBM chose their 8088 model for the IBM PC computer.

Many software industry leaders expected the transition from 8-bit to 16-bit computing to bring more capable operating systems, including multitasking and networking support. That did not happen: the IBM PC shipped with a clone of the old, single-tasking CP/M operating system, called PC DOS and most of the industry followed.

In early 1982, Intel announced the next generation of its x86 processors, the 80286, and there was hope the operating systems would finally catch up. That did not happen either.

Two Modes of Operation

The 80286 could be run in one of the two modes: Real Address Mode or Protected Virtual Address Mode .

Real Address Mode, usually called “Real Mode” was a compatibility feature. The CPU would start in Real Mode and would be able to run software written for the 8086 architecture without any modifications. Strictly speaking, there were a few instructions available in the 286 Real Mode that did not exist in the original 8086 set, but they would not interfere with the compatibility.

To have access to the full set of advanced CPU’s features, an application could switch to Protected Virtual Address Mode (often called “Protected Mode”). Once in Protected Mode, the CPU offered much larger address space (16 MB vs 1 MB in Real Mode), built-in support for virtual memory, efficient context switching to better support multitasking, and four protection rings which enabled different levels of trust for kernel and user modes.

The new features were very attractive to the implementors of multitasking operating systems such as various UNIX derivatives. Even the Real Mode provided significant performance improvements to the older generation of the Intel’s 16-bit chips, and already in June of 1982 IBM decided to base its next generation model of the IBM PC on the 80286. The new computer, named IBM PC AT was announced in August 1984, and was soon followed by numerous clones from companies such as Compaq, Tandy, HP and even Commodore. It was positioned as a high-end model and did not replace the existing IBM PC and XT ones that continued to be produced until the PS/2 line was released in 1987.

It was clear that a simple single-user operating system such as MS-DOS/PC-DOS could not take the full advantage of the Intel’s new “superchip” . The IBM PC AT shipped with PC-DOS 3.0 which was specifically written to preserve backwards compatibility to the highest degree possible. For all its new features, PC-DOS ran in Real Mode, could see only 1 MB of memory (with only 640 KB being available to software and data) and offered no real multitasking or protection from an application crashing the entire system. The challenge was not designing a modern operating system but designing one that could still run Lotus 1-2-3.

There was a detail that made the task of creating such an operating system much more difficult than anybody expected: once an 80286 CPU started, it was in the Real Mode. Switching to the Protected Mode was well supported and trivially easy. Going back from the Protected Mode to Real Mode was not supported - it required a restart. This feature frustrated the operating system designers to the point that Bill Gates famously called the 80286 a “brain damaged chip”.

Why was that such a problem? A modern, multitasking, operating system needed to run in the Protected Mode to use the advanced 80286 features. From there, running applications written for the new environment would be easy - such programs would not assume they could access the hardware directly, or mess up with arbitrary memory locations. However, by 1984 it became critically important to run existing software, like Lotus 1-2-3 which was written with no such limitations in mind - it assumed complete control over the machine it ran on, which is what the old 8086 and PC-DOS enabled.

Two Ways to Run DOS Programs on 286

To run a legacy DOS application from Protected Mode, there were two possible approaches:

The first technique was to run even the DOS applications in Protected Mode. Strictly speaking, software written with the 8086 instruction set could run in Protected Mode just fine as long as it never accessed hardware directly, or used Protected Mode only instructions, or trying to make tricks with segments arithmetic… Of course, pretty much no real-life programs adhered to these limits and sooner or later they would cause protection faults. The goal was to handle all the faults caused by the 8086 programs and restart them at the point where they were interrupted. If implemented correctly, this mechanism could run multiple legacy applications simultaneously and safely while keeping the operating system in control. Of course, the implementation was extremely tricky, and the performance was going to suffer when there were many faults. This technique was sometimes referred as an “emulation” of Real Mode, although the term was somewhat misleading; the 8086 instructions ran natively in Protected Mode without any emulation.

The second mechanism was to run the operating system in Protected Mode and switch to Real Mode to run a DOS program. As mentioned, the only way to do it was to involve the reset and a few techniques to do that were used in different situations. This technique was simpler (once the switch actually happened), the legacy applications could access hardware without supervision from the OS and generally performed better. One major downside was that once the CPU was in Real Mode, there was no more multitasking - it was not possible to run multiple 8086 applications simultaneously.

The two major system software providers: Digital Research Inc (DRI) and Microsoft took two different approaches: DRI with support from Intel went with pure Protected Mode, and Microsoft along with IBM opted for restarting the CPU into Real Mode. Let’s see how that went.

DRI: Running in Protected Mode

Soon after announcing the 80286, Intel presented it to the legendary Homebrew Computer Club. A Michael Swaine’s report from the meeting published in the April 5th, 1982, edition of InfoWorld mentions the announcement of Digital Research already working on an operating system for the new CPU, named MP/M-286.

At that time, DRI was already heavily betting on multitasking operating systems - even on the much less capable 8086 platform. MP/M-86 2.1 was released in September 1981, and in early 1982 its single-user multitasking descendant called Concurrent CP/M was announced. By 1984, DRI’s Operating Systems Division was busy developing not only the next generation of the 8086-based multitasking systems, but also porting UNIX to the 80286 in cooperation with Intel and AT&T. That project went on a back burner after AT&T pulled off and DRI and Intel then focused on completing MP/M-286, which was in the meantime renamed to Concurrent DOS 286 and pre-released to select customers in January 1985.

Concurrent DOS 286 was not merely a port of the previous generation of “Concurrent” systems to the 80286. It was designed from scratch by DRI’s chief operating systems architect Frank Holsworth and implemented in the C programming language. Legacy 8086 applications could be run via the optional PC DOS front end, which ran in Protected Mode, just like the rest of the operating system.

Unfortunately, the source code of Concurrent DOS 286 and its descendants has not been released to public. As a result, the details of its compatibility layer must be reconstructed from Intel documentation, magazine articles, and contemporary accounts.

It seems that the original PC front end would set the privilege level of all local and global descriptor table entries lower than that of the current program, so that a protection violation occurs whenever the program attempts to load a segment register. The kernel could catch exceptions caused by Real Mode style instructions, handle them by mapping invalid addresses to valid descriptors and restart the execution. Obviously, a DOS application could generate arbitrary segment values, and the operating system could never know in advance which descriptors would be required, which put even more pressure on the performance.

At the time of the initial announcement DRI went to great lengths to emphasize that the PC DOS front end was able to run a wide range of legacy applications, including the notorious Lotus 1-2-3.

Then a disaster struck: DRI’s new operating system was developed on an early version of the 80826 called stepping B1. In the first half of 1985, Intel started shipping stepping C1 which fixed some problems (or “errata” in Intel’s terminology) but also stopped preserving the state of the CX register when a general protection fault occurred, which made restarting the execution from the handler impossible.

The release of Concurrent DOS 286 was delayed while Intel went to work to produce a new stepping: E1. In that version, Intel restored preserving the state of the CX register during general protection faults. DRI adapted the compatibility layer to the E1 chips, but there were still some performance issues left, and it was only when Intel released the E2 stepping that the operating system could finally be released.

By that point, DRI abandoned the goal of making Concurrent DOS 286 a general- purpose operating system. A few big customers such as Siemens and IBM were more interested in the system’s real time capabilities and modularity than in its ability to run legacy DOS applications, and they licensed it to use it on their specialized hardware. DRI renamed Concurrent DOS 286 to FlexOS and marketed it as a real-time system for industrial control applications.

The PC front end remained an optional feature of FlexOS 286. It required the E2 stepping of the 80286 and allowed “certified PC DOS version 1.0 and 2.x applications to run in the protected environment of a multitasking operating system”.

Microsoft/IBM: Switching to Real Mode

The IBM PC AT shipped with PC-DOS 3.0 which did not even try to take advantage of the Protected Mode goodies the new chip offered. The name of the game was backward compatibility: existing PC applications had to continue running on the new model - just faster. At the time of the launch, IBM showed XENIX 286 as an example of the system that was able to use Protected Mode, even though XENIX was not a product IBM supported.

However, even before the AT was released, Microsoft started working on a multitasking version of MS-DOS . Like DRI, Microsoft hoped to develop an operating system that would run both legacy 8086 applications and exploit the new 286 Protected Mode. Unlike Concurrent DOS 286, it was initially not imagined as a brand-new code base and architecture; rather it was an extension of MS-DOS 2 and internally named DOS 4. The brain behind the multitasking DOS was Gordon Letwin.

It soon became clear that the task was more complex than anybody imagined. Microsoft decided to scale down DOS 4; Protected Mode was abandoned and only a limited version of multitasking was introduced. The operating system was released to a select group of OEMs in May 1986 under the name MS-DOS 4.0 and quickly forgotten as it did not receive support from IBM.

However, the idea of developing a DOS descendent that would use Protected Mode was not abandoned. The project was at first internally called DOS 5 (unrelated to MS-DOS 5.0, released in 1991), but in the summer of 1985 the agreement was made with IBM to merge the project with IBM’s similar effort, led by Ed Iacobucci, code named CP/DOS. The operating system would eventually be released under the name OS/2.

About the time the joint work on CP/DOS started, Intel announced it next x86 compatible processor - 80386. It solved all problems with Real Mode compatibility by introducing the Virtual 8086 Mode specifically to run multiple legacy DOS applications, and according to some sources, Bill Gates wanted to abandon the 80286 and develop the new operating system for the 386 only. IBM insisted on supporting the 286 and at that point IBM was still the senior partner in the relationship.

According to Letwin, both teams independently came to the conclusion that implementing Real Mode emulation from Protected Mode was a dead end and opted for switching to Real Mode by resetting the CPU. The challenging part, of course, was executing the reset without losing the state of the operating system, and Letwin compared it to “to switching gears in a car by turning the motor off”. The reset had to be initiated by hardware: on PC AT machines, the keyboard controller was connected to the CPU reset line and it was possible to restart the CPU by sending a command to the controller. RAM would be refreshed simultaneously to keep the operating system in memory. The BIOS for IBM PC AT contained code that would distinguish between a “real” restart and a mode switch and execute code in a different location for each scenario.

The switching mechanism was not without problems: only one DOS application could be executed at any time, and it needed to run in the foreground, although multiple protected mode processes could be run simultaneously. The mode switch itself was slow, and the operating system was designed to minimize number of switches.

Yet, the approach led to decent performance and backward compatibility, and most importantly was not affected by the bugs in the 80286. That allowed the OS/2 to be sold as a general-purpose operating system, although it was not nearly as successful as its creators had hoped.

The Aftermath

For all the effort spent by the leading software companies of the time, multitasking operating systems did not become mainstream on personal computers until well into the 1990s, and by that time the restrictions of the 80286 did not matter anymore. Microsoft and IBM ended their cooperation in 1990 with IBM continuing the development of the OS/2 for another decade, and Microsoft focusing on Windows. Digital Research survived as a developer of alternative operating systems until 1991 when it was acquired by Novell which was not able to make use of DRI’s technologies and by 1996 sold them all off for a fraction of the cost it spent on the acquisition.

The official names for the two modes are quoted directly from Intel 80286 and 80287 Programmer’s Reference Manual, chapter 1

InfoWorld Apr 5, 1982 announced the 80286 in its article named “Homebrew Computer Club views Intel’s superchip”

It was unfair: development of the 80286 started in 1978 with the goal to be primarily used in embedded/industrial systems. Custom PC’s article Intel 286 - 16-bit CPU swansong quotes Intel’s 1984 annual report in which it is stated that in the planning documentation for the 80286 the personal computer “wasn’t mentioned once”.

If you are interested in history of Digital Research, my article The Peak and Collapse of Digital Research is a pretty comprehensive work on the topic. It’s quite long and if you don’t have patience to read it all, I recommend section Round Three which describes some history behind the 80286 race.

Francis R. Holsworth was a DRI employee number 14 and the lead designer of its advanced operating systems including MP/M-86, Concurrent CP/M and FlexOS. He served as Director of Flexible Automation Unit from 1986 until Novell’s acquisition of the company in 1991 and then led the Novell’s Monterey Development Center until September 1992 when he left the company.

Source code for many other DRI products can be found at http://www.cpm.z80.de/source.html

The method was described in Intel’s unofficial documentation for the LOADALL instruction. LOADALL itself was probably not used in the original implementation of Concurrent DOS 286, due to bugs in this instruction in early 80286 versions.

The quote is from FlexOS Supplement for Intel iAPX 286-based Computers released by DRI in November 1986.

From the book “Gates”, by S. Manes and P. Andrews: “The AT would run XENIX 286, a new version of Microsoft’s forgotten solution to the 16-bit software crisis, which Bill Gates had managed to sell IBM at the last minute when Boca suddenly realized it had nothing at all to show off the new chip’s protected mode.”

Much of the information about Microsoft’s initial efforts to develop a multitasking operating system comes from Microsoft Systems Journal’s 1987 interview with Gordon Letwin.

Gordon Letwin was hired by Microsoft in 1978 where he started by writing their BASIC compiler. He managed development of XENIX until in late 1983 he took over development of multitasking DOS. Letwin is best known as a lead architect for OS/2. He left Microsoft in 1993.

Edward E Iacobucci joined IBM’s Boca Raton office in 1979 and became the chief architect for their multitasking operating system code named CP/DOS in 1983. He left IBM in 1989 to create Citrix. Iacobucci passed away in 2013.

For more technical details like this, see The Design of OS/2 by Harvey M. Deitelan Michael S. Kogan. The book in the PDF format is freely and legally available.