What MEO’s outage looked like from the routing table

Portugal’s largest telecom operator suffered two disruptions in one day. Before we knew why, BGP was already showing that something unusual was happening.

Last week I wrote that BGP, one of the Internet’s most important protocols, has no map. Last night (and again this afternoon, Lisbon time) Portugal reminded us of that.

Around 00:00 UTC, MEO suffered a major disruption. Traffic on its mobile network (AS42863) dropped by as much as 69%, while its residential network (AS3243) fell by around 65%. By 02:00 UTC it had largely recovered, so most people in Portugal slept through it.

But the traffic drop wasn’t the only thing that stood out. MEO’s four networks also saw a surge in BGP announcements beginning around 22:30 UTC and peaking at 00:15. Looking across 48 hours, announcement volume was roughly 70 to 85 times higher than during the previous week.

No central map

BGP, or Border Gateway Protocol, is how networks tell each other which blocks of IP addresses they can reach. There is no central map of the Internet: each network builds its own view from what its neighbours tell it.

I used a courier analogy last week. Imagine thousands of delivery companies, each putting up notices saying: “We deliver to these postcodes.” Other couriers read those notices and use them to decide where to send the next parcel. Nobody needs to know the entire journey, only where to send the parcel next.

Those notices change constantly. Networks announce new routes, change existing ones and withdraw routes they can no longer reach, so BGP updates are part of the Internet’s normal background noise. A sudden burst across several networks belonging to the same ISP is more interesting.

During the MEO disruption, its prefixes remained announced. In the courier analogy, MEO never took its notice down. It just kept rewriting it while many of the parcels weren’t moving.

Doug Madory, one of the best-known Internet routing analysts, spotted another part of the picture. As traffic dropped, MEO’s customer networks were routing away from its backbone network, AS8657 — and those changes themselves generate BGP activity.

From IPinfo’s vantage point, the burst looked unusually even across all four of MEO’s networks. Two measurement systems, two angles on the same night. Something was clearly happening inside the routing system; what the routing data couldn’t tell us on its own was why.

Symptom versus cause

MEO has since said that it was hit by a large-scale attack that caused congestion and temporary degradation of its international network, with no intrusion or access to customer data. What kind of attack caused it is now the obvious next question.

That’s an important distinction when looking at Internet measurement data. The BGP spike tells us what the disruption looked like from the routing system, but it doesn’t by itself tell us what caused it.

There’s a useful precedent here. During the Rogers outage in Canada in 2022, Cloudflare saw Rogers routes being withdrawn and initially described the network as effectively disappearing from the Internet. But that was the view from Cloudflare’s observation points. Other Rogers routes remained announced elsewhere, yet weren’t carrying traffic either. The withdrawals were a symptom of a deeper internal routing failure, not the cause of the outage.

It’s a useful lesson in observability: what you see depends partly on where you’re looking from.

And MEO’s problems weren’t over. A second disruption hit AS3243 and AS42863 between 16:30 and 17:30 UTC, with traffic falling by as much as 51% and the unusual BGP activity returning, before traffic recovered again.

There is a much more extreme version of this story. It happens when the information being passed through BGP isn’t merely changing very quickly, but is actually wrong.

The day one router tried to become the Internet

In April 1997, a router at a US Internet provider received a full routing table from a downstream customer and then re-announced around 23,000 of those routes as if they belonged to its own network.

Many were more specific than the legitimate routes already circulating, and BGP generally prefers the more specific route. Networks around the world therefore began sending traffic towards the wrong place, creating one of the earliest major global BGP incidents.

The following day — once the company was back online, having been disconnected in the aftermath — an engineer explained what had happened on the NANOG mailing list. The immediate technical failure was wonderfully mundane: there was no inbound filter preventing those routes from being accepted and re-announced.

The apology is still sitting in the NANOG archives, typos included. My favourite phrase comes when the engineer tries to describe what the router did next:

“In an as yet unexplained twist of bits, the 7007 router then began to de-aggregate the 23K route view *and* strip the AS path out of it.”

Nearly thirty years later, the Internet is vastly larger, and routing security has improved. But the fundamental architecture remains remarkably familiar. BGP still works because networks tell other networks what they can reach, and those networks make decisions based on the information they receive. There is no omniscient controller checking a global map before every packet starts its journey.

That is both one of the Internet’s great strengths and one of its enduring weaknesses. It’s also what makes routing data so interesting to watch: you can sometimes see that something is going wrong, and even see how networks are reacting to it, before you know why.

Thanks to Andree Toonk for pointing me towards the MEO disruption in the first place.