Notes from RIM’s press conference on its apocalyptic outage

It’s been a wild three days — and counting — for millions of BlackBerry users across several continents. Facing the BlackBerry Internet Service’s biggest outage in history, and compounded by a hoax BBM message that asks users to forward it to their contacts in order to maintain BBM service, RIM held an emergency press conference on Wednesday to mitigate some of the growing animosity of its weary customer base.

Following sporadic service status updates on its website since Monday, RIM addressed the press in a conference call that included a Q&A session.

David Yach, RIM’s CTO for software acknowledged the severity of the outage that started on Monday by confirming a “major outage for customers around the world”. He said that it is RIM’s top priority to return BlackBerry service to its customers and that they have separate teams still investigating root cause.

As noted before, Yach re-iterated that RIM’s infrastructure based in Europe experienced a core switch failure. Despite having multiple redundancies for its network switches that would fail over to a redundant switch with no impact to users, in this case, the fail over did not function as expected. Despite regularly testing the fail over systems, RIM is currently investigating what caused this unexpected behaviour.

As a result a substantial backlog of messages has been generated and RIM had to throttle service operating speeds to process it. The throttling is why users are seeing ongoing issues in the EMEA region, and now, the Americas.

RIM will be launching a full investigation once systems are restored fully to make definitive determination as to the root cause and the subsequent instability.

Yach noted that the impact on RIM’s customers has been varied. While some have experienced no impact at all, a large number have experience either delays or service interruptions while some have experienced a lethal combination of both.

Yach ruled out a security breach or hack as the cause for the service interruption, by saying that RIM has seen no evidence that this is the case.

Q&A Session

Q: How has this outage spread from one region to another?
A: BlackBerry has a global infrastructure with nodes located in various geographies. Messages being relayed via Europe will be delayed.

Q: Are the recent issues in the Americas caused by the backlog in Europe, a problem with the main NOC (Network Operations Centre) in Waterloo, or service throttling?
A: It’s a backlog issue in Europe based on the initial outage and the time needed to stabilise it. Other regions have not been throttled at this time.

Q: Is there any likelihood that messages that form part of the backlog could be purged or lost?
A: No, all of the email will be delivered. We will not be dropping any messages.

Q: There seems to be an awful lot of angry customers out there that are weighing on in Twitter, Facebook and elsewhere regarding this outage. Does RIM feel that it has done all that it can to communicate with its customers as it hasn’t seemed to be — from the public’s perspective — enough, and is there any thought on any “make good” actions to soothe those angry customers?
A: We have been posting regular updates on our Twitter handles as well as our website properties at rim.com and blackberry.com. Our priority is to get the service up and running, because at the end of the day, that’s what is going to make our customers happy.