TPG Submarine Cable Outage Expected to Last a Month

Fixed-line telecommunications provider TPG has advised its customers of a major submarine cable outage between Sydney and Guam, which is not expected to be repaired until March 7, 30 days after it occurred.

The outage, which occurred shortly after 8pm AEDT on February 5, resulted when the PPC-1 submarine cable system, which connects Australia with Papua New Guinea and Guam, lost its payload. Investigating engineers found a fibre fault on the Guam side of the cable.
​
Figure 1 - PPC-1 Cable Map

Source: PIPE Networks

"At approximately 20:00:10 AEDT on 5 February 2016, the TPG Network Operations Centre received multiple alarms for the PPC-1 cable system between Sydney and Guam. The alarms indicated that a submarine line card had lost its payload. A Network Incident ticket was created and the issue was immediately escalated to the PPC-1 transmission engineering team," TPG said in a statement on its website.

"The transmission engineering team, in collaboration with infrastructure vendors, completed preliminary investigations and have found a fibre fault approximately 4,590 kilometres from Guam. The nature for the fibre fault has resulted in an outage to the entire PPC-1 cable system. TPG has started to work with our maintenance contractor to understand when we can mobilise a repair ship to the specific location.

"Due to the nature of the fault, resolution is expected to take an extended period."
A spokesperson from TPG explained to ZDNet that it is mostly wholesale customers who have been affected by the fibre fault.

"There is a clear fibre fault about 4,652km from Guam at a depth of more than 2km. The PPC1 outage has meant that TPG Telecom has shifted all IP traffic to its redundant Southern Cross and AJC capacity.
"At this time, the affected customers are mainly wholesale customers who purchased unprotected services using the PCC1 path. Specific Asian destinations may experience an increase in latency, however TPG will further optimise traffic routing over the next few days to minimise customer impact."
The spokesperson added that TPG's maintenance contract provides for relatively fast shipments based out of New Caledonia for hardware to repair any cable faults that arise. However, the maintenance ship is already repairing a fault in the Basslink cable system, which will delay repairs on the Sydney-Guam TPG subsea cable.

"TPG Telecom has a maintenance agreement which provides the group access to the CS Ile De Re based in Noumea, New Caledonia. This ship is stocked with spare cables and other hardware for repairs to the cable system. We have been informed that the same ship is being used to repair the Basslink cable system. As a result of the Basslink repairs, extra delays can be expected and therefore TPG is currently considering other options for a faster repair using an alternative vessel. At this stage, TPG is not able to quote exact restoration times.

"Once TPG has a firm handle on the restoration times we will update our customers accordingly."
 
Figure 2 - Cable ship Ile de Ré
​

Source: Alcatel Submarine Networks

On February 21, TPG's maintenance contractor, Alcatel Submarine Networks, dispatched the cable ship "Lodbrog" to service the damaged cable. Neither Alcatel Submarine Networks nor PPC1's owner, TPG, announced which ship would handle the PPC-1 break, but online marine tracking services show it as departing Taiwan en route to Cairns.

TPG's announcement last week said the ship was due in Cairns on February 28, but Vessel Finder's current track for the ship suggests it will arrive in Cairns on 1 March (midnight UTC, 11am Australian Eastern Standard Time). If it continues to that schedule, the Lodbrog will be on station to commence repairs around March 4.

TPG customers have complained over broadband enthusiast website Whirlpool about slow international browsing.

"Since Friday night, has anyone else been noticing very slow international internet browsing? Often getting timeouts loading various web sites, or takes a very long time to load up," said one user.
 
Figure 3 - Cable ship Lodbrog

Source: MarineTraffic.com

Another user came up with a method to circumvent the affected cable: " I have been using a SSH tunnel into my VPS in Hong Kong to get around these issues. It goes through the Australia-Japan Cable. Singapore is also working fine, lots of cheap VPS options there, including the cheapie Digital Ocean. If TPG submarine cable outage expected to last a month you are willing to compromise on security, there are also some free vpns [sic] you can experiment with temporarily until the issue is fixed."

For the interim, TPG has rerouted customer connections to the Australian-Japan Cable system and the Southern Cross cable which connects Australia and New Zealand to the United States.
​
Another Whirlpool user complained that TPG's fix was not working well, however.

"TPG's alternative routing seems to be very poor, the performance is shocking. Alas I suppose if you don't pay much for the service you can't expect they will pay for a good backup system, think i'd [sic] better find another ISP.

"Not to mention they run a major internet service and didn't even notice there were problems, then when someone reported it they said nothing was wrong - that ain't active monitoring to ensure your customers get a good service."

The Southern Cross cable was itself hit by a "catastrophic failure" in 2012, when an unauthorised software change was made to the wavelength switching platform at its landing station in Sydney.
Rival telecommunications provider Telstra had its own Australia-Singapore subsea cable outage in October last year, which caused severe delays for Telstra customers attempting to download or update their operating systems or apps across the iTunes Store and the App Store, as well as use streaming services Apple Music and Apple Radio.

Source :  ZDNet.com, The Register

JRC ANALYSIS:

TPG has a private maintenance agreement with Alcatel Submarine Networks which is supposed to provide vessel cover in the so-called "South Pacific Zone", an area which stretches from Guam in the North Pacific to Tasmania in the South to the mid-Pacific. This would have been the only pre-arranged maintenance option available, since the alternative South East Asia & Indian Ocean Cable Maintenance Agreement (SEAIOCMA) which offers "club maintenance", does not stretch that far east.
​
Alcatel's maintenance vessel, "Ile de Ré", is normally stationed in Noumea, New Caledonia and covers the South Pacific Zone. However, as mentioned above, this vessel was already working for another customer when the outage occurred on PPC-1. The details of the maintenance contract between TPG and ASN are confidential but it is likely that ASN is, to a certain extent, obligated to provide a vessel. The "Lodbrog" usually operates out of Keelung, Taiwan under the Asia Pacific Private Maintenance Agreement (APMMSA) which is a private maintenance service offered jointly by ASN and TE SubCom. Getting their partner, TE SubCom, to agree to release the Lodbrog for PPC-1 was probably a costly endeavor in the first place but with ASE and possibly other cables landing in Singapore going down in early March, there is certainly pressure on maintenance providers in the Asia Pacific region at the moment.

Restoration for PPC-1 on Southern Cross and on AJC will also not come cheaply. TPG is known to have expressed interest in other cable initiatives that would touch Australia and this incident will add fuel to the fire.

Lastly, if the fault is really located at 2,000 meters water depth, it will be a challenge for the Lodbrog to retrieve the cable, let alone repair it.