Tue, 03/03/2020 - 16:32
Update 03-03-2020 16:28:20
Supplier has identified our affected circuit and rebuilt the circuits, we got an update from supplier that most of Ethernet services' issue would be fixed now. We contacted individual customer and confirmed that issue is resolved. There are two circuit which is still down and supplier is working on it. We shall provide you further update in an hour.
We apologise for any inconvenience caused. If you have any queries, please contact the Support Team on 0345 257 1335 or email at support@cerberusnetworks.co.uk.
Update 03-03-2020 15:07:16
Update - Network - P1 - A number of customers connecting via NPE002.THN may be experience a total loss of voice and data service.
Priority: 1
Status: Investigation & Diagnosis
Latest Update
The technical bridge between Network Support/Access and I Operations has concluded. Thy agreed that the invalid encaps configuration will been to be rectified however are not sure why the impact is only being seen by such a small number of circuits on the NPE (there are 130 customer reporting problems, 6.6k with invalid encaps and over 8.5k customers connected to the NPE). They have confirmed we will need to read the correct configuration to the NPE and then complete a LDP process restart in order for them to be saved. This can be done out of hours if we can mitigate the impact by rebuilding the impacted customer services. Fully rebuilding the circuits that have been reported as impacted is working, once committed the circuits come up. It takes between 5-10 minutes to rebuild each circuit and we have around 130 reported (so far we have rebuilt 7). As a plan to mitigate impact Operations have collated a full list of impacted circuits, this has been shared with Access and they will work with 24/7 to rebuild the remaining circuits asap. Its expected to take at least 2-3 hours to complete all of the rebuilds. MIM engaged CGI OSS and has tasked them with checking the HPSA server logs to see if there is anything which would elude to problems in the logs between 0100 0400 03/03. Next Actions Operations and Access working through all of the impacted circuits (around 120 remaining) and fully striping and rebuilding the circuits. It takes between 5 and 10 minutes to build each circuit so we have allocated resource from 24/7, Escalations and Access to hasten the progress. Network Support want to await the rebuild of circuits before progressing the next steps. They have confirmed they still see the invalid encaps setting on 6.6+ circuits on the NPE and this will need correcting. If we continue to mitigate the impact to customers be rebuilding circuits any corrective works can be done under a planned outage. CGI Oss have been engaged to review the HPSA logs an identify if there was any interaction between the HPSA servers and NPE.THN002.
SummaryFollowing network monitoring. It has been identified that some customers connecting via NPE002.THN will experience a experience a total loss of voice and data services. The affected users will interrupted voice and data services for the duration of this incident.
Date Created - 03/03/2020 03:40:03
Incident Duration - 0 day(s) 8 hour(s) 20 minute(s).
Service Affecting Status - Incident is Service Affecting.
Service Affecting Duration - 0 day(s) 8 hour(s) 20 minute(s).
Tickets Raised to Date: 0
Technical / Suspected Root CauseRoot cause has not currently been identified
We apologise for any inconvenience caused. If you have any queries, please contact the Support Team on 0345 257 1335 or email at support@cerberusnetworks.co.uk.
Posted 03-03-2020 12:23:09
We are currently experiencing disruption to Leased Line services. This is being investigated as a high-priority P1 incident with the present focus being on the possibility of a encaps related problem. Further updates will be posted as they become available.
We apologise for any inconvenience caused. If you have any queries, please contact the Support Team on 0345 257 1335 or at support@cerberusnetworks.co.uk.
