Some customers are experiencing issues with sending and receiving emails, starting yesterday and continuing into today. If you use the Business Email service you will be one of the unlucky customers. We apologise for the outage and would like to explain in a little more detail what happened, and what steps are being taken to address the problem.
The issue lies with the upstream service provider, who experienced a catastrophic failure of the email server in the early hours of yesterday morning (6th Dec) during a routine maintenance task. The server was stopped to swap some disks, and has subsequently failed to reboot.
Initial issues to repair the problem were unsuccessful. 4 or 5 hours in the decision was taken to open a separate, parallel stream of work. Since that point they have been attempting to...
- repair the issue on the original server, and
- restore from the latest backup to a new server
The repairs have, to date, been unsuccessful. The suspicion is that there is a significant hardware failure on the server that will make this impossible.
The restore process is continuing, albeit very, very slowly. This is the most likely means of restoring service, but as yet no timescale can be produced.
That's all the information we have at this point in time, but we will provide further updates as progress is made.
Thursday, December 7, 2023
