Just as we were about to report that service had been fully restored from the last major outage, unfortunately there has been another issue with this server. We can only apologise for the ongoing problems, and provide an explanation as to the cause (as far as we know).
Some background first. The previous outage was, essentially, because the server died with no prospect of being repaired. In order to restore service a new server was built, and after some time all user accounts were restored onto that server.
It was this mass restore process that illustrated one of the major failing of the backup process as it was, namely that restoring millions of files (each individual email on the server is a separate file) is painfully slow and causes unacceptable downtime for customers. As a result the backup process is going to change in future so that such outages can be fixed more quickly.
Unfortunately this new outage has come too soon to benefit from that revised process.
This latest outage is due to a failure of the disks connected to the new server. It is completely unrelated to the first outage and seems to be purely bad luck. The old backup/restore service is still being used, which is why recovery is taking so long.
We can only apologise wholeheartedly. Email is an essential service, and whilst no one can provide 100% uptime, we strive to offer a much better service than this. We will be offering compensation in the form of account credits once the dust has settled.
Thank you for your continued understanding.
Monday, December 18, 2023
