We've published a post-mortem on our blog: https://cloudplane.org/blog/outage-post-mortem

All apps have been restored. Apps using object storage (Mastodon) have been recovered in full, no data lost. The list of apps itself has been restored from a backup that was 20 hours old, so size and domain changes may need to be repeated. We were unable to fully restore one Gitea instance as all volumes including snapshots were deleted, the affected user will be contacted soon.

I'll put up a post to describe why all of this happened soon. For now, I need some sleep. Please message me if you have any questions.

A small mistake during an Helm upgrade ended up deleting our entire production cluster. We do have backups and we should be able to restore all Mastodon instances without any data loss, but we are running into some challenges. I do hope to have everything back by Sunday evening.

I am truly sorry for this event, we are a new provider and some areas are still under development. We have backups, but we haven't had time to actually test a worst-case situation like this one. So, unfortunately, things aren't as easy as I'd hoped.

You will of course not be billed while your apps are down. More details will follow soon.