Incidents reported on status page for Cloudplane https://status.cloudplane.org/ en https://status.cloudplane.org/incident/422574 Sun, 01 Sep 2024 15:15:00 -0000 https://status.cloudplane.org/incident/422574#6cc6c3615273b69802b7e97ec177f9630890d94f51c6dc2455bfa2e0cf2ce85c The issue has been resolved. We apologize for the downtime. https://status.cloudplane.org/incident/422574 Sun, 01 Sep 2024 08:54:00 -0000 https://status.cloudplane.org/incident/422574#90b8bb08fd7ea494e950904a24b170238fee7eb63d10ed3919a4f766ed0a794a This morning, our compute cluster experienced issues reaching internal APIs. This is connected to an issue at our hosting provider (https://status.scaleway.com/incidents/11x4s3y16p9g). This issue was eventually resolved by Scaleway staff, but it did create a scheduling problem that prevented many Redis instances from starting, thereby causing downtime for some applications (including Mastodon). We are currently taking steps to mitigate this issue. https://status.cloudplane.org/incident/378460 Mon, 03 Jun 2024 00:30:00 -0000 https://status.cloudplane.org/incident/378460#a64b94cf0469f406720b119d1280a35f8ae12f9e6253b033a8baf9c2af78db93 The issue has been resolved. https://status.cloudplane.org/incident/378460 Mon, 03 Jun 2024 00:00:00 -0000 https://status.cloudplane.org/incident/378460#54e1d4298fcffb24bde56e4cef6327905b64d7d67652b3d1763053940f023505 One of our database servers unexpectedly ran out of space during a routine backup operation. This caused approximately 30-60 minutes of downtime for a small number of applications. We apologize for the disruption. https://status.cloudplane.org/incident/377161 Thu, 30 May 2024 22:50:00 -0000 https://status.cloudplane.org/incident/377161#82ef267e658259d952e56df115626b5facfb812033b143e31ec36c186dc056ad The update has been rolled back. https://status.cloudplane.org/incident/377161 Thu, 30 May 2024 22:30:00 -0000 https://status.cloudplane.org/incident/377161#6db9eea47f4d2b0b4fa7f35620a64a7f869ff58ff1b9dca5a8f23c492bfebd36 The update to Mastodon 4.2.9 made it so all pages returned an error due to stricter proxy validation in the new version. This change was unfortunately not caught during testing, we apologize for the disruption. The update has been rolled back. https://status.cloudplane.org/incident/326268 Tue, 13 Feb 2024 15:40:00 -0000 https://status.cloudplane.org/incident/326268#7b162b85d34c8f256353898acd76cde1f327d2730d7283591c0aa5c3830cd539 All services are back to normal now. The cause was an issue at our provider, see https://status.scaleway.com/incidents/4ppbjnqwp7bb https://status.cloudplane.org/incident/326268 Tue, 13 Feb 2024 15:00:00 -0000 https://status.cloudplane.org/incident/326268#505985f86af8edd8774e84e91b8f3f3c02b29f305db46fccf3c5b76ad5735849 All our nodes are currently unavailable while are awaiting a resolution by our provider. This appears to be the same issue we experienced last night. We apologize for the disruption and we'll update you as soon as we know more. https://status.cloudplane.org/incident/326085 Tue, 13 Feb 2024 07:50:00 -0000 https://status.cloudplane.org/incident/326085#1fc3db8a8c94f800efa149fdac2ad851f0cfab528c94758a181351f8747e56ea The issue has been resolved. https://status.cloudplane.org/incident/326085 Tue, 13 Feb 2024 00:27:00 -0000 https://status.cloudplane.org/incident/326085#95c1131d136b83615581965cf0b710fa4425f22dd03a474e3531032915b95998 Last night at 01:27 CET, our hosting provider Scaleway experienced a [region-wide outage](https://status.scaleway.com/incidents/vnxbgsxqmt5x) which brought down most of our services. The problem was resolved by 03:42 CET, but there was another problem that prevented our Redis instances from coming back online. We were able to resolve the issue fully by 8:50 CET. https://status.cloudplane.org/incident/317484 Thu, 25 Jan 2024 00:50:00 -0000 https://status.cloudplane.org/incident/317484#847637dec9b3da47bc88be7310dbf381f0f978156a5a10e4022d08c87aa56c92 The issue has been resolved. https://status.cloudplane.org/incident/317484 Wed, 24 Jan 2024 22:50:00 -0000 https://status.cloudplane.org/incident/317484#0802822c8bec2ddddca31672ddb12e1bb10ad156f730b1df954c5019721cd7dd Last night at around 11:50pm CET, all our services became unavailable. Normal operation was mostly restored by 12:50am, but there was another outage at 1:40am lasting around 10 minutes. The source of these issues was an outage of Kubernetes services across the fr-par region. More information at https://status.scaleway.com/incidents/qqv67gfnxg7m We automatically mitigate many failure scenarios like node failures, but it's these global outages we unfortunately don't have a solution for. We apologize for the disruption. https://status.cloudplane.org/incident/314002 Wed, 17 Jan 2024 16:25:00 -0000 https://status.cloudplane.org/incident/314002#9c48b0ebf3399436404e7c0e1a87c755d89fdac0c164d3aa78f024a2e969e624 The issue has been resolved. https://status.cloudplane.org/incident/314002 Wed, 17 Jan 2024 16:18:00 -0000 https://status.cloudplane.org/incident/314002#6a42ea2c6e1f9e0193c7a8fe3ec26243bfded7a85ab5d7fe419a3366749ae732 We've detected intermittent outages of our load balancer over the last hour. This is caused by an issue at our hosting provider Scaleway (https://status.scaleway.com/incidents/j75p1pq14q20). We're being told they to have it under control now and are investigating the causes. We apologize for the inconvenience. https://status.cloudplane.org/incident/278517 Mon, 30 Oct 2023 17:00:00 +0000 https://status.cloudplane.org/incident/278517#2e91f08c1fb352a673c2e62e9968a2624bf3c2b37a380c74767c04d5b311f70b Maintenance completed https://status.cloudplane.org/incident/278517 Mon, 30 Oct 2023 09:00:00 -0000 https://status.cloudplane.org/incident/278517#87dd8e785941fa971317e5e284bdd7a63e01e57356034343020c5cd42175a4c4 Our object storage provider is performing maintenance on Oct 30 from 10 am to 6pm CEST. During this time, you may experience increased loading times or errors when accessing files. https://status.cloudplane.org/incident/276901 Tue, 24 Oct 2023 06:35:00 -0000 https://status.cloudplane.org/incident/276901#644bdb35b29e673ea37c33926102be9cb3b0829617b5d7e13b9a121753978b55 We are still monitoring the situation, but as of right now, the issue appears to be resolved. https://status.cloudplane.org/incident/276901 Tue, 24 Oct 2023 06:20:00 -0000 https://status.cloudplane.org/incident/276901#ca48a8d064419f1540cfe59c4c5e5489927cb2405b267a630caa95922129f8f9 We've detected a database failure followed by a delayed switchover, which caused issues for a small number of applications. https://status.cloudplane.org/incident/260882 Tue, 19 Sep 2023 16:22:59 +0000 https://status.cloudplane.org/incident/260882#bf0c706a5270af7f8a7e426a2e1dffd8b1fb5e4bcd1025de43fe7b8220df3756 Maintenance completed https://status.cloudplane.org/incident/260882 Tue, 19 Sep 2023 16:22:59 -0000 https://status.cloudplane.org/incident/260882#2ed7a0f2525ae3187d6cb75335cc2fd7c0f3b3bc0f30293d16cb4542389dd7ac We're deploying database optimizations and important security updates. Brief downtime of under 1 minute may be experienced. https://status.cloudplane.org/incident/260882 Tue, 19 Sep 2023 15:00:00 -0000 https://status.cloudplane.org/incident/260882#2ed7a0f2525ae3187d6cb75335cc2fd7c0f3b3bc0f30293d16cb4542389dd7ac We're deploying database optimizations and important security updates. Brief downtime of under 1 minute may be experienced. https://status.cloudplane.org/incident/239670 Mon, 31 Jul 2023 20:00:00 +0000 https://status.cloudplane.org/incident/239670#d6fc50d14ce7cc2bd3f09fe98099607b993a72a017b491dd7c99aa11130b895a Maintenance completed https://status.cloudplane.org/incident/239670 Mon, 31 Jul 2023 18:00:00 -0000 https://status.cloudplane.org/incident/239670#172803d3e075bfc5d252b8e4b09869067431a19bbb8636191a196c38c459b71e We're deploying changes to core components of our system. Brief downtime may be experienced during this maintenance. https://status.cloudplane.org/incident/225233 Wed, 28 Jun 2023 04:30:00 +0000 https://status.cloudplane.org/incident/225233#b38419d6c8d354685b06d7f09f07e50b738eb39cc05a04528a88029dd420286b Maintenance completed https://status.cloudplane.org/incident/225233 Wed, 28 Jun 2023 03:00:00 -0000 https://status.cloudplane.org/incident/225233#ebb7f6c6f107eb98cb25482773ee8f1cb55caa0e0de8c134981fb5913ab44aac Our provider is once again migrating one of our nodes to a different server. We're also using the opportunity to update our databases. We expect less than 2 minutes of downtime to occur. https://status.cloudplane.org/incident/221793 Mon, 19 Jun 2023 13:00:00 +0000 https://status.cloudplane.org/incident/221793#7ba521c497272c39dec5f1d6fc76a4ef794e718364bc2ae169020600792d67dd Maintenance completed https://status.cloudplane.org/incident/221793 Mon, 19 Jun 2023 12:00:00 -0000 https://status.cloudplane.org/incident/221793#d6652f8529f6ad45bcd185e1310682001d338a8ebb8fe89fa70da6f95be355e6 Maintenance by our cloud provider will require migrating two nodes to a different server. A small number of applications will experience a brief disruption (under 5 minutes). https://status.cloudplane.org/incident/219096 Mon, 12 Jun 2023 10:20:00 -0000 https://status.cloudplane.org/incident/219096#7c8b7e8a1a2e70bb3e8009e6f5a90fbd95370337b04d5c81540d4aed462f727d The issue has been resolved. https://status.cloudplane.org/incident/219096 Mon, 12 Jun 2023 09:47:00 -0000 https://status.cloudplane.org/incident/219096#abe9be73c1ac1bdb362995451c5c96ab10e4958a6abf6452704ce7f9b48ccef2 We have detected a large number of applications being unable to access their database. https://status.cloudplane.org/incident/210424 Thu, 25 May 2023 07:00:00 +0000 https://status.cloudplane.org/incident/210424#38400a3404ea9b79139e586dad2771089cd5935b801c39d7bbe59d678ddaa311 Maintenance completed https://status.cloudplane.org/incident/210424 Thu, 25 May 2023 07:00:00 +0000 https://status.cloudplane.org/incident/210424#38400a3404ea9b79139e586dad2771089cd5935b801c39d7bbe59d678ddaa311 Maintenance completed https://status.cloudplane.org/incident/210424 Thu, 25 May 2023 06:00:00 -0000 https://status.cloudplane.org/incident/210424#dd4b7ac29255e18027eef3c3eaf1ab5376fe61ba74dd6727cb1161206fe7a55a We're taking steps to improve the security of our cluster. Expected downtime of 5-10 minutes. https://status.cloudplane.org/incident/210424 Thu, 25 May 2023 06:00:00 -0000 https://status.cloudplane.org/incident/210424#dd4b7ac29255e18027eef3c3eaf1ab5376fe61ba74dd6727cb1161206fe7a55a We're taking steps to improve the security of our cluster. Expected downtime of 5-10 minutes. https://status.cloudplane.org/incident/204895 Tue, 09 May 2023 00:30:00 +0000 https://status.cloudplane.org/incident/204895#aa347ea67b26b6cfe4618be34d6be25299447040802c058a47963b3ac867baea Maintenance completed https://status.cloudplane.org/incident/204895 Tue, 09 May 2023 00:00:00 -0000 https://status.cloudplane.org/incident/204895#262d0d176e98524f841760c929415ac74b047abe36528f0a33831be384ccb1c5 We're deploying a security update for Redis. Mastodon instances might experience partial degradation lasting less than 1 minute. https://status.cloudplane.org/incident/203042 Thu, 04 May 2023 11:00:00 +0000 https://status.cloudplane.org/incident/203042#c7e7a3eedb3914a5d77f3df5d926b50a81d432bd7451951b614a109ffc2943b6 Maintenance completed https://status.cloudplane.org/incident/203042 Thu, 04 May 2023 06:00:00 -0000 https://status.cloudplane.org/incident/203042#babebc1a8d24e92573f4626d8e7317ff4f374428a4e9da3d91575fdc9088cfd6 Our provider is performing scheduled server maintenance. A small number of apps will experience downtime of less than 5 minutes. https://status.cloudplane.org/incident/197967 Thu, 27 Apr 2023 13:30:00 +0000 https://status.cloudplane.org/incident/197967#ffe6c639c6e6ec4980228ff2fbf15103a88ee95dc573e6f03a46f39442e539f5 Maintenance completed https://status.cloudplane.org/incident/197967 Thu, 27 Apr 2023 13:30:00 +0000 https://status.cloudplane.org/incident/197967#ffe6c639c6e6ec4980228ff2fbf15103a88ee95dc573e6f03a46f39442e539f5 Maintenance completed https://status.cloudplane.org/incident/197967 Thu, 27 Apr 2023 12:15:00 -0000 https://status.cloudplane.org/incident/197967#b23dfd6063f51ed5fb8fe6d3bf392013e4081243be981b2c533e8c2e27f5f5c5 Our provider is performing maintenance on one of their servers, which will require the migration of a small number of instances to a different node. Expected downtime is 5 minutes or less. https://status.cloudplane.org/incident/197967 Thu, 27 Apr 2023 12:15:00 -0000 https://status.cloudplane.org/incident/197967#b23dfd6063f51ed5fb8fe6d3bf392013e4081243be981b2c533e8c2e27f5f5c5 Our provider is performing maintenance on one of their servers, which will require the migration of a small number of instances to a different node. Expected downtime is 5 minutes or less. https://status.cloudplane.org/incident/200578 Wed, 26 Apr 2023 14:17:00 -0000 https://status.cloudplane.org/incident/200578#dd4769904785149e205483e094e44498d548a86f738b43c5c64489847358417a We have detected an issue with a malfunctioning proxy server. Normal operation has been restored. https://status.cloudplane.org/incident/197966 Wed, 26 Apr 2023 12:30:00 +0000 https://status.cloudplane.org/incident/197966#3c3b9e923b9b52deb3a1eb27bf20b3f796674da3913db47abf736a43b764cc3a Maintenance completed https://status.cloudplane.org/incident/200578 Wed, 26 Apr 2023 11:00:00 -0000 https://status.cloudplane.org/incident/200578#325c3e2eb84ea2f6158f8417a3d0f65b25109d12a3473cc29f7c2f861c764cbc We have detected connection issues since our scheduled maintenance and are investigating the issue. https://status.cloudplane.org/incident/197966 Wed, 26 Apr 2023 03:00:00 -0000 https://status.cloudplane.org/incident/197966#0c576ecba42964a1f1eec268a231de53d2460774557c465852201746f1cc38e7 Our provider is performing maintenance on a few of their servers, which will require the migration of a number of instances to a different node. Expected downtime is 5 minutes or less. https://status.cloudplane.org/incident/166827 Sun, 22 Jan 2023 08:00:00 +0000 https://status.cloudplane.org/incident/166827#e779ea08ab3edfbae27f34e5b8f5f4c12d1feb073782ee0261d24d30c41475e6 Maintenance completed https://status.cloudplane.org/incident/166827 Sun, 22 Jan 2023 07:00:00 -0000 https://status.cloudplane.org/incident/166827#1d587fac218ff2eee8cbc7bb36a859149c357fc3248bf3a56068c3e0a10af3af We are deploying a security update for Redis. Some apps may see a brief disruption lasting approximately 1 minute. https://status.cloudplane.org/incident/146051 Tue, 20 Dec 2022 15:00:00 +0000 https://status.cloudplane.org/incident/146051#281959b8e16408cf4b929f24f128669a06426cd3b205629c6c80e7cad996f111 Maintenance completed https://status.cloudplane.org/incident/146051 Tue, 20 Dec 2022 11:00:00 -0000 https://status.cloudplane.org/incident/146051#96aedfa15085608f42f75f3727055a92becffbebfd45967ddbb01965a100cf4a We are performing maintenance and deploying updates. Disruptions may occur. https://status.cloudplane.org/incident/142503 Sat, 10 Dec 2022 12:28:04 -0000 https://status.cloudplane.org/incident/142503#f1e3ba0d1d073b195e77e10f3ce39dd1667c2e0d9bf91e4f5de230583df4cb00 We are deploying several changes to improve stability. There may be brief disruptions during this rollout. https://status.cloudplane.org/incident/142503 Sat, 10 Dec 2022 12:00:00 +0000 https://status.cloudplane.org/incident/142503#c98d8caf46ee5b077d428c6ab93a9e08648a8ac1a04792a7bc43f04c8fda99f6 Maintenance completed https://status.cloudplane.org/incident/142503 Sat, 10 Dec 2022 10:00:00 -0000 https://status.cloudplane.org/incident/142503#f1e3ba0d1d073b195e77e10f3ce39dd1667c2e0d9bf91e4f5de230583df4cb00 We are deploying several changes to improve stability. There may be brief disruptions during this rollout. https://status.cloudplane.org/incident/142023 Tue, 06 Dec 2022 08:04:00 -0000 https://status.cloudplane.org/incident/142023#8de3f1093c9e3713ea93d26a69077ebea5be7558971e43e2527d948a240170b6 The issue has been resolved. https://status.cloudplane.org/incident/142023 Mon, 05 Dec 2022 22:38:00 -0000 https://status.cloudplane.org/incident/142023#748ddec58ff796a8c59faa71381a06d3daae7d496335b6beb3097b5408be5103 We've received reports of applications not being available, and we are currently unable to access our cluster. We have raised an issue with our cloud provider and hope for resolution ASAP. Sorry for the inconvenience. https://status.cloudplane.org/incident/141495 Sun, 04 Dec 2022 01:56:00 -0000 https://status.cloudplane.org/incident/141495#10c48c8379b41518795ca14b6476f8b5fc50f06f95c34f49c80442f163b1650d Normal service has been restored, sorry for the inconvenience. https://status.cloudplane.org/incident/141495 Sun, 04 Dec 2022 01:07:00 -0000 https://status.cloudplane.org/incident/141495#23ec2782b210d9a63b6be36638b670fde908f10346055d193152abd8d46ae8df One of our databases is experiencing issues, which affects a number of applications. Investigation ongoing. https://status.cloudplane.org/incident/139714 Tue, 29 Nov 2022 09:26:33 +0000 https://status.cloudplane.org/incident/139714#4ef3b9f2a42e4cdc6d82afedc865e55989cceaf86c1117787799734380d976ee Maintenance completed https://status.cloudplane.org/incident/139714 Tue, 29 Nov 2022 09:26:33 -0000 https://status.cloudplane.org/incident/139714#03de33e7b31c2cb8a8baa11044fa09e5c21bf57447ea68e4acbcb117eedb25be We are turning off a few nodes that were found to be unsuitable due to the volume limit of our cloud provider. This may cause brief downtime (~1 min) on a small number of instances. https://status.cloudplane.org/incident/139714 Tue, 29 Nov 2022 07:00:00 -0000 https://status.cloudplane.org/incident/139714#03de33e7b31c2cb8a8baa11044fa09e5c21bf57447ea68e4acbcb117eedb25be We are turning off a few nodes that were found to be unsuitable due to the volume limit of our cloud provider. This may cause brief downtime (~1 min) on a small number of instances. https://status.cloudplane.org/incident/139024 Fri, 25 Nov 2022 06:50:00 -0000 https://status.cloudplane.org/incident/139024#e16de5e84c3a12f7339a27f56d1235fde4ca22bd7e887e5854baf635f3262a1b We have found the cause for the issue, our object storage was unable to keep up with the increase in users. Scaling up the object storage servers resolved the issue. https://status.cloudplane.org/incident/139024 Fri, 25 Nov 2022 01:02:00 -0000 https://status.cloudplane.org/incident/139024#240b5fc27d376f32f6b778ca3eaf2d29c688bc2e13d05b16a42d19dd98153dad We received reports of 503 errors for actions such as uploading images and searching. https://status.cloudplane.org/incident/137766 Mon, 21 Nov 2022 07:00:00 -0000 https://status.cloudplane.org/incident/137766#519b1d0e81717dc26af7db3ea89463244c6fce06df2f81084b584ffcd1d8e0e6 We are still awaiting resolution of our ticket, but it appears that storage provisioning is back to working normally. https://status.cloudplane.org/incident/137766 Mon, 21 Nov 2022 00:04:00 -0000 https://status.cloudplane.org/incident/137766#be469cce87183f59e6b39eb4e0604739b05021401572937673cbdf4b8b5427e8 Some object storage endpoints of our infrastructure provider are currently unavailable or malfunctioning. This may prevent new applications from being provisioned. We have raised a ticket with them and will update you as soon as we know more. https://status.cloudplane.org/incident/137540 Sun, 20 Nov 2022 00:47:00 -0000 https://status.cloudplane.org/incident/137540#77bba13bbd5cb1a86f6cbbe3fdc84715c544f947d180e75d8464f327d1d1f8fd The issue has been resolved, I apologize for the inconvenience. Some apps may have experienced downtime of approximately 10-30 minutes. https://status.cloudplane.org/incident/137540 Sun, 20 Nov 2022 00:07:00 -0000 https://status.cloudplane.org/incident/137540#e351c48b2eb4243dcdd34076457ede022978f23abe6580138e811e4bf85b23f4 We detected an issue with our database that affects a number of applications. https://status.cloudplane.org/incident/137409 Sat, 19 Nov 2022 08:53:00 -0000 https://status.cloudplane.org/incident/137409#36931ad37efc72b35625058c999a46e1b46d1088cf5ffd96798329d30fed12ad The issue has been resolved, Mastodon will resend all previously failed emails. We found no problem with our SMTP configuration, but we believe that the issue is related to ongoing issues at our hosting provider. We are still monitoring the situation, but for now, everything seems to be back to normal. https://status.cloudplane.org/incident/137409 Sat, 19 Nov 2022 07:23:00 -0000 https://status.cloudplane.org/incident/137409#042b0423058fc370229315b277cc34025cd961d111361244007c3577fe2c5eab We have received reports that emails are not being sent and are investigating the issue. https://status.cloudplane.org/incident/135380 Sun, 13 Nov 2022 18:30:00 -0000 https://status.cloudplane.org/incident/135380#9d60e8ce11aa2cff71f6868fe86bd77726d7e3ddfdcbc6b8bb23413ec2e2f16e We've published a post-mortem on our blog: https://cloudplane.org/blog/outage-post-mortem https://status.cloudplane.org/incident/135380 Sun, 13 Nov 2022 09:32:00 -0000 https://status.cloudplane.org/incident/135380#f6dff6ed0a3bde1fc356c9bce268666c00e0a9c252d1a505eafd54905fc7299f All apps have been restored. Apps using object storage (Mastodon) have been recovered in full, no data lost. The list of apps itself has been restored from a backup that was 20 hours old, so size and domain changes may need to be repeated. We were unable to fully restore one Gitea instance as all volumes including snapshots were deleted, the affected user will be contacted soon. I'll put up a post to describe why all of this happened soon. For now, I need some sleep. Please message me if you have any questions. https://status.cloudplane.org/incident/135380 Sun, 13 Nov 2022 03:21:00 -0000 https://status.cloudplane.org/incident/135380#19df065b8f832bea261bb504fc183e73d02c9e86e00ea5fe50c44200f034a9ad A small mistake during an Helm upgrade ended up deleting our entire production cluster. We do have backups and we should be able to restore all Mastodon instances without any data loss, but we are running into some challenges. I do hope to have everything back by Sunday evening. I am truly sorry for this event, we are a new provider and some areas are still under development. We have backups, but we haven't had time to actually test a worst-case situation like this one. So, unfortunately, things aren't as easy as I'd hoped. You will of course not be billed while your apps are down. More details will follow soon. https://status.cloudplane.org/incident/131019 Sun, 30 Oct 2022 09:44:00 -0000 https://status.cloudplane.org/incident/131019#34e549c30fa0e31ec563f3207c2d92f96037162d7ca2196d302fe4931ad80d71 All services operational. https://status.cloudplane.org/incident/131019 Sun, 30 Oct 2022 09:43:00 -0000 https://status.cloudplane.org/incident/131019#1bc3ebc03ba9b2334c539a1adc8930142e99f7c10fe9d5b58d8a220bdb1fdec1 At 1:19 (GMT), our system detected an issue with our object storage service. We began our investigation at 9:10 and deployed a fix soon after. Full service was restored by 9:40.