Incidents reported on status page for UrbanPiper https://status2.urbanpiper.com/ https://d1lppblt9t2x15.cloudfront.net/logos/44d75fab330f4641a505e00fc0350bdc.png https://status2.urbanpiper.com/ en https://status2.urbanpiper.com/incident/121747 Wed, 28 Sep 2022 04:29:06 -0000 https://status2.urbanpiper.com/incident/121747#952b71111c21610a60ff4aaf933236504d45a4ce8cc71285cc70036171d20f85 Merchant API Services recovered. https://status2.urbanpiper.com/incident/121747 Wed, 28 Sep 2022 04:26:07 -0000 https://status2.urbanpiper.com/incident/121747#2fae4df2e1b19818835123b84edd2919e7864210464e4d32caa3b72339b60ba5 Merchant API Services and Meraki API Services went down. https://status2.urbanpiper.com/incident/120433 Thu, 22 Sep 2022 19:09:38 -0000 https://status2.urbanpiper.com/incident/120433#bb9cc8065379c31775483f20ccbffaeb244c1a9cde4c6cfdd072baa89807b60f Atlas Portal recovered. https://status2.urbanpiper.com/incident/120433 Thu, 22 Sep 2022 18:56:40 -0000 https://status2.urbanpiper.com/incident/120433#776201d952b64a3f3aaff3b6f9bab0c8efbcf77936566831512098671fe5423d Atlas Portal went down. https://status2.urbanpiper.com/incident/113103 Sat, 27 Aug 2022 05:30:59 -0000 https://status2.urbanpiper.com/incident/113103#34c998abbb6d7d73f289df75ba440ff0a7c5138cf997ac765431a206b1b442f5 Atlas Portal recovered. https://status2.urbanpiper.com/incident/113103 Sat, 27 Aug 2022 05:09:49 -0000 https://status2.urbanpiper.com/incident/113103#a3e8047a55cc8eb77aa615d8aa91f03a9eaeef185bb0b8f878a496dd628663c2 Atlas Portal went down. https://status2.urbanpiper.com/incident/113103 Sat, 27 Aug 2022 05:07:57 -0000 https://status2.urbanpiper.com/incident/113103#35257d940968e21cef89c8aa666af044cf91f4a75ad4e88b4106bd93da2e978f Atlas Portal recovered. https://status2.urbanpiper.com/incident/113103 Sat, 27 Aug 2022 03:50:56 -0000 https://status2.urbanpiper.com/incident/113103#bc20b110567072835d8155a683128da0b03783d0acda967e5f067b68f1c77004 Atlas Portal went down. https://status2.urbanpiper.com/incident/113012 Fri, 26 Aug 2022 18:08:16 -0000 https://status2.urbanpiper.com/incident/113012#6d6876a4135460111aea6a30c135e30cbecb6a8e39e5ba9c8f186decdba9deca Atlas Portal recovered. https://status2.urbanpiper.com/incident/113012 Fri, 26 Aug 2022 17:57:41 -0000 https://status2.urbanpiper.com/incident/113012#93a2bd92456e0be0be57baed28a1b1fc304896576c5d3654b9b8e56ab31c1260 Atlas Portal went down. https://status2.urbanpiper.com/incident/110903 Fri, 19 Aug 2022 09:17:37 -0000 https://status2.urbanpiper.com/incident/110903#36e637a98b5a2760de68cdbd2c8d66be8755c14684a00f7a47d9ea2508dabe30 Atlas Portal recovered. https://status2.urbanpiper.com/incident/110903 Fri, 19 Aug 2022 09:11:26 -0000 https://status2.urbanpiper.com/incident/110903#0099ec5d6fa497601d63e01f382d3302e13939b73ab09c6f45e17ad0bb0da863 Atlas Portal went down. https://status2.urbanpiper.com/incident/110903 Fri, 19 Aug 2022 04:13:25 -0000 https://status2.urbanpiper.com/incident/110903#62b1f2fb7e313281f86a6ab3602728231736f532acc24779be3c2fe4447a30de Atlas Portal recovered. https://status2.urbanpiper.com/incident/110903 Fri, 19 Aug 2022 04:00:23 -0000 https://status2.urbanpiper.com/incident/110903#e54931e4b75014b1ac0f727a457dfd1b3a41fd227c7aafcb440a00032e1a1575 Atlas Portal went down. https://status2.urbanpiper.com/incident/110696 Thu, 18 Aug 2022 11:17:07 -0000 https://status2.urbanpiper.com/incident/110696#cc4b9034027c9a07f3e7d1a5ce7c12bdcb2af8a536edee227e77c6f6b4b5a1a2 Atlas Portal recovered. https://status2.urbanpiper.com/incident/110696 Thu, 18 Aug 2022 11:04:57 -0000 https://status2.urbanpiper.com/incident/110696#38d4fd080011c319954888ad42b3f291c7d419ae63acfbbc45a71b76a337f21a Atlas Portal went down. https://status2.urbanpiper.com/incident/110152 Tue, 16 Aug 2022 17:11:43 -0000 https://status2.urbanpiper.com/incident/110152#515bd2f49c49b25e536c80d010c656cc732f9848226dc2bc483e432612d408eb Merchant API Services recovered. https://status2.urbanpiper.com/incident/110152 Tue, 16 Aug 2022 17:05:17 -0000 https://status2.urbanpiper.com/incident/110152#94c9c6b3e493cd5556bf7b152ebc36d498f7a7ee371bc44d69ae4129fe8fd49e Merchant API Services and Meraki API Services went down. https://status2.urbanpiper.com/incident/108201 Tue, 09 Aug 2022 18:38:32 -0000 https://status2.urbanpiper.com/incident/108201#24750dd362592964a3130fabb3785d0a643f6a284538ee62109ea9321caaf65b Atlas Portal recovered. https://status2.urbanpiper.com/incident/108201 Tue, 09 Aug 2022 18:29:34 -0000 https://status2.urbanpiper.com/incident/108201#d703302b3d33eb7e4d75e293edc36faca39afc66bd6b3d877366eec20d539cbb Atlas Portal went down. https://status2.urbanpiper.com/incident/102182 Fri, 15 Jul 2022 07:09:46 -0000 https://status2.urbanpiper.com/incident/102182#27cc9c7efef34548ea5ddd6e79436852f09a8f31cd176d46e96bf802313218a8 Atlas Portal recovered. https://status2.urbanpiper.com/incident/102182 Fri, 15 Jul 2022 07:02:52 -0000 https://status2.urbanpiper.com/incident/102182#a40cd8b7e24fb63aee1b0f9ed64948457d5ad5c7808cf9494324214d40101557 Atlas Portal went down. https://status2.urbanpiper.com/incident/100748 Sat, 09 Jul 2022 10:25:34 -0000 https://status2.urbanpiper.com/incident/100748#55c3933c0d92b8e4947d5f5fcdf931780896fb48112f0f55061189112e1497b2 Atlas Portal recovered. https://status2.urbanpiper.com/incident/100748 Sat, 09 Jul 2022 10:19:06 -0000 https://status2.urbanpiper.com/incident/100748#81697018bbb7d688fa54e0c2bb70c4bbd559aaf06cc0f9a68159c6d3074ade69 Atlas Portal and Meraki API Services went down and Aggregator API Services recovered. https://status2.urbanpiper.com/incident/100748 Sat, 09 Jul 2022 10:19:06 -0000 https://status2.urbanpiper.com/incident/100748#81697018bbb7d688fa54e0c2bb70c4bbd559aaf06cc0f9a68159c6d3074ade69 Atlas Portal and Meraki API Services went down and Aggregator API Services recovered. https://status2.urbanpiper.com/incident/100748 Sat, 09 Jul 2022 10:06:53 -0000 https://status2.urbanpiper.com/incident/100748#cc144a564f61dfe7331a0ebafb42148500bd23ed6515759540d6441ed91a5237 Merchant API Services, Atlas Portal, and Meraki API Services recovered. https://status2.urbanpiper.com/incident/100748 Sat, 09 Jul 2022 10:06:53 -0000 https://status2.urbanpiper.com/incident/100748#cc144a564f61dfe7331a0ebafb42148500bd23ed6515759540d6441ed91a5237 Merchant API Services, Atlas Portal, and Meraki API Services recovered. https://status2.urbanpiper.com/incident/100748 Sat, 09 Jul 2022 09:47:30 -0000 https://status2.urbanpiper.com/incident/100748#07a43d06378d9e91b5e95d388d2517d45e34cecd7fddcbd9ad620d4fb05edffa Aggregator API Services, Merchant API Services, Atlas Portal, and 1 other resource went down. https://status2.urbanpiper.com/incident/100748 Sat, 09 Jul 2022 09:47:30 -0000 https://status2.urbanpiper.com/incident/100748#07a43d06378d9e91b5e95d388d2517d45e34cecd7fddcbd9ad620d4fb05edffa Aggregator API Services, Merchant API Services, Atlas Portal, and 1 other resource went down. https://status2.urbanpiper.com/incident/100748 Sat, 09 Jul 2022 09:47:30 -0000 https://status2.urbanpiper.com/incident/100748#07a43d06378d9e91b5e95d388d2517d45e34cecd7fddcbd9ad620d4fb05edffa Aggregator API Services, Merchant API Services, Atlas Portal, and 1 other resource went down. https://status2.urbanpiper.com/incident/96366 Tue, 21 Jun 2022 07:55:39 -0000 https://status2.urbanpiper.com/incident/96366#29e4672c0cde2d87648774617ff587f3349e864eac88502ad179cc2a74e255d8 Atlas Portal recovered. https://status2.urbanpiper.com/incident/96366 Tue, 21 Jun 2022 06:39:30 -0000 https://status2.urbanpiper.com/incident/96366#b1e8350af44fc4d118828bba465b9dbeea4ae4bb12406f44af9bd56f2236349e Atlas Portal went down. https://status2.urbanpiper.com/incident/92067 Thu, 02 Jun 2022 21:08:34 -0000 https://status2.urbanpiper.com/incident/92067#0c690232cc6b8838b983df3776c48be1c95b8bad1ba46905eb3dac7d7b754987 Persistence availability recovered. https://status2.urbanpiper.com/incident/92067 Thu, 02 Jun 2022 20:52:37 -0000 https://status2.urbanpiper.com/incident/92067#375fd48d6a88c3b4f247f77e8cead337c66e93b177c32ffafad509acc00c9b64 Persistence availability went down. https://status2.urbanpiper.com/incident/92067 Thu, 02 Jun 2022 12:11:18 -0000 https://status2.urbanpiper.com/incident/92067#bb34f88bf166065448285beac4f87673d45b0d0f5b71ca663bb06f0a3ac6e9da Atlas Portal recovered. https://status2.urbanpiper.com/incident/92067 Thu, 02 Jun 2022 12:05:00 -0000 https://status2.urbanpiper.com/incident/92067#56c8fb9349d4b60e712d811ab1b2c573e1cb5179d3a547eb9e085dac19930556 Atlas Portal went down. https://status2.urbanpiper.com/incident/89261 Fri, 20 May 2022 09:43:57 -0000 https://status2.urbanpiper.com/incident/89261#e287abc6a0114bb58d6c122aa0d79beec4538d6d5a8796c5744789d04ef64c09 Atlas Portal recovered. https://status2.urbanpiper.com/incident/89261 Fri, 20 May 2022 09:37:38 -0000 https://status2.urbanpiper.com/incident/89261#6a650fc626788e8d7a9be5bcd46f0a7c74e46d216360a1f8096ecbfcceb819eb Atlas Portal went down. https://status2.urbanpiper.com/incident/85528 Tue, 03 May 2022 10:26:00 -0000 https://status2.urbanpiper.com/incident/85528#a6bdbbf0a9a24d34304c2cebc81d96ba84766bd4c23a7d73614372327010effa # Post-mortem At 1310 hrs due to a manual switch in network configuration, the central RabbitMQ server went offline for a duration of 4-5 mins. The situation was identified immediately and corrective measures were taken to divert the traffic to the standby queue. In the intervening period, since the API servers couldn't establish connection to the queue, their CPU's got blocked on network I/O. As soon as the standby queue kicked in, the situation auto-recovered in a few seconds. Corrective actions - it was found that the primary queue server had no termination protection associated with it, which is an incorrect setup for any critical piece of infra. - a lot of similarly named servers were also found to be in the _stopped_ state. The cleanup of these servers ended up triggering the issue. These servers have now been removed. https://status2.urbanpiper.com/incident/85528 Tue, 03 May 2022 07:51:20 -0000 https://status2.urbanpiper.com/incident/85528#5abfba9d4b0a91473264e458d53d24714a02794acb64437f637357a30669a11f Aggregator API Services recovered. https://status2.urbanpiper.com/incident/85528 Tue, 03 May 2022 07:43:51 -0000 https://status2.urbanpiper.com/incident/85528#807553fa0d34b47915f23a758b3f5004e3c362c33aa37ecd19dce76609eb6cd8 Aggregator API Services went down. https://status2.urbanpiper.com/incident/85528 Tue, 03 May 2022 06:45:18 -0000 https://status2.urbanpiper.com/incident/85528#598824073ebf7c87d9246ebdd286c389f42c76baf944e5601ac03e693dcd33eb Aggregator API Services recovered. https://status2.urbanpiper.com/incident/85528 Tue, 03 May 2022 06:44:48 -0000 https://status2.urbanpiper.com/incident/85528#73c4cb43aa2c45c2aa8944c60bdd0d17db2f342dd4625a8dbe11d40cff32bfbb Merchant API Services recovered. https://status2.urbanpiper.com/incident/85528 Tue, 03 May 2022 06:40:15 -0000 https://status2.urbanpiper.com/incident/85528#1f63b05600bccdb26dbc97fdb0f0f92712942c0473d08c7942ecfe8c55fde96b Merchant API Services went down. https://status2.urbanpiper.com/incident/85528 Tue, 03 May 2022 06:39:47 -0000 https://status2.urbanpiper.com/incident/85528#44849c3234e3897cc6d9bdd9df690726f88ea1e1073a870b7fce1d5f1eae7a7f Aggregator API Services went down.