Incidents reported on status page for Crowdpurr https://status.crowdpurr.com/ https://d1lppblt9t2x15.cloudfront.net/logos/0c6e4a46d471aa297ff98fe03b62a548.png https://status.crowdpurr.com/ en https://status.crowdpurr.com/incident/610674 Sat, 28 Jun 2025 14:23:07 -0000 https://status.crowdpurr.com/incident/610674#3ec0bebbf9ad030ef5bb255614304d6abc6ea1de6c5db0fb7cf9f11783ed15d0 Community Forum recovered. https://status.crowdpurr.com/incident/610674 Sat, 28 Jun 2025 14:19:35 -0000 https://status.crowdpurr.com/incident/610674#7fcfe2f65da1ada46e09e8b865bf0800b03105f4b50823ae213f887af37bfaa3 Community Forum went down. https://status.crowdpurr.com/incident/610064 Fri, 27 Jun 2025 07:32:20 -0000 https://status.crowdpurr.com/incident/610064#1ac946996e9d62541dc7eea68c69d82d243de7735a0a634674cd88872bbd27f4 Community Forum recovered. https://status.crowdpurr.com/incident/610064 Fri, 27 Jun 2025 07:28:53 -0000 https://status.crowdpurr.com/incident/610064#1b706dccd379bc6f9570ea55abff93bc178d1d2fd045e258b993816ebc7b32cc Community Forum went down. https://status.crowdpurr.com/incident/608552 Tue, 24 Jun 2025 23:54:38 -0000 https://status.crowdpurr.com/incident/608552#153d0e64f13209328eccd00cec74b7113d1e4ec59464c0a20041d5f6521eb459 Community Forum recovered. https://status.crowdpurr.com/incident/608552 Tue, 24 Jun 2025 23:50:40 -0000 https://status.crowdpurr.com/incident/608552#f3bc7973ab3c7cce136b93640dee1aea171c0fe36f9cf0512b8853638ad95b30 Community Forum went down. https://status.crowdpurr.com/incident/603742 Mon, 16 Jun 2025 07:29:03 -0000 https://status.crowdpurr.com/incident/603742#d05ccf1cc65cc17c07f6dce2aab7e989958d5297d63ac23fa4fa222926b5529c Community Forum recovered. https://status.crowdpurr.com/incident/603742 Mon, 16 Jun 2025 07:25:34 -0000 https://status.crowdpurr.com/incident/603742#c3f7a32f6853cf4027f3facec39df97e8241bb848e459202c4f292652a18cdd6 Community Forum went down. https://status.crowdpurr.com/incident/603063 Sat, 14 Jun 2025 18:44:38 -0000 https://status.crowdpurr.com/incident/603063#22678a1fb3be416c4d66281b8cc4a37dbc22c00a4ee8f2c2b5283acf0433ed98 Community Forum recovered. https://status.crowdpurr.com/incident/603063 Sat, 14 Jun 2025 18:40:53 -0000 https://status.crowdpurr.com/incident/603063#e8d906d9ffa42a85c3206639d27c792b2f3c0170ad1687b5e51d9a7407b1d1b2 Community Forum went down. https://status.crowdpurr.com/incident/561807 Tue, 13 May 2025 05:10:18 -0000 https://status.crowdpurr.com/incident/561807#c9da9cb9d91d08f2a37be176dbea8f66ead621debe68c501407a17b88effe25d Community Forum recovered. https://status.crowdpurr.com/incident/561807 Tue, 13 May 2025 05:02:09 -0000 https://status.crowdpurr.com/incident/561807#cf2e28401a9e72779becc462cf5a7d6301fe3d8b46b968b3930b0163f9c0cf57 Community Forum went down. https://status.crowdpurr.com/incident/534393 Tue, 25 Mar 2025 17:59:34 -0000 https://status.crowdpurr.com/incident/534393#d04562db002b9d36bdb61c39f02397d19b642c37f827aefea94a88f49f22fe26 Community Forum recovered. https://status.crowdpurr.com/incident/534393 Tue, 25 Mar 2025 17:51:57 -0000 https://status.crowdpurr.com/incident/534393#99ade01380cfbe3745a858fa2be89b6f09a76f3a1c92c8ba415d415233bfe708 Community Forum went down. https://status.crowdpurr.com/incident/533408 Mon, 24 Mar 2025 05:26:39 -0000 https://status.crowdpurr.com/incident/533408#c48c2c604302ace2db7839211320f778fadcf1996fc379eae13a31d16b27a2f7 Help Center recovered. https://status.crowdpurr.com/incident/533408 Mon, 24 Mar 2025 05:16:38 -0000 https://status.crowdpurr.com/incident/533408#6785ca52a66cd08394d5f447c8ddf2e7e1a987e6606d6372d51d38bb6fd624e5 Help Center went down. https://status.crowdpurr.com/incident/521080 Sat, 01 Mar 2025 16:15:00 -0000 https://status.crowdpurr.com/incident/521080#11149b560e71850223a888f1c4295bef194a1becdddf4668099b0ed980ced5a0 The Crowdpurr Ingress server has been restored. Our team quickly responded to the outage and restored the service within seven minutes. This is yet another unusual outage for Crowdpurr's Ingress server which is the load balancer that spreads out all joining participants across our various Participant View application servers. Crowdpurr's application servers did not experience an issue. Our team is currently reviewing the incident. We have traced it to a software bug in a third party web server software that we use in our stack. It is not a Crowdpurr software issue. Our team is checking with the third party to see if they are aware of the issue. Our team has already updated the load balancer machine with a configuration that will automatically restart the machine if this issue happens again, thus drastically reducing any downtime. Lastly, after several issues with this particular service, we are reviewing the option of replacing it with with a more well-known industry standard web server software. While no incident involving downtime is acceptable, we are pleased that the service was only down for seven minutes compared to the previous incident. https://status.crowdpurr.com/incident/521080 Sat, 01 Mar 2025 16:04:00 -0000 https://status.crowdpurr.com/incident/521080#c3cac43ecdbc580d62fe8ffc9e78fd10901e1487743c7ad878b2bb96cc314eb3 The Crowdpurr Ingress server is currently experiencing an outage. Our team was notified and are currently diagnosing the issue. https://status.crowdpurr.com/incident/504572 Fri, 17 Jan 2025 01:30:00 -0000 https://status.crowdpurr.com/incident/504572#aa2f4c7aa5103871af627d122ac486791a593d965cca32a0351cd967ce573ede Last night around 7:30PM EST we experienced a forty-five minute outage of the Crowdpurr Ingress service of Crowdpurr. Our team became aware of the issue around 7:45PM EST. The issue was resolved around 8:15PM EST. The root cause of the outage was due to an Amazon Web Services (AWS) hardware failure in the Participant View's load balancer server (Crowdpurr Ingress). This is the server that balances all the thousands of incoming participants evenly and sends them on to our underlying set of application servers (which were running fine). This is why the Participant View completely failed to load. Historically, when a server crashes or hangs it’s usually due to a Crowdpurr software/bug issue. This failure was truly due to the underlying cloud hardware provided by AWS. While these AWS hardware incidents are very rare, they do happen. Cloud servers are meant to be monitored and replaced quickly. Once we knew the server was down, our team assembled, and it was quickly replaced and service was restored. In order to quickly know when any part of the Crowdpurr server ecosystem goes down, we make use of a third-party monitoring service that checks on every server in the system every sixty seconds from eighteen different locations all over the world. If any server goes down for even a minute, our team is notified via email and SMS text to emergency inboxes reserved for critical service events. A secondary issue also occurred in alerting our team of the server issue. Our monitoring service uses an SMS texting service that is experiencing issues with west Los Angeles phone numbers on the T-Mobile network. Our phone number for receiving alerts happens to be a west Los Angeles phone number on T-Mobile. Consequently, our Engineering Team didn’t receive the immediate SMS text messages about the app’s health which is critical in responding after hours. We did receive alerts via email and our Customer Support Team was online and quickly manually alerted our Engineering Team of the issue. Once our Engineering Team knew about the outage, we successfully replaced the Crowdpurr Ingress server. We also have dropped the above monitoring service vendor and started up with a new vendor that is much more powerful and offers several additional features. You can even subscribe to our status page now to be updated about incidents in real-time. Additionally, we are adding several extra layers to the design of our notification and response procedures to make sure we don’t have a single point of failure for notifications when Crowdpurr experiences an issue. This includes adding more email and phone numbers inboxes, and a second, redundant monitoring system in the event one doesn’t work. Additionally, we added a redundant load balancing server that we can quickly switch to in the event something like this happens again. We understand Crowdpurr is a software for live events and it’s of utmost importance that it work for every rehearsal and every production live event. We take application uptime very seriously. If you’re a seasoned Crowdpurr user, you know that the application rarely has an outage. However, due to the nature of Internet base software, outages do sometimes occur. This has been the most severe outage for Crowdpurr in about five years. That's how rare downtime is for us. However, even with a forty-five minute outage, Crowdpurr is still far and well above 99.9% uptime over the last twelve months. But, with live events, we’re aware that even 99.9% uptime is not good enough. We will always strive for 100% uptime. Please accept our deepest apologies and always know that we are learning and working hard to prevent something like this from happening again. We deeply respect and appreciate your patronage. You pay for our service, you expect it to work. Please let us know how we can make this right. https://status.crowdpurr.com/incident/504572 Fri, 17 Jan 2025 01:30:00 -0000 https://status.crowdpurr.com/incident/504572#aa2f4c7aa5103871af627d122ac486791a593d965cca32a0351cd967ce573ede Last night around 7:30PM EST we experienced a forty-five minute outage of the Crowdpurr Ingress service of Crowdpurr. Our team became aware of the issue around 7:45PM EST. The issue was resolved around 8:15PM EST. The root cause of the outage was due to an Amazon Web Services (AWS) hardware failure in the Participant View's load balancer server (Crowdpurr Ingress). This is the server that balances all the thousands of incoming participants evenly and sends them on to our underlying set of application servers (which were running fine). This is why the Participant View completely failed to load. Historically, when a server crashes or hangs it’s usually due to a Crowdpurr software/bug issue. This failure was truly due to the underlying cloud hardware provided by AWS. While these AWS hardware incidents are very rare, they do happen. Cloud servers are meant to be monitored and replaced quickly. Once we knew the server was down, our team assembled, and it was quickly replaced and service was restored. In order to quickly know when any part of the Crowdpurr server ecosystem goes down, we make use of a third-party monitoring service that checks on every server in the system every sixty seconds from eighteen different locations all over the world. If any server goes down for even a minute, our team is notified via email and SMS text to emergency inboxes reserved for critical service events. A secondary issue also occurred in alerting our team of the server issue. Our monitoring service uses an SMS texting service that is experiencing issues with west Los Angeles phone numbers on the T-Mobile network. Our phone number for receiving alerts happens to be a west Los Angeles phone number on T-Mobile. Consequently, our Engineering Team didn’t receive the immediate SMS text messages about the app’s health which is critical in responding after hours. We did receive alerts via email and our Customer Support Team was online and quickly manually alerted our Engineering Team of the issue. Once our Engineering Team knew about the outage, we successfully replaced the Crowdpurr Ingress server. We also have dropped the above monitoring service vendor and started up with a new vendor that is much more powerful and offers several additional features. You can even subscribe to our status page now to be updated about incidents in real-time. Additionally, we are adding several extra layers to the design of our notification and response procedures to make sure we don’t have a single point of failure for notifications when Crowdpurr experiences an issue. This includes adding more email and phone numbers inboxes, and a second, redundant monitoring system in the event one doesn’t work. Additionally, we added a redundant load balancing server that we can quickly switch to in the event something like this happens again. We understand Crowdpurr is a software for live events and it’s of utmost importance that it work for every rehearsal and every production live event. We take application uptime very seriously. If you’re a seasoned Crowdpurr user, you know that the application rarely has an outage. However, due to the nature of Internet base software, outages do sometimes occur. This has been the most severe outage for Crowdpurr in about five years. That's how rare downtime is for us. However, even with a forty-five minute outage, Crowdpurr is still far and well above 99.9% uptime over the last twelve months. But, with live events, we’re aware that even 99.9% uptime is not good enough. We will always strive for 100% uptime. Please accept our deepest apologies and always know that we are learning and working hard to prevent something like this from happening again. We deeply respect and appreciate your patronage. You pay for our service, you expect it to work. Please let us know how we can make this right. https://status.crowdpurr.com/incident/504572 Fri, 17 Jan 2025 00:30:00 -0000 https://status.crowdpurr.com/incident/504572#71f6000e97d9233f4e12bdff6507e30f987c4e7ad68f257916aaca1e24f1bf99 At 7:32PM EST the Participant View experienced an outage. Our team was able to restore the service around forty minutes later at 8:15PM EST. We corrected the problematic server and are investigating the cause. https://status.crowdpurr.com/incident/504572 Fri, 17 Jan 2025 00:30:00 -0000 https://status.crowdpurr.com/incident/504572#71f6000e97d9233f4e12bdff6507e30f987c4e7ad68f257916aaca1e24f1bf99 At 7:32PM EST the Participant View experienced an outage. Our team was able to restore the service around forty minutes later at 8:15PM EST. We corrected the problematic server and are investigating the cause. https://status.crowdpurr.com/incident/504576 Wed, 17 Jul 2024 05:05:00 -0000 https://status.crowdpurr.com/incident/504576#fd0f4478ca0f6216778628cfcfbd78aae056f9a0b7b047aa1761a59f2f1c1145 On Tuesday July 16th around 8:45PM to 1AM EST, a trivia game was ran in an edge-case configuration that caused one of Crowdpurr’s micro-services, our sync server that is responsible for updating Participant Views and Presentation Views with host actions triggered on the Experience Dashboard, to overload and become slow and unresponsive. A fairly recent change to our codebase caused an unanticipated load on the microservice which normally and historically has never overloaded and has always been run with minimal resources. We have updated the microservice with twice the processing power, six times the memory, and ten times the throughput allowance. This will resolve the issue moving forward. Please accept our sincere apologies for your disrupted events and frustration experienced. We strive to deliver a smooth experience that can always be relied upon 100% of the time. As we know your live events depend on it. https://status.crowdpurr.com/incident/504576 Wed, 17 Jul 2024 05:05:00 -0000 https://status.crowdpurr.com/incident/504576#fd0f4478ca0f6216778628cfcfbd78aae056f9a0b7b047aa1761a59f2f1c1145 On Tuesday July 16th around 8:45PM to 1AM EST, a trivia game was ran in an edge-case configuration that caused one of Crowdpurr’s micro-services, our sync server that is responsible for updating Participant Views and Presentation Views with host actions triggered on the Experience Dashboard, to overload and become slow and unresponsive. A fairly recent change to our codebase caused an unanticipated load on the microservice which normally and historically has never overloaded and has always been run with minimal resources. We have updated the microservice with twice the processing power, six times the memory, and ten times the throughput allowance. This will resolve the issue moving forward. Please accept our sincere apologies for your disrupted events and frustration experienced. We strive to deliver a smooth experience that can always be relied upon 100% of the time. As we know your live events depend on it. https://status.crowdpurr.com/incident/504576 Wed, 17 Jul 2024 05:05:00 -0000 https://status.crowdpurr.com/incident/504576#fd0f4478ca0f6216778628cfcfbd78aae056f9a0b7b047aa1761a59f2f1c1145 On Tuesday July 16th around 8:45PM to 1AM EST, a trivia game was ran in an edge-case configuration that caused one of Crowdpurr’s micro-services, our sync server that is responsible for updating Participant Views and Presentation Views with host actions triggered on the Experience Dashboard, to overload and become slow and unresponsive. A fairly recent change to our codebase caused an unanticipated load on the microservice which normally and historically has never overloaded and has always been run with minimal resources. We have updated the microservice with twice the processing power, six times the memory, and ten times the throughput allowance. This will resolve the issue moving forward. Please accept our sincere apologies for your disrupted events and frustration experienced. We strive to deliver a smooth experience that can always be relied upon 100% of the time. As we know your live events depend on it. https://status.crowdpurr.com/incident/504576 Wed, 17 Jul 2024 00:45:00 -0000 https://status.crowdpurr.com/incident/504576#7a1e1712b90ba1727cfa7723bd81f3a3cd739c55a9669ff513d4252466228082 Currently the Participant View is updating very slowly or not at all when actions are triggered on the Experience Dashboard. Our team is investigating the issue. https://status.crowdpurr.com/incident/504576 Wed, 17 Jul 2024 00:45:00 -0000 https://status.crowdpurr.com/incident/504576#7a1e1712b90ba1727cfa7723bd81f3a3cd739c55a9669ff513d4252466228082 Currently the Participant View is updating very slowly or not at all when actions are triggered on the Experience Dashboard. Our team is investigating the issue. https://status.crowdpurr.com/incident/504576 Wed, 17 Jul 2024 00:45:00 -0000 https://status.crowdpurr.com/incident/504576#7a1e1712b90ba1727cfa7723bd81f3a3cd739c55a9669ff513d4252466228082 Currently the Participant View is updating very slowly or not at all when actions are triggered on the Experience Dashboard. Our team is investigating the issue.