tag:status.eventive.org,2005:/historyEventive Status - Incident History2024-03-18T21:29:49-05:00Eventivetag:status.eventive.org,2005:Incident/201958222024-03-07T10:00:00-06:002024-03-08T12:23:26-06:00Delayed virtual content transcodes<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>10:00</var> CST</small><br><strong>Resolved</strong> - Partial outage - a webhook outage resulted in some successful transcodes not updating status in the dashboard, resulting in an extended delay in the content ingestion process. The issue has been resolved and all new uploads are processing as expected. For any assets still stuck in an extended (over 3 hour) VOD transcoding state, please remove the asset and try uploading again.</p>tag:status.eventive.org,2005:Incident/190592102023-11-08T09:58:37-06:002023-11-08T09:58:38-06:00Users unable to access Eventive<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>09:58</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>09:55</var> CST</small><br><strong>Monitoring</strong> - This issue was caused by an upstream service provider DNS outage. The incident has been resolved by the upstream provider and we are monitoring results.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>09:52</var> CST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.eventive.org,2005:Incident/184731152023-09-11T22:02:37-05:002023-09-11T22:02:37-05:00Virtual content transcode failures<p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>22:02</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>15:25</var> CDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Sep <var data-var='date'>11</var>, <var data-var='time'>15:22</var> CDT</small><br><strong>Identified</strong> - Our video encoding provider has reported an issue with cloud server architecture that is causing virtual content trancode failures. The cause has been identified and a fix is in progress. All affected uploads will have to be re-submitted once the fix is complete. Estimated resolution by 09:00 UTC on 12 September.</p>tag:status.eventive.org,2005:Incident/108952732022-08-23T21:07:30-05:002022-08-23T21:07:31-05:00Platform outage<p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>21:07</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>19:57</var> CDT</small><br><strong>Monitoring</strong> - We have implemented a temporary mitigation to bypass the Heroku DNS issues until they resolve the underlying incident. As a result of our temporary mitigation, the Eventive platform should now be accessible again.</p><p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>19:28</var> CDT</small><br><strong>Update</strong> - Heroku is still actively working to mitigate the issue. We will provide another update within 45 minutes.</p><p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>18:45</var> CDT</small><br><strong>Update</strong> - One of our service providers, Heroku, is currently experiencing a DNS-related incident that is preventing users from accessing the Eventive platform. They are actively working to mitigate the issue. We will provide another update within 45 minutes.</p><p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>18:21</var> CDT</small><br><strong>Investigating</strong> - The Eventive platform is currently inaccessible due to a networking related incident at one of our DNS providers. We will provide an update within 30 minutes.</p>tag:status.eventive.org,2005:Incident/94001092022-02-24T11:29:51-06:002022-02-24T11:29:52-06:00Eventive platform outage<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>11:29</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>11:14</var> CST</small><br><strong>Monitoring</strong> - At this time we are observing that accessibility to the Eventive platform is gradually being restored. Heroku has indicated that a gradual restoration is in progress. We will provide another update within 30 minutes.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>10:42</var> CST</small><br><strong>Update</strong> - One of our service providers, Heroku, is currently experiencing a widespread platform outage that is impacting accessibility of the Eventive platform. The Eventive platform may be inaccessible to users. We will provide an update within 30 minutes.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>10:39</var> CST</small><br><strong>Investigating</strong> - We are currently investigating an issue impacting access to the Eventive organizer dashboard. We will provide another update within 30 minutes.</p>tag:status.eventive.org,2005:Incident/93153422022-02-22T02:00:30-06:002022-02-22T02:00:30-06:00Virtual platform security infrastructure upgrade<p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>02:00</var> CST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>00:31</var> CST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>12:46</var> CST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance to upgrade important security infrastructure for the Eventive Advanced Antipiracy platform between 12:30-2:00 AM Central Time on Tuesday, Feb 22nd. Some users may be unable to initiate new viewing sessions for up to 1 hour during this window.</p>tag:status.eventive.org,2005:Incident/92963472022-02-12T12:19:28-06:002022-02-12T12:19:28-06:00Premium Livestream issues<p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>12:19</var> CST</small><br><strong>Resolved</strong> - We have resolved the issue impacting Premium Livestream playback. All functionality has been restored. We sincerely apologize for the service disruption, and we will conduct an extensive review to ensure such an issue does not occur again.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>11:59</var> CST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented. We will provide an update within 1 hour.</p><p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>23:22</var> CST</small><br><strong>Update</strong> - In collaboration with our primary premium livestreaming partner Microsoft Azure, we continue to investigate the issue impacting Premium Livestreams. At this time, Basic Livestreams are recommended to ensure this error scenario is avoided.</p><p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>21:21</var> CST</small><br><strong>Investigating</strong> - We are currently investigating an elevated rate of Premium Livestream failures. Failures may manifest as only a black screen visible (instead of expected content) in preview and/or playback mode. As a temporary remediation while we continue our investigation, we recommend utilizing Basic Livestreams, which do not appear impacted by this issue.</p>tag:status.eventive.org,2005:Incident/91924952022-01-29T16:37:29-06:002022-01-29T16:37:30-06:00Elevated rate of encoding failures<p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>16:37</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved. All affected assets are being automatically queued for re-encoding. No further action should be necessary. We apologize again for the inconvenience.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>15:38</var> CST</small><br><strong>Identified</strong> - An issue at an external vendor that is causing some new asset encodings to fail has been identified. The vendor is working towards a remediation now. We apologize for the inconvenience, and will provide another update within 3 hours.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>14:26</var> CST</small><br><strong>Update</strong> - At this time we are continuing to investigate the elevated rate of failures impacting new asset encodings on the Eventive platform. We will provide another update within 90 minutes.</p><p><small>Jan <var data-var='date'>29</var>, <var data-var='time'>13:32</var> CST</small><br><strong>Investigating</strong> - We are currently investigating an elevated rate of encoding failures for newly uploaded assets. At this time, please do not retry failed uploads. We will provide an update within 60 minutes.</p>tag:status.eventive.org,2005:Incident/87861282021-12-07T18:05:53-06:002021-12-07T18:05:53-06:00Return of elevated DRM license key delivery failures<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>18:05</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>18:05</var> CST</small><br><strong>Monitoring</strong> - The incident has been resolved. Playback success has returned to normal levels, and AWS has confirmed a resolution of the underlying networking issue.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>15:17</var> CST</small><br><strong>Investigating</strong> - We are again observing an increase in DRM license key delivery failures. A current widespread Amazon Web Services outage impacting platforms including Disney+, Netflix, and others, is also affecting the Eventive DRM licensing system. Viewers may be unable to view protected content. This incident will remain open until the issue is exhaustively confirmed resolved by AWS.</p>tag:status.eventive.org,2005:Incident/87851062021-12-07T14:21:24-06:002021-12-07T14:21:24-06:00DRM license key delivery failures<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>14:21</var> CST</small><br><strong>Resolved</strong> - A fix has been implemented by Amazon, and DRM playback success is returning to normal levels.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>13:27</var> CST</small><br><strong>Investigating</strong> - A current widespread Amazon Web Services outage impacting platforms including Disney+, Netflix, and others, is also affecting the Eventive DRM licensing system. Viewers may be unable to view protected content. We will provide another update soon.</p>tag:status.eventive.org,2005:Incident/66594132021-04-01T17:56:24-05:002021-04-01T17:56:25-05:00Video playback issue<p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>17:56</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>17:15</var> CDT</small><br><strong>Monitoring</strong> - Microsoft has implemented a fix and we are beginning to see traffic return to normal levels. Please try viewing content again. We will continue to monitor the resolution process.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>17:09</var> CDT</small><br><strong>Update</strong> - Microsoft continues to investigate the issue impacting Azure DNS. Although their primary status page is impacted by the issue, updates are available at https://status2.azure.com/ We will provide an additional update soon.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>17:08</var> CDT</small><br><strong>Update</strong> - Microsoft continues to investigate the issue impacting Azure DNS. Although their primary status page is impacted by the issue, updates are available at https://status2.azure.com/ We will provide an additional update soon.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>16:49</var> CDT</small><br><strong>Identified</strong> - A Microsoft Azure DNS outage affecting a number of top internet properties is currently preventing playback of some assets. Azure is investigating the incident. We will provide an update soon.</p>tag:status.eventive.org,2005:Incident/62494142021-02-11T19:53:43-06:002021-02-11T19:53:44-06:00Virtual platform hosting provider partial outage<p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>19:53</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved by our upstream provider.</p><p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>19:34</var> CST</small><br><strong>Monitoring</strong> - A mitigation has been put in place by our hosting provider and access should soon be fully restored for affected customers.</p><p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>19:13</var> CST</small><br><strong>Investigating</strong> - One of our hosting providers, Vercel, is currently experiencing a partial outage that may impact the accessibility of the Eventive Virtual screening platform. Trying another device, browser, or wireless network may improve accessibility. We will provide another update shortly. You can follow along for updates here: https://www.vercel-status.com/</p>tag:status.eventive.org,2005:Incident/61231852021-01-26T22:20:42-06:002021-01-26T22:20:43-06:00DRM license key delivery failures<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>22:20</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>22:15</var> CST</small><br><strong>Update</strong> - Playback has returned to normal levels. We are continuing to monitor for any further issues.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>22:02</var> CST</small><br><strong>Monitoring</strong> - A fix has been implemented and DRM-protected playback should now be restored for all customers and working properly. We will continue to monitor the resolution and will provide another update within an hour.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>21:34</var> CST</small><br><strong>Identified</strong> - The issue affecting DRM license key delivery has been identified and a fix is being implemented by Microsoft. We will provide another update soon.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>21:11</var> CST</small><br><strong>Investigating</strong> - Our primary DRM provider, Microsoft Azure, is currently experiencing an outage impacting DRM license key delivery. Viewers may be unable to view protected content. We will provide another update soon.</p>tag:status.eventive.org,2005:Incident/50397042020-09-08T01:05:30-05:002020-09-08T01:05:30-05:00Database migration<p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>01:05</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>01:01</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>23:00</var> CDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance to perform a database migration between 1:00-1:05 AM Central Time on Tuesday, Sept 8th. Some services will be unavailable during this window.</p>tag:status.eventive.org,2005:Incident/49945492020-09-01T11:24:21-05:002020-09-01T11:24:21-05:00Live chat issues<p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>11:24</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>11:18</var> CDT</small><br><strong>Monitoring</strong> - Our live chat provider has implemented a fix for this issue and we are monitoring results.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>11:07</var> CDT</small><br><strong>Identified</strong> - Our live chat provider, Intercom, is currently experiencing a partial outage that reduces our ability to provide live support. Please contact us via email, team@eventive.org, with any urgent issues. https://www.intercomstatus.com/</p>tag:status.eventive.org,2005:Incident/46444842020-07-17T17:02:40-05:002020-07-17T17:02:40-05:00Partial networking outage<p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>17:02</var> CDT</small><br><strong>Resolved</strong> - The incident has been resolved by Cloudflare. We will continue to monitor the situation.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>16:49</var> CDT</small><br><strong>Update</strong> - CloudFlare is implementing a fix for the issue. In our logs, we see that traffic has now returned to near-normal levels and it appears that ~5% of users are currently being affected. Data Centers impacted include: SJC, DFW, SEA, LAX, ORD, IAD, EWR, ATL, LHR, AMS, FRA, CDG.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>16:40</var> CDT</small><br><strong>Identified</strong> - Approximately 20% of Eventive users appear to be affected by the Cloudflare outage. We will continue to monitor the situation as Cloudflare works toward a resolution. Updates are available at https://www.cloudflarestatus.com/</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>16:36</var> CDT</small><br><strong>Investigating</strong> - Our DNS and DDoS attack protection provider, Cloudflare, is currently experiencing a partial outage that is affecting us and many other platforms across the internet. Access to the Eventive platform is affected.</p>tag:status.eventive.org,2005:Incident/40863432020-05-11T11:58:34-05:002020-05-11T11:58:34-05:00Payment processing issues<p><small>May <var data-var='date'>11</var>, <var data-var='time'>11:58</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>11:51</var> CDT</small><br><strong>Monitoring</strong> - Stripe has implemented a fix for the issue. We are monitoring the results.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>11:35</var> CDT</small><br><strong>Investigating</strong> - Our payment processor, Stripe, is currently reporting elevated error rates. https://status.stripe.com/</p>tag:status.eventive.org,2005:Incident/36950972020-03-16T15:22:17-05:002020-03-16T17:54:35-05:00Platform outage<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>15:22</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>15:00</var> CDT</small><br><strong>Monitoring</strong> - Operation appears to have been restored. We will continue to monitor the situation.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>14:51</var> CDT</small><br><strong>Investigating</strong> - We are again investigating increased error rates with our hosting provider, @heroku.</p>tag:status.eventive.org,2005:Incident/36941922020-03-16T11:39:36-05:002020-03-16T11:39:36-05:00Global outage<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>11:39</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>11:26</var> CDT</small><br><strong>Monitoring</strong> - Our hosting provider, @heroku, experienced a global outage (https://status.heroku.com/incidents/1973). They appear to be recovering now. We will continue to monitor the platform as it recovers.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>11:18</var> CDT</small><br><strong>Investigating</strong> - We are currently investigating a global outage that appears to be affecting many internet platforms.</p>tag:status.eventive.org,2005:Incident/26873112019-07-10T18:05:01-05:002019-07-10T18:05:01-05:00There is an issue with our payments provider<p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>18:05</var> CDT</small><br><strong>Resolved</strong> - Our payments provider has resolved the issue and full payments functionality has been restored.</p><p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>17:50</var> CDT</small><br><strong>Update</strong> - Our payments provider is working on resolving the issue. We are continuing to monitor the situation.</p><p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>17:31</var> CDT</small><br><strong>Identified</strong> - Our payments provider, Stripe, is currently experiencing a global outage that is preventing some payments from being completed successfully.</p><p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>17:29</var> CDT</small><br><strong>Investigating</strong> - We are currently investigating elevated failure rates for payments processed through Eventive.</p>