tag:status.pactsafe.com,2005:/historyIronclad Clickwrap Status - Incident History2024-03-29T03:53:40-04:00Ironclad Clickwraptag:status.pactsafe.com,2005:Incident/166981072023-03-30T13:20:43-04:002023-03-30T13:20:43-04:00Activity API Service Disruption<p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>13:20</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>10:03</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>09:45</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Mar <var data-var='date'>30</var>, <var data-var='time'>09:34</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating an issue with our Activity API</p>tag:status.pactsafe.com,2005:Incident/151961352022-12-14T15:01:15-05:002022-12-14T15:01:15-05:00Activity API currently experiencing a disruption<p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>15:01</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>14:20</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>14:16</var> EST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>14:15</var> EST</small><br><strong>Investigating</strong> - We're currently experiencing errors with our Activity API. We are currently investigating and working to resolve ASAP.</p>tag:status.pactsafe.com,2005:Incident/114285662022-10-04T18:10:52-04:002022-10-04T18:10:52-04:00Issue with async record generation for Clickwrap for CLM<p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>18:10</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>17:42</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>17:05</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>16:06</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating issues with async record generation for Clickwrap for CLM. We are investigating the cause and will update accordingly.</p>tag:status.pactsafe.com,2005:Incident/113319642022-09-30T18:22:03-04:002022-09-30T18:22:03-04:00Issue with available features in Ironclad Clickwrap<p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>18:22</var> EDT</small><br><strong>Resolved</strong> - All of the impacted functionality has been restored, and the system should again be functioning as expected!</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>16:48</var> EDT</small><br><strong>Update</strong> - We have identified an issue where certain customers may have had certain system functionality--such as SSO and Legacy Sign capabilities--restored to a previous state that has affected their access to said features. We've identified the root cause and are actively working on a fix.</p><p><small>Sep <var data-var='date'>30</var>, <var data-var='time'>16:48</var> EDT</small><br><strong>Identified</strong> - We've recently identified an issue where a subset of customers had a feature set inadvertently modified when system changes were being made today.</p>tag:status.pactsafe.com,2005:Incident/110268872022-08-31T15:13:08-04:002022-08-31T15:13:08-04:00Error loading the new Clickwrap experience<p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>15:13</var> EDT</small><br><strong>Resolved</strong> - This issue has been resolved, but we are implementing changes over the course of the day to ensure we prevent this from happening going forward.</p><p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>14:15</var> EDT</small><br><strong>Monitoring</strong> - We have restored functionality to our clickwrap.pactsafe.com service, and it should be now up and available. We are continuing to proactively monitor and will post an update here shortly once these changes are confirmed to be effective.</p><p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>13:37</var> EDT</small><br><strong>Identified</strong> - Update: our underlying infrastructure for the application was updated this morning and resulted in some unexpected side effects that broke app functionality. We are currently working on a few paths to resolution, including rolling back the config or fixing the current cluster or creating a new cluster in the same subnet.</p><p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>13:06</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>13:06</var> EDT</small><br><strong>Investigating</strong> - We are investigating issues with our Clickwrap application at clickwrap.pactsafe.com. Please use Clickwrap Classic at https://app.pactsafe.com for the time being.</p>tag:status.pactsafe.com,2005:Incident/108063402022-08-11T13:10:03-04:002022-08-11T13:10:03-04:00Issue creating contract templates and new clickwrap groups<p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>13:10</var> EDT</small><br><strong>Resolved</strong> - Everything is functioning as normal after the deployment. This issue is resolved.</p><p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>13:03</var> EDT</small><br><strong>Monitoring</strong> - The fix has been implemented and deployed. Currently monitoring to ensure this issue has been resolved.</p><p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>12:58</var> EDT</small><br><strong>Update</strong> - We have identified the issue and are currently deploying a fix to all regions and servers.</p><p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>12:41</var> EDT</small><br><strong>Identified</strong> - We believe we have a fix that is being reviewed. Current impact is the creation of: groups, snapshot locations, contract templates, and potentially new versions of contracts.</p><p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>12:35</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating an issue related to creating new contract templates and new clickwrap groups via our application and REST API. We believe we have identified the root of the issue and are exploring a fix. Acceptance tracking and Legal Centers are NOT affected.</p>tag:status.pactsafe.com,2005:Incident/104142022022-06-27T12:27:41-04:002022-06-27T12:27:41-04:00*.pactsafe.io Legal Center domains displaying "insecure" alert<p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>12:27</var> EDT</small><br><strong>Resolved</strong> - We believe all Legal Center sites to be resolved at this point. If you continue to have any more issues, please reach out to Ironclad Customer Support.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>12:26</var> EDT</small><br><strong>Update</strong> - We have redeployed all Legal Centers with the fix in place and are currently monitoring all customer Legal Centers coming back online.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>12:14</var> EDT</small><br><strong>Update</strong> - Around 75% of customers are back online, and we are still working to correct the remainder of Legal Center domains and should have the rest online shortly.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>11:43</var> EDT</small><br><strong>Update</strong> - We've updated the incident to reflect that some custom domains are also affected.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>11:26</var> EDT</small><br><strong>Monitoring</strong> - It appears that old configurations were deployed to our CDN distributions, which in many cases has resulted in expired SSL certs being assigned. We are rolling out a fix to update all Legal Centers affected currently.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>11:11</var> EDT</small><br><strong>Investigating</strong> - This morning, we noticed an issue that was preventing a number of Legal Centers hosted on pactsafe.io to show an alert about the connection not being private/secure. Our engineering team is currently investigating the issue and working to resolve.</p>tag:status.pactsafe.com,2005:Incident/88034982021-12-09T15:59:29-05:002021-12-09T15:59:47-05:00Increased response times on the Activity API<p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>15:59</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>15:59</var> EST</small><br><strong>Update</strong> - We have confirmed that the issue is resolved and response times are back to normal.</p><p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>14:19</var> EST</small><br><strong>Update</strong> - We are seeing improved performance, but continue to investigate to ensure performance has returned to normal.</p><p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>12:38</var> EST</small><br><strong>Investigating</strong> - We are currently seeing increased response times on some requests to the Activity API and are actively investigating to identify a root cause.</p>tag:status.pactsafe.com,2005:Incident/87881902021-12-07T20:38:47-05:002021-12-07T20:39:01-05:00Issue with notification and caching service for Activity API<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>20:38</var> EST</small><br><strong>Resolved</strong> - At approximately 3:24pm EST, Ironclad Clickwrap experienced an incident where the service we use to cache, process notifications, and deliver webhooks ran into an out of memory issue. The cluster was upgraded and service was restored at 8:05pm.</p>tag:status.pactsafe.com,2005:Incident/78340062021-08-24T14:52:04-04:002021-08-24T14:52:04-04:00Experiencing an intermittent issue on core PactSafe services<p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>14:52</var> EDT</small><br><strong>Resolved</strong> - We've narrowed this issue to a change that was implemented within the past 24 hours or so. This issue is resolved for now and we will be writing a post-mortem which we will be delivering to all customers to provide more insight into what happened with this incident, what was impacted, and steps we'll take to prevent this from happening in the future. Thank you for your patience and we apologize for the inconvenience caused by this issue.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>14:27</var> EDT</small><br><strong>Monitoring</strong> - We have implemented a change that we believe has resolved the issue. We're currently monitoring the situation to ensure that the issue is resolved for all customers across all locations and ISPs.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>14:10</var> EDT</small><br><strong>Update</strong> - Update: We are continuing to investigate this issue and will update again within the next 30 minutes.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>13:58</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>13:38</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating an issue that seems to be isolated on specific Internet Service Providers on the following domains:<br />- pactsafe.io<br />- app.pactsafe.com<br />- api.pactsafe.com</p>tag:status.pactsafe.com,2005:Incident/49449392020-08-26T14:04:56-04:002020-08-26T14:04:56-04:00Application dashboard seeing an unusual amount of errors<p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>14:04</var> EDT</small><br><strong>Resolved</strong> - Servers did restart and are now back to normal. We apologize for the brief issue with the service and hope it didn't cause a major inconvenience.</p><p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>14:01</var> EDT</small><br><strong>Monitoring</strong> - The application servers have restarted and are stabilizing. We will monitor briefly to ensure everything is back online.</p><p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>13:55</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Aug <var data-var='date'>26</var>, <var data-var='time'>13:54</var> EDT</small><br><strong>Investigating</strong> - We're currently investigating a large number of errors coming from our application dashboard. Logging into app.pactsafe.com is currently affected.</p>tag:status.pactsafe.com,2005:Incident/49286222020-08-24T20:30:09-04:002020-08-24T22:25:02-04:00Links to sign contracts resulting in security error<p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>20:30</var> EDT</small><br><strong>Resolved</strong> - Emails have been working for a few hours now. We will be reaching out to customers affected by the issue directly (between 3pm and 5pm EST) to resolve issues with links in emails during this time period.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>17:23</var> EDT</small><br><strong>Update</strong> - We have currently resolved all links going forward, but we are currently working to fix all links delivered between 3pm and 5pm EST. We'll mark this issue as "Resolved" once that fix is in place.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>17:23</var> EDT</small><br><strong>Monitoring</strong> - We have currently resolved all links going forward, but we are currently working to fix all links delivered between 3pm and 5pm EST. We'll mark this issue as "Resolved" once that fix is in place.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>17:22</var> EDT</small><br><strong>Identified</strong> - Emails between 3pm and 5pm EST today have been identified as problematic and raising this security issue when signing due to a change in DNS configuration between our email provider and PactSafe.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>17:19</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating an issue that's caused link redirects to interrupt the user signing with a security error. This is due to a configuration issue that was recently introduced with our email provider that we did not anticipate. We have resolved the issue for emails going forward, but we're now working to introduce a new function to retroactively redirect links that are currently failing.<br /><br />There is a workaround: To "Proceed" through the signing process, users will successfully be redirected to sign.</p>tag:status.pactsafe.com,2005:Incident/47265122020-07-28T14:26:26-04:002020-07-31T13:44:41-04:00Email sending delay (SendGrid)<p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>14:26</var> EDT</small><br><strong>Resolved</strong> - SendGrid has confirmed that they are now receiving and processing mail normally, which we have also confirmed in our reporting.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>13:39</var> EDT</small><br><strong>Monitoring</strong> - Emails delivery has resumed and should be functioning normally. A slight delay may still be possible between sending and delivery. We are still actively monitoring to confirm.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>12:48</var> EDT</small><br><strong>Investigating</strong> - The third party service used to deliver emails on behalf of PactSafe is currently experiencing an outage.</p>tag:status.pactsafe.com,2005:Incident/42584552020-06-02T11:22:29-04:002020-06-16T13:17:02-04:00Experiencing an issue with our Activity API and signing requests in PactSafe<p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>11:22</var> EDT</small><br><strong>Resolved</strong> - We have been monitoring and feel confident we’ve resolved this issue at this time. Within the next 48 hours we will be providing a post-mortem outlining more detail on what happened.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>11:21</var> EDT</small><br><strong>Update</strong> - We are fully operational but are still monitoring for ongoing issues. Our engineering and architecture team is working to ensure that we shore up any issues. At this time, we do not believe there was any data lost when processing acceptances during the outage. We'll be writing up a full post-mortem summarizing what the issue was, why it happened, and how we'll be improving our system architecture going forward.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>11:07</var> EDT</small><br><strong>Monitoring</strong> - Our application, REST APIs, and Activity APIs are starting to come back online and we are monitoring their performance.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>10:50</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate and isolate the nature of the issue.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>10:20</var> EDT</small><br><strong>Update</strong> - We have restarted a number of servers and are seeing a significant decrease in errors in our Activity API but are still seeing issues in our REST API.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>10:07</var> EDT</small><br><strong>Update</strong> - We are still continuing to see timeouts on both our Activity and REST APIs and are still investigating the issue.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>09:53</var> EDT</small><br><strong>Update</strong> - We are now seeing an increased volume of errors in our REST API as well.</p><p><small>Jun <var data-var='date'> 2</var>, <var data-var='time'>09:49</var> EDT</small><br><strong>Investigating</strong> - We are seeing an increased number of errors (504 Gateway Timeout) on our Activity API which is causing an outage with capturing acceptance of contracts. We are investigating with the highest priority.</p>tag:status.pactsafe.com,2005:Incident/40043302020-04-30T19:33:35-04:002020-04-30T19:33:41-04:00Salesforce integration is experiencing a partial outage<p><small>Apr <var data-var='date'>30</var>, <var data-var='time'>19:33</var> EDT</small><br><strong>Resolved</strong> - We've seen resolution for at least 30 minutes. This issue has now been resolved.</p><p><small>Apr <var data-var='date'>30</var>, <var data-var='time'>18:32</var> EDT</small><br><strong>Monitoring</strong> - We believe our integration is now back online. We'll be monitoring for the next 10-15 minutes to ensure we're back online.</p><p><small>Apr <var data-var='date'>30</var>, <var data-var='time'>18:13</var> EDT</small><br><strong>Identified</strong> - We have isolated the issue to customers that are on the Legacy Salesforce integration. Customer that have migrated to the new integration are unaffected. We believe our legacy integration may be coming back online.</p><p><small>Apr <var data-var='date'>30</var>, <var data-var='time'>18:08</var> EDT</small><br><strong>Investigating</strong> - We are investigating an issue with our integration partner that powers the delivery of contracts from Salesforce.</p>tag:status.pactsafe.com,2005:Incident/36235922020-02-27T22:54:28-05:002020-02-27T22:54:28-05:00Issue publishing new versions of contracts<p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>22:54</var> EST</small><br><strong>Resolved</strong> - We've successfully rolled back our code and the issue is resolved. We will be investigating the root cause of the issue and will be re-deploying the release once a fix is in place.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>22:51</var> EST</small><br><strong>Monitoring</strong> - We have rolled back our release (which deployed at 9:44pm EST) and are testing that everything is back in order.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>22:49</var> EST</small><br><strong>Identified</strong> - We've narrowed down the issue in connection with a change that we deployed with our most recent release which was completed this evening. We have rolled back the release and are testing that everything is back in order.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>22:47</var> EST</small><br><strong>Investigating</strong> - We are currently seeing an issue with our REST API that's causing problems when new versions of contracts are being published. No clickwrap acceptance events or Legal Center interactions are being affected.</p>tag:status.pactsafe.com,2005:Incident/30316102019-09-24T12:55:04-04:002019-09-24T12:55:04-04:00Electronic signing currently experiencing issues<p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>12:55</var> EDT</small><br><strong>Resolved</strong> - Signing is now back to normal and all records generated going forward will contain the correct information. As mentioned, we will be sending a follow-up bulletin this afternoon including our plan to retroactively any contract signed during this partial outage. Thank you for your patience.</p><p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>12:39</var> EDT</small><br><strong>Monitoring</strong> - We have successfully rolled back our code, and we are seeing signing and records return to normal. We will be putting a plan together to notify anyone that had contracts that we're affected during this time with details on how we plan to recover and recreate records for signatures that happened during this time.</p><p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>12:35</var> EDT</small><br><strong>Identified</strong> - This issue was identified and connected to a hotfix deployed this morning. We are currently doing one more rollback to confirm all fields will be shown in records moving forward. We will know more on specific timings ASAP. We will also be putting a plan together to retroactively address records that were signed in the meantime.<br /><br />PDFs generated during this time will need to be regenerated once our fix is in place.</p><p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>12:28</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate. Currently, no field values or signatures are being shown on the electronic record. We are seeing data stored on the activity, so we believe at this point we do not have data loss at this point.</p><p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>12:18</var> EDT</small><br><strong>Investigating</strong> - We are seeing intermittent issues with electronically signing contracts in PactSafe. We are investigating a change related to a deployment this morning that we are currently rolling back. Once rolled back, we will confirm that this change fixes the issue.</p>tag:status.pactsafe.com,2005:Incident/28562912019-08-13T08:58:57-04:002019-08-13T08:58:57-04:00Issue with signing SmartPacts<p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>08:58</var> EDT</small><br><strong>Resolved</strong> - We’ve rolled back the change and will fix and test a patch to production. All SmartPacts are currently back online.</p><p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>08:46</var> EDT</small><br><strong>Identified</strong> - We have rolled back a change we deployed to production yesterday to temporarily resolve signing issues. We are working through a fix and will be testing shortly.</p><p><small>Aug <var data-var='date'>13</var>, <var data-var='time'>08:36</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating an issue that is affecting the signing of SmartPacts in PactSafe. All API, esignatures, and clickwraps are otherwise being processed and our interface is accessible. We'll post an update soon as we have a fix identified.</p>tag:status.pactsafe.com,2005:Incident/26990962019-07-12T22:33:22-04:002019-07-12T22:33:22-04:00Salesforce integration experiencing sync issues<p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>22:33</var> EDT</small><br><strong>Resolved</strong> - Everything should be good. Let us know if you are experiencing sync issues with Salesforce by emailing help@pactsafe.com.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>22:02</var> EDT</small><br><strong>Monitoring</strong> - We have rolled back changes. Things should be returning to normal and data should begin syncing shortly.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>20:09</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and changes will be rolled back shortly.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>19:31</var> EDT</small><br><strong>Investigating</strong> - We are currently experiencing an issue with our processing engine that synchronizes data into Salesforce contracts. <br /><br />This is only affecting customers using the Salesforce integration with PactSafe. It would be preventing users from populating data from Salesforce into a contract. <br /><br />No other services are impacted.</p>tag:status.pactsafe.com,2005:Incident/26428522019-07-02T11:05:05-04:002019-07-02T11:05:05-04:00Cloudflare CDN outage<p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>11:05</var> EDT</small><br><strong>Resolved</strong> - PactSafe experienced some delays and intermittent issues due to services we use that rely on Cloudflare's CDN. From approximately 13:52 UTC today to Jul 2, 14:57 UTC, you may have experience loading issues, console errors, and more in using PactSafe's application and signature request signing pages.<br /><br />This issue has already been resolved by Cloudflare so everything is back to normal.</p>tag:status.pactsafe.com,2005:Incident/21745182019-01-28T11:48:43-05:002019-01-29T14:18:08-05:00Intermittent issues with Actity API<p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>11:48</var> EST</small><br><strong>Resolved</strong> - This issue has been resolved, but we will be researching and providing a full post-mortem as to what happened and how we'll be preventing this from happening in the future.</p><p><small>Jan <var data-var='date'>28</var>, <var data-var='time'>08:37</var> EST</small><br><strong>Monitoring</strong> - Due to an increased amount of activity over the weekend, we identified an issue with how our logs were written that caused issues with writing new values to our database. We have temporarily resolved the issue and will provide a post-mortem with full outage timings and path to prevent in the future.</p>tag:status.pactsafe.com,2005:Incident/17341832018-05-17T15:33:53-04:002018-05-17T15:33:54-04:00Issue with email delivery service<p><small>May <var data-var='date'>17</var>, <var data-var='time'>15:33</var> EDT</small><br><strong>Resolved</strong> - Mandrill confirmed at 2:26pm EST that this issue was resolved. Mandrill/MailChimp will be monitoring everything to make we're good. We are marking this resolved for now. Thank you for your patience.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>11:45</var> EDT</small><br><strong>Update</strong> - We finally have confirmation from Mandrill that they are experiencing connectivity issues. We are in touch with them and will follow the status of their connectivity issues closely. If you'd like to follow these issues yourself, please visit: http://status.mandrillapp.com/</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>11:39</var> EDT</small><br><strong>Update</strong> - UPDATE: We are still able to send some emails through the system and are seeing less intermittent errors from our email service provider, but we are still logging some errors preventing some emails from sending.<br /><br />We're still investigating what might be happening and will update in 15 minutes.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>11:13</var> EDT</small><br><strong>Update</strong> - We are seeing emails delivered, but are still receiving a large number of errors from our email service at this time. NOTE: No other PactSafe service or API is currently affected.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>11:08</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>11:07</var> EDT</small><br><strong>Investigating</strong> - We are currently receiving a high number of errors connected to our email delivery service Mandrill. We're currently investigating through Mandrill what the problem is.<br /><br />**What is affected**: Currently, this could affect emails delivered from the PactSafe system including emails for sending requests, completing requests, and notifications.</p>tag:status.pactsafe.com,2005:Incident/14248942017-10-18T14:29:37-04:002017-10-18T14:29:37-04:00Activity API currently experiencing an outage<p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>14:29</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>14:21</var> EDT</small><br><strong>Monitoring</strong> - We're seeing everything coming back online and are monitoring all core services to ensure we're back to normal.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>14:20</var> EDT</small><br><strong>Update</strong> - Activity API is back online and we're starting to see the User Interface start to come back online.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>14:17</var> EDT</small><br><strong>Update</strong> - We're starting to see about half of all Activity API calls coming back online. We're still working to resolve the User Interface/Signing portion of our application.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>14:07</var> EDT</small><br><strong>Update</strong> - We're currently in the process of rolling back the security/authentication update we made to our database late last night and should start to see a resolution shortly.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>13:58</var> EDT</small><br><strong>Update</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>13:58</var> EDT</small><br><strong>Identified</strong> - We believe we've isolated the incident to an update we made in securing our database last night. We're currently reverting this change to see if this resolves the issue.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>13:55</var> EDT</small><br><strong>Investigating</strong> - We're currently experiencing an outage with our Activity API, which is affecting authentication and authorization into the application. We are currently investigating and working to resolve ASAP.</p>tag:status.pactsafe.com,2005:Incident/13888742017-09-29T09:49:12-04:002017-09-29T09:49:12-04:00Application servers experiencing intermittent issues<p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>09:49</var> EDT</small><br><strong>Resolved</strong> - All services are now back online, and our team is working through a better solution as to how we can ensure dependencies in some of the libraries we use.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>09:39</var> EDT</small><br><strong>Update</strong> - Servers are coming back online and we discovered that there was a dependency in our code change this morning. All services should be back online momentarily.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>09:29</var> EDT</small><br><strong>Monitoring</strong> - We've rolled back the changes and should be back online in about 4-5 minutes.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>09:26</var> EDT</small><br><strong>Identified</strong> - We are rolling back code that was deployed early this morning that may have caused the issue.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>09:22</var> EDT</small><br><strong>Investigating</strong> - We are seeing some issues in loading the PactSafe application services for our user interface. We're investigating and troubleshooting and will post an update shortly.</p>tag:status.pactsafe.com,2005:Incident/13685442017-09-20T13:20:10-04:002018-07-31T13:06:33-04:00DNS issue on pactsafe.io<p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>13:20</var> EDT</small><br><strong>Resolved</strong> - This issue was resolved around 11:45am EST and all systems are operational.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>09:35</var> EDT</small><br><strong>Monitoring</strong> - The DNS seems to have resolved—we are monitoring closely to ensure we're fully back online.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>09:32</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating a DNS issue that's causing pactsafe.io to return a 404 error. We will post an update on the status of this issue in the next 10-15 minutes.</p>