A root cause for this incident has been determined:
A bug related to how data exporting manages resources in the platform's data indexer was uncovered during unrelated data processing, which led to an unexpected consumption of resources in the indexer.
That data processing was stopped shortly after system degradation was discovered, which allowed the indexer to automatically recover. Normal access to the platform was restored shortly after.
An immediate fix for this bug is currently being tested and will likely be deployed today.
Longer term investigation will continue, to determine any additional changes that may prevent this and similar scenarios from occurring.
Posted Apr 07, 2020 - 14:23 CDT
Monitoring
Platform accessibility has returned to expected levels. The Kenna Engineering team is investigating a suspected root cause of the original high load and continues to monitor the current status of the platform.
Posted Apr 07, 2020 - 12:08 CDT
Update
Load on the indexing system has begun to lessen and the platform is currently accessible. Investigation on the root cause is ongoing.
Posted Apr 07, 2020 - 11:55 CDT
Investigating
At 11:35am CDT the Kenna US Production platform was automatically reported slow and/or inaccessible, due to a high load on the indexing system. The Kenna Engineering team is currently investigating the current cause of this degredation.