# CVJointOps outage - Post Mortem

**URL:** <https://forum.ssv.network/t/cvjointops-outage-post-mortem/788>\
**Category:** Issues\
**Created:** [October 22, 2022, 5:02pm UTC](https://forum.ssv.network/t/cvjointops-outage-post-mortem/788 "2022-10-22T17:02:11Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![CVJointOps](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.ssv.network/cvjointops/32/29_2.png) [@CVJointOps](https://forum.ssv.network/u/CVJointOps)\
**Post date:** [October 22, 2022, 5:02pm UTC](https://forum.ssv.network/t/cvjointops-outage-post-mortem/788/1 "2022-10-22T17:02:11Z")

</div>

**Background:**  
On Sep 30th, 2022 at epoch 125067 the CVJointOps node went offline and was no longer attesting as part of the sync committee.

> [@Rectify CVJointOps outage, or remove verified operator status](https://forum.ssv.network/t/rectify-cvjointops-outage-or-remove-verified-operator-status/777):
>
> Background: Verified operators warranted to the DAO that they would resolve outages within 2 weeks and engage their users when they have outages. CVJointOps has been offline since epoch 125067 on Sep 30th 2022. They have not engaged their users on Discord as far as I can see. Proposal: Give CVJointOps an opportunity to rectify their outage If CVJointOps is unwilling or unable to rectify their outage by Oct 31st 2022, remove their verified operator status

**Investigation:**  
After being notified via Discord on Oct 15, 2022 that the CVJointOps node was offline, an investigation began into the issue which stopped the node from attesting. When logging into the node, it was clear that there were no issues with the machine itself. It was online and operating normally. Checking the logs for the clients, it was found that the EL client (besu) had failed and was no longer keeping up with the chain. The failure of the EL client caused the CL client (teku) as well as the SSV container to go down and stop attesting.

**Root Cause:**  
The EL client software was updated many times post-merge to improve EL operation. The last of these updates were not applied to the node, which is believed to have caused the failure.

**Resolution:**  
All clients were updated to their latest release, and the EL and CL databases were removed to avoid any corruption of the data. Once the databases were re-synced, the SSV container was able to continue attesting properly. By Oct 17, 2022 the node was fully operational.

**Prevent Recurrence:**  
When the node went down, it was unknown to the operator due to failures in alerting. New alerts have been set up to push an alert to the operator when either the EL, CL, or SSV container are no longer responding.

---

<div class="post-metadata">

**Author:** ![alonmuroch](https://avatars.discourse-cdn.com/v4/letter/a/9de053/32.png) [@alonmuroch](https://forum.ssv.network/u/alonmuroch)\
**Post date:** [October 30, 2022, 6:17am UTC](https://forum.ssv.network/t/cvjointops-outage-post-mortem/788/2 "2022-10-30T06:17:58Z")

</div>

Thank you for the update