SWAP Function Post-Mortem Report
Prepared By: Phillip (MarPhil), CTO
Included: AG - Lead Developer, Dublin
Issue: Swaps within the last 12 hours 47 minutes failed to pay out ONG
Cause: Oracle didn't have enough ETH
Root Cause: Code didn't check if the gas was there on the oracle before initiating the entire swap
Remediation: Update the code to validate the gas for the user or oracle (depending on direction of swap/buy) to ensure that both points, ong and gas are available before initiating a swap.
RESOLUTION PROGRESS: Published - active, issue remedied
Any user who did a swap during the outage period will be manually compensated for any missing ONG token - we'll get to that just as soon as our Indian team is on shift.
While this is a good remediation, it isn't a perfect solution; it's possible that the state of the oracle may change during parallel transfers, meaning that there still may not be enough ETH on the oracle if multiple transfers hit it at the same time. However, the affect case will be significantly limited (meaning of 50 users, instead of 20 being denied, maybe 2 will be), which is an acceptable customer service management situation. Our development team is going to work on some additional code level changes to ensure that failed transactions are either retried, or rolled back, so that users are not missing points going forward (which will take care of the potential 2 failures for parallel transactions). We anticipate that change in the next major release of the software backend.
We have requested developers put an automated method in place to monitor the oracle, to keep a record of the ETH/ONG in place at any given time, so that we can better figure out any missing transactions over time, and potentially automate paying out missed ong/points as needed.
//END OF REPORT