Instead of implementing the tracking everywhere, create a RAII-guard that will track the state as it is held and dropped.