Add extra asserts to addrman