In order to iteratively find the best memory allocation, we need to be able to free backend memory so we can try again.