I think it would be ok to destroy the nest if there are no eggs. But if there are eggs and you move it, I think the parents would abandon it.
In the US, I believe it is not legal to destroy a nest once there are eggs if it is a migratory or native bird. Non-native invasive species (like pigeon) wouldn't be protected here.