I think it depends. Like if I wanna release a photobook of artist XY I gotta get the permission of XY to release the photo and make money with. If the photo was done at a public event (like an award show) I can release the photo like that.
Same the other way around, if artist XY want to publish a photo I took, he needs my permission.
It all depends if stuff is for commercial use or not. Also if the person / building is NOT the main subject on the building, I can release it how I want it.