Having an office or two inside doesn't seem to be decisive for towers to become buildings.
I don't even think most monarchs actually like the place!
Because it's easily commutable to London, and far nicer than Buckingham Palace (and yes monarchs do hate it - when Elizabeth ascended the throne, she wanted to stay in her much smaller residence nearby), it has tended to be where the Queen has actually lived most of the time, with Buck House basically being the office that has a house attached - typically only used when there's a reason for the Queen to be in London for a few days.
Buckingham Palace is where the Queen primarily does her Queening though.
My country tis of thee is not the UK anthem the main one in god save the queen along with:
Flower of Scotland
Land of my fathers
Bozo? Boozo? Drunky McDrunkface?
Everything is small there; even the largest non-human predator is tiny! (Badger. I do jest with these comments a bit.)