Happened at our table a couple of months back.
Sounds railroady and yeah BS, but I know some that don´t have buildings. Like a Dungeon, a Turret with just arrow slits same with the Burgfried. Or a Barn, maybe a church that abhorres the Light and or Sun? Just spitballing :)