"The name was misleading— it was actually a low, rambling house. Surrounded, true, by a remarkable garden, gift of the Vicereine’s even more remarkable daughter-in-law, which was growing up lushly these days to lend color and privacy, or the illusion of it. The old, hand-painted sign still hung by the gate."
―The Viceroy's Palace[src]
The Viceroy's Palace was the home of the Viceroy of Sergyar. At the time of Gentleman Jole, it was not particularly palace-like, but it was a big step up from prior versions in Kareenburg: The first had been a relocated field shelter, then a group of field shelters stuck together variously, followed by a remarkably ugly semi-fortified prefab dwelling. The Vorkosigans cleared it out and replaced it with the rambling house. Upon moving the planetary capital to Gridgrad, a new Palace with new gardens designed by Ekaterin Vorkosigan was begun.

In addition to the main house, there were barracks at the back where armsmen stayed while the Count was Viceroy; they were converted to offices for Viceregal officials afterwards. There was a gate, and presumably a fence around the house. Sadly, it lacked an oubliette for disposing of troublemakers. Major (by Sergyaran standards) social events were held at the palace from time to time.