You should perhaps consider a summer residence or some such.. A regular castle would most certainly be defensible, whereas a mansion on a hill somewhere that was generally considered safe could quite possibly "only" be surrounded by low walls with numerable gates and such...
And these walls could also be rather fra from the mansion.. and even broken or burned...
Imagine part of a mansion in one corner of your deployment zone.. Positioned on a hill... Standing in a park that bears the marks of a previous battle...
Make a few pieces of broken wall, burned bushes and noblemen / womens corpses scattered arround the mansion where the brave imperial forces make a last stand to save their count and his family...
Consider the fluff possibilities.. and arguments...
Why would an elector count ever consider fighting near his mansion/palace/castle?? Only as a last resort...
Why are there enemies so close?? Because his defences has been broken and only a handfull of loyal men remain to protect the lands...
All you need would be part of a mansion, broken pieces of wall (placed fairly on the battlefield so noone gains too big a benefit from them) and assorted things to make part of the battlefield resemble a palace garten after a major battle.