The start of something great! I wouldn't worry about stone front with wood walls. Years ago I use to work at a historical site, one of those pioneer village places but it was a French fort/fur trading outpost. The recreation had, and historically did as well, a stone gatehouse with two stone towers beside it and a native American style wood palisade around the rest of it. Historically they just didn't have the time, manpower or resources to make a fancy stone wall around the whole thing. Just for the main gate!
So maybe your building the start of a fort or outpost! If it was good enough for the French in the 1600's, it'll do your lads nicely in the mortal realms.