To answer you points
1: The area is cursed and everybody know it. Really, do you really think that. This is the empire. Outside a 20-30 mile radius nobody knows anything about anything. It is entirely possible that people in the village/town or whatever fifty miles away have no idea how badly off this area is. The travel of information is slow, really slow and very unreliable. There is no reason to assume that everybody knows that this place is messed up.
2: We don't leave because we were invited to dinner. We had intended to leave tomorrow, after all we were just looking for a village to buy food in. As there is no food to be bought we were just going to move on. However the invite to dinner would be rude to refuse.
3: We didn't need to get lost in the woods, we could simply have been following the river, a fine way to travel when you have no maps.
Rousseaux may well be evil, it is even likely that he is, but we don't know that he is. People in this group seem to be jumping on the bandwagon that everybody is evil, which may or may not be the case.
Of course they care if they have a powerful noble after them. A powerful noble would level the place and kill them all. Indeed the fact that nobody, knights griffon, witch hunters etc. is after them shows how little people know. A hint of chaos should be enough to get the knights griffon, witch hunters and the entire army of the church camped outside their door.
It is not that they represent no threat, it is that we represent no threat and therefore they don't represent the threat that people are expecting them to.
