Very large units tend to be inefficient, and dreadfully vulnerable to certain things. They are also extremely awkward to maneuver.
In the case of infantry, there is little to gain by taking a unit of greater than thirty men, unless they are unbreakable or stubborn (and in the second case, also immune to terror and panic). This is because a unit of fifty swordsmen is no harder to break in combat than a unit of twenty-five, except that it is more likely to outnumber the enemy. If it fails a terror/panic check, or is flank charged, a large number of points will be lost. Two units of twenty-five spread the points out, and more importantly are able to maneuver independantly to accomplish different goals.
Very large knight units are insanely expensive, and turn like bricks. They can certainly crush almost anything they charge, but can also be diverted and delayed to the point where they have no influence on the game. Once again, a flank charge by a much smaller unit can see them off quite effectively.
For these reasons, it is generally better to take a large number of moderate to small units than a few huge ones.