The name originally applied to especially bright stars, i.e., a large Luminosity or Absolute Magnitude. Because the brightness is directly related to the area facing us (the square of the radius), the brightest stars are indeed large. The general term Dwarf Star actually applies to any star that is not a giant, though it is now usually used with a qualifier, with a more specific meaning (White Dwarf, Red Dwarf, Brown Dwarf).
H-R Diagram (HRD)
Stellar Age Determination