Is it harmful/toxic to handle small quantities of mercury for short periods of time?

Lots of us as kids busted thermomoters & played with a drop of mercury in the palm of the hand, to no ill effect, though most will say 'it'll killya' Someone told me it needs to be large quantities, or actually ingested to harm a person. a chemist gave a friend a half-dollar sized amt of mercury. Whats the 'real deal' on this stuff?