Yes! Breathing helium can, in fact, kill you — but not because of the helium, rather because the lack of oxygen when you inhale the helium. As you breathe in a balloon full of helium, you are not breathing in any oxygen, which your cells need - usually we get this from the air we breathe.
It is not safe to inhale helium from a balloon (several times in a row). Helium is an asphyxiant. If you breathe mostly helium, it will suffocate you and you will die. Taking a single suck of helium from a balloon so you will talk funny is not dangerous.