There's no need to worry about not being safe during your visit, as Ovelikios & Mimhs.gr mentioned before me, the refufees aren't causing any trouble. If that makes you feel any better, I was at the port of Pureaus the other day to catch the ferry to Syros, now there are about 3000+ refugees hosted there, if I didn't kbow about them, I wouldn't even notice them. Aside what the media around the world are projecting, the refugees don't cause trouble, they just seek a peaceful, safe future. Would you (generally speaking) cause trouble if you had just escaped the situation they are running aeay from? I know I wouldn't.
Asking here was the best thing you could do. People and media are -let's say because it isn't the right word- overreacting. You will be perfectly safe in Athens