Can a girl wear a leather jacket without looking.....tough?

I bought a simple cut leather jacket when I was shopping, because it was very figure flattering (I have an hourgl figure). But now that I'm home I'm having second thoughts. I'm not a biker chick or a tomboy, but neither am I a girly-girl. Can I pull it off, or will people watch me out of the corner of their eyes like I'm about to pull out a knife or something?