Don't worry about it Kirlia. It's just like with legal lolis; some people just can't wrap their heads around the fact that you are NOT a child, just because you look smaller than others. You're not bad, you're just a bit niche. :D
I actually know a few middle-aged people who are barely above the four feet mark, so I guess it's not as weird to me. Then again... Nothing is really THAT weird to me anymore. X)