strange neighbor

I'm confused with how to deal with this topic...
On one hand I want to appear neighborly, and on the other hand I want nothing to do with this guy...
Story: For the past 7 month (perhaps a year?) I've had a new neighbor that lives a few doors down from me (not sure exactly which apartment he lives in). He's a strange looking fella... skinny, blond hair, perhaps in his mid 50's, though I suspect he might be younger than he looks, and that his 'life experiences' have aged him prematurely. He kinda looks like an ugly version of Tom Petty (I know, that's not saying much, apologies). He has an odd habit... he walks the parking lot almost constantly with shorts and no shirt on. I'm not sure if he's on disability, but I don't think he works, and he walks with a strange limp - not to mention about 1/4 the speed of an average walking pace. Well, I've passed him in the past, and I always try to keep my distance from him. Just one of those 'gut feelings' that I can't explain that the guy's bad news. Well, about 2-3 weeks ago, I cracked... I walked out to take my garbage to the dumpster, and he was out strolling. I finally acknowledged him just by giving him a "head nod" as if to say "hey, what's up" without actually saying the words. Boom... that's all he needed for an invite. A few days later he approached me as I was headed out somewhere, and proceeded to tell me that he used to have long hair (which I do - what's left of it anyway). He went on to say that he had long hair up until one day when some strangers stopped on the side of the road thinking he was a girl - so he cut it off.
Um... great. You're telling me this why? Are you calling me a girl? Are you trying to say you're 'cooler' than me cause you cut it? I didn't get it, nor did I wish to find out. I simply laughed, nodded, then went on my way.
Tonight, he cornered me again...
He asked me what I did for a living and what hours I worked, because he noticed my vehicle in the parking lot is always out there in the morning and that I usually come home late. Yes, I said, I have odd hours (without telling him what I did for a living). Then he said, "Oh you mean you work the 'lazy a$$' second shift?" - Um... yeah I said, if you say so.
So now he's basically called me a girl, and a "lazy a$$" - how's that for neighborly?
Actually, I guess I just answered my question. I think I'll go back to ignoring him. If I just have to 'nod' and acknowledge him and leads him to insulting me - I guess the answer is pretty obvious. But I'm not that type of person, so I'll continue to fight with this internally - AND I'm also aware too that this is 'twice' now that he's spoken with me, and with each time it will get easier to bring up things to talk about. Ugh.