It does take two or three tries for the effects of weed to kick in, which matches what you've told us.
My first time wasn't great either. Memorable, yes, but not great. I'd recommend that you at least try it one more time, and this time just remember to chill out a little bit. Weed ain't gonna do shit to you, dude; if you're not breaking out in a rash or hives, I'd say you're not technically "allergic."
But, of course, weed is not everyone's bag.