All things considered... Yoshi probably hates you.

He's actually pointing to what Yoshi's supposed to stick out his tongue to.

/ruinedit

Well it looks a lot like a punch in the back of the head to me, but I guess the sprites were too small to make a finger. I still think Mario is punching him though.