Should I ditch my mage?

Well if bossing's your thing and ultimates aren't, then I'd say go on ahead.

But think about why you chose I/L in the first place and what motivated you to train to level 81. Anything you were aiming for, that you're close to achieving and won't be able to if you quit now?

Also... quitting doesn't have to be final, unless you don't have enough character space. You can just make a gunslinger and leave the mage, so you can play it if you feel like it.