Phantom - Stolen Skills and Auto Buff Gem

Yes, that's how it works.

And if you switch a gem buff, the buff disappears from the gem, and you have to re apply.

Well, that's disappointing.

Then what's the point of stealing 2 buffs for the same skill? If you cant use them at the same time seems pretty pointless.

Here's at least two good reasons:
1) The 2+ buffs you've stolen for the same job have different PURPOSES (e.g., Meso Guard for solo defense, HB for party defense, Rage for party offense, etc.) Just switch to the one that suits the situation the best.
2) If another phantom casts buff X, you can switch your X to Y and share two party skills instead of one.