Am I the only one annoyed by the fact you can't see each others skills? (Or damage)

I definitely disagree with this. Ultimates still lag me like crazy, even with graphics down, and so does seeing 394839483093 numbers on the screen all at once from an entire expedition of people, so I'm glad they at least acknowledged that this is a problem.

I do agree that there should at least be an on/off option, though.

Same. I was just training with a party and the skills lagged me horribly.
Turning down the graphics just makes everything look like static, but does almost nothing for performance.

The solution should be simplified graphics, not the lazy low-res version of the regular sprites that they have now, which are nearly impossible to understand.
Make something completely new, like the simple mini-map. Show me what I need to know, without all the bells and whistles. Schematics will do. Text, even. Make other people's damage numbers simple text, while mine are still big and fat and impressive (especially if red). Make their skills and status effects monochrome outlines that don't extend more than a few pixels beyond the character.
All that - if we choose simplified graphics, of course. Those who like the fireworks and have computers that can handle them - enjoy.