X isn't too bad on real server hardware. If you have a multiple cores on dedicated hardware with a few DP servers and website, you won't notice it. But.. if you rent a VPS that's running off some guy's white box reseller server, it's probably so loaded that you can't afford the extra CPU cycles to run X.
I also think that X sessions aren't encrypted unless you tell it to forward through SSH, so don't put in root password through xterm
I would take the time to learn the unix/linux commands, might come in handy later during university/career.