Well, the dm server was idling at around 47% CPU, so I decided to switch window managers...
Now running Enlightenment, the server is idling at max 17% and as low as 3% cpu with 2 desktop sessions running... awesomeness.
It is the VNC session in the above screenshot, not the XP desktop I'm remoting it with.