I run debian with XFCE on my htc universal, and it easily uses up the 128mb RAM - so swap space is a requirement.
33% CPU? Bollocks does it! - 6% is more accurate.
If you really had to I guess you could kill X to free up some RAM, but AFAIK not many graphics drivers support OpenGL on FB...