There is already an open source Java Solution for Linux based embedded devices. Look at Jalimo. At the moment packages for the Maemo and Openmoko platform exist.
What have X to do with real multitasking? Qtopia, for example, is not based on X and have real multitasking. Many servers have no X but real multitasking.
In my opinion X can be useful and can work well on embedded devices. An example for this is the Nokia Internet Tablet and hopefully soon the...
As far as I know, Ubuntu Mobile will also use parts of Maemo. More information about it here:
Ubuntu Mobile
Wiki Ubuntu Mobile
It is targeted to MIDs, but maybe it can be useful at the OpenPandora too.