The problem is not the time it takes to launch.... if not when you benchmark graphic primitives like: lines, fills, bitblt, etc.
Some tests in NEO1973 shows X11 with an overhead of 50% vs framebuffer.
Sure that more layers above that will make it slower... (this is the price you pay for...