Okay, I've browsed the internet to find out more about what X drivers are available to us and which one is the best to use.
It's mostly about the Pandaboard with an OMAP4, but it should be similar or even the same for the OMAP5.
There's a pretty old but interesting blog entry about that as well: http://alastordmcblog.blogspot.de/2012/09/debian-on-pandaboard-graphics.html
Basically, there are the following within the debian repositories * xf86-video-omap * xf86-video-omapfb * xf86-video-omap3 * pvr-omap4
xf86-video-omapfb is an enhanced version of the normal framebuffer driver. It should still be usable with the OMAP5, but does not feature hardware acceleration. Maybe we can use the DSS scaler / rotator as low-level layer and put omapfb on top of it?
xf86-video-omap3 is the same thing as omapfb, but with NEON support (so probably what we want).
xf86-video-omap is an open source driver for the DSS. As far as I understood, it's similar to omapfb, but in a more standard way, as xrandr can be used with it for rotation, scaling, etc. It needs to have omapdrm enabled in the kernel, but apart from that it should simply work.
However, without the closed source EXA binary from TI, it does not use hardware acceleration.
As far as I understood, that binary needs to be compiled for the correct X server version... I haven't yet checked where you can get these binaries from TI and which versions are available.
pvr-omap4 is the closed source driver by TI. It offers everything (overlays, XV, GLES, GLES2, EGL), but seems to depend on a kernel version and X Server version.
So it seems we first should check what binaries TI provides and supports, and then decide and test what driver to use.
We should also check them for performance, power consumption, etc. The OS omap driver might be our best guess - thanks to xrandr and being standard, it supports HDMI and multi monitors out of the box.
No idea whether that would easily work with fbdev.
I wish we had a way getting the sourcecode for the closed source TI driver so we could recompile it for any kernel or X-Server, but I doubt that's possible, even with an NDA :/