I'm now using the 3.19.0-rc5-gta04 , with a few config changes, and everything seems extremely stable. Even running the debian xorg causes no problems. Haven't seen any segfaults when compiling either.
The (major) things i enabled were lpae and virtualization, which also required disabling armv6 support. So i suspect one of those to be the cause of the problems. Without those changes, the debian xorg crashes almost immediately (unless the non thumb version from our repo is used)
compiled version here : http://next.openpandora.org/omap5/kernel/3.19.0-rc5-gta04-av-2.tar.xz