You can make a kernel module in Linux and do what you want, or you can blow away Linux altogether. There are some CPU features that require "secure" mode as opposed to just supervisor level access, and there's a complex ritual for gaining this, but it can still be done. There probably isn't much you'd want to enter this for.
Most of the peripheral hardware on the OMAP3530 has documentation and register interfaces available, the SGX not really so much. So you'd have to reverse engineer it to do what the OGL ES driver does and it probably won't be very easy. Some of the thoroughly documented parts of the hardware are also very complex and I wouldn't want to have to code for it. The WiFi/Bluetooth chip is also not openly documented; a datasheet unofficially floated around before but that doesn't really constitute as a user's manual.