AFAIK it's made for exactly that: video and sound en- and decoding.
For me it would be use enough if all the audio/video codecs would be handled via DSP and M4.
Actually the subsystem for that is IVA-HD. Unlike its precedessor IVA2 on the omap3 it doesn't embed a C64x+ DSP. Instead there's a separate C64x-derivative "Tesla" DSP, which seems to be focussed on audio since it has a direct connection to the Audio Backend (ABE) subsystem. The main intended purpose of the M4 subsystem is looking after the camera subsystem (ISS) and real-time management of video streams in general.
I'm idly wondering if one of the M4s could be used to keep the DSP fed with data. Is there a need to make that easier to schedule?
These are most useful when you need to sample inputs very fast, like an oscilloscope or logic analyser does,
I'd consider using timer-triggered DMA for that if nothing better is available.
How about a hypervisor boot environment on the m4's that could then boot different VMs running different kernels for the a15 cores?
LOL, how about a hypervisor boot environment on the Cortex-A15 since it has actual hardware virtualization support? (which is separate and distinct from TrustZone btw)
If it's possible to run a self written programm on the M4s (I assume without an operating system), while running the Pyra with it's standard operating system, will it be possible to just write something on to the screen? (So simply overtake the screen, and give it back afterwards)
Yes, although no "overtaking" would be involved... you could just scribble onto framebuffers if you know where they are located. A nicer solution would be to allocate a private framebuffer for it and stick it on an overlay, or have JonnyH render it on a rotating cube for you
Developing software for both cores simultaneously is a bit tricky since they share the same address space (apart from the Private Peripheral Bus) so it behaves mostly like two threads rather than like two processes.
I'm pretty sure that's how they're intended. The main CPU has some kind of debug system allowing you to write code to the M4's
It has no access to the debug functionality of those cores, you'd need JTAG for that. It does however have direct access to the local RAM and can set up the MMUs.
For starters we'd need the appropriate software for writing code to the M4's. There is probably some software development kit for this, and/or some kernel driver is needed.
You can just use /dev/mem, and the procedure is quite simple: release subsystem from reset, write code to subsystem RAM, optionally configure the MMUs, release core(s) from reset.