Use of the (Co-)Processors


Can I use an M4 to listen to the microphone and wake up the main CPU when there's audio to be recorded?

I seem to recall testing constant microphone listening on Pandora and it used a lot of CPU. Or maybe that was Mumble, which uses a lot of CPU even on desktop, even when it's totally not connected to any server.
 
Can I use an M4 to listen to the microphone and wake up the main CPU when there's audio to be recorded?
What Exophase wrote applies for this too - M4s won't have access to the hardware because A15s are already driving it. It is possible to coordinate this but it's highly problematic. It's even possible on pandora right now as the DSP has direct audio hardware access, but nobody ever implemented it and almost certainly never will.
I'm quite sure the coprocessors on Pyra won't be used at all. Of course it's possible that some enthusiastic person shows up and codes something interesting, but I don't really see this happening. On pandora we have a DSP that's even faster that A8 on certain tasks, yet I've failed to find any good use for it (that would not take forever to implement) even after putting some effort to find something.
 
Last edited by a moderator:
What about the Notification LED idea mentioned earlier.  Could one of the M4s be given exclusive access to the RGB LEDs and accept signals from the A15s to adjust them as needed?   I remember a discussion about notification colour schemes in the absence of dedicated LEDs  for each indicated function and it occurs to me that there could be some processing involved just to multiplex the various different indications into specific RGB LED sequences...

-Neelix
 
Would it be feasible to use the M4 to make some kind of very-low-power suspend mode, to increase standby time by completely shutting down the A15s? Or would that make nearly no difference compared to having the A15s in their deepest sleep mode?

I agree that the OMAP5 coprocessors are probably mostly useless from the point of view of performance. But from the point of view of battery life, perhaps they can be somewhat useful. Maybe they could manage some hardware that is mostly isolated/orthogonal but generates relatively many interrupts (say more than one per second on average)? E.g. all those sensors (accelerometer, altimeter, nubs, touch screen), do they have their own microcontroller doing everything already, or is there some amount of processing/calibration/filtering going on in a driver running on the A15?

If there's a way to reduce the number of irqs to the A15s, it would slightly improve performance and potentially significantly improve battery life, right?
 
The M4's also have floating point units, but I don't know at which speed they are clocked. Besides why would TI call them "image processing unit", if they are so weak?
 
Last edited by a moderator:
Reference? Because when I checked I couldn't find any evidence of FPUs.
 
The ARM Cortex-M4 processor is a low-power processor that features low gate count, low interrupt latency, and low-cost debug. The Cortex-M4F is a processor with the same capability as the Cortex-M4 processor, and includes floating point arithmetic functionality. These processors are intended for applications requiring digital signal processing functionality. The ARM Cortex-M4 processor implements the ARMv7-M architecture profile.
does this count ?
 
Last edited by a moderator:
The Cortex-M4 can be configured in a number of ways. FPU is optional. A source from Ti would be needed to say anything about the M4's in the OMAP5.
 
The Cortex-M4 can be configured in a number of ways. FPU is optional. A source from Ti would be needed to say anything about the M4's in the OMAP5.
Here are some Specs out of the OMAP543x Datasheet:

1.3.1 MPU Subsystem Description


The Cortex-A15 MPU subsystem integrates the following submodules:


. . ARM Cortex-A15 MPCore


- Two central processing units (CPUs)


- ARM Version 7 ISA: Standard ARM instruction set plus Thumb-®-2, Jazelle-® RCT Java-¢


accelerator, hardware virtualization support, and large physical address extensions (LPAE)


- Neon-¢ SIMD coprocessor and VFPv4 per CPU


- Interrupt controller with up to 160 interrupt requests


- One general-purpose timer and one watchdog timer per CPU


- Debug and trace features


- 32-KiB instruction and 32-KiB data level 1 (L1) caches per CPU


. . Shared 2-MiB level 2 (L2) cache


. . 48-KiB bootable ROM


. . Local power, reset, and clock management (PRCM) module


. . Emulation features


. . Digital phase-locked loop (DPLL)


1.3.2 DSP Subsystem Description


The DSP subsystem contains the following submodules:


. . TMS320DM64-¢ 32-bit fixed DSP core for audio processing and general-purpose imaging and video


processing. It is backward compatible with existing C64x-¢ video codecs.


- 32-KiB L1 4-way set associative cache


- 128-KiB L2 8-way set associative cache


. . Enhanced direct memory access (DMA_DSP) engine with 128 channels for video/audio data transfer


between IVA3 local memories and external SDRAM memories or peripherals


. . Interrupt controller (INTC) with up to 128 interrupt requests


. . Memory management unit (MMU) for address management. It translates virtual addresses handled by


the DSP CPU into physical addresses.


. . DPLL


The DSP subsystem allows direct access to the ABE subsystem, and has emulation capabilities.


1.3.4 Image Processing Unit (IPU) Subsystem


The Cortex-M4 IPU subsystem includes the following components:


. . Two Cortex-M4 CPUs


. . ARMv7E-M and Thumb-2 instruction set architecture


. . Hardware division and single-cycle multiplication acceleration


. . Dedicated INTC with up to 63 physical interrupt events with 16-level priority


. . Two-level memory subsystem hierarchy


- L1


32-KiB shared cache memory


- L2 ROM + RAM


. . 64-KiB RAM


. . 16-KiB bootable ROM


. . Cortex-M4 system bus directly connected to the ISS interconnect


. . MMU for address translation


. . Integrated power management


. . Emulation feature embedded in the Cortex-M4


1.3.7 Imaging Subsystem (ISS)


The ISS processes data coming from the image sensor, memory, and IVA-HD subsystem. The ISS is


responsible for multimedia applications such as: camera viewfinder; video record with up to 1080p at 60fps or stereoscopic record up to 1080p at 30 fps with digital zoom, video stabilization, rotation and auto-convergence. The ISS supports a pixel throughput of up to 304 MP/s. It assures good performance with


sensors up to 24 MP and more (higher resolution can be achieved using vertical frame division mode).


ISS supports still image capture during video record. The ISS can implement third-party algorithms for


further flexibility when working with image sensors.


The ISS consists of:


. . The ISP, which deals with on-the-fly or memory-to-memory data processing. It allows data collection


for autoexposure, autowhite balance, autofocus, resizing, and histogram generation. The ISP consists


of:


- Image pipe interface (IPIPEIF) for synchronization signals (HD, VD) for the ISIF, IPIPE, RSZ, and


hardware 3A (H3A) modules, and data transfer from video port, SDRAM, ISIF. Various pixel data


manipulation functions.


- Image pipe (IPIPE) front-end and back-end modules for raw data processing and RGB and YUV


data processing, respectively. They support:


. . Sensor data linearization for dynamic range extension


. . Programmable 2D lens shading compensation correction


. . Black-level compensation


. . Defect pixel correction (LUT_DPC) with look-up table


. . Defect pixel correction (OTF_DPC) with on-the-fly detection and correction


. . Green imbalance correction (GIC)


. . Gamma correction


. . RGB color correction


. . RGB to YUV4:2:2 color conversion


. . 3D look up table (LUT) for color correction


. . 2D noise filtering


. . 2D edge enhancement


. . False chroma suppression


- H3A for autowhite balance, autoexposure, and autofocus


- Two independent resizers, which allow YUV4:2:2 to YUV4:2:0 planar chroma filtering and


downsampling. The resizers support input and output flows with up to 304 MP/s, and memory-to-


memory rescaling in the range Ã1/4096 scale down, and Ã20 scale up. RGB565, ARGB888,


YUV422 co sited and YUV4:2:0 planar (NV12 or NV21) data output.


- Image sensor interface (ISIF) can process the incoming data and supports the following main


functions:


. . Sensor data linearization


. . Supports VGA read out mode


. . Color space conversion


. . Digital clamp with horizontal/vertical offset drift compensation


. . Vertical line defect correction


. . Programmable 2D-matrix lens shading correction


. . 10-to-8 bits A-Law compression table inside


- Buffer logic (BL), which processes and manages the requests to the module and memory


subsystem


. . Peripheral serial interfaces for connection with sensors and memories:


- One MIPI CSI-2 compliant with four RX data lanes


- One MIPI CSI-2 compliant with two RX data lanes


- One MIPI CSI-2 compliant with one RX data lane / MIPI CSI-1 / CCP2


. . Peripheral 16-bit parallel interface, BT656 and SYNC mode


. . SIMCOP coprocessor for memory-to-memory operation; JPEG encode/decode hardware acceleration;


high-ISO filtering; block-based rotation; warping and fusion; and general-purpose imaging acceleration.


The SIMCOP includes the following main submodules:


- Two imaging extension (iMX) modules - programmable image and video processing engines


- Noise filter 2 (NSF2) - for advanced noise filtering and edge-enhancement


- Noise filter 3 (NSF3) - next generation NSF for advanced noise filtering and edge-enhancement,


supporting Bayer RGB format


- Variable-length coder/decoder for JPEG (VLCDJ) module


- Discrete cosine transform (DCT) module


- Warping accelerator for lens distortion correction (LDC) and chromatic aberration correction


- Rotation accelerator (ROT) engine


- Hardware sequencer, which offloads sequencing tasks from the MPU


- Shared buffers/memories


- DMA controller


. . Timing control module for CAM global reset control, CAM flash strobe, and CAM shutter


. . Bayer scaler module allowing low latency still image capture and stabilization statistics collection.
I think the M4s are used to feed the Imaging Subsystem

(I hope TI is Okay with that :) )
 
Last edited by a moderator:
E.g. all those sensors (accelerometer, altimeter, nubs, touch screen), do they have their own microcontroller doing everything already, or is there some amount of processing/calibration/filtering going on in a driver running on the A15?
I hope normally those devices would be silent while idle, like they are now on pandora. When the device is being used (like playing a game) it doesn't matter, as at least one A15 will be active anyway.
If there's a way to reduce the number of irqs to the A15s, it would slightly improve performance and potentially significantly improve battery life, right?
Only subset of irqs are routed to M4s (like only the first 2 GPIO banks), so this may or may not work, depends on how Nikolaus designs things. Still I don't see anyone coding special drivers for both M4s (to drive hardware) and A15s (for communication with M4s), it's far easier to just use already available Linux drivers that will just run on A15s.
The M4's also have floating point units, but I don't know at which speed they are clocked.
The manual doesn't mention FPUs in M4 section so it's almost certain the M4s don't have them. Also very early revision of M4s are used (like A15s), expect many bugs...
Besides why would TI call them "image processing unit", if they are so weak?
I guess to drive the cameras (focus, etc), set up image/video transfers from the cameras, set up image/video compressor hardware blocks and so on...
 
Last edited by a moderator:
The ARM Cortex-M4 processor is a low-power processor that features low gate count, low interrupt latency, and low-cost debug. The Cortex-M4F is a processor with the same capability as the Cortex-M4 processor, and includes floating point arithmetic functionality. These processors are intended for applications requiring digital signal processing functionality. The ARM Cortex-M4 processor implements the ARMv7-M architecture profile.
does this count ?
No. The FPU is optional. It's saying the Cortex M4 is a low-power processor yadda yadda, the M4F is the same thing plus FPU.If the OMAP5 M4s had the FPU I'm pretty sure they'd say M4f, not just M4.

I think the M4s are used to feed the Imaging Subsystem
You can do image manipulation easily with integer or even fixed point arithmatic.
I am 99.5% certain the M4s do not have an FPU.
 
The ARM Cortex-M4 processor is a low-power processor that features low gate count, low interrupt latency, and low-cost debug. The Cortex-M4F is a processor with the same capability as the Cortex-M4 processor, and includes floating point arithmetic functionality. These processors are intended for applications requiring digital signal processing functionality. The ARM Cortex-M4 processor implements the ARMv7-M architecture profile.
does this count ?
No. The FPU is optional. It's saying the Cortex M4 is a low-power processor yadda yadda, the M4F is the same thing plus FPU.
If the OMAP5 M4s had the FPU I'm pretty sure they'd say M4f, not just M4.

I think the M4s are used to feed the Imaging Subsystem
You can do image manipulation easily with integer or even fixed point arithmatic.

I am 99.5% certain the M4s do not have an FPU.
Sorry I should have trimmed a little on my post, the omap5 definitely only has regular M4s and not the M4F
 
AFAIK:

- there is something that TI calls the Ducati system, which is built around the M4s and some other components and is used for video decoding and encoding. See here: http://www.omappedia.com/wiki/Gstreamer_Project_Main

- something which as I see is often forgotten is the built-in Vivante GC320, which is a hardware 2D accelerator which can do cool things like scaling or even be used by Wayland for compositing: http://processors.wiki.ti.com/index.php/GLSDK_Datasheet_OMAP5_6.03.00.01#GC320. Examples: http://archlinuxarm.org/forum/viewtopic.php?f=45&t=6815 and http://jas-hacks.blogspot.de/2014/03/imx6-waylandweston-31017beta.html

I don't know how much of these systems we are able to use on the OMAP without TIs support, but it may be worth looking into it for both performance and battery saving.
 
Could the M4 be used to control battery charge? ie, so you could charge the Pyra without it booting up into Linux for example; or if it's in sleep mode, and the battery gets too low, it could wake the cpu to shutdown properly? (I've had a few occasions where my battery went flat in sleep mode on the pandy).

One of the things people mentioned about DSP on the Pandora was an MP3 player, could the M4 be used for that?

Though, they might both suffer the same problem of the A15 driving the hardware?

Hmmm. Could the M4 control the battery, and just provide some sort of signal to the A15 for reporting battery information? or would that be too complicated?
 
Maybe they could, maybe they could not, all this is irrelevant as like we have seen on pandora, there is like 99% chance that nobody will be implementing such things anyway.

Edit:

I've had a few occasions where my battery went flat in sleep mode on the pandy.
This could be solved on pandora even with periodic wakeups by rtc...
 
Last edited by a moderator:
Maybe they could, maybe they could not, all this is irrelevant as like we have seen on pandora, there is like 99% chance that nobody will be implementing such things anyway.
Fair enough. I always like to think these things I'd like to dabble at, but these days I rarely finish projects where I know what I'm doing, so something totally new... :(
Edit:

I've had a few occasions where my battery went flat in sleep mode on the pandy.
This could be solved on pandora even with periodic wakeups by rtc...
Oh? How would your Pandora know if it needs to return to sleep? Can you tell whether it was the RTC or the button used to wake the Pandora?
 
Can you tell whether it was the RTC or the button used to wake the Pandora?
Yes. I don't have the information on hand right now, it's all buried in the documentation, but there's an i2c register you can look at to tell whether it was woken up because of RTC, power switch, power applied (ie, pluged in), or interrupt.
 
Can you tell whether it was the RTC or the button used to wake the Pandora?
Yes. I don't have the information on hand right now, it's all buried in the documentation, but there's an i2c register you can look at to tell whether it was woken up because of RTC, power switch, power applied (ie, pluged in), or interrupt.
Without wanting to sound stupid, but the documentation for what? The OMAP, Pandora, or Linux?
I found this post of yours about waking up via RTC. But would the wakeup reason be in the /sys/class/rtc tree or the sys/class/i2c tree?

I also found this, not sure if that's what I'm looking for though?
 
Last edited by a moderator:
Oh, right, sorry. I'm having trouble finding it now, but what you need is the detailed reference manual for either the TWL4030 or the TPS65950, they're basically the same thing for our purposes. Somewhere in that manual it'll give a list of i2c register addresses, one of which will be something along the lines of "wake response" or whatever. It's a very large manual, unfortunately.

Once you know that, you can use i2cget ("sudo opkg install i2c-tools" and then "i2cget <address>") to get a number which you can use to make a decision.
 
Back
Top