Am 31.07.2020 um 16:38 schrieb David Shah dave@ds0.me:
It is really looking like the OMAP5, or possibly TWL is _very_ sensitive to noise on the McPDM interface.
- My previous loopback experiment was not correct for some reason (I
think ALSA was recording from the wrong device). I tried again this time looking at the raw data coming back via McPDM and indeed the data coming back is corrupted when the audio output is corrupted.
- I tried deliberately glitching the ABE_CLKS signal (from TWL to
OMAP) on the uEVM deliberately by touching it with some tweezers to inject noise and capacitance - it is easily exposed by the inverter footprint.
Interesting! This may help Peter to mimic the issue on the uEVM.
This causes the audio to fail in exactly the same way as the Pyra. Also identically to the Pyra, disabling uplink in the OMAP fixes the problem.
Checking the signal with an oscilloscope, the clock signal does not change during while the audio is glitching after the tweezers are removed. Sadly there is no easy access to the other McPDM signals.*
Looks all very plausible now. Sampling the input data by an edge triggered flipflop in the moment the input swings from one data level to the other (this is where digital logic becomes an analog device) the bit value sampled makes it very dependent on noise on the input line or some pico seconds fluctuations of the sample point and this may even drive the flipflop into https://en.wikipedia.org/wiki/Metastability_(electronics)
So the sampling should take place while the data is stable. This is why inverting the clock polarity did work for the Pandora.
Unfortunately this doesn't help to find a good solution :(
The only question now is whether there is a less drastic way to 'desensitize' it other than disabling uplink.
I have tried to better understand omap5 and twl6040 TRM but all I found was that it should magically synchronize through the clock feedback and simply work. No configuration registers.
On Thu, 2020-07-30 at 20:37 +0200, H. Nikolaus Schaller wrote:
Am 30.07.2020 um 20:29 schrieb David Shah dave@ds0.me:
A tiny bit more insight.
It also seems possible to stop the glitching by disabling the _up_link channels on the OMAP side, either by clearing PDM_UP[1-3]_EN or setting SW_UP_RST in MCPDM_CTRL.
This actually seems slightly more reliable than disabling the loopback clock input buffer. But I suspect disabling that input buffer also breaks the uplink path, I haven't tried yet.
Disabling the uplink by disabling the uplink data input buffer, on the other hand, does not fix the glitching, so the glitching is not some strange result of the uplink data stream itself. Instead there must be some strange cross-connection between the up and down link in the OMAP5 that gets messed up by electrical noise (???). All the more mysterious...
That is certainly something to discuss with TI people (Peter Ujfalusi is still active - sent a mail a week ago about "omap-mcbsp 49022000.mcbsp: TX Buffer Overflow!").
As a worst case scenario, a patch that fully disabled the uplink channels when not being used would already improve reliability a lot. The mechanical noise of the vibra would make the internal microphone useless anyway.
Not necessarily since the mic has a sound guide and small hole on the left side of the keyboard. So it may be possible to hear the rumble but it would not make the recorded sound useless. It may even be possible to do noise cancelling for the vibra activities through aess (yes I know it is a dream)...
Best regards, Nikolaus
Best
David
On Thu, 2020-07-30 at 13:10 +0200, H. Nikolaus Schaller wrote:
Am 30.07.2020 um 13:02 schrieb David Shah dave@ds0.me:
On Thu, 2020-07-30 at 12:37 +0200, H. Nikolaus Schaller wrote:
Yes, I can confirm the effect.
After doing devmem2 0x4A0029A0 w 0x0080118 sound output stops (but aplay continues to run without notice about buffer under/overflow or alike). After restarting aplay sound is better. And vibra influence is no longer noticeable. Restoring devmem2 0x4A0029A0 w 0x1080118 makes problem come back
even while aplay runs. But still it depends on volume. So high volume + 0x1080118 is unreliable.
Hm. Is there a clock polarity setting? Either in McPDM or the twl6040? If clock polarity would be wrong, then it may depend on power supply noise if the data line is sampled correctly and the previous bit value creeps in or some random bit is read.
As far as I know there isn't one; the errata does suggest adding an external inverter to work around a clock phase issue (but I don't think that's exactly the same problem as what we see here.)
Ok, that could be the missing inverter I had hoped that it can be software controlled... Adding one in hardware would be more difficult than the missing capacitor :(
There are also the OMAP IO drive strength and slew settings to play with though.
Indeed. If the drive strengh is higher the slew rate is higher and the risk of sampling the wrong value should go down. Let's hope it goes this way...
BR, Nikolaus
> Am 30.07.2020 um 12:20 schrieb H. Nikolaus Schaller < > hns@goldelico.com>: > > Wow! > Maybe you can ask Peter Ujfalusi as he has done a lot of > previous > work for twl audio. > > I'll check all this as soon as I find time. > > BR and thanks, > Nikolaus > >> Am 30.07.2020 um 12:15 schrieb David Shah dave@ds0.me: >> >> Well, I seem to have found a solution. It is totally and >> utterly >> inexplicable, but there you go. >> >> The OMAP5 TRM says that the input buffer for MCPDM_LB_CLK >> - >> the >> clock >> going out from the OMAP5 to the TWL, that clocks the PDM >> stream >> and >> mirrors the TWL clock output - must be enabled; as the >> input >> signal is >> used internally for retiming purposes. >> >> But using devmem2 to DISABLE this input buffer, against >> the >> TRM >> advice, >> seems to fix the stream desynchronisation issue! >> >> # devmem2 0x4A0029A0 w 0x0080118 >> >> There is still a quiet a crackle when the vibra turns on >> or >> off, >> which >> really does seem like switching noise from the motor, but >> the >> stream is >> never actually corrupted and the crackle does not disrupt >> what is >> playing! >> >> I'm done with OMAP stuff for today :) but I'll experiment >> more >> over the >> weekend. If you get a chance, I would be interested to >> see if >> this is >> reproducible on your unit, too. >> >> I need to think a bit more about how this could actually >> look >> from a >> silicon point of view to see if there is a better fix. It >> is >> possible >> that disabling the input buffer means that the OMAP5 is >> using >> some >> internal clock instead that isn't disrupted by noise on >> the >> pad. >> >> It could also be related to OAMP5 errata i800 / TWL >> errata 6, >> which >> involves relative clock phases. I had ruled that out >> because >> it >> would >> be odd for power supply noise to affect that; and it >> reads >> more >> like >> dropped/duplicated samples than total loss of stream. >> >> Best >> >> David >> >> >> On Wed, 2020-07-29 at 18:32 +0100, David Shah wrote: >>> Hi Nikolaus, >>> >>> Couple of thoughts below. >>> >>> On Wed, 2020-07-29 at 19:19 +0200, H. Nikolaus Schaller >>> wrote: >>>>> Am 29.07.2020 um 18:58 schrieb H. Nikolaus Schaller >>>>> < >>>>> hns@goldelico.com>: >>>>> >>>>> >>>>>> Am 29.07.2020 um 18:04 schrieb David Shah < >>>>>> dave@ds0.me>: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> I tried adding a 470nF 0402 capacitor in what I >>>>>> hope >>>>>> was >>>>>> the >>>>>> right >>>>>> place (see attached), >>>>> >>>>> yes, from a first glance it looks to be right! >>>>> >>>>>> and replacing the associated resistor with a >>>>>> short. Sadly that made no difference, but maybe >>>>>> 470n >>>>>> isn't high >>>>>> enough >>>>>> (I also have 4.7u 0805 but I suspect that might >>>>>> be >>>>>> too >>>>>> tall. >>>>> >>>>> Hm. I hope I find some minutes to open my sample >>>>> and >>>>> try >>>>> the >>>>> same. >>>>> Then we know. >>>> >>>> Here is a full picture where I tried tp mimic the >>>> colours: >>>> >>> >>> Yes, that matches what I did. >>> >>>> Maybe the copper trace between the C and the R is too >>>> small. >>>> So a >>>> wire directly >>>> across all three solder pads could be better. Or >>>> scratching >>>> off the >>>> solder mask from >>>> the ground plane around and soldering the capacitor >>>> directly >>>> to >>>> GND. >>>> >>> >>> In my second attempt with 4.7u, I did effectively end >>> up >>> with >>> one >>> solder blob all the way across that should be pretty >>> low >>> impedance. >>> >>>> If I find time (which is unfortunately unlikely) I >>>> could >>>> add >>>> two >>>> wires to the pads of the >>>> capacitor and get them outside the Pyra to be able to >>>> measure >>>> the >>>> voltage with the oscilloscope. >>>> >>>> Some thoughts about the software side: I remember >>>> that >>>> Peter >>>> Ujfalusi >>>> long time ago mentioned >>>> different clocks. Maybe the 32kHz clock or the McPDM >>>> clock >>>> which >>>> could be used as a >>>> reference. And there was a kernel version where sound >>>> became >>>> a >>>> little >>>> better (but I wouldn't >>>> bet for it). So there may still be another effect in >>>> software. >>>> Maybe >>>> in addition so that >>>> what we observe is the sum of several effects. >>>> >>> >>> Interesting, the clocking definitely isn't quite right >>> in >>> the >>> latest >>> as >>> high power mode also used to work on 4.19 but doesn't >>> seem >>> to >>> any >>> more. >>> >>>> One question about the loopback: does it show if the >>>> PCM >>>> data >>>> is >>>> coming in the right >>>> and stable timing? E.g. speed may fluctuate but still >>>> provide >>>> perfect >>>> data mirroring. >>>> Nevertheless, fluctuating speed may become audible. >>>> >>> >>> This is true, there is some possibility for "common >>> mode >>> failure" >>> that >>> would symmetrically affect both directions. >>> >>>> But it is difficult to see how this would be >>>> influenced >>>> by >>>> the >>>> Vibra. >>>> Well, the Vibra >>>> is also controlled through some PCM channel because >>>> the >>>> motor >>>> can >>>> "play" the bass sound... >>>> >>> The Vibra seems to be set to "Input FF" mode by default >>> AIUI it >>> is >>> controlled by I2C not the PDM stream when using >>> vibra.py. >>> But >>> you can >>> set it to PDM mode in alsamixer which seems to turn it >>> always >>> on >>> (presumably a 1 is sent in the PDM stream for that >>> channel >>> by >>> default?) >>> noise that always goes away as soon as you turn it off >>> again >>> rather >>> than glitching the stream. >>> >>> It also seems to differ significantly less depending on >>> the >>> volume >>> setting - it isn't directly scaled by volume in the >>> same >>> way as >>> Vibra >>> Right glitching is; as if it is affecting something >>> later >>> in >>> the >>> signal >>> path. >>> >>> Strange that an unconnected channel also makes a >>> difference, >>> but the >>> H- >>> bridge is still presumably still switching and any H- >>> bridge >>> does >>> produce some shoot-through noise. >>> >>> I am wondering now if some config issue is making the >>> TWL >>> much >>> more >>> sensitive than it should be for some reason. >>> >>> Best >>> >>> David >>> >>>>>> I'm still proud to own the first modded Pyra, >>>>>> even >>>>>> though >>>>>> it >>>>>> didn't >>>>>> work :D >>>>> >>>>> ++ >>>>> >>>>> BR, Nikolaus >>>>> >>>>>> Best >>>>>> >>>>>> David >>>>>> >>>>>> On Wed, 2020-07-29 at 17:37 +0200, Michael Mrozek >>>>>> wrote: >>>>>>> Am Mittwoch, den 29.07.2020, 17:25 +0200 >>>>>>> schrieb H. >>>>>>> Nikolaus >>>>>>> Schaller: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>>> What I have seen is that there is room for an >>>>>>>> optional LED >>>>>>>> on >>>>>>>> the >>>>>>>> backside of the CPU board directly connected >>>>>>>> to >>>>>>>> these >>>>>>>> pins. >>>>>>>> It was >>>>>>>> left there from initial hardware debugging. >>>>>>>> It >>>>>>>> should >>>>>>>> be >>>>>>>> not >>>>>>>> too >>>>>>>> difficult >>>>>>>> to try to add a 4.7uF or 10uF capacitor there >>>>>>>> to >>>>>>>> test >>>>>>>> the >>>>>>>> result. >>>>>>> >>>>>>> Oh, that's good news! >>>>>>> Can you let me know where to solder them so >>>>>>> I'll >>>>>>> try >>>>>>> that >>>>>>> here? >>>>>>> >>>>>>>> I am not sure if that would be a good mass >>>>>>>> production >>>>>>>> solution, >>>>>>>> because it >>>>>>>> sticks to the battery bay plastics and/or the >>>>>>>> heat >>>>>>>> shield >>>>>>>> and >>>>>>>> slows >>>>>>>> down >>>>>>>> the production process and/or must be hand- >>>>>>>> soldered. >>>>>>> >>>>>>> Well, we tested various materials between the >>>>>>> CPU >>>>>>> board >>>>>>> and >>>>>>> the >>>>>>> plastics - but >>>>>>> the best solution is to put NOTHING in between. >>>>>>> >>>>>>> If we put a metal sheet in there, the heat goes >>>>>>> into >>>>>>> the >>>>>>> metal >>>>>>> and >>>>>>> heats the >>>>>>> battery - whereas almost nothing moves into the >>>>>>> heatsink >>>>>>> (where >>>>>>> it >>>>>>> should). >>>>>>> >>>>>>> So there's a gap of 0.4mm between the CPU board >>>>>>> and >>>>>>> the >>>>>>> backside, and >>>>>>> as the >>>>>>> keymat acts a bit like a cushion as well, I >>>>>>> guess a >>>>>>> small >>>>>>> part >>>>>>> like a >>>>>>> capacitor >>>>>>> could even use up to 0.6 to 0.7mm. >>>>>>> >>>>>>> So there should be enough space to put a small >>>>>>> capacitor in >>>>>>> there. >>>>>>> >>>>>>>> But there would be room and easy access on >>>>>>>> component >>>>>>>> side >>>>>>>> of >>>>>>>> the >>>>>>>> LC15 >>>>>>>> processor board to add such capacitor(s) by >>>>>>>> modified >>>>>>>> copper >>>>>>>> trace >>>>>>>> layout. >>>>>>>> So it could be something for the next mass >>>>>>>> production >>>>>>>> run >>>>>>>> (unless >>>>>>>> ED has >>>>>>>> already ordered thousands of PCBs). >>>>>>> >>>>>>> We have 500 already-produced PCBs - so I guess >>>>>>> we'd >>>>>>> need to >>>>>>> fix >>>>>>> these >>>>>>> manually. >>>>>>> But hey, MWeston had to replace 500 capacitors >>>>>>> on >>>>>>> the >>>>>>> Pandora >>>>>>> PCBs >>>>>>> manually as >>>>>>> well. >>>>>>> >>>>>>> Only bigger downside would be that we have >>>>>>> another >>>>>>> setup >>>>>>> costs >>>>>>> for >>>>>>> the next CPU >>>>>>> boards... that's around 5000 EUR alltogether :/ >>>>>>> >>>>>>> >>>>>> <photo_2020-07-29_17-03-18.jpg> >>>>> >>>>> _______________________________________________ >>>>> Kernel mailing list >>>>> Kernel@pyra-handheld.com >>>>> http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel >>> >>> _______________________________________________ >>> Kernel mailing list >>> Kernel@pyra-handheld.com >>> http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel > > _______________________________________________ > Kernel mailing list > Kernel@pyra-handheld.com > http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel