Hi,
sometimes on our OS, the USB ports (as well as the modem) don't work. Not even a mouse or a similar low-power device works in that case.
The hardware should be fine, as IF it works after booting up, it works totally reliably.
I've attached a dmesg output from a user of the boot process when USB does NOT work.
I've also asked him to send a dmesg after he plugs in a device, so we can see what error USB produces.
But there are already a few interesting error messages in the dmesg. It seems the USB bus is found.. but produces an error. What's error -110 ?
Any idea how to debug this further?
AFAIR this effect changes based on the U-Boot / MLO setup you use. I remember that I tried various U-Boot / MLO combinations when starting the mass production - and one ALWAYS had a failing USB. Didn't test that thoroughly, so take it with a grain of salt. But maybe some initialization parameters in the board file or similar are off?
[7.084037] ehci-omap 4a064c00.ehci: EHCI Host Controller [7.100253] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1 [7.114047] ehci-omap 4a064c00.ehci: irq 106, io mem 0x4a064c00 [7.154177] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00 [7.170718] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.06 [7.183576] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [7.195528] usb usb1: Product: EHCI Host Controller [7.204562] usb usb1: Manufacturer: Linux 5.6.19-daveiii-pyradef-aufs ehci_hcd [7.216372] usb usb1: SerialNumber: 4a064c00.ehci [7.225588] hub 1-0:1.0: USB hub found [7.233218] hub 1-0:1.0: 3 ports detected [7.604098] usb 1-2: new high-speed USB device number 2 using ehci-omap [7.706183] ehci-omap 4a064c00.ehci: port 2 reset error -110 [16.474154] usb usb1-port2: Cannot enable. Maybe the USB cable is bad? [16.489352] usb usb1-port2: attempt power cycle
Hi, thanks for the console logs! They at least give a first idea about the issue.
I have added Andreas and Tony to get more experience into the discussion.
The history of similar bug reports seem to be very old (ca. 13 years for BeagleBoard).
Googling did reveal that we (LetuxOS) had seen a similar issue on the jz4780 based CI20 board:
https://lists.goldelico.com/pipermail/letux-kernel/2020-September/005730.htm...
But the reason is likely completely different (because there is no ehci-omap in an Ingenic SoC). But EHCI (USB2 interface) is a standard module used in many chipsets.
Am 09.06.2021 um 12:31 schrieb Michael Mrozek EvilDragon@openpandora.org:
Hi,
sometimes on our OS, the USB ports (as well as the modem) don't work. Not even a mouse or a similar low-power device works in that case.
The hardware should be fine, as IF it works after booting up, it works totally reliably.
I've attached a dmesg output from a user of the boot process when USB does NOT work.
I've also asked him to send a dmesg after he plugs in a device, so we can see what error USB produces.
But there are already a few interesting error messages in the dmesg. It seems the USB bus is found.. but produces an error. What's error -110 ?
Any idea how to debug this further?
AFAIR this effect changes based on the U-Boot / MLO setup you use. I remember that I tried various U-Boot / MLO combinations when starting the mass production - and one ALWAYS had a failing USB.
During my quick google searches I found some connection with U-Boot. U-Boot usually also configures/resets the EHCI module so that a device can boot from an USB disk.
E.g.: https://lists.denx.de/pipermail/u-boot/2013-December/168271.html
There is also very old (!) omap5 specific ehci u-boot discussion:
https://patchwork.ozlabs.org/project/uboot/patch/5193AFEC.60700@mm-sol.com/
But this may not be the code anyone of us is using. So it may be the wrong direction to look for hints.
Anyways: yes there is likely a connection between U-Boot and kernel EHCI (unless kernel driver does a full reset and fresh initialization and wipes out any settings done by u-boot).
Therefore, it could also depend on whether the mouse is already connected during U-Boot or plugged in afterwards.
Didn't test that thoroughly, so take it with a grain of salt. But maybe some initialization parameters in the board file or similar are off?
[7.084037] ehci-omap 4a064c00.ehci: EHCI Host Controller [7.100253] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1 [7.114047] ehci-omap 4a064c00.ehci: irq 106, io mem 0x4a064c00 [7.154177] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00 [7.170718] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.06 [7.183576] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [7.195528] usb usb1: Product: EHCI Host Controller [7.204562] usb usb1: Manufacturer: Linux 5.6.19-daveiii-pyradef-aufs ehci_hcd [7.216372] usb usb1: SerialNumber: 4a064c00.ehci [7.225588] hub 1-0:1.0: USB hub found [7.233218] hub 1-0:1.0: 3 ports detected [7.604098] usb 1-2: new high-speed USB device number 2 using ehci-omap [7.706183] ehci-omap 4a064c00.ehci: port 2 reset error -110
https://elixir.bootlin.com/linux/v5.6.19/source/drivers/usb/host/ehci-hub.c#...
Error -110 is ETIMEDOUT which could be that the EHCI module is no longer responding. For whatever reasons.
Maybe the timout of 1000us is too short?
[16.474154] usb usb1-port2: Cannot enable. Maybe the USB cable is bad?
https://elixir.bootlin.com/linux/v5.6.19/source/drivers/usb/core/hub.c#L2953
[16.489352] usb usb1-port2: attempt power cycle
https://elixir.bootlin.com/linux/v5.6.19/source/drivers/usb/core/hub.c#L5243
BR, Nikolaus
Hi,
On 6/9/21 12:31 PM, Michael Mrozek wrote:
Hi,
sometimes on our OS, the USB ports (as well as the modem) don't work. Not even a mouse or a similar low-power device works in that case.
The hardware should be fine, as IF it works after booting up, it works totally reliably.
I've attached a dmesg output from a user of the boot process when USB does NOT work.
I've also asked him to send a dmesg after he plugs in a device, so we can see what error USB produces.
But there are already a few interesting error messages in the dmesg. It seems the USB bus is found.. but produces an error. What's error -110 ?
I believe -110 is -ETIMEDOUT. From what I know of the Linux kernel USB code, it's fairly common to send a USB request to the HW USB controller, and then wait a fixed time for the controller to run the callback function of the request. If this doesn't happen within the given time span, the request will fail with -ETIMEDOUT as error code.
Any idea how to debug this further?
AFAIR this effect changes based on the U-Boot / MLO setup you use. I remember that I tried various U-Boot / MLO combinations when starting the mass production - and one ALWAYS had a failing USB. Didn't test that thoroughly, so take it with a grain of salt. But maybe some initialization parameters in the board file or similar are off?
[7.084037] ehci-omap 4a064c00.ehci: EHCI Host Controller [7.100253] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1 [7.114047] ehci-omap 4a064c00.ehci: irq 106, io mem 0x4a064c00 [7.154177] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00 [7.170718] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.06 [7.183576] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [7.195528] usb usb1: Product: EHCI Host Controller [7.204562] usb usb1: Manufacturer: Linux 5.6.19-daveiii-pyradef-aufs ehci_hcd [7.216372] usb usb1: SerialNumber: 4a064c00.ehci [7.225588] hub 1-0:1.0: USB hub found [7.233218] hub 1-0:1.0: 3 ports detected [7.604098] usb 1-2: new high-speed USB device number 2 using ehci-omap [7.706183] ehci-omap 4a064c00.ehci: port 2 reset error -110 [16.474154] usb usb1-port2: Cannot enable. Maybe the USB cable is bad? [16.489352] usb usb1-port2: attempt power cycle
Looks like it manages to do at least partial arbitration, and find some devices, but then times out on a reset request to one of the devices. There are a lot more USB debug and tracing options to enable in the kernel, but I never really figured out how...
I've done some minor dives in the Linux USB stack, but it is far from trivial (to me at least), and many drivers are unique in their own way. I wish I could provide you with a better answer
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
The main source of USB unreliability I've seen was the hub chip, which takes HSIC from the OMAP and provides the various downstream USB ports.
In particular, I think there was an ordering sensitivity between the initialisation/reset of the hub chip that the kernel driver did out-of- band via I2C/GPIO and the actual USB probing of the hub chip (perhaps because HSIC doesn't deal with a device dropping off the bus, as it does when the hub is reset as well as conventional USB).
I did attempt to improve things with a U-boot patch: https://dev.pyra-handheld.com/kernel/pyra-uboot/-/commit/6cfcca4bfb30716d807...
This patch kept the hub in reset until the kernel initialised it, so it didn't appear as a USB device due to being brought out of reset by U- boot; be probed as a USB device by the kernel; then be reset by the kernel hub driver but not re-probed again as a USB device.
But evidently, this might not be an entire solution after all.
Best
gatecat
On Wed, 2021-06-09 at 12:31 +0200, Michael Mrozek wrote:
Hi,
sometimes on our OS, the USB ports (as well as the modem) don't work. Not even a mouse or a similar low-power device works in that case.
The hardware should be fine, as IF it works after booting up, it works totally reliably.
I've attached a dmesg output from a user of the boot process when USB does NOT work.
I've also asked him to send a dmesg after he plugs in a device, so we can see what error USB produces.
But there are already a few interesting error messages in the dmesg. It seems the USB bus is found.. but produces an error. What's error -110 ?
Any idea how to debug this further?
AFAIR this effect changes based on the U-Boot / MLO setup you use. I remember that I tried various U-Boot / MLO combinations when starting the mass production - and one ALWAYS had a failing USB. Didn't test that thoroughly, so take it with a grain of salt. But maybe some initialization parameters in the board file or similar are off?
[7.084037] ehci-omap 4a064c00.ehci: EHCI Host Controller [7.100253] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1 [7.114047] ehci-omap 4a064c00.ehci: irq 106, io mem 0x4a064c00 [7.154177] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00 [7.170718] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.06 [7.183576] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [7.195528] usb usb1: Product: EHCI Host Controller [7.204562] usb usb1: Manufacturer: Linux 5.6.19-daveiii-pyradef-aufs ehci_hcd [7.216372] usb usb1: SerialNumber: 4a064c00.ehci [7.225588] hub 1-0:1.0: USB hub found [7.233218] hub 1-0:1.0: 3 ports detected [7.604098] usb 1-2: new high-speed USB device number 2 using ehci-omap [7.706183] ehci-omap 4a064c00.ehci: port 2 reset error -110 [16.474154] usb usb1-port2: Cannot enable. Maybe the USB cable is bad? [16.489352] usb usb1-port2: attempt power cycle
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Hi, can you please copy to Tony and Andreas and the letux mailing list?
Am 09.06.2021 um 19:37 schrieb gatecat gatecat@ds0.me:
The main source of USB unreliability I've seen was the hub chip, which takes HSIC from the OMAP and provides the various downstream USB ports.
In particular, I think there was an ordering sensitivity between the initialisation/reset of the hub chip that the kernel driver did out-of- band via I2C/GPIO and the actual USB probing of the hub chip (perhaps because HSIC doesn't deal with a device dropping off the bus, as it does when the hub is reset as well as conventional USB).
Yes, that is right. I remember that the chip was initially a little flaky but was stable for years after fixing device trees and overall setup.
One issue is AFAIR the sequence of reset and i2c.
The driver can handle reset of the chip but the driver is only loaded if the chip responds to i2c. AFAIR i2c stops working while in reset state (but I may be wrong and the issue was slightly different).
I did attempt to improve things with a U-boot patch: https://dev.pyra-handheld.com/kernel/pyra-uboot/-/commit/6cfcca4bfb30716d807... https://dev.pyra-handheld.com/kernel/pyra-uboot/-/commit/6cfcca4bfb30716d807f9e2148e6c7b1d644847e
This patch kept the hub in reset until the kernel initialised it, so it didn't appear as a USB device due to being brought out of reset by U- boot; be probed as a USB device by the kernel; then be reset by the kernel hub driver but not re-probed again as a USB device.
Hm. Does this work at all? AFAIR if the chip is in reset by U-Boot it won't respond to i2c and the driver is not probed/loaded.
But evidently, this might not be an entire solution after all.
But be part of a better understanding.
BR and thanks, Nikolaus
Best
gatecat
On Wed, 2021-06-09 at 12:31 +0200, Michael Mrozek wrote:
Hi,
sometimes on our OS, the USB ports (as well as the modem) don't work. Not even a mouse or a similar low-power device works in that case.
The hardware should be fine, as IF it works after booting up, it works totally reliably.
I've attached a dmesg output from a user of the boot process when USB does NOT work.
I've also asked him to send a dmesg after he plugs in a device, so we can see what error USB produces.
But there are already a few interesting error messages in the dmesg. It seems the USB bus is found.. but produces an error. What's error -110 ?
Any idea how to debug this further?
AFAIR this effect changes based on the U-Boot / MLO setup you use. I remember that I tried various U-Boot / MLO combinations when starting the mass production - and one ALWAYS had a failing USB. Didn't test that thoroughly, so take it with a grain of salt. But maybe some initialization parameters in the board file or similar are off?
[7.084037] ehci-omap 4a064c00.ehci: EHCI Host Controller [7.100253] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1 [7.114047] ehci-omap 4a064c00.ehci: irq 106, io mem 0x4a064c00 [7.154177] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00 [7.170718] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.06 [7.183576] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [7.195528] usb usb1: Product: EHCI Host Controller [7.204562] usb usb1: Manufacturer: Linux 5.6.19-daveiii-pyradef-aufs ehci_hcd [7.216372] usb usb1: SerialNumber: 4a064c00.ehci [7.225588] hub 1-0:1.0: USB hub found [7.233218] hub 1-0:1.0: 3 ports detected [7.604098] usb 1-2: new high-speed USB device number 2 using ehci-omap [7.706183] ehci-omap 4a064c00.ehci: port 2 reset error -110 [16.474154] usb usb1-port2: Cannot enable. Maybe the USB cable is bad? [16.489352] usb usb1-port2: attempt power cycle
Kernel mailing list Kernel@pyra-handheld.com mailto:Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Kernel mailing list Kernel@pyra-handheld.com mailto:Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
On Wed, 2021-06-09 at 20:39 +0200, H. Nikolaus Schaller wrote:
Hi, can you please copy to Tony and Andreas and the letux mailing list?
Am 09.06.2021 um 19:37 schrieb gatecat gatecat@ds0.me:
The main source of USB unreliability I've seen was the hub chip, which takes HSIC from the OMAP and provides the various downstream USB ports.
In particular, I think there was an ordering sensitivity between the initialisation/reset of the hub chip that the kernel driver did out-of- band via I2C/GPIO and the actual USB probing of the hub chip (perhaps because HSIC doesn't deal with a device dropping off the bus, as it does when the hub is reset as well as conventional USB).
Yes, that is right. I remember that the chip was initially a little flaky but was stable for years after fixing device trees and overall setup.
One issue is AFAIR the sequence of reset and i2c.
The driver can handle reset of the chip but the driver is only loaded if the chip responds to i2c. AFAIR i2c stops working while in reset state (but I may be wrong and the issue was slightly different).
Hopefully, this shouldn't happen because the driver brings the chip out of reset before trying to talk to it over I²C...
I did attempt to improve things with a U-boot patch:
https://dev.pyra-handheld.com/kernel/pyra-uboot/-/commit/6cfcca4bfb30716d807...
This patch kept the hub in reset until the kernel initialised it, so it didn't appear as a USB device due to being brought out of reset by U- boot; be probed as a USB device by the kernel; then be reset by the kernel hub driver but not re-probed again as a USB device.
Hm. Does this work at all? AFAIR if the chip is in reset by U-Boot it won't respond to i2c and the driver is not probed/loaded.
But evidently, this might not be an entire solution after all.
But be part of a better understanding.
BR and thanks, Nikolaus
Best
gatecat
On Wed, 2021-06-09 at 12:31 +0200, Michael Mrozek wrote:
Hi,
sometimes on our OS, the USB ports (as well as the modem) don't work. Not even a mouse or a similar low-power device works in that case.
The hardware should be fine, as IF it works after booting up, it works totally reliably.
I've attached a dmesg output from a user of the boot process when USB does NOT work.
I've also asked him to send a dmesg after he plugs in a device, so we can see what error USB produces.
But there are already a few interesting error messages in the dmesg. It seems the USB bus is found.. but produces an error. What's error -110 ?
Any idea how to debug this further?
AFAIR this effect changes based on the U-Boot / MLO setup you use. I remember that I tried various U-Boot / MLO combinations when starting the mass production - and one ALWAYS had a failing USB. Didn't test that thoroughly, so take it with a grain of salt. But maybe some initialization parameters in the board file or similar are off?
[7.084037] ehci-omap 4a064c00.ehci: EHCI Host Controller [7.100253] ehci-omap 4a064c00.ehci: new USB bus registered, assigned bus number 1 [7.114047] ehci-omap 4a064c00.ehci: irq 106, io mem 0x4a064c00 [7.154177] ehci-omap 4a064c00.ehci: USB 2.0 started, EHCI 1.00 [7.170718] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.06 [7.183576] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [7.195528] usb usb1: Product: EHCI Host Controller [7.204562] usb usb1: Manufacturer: Linux 5.6.19-daveiii-pyradef-aufs ehci_hcd [7.216372] usb usb1: SerialNumber: 4a064c00.ehci [7.225588] hub 1-0:1.0: USB hub found [7.233218] hub 1-0:1.0: 3 ports detected [7.604098] usb 1-2: new high-speed USB device number 2 using ehci-omap [7.706183] ehci-omap 4a064c00.ehci: port 2 reset error -110 [16.474154] usb usb1-port2: Cannot enable. Maybe the USB cable is bad? [16.489352] usb usb1-port2: attempt power cycle
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel
Kernel mailing list Kernel@pyra-handheld.com http://pyra-handheld.com/cgi-bin/mailman/listinfo/kernel