Slow start caused by USB errors

Running Debian Forky, for the last few weeks the boot up process has been slowed down 20-30 seconds by a misbehaving USB device/port.

All of my connected usb devices appear to work fine after booting, and unplugging things did not appear to speed up boot time.

How can I figure out which device is malfunctioning? And if it is the port on the hub itself that is bad, how can I disable that port? I was only able to find instructions to disable the entire hub, which would be bad because it is a reasonably expensive KVM usb switch I use to toggle between personal and work computers.

Logs from journalctl show:

Sep 13 11:44:58 pc kernel: usb 1-1: new high-speed USB device number 2 using xhci_hcd
Sep 13 11:44:58 pc kernel: usb 1-1: New USB device found, idVendor=05e3, idProduct=0610, bcdDevice=93.07
Sep 13 11:44:58 pc kernel: usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
Sep 13 11:44:58 pc kernel: usb 1-1: Product: USB2.1 Hub
Sep 13 11:44:58 pc kernel: usb 1-1: Manufacturer: GenesysLogic
// ... succesful enumerations
Sep 13 11:44:58 pc kernel: usb 1-1.4: new high-speed USB device number 10 using xhci_hcd
Sep 13 11:44:58 pc kernel: usb 1-1.4: device descriptor read/64, error -110
Sep 13 11:44:58 pc kernel: usb 1-1.4: device descriptor read/64, error -110
Sep 13 11:44:58 pc kernel: usb 1-1.4: new high-speed USB device number 11 using xhci_hcd
Sep 13 11:44:58 pc kernel: usb 1-1.4: device descriptor read/64, error -110
Sep 13 11:44:58 pc kernel: usb 1-1.4: device descriptor read/64, error -110
Sep 13 11:44:58 pc kernel: usb 1-1-port4: attempt power cycle
Sep 13 11:44:58 pc kernel: usb 1-1.4: new high-speed USB device number 12 using xhci_hcd
Sep 13 11:44:58 pc kernel: usb 1-1.4: Device not responding to setup address.
Sep 13 11:44:58 pc kernel: usb 1-1.4: Device not responding to setup address.
Sep 13 11:44:58 pc kernel: usb 1-1.4: device not accepting address 12, error -71
Sep 13 11:44:58 pc kernel: usb 1-1.4: new high-speed USB device number 13 using xhci_hcd
Sep 13 11:44:58 pc kernel: usb 1-1.4: Device not responding to setup address.
Sep 13 11:44:58 pc kernel: usb 1-1.4: Device not responding to setup address.
Sep 13 11:44:58 pc kernel: usb 1-1.4: device not accepting address 13, error -71
Sep 13 11:44:58 pc kernel: usb 1-1-port4: unable to enumerate USB device

lsusb output:

$ sudo lsusb -v -s 1:001

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Negotiated speed: High Speed (480Mbps)
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            9 Hub
  bDeviceSubClass         0 [unknown]
  bDeviceProtocol         1 Single TT
  bMaxPacketSize0        64
  idVendor           0x1d6b Linux Foundation
  idProduct          0x0002 2.0 root hub
  bcdDevice            7.01
  iManufacturer           3 Linux 7.1.13+deb14-amd64 xhci-hcd
  iProduct                2 xHCI Host Controller
  iSerial                 1 0000:02:00.0
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0019
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xe0
      Self Powered
      Remote Wakeup
    MaxPower                0mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         9 Hub
      bInterfaceSubClass      0 [unknown]
      bInterfaceProtocol      0 Full speed (or root) hub
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0004  1x 4 bytes
        bInterval              12
Hub Descriptor:
  bLength              11
  bDescriptorType      41
  nNbrPorts            10
  wHubCharacteristic 0x000a
    No power switching (usb 1.0)
    Per-port overcurrent protection
    TT think time 8 FS bits
  bPwrOn2PwrGood       10 * 2 milli seconds
  bHubContrCurrent      0 milli Ampere
  DeviceRemovable    0x00 0x00
  PortPwrCtrlMask    0xff 0xff
 Hub Port Status:
   Port 1: 0000.0503 highspeed power enable connect
   Port 2: 0000.0103 power enable connect
   Port 3: 0000.0103 power enable connect
   Port 4: 0000.0100 power
   Port 5: 0000.0100 power
   Port 6: 0000.0100 power
   Port 7: 0000.0100 power
   Port 8: 0000.0100 power
   Port 9: 0000.0100 power
   Port 10: 0000.0100 power
Device Status:     0x0001
  Self Powered

Edit: dmesg logs that actually show the boot delay:

[    5.264210] usb 1-1.4: new high-speed USB device number 10 using xhci_hcd
[   10.608714] usb 1-1.4: device descriptor read/64, error -110
[   26.224710] usb 1-1.4: device descriptor read/64, error -110
[   26.436688] usb 1-1.4: new high-speed USB device number 11 using xhci_hcd
[   31.604717] usb 1-1.4: device descriptor read/64, error -110
[   47.220717] usb 1-1.4: device descriptor read/64, error -110
[   47.332085] usb 1-1-port4: attempt power cycle
[   47.976700] usb 1-1.4: new high-speed USB device number 12 using xhci_hcd
[   52.777712] usb 1-1.4: Device not responding to setup address.
[   57.785313] usb 1-1.4: Device not responding to setup address.
[   57.992786] usb 1-1.4: device not accepting address 12, error -71
[   58.100773] usb 1-1.4: new high-speed USB device number 13 using xhci_hcd
[   62.901989] usb 1-1.4: Device not responding to setup address.
[   67.909646] usb 1-1.4: Device not responding to setup address.
[   68.116673] usb 1-1.4: device not accepting address 13, error -71
[   68.123207] usb 1-1-port4: unable to enumerate USB device