PCI: apple: bring up T600x/T602x PCIe-C tunnels without an m1n1 handoff

[wesleygrimes]

Summary

Brings up the Thunderbolt PCIe tunnel (PCIe-C) on M2 Pro/Max (T602x) and M1 Pro/Max (T600x) without an m1n1 handoff, building on the existing M1 (t8103) path. Devices behind Thunderbolt 3 docks and displays now come up. For an Apple Studio Display, that's its USB hubs and USB-C ports, camera, speakers and microphone.

PCIe-C on these chips requires apple,pciec-preinit-status = 1 from m1n1, and no m1n1 sets it, so today the tunnel is always refused. This series is opt-in with pcie_apple.tunnel_kernel_init=1. With the parameter off, nothing changes on any machine.

Commits

Commit	What it does
PCI: apple: Cold-init T600x/T602x PCIe-C ports without an m1n1 handoff	Adds the parameter and the predicate apple_pcie_tunnel_needs_cold_init(), which also refuses ports whose DART has no apple,tunable. Reuses the t8103 cold-init, prepare and setup paths, with one difference: these ports must report RUN before the root-complex config space is written, because the t8103 order hard-resets a J416c. They use Intr2AXI instead of oe-fabric. The t8103 power and resume paths (pcie->kernel_init) are not used.
thunderbolt: apple: Accept PCIe-C kernel cold init on T600x/T602x	ACIO populates the tunnel for these ports.
PCI: apple: Keep cold-initialized PCIe-C functions runtime-active	A PCI bus notifier holds a runtime-PM reference on each tunneled function. Without it, after a display power cycle the Titan Ridge xHCI suspends, its wakeup never arrives, and the display's hub never comes back. It's a notifier rather than a fixup so that it also works when pcie-apple is built as a module.
PCI: apple: Keep cold-initialized PCIe-C links up through suspend-to-idle	Uses the existing s2idle keep-link path. The stop/restart path left the port unable to train until reboot. If the link is lost while asleep, the host fails like a surprise unplug instead of restarting in place.
PCI: apple: Don't restart a quiesced PCIe-C host on resume	Pre-existing on t8103 too. A host quiesced by tunnel deactivation had all its ports restarted in resume_noirq(), which fails without a tunnel and leaves the host marked resume_failed. apple_pcie_tunnel_restore() already restarts them on reactivation.
arm64: dts: apple: t602x: Add PCIe-C DART tunables	m1n1 doesn't copy dart-tunables-instance-0 for dart-apciec*, and without it the DART fails to probe. Values were read from J416c and J414s ADTs and are identical on both machines and all ports.
arm64: dts: apple: t600x: Add PCIe-C DART tunables	Values read from a J314s (MacBookPro18,3) ADT. They differ from T602x.
tools: aurora-tb: Add a Thunderbolt PCIe test report	tools/aurora-tb/tb-pcie-report, a read-only report for testers, with a README.

Why a parameter rather than a DT property: m1n1 bundles one DTB into boot.bin for every installed kernel, so an older kernel would also see such a property. The DART tunables are harmless to older kernels, because without the handoff they never populate the tunnel.

Testing

MacBook Pro 16" M2 Pro (apple,j416s), Aurora 11.25 base and config, Apple Studio Display:

• Hot plug after boot, and connected at boot: the port cold-initializes, the DART probes with its tunables, and the xHCI (8086:15f0) enumerates the display's USB2/USB3 hubs and 05ac:1114: UVC camera, USB audio (speakers and mic) and its HID interfaces.
• All three USB-C ports (tunnel-0/1/2).
• Display power cycle: the tunnel tears down and comes back, and the hub, camera and audio return on their own.
• s2idle (from a few seconds up to 3 min): the link stays up, and the hub, camera and audio keep working.
• Unplugging the display while asleep: handled as an unplug, and replugging works.
• Without the DART tunables: the DART fails to probe and the xHCI dies about 20 s later. Without the runtime-PM reference: after a power cycle the hub only returns once power/control=on is set.

Not tested

• T600x (M1 Pro/Max) hardware. The tunables come from a macOS dump, and the cold-init order is the T602x one.
• Other T602x machines, including the 4th port on desktops (apciec3, which has no tunables, so it's refused).
• pcie-apple built as a module.
• The lost-link-while-asleep branch. In testing, an unplug during sleep was caught by the normal unplug path instead.
• The quiesced-host resume path.

Testers: boot with pcie_apple.tunnel_kernel_init=1, plug in a Thunderbolt 3 dock or display, and run tools/aurora-tb/tb-pcie-report.

Studio Display brightness isn't part of this series.

[wesleygrimes]

Known compromises in this series, and the cleaner follow-ups for each:

1. DART tunables are hardcoded in the DT (t602x-/t600x-pciec-dart-tunables.dtsi). The binding expects the bootloader to provide them. The proper fix is for m1n1 to copy dart-tunables-instance-0 for dart-apciec*, as it already does for the other PCIe-C tunables. The DT files could then be dropped, and every machine would get its own values.
2. Opt-in is a boot parameter rather than a DT property. m1n1 hands the same DTB to every installed kernel, so a DT property would also reach older kernels. Once kernels without this series are gone, the parameter could become a default or a DT property.
3. Machine compatibles are checked in the driver (apple,t6000/t6001/t6020/t6021), because T600x and T602x share apple,t6000-pciec. Adding per-SoC compatibles (e.g. "apple,t6020-pciec", "apple,t6000-pciec") to the binding and DTs would remove that check.
4. Tunneled functions are kept runtime-active instead of being woken. The underlying problem is that a runtime-suspended function behind the tunnel never gets its wakeup (PME), so its runtime PM is blocked. Delivering that wakeup would let these devices runtime-suspend again. The cost meanwhile is a little power while a Thunderbolt device is connected.

The RUN-before-config-space order in the cold init is also empirical: writing the RC tunables first hard-resets a J416c. It isn't documented anywhere.

None of these affect machines booted without pcie_apple.tunnel_kernel_init=1.

[iconidentify]

Thanks @wesleygrimes, this is great work, and thanks for the "known compromises" note. We ran two independent reviews.

Already in good shape:

• It merges cleanly onto custom/sep and onto our Thunderbolt branch (aurora-silicon#8), and builds with W=1 clean.
• DTB changes are confined to the 11 T600x/T602x boards. t8103, t8112, Neo and the Ultras are byte-identical.
• No hunk touches the DP tunnel paths or the Type-C display routing.
• With the parameter off, T600x/T602x behave as today.

M1 Pro data check: the values in t600x-pciec-dart-tunables.dtsi match a second J314s (MacBookPro18,3, macOS 27.0.1) exactly. That covers all three tuples (0x20c/0x220/0x224 with the same masks and values) on apciec0–2, and apciec3 has none. We're installing Omarchy on that J314s now, to run the cold init with a Thunderbolt 3 dock. So please keep T600x in, pending that run.

Requested changes

1. PR text and the M1 change. "Don't restart a quiesced PCIe-C host on resume" changes t8103 even with the parameter off (pcie-apple.c resume_noirq).

   • It's a correct fix: today a host quiesced by tunnel teardown gets resume_failed, and the next restore returns -EIO until replug.
   • Please say so in the description.
   • It's the same change as rryan's 18b9e789 in aurora-silicon#50 (Sep 28). Please use his commit (see 6).
   • We'll test it on an M1 with a TB3 dock: tunnel deactivated, suspend, resume, reactivate.

2. A lost link at resume on cold-initialised hosts (~3104 and ~3184).

   • The stop/restart path is still reachable, with s2idle_keep_link=0 or a stale link-down bit at suspend.
   • Your commit says that path leaves the port unable to train until reboot, and T602x has no PCIe-C block reset.
   • Please fail such a host (-ENOLINK, handled as a surprise unplug) instead of restarting it.

3. A failed cold init leaves the port clocked (~950–959). On the next plug, prepare and setup_port see it running and skip cold init, so the root-complex tunables are never written. Please stop the port on failure.

4. Runtime-PM reference (~2157–2176, ~2493).

   • It pins every tunneled function, NICs and NVMe included, and overrides power/control=auto and TLP. That costs battery whenever a dock is connected.
   • Could it be limited to the xHCI class that loses its PME, or use pm_runtime_forbid() so a user can override it?
   • Your note 4, finding the root cause, is the real fix later.

5. Nits:

   • The 250 ms busy-wait under pcie_tunnel_lock (~951) should be read_poll_timeout.
   • pr_warn_once (~110) hides the second and later refused ports.
   • tb-pcie-report reports M1 DART tunables as missing and sends M1 testers to macOS, but M1's PCIe-C DART has none by design. Please skip that check on t8103.
   • An unplug log, please. apple_nhi_pci_tunnel_deactivate() holds tb->lock while waiting up to 1 s for the reset ack, which is where rryan logged -110. Could you share a log of an unplug or display power cycle with the parameter on, so we can see whether DP teardown or replug stalls?

6. Credit. aurora-silicon#50 (rryan, Sep 27) got to the same place first:

   • the RUN-before-config-space order (the J416c hard reset comes from his reports);
   • byte-identical T602x DART tunables;
   • the same quiesced-host resume guard.

   Your series is the better base: safer opt-in, built on #8's functions, plus s2idle handling. Please add Co-developed-by: RJ Skerry-Ryan <rryan@alum.mit.edu> to the cold-init commit, use his resume commit in place of yours, and credit him for the T602x tunables. We'll close #50 as superseded, with credit, when this lands.

After the fixes:

• Default on: per our default-on policy, the plan is to turn this on by default for M2 Pro/Max once 2–4 are in, keeping the parameter as an off switch. T600x follows once the J314s run passes.
• Upstream follow-ups: your known compromises are the right ones (m1n1 copying dart-tunables-instance-0, per-SoC compatibles instead of the compatible check).
• On our side: we'll update our installer's test plan, which still calls the old "m1n1 handoff is not initialized" line normal.

Also, thanks for the Touch ID report on sepOS 27.0 (#8). It's the first M2-family keybag on 27.0.

[iconidentify]

Merged and released in 11.31. Thank you, @wesleygrimes. Your cold init worked first time on an M1 Pro. Testing it there with docks that have several controllers inside found two bugs, now fixed in follow-up commits on custom/sep:

1. DART table depth on T600x (ae0b063). The fixed PCIe-C topology sets up 42-bit, four-level tables, and the T600x PCIe-C DART walks only three levels. A dock's xHCI therefore faulted on its first DMA (NO PMD FOR IOVA) and then died. apple,dma-range = <0x0 0x0 0x1 0x0> on the four T600x DART nodes keeps the domain at three levels. T602x is unchanged.
2. Bus scan before the dock is ready (f44aa74). On a CalDigit TS3 Plus, the FL1100 and ASM1142 controllers train their links after the tunnel does, behind switch ports that aren't hotplug-capable. The scan right after cold init missed them for good, so the USB ports they serve stayed dead. The one-second wait from the t8103 path only ran with kernel_init set. A shared helper now waits until the link has been up for one second, before the first scan and before the restore rescan, on every PCIe-C host. Your Studio Display has a single xHCI that's ready at link-up, which is why this didn't show up there.

Also:

• tunnel_kernel_init is now on by default, with =0 as the off switch (77f974b). tb-pcie-report and its README are updated to match.
• igb and igc are built for dock Ethernet.

Tested on builds carrying exactly this Thunderbolt code (11.29–11.31):

• M1 Pro (J314s), CalDigit TS3 Plus: all five dock functions bind, with zero DART faults. A USB keyboard and mouse, webcam, audio, Ethernet at 1 Gb/s and a USB SSD all work behind it.
• M1 Pro (J314s), CalDigit TS4: cold init on the new default with no parameter, and sleep/wake with the dock attached; both displays came back.
• M1 13" (J293), TS4: sleep/wake with the dock attached, which exercises your resume change on t8103.

Still untested:

• M1 Max and M2 Max;
• an M2 Pro/Max with a multi-controller dock.

A request: 11.31 also turns the Neural Engine on for the M2 Pro. It has the same die-0 ANE as the M2 Max and uses the same ane_t6021 driver, but has never run on an M2 Pro. If you can test it on your J416s:

1. It needs the ANE firmware, fetched with Joshua Warren's omarchy-ane-firmware-fetch from omarchy-ane.
2. Then quote:

   modinfo -F filename ane_t6021
   ls -l /dev/accel/
   sudo dmesg | grep -iE 'ane_t6021|\bane\b'
   

3. If you have omarchy-ane's tools, also run omarchy-ane-check --smoke.

A run with a multi-controller TB3 dock would be welcome too. The installer's --agent-prompt has the full plan. Reports go to https://github.com/iconidentify/aurora-linux/issues (the format is in #6).

🤖 Generated with Claude Code

https://claude.ai/code/session_01FwdG63dRaMrRbYrUBkSa1i