Discussion about touchpad gesture implementation details

[gexgd0419]

It's from the comment here: #3358 (comment)

To make it easier to track, I opened a discussion here. It's about my touchpad gesture implementation in PR #19823.

Touchpad exploring area

Touchpads are much smaller than the screen, so the exploring area needs to be smaller, too. So what should the touchpad area be mapped to?

• Map to the current screen.
  • Touchpads are smaller, yes, but my touchpad actually has a higher resolution (3209 x 2097) than my screen (1920 x 1080)! So at least on my laptop, mapping the touchpad to the screen is feasible.
  • Pros: Instant access to anywhere on the screen, especially for things like the Close button at the top right corner, the taskbar at the bottom, and the system tray at the bottom right corner. Also similar to the touchscreen logic.
  • Cons: Harder to navigate among very small objects.
  • VoiceOver allows to double-tap the top area of the touchpad with two fingers to go to the menu bar, and do this on the bottom area to go to the dock.
• Map to the current window.
  • Pros: A clear rule for users to understand, and a window's area is usually smaller than the entire screen.
  • Cons: Fullscreen windows and maximized windows are still quite large.
• Map to the current navigator object's parent.
  • Pros: More similar to what VoiceOver is doing, and allows granular exploring in smaller areas.
  • Cons: We'll have to limit exploring to one level at a time to avoid sudden changes of the exploring scope, and users will have to go down a level manually if they want to interact with inner objects.

We can also include all of them and provide a way to switch between them, but then we'll still need to decide which one is the default, and for users there's one more status to keep track of.

Touchpad gesture conflicts

Windows will enable built-in touchpad gestures by default when there is a precision touchpad present. NVDA can receive the raw interaction data from touchpad devices to recognize touchpad gestures, but this won't prevent the gestures from sending to the system, and the same gesture may be recognized by both NVDA and the system, causing gesture conflicts.

Unfortunately, due to system API limitation, NVDA can neither intercept touchpad gestures completely, nor change the system touchpad gesture settings on user's behalf, due to system API limitation.

There is a registry key HKLM\Software\Microsoft\Windows\CurrentVersion\PrecisionTouchpad, and changing the touchpad settings will affect the registry values, but changing the registry values won't affect the touchpad settings until explorer restarts.

On Windows 11 24H2 or later, you can call SystemParametersInfoW with SPI_SETTOUCHPADPARAMETERS to change some of the touchpad gestures programmatically, but three-finger and four-finger gesture settings are still out of reach.

So to change all the system gesture settings, we have the following options:

• Guide users to turn off touchpad gestures manually in Windows Settings.
• Modify the registry, and then restart the explorer process. More acceptable than a complete reboot, but can still interrupt user's workflow, since all file explorer windows will be closed.
• Provide both options and allow users to choose. Might confuse users further.
• Use an undocumented, internal interface IImmersiveSettingsCache. See comment.

Additionally, there are still some cases to be considered.

• Should we prevent enabling NVDA touchpad gestures completely when there's a system touchpad gesture enabled?
• What should NVDA do when the user enables a system touchpad gesture after enabling NVDA touchpad gestures?
• Should we allow NVDA touchpad gestures and system touchpad gestures to co-exist? If so, how should we avoid gesture conflicts in this case?

More advanced touchpad gestures

Currently, only taps and simple swipes are supported by the NVDA touch tracker.

The author of #19414 planned to implement more gestures, such as pinching/spreading, rotation, edge gestures, and multi-part gestures, which is great, and can benefit touchpad gestures as well, since they share the same touch tracker. Maybe we can consider merging some touchpad and touchscreen infrastructures into one base class/module later.

Compared to touchscreens, touchpads are near the keyboard, so it's also convenient to combine modifier keys with touchpad gestures, since users can press the key with one hand and perform gestures with the other hand, and this can add more layers to simple touchpad gestures.

So my question: Is it possible, or how difficult it would be, to implement modifier keys plus touchpad gestures, in the current NVDA architecture?

Also, touchpads can be pressed or clicked, which is treated as a mouse button, and this can be used as additional available gestures as well. But since many touchpads on Windows PCs are not easy to press, or can only be pressed at the bottom, I'm not sure whether it's worthy to implement this.

Default touchpad gestures

As for now, I copied the default touchscreen gestures to be the default touchpad gestures, which is not necessarily the most suitable for touchpads. But there aren't many gesture solutions I can study from, especially for touchpad gestures.

Should I keep the default touchpad gestures aligned with the NVDA touchscreen gestures, or should I learn from VoiceOver and use a set of gestures more similar to VoiceOver's gestures?

How to enable touchpad gestures

macOS VoiceOver allows enabling touchpad gestures with the VoiceOver modifier key plus a specific touchpad gesture. In order to use touchpad gestures to enable touchpad gestures, gesture recognition must be run when touchpad gestures are disabled.

Should we try this?

Or we can also use a traditional keyboard gesture to enable touchpad gestures. I've currently set it to NVDA+Ctrl+Alt+P.

[gexgd0419]

I've discovered how changes to touchpad gestures can be applied immediately.

The explorer has an internal cache for some settings, possibly to avoid reading from the registry every time. When the user changes a setting via the Settings UI, Settings will inform the explorer about this change, via an undocumented COM interface IImmersiveSettingsCache (IID = {4214f6fa-eb36-4e2f-9ca2-23fdc1832df7}), so that the explorer can reload the cache and apply the change immediately. If we change the registry without informing the explorer, it won't take effect, because the cache isn't updated. Unfortunately, touchpad gesture settings are also cached, which explains why restarting the explorer is needed, to force it to reload the cache.

The problem is that this interface is internal and undocumented, so it could be changed in future Windows versions. However, since its introduction in the first version of Windows 10 (Windows 10 1511), this interface hasn't been changed until today's Windows 11 25H2, so it's still somewhat stable.

Here's the C++ definition for the undocumented interface:

MIDL_INTERFACE("4214f6fa-eb36-4e2f-9ca2-23fdc1832df7")
IImmersiveSettingsCache : public IUnknown
{
    STDMETHOD(OnSettingChanged)(SETTING_IDENTIFIER) = 0;
    STDMETHOD(GetBOOL)(SETTING_IDENTIFIER, LPBOOL) = 0;
    STDMETHOD(GetDWORD)(SETTING_IDENTIFIER, LPDWORD) = 0;
    STDMETHOD(RegisterForSettingChange)(/* IImmersiveSettingsCacheNotification* */ LPUNKNOWN, LPDWORD) = 0;
    STDMETHOD(UnregisterForSettingChange)(DWORD) = 0;
};

In order to get a IImmersiveSettingsCache pointer, first create an instance of CLSID_ImmersiveShell ({c2f03a33-21f5-47fa-b4bb-156362a2f239}), get its IServiceProvider pointer, then call IServiceProvider::QueryService with service GUID SID_ImmersiveSettingsCache ({53660488-8855-460b-a9ab-5cfc6b5012ca}) and interface IID_IImmersiveSettingsCache.

SETTING_IDENTIFER is an integer enum, starting from 0, representing an "index" of a specific setting. When a setting is changed, OnSettingChanged should be called with the ID of the changed setting, and when the explorer receives the notification, that setting will be reloaded from the registry.

Although the interface itself is quite stable across those versions, the setting identifiers are not. Since it's an internal enum, Microsoft is free to reassign the IDs once new settings are introduced, and the IDs of touchpad gesture settings may change. Following is a table of the ID of some touchpad gesture settings across different Windows versions.

Windows version	ThreeFingerTapEnabled	FourFingerTapEnabled	ThreeFingerSlideEnabled	FourFingerSlideEnabled
Windows 10 1511 ~ 2004	0x9	0xA	0xB	0xC
Windows 11 21H2	0xE	0xF	0x10	0x11
Windows 11 22H2 ~ 25H2	0xF	0x10	0x11	0x12

So in order to be somewhat futureproof, we might need to broadcast change notifications on a larger range of setting IDs. Although a lot of IDs are for unrelated settings, since this only triggers cache reload from registry, it shouldn't have serious side effects.

Unfortunately, we can't know whether the explorer actually recognizes a specific setting ID, from the return value of OnSettingChanged. It only notifies the explorer (and other registered listeners) about a certain setting change, but you won't know how the listeners handle the notification. This makes it more difficult to ensure that the explorer actually applies the touchpad changes instead of just ignoring it.

I know that using undocumented APIs should be avoided, but as for now, this is the only way that can disable the built-in touchpad gestures immediately, without requiring an explorer restart.

What's your thought on this, and on the other questions above? @seanbudd @SaschaCowley

[harun-342]

Hello, I would be very grateful if NV Access could respond to my questions below.
Will you agree to add NVDA touchpad functionality to the NVDA kernel?
What do you think should be done to prevent conflicts between NVDA touchpad gestures and Windows touchpad gestures? As @gexgd0419 informed me, there is no way for NVDA to disable Windows touchpad gestures on its own, since NVDA is not part of Windows and Microsoft Windows currently does not offer any public API that allows us to provide touchpad-specific access or programmatically disable system touchpad gestures. Are you considering discussing this with Microsoft?
Whatever the outcome, I kindly request that you add NVDA Touchpad support to the kernel. See thread #19823 for related comments. Thank you.

[gerald-hartig]

We're not yet happy with the implementations suggested. We will work with Microsoft to establish a stable API.