Swipe Gestures

[SaswatB]

Based on #2355
Resolves #730
Thanks to @ris58h for the proof of concept!

This PR builds on top of the original PR:

• merges with the latest master & updates the control panel to use the new settings components
• implements 360 gesture navigation
• adds a popover for explaining how these new gestures can conflict with system gestures

The only outstanding issue is #2355 (comment) AltTab uses background threads to process system events but I couldn't make a separate thread that accepts touch events., but I wasn't able to find any actual issues when using this.

[lwouis]

Hi,

Thank you for sharing this great PR! I've reviewed the code and it was easy to follow. I'll test it as soon as I'm done with a big rework I'm working on.

Some topics to discuss meanwhile:

• There may be a way for us to disable System Settings 3-finger shortcuts. We probably detect if they are enabled, and either disable it silently, or ask the user yes/no. If we can't set toggle these, we could at least open System Settings in the right place. We already do that for permissions.
• Should this be off by default? I'm not sure which way would be better. On by default is surprising, but it creates discoverability. I fear that otherwise, being hidden away in the corner tab, it may be unseen by > 90% of users
• No 4-finger swipe option? It may be useful, no? Some users may want to keep 3-finger macOS swipes, and use AltTab on 4-finger swipes.
• I wonder how the swipe-to-navigate-thumbnails feel. I'll have to QA it carefully. Based on the code, I'm not sure it will feel great. The way Windows 11 does it is amazing. They essentially track a hidden cursor I believe. So you can swipe slow or fast, with great precision, and when you feel it should select the other thumbnails, it does.

Thank you 🙇

[SaswatB]

Thanks for the review and feedback @lwouis!

1. I don't think we should disable existing gestures, I've added a button that directly opens the settings pane instead.
2. I'm fine either way here, I've updated it to be enabled by default for now and set to 3 finger mode as that should fit well with macOS's default four finger horizontal and three finger vertical swipes.
3. I've added a four finger option
4. I've rewritten the gesture handling logic in an attempt to match Windows more closely, let me know what you think!

[lwouis]

Hey @SaswatB,

Thank you for your message

I could finally deep dive on this PR today. I made some changes to it. Here's the current code with my changes.

As I suspected, navigating the thumbnails with the swipe motion is not the best experience. It's surprisingly better than I expected, but it's still a bit clunky. For instance, there is no diagonal navigation. I think we should rework it to create some kind of virtual cursor, as I believe Windows 11 does.

I'm not sure how to move forward:

• Not ship anything until it's reworked
• Ship with no navigation, but can open the switcher with a swipe
• Ship with everything as-is, and improve later

I'm leaning towards the last option.

Another thing we need to figure out is to avoid the gesture passing-through to the active app when AltTab is open. I've started trying to implement that in the branch I linked.

Do you think you could implement the virtual cursor navigation? I think it shouldn't be too hard. We want the gestures to move this invisible cursor around. They we can check if it's above a thumbnail, in which case we select it. This is actually probably easier than the current implementation.

What do you think?

Thank you 🙇

[lwouis]

I'd dug a bit if we could detect if there are swipe conflicts.

There are 2 settings in System Settings which can bind swipes an action:

We can read the state of swipe-binding from the OS: defaults read com.apple.AppleMultitouchTrackpad. Here are the 2 relevant keys for 3 and 4 fingers, horizontal swipes: TrackpadFourFingerHorizSwipeGesture, TrackpadThreeFingerHorizSwipeGesture.

These preferences can have each the following values:

• 0: swipe will not trigger any action
• 1: swipe will trigger Swipe between pages
• 2: swipe will trigger Swipe between full-screen applications

It's interesting to note that if the user select Swipe between full-screen applications: with 3 fingers, it will work with 3 and also with 4 fingers. This means that both keys are set to 2.

Overriding the values doesn't seem to work. For instance, running defaults write com.apple.AppleMultitouchTrackpad TrackpadThreeFingerHorizSwipeGesture 0 will set the value to 0, but System Settings will still show the binded action instead of Off. And Swiping is enabled in practice. killall Dock didn't help. Maybe a restart would take those into account. It's not usable for us though.

I'm thinking that it would be nice to detect conflicts on launch, and ask the user with a dialog:

• Open Trackpad Settings and review?
• Disable swiping to trigger AltTab?
• Ignore conflicts?

Update: there is also one Accessibility feature which uses 3-finger mouvement:

We can also detect its usage using defaults. The key is TrackpadThreeFingerDrag. Value is 1 if it's enabled.

Thus we could detect a conflict if any of the 3 keys has a non-0 value.

[lwouis]

Update: more problems. macOS seems to have hidden gestures. They are not documented, but users can, by default:

• 3-finger swipe in any direction, to select text on a page
• 4-finger swipe in any direction, to scroll on a page

I don't think we can detect these. I don't think users can turn them off as well. This makes it so that if AltTab is running, the user may lose those features. More precisely, AltTab may appear and react, and at the same time the user is using the native features in the background (selecting or scrolling). The code in the branch I shared does exactly that.

In general, when we call CGEvent.tapCreate with options: .defaultTap, it's supposed to let us block events. That's how it works for keyboard taps today in AltTab. We don't want the active app to receive key presses that are meant for AltTab.

However for gesture taps, it seems that we can't absorb gesture events. Even when we return nil to the callback, the system is getting the select text or scroll behavior. Perhaps these built-in behavior are triggered before the tap, as is the case for things like Mission Control.

I did a lot of research on various git repos to try and find a way to intercept these events, so the active app doesn't receive them. I thought I found this app which does it. It turns out that it triggers the native command+tab, and that switcher will block multitouch events while it's open. That's very convenient. I wish we could do that, but it likely uses private APIs we can't access.

[lwouis]

One avenue we could explore is replacing the CGEvent.createTap implementation with calls to the private APIs of the MultitouchSupport.framework.

It seems that Karabineer is using it. Also this small app Origami could be a good case-study. I tried to use it to understand. However, for some reason, I don't seem to have the MultitouchSupport.framework on my mac. Maybe I deleted it by mistake? If anyone could share a zip here, it would really help. In the meanwhile, I can't compile any of these project since it can't link to the framework at build-time.

My goal would be to see if using these APIs, we are able to capture the events, and avoid having them pass-through to the active app like is the case with CGEvent.createTap

[SaswatB]

Hey @lwouis!
I'm surprised diagonal doesn't work for you, I've tested with a large number of windows and swiping diagonally works fine for me (it technically flashes with the thumbnail between the diagonal ones, but I've observed the same behavior on Windows).
If you can show me with a screen recording or something visual what the issue is I can take an attempt at fixing, or worst case possibly rewriting the swiping logic again. I've used Window's swiping before (which is why I'm so interested in this feature) and this felt similar to me. I'm using displacement instead of velocity to track the gesture which is already similar to a virtual cursor.

I don't really have a solution for preventing the gesture passthrough, there's hypothetically a way in the quartz tap to stop event propagation but I haven't gotten that to work.
I have a poc using private apis here, but there doesn't seem to be a way to prevent event propagation. Overall I don't see much difference between this and the quartz tap so I'd recommend against using it.
https://github.com/SaswatB/alt-tab-macos/tree/private-swipes

I think adding conflict detection like that could be nice for a follow-up pr when there's time, and in the mean time a compromise could be keeping this feature disabled by default. As it stands this has been a feature that's been attempted for a number of years, I'm hoping we can make it in with the momentum we have right now.

[lwouis]

Hi,

Indeed, I think there is no benefit to the private APIs:

• They don't seem to be runnable off main-thread, which we would like ideally
• They don't seem to be able to block/absorb events. I looked through every github repo using those and found nothing interesting. The last param of MTDeviceStart seems to be about verbosity. Regarding MTFrameCallbackFunction, it's not clear that it actually has a return value at all. I tried passing -1, 0, and 1, and couldn't see any difference.

Going forwards, since there are so many issues with this feature, indeed we have to disable it by default. Power-users can then enable it and accept the limitations and actively mitigate them on their side.

The last thing before we ship this is to fix the UI. Currently there are 2 issues:

• Broken tabs layout
• The dropdown doesn't extend when it contains a long text

Normal tabs	Broken tabs

Text fits	Text is truncated

Could you please look into fixing those? I've pushed the swipe branch which is up to date with master and with this PR. Please work on top of it if possible.

Thank you 🙇

[lwouis]

Regarding the swiping conflicts and thumbnails browsing, I've recorded this video. I thought it's complex to discuss with text only, and a video would be much clearer 👍

Could you please have a look?

Note: in the video I mentioned that I email BTT's author. He's already replied to me. What a legend. Here's his reply:

unfortunately I also can/do not really block trackpad events.
For some gestures BTT might freeze the mouse cursor,
but it uses simple tricks to do that (basically calling cgswarpmouseposition whenever a trackpad event happens and maybe hiding the mouse cursor when necessary ). For some gestures I think I use CGEventTaps to block scroll or click events - this can get quite tricky.

But especially for three finger gestures I don’t think I do anything at all - just recommending users to disable the native trackpad gestures.

Warping the mouse is interesting. I'm thinking that my solution of having a big invisible window may be even better for our use-case since it would absorb every event natively. It would not deal with the OS gestures if they are ON, but it would deal with everything else. And for those, we could ask the user to disable them. That would also allow us to not mention the Accessibility Trackpad Dragging since that case would be neutralized by the transparent window.

[SaswatB]

Thanks for looking through this, I've updated this branch to include your changes and I've fixed the truncated text in the dropdown as well as the gesture trigger row height.

I noticed an interesting behavior based on your video, I wasn't having issues with the three finger drag affecting page scroll because I use three finger swipe up for mission control. For reference this is what I use with system shortcuts along with the three finger swipe to switch windows:

If we want a more robust solution to capture trackpad events such as a large transparent window it might be best for a follow-up pr as at least I likely won't have time to investigate such for a few weeks.

[lwouis]

Thank you for your message

I'll try to implement the fullscreen invisible window trick. I'll release the feature soon, with or without the trick 👍

I've also decided to remove Shortcut 4 and 5. Looking at the Debug Profiles in the github issues here, there are barely used. It will clean things up to have only 3 + the gesture.

Thank you

[lwouis]

I was able to find a solution for the issue. I generate a synthetic mousewhell event when we get a 3/4-finger swipe event. This way, any ongoing scroll is stopped.

Using an invisible didn't work because once a scrolling event is started, it will continue, even if the window is no longer under the mouse, or in focus. we have to break that scrolling event.

The native command+tab launcher doesn't have this issue because it's not started by a swipe. However, you can start a scroll on the trackpad, and keep it going as you command+tab. You will see that they don't stop it. It's not a problem for them since it's trackpad (scroll) + keyboard (command+tab). But in our case, we are trackpad (scroll) + trackpad (alttab), so we need to deal with it. The synthetic event does the trick nicely. I don't think it will cause any undesired side-effect either. Hopefully I'm not missing a scenario here.

I thought to still add the invisible window for another reason. That is: to stop mouse and trackpad events from reaching behind the switcher. I noticed that the native command+tab doesn't let anything through.

After thinking about it more, I think it's actually nice to be able to scroll behind to see something without stopping AltTab. I may revisit this, but for now I'm fine with it. Swipping works well with the new synthetic event!

Thank you 🙇

[lwouis]

The feature shipped in the latest release 🥳

The only thing I've now like to see improved with the trackpad, is the navigation. Using the virtual cursor I mentioned. That would be really nice 👍

[SaswatB]

Awesome to hear, thanks for everything!