Improve heuristic to detect real windows

[lwouis]

I've added 3 exceptions for buggy apps in the last 2 days: #441 #450 #455. I really don't know how to avoid having to hardcode exceptions. The risk is that we would display objects which are not windows. You can imagine for example that some apps will have a fullscreen transparent window to draw overlays on top of existing windows, or such creative tricks. AltTab uses the accessibility API, and only lists windows that declare themselves as standard windows or dialog windows.

If we only keep these, we miss out on the buggy windows like Dr Betotte which identifies as AXUnknown. On the flip-side, if we remove that restriction, we risk listing non-windows in the app.

Some ideas I had to mitigate that:

Add a whitelist preference where you can add apps bundleId and that will force AltTab to display all their "windows". The issue here is that it's not a good out-of-the-box experience since people need to know a lot to realize the issue, understand how to fix it (how to get the bundleId is non-trivial for most users). Another issue is that for these faulty apps, some of them actually have non-windows... This is the case for Steam for instance. So that whitelist approach wouldn't work here as you can blanket-approve all "windows" from Steam. You have to distinguish within the AXUnknown windows, which one is the Steam main window, and which ones are buggy dropdown menus being coded as windows.

Better heuristic to discriminate real windows from non-window. Maybe checking the size of them, and making sure it's at least 200x200 or something like that. That would avoid showing small UI pieces coded as windows. That's not enough though, as I gave the example earlier of the fullscreen transparent window trick. We would also need to check for opacity (not sure if possible).

Overall I'm going to keep hardcoding for now, but I really wish someone would share a better way of approaching this issue that doesn't involve me hardcoding things. Many users will not open an issue when they don't see the window listed, so they have a bad experience and we never know about it. There must be a way to improve our approach

[ghost]

Those are some good thoughts, I assume it's impossible to cater to every macOS app in the world due to non-standard and diverse ways of presenting 'windows' or dialog boxes in the macOS system (and yes some screen dimming apps IIRC apply a full-screen overlay over all other apps?).

A set of criteria to judge whether something is a 'window' or not sounds like a logical way to go. A human-centric logic following what a human would usually say 'yeah that's a window I'd want to alt-tab to.'

Interestingly enough: I've often wanted small dialog boxes to be alt-tabbale. They can pop up and then disappear down the very bottom where I have to minimize/hide other windows just to get to it. Really frustrating. Since it's something I want to swat away anyway, why not make such tiny 'windows' appear in AltTab?

An idea then to think about: 3-tier setting for the user for how liberally it wants to judge what's a window - liberal, moderate, conservative. (OK, maybe not political words LOL.) In conjunction, you could combine that with a some manual blocking/allowing for the odd unorthodox case that's best to be treated as a manual yes or no depending on 1/2/3 tier, but most cases can be automatic based on your automatic criteria.

Usefulness of an extreme reveal mode is to let user know more what's going on in their macOS system. Maybe hidden windows are around which they'd then rather turn off. A lot of apps do sneaky or unwanted things, like Zoom etc.

Another idea (and I think you were saying it yourself) is to extend the "Blacklists" tab to allow user to customise all this to their liking per-app such as: for each app added to list to customise: fullscreen (yes/no), 'normal' (yes/no), 'anything including dialog box' yes/no.

To improve that experience for non-advanced users, you can have a button somewhere in every thumbnail for 'Don't show windows like this' with maybe a pop-up or it brings up blacklist tab to show all the options. Then they see and can learn the options.

[lwouis]

Thanks for sharing your thoughts! Here is a response in video format. For anyone else interested in this topic, please watch the video. I go over the problem space, and why it's not easy to do, in details 👍

[ghost]

Meta comment: I watched the video, and it's very interesting, everything said makes sense, and it's fun to solve problems with you...

But, I'm slowly moving to Linux-primary in my world. I don't like Apple's trends of walled gardens, changing things without user consent, or being a closed-source black box from a privacy POV!

Your dedication to this app's development is inspiring, but there's only so much time I can give to macOS FOSS projects. Linux needs a lot of help. :) I need to spend more of my upcoming time contributing and learning on the Linux side.

If you developed a Linux AltTab (Debian+Ubuntu-compatible), I'd spend more time with you there. :)

(No idea how much the fragmentation of the 7 current most popular Linux DE's - Xfce, GNOME, Cinnamon, KDE, MATE, LXDE, LXQt - makes that an unfeasible nightmare, or if it can be a universal enough Wayland/X11 app.)

Sorry to share my news, been meaning to say it for a week. Whatever you contribute is amazing, and thank you for that. I might still continue working on random things like the traffic lights icons, but can't spend time thinking further on this particular problem, it's too big. Hope someone else can help think with you and solve it. 👍

[lwouis]

No worries at all 👍

You may find amusing that I started AltTab right after 1 year of being on Linux full time. I was running ubuntu kernel + i3. I was unhappy with the fixed window management in i3. I loved the screen real-estate management, but I hated that it was made mostly for people who want to interact with terminals mostly, and don't introduce new tools in their workflows much. For them, direct mapping is amazing. For me, I install apps every months or so, I'm discovering tools, tech, experimenting around, and I wanted something more dynamic.

I looked at Expose solutions on Linux, and found only SkippyXD as a standlone solution. As you can see the project is abandoned. There are interesting implementations in the big distros, but they are integrated in the graphical stacks and not standlone. I was not interested in running these large graphic stacks like KDE/Gnome. I liked i3. Everything snaps with almost no latency, and every pixel on screen is used. I went on some IRC channel (archlinux? I don't remember) and asked if someone could give me pointers on how to develop a successor to SkippyXD. I was told: "if you need help getting started, then you probably don't have the skills to do it". I had just experienced an amazingly welcoming community when learning Rust the previous year, and I remember being stupefied by how terrible this community was in comparison.

Anyway that's for the trivia. Basically I realized that I like drivers support, a nice cohesive menubar with wifi, sound, battery, and other critical UIs, and an ecosystem with a legacy of highly polish apps. I realized I like that better than the upsides of Linux. I don't want to sound like I dislike Linux though. It's not the case at all. This was to explain how AltTab was born, and why it was born on macOS

[ghost]

I went on some IRC channel (archlinux? I don't remember) and asked if someone could give me pointers on how to develop a successor to SkippyXD. I was told: "if you need help getting started, then you probably don't have the skills to do it".

Oh no, well I've had surprisingly good experiences with Linux non-elitism these days, I guess it depends where you're hanging out. I have heard arch tend to be the 'elitist' ones, and that across the landscape it's changed a lot now.

Your skills are obviously up to the task, no matter the platform. I'll still be around as long as I'm on macOS, and if you ever take a stab at an X11 window switcher, the Linux world would have your back big time, I know it. :) macOS indeed is very beautiful and powerful.

[lwouis]

@ethenhunt on twitter shared this tweet with me. It would be interesting to test and see which windows the Monterey Shortcuts app shows. Does it show pop-ups or HUDs as windows? Does it show windows of non-native apps (e.g. IntelliJ, Steam, Adobe apps, etc). Does it show windows on all Spaces?

[lwouis]

Does it show windows on all Spaces?

I just tested, and it does not.

[lwouis]

Closing this as I can't imagine any improvement at the moment, and there are other tickets dealing with related topics.