Option to keep center button always visible as a way to activate Fly Pie.

[Vinnyboiler]

I'm a bit nervous about discussing this as it's something so specific to my setup that it's probably not even worth bringing up but I would love an option to have a button on screen at all times to activate a custom menu.

I'm currently running Fly-Pie on my Surface Pro 7 running Pop-OS with the Surface Kernel and when I don't have my keyboard attached I don't have an easy way to activate Fly-Pie.

You could have the button be moved while held down and activate on release.

[Schneegans]

This is definitely something worth a discussion as Fly-Pie looks like being made for touch devices but having no obvious way to summon a menu without a keyboard is problematic. The idea of a "Floating Button" came up in several comments around the internet but personally I am a bit hesitant as I see some problems:

• Where should the menu show up when the floating button is activated? At the button position or in the center of the screen?
• If opened in the screen center, this requires quite a lot of hand travel and precise aiming which would reduce the efficiency.
• If opened at the button position, some items will not be selectable as they end up off-screen. Usually you would place this button at a screen edge or corner to prevent occlusion of important content. Of course, you could design the menu in such a way that all items are positioned in the opposite direction but then you couldn't move the button around easily....

If you use touch input, an alternative solution would be to assign a Fly-Pie menu to a specific multitouch gesture. I haven't tried this yet (I have no touch device). There are multiple approaches here (also depending on whether you are using X11 or Wayland), but for example Fusuma looks promising.

Also, for Fly-Pie 9 I'll investigate whether it could be possible to summon a menu by pressing a mouse button only. This could be an alternative if you use a stylus for input.

What do you think?

[Vinnyboiler]

I believe it should show up from the button position, you could have the button push away from the edge to include all items so it doesn't require too much hand travel. I made a rough demonstration of the feature, Click activates the menu, hold moves the menu and click again on the centre closes the menu.

Click.and.drag.mp4

[Schneegans]

Nice mockup video 😄

I'll think a about this. It seems to be a great way to make Fly-Pie usable on keyboard-less PCs. However, this could also be useful beyond Fly-Pie. We could consider creating a separate extension for placing such buttons on the screen. They could simply execute any shell command when activated...

[Schneegans]

I started working on this! It's still work-in-progress, but you can test it here: #165!

[Schneegans]

So... #165 got merged to the develop branch! As I have no touch device, I would appreciate it if you could test the new feature! You can test it by installing the current development version of Fly-Pie! (Feel free to ask is anything is unclear in this guide). Then:

• Select a menu in the menu editor.
• Check "Show Touch Button".
• The touch button should show up on your screen.
  • You can long-press it in order to move it somewhere else.
  • You can click it to open the menu.
  • You can click and move over the button to open the menu and directly enter marking mode!

Does it work properly on a touch screen?

[Vinnyboiler]

Hi I managed to install the development branch and am currently testing out the new feature. So far this seems to be exactly what I suggested, I do have issues with it working using touch gestures though.

Holding down does not function properly on my touch screen and instead simply opens the menu. Also it seems to also lock up which can be fixed by enabling/disabling the button.

2021-11-15.00-38-44.mp4

I'll be more the happy to assist in debugging this issue though you may need to bear with me since I'm a novice to Linux and development in general.

EDIT: Additionally when in the "locked in" state Application and Workspaces are also stuck and Alt+Tab soft locks the system being unable to get off the menu.

2021-11-15.01-00-53.mp4

[Schneegans]

Then you'll have to learn some JavaScript 😛. Or, if it turns out to be more complicated, I'll have to find a touch screen somewhere...

All the important code is in src/extension/TouchButtons.js. You can edit this file in any text editor. After you changed some code, save it and call make install in a terminal in this directory. Then you'll have to restart GNOME Shell to reload the extension. You should use X11 (not Wayland) because then you can simply reload GNOME Shell with Alt+F2, then r and Enter. On Wayland you have to logout and login again.

For simple debugging you can insert a line like utils.debug("Hello Vinnyboiler"); anywhere in the file. The "Hello Vinnyboiler" will be written to a log which you can watch with this command:

journalctl -f -o cat | grep -E 'flypie|'

For example, you could add the above line 374 of the TouchButtons file, run make install, then run (for convenience in another terminal) the log-watch command, and finally restart GNOME Shell. From now on, "Hello Vinnyboiler" will be written to the log whenever you click on a touch button. This might be helpful if you want to print the value of some variable, for example. Most importantly, you could try to understand what the different behavior between touch and mouse input is.

I am not sure whether the long press action is not recognized or whether the dragging of the button is not working...
Anyway, there are several things to try. For example, you could replace line 334 with this code:

const action = new Clutter.ClickAction({
  long_press_duration: 2000, // Duration required for long press, in milliseconds.
  long_press_threshold: 200  // Maximum allowed movement during long press in pixels.
});

Then you could play with the values and see whether this changes the behavior....

Another thing is that I grab the complete pointer input during the movement of the button (line 359 and 360). This is done to prevent "loosing" the button during quick dragging movements. You could try to disable the two lines with comments (//) like this:

// Clutter.get_default_backend().get_default_seat().get_pointer().grab(actor);
// global.begin_modal(global.get_current_time(), 0);

And test the difference.

The more I think about it, the clearer I see a potential problem here: I grab the pointer device. However, quite likely, touch input is not considered to be pointer input. Therefore the UI may feel like being "locked"... Maybe it only reacts to mouse input in this state...? Maybe I'll have to find out which device triggered the long press action (touch or pointer) and grab this device... Anyway, you should try the things above and report any findings!

If anything of the above wall of text is unclear, feel free to ask for more details 😉 !

Happy debugging!

[Schneegans]

I just discovered that one can actually connect to touch events. Now I wonder whether the touch events are automatically translated to button-press and button-release events...? You could add

actor.connect('button-press-event', () => {
  utils.debug('button press!');
  return true;
});

actor.connect('touch-event', () => {
  utils.debug('touch!');
  return true;
});

to the other handlers (e.g. line 296) and test when they get called! On mouse clicks, only the top one should be fired. But on touch-clicks...?

Thanks for your time!

[Vinnyboiler]

Setting the long_press_duration to 100 seems to work, also opening the menu while my figure is sliding is what is causing it to lock up:

2021-11-16.04-58-19.mp4

[Schneegans]

Today, I managed to get hands on a touch screen and had some time to implement several fixes. There is some weird behavior in Clutter and I needed several workarounds to get touch and mouse input working at the same time on both X11 and Wayland. But I think it works now quite well. I'll also try stylus input on a Wacom tablet in the coming days. I would really appreciate it if you could test the changes and report any bugs you find!

Here is the pull request: #168. You can test it with the following commands in your cloned Fly-Pie repository:

git pull
git checkout feature/touch-input
make install

And then restart GNOME Shell.

[Vinnyboiler]

All my current issues seems to be solved 🙂 I will report any further issues I come across. Thank you so much for this feature, it has so far been as every bit as valuable as I thought having this as a feature would be.

[Schneegans]

Thank you very much for your donation! Do you want to be displayed in the list of sponsors? I could also add you to the respective lists of the Desktop-Cube and Burn-My-Windows if you happen to like these extensions as well :)

[Vinnyboiler]

I'll very much appreciate it. Thank you for a fantastic extension, it's the one of the major reasons I went from experimenting with Linux to sticking Windows on a external drive.