Better and more seamless mouse/pointer and keyboard control from computer to phone

[ShapeShifter499]

Describe your request

I'm not sure if I'm experiencing a bug in software. Right now in GSConnect I have to click to even show a pointer on the phone side, this makes it pretty useless to figure out where and when to click. And makes swipe gestures impossible. The issue is it doesn't work well and it doesn't always register as a swipe.

It would be nice if you could have it simulate a connected bluetooth or wired mouse on the phone. If I connect a mouse directly to my android phone it will show a pointer and I can use that to click and drag to simulate swipes or just click on things to simulate touches.

Proposed solution

Ideas I would love to see.

Have the mouse/touch feature simulate how connecting a mouse directly to the phone would work. The area that you would click in GSConnect would just instead detect the pointer and a pointer would appear on the phone, clicking and dragging inside the window then would end up working as if you had the mouse directly connected.

Allow one to "pin" gsconnect to a screen edge so that you could seamlessly have the pointer appear to go off screen from the computer to the phone. Basically a multipoint keyboard and mouse control feature with only one keyboard and mouse. Useful if you have your phone (or other mobile device) next to your main laptop or computer.

Alternatives

If there are alternatives that would work, I'd love to know them.

GSConnect version

56

Installed from

OS package manager

GNOME Shell version

45.4

Linux distribution/release

Arch Linux

Additional context

No response

[andyholmes]

This issue seems like it needs a little discussion to narrow down to scope, so moving to discussions.

[ShapeShifter499]

fair enough.

Maybe it's just me but the issue is that in it's current form it's so clunky to use the same keyboard and mouse between a computer and phone (or other android device). It's harder to use since you have to click the mouse or trackpad of your main computer just to get the pointer to show up on the connected device. If it detected your mouse inside of the active area of the window and sent that to the phone, like how the pointer would act if you direct connected a mouse to said device, would work better I think.

I guess kinda like a "headless" VNC connection, no screen streaming but just sending pointer or stylus inputs over.

The use case I have in mind is you could have both a regular computer or laptop and a android device to the side in a "multi monitor" type setup. Being able to use the same keyboard and mouse for both in a more seamless way.

[ferdnyc]

I guess kinda like a "headless" VNC connection, no screen streaming but just sending pointer or stylus inputs over.

The use case I have in mind is you could have both a regular computer or laptop and a android device to the side in a "multi monitor" type setup. Being able to use the same keyboard and mouse for both in a more seamless way.

What you're describing is basically Synergy, which is something I've raved about in the past as one of my favorite tools for multi-computer setups. There are some barriers to implementing a similar system via KDE Connect, though:

First and foremost, it doesn't work on Wayland. Even Synergy itself requires an X server to function.

With X11 it was possible to grab pointer movements desktop-wide and do basically anything you wanted to with them, because applications had free reign to abuse the display protocols however they wished. But Wayland's increased isolation and security means that you can't observe any interactions that happen outside the confines of your own application window(s), and within them you only get local positioning, no global coordinates.

Remote desktop protocols allow for better global control, since they're implemented by the window manager and have access to underlying APIs that applications don't. That's how GSConnect translates inputs in the other direction, when it's delivering KDE Connect mouse inputs to the local screen, for example. The pointer can be moved anywhere on the screen instead of being confined to a particular application, since it's the window manager receiving the motion events. But that setup still relies on relative mouse movements rather than absolute positioning, effectively using the device touchscreen as a trackpad.

If it detected your mouse inside of the active area of the window and sent that to the phone, like how the pointer would act if you direct connected a mouse to said device, would work better I think.

To make an interface like that work comfortably, what you really want is absolute pointer positioning, rather than relative positioning. That's how, for example, most virtual machine managers implement mouse input, so that they can provide seamless pointer capture and release when crossing the boundaries of the VM's display window.

Even though a mouse (or trackpad) is a relative-motion device, the VM's guest OS is configured with a virtual tablet device that has the same dimensions as the VM screen. When you move the pointer around inside the window, your relative mouse movements are translated into local window coordinates by the host window manager, and the VM manager takes those and translates them into absolute coordinates on the guest's "tablet" device. That way, whenever the mouse is over the window, whatever local coordinates it's at, those become the absolute pointer position inside the VM. And whenever you move the guest pointer to the edge of the screen, the VM manager can know to release the pointer back to the host window manager. (Added bonus, the pointer ends up in a screen location adjacent to its last position inside the VM.)

But that only works because the window you're interacting with is the virtual display you're controlling, meaning that the mapping from local window coordinates to absolute guest display positioning is basically intrinsic. Heck, there's no actual "mapping" being done, really — the local window coordinates just become the VM pointer position, directly. When that's not the case, things get trickier:

1. GSConnect and KDE Connect would have to negotiate their scaling and mapping when setting up for remote input: KDE Connect would have to send the dimensions of the device screen, and GSConnect's window would have to have at least the same aspect ratio. That's the only way local window coordinates could be translated into absolute pointer positions on the device.

2. And even then, because the window would typically be smaller than the device screen (most laptop & desktop displays couldn't even fit an Android device's full screen dimensions), the coordinates would have to be scaled up by some factor in order to map correctly. That'd create a somewhat disconcerting acceleration effect, whenever your mouse entered the GSConnect input window. (The smaller the window, the more jarring the acceleration.)

3. (This is the big one:) I have absolutely no idea if Android's pointer control allows for absolute positioning of the pointer. If it only supports relative motion, then there's no way to provide an absolute-coordinate-mapped input surface.

4. Even if this is possible, it would have to be implemented in KDE Connect first.

[SMANahian]

I also felt the need for this.
The current feature seems quite unintuitive to use.