[terminal] Cmd + click doesn't work on iPad (iOS 12)

[jankeromnes]

Description

There is currently no way to click on Terminal links on iPad.

The Cmd + click to follow link tooltip does show up, but pressing Cmd on my external bluetooth keyboard and tapping the link doesn't work.

Reproduction Steps

1. Open Theia on an iPad (I tried iOS 12), in any browser (they're all Safari anyway)
2. Run echo "https://theia-ide.org" in a Terminal
3. Try to click the link. Also try pressing Cmd as indicated and click. It doesn't work.

OS and Theia version: master

Diagnostics:

The good news is that tapping on the link is detected, because this listener is being called:

theia/packages/terminal/src/browser/terminal-linkmatcher.ts

Lines 62 to 64 in 629240b

	protected isCommandPressed(event: MouseEvent): boolean {
	return isOSX ? event.metaKey : event.ctrlKey;
	}

The bad news is that isOSX ? event.metaKey : event.ctrlKey is always false on iPad.

In fact, I've tried to detect several different modifier keys, but pressing any of these on my iPad's external bluetooth keyboard is never detected by the event listener. I've tried:

• event.altKey
• event.metaKey
• event.ctrlKey
• event.shiftKey
• event.getModifierState('Alt')
• event.getModifierState('Control')
• event.getModifierState('Shift')
• event.getModifierState('CapsLock')
• event.getModifierState('Fn')
• event.getModifierState('Super')
• event.getModifierState('Accel')

They're all false, always. And even on https://keycode.info/ it's quite hard to get the Cmd / Control / Alt / Shift keys of my iPad keyboard detected.

Maybe we should make Terminal links directly clickable on iPad? I think that would be a huge improvement over the current situation.

[jankeromnes]

Part of #3557.

[jankeromnes]

@akosyakov @vince-fugnitto @marcdumais-work Do you have any thoughts on making all Terminal links immediately clickable on touch devices?

I personally believe that accidental link clicks are less likely with touch-screens than on desktop, and having some way to follow links there would be great.

[marcdumais-work]

Hi @jankeromnes ,

I personally believe that accidental link clicks are less likely with touch-screens than on desktop, and having some way to follow links there would be great.

Sounds reasonable.

Is an iPad detected as "OSX" in the code above? If so is there a way to distinguish from a Mac laptop? Would it be possible and worth the trouble to do this generally for other devices similar to an iPad, like Android tablets?

[vince-fugnitto]

@akosyakov @vince-fugnitto @marcdumais-work Do you have any thoughts on making all Terminal links immediately clickable on touch devices?

It might be worthwhile to attempt to debug and see what is happening, and also try out to see what happens when listening to touchevents explicitly (reference).

[jankeromnes]

Thanks a lot for your input, Marc and Vince!

Is an iPad detected as "OSX" in the code above? If so is there a way to distinguish from a Mac laptop? Would it be possible and worth the trouble to do this generally for other devices similar to an iPad, like Android tablets?

I'm not sure how to detect these different platforms yet, but I agree that the "always-clickable terminal link" special case should apply only to tablets like iPad and Android, and not to laptops (even if they have touchscreens).

It might be worthwhile to attempt to debug and see what is happening, and also try out to see what happens when listening to touchevents explicitly (reference).

I already did some debugging (see top comment), and what's happening is that the click is detected, but not the modifier keys, so we currently ignore the link "taps".

But you're right, maybe listening to touchevents can allow us to detect whether the link was "tapped" (with a finger on a touch screen) or actually "clicked" (with a mouse or touchpad). I think we'll want to enforce the modifier key only for mouse/pointer clicks, and make "taps" always open links regardless of keyboard state. I think this behavior even makes sense on laptops with touchscreens.

[marcdumais-work]

But you're right, maybe listening to touchevents can allow us to detect whether the link was "tapped" (with a finger on a touch screen) or actually "clicked" (with a mouse or touchpad). I think we'll want to enforce the modifier key only for mouse/pointer clicks, and make "taps" always open links regardless of keyboard state. I think this behavior even makes sense on laptops with touchscreens.

👍

[jankeromnes]

Update: I'm able to detect Terminal touchstart and touchend events by adding listeners next to the existing touchmove listener here:

theia/packages/core/src/browser/frontend-application.ts

Line 196 in 10d526f

document.addEventListener('touchmove', event => { event.preventDefault(); }, { passive: false });

Unfortunately, touchevents listeners only seem to work when attached to the document, not the Terminal element, and XtermJS can only tell us about the click events on its links, so some work needs to be done to correlate touch and click events.

Here is the handler that XtermJS calls when a Terminal link is clicked or tapped (even when the link won't open):

theia/packages/terminal/src/browser/terminal-linkmatcher.ts

Lines 38 to 45 in 10d526f

	const wrappedHandler = (event: MouseEvent, match: string) => {
	event.preventDefault();
	if (this.isCommandPressed(event)) {
	handler(event, match);
	} else {
	term.focus();
	}
	};

Maybe we can correlate event times / targets / coordinates in order to known when the click handler was triggered by a touch?

In my testing on iPad, here is the full sequence of events I see:

Time	Event	event.target	event.pageX x Y
2020-01-13T11:32:32.818Z	touchstart	canvas.xterm-cursor-layer	160 x 493
2020-01-13T11:32:32.912Z	touchend	canvas.xterm-cursor-layer	160 x 493
2020-01-13T11:32:32.962Z	willLinkActivate	canvas.xterm-cursor-layer	160 x 493
2020-01-13T11:32:32.964Z	click	canvas.xterm-cursor-layer	160 x 493
2020-01-13T11:32:33.431Z	tooltipCallback	canvas.xterm-cursor-layer	160 x 493

(In this run, all coordinates are exactly the same, but I've seen many cases where the touch and click coordinates are off by one, so some fuzzy-matching may be necessary.)

[jankeromnes]

Actually, with iOS 1.13, Cmd + click now works now on iPad if you have an external keyboard.