Skip to content

Commit

Permalink
Accessibility: Set-up UIA Tree (#1691)
Browse files Browse the repository at this point in the history
**The Basics of Accessibility**
- [What is a User Interaction Automation (UIA) Tree?](https://docs.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-tree-overview)
- Other projects (i.e.: Narrator) can take advantage of this UIA tree and are used to present information within it.
- Some things like XAML already have a UIA Tree. So some UIA tree navigation and features are already there. It's just a matter of getting them hooked up and looking right.

**Accessibility in our Project**
There's a few important classes...
regarding Accessibility...
- **WindowUiaProvider**: This sets up the UIA tree for a window. So this is the top-level for the UIA tree.
- **ScreenInfoUiaProvider**: This sets up the UIA tree for a terminal buffer.
- **UiaTextRange**: This is essential to interacting with the UIA tree for the terminal buffer. Actually gets portions of the buffer and presents them.

regarding the Windows Terminal window...
- **BaseWindow**: The foundation to a window. Deals with HWNDs and that kind of stuff.
- **IslandWindow**: This extends `BaseWindow` and is actually what holds our Windows Terminal
- **NonClientIslandWindow**: An extension of the `IslandWindow`

regarding ConHost...
- **IConsoleWindow**: This is an interface for the console window.
- **Window**: This is the actual window for ConHost. Extends `IConsoleWindow`

- `IConsoleWindow` changes:
  - move into `Microsoft::Console::Types` (a shared space)
  - Have `IslandWindow` extend it
- `WindowUiaProvider` changes:
  - move into `Microsoft::Console::Types` (a shared space)
- Hook up `WindowUiaProvider` to IslandWindow (yay! we now have a tree)

### Changes to the WindowUiaProvider
As mentioned earlier, the WindowUiaProvider is the top-level UIA provider for our projects. To reuse as much code as possible, I created `Microsoft::Console::Types::WindowUiaProviderBase`. Any existing functions that reference a `ScreenInfoUiaProvider` were virtual-ized.

In each project, a `WindowUiaProvider : WindowUiaProviderBase` was created to define those virtual functions. Note that that will be the main difference between ConHost and Windows Terminal moving forward: how many TextBuffers are on the screen.

So, ConHost should be the same as before, with only one `ScreenInfoUiaProvider`, whereas Windows Terminal needs to (1) update which one is on the screen and (2) may have multiple on the screen.

🚨 Windows Terminal doesn't have the `ScreenInfoUiaProvider` hooked up yet. We'll have all the XAML elements in the UIA tree. But, since `TermControl` is a custom XAML Control, I need to hook up the `ScreenInfoUiaProvider` to it. This work will be done in a new PR and resolve GitHub Issue #1352.


### Moved to `Microsoft::Console::Types`
These files got moved to a shared area so that they can be used by both ConHost and Windows Terminal.
This means that any references to the `ServiceLocator` had to be removed.

- `IConsoleWindow`
  - Windows Terminal: `IslandWindow : IConsoleWindow`
- `ScreenInfoUiaProvider`
  - all references to `ServiceLocator` and `SCREEN_INFORMATION` were removed. `IRenderData` was used to accomplish this. Refer to next section for more details.
- `UiaTextRange`
  - all references to `ServiceLocator` and `SCREEN_INFORMATION` were removed. `IRenderData` was used to accomplish this. Refer to next section for more details.
  - since most of the functions were `static`, that means that an `IRenderData` had to be added into most of them.


### Changes to IRenderData
Since `IRenderData` is now being used to abstract out `ServiceLocator` and `SCREEN_INFORMATION`, I had to add a few functions here:
- `bool IsAreaSelected()`
- `void ClearSelection()`
- `void SelectNewRegion(...)`
- `HRESULT SearchForText(...)`

`SearchForText()` is a problem here. The overall new design is great! But Windows Terminal doesn't have a way to search for text in the buffer yet, whereas ConHost does. So I'm punting on this issue for now. It looks nasty, but just look at all the other pretty things here. :)
  • Loading branch information
carlos-zamora authored Jul 29, 2019
1 parent ed18c1e commit 96496d8
Show file tree
Hide file tree
Showing 46 changed files with 1,732 additions and 1,149 deletions.
18 changes: 17 additions & 1 deletion src/cascadia/TerminalCore/Terminal.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,23 @@ class Microsoft::Terminal::Core::Terminal final :
const std::vector<Microsoft::Console::Render::RenderOverlay> GetOverlays() const noexcept override;
const bool IsGridLineDrawingAllowed() noexcept override;
std::vector<Microsoft::Console::Types::Viewport> GetSelectionRects() noexcept override;
bool IsAreaSelected() const override;
void ClearSelection() override;
void SelectNewRegion(const COORD coordStart, const COORD coordEnd) override;

// TODO GitHub #605: Search functionality
// For now, just adding it here to make UiaTextRange easier to create (Accessibility)
// We should actually abstract this out better once Windows Terminal has Search
HRESULT SearchForText(_In_ BSTR text,
_In_ BOOL searchBackward,
_In_ BOOL ignoreCase,
_Outptr_result_maybenull_ ITextRangeProvider** ppRetVal,
unsigned int _start,
unsigned int _end,
std::function<unsigned int(IRenderData*, const COORD)> _coordToEndpoint,
std::function<COORD(IRenderData*, const unsigned int)> _endpointToCoord,
std::function<IFACEMETHODIMP(ITextRangeProvider**)> Clone) override;

const std::wstring GetConsoleTitle() const noexcept override;
void LockConsole() noexcept override;
void UnlockConsole() noexcept override;
Expand All @@ -122,7 +139,6 @@ class Microsoft::Terminal::Core::Terminal final :
void SetSelectionAnchor(const COORD position);
void SetEndSelectionPosition(const COORD position);
void SetBoxSelection(const bool isEnabled) noexcept;
void ClearSelection() noexcept;

const std::wstring RetrieveSelectedTextFromBuffer(bool trimTrailingWhitespace) const;
#pragma endregion
Expand Down
2 changes: 1 addition & 1 deletion src/cascadia/TerminalCore/TerminalSelection.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -210,7 +210,7 @@ void Terminal::SetBoxSelection(const bool isEnabled) noexcept

// Method Description:
// - clear selection data and disable rendering it
void Terminal::ClearSelection() noexcept
void Terminal::ClearSelection()
{
_selectionActive = false;
_selectionAnchor = { 0, 0 };
Expand Down
27 changes: 27 additions & 0 deletions src/cascadia/TerminalCore/terminalrenderdata.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,33 @@ std::vector<Microsoft::Console::Types::Viewport> Terminal::GetSelectionRects() n
return result;
}

bool Terminal::IsAreaSelected() const
{
return _selectionActive;
}

void Terminal::SelectNewRegion(const COORD coordStart, const COORD coordEnd)
{
SetSelectionAnchor(coordStart);
SetEndSelectionPosition(coordEnd);
}

// TODO GitHub #605: Search functionality
// For now, just adding it here to make UiaTextRange easier to create (Accessibility)
// We should actually abstract this out better once Windows Terminal has Search
HRESULT Terminal::SearchForText(_In_ BSTR /*text*/,
_In_ BOOL /*searchBackward*/,
_In_ BOOL /*ignoreCase*/,
_Outptr_result_maybenull_ ITextRangeProvider** /*ppRetVal*/,
unsigned int /*_start*/,
unsigned int /*_end*/,
std::function<unsigned int(IRenderData*, const COORD)> /*_coordToEndpoint*/,
std::function<COORD(IRenderData*, const unsigned int)> /*_endpointToCoord*/,
std::function<IFACEMETHODIMP(ITextRangeProvider**)> /*Clone*/)
{
return E_NOTIMPL;
}

const std::wstring Terminal::GetConsoleTitle() const noexcept
{
return _title;
Expand Down
30 changes: 28 additions & 2 deletions src/cascadia/WindowsTerminal/BaseWindow.h
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,16 @@

#pragma once

#include "..\types\IConsoleWindow.hpp"
#include "..\types\WindowUiaProviderBase.hpp"

// Custom window messages
#define CM_UPDATE_TITLE (WM_USER)

#include <wil/resource.h>

using namespace Microsoft::Console::Types;

template<typename T>
class BaseWindow
{
Expand Down Expand Up @@ -51,6 +56,11 @@ class BaseWindow
return HandleDpiChange(_window.get(), wparam, lparam);
}

case WM_GETOBJECT:
{
return HandleGetObject(_window.get(), wparam, lparam);
}

case WM_DESTROY:
{
PostQuitMessage(0);
Expand Down Expand Up @@ -121,6 +131,22 @@ class BaseWindow
return 0;
}

[[nodiscard]] LRESULT HandleGetObject(const HWND hWnd, const WPARAM wParam, const LPARAM lParam)
{
LRESULT retVal = 0;

// If we are receiving a request from Microsoft UI Automation framework, then return the basic UIA COM interface.
if (static_cast<long>(lParam) == static_cast<long>(UiaRootObjectId))
{
retVal = UiaReturnRawElementProvider(hWnd, wParam, lParam, _GetUiaProvider());
}
// Otherwise, return 0. We don't implement MS Active Accessibility (the other framework that calls WM_GETOBJECT).

return retVal;
}

virtual IRawElementProviderSimple* _GetUiaProvider() = 0;

virtual void OnResize(const UINT width, const UINT height) = 0;
virtual void OnMinimize() = 0;
virtual void OnRestore() = 0;
Expand All @@ -135,7 +161,7 @@ class BaseWindow
HWND GetHandle() const noexcept
{
return _window.get();
};
}

float GetCurrentDpiScale() const noexcept
{
Expand Down Expand Up @@ -187,7 +213,7 @@ class BaseWindow
{
_title = newTitle;
PostMessageW(_window.get(), CM_UPDATE_TITLE, 0, reinterpret_cast<LPARAM>(nullptr));
};
}

protected:
using base_type = BaseWindow<T>;
Expand Down
24 changes: 24 additions & 0 deletions src/cascadia/WindowsTerminal/IslandWindow.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,30 @@ void IslandWindow::OnSize(const UINT width, const UINT height)
return base_type::MessageHandler(message, wparam, lparam);
}

// Routine Description:
// - Creates/retrieves a handle to the UI Automation provider COM interfaces
// Arguments:
// - <none>
// Return Value:
// - Pointer to UI Automation provider class/interfaces.
IRawElementProviderSimple* IslandWindow::_GetUiaProvider()
{
if (nullptr == _pUiaProvider)
{
try
{
_pUiaProvider = WindowUiaProvider::Create(this);
}
catch (...)
{
LOG_HR(wil::ResultFromCaughtException());
_pUiaProvider = nullptr;
}
}

return _pUiaProvider;
}

// Method Description:
// - Called when the window has been resized (or maximized)
// Arguments:
Expand Down
40 changes: 39 additions & 1 deletion src/cascadia/WindowsTerminal/IslandWindow.h
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,14 @@

#include "pch.h"
#include "BaseWindow.h"
#include "../types/IUiaWindow.h"
#include "WindowUiaProvider.hpp"
#include <winrt/Microsoft.Terminal.TerminalControl.h>
#include <winrt/TerminalApp.h>

class IslandWindow : public BaseWindow<IslandWindow>
class IslandWindow :
public BaseWindow<IslandWindow>,
public IUiaWindow
{
public:
IslandWindow() noexcept;
Expand All @@ -17,6 +21,7 @@ class IslandWindow : public BaseWindow<IslandWindow>
virtual void OnSize(const UINT width, const UINT height);

[[nodiscard]] virtual LRESULT MessageHandler(UINT const message, WPARAM const wparam, LPARAM const lparam) noexcept override;
IRawElementProviderSimple* _GetUiaProvider();
void OnResize(const UINT width, const UINT height) override;
void OnMinimize() override;
void OnRestore() override;
Expand All @@ -29,6 +34,38 @@ class IslandWindow : public BaseWindow<IslandWindow>

void UpdateTheme(const winrt::Windows::UI::Xaml::ElementTheme& requestedTheme);

#pragma region IUiaWindow
void ChangeViewport(const SMALL_RECT NewWindow)
{
// TODO GitHub #1352: Hook up ScreenInfoUiaProvider to WindowUiaProvider
// Relevant comment from zadjii-msft:
/*
In my head for designing this, I'd then have IslandWindow::ChangeViewport
call a callback that AppHost sets, where AppHost will then call into the
TerminalApp to have TerminalApp handle the ChangeViewport call.
(See IslandWindow::SetCreateCallback as an example of a similar
pattern we're using today.) That way, if someone else were trying
to resuse this, they could have their own AppHost (or TerminalApp
equivalent) handle the ChangeViewport call their own way.
*/
return;
};

HWND GetWindowHandle() const noexcept override
{
return BaseWindow::GetHandle();
};

[[nodiscard]] HRESULT SignalUia(_In_ EVENTID id) override { return E_NOTIMPL; };
[[nodiscard]] HRESULT UiaSetTextAreaFocus() override { return E_NOTIMPL; };

RECT GetWindowRect() const noexcept override
{
return BaseWindow::GetWindowRect();
};

#pragma endregion

protected:
void ForceResize()
{
Expand All @@ -38,6 +75,7 @@ class IslandWindow : public BaseWindow<IslandWindow>
}

HWND _interopWindowHandle;
WindowUiaProvider* _pUiaProvider;

winrt::Windows::UI::Xaml::Hosting::DesktopWindowXamlSource _source;

Expand Down
Loading

0 comments on commit 96496d8

Please sign in to comment.