Microsoft UI Automation/Get The Item That The User Clicks

Question

I am trying to figure out what item (for example document, web page tab, window, picture, folder) the user has clicked on. I started by using the following code when I detect a global left mouse click:

System.Drawing.Point MousePoint = System.Windows.Forms.Cursor.Position;
AutomationElement AutomationElement = AutomationElement.FromPoint(new System.Windows.Point(MousePoint.X, MousePoint.Y));
Console.WriteLine(AutomationElement.Current.Name);

This seems to work well in most conditions. However, I need to (if possible) get names of documents/images/folders inside Windows Explorer for example. The value returned when I click a document in the right hand pane of Windows Explorer (not the tree view) is "Name". Is there anyway to get the actual document name? For some reason, clicking sub-folders in the tree view returns the name of the folder, which is what I want.

I also notice that the code seems to display the document/image/folder name when clicked if the Windows Explorer view is set to icons (medium, large or extra large). Is there any reason why other views return "Name" or empty string while medium, large and extra large icons return the actual document/image/folder name? Is it to do with the size of the object clicked? I could really do with a way round this if possible?

I apologise, I am new to UI Automation and just really want a way to find the name of the object (file, folder, document, picture, web page tab etc.) that the user has clicked on. Any help anyone could give would be great.

Answer 1

You need to listen to InvokePattern.InvokedEvent UI automation event.

Example

The following example assumes you have an open instance of windows "Calculator" app. Then if you run the application, when you click on any button in Calculator, we handle the click event and show what button has clicked.

[DllImport("user32.dll")]
private extern static IntPtr FindWindow(string lpClassName, string lpWindowName);
protected override void OnLoad(EventArgs e)
{
    base.OnLoad(e);
    var proc = System.Diagnostics.Process.GetProcessesByName("Calculator")
                        .FirstOrDefault();
    if (proc != null)
    {
        var mainHwnd = FindWindow(null, "Calculator");
        var mainWndElement = AutomationElement.FromHandle(mainHwnd);
        Automation.AddAutomationEventHandler(
            InvokePattern.InvokedEvent, mainWndElement,
            TreeScope.Subtree, (s1, e1) =>
            {
                var element = s1 as AutomationElement;
                MessageBox.Show($"'{element.Current.Name}' clicked by user!");
            });
    }
}
protected override void OnFormClosing(FormClosingEventArgs e)
{
    Automation.RemoveAllEventHandlers();
    base.OnFormClosing(e);
}

You need to add reference to UIAutomationClient and UIAutomationTypes assemblies.

Note

• You can easily extend the example and listen to Win32_ProcessStartTrace to detect when Calculator opens. You can see an example to detect when an external process window closes which is using Win32_ProcessStopTrace.

• Another option is using SetWinEventHook you can listen to some events from other processes and register a WinEventProc callback method to receive the event when the event raised. Here, you are interested in EVENT_OBJECT_INVOKED event.