Integration Tests for Plugin Developers: UI Testing

In our previous blog post, we created a basic integration test that: Now, let’s explore how to interact with UI elements in our tests. UI hierarchy IntelliJ-based IDEs primarily use Swing and AWT for their user interface, while JCEF is used in specific cases like Markdown rendering. This UI framework organizes elements in a parent-child […]

Feb 28, 2025 - 19:53

Integration Tests for Plugin Developers: UI Testing

In our previous blog post, we created a basic integration test that:

Installed a plugin in the IDE.
Opened a project.
Waited for background processes to complete.
Performed an exit.

Now, let’s explore how to interact with UI elements in our tests.

UI hierarchy

IntelliJ-based IDEs primarily use Swing and AWT for their user interface, while JCEF is used in specific cases like Markdown rendering. This UI framework organizes elements in a parent-child hierarchy, similar to HTML’s DOM structure:

Top-level containers (IDE frame and dialogs).
Nested containers.
Individual components (buttons, text fields, and lists).

Every UI element (except top-level containers) must have a parent container, creating a clear hierarchical structure.

The Driver framework provides a Kotlin DSL that mirrors this hierarchy. Here’s an example:

ideFrame {
     invokeAction("SearchEverywhere")
     searchEverywherePopup {
          actionButtonByXpath(xQuery { byAccessibleName("Preview")}).click()
     }
}

This code demonstrates hierarchical navigation:

Find the main IDE window (ideFrame).
Trigger the Search Everywhere action (invokeAction("SearchEverywhere")).
Locate the Search Everywhere popup (searchEverywherePopup).
Find and click the Preview button within the popup (actionButtonByXpath(xQuery { byAccessibleName("Preview")}).click()).

You could write more concise code:

ideFrame { 
     actionButtonByXpath(xQuery { byAccessibleName("Preview")}).click() 
}

But the shorter code has two significant drawbacks:

Reduced precision: The code searches for the Preview button throughout the entire IDE frame. It might find unintended matches in the project explorer, tool windows, or other UI elements. This can make your tests unreliable and prone to breaking when the UI content changes.
Decreased readability: While the code is more concise, it doesn’t communicate the intended navigation path. The longer version makes it clear exactly where we expect to find the Preview button, making the code more maintainable and easier to debug.

So, being explicit about the component hierarchy helps create more robust and maintainable UI automation code, even though it requires writing more code.

Searching components

While the Driver framework provides many pre-built components (like ideFrame, codeEditor, button, tree, etc.), you’ll sometimes need to locate custom elements. Let’s explore how to find any UI component in your tests.

First, let’s modify our test to pause the IDE so we can examine its UI structure:

@Test
fun simpleTest() {
   Starter.newContext(
       "testExample",
       TestCase(
           IdeProductProvider.IC,
           GitHubProject.fromGithub(branchName = "master",
               repoRelativeUrl = "JetBrains/ij-perf-report-aggregator"))
           .withVersion("2024.3")
   ).apply {
       val pathToPlugin = System.getProperty("path.to.build.plugin")
       PluginConfigurator(this).installPluginFromPath(Path(pathToPlugin))
   }.runIdeWithDriver().useDriverAndCloseIde {
       Thread.sleep(30.minutes.inWholeMilliseconds)
   }
}

When you run this test, look for this line in the logs: http://localhost:63343/api/remote-driver/.

Opening this URL reveals an HTML representation of the IDE’s Swing component tree:

Using Developer Tools in the browser, you can inspect detailed component attributes. Here’s an example component:

The element corresponds to the following button:

Similar to web testing frameworks like Selenium, we use XPath to locate components. The Driver framework provides a simple XPath builder. Here are several ways to find the same component:

xQuery { byVisibleText("Current File") }
xQuery { byAccessibleName("Current File") }
xQuery { byType("com.intellij.execution.ui.RedesignedRunConfigurationSelector\$createCustomComponent$1") }

For reliable component identification, prioritize these attributes: accessiblename, visible_text, icon, javaclass. You can combine multiple attributes for a more precise selection.

Interaction with components

Once you’ve located a component, you’ll want to interact with it or verify its properties. So, let’s now explore how to perform various UI interactions.
To click the Current File button we found earlier, we need to enter:

x(xQuery { byVisibleText("Current File") }).click()

The x() call creates a lazy reference to the component. It means that the XPath query isn’t executed immediately and component lookup happens only when an action (like click()) is invoked.
Here’s our test that incorporates UI interaction:

fun simpleTestForCustomUIElement() {
   Starter.newContext(
       "testExample",
       TestCase(
           IdeProductProvider.IC,
           GitHubProject.fromGithub(branchName = "master",
               repoRelativeUrl = "JetBrains/ij-perf-report-aggregator"))
           .withVersion("2024.3")
   ).apply {
       val pathToPlugin = System.getProperty("path.to.build.plugin")
       PluginConfigurator(this).installPluginFromPath(Path(pathToPlugin))
   }.runIdeWithDriver().useDriverAndCloseIde {
       waitForIndicators(1.minutes)
       ideFrame {
           x(xQuery { byVisibleText("Current File") }).click()
       }
   }
}

Beyond mouse clicks, you can simulate keyboard input and shortcuts:

keyboard {
   enterText("Sample text")
   enter()
   hotKey(if (SystemInfo.isMac) KeyEvent.VK_META else KeyEvent.VK_CONTROL, KeyEvent.VK_A)
   backspace()
}

Note: On macOS, the interaction via java.awt.Robot requires special permissions. IntelliJ IDEA should be granted the necessary permissions via the Accessibility page, which can be found under System Settings | Privacy & Security.

Asserting properties and putting it all together

Let’s combine everything we’ve learned and add property assertions to create a complete UI test:

@Test
fun simpleTestForCustomUIElement() {
   Starter.newContext(
       "testExample",
       TestCase(
           IdeProductProvider.IC,
           GitHubProject.fromGithub(branchName = "master",
               repoRelativeUrl = "JetBrains/ij-perf-report-aggregator"))
           .withVersion("2024.3")
   ).apply {
       val pathToPlugin = System.getProperty("path.to.build.plugin")
       PluginConfigurator(this).installPluginFromPath(Path(pathToPlugin))
   }.runIdeWithDriver().useDriverAndCloseIde {
       waitForIndicators(1.minutes)
       ideFrame {
           x(xQuery { byVisibleText("Current File") }).click()
           val configurations = popup().jBlist(xQuery { contains(byVisibleText("Edit Configurations")) })
           configurations.shouldBe("Configuration list is not present", present)
           Assertions.assertTrue(configurations.rawItems.contains("backup-data"),
               "Configurations list doesn't contain 'backup-data' item: ${configurations.rawItems}")
       }
   }
}

Let’s break down each step:

Opening the popup
- Click the Current File button.
- Popup menu appears.
Finding the list
- Use popup() to locate the popup with a configuration list. Note: This works without any XPath because at the moment of the call, there are no other popups shown on the UI.
- Find the list containing the text Edit Configurations by using the following query:

jBlist(xQuery { contains(byVisibleText("Edit Configurations")) }).

XQuery searches for the list component that contains the visible text "Edit Configurations".

Verifying list presence
- Use shouldBe(, present) to ensure the list exists.
  - This is important because popup().jBlist creates a lazy reference without actually checking the results.
- The actual check happens when shouldBe calls the present method.
- The shouldBe method waits 15 seconds until the condition is met and can be used to assert various properties.
Checking list contents
- Access the rawItems property to get all list items.
- Verify the 'backup-data' exists in the list.
- Include full list content in the error message for debugging.

You can find the full source code here.

What’s next?

You now have the foundation to create UI tests that can interact with IDE components, verify user scenarios, and catch UI-related regressions. But this is just the beginning of your testing journey!

Stay tuned for upcoming blog posts in this series, where we’ll cover:

API Testing: working with plugin APIs effectively via JMX calls.
GitHub Actions: setting up continuous integration.
Common Pitfalls: tips and tricks for stable UI tests.