Sikuli Tutorial for Selenium Automation

What is Sikuli in Selenium?

Sikuli is an open-source GUI based test automation tool. It is mainly used for interacting with elements of web pages and handling windows based popups. Sikuli uses the technique of “Image Recognition” and “Control GUI” to interact with elements of web pages and windows popups. In Sikuli, all the web elements are taken as images and stored inside the project.

⚠️ Naming note: the project is maintained today as SikuliX, published on Maven Central as com.sikulix:sikulixapi. The class names below are unchanged.

How to use Sikuli with Selenium Webdriver

Sikuli can be integrated with Selenium WebDriver using the Sikuli JAR file, in the two steps below.

Step 1) Download the Sikuli JAR file from the below URL, and extract the contents of the ZIP file to a folder.

https://mvnrepository.com/artifact/com.sikulix/sikulixapi/2.0.5

Step 2) Create a new JAVA project in Eclipse and add the JAR file to build path, along with selenium jar files using Right Click on the project -> Build Path -> Configure Build Path, as shown below.

Once you have added the JAR file to project build path, classes provided by Sikuli can be used. A Maven project can declare the same artifact as a dependency instead.

Screen class in Sikuli

Screen class is the base class for all the methods provided by Sikuli. Screen class contains predefined methods for all the commonly performed operations on screen elements such as click, double-click, providing input to a text box, hover, etc. The below is the list of commonly used methods provided by Screen class.

Pattern class in Sikuli

Where Screen acts on a plain image name, Pattern class is used to associate the image file with additional attributes to uniquely identify the element. It takes the path of the image as a parameter.

Pattern p = new Pattern(“Path of image”);

The following are the most commonly used methods of Pattern class.

Code Example for File Upload using Sikuli

Below code explains the use of Sikuli for file upload in Chrome.

package com.sikuli.demo;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.sikuli.script.FindFailed;
import org.sikuli.script.Pattern;
import org.sikuli.script.Screen;
import org.openqa.selenium.chrome.ChromeDriver;

public class SikuliDemo {

    public static void main(String[] args) throws FindFailed {

        System.setProperty("webdriver.chrome.driver", "D:\\chromedriver.exe");
        String filepath = "D:\\Guru99Demo\\Files\\";
        String inputFilePath = "D:\\Guru99Demo\\Files\\";
        Screen s = new Screen();
        Pattern fileInputTextBox = new Pattern(filepath + "FileTextBox.PNG");
        Pattern openButton = new Pattern(filepath + "OpenButton.PNG");
        WebDriver driver;

        // Open Chrome browser    
        driver = new ChromeDriver();
        driver.get("https://demo.guru99.com/test/image_upload/index.php");

        // Click on Browse button and handle windows pop up using Sikuli
        driver.findElement(By.xpath(".//*[@id='photoimg']")).click();
        s.wait(fileInputTextBox, 20);
        s.type(fileInputTextBox, inputFilePath + "Test.docx");
        s.click(openButton);

        // Close the browser
        driver.close();
    }
}

Code Explanation

Step 1) The first statement involves setting the driver executable path for chrome.

System.setProperty("webdriver.chrome.driver", "D:\\ chromedriver.exe");

⚠️ Version note: Selenium 4 resolves the driver binary through Selenium Manager, so this line is now optional. It is kept as the source wrote it.

Step 2) Use a screengrab tool such as Snipping Tool to take screenshots of windows popup ‘FileTextBox’ and ‘Open’ button, as below.

This is how your screenshot should look:

Images for windows file input text box and open button are stored onto ‘FileTextBox.PNG’ and ‘OpenButton.PNG’.

Example: If you want to automate the operation of opening notepad, then you need to store the image of a desktop icon for notepad onto a PNG file and perform click operation on it.

In our case, it recognizes the file input text box and Open button on Windows popup using the images stored. If the screen resolution changes from image capture to test script execution, the behavior of Sikuli would be inconsistent. Hence it is always advisable to run the test script on the same resolution at which images are captured. Change in pixel size of images will result in Sikuli throwing a FindFailed exception.

Step 3) The next statements include the creation of objects for Screen and Pattern classes. Create a new screen object. Set the path of the file you want to upload as a parameter to the Pattern object.

Screen s = new Screen();
Pattern fileInputTextBox = new Pattern(filepath + "FileTextBox.PNG");
Pattern openButton = new Pattern(filepath + "OpenButton.PNG");

Step 4) The below statements involve opening chrome browser with the URL of a demo file upload application: https://demo.guru99.com/test/image_upload/index.php

driver = new ChromeDriver();
driver.get("https://demo.guru99.com/test/image_upload/index.php");

Step 5) Click the choose file button using below statement

driver.findElement(By.xpath(".//*[@id='photoimg']")).click();

Step 6) Wait for the windows popup to appear. Wait method is used to handle the delay associated with opening windows pop up after clicking on the browse button.

s.wait(fileInputTextBox, 20);

Step 7) Type the file path onto input file text box and click on Open button

s.type(fileInputTextBox, inputFilePath + "Test.docx");
s.click(openButton);

Step 8) Close the browser

driver.close();

Output

Initially, script opens chrome browser

Clicks on the ‘Choose File’ button, windows file popup screen will appear. Enters data into File Input textbox and clicks on ‘Open’ button

Below screen is displayed once the file upload is complete and closes the browser

Because matching depends on pixels, recapture every reference image whenever the interface is restyled. That fragility is why teams reserve Sikuli for popups plain WebDriver file handling cannot reach.

FAQs