HTMLUnitDriver in Selenium: Headless Browser Tutorial

A headless browser is a web browser without a graphical user interface. It can run in the background without visual distractions. It offers an efficient and effective way to test web applications while saving time and resources. In this article, we explore what a headless browser is, when to use headless browser testing, and how to perform it using Selenium HtmlUnitDriver and PhantomJS.

What is a Headless Browser?

A headless browser is a web browser without a graphical user interface. This program behaves just like a regular browser, but it does not display any GUI window during execution.

Some examples of headless browser drivers include:

• HtmlUnit
• Ghost
• PhantomJS (deprecated)
• ZombieJS
• Watir-webdriver
• Headless Chrome
• Headless Firefox

When to Use Headless Browser Testing?

In today’s digital age, web applications are built to be compatible with a wide range of devices and platforms. This creates a challenge for developers who must ensure their applications work seamlessly across different environments. Headless browser testing is an ideal solution because it lets developers validate their web applications without a graphic user interface. By using headless browser testing, developers can quickly cover complex web applications with many components and dependencies, paving the way for faster development, fewer bugs, and happier users.

Headless Browser Testing with Selenium

Selenium is a powerful tool for headless browser testing, allowing developers to run automated tests without a visible user interface. By running tests in the background, Selenium saves time and resources while also helping identify issues that may not be apparent in a traditional UI-based testing environment. This includes performance and layout problems that only become evident in a headless setting. However, it is important to balance headless testing with traditional UI-based tests to ensure full coverage.

Popular Examples of Headless Browsers

Many headless browsers are available, each with its own features and benefits, making them suitable for different use cases. We discuss them below.

PhantomJS

Deprecation notice: Active development of PhantomJS stopped in 2018, and the project is no longer maintained. New projects should prefer headless Chrome, headless Firefox, or Playwright for modern headless automation. The PhantomJS example below is included for reference only.

PhantomJS is a headless browser that uses WebKit as its rendering engine and supports web standards such as HTML5, CSS3, and JavaScript. It can be used for screen capture and page automation tasks. It is open-source and was compatible with multiple operating systems.

Example of Selenium with Headless PhantomJS in Python:

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
# Set up PhantomJS options
phantomjs_options = webdriver.DesiredCapabilities.PHANTOMJS.copy()
phantomjs_options['phantomjs.page.settings.userAgent'] = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'
# Set up the PhantomJS driver
driver = webdriver.PhantomJS('/path/to/phantomjs', desired_capabilities=phantomjs_options)
# Perform actions using the driver
driver.get('https://www.example.com')
print(driver.title)
# Close the driver
driver.quit()

Chrome

Chrome is the most popular browser on the planet and offers a headless mode as well. It can be used across many platforms and supports popular programming languages and frameworks. Its built-in debugging tools and extensive documentation make it easy to use and troubleshoot any issues that may arise during testing.

Example of Headless Chrome with Selenium in Python:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
# Set up Chrome options
chrome_options = Options()
chrome_options.add_argument('--headless')  # Run Chrome in headless mode
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
# Set up the Chrome driver
driver = webdriver.Chrome('/path/to/chromedriver', options=chrome_options)
# Perform actions using the driver
driver.get('https://www.example.com')
print(driver.title)
# Close the driver
driver.quit()

Firefox

Firefox is a popular web browser that can also be used as a headless browser for testing. One benefit of using Firefox as a headless browser is its lightweight footprint, which makes it a versatile option for testing on different operating systems. Combined with its extensive documentation and community support, Firefox is an excellent choice for teams experimenting with headless browser automation.

Example of Headless Firefox with Selenium in Python:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
# Set up Firefox options
firefox_options = Options()
firefox_options.add_argument('--headless')  # Run Firefox in headless mode
# Set up the Firefox driver
driver = webdriver.Firefox(options=firefox_options)
# Perform actions using the driver
driver.get('https://www.example.com')
print(driver.title)
# Close the driver
driver.quit()

Advantages of Headless Browser Testing

• Faster test execution
• Cost-effective testing
• Better test coverage
• Flexibility in running tests
• Easier integration with CI/CD pipelines

Disadvantages of Headless Browser Testing

• Lack of visible GUI for manual inspection
• Debugging can be harder without a screen
• Limited browser-specific behavior coverage

HtmlUnitDriver

HtmlUnitDriver is the most lightweight and fastest implementation of a headless browser for Selenium WebDriver. It is based on HtmlUnit and is also known as the Headless Browser Driver. It behaves like the Chrome, IE, or Firefox driver, but it does not have a GUI, so the test execution is not visible on screen. HtmlUnitDriver is still maintained and is distributed through the org.seleniumhq.selenium:htmlunit3-driver Maven artifact.

Features of HtmlUnitDriver:

• Support for the HTTPS and HTTP protocols
• Support for HTML responses (clicking links, submitting forms, walking the DOM model of the HTML document, and more)
• Support for cookies
• Proxy server support
• Support for basic and NTLM authentication
• Solid JavaScript support
• Support for GET and POST submit methods
• Ability to customize the request headers being sent to the server
• Ability to control whether failing responses from the server should throw exceptions or be returned as pages of the appropriate type

Steps to Use HtmlUnitDriver with Selenium

Step 1) In Eclipse, copy the following code. Add the standard Selenium library files to the project. No additional JAR files are required.

package  htmldriver;
import org.openqa.selenium.By;		
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;	
import org.openqa.selenium.htmlunit.HtmlUnitDriver;		
public class htmlUnitYest {				
		public static void main(String[] args) {
                     // Creating a new instance of the HtmlUnit driver
                      
                     WebDriver driver = new HtmlUnitDriver();
                      
           		     // Navigate to Google		
                     driver.get("http://www.google.com");					
          
					 // Locate the searchbox using its name		
                     WebElement element = driver.findElement(By.name("q"));	
                     
                    // Enter a search query		
                    element.sendKeys("Guru99");	
                   
		            // Submit the query. WebDriver searches for the form using the text input element automatically		
                    // No need to locate/find the submit button		
                    element.submit();			
                    
            		// This code will print the page title		
                    System.out.println("Page title is: " + driver.getTitle());		
                    
                    driver.quit();			
         }		
}

Step 2) Run the code. You will observe no browser is launched and the results are shown in the console.

Benefits of HtmlUnitDriver

• Since it does not use any GUI, tests run in the background without visual interruption
• Compared to other driver instances, execution is faster
• You can also emulate different browser versions when running tests through HtmlUnitDriver
• It is platform-independent and easy to run several tests concurrently. It is ideal for load testing.

Limitations:

• It cannot fully emulate JavaScript behavior of every real browser engine

PhantomJS

Reminder: PhantomJS is deprecated and no longer maintained. For new headless work, use Selenium with headless Chrome or Firefox, or move to Playwright. The section below remains for legacy reference only.

PhantomJS is a headless browser with a JavaScript API. It was historically used for headless website testing, accessing and manipulating webpages, and ships with the standard DOM API.

To use PhantomJS with Selenium, you need GhostDriver. GhostDriver is an implementation of the WebDriver Wire Protocol in plain JavaScript for PhantomJS.

The later releases of PhantomJS integrated GhostDriver, so there is no need to install it separately.

Here is how the system works:

Steps to Run Selenium with PhantomJS

Step 1) You need Eclipse with Selenium installed.

Step 2) Download PhantomJS here.

Step 3) Extract the downloaded folder into Program Files.

Step 4) Download the PhantomJS Driver from here. Add the JAR to your project.

Step 5) Paste the following code in Eclipse.

package htmldriver;	
import java.io.File;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.phantomjs.PhantomJSDriver;	

public class phantom {				
		public static void main(String[] args) {
        	        File file = new File("C:/Program Files/phantomjs-2.0.0-windows/bin/phantomjs.exe");				
                    System.setProperty("phantomjs.binary.path", file.getAbsolutePath());		
                    WebDriver driver = new PhantomJSDriver();	
                    driver.get("http://www.google.com");         
                    WebElement element = driver.findElement(By.name("q"));	
                    element.sendKeys("Guru99");					
                    element.submit();         			
                    System.out.println("Page title is: " + driver.getTitle());		
                    driver.quit();			
           }		
}

Step 6) Run the code. You will see the output in the console and no browser will be launched.

NOTE: On the first run, depending on your settings, you may see a security warning from Windows to allow PhantomJS to run. Click Allow Access.

Many organizations have historically used PhantomJS for tasks such as:

• Headless testing
• Screen capture
• Page automation
• Network monitoring
• Rendering dashboard screenshots for their users
• Running unit tests on the command line
• Generating employee handbooks from HTML to PDF
• Combining with QUnit for the test suite

For new work, replace these PhantomJS use cases with headless Chrome, headless Firefox, or Playwright, which receive ongoing security and rendering updates.

FAQs