Preparing for Browser Automation by Using Web Developer Tools to Identify HTML Elements

Background

In a separate article, we introduced the concept of performing synthetic transactions with LogicMonitor. An obviously crucial step of that process is actually scripting the automation. You can have all the right automation tools installed but how do you get started with the actual script? This article will help illustrate some of the preparation needed for developing a browser automation script.

Choose a Browser

You will need to choose a browser for visually inspecting a website and performing automation. While it makes sense to prioritize automated service testing through the same browser as most of your customers would, you should be able to get the same results visually inspecting a website’s elements in any popular browser.

Consider browsers that are compatible with Selenium, a browser automation tool we’re using in our series of articles on executing synthetic transactions with LogicMonitor. Selenium offers webdrivers for some of the most popular browsers:

Mozilla Firefox
Google Chrome
Microsoft Internet Explorer

While Firefox and Chrome generally render websites similarly, Internet Explorer is traditionally an outlier. If you can be flexible, consider starting with Firefox or Chrome.

Choose the Right Website

Your first website choice for browser automation might not be the best choice. How can you tell? The largest factor is whether the webpage elements can be easily identified by an id or class name rather than an XPath. If none of that made sense to you, it’s okay. For now, choose a website but keep in mind that as you continue reading, you may discover it is not the most conducive to automation.

Keep Responsive Web Design In Mind

Many modern websites are designed to adapt or respond, to various screen sizes in order to accommodate desktops, laptops, and handheld devices such as mobile phones. Sometimes web page elements rearrange themselves in order to fit smaller screen sizes. In some cases, elements disappear entirely! You want to keep window size in mind before you attempt to identify elements and automate browser activity.

One way to test responsiveness is to simply resize your web browser. In most cases, resizing your browser to 50% of the screen’s width will be enough to trigger a responsive design. Alternatively, those who are already familiar with browser web developer tools can use those options to simulate other device screen sizes.

Locating Web Page Elements

The Challenge

Consider the login form illustrated below.

Visually, it is obvious what actions you would take to use a login form: click the username field, type username, click the password field, type password, and click the login button. Without thinking, you visually locate the elements you need to use and take action. What about programmatically?

The Solution

HTML elements can be located in many ways, three of which being:

ID
Class name
Xpath

One of the benefits of developer tools in modern browsers is the ability to inspect HTML elements in order to see details such as their id, class, and xpath. The inspection process is mostly visual, resulting in a favorable learning curve. You don’t need to be an expert on HTML, CSS, or JavaScript. Consider again the following user form.

Assuming the default mouse configuration, you could perform an inspection on the username field by right-clicking on the username field to expose a contextual menu and left clicking on the “Inspect Element” menu option. See the illustration below.

After inspecting the username field you would see the web developer tools open automatically and highlight, in blue, the HTML that represents the username input field. You would focus your attention only in the section highlighted in blue for now. The screenshot below is from Firefox’s web developer tools.

You don’t need to understand everything you see. Just look for “id=”. While we technically could also use the class name (“class=”) and Xpath (not illustrated), it is best to prioritize the id because element ids are unique in HTML. Locating HTML elements in any other way either increases the risk or the complexity of locating the right elements. Repeat the inspection process for both the password and login buttons, capturing ids for both when possible.

Remember Element Actions and Wait Times

Some HTML elements such as buttons trigger an action. A login button, for example, typically will submit information to a website and cause the browser to navigate to another page. This loading time is never instantaneous. There is at least some delay that should be considered during browser automation. Remember where these delays are.

You can ballpark the times as long as you are generous (remember not all network connections are equal). You also have the option of using the browser’s web developer tools to measure when a page finishes loading. Be sure to have the developer tools open to the Network section before navigating to a page. The screen capture below illustrates a load time measurement for Google’s home page in–simply to demonstrate flexibility–Chrome’s developer tools rather than Firefox.

You might wonder why the “Finish” time is almost 6 seconds. The Google homepage practically loads into a usable state in under a second; however, the homepage continues to transfer some background data. If you continue waiting, you would see the finish measurement update from 1 second, to almost 6, to as much as 5 minutes. A 5 second wait time would be fine, but, 5 minutes is overdoing it. This is where visual inspection and human discern
ment is important.

Conclusion

In this article, we expanded upon the idea of executing synthetic transactions within LogicMonitor using Selenium, a tried and true automation tool, and Python, one of the most popular scripting languages. By visually inspecting and navigating web content with browser developer tools, you can start planning advanced browser automation that goes beyond recording rigid steps.

Implementing browser automation for your business doesn’t have to be daunting or cumbersome. The LogicMonitor Professional Services team has the talent and is up for the challenge! Ask about a free LogicMonitor Professional Services scoping call or free trial.

Jeoffri Davis

Jeoffri enjoys serving LogicMonitor-sponsored scripting meals that are proudly low-calorie, diverse, vegan, innovative, free-range, local, and BPA-free. As far as he knows, none of the code confections have been solar powered but he still joins his colleagues in demonstrating how much he cares about the environment. You can often find Jeoffri in the common area of LM’s Austin WeWork Shazaming a song that’s probably stacked with 7th chords, walking boxes around Austin parking lots, training his dogs on loose leash walking, or buying another lavender plant because he failed to keep the last one alive.

Subscribe to our LogicBlog to stay updated on the latest developments from LogicMonitor and get notified about blog posts from our world-class team of IT experts and engineers, as well as our leadership team with in-depth knowledge and decades of collective experience in delivering a product IT professionals love.

Preparing for Browser Automation by Using Web Developer Tools to Identify HTML Elements

Background

Choose a Browser

Choose the Right Website

Keep Responsive Web Design In Mind