Selenium 141 - Alternatives to BrowserMob Proxy

15 Aug 2013 | By Matt Robbins | Comments | Tags testing webdriver selenium browsermob-proxy

Selenium Issue 141

The long story of Selenium Issue 141 is not worth recounting here as it has been covered in depth by Selenium core committers Simon Stewart and David Burns.

In a nutshell issue 141 is a hotly contested debate as to whether Selenium-Webdriver should provide public API for querying HTTP status codes and response headers.

I pretty much agree with the Selenium guys and can see how introducing this API would be the start of a very long and slippery slope for the project. However in the day to day work of an automation developer it is often essential to have access this information, for example most Web Apps have to do some kind of tracking which takes the form of a request for a 1x1 image with a query string containing the tracking key value pairs and we are frequently asked to validate these calls.

BrowserMob - the solution….maybe

The conventional wisdom is that you should use the Selenium approved scriptable proxy BroweserMob Proxy to get around this problem.

Up until recently I followed this thinking and had used BrowserMob with reasonable success (see my earlier post about using BrowserMob through a proxy, however I had never had to use it in a critical CI pipeline and when I did things got a little less appealing.

There is no doubt that BrowserMob is a great project but using it via the Ruby bindings I found that all too often the Java process would not shut down cleanly after a test run (particularly if the test run failed for some other reason) meaning that on subsequent runs the port was locked and the proxy would not restart.

All these problems are solvable of course and I intend to keep using BrowserMob and contribute any fixes if I can.

However for the project I was on I needed something bullet proof which limited the amount of moving parts in my CI pipeline to capture and validate tracking calls from a mediaplayer.

Browser Extensions - an alternative approach…

These trials took me to an alternative approach - browser extensions!

Starting with Chrome I knocked together an extension that uses the chrome.webRequest api to capture network requests and push them back into the page under test via local storage. I then raise custom events which the page under test can register for.

Of course this assumes you are in control of the page under test, in my case I was testing a component on a test page so that was fine, I listened for the custom events and pushed them out into the DOM so I could scrape them using Selenium as normal. If this is not the case for you (which is likely) I think you could still use this solution, for example by updating the extension to push the events into a global object which you could query by injecting JavaScript from Selenium. Of course this is not an uber-clean but no solution in this area is!

I won’t go into the code as you can have a play for yourself, it turns out to be very simple and of course it is a bullet proof and fast way of capturing this information.

But what about the other browsers?

Of course this only addresses Chrome…to compete with BrowserMob we need to be able to cover as many of the other browsers supported by Selenium.

For this I used Crossrider which provides a limited api but allows you to generate extensions for Chrome, Firefox, IE and Safari.

You cannot be as sophisticated as you can using the Chrome APIs directly but you can still accomplish a fair amount.

Again feel free to have a play around with the simple crossrider extension I wrote to scrape net requests.


There is no ‘right’ way of dealing with the lower level aspects of web automation. In some cases using BrowserMob makes perfect sense and should be your go to solution. However if you need a resilient way of capturing net requests, blocking net requests, accessing response codes or headers then a browser extension might just be the way to go.

Building Erlang-r16 and Elixir on OSX using HomeBrew

06 Aug 2013 | By Matt Robbins | Comments | Tags erlang elixir osx homebrew

A quick one just in case anybody is struggling to build Erlang-r16 on OSX. I wanted to play around with Elixir the latest and slightly more offbeat (dare I say hipster) functional language to hit the big time, Elixir runs on the Erlang VM and you need Erlang-r16 to build and run Elixir.

To install Erlang-r16 on OSX using homebrew you in theory just need to add the homebrew-versions tap and brew install:

brew tap homebrew/versions
brew install erlang-r16

However I got compile errors when running the install (note I think this may be even specific to the latest MacBooks with the new Haswell chips).

erl -noshell -noinput -run prepare_templates gen_asn1ct_rtt \
           asn1rtt_check.beam asn1rtt_ext.beam asn1rtt_per_common.beam asn1rtt_real_common.beam asn1rtt_ber.beam asn1rtt_per.beam asn1rtt_uper.beam >asn1ct_rtt.erl
erlc -W  +debug_info -I/private/tmp/erlang-r16-HFuS/otp-OTP_R16B/lib/stdlib -Werror -o../ebin asn1ct_rtt.erl
Running Erlang
asn1ct_rtt.erl:1: syntax error before: Running
asn1ct_rtt.erl:14: no module definition
make[2]: *** [../ebin/asn1ct_rtt.beam] Error 1
make[1]: *** [opt] Error 2
make: *** [secondary_bootstrap_build] Error 2

The reason for this is that newer Macs with the latest versions of X-Code use LLVM-GCC and not GNU GCC. Homebrew now attempts to build with LLVM by default which is probably sensible as this is the platform compiler, however it is likely for a fair number of packages this won’t work as their builds have not been tested on LLVM. This seems to be the case with the latest Erlang versions.

The additional problem at the moment seems to be that the Homebrew project is currently changing the way they handle the situation around cross compiling with different C compilers.

You can use the homebrew-versions tap to install a specific gcc e.g.

brew tap homebrew/versions
brew install gcc48

However in Homebrew 0.9.4 at least there seems to be no way of telling it to use these GCC versions when building. You can specify --use-gcc or set the HOMEBREW_CC environment variable but neither worked for me. This issue got me thinking as the only success people had seemed to have is where they had the apple-gcc42 compiler.

When you do export HOMEBREW_CC=gcc Homebrew seems to be hardcoded to look for the apple-gcc42 package and not any from the homebrew-versions tap. I think this pull request will make this integration smoother.

For the time being then, to get a working Elixir and Erlang do the following:

export HOMEBREW_CC=gcc
brew update
brew tap homebrew/dupes
brew install apple-gcc42
brew install erlang-r16
brew install elixir

You can probably avoid exporting HOMEBREW_CC and just use (but I haven’t tested this method):

brew update
brew tap homebrew/dupes
brew install apple-gcc42
brew install --use-gcc erlang-r16
brew install elixir

This gets the 4.2 GCC that currently Homebrew is hardcoded to look for, indeed you can see this if you run brew --config as it now lists a GCC-4.2 value which would have been absent before:

HEAD: 6b5b347332f7fdad35a5deab79fa71018e02a2b4
HOMEBREW_CELLAR: /usr/local/Cellar
CPU: quad-core 64-bit haswell
OS X: 10.8.4-x86_64
Xcode: 4.6.3
GCC-4.2: build 5666
LLVM-GCC: build 2336
Clang: 4.2 build 425
X11: N/A
System Ruby: 1.8.7-358
Perl: /usr/bin/perl
Python: /usr/bin/python
Ruby: /Users/matthewrobbins/.rbenv/shims/ruby

Git Clone via SSH using Jenkins on Windows

28 Jun 2013 | By Matt Robbins | Comments | Tags ci jenkins

This blog post is nothing revolutionary but more of a note to self so that I never have to learn this painful lesson again!

My current team has Windows specific CI dependencies, hence I had no choice but to setup our Jenkins instance on Windows. The cleanest way to do this is to use the Jenkins Installer which installs Jenkins as a Windows Service.

That works fine, but of course the first challenge you face is cloning your Git repo over SSH (the default for the Jenkins Git plugin).

I’ll assume you have done the following:

  • Installed Git Client

  • Created SSH Keys and ensured that you can successfully clone a repo from your Git Bash (running as current user).

    Note: In theory you should not use passwordless keys as anybody obtaining your key will have access to your account. However for Jenkins you cannot have a password as the Job would hang when attempting to clone. From Git Bash this problem is solved using ssh-agent but this won’t work in Jenkins as we are using the Git plugin and not the shell.

    Bottom line - for Jenkins use a passwordless key - you can mitigate the risk by having a dedicated Jenkins Git User and only allowing them pull access.

  • Set up the Git plugin in Jenkins via ‘Manage Jenkins’ and point it to the Git cmd file on the system, as in the screenshot below:


Now we get to the tricky part. When Jenkins runs as a Windows Service it does not run under a normal user account, it runs under the “Local System Account”. Hence even though you have Git Clone working as the current user it will fail in Jenkins.

To solve this problem we need to put our private ssh key in the correct location for the Local System Account.

First of all we need to find out where this user’s home directory is, since it will not exist under C:\Users…

  • Download PSTools
  • Extract to a conventiant location and using a command prompt navigate to that location (or add to path if you prefer)
  • Start the PsExec tool by running:
PsExec.exe -i -s cmd.exe
  • Now we need to find out where this user’s home directory is:
echo %userprofile%
  • For me this location was:
  • Now you can open Git Bash within this shell by running the following command:
C:\Program Files (x86)\Git\bin\sh –login –i
  • You can copy your existing private key into the correct location for the Local System User, for me this was something like the following:
cd ~
cp /c/Users/matt/.ssh/id_rsa ~/.ssh/
  • Finally you can test that you can successfully authenticate to Github as the Local System User:

ssh -T

  • If all is successful you should see the usual message:
Hi username! You've successfully authenticated, but GitHub does not provide shell access.

All that’s left to do now is test from Jenkins…Huzzah!

3DES Zeros Padding with Ruby and Openssl

29 Nov 2012 | By Matt Robbins | Comments | Tags testing ruby

Although many encryption libraries exist for Ruby (and here I am talking ‘C’ Ruby in the main) you will most typically want to use Openssl.

Recently I needed to encrypt a simple JSON string using 3DES (ECB mode).

IMPORTANT - You should never use ECB mode it is insecure and highly prone to brute force attacks….but I had no choice this is what the external service required.

It all seemed so simple and my initial script looked something like this:

require 'openssl'
require 'base64'
require 'json'

des ='des-ede3')
des.key = 'yourchosendeskey' 

edata = des.update('{"somekey":"somevalue"}') +
b64data = Base64.encode64(edata).gsub("\n",'') #remove EOL insterted in Base64 encoding

Although this worked perfectly in my local test harness the external PHP service failed to decrypt the data. The reason for this is all to do with padding.

Certain encryption schemes (such as 3DES) are known as ‘Block Cipers’ and operate on a fixed block size i.e. in order for the data to encrypt/decrypt successfully it must be presented as a multiple of the required block size (in this case 8 bytes).

Openssl ensures this by inserting PKCS#5 padding, this padding scheme is preferred because it can be easily be distinguished from the actual data and allows for basic integrity or password checking.

Sadly (and possibly unsuprisingly) the PHP library I was integrating with did not use Openssl and instead used MCRYPT which uses ‘Zeros Padding’ and not PKCS#5.

Openssl does not support Zeros Padding out of the box but you can do this yourself by telling Openssl not to pad the data and ensuring the data is a multiple of the block size by adding ‘/0’ characters at the end of the data, the code looks something like this:

block_length = 8
des.padding = 0 #Tell Openssl not to pad
json = '{"somekey":"somevalue"}'
json += "\0" until json.bytesize % block_length == 0 #Pad with zeros
edata = des.update(json) + 
b64data = Base64.encode64(edata).gsub("\n",'')

IMPORTANT - you should choose a much better encryption scheme and padding mechanism, but if you really have to integrate with some ‘interesting’ PHP this may help!

Setting two space indent for Gherkin Features in Vim

26 Nov 2012 | By Matt Robbins | Comments | Tags testing vim cucumber

Users of Vim (or GVim, Macvim etc) who write Gherkin Feature files will know that for most installations (on Mac certainly) you appear to have auto-magical support for Gherkin Feature file syntax i.e. nice colours, auto format etc.

This is possible because the distribution of Vim installed on the machine, or which you installed comes shipped with Tim Pope’s excellent vim-cucumber plugin.

If your Vim does not ship with this you should most certainly install it right away!

I have always auto-indented using gg=G and never really thought too much about the indent level. It turns out that presently the vim-cucumber plugin does not specify this (although I note that it is commented out in the source at the bottom of the syntax file)

My vimrc defines this globally as follows:

set expandtab
set tabstop=4
set shiftwidth=4

So of course I was getting 4 level indent in my feature files.

The current team I am working with was not keen on this and I can see their point it does seem excessive, 2 would seem more than enough and in fact is what is used in the examples on

So how best to do this?

One option would be to navigate to the file relevant to your Vim install location:


And change this line:

" vim:set sts=2 sw=2:

To this:

set sts=2 sw=2

This is a hack though (when you reinstall vim this change would be lost) and will need to be done for all copies of vim on the machine.

The answer is to use Vim’s ftplugin functionality.

First of all you need to switch this on in your vimrc, adding the following:

filetype plugin indent on

The you have two options, firstly you can create a file ‘~.vim/after/ftplugin/cucumber.vim’ and add the following:

setlocal expandtab
setlocal shiftwidth=2
setlocal softtabstop=2

Or add the config directly into your .vimrc:

au FileType cucumber setl sw=2 sts=2 et

This is obviously common knowledge to experienced Vim users but may be useful for anybody out there who is relatively new to Vim and wants to customize syntax on a per language basis in a clean way.

Cucumber and the global scope problem - bring on page objects and the MGF pattern

06 Aug 2012 | By Matt Robbins | Comments | Tags testing cucumber cpaybara

There is no argument, Cucumber has added a new dimension to acceptance testing, Gherkin Features are living executable requirements and that for me is the benefit we should never loose sight of.

So you’ve clocked this and created a bunch of automated tests driven from you’re features….but can too much Gherkin be a problem?

In my experience it can and it’s something you need to keep a close eye on to avoid you’re tests becoming as big a maintenance burden as the application code.

The Global Scope Problem

For a project with just a few features you can quite happily get away with a project structure that looks something like that below. All your steps in a single file containing the automation code for your features.

/ /~step-definitions/
/ / /-steps.rb
/ /-example1.feature
/ /-example2.feature
/ /~support/
/ / /-env.rb/

This is fine for a small project, but let’s say you get above 10-15 scenarios, soon the steps file will become very big and you will start spreading steps over a few files with roughly representative names. Then you run headfirst into the ‘Global Scope Problem’….

Cucmber has the concept of a ‘World’ object, a kind of giant mixin which allows your Features, Steps and Support files to share context through the use of Ruby instance variables (the ones preceded by @ characters). Whilst this is essential to the functioning of Cucumber it is also our enemy.

As you’re project grows this shared scope can cause chaos, an IDE or editor that understands Cucumber syntax will help, but ultimately you will have features mapping to steps which could be in any steps file and where state can be affected by any other step in any file via instance variables.

This can end up as mess of procedural spaghetti.

The Page Object solution and the MGF Pattern

An excellent blog post by Joseph Wilk got me thinking seriously about this when I was working on a project which was being bitten hard by the global scope problem. Integrating the Page Object Pattern into your Cucumber tests is really your only option for containing the problem of global scope.

At this point I am going to introduce what I am calling the MGF pattern, which stands for ‘Model, Glue, Feature’.

  • Model - You’re page objects
  • Glue - The step definitions
  • Feature - The Gherkin Features

The Page Object Pattern will be familiar to most of those reading this blog so I won’t go over it again. However I will suggest the following ‘Golden Rules’. I have adapted these from Simon Stewart’s wiki page on the Selenium site because as with most things he has it nailed:

  1. PageObject methods return other PageObjects or Self (whenever a page load is triggered)…
  2. Assertions should nearly always be in the Glue (step defs) not Page Objects (exception - ‘correct location’ check)
  3. Different results for the same action are different methods (‘save_expecting_error’, ‘save_expecting_submit’)
  4. Never call steps from steps (add helpers to objects if compounding steps are needed)
  5. Have a hash of element locators in each PageObject (or register custom locators with Capybara)
  6. NEVER pollute test code with page internals i.e. no css xpath in test code

Page objects possess all the knowledge about the application under test, the Glue simply binds the objects to the Gherkin features.

If you follow this pattern I would expect you to end up with a project layout something like that below.

/ /~examples/
/ / /~step-definitions/
/ / / /-steps.rb
/ / /-signin.feature
/ / /-dashboard.feature
/ /~support/
/ / /-env.rb/
/ / /~objects/
/ / / /-signin.rb
/ / / /-dashboard.rb

Intergrating Capybara

Many people including myself like using Capybara and so it is worth noting how to integrate this into your Page Objects. Because Page Objects will live outside the Cucumber ‘World’ then you have to options available:

  1. Include the Capybara::DSL module in you’re page objects
  2. Pass around an instance of a Capybara Session

The advantage of the first method is that you only have to do this in a ‘base’ class which all other objects can inherit from and obtain those methods, the advantage of the second method is that you could potentially pass in a mock if you wanted to unit test you’re page objects (though the value of doing this is debatable and could fill an entire blog post).

Below is an example base class page object which may help get you started. In this example @session is an instance of an object which acts as a container for our current page and is the only variable that couples the step definitions (or Glue) it also acts as a way of storing data that needs to persist across a test run e.g. user.

class GenericPage
  include Capybara::DSL
  include Capybara::Node::Matchers
  include RSpec::Matchers

  attr_accessor :url

  def initialize(session)
    @session = session

  def correct_page?
    page.should have_css self.class::ELEMENTS['page check element']

  def element_exists?(element)
    page.has_selector? self.class::ELEMENTS[element]

  def current_url

class SigninPage < GenericPage


    'page check element'            => '.myapp-signin > h1',
    'page check value'              => 'Weclcome to MyApp',
    'username'                      => '#username',
    'password'                      => '#password',
    'save'                          => '#submit_button',

  def initialize(session)

  def visit
    Capybara::visit LOCATION

  def signin_expecting_success

  def signin_expecting_error_using(unique)


  def signin
    fill_in ELEMENTS['username'], :with => @session.user.username
    fill_in ELEMENTS['password'], :with => @session.user.password
    click_on ELEMENTS['save']


Adding Page Objects and following the MGF pattern will help you combat the ‘Global Scope’ problem that can hit on large projects using Cucumber to drive automated tests. It will mean changes can be isolated to single points in the code and add logical structure which will reduce the maintenance burden.


I have just finshed reading some excellent posts on Nathaniel Ritmeyer’s blog which elaborate on how we can manage page objects thus tackling the issues of step coupling. He also maintains SitePrism which looks like an interesting dsl for page objects (and removes the need for the ugly constants I have in the examples above).

Anyway I think I will follow up this post in a couple of months and elaborate on the issues around coupling in our glue code and how best we can minimize what is one of the major problems with Cucumber.

Passing parameters to a downstream Hudson (Jenkins) job

03 Aug 2012 | By Matt Robbins | Comments | Tags testing ci

I wanted to run an acceptance test in a downstream job. However I needed to grab the value of a build parameter from the upstream job. It turns out Hudson (Jenkins) does not support this ‘out of the box’.

You can add a plugin, however this was going to cause a time delay in my environment as I did not have admin rights to do this.

So a thought struck me…trigger the downstream job from the shell in the upstream and use the REST API.

Now this is all fine and to be fair there is reference on the Hudson wiki but it is not totally obvious.

You can’t do a get with a query string it needs to be a POST with params, below is an example using curl.

json="{\"parameter\": [{\"name\": \"NAMESPACE\", \"value\": \"test\"}, {\"name\": \"ENVIRONMENT\", \"value\": \"production\"}], \"\": \"\"}"
curl -X POST $url -d token=FRAMEWORKS --data-urlencode json="$json" -k -E "/data/certs/mycert.pem"

You’ll notice you need the security token in order to run this. This can be set in the upstream job ‘Configure > Trigger Builds Remotely > Authentication Token’

The funny thing is I did all of this and then realised the version of curl on the Red Hat 5 system I was on didn’t support –data-urlencode, luckily –data sufficed for my needs.

Modal Dialog Present exception Selenium-Webdriver

23 Mar 2012 | By Matt Robbins | Comments | Tags testing webdriver selenium capybara ruby

Let me tell you an incredibly boring story…

We have a scenario where a user clicks a link on a test page, some JS runs creating a cookie, the page reloads and then the cookie is read by JS on the next page as it loads and then is deleted.

In my test I wanted to assert that the cookie was created and the only way I could do this using Webdriver was by popping up a JS alert on the test page when the link is clicked to delay the page reload and then using Webdriver read the cookies.

#whilst the page reload is blocked by the JS alert
cookie = page.driver.browser.manage.cookie_named 'foo'
alert = page.driver.browser.switch_to.alert #using js alert on acceptance test page to stop execution and allow time to check presence of cookie

This hack worked fine and served the purpose it was intended for….

That was until the guys at Selenium got all responsible and my tests started failing with webdriver throwing a ‘Modal dialog present’ exception.

I can’t find anything in the change logs but it looks like they have done this deliberately which makes sense…why would you want to interact with the page when a modal dialog is present????

Oh well…thought I would just mention it in case anybody else was wondering why this had happened!

Using Selenium Webdriver and Browsermob Proxy through a corporate proxy

04 Jan 2012 | By Matt Robbins | Comments | Tags testing capybara webdriver selenium browsermob

Browsermob Proxy proxy is a really useful tool that enables the user to effectively have an add-on api to ‘out of the box’ Selenium Webdriver which enables you to do things such as set header values, dump out har data etc. There is also a Ruby binding maintained as a separate gem which can be found here.

The problem I had was how to use this through/behind a corporate proxy, obviously it works by starting Firefox with a profile pointing to the browsermob proxy, which would then need to be told to navigate through a corporate proxy/firewall.

It turns out that the Java project does support this in Beta 4 and is documented here. Note this feature is not currently in the released binary (Beta 3) so you will need to ‘git clone’ the project and ‘mvn install’ to generate the jar file.

I then added a crude monkey patch which can be seen below, obviously when the next Beta of the Java code is released I can hopefuly submit a pull request for the Gem, which will be more elegant and better thought out, but for now if you need the hack here is the code:

require 'rubygems'
require 'bundler/setup'
require 'selenium/webdriver'
require 'browsermob/proxy'

#Patch required for proxy support - note: proxy support only added in beta-4 (not yet released as binary) and the patch is required for the gem to use this
#functionality.  Once beta-4 is released (Java Code) then a pull request could be submitted for the Gem.
module BrowserMob
  module Proxy
    class Client
      def self.from(server_url)
        port = JSON.parse(
, "proxy?httpProxy=myproxyhost:80").to_s, '')

        uri = URI.parse(File.join(server_url, "proxy", port.to_s))
        resource = resource,, port

server ="/Users/robbim02/dev/utils/browsermob-proxy-2.0-beta-3/bin/browsermob-proxy") #=> #<BrowserMob::Proxy::Server:0x000001022c6ea8 ...>


proxy = server.create_proxy #=> #<BrowserMob::Proxy::Client:0x0000010224bdc0 ...>

profile = #=> #<Selenium::WebDriver::Firefox::Profile:0x000001022bf748 ...>
profile.proxy = proxy.selenium_proxy

driver = Selenium::WebDriver.for :firefox, :profile => profile

proxy.new_har "google"
driver.get ""

har = proxy.har #=> #<HAR::Archive:0x-27066c42d7e75fa6>
har.entries.first.request.url #=> ""
har.save_to "tmp/google.har"


Cucumber 'requires' option - autoload magic for your objects

24 Nov 2011 | By Matt Robbins | Comments | Tags testing cucumber ruby

Once your cucumber projects get to a certain size you almost certainly start to think about moving away from monolithic steps files and toward a more object based approach.

This blog post describes how you might go about doing this and is definitely worth a read.

So, I wanted to have a project structure something like that depicted below:

/ /~examples/
/ / /~step-definitions/
/ / / /-steps.rb
/ / /-example.feature
/ /~support/
/ / /-env.rb/
/ / /~objects/
/ / / /-myobjects.rb

And the good news is that this worked fine when I ran my cukes as follows:

cucumber -p default

However, when I ran with a specific feature it blew up as it failed to load my custom objects (myobjects.rb).

cucumber -p default features/examples/example.feature
uninitialized constant MyCustomObject (NameError)

So like every good hacker I started to delve down into the innards of the Cucumber runtime code, which whilst interesting was unnecessary had I read the help!!!

-r, --require LIBRARY|DIR        Require files before executing the features. If this
                                     option is not specified, all *.rb files that are
                                     siblings or below the features will be loaded auto-
                                     matically. Automatic loading is disabled when this
                                     option is specified, and all loading becomes explicit.
                                     Files under directories named "support" are always
                                     loaded first.
                                     This option can be specified multiple times.

Note if you run a feature with tags then all code under /features will be pulled in, as when you run with no specificity.

So to always include all your code under /features, but still run a single feature - do the following:

cucumber -p default --require features features/examples/example.feature
1 2