How we made our Rails feature test suite twice as fast

Feature specs are slow. Everyone knows that! But that’s no reason to avoid them. Our team at Railsware relies on feature specs as the first line of defense against integration bugs and regressions.

Here are some techniques that helped us reduce our feature spec execution time twofold.

Skip login (and other repetitive parts)

Most of our specs cover authenticated app access. So all of them started with a login helper. The rather standard helper visited the login form and filled it in. Textbook stuff.

The only problem that it happened hundreds of times per suite, each time taking seconds.

What’s particularly bad is that in our app, the unauthenticated part is a separate application from the main app. So for every spec, the browser had to load not one but two frontend apps. (Same would be true if in your SPA login causes a full page reload).

The alternative is to set session state programmatically. As we use Devise for authentication, and Devise uses Warden under the hood, we could simply use the Warden test helper.

include Warden::Test::Helpers

login_as(user)

This measure alone saved us 16 minutes of suite run time! Shoutout to Juan for spotting this opportunity.

Other things to consider are:

• Repetitive frontend flows. Don’t ever seed data with frontend interaction. It may seem more “real worldly” but it will kill your performance.
• Long pre-test navigation. If every test starts with clicking through several links, consider adding deep URLs to open page at precisely the view you need it to be. It helps regular users, too!

Avoid loading external resources

You should never load external scripts like Google Analytics or Intercom in your test suite. It’s creating noise for your reporting, and it is slowing tests down.

Such scripts are usually loaded from templates, so avoiding them is best done with an environment check:

<% if Rails.env.production? %>
   <script><!-- Google Analytics snippet --></script>
   <script><!-- Intercom snippet --></script>
<% end %>

But there is a more devious source of test slowdown: loading external assets like fonts, images, videos, or maps. Unbeknownst to you, the browser will load (or attempt to load) everything that is requested, blocking rendering and script execution.

Usually excluding external assets entirely is unfeasible, so you use a proxy to cache and block requests coming from the browser. A rubyist’s proxy of choice is Puffing Billy - you can managed it entirely from your Ruby test suite.

Billy.configure do |c|
  c.cache = true
  c.persist_cache = true
  c.cache_request_headers = false
  c.ignore_cache_port = true
  c.whitelist = %w[localhost 127.0.0.1 and other domains you want to allow]
end

Capybara.register_driver :chrome do |app|
  Capybara::Cuprite::Driver.new(
    app,
    # ... other options here
    browser_options: {
      # ... other options here
      # Billy can't provide real certificates
      'ignore-certificate-errors': nil,
      # Connect to Billy
      'proxy-server': "#{Billy.proxy.host}:#{Billy.proxy.port}",
    }
  )
end

Billy has more options. You typically set it up to block everything, and then you cache (or whitelist) requests that your app cannot work without.

Thanks to leopard for setting up Puffing Billy in our project.

Make the app faster to load

Here’s a no-brainer way to make feature tests faster - make your frontend code faster!

Specifically, making the app load faster makes a big difference for tests, because it happens so many times. Actually, most of the time of your feature test suite is spent downloading assets and parsing Javascript again and again - so profiling and optimizing Ruby code is not very useful.

We took this as an opportunity to introduce Webpack chunking and lazy loading.

Lazy loading of React components is almost trivial, you should try it. In our case, I found that the charts library was only needed for report pages, and that a huge billing dropin was only needed for billing. So, by separating these two features into lazy components, we reduced the initial JS bundle size by 50%.

Separate feature specs when running parallel jobs

If your test suite runs in parallel jobs, separate feature specs and non-feature specs into different jobs. Then non-feature specs do not have to initialize the browser, or precompile assets.

This certainly saves total job time - important for billing - but not necessarily build duration. Your mileage may vary depending on how many specs you have and of which kind.

We use a tool I wrote called split_tests to filter and parallelize tests.

# only feature specs:
./split_tests -line-count \
  -glob 'spec/features/**/*_spec.rb' \
  -split-index $SPLIT_INDEX \
  -split-total $SPLIT_COUNT

# no feature specs:
./split_tests -line-count \
  -exclude-glob 'spec/features/**/*_spec.rb' \
  -split-index $SPLIT_INDEX \
  -split-total $SPLIT_COUNT

Optimize asset compilation

We want to precompile assets and we want to cache the assets between builds, so backend-only commits don’t have to precompile again. By the way, Webpack 5 introduced persistent caching. Before that, only final results of a build could be cached, and if anything changed at all Webpack would rebuild from scratch. With Webpack 5, intermediate results - for example, preprocessed files - can be reused, significantly reducing precompilation time.

Also, we want to disable just-in-time compilation (compile: false flag in Webpacker), because otherwise Webpacker will re-check assets on every request to the app.

And even after that, we discovered that webpacker was consuming not insignificant time even in our non-feature specs, where it is presumably useless. As it turned out, specs that render views - controller, request, or mailer specs - can also trigger asset lookup.

To avoid this, in non-feature specs we stub out Webpacker lookup entirely:

module NoAssetsManifestStub
  extend ActiveSupport::Concern
  included do
    def lookup!(asset_name, _pack_type = {})
      "/asset_stubbed_for_tests/#{asset_name}"
    end
    alias_method :lookup_pack_with_chunks!, :lookup!
  end
end

Webpacker::Manifest.include(NoAssetsManifestStub) if ENV['CI'] && ENV['NO_FEATURE_SPECS']

Conclusion

Specs have a tendency to become slower, not faster, over time. So consider reviewing your suite regularly. Just as new issues creep in, new opportunities to make specs faster arise.

Liked the post? Treat me to a coffee