Testing MapLibre GL JS applications with MapGrab and Playwright

Testing map-based applications can be different from testing regular apps, but it doesn't have to be difficult. In this article, I will show you how to use MapGrab, a library designed to make testing map apps easier. I'll provide tips on configuring Playwright, writing visual test, and avoiding flaky tests.]

Unique challenges

Maps are canvas based. Map features are rendered on canvas, so, you cannot query elements using DOM API, neither can you run tests in Node with JSDOM.

This implies the use of a testing environment that executes tests in a real browser and leaves us with a few testing library choices:

• Selenium - the oldest and most popular. Your grandpa wrote tests in Selenium. If you already know it, go for it. Also, choose it if you plan on integrating with BrowserStack Device Farm.

• Cypress - the proprietary one. Choose it if you like weird APIs. Running parallel tests requires a third-party plugin since Cypress makes money from selling Cypress Cloud subscriptions. Use it if you don’t mind paying for a better user experience.

• Playwright - modern API, fast parallel tests, easy cross-browser testing. Great docs. Built by Microsoft and fully open source. My choice for this article.

Now, all these libraries were mainly created for querying and running assertions on DOM elements. They also provide APIs to wait for events like page load or network requests. However, this doesn't help when we need to wait for an event sent from MapLibre, like map load or render. Similarly, with navigation, you can navigate to a page, but not to a map location. For these needs, we require another tool.

MapGrab testing library

MapGrab is for maplibre applications what react-testing-library is for React applications. MapGrab integrates with all libraries mentioned in previous paragraph. It has a simple but powerful async API that let’s you zoom to the area of interest, wait for map loaded and other events, query rendered features from within Playwright test cases. This let’s you keep your fixtures clean.

Waiting for events is particularly useful. By default Playwright support only a handful of events and if you need more, you need to implement a connection between fixture context (the browser) and NodeJS, which is not straightforward.

With MapGrab you just need would instrument your fixtures by calling installMapGrab:

import { installMapGrab } from '@mapgrab/map-interface';

const map = new maplibregl.Map({
  container: 'map',
  style: 'https://demotiles.maplibre.org/style.json',
});

// Instrument fixtures
installMapGrab(map, 'mainMap');

And then in tests you can wait for map load and even zoom to a given location:

// Extended test
import { test } from './setup';

test("foo test", async ({ mapController, mapLocator page }) => {
  await mapController('mainMap').waitToMapLoaded();
  // Assertions...
});

The API is quite powerful, you can also center map in a given location or query map feature and fit map view to its bbox.

Read the Getting started for the full instructions.

Verifying map features

Depending on what you want to verify, you need different tests. Usually we render something and first thing we check is the visibility. For that we just need to verify if a feature was rendered.

const controller = mapController('mainMap');
const marker = mapLocator('layer[id=marker]').first();

await expect(marker).toBeVisibleOnMap();

Visual tests

Visual tests are the most accurate way to verify what has been rendered on the map. How do they work? When visual test is first run, screenshot of the map is saved on the disk. On subsequent test runs, new screenshot is taken and compared pixel by pixel to check if new version matches with the saved snapshot.

In practice

Only thing that differentiates visual tests from other types, is the toHaveScreenshot() matcher:

test('renders grid: zl: 0', async ({ page, mapController }) => {
  const controller = () => mapController('mainMap');
  await page.goto(pageUrl);

  await controller().setView({ zoom: 0, center: [0, 0] });
  await controller().waitToMapLoaded();

  await expect(page).toHaveScreenshot();
});

When test fail, Playwrights runs a server with a test report page on which you can see the expected and actual screenshot, and compare them.

Remove snapshot platform suffix

By default, Playwright adds a platform suffix to the snapshot name. For example: example-test-1-chromium-darwin.png. You will want to remove this suffix because if you use a different platform than your CI, or if your colleague uses Linux while you use a Mac, renaming these files can become a big hassle. To remove the platform suffix, add a custom snapshotPathTemplate to your Playwright configuration.

Fighting flakiness

Visual tests can be flaky for several reasons:

• Base style changes - For example, if you are testing whether your MapLibre drawing library renders a rectangle on the map, and you have set the base map style to always use the latest version or a wildcard for the version, a new style release could introduce visual changes. These changes might break your visual tests without you realizing it.

• Base map data changes - Even if the map style is the same, the tile data may be updated, altering what is rendered on the map.

• Network errors - The base map style or tiles may not load due to network errors or slow server response times.

• Different rendering depending on the browser and platform - If you work in a team where people use both Macs and Linux machines, or if your CI uses a different OS than you do, you might experience some flakiness due to rendering differences across platforms. This is quite rare when it comes to MapLibre.

To eliminate flakiness:

• Remove the base map if it’s not crucial for the test case. If you need the base map for some cases but not others, you can use MapGrab’s exposeLayers to hide all layers except the ones you need.

• Use a fixed version of the base map style. If you need the base map style in your tests, use a fixed version of the map style.

• Use offline map styles to eliminate network errors and speed up the tests. Instead of using online styles, opt for offline ones.

• Run your tests inside a Docker container. I suggest doing this only if you encounter flakiness caused by rendering differences on different platforms. Setting up Docker requires some effort, and it can make tests take longer and be more cumbersome to run.