Ever since I published the Turkey repository for crowd-sourcing image segmentation annotation on Amazon Mechanical Turk, I have been thinking about how to keep improving it using the best practices of software engineering. One key improvement that I implemented recently is getting the GitHub repo to run full system tests on every code push. It is fully automated, consistent, environment agnostic, and free.

1. Turkey

The product I am testing is called Turkey. It is a HTML file that you can paste into Amazon Mechanical Turk as a custom job. It allows crowd sourcing image annotation data, such as instance segmentation, bounding boxes, and lines, in the same job. It supports undo, redo, zooming in, zooming out, drag, and reset. I made this when I was in college because there wasn't a good and free solution for online image annotation crowd sourcing, and we needed one for a research project.

2. Cypress

Cypress is great for system tests. I plan to write a separate article on writing good Cypress tests. For the scope of this article, I will only talk about what I use it for. As the functionalities of Turkey expands, it is hard to keep track of all the things it does. It is even harder to manually test all the features reliably. Cypress programmatically simulates clicking, dragging, and typing. All the things that a user would typically do when using my product. Due to the nature of the image annotation tool, it is not enough to just know that Cypress found the rendered components and interacted with them successfully. I also need to make sure that the graphical user interface (GUI) looks exactly the way I expect it to. So I use a Cypress plugin package called cypress-image-snapshot(npm link) to take screenshots and compare them with the stored version. A typical test for Turkey looks like this:

describe("buttons", () => {
  it("can change class", () => {
    cy.getById("label_class").select("class1");
    dot_mode_coordinates_2.forEach((coordinate) => {
      cy.get("canvas").click(coordinate.x, coordinate.y);
    });
    cy.get("canvas").matchImageSnapshot("multi_class");
  });
}

the command matchImageSnapshot would take a screenshot of only what cy.get("canvas") returned, compare it with the stored screenshot of the same name, and check of percentage of different pixels. It fails if the percentage is above a customizable threshold. If there is not a stored screenshot by that name, then it stores the freshly captured one as the correct answer.

To run the tests, first you need to serve the website on localhost, and then run cypress run. It will automatically run all the tests found within the cypress folder. You can also use cypress open to watch it in action, but be aware that this mode could be using a different viewport than headless mode unless overridden. The resolution difference will cause snapshot comparison to fail.

3. GitHub Actions

GitHub released their Actions feature in 2018. It is great to see native support of CI/CD on GitHub. By specifying a \\.github\\workflows\\github-actions.ymlfile in your repo, GitHub automatically picks up the action definitions. The action then gets triggered on every pull request or push. Cypress now has their official GitHub action, and the options are quite impressive. So I started experimenting with the most basic version:

name: Turkey end-to-end tests chrome headless
on: [push]
jobs:
  cypress-run:
    runs-on: ubuntu-20.04
    steps:
      - uses: actions/checkout@v2
      - uses: cypress-io/github-action@v2
        with:
          start: npm start
					wait-on: "<http://localhost:5000>"
          browser: chrome
          headless: true

Let's go through the file line by line:

name: Turkey end-to-end tests chrome headless: name of the action. Doesn't affect any functionality other than helping you remember what it does

on: [push]: this action will be triggered on every git push

jobs:defines what runs when action is triggered.

cypress-run:: this is also just a name for the job. You can have multiple jobs defined. Jobs run in parallel by default, but they can also run in sequence. So in our case, we can have multiple jobs running, one for each supported browser, or a slightly different variation in settings.

runs-on: ubuntu-20.04: operating system that the action runs on. As it turns out, this will have an impact on our system tests results. More on that later.

steps: steps of the job. These run in sequence.

- uses: actions/checkout@v2: uses version 2 of a community action that downloads your repo to the host.

- uses: cypress-io/github-action@v2: uses the cypress action that runs the job. It runs cypress run. Note that, for this to work, there must be a package.json file in the repo and has cypress as a dependency in it.