chemecse

Issues running headless chrome on heroku

2020-02-01

TL;DR use the `--disable-dev-shm-usage` flag when running headless chrome on the heroku private spaces runtime.

At CBRE Build we run headless chrome on heroku worker dynos to generate PDFs for a production application. When attempting to generate larger PDFs (high page and image count) we would encounter the following error:

  app[worker.1]: [0411/255147.243199:ERROR:crash_handler_host_linux.cc(433)] Failed to write crash dump for pid 86
        

This error was not reproducible in development or staging environments, only production. After some headbanging investigating, it turns out that staging and production environments use different heroku runtimes.

Heroku currently offers 2 runtimes: common and private spaces. The main difference between the runtimes is that common is multi-tenant and private spaces are single tenant. We use the common runtime for staging and the private spaces runtime for production. Since the error message was about writing to the file system, running `df -h` on each runtime was the next step:

common runtime (staging environment)

  $ df -h
  Filesystem              Size  Used Avail Use% Mounted on
  /dev/loop6              2.3G  507M  1.8G  23% /
  /dev/mapper/evg0-evol0  376G   84G  273G  24% /app
  none                    1.0M     0  1.0M   0% /dev
  udev                     30G   12K   30G   1% /proc/kcore
  none                     64M     0   64M   0% /dev/shm
  tmpfs                   6.1G  396K  6.0G   1% /etc/resolv.conf
        

private space (production environment)

  $ df -h
  Filesystem      Size  Used Avail Use% Mounted on
  overlay          65G  1.3G   61G   2% /
  tmpfs            64M     0   64M   0% /dev
  shm              64M     0   64M   0% /dev/shm
  tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
  /dev/xvdh        65G  1.3G   61G   2% /etc/group
  /dev/xvda1       20G   18G  1.6G  92% /sbin/dyno-init
  tmpfs           1.9G     0  1.9G   0% /proc/scsi
        

Looking at the results shows that only the private space runtime mounts shm on /dev/shm. Some searching on the internet reveals that allocating too little shared memory space to /dev/shm will cause chrome to crash when rendering large pages.

The simple fix is to use the `--disable-dev-shm-usage` flag which forces headless chrome to use /tmp instead of /dev/shm.

Update (2020-05-22): There was a regression in Chrome v83 which reintroduced this issue.

Update (2020-06-02): The regression was fixed.