Issues running headless chrome on heroku
TL;DR use the `--disable-dev-shm-usage` flag when running headless chrome on the heroku private spaces runtime.
At CBRE Build we run headless chrome on heroku worker dynos to generate PDFs for a production application. When attempting to generate larger PDFs (high page and image count) we would encounter the following error:
app[worker.1]: [0411/255147.243199:ERROR:crash_handler_host_linux.cc(433)] Failed to write crash dump for pid 86
This error was not reproducible in development or staging environments, only production. After some
headbanging investigating, it turns out that staging and production environments use different heroku runtimes.
Heroku currently offers 2 runtimes: common and private spaces. The main difference between the runtimes is that common is multi-tenant and private spaces are single tenant. We use the common runtime for staging and the private spaces runtime for production. Since the error message was about writing to the file system, running `df -h` on each runtime was the next step:
common runtime (staging environment)
$ df -h Filesystem Size Used Avail Use% Mounted on /dev/loop6 2.3G 507M 1.8G 23% / /dev/mapper/evg0-evol0 376G 84G 273G 24% /app none 1.0M 0 1.0M 0% /dev udev 30G 12K 30G 1% /proc/kcore none 64M 0 64M 0% /dev/shm tmpfs 6.1G 396K 6.0G 1% /etc/resolv.conf
private space (production environment)
$ df -h Filesystem Size Used Avail Use% Mounted on overlay 65G 1.3G 61G 2% / tmpfs 64M 0 64M 0% /dev shm 64M 0 64M 0% /dev/shm tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/xvdh 65G 1.3G 61G 2% /etc/group /dev/xvda1 20G 18G 1.6G 92% /sbin/dyno-init tmpfs 1.9G 0 1.9G 0% /proc/scsi
Looking at the results shows that only the private space runtime mounts shm on /dev/shm. Some searching on the internet reveals that allocating too little shared memory space to /dev/shm will cause chrome to crash when rendering large pages.
The simple fix is to use the `--disable-dev-shm-usage` flag which forces headless chrome to use /tmp instead of /dev/shm.
Update (2020-05-22): There was a regression in Chrome v83 which reintroduced this issue.
Update (2020-06-02): The regression was fixed.