Securing Development with Docker and gVisor

6 minute read

Enclosing the development environment in a container might improve the security and help against certain software supply-chain attacks. However, containers were first introduced for resource management and bundling dependencies. As some might say - containers are not a security boundary.

Malicious actors can still escape from containers, especially when there is a kernel bug - Dirty Pipe container escape PoC. There have been more than a few Linux vulnerabilities this year and Bruce Schneier concludes that Zero-Day Vulnerabilities Are on the Rise.

The good thing is that you can trade some performance for additional security of your containers.

What is gVisor?

The gVisor team calls it an “Application Kernel for Containers”. It is an OCI container runtime for Docker (and k8s). Simply said, the system calls to the Linux kernel from the applications in the container are trapped and handled by gVisor. It acts as an additional layer of protection and reduces the attack surface of the host.

You can read more on the topic in their documentation and Security Model.

Download & Install gVisor

The gVisor container runtime is known as runsc.

Download, check and move the runsc and containerd shim to /usr/local/bin, as shown in the gVisor’s installation guide:

(
  set -e
  ARCH=$(uname -m)
  URL=https://storage.googleapis.com/gvisor/releases/release/latest/${ARCH}
  wget ${URL}/runsc ${URL}/runsc.sha512 \
    ${URL}/containerd-shim-runsc-v1 ${URL}/containerd-shim-runsc-v1.sha512
  sha512sum -c runsc.sha512 \
    -c containerd-shim-runsc-v1.sha512
  rm -f *.sha512
  chmod a+rx runsc containerd-shim-runsc-v1
  sudo mv runsc containerd-shim-runsc-v1 /usr/local/bin
)

Install the runtime and restart Docker:

sudo /usr/local/bin/runsc install
sudo service docker reload 

What runsc install does is to update /etc/docker/daemon.json and add runsc runtime to it. The Docker’s daemon.json configuration file would become something like:

{
    "runtimes": {
        "runsc": {
            "path": "/usr/local/bin/runsc"
        }
    }
}

You can now run a container using runsc like this:

docker run --rm --runtime=runsc hello-world

or you can set the default-runtime to runsc in your daemon.json, if you don’t mind the drawbacks.

Note. The containter runtime introduces a performance penalty. It would increase the container start-up time and make some processes slower.

Use runsc in Your Development Environment

In my previous post Language Runtime Version Management, we have used Docker to manage multiple versions of language runtimes and compilers.

In order to improve the security, you only need to add --runtime=runsc to the Docker command. For instance in the .bash_profile:

here() {
  docker run -it --rm --runtime=runsc -v "$PWD:/work" -w /work -u "$(id -u):$(id -g)" "[email protected]"
}

Or in the shell script located at $HOME/bin/here:

#!/bin/sh

exec docker run -it --rm --runtime=runsc -v "$PWD:/work" -w /work -u "$(id -u):$(id -g)" "[email protected]"

Benchmarks

/* because everybody loves benchmarks */

As we have already said, securing your containers using runsc would impact the overall performance. In order to evaluate how significant this impact is, I would build a React app and time it.

Testbed:

CPU: AMD Ryzen 7 5700G
Mem: 2 x 16GB DDR4-3200
Linux kernel: 5.16.12
runsc version release-20220425.0

Running create-react-app build using the default Docker container runtime runc:

[email protected]:~/project$ $ here --runtime=runc node:16 bash -c 'time yarn build'
yarn run v1.22.18
$ react-scripts build
Creating an optimized production build...
Compiled with warnings.

File sizes after gzip:

  499.72 kB  build/static/js/main.512bf5e1.js
  2.97 kB    build/static/css/main.1be55ed1.css


Done in 16.87s.

real	0m17.040s
user	0m30.596s
sys	0m1.418s

Running create-react-app build using gVisor container runtime runsc (with ptrace, the default):

[email protected]:~/project$ $ here --runtime=runsc node:16 bash -c 'time yarn build'
yarn run v1.22.18
$ react-scripts build
Creating an optimized production build...
Compiled with warnings.

File sizes after gzip:

  499.72 kB  build/static/js/main.512bf5e1.js
  2.97 kB    build/static/css/main.1be55ed1.css

Done in 41.23s.

real	0m41.688s
user	0m57.270s
sys	0m6.600s

The gVisor performance penalty increases the build time more than two times, from 17s to 42s. There is an additional 83% increase of userspace CPU time (31 vs 57 seconds). However, the actual userspace CPU time could even be a bit higher, since the runsc starts multiple processes that might not be accounted. The system time is expectedly many times higher than the baseline.

runsc version release-20220228.0

Running create-react-app build using the default Docker container runtime runc:

[email protected]:~/project$ $ here --runtime=runc node:16 bash -c 'time yarn build'
yarn run v1.22.18
$ react-scripts build
Creating an optimized production build...
Compiled with warnings.

File sizes after gzip:

  499.72 kB  build/static/js/main.512bf5e1.js
  2.97 kB    build/static/css/main.1be55ed1.css

Done in 16.25s.

real	0m16.401s
user	0m29.148s
sys	0m1.121s

Running create-react-app build using gVisor container runtime runsc (with ptrace, the default):

[email protected]:~/project$ $ here --runtime=runsc node:16 bash -c 'time yarn build'
yarn run v1.22.18
$ react-scripts build
Creating an optimized production build...
Compiled with warnings.

File sizes after gzip:

  499.72 kB  build/static/js/main.512bf5e1.js
  2.97 kB    build/static/css/main.1be55ed1.css

Done in 37.74s.

real	0m38.103s
user	0m53.840s
sys	0m5.960s

The gVisor performance penalty increases the build time more than two times, from 16s to 38s. There is an additional 84% increase of userspace CPU time (29 vs 54 seconds). However, the actual userspace CPU time could even be a bit higher, since the runsc starts multiple processes that might not be accounted. The system time is expectedly many times higher than the baseline.

Bonus benchmark - running create-react-app build using gVisor container runtime runsc (with KVM):

[email protected]:~/project$ $ here --runtime=runsc node:16 bash -c 'time yarn build'
yarn run v1.22.18
$ react-scripts build
Creating an optimized production build...
Compiled with warnings.

File sizes after gzip:

  499.72 kB  build/static/js/main.512bf5e1.js
  2.97 kB    build/static/css/main.1be55ed1.css

Done in 43.62s.

real	0m44.454s
user	1m4.760s
sys	0m11.880s

For more information you can consult gVisor’s Performance Guide in the documentation.

An Example

With the default container runtime runC:

$ docker run --name isolated -it --rm -u "$(id -u):$(id -g)" -v "$PWD:/work" -w /work debian:10 bash
I have no [email protected]:/work$ id
uid=1000 gid=1000 groups=1000

# Then in a different terminal...
$ docker exec -it -u '0:0' isolated bash
[email protected]:/work# id
uid=0(root) gid=0(root) groups=0(root)
[email protected]:/work# apt update
Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:2 http://security.debian.org/debian-security buster/updates/main amd64 Packages [318 kB]
Get:3 http://deb.debian.org/debian buster InRelease [122 kB]
Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7911 kB]
21% [5 Packages 2856 B/7911 kB 0%]^C

With gVisor as a container runtime runsc:

$ docker run --name isolated -it --rm --runtime=runsc -u "$(id -u):$(id -g)" -v "$PWD:/work" -w /work debian:10 bash
I have no [email protected]:/work$ id
uid=1000 gid=1000 groups=1000

# Then in a different terminal...
$ docker exec -it -u '0:0' isolated bash
OCI runtime exec failed: executing processes for container: executing command "bash" in sandbox: failed to find initial working directory "/work": permission denied: unknown

$ docker exec -it -u '0:0' -w / isolated bash
[email protected]:/# id
uid=0(root) gid=0(root) groups=0(root)
[email protected]:/# apt update
E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (1: Operation not permitted)
E: seteuid 100 failed - seteuid (1: Operation not permitted)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
Reading package lists... Done
W: chown to _apt:root of directory /var/lib/apt/lists/partial failed - SetupAPTPartialDirectory (1: Operation not permitted)
W: chown to _apt:root of directory /var/lib/apt/lists/auxfiles failed - SetupAPTPartialDirectory (1: Operation not permitted)
E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (1: Operation not permitted)
E: seteuid 100 failed - seteuid (1: Operation not permitted)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
E: Method gave invalid 400 URI Failure message: Failed to setgroups - setgroups (1: Operation not permitted)
E: Method gave invalid 400 URI Failure message: Failed to setgroups - setgroups (1: Operation not permitted)
E: Method http has died unexpectedly!
E: Sub-process http returned an error code (112)