Playground v3

2019-10-08

Overview

• This is talk about play.golang.org
• What it does
• History of implementations
• Latest incarnation

What does it does

• Lets users run untrusted code in browser
• Used by play.golang.org, tour.golang.org, godoc example snippets, etc.

Time

• time is faked, cacheable
• starts in Nov 2009, when Go was announced publicly

Sleep replays

• sleeps are emulated server-side (special runtime support)
• stdout/stderr writes prefixed with binary header containing virtual time
• sandbox parses output, simulates passage of time in browser
• doesn't tie up server resources

Fun: clear screen with \f

animation example

func main() {
   const x = `.oOo`
   for i := 0; i < 100; i++ {
       fmt.Printf("\f%c", x[i%len(x)])
       time.Sleep(200 * time.Millisecond)
   }
}

Fun: images

image example

History

Caveat: fake, retroactive version numbers follow for ease of discussion.

In reality, v0.0.0-20141209125517-dba9ba1953c6

Native Client (NaCl)

• July 2009, four months before Go
• GOOS=nacl, GOARCH={386,amd64p32,arm}
• restricted, verifiable subset of {386,amd64,arm}
• some had ambitions to add it to Chrome, web
• most people opposed
• "Portable NativeClient" (PNaCl) later proposed, opposed
• GOARCH=amd64p32: amd64, but 32-bit pointers

Playground ~v1

• App Engine frontend, in Python, made RPC calls to pass source to:
• C++ Borg backend, compiled Go to NaCl, ran under ptrace syscall filter
• standard two layers of paranoia; either sufficient in theory alone, ptrace layer logs/alerts hard on any violation

Playground ~v1.1

• GAE frontend rewritten from Python to Go
• backend still C++ running NaCl on Borg

Playground ~v2

• move to App Engine Flex, where we can do more stuff
• unify frontend + backend
• run Native Client sel_ldr (NaCl runner binary) ourselves
• no more C++
• no more Borg

2017

• May 30, 2017: Portable Native Client (PNaCl) deprecated
• Go never even implemented PNaCl (fortunately, in retrospect)
• Go still implements the doubly deprecated arch-specific NaCl variants

2018, 2019...

• WebAssembly rise continues
• NaCl looking increasingly odd, high maintenance cost relative to its worth
• maintenance: builder resources
• maintenance: hoping runtime faketime patches still apply at every release
• maintenance: amd64p32
• but play.golang.org still depends on it

How to sandbox play.golang.org without NaCl?

Plan: Playground v3

• delete nacl/*
• move Nacl's funky faketime support to linux/amd64 at least...
• +build faketime (in master for Go 1.14; thanks, Austin!)
• run otherwise regular linux/amd64 binaries in some sandbox

gVisor

• gVisor is a mostly complete re-implementation of the Linux system call interface, in Go
• runs under various modes ("platforms"): KVM, ptrace
• everything is hookable: ~FUSE on steroids
• performance pretty good, especially with KVM, not that we really need performance
• was Google-internal, open source May 2018

Who uses gVisor?

• App Engine 2nd gen runtimes (all new ones)
• Cloud Run
• Cloud Functions
• GKE Sandbox (opt-in hardened containers)
• ...

How to use gVisor?

• tricky, ~unsupported to use directly
• comes with a "runsc" wrapper command that team recommends

runsc woes

• runsc ("run secure container") is an OCI "runc"-compatible frontend to gVisor
• runc interface itself is tricky
• kinda gross to use directly
• syzkaller fuzzer does, though, so Go driver code exists
• but ...

root

• runsc requires root
• without networking and ptrace instead of kvm, it almost doesn't require root, but gvisor team isn't quite done with rootless
• App Engine, etc doesn't give us real root (uid=0, but fake, not enough caps, under gVisor itself)
• gvisor can't run nested under gvisor (can't run on Cloud Run, for example)

Docker

• But Docker supports runsc easily:

docker run --runtime=runsc ...

• So we can just run a dynamically scaled pool of VMs running Docker + a backend HTTP server driving Docker, with a load balancer in front of the pool of VMs.

v3 architecture

• existing HTTP frontend takes source, compiles binary (for now)
• HTTP PUTs compiled binary to backend HTTP server

v3 architecture

On each VM

• one privileged container that speaks HTTP
• has bind-mounted /var/run/docker.dock, to control its parent docker
• creates sibling-but-gvisor'ed containers for each play
• HTTP server reads PUT'ed binary, pipes into secure container, reads stdout/stderr response & exit status
• replies over HTTP to frontend

VM view

Transition plan

Today

• Frontend compiles to nacl/amd64p32, runs binary

Today+1

• Frontend compiles to nacl/amd64p32, runs binary
• opt-in support for using new backend (in either linux/amd64 or nacl/amd64p32 mode)
• temporary support in new backend to understand how to wrap nacl with sel_ldr binary

Today+2:

• Frontend compiles to nacl/amd64p2, PUTs all requests to new backend
• gets us operational experience with backend before Go 1.14 release day chaos

Go 1.14 release day

• flip a bit in frontend that forces linux/amd64
• NaCl no longer allowed
• can delete all its playground code (minimal)

Future possibilities

• stop shipping compiled binaries around the network?
• ... minimal win, might not be worth it
• unify compilation & running in the backend?
• 0, 1, or 2 sandbox services to update per Go release?
• 2 is too many
• 1 is today, acceptable
• 0 might be too risky

Alternatives

Cloud Run?

• Almost perfect
• but doesn't let you disable network (yet? Cloud Functions has VPC connectors, can route traffic to Linux VM that drops packets)
• if root-less runsc happens, can run in ptrace mode under Cloud Run

GKE Sandbox?

• not obviously any simpler, faster than this setup
• but might be alternative later
• worry about node pool elasticity latency? COS boots quickly.

WebAssembly?

• large binaries for users
• still experimental support, but improving
• use WebAssembly for x/tools/cmd/present?
• ... because this Docker runsc contraption is too complicated for casual users giving talks on laptops
• or just run unsandboxes in cmd/present

Thank you