Go: 90% Perfect, 100% of the time.

"GoCon Tokyo"

31 May 2014

Hello!

"60% of the time, it works every time...."

"That makes no sense"

Starting a new project, picking a language...

Disclaimer

• Personal experience only
• My opinions
• Fake, fun numbers (e.g. "90% Perfect, 100% of the time.")
• Maybe it's 91%.
• All languages are both wonderful and terrible
• <3

2010

Idea: Camlistore

• All my life's data from everywhere,
• Import, backup, serve, browse, search, share, index
• Designed to last 100+ years
• Don't trust any company
• In 2010: an idea
• In 2014: busy, fun, active project

Which language to use for Camlistore?

Before Go

• Starting a new project from 1994 ~ 2010...
• Perl or C?
• Rarely but occasionally Java.

Perl: The Good

• Fun
• Expressive
• Get things done, quickly

Python, Ruby, Javascript: The Good

• Same

Perl: The Bad

• Not fast like C
• Sometimes need to write in C
• ... or write in XS (Perl-C glue language)
• Perl-C, C-Perl calls are slow
• Single-threaded, single-core, callback hell

Python, Ruby, Javascript: The Bad

• Same, but even slower :)
• JavaScript is faster now, but not as fast as C

Summary

• Perlythonubyscript: slow for computer, fast & fun for human
• C: slow for human, fast for computer
• C++: little more fun than C, but more complexity
• Java: tedious (verbose), slow start-up, easy to find programmers
• PHP: if somebody's paying you enough, maybe

Fun vs. fast

Maybe mix of two languages?

• Fun, slow parts in Perlythonubyscript?
• Important, fast parts in C?

But...

• You'll want to share code.
• Calling from C to X or X to C is slow.
• The C-X glue language is the least fun language
• You'll be lazy, and write in the fun language
• Or be lazy and waste time, and write too much stuff in C++ (like Google)

Server? Threads or events?

Threads

• C, C++, Java, ...
• Threads? Heavy. Lots of memory per thread for stacks. Be careful! Executor pool = new ScheduledThreadPoolExecutor(/* ???? guess a number */);

Events

• Perlythonubyscript: events. no choice. Callback hell.
• C, C++: libevent, etc. Callback hell.

• C#: await async await async await async await async noise

Concurrency and performance

It's really hard to pick a language!

2010. Let's try Go!

Go: 90% Perfect, 100% of the time.

Go: 90% Perfect, 100% of the time.

• Go is very good at lots of things
• High-level code when you want
• Low-level code when you want
• Start high, profile, go low-level when needed
• Static typing without too much keyboard typing

Before Go

After Go

Go's Concurrency

• Built-in! Lightweight threads, no callback hell
• Readable, top-down code
• So easy to write servers

for {
  conn, err := listener.Accept()
  // check err
  go serve(conn)
}

• Goroutine-per-request can scale

Concurrency before Go

After Go

Go is showing up for all sorts of tasks

Web frameworks

• Typically a job for scripting languages: Node.js, Python, Ruby, Perl, PHP, etc.
• net/http in the standard library
• Other http frameworks on top: gorilla, martini, Revel etc...
• No need for nginx: standard library's http server scales

Image processing

• no libpng, imagemagick, etc
• pure Go PNG, JPEG, GIF encoding & decoding in the standard library
• slower, but safer
• getting faster

Crypto

• no OpenSSL, no GnuTLS
• pure Go AES, TLS (SSL), OpenPGP, etc
• slower, but safer
• getting faster

Replacing shell scripts

• Instead of little Perl & shell scripts, I now write in Go
• os/exec package & goroutines makes subprocess management easier than shell

Camlistore

• web handlers
• using third-party web APIs
• image thumbnails: decode, resize, encode
• EXIF, ID3 parsing
• database drivers
• crypto, image decoding/resizing/encoding
• search, indexing: tight memory layout control
• WebSockets, HTTPS
• all pure Go!

Controlling flying drones

www.godrone.io

• "Isn’t Go unsuitable for real-time applications like this?"
• "However, for all practical purposes the GC just needs to keep up with the stabilization loop which runs at 200 Hz. This means that GC pauses below 5ms have no impact on performance. Longer pauses will degrade stabilization performance, but the tolerance threshold may be up to a second ..." (small heap)

Emulators

Go: The Language For Emulators

Disassembler, linker, compiler.

• llgo compiler (Go -> LLVM IR -> ...)
• Go 1.3's disassembler
• Go 1.4's linker
• Go 1.4 or Go 1.5's compiler

Mobile

• Go runs on ARM
• minux's iOS port of Go
• Camlistore child process, goandroid

Audio synthesis

• https://github.com/nf/sigourney

Cloud infrastructure

Go: the emerging language of cloud infrastructure

• Docker, Packer
• CoreOS’s etcd and fleet
• Ubuntu Juju, Mozilla Heka, Apcera’s NATS, gnatsd

Load balancers & servers

• YouTube's vitess: MySQL
• net/http/httputil.ReverseProxy
• dl.google.com
• etc

Raspberry Pi GPIO

github.com/davecheney/gpio

github.com/stianeikeland/go-rpio

github.com/luismesas/goPi

What other language is used for all these?

• web apps
• scripts
• system administration
• image processing
• load balancers, servers
• crypto
• hardware

Go!

And even better...

Go built-in tools

• testing
• benchmarking
• profiling (CPU, memory, blocking)
• huge standard library
• "go get"
• godoc
• gofmt, goimports
• race detector
• static binaries

So why isn't Go perfect?

Generics

• No generics
• It sucks sometimes, but rarely
• Use maps, slices, interfaces
• Occasionally a gross interface{}
• No algorithms in stdlib
• No great proposal yet

No generics (Simon Peyton Jones: "Haskell is useless")

Data races can happen

• shared, mutable state is possible
• use channels
• be careful
• conventions and documentation help
• runtime race detector helps too $ go test -race

Code generation

• The default Go compiler often generates pretty dumb code
• Especially on ARM
• But getting better
• In Go 1.4, 1.5+: Go compiler in Go, refactor, add SSA, more optimizations

gccgo

• gccgo generates very good code
• ... but lacks escape analysis: kills performance with many small allocs + garbage
• ... GC isn't precise. Bad for 32-bit.

Compiling to JavaScript isn't yet great

• Would be nice to write frontend & backend in the same language, share code
• go/types + go/ssa enables many things:

TARDIS Go Compiler

Limited Mobile Support

• No built-in Android support (but goandroid, child processes...)
• No built-in iOS support (but unmerged port...)

Embedding Go in C/C++/Java/etc

• Can't embed Go in other languages, environments.
• e.g. Android, iOS, C/C++ programs
• problems with memory setup, GC, signals, ...
• need embedding API design
• In Go 1.4?

Shared libraries

• no shared libraries
• can't load Go code at runtime

Garbage collector

• Pauses: faster each release, good people working on it.
• Precision: heap in 1.2, most stacks in 1.3, more in 1.4.
• 4 versions of memcached: Perl, C, C++, Go

Hot stack splits

• Almost entirely fixed in Go 1.3's contiguous stacks
• Rest should be in Go 1.4.

But...

• Problems are fixable.
• It all keeps getting better with each release.

Go for everything

• Since mid-2010, I prefer 90%-perfect Go for all my hacking:

• Go is flexible, fun, readable
• Go is fast (for computers & humans)
• Go is good for working with others
• Go for everything!

Thank you