← writing

What a goroutine actually is

Rudra Narayan Panda ·

A goroutine looks like a function call with go in front of it. That is the entire syntax. What took me longer to understand was what the runtime does with it, and why you can start a hundred thousand of them on a laptop without the machine noticing.

This post is the mental model I wish I had started with: what a goroutine is made of, how the scheduler decides what runs, and what happens when one of them blocks. Most of it you can watch happen in the simulator further down. I ran the code and the numbers myself, on Go 1.27.4 on an Apple Silicon Mac. Your numbers will differ a little.

the smallest example

package main

import (
	"fmt"
	"sync"
)

func main() {
	var wg sync.WaitGroup
	for i := 1; i <= 3; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			fmt.Println("hello from goroutine", i)
		}()
	}
	wg.Wait()
}

Run it a few times and the lines come out in a different order. That is the first thing to accept: once you write go, you no longer control the order. The runtime decides.

The WaitGroup is there for a reason. When main returns, the program exits, and every other goroutine is killed on the spot, mid-sentence if it has to be. Without wg.Wait() you would probably see nothing printed at all.

Since Go 1.22 each loop iteration gets its own i. On older versions all three goroutines could print the same number, and you had to pass i in as an argument.

why not just use threads

An operating system thread is a kernel object. The kernel schedules it, and it gets a stack whose size is fixed when the thread is created. On Linux that is commonly 8 MB of address space, and other systems pick other defaults, usually somewhere from 1 to 8 MB. Most of that is not backed by real memory until it is touched, so it is not as bad as it sounds. But the size cannot change later, and every switch from one thread to another goes through the kernel.

That is fine for a few hundred threads. It gets uncomfortable at tens of thousands, and a server that wants one unit of work per connection or per request wants far more than that. Eight megabytes times a hundred thousand is 800 GB of address space, and most systems will refuse before you get there, on a process or memory limit.

Go's answer is to not give every unit of work a thread. It gives each one a goroutine, and runs many goroutines on a small number of threads.

a goroutine is a small struct and a small stack

Inside the runtime a goroutine is a struct, usually called g. It holds the things needed to pause the work and resume it later: where its stack is, the saved stack pointer and program counter, and its current status (runnable, running, waiting, and so on). It does not hold a thread.

The stack is the interesting part. It starts at 2 KB. Every function begins with a tiny check that asks whether there is enough stack left for this call. When there is not, the runtime allocates a stack twice the size, copies the old one across, fixes up the pointers into it, and carries on. The stack can grow like this up to 1 GB on a 64-bit system, and the garbage collector shrinks stacks that are mostly unused. Since Go 1.19 the starting size is not even always 2 KB, because the runtime adapts it to what your goroutines actually use.

One thread stack divided into 4,096 goroutine stacks A square representing 8 megabytes, filled with a 64 by 64 grid of small squares. Each small square is one 2 kilobyte goroutine stack. The top left square is highlighted.
  • The whole square is the 8 MB a typical Linux thread reserves for its stack.
  • Each small square is a 2 KB goroutine stack.
  • The circled orange one in the corner is a single goroutine.
  • There are 64 × 64 = 4,096 of them in the square.
Drawn to scale, using the 8 MB Linux default and the 2 KB starting stack.

Here is a program that starts 100,000 goroutines, parks them all on a channel, and asks the runtime how much stack they are using:

package main

import (
	"fmt"
	"runtime"
	"sync"
)

func main() {
	var before, after runtime.MemStats
	runtime.GC()
	runtime.ReadMemStats(&before)

	const n = 100_000
	var wg sync.WaitGroup
	stop := make(chan struct{})
	wg.Add(n)
	for i := 0; i < n; i++ {
		go func() {
			defer wg.Done()
			<-stop
		}()
	}

	runtime.Gosched()
	runtime.ReadMemStats(&after)
	fmt.Println("goroutines:", runtime.NumGoroutine())
	fmt.Printf("stack in use: %.0f MB\n", float64(after.StackInuse-before.StackInuse)/1e6)
	fmt.Printf("per goroutine: %d bytes\n", (after.StackInuse-before.StackInuse)/n)
	fmt.Printf("total sys: %.0f MB\n", float64(after.Sys-before.Sys)/1e6)
	close(stop)
	wg.Wait()
}

On my machine it printed:

goroutines: 100001
stack in use: 205 MB
per goroutine: 2049 bytes
total sys: 276 MB

About 2 KB each, 205 MB of stack for 100,000 of them. The 100,001 is the 100,000 plus main. The 276 MB is everything the process asked the OS for, which also covers the g structs and other bookkeeping. So the claim is real, and also not free. A million of these would be a couple of gigabytes.

G, M and P

The scheduler is easier to follow once you have three names straight. They come from the runtime source, and they are one letter each:

An M has to hold a P to run Go code. The number of Ps is GOMAXPROCS, which defaults to the number of CPU cores. So with 12 cores you have 12 Ps, and at most 12 goroutines executing Go code at the same instant. Everything else is waiting its turn.

The G, M and P model A global run queue feeds three Ps. Each P has a local run queue and a runnext slot. Each P is attached to one M, which is an OS thread running the current G. The threads sit on top of CPU cores. global run queue G9 G10 G11 P0 local queue G2 G3 runnext G4 P1 local queue G6 runnext empty P2 local queue empty runnext empty holds holds holds M0OS thread runs G1 M1OS thread runs G5 M2OS thread idle, looking for work the kernel schedules the threads onto CPU cores
One snapshot with 3 Ps. G1 and G5 are running. Dashed lines are goroutines moving from the global queue to a P.

Why have Ps at all, when you could just have goroutines and threads? Mostly so the run queues and caches belong to a P instead of one shared structure behind one lock. And when a thread gets stuck in the kernel, its P can be handed to a different thread, which the next sections show.

Each P keeps its own local run queue, plus a runnext slot for the one goroutine that should run next. There is also a global run queue for everything else. When a P needs work it looks in roughly this order: its own runnext and queue, then the global queue, then the network poller, and finally it steals from another P. Every 61st time around it checks the global queue first, so goroutines parked there cannot starve.

watch it run

Pick a scenario, then press Step to advance one tick, or Play to let it run. One tick is a unit of time, call it a millisecond. A running goroutine uses up one tick of its work per tick. The log underneath explains what the scheduler just did, newest line on top.

Chips show the goroutine number and what kind of goroutine it is. cpu just computes and finishes. chan computes, then blocks waiting on a channel, then computes again. syscall computes, then makes a blocking system call, then computes again. loop computes for a long time without ever yielding.

    The simulator simplifies things. Every goroutine is created on P0, ticks are all the same length, it steals from the busiest P instead of a random one, and a stuck syscall hands over its P immediately instead of after a short delay. It leaves out the network poller and the every-61st-tick rule. The shape of the behavior is right, the timing is not.

    starting one

    When you write go f(), the runtime takes a g (reusing a finished one when it can), gives it a small stack, and puts it in the runnext slot of the P you are on. If something was already in runnext, that goroutine gets bumped to the back of the local queue. That is why, in the first scenario, the goroutine created last is the one that runs first.

    The reason for runnext is communication. If goroutine A creates B, or wakes B by sending on a channel, the two probably share data that is still warm in the CPU cache, and A is likely to block soon. Running B straight away is usually the cheapest thing to do.

    blocking on a channel

    Try the channel block scenario. When a chan goroutine reaches its wait, it does not hold a thread hostage. The runtime marks the goroutine as waiting, files it in the channel's wait list, and the thread goes straight back to the scheduler to pick something else. No kernel is involved. When the other side finally sends, the waiting goroutine is marked runnable and goes into a runnext slot, normally on the P that woke it.

    The same thing happens for mutexes, time.Sleep, select and WaitGroup.Wait. Blocking a goroutine is cheap, which is the whole reason the style of one goroutine per task works.

    Network I/O follows the same idea with some help. Sockets are set to non-blocking, and the runtime asks the OS (epoll on Linux, kqueue on macOS) to say when they are ready. A goroutine that reads from a socket with no data is just parked until the poller reports data. No thread sits blocked inside read.

    blocking in a syscall

    Some calls cannot be parked this way. Regular file reads and writes, and most cgo calls, block the whole thread inside the kernel, and the goroutine is stuck on it. Try the blocking syscall scenario, which runs on a single P.

    The thread, M0, is stuck with its goroutine, so it cannot keep the P. A background thread in the runtime called sysmon notices a P that has been sitting in a syscall and hands it to another thread, creating a new one if none is idle. That is why you will often see more OS threads in a Go process than GOMAXPROCS. When the syscall finally returns, the goroutine needs a P again. It tries to grab an idle one, and if there is none it goes onto the global queue while its thread goes to sleep.

    work stealing

    In the work stealing scenario everything starts on P0, but there are three Ps. P1 and P2 begin with nothing local and nothing global, so they go looking. A P that steals takes about half of another P's local queue, runs one of those goroutines, and keeps the rest. Work spreads out without a central dispatcher, and the lock-free local queues stay cheap in the common case where nobody needs to steal.

    preemption

    In the preemption scenario one goroutine does a very long computation and never blocks. After 10 ticks the scheduler takes it off the P and puts it on the global queue, so the short goroutines are not stuck behind it. Real Go uses roughly 10 ms for this.

    This got better in Go 1.14. Before that, preemption was cooperative: it could only happen at function calls, so a tight loop with no calls could hold a P forever. Since 1.14 the runtime can interrupt a goroutine with a signal. You can see the difference with this program, which has one P and a goroutine that never yields:

    package main
    
    import (
    	"fmt"
    	"runtime"
    	"time"
    )
    
    func main() {
    	runtime.GOMAXPROCS(1)
    
    	go func() {
    		for {
    		}
    	}()
    
    	time.Sleep(100 * time.Millisecond)
    	fmt.Println("main got the only P back")
    }

    On a current Go it prints the message after about 100 ms. On Go 1.13 or earlier, this program could hang, because the spinning goroutine never reached a point where it could be stopped.

    looking at the real scheduler

    The runtime will tell you what it is doing. Set GODEBUG=schedtrace with an interval in milliseconds and it prints a line per interval. I ran 8 goroutines that each spin the CPU for 1.5 seconds:

    GODEBUG=schedtrace=500 ./spin
    
    SCHED 0ms: gomaxprocs=12 idleprocs=10 threads=3 spinningthreads=1 needspinning=0 idlethreads=0 runqueue=0 [ 0 0 0 0 0 0 0 0 0 0 0 0 ] schedticks=[ 0 0 0 0 0 0 0 0 0 0 0 0 ]
    SCHED 511ms: gomaxprocs=12 idleprocs=4 threads=11 spinningthreads=0 needspinning=0 idlethreads=2 runqueue=0 [ 0 0 0 0 0 0 0 0 0 0 0 0 ] schedticks=[ 23 0 0 11 15 22 22 11 7 22 23 25 ]

    Reading the second line: 12 Ps (gomaxprocs), 4 of them idle, so 8 busy, one per spinning goroutine. 11 OS threads exist. runqueue is the global queue, and the bracketed list is each P's local queue length. All zero, because there is a P for every goroutine. schedticks counts how many times each P has gone back to the scheduler.

    Two other tools are worth knowing. go tool trace shows a timeline of goroutines on Ps. And runtime.NumGoroutine(), or the goroutine profile at /debug/pprof/goroutine, tells you how many exist and where each one is stuck.

    where goroutines bite

    things to poke at