What a goroutine actually is
Rudra Narayan Panda ·
A goroutine looks like a function call with go in front of it. That is the entire syntax. What took me longer to understand was what the runtime does with it, and why you can start a hundred thousand of them on a laptop without the machine noticing.
This post is the mental model I wish I had started with: what a goroutine is made of, how the scheduler decides what runs, and what happens when one of them blocks. Most of it you can watch happen in the simulator further down. I ran the code and the numbers myself, on Go 1.27.4 on an Apple Silicon Mac. Your numbers will differ a little.
the smallest example
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
for i := 1; i <= 3; i++ {
wg.Add(1)
go func() {
defer wg.Done()
fmt.Println("hello from goroutine", i)
}()
}
wg.Wait()
}
Run it a few times and the lines come out in a different order. That is the first thing to accept: once you write go, you no longer control the order. The runtime decides.
The WaitGroup is there for a reason. When main returns, the program exits, and every other goroutine is killed on the spot, mid-sentence if it has to be. Without wg.Wait() you would probably see nothing printed at all.
Since Go 1.22 each loop iteration gets its own i. On older versions all three goroutines could print the same number, and you had to pass i in as an argument.
why not just use threads
An operating system thread is a kernel object. The kernel schedules it, and it gets a stack whose size is fixed when the thread is created. On Linux that is commonly 8 MB of address space, and other systems pick other defaults, usually somewhere from 1 to 8 MB. Most of that is not backed by real memory until it is touched, so it is not as bad as it sounds. But the size cannot change later, and every switch from one thread to another goes through the kernel.
That is fine for a few hundred threads. It gets uncomfortable at tens of thousands, and a server that wants one unit of work per connection or per request wants far more than that. Eight megabytes times a hundred thousand is 800 GB of address space, and most systems will refuse before you get there, on a process or memory limit.
Go's answer is to not give every unit of work a thread. It gives each one a goroutine, and runs many goroutines on a small number of threads.
a goroutine is a small struct and a small stack
Inside the runtime a goroutine is a struct, usually called g. It holds the things needed to pause the work and resume it later: where its stack is, the saved stack pointer and program counter, and its current status (runnable, running, waiting, and so on). It does not hold a thread.
The stack is the interesting part. It starts at 2 KB. Every function begins with a tiny check that asks whether there is enough stack left for this call. When there is not, the runtime allocates a stack twice the size, copies the old one across, fixes up the pointers into it, and carries on. The stack can grow like this up to 1 GB on a 64-bit system, and the garbage collector shrinks stacks that are mostly unused. Since Go 1.19 the starting size is not even always 2 KB, because the runtime adapts it to what your goroutines actually use.
- The whole square is the 8 MB a typical Linux thread reserves for its stack.
- Each small square is a 2 KB goroutine stack.
- The circled orange one in the corner is a single goroutine.
- There are 64 × 64 = 4,096 of them in the square.
Here is a program that starts 100,000 goroutines, parks them all on a channel, and asks the runtime how much stack they are using:
package main
import (
"fmt"
"runtime"
"sync"
)
func main() {
var before, after runtime.MemStats
runtime.GC()
runtime.ReadMemStats(&before)
const n = 100_000
var wg sync.WaitGroup
stop := make(chan struct{})
wg.Add(n)
for i := 0; i < n; i++ {
go func() {
defer wg.Done()
<-stop
}()
}
runtime.Gosched()
runtime.ReadMemStats(&after)
fmt.Println("goroutines:", runtime.NumGoroutine())
fmt.Printf("stack in use: %.0f MB\n", float64(after.StackInuse-before.StackInuse)/1e6)
fmt.Printf("per goroutine: %d bytes\n", (after.StackInuse-before.StackInuse)/n)
fmt.Printf("total sys: %.0f MB\n", float64(after.Sys-before.Sys)/1e6)
close(stop)
wg.Wait()
}
On my machine it printed:
goroutines: 100001
stack in use: 205 MB
per goroutine: 2049 bytes
total sys: 276 MB
About 2 KB each, 205 MB of stack for 100,000 of them. The 100,001 is the 100,000 plus main. The 276 MB is everything the process asked the OS for, which also covers the g structs and other bookkeeping. So the claim is real, and also not free. A million of these would be a couple of gigabytes.
G, M and P
The scheduler is easier to follow once you have three names straight. They come from the runtime source, and they are one letter each:
- G is a goroutine.
- M is a machine, which means an OS thread. It does the actual running.
- P is a processor, which is a scheduling context. It is not a CPU. It is a token that says "this thread is allowed to run Go code right now", and it carries the run queue and some per-P memory caches with it.
An M has to hold a P to run Go code. The number of Ps is GOMAXPROCS, which defaults to the number of CPU cores. So with 12 cores you have 12 Ps, and at most 12 goroutines executing Go code at the same instant. Everything else is waiting its turn.
Why have Ps at all, when you could just have goroutines and threads? Mostly so the run queues and caches belong to a P instead of one shared structure behind one lock. And when a thread gets stuck in the kernel, its P can be handed to a different thread, which the next sections show.
Each P keeps its own local run queue, plus a runnext slot for the one goroutine that should run next. There is also a global run queue for everything else. When a P needs work it looks in roughly this order: its own runnext and queue, then the global queue, then the network poller, and finally it steals from another P. Every 61st time around it checks the global queue first, so goroutines parked there cannot starve.
watch it run
Pick a scenario, then press Step to advance one tick, or Play to let it run. One tick is a unit of time, call it a millisecond. A running goroutine uses up one tick of its work per tick. The log underneath explains what the scheduler just did, newest line on top.
Chips show the goroutine number and what kind of goroutine it is. cpu just computes and finishes. chan computes, then blocks waiting on a channel, then computes again. syscall computes, then makes a blocking system call, then computes again. loop computes for a long time without ever yielding.
The simulator simplifies things. Every goroutine is created on P0, ticks are all the same length, it steals from the busiest P instead of a random one, and a stuck syscall hands over its P immediately instead of after a short delay. It leaves out the network poller and the every-61st-tick rule. The shape of the behavior is right, the timing is not.
starting one
When you write go f(), the runtime takes a g (reusing a finished one when it can), gives it a small stack, and puts it in the runnext slot of the P you are on. If something was already in runnext, that goroutine gets bumped to the back of the local queue. That is why, in the first scenario, the goroutine created last is the one that runs first.
The reason for runnext is communication. If goroutine A creates B, or wakes B by sending on a channel, the two probably share data that is still warm in the CPU cache, and A is likely to block soon. Running B straight away is usually the cheapest thing to do.
blocking on a channel
Try the channel block scenario. When a chan goroutine reaches its wait, it does not hold a thread hostage. The runtime marks the goroutine as waiting, files it in the channel's wait list, and the thread goes straight back to the scheduler to pick something else. No kernel is involved. When the other side finally sends, the waiting goroutine is marked runnable and goes into a runnext slot, normally on the P that woke it.
The same thing happens for mutexes, time.Sleep, select and WaitGroup.Wait. Blocking a goroutine is cheap, which is the whole reason the style of one goroutine per task works.
Network I/O follows the same idea with some help. Sockets are set to non-blocking, and the runtime asks the OS (epoll on Linux, kqueue on macOS) to say when they are ready. A goroutine that reads from a socket with no data is just parked until the poller reports data. No thread sits blocked inside read.
blocking in a syscall
Some calls cannot be parked this way. Regular file reads and writes, and most cgo calls, block the whole thread inside the kernel, and the goroutine is stuck on it. Try the blocking syscall scenario, which runs on a single P.
The thread, M0, is stuck with its goroutine, so it cannot keep the P. A background thread in the runtime called sysmon notices a P that has been sitting in a syscall and hands it to another thread, creating a new one if none is idle. That is why you will often see more OS threads in a Go process than GOMAXPROCS. When the syscall finally returns, the goroutine needs a P again. It tries to grab an idle one, and if there is none it goes onto the global queue while its thread goes to sleep.
work stealing
In the work stealing scenario everything starts on P0, but there are three Ps. P1 and P2 begin with nothing local and nothing global, so they go looking. A P that steals takes about half of another P's local queue, runs one of those goroutines, and keeps the rest. Work spreads out without a central dispatcher, and the lock-free local queues stay cheap in the common case where nobody needs to steal.
preemption
In the preemption scenario one goroutine does a very long computation and never blocks. After 10 ticks the scheduler takes it off the P and puts it on the global queue, so the short goroutines are not stuck behind it. Real Go uses roughly 10 ms for this.
This got better in Go 1.14. Before that, preemption was cooperative: it could only happen at function calls, so a tight loop with no calls could hold a P forever. Since 1.14 the runtime can interrupt a goroutine with a signal. You can see the difference with this program, which has one P and a goroutine that never yields:
package main
import (
"fmt"
"runtime"
"time"
)
func main() {
runtime.GOMAXPROCS(1)
go func() {
for {
}
}()
time.Sleep(100 * time.Millisecond)
fmt.Println("main got the only P back")
}
On a current Go it prints the message after about 100 ms. On Go 1.13 or earlier, this program could hang, because the spinning goroutine never reached a point where it could be stopped.
looking at the real scheduler
The runtime will tell you what it is doing. Set GODEBUG=schedtrace with an interval in milliseconds and it prints a line per interval. I ran 8 goroutines that each spin the CPU for 1.5 seconds:
GODEBUG=schedtrace=500 ./spin
SCHED 0ms: gomaxprocs=12 idleprocs=10 threads=3 spinningthreads=1 needspinning=0 idlethreads=0 runqueue=0 [ 0 0 0 0 0 0 0 0 0 0 0 0 ] schedticks=[ 0 0 0 0 0 0 0 0 0 0 0 0 ]
SCHED 511ms: gomaxprocs=12 idleprocs=4 threads=11 spinningthreads=0 needspinning=0 idlethreads=2 runqueue=0 [ 0 0 0 0 0 0 0 0 0 0 0 0 ] schedticks=[ 23 0 0 11 15 22 22 11 7 22 23 25 ]
Reading the second line: 12 Ps (gomaxprocs), 4 of them idle, so 8 busy, one per spinning goroutine. 11 OS threads exist. runqueue is the global queue, and the bracketed list is each P's local queue length. All zero, because there is a P for every goroutine. schedticks counts how many times each P has gone back to the scheduler.
Two other tools are worth knowing. go tool trace shows a timeline of goroutines on Ps. And runtime.NumGoroutine(), or the goroutine profile at /debug/pprof/goroutine, tells you how many exist and where each one is stuck.
where goroutines bite
- A goroutine that blocks forever is never collected. The garbage collector does not clean up a goroutine just because nobody can reach it any more. This leaks 1,000 of them:
Callingfunc leak() { ch := make(chan int) go func() { ch <- 1 }() // nobody ever receives }leak()1,000 times printedgoroutines: 1001. Giving the channel a buffer,make(chan int, 1), lets the send finish, and the count drops to 1. In real code the usual fix is acontextor a channel that tells the goroutine when to stop. - Cheap is not free. 100,000 parked goroutines cost about 200 MB. Goroutines that hold connections, buffers or files cost whatever those cost. Put a limit on how many you start. A buffered channel used as a semaphore works, and so does
SetLimitonerrgroupfromgolang.org/x/sync. - Shared memory still needs protecting. Two goroutines writing the same variable is a data race, however light they are. Run your tests with
go test -race. - The program does not wait for them. When
mainreturns, they are gone. Use aWaitGroupor a channel.
things to poke at
- In the simulator, spawn a few loop goroutines on the preemption scenario and watch how the global queue builds up.
- Run the 100,000 goroutine program with 1,000,000 and see how the memory scales.
- Run any of your own programs with
GODEBUG=schedtrace=1000and check whether the local queues ever fill up. - Read the long comment at the top of
src/runtime/proc.goin the Go source. It is the scheduler's own explanation of the design.