GC-caused scheduler deadlock on RP2350

[herczegzsolt]

Hi!

If I run the following simple code on an RP2350 board, it'll lock-up after only a few (2 or 3) cycles:

package main

import (
	"runtime"
	"machine"


	"time"
)

var (
	uart = machine.Serial
)

func main() {
	uart.Configure(machine.UARTConfig{})
	time.Sleep(1000 * time.Millisecond)
	uart.Write([]byte("\n Start"))
	for {
		uart.Write([]byte("\n Gosched: "))
		runtime.Gosched()
		uart.Write([]byte(" GC: "))
		runtime.GC()
		uart.Write([]byte("done"))
	}
}

The code freezes on the runtime.Gosched() line. By adding println statements to the tinygo standard library, I was able to see that the cause of the stop is that the runtime.schedulerLock mutex is already locked when Gosched() is called. I don't know if anything is happening on the other core, but this locked-up state never resolves itself.

The issue does not seem to happen, if I don't call runtime.GC() - so the lockup is somehow related.

I understand that there's no need to call runtime.GC() in a tight loop like that, but this is only a very simple reproducer of the issue that I've found while doing some memory-heavy math.

Switching the scheduler to none -scheduler=none does workaround the issue, but it means I cannot use any goroutines.

[herczegzsolt]

I'd be happy to work on this and even potentially create a patch, but I'd need some pointers for how the tinygo internals supposed to work. The issue does not occur if I simply use tinygo run, so I'm stuck with on-device debugging for now.

[eliasnaur]

I don't know the answer to your questions, but -scheduler tasks should switch to the single core scheduler and possibly work around the issue while supporting goroutines.

[herczegzsolt]

Thanks! I've checked and -scheduler tasks is indeed a suitable workaround.

Seems like locks are working weirdly on RP2350 in general? #4988

[squidkeeper]

This is good to know, thanks. I find that the freeze just requires the GC to have to run; manipulating it is not necessary.
I have an RP2350 program that freezes after a few GC cycles using -scheduler=cores, but the more extreme test below hangs on the first cycle. -scheduler=tasks makes them happy. It looks like cores should not be the default for now. TinyGo 0.40.1.

package main

import (
	"runtime"
	"time"
)

var leaker *[]byte
var stats runtime.MemStats

func main() {
	for loops := 0; ; loops += 1 {
		foo := make([]byte, 50000)
		leaker = &foo

		runtime.ReadMemStats(&stats)
		println("heap=", stats.HeapInuse, "mallocs=", stats.Mallocs, "frees=", stats.Frees, "loops=", loops)
		time.Sleep(100 * time.Millisecond)
	}
}
// Happy: tinygo build -target=pico2-w -serial=uart -scheduler=tasks -o leaker.elf cmd/leaker/main.go
// Sad:   tinygo build -target=pico2-w -serial=uart -scheduler=cores -o leaker.elf cmd/leaker/main.go
// Happy keeps the GC busy but runs forever. Sad freezes before the GC has finished once.