Long ago, I learned the power of cycle reboot testing for Embedded Linux systems:
http://bec-systems.com/site/942/running-a-reboot-cycle-test-shell-script-with-systemd
One of the most stressful things a system does is boot, and if you can boot 1000’s of times with no failures, that is good.
I’m learning similar things with testing complex Go apps – if you run your tests in a loop, you often uncover obscure bugs that don’t show up if you only run your tests once – especially in highly concurrent systems where there is lots going on.
Something like this is very useful:
#!/bin/bash
count=0
while true; do
echo "running test ..."
go clean -testcache
if ! go test -race ./client/; then
echo "test failed at count: $count"
break
fi
count=$((count + 1))
echo "count: $count"
done