Problem with Go binary portability: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found

Software
1

So today I was building Go binaries on a Arch Linux machine and deploying to an Ubuntu server, and go the following error when trying to run them:

/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found

This was odd because I’ve never had any problems with Go binary portability in the past. On the Gopher slack, it was suggested:

cgo is used by default if you use net or os/user

The go binary package from golang.org produced working binaries using the same combination of machines. Another solution suggested on slack was:

just build your stuff properly CGO_ENABLED=0 go build ...

Disabling CGO worked.

Digging a little more, before I disabled CGO, the binaries had the following dependencies:

[cbrake@mars go]$ ldd is-portal 
        linux-vdso.so.1 (0x00007ffe5dcd6000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f082b446000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f082b27d000)
        /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f082b4bb000)

After enabling CGO, there are no dynamically linked libraries:

[cbrake@mars go]$ ldd is-portal 
        not a dynamic executable

After asking more questions, the reasons for this differing behavior was given:

because name resolution is complex and people sometimes want compatible behaviour, see net package - net - Go Packages

And then from that page:

By default the pure Go resolver is used, because a blocked DNS request consumes only a goroutine, while a blocked C call consumes an operating system thread. When cgo is available, the cgo-based resolver is used instead under a variety of conditions: on systems that do not let programs make direct DNS requests (OS X), when the LOCALDOMAIN environment variable is present (even if empty), when the RES_OPTIONS or HOSTALIASES environment variable is non-empty, when the ASR_CONFIG environment variable is non-empty (OpenBSD only), when /etc/resolv.conf or /etc/nsswitch.conf specify the use of features that the Go resolver does not implement, and when the name being looked up ends in .local or is an mDNS name.

So the bottom line – if you want to make sure you have portable Go binaries, disable CGO.

Thanks for your help Sean Liao – I learned something today!

2

right, you ran into this issue because your build machine (arch) is running newer glibc compared to your deployment host ( ubuntu ), usually, if it was otherway around it would have worked without any issues.

3

This is interesting, especially with other mDNS discussions. So to get mDNS functionality in Go, you have to use CGO? It seems that way from the doc you linked to.

I feel like this is similar to the debates about browsers now often defaulting to using DNS over HTTPS rather than using the system resolver. As much as DNS can benefit from these types of optimizations (both Go’s and the browsers’) it feels like we’re starting to really get away from relying on the OS to provide the core tools for networking within modern computing. I don’t particularly like this, I think because it makes me fear that we’ll end up with a million poorly implemented solutions rather than a handful of very well implemented and tested solutions at the system level.

4

I ran a quick test:

The behavior is different based on when CGO is enabled (uses libc resolver) and CGO disabled (uses go resolver):

[cbrake@ceres go]$ CGO_ENABLED=1 go run mdns-test.go
MDNS test
IP:  fe80::7a02:43d1:6997:fe3


[cbrake@ceres go]$ CGO_ENABLED=0 go run mdns-test.go
MDNS test
IP:  10.0.0.118
IP:  fe80::7a02:43d1:6997:fe3

Can we assume from the above that Go can resolve mDNS addresses?

5

timezone data/functions may be another area. It seems there are other examples of this in OSS:

  • clang vs GCC
  • Musl vs GLIBC

Choice like this has its downsides as we are continually fixing problems where stuff will build in case, but not the other. But there are also advantages of starting fresh at times. @khem probably has an opinion on this as he spends much of his time in system software land.

Being able to easily cross compile Go and deploy it to any OS is pretty neat. For example, I can build all of the following on my Arch box (has latest libc) and it will run anywhere (assuming I disable CGO):

image
image733×896 75.6 KB

For getting stuff done, I’ve never experienced anything like Go before. And some claim the new Go stuff is more secure than the old and proven C stuff.

But then we have the other perspective we discussed before.

Interesting to think about – especially as we make architecture decisions and chose technologies and approaches.

6

I think yes! Although it’s interesting that you get both IPv4 and IPv6 in one case but only IPv6 in the other.

You’ve gotten me re-interested in learning Go :slight_smile:

7

And related, today I was pointed at GitHub - microsoft/go-crypto-openssl: Go crypto backend for Linux using OpenSSL for situations where you are required to use OpenSSL but you want to write in Go. You need CGO but that seems OK.

I’m not confident that it will show through readelf-like tools that there’s any dependency on the openssl libs when using this mechanism, which is kind of annoying as it will make automatic dependency checking harder, but at least it can work at runtime if the right things are present.

8

@bradfa you have interesting point and I partially agree with you. However, I think that open source paradigm thrived because it offers open innovation and may the best implementation win philosophy, everything is replaceable and this has an interesting side effect that good stuff tends to stick and other stuff has shorter self life. So I usually do not worry about this, on the contrary I like that someone is trying to provide an alternative and if someone has done a good job, it will stick and perhaps even replace long
used components and thats good as our computing needs as well as equipments change, its always good to keep rolling.