An account of using Debian in an Embedded Linux project

Michael’s retrospectives are always interesting – lots of good stuff in this one. In this one, he details the pain of building a production system on top of Debian (rPI). He started using Ansible to install everything, but that proved to not be a good fit. Ansible is great for small scale server deployments (I use it). But once you get to the production line, it is too slow. They are now in the process of switching to Debian packages, but it appears this is not simple either.

With Embedded Linux, there are many ways to do things. But one pattern I see over and over again is people not using Yocto, because it is too hard. I agree, it is hard – when things don’t work it can be nearly impossible. However, the alternatives don’t look all that attractive either. At least with Yocto, it was built from the ground up with goal of building custom Embedded Linux images. And when you are shipping products based on Embedded Linux, that is exactly what you are doing – there are no shortcuts. You can pay now, or pay later – take your choice. If you want to scale, you have to do the hard work of build/deployment systems that scale. Yocto has excellent tooling for building custom kernels and your custom applications. The Yoe distribution adds sane defaults, a simple but robust update mechanism, etc. Once you get a Yocto build set up, you can easily build production images for years with a single command. Yocto is high in initial development costs, but ongoing production costs are much lower.

One similarity I’ve noticed with both Ansible and Yocto (both written in Python) – when things are not working, the error messages are in the form of a long, hard to decipher Python stacktrace – on the scale of some heavily templatized C++ code that won’t compile. This probably indicates Python is probably not the best language to implement tooling like this. It may have been the best option at the time, but there are likely better ways to do things now.

Using Debian correctly has a similar learning curve to OpenEmbedded, in my experience. At first it is hard, but once you do it a bit it becomes second nature and you can easily use existing solutions as a set of exact steps on how to package and distribute other packages.

The one big advantage that OpenEmbedded has over Debian is the documentation. The Yocto Project documentation is quite extensive and useful, once you’re beyond the very early parts of the learning curve. Debian, however, is horribly documented for developers. The docs are distributed over a million different places, often the wiki is wrong, and because there’s so many different ways to make a Debian package there’s too many conflicting recommendations and examples.

My opinion is that if you’re making an embedded appliance which is not supposed to look like a general purpose computer, then a build system like OpenEmbedded is the right choice. But if you’re making a system which should somewhat look like a general purpose computer that you should use Debian (or RHEL) as your base because people who admin Linux boxes will be easily able to use your system. When people can’t Google for solutions and find a Stack Overflow post telling them exact steps on how to mess with their general purpose computer, they get mad. So you avoid this by either making it very clear that the system isn’t a general purpose computer or by simply making it work like people expect.

1 Like