The future direction of InfuxDB (Rust, Apache Arrow, OSS, Distributed, etc)

Very interesting article about the future direction of InfluxDB:

This article covers a lot of ground – its use of technologies like Apache Arrow is interesting. From the Apache Arrow FAQ:

What is the difference between Apache Arrow and Apache Parquet?

Parquet is not a “runtime in-memory format”; in general, file formats almost always have to be deserialized into some in-memory data structure for processing. We intend for Arrow to be that in-memory data structure.

Parquet is a storage format designed for maximum space efficiency, using advanced compression and encoding techniques. It is ideal when wanting to minimize disk usage while storing gigabytes of data, or perhaps more. This efficiency comes at the cost of relatively expensive reading into memory, as Parquet data cannot be directly operated on but must be decoded in large chunks.

Conversely, Arrow is an in-memory format meant for direct and efficient use for computational purposes. Arrow data is not compressed (or only lightly so, when using dictionary encoding) but laid out in natural format for the CPU, so that data can be accessed at arbitrary places at full speed.

Therefore, Arrow and Parquet complement each other and are commonly used together in applications. Storing your data on disk using Parquet and reading it into memory in the Arrow format will allow you to make the most of your computing hardware.

Also, an interesting comment on OSS in the article:

Community and open source aren’t about how much value your individual company manages to capture. Open source is about creating a cambrian explosion in software that creates value that far outstrips what a single vendor manages to monetize. Open source is not a zero sum game. A larger community and ecosystem brings more opportunity to all vendors.

We have a vision of InfluxDB becoming the basis for countless analytics, sensor, monitoring and data analysis projects and companies. The only way this might become a reality is if we come from a starting point that is extensible, adoptable and commercializable by anyone.

Community and open source aren’t about how much value your individual company manages to capture. Open source is about creating a cambrian explosion in software that creates value that far outstrips what a single vendor manages to monetize. Open source is not a zero sum game. A larger community and ecosystem brings more opportunity to all vendors.

This is growing the pie mindset, I like it