The following discussion is interesting:
Includes a video:
Marmot is focused on eventual consistency:
Once changes are committed to DB (physically written) they are written to a change log table as well. Marmot picks up these changes and then proposes to rest of nodes over raft. There can be 1 and only 1 sequence of agreed upon order of applying changes. So if there is a race condition on same row being changed by two nodes, these operations will be sequenced due to raft and the last writer will win.
In eventually consistent systems, it seems critical that your data granularity be small. This way, the possibility of collisions is greatly reduced. Simple IoT does this by storing every field in a node as a separate point (row) in the database. A classical DB row may be too large for many applications. Something like Marmot could be used to scale two SIOT instances horizontally. However, there is a disadvantage to synchronizing at the DB level which is demonstrated by the above video – a page refresh is required to see the new changes. With Simple IoT, any data changes are required to go over NATS so any client can have real-time visibility into changes. It seems there is a lot of value to synchronizing data at the application or message bus level. I think NATs gets it right. However, it’s not clear yet how to best use NATs streaming in applications like Simple IoT where you need granular, bi-direction flow of data with CRDT properties. Event bus streaming platforms are optimized for data flowing in one direction (common in manufacturing, order processing, etc).
SIOT points could be stored in a NATS KV store, which can be automatically synchronized. However, it’s unclear how a KV store would be efficient in reassembling nodes without SQL like features where you can store the node ID in each point, and have that indexed. One alternative would be to manually manage an index, or store everything in memory.