Fileservers are a technology we used a lot in the past – NFS, Samba, etc. If we needed to share files with remote people, we might use a VPN or a cloud service like Dropbox. These all worked pretty well, but there are a few pain points:
- you have to be connected to the same network as the file server to use the files. So you had to copy files to your computer to work on a plane, copy them back, etc.
- users and permissions are always a pain with a fileserver. There are often problems where people “don’t have access” to a new file, etc.
- cloud file sharing requires a round trip through the cloud, which in some cases is not necessary and slow
Other technologies of this era focused on the centralized server concept, like SVN. Then came Git, and we learned we could work in a completely disconnected, distributed fashion. So we started using Git for more and more, and there are even extensions for Git that make it work well with large files. But for some things like photos, music, etc, Git-lfs is overkill.
Then Syncthing came along. Syncthing automatically synchronizes directories between two or more systems. You can have multiple shares (or directories). After experimenting a bit, I concluded that sharing individual directories on the traditional fileserver was the way to go. This way a group of files could be shared only with the people/devices that needed those files. Gone was the need to worry about access or permissions – if something was shared, then that device/person had access. Files are replicated, so you could work offline, and things were sync’d up the next time you were on the network.
My password database gets shared with all my devices (workstation, notebook, phone, and e-reader) and is always up to date. Books get shared to e-reader. music gets shared to phone, etc.
It works great.
However, what about cloud backups or remote people? I’ve run various backup programs in the past: plain rsync, restic, autorestic, and rsnapshot. Some of these tools have very nice features like keeping multiple snapshots of the past. But they all have significant overhead to set up and operate. So, now I just use plain rsync, and Dropbox for a few things. However, what if we could “syncthing” files to the cloud? This would require a VPN, which I’ve typically eschewed until I met Wireguard and Tailscale – now VPNs are easy, fast, and reliable. So if you had a VPN connection to a cloud server with some storage, you could easily sync your files there, and essentially have Dropbox.
Linode charges ~ $1/10GB per month for block storage. Dropbox’s cheapest plan charges $9.99 for 2TB. The equivalent cost in Linode block storage is over $200/mo, so if you need a lot of storage, then Dropbox is clearly a better deal. However, with syncthing, you don’t have to share everything – you can choose what you send to the cloud, and perhaps it is unnecessary to back up your TB’s of video to the cloud? Another option would be to put several large drives at two locations and sync large files to those for backups?
Syncthing is not as convenient as Dropbox, but it is a lot more flexible and in many cases faster for an internal team/family/devices in that files don’t have to make a round trip through the cloud. There may also be security concerns where you don’t want some files residing on public cloud services.
One thing that seems clear in thinking through this – I need to be doing more with Wireguard/Tailscale. Perhaps Syncthing + Wireguard could have a place in Ops for deployment. Ansible works pretty well, but it is fairly slow as it runs everything over ssh connections. So instead of deploying to the server, we deploy to a local syncthing share that gets sync’d to the server. There would need to be a little more infrastructure to detect when things change and restart services.
This recent article by DHH regarding CI is interesting and perhaps related:
Centralized CI/CD services like Jenkins/Github/Gitea were a step forward, but perhaps the next step is pushing this back to the developer’s workstation where we have so many cheap and powerful resources, and then we deploy from the workstation, not from the Git hosting service.