NATS notes/tips

As I am digging more into advanced features of NATS (auth, kv store, Jetstream, NGS, etc), I thought it might be useful to put together a series of notes on how to do various things. Through this series, I will be using the Synadia NGS hosted offering as well as a local server.

Feel free to add your own entries of NATS tips to this thread.

Install the tools

  • nats: NATS Utility, server and JetStream admin
  • nsc: manage operators, accounts, users and permissions
  • ngs: used to manage your Synadia NATS Global service account
  • kv: generate NKeys
  • nats-server: the server itself

Arch Linux has nats-server and natscli-bin packages in the AUR, which I installed with yay.

nats context

The nats utility has a useful feature called contexts. This allows you to set up various contexts and quickly switch between them. Lets set a context for the local server:

nats context save local
nats context select local

This creates and selects a default context which connects to your a local server.

more /home/cbrake/.config/nats/context/local.json
{
  "description": "",
  "url": "nats://127.0.0.1:4222",
  "token": "",
  "user": "",
  "password": "",
  "creds": "",
  "nkey": "",
  "cert": "",
  "key": "",
  "ca": "",
  "nsc": "",
  "jetstream_domain": "",
  "jetstream_api_prefix": "",
  "jetstream_event_prefix": ""
}

Basic operations with local server

nats-server 
[91720] 2021/10/22 09:17:19.187352 [INF] Starting nats-server
[91720] 2021/10/22 09:17:19.187403 [INF]   Version:  2.6.1
[91720] 2021/10/22 09:17:19.187407 [INF]   Git:      [c91f0fe]
[91720] 2021/10/22 09:17:19.187414 [INF]   Name:     NCH4L4TAGXQK3QXTUTZXEPZJHOOGZRJVGPPMDQYFFKQ2EJHNUO2BMSMS
[91720] 2021/10/22 09:17:19.187418 [INF]   ID:       NCH4L4TAGXQK3QXTUTZXEPZJHOOGZRJVGPPMDQYFFKQ2EJHNUO2BMSMS
[91720] 2021/10/22 09:17:19.187982 [INF] Listening for client connections on 0.0.0.0:4222
[91720] 2021/10/22 09:17:19.188214 [INF] Server is ready
[cbrake@ceres ~]$ nats sub test
09:17:35 Subscribing on test
[#1] Received on "test"
hi

[#2] Received on "test"
hi
[cbrake@ceres ~]$ nats pub test "hi"
09:26:16 Published 2 bytes to "test"
[cbrake@ceres ~]$ nats pub test "hi"
09:26:17 Published 2 bytes to "test"

Pub/Sub with NGS account

  • create an account as described here
    • create account: nsc init -o synadia -n Cliff
    • check status limits: ngs status
  • Set up context for NGS
    • create context: nats context save ngs
    • edit context: nats context edit ngs
      • change the following lines
      • "url": "connect.ngs.global"
      • "creds": "/home/cbrake/.nkeys/creds/synadia/Cliff/Cliff.creds"
    • select context: nats context select ngs

(It seems the complete path is required for creds to work)

Now you should be able to nats sub and nats pub without your local server running as it will use the NGS global NATS servers.

Leaf nodes

Create a file named nsc-leaf.conf:

host: 127.0.0.1

leafnodes {
  remotes = [
    {
        url: "tls://connect.ngs.global"
        creds: "~/.nkeys/creds/synadia/Cliff/Cliff.creds"
    }
  ]
}

Start the NATS server with the above config:

  • nats-server -c ngs-leaf.conf

However, I got errors because the free plan does not support leaf nodes, so we need to change our plan to the developer:

  • ngs edit
  • select developer plan
  • you will then receive an email to verify email address and enter credit card #. Your card will not be charged with the developer plan.

Run nsc pull -A to sync your account information.

Now, I can sub a topic on my NGS server account, pub to a topic on my local server, and because the local server is a leaf node of NGS, the message will be sent to NGS.

> nats context select ngs
> nats sub test
10:35:26 Subscribing on test
[#1] Received on "test"
hi there

[#2] Received on "test"
hi there
> nats context select local
NATS Configuration Context "local"

      Server URLs: nats://127.0.0.1:4222
             Path: /home/cbrake/.config/nats/context/local.json

[cbrake@ceres ~]$ nats pub test "hi there"
10:35:50 Published 8 bytes to "test"
[cbrake@ceres ~]$ nats pub test "hi there"
10:35:52 Published 8 bytes to "test"

Next steps:

  • leaf node subscribes to a subset of topic namespace
  • accounts
  • authz so leaf node only has access to his subject namespace
  • JetStream
  • kv store

We do this in Simple IoT – it works great!

NATS 2.9.0 released:

NATS 2.10 has been released:

A few things caught my attention:

  • the extreme amount of testing that is going into NATS
  • they are designing it to scale
  • a lot of focus went into making the jetstream disk format more efficient.
  • there is now an Auth Callout extension point.

One other thing that is interesting is the NATS KV and Object store are client-side abstractions on top of Jetstream. No extra server functionality is required. This is pretty neat.

Evaluating Jetstream as a store for Simple IoT

The nats-expGitHub - simpleiot/nats-exp repo has several experiments where we attempt to determine if we can use NATS Jetstream as a store for Simple IoT. So far, it looks promising.

Time-series metrics:

2024/01/03 14:20:10 NATS TSD experiment
2024/01/03 14:20:10 js.Publish insert rate for 10,000 points: 42028 pts/sec
2024/01/03 14:20:10 js.PublishAsync insert rate for 10,000 points: 412363 pts/sec
2024/01/03 14:20:10 nats.Publish insert rate for 10,000 points: 17937960 pts/sec
2024/01/03 14:20:10 ack: false, Get 30000 points took 0.04, 668062 pts/sec
2024/01/03 14:20:10 ack: true, Get 30000 points took 0.04, 827364 pts/sec
2024/01/03 14:20:10 ack: false, Get 30000 points took 0.03, 881960 pts/sec

Seems fast enough.

Syncing streams between hub and leaf nodes:

[cbrake@quark nats-exp]$ go run ./bi-directional-sync/
=====================================================
       Create stream on hub and source to leaf
=====================================================
* Create stream: server:hub domain:hub stream:NODES-HUB subject:n.hub.>
* Publish: hub -> n.hub.123.value -> 12 13 14 15
* Stream count: server:hub domain:hub stream:NODES-HUB count:4
* Stream count: server:leaf domain:hub stream:NODES-HUB count:4
* Source stream: server:leaf domain:leaf stream:NODES-HUB source-domain:hub subject:n.hub.>
* Stream count: server:leaf domain:leaf stream:NODES-HUB count:4
=====================================================
       Create stream on leaf and source to hub
=====================================================
* Create stream: server:leaf domain:leaf stream:NODES-LEAF subject:n.leaf.>
* Publish: leaf -> n.leaf.456.value -> 22 23 24 25 26
* Source stream: server:hub domain:hub stream:NODES-LEAF source-domain:leaf subject:n.leaf.>
* Stream count: server:hub domain:hub stream:NODES-LEAF count:5
=====================================================
       Shut down leaf node
=====================================================
* Stream count: server:hub domain:hub stream:NODES-LEAF count:5
* Publish: hub -> n.hub.123.value -> 16 17 18 19
* Stream count: server:hub domain:hub stream:NODES-HUB count:8
=====================================================
       Start up leaf node
=====================================================
* Stream count: server:leaf domain:leaf stream:NODES-HUB count:8
* Publish: hub -> n.hub.123.value -> 20 21 22 23
* Stream count: server:leaf domain:leaf stream:NODES-HUB count:12
=====================================================
       Publish more messages to leaf
=====================================================
* Publish: leaf -> n.leaf.456.value -> 27 28 29 30 31
* Stream count: server:leaf domain:leaf stream:NODES-LEAF count:10
* Stream count: server:leaf domain:leaf stream:NODES-LEAF count:10