High rate data, example of Go concurrency

Doing some load testing in SIOT by pumping 15KHz data through it – we have some work to do as we have too many NATS clients subscribing to all the data flowing through the system and we start getting NATS slow consumer messages.

It is interesting to view how much CPU is being used by a heavily loaded Go app. Below is a top snapshot. The cumulative processor usage is around 220%. Appears we have lots of threads using parts of a CPU. May not be the most efficient, but at least it’s highly concurrent at this point.

In this test setup, I’m writing data to InfluxDB.

Decided that we need to bypass most of the normal stuff in SIOT for high-rate data and create some new NATs subjects specially for high rate data. Once I did that, 15KHz data started flowing through just fine. In viewing the data, learned that Grafana and the Influx UI’s only support ms time resolution:

Apparently this is a limitation of the browser that is not easily overcome:

Influx held up to this load pretty well – could handle the 14KHz stream using about 17% of one CPU (out of the 12 I have on this machine).