TPU and GPU inference support

[novikov-alexander]

Are there any plans to support inference on some accelerators?

Let's say use ONNXRuntime or TensorRT to free up CPU resources.

[olilarkin]

NeuralAmpModelerCore is a bespoke inference engine written in C++ using Eigen. ORT etc are generic inference engines that aren't designed for Realtime usage and may not support the kind of stateful model that is used in the NAM architectures.

GPUAudio have ported NAM to their SDK which is probably the most interesting thing in this regard since it is designed and optimized for realtime

https://github.com/gpuaudio/gpuaudio-sdk/tree/main/NeuralAmpModeler

[sdatkinson]

What @olilarkin says is right--only certain kinds of standard NAMs like the stacked LSTM can be "off the shelf" run with something like ONNX and work acceptably in real-time.

That said, I'm not necessarily against supporting other devices; it's just not straightforward--and why things like GPU Audio's tools are useful :)

If your actual goal is just to free up CPU, then the biggest thing you could do is to train your models smaller.

I'm going to close this since there's not a concrete plan for this in this repo for the near future. If there is, I'll flesh it out a bit more.