Batch Inference On 8255 device

[imjking]

Hi, I want to perform batch inference on the 8255 device now.
I noticed there is a --num_iters parameter in qnn_llama_runner. Is this parameter for batch inference? Additionally, how can I use the KV cache, that is, load the model and system_prompt once and then perform multiple inferences.
Looking forward to your reply.

cc @cccclai @winskuo-quic @shewu-quic @haowhsu-quic @DannyYuyang-quic @cbilgin

[cccclai]

I noticed there is a --num_iters parameter in qnn_llama_runner. Is this parameter for batch inference?

I believe --num_iters is for running multiple same inference, but not batch inference. Do you expect users sending multiple prompts at the same time for inference?

Additionally, how can I use the KV cache, that is, load the model and system_prompt once and then perform multiple inferences.

Are you trying to have multi turn conversation? If so, I think you can do `--prompt first_prompt --prompt second_prompt" https://github.com/pytorch/executorch/blob/main/examples/qualcomm/oss_scripts/llama/qnn_llama_runner.cpp#L44-L47

[haowhsu-quic]

Batch inference will impact the performance of NPU, currently we only support graph with batch size 1. Since the kv cache is not very straight forward to be reused now, the easiest way to do it is to define your own token generation loop if you're looking for something like beam search.
Once the flow enters the token generation loop, prompt kv caches are already evaluated. You can apply different sampling strategy by resetting the start_pos.