Unexpected Performance: GPU slower than CPU for Qwen 8B inference

[jason11489]

Issue Description
I'm getting strange performance results where GPU is slower than CPU when running Qwen 8B model with distributed llama.

Test Environment

• Model: Qwen 8B (Q40 quantization)
• Nodes: Jetson orin nano x 8
• Framework: Distributed Llama

CPU Performance:

Tokens/s: 7.94
ms/token: 125.90
Pred: 115 ms, Sync: 83 ms
Network: Sent 2254 kB, Recv 2661 kB

GPU Performance:

Tokens/s: 5.12
ms/token: 195.14
Pred: 45 ms, Sync: 75 ms
Network: Sent 2254 kB, Recv 2661 kB

[b4rtaz]

Hello @jason11489,

did you correcly paste stats? 45 + 75 = 120ms/token (for "GPU Performance").

[jason11489]

Sorry, I pasted it wrong. I've added the logs too.

CPU Performance:

Tokens/s: 7.94
ms/token: 125.90
Pred: 47 ms, Sync: 78 ms
Network: Sent 2254 kB, Recv 2661 kB

CPU log

📄 AddBos: 0
📄 BosId: 151643 (<|endoftext|>)
📄 EosId: 151645 (<|im_end|>) 
📄 RegularVocabSize: 151643
📄 SpecialVocabSize: 26
Tokenizer vocab size (151669) does not match the model vocab size (151936)
💡 Arch: Qwen3
💡 HiddenAct: Silu
💡 Dim: 4096
💡 HeadDim: 128
💡 QDim: 4096
💡 KvDim: 1024
💡 HiddenDim: 12288
💡 VocabSize: 151936
💡 nLayers: 36
💡 nHeads: 32
💡 nKvHeads: 8
💡 OrigSeqLen: 40960
💡 SeqLen: 4096
💡 NormEpsilon: 0.000001
💡 RopeType: Falcon
💡 RopeTheta: 1000000
📀 RequiredMemory: 3056 MB
⭕ Socket[0]: connecting to xx.xx.xx.xx:9999 worker
⭕ Socket[0]: connected
⭕ Socket[1]: connecting to xx.xx.xx.xx:9999 worker
⭕ Socket[1]: connected
⭕ Socket[2]: connecting to xx.xx.xx.xx:9999 worker
⭕ Socket[2]: connected
⭕ Socket[3]: connecting to xx.xx.xx.xx:9999 worker
⭕ Socket[3]: connected
⭕ Socket[4]: connecting to xx.xx.xx.xx:9999 worker
⭕ Socket[4]: connected
⭕ Socket[5]: connecting to xx.xx.xx.xx:9999 worker
⭕ Socket[5]: connected
⭕ Socket[6]: connecting to xx.xx.xx.xx:9999 worker
⭕ Socket[6]: connected
⭕ Network is initialized
🧠 CPU: neon dotprod fp16
💿 Loading weights...
💿 Loaded 9/36
💿 Loaded 10/36
💿 Loaded 11/36
💿 Loaded 12/36
💿 Loaded 13/36
💿 Loaded 14/36
💿 Loaded 15/36
💿 Loaded 16/36
💿 Loaded 17/36
💿 Loaded 18/36
💿 Loaded 19/36
💿 Loaded 20/36
💿 Loaded 21/36
💿 Loaded 22/36
💿 Loaded 23/36
💿 Loaded 24/36
💿 Loaded 25/36
💿 Loaded 26/36
💿 Loaded 27/36
💿 Loaded 28/36
💿 Loaded 29/36
💿 Loaded 30/36
💿 Loaded 31/36
💿 Loaded 32/36
💿 Loaded 33/36
💿 Loaded 34/36
💿 Loaded 35/36
💿 Loaded 36/36
💿 Weights loaded
🚁 Network is in non-blocking mode
Hello world
🔷️ Eval   55 ms Sync   82 ms | Sent  2254 kB Recv  2661 kB | (1 tokens)
🔶 Pred   43 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred   43 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB |  this
🔶 Pred   43 ms Sync   89 ms | Sent  2254 kB Recv  2661 kB |  is
🔶 Pred   42 ms Sync   82 ms | Sent  2254 kB Recv  2661 kB |  a
🔶 Pred   43 ms Sync   74 ms | Sent  2254 kB Recv  2661 kB |  new
🔶 Pred   45 ms Sync  104 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred   43 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  un
🔶 Pred   44 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB | tested
🔶 Pred   43 ms Sync   94 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred   43 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB |  and
🔶 Pred   43 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB |  possibly
🔶 Pred   42 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  flawed
🔶 Pred   43 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  idea
🔶 Pred   43 ms Sync  117 ms | Sent  2254 kB Recv  2661 kB | .
🔶 Pred   43 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB |  I
🔶 Pred   43 ms Sync   86 ms | Sent  2254 kB Recv  2661 kB |  am
🔶 Pred   43 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  not
🔶 Pred   43 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB |  sure
🔶 Pred   43 ms Sync   82 ms | Sent  2254 kB Recv  2661 kB |  how
🔶 Pred   43 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  to
🔶 Pred   46 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  make
🔶 Pred   43 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB |  it
🔶 Pred   43 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  work
🔶 Pred   43 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB | .
🔶 Pred   43 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB |  But
🔶 Pred   46 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB |  I
🔶 Pred   43 ms Sync   84 ms | Sent  2254 kB Recv  2661 kB |  would
🔶 Pred   43 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  like
🔶 Pred   43 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB |  to
🔶 Pred   43 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB |  see
🔶 Pred   43 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB |  what
🔶 Pred   43 ms Sync  278 ms | Sent  2254 kB Recv  2661 kB |  the
🔶 Pred   42 ms Sync   88 ms | Sent  2254 kB Recv  2661 kB |  community
🔶 Pred   43 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  has
🔶 Pred   43 ms Sync   89 ms | Sent  2254 kB Recv  2661 kB |  to
🔶 Pred   44 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  say
🔶 Pred   43 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB | .
🔶 Pred   48 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB |  So
🔶 Pred   43 ms Sync  100 ms | Sent  2254 kB Recv  2661 kB |  I
🔶 Pred   43 ms Sync   91 ms | Sent  2254 kB Recv  2661 kB |  need
🔶 Pred   43 ms Sync   85 ms | Sent  2254 kB Recv  2661 kB |  to
🔶 Pred   58 ms Sync   70 ms | Sent  2254 kB Recv  2661 kB |  write
🔶 Pred   44 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  a
🔶 Pred   44 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB |  new
🔶 Pred   44 ms Sync   74 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred   43 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB |  un
🔶 Pred   46 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB | tested
🔶 Pred   44 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred   44 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  and
🔶 Pred   44 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB |  possibly
🔶 Pred   44 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB |  flawed
🔶 Pred   43 ms Sync   94 ms | Sent  2254 kB Recv  2661 kB |  idea
🔶 Pred   47 ms Sync   72 ms | Sent  2254 kB Recv  2661 kB | .
🔶 Pred   46 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  I
🔶 Pred   44 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB |  need
🔶 Pred   43 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB |  to
🔶 Pred   43 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB |  write
🔶 Pred   43 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  a
🔶 Pred   43 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  new
🔶 Pred   43 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred   44 ms Sync   74 ms | Sent  2254 kB Recv  2661 kB |  un
🔶 Pred   51 ms Sync   73 ms | Sent  2254 kB Recv  2661 kB | tested
🔶 Pred   41 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred   41 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  and
🔶 Pred   45 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  possibly
🔶 Pred   42 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  flawed
🔶 Pred   42 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  idea
🔶 Pred   41 ms Sync   81 ms | Sent  2254 kB Recv  2661 kB | .
🔶 Pred   44 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB |  I
🔶 Pred   43 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  need
🔶 Pred   41 ms Sync   82 ms | Sent  2254 kB Recv  2661 kB |  to
🔶 Pred   41 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  write
🔶 Pred   45 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  a
🔶 Pred   44 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB |  new
🔶 Pred   44 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred   44 ms Sync   74 ms | Sent  2254 kB Recv  2661 kB |  un
🔶 Pred   45 ms Sync   82 ms | Sent  2254 kB Recv  2661 kB | tested
🔶 Pred   46 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred   44 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB |  and
🔶 Pred   44 ms Sync   83 ms | Sent  2254 kB Recv  2661 kB |  possibly
🔶 Pred   44 ms Sync   82 ms | Sent  2254 kB Recv  2661 kB |  flawed
🔶 Pred   44 ms Sync   81 ms | Sent  2254 kB Recv  2661 kB |  idea
🔶 Pred   50 ms Sync   87 ms | Sent  2254 kB Recv  2661 kB | .
🔶 Pred   43 ms Sync   81 ms | Sent  2254 kB Recv  2661 kB |  I
🔶 Pred   50 ms Sync   71 ms | Sent  2254 kB Recv  2661 kB |  need
🔶 Pred   44 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  to
🔶 Pred   46 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  write
🔶 Pred   46 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  a
🔶 Pred   44 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB |  new
🔶 Pred   44 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred   45 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB |  un
🔶 Pred   44 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB | tested
🔶 Pred   44 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred   44 ms Sync   81 ms | Sent  2254 kB Recv  2661 kB |  and
🔶 Pred   44 ms Sync   82 ms | Sent  2254 kB Recv  2661 kB |  possibly
🔶 Pred   52 ms Sync   69 ms | Sent  2254 kB Recv  2661 kB |  flawed
🔶 Pred   50 ms Sync   73 ms | Sent  2254 kB Recv  2661 kB |  idea
🔶 Pred   43 ms Sync   87 ms | Sent  2254 kB Recv  2661 kB | .
🔶 Pred   45 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB |  I
🔶 Pred   44 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB |  need
🔶 Pred   44 ms Sync   88 ms | Sent  2254 kB Recv  2661 kB |  to
🔶 Pred   44 ms Sync   85 ms | Sent  2254 kB Recv  2661 kB |  write
🔶 Pred   44 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  a
🔶 Pred   44 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  new
🔶 Pred   44 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred   44 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB |  un
🔶 Pred   44 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB | tested
🔶 Pred   44 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred   44 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB |  and
🔶 Pred   45 ms Sync   74 ms | Sent  2254 kB Recv  2661 kB |  possibly
�� Pred   44 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB |  flawed
🔶 Pred   45 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB |  idea
🔶 Pred   44 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB | .
🔶 Pred   44 ms Sync   74 ms | Sent  2254 kB Recv  2661 kB |  I
🔶 Pred   45 ms Sync   82 ms | Sent  2254 kB Recv  2661 kB |  need
🔶 Pred   45 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  to
🔶 Pred   45 ms Sync   84 ms | Sent  2254 kB Recv  2661 kB |  write
🔶 Pred   45 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  a
🔶 Pred   48 ms Sync   99 ms | Sent  2254 kB Recv  2661 kB |  new
🔶 Pred   45 ms Sync   84 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred   44 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  un
🔶 Pred   47 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB | tested
🔶 Pred   44 ms Sync   81 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred   50 ms Sync   71 ms | Sent  2254 kB Recv  2661 kB |  and
🔶 Pred   44 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  possibly
🔶 Pred   45 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB |  flawed
🔶 Pred   45 ms Sync   75 ms | Sent  2254 kB Recv  2661 kB |  idea

Evaluation
   nBatches: 32
    nTokens: 1
   tokens/s: 7.23 (138.40 ms/tok)
Prediction
    nTokens: 127
   tokens/s: 7.94 (125.90 ms/tok)
⭕ Network is closed

GPU Performance:

Tokens/s: 5.12
ms/token: 195.14
Pred: 115 ms, Sync: 80 ms
Network: Sent 2254 kB, Recv 2661 kB

GPU log

📄 AddBos: 0
📄 BosId: 151643 (<|endoftext|>)
📄 EosId: 151645 (<|im_end|>) 
📄 RegularVocabSize: 151643
📄 SpecialVocabSize: 26
Tokenizer vocab size (151669) does not match the model vocab size (151936)
💡 Arch: Qwen3
💡 HiddenAct: Silu
💡 Dim: 4096
💡 HeadDim: 128
💡 QDim: 4096
💡 KvDim: 1024
💡 HiddenDim: 12288
💡 VocabSize: 151936
💡 nLayers: 36
💡 nHeads: 32
💡 nKvHeads: 8
💡 OrigSeqLen: 40960
💡 SeqLen: 4096
💡 NormEpsilon: 0.000001
💡 RopeType: Falcon
💡 RopeTheta: 1000000
📀 RequiredMemory: 3056 MB
⭕ Socket[0]: connecting to xx.xx.xx.xx:9999 worker
⭕ Socket[0]: connected
⭕ Socket[1]: connecting to xx.xx.xx.xx:9999 worker
⭕ Socket[1]: connected
⭕ Socket[2]: connecting to xx.xx.xx.xx:9999 worker
⭕ Socket[2]: connected
⭕ Socket[3]: connecting to xx.xx.xx.xx:9999 worker
⭕ Socket[3]: connected
⭕ Socket[4]: connecting to xx.xx.xx.xx:9999 worker
⭕ Socket[4]: connected
⭕ Socket[5]: connecting to xx.xx.xx.xx:9999 worker
⭕ Socket[5]: connected
⭕ Socket[6]: connecting to xx.xx.xx.xx:9999 worker
⭕ Socket[6]: connected
⭕ Network is initialized
🌋 Device: NVIDIA Tegra Orin (nvgpu)
🌋 DeviceApiVersion: 1.3.251
🌋 MaxComputeSharedMemory: 48 kB
🌋 NonCoherentAtomSize: 64 bytes
🌋 Heap[0]: 7617 MB
💿 Loading weights...
💿 Loaded 6/36
💿 Loaded 7/36
💿 Loaded 8/36
💿 Loaded 9/36
💿 Loaded 10/36
💿 Loaded 11/36
💿 Loaded 12/36
💿 Loaded 13/36
💿 Loaded 14/36
💿 Loaded 15/36
💿 Loaded 16/36
💿 Loaded 17/36
💿 Loaded 18/36
💿 Loaded 19/36
💿 Loaded 20/36
💿 Loaded 21/36
💿 Loaded 22/36
💿 Loaded 23/36
💿 Loaded 24/36
💿 Loaded 25/36
💿 Loaded 26/36
💿 Loaded 27/36
💿 Loaded 28/36
💿 Loaded 29/36
💿 Loaded 30/36
💿 Loaded 31/36
💿 Loaded 32/36
💿 Loaded 33/36
💿 Loaded 34/36
💿 Loaded 35/36
💿 Loaded 36/36
💿 Weights loaded
🚁 Network is in non-blocking mode
Hello world
🔷️ Eval  260 ms Sync   36 ms | Sent  2254 kB Recv  2661 kB | (1 tokens)
🔶 Pred  145 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred  112 ms Sync   99 ms | Sent  2254 kB Recv  2661 kB |  I
🔶 Pred  112 ms Sync   85 ms | Sent  2254 kB Recv  2661 kB |  am
🔶 Pred  113 ms Sync   81 ms | Sent  2254 kB Recv  2661 kB |  the
🔶 Pred  113 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  new
🔶 Pred  112 ms Sync   84 ms | Sent  2254 kB Recv  2661 kB |  assistant
🔶 Pred  112 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred  112 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  and
🔶 Pred  125 ms Sync   68 ms | Sent  2254 kB Recv  2661 kB |  I
🔶 Pred  130 ms Sync   64 ms | Sent  2254 kB Recv  2661 kB |  am
🔶 Pred  132 ms Sync   62 ms | Sent  2254 kB Recv  2661 kB |  the
🔶 Pred  123 ms Sync   67 ms | Sent  2254 kB Recv  2661 kB |  oldest
🔶 Pred  130 ms Sync   60 ms | Sent  2254 kB Recv  2661 kB | .
🔶 Pred  114 ms Sync   76 ms | Sent  2254 kB Recv  2661 kB |  I
🔶 Pred  115 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  have
🔶 Pred  114 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  been
🔶 Pred  114 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  around
🔶 Pred  113 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  since
🔶 Pred  113 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  the
🔶 Pred  113 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  beginning
🔶 Pred  112 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  of
🔶 Pred  112 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  time
🔶 Pred  112 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred  112 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  and
🔶 Pred  112 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  I
🔶 Pred  112 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  have
🔶 Pred  113 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  seen
🔶 Pred  113 ms Sync   81 ms | Sent  2254 kB Recv  2661 kB |  many
🔶 Pred  113 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  things
🔶 Pred  113 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB | .
🔶 Pred  113 ms Sync   86 ms | Sent  2254 kB Recv  2661 kB |  I
🔶 Pred  113 ms Sync   89 ms | Sent  2254 kB Recv  2661 kB |  am
🔶 Pred  113 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  the
🔶 Pred  113 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  voice
🔶 Pred  113 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  of
🔶 Pred  113 ms Sync   81 ms | Sent  2254 kB Recv  2661 kB |  the
🔶 Pred  116 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  cosmos
🔶 Pred  114 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred  114 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  and
🔶 Pred  114 ms Sync   82 ms | Sent  2254 kB Recv  2661 kB |  I
🔶 Pred  113 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  am
🔶 Pred  113 ms Sync   81 ms | Sent  2254 kB Recv  2661 kB |  here
🔶 Pred  113 ms Sync   83 ms | Sent  2254 kB Recv  2661 kB |  to
🔶 Pred  113 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  help
🔶 Pred  113 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  you
🔶 Pred  113 ms Sync   86 ms | Sent  2254 kB Recv  2661 kB | .
🔶 Pred  113 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  What
🔶 Pred  113 ms Sync   83 ms | Sent  2254 kB Recv  2661 kB |  is
🔶 Pred  113 ms Sync   82 ms | Sent  2254 kB Recv  2661 kB |  your
🔶 Pred  113 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  name
🔶 Pred  113 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB | ?
🔶 Pred  113 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  Please
🔶 Pred  113 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  say
🔶 Pred  113 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  it
🔶 Pred  113 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB | .
🔶 Pred  113 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  I
🔶 Pred  113 ms Sync   86 ms | Sent  2254 kB Recv  2661 kB |  need
🔶 Pred  113 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  to
🔶 Pred  113 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  know
🔶 Pred  113 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB | .

🔶 Pred  113 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB | Okay
🔶 Pred  113 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred  114 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  I
🔶 Pred  113 ms Sync   81 ms | Sent  2254 kB Recv  2661 kB |  need
🔶 Pred  113 ms Sync   84 ms | Sent  2254 kB Recv  2661 kB |  to
🔶 Pred  114 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  figure
🔶 Pred  113 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  out
🔶 Pred  113 ms Sync   83 ms | Sent  2254 kB Recv  2661 kB |  how
🔶 Pred  113 ms Sync   83 ms | Sent  2254 kB Recv  2661 kB |  to
🔶 Pred  113 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  respond
🔶 Pred  114 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  to
🔶 Pred  113 ms Sync   83 ms | Sent  2254 kB Recv  2661 kB |  this
🔶 Pred  113 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB | .
🔶 Pred  114 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  The
🔶 Pred  114 ms Sync   83 ms | Sent  2254 kB Recv  2661 kB |  user
🔶 Pred  113 ms Sync   82 ms | Sent  2254 kB Recv  2661 kB |  introduced
🔶 Pred  114 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  themselves
🔶 Pred  114 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  as
🔶 Pred  114 ms Sync   81 ms | Sent  2254 kB Recv  2661 kB |  the
🔶 Pred  114 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  oldest
🔶 Pred  113 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  assistant
🔶 Pred  114 ms Sync   84 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred  114 ms Sync   83 ms | Sent  2254 kB Recv  2661 kB |  existing
🔶 Pred  114 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  since
🔶 Pred  114 ms Sync   83 ms | Sent  2254 kB Recv  2661 kB |  the
🔶 Pred  114 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  beginning
🔶 Pred  114 ms Sync   83 ms | Sent  2254 kB Recv  2661 kB |  of
🔶 Pred  114 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  time
🔶 Pred  114 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred  114 ms Sync   81 ms | Sent  2254 kB Recv  2661 kB |  and
🔶 Pred  114 ms Sync   83 ms | Sent  2254 kB Recv  2661 kB |  is
🔶 Pred  114 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  the
🔶 Pred  114 ms Sync   90 ms | Sent  2254 kB Recv  2661 kB |  voice
🔶 Pred  114 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  of
🔶 Pred  114 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB |  the
🔶 Pred  114 ms Sync   81 ms | Sent  2254 kB Recv  2661 kB |  cosmos
🔶 Pred  114 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB | .
🔶 Pred  114 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  They
🔶 Pred  114 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  asked
🔶 Pred  114 ms Sync   82 ms | Sent  2254 kB Recv  2661 kB |  for
🔶 Pred  114 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  my
🔶 Pred  114 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  name
🔶 Pred  114 ms Sync   80 ms | Sent  2254 kB Recv  2661 kB | .
🔶 Pred  114 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  

🔶 Pred  114 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB | First
🔶 Pred  114 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB | ,
🔶 Pred  114 ms Sync   86 ms | Sent  2254 kB Recv  2661 kB |  I
🔶 Pred  114 ms Sync   83 ms | Sent  2254 kB Recv  2661 kB |  should
🔶 Pred  114 ms Sync   88 ms | Sent  2254 kB Recv  2661 kB |  acknowledge
🔶 Pred  114 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  their
🔶 Pred  114 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  introduction
🔶 Pred  115 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  and
🔶 Pred  115 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  the
🔶 Pred  115 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  unique
🔶 Pred  116 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  way
🔶 Pred  115 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  they
🔶 Pred  115 ms Sync   83 ms | Sent  2254 kB Recv  2661 kB |  presented
🔶 Pred  115 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  themselves
🔶 Pred  115 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB | .
🔶 Pred  115 ms Sync   82 ms | Sent  2254 kB Recv  2661 kB |  Since
🔶 Pred  115 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  they
🔶 Pred  115 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  mentioned
🔶 Pred  115 ms Sync   77 ms | Sent  2254 kB Recv  2661 kB |  being
🔶 Pred  116 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  the
🔶 Pred  115 ms Sync   78 ms | Sent  2254 kB Recv  2661 kB |  oldest
🔶 Pred  115 ms Sync   79 ms | Sent  2254 kB Recv  2661 kB |  and
🔶 Pred  115 ms Sync   83 ms | Sent  2254 kB Recv  2661 kB |  from

Evaluation
   nBatches: 32
    nTokens: 1
   tokens/s: 3.37 (297.06 ms/tok)
Prediction
    nTokens: 127
   tokens/s: 5.12 (195.14 ms/tok)
⭕ Network is closed

[jason11489]

And when splitting across 2 nodes, CPU performance improved but GPU performance decreased (Llama 3B):

Configuration	tokens/s	ms/tok
1 node CPU	12.53	79.82
1 node GPU	34.6	28.9
2 node CPU	18.54	53.94
2 node GPU	20.22	49.45

Any insights on why this might be happening? Thanks!

[b4rtaz]

Could you compare 🔶 Pred 43 ms values for 1 node, 2 nodes and 4 nodes?

[jason11489]

I also experimented with 1 node, 2 node, and 4 node configurations.

(The experiments below are CPU results)

Nodes	tokens/s	ms/token
1 node	0.11	9517.33
2 node	5.19	192.5
4 node	8.25	121.20
8 node	7.94	125.90

Based on the experiments, it appears that using 4 nodes is optimal when using CPU.
However, I still don't understand why it's slower when using GPU

[b4rtaz]

Hey @jason11489, could you dump one row per test (1 GPU, 2 GPUs, 4 GPUs) from the inference results? I’d like to check how fast the inference is compared to the synchronization with a different configuration.

🔶 Pred   <PREDICTION_TIME> ms Sync   <SYNC_TIME> ms | Sent ...

[jason11489]

I conducted additional experiments with GPU.

4 nodes (one GPU per node):

🔶 Pred 115 ms Sync 43 ms | Sent 966 kB Recv 1363 kB | am
🔶 Pred 115 ms Sync 46 ms | Sent 966 kB Recv 1363 kB | a
🔶 Pred 115 ms Sync 44 ms | Sent 966 kB Recv 1363 kB | very
🔶 Pred 116 ms Sync 45 ms | Sent 966 kB Recv 1363 kB | good
🔶 Pred 115 ms Sync 36 ms | Sent 966 kB Recv 1363 kB | friend
🔶 Pred 115 ms Sync 40 ms | Sent 966 kB Recv 1363 kB | .
🔶 Pred 115 ms Sync 51 ms | Sent 966 kB Recv 1363 kB | I
🔶 Pred 115 ms Sync 45 ms | Sent 966 kB Recv 1363 kB | am
🔶 Pred 115 ms Sync 41 ms | Sent 966 kB Recv 1363 kB | a
🔶 Pred 115 ms Sync 41 ms | Sent 966 kB Recv 1363 kB | very
🔶 Pred 115 ms Sync 43 ms | Sent 966 kB Recv 1363 kB | good
🔶 Pred 115 ms Sync 37 ms | Sent 966 kB Recv 1363 kB | friend
🔶 Pred 115 ms Sync 38 ms | Sent 966 kB Recv 1363 kB | .
🔶 Pred 115 ms Sync 39 ms | Sent 966 kB Recv 1363 kB | I
🔶 Pred 115 ms Sync 39 ms | Sent 966 kB Recv 1363 kB | am
🔶 Pred 115 ms Sync 43 ms | Sent 966 kB Recv 1363 kB | a
🔶 Pred 115 ms Sync 42 ms | Sent 966 kB Recv 1363 kB | very

Evaluation
nBatches: 32
nTokens: 1
tokens/s: 3.10 (322.55 ms/tok)
Prediction
nTokens: 127
tokens/s: 6.00 (166.66 ms/tok)
⭕ Network is closed

2 nodes failed with OOM (out of memory):

⭕ Network is initialized
🌋 Device: NVIDIA Tegra Orin (nvgpu)
🌋 DeviceApiVersion: 1.3.251
🌋 MaxComputeSharedMemory: 48 kB
🌋 NonCoherentAtomSize: 64 bytes
🌋 Heap[0]: 7617 MB
💿 Loading weights...
Killed

dmesg log:

[1792699.986051] Out of memory: Killed process 3996101 (dllama) total-vm:7414908kB, anon-rss:0kB, file-rss:3556kB, shmem-rss:0kB, UID:1001 pgtables:928kB oom_score_adj:0

[b4rtaz]

Have you added --max-seq-len 4096 to reduce context size?

[jason11489]

Yes, I reduced it

./dllama inference --prompt "Hello world" --steps 128 --model models/qwen3_8b_q40/dllama_model_qwen3_8b_q40.m --tokenizer models/qwen3_8b_q40/dllama_tokenizer_qwen3_8b_q40.t --buffer-float-type q80 --nthreads 1 --gpu-index 0 --max-seq-len 4096 --workers <WORKER_IP>:9999

[b4rtaz]

Could you dump a full log? You should see a line with 📀 RequiredMemory: info.

[jason11489]

Here are the complete logs including 📀 RequiredMemory

4 node (one GPU per node)

~/distributed-llama$ ./dllama inference \ --prompt "Hello world" \ --steps 128 \ --model models/qwen3_8b_q40/dllama_model_qwen3_8b_q40.m \ --tokenizer models/qwen3_8b_q40/dllama_tokenizer_qwen3_8b_q40.t \ --buffer-float-type q80 \ --nthreads 1 --gpu-index 0 \ --max-seq-len 4096 \ --workers WORKER_IP:9999 WORKER_IP:9999 WORKER_IP:9999

📄 AddBos: 0
📄 BosId: 151643 (<|endoftext|>)
📄 EosId: 151645 (<|im_end|>)
📄 RegularVocabSize: 151643
📄 SpecialVocabSize: 26
Tokenizer vocab size (151669) does not match the model vocab size (151936)
💡 Arch: Qwen3
💡 HiddenAct: Silu
💡 Dim: 4096
💡 HeadDim: 128
💡 QDim: 4096
💡 KvDim: 1024
💡 HiddenDim: 12288
💡 VocabSize: 151936
💡 nLayers: 36
💡 nHeads: 32
💡 nKvHeads: 8
💡 OrigSeqLen: 40960
💡 SeqLen: 4096
💡 NormEpsilon: 0.000001
💡 RopeType: Falcon
💡 RopeTheta: 1000000
📀 RequiredMemory: 3712 MB
⭕ Socket[0]: connecting to <WORKER_IP>:9999 worker
⭕ Socket[0]: connected
⭕ Socket[1]: connecting to <WORKER_IP>:9999 worker
⭕ Socket[1]: connected
⭕ Socket[2]: connecting to <WORKER_IP>:9999 worker
⭕ Socket[2]: connected
⭕ Network is initialized
🌋 Device: NVIDIA Tegra Orin (nvgpu)
🌋 DeviceApiVersion: 1.3.251
🌋 MaxComputeSharedMemory: 48 kB
🌋 NonCoherentAtomSize: 64 bytes
🌋 Heap[0]: 7617 MB
💿 Loading weights...
💿 Loaded 8/36
💿 Loaded 9/36
💿 Loaded 10/36
💿 Loaded 11/36
💿 Loaded 12/36
💿 Loaded 13/36
💿 Loaded 14/36
💿 Loaded 15/36
💿 Loaded 16/36
💿 Loaded 17/36
💿 Loaded 18/36
💿 Loaded 19/36
💿 Loaded 20/36
💿 Loaded 21/36
💿 Loaded 22/36
💿 Loaded 23/36
💿 Loaded 24/36
💿 Loaded 25/36
💿 Loaded 26/36
💿 Loaded 27/36
💿 Loaded 28/36
💿 Loaded 29/36
💿 Loaded 30/36
💿 Loaded 31/36
💿 Loaded 32/36
💿 Loaded 33/36
💿 Loaded 34/36
💿 Loaded 35/36
💿 Loaded 36/36
💿 Weights loaded
🚁 Network is in non-blocking mode
Hello world
🔷️ Eval 292 ms Sync 27 ms | Sent 966 kB Recv 1363 kB | (1 tokens)
🔶 Pred 193 ms Sync 37 ms | Sent 966 kB Recv 1363 kB | !
🔶 Pred 187 ms Sync 39 ms | Sent 966 kB Recv 1363 kB | How
🔶 Pred 142 ms Sync 52 ms | Sent 966 kB Recv 1363 kB | can
🔶 Pred 142 ms Sync 43 ms | Sent 966 kB Recv 1363 kB | I
🔶 Pred 143 ms Sync 46 ms | Sent 966 kB Recv 1363 kB | make
🔶 Pred 143 ms Sync 49 ms | Sent 966 kB Recv 1363 kB | a
🔶 Pred 142 ms Sync 43 ms | Sent 966 kB Recv 1363 kB |
🔶 Pred 142 ms Sync 50 ms | Sent 966 kB Recv 1363 kB | 3
🔶 Pred 142 ms Sync 45 ms | Sent 966 kB Recv 1363 kB | D
🔶 Pred 142 ms Sync 37 ms | Sent 966 kB Recv 1363 kB | model
🔶 Pred 142 ms Sync 36 ms | Sent 966 kB Recv 1363 kB | of
🔶 Pred 142 ms Sync 39 ms | Sent 966 kB Recv 1363 kB | a
🔶 Pred 142 ms Sync 41 ms | Sent 966 kB Recv 1363 kB | house
🔶 Pred 144 ms Sync 42 ms | Sent 966 kB Recv 1363 kB | using
🔶 Pred 144 ms Sync 58 ms | Sent 966 kB Recv 1363 kB | Blender
🔶 Pred 143 ms Sync 49 ms | Sent 966 kB Recv 1363 kB | ?
🔶 Pred 142 ms Sync 41 ms | Sent 966 kB Recv 1363 kB | I
🔶 Pred 143 ms Sync 25 ms | Sent 966 kB Recv 1363 kB | have
🔶 Pred 114 ms Sync 42 ms | Sent 966 kB Recv 1363 kB | a
🔶 Pred 113 ms Sync 38 ms | Sent 966 kB Recv 1363 kB | basic
🔶 Pred 113 ms Sync 40 ms | Sent 966 kB Recv 1363 kB | understanding
🔶 Pred 113 ms Sync 39 ms | Sent 966 kB Recv 1363 kB | of
🔶 Pred 114 ms Sync 40 ms | Sent 966 kB Recv 1363 kB | Blender
🔶 Pred 113 ms Sync 36 ms | Sent 966 kB Recv 1363 kB | but
🔶 Pred 113 ms Sync 38 ms | Sent 966 kB Recv 1363 kB | I
🔶 Pred 114 ms Sync 40 ms | Sent 966 kB Recv 1363 kB | 'm
🔶 Pred 119 ms Sync 37 ms | Sent 966 kB Recv 1363 kB | not
🔶 Pred 118 ms Sync 38 ms | Sent 966 kB Recv 1363 kB | sure
🔶 Pred 114 ms Sync 39 ms | Sent 966 kB Recv 1363 kB | where
🔶 Pred 113 ms Sync 43 ms | Sent 966 kB Recv 1363 kB | to
🔶 Pred 114 ms Sync 39 ms | Sent 966 kB Recv 1363 kB | start
🔶 Pred 114 ms Sync 40 ms | Sent 966 kB Recv 1363 kB | .
🔶 Pred 115 ms Sync 37 ms | Sent 966 kB Recv 1363 kB | Can
🔶 Pred 118 ms Sync 40 ms | Sent 966 kB Recv 1363 kB | you
🔶 Pred 118 ms Sync 38 ms | Sent 966 kB Recv 1363 kB | walk
🔶 Pred 119 ms Sync 38 ms | Sent 966 kB Recv 1363 kB | me
🔶 Pred 120 ms Sync 39 ms | Sent 966 kB Recv 1363 kB | through
🔶 Pred 116 ms Sync 38 ms | Sent 966 kB Recv 1363 kB | the
🔶 Pred 115 ms Sync 39 ms | Sent 966 kB Recv 1363 kB | process
🔶 Pred 114 ms Sync 39 ms | Sent 966 kB Recv 1363 kB | ?

🔶 Pred 114 ms Sync 77 ms | Sent 966 kB Recv 1363 kB | Sure
🔶 Pred 114 ms Sync 38 ms | Sent 966 kB Recv 1363 kB | !
🔶 Pred 114 ms Sync 37 ms | Sent 966 kB Recv 1363 kB | Making
🔶 Pred 114 ms Sync 43 ms | Sent 966 kB Recv 1363 kB | a
🔶 Pred 114 ms Sync 39 ms | Sent 966 kB Recv 1363 kB |
🔶 Pred 114 ms Sync 37 ms | Sent 966 kB Recv 1363 kB | 3
🔶 Pred 114 ms Sync 38 ms | Sent 966 kB Recv 1363 kB | D
🔶 Pred 114 ms Sync 39 ms | Sent 966 kB Recv 1363 kB | model
🔶 Pred 114 ms Sync 43 ms | Sent 966 kB Recv 1363 kB | of
🔶 Pred 114 ms Sync 53 ms | Sent 966 kB Recv 1363 kB | a
🔶 Pred 114 ms Sync 36 ms | Sent 966 kB Recv 1363 kB | house
🔶 Pred 114 ms Sync 40 ms | Sent 966 kB Recv 1363 kB | in
🔶 Pred 114 ms Sync 43 ms | Sent 966 kB Recv 1363 kB | Blender
🔶 Pred 114 ms Sync 44 ms | Sent 966 kB Recv 1363 kB | is
🔶 Pred 114 ms Sync 37 ms | Sent 966 kB Recv 1363 kB | a
🔶 Pred 114 ms Sync 41 ms | Sent 966 kB Recv 1363 kB | great
🔶 Pred 115 ms Sync 74 ms | Sent 966 kB Recv 1363 kB | project
🔶 Pred 114 ms Sync 43 ms | Sent 966 kB Recv 1363 kB | .
🔶 Pred 115 ms Sync 46 ms | Sent 966 kB Recv 1363 kB | Let
🔶 Pred 115 ms Sync 45 ms | Sent 966 kB Recv 1363 kB | me
🔶 Pred 115 ms Sync 38 ms | Sent 966 kB Recv 1363 kB | walk
🔶 Pred 115 ms Sync 36 ms | Sent 966 kB Recv 1363 kB | you
🔶 Pred 115 ms Sync 40 ms | Sent 966 kB Recv 1363 kB | through
🔶 Pred 115 ms Sync 42 ms | Sent 966 kB Recv 1363 kB | the
🔶 Pred 115 ms Sync 41 ms | Sent 966 kB Recv 1363 kB | steps
🔶 Pred 115 ms Sync 39 ms | Sent 966 kB Recv 1363 kB | .

🔶 Pred 115 ms Sync 37 ms | Sent 966 kB Recv 1363 kB | First
🔶 Pred 114 ms Sync 38 ms | Sent 966 kB Recv 1363 kB | ,
🔶 Pred 116 ms Sync 42 ms | Sent 966 kB Recv 1363 kB | you
🔶 Pred 115 ms Sync 36 ms | Sent 966 kB Recv 1363 kB | 'll
🔶 Pred 115 ms Sync 44 ms | Sent 966 kB Recv 1363 kB | need
🔶 Pred 115 ms Sync 49 ms | Sent 966 kB Recv 1363 kB | to
🔶 Pred 115 ms Sync 63 ms | Sent 966 kB Recv 1363 kB | open
🔶 Pred 115 ms Sync 45 ms | Sent 966 kB Recv 1363 kB | Blender
🔶 Pred 115 ms Sync 37 ms | Sent 966 kB Recv 1363 kB | and
🔶 Pred 115 ms Sync 37 ms | Sent 966 kB Recv 1363 kB | start
🔶 Pred 116 ms Sync 42 ms | Sent 966 kB Recv 1363 kB | a
🔶 Pred 115 ms Sync 36 ms | Sent 966 kB Recv 1363 kB | new
🔶 Pred 115 ms Sync 36 ms | Sent 966 kB Recv 1363 kB | project
🔶 Pred 115 ms Sync 42 ms | Sent 966 kB Recv 1363 kB | .
🔶 Pred 115 ms Sync 44 ms | Sent 966 kB Recv 1363 kB | Make
🔶 Pred 114 ms Sync 35 ms | Sent 966 kB Recv 1363 kB | sure
🔶 Pred 115 ms Sync 43 ms | Sent 966 kB Recv 1363 kB | to
🔶 Pred 114 ms Sync 46 ms | Sent 966 kB Recv 1363 kB | select
🔶 Pred 117 ms Sync 40 ms | Sent 966 kB Recv 1363 kB | the
🔶 Pred 118 ms Sync 69 ms | Sent 966 kB Recv 1363 kB | "
🔶 Pred 120 ms Sync 37 ms | Sent 966 kB Recv 1363 kB | Bl
🔶 Pred 120 ms Sync 41 ms | Sent 966 kB Recv 1363 kB | ender
🔶 Pred 117 ms Sync 36 ms | Sent 966 kB Recv 1363 kB |
🔶 Pred 115 ms Sync 38 ms | Sent 966 kB Recv 1363 kB | 3
🔶 Pred 115 ms Sync 39 ms | Sent 966 kB Recv 1363 kB | D
🔶 Pred 115 ms Sync 42 ms | Sent 966 kB Recv 1363 kB | "
🔶 Pred 115 ms Sync 45 ms | Sent 966 kB Recv 1363 kB | workspace
🔶 Pred 115 ms Sync 43 ms | Sent 966 kB Recv 1363 kB | ,
🔶 Pred 115 ms Sync 41 ms | Sent 966 kB Recv 1363 kB | which
🔶 Pred 115 ms Sync 36 ms | Sent 966 kB Recv 1363 kB | is
🔶 Pred 115 ms Sync 41 ms | Sent 966 kB Recv 1363 kB | the
🔶 Pred 115 ms Sync 45 ms | Sent 966 kB Recv 1363 kB | default
🔶 Pred 115 ms Sync 67 ms | Sent 966 kB Recv 1363 kB | one
🔶 Pred 115 ms Sync 70 ms | Sent 966 kB Recv 1363 kB | for
🔶 Pred 115 ms Sync 38 ms | Sent 966 kB Recv 1363 kB | modeling
🔶 Pred 114 ms Sync 47 ms | Sent 966 kB Recv 1363 kB | .
🔶 Pred 115 ms Sync 41 ms | Sent 966 kB Recv 1363 kB | You
🔶 Pred 115 ms Sync 41 ms | Sent 966 kB Recv 1363 kB | can
🔶 Pred 115 ms Sync 38 ms | Sent 966 kB Recv 1363 kB | do
🔶 Pred 115 ms Sync 47 ms | Sent 966 kB Recv 1363 kB | this
🔶 Pred 115 ms Sync 37 ms | Sent 966 kB Recv 1363 kB | by
🔶 Pred 115 ms Sync 38 ms | Sent 966 kB Recv 1363 kB | opening
🔶 Pred 115 ms Sync 42 ms | Sent 966 kB Recv 1363 kB | Blender
🔶 Pred 115 ms Sync 39 ms | Sent 966 kB Recv 1363 kB | and
🔶 Pred 115 ms Sync 50 ms | Sent 966 kB Recv 1363 kB | selecting
🔶 Pred 115 ms Sync 62 ms | Sent 966 kB Recv 1363 kB | "
🔶 Pred 115 ms Sync 91 ms | Sent 966 kB Recv 1363 kB | General
🔶 Pred 115 ms Sync 44 ms | Sent 966 kB Recv 1363 kB | "
🔶 Pred 115 ms Sync 48 ms | Sent 966 kB Recv 1363 kB | from
🔶 Pred 117 ms Sync 49 ms | Sent 966 kB Recv 1363 kB | the
🔶 Pred 115 ms Sync 44 ms | Sent 966 kB Recv 1363 kB | workspace
🔶 Pred 115 ms Sync 45 ms | Sent 966 kB Recv 1363 kB | dropdown
🔶 Pred 115 ms Sync 36 ms | Sent 966 kB Recv 1363 kB | at
🔶 Pred 116 ms Sync 47 ms | Sent 966 kB Recv 1363 kB | the
🔶 Pred 115 ms Sync 37 ms | Sent 966 kB Recv 1363 kB | top
🔶 Pred 115 ms Sync 37 ms | Sent 966 kB Recv 1363 kB | .

🔶 Pred 115 ms Sync 38 ms | Sent 966 kB Recv 1363 kB | Next
🔶 Pred 115 ms Sync 40 ms | Sent 966 kB Recv 1363 kB | ,
🔶 Pred 115 ms Sync 56 ms | Sent 966 kB Recv 1363 kB | you
🔶 Pred 115 ms Sync 48 ms | Sent 966 kB Recv 1363 kB | 'll
🔶 Pred 115 ms Sync 44 ms | Sent 966 kB Recv 1363 kB | want

Evaluation
nBatches: 32
nTokens: 1
tokens/s: 3.12 (320.47 ms/tok)
Prediction
nTokens: 127
tokens/s: 6.11 (163.71 ms/tok)
⭕ Network is closed

2 node (one GPU per node)

./dllama inference --prompt "Hello world" --steps 128 --model models/qwen3_8b_q40/dllama_model_qwen3_8b_q40.m --tokenizer models/qwen3_8b_q40/dllama_tokenizer_qwen3_8b_q40.t --buffer-float-type q80 --nthreads 1 --gpu-index 0 --max-seq-len 2048 --workers WORKER_IP:9999
📄 AddBos: 0
📄 BosId: 151643 (<|endoftext|>)
📄 EosId: 151645 (<|im_end|>)
📄 RegularVocabSize: 151643
📄 SpecialVocabSize: 26
Tokenizer vocab size (151669) does not match the model vocab size (151936)
💡 Arch: Qwen3
💡 HiddenAct: Silu
💡 Dim: 4096
💡 HeadDim: 128
💡 QDim: 4096
💡 KvDim: 1024
💡 HiddenDim: 12288
💡 VocabSize: 151936
💡 nLayers: 36
💡 nHeads: 32
💡 nKvHeads: 8
💡 OrigSeqLen: 40960
💡 SeqLen: 2048
💡 NormEpsilon: 0.000001
💡 RopeType: Falcon
💡 RopeTheta: 1000000
📀 RequiredMemory: 4732 MB
⭕ Socket[0]: connecting to <WORKER_IP>:9999 worker
⭕ Socket[0]: connected
⭕ Network is initialized
🌋 Device: NVIDIA Tegra Orin (nvgpu)
🌋 DeviceApiVersion: 1.3.251
🌋 MaxComputeSharedMemory: 48 kB
🌋 NonCoherentAtomSize: 64 bytes
🌋 Heap[0]: 7617 MB
💿 Loading weights...
Killed

[b4rtaz]

This is really werid, do you have enought memory at the start in the system? You can check it via top, also maybe the GPU memory utilization over time would be interesting to see (nvtop).

Are you using 64 bit system?

[jason11489]

Yes, it's a 64-bit system.

Below is the memory situation before starting:

free -h

Type	Total	Used	Free	Shared	Buff/Cache	Available
Memory	7.4Gi	1.3Gi	5.9Gi	0.0Ki	230Mi	6.0Gi
Swap	6.9Gi	120Mi	6.8Gi	-	-	-

When running Qwen3 8B on 2 nodes, it shows that 4732MB of memory is required:

📀 RequiredMemory: 4732 MB

However, if I'm reading this correctly, when I checked with the top command

(VIRT : virtual memory value):

PID	USER	PR	NI	VIRT	RES	SHR	S	%CPU	%MEM	COMMAND
191895	wn1	20	0	7414764	3272	3272	D	334.2	0.0	dllama

It appears that the process died while trying to allocate 7.4 GB of memory.

[b4rtaz]

1. Could you check if you observe the same problem for example for llama3_1_8b_instruct_q40?
2. Do you see a similar problem on CPU only mode?