Eval bug: asymmetric layer splitting of reduced models on multiple CUDA GPUs

[recallmenot]

Name and Version

./llama-cli --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
version: 4434 (a3d50bc0)
built with cc (GCC) 14.2.1 20240910 for x86_64-pc-linux-gnu

Operating systems

Linux

GGML backends

CUDA

Hardware

2 * RTX 3090

Models

Llama-3_1-Nemotron-51B-Instruct

Problem description & steps to reproduce

model download:

huggingface-cli download tensorblock/Llama-3_1-Nemotron-51B-Instruct-GGUF --include "Llama-3_1-Nemotron-51B-Instruct-Q5_K_M.gguf" --local-dir ~/llama.cpp/models/

Bartowski's GGUF behaves the same:

huggingface-cli download bartowski/Llama-3_1-Nemotron-51B-Instruct-GGUF --include "Llama-3_1-Nemotron-51B-Instruct-Q5_K_M.gguf" --local-dir ~/llama.cpp/models/

When I try to load this model, llama.cpp tries to load most of the layers into CUDA0 and only some into CUDA1.

I had to reduce context size a lot to get it to even load.

~/llama.cpp/build/bin/llama-server \
	--model ~/llama.cpp/models/Llama-3_1-Nemotron-51B-Instruct-Q5_K_M.gguf \
	--n-gpu-layers 81 \
	--ctx-size 10000

First Bad Commit

No response

Relevant log output

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
build: 4434 (a3d50bc0) with cc (GCC) 14.2.1 20240910 for x86_64-pc-linux-gnu
system info: n_threads = 6, n_threads_batch = 6, total_threads = 16

system_info: n_threads = 6 (n_threads_batch = 6) / 16 | CUDA : ARCHS = 860 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 | 

main: HTTP server is listening, hostname: 0.0.0.0, port: 9090, http threads: 15
main: loading model
srv    load_model: loading model '/home/recallmenot/llama.cpp/models/Llama-3_1-Nemotron-51B-Instruct-Q5_K_M.gguf'
llama_model_load_from_file: using device CUDA0 (NVIDIA GeForce RTX 3090) - 23873 MiB free
llama_model_load_from_file: using device CUDA1 (NVIDIA GeForce RTX 3090) - 23873 MiB free
llama_model_loader: loaded meta data with 34 key-value pairs and 630 tensors from /home/recallmenot/llama.cpp/models/Llama-3_1-Nemotron-51B-Instruct-Q5_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = deci
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Nvidia Llama 3_1 Nemotron 51B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = 3_1-Nemotron-Instruct
llama_model_loader: - kv   4:                           general.basename str              = nvidia-Llama
llama_model_loader: - kv   5:                         general.size_label str              = 51B
llama_model_loader: - kv   6:                            general.license str              = other
llama_model_loader: - kv   7:                       general.license.name str              = nvidia-open-model-license
llama_model_loader: - kv   8:                       general.license.link str              = https://developer.download.nvidia.com...
llama_model_loader: - kv   9:                               general.tags arr[str,4]       = ["nvidia", "llama-3", "pytorch", "tex...
llama_model_loader: - kv  10:                          general.languages arr[str,1]       = ["en"]
llama_model_loader: - kv  11:               deci.attention.head_count_kv arr[i32,80]      = [8, 4, 8, 8, 8, 2, 2, 1, 1, 2, 2, 0, ...
llama_model_loader: - kv  12:                  deci.attention.head_count arr[i32,80]      = [64, 64, 64, 64, 64, 64, 64, 64, 64, ...
llama_model_loader: - kv  13:                   deci.feed_forward_length arr[i32,80]      = [7168, 14336, 28672, 28672, 28672, 14...
llama_model_loader: - kv  14:                           deci.block_count u32              = 80
llama_model_loader: - kv  15:                        deci.context_length u32              = 131072
llama_model_loader: - kv  16:                      deci.embedding_length u32              = 8192
llama_model_loader: - kv  17:      deci.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  18:                  deci.attention.key_length u32              = 128
llama_model_loader: - kv  19:                deci.attention.value_length u32              = 128
llama_model_loader: - kv  20:                          general.file_type u32              = 17
llama_model_loader: - kv  21:                            deci.vocab_size u32              = 128256
llama_model_loader: - kv  22:                  deci.rope.dimension_count u32              = 128
llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = llama-bpe
llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,128256]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,128256]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  27:                      tokenizer.ggml.merges arr[str,280147]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  28:                tokenizer.ggml.bos_token_id u32              = 128000
llama_model_loader: - kv  29:                tokenizer.ggml.eos_token_id u32              = 128009
llama_model_loader: - kv  30:                tokenizer.ggml.eom_token_id u32              = 128008
llama_model_loader: - kv  31:                tokenizer.ggml.eot_token_id u32              = 128009
llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {{- bos_token }}\n{%- if custom_tools ...
llama_model_loader: - kv  33:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  154 tensors
llama_model_loader: - type q5_K:  409 tensors
llama_model_loader: - type q6_K:   67 tensors
llm_load_vocab: special tokens cache size = 256
llm_load_vocab: token to piece cache size = 0.7999 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = deci
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 128256
llm_load_print_meta: n_merges         = 280147
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 131072
llm_load_print_meta: n_embd           = 8192
llm_load_print_meta: n_layer          = 80
llm_load_print_meta: n_head           = [64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 0, 64, 64, 0, 64, 0, 64, 64, 0, 64, 64, 0, 0, 64, 0, 0, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64, 64]
llm_load_print_meta: n_head_kv        = [8, 4, 8, 8, 8, 2, 2, 1, 1, 2, 2, 0, 1, 2, 2, 0, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 0, 8, 8, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8]
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = [8, 16, 8, 8, 8, 32, 32, 64, 64, 32, 32, 0, 64, 32, 32, 0, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 0, 8, 8, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8]
llm_load_print_meta: n_embd_k_gqa     = [1024, 512, 1024, 1024, 1024, 256, 256, 128, 128, 256, 256, 0, 128, 256, 256, 0, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 0, 1024, 1024, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024]
llm_load_print_meta: n_embd_v_gqa     = [1024, 512, 1024, 1024, 1024, 256, 256, 128, 128, 256, 256, 0, 128, 256, 256, 0, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 0, 1024, 1024, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 0, 0, 1024, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024, 1024]
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = [7168, 14336, 28672, 28672, 28672, 14336, 14336, 14336, 14336, 14336, 14336, 14336, 14336, 14336, 14336, 7168, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 14336, 28672, 28672, 14336, 28672, 14336, 14336, 14336, 7168, 7168, 28672, 7168, 7168, 7168, 28672, 7168, 7168, 7168, 7168, 7168, 7168, 7168, 7168, 7168, 7168, 7168, 7168, 7168, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672, 28672]
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 131072
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = 70B
llm_load_print_meta: model ftype      = Q5_K - Medium
llm_load_print_meta: model params     = 51.50 B
llm_load_print_meta: model size       = 33.95 GiB (5.66 BPW) 
llm_load_print_meta: general.name     = Nvidia Llama 3_1 Nemotron 51B Instruct
llm_load_print_meta: BOS token        = 128000 '<|begin_of_text|>'
llm_load_print_meta: EOS token        = 128009 '<|eot_id|>'
llm_load_print_meta: EOT token        = 128009 '<|eot_id|>'
llm_load_print_meta: EOM token        = 128008 '<|eom_id|>'
llm_load_print_meta: LF token         = 128 'Ä'
llm_load_print_meta: EOG token        = 128008 '<|eom_id|>'
llm_load_print_meta: EOG token        = 128009 '<|eot_id|>'
llm_load_print_meta: max token length = 256
llm_load_tensors: offloading 80 repeating layers to GPU
llm_load_tensors: offloading output layer to GPU
llm_load_tensors: offloaded 81/81 layers to GPU
llm_load_tensors:   CPU_Mapped model buffer size =   688.88 MiB
llm_load_tensors:        CUDA0 model buffer size = 20036.81 MiB
llm_load_tensors:        CUDA1 model buffer size = 14042.92 MiB
...................................................................................................
llama_new_context_with_model: n_seq_max     = 1
llama_new_context_with_model: n_ctx         = 10016
llama_new_context_with_model: n_ctx_per_seq = 10016
llama_new_context_with_model: n_batch       = 2048
llama_new_context_with_model: n_ubatch      = 512
llama_new_context_with_model: flash_attn    = 0
llama_new_context_with_model: freq_base     = 10000.0
llama_new_context_with_model: freq_scale    = 1
llama_new_context_with_model: n_ctx_per_seq (10016) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
llama_kv_cache_init: kv_size = 10016, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 80, can_shift = 1
llama_kv_cache_init:      CUDA0 KV buffer size =  1227.55 MiB
llama_kv_cache_init:      CUDA1 KV buffer size =   586.88 MiB
llama_new_context_with_model: KV self size  = 1814.42 MiB, K (f16):  907.21 MiB, V (f16):  907.21 MiB
llama_new_context_with_model:  CUDA_Host  output buffer size =     0.49 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =  1335.56 MiB
llama_new_context_with_model:      CUDA1 compute buffer size =  1335.57 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    35.57 MiB
llama_new_context_with_model: graph nodes  = 2014
llama_new_context_with_model: graph splits = 3
common_init_from_params: setting dry_penalty_last_n to ctx_size = 10016
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 10016
main: model loaded
main: chat template, chat_template: (built-in), example_format: '<|start_header_id|>system<|end_header_id|>

You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>

Hello<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Hi there<|eot_id|><|start_header_id|>user<|end_header_id|>

How are you?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

'
main: server is listening on http://0.0.0.0:9090 - starting the main loop
srv  update_slots: all slots are idle

nvidia-smi confirms this asymmetric usage pattern:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.77                 Driver Version: 565.77         CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        On  |   00000000:01:00.0 Off |                  N/A |
|  0%   48C    P8             27W /  420W |   22897MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        On  |   00000000:06:00.0 Off |                  N/A |
|  0%   40C    P8             33W /  390W |   16263MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      1004      G   /usr/lib/Xorg                                   4MiB |
|    0   N/A  N/A    251610      C   .../a/llama.cpp/build/bin/llama-server      22874MiB |
|    1   N/A  N/A      1004      G   /usr/lib/Xorg                                   4MiB |
|    1   N/A  N/A    251610      C   .../a/llama.cpp/build/bin/llama-server      16240MiB |
+-----------------------------------------------------------------------------------------+

[slaren]

By default, the layers of the model are distributed among the GPUs based on their free memory. If both GPUs have the same memory, they should get each the same number of layers. This assumes that each layer is the same size, but I think that's not the case with this model. You can change the default distribution by using the -ts parameter, this should allow you to get a more even usage of the GPUs.