mirror of
https://github.com/ggml-org/whisper.cpp.git
synced 2026-08-12 22:31:22 +04:00
sync : ggml (#3962)
* hexagon: tiling, tracing and optimizations for unary ops (llama/25474) * hexagon: tile wide rows in pointwise unary ops to avoid VTCM overflow * unary: reject permuted tensors for now (not used by models) * hex-unary: replace divs with fastdiv * hex-unary: add vtcm layout and host computed kernel params * hex-unary: move fastdiv init into kernel params * hex-unary: add specialized thread functions to improve generated code * hex-unary: tracing instrumentation for unary ops * hex-unary: factor out hvx kernels, streamline and remove more duplication * ggml-hexagon: fix std::min collision with Windows min macro * hex-cmake: make lto build happy --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> * ggml : process data in smaller chunks in CUDA ggml_top_k() and ggml_argsort() to reduce temporary buffers memory usage (llama/24776) * ggml : process data in smaller chunks in CUDA ggml_top_k() implementation to reduce temporary buffers memory usage * ggml : allocate tmp_dst only only once before the loop * chore : whitespaces Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * ggml : use chunked processing in both CUDA CUB top-k and argsort implementations * chore : separate argsort_f32_i32_cuda_bitonic() call from return statement Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * chore : replace ternary operators with min/max --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Johannes Gäßler <johannesg@5d6.de> * opencl: cluster-parallel decode FA for Adreno (llama/25473) * ggml-et: Initial ET backend (llama/24179) * ggml-et: Add performance logging * ggml-et: Quants helpers * ggml-et: Add MUL_MAT kernel * ggml-et: Add ROPE kernel * ggml-et: Add RMS_NORM kernel * ggml-et: Add GLU kernel * ggml-et: Add SOFT_MAX kernel * ggml-et: Add GET_ROWS kernel * ggml-et: Add CONT kernel * ggml-et: Add SET_ROWS kernel * ggml-et: Add MUL_MAT_ID kernel * ggml-et: Build et kernels as part of ggml * ggml-et: Embed kernels with fs fallback * ggml-et: Build fixes * ggml-et: Add MUL_MAT F32xF32 op * ggml_et: Add MUL_MAT_ID op * ggml-et: Disable offloading for debug * ggml-et: Refactor out block ops * ggml-et: ggml backend API changes * ggml-et: Add RESHAPE/TRANSPOSE to supported * ggml-et: Add CONT_F16 * ggml-et: Add supported ops doc * gglm-et: Initial doc * ggml-et: Remove runtime import hacks We can now import the runtime by a simple find_package(), so we can cleanup the CMakeLists.txt. * ggml-et: Fix GET_ROWS kernel Fix lost batch dimension. Also clean vibe-comments. * ggml-et: Fix SET_ROWS kernel Remove incorrect broadcasting guard. * ggml-et: Use custom instruction for fp32->fp16 * ggml-et: Vectorize set_rows fp32->fp16 * ggml-et: Fix ROPE kernel (yarn) ggml-et: fix et_logf WIP: Fix ramp WIP: fix ROPE! * ggml-et: Better sinf * ggml-et: Fix SOFT_MAX Add `max_bias` and `sink` support. * ggml-et: Fix CONT Reorder from contiguous write to read with atomic stores. * ggml-et: Fix elmap kernel Remainder handlin * ggml-et: Fix MUL_MAT MUL_MAT_ID remainders * ggml-et: Fix ET-SOC reference * ggml-et: Fix embed kernels scripts for old python This allows GGML-ET to build on pre-3.8 python. * Add sysemu support with compile time flag `-DGGML_ET_SYSEMU=ON` (llama/6) * Example using ET-Soc-1 emulator configuration Example usage: ```bash cmake -B build -DGGML_CUDA=OFF -DGGML_ET=ON -DLLAMA_CURL=OFF -DGGML_CCACHE=ON cmake --build build --config Release -j $(nproc) time ./build/bin/test-backend-ops ./build/bin/llama-server \ --model Qwen3-0.6B-Q8_0.gguf \ --alias Qwen3-0.6B-Q8_0 \ -fa 0 \ --ctx-size 1024 \ --no-warmup \ --host 127.0.0.1 \ --port 8080 ``` * build: proper dep tracking for kernels * support host using MOLD linker * initial multi core GET_ROW F32 implementation * vectorized q8 dequant * wip: cland warning clenaups and initial logging refactor * wip: message default message cleanup * chore: message cleanups * cmake cleanup * migrate to use platform provided functions * cmake back into subdir * support et_print() in kernels * fix: repair kernel building * perf: operations run async by default * debug: proper kernel dep tracking and error detection on kenrel launch * fix: kernel binary dep tracking and fixing get_rows_f32 erroring * perf: back to doing async kernel runs by default * perf: vectorize and parallel device memset * merge matmul work * misc: align allocation and enable all offload * misc: delete deadcode and respect memory limits * fix: repair tensor debug print * fix: loosen RMS_NORM op percision * feat: Q4_0 GET_ROWS * perf: FP32 MUL_MAT using TensorFMA * update limitations * perf: redue L1 load in compute_block_dot_product_q8_0 * feat: save kernel mapping (name to id) when profiling is enabled * chore: memops cleanup * perf: parallelize softmax by rows * perf: vectorize 2nd phase of softmax * perf: ban GET_ROWS from offloaded * perf: vectorize and non-atomic for eltwise ops and sub support * perf: vectorize normal rope * perf: glu runs in parallel * merge: manually merge saqib's work on kernel fixes * perf: more vectorized RoPE * perf: parallelize mul_mat_id * perf: parallelize set_rows_f32 * perf: vectorize softmax * feat: support kernel fusion and fuse RMS_NORM + MUL * fix: mostly resolve test-backend-ops failure in SOFT_MAX and ROPE * fix: bump max rope dims for gemma * feat: GeGLU and SCALE support to fully offload Gemma * perf: faster device memset * feat: get_rows supporting Q4_K and avoid cont cache coherent issues * better F32 MM * feat: NORM for ET backend * feat: SQR for ET backend * feat: UNARY on ET * feat: el_map support broadcasting for ET * feat: SUM_ROWS in ET backend * feat: more ops in ET backend * feat: WKV* operators in ET backend * perf: parallelize operators across cacheline instead of row * perf: parallelize get_rows on cacheline * wip: baseline FlashAttention for ET backend * wip: enough FA and CPY f32->f16 to run llama 3.1 fully offloaded with FA on * feat: f16 x f16 -> f32 MM using matrix engine * wip: f16 FlashAttention using matrix engine * wip: clean up * feat: barriers * perf: optimize FA_F16 in ET * perf: vectorize pack_k_for_transpose16 * perf: prefetch next loop matrix tile * perf: FlashAttention 2nd MM uses TensorFMA and optimizations * cleanup: flashattention reorg * perf: optimizations and fixes * feat: L2SCP API and make FlashAttention support DV = 256 for gemma * perf: parallelize norms beyond single row * feat: GATED_DELTA_NET support and relaxed L2_NORM requirment * feat: loosen RMS_NORM, NORM, ROPE contingous req too * feat: repeat supports brocasting on dim 0 and loosen cont check * feat: FILL and DIAG operator * feat: loosen UNARY support chcek * feat: TRI support * feat: SOLVE_TRI support * feat: basic SET support * feat: loosen CONT req * perf: fp16_to_fp32 use ASM * feat: IMROPE support * feat: PAD support * feat: global barrier * fix: view must live on the same backend as backing tensor * feat: relax CONCAT in ET backend * feat: dead simple CUMSUM implementation * feat: basic SSM_CONV support * feat: loosen CONCAT req * feat: relax GATED_DELTA_NET and add SET support proper * cleanup: cleanup LCM math * feat: SWIGLU single input * feat: SSM_SCAN support * feat: el_map supports non aligned tensors in best effort * feat: basic GROUP_NORM support * feat: loosen MUL_MAT capablities slightly * feat: loosen MUL_MAT and GET_ROWS and add IM2COL * feat: special case for softmax 1x1x1x1 * feat: loosen SOFT_MAX req in ET backend * fix: el_map unaligned acse fixes * perf: optimize zero_acc_vec in flash_attn_ext_f16_me * perf: use hart 1 for packing in MM and FA for FP16 * feat: kernel semaphore * perf: better instruction sequence in FlashAttention * fix: gated_delta_net with proper masking * perf: better parallelization for GATED_DELTA_NET * perf: parallelize SSM_CONV over nr * perf: vectorize SSM_CONV * perf: optimize MUL_MAT for q8 * feat: support Gemma 4 * fix: support multi-device * feat: broader GLU support * feat: unary ops supports view * fix: repair fp16 MM using matrix engine * perf: handle large N GEMV better * perf: better q8_0 MM * perf: better set_rows * add back deleted files * fix: repair after merge * feat: POC version of uberkernel * feat: RMS_NORM in uberkernel * feat: add more kernels into usage * chore: clean up uberkernel compilation * perf: faster flash attention * perf: opt flash attention for large seq length * feat: loosen op bounds. clamp and mean support * perf: vectorize ssm_scan * perf: slightly faster FA * perf: FlashAttention parallel MM and load * perf: fuse Q8 MM and ADD * feat: basic conv kernel for ET * softMAx_test * set_rows_f32 * get_rows and cont * testing * set_rows_exp * Junk addition * Narrowing the issue * Update flash_attn_ext_f16_me.c Focusing FA_ext_f16_me * test * Eviction updated * Detailed cache eviction debug * mulmat * removeal of `BUILD_FOR_UBERKERNEL` flag * cleaning... * fix: balance FCC0 count * feat: implement mul_mat and mul_mat_id for Q4_0 type * optimize uberkernel plan upload * add mul_mat q4 into uberkernel * enable gating flush to just uberkernel * update docs for ET * update op support for ET * et-backend: optimize Q4_0 and Q8_0 mul_mat_id row accumulations * et-backend: specialize mul_mat_id kernels for Q4_0 and Q8_0 * et-backend: fix RoPE YaRN corr_dim formula and handle degenerate inputs * test-backend-ops: add DeepSeek-V2-Lite RoPE test coverage * et-backend: add Q4_0 mul_mat matrix-engine kernel using TensorFMA32 * et-backend: vectorize Q4_0 matrix-engine dequantization * et-backend: support hybrid matrix/vector engine execution for Q4_0 mul_mat tail * et-backend: run partial-N tiles on matrix engine for Q4_0 mul_mat * et-backend: route Q4_0 mul_mat N < 53 to vecdot for better prefill latency * Update uberkernel.c * Update unary_f32.c * gemma 4 * bisect gemma4: enable scale_f32 only * bisect gemma4: +rms_norm_f32 * bisect gemma4: +rms_norm_mul_f32 * bisect gemma4: disable rms_norm_mul_f32 -- BREAKS OUTPUT * bisect gemma4: +rope_f32 (skip rms_norm_mul) * bisect gemma4: +el_map_f32 * bisect gemma4: +softmax_f32 * bisect gemma4: +get_rows_f32 * bisect gemma4: +glu_f32 * bisect gemma4: +mul_mat_f32 +mul_mat_f32_matrix_engine * bisect gemma4: +mul_mat_f16 +mul_mat_f16_matrix_engine * bisect gemma4: +mul_mat_Q8_0 +mul_mat_Q4_0 * bisect gemma4: +flash_attn_ext_f32 +flash_attn_ext_f16_me * bisect gemma4: +mul_mat_id_f32 * bisect gemma4: +sum_rows_f32 * bisect gemma4: +cont_f16 * bisect gemma4: +fill_f32 * bisect gemma4: +unary_f32 (all ops re-enabled except rms_norm_mul) * Update rms_norm_mul_f32.c * bisect2 gemma4 n64: +scale_f32 only * bisect2 gemma4 n64: +rms_norm_f32 +rope_f32 * bisect2 gemma4 n64: +rms_norm_mul_f32 (with ET_UBERKERNEL eviction fix) * bisect2 gemma4 n64: +el_map +get_rows +glu +softmax (skip rms_norm_mul) * bisect2 gemma4 n64: all ops enabled except rms_norm_mul * bisect2 n64: test unary+cont+fill+sum_rows (no mul_mat/flash_attn) * bisect2 n64: +mul_mat_f32 +mul_mat_f32_matrix_engine * bisect2 n64: +mul_mat_f16 +mul_mat_f16_matrix_engine * bisect2 n64: +mul_mat_Q8_0 +mul_mat_Q4_0 * bisect2 n64: +mul_mat_Q8_0 only (disable Q4_0) * bisect2 n64: +mul_mat_Q4_0 only (Q8_0 breaks) * bisect2 n64: +mul_mat_id +flash_attn_ext (skip Q8_0) * run-3: matmul + rms_norm_mul * run-4 * Revert "run-4" * run5 * changes after cleanup * cleanup before upstream * restrict changes into ET backend * move kernel embedding from Python to CMake * move uberkernel gen into CMake * apply clang format * update CMake style * update to match C and C++ style * use source ggml and quant headers instead of ET's * MROPE support * absorb view ops into same branch as none * fix bad rebase * add marty1885 to codeowners * oops * remove redundant newline * fix CI editor warnings --------- Co-authored-by: Vidas <vidas@nuolat.lt> Co-authored-by: Gianluca Guida <glguida@tlbflush.org> Co-authored-by: Gianluca Guida <gianluca@nekko.ai> Co-authored-by: ubergarm <leimgrub@gmail.com> Co-authored-by: SaqibAkram-10xE <saqib.akram@10xengineers.ai> Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai> * hexagon: improve ARGSORT performance for small tensors (llama/25512) * hex-sort: add efficient bitomic sort in hvx regs up to 1024 elements * hex-sort: fix inverted vrors * hex-sort: specialize sort functions for the common cases * hex-sort: add tracing and local context * opencl: add int8 dp4 dense and MoE prefill optimization for Adreno GPUs (llama/25537) * opencl: add int8 dp4 dense and moe GEMM * opencl: refactor --------- Co-authored-by: Li He <lih@qti.qualcomm.com> * Vulkan: route large matmuls to medium tile on Adreno (llama/24877) * [Vulkan] Fixes llama-cli breaking over longer promts sizes The llama-cli was breaking for longer promts sizes for q4_0 quantized networks. Causing due to insufficient shared memory. * Removed the un-used Adreno device * Updated matmul for small pipeline. * ggml : add GGML_OP_LIGHTNING_INDEXER that implements DeepSeek V3.2/V4 lightning indexer (llama/24231) * ggml : add GGML_OP_LIGHTNING_INDEXER that implements DeepSeek V3.2/V4 lightning indexer * ggml : remove scale parameters from lightning indexer OP, add f16 mask parameter * tests : add GGML_OP_LIGHTNING_INDEXER tests * ggml : bump RPC version * chore : check if lightning indexer input tensors are not transposed * tests : count flops instead of bandwidth in lightning indexer test * chore : add missing const * chore : whitespace * ggml : renamed variables in CPU lightning indexer implementation * ggml : fix lightning indexer mask broadcasting * tests : tests for lightning indexer mask broadcasting * chore : whitespace * llama : use GGML_OP_LIGHTNING_INDEXER in DeepSeek V3.2 and DeepSeek V4 models --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> * cuda: Don't crash when querying memory on device with no free memory. (llama/25157) If a Cuda device has no or limited available memory, the actual call to cudaMemGetInfo() itself can cause a fatal crash due to a cuda out of memory error (there is not enough memory to actually query memory) This causes an issue because we query memory for all devices at startup even if the user isn't trying to use the device for inference. Fix this by making the error non-fatal and assigning zero total/free memory to the device. This will have the downstream effect of the fit algorithm not trying to put any layers on it, which is desired outcome vs hard crashing. this also prevents crashes in cuda enabled builds when user explicitly passes '-dev none' * gguf : reject empty metadata keys (llama/24917) * sycl: add Q2_K to DMMV reorder path (llama/25064) Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * sycl: add fused top-k MoE (llama/25217) * sycl: add fused top-k MoE * sycl: address review: GGML_SYCL_ENABLE_FUSION env, move fusion dispatch to topk-moe * sycl: print GGML_SYCL_ENABLE_FUSION at startup like other env vars Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * gguf : add tensor shape accessor (llama/24405) * gguf : add tensor shape accessors * gguf : return tensor shape as const int64_t * * gguf : remove n_dims accessor, keep only gguf_get_tensor_ne * vulkan: Use native e2m1 and e4m3 conversions for mxfp4/nvfp4 (llama/25338) This uses the new VK_EXT_shader_ocp_microscaling_types extension to do fp4 type promotions, and also uses the float8 extension to do ue4m3 promotions for nvfp4. It's reasonable to assume that an implementation that supports fp4 will also support fp8, so we don't need to handle all possible combinations of support. * CUDA: refactor MMQ kernel configuration (llama/24127) * CUDA: refactor MMQ kernel configuration * fix Blackwell config * remove legacy code * metal : add Q2_0 support (llama/25419) * sycl: set fattn_vec_nthreads to 256 for Battlemage (llama/25205) Currently detects lunarlake + battlemage / xe2 and sets the value to 256. Keeps default at 128, Intel's ARC Alchemist's prefered value. * kleidiai : add SME2 f32 kernel (llama/24414) * kleidiai : add SME2 f32 kernel * enable dynamic scheduling for SME2 f32 kernel * ggml: uniformize im2col dst_type for all conv ops (llama/23660) * ggml: uniformize im2col dst_type for all conv ops * Update ggml/src/ggml.c Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * ggml : uniformize im2col casting logic across all conv ops * fix : allow im2col_f16 to accept any kernel type --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * ggml : add a set of functions for checking contiguity of inner tensor dimensions (llama/25650) Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> * vulkan/cpu: Support f16 as SET_ROWS src. (llama/25432) * vulkan/cpu: Support f16 as SET_ROWS src. This adds full support for f16 SET_ROWS (equivalent to f32) to vulkan and CPU backends, and adds more backend tests. * Set DenormPreserve 16 when supported, to try to fix failures on Intel * tune error threshold * update metal supports_op * opencl: fix a dp4a bug for devices where cl_khr_integer_dot_product is unavailable (llama/25639) * opencl: do not fail backend init on devices without cl_khr_integer_dot_product * opencl: do not call dp4 kernels when dp is unavailable --------- Co-authored-by: Li He <lih@qti.qualcomm.com> * hexagon: fix hmx-queue signal enum-narrowing problem (llama/25677) * opencl: avoid the vec path in GEMV for unaligned row stride (llama/25671) The f16 GEMV kernels take a vectorized path for ne00 >= 128 that casts the row pointers to half4 or float4. When the row stride is not aligned, the wide load becomes misaligned. On devices that require natural alignment for vector loads, the kernel reads garbage. This is the case Intel GPUs and the kernels produce incorrect results there. Adreno happpens to be byte addressable and the kernels happen to work. * opencl: handle OOB write in noshuffle GEMV kernels (odd ne01) (llama/25640) * opencl: do not use `clCreateBufferWithProperties` when targeting CL 2.x (llama/25673) * Flash Attention with XMX engine via oneDNN (llama/25222) * [SYCL] F16 (default) Flash Attention with XMX engine via oneDNN graph API; Qwen3.6-27b-Q8_0 prefill speed up x1.21 at p=512 and x4.26 at p=80k * [SYCL] Address review on FA oneDNN path. Result: llama-bench---pp512; 32% increase with fa1; llama-perplexity---0.11% difference; tested model: mradermacher/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf * PR-25222 revision v2: addressed audits * [SYCL] flash-attn oneDNN SDPA KV F16 rev 3.0: add BMG gate + multi-device sync. Narrow the scrope of this PR to Battlemage only (bmg; Xe2). Other archs (e.g., alchemist) fall back to existing FA kernel. When device_count >1, apply stream -> wait_and_throw(), validated working path for multi-gpu sync fix by @maxious. Co-authored-by: maxious <81432+maxious@users.noreply.github.com> * updated comment on bmg gate, noted the issue --------- Co-authored-by: scientist3 <scientist.3@users.noreply.github.com> Co-authored-by: hmscider <hmscider@users.noreply.github.com> Co-authored-by: maxious <81432+maxious@users.noreply.github.com> * sycl: Increase minimum buffer size for USM system allocations (llama/25525) Raise the threshold for minimum buffer size from 1 GiB to 4 GiB, based on real-world experiments of overcommitting device memory with model weights larger than available VRAM, for example Qwen3.5-35B-A3B-Q8 running on a B70. Also add a debug message to better track USM system allocations. Signed-off-by: Francois Dugast <francois.dugast@intel.com> * sycl : implement xielu op (llama/25550) * sycl : support kernel type fp16 for conv2d_dw (llama/25653) * sycl : fix get_rows Q2_K, Q4_K, Q5_K (llama/25656) * ggml: add f16 out_prod support for CPU and out_prod op for Vulkan (llama/23997) * metal: fuse snake activation (mul, sin, sqr, mul, add) (llama/25459) * metal: fuse snake activation (mul, sin, sqr, mul, add) Mirror the CUDA, Vulkan and CPU snake fusion: same matcher on the naive 5-op chain, same F32 contract on a and inv_b, same F32/F16/BF16 kernel with F32 compute. Follows the Metal backend idioms: bf16 instantiation gated behind GGML_METAL_HAS_BF16 and concurrency ranges checked on the remaining chain nodes before encoding, as done by the bin fusion. Covered by the existing backend-agnostic SNAKE_FUSE tests. * metal: absorb snake fusion into ggml_metal_op_bin Extract the matcher to ggml_metal_op_can_fuse_snake, mirroring the Vulkan naming, and dispatch the fused path from ggml_metal_op_bin. The encode loop switch is back to a single call per case. Address review from ggerganov * metal: fix indentation in ggml_metal_op_can_fuse_snake * cuda : relax tensor contiguity requirements for quantized concat (llama/25678) * cuda : relax tensor contiguity requirements for quantized concat * tests : add test cases for non-contiguous quantized concat * ggml : relax contiguity requirements for quantized concat --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> * CUDA: tighter MMQ src1 buffer size for native fp4 (llama/25613) * opencl: fix two issues on flash attention for Adreno a7x (llama/25697) * opencl: route `sub_group_shuffle_xor` to qcom ext when KHR ext is unavailable KHR `sub_group_shuffle_xor` is not defined by compiler when `cl_qcom_subgroup_shuffle` is present, causing certain FA kernels fail to build. Define the KHR shuffle_xor using the qcom extension. * opencl: skip FA kernels with mixed and quant types for A7x to avoid compiler crash * cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel) (llama/25545) * cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel) * chore : remove indentation of #pragma unroll * cuda : remove unnecessary kernel template declarations * cuda : add WARPS_PER_BLOCK and K_VECS_PER_BLOCK template parameters in lightning indexer kernels to avoid duplication of constants. * cuda : relax MMA architecture requirements to Turing in lightning indexer implementation * chore : renamed variables * chore : rename ggml_cuda_op_lightning_indexer() to ggml_cuda_lightning_indexer() * chore : TODO for AMD rocWMMA * chore : whitespace formatting * chore : another variable rename to fix problems caused by shadowing * chore : yet another rename, this time uppercased all constants * cuda : added alignment checks for Q and K tensors in lightning indexer implementation --------- Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> * opencl: exclude some moe kernels on Adreno a7x (llama/25698) * opencl: exclude Adreno A7x from using Adreno MoE kernels Some compilers for A7x devices miscompile the repack kernels, corrupting the weights and causing MoE models to generate garbage output * opencl: exclude A6x and unknown Adreno from MoE weights repack * cuda: extract Q1_0 elements via __byte_perm (llama/25628) * opencl: disable FA and MoE weights repack to work around compiler issues for Adreno 850 GPU (llama/25745) * opencl: workaround for A850 compiler compat * opencl: fix DX compiler version parsing and cleanup --------- Co-authored-by: Li He <lih@qti.qualcomm.com> * CUDA: dedup MoE gate/up activation quantization (llama/25441) * CUDA: dedup MoE gate/up activation quantization (fp4) For MoE gate/up projections the src1 activation is broadcast across the routed experts (ne11 == 1), so ids_src1 maps every one of a token's n_expert_used slots to the same physical row. The MMQ path therefore re-quantized each token's activation n_expert_used times. For fp4 (NVFP4/MXFP4) src0, quantize each unique token row once instead of once per expert. For NVFP4 a single quantize+scatter kernel (quantize_scatter_mmq_nvfp4) quantizes each token once and writes the resulting block_fp4_mmq straight to all n_expert_used slots, using an inverse token->compact-row map (build_tok2c). MXFP4, and GGML_CUDA_MOE_QUANT_GATHER=1, use a two-kernel variant: quantize unique rows then gather into the expert-sorted layout (gather_mmq_fp4_blocks). Both are bit-identical to the previous gather-then-quantize path (identical source data, deterministic per-block quantization), verified by test-backend-ops MUL_MAT_ID (type_a=nvfp4, broadcast b=1; 790/790 for the default, gather, and per-expert paths) and by coherent end-to-end generation. Set GGML_CUDA_NO_MOE_QUANT_DEDUP=1 to force the original per-expert path. Same-binary A/B on RTX 5090 (sm_120), Qwen3.6-35B-A3B-NVFP4 prefill @8192 (nsys, graphs-off; the unchanged mul_mat_q GEMM confirms stable clocks): activation-quant GPU-busy drops 61% (78.2 -> 30.4 ms) with the fused quantize+scatter, vs 33% (78.2 -> 52.8 ms) for the two-kernel gather. The fused path avoids materializing and re-reading the 8x compact buffer, writing the expert copies directly from registers. * CUDA: bounds-check token ids in build_tok2c_kernel Guard against malformed ids_src1: skip out-of-range token ids (t < 0 or t >= n_tokens) and drop entries beyond n_expert_used per token instead of writing past the token's tok2c region. No behavior change for valid MoE routing data; test-backend-ops MUL_MAT_ID 790/790. * Refactor the code based on review comments - Removed previously added kernels that were not necessary anymore\ - Added an inverse mapping from (token, slot) to compact row. Each token is quantized once and scattered to its compact rows. * Adding q8_1 support for dedup and addressing review comments * Add pragma unrolls * Remove redundant cudaMemsetAsync call * Removing follow up redundancies --------- Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com> * ggml-cuda : restore prop.integrated on HIP builds (llama/24233) PR #16308 set info.devices[id].integrated = false unconditionally for all CUDA/HIP devices as a workaround for corrupted output on Jetson Orin (#15034). On HIP/ROCm the device's real hipDeviceProp_t.integrated flag is needed: with the cached field forced to false, supports_buft() refuses CUDA host buffers on AMD APU/UMA parts, while get_type() already reads prop.integrated (#23007) — an inconsistency that breaks integrated-GPU host-buffer use on ROCm. Guard the workaround so it only applies to non-HIP (CUDA) builds and restore prop.integrated for HIP, keeping the Jetson workaround intact for CUDA. Fixes #23977 Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com> * Enable CUDA graphs on volta+turing (llama/25749) * CUDA: Support CUDA Virtual Devices (llama/25228) * support cuda virtual devices * disable NCCL path when virtual devices are used * label virtual devices in description; add GPUx2 server CI jobs * code refactor * vulkan: when using transfer queue for async copies, sync on event_wait to avoid race (llama/25229) * kleidiai: Add SME vs SME2 distinction in kernel dispatch (llama/25478) The current integration treats SME as a single capability (CPU_FEATURE_SME) with no distinction between SME(v1) and SME2. The kernels dispatched under CPU_FEATURE_SME use SME2-specific instructions, making dispatch incorrect on SME(v1)-only hardware. We introduce build-time and runtime distinction between SME and SME2, and wire SME(v1) and SME2 kernels based on actual hardware support. * hexagon: L2 cache handling rework (dirty bit tracking with lazy flushing) and more MUL_MAT updates (llama/25762) * hex-mm: fix artificial limit in the solver that restricted number of act-prep threads * hex-mm: fix warning * hex-prof: do not apply --top to the timeline report * hmx-mm: add suport for tiled act-processing to better distribute hvx work * hex-l2: add tracing for l2flush events * workqueue: redo the legacy workpool api to match hmx-queue and dma-queue * hmx-mm: fix f32 activation buffer alignmnet for nhvx=5,6,7 * hex-work: minor cleanup for work-queue apis * hex-work: further cleanup of the work-queue api * hex-l2: optimize l2flushes at the opbatch level * hex-work: remove unused mask * hex-work: no need to drop hvx ctx in the work-queue * hex-work: add explicit wakeup/suspend and make threads spin * hex-bufs: mark any non-weight tensor as compute * hex-dma: dma-queue support for alias queues and cached dma * hex-l2: track tensor aliases and delay or skip flushes as much as possible * hex-l2: simplify tensor alias handling * hex-l2: handle overlapping views as a circular list of aliases * hex-tens: add flags helper * hex-l2: add helper for marking tensors clearn/dirty * hex-l2: mark binary and rope outputs as l2-clean and keep the rest as is for now * hex-l2: proper support for handling all tensor overlap scenarios * hex-trace: instrument matmul init code and cleanup trace checks * hex-thread: introduce dedicated main thread with explicit stack and priority * hex-l2: track dirty state as bitmap and introduce threaded flush * hex-trace: remove redundant checks for ctx != null * hex-l2: allocate entire context as one buffer and l2fetch it after big flushes * hex-l2: disable tensor clearing in binary and rope for now seems to cause issues with fusion * hmx-mm: update act proc to use fastdivs and fix DMA overflow * hmx-mm: make MUL_MAT_ID kernels robust to multi-chunk cases (start_row>0) * hex-queue: remove obsolete queue interfaces and flush hmx-queue at the end of the op-batch * hex-queue: dont use early wakeup for small op-batches * hex-tensors: properly cap max_tensors in op-batches and dirty_map * hex-l2: make sure threaded l2flush does proper rounding * hex-l2: factor out htp_tensor_flush for reuse (if needed) * hex-l2: optimize tensor flushes by coalescing flush-all * hex-l2: optimize multi-threaded flush * hex-drv: futureproof version checks * hexagon: fix errors and warnings on windows * hex-main: update main thread to only use dspqueue_read, dspqueue_peek is not available on some platforms * hex-main: add fallback mode for dspqueue with callbacks * hex-main: introduce fallback mode for using dspqueue callbacks for full op processing * hex-main: remove early wakeup, not helping and seems to cause some errors with certain batch sizes * hex-l2: make sure to use invalidate version of flushall * hex-l2: dont try to trace early l2flush at the start of op-batch * hex-main: remove offset_ctx that must be zero anyway * hex-hmx: fix hmx_queue_depth to use idx_write - idx_read * hex-hmx: use atomic_load for idx_read/write * hex-main: add static assert to make sure n_threads are aligned * DeepseekV4: Add fused hyper-connection ops (llama/25585) * dsv4 hc-ops * add missing files; * add cparams * update rpc version * address review comments * address review comments * docs: added a note about using OpenCl with Adreno 810 (llama/25786) * opencl: loads quants as uint for q4_K and q5_K flat mv (optimization for Adreno A7x GPUs) (llama/25780) * opencl: load quant as uint in mv_q4_k_f32_flat helps older compilers (e.g., E031.41, boosts 2x), no impact on newer compilers (e.g., E031.45 or newer) * opencl: load quant as uint in mv_q5_K_f32_flat helps older compilers (e.g., E031.41) * opencl: format --------- Co-authored-by: Li He <lih@qti.qualcomm.com> * opencl: add ABS op (llama/25115) * sycl: fix row calculation when K_QUANTS_PER_ITERATION is 1 (llama/25690) * sycl: fix incorrect row calculation when K_QUANTS_PER_ITERATION=1 Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * sycl: use K_QUANTS_PER_ITERATION for non-reordered Q5_K kernel This is the only Q5_K kernel that was not using KQPI. Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * sycl: add missing second half processing to reordered q5_k Error found while running GGML_SYCL_PRIORITIZE_DMMV=1 \ build/bin/test-backend-ops test -o MUL_MAT Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * sycl: fix potential off-by-one error Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * sycl: fix missing row > nrows check Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> --------- Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> * vulkan: Support Q2_0 (llama/25430) * vulkan: Support Q2_0 The backend perf tests for mat-vec-mul weren't very good at first (worse than q2_k), doubling the rows per workgroup made a big difference. * reorder * resolve merge conflict, adjust err threshold for f16->q2_0 set_rows * ggml-blas: default hadamard mul_mat to cpu routine (llama/25710) Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> * ggml : bump version to 0.17.0 (ggml/1568) * opencl: transpose q4_K noshuffle scales for coalesced reads (llama/25805) * opencl: read/write MoE dp4a activation tiles to local memory as 128-bit (vectorized LD/ST perf opt) for Adreno GPUs (llama/25810) * opencl: read MoE dp4a activation tile as 128-bit local loads * opencl: vectorize MoE dp4a activation staging as 128-bit loads * opencl: load and use `kernel_gemm_moe_q6_k_f32_ns` from bin kernel lib (llama/25797) * opencl: Support broadcast for Adreno MUL_MAT and honor `view_offs` for Adreno Q8_0 MUL_MAT for llama-server multi-stream (llama/25910) * opencl: handle broadcast for adreno gemm/gemv_noshuffle * opencl: honor view_offs for adreno noshuffle gemm/gemv * opencl: general GEMM/GEMV support broadcast * opencl: remove unnecessary tests * opencl: remove unnecessary comments --------- Co-authored-by: Li He <lih@qti.qualcomm.com> * hexagon: add CLAMP op (llama/25934) * CUDA: vectorize same-type get_rows with int4 copy (llama/25929) k_get_rows_float did a scalar one-element-per-thread copy and recomputed the row-invariant work (index load, fast_div_modulo, src/dst row pointers) for every element. Hoist that out of the per-element loop, and add a vectorized path (k_get_rows_float_vec) that copies one int4 (16 B) per thread for the contiguous same-type (no-cast) case. The vectorized path is gated at compile time (is_same<src0_t, dst_t>) and at runtime on 16-byte alignment of the base pointers and all row strides and on ne00 % VEC == 0. Vectorizing divides the block count by VEC, so a small single-row gather can drop below the device CU count and regress; an occupancy gate keeps those on the block-rich scalar path. On Strix Halo (gfx1151) the DeltaNet recurrent-state gather (ne00=524288) drops 18.6us -> 13.0us (rocprofv3 HW timestamps), faster than the Vulkan backend, with no regression on the small conv-state gather; total get_rows -27%. test-backend-ops GET_ROWS passes (47/47). Assisted-by: Claude Opus 4.8 * ggml-openvino: Add GGML_BACKEND_DL_IMPL invocation for OpenVINO backend (llama/25795) This adds the missing `GGML_BACKEND_DL_IMPL()` macro invocation, that other backends have. Fixes #25586 for me * vulkan: Refactor vk_queue to use per-instance mutexes and unique handles (llama/23570) * Refactor vk_queue to use per-instance mutexes and unique handles * integrates VK_KHR_internally_synchronized_queues, abstracting the queue submission into a polymorphic interface that completely bypasses host-side mutex locking when driver-side synchronization is supported * fix compilation error * fix duplicate pNext chain for VkPhysicalDeviceInternallySynchronizedQueuesFeaturesKHR * add fallback defines for VK_KHR_internally_synchronized_queues * add null checks for queues in vk_device_struct destructor * use unique_ptr for outer queues to enforce exclusive ownership and optimize lifetime * use static constexpr for eInternallySynchronizedKHR * add lock guard to ggml_vk_create_aliased_queue for thread safety * initialize sync_query_features.internallySynchronizedQueues to VK_FALSE * reuse sync_query_features for internallySynchronizedQueues and simplify chaining * refactor internallySynchronizedQueues detection * fix internallySynchronizedQueues query guard * use eInternallySynchronizedKHR constant * fix self-referential alias for eInternallySynchronizedKHR * use macro for eInternallySynchronizedKHR fallback * fix internallySynchronizedQueues query timing in ggml-vulkan.cpp to prevent device creation mismatch * reset sync_query_features.pNext before reusing in device creation chain, also removed the redundant second probe call * refactor internally synchronized queues detection to use chained feature query and avoid redundant API calls * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * rename sync_enable_features to internally_synchronized_queues_features * queue_flags is still computed before has_internally_synchronized_queues is set * fix trailing whitespace * replace eInternallySynchronizedKHR macro with static constexpr * preserve source queue semantics in single-queue aliased transfer queue * vulkan: fix cmd_pool access via pointer for compute_queue unique_ptr * vulkan: lock queue during debug label emission when not internally synchronized --------- Co-authored-by: Jeff Bolz <jbolz@nvidia.com> * kleidiai : warn once when a weight type has no KleidiAI kernel (llama/25701) * cuda: add sqrt_softplus in topk-moe for dsv4 (llama/25896) * hexagon: check tensor type when reusing descriptors (llama/25968) * cuda: GET_ROWS quants (llama/25962) * cuda: add k-quant support to GET_ROWS Device-side embedding lookups require GET_ROWS to handle the k-quants used by common GGUF recipes (Q4_K_M stores token_embd as q6_K). Without it the backend rejects the op and the scheduler falls back to the host, copying the full embedding matrix back on every token in single-device graphs. Factor the super-block dequantizers out of the dequantize_block kernels in convert.cu into shared device functions in dequantize.cuh and reuse them from a new k_get_rows_kq kernel : one thread block dequantizes one (dst row, super-block) pair with the existing thread layouts, 32 threads for q4_K and 64 for the other k-quants. Covers q2_K to q6_K in get_rows_cuda and supports_op. i-quants are left as a TODO. * cuda: add i-quant support to GET_ROWS Extends the shared super-block dequantizers to the nine i-quants and reuses them from k_get_rows_kq with the 32-thread layout of the matching convert.cu kernels. supports_op gates the k-quant and i-quant path on ne0 being a multiple of QK_K, which iq4_nl does not guarantee on its own (QK4_NL sub-blocks). mxfp4 is left as a TODO. * cuda: add mxfp4 support to GET_ROWS Moves the mxfp4 dequantizer into the shared super-block helpers and reuses it from k_get_rows_kq with the 32-thread layout of the matching convert.cu kernel. mxfp4 joins the ne0 % QK_K gate in supports_op since its 32-value sub-blocks do not guarantee QK_K-aligned rows on their own. This closes GET_ROWS type coverage on CUDA: every quantized GGML type now takes the direct device path. * cuda: gate the GET_ROWS row size only for 32-value sub-block types Address review from @pwilkin: the i-quant commit replaced the return shared by the whole supported type cascade, so f16/f32/bf16/i32 and the legacy quants also inherited the ne0 % QK_K == 0 gate and any row size that is not a multiple of 256 fell back to the scheduler. Split the cascade: unconditional support is restored everywhere, the gate stays only on iq4_nl and mxfp4 whose 32-value sub-blocks do not guarantee the QK_K super-blocks the kernel iterates on. * webgpu : add CONV_2D_DW (depthwise conv2d) kernel (llama/25847) * webgpu : add CONV_2D_DW (depthwise conv2d) kernel Implement GGML_OP_CONV_2D_DW for the WebGPU backend, ported from the Vulkan backend's conv2d_dw.comp. Assisted-by: Claude Opus-4.8 * Remove unnecessary comments in webgpu support * update supported ops tables, triggered by adding webgpu CONV_2D_DW * ggml: enable PowerPC backend variants on AIX (llama/25983) * ggml: enable PowerPC backend variants on AIX Allow the PowerPC CPU backend variants to be built on AIX by extending the platform check in the CMake configuration. This reuses the existing PowerPC backend implementations without changing their behavior. Also fix a missing semicolon in the PowerPC Q0 matmul implementation. * Fix missing semicolon in sgemm.cpp * hexagon: activation ops update (llama/25974) * hex-geglu: optimized all-in-one geglu microkernel * hex-geglu: enable non-contiguous src and strided DMA * hex-act: enable non-contiguous srs and strided DMA for rest of ACT ops * hex-act: generalize GLU per-thread functions via DEFINE_GLU_PER_THREAD macro * hexagon: move UNARY_SILU and UNARY_GELU to unary-ops * hex-act: replace the generic ops_context scratchpad usage with a local htp_vtcm_layout computation per act op. --------- Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> * CUDA: Improve NVFP4 W4A4 activation quantization (llama/25730) * Squash history before conflict-resolution during rebase on master WIP commit Add 32-byte loads, restore per-block amax Use nvfp4x4 intrinsic when available Fuse per-channel amax and quantization kernels Do pointer arithmetic only once on x Remove unnecessary ternary in the load We assert on host side that ne00 is 64-aligned Add back scale-search, but optimize it with intrinsics Code cleanup Make scale in MMQ-epilogue NVFP4-specific/restrictive for now Remove unneeded include, add comment Fix trailing whitespace Guard __builtin_align__(32) struct to NVIDIA Seems like HIP doesn't have this available, see https://github.com/ggml-org/llama.cpp/actions/runs/29438651734/job/87431623001 * compiler massaging to avoid unnecessary LDCs * kvalues_mxfp4 -> kvalues_nvfp4 in quantize_mmq_nvfp4 * Always pass in src1_scale.ptr * Extract ggml_cuda_is_aligned helper * metal : add f16 type support to leaky relu (llama/25981) * CUDA: fix external compilation of q1_0 MMQ (llama/25778) * hexagon: fix Windows crash when op_poll is enabled (llama/26029) * hexagon: further improved pipeline of the core bits (L2, DMA, MM, FA) (llama/26049) * hex-l2: use dirty ranges for flushing * hex-l2: simplify range based flush logic * hex-l2: optimize dirty range scans * hex-hvx: support for reduce_max_i32 * hex-mm: optimize fused MUL_MAT+ADD to use vtcm for bias when it fits * hex-mmid: optimize mmid row-mapping generation * hex-mmid: optimize mmid row-mapping generation * hex-mmid: optimize mmid row-mapping generation (round2) * hmx-mm: optimize output proc by tiling (col-chunking) * hex-fa: start the next q dmas a bit earlier * hex-fa: prefetch Q even earlier * hvx-fa: optimize softmax to keep things in hvx registers * hex-fa: hoist const register init in softmax loop * hmx-fa: kick off next-qkv DMAs before o-proc * hmx-fa: hoist various checks out of the inner loop * hmx-fa: adjust the cost model to better balance softmax work across hvx threads * hmx-fa: overlap diag rescale build with last HMX task * hmx-fa: optimize idx update in output proc * hmx-fa: unroll the softmax loops for improved perf * hmx-fa: overlap qk-dot with softmax, double-buffer p and s tiles * hex-trace: double the default number of trace entries * hex-trace: add trace events for opbatch and buffer mgmt * hex-trace: overhaul tracing to simplify runtime event handling and support opbatch stats * hex-trace: replace ascii timeline diagram with pipeline bubbles detector * hex-trace: handle missing start/stop events * hex-dma: always log stop/start trace events even for dummy dmas * hex-scripts: fix flake warnings * opencl: do not treat NULL-mask flash attention as causal (llama/25771) * opencl: cache compiled cl_program binaries on disk (llama/26050) * HIP: remove rocWMMA FlashAttention (llama/26046) * Update ggml/src/gguf.cpp : Defined virtual keyword for destructor of gguf_writer_base (llama/25867) Without a virtual destructor, deleting a derived object through a base-class pointer only invokes the base destructor, skipping the derived one. * hexagon: partial im2col support (llama/26007) * hexagon: add IM2COL op Add Hexagon IM2COL support targeting only patch-embedding convolutions. * hexagon: im2col refactor and cleanup * hex-im2col: instrument and update im2col. * hex-im2col: add local htp_vtcm_layout computation. * opencl: fix fused RMS norm mul view offset (llama/26085) * ggml-cpu: Enable BF16 tiled gemm optimization on PowerPC (llama/26068) * ggml : adjust logic for offloading ops to weight's backend (llama/25832) * ggml : adjust logic for offloading ops to weight's backend * llama : dsv4 graph fixes * sycl(build): parallelize ocloc invocations (llama/25903) * Disable -ffast-math on HIP (llama/25495) * ggml-metal: FWHT kernel for metal backend (llama/25924) * metal fwht wip * shape guard and formatting * formatting * Formatting and typos Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com> * fix narrowing issue Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com> * cont : minor style --------- Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * sycl: fix use-after-return of the SDPA scale in the oneDNN flash-attention path (llama/25880) * sycl: fix use-after-return of the SDPA scale in the oneDNN flash-attention path The scale was uploaded with an async memcpy sourced from a stack local. On the in-order queue that copy is ordered behind the K/V staging kernels; once n_kv is large enough (>= ~26k observed on Arc Pro B70) the staging outlives the host stack frame and the copy reads recycled memory, feeding the SDPA a garbage scale. Output then collapses to a single repeated token and the KV cache is poisoned for the rest of the session. Short contexts win the race by accident, and test-backend-ops caps FLASH_ATTN_EXT at kv=1024, which is why CI never caught it. The previous device_count > 1 wait_and_throw() gate (and reverting it, PR #25741) fixes the symptom only by keeping the frame alive across the copy at the cost of a host sync on every FA call. Fix: cache one device scalar per (device, value) -- the scale is constant per model -- and upload it synchronously once. The single-device fast path (no per-call host sync) is then safe: every device-side hazard already serializes on the in-order queue. The multi-GPU conservative wait is kept unchanged. Also: - GGML_SYCL_FA_ONEDNN_MAX_KV env (0 = unlimited): optional n_kv ceiling that routes very long sequences to the native FA kernel. - test-backend-ops: FLASH_ATTN_EXT F16 cases up to kv=65536 (Qwen3.6-27B geometry hsk=hsv=256 GQA 6, and hsk=128 GQA 4), closing the kv=1024 blind spot. Note the race itself needs a live multi-op pipeline to reproduce; single-op runs pass even on broken builds. Verified on Arc Pro B70 (bmg_g31), Qwen3.6-27B Q4_K, -c 131072: output byte-identical at temp 0 to the native FA path through 32k-deep prefill, with prefill depth-flat at 820-840 t/s (vs 340-350 native at 32k depth). Assisted-by: Claude Fable 5 * sycl: handle GGML_SYCL_FA_ONEDNN_MAX_KV like the other runtime env vars and document it Review feedback on #25880: - read the variable once at backend init into g_ggml_sycl_fa_onednn_max_kv via ggml_sycl_get_env, and print it in the startup env listing (-lv 4 shows it) - document GGML_SYCL_FA_ONEDNN and GGML_SYCL_FA_ONEDNN_MAX_KV in the SYCL.md runtime table Also trim the added FLASH_ATTN_EXT cases to kv={4096,16384}: the 32768/65536 shapes exceed the legacy NMSE threshold on both the oneDNN and native kernels (long-sequence fp16 accumulation drift, present before this PR) and would fail CI for an unrelated reason. Assisted-by: Claude Fable 5 * sycl: clarify GGML_SYCL_FA_ONEDNN_MAX_KV default is disabled Assisted-by: Claude Fable 5 * sycl: state default behavior of GGML_SYCL_FA_ONEDNN_MAX_KV explicitly Assisted-by: Claude Fable 5 * Update ggml/src/ggml-sycl/fattn-onednn.cpp Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com> * sycl: write the SDPA scale from a kernel instead of caching it The per-(device, value) scale cache was a function-local static unordered_map with no synchronization, so concurrent backend instances could access and rehash it at the same time. Write the scalar with a single_task instead. The value is captured into the command, so no host memory has to outlive the call -- which is what the use-after-return fix needed in the first place. That removes the shared container, the leaked device allocation and the string key, and it also closes the remaining async-memcpy-from-a-stack-local on the first flash-attention call. Ordering does not rely on timing: the queue is created with sycl::property::queue::in_order and the dnnl stream wraps that same queue, so the write completes before the SDPA reads the scalar. The multi-GPU wait_and_throw() branch is unchanged. Also drop the <cstdlib> include, which is unused. Assisted-by: Claude Opus 5 --------- Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com> * ggml-cuda: add chunked SSD matmul for Mamba-2 prefill acceleration (llama/22675) * ggml-cuda: add chunked SSD matmul for Mamba-2 prefill acceleration * cuda: added SSD CICD fixes for CUDA / HIP / MUSA / MSVC. * ggml-cuda: review comments fixed. * ggml-cuda: Fuse M matrix materialization into pre_matmul kernel and enabled test. * ggml-cuda: test updates and fixes * ggml-cuda: test updates to remove hardcoding of tensor initialise data limits. * ggml-cuda: ssd minor review comment fixed. * ggml-cuda: ssd minor CICD fixed. * CUDA SSD: Fixes correctness by promoting s0_stride_seq to int64_t, improves memory coalescing in ssm_ssd_prepare_dt_kernel, and boosts efficiency by merging B_weighted and C_scaled; also addresses prior review comments. * cuda: fix sdata read-write race in prepare_dt fallback scan loop * vulkan: add iq4_nl support back to FA (llama/24585) * vulkan: add iq4_nl support back to FA I was originally concerned about wasting shared memory on the LUT, but it's small and unlikely to matter in practice. Also support q1_0 for non-coopmat2. Fixes #23681 * remove q1_0 FA support * ggml : set output of view src (llama/25729) * llama-graph: set_outputs to t->view_src * change set_output to GGML_ASSERT about views not being outputs * sampler : avoid views in outputs * cont : fix dist sampler * cont : consistent logits handling * ggml : set output of view src * graph : simplify set_outputs() * cont : cleanup Co-authored-by: Gaurav Garg <gaugarg@nvidia.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> Co-authored-by: Gaurav Garg <gaugarg@nvidia.com> * opencl: skip the Adreno KQ/KQV image kernels for multi-stream batches (llama/26189) The Adreno KQ/KQV image1d kernels (ggml_cl_mul_mat_kq_kqv_adreno) ignore dim 3 entirely: the sub-buffer covers only nb02*ne02 bytes and the kernel receives no ne03/ne13/nb03/nb13 arguments. With the unified KV cache, multi-sequence batches (e.g. llama-perplexity with its default -b 2048, n_seq=4, or a multi-slot llama-server) present KQ/KQV as 4D tensors with ne3 = n_stream, so every stream past the first reads the first stream's K/V and produces garbage. Flash attention masks the bug where it is enabled; devices where FA is declined (e.g. Adreno 740) hit it with default settings. Route ne03/ne13 > 1 to the general path, which handles dim 3, and honor view_offs when creating the sub-buffers (currently always 0 for tensors reaching this function, but the function would silently misread any future view). Llama-3.2-1B-Instruct Q4_0, wiki.test.raw, 8 chunks, -ngl 99: - Adreno 740, default: PPL 1817.64 -> 15.61 - Adreno 740, -fa 0: PPL 1941.64 -> 15.61 - Adreno 840, -fa 0: PPL 1943.90 -> 15.50 - single-stream (-b 512) results unchanged (15.6090) - test-backend-ops -o MUL_MAT on 740: identical before/after (909 OK, 12 pre-existing q6_K failures) * ggml-webgpu: Fix some binding alias issues to support all archs, fix recurrent-state-rollback test (llama/25931) * Add overlap glu variant to support all archs, fix recurrent-state-rollback test * format * Fix all arch overlapped ranges * format * diagnose bus error on apple ci * More testing * more testing * more targeted testing * Fix bug in alignment for > 4gb buffer offsets * Fix bug in view offsets * Try avoiding multi_buffers * not fixed yet, more logging :( * Handle edge case in set_rows * Try looking at view source * Skip deepseek32 for now and clean up trace infrastructure * simplify skipping * last cleanup * actually final cleanup * update handling of overlap * format * try skipping other failing model * add rdna3.5, and 3 to mmq configs so they can be tuned independently. (llama/26199) * RPC: add tensor_memset (llama/25912) * sycl: contiguous fast path + 32-bit index math for unary elementwise ops (llama/25946) * sycl: contiguous fast path + 32-bit index math for unary elementwise ops * sycl: use fastdiv for elementwise index math * ggml-cuda : disable MMQ on devices with less than 48 KiB shared memory (llama/26141) ggml_cuda_should_use_mmq() selects MMQ purely from the quantization type. The current MMQ configurations are designed and maintained against a minimum of 48 KiB per-block shared memory, the limit provided by NVIDIA Pascal GPUs and later. On devices that report less, no supported MMQ tile fits and mul_mat_q_switch_J() aborts when every tile size exceeds the device's per-block shared memory budget. Disable MMQ when smpbo < 48 KiB so the caller falls back to the BLAS path instead of hitting GGML_ABORT. Some current MUSA QY1 devices report only 28 KiB and are covered by this guard. Reproduced on a Moore Threads MTT S70 (arch mp_21, 28 KiB shared memory per block) with an RWKV-7 0.1B Q8_0 model: $ llama-bench -m rwkv7-g1d-0.1b-Q8_0.gguf -p 128 -n 0 J_best=0 ggml/src/ggml-cuda/template-instances/../mmq.cuh:1521: fatal error (core dumped) Only prefill (batch > 1) is affected; token generation is fine. After the fix the same device falls back to the BLAS path: Q8_0 pp128 1470.7 t/s, tg8 55.3 t/s (was: abort) FP16 unchanged Q4_K_M unchanged This matches a -DGGML_CUDA_FORCE_CUBLAS=ON build (pp128 1464.2 t/s), which confirms the fallback path is the one being taken. This is not MUSA-specific: any device with less than 48 KiB per-block shared memory is affected. Co-authored-by: KakaruHayate <KakaruHayate@users.noreply.github.com> * ggml : Fix issue with kleidiai ci and stringop overflow warning (llama/26277) Signed-off-by: Jonathan Clohessy <Jonathan.Clohessy@arm.com> * metal: fix memory unwire if model is freed without any GPU operations (llama/26082) * metal: fix memory leak if model is freed without any GPU operations * metal: run dummy work only if residency sets are used * metal: wrap function in #if defined * metal: measure system-wide wired memory in test * metal: always build regression test Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com> --------- Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com> * CUDA: add Q2_0 support (llama/25707) * ggml : bump version to 0.18.0 (ggml/1576) * sync : ggml * parakeet : update parakeet test models generation This commit updates the parakeet test models to ensure that the random values generated for mel filters are not negative. This will otherwise cause a debug assertion and fail the parakeet-test. Refs: https://github.com/ggml-org/whisper.cpp/actions/runs/30547589734/job/90887624319?pr=3962 --------- Signed-off-by: Todd Malsbary <todd.malsbary@intel.com> Signed-off-by: Francois Dugast <francois.dugast@intel.com> Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com> Signed-off-by: Aaron Teo <aaron.teo1@ibm.com> Signed-off-by: Jonathan Clohessy <Jonathan.Clohessy@arm.com> Co-authored-by: Aparna M P <aparmp@qti.qualcomm.com> Co-authored-by: Max Krasnyansky <maxk@qti.qualcomm.com> Co-authored-by: fairydreaming <166155368+fairydreaming@users.noreply.github.com> Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com> Co-authored-by: Johannes Gäßler <johannesg@5d6.de> Co-authored-by: Hongqiang Wang <wangh@qti.qualcomm.com> Co-authored-by: Martin Chang <marty1885@users.noreply.github.com> Co-authored-by: Vidas <vidas@nuolat.lt> Co-authored-by: Gianluca Guida <glguida@tlbflush.org> Co-authored-by: Gianluca Guida <gianluca@nekko.ai> Co-authored-by: ubergarm <leimgrub@gmail.com> Co-authored-by: SaqibAkram-10xE <saqib.akram@10xengineers.ai> Co-authored-by: Rehan Qasim <rehan.qasim@10xengineers.ai> Co-authored-by: Li He <lih@qti.qualcomm.com> Co-authored-by: Raman Shinde <raman.shinde15@gmail.com> Co-authored-by: cphlipot <9103367+cphlipot@users.noreply.github.com> Co-authored-by: Rohit Mahesh <74331568+rohitmahesh1@users.noreply.github.com> Co-authored-by: Todd Malsbary <todd.malsbary@intel.com> Co-authored-by: Frosty40 <newjordan@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: QuintinShaw <yx6f20@soton.ac.uk> Co-authored-by: Jeff Bolz <jbolz@nvidia.com> Co-authored-by: Pasha Khosravi <khosravipasha@users.noreply.github.com> Co-authored-by: Titaniumtown <titaniumtown@proton.me> Co-authored-by: Charles Xu <charles.xu@arm.com> Co-authored-by: JusteLeo <leonard.adamo66@gmail.com> Co-authored-by: Chyan <163109379+chyan8@users.noreply.github.com> Co-authored-by: hmscider <201289679+hmscider@users.noreply.github.com> Co-authored-by: scientist3 <scientist.3@users.noreply.github.com> Co-authored-by: hmscider <hmscider@users.noreply.github.com> Co-authored-by: maxious <81432+maxious@users.noreply.github.com> Co-authored-by: Francois Dugast <francois.dugast@intel.com> Co-authored-by: Andrew Smith <atsmith19@comcast.net> Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com> Co-authored-by: Michael Lamothe <michael.lamothe@gmail.com> Co-authored-by: Pascal <admin@serveurperso.com> Co-authored-by: leonardHONG <2695316095@qq.com> Co-authored-by: David Friehs <david@friehs.info> Co-authored-by: Pranesh Gonegandla <pranesh.iitp@gmail.com> Co-authored-by: praneshgo <227579474+praneshgo@users.noreply.github.com> Co-authored-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com> Co-authored-by: Alexander Heisler <126129661+heislera763@users.noreply.github.com> Co-authored-by: Anav Prasad <anavp@nvidia.com> Co-authored-by: Ruben Ortlam <rortlam@redhat.com> Co-authored-by: Rajendra Matcha <matcraje@qti.qualcomm.com> Co-authored-by: Aman Gupta <amangupta052@gmail.com> Co-authored-by: akleine <alb.kleine@gmx.de> Co-authored-by: Gezahegne <gezahegne.yirefu@gmail.com> Co-authored-by: Aaron Teo <aaron.teo1@ibm.com> Co-authored-by: Todor Boinovski <todorb@qti.qualcomm.com> Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com> Co-authored-by: Markus Ebner <seijikun@users.noreply.github.com> Co-authored-by: Winston Ma <winstonma@ymail.com> Co-authored-by: Kamalesh VS <76260512+kkjjkamal123@users.noreply.github.com> Co-authored-by: Wei Wang <w10493wang@163.com> Co-authored-by: m1el <m1el@ya.ru> Co-authored-by: shalinib-ibm <Shalini.Salomi.Bodapati@ibm.com> Co-authored-by: Oliver Simons <osimons@nvidia.com> Co-authored-by: Ilia Ilmer <iliailmer@users.noreply.github.com> Co-authored-by: adgup-qti <adgup@qti.qualcomm.com> Co-authored-by: kumaal <44551860+kumaal@users.noreply.github.com> Co-authored-by: Yongmin Yoo 유용민 <yymin1022@gmail.com> Co-authored-by: yzyyzyhhh <96101183+happyyzy@users.noreply.github.com> Co-authored-by: Beinsezii <39478211+Beinsezii@users.noreply.github.com> Co-authored-by: Nick Lafleur <55208706+nicklafleur@users.noreply.github.com> Co-authored-by: YiChen Lv <63285796+forforever73@users.noreply.github.com> Co-authored-by: meatposes <computerdork@verizon.net> Co-authored-by: Bhavik Sharda <10757940+BLSharda@users.noreply.github.com> Co-authored-by: Gaurav Garg <gaugarg@nvidia.com> Co-authored-by: Reese Levine <reeselevine1@gmail.com> Co-authored-by: Geramy Loveless <gloveless@jqluv.com> Co-authored-by: Kakaru <97896816+KakaruHayate@users.noreply.github.com> Co-authored-by: KakaruHayate <KakaruHayate@users.noreply.github.com> Co-authored-by: Jonathan Clohessy <jonathan.clohessy@arm.com> Co-authored-by: Niklas Wenzel <dev@nikwen.de> Co-authored-by: Daniel Bevenius <daniel.bevenius@gmail.com>
This commit is contained in:
Georgi Gerganov
co-authored by
Aparna M P
Max Krasnyansky
fairydreaming
Stanisław Szymczyk
Johannes Gäßler
Hongqiang Wang
Martin Chang
Vidas
Gianluca Guida
Gianluca Guida
ubergarm
SaqibAkram-10xE
Rehan Qasim
Li He
Raman Shinde
cphlipot
Rohit Mahesh
Todd Malsbary
Frosty40
Claude Fable 5
QuintinShaw
Jeff Bolz
Pasha Khosravi
Titaniumtown
Charles Xu
JusteLeo
Chyan
hmscider
scientist3
hmscider
maxious
Francois Dugast
Andrew Smith
Neo Zhang
Michael Lamothe
Pascal
leonardHONG
David Friehs
Pranesh Gonegandla
praneshgo
liminfei-amd
Alexander Heisler
Anav Prasad
Ruben Ortlam
Rajendra Matcha
Aman Gupta
akleine
Gezahegne
Aaron Teo
Todor Boinovski
Piotr Wilkin
Markus Ebner
Winston Ma
Kamalesh VS
Wei Wang
m1el
shalinib-ibm
Oliver Simons
Ilia Ilmer
adgup-qti
kumaal
Yongmin Yoo 유용민
yzyyzyhhh
Beinsezii
Nick Lafleur
YiChen Lv
meatposes
Bhavik Sharda
Gaurav Garg
Reese Levine
Geramy Loveless
Kakaru
KakaruHayate
Jonathan Clohessy
Niklas Wenzel
Daniel Bevenius
Aparna M P
Max Krasnyansky
fairydreaming
Stanisław Szymczyk
Johannes Gäßler
Hongqiang Wang
Martin Chang
Vidas
Gianluca Guida
Gianluca Guida
ubergarm
SaqibAkram-10xE
Rehan Qasim
Li He
Raman Shinde
cphlipot
Rohit Mahesh
Todd Malsbary
Frosty40
Claude Fable 5
QuintinShaw
Jeff Bolz
Pasha Khosravi
Titaniumtown
Charles Xu
JusteLeo
Chyan
hmscider
scientist3
hmscider
maxious
Francois Dugast
Andrew Smith
Neo Zhang
Michael Lamothe
Pascal
leonardHONG
David Friehs
Pranesh Gonegandla
praneshgo
liminfei-amd
Alexander Heisler
Anav Prasad
Ruben Ortlam
Rajendra Matcha
Aman Gupta
akleine
Gezahegne
Aaron Teo
Todor Boinovski
Piotr Wilkin
Markus Ebner
Winston Ma
Kamalesh VS
Wei Wang
m1el
shalinib-ibm
Oliver Simons
Ilia Ilmer
adgup-qti
kumaal
Yongmin Yoo 유용민
yzyyzyhhh
Beinsezii
Nick Lafleur
YiChen Lv
meatposes
Bhavik Sharda
Gaurav Garg
Reese Levine
Geramy Loveless
Kakaru
KakaruHayate
Jonathan Clohessy
Niklas Wenzel
Daniel Bevenius
parent
4523d0ce37
commit
2ca53bb45e
+3
-2
@@ -4,7 +4,7 @@ project("ggml" C CXX ASM)
|
|||||||
|
|
||||||
### GGML Version
|
### GGML Version
|
||||||
set(GGML_VERSION_MAJOR 0)
|
set(GGML_VERSION_MAJOR 0)
|
||||||
set(GGML_VERSION_MINOR 16)
|
set(GGML_VERSION_MINOR 18)
|
||||||
set(GGML_VERSION_PATCH 0)
|
set(GGML_VERSION_PATCH 0)
|
||||||
set(GGML_VERSION_BASE "${GGML_VERSION_MAJOR}.${GGML_VERSION_MINOR}.${GGML_VERSION_PATCH}")
|
set(GGML_VERSION_BASE "${GGML_VERSION_MAJOR}.${GGML_VERSION_MINOR}.${GGML_VERSION_PATCH}")
|
||||||
|
|
||||||
@@ -216,7 +216,6 @@ option(GGML_HIP "ggml: use HIP"
|
|||||||
option(GGML_HIP_GRAPHS "ggml: use HIP graph" ON)
|
option(GGML_HIP_GRAPHS "ggml: use HIP graph" ON)
|
||||||
option(GGML_HIP_RCCL "ggml: use ROCm Collective Comm. Library" OFF)
|
option(GGML_HIP_RCCL "ggml: use ROCm Collective Comm. Library" OFF)
|
||||||
option(GGML_HIP_NO_VMM "ggml: do not try to use HIP VMM" ON)
|
option(GGML_HIP_NO_VMM "ggml: do not try to use HIP VMM" ON)
|
||||||
option(GGML_HIP_ROCWMMA_FATTN "ggml: enable rocWMMA for FlashAttention" OFF)
|
|
||||||
option(GGML_HIP_MMQ_MFMA "ggml: enable MFMA MMA for CDNA in MMQ" ON)
|
option(GGML_HIP_MMQ_MFMA "ggml: enable MFMA MMA for CDNA in MMQ" ON)
|
||||||
option(GGML_HIP_EXPORT_METRICS "ggml: enable kernel perf metrics output" OFF)
|
option(GGML_HIP_EXPORT_METRICS "ggml: enable kernel perf metrics output" OFF)
|
||||||
option(GGML_MUSA_GRAPHS "ggml: use MUSA graph, experimental, unstable" OFF)
|
option(GGML_MUSA_GRAPHS "ggml: use MUSA graph, experimental, unstable" OFF)
|
||||||
@@ -257,6 +256,8 @@ set (GGML_SYCL_DEVICE_ARCH "" CACHE STRING
|
|||||||
"ggml: sycl device architecture")
|
"ggml: sycl device architecture")
|
||||||
|
|
||||||
option(GGML_OPENVINO "ggml: use OPENVINO" OFF)
|
option(GGML_OPENVINO "ggml: use OPENVINO" OFF)
|
||||||
|
option(GGML_ET "ggml: use ET backend" OFF)
|
||||||
|
option(GGML_ET_SYSEMU "ggml: use ET backend via sysemu" OFF)
|
||||||
|
|
||||||
option(GGML_OPENCL "ggml: use OpenCL" OFF)
|
option(GGML_OPENCL "ggml: use OpenCL" OFF)
|
||||||
option(GGML_OPENCL_PROFILING "ggml: use OpenCL profiling (increases overhead)" OFF)
|
option(GGML_OPENCL_PROFILING "ggml: use OpenCL profiling (increases overhead)" OFF)
|
||||||
|
|||||||
@@ -100,6 +100,7 @@ extern "C" {
|
|||||||
GGML_BACKEND_API int ggml_cpu_has_sve (void);
|
GGML_BACKEND_API int ggml_cpu_has_sve (void);
|
||||||
GGML_BACKEND_API int ggml_cpu_get_sve_cnt (void); // sve vector length in bytes
|
GGML_BACKEND_API int ggml_cpu_get_sve_cnt (void); // sve vector length in bytes
|
||||||
GGML_BACKEND_API int ggml_cpu_has_sme (void);
|
GGML_BACKEND_API int ggml_cpu_has_sme (void);
|
||||||
|
GGML_BACKEND_API int ggml_cpu_has_sme2 (void);
|
||||||
// other
|
// other
|
||||||
GGML_BACKEND_API int ggml_cpu_has_riscv_v (void);
|
GGML_BACKEND_API int ggml_cpu_has_riscv_v (void);
|
||||||
GGML_BACKEND_API int ggml_cpu_get_rvv_vlen (void); // risc-v vector length in bytes
|
GGML_BACKEND_API int ggml_cpu_get_rvv_vlen (void); // risc-v vector length in bytes
|
||||||
|
|||||||
@@ -0,0 +1,28 @@
|
|||||||
|
#pragma once
|
||||||
|
|
||||||
|
#include "ggml.h"
|
||||||
|
#include "ggml-backend.h"
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
extern "C" {
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#define GGML_ET_NAME "ET"
|
||||||
|
|
||||||
|
// backend API
|
||||||
|
GGML_BACKEND_API ggml_guid_t ggml_backend_et_guid(void);
|
||||||
|
GGML_BACKEND_API ggml_backend_t ggml_backend_et_init(size_t devidx);
|
||||||
|
|
||||||
|
GGML_BACKEND_API bool ggml_backend_is_et(ggml_backend_t backend);
|
||||||
|
GGML_BACKEND_API int ggml_backend_et_get_device_count(void);
|
||||||
|
GGML_BACKEND_API void ggml_backend_et_get_device_description(int devidx, char * description, size_t description_size);
|
||||||
|
GGML_BACKEND_API void ggml_backend_et_get_device_memory(int devidx, size_t * free, size_t * total);
|
||||||
|
|
||||||
|
GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_et_buffer_type(size_t dev_num);
|
||||||
|
GGML_BACKEND_API ggml_backend_buffer_type_t ggml_backend_et_host_buffer_type(void);
|
||||||
|
|
||||||
|
GGML_BACKEND_API ggml_backend_reg_t ggml_backend_et_reg(void);
|
||||||
|
|
||||||
|
#ifdef __cplusplus
|
||||||
|
}
|
||||||
|
#endif
|
||||||
@@ -6,12 +6,12 @@
|
|||||||
extern "C" {
|
extern "C" {
|
||||||
#endif
|
#endif
|
||||||
|
|
||||||
#define RPC_PROTO_MAJOR_VERSION 4
|
#define RPC_PROTO_MAJOR_VERSION 5
|
||||||
#define RPC_PROTO_MINOR_VERSION 0
|
#define RPC_PROTO_MINOR_VERSION 0
|
||||||
#define RPC_PROTO_PATCH_VERSION 1
|
#define RPC_PROTO_PATCH_VERSION 0
|
||||||
|
|
||||||
#ifdef __cplusplus
|
#ifdef __cplusplus
|
||||||
static_assert(GGML_OP_COUNT == 97, "GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION");
|
static_assert(GGML_OP_COUNT == 101, "GGML_OP_COUNT has changed - update RPC_PROTO_PATCH_VERSION");
|
||||||
#endif
|
#endif
|
||||||
|
|
||||||
#define GGML_RPC_MAX_SERVERS 16
|
#define GGML_RPC_MAX_SERVERS 16
|
||||||
|
|||||||
@@ -570,6 +570,10 @@ extern "C" {
|
|||||||
GGML_OP_RWKV_WKV7,
|
GGML_OP_RWKV_WKV7,
|
||||||
GGML_OP_SOLVE_TRI,
|
GGML_OP_SOLVE_TRI,
|
||||||
GGML_OP_GATED_DELTA_NET,
|
GGML_OP_GATED_DELTA_NET,
|
||||||
|
GGML_OP_LIGHTNING_INDEXER,
|
||||||
|
GGML_OP_DSV4_HC_COMB,
|
||||||
|
GGML_OP_DSV4_HC_PRE,
|
||||||
|
GGML_OP_DSV4_HC_POST,
|
||||||
|
|
||||||
GGML_OP_UNARY,
|
GGML_OP_UNARY,
|
||||||
|
|
||||||
@@ -779,6 +783,10 @@ extern "C" {
|
|||||||
GGML_API bool ggml_is_contiguous_1(const struct ggml_tensor * tensor); // contiguous for dims >= 1
|
GGML_API bool ggml_is_contiguous_1(const struct ggml_tensor * tensor); // contiguous for dims >= 1
|
||||||
GGML_API bool ggml_is_contiguous_2(const struct ggml_tensor * tensor); // contiguous for dims >= 2
|
GGML_API bool ggml_is_contiguous_2(const struct ggml_tensor * tensor); // contiguous for dims >= 2
|
||||||
|
|
||||||
|
GGML_API bool ggml_is_contiguous_to_1(const struct ggml_tensor * tensor); // contiguous for dims < 1
|
||||||
|
GGML_API bool ggml_is_contiguous_to_2(const struct ggml_tensor * tensor); // contiguous for dims < 2
|
||||||
|
GGML_API bool ggml_is_contiguous_to_3(const struct ggml_tensor * tensor); // contiguous for dims < 3
|
||||||
|
|
||||||
// returns whether the tensor elements are allocated as one contiguous block of memory (no gaps, but permutation ok)
|
// returns whether the tensor elements are allocated as one contiguous block of memory (no gaps, but permutation ok)
|
||||||
GGML_API bool ggml_is_contiguously_allocated(const struct ggml_tensor * tensor);
|
GGML_API bool ggml_is_contiguously_allocated(const struct ggml_tensor * tensor);
|
||||||
|
|
||||||
@@ -2575,6 +2583,63 @@ extern "C" {
|
|||||||
struct ggml_tensor * state,
|
struct ggml_tensor * state,
|
||||||
int64_t K);
|
int64_t K);
|
||||||
|
|
||||||
|
// DSA lightning indexer
|
||||||
|
//
|
||||||
|
// q: [n_embd_idx, n_head_idx, n_batch, ne3 ]
|
||||||
|
// k: [n_embd_idx, 1, n_kv, ne3 ]
|
||||||
|
// weights: [n_head_idx, n_batch, 1, ne3 ] !! prescaled !!
|
||||||
|
// mask: [n_kv, n_batch, 1, ne33] !! f16 !!
|
||||||
|
// res: [n_kv, n_batch, 1, ne3 ]
|
||||||
|
//
|
||||||
|
// broadcast:
|
||||||
|
// ne3 % ne33 == 0
|
||||||
|
//
|
||||||
|
GGML_API struct ggml_tensor * ggml_lightning_indexer(
|
||||||
|
struct ggml_context * ctx,
|
||||||
|
struct ggml_tensor * q,
|
||||||
|
struct ggml_tensor * k,
|
||||||
|
struct ggml_tensor * weights,
|
||||||
|
struct ggml_tensor * mask);
|
||||||
|
|
||||||
|
// DeepSeek V4 hyper-connections (ref. https://arxiv.org/pdf/2512.24880)
|
||||||
|
// In short these operations are replacements for the original residual connection (x = transformer(x) + x)
|
||||||
|
// using a richer representation through streams.
|
||||||
|
//
|
||||||
|
// hc_comb: mixes [(2 + hc)*hc, n_tokens], scale [3], base [(2 + hc)*hc]
|
||||||
|
// -> [dst_hc, src_hc, n_tokens]
|
||||||
|
// logits[dst, src, t] = mixes[2*hc + dst + hc*src, t]*scale[2]
|
||||||
|
// + base[2*hc + dst + hc*src]
|
||||||
|
// Softmax over dst, add eps, normalize over src, then repeat normalization
|
||||||
|
// over dst followed by src for iterations 1 through n_iter - 1.
|
||||||
|
GGML_API struct ggml_tensor * ggml_dsv4_hc_comb(
|
||||||
|
struct ggml_context * ctx,
|
||||||
|
struct ggml_tensor * mixes,
|
||||||
|
struct ggml_tensor * scale,
|
||||||
|
struct ggml_tensor * base,
|
||||||
|
float eps,
|
||||||
|
int32_t n_iter);
|
||||||
|
|
||||||
|
// hc_pre: x [n_embd, hc, n_tokens], weights [hc, n_tokens] -> [n_embd, n_tokens]
|
||||||
|
// result[i, t] = sum_h x[i, h, t]*weights[h, t]
|
||||||
|
//
|
||||||
|
GGML_API struct ggml_tensor * ggml_dsv4_hc_pre(
|
||||||
|
struct ggml_context * ctx,
|
||||||
|
struct ggml_tensor * x,
|
||||||
|
struct ggml_tensor * weights);
|
||||||
|
|
||||||
|
// hc_post: x [n_embd, n_tokens], residual [n_embd, hc, n_tokens],
|
||||||
|
// post [hc, n_tokens], comb [dst_hc, src_hc, n_tokens]
|
||||||
|
// -> [n_embd, hc, n_tokens]
|
||||||
|
// result[i, dst, t] = x[i, t]*post[dst, t]
|
||||||
|
// + sum_src residual[i, src, t]*comb[dst, src, t]
|
||||||
|
//
|
||||||
|
GGML_API struct ggml_tensor * ggml_dsv4_hc_post(
|
||||||
|
struct ggml_context * ctx,
|
||||||
|
struct ggml_tensor * x,
|
||||||
|
struct ggml_tensor * residual,
|
||||||
|
struct ggml_tensor * post,
|
||||||
|
struct ggml_tensor * comb);
|
||||||
|
|
||||||
// custom operators
|
// custom operators
|
||||||
|
|
||||||
typedef void (*ggml_custom1_op_t)(struct ggml_tensor * dst , const struct ggml_tensor * a, int ith, int nth, void * userdata);
|
typedef void (*ggml_custom1_op_t)(struct ggml_tensor * dst , const struct ggml_tensor * a, int ith, int nth, void * userdata);
|
||||||
|
|||||||
+7
-6
@@ -125,12 +125,13 @@ extern "C" {
|
|||||||
// get ith C string from array with given key_id
|
// get ith C string from array with given key_id
|
||||||
GGML_API const char * gguf_get_arr_str (const struct gguf_context * ctx, int64_t key_id, size_t i);
|
GGML_API const char * gguf_get_arr_str (const struct gguf_context * ctx, int64_t key_id, size_t i);
|
||||||
|
|
||||||
GGML_API int64_t gguf_get_n_tensors (const struct gguf_context * ctx);
|
GGML_API int64_t gguf_get_n_tensors (const struct gguf_context * ctx);
|
||||||
GGML_API int64_t gguf_find_tensor (const struct gguf_context * ctx, const char * name); // returns -1 if the tensor is not found
|
GGML_API int64_t gguf_find_tensor (const struct gguf_context * ctx, const char * name); // returns -1 if the tensor is not found
|
||||||
GGML_API size_t gguf_get_tensor_offset(const struct gguf_context * ctx, int64_t tensor_id);
|
GGML_API size_t gguf_get_tensor_offset(const struct gguf_context * ctx, int64_t tensor_id);
|
||||||
GGML_API const char * gguf_get_tensor_name (const struct gguf_context * ctx, int64_t tensor_id);
|
GGML_API const char * gguf_get_tensor_name (const struct gguf_context * ctx, int64_t tensor_id);
|
||||||
GGML_API enum ggml_type gguf_get_tensor_type (const struct gguf_context * ctx, int64_t tensor_id);
|
GGML_API const int64_t * gguf_get_tensor_ne (const struct gguf_context * ctx, int64_t tensor_id); // returns ne, an array of GGML_MAX_DIMS elements; ne[dim] is 1 for dim >= n_dims
|
||||||
GGML_API size_t gguf_get_tensor_size (const struct gguf_context * ctx, int64_t tensor_id);
|
GGML_API enum ggml_type gguf_get_tensor_type (const struct gguf_context * ctx, int64_t tensor_id);
|
||||||
|
GGML_API size_t gguf_get_tensor_size (const struct gguf_context * ctx, int64_t tensor_id);
|
||||||
|
|
||||||
// removes key if it exists, returns id that the key had prior to removal (-1 if it didn't exist)
|
// removes key if it exists, returns id that the key had prior to removal (-1 if it didn't exist)
|
||||||
GGML_API int64_t gguf_remove_key(struct gguf_context * ctx, const char * key);
|
GGML_API int64_t gguf_remove_key(struct gguf_context * ctx, const char * key);
|
||||||
|
|||||||
@@ -430,7 +430,7 @@ if (GGML_CPU_ALL_VARIANTS)
|
|||||||
message(FATAL_ERROR "Unsupported ARM target OS: ${CMAKE_SYSTEM_NAME}")
|
message(FATAL_ERROR "Unsupported ARM target OS: ${CMAKE_SYSTEM_NAME}")
|
||||||
endif()
|
endif()
|
||||||
elseif (GGML_SYSTEM_ARCH STREQUAL "PowerPC")
|
elseif (GGML_SYSTEM_ARCH STREQUAL "PowerPC")
|
||||||
if (CMAKE_SYSTEM_NAME MATCHES "Linux")
|
if (CMAKE_SYSTEM_NAME MATCHES "Linux|AIX")
|
||||||
ggml_add_cpu_backend_variant(power0)
|
ggml_add_cpu_backend_variant(power0)
|
||||||
ggml_add_cpu_backend_variant(power7_1 POWER7)
|
ggml_add_cpu_backend_variant(power7_1 POWER7)
|
||||||
ggml_add_cpu_backend_variant(power7_2 POWER7 VSX)
|
ggml_add_cpu_backend_variant(power7_2 POWER7 VSX)
|
||||||
@@ -473,6 +473,7 @@ endif()
|
|||||||
ggml_add_backend(BLAS)
|
ggml_add_backend(BLAS)
|
||||||
ggml_add_backend(CANN)
|
ggml_add_backend(CANN)
|
||||||
ggml_add_backend(CUDA)
|
ggml_add_backend(CUDA)
|
||||||
|
ggml_add_backend(ET)
|
||||||
ggml_add_backend(HIP)
|
ggml_add_backend(HIP)
|
||||||
ggml_add_backend(METAL)
|
ggml_add_backend(METAL)
|
||||||
ggml_add_backend(MUSA)
|
ggml_add_backend(MUSA)
|
||||||
|
|||||||
@@ -984,6 +984,11 @@ static struct ggml_backend_meta_split_state ggml_backend_meta_get_split_state(
|
|||||||
case GGML_OP_GATED_DELTA_NET: {
|
case GGML_OP_GATED_DELTA_NET: {
|
||||||
split_state = handle_gated_delta_net(src_ss);
|
split_state = handle_gated_delta_net(src_ss);
|
||||||
} break;
|
} break;
|
||||||
|
case GGML_OP_DSV4_HC_COMB:
|
||||||
|
case GGML_OP_DSV4_HC_PRE:
|
||||||
|
case GGML_OP_DSV4_HC_POST: {
|
||||||
|
split_state = handle_generic(src_ss, /*scalar_only =*/ true);
|
||||||
|
} break;
|
||||||
case GGML_OP_UNARY: {
|
case GGML_OP_UNARY: {
|
||||||
split_state = handle_generic(src_ss, /*scalar_only =*/ false);
|
split_state = handle_generic(src_ss, /*scalar_only =*/ false);
|
||||||
} break;
|
} break;
|
||||||
|
|||||||
@@ -86,6 +86,10 @@
|
|||||||
#include "ggml-openvino.h"
|
#include "ggml-openvino.h"
|
||||||
#endif
|
#endif
|
||||||
|
|
||||||
|
#ifdef GGML_USE_ET
|
||||||
|
#include "ggml-et.h"
|
||||||
|
#endif
|
||||||
|
|
||||||
namespace fs = std::filesystem;
|
namespace fs = std::filesystem;
|
||||||
|
|
||||||
static std::string path_str(const fs::path & path) {
|
static std::string path_str(const fs::path & path) {
|
||||||
@@ -161,6 +165,9 @@ struct ggml_backend_registry {
|
|||||||
#ifdef GGML_USE_OPENVINO
|
#ifdef GGML_USE_OPENVINO
|
||||||
register_backend(ggml_backend_openvino_reg());
|
register_backend(ggml_backend_openvino_reg());
|
||||||
#endif
|
#endif
|
||||||
|
#ifdef GGML_USE_ET
|
||||||
|
register_backend(ggml_backend_et_reg());
|
||||||
|
#endif
|
||||||
#ifdef GGML_USE_CPU
|
#ifdef GGML_USE_CPU
|
||||||
register_backend(ggml_backend_cpu_reg());
|
register_backend(ggml_backend_cpu_reg());
|
||||||
#endif
|
#endif
|
||||||
|
|||||||
+26
-17
@@ -906,26 +906,35 @@ static int ggml_backend_sched_backend_id_from_cur(ggml_backend_sched_t sched, st
|
|||||||
}
|
}
|
||||||
|
|
||||||
// operations with weights are preferably run on the same backend as the weights
|
// operations with weights are preferably run on the same backend as the weights
|
||||||
for (int i = 0; i < GGML_MAX_SRC; i++) {
|
// TODO: there are exceptions (see below) - not an ideal solution
|
||||||
const struct ggml_tensor * src = tensor->src[i];
|
bool allow = true;
|
||||||
if (src == NULL) {
|
|
||||||
continue;
|
// skip ROPE since the rope freqs tensor is too small to choose a backend based on it
|
||||||
}
|
allow = allow && tensor->op != GGML_OP_ROPE;
|
||||||
// skip ROPE since the rope freqs tensor is too small to choose a backend based on it
|
|
||||||
// not an ideal solution
|
// skip FLASH_ATTN_EXT since the sinks tensor is too small to choose a based based on it
|
||||||
if (tensor->op != GGML_OP_ROPE && src->buffer != NULL && src->buffer->usage == GGML_BACKEND_BUFFER_USAGE_WEIGHTS) {
|
allow = allow && tensor->op != GGML_OP_FLASH_ATTN_EXT;
|
||||||
int src_backend_id = ggml_backend_sched_backend_from_buffer(sched, src, tensor);
|
|
||||||
// check if a backend with higher prio wants to offload the op
|
if (allow) {
|
||||||
if (sched->op_offload && src_backend_id == sched->n_backends - 1 && ggml_backend_buffer_is_host(src->buffer)) {
|
for (int i = 0; i < GGML_MAX_SRC; i++) {
|
||||||
for (int b = 0; b < src_backend_id; b++) {
|
const struct ggml_tensor * src = tensor->src[i];
|
||||||
if (ggml_backend_supports_op(sched->backends[b], tensor) && ggml_backend_offload_op(sched->backends[b], tensor)) {
|
if (src == NULL) {
|
||||||
SET_CAUSE(tensor, "1.off");
|
continue;
|
||||||
return b;
|
}
|
||||||
|
if (src->buffer != NULL && src->buffer->usage == GGML_BACKEND_BUFFER_USAGE_WEIGHTS) {
|
||||||
|
int src_backend_id = ggml_backend_sched_backend_from_buffer(sched, src, tensor);
|
||||||
|
// check if a backend with higher prio wants to offload the op
|
||||||
|
if (sched->op_offload && src_backend_id == sched->n_backends - 1 && ggml_backend_buffer_is_host(src->buffer)) {
|
||||||
|
for (int b = 0; b < src_backend_id; b++) {
|
||||||
|
if (ggml_backend_supports_op(sched->backends[b], tensor) && ggml_backend_offload_op(sched->backends[b], tensor)) {
|
||||||
|
SET_CAUSE(tensor, "1.off");
|
||||||
|
return b;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
SET_CAUSE(tensor, "1.wgt%d", i);
|
||||||
|
return src_backend_id;
|
||||||
}
|
}
|
||||||
SET_CAUSE(tensor, "1.wgt%d", i);
|
|
||||||
return src_backend_id;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,3 +1,4 @@
|
|||||||
|
#include "ggml.h"
|
||||||
#include "ggml-impl.h"
|
#include "ggml-impl.h"
|
||||||
#include "ggml-blas.h"
|
#include "ggml-blas.h"
|
||||||
#include "ggml-backend-impl.h"
|
#include "ggml-backend-impl.h"
|
||||||
@@ -415,6 +416,12 @@ static bool ggml_backend_blas_device_supports_op(ggml_backend_dev_t dev, const s
|
|||||||
// TODO: find the optimal value
|
// TODO: find the optimal value
|
||||||
const int64_t min_batch = 32;
|
const int64_t min_batch = 32;
|
||||||
|
|
||||||
|
// default back to CPU fast path
|
||||||
|
// see: https://github.com/ggml-org/llama.cpp/issues/25565
|
||||||
|
if (ggml_get_op_params_i32(op, 1) == GGML_HINT_SRC0_IS_HADAMARD) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
return ggml_is_contiguous(src0) &&
|
return ggml_is_contiguous(src0) &&
|
||||||
ggml_is_contiguous(src1) &&
|
ggml_is_contiguous(src1) &&
|
||||||
src1->type == GGML_TYPE_F32 &&
|
src1->type == GGML_TYPE_F32 &&
|
||||||
|
|||||||
@@ -638,6 +638,7 @@ function(ggml_add_cpu_backend_variant_impl tag_name)
|
|||||||
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qai8dxp_qsi8cxp/
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qai8dxp_qsi8cxp/
|
||||||
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_fp32_bf16p_bf16p/
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_fp32_bf16p_bf16p/
|
||||||
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f16p_qsi4c32p/
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f16p_qsi4c32p/
|
||||||
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f32p_f32p/
|
||||||
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/)
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/)
|
||||||
|
|
||||||
set(ARCH_FLAGS_TEMP "${ARCH_FLAGS}")
|
set(ARCH_FLAGS_TEMP "${ARCH_FLAGS}")
|
||||||
@@ -677,7 +678,18 @@ function(ggml_add_cpu_backend_variant_impl tag_name)
|
|||||||
endif()
|
endif()
|
||||||
|
|
||||||
if (NOT SME_ENABLED MATCHES -1)
|
if (NOT SME_ENABLED MATCHES -1)
|
||||||
list(APPEND GGML_KLEIDIAI_SOURCES
|
list(APPEND GGML_KLEIDIAI_SME_SOURCES
|
||||||
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qai8dxp_qsi8cxp/kai_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa.c
|
||||||
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qai8dxp_qsi8cxp/kai_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa_asm.S
|
||||||
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qai8dxp_qsi8cxp/kai_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot.c
|
||||||
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qai8dxp_qsi8cxp/kai_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot_asm.S
|
||||||
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f32p_f32p/kai_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa.c
|
||||||
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f32p_f32p/kai_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa_asm.S)
|
||||||
|
set_source_files_properties(${GGML_KLEIDIAI_SME_SOURCES}
|
||||||
|
PROPERTIES COMPILE_OPTIONS "-fno-tree-vectorize;${ARCH_FLAGS_TEMP}+sve+sve2+sme")
|
||||||
|
list(APPEND GGML_CPU_SOURCES ${GGML_KLEIDIAI_SME_SOURCES})
|
||||||
|
|
||||||
|
list(APPEND GGML_KLEIDIAI_SME2_SOURCES
|
||||||
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qsi8d32p_qsi4c32p/kai_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4vlx4_1x4vl_sme2_sdot.c
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qsi8d32p_qsi4c32p/kai_matmul_clamp_f32_qsi8d32p1x4_qsi4c32p4vlx4_1x4vl_sme2_sdot.c
|
||||||
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qai8dxp_qsi8cxp/kai_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme2_mopa.c
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qai8dxp_qsi8cxp/kai_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme2_mopa.c
|
||||||
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qai8dxp_qsi8cxp/kai_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme2_mopa_asm.S
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_qai8dxp_qsi8cxp/kai_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme2_mopa_asm.S
|
||||||
@@ -687,11 +699,20 @@ function(ggml_add_cpu_backend_variant_impl tag_name)
|
|||||||
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_fp32_bf16p_bf16p/kai_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa_asm.S
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_fp32_bf16p_bf16p/kai_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa_asm.S
|
||||||
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f16p_qsi4c32p/kai_matmul_clamp_f32_f16p1vlx2_qsi4c32p4vlx2_1vlx4vl_sme2_mopa.c
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f16p_qsi4c32p/kai_matmul_clamp_f32_f16p1vlx2_qsi4c32p4vlx2_1vlx4vl_sme2_mopa.c
|
||||||
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f16p_qsi4c32p/kai_matmul_clamp_f32_f16p1vlx2_qsi4c32p4vlx2_1vlx4vl_sme2_mopa_asm.S
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f16p_qsi4c32p/kai_matmul_clamp_f32_f16p1vlx2_qsi4c32p4vlx2_1vlx4vl_sme2_mopa_asm.S
|
||||||
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f32p_f32p/kai_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa.c
|
||||||
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/matmul_clamp_f32_f32p_f32p/kai_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa_asm.S
|
||||||
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_lhs_pack_bf16p2vlx2_f32_sme.c
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_lhs_pack_bf16p2vlx2_f32_sme.c
|
||||||
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_rhs_pack_kxn_bf16p2vlx2b_f32_x32_sme.c
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_rhs_pack_kxn_bf16p2vlx2b_f32_x32_sme.c
|
||||||
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_lhs_pack_f16pmrx2_f32_neon.c
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_lhs_pack_f16pmrx2_f32_neon.c
|
||||||
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_lhs_pack_f32p2vlx1_f32_sme.c
|
||||||
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_lhs_pack_f32p2vlx1_f32_sme_asm.S
|
||||||
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme.c
|
||||||
|
${KLEIDIAI_SRC}/kai/ukernels/matmul/pack/kai_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme_asm.S
|
||||||
${KLEIDIAI_SRC}/kai/kai_common_sme_asm.S)
|
${KLEIDIAI_SRC}/kai/kai_common_sme_asm.S)
|
||||||
set(PRIVATE_ARCH_FLAGS "-fno-tree-vectorize;${PRIVATE_ARCH_FLAGS}+sve+sve2+sme2+fp16")
|
set_source_files_properties(${GGML_KLEIDIAI_SME2_SOURCES}
|
||||||
|
PROPERTIES COMPILE_OPTIONS "-fno-tree-vectorize;${ARCH_FLAGS_TEMP}+sve+sve2+sme2+fp16")
|
||||||
|
list(APPEND GGML_CPU_SOURCES ${GGML_KLEIDIAI_SME2_SOURCES})
|
||||||
|
set(PRIVATE_ARCH_FLAGS "-fno-tree-vectorize;${PRIVATE_ARCH_FLAGS}")
|
||||||
endif()
|
endif()
|
||||||
|
|
||||||
if (NOT SVE_ENABLED MATCHES -1)
|
if (NOT SVE_ENABLED MATCHES -1)
|
||||||
|
|||||||
@@ -28,6 +28,7 @@ struct aarch64_features {
|
|||||||
bool has_sve2 = false;
|
bool has_sve2 = false;
|
||||||
bool has_i8mm = false;
|
bool has_i8mm = false;
|
||||||
bool has_sme = false;
|
bool has_sme = false;
|
||||||
|
bool has_sme2 = false;
|
||||||
|
|
||||||
aarch64_features() {
|
aarch64_features() {
|
||||||
#if defined(__linux__)
|
#if defined(__linux__)
|
||||||
@@ -56,6 +57,10 @@ struct aarch64_features {
|
|||||||
has_sme = static_cast<bool>(oldp);
|
has_sme = static_cast<bool>(oldp);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (sysctlbyname("hw.optional.arm.FEAT_SME2", &oldp, &size, NULL, 0) == 0) {
|
||||||
|
has_sme2 = static_cast<bool>(oldp);
|
||||||
|
}
|
||||||
|
|
||||||
// Apple apparently does not implement SVE yet
|
// Apple apparently does not implement SVE yet
|
||||||
#endif
|
#endif
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2060,6 +2060,22 @@ static void ggml_compute_forward(struct ggml_compute_params * params, struct ggm
|
|||||||
{
|
{
|
||||||
ggml_compute_forward_gated_delta_net(params, tensor);
|
ggml_compute_forward_gated_delta_net(params, tensor);
|
||||||
} break;
|
} break;
|
||||||
|
case GGML_OP_LIGHTNING_INDEXER:
|
||||||
|
{
|
||||||
|
ggml_compute_forward_lightning_indexer(params, tensor);
|
||||||
|
} break;
|
||||||
|
case GGML_OP_DSV4_HC_COMB:
|
||||||
|
{
|
||||||
|
ggml_compute_forward_dsv4_hc_comb(params, tensor);
|
||||||
|
} break;
|
||||||
|
case GGML_OP_DSV4_HC_PRE:
|
||||||
|
{
|
||||||
|
ggml_compute_forward_dsv4_hc_pre(params, tensor);
|
||||||
|
} break;
|
||||||
|
case GGML_OP_DSV4_HC_POST:
|
||||||
|
{
|
||||||
|
ggml_compute_forward_dsv4_hc_post(params, tensor);
|
||||||
|
} break;
|
||||||
case GGML_OP_MAP_CUSTOM1:
|
case GGML_OP_MAP_CUSTOM1:
|
||||||
{
|
{
|
||||||
ggml_compute_forward_map_custom1(params, tensor);
|
ggml_compute_forward_map_custom1(params, tensor);
|
||||||
@@ -2240,6 +2256,9 @@ static int ggml_get_n_tasks(struct ggml_tensor * node, int n_threads) {
|
|||||||
case GGML_OP_COUNT_EQUAL:
|
case GGML_OP_COUNT_EQUAL:
|
||||||
case GGML_OP_SOLVE_TRI:
|
case GGML_OP_SOLVE_TRI:
|
||||||
case GGML_OP_GATED_DELTA_NET:
|
case GGML_OP_GATED_DELTA_NET:
|
||||||
|
case GGML_OP_DSV4_HC_COMB:
|
||||||
|
case GGML_OP_DSV4_HC_PRE:
|
||||||
|
case GGML_OP_DSV4_HC_POST:
|
||||||
{
|
{
|
||||||
n_tasks = n_threads;
|
n_tasks = n_threads;
|
||||||
} break;
|
} break;
|
||||||
@@ -2380,6 +2399,7 @@ static int ggml_get_n_tasks(struct ggml_tensor * node, int n_threads) {
|
|||||||
case GGML_OP_FLASH_ATTN_BACK:
|
case GGML_OP_FLASH_ATTN_BACK:
|
||||||
case GGML_OP_SSM_CONV:
|
case GGML_OP_SSM_CONV:
|
||||||
case GGML_OP_SSM_SCAN:
|
case GGML_OP_SSM_SCAN:
|
||||||
|
case GGML_OP_LIGHTNING_INDEXER:
|
||||||
{
|
{
|
||||||
n_tasks = n_threads;
|
n_tasks = n_threads;
|
||||||
} break;
|
} break;
|
||||||
@@ -2854,7 +2874,14 @@ struct ggml_cplan ggml_graph_plan(
|
|||||||
} break;
|
} break;
|
||||||
case GGML_OP_OUT_PROD:
|
case GGML_OP_OUT_PROD:
|
||||||
{
|
{
|
||||||
if (ggml_is_quantized(node->src[0]->type)) {
|
if (ggml_is_quantized(node->src[0]->type) ||
|
||||||
|
node->src[0]->type == GGML_TYPE_F16) {
|
||||||
|
cur = ggml_type_size(GGML_TYPE_F32) * node->src[0]->ne[0] * n_tasks;
|
||||||
|
}
|
||||||
|
} break;
|
||||||
|
case GGML_OP_SET_ROWS:
|
||||||
|
{
|
||||||
|
if (node->src[0]->type == GGML_TYPE_F16 && node->type != GGML_TYPE_F16) {
|
||||||
cur = ggml_type_size(GGML_TYPE_F32) * node->src[0]->ne[0] * n_tasks;
|
cur = ggml_type_size(GGML_TYPE_F32) * node->src[0]->ne[0] * n_tasks;
|
||||||
}
|
}
|
||||||
} break;
|
} break;
|
||||||
@@ -2965,6 +2992,12 @@ struct ggml_cplan ggml_graph_plan(
|
|||||||
{
|
{
|
||||||
GGML_ABORT("fatal error");
|
GGML_ABORT("fatal error");
|
||||||
}
|
}
|
||||||
|
case GGML_OP_LIGHTNING_INDEXER:
|
||||||
|
{
|
||||||
|
// temp buffer for dequantizing lightning indexer keys
|
||||||
|
const int64_t ne10 = node->src[1]->ne[0];
|
||||||
|
cur += sizeof(float)*ne10*n_tasks;
|
||||||
|
} break;
|
||||||
default:
|
default:
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
@@ -3774,6 +3807,14 @@ int ggml_cpu_has_sme(void) {
|
|||||||
#endif
|
#endif
|
||||||
}
|
}
|
||||||
|
|
||||||
|
int ggml_cpu_has_sme2(void) {
|
||||||
|
#if defined(__ARM_ARCH) && defined(__ARM_FEATURE_SME2)
|
||||||
|
return 1;
|
||||||
|
#else
|
||||||
|
return 0;
|
||||||
|
#endif
|
||||||
|
}
|
||||||
|
|
||||||
void ggml_cpu_init(void) {
|
void ggml_cpu_init(void) {
|
||||||
// needed to initialize ggml_time
|
// needed to initialize ggml_time
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -462,11 +462,12 @@ static bool ggml_backend_cpu_device_supports_op(ggml_backend_dev_t dev, const st
|
|||||||
return max_bias == 0.0f;
|
return max_bias == 0.0f;
|
||||||
}
|
}
|
||||||
case GGML_OP_IM2COL_BACK:
|
case GGML_OP_IM2COL_BACK:
|
||||||
return src0->type == GGML_TYPE_F32 && src1->type == GGML_TYPE_F32;
|
return src0->type == GGML_TYPE_F32 && (src1->type == GGML_TYPE_F32 || src1->type == GGML_TYPE_F16);
|
||||||
case GGML_OP_GET_ROWS_BACK:
|
case GGML_OP_GET_ROWS_BACK:
|
||||||
return src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16;
|
return src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16;
|
||||||
case GGML_OP_OUT_PROD:
|
case GGML_OP_OUT_PROD:
|
||||||
return (src0->type == GGML_TYPE_F32 || (ggml_is_quantized(src0->type) && src0->ne[2] == src1->ne[2] && src0->ne[3] == src1->ne[3])) &&
|
return (src0->type == GGML_TYPE_F32 ||
|
||||||
|
((src0->type == GGML_TYPE_F16 || ggml_is_quantized(src0->type)) && src0->ne[2] == src1->ne[2] && src0->ne[3] == src1->ne[3])) &&
|
||||||
src1->type == GGML_TYPE_F32 && op->type == GGML_TYPE_F32;
|
src1->type == GGML_TYPE_F32 && op->type == GGML_TYPE_F32;
|
||||||
default:
|
default:
|
||||||
return true;
|
return true;
|
||||||
@@ -594,6 +595,9 @@ static ggml_backend_feature * ggml_backend_cpu_get_features(ggml_backend_reg_t r
|
|||||||
if (ggml_cpu_has_sme()) {
|
if (ggml_cpu_has_sme()) {
|
||||||
features.push_back({ "SME", "1" });
|
features.push_back({ "SME", "1" });
|
||||||
}
|
}
|
||||||
|
if (ggml_cpu_has_sme2()) {
|
||||||
|
features.push_back({ "SME2", "1" });
|
||||||
|
}
|
||||||
if (ggml_cpu_has_riscv_v()) {
|
if (ggml_cpu_has_riscv_v()) {
|
||||||
features.push_back({ "RISCV_V", "1" });
|
features.push_back({ "RISCV_V", "1" });
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -13,6 +13,8 @@
|
|||||||
#include "kai_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa.h"
|
#include "kai_matmul_clamp_f32_bf16p2vlx2_bf16p2vlx2_2vlx2vl_sme2_mopa.h"
|
||||||
#include "kai_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme2_mopa.h"
|
#include "kai_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme2_mopa.h"
|
||||||
#include "kai_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme2_dot.h"
|
#include "kai_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme2_dot.h"
|
||||||
|
#include "kai_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa.h"
|
||||||
|
#include "kai_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot.h"
|
||||||
#include "kai_matmul_clamp_f32_qai8dxp1x8_qsi8cxp4x8_1x4_neon_dotprod.h"
|
#include "kai_matmul_clamp_f32_qai8dxp1x8_qsi8cxp4x8_1x4_neon_dotprod.h"
|
||||||
#include "kai_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4x4_1x4_neon_dotprod.h"
|
#include "kai_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4x4_1x4_neon_dotprod.h"
|
||||||
#include "kai_matmul_clamp_f32_qai8dxp4x4_qsi8cxp4x4_16x4_neon_dotprod.h"
|
#include "kai_matmul_clamp_f32_qai8dxp4x4_qsi8cxp4x4_16x4_neon_dotprod.h"
|
||||||
@@ -20,14 +22,18 @@
|
|||||||
#include "kai_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p8x8_16x8_sve_i8mm.h"
|
#include "kai_matmul_clamp_f32_qsi8d32p4x8_qsi4c32p8x8_16x8_sve_i8mm.h"
|
||||||
#include "kai_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p8x8_1x8_sve_dotprod.h"
|
#include "kai_matmul_clamp_f32_qsi8d32p1x8_qsi4c32p8x8_1x8_sve_dotprod.h"
|
||||||
#include "kai_matmul_clamp_f32_f16p1vlx2_qsi4c32p4vlx2_1vlx4vl_sme2_mopa.h"
|
#include "kai_matmul_clamp_f32_f16p1vlx2_qsi4c32p4vlx2_1vlx4vl_sme2_mopa.h"
|
||||||
|
#include "kai_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa.h"
|
||||||
|
#include "kai_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa.h"
|
||||||
|
|
||||||
#include "kai_lhs_pack_bf16p2vlx2_f32_sme.h"
|
#include "kai_lhs_pack_bf16p2vlx2_f32_sme.h"
|
||||||
|
#include "kai_lhs_pack_f32p2vlx1_f32_sme.h"
|
||||||
#include "kai_lhs_quant_pack_qsi8d32p_f32.h"
|
#include "kai_lhs_quant_pack_qsi8d32p_f32.h"
|
||||||
#include "kai_lhs_quant_pack_qsi8d32p4x8sb_f32_neon.h"
|
#include "kai_lhs_quant_pack_qsi8d32p4x8sb_f32_neon.h"
|
||||||
#include "kai_lhs_quant_pack_qsi8d32p_f32_neon.h"
|
#include "kai_lhs_quant_pack_qsi8d32p_f32_neon.h"
|
||||||
#include "kai_lhs_quant_pack_qai8dxp_f32.h"
|
#include "kai_lhs_quant_pack_qai8dxp_f32.h"
|
||||||
|
|
||||||
#include "kai_rhs_pack_kxn_bf16p2vlx2b_f32_x32_sme.h"
|
#include "kai_rhs_pack_kxn_bf16p2vlx2b_f32_x32_sme.h"
|
||||||
|
#include "kai_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme.h"
|
||||||
#include "kai_rhs_pack_nxk_qsi4c32pscalef16_qsu4c32s16s0.h"
|
#include "kai_rhs_pack_nxk_qsi4c32pscalef16_qsu4c32s16s0.h"
|
||||||
#include "kai_rhs_pack_nxk_qsi4c32ps1s0scalef16_qsu4c32s16s0_neon.h"
|
#include "kai_rhs_pack_nxk_qsi4c32ps1s0scalef16_qsu4c32s16s0_neon.h"
|
||||||
#include "kai_rhs_pack_nxk_qsi8cxp_qsi8cx_neon.h"
|
#include "kai_rhs_pack_nxk_qsi8cxp_qsi8cx_neon.h"
|
||||||
@@ -356,7 +362,7 @@ static ggml_kleidiai_kernels gemm_gemv_kernels[] = {
|
|||||||
/* .packed_stride_ex = */ &rhs_stride_fn4<kai_get_rhs_packed_stride_rhs_pack_nxk_qsi4c32ps1s0scalef16_qsu4c32s16s0_neon>,
|
/* .packed_stride_ex = */ &rhs_stride_fn4<kai_get_rhs_packed_stride_rhs_pack_nxk_qsi4c32ps1s0scalef16_qsu4c32s16s0_neon>,
|
||||||
/* .pack_func_ex = */ &rhs_pack_fn12<kai_run_rhs_pack_nxk_qsi4c32ps1s0scalef16_qsu4c32s16s0_neon>,
|
/* .pack_func_ex = */ &rhs_pack_fn12<kai_run_rhs_pack_nxk_qsi4c32ps1s0scalef16_qsu4c32s16s0_neon>,
|
||||||
},
|
},
|
||||||
/* .required_cpu = */ CPU_FEATURE_SME,
|
/* .required_cpu = */ CPU_FEATURE_SME2,
|
||||||
/* .lhs_type = */ GGML_TYPE_F32,
|
/* .lhs_type = */ GGML_TYPE_F32,
|
||||||
/* .rhs_type = */ GGML_TYPE_Q4_0,
|
/* .rhs_type = */ GGML_TYPE_Q4_0,
|
||||||
/* .op_type = */ GGML_TYPE_F32,
|
/* .op_type = */ GGML_TYPE_F32,
|
||||||
@@ -409,7 +415,7 @@ static ggml_kleidiai_kernels gemm_gemv_kernels[] = {
|
|||||||
/* .packed_stride_ex = */ &rhs_stride_fn1<kai_get_rhs_packed_stride_rhs_pack_kxn_bf16p2vlx2b_f32_x32_sme>,
|
/* .packed_stride_ex = */ &rhs_stride_fn1<kai_get_rhs_packed_stride_rhs_pack_kxn_bf16p2vlx2b_f32_x32_sme>,
|
||||||
/* .pack_func_ex = */ &rhs_pack_fn13<kai_run_rhs_pack_kxn_bf16p2vlx2b_f32_x32_sme>,
|
/* .pack_func_ex = */ &rhs_pack_fn13<kai_run_rhs_pack_kxn_bf16p2vlx2b_f32_x32_sme>,
|
||||||
},
|
},
|
||||||
/* .required_cpu = */ CPU_FEATURE_SME,
|
/* .required_cpu = */ CPU_FEATURE_SME2,
|
||||||
/* .lhs_type = */ GGML_TYPE_F32,
|
/* .lhs_type = */ GGML_TYPE_F32,
|
||||||
/* .rhs_type = */ GGML_TYPE_F16,
|
/* .rhs_type = */ GGML_TYPE_F16,
|
||||||
/* .op_type = */ GGML_TYPE_F32,
|
/* .op_type = */ GGML_TYPE_F32,
|
||||||
@@ -746,6 +752,59 @@ static ggml_kleidiai_kernels gemm_gemv_kernels_q8[] = {
|
|||||||
/* .packed_stride_ex = */ &rhs_stride_fn4<kai_get_rhs_packed_stride_rhs_pack_nxk_qsi8cxp_qsi8cx_neon>,
|
/* .packed_stride_ex = */ &rhs_stride_fn4<kai_get_rhs_packed_stride_rhs_pack_nxk_qsi8cxp_qsi8cx_neon>,
|
||||||
/* .pack_func_ex = */ &rhs_pack_scale_fn12<kai_run_rhs_pack_nxk_qsi8cxp_qsi8cx_neon>,
|
/* .pack_func_ex = */ &rhs_pack_scale_fn12<kai_run_rhs_pack_nxk_qsi8cxp_qsi8cx_neon>,
|
||||||
},
|
},
|
||||||
|
/* .required_cpu = */ CPU_FEATURE_SME2,
|
||||||
|
/* .lhs_type = */ GGML_TYPE_F32,
|
||||||
|
/* .rhs_type = */ GGML_TYPE_Q8_0,
|
||||||
|
/* .op_type = */ GGML_TYPE_F32,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
/* SME GEMM (pure SME, no SME2 required) */
|
||||||
|
{
|
||||||
|
/* .get_m_step = */ kai_get_m_step_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa,
|
||||||
|
/* .get_n_step = */ kai_get_n_step_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa,
|
||||||
|
/* .get_mr = */ kai_get_mr_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa,
|
||||||
|
/* .get_nr = */ kai_get_nr_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa,
|
||||||
|
/* .get_kr = */ kai_get_kr_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa,
|
||||||
|
/* .get_sr = */ kai_get_sr_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa,
|
||||||
|
/* .get_dst_offset = */ kai_get_dst_offset_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa,
|
||||||
|
/* .get_dst_size = */ kai_get_dst_size_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa,
|
||||||
|
/* .get_lhs_offset_ex = */ &kernel_offs_fn2<kai_get_lhs_packed_offset_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa>,
|
||||||
|
/* .get_rhs_packed_offset_ex = */ &kernel_offs_fn2<kai_get_rhs_packed_offset_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa>,
|
||||||
|
/* .run_kernel_ex = */ &kernel_run_float_fn10<kai_run_matmul_clamp_f32_qai8dxp1vlx4_qsi8cxp4vlx4_1vlx4vl_sme_mopa>,
|
||||||
|
},
|
||||||
|
/* .gemm_lhs_info = */ {
|
||||||
|
/* .get_offset = */ kai_get_lhs_offset_lhs_quant_pack_qai8dxp_f32,
|
||||||
|
/* .get_packed_offset_ex = */ &lhs_offs_fn5<kai_get_lhs_packed_offset_lhs_quant_pack_qai8dxp_f32>,
|
||||||
|
/* .packed_size_ex = */ &lhs_ps_fn5<kai_get_lhs_packed_size_lhs_quant_pack_qai8dxp_f32>,
|
||||||
|
/* .pack_func_ex = */ &lhs_pack_float_fn9_no_bl<kai_run_lhs_quant_pack_qai8dxp_f32>,
|
||||||
|
},
|
||||||
|
/* SME GEMV (pure SME, no SME2 required) */
|
||||||
|
{
|
||||||
|
/* .get_m_step = */ kai_get_m_step_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot,
|
||||||
|
/* .get_n_step = */ kai_get_n_step_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot,
|
||||||
|
/* .get_mr = */ kai_get_mr_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot,
|
||||||
|
/* .get_nr = */ kai_get_nr_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot,
|
||||||
|
/* .get_kr = */ kai_get_kr_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot,
|
||||||
|
/* .get_sr = */ kai_get_sr_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot,
|
||||||
|
/* .get_dst_offset = */ kai_get_dst_offset_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot,
|
||||||
|
/* .get_dst_size = */ kai_get_dst_size_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot,
|
||||||
|
/* .get_lhs_offset_ex = */ &kernel_offs_fn2<kai_get_lhs_packed_offset_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot>,
|
||||||
|
/* .get_rhs_packed_offset_ex = */ &kernel_offs_fn2<kai_get_rhs_packed_offset_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot>,
|
||||||
|
/* .run_kernel_ex = */ &kernel_run_float_fn10<kai_run_matmul_clamp_f32_qai8dxp1x4_qsi8cxp4vlx4_1x4vl_sme_dot>,
|
||||||
|
},
|
||||||
|
/* .gemv_lhs_info = */ {
|
||||||
|
/* .get_offset = */ kai_get_lhs_offset_lhs_quant_pack_qai8dxp_f32,
|
||||||
|
/* .get_packed_offset_ex = */ &lhs_offs_fn5<kai_get_lhs_packed_offset_lhs_quant_pack_qai8dxp_f32>,
|
||||||
|
/* .packed_size_ex = */ &lhs_ps_fn5<kai_get_lhs_packed_size_lhs_quant_pack_qai8dxp_f32>,
|
||||||
|
/* .pack_func_ex = */ &lhs_pack_float_fn9_no_bl<kai_run_lhs_quant_pack_qai8dxp_f32>,
|
||||||
|
},
|
||||||
|
/* .rhs_info = */ {
|
||||||
|
/* .packed_stride = */ kai_get_rhs_packed_stride_rhs_pack_nxk_qsi8cxp_qsi8cx_neon,
|
||||||
|
/* .to_float = */ dequantize_row_qsi8cxp,
|
||||||
|
/* .packed_size_ex = */ &rhs_ps_fn5<kai_get_rhs_packed_size_rhs_pack_nxk_qsi8cxp_qsi8cx_neon>,
|
||||||
|
/* .packed_stride_ex = */ &rhs_stride_fn4<kai_get_rhs_packed_stride_rhs_pack_nxk_qsi8cxp_qsi8cx_neon>,
|
||||||
|
/* .pack_func_ex = */ &rhs_pack_scale_fn12<kai_run_rhs_pack_nxk_qsi8cxp_qsi8cx_neon>,
|
||||||
|
},
|
||||||
/* .required_cpu = */ CPU_FEATURE_SME,
|
/* .required_cpu = */ CPU_FEATURE_SME,
|
||||||
/* .lhs_type = */ GGML_TYPE_F32,
|
/* .lhs_type = */ GGML_TYPE_F32,
|
||||||
/* .rhs_type = */ GGML_TYPE_Q8_0,
|
/* .rhs_type = */ GGML_TYPE_Q8_0,
|
||||||
@@ -865,6 +924,118 @@ static ggml_kleidiai_kernels gemm_gemv_kernels_q8[] = {
|
|||||||
{ /* Sentinel */ }
|
{ /* Sentinel */ }
|
||||||
};
|
};
|
||||||
|
|
||||||
|
static ggml_kleidiai_kernels ggml_kleidiai_kernels_f32[] = {
|
||||||
|
#if defined(__ARM_FEATURE_SME)
|
||||||
|
{
|
||||||
|
/* SME2 GEMM */
|
||||||
|
{
|
||||||
|
/* .get_m_step = */ kai_get_m_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
|
||||||
|
/* .get_n_step = */ kai_get_n_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
|
||||||
|
/* .get_mr = */ kai_get_mr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
|
||||||
|
/* .get_nr = */ kai_get_nr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
|
||||||
|
/* .get_kr = */ kai_get_kr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
|
||||||
|
/* .get_sr = */ kai_get_sr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
|
||||||
|
/* .get_dst_offset = */ kai_get_dst_offset_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
|
||||||
|
/* .get_dst_size = */ kai_get_dst_size_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
|
||||||
|
/* .get_lhs_offset_ex = */ &kernel_offs_fn2<kai_get_lhs_packed_offset_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa>,
|
||||||
|
/* .get_rhs_packed_offset_ex = */ &kernel_offs_fn2<kai_get_rhs_packed_offset_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa>,
|
||||||
|
/* .run_kernel_ex = */ &kernel_run_fn10<kai_run_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa>,
|
||||||
|
},
|
||||||
|
/* .gemm_lhs_info = */ {
|
||||||
|
/* .get_offset = */ kai_get_lhs_offset_lhs_pack_f32p2vlx1_f32_sme,
|
||||||
|
/* .get_packed_offset_ex = */ &lhs_offs_fn5<kai_get_lhs_packed_offset_lhs_pack_f32p2vlx1_f32_sme>,
|
||||||
|
/* .packed_size_ex = */ &lhs_ps_fn5<kai_get_lhs_packed_size_lhs_pack_f32p2vlx1_f32_sme>,
|
||||||
|
/* .pack_func_ex = */ &lhs_pack_void_fn9<kai_run_lhs_pack_f32p2vlx1_f32_sme>,
|
||||||
|
},
|
||||||
|
/* SME GEMV */
|
||||||
|
{
|
||||||
|
/* .get_m_step = */ kai_get_m_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
|
||||||
|
/* .get_n_step = */ kai_get_n_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
|
||||||
|
/* .get_mr = */ kai_get_mr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
|
||||||
|
/* .get_nr = */ kai_get_nr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
|
||||||
|
/* .get_kr = */ kai_get_kr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
|
||||||
|
/* .get_sr = */ kai_get_sr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
|
||||||
|
/* .get_dst_offset = */ kai_get_dst_offset_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
|
||||||
|
/* .get_dst_size = */ kai_get_dst_size_matmul_clamp_f32_f32p2vlx1_f32p2vlx1biasf32_sme2_mopa,
|
||||||
|
/* .get_lhs_offset_ex = */ nullptr,
|
||||||
|
/* .get_rhs_packed_offset_ex = */ nullptr,
|
||||||
|
/* .run_kernel_ex = */ nullptr,
|
||||||
|
},
|
||||||
|
/* .gemv_lhs_info = */ {
|
||||||
|
/* .get_offset = */ kai_get_lhs_offset_lhs_pack_f32p2vlx1_f32_sme,
|
||||||
|
/* .get_packed_offset_ex = */ &lhs_offs_fn5<kai_get_lhs_packed_offset_lhs_pack_f32p2vlx1_f32_sme>,
|
||||||
|
/* .packed_size_ex = */ &lhs_ps_fn5<kai_get_lhs_packed_size_lhs_pack_f32p2vlx1_f32_sme>,
|
||||||
|
/* .pack_func_ex = */ &lhs_pack_void_fn9<kai_run_lhs_pack_f32p2vlx1_f32_sme>,
|
||||||
|
},
|
||||||
|
/* .rhs_info = */ {
|
||||||
|
/* .packed_stride = */ nullptr,
|
||||||
|
/* .to_float = */ nullptr,
|
||||||
|
/* .packed_size_ex = */ &rhs_ps_fn2<kai_get_rhs_packed_size_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme>,
|
||||||
|
/* .packed_stride_ex = */ &rhs_stride_fn1<kai_get_rhs_packed_stride_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme>,
|
||||||
|
/* .pack_func_ex = */ &rhs_pack_fn13<kai_run_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme>,
|
||||||
|
},
|
||||||
|
/* .required_cpu = */ CPU_FEATURE_SME2,
|
||||||
|
/* .lhs_type = */ GGML_TYPE_F32,
|
||||||
|
/* .rhs_type = */ GGML_TYPE_F32,
|
||||||
|
/* .op_type = */ GGML_TYPE_F32,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
/* SME GEMM */
|
||||||
|
{
|
||||||
|
/* .get_m_step = */ kai_get_m_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
|
||||||
|
/* .get_n_step = */ kai_get_n_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
|
||||||
|
/* .get_mr = */ kai_get_mr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
|
||||||
|
/* .get_nr = */ kai_get_nr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
|
||||||
|
/* .get_kr = */ kai_get_kr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
|
||||||
|
/* .get_sr = */ kai_get_sr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
|
||||||
|
/* .get_dst_offset = */ kai_get_dst_offset_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
|
||||||
|
/* .get_dst_size = */ kai_get_dst_size_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
|
||||||
|
/* .get_lhs_offset_ex = */ &kernel_offs_fn2<kai_get_lhs_packed_offset_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa>,
|
||||||
|
/* .get_rhs_packed_offset_ex = */ &kernel_offs_fn2<kai_get_rhs_packed_offset_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa>,
|
||||||
|
/* .run_kernel_ex = */ &kernel_run_fn10<kai_run_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa>,
|
||||||
|
},
|
||||||
|
/* .gemm_lhs_info = */ {
|
||||||
|
/* .get_offset = */ kai_get_lhs_offset_lhs_pack_f32p2vlx1_f32_sme,
|
||||||
|
/* .get_packed_offset_ex = */ &lhs_offs_fn5<kai_get_lhs_packed_offset_lhs_pack_f32p2vlx1_f32_sme>,
|
||||||
|
/* .packed_size_ex = */ &lhs_ps_fn5<kai_get_lhs_packed_size_lhs_pack_f32p2vlx1_f32_sme>,
|
||||||
|
/* .pack_func_ex = */ &lhs_pack_void_fn9<kai_run_lhs_pack_f32p2vlx1_f32_sme>,
|
||||||
|
},
|
||||||
|
/* SME GEMV */
|
||||||
|
{
|
||||||
|
/* .get_m_step = */ kai_get_m_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
|
||||||
|
/* .get_n_step = */ kai_get_n_step_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
|
||||||
|
/* .get_mr = */ kai_get_mr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
|
||||||
|
/* .get_nr = */ kai_get_nr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
|
||||||
|
/* .get_kr = */ kai_get_kr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
|
||||||
|
/* .get_sr = */ kai_get_sr_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
|
||||||
|
/* .get_dst_offset = */ kai_get_dst_offset_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
|
||||||
|
/* .get_dst_size = */ kai_get_dst_size_matmul_clamp_f32_f32p2vlx1_f32p2vlx1b_2vlx2vl_sme_mopa,
|
||||||
|
/* .get_lhs_offset_ex = */ nullptr,
|
||||||
|
/* .get_rhs_packed_offset_ex = */ nullptr,
|
||||||
|
/* .run_kernel_ex = */ nullptr,
|
||||||
|
},
|
||||||
|
/* .gemv_lhs_info = */ {
|
||||||
|
/* .get_offset = */ kai_get_lhs_offset_lhs_pack_f32p2vlx1_f32_sme,
|
||||||
|
/* .get_packed_offset_ex = */ &lhs_offs_fn5<kai_get_lhs_packed_offset_lhs_pack_f32p2vlx1_f32_sme>,
|
||||||
|
/* .packed_size_ex = */ &lhs_ps_fn5<kai_get_lhs_packed_size_lhs_pack_f32p2vlx1_f32_sme>,
|
||||||
|
/* .pack_func_ex = */ &lhs_pack_void_fn9<kai_run_lhs_pack_f32p2vlx1_f32_sme>,
|
||||||
|
},
|
||||||
|
/* .rhs_info = */ {
|
||||||
|
/* .packed_stride = */ nullptr,
|
||||||
|
/* .to_float = */ nullptr,
|
||||||
|
/* .packed_size_ex = */ &rhs_ps_fn2<kai_get_rhs_packed_size_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme>,
|
||||||
|
/* .packed_stride_ex = */ &rhs_stride_fn1<kai_get_rhs_packed_stride_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme>,
|
||||||
|
/* .pack_func_ex = */ &rhs_pack_fn13<kai_run_rhs_pack_nxk_f32p2vlx1biasf32_f32_f32_sme>,
|
||||||
|
},
|
||||||
|
/* .required_cpu = */ CPU_FEATURE_SME,
|
||||||
|
/* .lhs_type = */ GGML_TYPE_F32,
|
||||||
|
/* .rhs_type = */ GGML_TYPE_F32,
|
||||||
|
/* .op_type = */ GGML_TYPE_F32,
|
||||||
|
},
|
||||||
|
#endif
|
||||||
|
{ /* Sentinel */ }
|
||||||
|
};
|
||||||
|
|
||||||
ggml_kleidiai_kernels * ggml_kleidiai_select_kernels(cpu_feature cpu_features, const ggml_tensor * tensor) {
|
ggml_kleidiai_kernels * ggml_kleidiai_select_kernels(cpu_feature cpu_features, const ggml_tensor * tensor) {
|
||||||
ggml_kleidiai_kernels * kernel = nullptr;
|
ggml_kleidiai_kernels * kernel = nullptr;
|
||||||
|
|
||||||
@@ -888,12 +1059,15 @@ ggml_kleidiai_kernels * ggml_kleidiai_select_kernels(cpu_feature cpu_features, c
|
|||||||
|
|
||||||
if (tensor->src[0]->type == GGML_TYPE_Q8_0) {
|
if (tensor->src[0]->type == GGML_TYPE_Q8_0) {
|
||||||
try_table(gemm_gemv_kernels_q8);
|
try_table(gemm_gemv_kernels_q8);
|
||||||
|
} else if (tensor->src[0]->type == GGML_TYPE_F32) {
|
||||||
|
try_table(ggml_kleidiai_kernels_f32);
|
||||||
} else {
|
} else {
|
||||||
try_table(gemm_gemv_kernels);
|
try_table(gemm_gemv_kernels);
|
||||||
}
|
}
|
||||||
#else
|
#else
|
||||||
GGML_UNUSED(gemm_gemv_kernels);
|
GGML_UNUSED(gemm_gemv_kernels);
|
||||||
GGML_UNUSED(gemm_gemv_kernels_q8);
|
GGML_UNUSED(gemm_gemv_kernels_q8);
|
||||||
|
GGML_UNUSED(ggml_kleidiai_kernels_f32);
|
||||||
GGML_UNUSED(cpu_features);
|
GGML_UNUSED(cpu_features);
|
||||||
#endif
|
#endif
|
||||||
}
|
}
|
||||||
@@ -937,3 +1111,20 @@ ggml_kleidiai_kernels * ggml_kleidiai_select_kernels_q8_0(cpu_feature features)
|
|||||||
|
|
||||||
return kernels;
|
return kernels;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
ggml_kleidiai_kernels * ggml_kleidiai_select_kernels_f32(cpu_feature features) {
|
||||||
|
ggml_kleidiai_kernels * kernels = nullptr;
|
||||||
|
|
||||||
|
#if defined(__ARM_FEATURE_SME)
|
||||||
|
for (size_t i = 0; i < NELEMS(ggml_kleidiai_kernels_f32) - 1; ++i) {
|
||||||
|
if ((features & ggml_kleidiai_kernels_f32[i].required_cpu) == ggml_kleidiai_kernels_f32[i].required_cpu) {
|
||||||
|
kernels = &ggml_kleidiai_kernels_f32[i];
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
#else
|
||||||
|
GGML_UNUSED(features);
|
||||||
|
#endif
|
||||||
|
|
||||||
|
return kernels;
|
||||||
|
}
|
||||||
|
|||||||
@@ -11,7 +11,8 @@ enum cpu_feature {
|
|||||||
CPU_FEATURE_DOTPROD = 1,
|
CPU_FEATURE_DOTPROD = 1,
|
||||||
CPU_FEATURE_I8MM = 2,
|
CPU_FEATURE_I8MM = 2,
|
||||||
CPU_FEATURE_SVE = 4,
|
CPU_FEATURE_SVE = 4,
|
||||||
CPU_FEATURE_SME = 8
|
CPU_FEATURE_SME = 8,
|
||||||
|
CPU_FEATURE_SME2 = 16
|
||||||
};
|
};
|
||||||
|
|
||||||
inline cpu_feature& operator|=(cpu_feature& lhs, cpu_feature rhs) {
|
inline cpu_feature& operator|=(cpu_feature& lhs, cpu_feature rhs) {
|
||||||
@@ -55,6 +56,12 @@ struct lhs_packing_info {
|
|||||||
size_t m_idx_start, const void * lhs, size_t lhs_stride, void * lhs_packed);
|
size_t m_idx_start, const void * lhs, size_t lhs_stride, void * lhs_packed);
|
||||||
};
|
};
|
||||||
|
|
||||||
|
enum rhs_repack_mode {
|
||||||
|
RHS_REPACK_PER_KERNEL,
|
||||||
|
RHS_REPACK_SHARED,
|
||||||
|
RHS_REPACK_SINGLE_ONLY,
|
||||||
|
};
|
||||||
|
|
||||||
struct rhs_packing_info {
|
struct rhs_packing_info {
|
||||||
size_t (*packed_stride)(size_t k, size_t nr, size_t kr, size_t bl);
|
size_t (*packed_stride)(size_t k, size_t nr, size_t kr, size_t bl);
|
||||||
|
|
||||||
@@ -68,6 +75,8 @@ struct rhs_packing_info {
|
|||||||
|
|
||||||
void (*pack_func_ex)(size_t num_groups, size_t n, size_t k, size_t nr, size_t kr, size_t sr, size_t bl,
|
void (*pack_func_ex)(size_t num_groups, size_t n, size_t k, size_t nr, size_t kr, size_t sr, size_t bl,
|
||||||
size_t rhs_stride, const void * rhs, const void * bias, const void * scale, void * rhs_packed, size_t extra_bytes, const void * params);
|
size_t rhs_stride, const void * rhs, const void * bias, const void * scale, void * rhs_packed, size_t extra_bytes, const void * params);
|
||||||
|
|
||||||
|
rhs_repack_mode repack_mode = RHS_REPACK_PER_KERNEL;
|
||||||
};
|
};
|
||||||
|
|
||||||
struct ggml_kleidiai_kernels {
|
struct ggml_kleidiai_kernels {
|
||||||
@@ -88,3 +97,4 @@ struct ggml_kleidiai_kernels {
|
|||||||
ggml_kleidiai_kernels * ggml_kleidiai_select_kernels(cpu_feature cpu_features, const ggml_tensor * tensor);
|
ggml_kleidiai_kernels * ggml_kleidiai_select_kernels(cpu_feature cpu_features, const ggml_tensor * tensor);
|
||||||
ggml_kleidiai_kernels * ggml_kleidiai_select_kernels_q4_0(cpu_feature features);
|
ggml_kleidiai_kernels * ggml_kleidiai_select_kernels_q4_0(cpu_feature features);
|
||||||
ggml_kleidiai_kernels * ggml_kleidiai_select_kernels_q8_0(cpu_feature features);
|
ggml_kleidiai_kernels * ggml_kleidiai_select_kernels_q8_0(cpu_feature features);
|
||||||
|
ggml_kleidiai_kernels * ggml_kleidiai_select_kernels_f32(cpu_feature features);
|
||||||
|
|||||||
@@ -26,6 +26,9 @@
|
|||||||
#include <sys/types.h>
|
#include <sys/types.h>
|
||||||
#include <sys/stat.h>
|
#include <sys/stat.h>
|
||||||
#include <unistd.h>
|
#include <unistd.h>
|
||||||
|
#ifndef HWCAP2_SME2
|
||||||
|
#define HWCAP2_SME2 (1UL << 37)
|
||||||
|
#endif
|
||||||
#elif defined(__APPLE__)
|
#elif defined(__APPLE__)
|
||||||
#include <string_view>
|
#include <string_view>
|
||||||
#include <sys/sysctl.h>
|
#include <sys/sysctl.h>
|
||||||
@@ -60,14 +63,21 @@ struct ggml_kleidiai_context {
|
|||||||
cpu_feature features;
|
cpu_feature features;
|
||||||
ggml_kleidiai_kernels * kernels_q4;
|
ggml_kleidiai_kernels * kernels_q4;
|
||||||
ggml_kleidiai_kernels * kernels_q8;
|
ggml_kleidiai_kernels * kernels_q8;
|
||||||
|
ggml_kleidiai_kernels * kernels_f32;
|
||||||
int sme_thread_cap; // <= 0 means “SME disabled/unknown”;
|
int sme_thread_cap; // <= 0 means “SME disabled/unknown”;
|
||||||
int thread_hint; // <= 0 means “no hint”
|
int thread_hint; // <= 0 means “no hint”
|
||||||
int chunk_multiplier;
|
int chunk_multiplier;
|
||||||
} static ctx = { CPU_FEATURE_NONE, nullptr, nullptr, 0, -1, 4 };
|
} static ctx = { CPU_FEATURE_NONE, nullptr, nullptr, nullptr, 0, -1, 4 };
|
||||||
|
|
||||||
|
static inline bool is_sme_family(cpu_feature f) {
|
||||||
|
return (f & (CPU_FEATURE_SME | CPU_FEATURE_SME2)) != CPU_FEATURE_NONE;
|
||||||
|
}
|
||||||
|
|
||||||
static const char* cpu_feature_to_string(cpu_feature f) {
|
static const char* cpu_feature_to_string(cpu_feature f) {
|
||||||
if (f == CPU_FEATURE_NONE) {
|
if (f == CPU_FEATURE_NONE) {
|
||||||
return "NONE";
|
return "NONE";
|
||||||
|
} else if ((f & CPU_FEATURE_SME2) == CPU_FEATURE_SME2) {
|
||||||
|
return "SME2";
|
||||||
} else if ((f & CPU_FEATURE_SME) == CPU_FEATURE_SME) {
|
} else if ((f & CPU_FEATURE_SME) == CPU_FEATURE_SME) {
|
||||||
return "SME";
|
return "SME";
|
||||||
} else if ((f & CPU_FEATURE_SVE) == CPU_FEATURE_SVE) {
|
} else if ((f & CPU_FEATURE_SVE) == CPU_FEATURE_SVE) {
|
||||||
@@ -156,10 +166,10 @@ static size_t detect_num_smcus() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return 1;
|
return 0;
|
||||||
|
|
||||||
#else
|
#else
|
||||||
return 1;
|
return 0;
|
||||||
#endif
|
#endif
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -192,7 +202,6 @@ static void init_kleidiai_context(void) {
|
|||||||
const char *env_threads = getenv("GGML_TOTAL_THREADS");
|
const char *env_threads = getenv("GGML_TOTAL_THREADS");
|
||||||
const char *env_chunk_mult = getenv("GGML_KLEIDIAI_CHUNK_MULTIPLIER");
|
const char *env_chunk_mult = getenv("GGML_KLEIDIAI_CHUNK_MULTIPLIER");
|
||||||
|
|
||||||
const bool cpu_has_sme = ggml_cpu_has_sme();
|
|
||||||
size_t detected_smcus = 0;
|
size_t detected_smcus = 0;
|
||||||
|
|
||||||
ctx.features = (ggml_cpu_has_dotprod() ? CPU_FEATURE_DOTPROD : CPU_FEATURE_NONE) |
|
ctx.features = (ggml_cpu_has_dotprod() ? CPU_FEATURE_DOTPROD : CPU_FEATURE_NONE) |
|
||||||
@@ -216,56 +225,59 @@ static void init_kleidiai_context(void) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// SME policy:
|
// SME policy:
|
||||||
// - If CPU doesn't support SME: SME always off.
|
// - env unset => auto-detect SMCUs; enable SME only if detected > 0.
|
||||||
// - Else:
|
// - env=0 => force off.
|
||||||
// - env unset => auto-detect cores; enable if detected > 0.
|
// - env>0 => force N cores, if the binary was built with SME.
|
||||||
// - env=0 => force off.
|
|
||||||
// - env>0 => force N cores (skip detection).
|
|
||||||
int sme_cores = 0;
|
int sme_cores = 0;
|
||||||
bool sme_env_ok = false;
|
bool sme_env_ok = false;
|
||||||
bool sme_env_set = (env_sme != nullptr);
|
bool sme_env_set = (env_sme != nullptr);
|
||||||
|
|
||||||
if (!cpu_has_sme) {
|
if (sme_env_set) {
|
||||||
if (sme_env_set) {
|
bool ok = false;
|
||||||
bool ok = false;
|
int v = parse_uint_env(env_sme, "GGML_KLEIDIAI_SME", &ok);
|
||||||
int req = parse_uint_env(env_sme, "GGML_KLEIDIAI_SME", &ok);
|
sme_env_ok = ok;
|
||||||
if (ok && req > 0) {
|
|
||||||
GGML_LOG_WARN("kleidiai: GGML_KLEIDIAI_SME=%d but SME is not supported on this CPU; disabling SME\n", req);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
sme_cores = 0;
|
|
||||||
} else {
|
|
||||||
if (sme_env_set) {
|
|
||||||
bool ok = false;
|
|
||||||
int v = parse_uint_env(env_sme, "GGML_KLEIDIAI_SME", &ok);
|
|
||||||
sme_env_ok = ok;
|
|
||||||
|
|
||||||
if (!ok) {
|
if (!ok) {
|
||||||
GGML_LOG_WARN("kleidiai: GGML_KLEIDIAI_SME set but parsing failed; falling back to runtime SME-core detection\n");
|
GGML_LOG_WARN("kleidiai: GGML_KLEIDIAI_SME set but parsing failed; falling back to runtime SME-core detection\n");
|
||||||
detected_smcus = detect_num_smcus();
|
|
||||||
sme_cores = detected_smcus > 0 ? (int)detected_smcus : 0;
|
|
||||||
} else if (v == 0) {
|
|
||||||
sme_cores = 0;
|
|
||||||
} else {
|
|
||||||
sme_cores = v;
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
detected_smcus = detect_num_smcus();
|
detected_smcus = detect_num_smcus();
|
||||||
sme_cores = detected_smcus > 0 ? (int)detected_smcus : 0;
|
sme_cores = detected_smcus > 0 ? (int)detected_smcus : 0;
|
||||||
|
} else if (v == 0) {
|
||||||
|
sme_cores = 0;
|
||||||
|
} else if (!ggml_cpu_has_sme()) {
|
||||||
|
GGML_LOG_WARN("kleidiai: GGML_KLEIDIAI_SME=%d but the binary was not built with SME; disabling SME\n", v);
|
||||||
|
sme_cores = 0;
|
||||||
|
} else {
|
||||||
|
sme_cores = v;
|
||||||
}
|
}
|
||||||
|
} else {
|
||||||
|
detected_smcus = detect_num_smcus();
|
||||||
|
sme_cores = detected_smcus > 0 ? (int)detected_smcus : 0;
|
||||||
|
}
|
||||||
|
|
||||||
if (!sme_env_set && sme_cores == 0) {
|
if (!sme_env_set && ggml_cpu_has_sme() && sme_cores == 0) {
|
||||||
GGML_LOG_WARN("kleidiai: SME supported but runtime SME-core detection returned 0; falling back to NEON\n");
|
GGML_LOG_WARN("kleidiai: runtime SME-core detection returned 0; falling back to NEON\n");
|
||||||
}
|
}
|
||||||
|
|
||||||
if (sme_cores > 0) {
|
if (sme_cores > 0) {
|
||||||
ctx.features |= CPU_FEATURE_SME;
|
ctx.features |= CPU_FEATURE_SME;
|
||||||
|
#if defined(__aarch64__) && defined(__linux__)
|
||||||
|
// ARM guarantees SME2 implies SME, so only check SME2 when SME is enabled.
|
||||||
|
if (getauxval(AT_HWCAP2) & HWCAP2_SME2) {
|
||||||
|
ctx.features |= CPU_FEATURE_SME2;
|
||||||
}
|
}
|
||||||
|
#elif defined(__aarch64__) && defined(__APPLE__)
|
||||||
|
int feat_sme2 = 0;
|
||||||
|
size_t size = sizeof(feat_sme2);
|
||||||
|
if (sysctlbyname("hw.optional.arm.FEAT_SME2", &feat_sme2, &size, NULL, 0) == 0 && feat_sme2) {
|
||||||
|
ctx.features |= CPU_FEATURE_SME2;
|
||||||
|
}
|
||||||
|
#endif
|
||||||
}
|
}
|
||||||
|
|
||||||
// Kernel selection
|
// Kernel selection
|
||||||
ctx.kernels_q4 = ggml_kleidiai_select_kernels_q4_0(ctx.features);
|
ctx.kernels_q4 = ggml_kleidiai_select_kernels_q4_0(ctx.features);
|
||||||
ctx.kernels_q8 = ggml_kleidiai_select_kernels_q8_0(ctx.features);
|
ctx.kernels_q8 = ggml_kleidiai_select_kernels_q8_0(ctx.features);
|
||||||
|
ctx.kernels_f32 = ggml_kleidiai_select_kernels_f32(ctx.features);
|
||||||
|
|
||||||
if (!ctx.kernels_q4) {
|
if (!ctx.kernels_q4) {
|
||||||
GGML_LOG_INFO("kleidiai: no compatible q4 kernels found for CPU features mask %d\n", (int)ctx.features);
|
GGML_LOG_INFO("kleidiai: no compatible q4 kernels found for CPU features mask %d\n", (int)ctx.features);
|
||||||
@@ -279,13 +291,22 @@ static void init_kleidiai_context(void) {
|
|||||||
GGML_LOG_INFO("kleidiai: primary q8 kernel feature %s\n", cpu_feature_to_string(ctx.kernels_q8->required_cpu));
|
GGML_LOG_INFO("kleidiai: primary q8 kernel feature %s\n", cpu_feature_to_string(ctx.kernels_q8->required_cpu));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (!ctx.kernels_f32) {
|
||||||
|
GGML_LOG_INFO("kleidiai: no compatible f32 kernels found for CPU features mask %d\n", (int)ctx.features);
|
||||||
|
} else {
|
||||||
|
GGML_LOG_INFO("kleidiai: primary f32 kernel feature %s\n", cpu_feature_to_string(ctx.kernels_f32->required_cpu));
|
||||||
|
}
|
||||||
|
|
||||||
ctx.sme_thread_cap = (ctx.features & CPU_FEATURE_SME) ? sme_cores : 0;
|
ctx.sme_thread_cap = (ctx.features & CPU_FEATURE_SME) ? sme_cores : 0;
|
||||||
|
|
||||||
if (ctx.features & CPU_FEATURE_SME) {
|
if (ctx.features & CPU_FEATURE_SME) {
|
||||||
|
const bool has_sme2 = (ctx.features & CPU_FEATURE_SME2) != CPU_FEATURE_NONE;
|
||||||
if (sme_env_set && sme_env_ok && sme_cores > 0) {
|
if (sme_env_set && sme_env_ok && sme_cores > 0) {
|
||||||
GGML_LOG_INFO("kleidiai: SME enabled (GGML_KLEIDIAI_SME=%d override)\n", sme_cores);
|
GGML_LOG_INFO("kleidiai: SME%s enabled (GGML_KLEIDIAI_SME=%d override)\n",
|
||||||
|
has_sme2 ? "2" : "", sme_cores);
|
||||||
} else {
|
} else {
|
||||||
GGML_LOG_INFO("kleidiai: SME enabled (runtime-detected SME cores=%d)\n", sme_cores);
|
GGML_LOG_INFO("kleidiai: SME%s enabled (runtime-detected SME cores=%d)\n",
|
||||||
|
has_sme2 ? "2" : "", sme_cores);
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
GGML_LOG_INFO("kleidiai: SME disabled\n");
|
GGML_LOG_INFO("kleidiai: SME disabled\n");
|
||||||
@@ -334,6 +355,13 @@ static inline size_t ceil_div_size(size_t a, size_t b) {
|
|||||||
return b == 0 ? 0 : (a + b - 1) / b;
|
return b == 0 ? 0 : (a + b - 1) / b;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static inline size_t kleidiai_chunk_cols(size_t n, int nth_total, bool disable_chunking, size_t n_step) {
|
||||||
|
const size_t multiplier = (nth_total == 1 || disable_chunking) ? 1 : std::max<size_t>(1, (size_t) ctx.chunk_multiplier);
|
||||||
|
const size_t divisor = std::max<size_t>(1, (size_t) nth_total * multiplier);
|
||||||
|
const size_t chunk_cols = align_up(std::max<size_t>(1, ceil_div_size(n, divisor)), n_step);
|
||||||
|
return chunk_cols ? chunk_cols : n_step;
|
||||||
|
}
|
||||||
|
|
||||||
struct kleidiai_block_args {
|
struct kleidiai_block_args {
|
||||||
size_t lhs_bl;
|
size_t lhs_bl;
|
||||||
size_t rhs_bl;
|
size_t rhs_bl;
|
||||||
@@ -418,6 +446,10 @@ static inline ggml_kleidiai_kernels * kleidiai_primary_kernel_q8() {
|
|||||||
return ctx.kernels_q8;
|
return ctx.kernels_q8;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static inline ggml_kleidiai_kernels * kleidiai_primary_kernel_f32() {
|
||||||
|
return ctx.kernels_f32;
|
||||||
|
}
|
||||||
|
|
||||||
template <typename SelectFallback>
|
template <typename SelectFallback>
|
||||||
static int kleidiai_collect_kernel_chain_common(
|
static int kleidiai_collect_kernel_chain_common(
|
||||||
ggml_kleidiai_kernels * primary,
|
ggml_kleidiai_kernels * primary,
|
||||||
@@ -430,11 +462,16 @@ static int kleidiai_collect_kernel_chain_common(
|
|||||||
}
|
}
|
||||||
out[count++] = primary;
|
out[count++] = primary;
|
||||||
|
|
||||||
if ((primary->required_cpu & CPU_FEATURE_SME) == CPU_FEATURE_SME) {
|
if (primary->rhs_info.repack_mode == RHS_REPACK_SINGLE_ONLY) {
|
||||||
const cpu_feature fallback_mask = static_cast<cpu_feature>(features & ~CPU_FEATURE_SME);
|
return count;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (is_sme_family(primary->required_cpu)) {
|
||||||
|
const cpu_feature fallback_mask = static_cast<cpu_feature>(features & ~CPU_FEATURE_SME & ~CPU_FEATURE_SME2);
|
||||||
if (fallback_mask != CPU_FEATURE_NONE) {
|
if (fallback_mask != CPU_FEATURE_NONE) {
|
||||||
ggml_kleidiai_kernels * fallback = select_fallback(fallback_mask);
|
ggml_kleidiai_kernels * fallback = select_fallback(fallback_mask);
|
||||||
if (fallback && fallback != primary &&
|
if (fallback && fallback != primary &&
|
||||||
|
fallback->rhs_info.repack_mode != RHS_REPACK_SINGLE_ONLY &&
|
||||||
fallback->lhs_type == primary->lhs_type &&
|
fallback->lhs_type == primary->lhs_type &&
|
||||||
fallback->rhs_type == primary->rhs_type &&
|
fallback->rhs_type == primary->rhs_type &&
|
||||||
fallback->op_type == primary->op_type) {
|
fallback->op_type == primary->op_type) {
|
||||||
@@ -465,6 +502,12 @@ static int kleidiai_collect_q8_chain(std::array<ggml_kleidiai_kernels *, GGML_KL
|
|||||||
[&](cpu_feature mask) { return ggml_kleidiai_select_kernels_q8_0(mask); });
|
[&](cpu_feature mask) { return ggml_kleidiai_select_kernels_q8_0(mask); });
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static int kleidiai_collect_f32_chain(std::array<ggml_kleidiai_kernels *, GGML_KLEIDIAI_MAX_KERNEL_SLOTS> & out) {
|
||||||
|
ggml_kleidiai_kernels * primary = kleidiai_primary_kernel_f32();
|
||||||
|
return kleidiai_collect_kernel_chain_common(primary, ctx.features, out,
|
||||||
|
[&](cpu_feature mask) { return ggml_kleidiai_select_kernels_f32(mask); });
|
||||||
|
}
|
||||||
|
|
||||||
static inline int64_t ggml_ne(const ggml_tensor * tensor, int dim) {
|
static inline int64_t ggml_ne(const ggml_tensor * tensor, int dim) {
|
||||||
GGML_ASSERT(dim >= 0 && dim < GGML_MAX_DIMS);
|
GGML_ASSERT(dim >= 0 && dim < GGML_MAX_DIMS);
|
||||||
return tensor->ne[dim];
|
return tensor->ne[dim];
|
||||||
@@ -539,6 +582,36 @@ class tensor_traits : public ggml::cpu::tensor_traits {
|
|||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (op->src[0]->type == GGML_TYPE_F32) {
|
||||||
|
size_t cursor = 0;
|
||||||
|
bool any_slot = false;
|
||||||
|
|
||||||
|
for (int slot = 0; slot < slot_count; ++slot) {
|
||||||
|
ggml_kleidiai_kernels * kernels = kernel_chain[slot];
|
||||||
|
lhs_packing_info * lhs_info = &kernels->gemm_lhs_info;
|
||||||
|
kernel_info * kernel = &kernels->gemm;
|
||||||
|
|
||||||
|
if (!lhs_info || !lhs_info->packed_size_ex || !kernel) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
const size_t mr = kernel->get_mr();
|
||||||
|
const size_t kr = kernel->get_kr();
|
||||||
|
const size_t sr = kernel->get_sr();
|
||||||
|
|
||||||
|
cursor = align_up(cursor, GGML_KLEIDIAI_PACK_ALIGN);
|
||||||
|
cursor += lhs_info->packed_size_ex(m, k, 0, mr, kr, sr);
|
||||||
|
any_slot = true;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!any_slot) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
size = cursor;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
if (op->src[0]->type == GGML_TYPE_F16) {
|
if (op->src[0]->type == GGML_TYPE_F16) {
|
||||||
const int64_t lhs_batch_size0 = op->src[1]->ne[2];
|
const int64_t lhs_batch_size0 = op->src[1]->ne[2];
|
||||||
const int64_t rhs_batch_size0 = op->src[0]->ne[2];
|
const int64_t rhs_batch_size0 = op->src[0]->ne[2];
|
||||||
@@ -595,6 +668,8 @@ class tensor_traits : public ggml::cpu::tensor_traits {
|
|||||||
if (dst->op == GGML_OP_MUL_MAT) {
|
if (dst->op == GGML_OP_MUL_MAT) {
|
||||||
if (dst->src[0]->type == GGML_TYPE_Q4_0 || dst->src[0]->type == GGML_TYPE_Q8_0) {
|
if (dst->src[0]->type == GGML_TYPE_Q4_0 || dst->src[0]->type == GGML_TYPE_Q8_0) {
|
||||||
return compute_forward_qx(params, dst);
|
return compute_forward_qx(params, dst);
|
||||||
|
} else if (dst->src[0]->type == GGML_TYPE_F32) {
|
||||||
|
return compute_forward_f32(params, dst);
|
||||||
} else if (dst->src[0]->type == GGML_TYPE_F16) {
|
} else if (dst->src[0]->type == GGML_TYPE_F16) {
|
||||||
return compute_forward_fp16(params, dst);
|
return compute_forward_fp16(params, dst);
|
||||||
}
|
}
|
||||||
@@ -606,6 +681,144 @@ class tensor_traits : public ggml::cpu::tensor_traits {
|
|||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
bool compute_forward_f32(ggml_compute_params * params, struct ggml_tensor * dst) {
|
||||||
|
GGML_ASSERT(dst->src[0]->type == GGML_TYPE_F32);
|
||||||
|
|
||||||
|
const ggml_tensor * src0 = dst->src[0];
|
||||||
|
const ggml_tensor * src1 = dst->src[1];
|
||||||
|
|
||||||
|
GGML_TENSOR_BINARY_OP_LOCALS
|
||||||
|
|
||||||
|
if (src1->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
ggml_kleidiai_kernels * kernels = kleidiai_primary_kernel_f32();
|
||||||
|
if (!kernels) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
kernel_info * kernel = &kernels->gemm;
|
||||||
|
lhs_packing_info * lhs_info = &kernels->gemm_lhs_info;
|
||||||
|
|
||||||
|
if (!kernel || !lhs_info || !lhs_info->get_offset || !lhs_info->get_packed_offset_ex ||
|
||||||
|
!lhs_info->packed_size_ex || !lhs_info->pack_func_ex ||
|
||||||
|
!kernel->get_rhs_packed_offset_ex || !kernel->run_kernel_ex || !kernel->get_dst_offset) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
const kleidiai_weight_header * header = kleidiai_weight_header_from_ptr(src0->data);
|
||||||
|
const bool has_header = kleidiai_is_weight_header_valid(header);
|
||||||
|
|
||||||
|
const uint8_t * rhs_base = has_header ? kleidiai_weight_slot_ptr(header, 0)
|
||||||
|
: static_cast<const uint8_t *>(src0->data);
|
||||||
|
if (!rhs_base) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int nth = params->nth > 0 ? params->nth : 1;
|
||||||
|
const int ith = params->ith;
|
||||||
|
|
||||||
|
const size_t k = ne00;
|
||||||
|
const size_t m = ne11;
|
||||||
|
const size_t n = ne01;
|
||||||
|
|
||||||
|
const size_t mr = kernel->get_mr();
|
||||||
|
const size_t kr = kernel->get_kr();
|
||||||
|
const size_t sr = kernel->get_sr();
|
||||||
|
|
||||||
|
const size_t lhs_packed_size = lhs_info->packed_size_ex(m, k, 0, mr, kr, sr);
|
||||||
|
GGML_ASSERT(lhs_packed_size <= params->wsize);
|
||||||
|
|
||||||
|
uint8_t * lhs_packed = static_cast<uint8_t *>(params->wdata);
|
||||||
|
const size_t dst_stride = dst->nb[1];
|
||||||
|
const size_t n_step = kernel->get_n_step() ? kernel->get_n_step() : 1;
|
||||||
|
const bool disable_chunking = ggml_is_numa();
|
||||||
|
GGML_ASSERT(n <= (size_t) INT_MAX);
|
||||||
|
|
||||||
|
for (int64_t batch_idx = 0; batch_idx < ne12; ++batch_idx) {
|
||||||
|
const uint8_t * lhs_batch_base = static_cast<const uint8_t *>(src1->data) + batch_idx * src1->nb[2];
|
||||||
|
uint8_t * dst_batch_base = static_cast<uint8_t *>(dst->data) + batch_idx * dst->nb[2];
|
||||||
|
|
||||||
|
{
|
||||||
|
const int64_t m_roundup_mr = kai_roundup((int64_t)m, (int64_t)mr);
|
||||||
|
int64_t max_threads = mr ? (m_roundup_mr / (int64_t)mr) : nth;
|
||||||
|
max_threads = std::max<int64_t>(1, max_threads);
|
||||||
|
const int64_t use_threads = std::min<int64_t>(nth, max_threads);
|
||||||
|
|
||||||
|
if (ith < use_threads) {
|
||||||
|
const int64_t num_m_per_thread0 = round_down((size_t)(m_roundup_mr / use_threads), mr);
|
||||||
|
const int64_t num_m_per_threadN_1 = (int64_t)m - (use_threads - 1) * num_m_per_thread0;
|
||||||
|
|
||||||
|
const int64_t m_start = (int64_t)ith * num_m_per_thread0;
|
||||||
|
const int64_t m_count = (ith == use_threads - 1) ? num_m_per_threadN_1 : num_m_per_thread0;
|
||||||
|
|
||||||
|
const size_t base_packed_off = lhs_info->get_packed_offset_ex(m_start, k, 0, mr, kr, sr);
|
||||||
|
const size_t next_block_off = lhs_info->get_packed_offset_ex(m_start + mr, k, 0, mr, kr, sr);
|
||||||
|
const size_t row_stride_bytes = mr ? (next_block_off - base_packed_off) / mr : 0;
|
||||||
|
|
||||||
|
int64_t remaining = m_count;
|
||||||
|
int64_t cur = m_start;
|
||||||
|
|
||||||
|
while (remaining > 0) {
|
||||||
|
const int64_t take = std::min<int64_t>((int64_t)m - cur, remaining);
|
||||||
|
const size_t src_off = lhs_info->get_offset(cur, src1->nb[1]);
|
||||||
|
const void * src_ptr = lhs_batch_base + src_off;
|
||||||
|
const size_t dst_off = base_packed_off + (size_t)(cur - m_start) * row_stride_bytes;
|
||||||
|
void * dst_ptr = lhs_packed + dst_off;
|
||||||
|
|
||||||
|
lhs_info->pack_func_ex(take, k, 0, mr, kr, sr, 0, src_ptr, src1->nb[1], dst_ptr);
|
||||||
|
|
||||||
|
cur += take;
|
||||||
|
remaining -= take;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (ith == 0) {
|
||||||
|
ggml_threadpool_chunk_set(params->threadpool, 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
ggml_barrier(params->threadpool);
|
||||||
|
|
||||||
|
const size_t chunk_cols = kleidiai_chunk_cols(n, nth, disable_chunking, n_step);
|
||||||
|
GGML_ASSERT(chunk_cols <= (size_t) INT_MAX);
|
||||||
|
|
||||||
|
int current_col = ggml_threadpool_chunk_add(params->threadpool, (int) chunk_cols);
|
||||||
|
while ((size_t) current_col < n) {
|
||||||
|
const size_t n_start = (size_t) current_col;
|
||||||
|
const size_t n_to_process = std::min(chunk_cols, n - n_start);
|
||||||
|
|
||||||
|
if (n_to_process > 0) {
|
||||||
|
const size_t lhs_packed_offset = lhs_info->get_packed_offset_ex(0, k, 0, mr, kr, sr);
|
||||||
|
const size_t rhs_packed_offset = kernel->get_rhs_packed_offset_ex(n_start, k, 0);
|
||||||
|
const size_t dst_offset = kernel->get_dst_offset(0, n_start, dst_stride);
|
||||||
|
|
||||||
|
const void * lhs_ptr = lhs_packed + lhs_packed_offset;
|
||||||
|
const void * rhs_ptr = rhs_base + rhs_packed_offset;
|
||||||
|
float * dst_ptr = reinterpret_cast<float *>(dst_batch_base + dst_offset);
|
||||||
|
|
||||||
|
kernel->run_kernel_ex(m, n_to_process, k, 0,
|
||||||
|
lhs_ptr,
|
||||||
|
rhs_ptr,
|
||||||
|
dst_ptr,
|
||||||
|
dst_stride,
|
||||||
|
sizeof(float),
|
||||||
|
-FLT_MAX,
|
||||||
|
FLT_MAX);
|
||||||
|
}
|
||||||
|
|
||||||
|
current_col = ggml_threadpool_chunk_add(params->threadpool, (int) chunk_cols);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (batch_idx != ne12 - 1) {
|
||||||
|
ggml_barrier(params->threadpool);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
bool compute_forward_fp16(ggml_compute_params * params, struct ggml_tensor * dst) {
|
bool compute_forward_fp16(ggml_compute_params * params, struct ggml_tensor * dst) {
|
||||||
const ggml_tensor * src0 = dst->src[0];
|
const ggml_tensor * src0 = dst->src[0];
|
||||||
const ggml_tensor * src1 = dst->src[1];
|
const ggml_tensor * src1 = dst->src[1];
|
||||||
@@ -865,14 +1078,14 @@ class tensor_traits : public ggml::cpu::tensor_traits {
|
|||||||
|
|
||||||
int sme_slot = -1;
|
int sme_slot = -1;
|
||||||
for (int i = 0; i < runtime_count; ++i) {
|
for (int i = 0; i < runtime_count; ++i) {
|
||||||
if ((runtime[i].kernels->required_cpu & CPU_FEATURE_SME) == CPU_FEATURE_SME) {
|
if (is_sme_family(runtime[i].kernels->required_cpu)) {
|
||||||
sme_slot = i;
|
sme_slot = i;
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
int non_sme_slot = -1;
|
int non_sme_slot = -1;
|
||||||
for (int i = 0; i < runtime_count; ++i) {
|
for (int i = 0; i < runtime_count; ++i) {
|
||||||
if ((runtime[i].kernels->required_cpu & CPU_FEATURE_SME) != CPU_FEATURE_SME) {
|
if (!is_sme_family(runtime[i].kernels->required_cpu)) {
|
||||||
non_sme_slot = i;
|
non_sme_slot = i;
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
@@ -910,7 +1123,7 @@ class tensor_traits : public ggml::cpu::tensor_traits {
|
|||||||
// Recompute SME slot based on the collapsed runtime[0]
|
// Recompute SME slot based on the collapsed runtime[0]
|
||||||
sme_slot = -1;
|
sme_slot = -1;
|
||||||
if (runtime_count > 0 &&
|
if (runtime_count > 0 &&
|
||||||
(runtime[0].kernels->required_cpu & CPU_FEATURE_SME) == CPU_FEATURE_SME) {
|
is_sme_family(runtime[0].kernels->required_cpu)) {
|
||||||
sme_slot = 0;
|
sme_slot = 0;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1214,7 +1427,7 @@ class tensor_traits : public ggml::cpu::tensor_traits {
|
|||||||
|
|
||||||
public:
|
public:
|
||||||
int repack(struct ggml_tensor * tensor, const void * data, size_t data_size) {
|
int repack(struct ggml_tensor * tensor, const void * data, size_t data_size) {
|
||||||
GGML_ASSERT(tensor->type == GGML_TYPE_Q4_0 || tensor->type == GGML_TYPE_Q8_0);
|
GGML_ASSERT(tensor->type == GGML_TYPE_Q4_0 || tensor->type == GGML_TYPE_Q8_0 || tensor->type == GGML_TYPE_F32);
|
||||||
const size_t n = tensor->ne[1];
|
const size_t n = tensor->ne[1];
|
||||||
const size_t k = tensor->ne[0];
|
const size_t k = tensor->ne[0];
|
||||||
|
|
||||||
@@ -1233,12 +1446,15 @@ public:
|
|||||||
|
|
||||||
std::array<ggml_kleidiai_kernels *, GGML_KLEIDIAI_MAX_KERNEL_SLOTS> kernel_chain;
|
std::array<ggml_kleidiai_kernels *, GGML_KLEIDIAI_MAX_KERNEL_SLOTS> kernel_chain;
|
||||||
const bool want_q8 = tensor->type == GGML_TYPE_Q8_0;
|
const bool want_q8 = tensor->type == GGML_TYPE_Q8_0;
|
||||||
const int slot_total = want_q8 ? kleidiai_collect_q8_chain(kernel_chain)
|
const bool want_f32 = tensor->type == GGML_TYPE_F32;
|
||||||
: kleidiai_collect_q4_chain(kernel_chain);
|
const int slot_total = want_f32 ? kleidiai_collect_f32_chain(kernel_chain)
|
||||||
|
: want_q8 ? kleidiai_collect_q8_chain(kernel_chain)
|
||||||
|
: kleidiai_collect_q4_chain(kernel_chain);
|
||||||
const bool allow_fallback = kleidiai_pack_fallback_allowed();
|
const bool allow_fallback = kleidiai_pack_fallback_allowed();
|
||||||
|
|
||||||
std::vector<int8_t> qdata;
|
std::vector<int8_t> qdata;
|
||||||
std::vector<float> scales;
|
std::vector<float> scales;
|
||||||
|
std::vector<float> bias;
|
||||||
|
|
||||||
if (want_q8 && slot_total > 0) {
|
if (want_q8 && slot_total > 0) {
|
||||||
qdata.resize(n * k, 0);
|
qdata.resize(n * k, 0);
|
||||||
@@ -1286,6 +1502,10 @@ public:
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (want_f32 && slot_total > 0) {
|
||||||
|
bias.resize(n, 0.0f);
|
||||||
|
}
|
||||||
|
|
||||||
for (int slot = 0; slot < slot_total && slot < GGML_KLEIDIAI_MAX_KERNEL_SLOTS; ++slot) {
|
for (int slot = 0; slot < slot_total && slot < GGML_KLEIDIAI_MAX_KERNEL_SLOTS; ++slot) {
|
||||||
if (!allow_fallback && slot > 0) {
|
if (!allow_fallback && slot > 0) {
|
||||||
break;
|
break;
|
||||||
@@ -1302,8 +1522,9 @@ public:
|
|||||||
const size_t sr = kernel->get_sr();
|
const size_t sr = kernel->get_sr();
|
||||||
const ggml_type rhs_type = kernels->rhs_type;
|
const ggml_type rhs_type = kernels->rhs_type;
|
||||||
const size_t block_len = rhs_type == GGML_TYPE_Q8_0 ? QK8_0 :
|
const size_t block_len = rhs_type == GGML_TYPE_Q8_0 ? QK8_0 :
|
||||||
rhs_type == GGML_TYPE_Q4_0 ? QK4_0 : 0;
|
rhs_type == GGML_TYPE_Q4_0 ? QK4_0 :
|
||||||
if (block_len == 0) {
|
rhs_type == GGML_TYPE_F32 ? 0 : SIZE_MAX;
|
||||||
|
if (block_len == SIZE_MAX) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1326,6 +1547,10 @@ public:
|
|||||||
rhs_info->pack_func_ex(1, n, k, nr, kr, sr, 0, 0,
|
rhs_info->pack_func_ex(1, n, k, nr, kr, sr, 0, 0,
|
||||||
qdata.data(), nullptr, scales.data(),
|
qdata.data(), nullptr, scales.data(),
|
||||||
dst_ptr, 0, ¶ms);
|
dst_ptr, 0, ¶ms);
|
||||||
|
} else if (rhs_type == GGML_TYPE_F32) {
|
||||||
|
rhs_info->pack_func_ex(1, n, k, nr, kr, sr, 0, tensor->nb[1],
|
||||||
|
data, bias.data(), nullptr,
|
||||||
|
dst_ptr, 0, nullptr);
|
||||||
} else {
|
} else {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
@@ -1400,7 +1625,7 @@ static size_t ggml_backend_cpu_kleidiai_buffer_type_get_alignment(ggml_backend_b
|
|||||||
static size_t ggml_backend_cpu_kleidiai_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const struct ggml_tensor * tensor) {
|
static size_t ggml_backend_cpu_kleidiai_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const struct ggml_tensor * tensor) {
|
||||||
GGML_UNUSED(buft);
|
GGML_UNUSED(buft);
|
||||||
|
|
||||||
if (tensor->type != GGML_TYPE_Q4_0 && tensor->type != GGML_TYPE_Q8_0) {
|
if (tensor->type != GGML_TYPE_Q4_0 && tensor->type != GGML_TYPE_Q8_0 && tensor->type != GGML_TYPE_F32) {
|
||||||
return ggml_nbytes(tensor);
|
return ggml_nbytes(tensor);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1412,8 +1637,10 @@ static size_t ggml_backend_cpu_kleidiai_buffer_type_get_alloc_size(ggml_backend_
|
|||||||
|
|
||||||
std::array<ggml_kleidiai_kernels *, GGML_KLEIDIAI_MAX_KERNEL_SLOTS> kernel_chain;
|
std::array<ggml_kleidiai_kernels *, GGML_KLEIDIAI_MAX_KERNEL_SLOTS> kernel_chain;
|
||||||
const bool want_q8 = tensor->type == GGML_TYPE_Q8_0;
|
const bool want_q8 = tensor->type == GGML_TYPE_Q8_0;
|
||||||
const int slot_total = want_q8 ? kleidiai_collect_q8_chain(kernel_chain)
|
const bool want_f32 = tensor->type == GGML_TYPE_F32;
|
||||||
: kleidiai_collect_q4_chain(kernel_chain);
|
const int slot_total = want_f32 ? kleidiai_collect_f32_chain(kernel_chain)
|
||||||
|
: want_q8 ? kleidiai_collect_q8_chain(kernel_chain)
|
||||||
|
: kleidiai_collect_q4_chain(kernel_chain);
|
||||||
const bool allow_fallback = kleidiai_pack_fallback_allowed();
|
const bool allow_fallback = kleidiai_pack_fallback_allowed();
|
||||||
|
|
||||||
size_t slot_count = 0;
|
size_t slot_count = 0;
|
||||||
@@ -1433,8 +1660,9 @@ static size_t ggml_backend_cpu_kleidiai_buffer_type_get_alloc_size(ggml_backend_
|
|||||||
|
|
||||||
const ggml_type rhs_type = kernels->rhs_type;
|
const ggml_type rhs_type = kernels->rhs_type;
|
||||||
const size_t block_len = rhs_type == GGML_TYPE_Q4_0 ? QK4_0 :
|
const size_t block_len = rhs_type == GGML_TYPE_Q4_0 ? QK4_0 :
|
||||||
rhs_type == GGML_TYPE_Q8_0 ? QK8_0 : 0;
|
rhs_type == GGML_TYPE_Q8_0 ? QK8_0 :
|
||||||
if (block_len == 0) {
|
rhs_type == GGML_TYPE_F32 ? 0 : SIZE_MAX;
|
||||||
|
if (block_len == SIZE_MAX) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1455,26 +1683,43 @@ class extra_buffer_type : ggml::cpu::extra_buffer_type {
|
|||||||
bool supports_op(ggml_backend_dev_t, const struct ggml_tensor * op) override {
|
bool supports_op(ggml_backend_dev_t, const struct ggml_tensor * op) override {
|
||||||
std::array<ggml_kleidiai_kernels *, GGML_KLEIDIAI_MAX_KERNEL_SLOTS> kernel_chain;
|
std::array<ggml_kleidiai_kernels *, GGML_KLEIDIAI_MAX_KERNEL_SLOTS> kernel_chain;
|
||||||
const int slot_total = kleidiai_collect_kernel_chain(op, kernel_chain);
|
const int slot_total = kleidiai_collect_kernel_chain(op, kernel_chain);
|
||||||
if ((op->op == GGML_OP_MUL_MAT || op->op == GGML_OP_GET_ROWS) &&
|
const bool src0_is_kleidiai =
|
||||||
(op->src[0]->type == GGML_TYPE_Q4_0 || op->src[0]->type == GGML_TYPE_Q8_0) &&
|
|
||||||
op->src[0]->buffer &&
|
op->src[0]->buffer &&
|
||||||
(ggml_n_dims(op->src[0]) == 2) &&
|
(ggml_n_dims(op->src[0]) == 2) &&
|
||||||
op->src[0]->buffer->buft == ggml_backend_cpu_kleidiai_buffer_type() &&
|
op->src[0]->buffer->buft == ggml_backend_cpu_kleidiai_buffer_type() &&
|
||||||
slot_total > 0) {
|
slot_total > 0;
|
||||||
|
|
||||||
|
if ((op->op == GGML_OP_MUL_MAT || op->op == GGML_OP_GET_ROWS) &&
|
||||||
|
(op->src[0]->type == GGML_TYPE_Q4_0 || op->src[0]->type == GGML_TYPE_Q8_0 || op->src[0]->type == GGML_TYPE_F32) &&
|
||||||
|
src0_is_kleidiai) {
|
||||||
if (op->src[0]->type == GGML_TYPE_Q4_0 && ctx.kernels_q4 == nullptr) {
|
if (op->src[0]->type == GGML_TYPE_Q4_0 && ctx.kernels_q4 == nullptr) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
if (op->src[0]->type == GGML_TYPE_Q8_0 && ctx.kernels_q8 == nullptr) {
|
if (op->src[0]->type == GGML_TYPE_Q8_0 && ctx.kernels_q8 == nullptr) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
if (op->src[0]->type == GGML_TYPE_F32 && ctx.kernels_f32 == nullptr) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
if (op->src[1]->buffer && !ggml_backend_buft_is_host(op->src[1]->buffer->buft)) {
|
if (op->src[1]->buffer && !ggml_backend_buft_is_host(op->src[1]->buffer->buft)) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
if ((op->src[1]->type == GGML_TYPE_F32 || op->src[1]->type == GGML_TYPE_I32) &&
|
|
||||||
ggml_ne(op->src[1], 3) == 1) {
|
if (op->src[0]->type == GGML_TYPE_Q4_0 || op->src[0]->type == GGML_TYPE_Q8_0) {
|
||||||
return true;
|
if ((op->src[1]->type == GGML_TYPE_F32 || op->src[1]->type == GGML_TYPE_I32) &&
|
||||||
|
ggml_ne(op->src[1], 3) == 1) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (op->op != GGML_OP_MUL_MAT || op->src[1]->type != GGML_TYPE_F32 || op->type != GGML_TYPE_F32) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1483,6 +1728,20 @@ class extra_buffer_type : ggml::cpu::extra_buffer_type {
|
|||||||
if (op->src[0]->buffer && op->src[0]->buffer->buft == ggml_backend_cpu_kleidiai_buffer_type()) {
|
if (op->src[0]->buffer && op->src[0]->buffer->buft == ggml_backend_cpu_kleidiai_buffer_type()) {
|
||||||
return (ggml::cpu::tensor_traits *) op->src[0]->extra;
|
return (ggml::cpu::tensor_traits *) op->src[0]->extra;
|
||||||
} else {
|
} else {
|
||||||
|
// KleidiAI only has kernels for Q4_0 and Q8_0. For a quantized weight of any
|
||||||
|
// other type (K-quants, IQ) it declines the op and returns nullptr below, so
|
||||||
|
// KleidiAI does not accelerate it. Another CPU backend may still take the op,
|
||||||
|
// and this can run during graph planning, so the message says what KleidiAI
|
||||||
|
// did rather than what ends up executing. Warn once per process.
|
||||||
|
if (ggml_is_quantized(op->src[0]->type) &&
|
||||||
|
op->src[0]->type != GGML_TYPE_Q4_0 && op->src[0]->type != GGML_TYPE_Q8_0) {
|
||||||
|
static std::atomic<bool> warned(false);
|
||||||
|
if (!warned.exchange(true)) {
|
||||||
|
GGML_LOG_WARN("kleidiai: no kernel for tensor type %s, not accelerated by KleidiAI "
|
||||||
|
"(kernels available for Q4_0 and Q8_0)\n",
|
||||||
|
ggml_type_name(op->src[0]->type));
|
||||||
|
}
|
||||||
|
}
|
||||||
if (op->src[0]->type != GGML_TYPE_F16) {
|
if (op->src[0]->type != GGML_TYPE_F16) {
|
||||||
return nullptr;
|
return nullptr;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1797,14 +1797,6 @@ class tinyBLAS_Q0_AVX {
|
|||||||
//PPC Implementation
|
//PPC Implementation
|
||||||
#if defined(__MMA__)
|
#if defined(__MMA__)
|
||||||
|
|
||||||
#define SAVE_ACC(ACC, ii, jj) \
|
|
||||||
__builtin_mma_disassemble_acc(vec_C, ACC); \
|
|
||||||
for (int I = 0; I < 4; I++) { \
|
|
||||||
for (int J = 0; J < 4; J++) { \
|
|
||||||
*((float*)(C+ii+((jj+J)*ldc)+I)) = *((float*)&vec_C[I]+J); \
|
|
||||||
} \
|
|
||||||
} \
|
|
||||||
|
|
||||||
template<typename T>
|
template<typename T>
|
||||||
struct mma_instr;
|
struct mma_instr;
|
||||||
|
|
||||||
@@ -1834,10 +1826,49 @@ class tinyBLAS_HP16_PPC {
|
|||||||
}
|
}
|
||||||
|
|
||||||
void matmul(int64_t m, int64_t n) {
|
void matmul(int64_t m, int64_t n) {
|
||||||
mnpack(0, m, 0, n);
|
int64_t mc = 256;
|
||||||
|
int64_t nc = 256;
|
||||||
|
int64_t kc = 256;
|
||||||
|
#if defined(_AIX) || defined(__BIG_ENDIAN__)
|
||||||
|
mc = 128;
|
||||||
|
nc = 128;
|
||||||
|
kc = 128;
|
||||||
|
#endif
|
||||||
|
if (k < kc) {
|
||||||
|
kc = k;
|
||||||
|
}
|
||||||
|
bool can_use_tiled = (m % mc == 0) && (n % nc == 0) && (k % kc == 0);
|
||||||
|
if (can_use_tiled) {
|
||||||
|
matmul_tiled(m, n, mc, nc, kc);
|
||||||
|
} else {
|
||||||
|
mnpack(0, m, 0, n);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private:
|
private:
|
||||||
|
__attribute__((always_inline))
|
||||||
|
inline void save_acc(acc_t * ACC, int64_t ii, int64_t jj) {
|
||||||
|
vec_t vec_C[4];
|
||||||
|
__builtin_mma_disassemble_acc(vec_C, ACC);
|
||||||
|
for (int I = 0; I < 4; I++) {
|
||||||
|
for (int J = 0; J < 4; J++) {
|
||||||
|
*((float *)(C+ii+((jj+J)*ldc)+I)) = *((float *)&vec_C[I]+J);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
__attribute__((always_inline))
|
||||||
|
inline void add_save_acc(acc_t * ACC, int64_t ii, int64_t jj) {
|
||||||
|
vec_t vec_C[4];
|
||||||
|
__builtin_mma_disassemble_acc(vec_C, ACC);
|
||||||
|
for (int I = 0; I < 4; I++) {
|
||||||
|
for (int J = 0; J < 4; J++) {
|
||||||
|
float * c_ptr = (float *)(C+ii+((jj+J)*ldc)+I);
|
||||||
|
*c_ptr += *((float *)&vec_C[I]+J);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
void vector_permute_store(vec_t *c, int numVec, unsigned char *vecOffset) {
|
void vector_permute_store(vec_t *c, int numVec, unsigned char *vecOffset) {
|
||||||
vec_t t[8], s[8];
|
vec_t t[8], s[8];
|
||||||
vec_t swiz1 = {0, 1, 2, 3, 16, 17, 18, 19, 4, 5, 6, 7, 20, 21, 22, 23};
|
vec_t swiz1 = {0, 1, 2, 3, 16, 17, 18, 19, 4, 5, 6, 7, 20, 21, 22, 23};
|
||||||
@@ -1896,6 +1927,7 @@ class tinyBLAS_HP16_PPC {
|
|||||||
j = (rows >> 3);
|
j = (rows >> 3);
|
||||||
if (j > 0) {
|
if (j > 0) {
|
||||||
do {
|
do {
|
||||||
|
aoffsets[0] = aoffset;
|
||||||
if (cols == 4) {
|
if (cols == 4) {
|
||||||
aoffsets[0] = aoffset;
|
aoffsets[0] = aoffset;
|
||||||
for (int it = 1; it < 4; ++it)
|
for (int it = 1; it < 4; ++it)
|
||||||
@@ -1910,17 +1942,17 @@ class tinyBLAS_HP16_PPC {
|
|||||||
}
|
}
|
||||||
i = (cols >> 3);
|
i = (cols >> 3);
|
||||||
if (i > 0) {
|
if (i > 0) {
|
||||||
aoffsets[0] = aoffset;
|
|
||||||
for (int it = 1; it < 8; ++it) {
|
for (int it = 1; it < 8; ++it) {
|
||||||
aoffsets[it] = aoffsets[it-1] + lda;
|
aoffsets[it] = aoffsets[it-1] + lda;
|
||||||
}
|
}
|
||||||
aoffset += 8 * lda;
|
aoffset += 8 * lda;
|
||||||
|
|
||||||
do {
|
do {
|
||||||
for (int it = 0; it < 8; ++it)
|
for (int it = 0; it < 8; ++it)
|
||||||
c_arr[it] = vec_xl(0, (vector unsigned char*)aoffsets[it]);
|
c_arr[it] = vec_xl(0, (vector unsigned char*)aoffsets[it]);
|
||||||
vector_permute_store(c_arr, 8, vecOffset);
|
vector_permute_store(c_arr, 8, vecOffset);
|
||||||
for (int it = 0; it < 8; ++it)
|
for (int it = 0; it < 8; ++it)
|
||||||
aoffsets[it] = aoffsets[it] + 8*lda;
|
aoffsets[it] = aoffsets[it] + 8;
|
||||||
vecOffset += 128;
|
vecOffset += 128;
|
||||||
i--;
|
i--;
|
||||||
} while(i > 0);
|
} while(i > 0);
|
||||||
@@ -2147,8 +2179,8 @@ class tinyBLAS_HP16_PPC {
|
|||||||
mma_instr<TA>::outer_product(&acc_1, vec_A[x], vec_B[x+4]);
|
mma_instr<TA>::outer_product(&acc_1, vec_A[x], vec_B[x+4]);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
SAVE_ACC(&acc_0, ii, jj);
|
save_acc(&acc_0, ii, jj);
|
||||||
SAVE_ACC(&acc_1, ii, jj+4);
|
save_acc(&acc_1, ii, jj+4);
|
||||||
}
|
}
|
||||||
|
|
||||||
void KERNEL_8x4(int64_t ii, int64_t jj) {
|
void KERNEL_8x4(int64_t ii, int64_t jj) {
|
||||||
@@ -2164,8 +2196,8 @@ class tinyBLAS_HP16_PPC {
|
|||||||
mma_instr<TA>::outer_product(&acc_1, vec_A[x+4], vec_B[x]);
|
mma_instr<TA>::outer_product(&acc_1, vec_A[x+4], vec_B[x]);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
SAVE_ACC(&acc_0, ii, jj);
|
save_acc(&acc_0, ii, jj);
|
||||||
SAVE_ACC(&acc_1, ii+4, jj);
|
save_acc(&acc_1, ii+4, jj);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
@@ -2186,13 +2218,64 @@ class tinyBLAS_HP16_PPC {
|
|||||||
mma_instr<TA>::outer_product(&acc_3, vec_A[x+4], vec_B[x+4]);
|
mma_instr<TA>::outer_product(&acc_3, vec_A[x+4], vec_B[x+4]);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
save_acc(&acc_0, ii, jj);
|
||||||
SAVE_ACC(&acc_0, ii, jj);
|
save_acc(&acc_1, ii, jj+4);
|
||||||
SAVE_ACC(&acc_1, ii, jj+4);
|
save_acc(&acc_2, ii+4, jj);
|
||||||
SAVE_ACC(&acc_2, ii+4, jj);
|
save_acc(&acc_3, ii+4, jj+4);
|
||||||
SAVE_ACC(&acc_3, ii+4, jj+4);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
inline void MMA_16x8(vec_t * vec_A0, vec_t * vec_A1, vec_t * vec_B, acc_t * acc) {
|
||||||
|
for (int x = 0; x < 4; x ++) {
|
||||||
|
mma_instr<TA>::outer_product(&acc[0], vec_A0[x], vec_B[x]);
|
||||||
|
mma_instr<TA>::outer_product(&acc[1], vec_A0[x], vec_B[x+4]);
|
||||||
|
mma_instr<TA>::outer_product(&acc[2], vec_A0[x+4], vec_B[x]);
|
||||||
|
mma_instr<TA>::outer_product(&acc[3], vec_A0[x+4], vec_B[x+4]);
|
||||||
|
mma_instr<TA>::outer_product(&acc[4], vec_A1[x], vec_B[x]);
|
||||||
|
mma_instr<TA>::outer_product(&acc[5], vec_A1[x], vec_B[x+4]);
|
||||||
|
mma_instr<TA>::outer_product(&acc[6], vec_A1[x+4], vec_B[x]);
|
||||||
|
mma_instr<TA>::outer_product(&acc[7], vec_A1[x+4], vec_B[x+4]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
void KERNEL(int64_t ii, int64_t jj, int64_t mc, int64_t nc, int64_t kc, vec_t * vec_A, vec_t * vec_B, int64_t kk) {
|
||||||
|
for (int64_t i = 0; i < mc; i += 16) {
|
||||||
|
int A_base_addr = (mc / 8) * (i / 8) * 8;
|
||||||
|
for (int64_t j = 0; j < nc; j += 8) {
|
||||||
|
int B_base_addr = (nc / 8) * (j / 8) * 8;
|
||||||
|
acc_t acc[8];
|
||||||
|
vec_t A0_block[8]; vec_t A1_block[8];
|
||||||
|
for (int x = 0; x < 8; x++)
|
||||||
|
__builtin_mma_xxsetaccz(&acc[x]);
|
||||||
|
for (int64_t l = 0; l < kc; l += 8) {
|
||||||
|
int A0_block_idx = A_base_addr + (l / 8) * 8;
|
||||||
|
int A1_block_idx = A0_block_idx + (mc / 8) * 8;
|
||||||
|
int B_block_idx = B_base_addr + (l / 8) * 8;
|
||||||
|
vec_t* A0_block = &vec_A[A0_block_idx];
|
||||||
|
vec_t* A1_block = &vec_A[A1_block_idx];
|
||||||
|
vec_t* B_block = &vec_B[B_block_idx];
|
||||||
|
MMA_16x8(A0_block, A1_block, B_block, acc);
|
||||||
|
}
|
||||||
|
if (kk == 0) {
|
||||||
|
save_acc(&acc[0], ii + i, jj + j);
|
||||||
|
save_acc(&acc[1], ii + i, jj + j + 4);
|
||||||
|
save_acc(&acc[2], ii + i + 4, jj + j);
|
||||||
|
save_acc(&acc[3], ii + i + 4, jj + j + 4);
|
||||||
|
save_acc(&acc[4], ii + i + 8, jj + j);
|
||||||
|
save_acc(&acc[5], ii + i + 8, jj + j + 4);
|
||||||
|
save_acc(&acc[6], ii + i + 12, jj + j);
|
||||||
|
save_acc(&acc[7], ii + i + 12, jj + j + 4);
|
||||||
|
} else {
|
||||||
|
add_save_acc(&acc[0], ii + i, jj + j);
|
||||||
|
add_save_acc(&acc[1], ii + i, jj + j + 4);
|
||||||
|
add_save_acc(&acc[2], ii + i + 4, jj + j);
|
||||||
|
add_save_acc(&acc[3], ii + i + 4, jj + j + 4);
|
||||||
|
add_save_acc(&acc[4], ii + i + 8, jj + j);
|
||||||
|
add_save_acc(&acc[5], ii + i + 8, jj + j + 4);
|
||||||
|
add_save_acc(&acc[6], ii + i + 12, jj + j);
|
||||||
|
add_save_acc(&acc[7], ii + i + 12, jj + j + 4);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
template<int RM, int RN>
|
template<int RM, int RN>
|
||||||
void gemm_small(int64_t m0, int64_t m, int64_t n0, int64_t n) {
|
void gemm_small(int64_t m0, int64_t m, int64_t n0, int64_t n) {
|
||||||
int64_t ytiles = (m - m0) / RM;
|
int64_t ytiles = (m - m0) / RM;
|
||||||
@@ -2281,6 +2364,29 @@ class tinyBLAS_HP16_PPC {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void matmul_tiled(int64_t m, int64_t n, int64_t mc, int64_t nc, int64_t kc) {
|
||||||
|
int64_t ytiles = m / mc;
|
||||||
|
int64_t xtiles = n / nc;
|
||||||
|
int64_t tiles = xtiles * ytiles;
|
||||||
|
int64_t duty = (tiles + nth - 1) / nth;
|
||||||
|
int64_t start = duty * ith;
|
||||||
|
int64_t end = start + duty;
|
||||||
|
if (end > tiles) {
|
||||||
|
end = tiles;
|
||||||
|
}
|
||||||
|
for (int64_t job = start; job < end; ++job) {
|
||||||
|
int64_t ii = (job / xtiles) * mc;
|
||||||
|
int64_t jj = (job % xtiles) * nc;
|
||||||
|
for (int64_t kk = 0; kk < k; kk += kc) {
|
||||||
|
vec_t A_pack[kc * mc / 8];
|
||||||
|
vec_t B_pack[kc * nc / 8];
|
||||||
|
packNormal(A + (ii * lda) + kk, lda, kc, mc, (uint8_t *)A_pack);
|
||||||
|
packNormal(B + (jj * ldb) + kk, ldb, kc, nc, (uint8_t *)B_pack);
|
||||||
|
KERNEL(ii, jj, mc, nc, kc, A_pack, B_pack, kk);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
template <int RM, int RN>
|
template <int RM, int RN>
|
||||||
NOINLINE void gemm(int64_t m0, int64_t m, int64_t n0, int64_t n) {
|
NOINLINE void gemm(int64_t m0, int64_t m, int64_t n0, int64_t n) {
|
||||||
int64_t ytiles = (m - m0) / RM;
|
int64_t ytiles = (m - m0) / RM;
|
||||||
@@ -2329,7 +2435,7 @@ class tinyBLAS_Q0_PPC {
|
|||||||
mc = 32;
|
mc = 32;
|
||||||
nc = 32;
|
nc = 32;
|
||||||
kc = 32;
|
kc = 32;
|
||||||
n_chunk = 32
|
n_chunk = 32;
|
||||||
#endif
|
#endif
|
||||||
int64_t n_aligned = 0;
|
int64_t n_aligned = 0;
|
||||||
if (n % n_chunk == 0) {
|
if (n % n_chunk == 0) {
|
||||||
|
|||||||
+454
-20
@@ -2081,8 +2081,8 @@ void ggml_compute_forward_concat(
|
|||||||
const ggml_tensor * src1 = dst->src[1];
|
const ggml_tensor * src1 = dst->src[1];
|
||||||
|
|
||||||
if (ggml_is_quantized(src0->type)) {
|
if (ggml_is_quantized(src0->type)) {
|
||||||
GGML_ASSERT(ggml_is_contiguous(src0));
|
GGML_ASSERT(ggml_is_contiguous_rows(src0));
|
||||||
GGML_ASSERT(ggml_is_contiguous(src1));
|
GGML_ASSERT(ggml_is_contiguous_rows(src1));
|
||||||
GGML_ASSERT(src0->ne[0] % ggml_blck_size(src0->type) == 0);
|
GGML_ASSERT(src0->ne[0] % ggml_blck_size(src0->type) == 0);
|
||||||
GGML_ASSERT(src1->ne[0] % ggml_blck_size(src1->type) == 0);
|
GGML_ASSERT(src1->ne[0] % ggml_blck_size(src1->type) == 0);
|
||||||
}
|
}
|
||||||
@@ -4449,6 +4449,70 @@ static void ggml_compute_forward_out_prod_q_f32(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static void ggml_compute_forward_out_prod_f16_f32(
|
||||||
|
const ggml_compute_params * params,
|
||||||
|
ggml_tensor * dst) {
|
||||||
|
|
||||||
|
const ggml_tensor * src0 = dst->src[0];
|
||||||
|
const ggml_tensor * src1 = dst->src[1];
|
||||||
|
|
||||||
|
GGML_TENSOR_BINARY_OP_LOCALS;
|
||||||
|
|
||||||
|
const int ith = params->ith;
|
||||||
|
const int nth = params->nth;
|
||||||
|
|
||||||
|
GGML_ASSERT(src0->type == GGML_TYPE_F16);
|
||||||
|
GGML_ASSERT(src1->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(dst->type == GGML_TYPE_F32);
|
||||||
|
|
||||||
|
GGML_ASSERT(ne02 == ne12);
|
||||||
|
GGML_ASSERT(ne03 == ne13);
|
||||||
|
GGML_ASSERT(ne2 == ne12);
|
||||||
|
GGML_ASSERT(ne3 == ne13);
|
||||||
|
|
||||||
|
GGML_ASSERT(nb00 == sizeof(ggml_fp16_t));
|
||||||
|
GGML_ASSERT(nb0 == sizeof(float));
|
||||||
|
|
||||||
|
GGML_ASSERT(ne0 == ne00);
|
||||||
|
GGML_ASSERT(ne1 == ne10);
|
||||||
|
GGML_ASSERT(ne2 == ne02);
|
||||||
|
GGML_ASSERT(ne3 == ne03);
|
||||||
|
|
||||||
|
if (ith == 0) {
|
||||||
|
ggml_vec_set_f32(ne0*ne1*ne2*ne3, (float *)dst->data, 0);
|
||||||
|
}
|
||||||
|
ggml_barrier(params->threadpool);
|
||||||
|
|
||||||
|
const int64_t nr = ne1*ne2*ne3;
|
||||||
|
const int64_t dr = (nr + nth - 1)/nth;
|
||||||
|
const int64_t ir0 = dr*ith;
|
||||||
|
const int64_t ir1 = MIN(ir0 + dr, nr);
|
||||||
|
|
||||||
|
float * wdata = (float *) params->wdata + (ne0 + CACHE_LINE_SIZE_F32) * ith;
|
||||||
|
|
||||||
|
for (int64_t ir = ir0; ir < ir1; ++ir) {
|
||||||
|
const int64_t i3 = ir/(ne2*ne1);
|
||||||
|
const int64_t i2 = (ir - i3*ne2*ne1)/ne1;
|
||||||
|
const int64_t i1 = (ir - i3*ne2*ne1 - i2*ne1);
|
||||||
|
|
||||||
|
const int64_t i02 = i2;
|
||||||
|
const int64_t i03 = i3;
|
||||||
|
|
||||||
|
const int64_t i12 = i2;
|
||||||
|
const int64_t i13 = i3;
|
||||||
|
|
||||||
|
float * d = (float *) ((char *) dst->data + (i1*nb1 + i2*nb2 + i3*nb3));
|
||||||
|
|
||||||
|
for (int64_t i01 = 0; i01 < ne01; ++i01) {
|
||||||
|
const int64_t i11 = i01;
|
||||||
|
ggml_fp16_t * s0 = (ggml_fp16_t *) ((char *) src0->data + (i01*nb01 + i02*nb02 + i03*nb03));
|
||||||
|
float * s1 = (float *) ((char *) src1->data + (i1*nb10 + i11*nb11 + i12*nb12 + i13*nb13));
|
||||||
|
ggml_fp16_to_fp32_row(s0, wdata, ne0);
|
||||||
|
ggml_vec_mad_f32(ne0, d, wdata, *s1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
void ggml_compute_forward_out_prod(
|
void ggml_compute_forward_out_prod(
|
||||||
const ggml_compute_params * params,
|
const ggml_compute_params * params,
|
||||||
ggml_tensor * dst) {
|
ggml_tensor * dst) {
|
||||||
@@ -4486,9 +4550,8 @@ void ggml_compute_forward_out_prod(
|
|||||||
} break;
|
} break;
|
||||||
case GGML_TYPE_F16:
|
case GGML_TYPE_F16:
|
||||||
{
|
{
|
||||||
GGML_ABORT("fatal error"); // todo
|
ggml_compute_forward_out_prod_f16_f32(params, dst);
|
||||||
// ggml_compute_forward_out_prod_f16_f32(params, dst);
|
} break;
|
||||||
}
|
|
||||||
case GGML_TYPE_F32:
|
case GGML_TYPE_F32:
|
||||||
{
|
{
|
||||||
ggml_compute_forward_out_prod_f32(params, dst);
|
ggml_compute_forward_out_prod_f32(params, dst);
|
||||||
@@ -5041,7 +5104,7 @@ static void ggml_compute_forward_set_rows_impl(
|
|||||||
assert(ne0 == nc);
|
assert(ne0 == nc);
|
||||||
assert(ne2 == ne02);
|
assert(ne2 == ne02);
|
||||||
assert(ne3 == ne03);
|
assert(ne3 == ne03);
|
||||||
GGML_ASSERT(src0->type == GGML_TYPE_F32 || (src0->type == GGML_TYPE_F16 && dst->type == GGML_TYPE_F16));
|
GGML_ASSERT(src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16);
|
||||||
assert(ne02 % ne11 == 0);
|
assert(ne02 % ne11 == 0);
|
||||||
assert(ne03 % ne12 == 0);
|
assert(ne03 % ne12 == 0);
|
||||||
|
|
||||||
@@ -5075,10 +5138,19 @@ static void ggml_compute_forward_set_rows_impl(
|
|||||||
(const float *) ((char *) src0->data + i*nb01 + i02*nb02 + i03*nb03),
|
(const float *) ((char *) src0->data + i*nb01 + i02*nb02 + i03*nb03),
|
||||||
((char *) dst->data + i1*nb1 + i02*nb2 + i03*nb3), nc);
|
((char *) dst->data + i1*nb1 + i02*nb2 + i03*nb3), nc);
|
||||||
} else if constexpr (std::is_same_v<src_t, ggml_fp16_t>) {
|
} else if constexpr (std::is_same_v<src_t, ggml_fp16_t>) {
|
||||||
memcpy(
|
if (dst->type == GGML_TYPE_F16) {
|
||||||
|
memcpy(
|
||||||
((char *) dst->data + i1*nb1 + i02*nb2 + i03*nb3),
|
((char *) dst->data + i1*nb1 + i02*nb2 + i03*nb3),
|
||||||
((char *) src0->data + i*nb01 + i02*nb02 + i03*nb03),
|
((char *) src0->data + i*nb01 + i02*nb02 + i03*nb03),
|
||||||
rs);
|
rs);
|
||||||
|
} else {
|
||||||
|
float * wdata = (float *) params->wdata + (nc + CACHE_LINE_SIZE_F32) * ith;
|
||||||
|
ggml_fp16_to_fp32_row(
|
||||||
|
(const ggml_fp16_t *) ((char *) src0->data + i*nb01 + i02*nb02 + i03*nb03),
|
||||||
|
wdata, nc);
|
||||||
|
from_float(wdata,
|
||||||
|
((char *) dst->data + i1*nb1 + i02*nb2 + i03*nb3), nc);
|
||||||
|
}
|
||||||
} else {
|
} else {
|
||||||
GGML_ABORT("src0->type = %d (%s) not supported", src0->type, ggml_type_name(src0->type));
|
GGML_ABORT("src0->type = %d (%s) not supported", src0->type, ggml_type_name(src0->type));
|
||||||
}
|
}
|
||||||
@@ -5107,16 +5179,12 @@ void ggml_compute_forward_set_rows(
|
|||||||
} break;
|
} break;
|
||||||
case GGML_TYPE_F16:
|
case GGML_TYPE_F16:
|
||||||
{
|
{
|
||||||
if (dst->type == GGML_TYPE_F16) {
|
if (src1->type == GGML_TYPE_I64) {
|
||||||
if (src1->type == GGML_TYPE_I64) {
|
ggml_compute_forward_set_rows_impl<ggml_fp16_t, int64_t>(params, dst);
|
||||||
ggml_compute_forward_set_rows_impl<ggml_fp16_t, int64_t>(params, dst);
|
} else if (src1->type == GGML_TYPE_I32) {
|
||||||
} else if (src1->type == GGML_TYPE_I32) {
|
ggml_compute_forward_set_rows_impl<ggml_fp16_t, int32_t>(params, dst);
|
||||||
ggml_compute_forward_set_rows_impl<ggml_fp16_t, int32_t>(params, dst);
|
|
||||||
} else {
|
|
||||||
GGML_ABORT("src1->type = %d (%s) not supported", src1->type, ggml_type_name(src1->type));
|
|
||||||
}
|
|
||||||
} else {
|
} else {
|
||||||
GGML_ABORT("dst->type = %d (%s) not supported with src0->type = %d (%s)", dst->type, ggml_type_name(dst->type), src0->type, ggml_type_name(src0->type));
|
GGML_ABORT("src1->type = %d (%s) not supported", src1->type, ggml_type_name(src1->type));
|
||||||
}
|
}
|
||||||
} break;
|
} break;
|
||||||
default:
|
default:
|
||||||
@@ -6362,7 +6430,6 @@ static void ggml_compute_forward_im2col_f16(
|
|||||||
const ggml_tensor * src0 = dst->src[0];
|
const ggml_tensor * src0 = dst->src[0];
|
||||||
const ggml_tensor * src1 = dst->src[1];
|
const ggml_tensor * src1 = dst->src[1];
|
||||||
|
|
||||||
GGML_ASSERT(src0->type == GGML_TYPE_F16);
|
|
||||||
GGML_ASSERT(src1->type == GGML_TYPE_F16 || src1->type == GGML_TYPE_F32);
|
GGML_ASSERT(src1->type == GGML_TYPE_F16 || src1->type == GGML_TYPE_F32);
|
||||||
GGML_ASSERT( dst->type == GGML_TYPE_F16);
|
GGML_ASSERT( dst->type == GGML_TYPE_F16);
|
||||||
|
|
||||||
@@ -6393,7 +6460,6 @@ static void ggml_compute_forward_im2col_f16(
|
|||||||
int ofs0 = is_2D ? nb13 : nb12;
|
int ofs0 = is_2D ? nb13 : nb12;
|
||||||
int ofs1 = is_2D ? nb12 : nb11;
|
int ofs1 = is_2D ? nb12 : nb11;
|
||||||
|
|
||||||
GGML_ASSERT(nb00 == sizeof(ggml_fp16_t));
|
|
||||||
GGML_ASSERT(nb10 == ggml_type_size(src1->type));
|
GGML_ASSERT(nb10 == ggml_type_size(src1->type));
|
||||||
|
|
||||||
// im2col: [N, IC, IH, IW] => [N, OH, OW, IC*KH*KW]
|
// im2col: [N, IC, IH, IW] => [N, OH, OW, IC*KH*KW]
|
||||||
@@ -6466,7 +6532,7 @@ void ggml_compute_forward_im2col_back_f32(
|
|||||||
const ggml_tensor * src1 = dst->src[1]; // convolution kernel
|
const ggml_tensor * src1 = dst->src[1]; // convolution kernel
|
||||||
|
|
||||||
GGML_ASSERT(src0->type == GGML_TYPE_F32);
|
GGML_ASSERT(src0->type == GGML_TYPE_F32);
|
||||||
GGML_ASSERT(src1->type == GGML_TYPE_F32);
|
GGML_ASSERT(src1->type == GGML_TYPE_F32 || src1->type == GGML_TYPE_F16);
|
||||||
GGML_ASSERT( dst->type == GGML_TYPE_F32);
|
GGML_ASSERT( dst->type == GGML_TYPE_F32);
|
||||||
|
|
||||||
GGML_TENSOR_BINARY_OP_LOCALS;
|
GGML_TENSOR_BINARY_OP_LOCALS;
|
||||||
@@ -6563,7 +6629,6 @@ static void ggml_compute_forward_im2col_3d_f16(
|
|||||||
const ggml_tensor * src0 = dst->src[0];
|
const ggml_tensor * src0 = dst->src[0];
|
||||||
const ggml_tensor * src1 = dst->src[1];
|
const ggml_tensor * src1 = dst->src[1];
|
||||||
|
|
||||||
GGML_ASSERT(src0->type == GGML_TYPE_F16);
|
|
||||||
GGML_ASSERT(src1->type == GGML_TYPE_F32);
|
GGML_ASSERT(src1->type == GGML_TYPE_F32);
|
||||||
GGML_ASSERT( dst->type == GGML_TYPE_F16);
|
GGML_ASSERT( dst->type == GGML_TYPE_F16);
|
||||||
|
|
||||||
@@ -10879,6 +10944,291 @@ void ggml_compute_forward_gated_delta_net(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
// ggml_compute_forward_dsv4_hc_comb
|
||||||
|
|
||||||
|
static void ggml_dsv4_hc_comb_norm_cols(float * comb, float eps) {
|
||||||
|
constexpr int64_t hc = 4;
|
||||||
|
|
||||||
|
for (int64_t idst = 0; idst < hc; ++idst) {
|
||||||
|
float sum = eps;
|
||||||
|
for (int64_t isrc = 0; isrc < hc; ++isrc) {
|
||||||
|
sum += comb[idst + hc*isrc];
|
||||||
|
}
|
||||||
|
|
||||||
|
const float inv_sum = 1.0f / sum;
|
||||||
|
for (int64_t isrc = 0; isrc < hc; ++isrc) {
|
||||||
|
comb[idst + hc*isrc] *= inv_sum;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static void ggml_dsv4_hc_comb_norm_rows(float * comb, float eps) {
|
||||||
|
constexpr int64_t hc = 4;
|
||||||
|
|
||||||
|
for (int64_t isrc = 0; isrc < hc; ++isrc) {
|
||||||
|
float sum = eps;
|
||||||
|
for (int64_t idst = 0; idst < hc; ++idst) {
|
||||||
|
sum += comb[idst + hc*isrc];
|
||||||
|
}
|
||||||
|
|
||||||
|
const float inv_sum = 1.0f / sum;
|
||||||
|
for (int64_t idst = 0; idst < hc; ++idst) {
|
||||||
|
comb[idst + hc*isrc] *= inv_sum;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static void ggml_compute_forward_dsv4_hc_comb_f32(
|
||||||
|
const ggml_compute_params * params,
|
||||||
|
ggml_tensor * dst) {
|
||||||
|
const ggml_tensor * mixes = dst->src[0];
|
||||||
|
const ggml_tensor * scale = dst->src[1];
|
||||||
|
const ggml_tensor * base = dst->src[2];
|
||||||
|
|
||||||
|
GGML_ASSERT(mixes->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(scale->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(base->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(dst->type == GGML_TYPE_F32);
|
||||||
|
|
||||||
|
constexpr int64_t hc = 4;
|
||||||
|
constexpr int64_t comb_offset = 2*hc;
|
||||||
|
constexpr int64_t hc_mix_dim = (2 + hc)*hc;
|
||||||
|
|
||||||
|
const int64_t n_tokens = mixes->ne[1];
|
||||||
|
|
||||||
|
GGML_ASSERT(mixes->ne[0] == hc_mix_dim);
|
||||||
|
GGML_ASSERT(dst->ne[0] == hc);
|
||||||
|
GGML_ASSERT(dst->ne[1] == hc);
|
||||||
|
GGML_ASSERT(dst->ne[2] == n_tokens);
|
||||||
|
GGML_ASSERT(scale->ne[0] >= 3);
|
||||||
|
GGML_ASSERT(base->ne[0] == hc_mix_dim);
|
||||||
|
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbm, mixes, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbs, scale, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbb, base, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbd, dst, nb);
|
||||||
|
|
||||||
|
const float eps = ggml_get_op_params_f32(dst, 0);
|
||||||
|
const int32_t n_iter = ggml_get_op_params_i32(dst, 1);
|
||||||
|
GGML_ASSERT(n_iter > 0);
|
||||||
|
|
||||||
|
const int ith = params->ith;
|
||||||
|
const int nth = params->nth;
|
||||||
|
|
||||||
|
const int64_t dr = (n_tokens + nth - 1) / nth;
|
||||||
|
const int64_t it0 = dr * ith;
|
||||||
|
const int64_t it1 = MIN(it0 + dr, n_tokens);
|
||||||
|
|
||||||
|
const float scale_comb = *(const float *) ((const char *) scale->data + 2*nbs0);
|
||||||
|
|
||||||
|
for (int64_t it = it0; it < it1; ++it) {
|
||||||
|
float comb[hc*hc];
|
||||||
|
|
||||||
|
for (int64_t isrc = 0; isrc < hc; ++isrc) {
|
||||||
|
float max = -INFINITY;
|
||||||
|
for (int64_t idst = 0; idst < hc; ++idst) {
|
||||||
|
const int64_t idx = idst + hc*isrc;
|
||||||
|
const float xv = *(const float *) ((const char *) mixes->data + (comb_offset + idx)*nbm0 + it*nbm1);
|
||||||
|
const float bv = *(const float *) ((const char *) base->data + (comb_offset + idx)*nbb0);
|
||||||
|
const float v = xv * scale_comb + bv;
|
||||||
|
comb[idx] = v;
|
||||||
|
max = MAX(max, v);
|
||||||
|
}
|
||||||
|
|
||||||
|
float sum = 0.0f;
|
||||||
|
for (int64_t idst = 0; idst < hc; ++idst) {
|
||||||
|
const int64_t idx = idst + hc*isrc;
|
||||||
|
const float v = expf(comb[idx] - max);
|
||||||
|
comb[idx] = v;
|
||||||
|
sum += v;
|
||||||
|
}
|
||||||
|
|
||||||
|
const float inv_sum = 1.0f / sum;
|
||||||
|
for (int64_t idst = 0; idst < hc; ++idst) {
|
||||||
|
const int64_t idx = idst + hc*isrc;
|
||||||
|
comb[idx] = comb[idx] * inv_sum + eps;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
ggml_dsv4_hc_comb_norm_cols(comb, eps);
|
||||||
|
for (int32_t i = 1; i < n_iter; ++i) {
|
||||||
|
ggml_dsv4_hc_comb_norm_rows(comb, eps);
|
||||||
|
ggml_dsv4_hc_comb_norm_cols(comb, eps);
|
||||||
|
}
|
||||||
|
|
||||||
|
for (int64_t isrc = 0; isrc < hc; ++isrc) {
|
||||||
|
for (int64_t idst = 0; idst < hc; ++idst) {
|
||||||
|
const int64_t idx = idst + hc*isrc;
|
||||||
|
*(float *) ((char *) dst->data + idst*nbd0 + isrc*nbd1 + it*nbd2) = comb[idx];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
void ggml_compute_forward_dsv4_hc_comb(
|
||||||
|
const ggml_compute_params * params,
|
||||||
|
ggml_tensor * dst) {
|
||||||
|
const ggml_tensor * src0 = dst->src[0];
|
||||||
|
|
||||||
|
switch (src0->type) {
|
||||||
|
case GGML_TYPE_F32:
|
||||||
|
{
|
||||||
|
ggml_compute_forward_dsv4_hc_comb_f32(params, dst);
|
||||||
|
} break;
|
||||||
|
default:
|
||||||
|
{
|
||||||
|
GGML_ABORT("fatal error");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ggml_compute_forward_dsv4_hc_pre
|
||||||
|
|
||||||
|
static void ggml_compute_forward_dsv4_hc_pre_f32(
|
||||||
|
const ggml_compute_params * params,
|
||||||
|
ggml_tensor * dst) {
|
||||||
|
const ggml_tensor * x = dst->src[0];
|
||||||
|
const ggml_tensor * weights = dst->src[1];
|
||||||
|
|
||||||
|
GGML_ASSERT(x->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(weights->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(dst->type == GGML_TYPE_F32);
|
||||||
|
|
||||||
|
const int64_t n_embd = x->ne[0];
|
||||||
|
const int64_t hc = x->ne[1];
|
||||||
|
const int64_t n_tokens = x->ne[2];
|
||||||
|
|
||||||
|
GGML_ASSERT(dst->ne[0] == n_embd);
|
||||||
|
GGML_ASSERT(dst->ne[1] == n_tokens);
|
||||||
|
GGML_ASSERT(weights->ne[0] == hc);
|
||||||
|
GGML_ASSERT(weights->ne[1] == n_tokens);
|
||||||
|
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbx, x, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbw, weights, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbd, dst, nb);
|
||||||
|
|
||||||
|
const int ith = params->ith;
|
||||||
|
const int nth = params->nth;
|
||||||
|
|
||||||
|
const int64_t nr = n_embd * n_tokens;
|
||||||
|
const int64_t dr = (nr + nth - 1) / nth;
|
||||||
|
const int64_t ir0 = dr * ith;
|
||||||
|
const int64_t ir1 = MIN(ir0 + dr, nr);
|
||||||
|
|
||||||
|
for (int64_t ir = ir0; ir < ir1; ++ir) {
|
||||||
|
const int64_t i0 = ir % n_embd;
|
||||||
|
const int64_t it = ir / n_embd;
|
||||||
|
|
||||||
|
float sum = 0.0f;
|
||||||
|
for (int64_t ih = 0; ih < hc; ++ih) {
|
||||||
|
const float xv = *(const float *) ((const char *) x->data + i0*nbx0 + ih*nbx1 + it*nbx2);
|
||||||
|
const float wv = *(const float *) ((const char *) weights->data + ih*nbw0 + it*nbw1);
|
||||||
|
sum += xv * wv;
|
||||||
|
}
|
||||||
|
|
||||||
|
*(float *) ((char *) dst->data + i0*nbd0 + it*nbd1) = sum;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
void ggml_compute_forward_dsv4_hc_pre(
|
||||||
|
const ggml_compute_params * params,
|
||||||
|
ggml_tensor * dst) {
|
||||||
|
const ggml_tensor * src0 = dst->src[0];
|
||||||
|
|
||||||
|
switch (src0->type) {
|
||||||
|
case GGML_TYPE_F32:
|
||||||
|
{
|
||||||
|
ggml_compute_forward_dsv4_hc_pre_f32(params, dst);
|
||||||
|
} break;
|
||||||
|
default:
|
||||||
|
{
|
||||||
|
GGML_ABORT("fatal error");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ggml_compute_forward_dsv4_hc_post
|
||||||
|
|
||||||
|
static void ggml_compute_forward_dsv4_hc_post_f32(
|
||||||
|
const ggml_compute_params * params,
|
||||||
|
ggml_tensor * dst) {
|
||||||
|
const ggml_tensor * x = dst->src[0];
|
||||||
|
const ggml_tensor * residual = dst->src[1];
|
||||||
|
const ggml_tensor * post = dst->src[2];
|
||||||
|
const ggml_tensor * comb = dst->src[3];
|
||||||
|
|
||||||
|
GGML_ASSERT(x->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(residual->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(post->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(comb->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(dst->type == GGML_TYPE_F32);
|
||||||
|
|
||||||
|
const int64_t n_embd = x->ne[0];
|
||||||
|
const int64_t n_tokens = x->ne[1];
|
||||||
|
const int64_t hc = residual->ne[1];
|
||||||
|
|
||||||
|
GGML_ASSERT(dst->ne[0] == n_embd);
|
||||||
|
GGML_ASSERT(dst->ne[1] == hc);
|
||||||
|
GGML_ASSERT(dst->ne[2] == n_tokens);
|
||||||
|
GGML_ASSERT(residual->ne[0] == n_embd);
|
||||||
|
GGML_ASSERT(residual->ne[2] == n_tokens);
|
||||||
|
GGML_ASSERT(post->ne[0] == hc);
|
||||||
|
GGML_ASSERT(post->ne[1] == n_tokens);
|
||||||
|
GGML_ASSERT(comb->ne[0] == hc);
|
||||||
|
GGML_ASSERT(comb->ne[1] == hc);
|
||||||
|
GGML_ASSERT(comb->ne[2] == n_tokens);
|
||||||
|
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbx, x, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbr, residual, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbp, post, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbc, comb, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbd, dst, nb);
|
||||||
|
|
||||||
|
const int ith = params->ith;
|
||||||
|
const int nth = params->nth;
|
||||||
|
|
||||||
|
const int64_t nr = n_embd * hc * n_tokens;
|
||||||
|
const int64_t dr = (nr + nth - 1) / nth;
|
||||||
|
const int64_t ir0 = dr * ith;
|
||||||
|
const int64_t ir1 = MIN(ir0 + dr, nr);
|
||||||
|
|
||||||
|
for (int64_t ir = ir0; ir < ir1; ++ir) {
|
||||||
|
const int64_t i0 = ir % n_embd;
|
||||||
|
const int64_t idst = (ir / n_embd) % hc;
|
||||||
|
const int64_t it = ir / (n_embd * hc);
|
||||||
|
|
||||||
|
const float xv = *(const float *) ((const char *) x->data + i0*nbx0 + it*nbx1);
|
||||||
|
const float pv = *(const float *) ((const char *) post->data + idst*nbp0 + it*nbp1);
|
||||||
|
|
||||||
|
float sum = xv * pv;
|
||||||
|
for (int64_t isrc = 0; isrc < hc; ++isrc) {
|
||||||
|
const float rv = *(const float *) ((const char *) residual->data + i0*nbr0 + isrc*nbr1 + it*nbr2);
|
||||||
|
const float cv = *(const float *) ((const char *) comb->data + idst*nbc0 + isrc*nbc1 + it*nbc2);
|
||||||
|
sum += rv * cv;
|
||||||
|
}
|
||||||
|
|
||||||
|
*(float *) ((char *) dst->data + i0*nbd0 + idst*nbd1 + it*nbd2) = sum;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
void ggml_compute_forward_dsv4_hc_post(
|
||||||
|
const ggml_compute_params * params,
|
||||||
|
ggml_tensor * dst) {
|
||||||
|
const ggml_tensor * src0 = dst->src[0];
|
||||||
|
|
||||||
|
switch (src0->type) {
|
||||||
|
case GGML_TYPE_F32:
|
||||||
|
{
|
||||||
|
ggml_compute_forward_dsv4_hc_post_f32(params, dst);
|
||||||
|
} break;
|
||||||
|
default:
|
||||||
|
{
|
||||||
|
GGML_ABORT("fatal error");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ggml_compute_forward_rwkv_wkv7
|
// ggml_compute_forward_rwkv_wkv7
|
||||||
|
|
||||||
static void ggml_compute_forward_rwkv_wkv7_f32(
|
static void ggml_compute_forward_rwkv_wkv7_f32(
|
||||||
@@ -11568,3 +11918,87 @@ void ggml_compute_forward_fwht(const ggml_compute_params * params, ggml_tensor *
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ggml_compute_forward_lightning_indexer
|
||||||
|
|
||||||
|
void ggml_compute_forward_lightning_indexer(
|
||||||
|
const ggml_compute_params * params,
|
||||||
|
ggml_tensor * dst) {
|
||||||
|
|
||||||
|
const ggml_tensor * q = dst->src[0];
|
||||||
|
const ggml_tensor * k = dst->src[1];
|
||||||
|
const ggml_tensor * w = dst->src[2]; // weights
|
||||||
|
const ggml_tensor * m = dst->src[3]; // mask
|
||||||
|
|
||||||
|
GGML_ASSERT(dst->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT( q->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT( w->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT( m->type == GGML_TYPE_F16);
|
||||||
|
|
||||||
|
GGML_TENSOR_LOCALS(int64_t, neq, q, ne)
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbq, q, nb)
|
||||||
|
GGML_TENSOR_LOCALS(int64_t, nek, k, ne)
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbk, k, nb)
|
||||||
|
GGML_TENSOR_LOCALS(int64_t, new, w, ne)
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbw, w, nb)
|
||||||
|
GGML_TENSOR_LOCALS(int64_t, nem, m, ne)
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbm, m, nb)
|
||||||
|
GGML_TENSOR_LOCALS(int64_t, ne, dst, ne)
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nb, dst, nb)
|
||||||
|
|
||||||
|
GGML_ASSERT( nb0 == ggml_type_size(dst->type));
|
||||||
|
GGML_ASSERT(nbq0 == ggml_type_size( q->type));
|
||||||
|
GGML_ASSERT(nbk0 == ggml_type_size( k->type));
|
||||||
|
GGML_ASSERT(nbw0 == ggml_type_size( w->type));
|
||||||
|
GGML_ASSERT(nbm0 == ggml_type_size( m->type));
|
||||||
|
|
||||||
|
const int n_embd = q->ne[0];
|
||||||
|
const int n_head = q->ne[1];
|
||||||
|
const int n_tokens = q->ne[2];
|
||||||
|
const int n_stream = q->ne[3];
|
||||||
|
const int n_kv = k->ne[2];
|
||||||
|
|
||||||
|
ggml_to_float_t const k_to_float = ggml_get_type_traits(k->type)->to_float;
|
||||||
|
GGML_ASSERT((k->type == GGML_TYPE_F32 || k_to_float) && "lightning indexer: unsupported K-type");
|
||||||
|
|
||||||
|
const int nr = n_kv;
|
||||||
|
const int ith = params->ith;
|
||||||
|
const int nth = params->nth;
|
||||||
|
|
||||||
|
// (temporary) buffer for K converted to float
|
||||||
|
float * k_row_f32 = (float *) params->wdata + ith*(1*n_embd + CACHE_LINE_SIZE_F32);
|
||||||
|
|
||||||
|
// rows per thread
|
||||||
|
const int dr = (nr + nth - 1)/nth;
|
||||||
|
|
||||||
|
// row range for this thread
|
||||||
|
const int ir0 = dr*ith;
|
||||||
|
const int ir1 = MIN(ir0 + dr, nr);
|
||||||
|
|
||||||
|
for (int s = 0; s < n_stream; ++s) {
|
||||||
|
for (int t = 0; t < n_tokens; ++t) {
|
||||||
|
const float * w_row = (float *) ((char *) w->data + t*nbw1 + s*nbw3);
|
||||||
|
const ggml_fp16_t * m_row = (ggml_fp16_t *) ((char *) m->data + t*nbm1 + (s%nem3)*nbm3);
|
||||||
|
float * dst_row = (float *) ((char *) dst->data + t*nb1 + s*nb3 );
|
||||||
|
for (int ik = ir0; ik < ir1; ++ik) {
|
||||||
|
char * k_row = (char *) k->data + ik*nbk2 + s*nbk3;
|
||||||
|
if (k_to_float) {
|
||||||
|
k_to_float(k_row, k_row_f32, n_embd);
|
||||||
|
} else {
|
||||||
|
k_row_f32 = (float *) k_row;
|
||||||
|
}
|
||||||
|
float score = 0.0f;
|
||||||
|
for (int h = 0; h < n_head; ++h) {
|
||||||
|
// dot product of q and k for head h
|
||||||
|
float qk = 0.0f;
|
||||||
|
const float * q_row = (float *) ((char *) q->data + h*nbq1 + t*nbq2 + s*nbq3);
|
||||||
|
ggml_vec_dot_f32(n_embd, &qk, 0, q_row, 0, k_row_f32, 0, 1);
|
||||||
|
// ReLU and weights (prescaled)
|
||||||
|
score += MAX(qk, 0.0f) * w_row[h];
|
||||||
|
}
|
||||||
|
// apply mask
|
||||||
|
dst_row[ik] = score + GGML_CPU_FP16_TO_FP32(m_row[ik]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -105,6 +105,10 @@ void ggml_compute_forward_rwkv_wkv7(const struct ggml_compute_params * params, s
|
|||||||
void ggml_compute_forward_solve_tri(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
void ggml_compute_forward_solve_tri(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
||||||
void ggml_compute_forward_gla(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
void ggml_compute_forward_gla(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
||||||
void ggml_compute_forward_gated_delta_net(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
void ggml_compute_forward_gated_delta_net(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
||||||
|
void ggml_compute_forward_lightning_indexer(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
||||||
|
void ggml_compute_forward_dsv4_hc_comb(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
||||||
|
void ggml_compute_forward_dsv4_hc_pre(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
||||||
|
void ggml_compute_forward_dsv4_hc_post(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
||||||
void ggml_compute_forward_map_custom1(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
void ggml_compute_forward_map_custom1(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
||||||
void ggml_compute_forward_map_custom2(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
void ggml_compute_forward_map_custom2(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
||||||
void ggml_compute_forward_map_custom3(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
void ggml_compute_forward_map_custom3(const struct ggml_compute_params * params, struct ggml_tensor * dst);
|
||||||
|
|||||||
@@ -2739,7 +2739,7 @@ static block_q8_0x4 make_block_q8_0x4(block_q8_0 * in, unsigned int blck_size_in
|
|||||||
return out;
|
return out;
|
||||||
}
|
}
|
||||||
|
|
||||||
static block_q4_0x4 make_block_q4_0x4(block_q4_0 * in, unsigned int blck_size_interleave) {
|
static block_q4_0x4 make_block_q4_0x4(block_q4_0 * in, int blck_size_interleave) {
|
||||||
block_q4_0x4 out;
|
block_q4_0x4 out;
|
||||||
|
|
||||||
for (int i = 0; i < 4; i++) {
|
for (int i = 0; i < 4; i++) {
|
||||||
|
|||||||
@@ -28,6 +28,20 @@ static __global__ void init_offsets(int * offsets, const int ncols, const int nr
|
|||||||
#endif // STRIDED_ITERATOR_AVAILABLE
|
#endif // STRIDED_ITERATOR_AVAILABLE
|
||||||
|
|
||||||
#ifdef GGML_CUDA_USE_CUB
|
#ifdef GGML_CUDA_USE_CUB
|
||||||
|
|
||||||
|
// returns the suggested maximum number of rows to process during one argsort_f32_i32_cuda_cub() call
|
||||||
|
int argsort_f32_i32_cuda_cub_chunk_nrows(const size_t nb01, const int64_t nrows) {
|
||||||
|
// perform argsort in chunks up to approximately this size (currently 64MB)
|
||||||
|
// to avoid excessive temporary buffers memory usage
|
||||||
|
const int chunk_bytes = 1 << 26;
|
||||||
|
|
||||||
|
// calculate how many rows will fit in one chunk (must be at least one)
|
||||||
|
const int chunk_nrows = std::max((int) (chunk_bytes / nb01), 1);
|
||||||
|
|
||||||
|
// limit the resulting amount to total nrows
|
||||||
|
return std::min((int64_t) chunk_nrows, nrows);
|
||||||
|
}
|
||||||
|
|
||||||
void argsort_f32_i32_cuda_cub(ggml_cuda_pool & pool,
|
void argsort_f32_i32_cuda_cub(ggml_cuda_pool & pool,
|
||||||
const float * x,
|
const float * x,
|
||||||
int * dst,
|
int * dst,
|
||||||
@@ -254,11 +268,23 @@ void ggml_cuda_op_argsort(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
|
|||||||
const size_t shared_mem = ncols_pad * sizeof(int);
|
const size_t shared_mem = ncols_pad * sizeof(int);
|
||||||
const size_t max_shared_mem = ggml_cuda_info().devices[ggml_cuda_get_device()].smpb;
|
const size_t max_shared_mem = ggml_cuda_info().devices[ggml_cuda_get_device()].smpb;
|
||||||
|
|
||||||
if (shared_mem > max_shared_mem || ncols > 1024) {
|
// early return if we can use bitonic argsort
|
||||||
ggml_cuda_pool & pool = ctx.pool();
|
if (shared_mem <= max_shared_mem && ncols <= 1024) {
|
||||||
argsort_f32_i32_cuda_cub(pool, src0_d, (int *) dst_d, ncols, nrows, order, stream);
|
|
||||||
} else {
|
|
||||||
argsort_f32_i32_cuda_bitonic(src0_d, (int *) dst_d, ncols, nrows, order, stream);
|
argsort_f32_i32_cuda_bitonic(src0_d, (int *) dst_d, ncols, nrows, order, stream);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int chunk_nrows = argsort_f32_i32_cuda_cub_chunk_nrows(src0->nb[1], nrows);
|
||||||
|
|
||||||
|
ggml_cuda_pool & pool = ctx.pool();
|
||||||
|
|
||||||
|
for (int64_t i = 0; i < nrows; i += chunk_nrows) {
|
||||||
|
int iter_nrows = std::min((int64_t) chunk_nrows, nrows - i);
|
||||||
|
|
||||||
|
argsort_f32_i32_cuda_cub(pool, src0_d, (int *) dst_d, ncols, iter_nrows, order, stream);
|
||||||
|
|
||||||
|
src0_d += ncols * iter_nrows;
|
||||||
|
dst_d += ncols * iter_nrows;
|
||||||
}
|
}
|
||||||
#else
|
#else
|
||||||
argsort_f32_i32_cuda_bitonic(src0_d, (int *) dst_d, ncols, nrows, order, stream);
|
argsort_f32_i32_cuda_bitonic(src0_d, (int *) dst_d, ncols, nrows, order, stream);
|
||||||
|
|||||||
@@ -3,6 +3,7 @@
|
|||||||
void ggml_cuda_op_argsort(ggml_backend_cuda_context & ctx, ggml_tensor * dst);
|
void ggml_cuda_op_argsort(ggml_backend_cuda_context & ctx, ggml_tensor * dst);
|
||||||
|
|
||||||
#ifdef GGML_CUDA_USE_CUB
|
#ifdef GGML_CUDA_USE_CUB
|
||||||
|
int argsort_f32_i32_cuda_cub_chunk_nrows(const size_t nb01, const int64_t nrows);
|
||||||
void argsort_f32_i32_cuda_cub(ggml_cuda_pool & pool,
|
void argsort_f32_i32_cuda_cub(ggml_cuda_pool & pool,
|
||||||
const float * x,
|
const float * x,
|
||||||
int * dst,
|
int * dst,
|
||||||
|
|||||||
@@ -362,6 +362,15 @@ static bool blackwell_mma_available(const int cc) {
|
|||||||
ggml_cuda_highest_compiled_arch(cc) < GGML_CUDA_CC_RUBIN;
|
ggml_cuda_highest_compiled_arch(cc) < GGML_CUDA_CC_RUBIN;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Checks whether the tensor's base data pointer and higher-dimensional strides are byte-aligned to `alignment` bytes.
|
||||||
|
static bool ggml_cuda_is_aligned(const ggml_tensor * tensor, const size_t alignment) {
|
||||||
|
GGML_ASSERT(tensor != nullptr);
|
||||||
|
return (reinterpret_cast<uintptr_t>(tensor->data) % alignment) == 0 &&
|
||||||
|
tensor->nb[1] % alignment == 0 &&
|
||||||
|
tensor->nb[2] % alignment == 0 &&
|
||||||
|
tensor->nb[3] % alignment == 0;
|
||||||
|
}
|
||||||
|
|
||||||
static constexpr __device__ int ggml_cuda_get_physical_warp_size() {
|
static constexpr __device__ int ggml_cuda_get_physical_warp_size() {
|
||||||
#if defined(GGML_USE_HIP) && (defined(__GFX9__) || defined(__GFX8__))
|
#if defined(GGML_USE_HIP) && (defined(__GFX9__) || defined(__GFX8__))
|
||||||
return 64;
|
return 64;
|
||||||
@@ -937,6 +946,9 @@ static __device__ __forceinline__ uint2 fast_div_modulo(uint32_t n, const uint3
|
|||||||
|
|
||||||
typedef void (*dequantize_kernel_t)(const void * vx, const int64_t ib, const int iqs, float2 & v);
|
typedef void (*dequantize_kernel_t)(const void * vx, const int64_t ib, const int iqs, float2 & v);
|
||||||
|
|
||||||
|
template<typename dst_t>
|
||||||
|
using dequantize_kq_t = void (*)(const void * vx, const int64_t ib, dst_t * y, const int tid);
|
||||||
|
|
||||||
static __device__ __forceinline__ float get_alibi_slope(
|
static __device__ __forceinline__ float get_alibi_slope(
|
||||||
const float max_bias, const uint32_t h, const uint32_t n_head_log2, const float m0, const float m1
|
const float max_bias, const uint32_t h, const uint32_t n_head_log2, const float m0, const float m1
|
||||||
) {
|
) {
|
||||||
@@ -965,6 +977,13 @@ struct ggml_cuda_type_traits<GGML_TYPE_Q1_0> {
|
|||||||
static constexpr int qi = QI1_0;
|
static constexpr int qi = QI1_0;
|
||||||
};
|
};
|
||||||
|
|
||||||
|
template<>
|
||||||
|
struct ggml_cuda_type_traits<GGML_TYPE_Q2_0> {
|
||||||
|
static constexpr int qk = QK2_0;
|
||||||
|
static constexpr int qr = QR2_0;
|
||||||
|
static constexpr int qi = QI2_0;
|
||||||
|
};
|
||||||
|
|
||||||
template<>
|
template<>
|
||||||
struct ggml_cuda_type_traits<GGML_TYPE_Q4_0> {
|
struct ggml_cuda_type_traits<GGML_TYPE_Q4_0> {
|
||||||
static constexpr int qk = QK4_0;
|
static constexpr int qk = QK4_0;
|
||||||
@@ -1115,7 +1134,8 @@ struct ggml_cuda_type_traits<GGML_TYPE_IQ3_S> {
|
|||||||
//////////////////////
|
//////////////////////
|
||||||
|
|
||||||
struct ggml_cuda_device_info {
|
struct ggml_cuda_device_info {
|
||||||
int device_count;
|
int device_count; // number of (possibly virtual) devices exposed to the rest of ggml
|
||||||
|
int physical_device_count; // number of physical CUDA devices actually present
|
||||||
|
|
||||||
struct cuda_device_info {
|
struct cuda_device_info {
|
||||||
int cc; // compute capability
|
int cc; // compute capability
|
||||||
@@ -1128,6 +1148,9 @@ struct ggml_cuda_device_info {
|
|||||||
size_t total_vram;
|
size_t total_vram;
|
||||||
int warp_size; // Number of threads in a dispatch
|
int warp_size; // Number of threads in a dispatch
|
||||||
bool supports_cooperative_launch; // whether cooperative launch is supported
|
bool supports_cooperative_launch; // whether cooperative launch is supported
|
||||||
|
int physical_device; // backing physical CUDA device for this (virtual) device
|
||||||
|
int physical_share_count; // number of (virtual) devices sharing this device's physical GPU
|
||||||
|
int virtual_index; // index of this (virtual) device among those sharing its physical GPU
|
||||||
};
|
};
|
||||||
|
|
||||||
cuda_device_info devices[GGML_CUDA_MAX_DEVICES] = {};
|
cuda_device_info devices[GGML_CUDA_MAX_DEVICES] = {};
|
||||||
|
|||||||
@@ -141,27 +141,25 @@ static __global__ void __launch_bounds__(CUDA_CONCAT_BLOCK_SIZE)
|
|||||||
|
|
||||||
template <typename T>
|
template <typename T>
|
||||||
static void concat_cuda(const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, int dim, cudaStream_t stream) {
|
static void concat_cuda(const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, int dim, cudaStream_t stream) {
|
||||||
if (ggml_is_contiguous(src0) && ggml_is_contiguous(src1)) {
|
if (dim != 3 && ggml_is_contiguous_to_3(src0) && ggml_is_contiguous_to_3(src1)) {
|
||||||
const T * src0_d = (const T *) src0->data;
|
const T * src0_d = (const T *) src0->data;
|
||||||
const T * src1_d = (const T *) src1->data;
|
const T * src1_d = (const T *) src1->data;
|
||||||
T * dst_d = (T *) dst->data;
|
T * dst_d = (T *) dst->data;
|
||||||
|
|
||||||
if (dim != 3) {
|
for (int64_t i3 = 0; i3 < dst->ne[3]; i3++) {
|
||||||
for (int64_t i3 = 0; i3 < dst->ne[3]; i3++) {
|
concat_cont_cuda(
|
||||||
concat_cont_cuda(
|
src0_d + i3*(src0->nb[3] / sizeof(T)),
|
||||||
src0_d + i3*(src0->nb[3] / sizeof(T)),
|
src1_d + i3*(src1->nb[3] / sizeof(T)),
|
||||||
src1_d + i3*(src1->nb[3] / sizeof(T)),
|
dst_d + i3*( dst->nb[3] / sizeof(T)),
|
||||||
dst_d + i3*( dst->nb[3] / sizeof(T)),
|
ggml_row_size(src0->type, src0->ne[0])/sizeof(T), src0->ne[1], src0->ne[2],
|
||||||
ggml_row_size(src0->type, src0->ne[0])/sizeof(T), src0->ne[1], src0->ne[2],
|
ggml_row_size(dst->type, dst->ne[0])/sizeof(T), dst->ne[1], dst->ne[2], dim, stream);
|
||||||
ggml_row_size(dst->type, dst->ne[0])/sizeof(T), dst->ne[1], dst->ne[2], dim, stream);
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
const size_t size0 = ggml_nbytes(src0);
|
|
||||||
const size_t size1 = ggml_nbytes(src1);
|
|
||||||
|
|
||||||
CUDA_CHECK(cudaMemcpyAsync((char *) dst->data, src0->data, size0, cudaMemcpyDeviceToDevice, stream));
|
|
||||||
CUDA_CHECK(cudaMemcpyAsync((char *) dst->data + size0, src1->data, size1, cudaMemcpyDeviceToDevice, stream));
|
|
||||||
}
|
}
|
||||||
|
} else if (dim == 3 && ggml_is_contiguous(src0) && ggml_is_contiguous(src1)) {
|
||||||
|
const size_t size0 = ggml_nbytes(src0);
|
||||||
|
const size_t size1 = ggml_nbytes(src1);
|
||||||
|
|
||||||
|
CUDA_CHECK(cudaMemcpyAsync((char *) dst->data, src0->data, size0, cudaMemcpyDeviceToDevice, stream));
|
||||||
|
CUDA_CHECK(cudaMemcpyAsync((char *) dst->data + size0, src1->data, size1, cudaMemcpyDeviceToDevice, stream));
|
||||||
} else {
|
} else {
|
||||||
GGML_ASSERT(!ggml_is_quantized(src0->type));
|
GGML_ASSERT(!ggml_is_quantized(src0->type));
|
||||||
|
|
||||||
@@ -208,12 +206,17 @@ void ggml_cuda_op_concat(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
|
|||||||
GGML_ASSERT(dst->type == src0->type);
|
GGML_ASSERT(dst->type == src0->type);
|
||||||
|
|
||||||
if (ggml_is_quantized(src0->type)) {
|
if (ggml_is_quantized(src0->type)) {
|
||||||
GGML_ASSERT(ggml_is_contiguous(src0));
|
if (dim == 3) {
|
||||||
GGML_ASSERT(ggml_is_contiguous(src1));
|
GGML_ASSERT(ggml_is_contiguous(src0));
|
||||||
|
GGML_ASSERT(ggml_is_contiguous(src1));
|
||||||
|
} else {
|
||||||
|
GGML_ASSERT(ggml_is_contiguous_to_3(src0));
|
||||||
|
GGML_ASSERT(ggml_is_contiguous_to_3(src1));
|
||||||
|
}
|
||||||
GGML_ASSERT(src0->ne[0] % ggml_blck_size(src0->type) == 0);
|
GGML_ASSERT(src0->ne[0] % ggml_blck_size(src0->type) == 0);
|
||||||
GGML_ASSERT(src1->ne[0] % ggml_blck_size(src1->type) == 0);
|
GGML_ASSERT(src1->ne[0] % ggml_blck_size(src1->type) == 0);
|
||||||
|
|
||||||
// if tensors are contiguous and ne[0] is multiple of the block size we can concat both tensors as byte tensors
|
// if first 3 dimensions are contiguous and ne[0] is multiple of the block size we can concat both tensors as byte tensors
|
||||||
concat_cuda<uint8_t>(src0, src1, dst, dim, stream);
|
concat_cuda<uint8_t>(src0, src1, dst, dim, stream);
|
||||||
} else {
|
} else {
|
||||||
GGML_ASSERT(ggml_blck_size(src0->type) == 1);
|
GGML_ASSERT(ggml_blck_size(src0->type) == 1);
|
||||||
|
|||||||
+38
-277
@@ -140,358 +140,107 @@ static __global__ void dequantize_block_q4_1(const void * __restrict__ vx, dst_t
|
|||||||
|
|
||||||
template<typename dst_t>
|
template<typename dst_t>
|
||||||
static __global__ void dequantize_block_q2_K(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
static __global__ void dequantize_block_q2_K(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
||||||
|
const int64_t i = blockIdx.x;
|
||||||
|
|
||||||
const int64_t i = blockIdx.x;
|
dequantize_q2_K(vx, i, yy + i*QK_K, threadIdx.x);
|
||||||
const block_q2_K * x = (const block_q2_K *) vx;
|
|
||||||
|
|
||||||
const int64_t tid = threadIdx.x;
|
|
||||||
const int64_t n = tid/32;
|
|
||||||
const int64_t l = tid - 32*n;
|
|
||||||
const int64_t is = 8*n + l/16;
|
|
||||||
|
|
||||||
const uint8_t q = x[i].qs[32*n + l];
|
|
||||||
dst_t * y = yy + i*QK_K + 128*n;
|
|
||||||
|
|
||||||
float dall = __low2half(x[i].dm);
|
|
||||||
float dmin = __high2half(x[i].dm);
|
|
||||||
y[l+ 0] = ggml_cuda_cast<dst_t>(dall * (x[i].scales[is+0] & 0xF) * ((q >> 0) & 3) - dmin * (x[i].scales[is+0] >> 4));
|
|
||||||
y[l+32] = ggml_cuda_cast<dst_t>(dall * (x[i].scales[is+2] & 0xF) * ((q >> 2) & 3) - dmin * (x[i].scales[is+2] >> 4));
|
|
||||||
y[l+64] = ggml_cuda_cast<dst_t>(dall * (x[i].scales[is+4] & 0xF) * ((q >> 4) & 3) - dmin * (x[i].scales[is+4] >> 4));
|
|
||||||
y[l+96] = ggml_cuda_cast<dst_t>(dall * (x[i].scales[is+6] & 0xF) * ((q >> 6) & 3) - dmin * (x[i].scales[is+6] >> 4));
|
|
||||||
}
|
}
|
||||||
|
|
||||||
template<typename dst_t>
|
template<typename dst_t>
|
||||||
static __global__ void dequantize_block_q3_K(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
static __global__ void dequantize_block_q3_K(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
||||||
|
|
||||||
const int64_t i = blockIdx.x;
|
const int64_t i = blockIdx.x;
|
||||||
const block_q3_K * x = (const block_q3_K *) vx;
|
|
||||||
|
|
||||||
const int64_t r = threadIdx.x/4;
|
dequantize_q3_K(vx, i, yy + i*QK_K, threadIdx.x);
|
||||||
const int64_t tid = r/2;
|
|
||||||
const int64_t is0 = r%2;
|
|
||||||
const int64_t l0 = 16*is0 + 4*(threadIdx.x%4);
|
|
||||||
const int64_t n = tid / 4;
|
|
||||||
const int64_t j = tid - 4*n;
|
|
||||||
|
|
||||||
uint8_t m = 1 << (4*n + j);
|
|
||||||
int64_t is = 8*n + 2*j + is0;
|
|
||||||
int shift = 2*j;
|
|
||||||
|
|
||||||
int8_t us = is < 4 ? (x[i].scales[is-0] & 0xF) | (((x[i].scales[is+8] >> 0) & 3) << 4) :
|
|
||||||
is < 8 ? (x[i].scales[is-0] & 0xF) | (((x[i].scales[is+4] >> 2) & 3) << 4) :
|
|
||||||
is < 12 ? (x[i].scales[is-8] >> 4) | (((x[i].scales[is+0] >> 4) & 3) << 4) :
|
|
||||||
(x[i].scales[is-8] >> 4) | (((x[i].scales[is-4] >> 6) & 3) << 4);
|
|
||||||
float d_all = x[i].d;
|
|
||||||
float dl = d_all * (us - 32);
|
|
||||||
|
|
||||||
dst_t * y = yy + i*QK_K + 128*n + 32*j;
|
|
||||||
const uint8_t * q = x[i].qs + 32*n;
|
|
||||||
const uint8_t * hm = x[i].hmask;
|
|
||||||
|
|
||||||
for (int l = l0; l < l0+4; ++l) {
|
|
||||||
y[l] = ggml_cuda_cast<dst_t>(dl * ((int8_t)((q[l] >> shift) & 3) - ((hm[l] & m) ? 0 : 4)));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
static inline __device__ void get_scale_min_k4(int j, const uint8_t * q, uint8_t & d, uint8_t & m) {
|
|
||||||
if (j < 4) {
|
|
||||||
d = q[j] & 63; m = q[j + 4] & 63;
|
|
||||||
} else {
|
|
||||||
d = (q[j+4] & 0xF) | ((q[j-4] >> 6) << 4);
|
|
||||||
m = (q[j+4] >> 4) | ((q[j-0] >> 6) << 4);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
template<typename dst_t>
|
template<typename dst_t>
|
||||||
static __global__ void dequantize_block_q4_K(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
static __global__ void dequantize_block_q4_K(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
||||||
const block_q4_K * x = (const block_q4_K *) vx;
|
|
||||||
|
|
||||||
const int64_t i = blockIdx.x;
|
const int64_t i = blockIdx.x;
|
||||||
|
|
||||||
// assume 32 threads
|
dequantize_q4_K(vx, i, yy + i*QK_K, threadIdx.x);
|
||||||
const int64_t tid = threadIdx.x;
|
|
||||||
const int64_t il = tid/8;
|
|
||||||
const int64_t ir = tid%8;
|
|
||||||
const int64_t is = 2*il;
|
|
||||||
const int64_t n = 4;
|
|
||||||
|
|
||||||
dst_t * y = yy + i*QK_K + 64*il + n*ir;
|
|
||||||
|
|
||||||
const float dall = __low2half(x[i].dm);
|
|
||||||
const float dmin = __high2half(x[i].dm);
|
|
||||||
|
|
||||||
const uint8_t * q = x[i].qs + 32*il + n*ir;
|
|
||||||
|
|
||||||
uint8_t sc, m;
|
|
||||||
get_scale_min_k4(is + 0, x[i].scales, sc, m);
|
|
||||||
const float d1 = dall * sc; const float m1 = dmin * m;
|
|
||||||
get_scale_min_k4(is + 1, x[i].scales, sc, m);
|
|
||||||
const float d2 = dall * sc; const float m2 = dmin * m;
|
|
||||||
for (int l = 0; l < n; ++l) {
|
|
||||||
y[l + 0] = ggml_cuda_cast<dst_t>(d1 * (q[l] & 0xF) - m1);
|
|
||||||
y[l +32] = ggml_cuda_cast<dst_t>(d2 * (q[l] >> 4) - m2);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
template<typename dst_t>
|
template<typename dst_t>
|
||||||
static __global__ void dequantize_block_q5_K(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
static __global__ void dequantize_block_q5_K(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
||||||
const block_q5_K * x = (const block_q5_K *) vx;
|
|
||||||
|
|
||||||
const int64_t i = blockIdx.x;
|
const int64_t i = blockIdx.x;
|
||||||
|
|
||||||
// assume 64 threads - this is very slightly better than the one below
|
dequantize_q5_K(vx, i, yy + i*QK_K, threadIdx.x);
|
||||||
const int64_t tid = threadIdx.x;
|
|
||||||
const int64_t il = tid/16; // il is in 0...3
|
|
||||||
const int64_t ir = tid%16; // ir is in 0...15
|
|
||||||
const int64_t is = 2*il; // is is in 0...6
|
|
||||||
|
|
||||||
dst_t * y = yy + i*QK_K + 64*il + 2*ir;
|
|
||||||
|
|
||||||
const float dall = __low2half(x[i].dm);
|
|
||||||
const float dmin = __high2half(x[i].dm);
|
|
||||||
|
|
||||||
const uint8_t * ql = x[i].qs + 32*il + 2*ir;
|
|
||||||
const uint8_t * qh = x[i].qh + 2*ir;
|
|
||||||
|
|
||||||
uint8_t sc, m;
|
|
||||||
get_scale_min_k4(is + 0, x[i].scales, sc, m);
|
|
||||||
const float d1 = dall * sc; const float m1 = dmin * m;
|
|
||||||
get_scale_min_k4(is + 1, x[i].scales, sc, m);
|
|
||||||
const float d2 = dall * sc; const float m2 = dmin * m;
|
|
||||||
|
|
||||||
uint8_t hm = 1 << (2*il);
|
|
||||||
y[ 0] = ggml_cuda_cast<dst_t>(d1 * ((ql[ 0] & 0xF) + (qh[ 0] & hm ? 16 : 0)) - m1);
|
|
||||||
y[ 1] = ggml_cuda_cast<dst_t>(d1 * ((ql[ 1] & 0xF) + (qh[ 1] & hm ? 16 : 0)) - m1);
|
|
||||||
hm <<= 1;
|
|
||||||
y[32] = ggml_cuda_cast<dst_t>(d2 * ((ql[ 0] >> 4) + (qh[ 0] & hm ? 16 : 0)) - m2);
|
|
||||||
y[33] = ggml_cuda_cast<dst_t>(d2 * ((ql[ 1] >> 4) + (qh[ 1] & hm ? 16 : 0)) - m2);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
template<typename dst_t>
|
template<typename dst_t>
|
||||||
static __global__ void dequantize_block_q6_K(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
static __global__ void dequantize_block_q6_K(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
||||||
const block_q6_K * x = (const block_q6_K *) vx;
|
|
||||||
|
|
||||||
const int64_t i = blockIdx.x;
|
const int64_t i = blockIdx.x;
|
||||||
|
|
||||||
// assume 64 threads - this is very slightly better than the one below
|
dequantize_q6_K(vx, i, yy + i*QK_K, threadIdx.x);
|
||||||
const int64_t tid = threadIdx.x;
|
|
||||||
const int64_t ip = tid/32; // ip is 0 or 1
|
|
||||||
const int64_t il = tid - 32*ip; // 0...32
|
|
||||||
const int64_t is = 8*ip + il/16;
|
|
||||||
|
|
||||||
dst_t * y = yy + i*QK_K + 128*ip + il;
|
|
||||||
|
|
||||||
const float d = x[i].d;
|
|
||||||
|
|
||||||
const uint8_t * ql = x[i].ql + 64*ip + il;
|
|
||||||
const uint8_t qh = x[i].qh[32*ip + il];
|
|
||||||
const int8_t * sc = x[i].scales + is;
|
|
||||||
|
|
||||||
y[ 0] = ggml_cuda_cast<dst_t>(d * sc[0] * ((int8_t)((ql[ 0] & 0xF) | (((qh >> 0) & 3) << 4)) - 32));
|
|
||||||
y[32] = ggml_cuda_cast<dst_t>(d * sc[2] * ((int8_t)((ql[32] & 0xF) | (((qh >> 2) & 3) << 4)) - 32));
|
|
||||||
y[64] = ggml_cuda_cast<dst_t>(d * sc[4] * ((int8_t)((ql[ 0] >> 4) | (((qh >> 4) & 3) << 4)) - 32));
|
|
||||||
y[96] = ggml_cuda_cast<dst_t>(d * sc[6] * ((int8_t)((ql[32] >> 4) | (((qh >> 6) & 3) << 4)) - 32));
|
|
||||||
}
|
}
|
||||||
|
|
||||||
template<typename dst_t>
|
template<typename dst_t>
|
||||||
static __global__ void dequantize_block_iq2_xxs(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
static __global__ void dequantize_block_iq2_xxs(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
||||||
|
const int64_t i = blockIdx.x;
|
||||||
|
|
||||||
const int64_t i = blockIdx.x;
|
dequantize_iq2_xxs(vx, i, yy + i*QK_K, threadIdx.x);
|
||||||
const block_iq2_xxs * x = (const block_iq2_xxs *) vx;
|
|
||||||
|
|
||||||
const int64_t tid = threadIdx.x;
|
|
||||||
const int64_t il = tid/8; // 0...3
|
|
||||||
const int64_t ib = tid%8; // 0...7
|
|
||||||
dst_t * y = yy + i*QK_K + 32*ib + 8*il;
|
|
||||||
const uint16_t * q2 = x[i].qs + 4*ib;
|
|
||||||
const uint8_t * aux8 = (const uint8_t *)q2;
|
|
||||||
const uint8_t * grid = (const uint8_t *)(iq2xxs_grid + aux8[il]);
|
|
||||||
const uint32_t aux32 = q2[2] | (q2[3] << 16);
|
|
||||||
const float d = (float)x[i].d * (0.5f + (aux32 >> 28)) * 0.25f;
|
|
||||||
const uint8_t signs = ksigns_iq2xs[(aux32 >> 7*il) & 127];
|
|
||||||
for (int j = 0; j < 8; ++j) {
|
|
||||||
y[j] = ggml_cuda_cast<dst_t>(d * grid[j] * (signs & kmask_iq2xs[j] ? -1.f : 1.f));
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
template<typename dst_t>
|
template<typename dst_t>
|
||||||
static __global__ void dequantize_block_iq2_xs(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
static __global__ void dequantize_block_iq2_xs(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
||||||
|
const int64_t i = blockIdx.x;
|
||||||
|
|
||||||
const int64_t i = blockIdx.x;
|
dequantize_iq2_xs(vx, i, yy + i*QK_K, threadIdx.x);
|
||||||
const block_iq2_xs * x = (const block_iq2_xs *) vx;
|
|
||||||
|
|
||||||
const int64_t tid = threadIdx.x;
|
|
||||||
const int64_t il = tid/8; // 0...3
|
|
||||||
const int64_t ib = tid%8; // 0...7
|
|
||||||
dst_t * y = yy + i*QK_K + 32*ib + 8*il;
|
|
||||||
const uint16_t * q2 = x[i].qs + 4*ib;
|
|
||||||
const uint8_t * grid = (const uint8_t *)(iq2xs_grid + (q2[il] & 511));
|
|
||||||
const float d = (float)x[i].d * (0.5f + ((x[i].scales[ib] >> 4*(il/2)) & 0xf)) * 0.25f;
|
|
||||||
const uint8_t signs = ksigns_iq2xs[q2[il] >> 9];
|
|
||||||
for (int j = 0; j < 8; ++j) {
|
|
||||||
y[j] = ggml_cuda_cast<dst_t>(d * grid[j] * (signs & kmask_iq2xs[j] ? -1.f : 1.f));
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
template<typename dst_t>
|
template<typename dst_t>
|
||||||
static __global__ void dequantize_block_iq2_s(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
static __global__ void dequantize_block_iq2_s(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
||||||
|
const int64_t i = blockIdx.x;
|
||||||
|
|
||||||
const int64_t i = blockIdx.x;
|
dequantize_iq2_s(vx, i, yy + i*QK_K, threadIdx.x);
|
||||||
const block_iq2_s * x = (const block_iq2_s *) vx;
|
|
||||||
|
|
||||||
const int64_t tid = threadIdx.x;
|
|
||||||
const int64_t il = tid/8; // 0...3
|
|
||||||
const int64_t ib = tid%8; // 0...7
|
|
||||||
dst_t * y = yy + i*QK_K + 32*ib + 8*il;
|
|
||||||
const uint8_t * grid = (const uint8_t *)(iq2s_grid + (x[i].qs[4*ib+il] | ((x[i].qh[ib] << (8-2*il)) & 0x300)));
|
|
||||||
const float d = (float)x[i].d * (0.5f + ((x[i].scales[ib] >> 4*(il/2)) & 0xf)) * 0.25f;
|
|
||||||
const uint8_t signs = x[i].qs[QK_K/8+4*ib+il];
|
|
||||||
for (int j = 0; j < 8; ++j) {
|
|
||||||
y[j] = ggml_cuda_cast<dst_t>(d * grid[j] * (signs & kmask_iq2xs[j] ? -1.f : 1.f));
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
template<typename dst_t>
|
template<typename dst_t>
|
||||||
static __global__ void dequantize_block_iq3_xxs(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
static __global__ void dequantize_block_iq3_xxs(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
||||||
|
const int64_t i = blockIdx.x;
|
||||||
|
|
||||||
const int64_t i = blockIdx.x;
|
dequantize_iq3_xxs(vx, i, yy + i*QK_K, threadIdx.x);
|
||||||
const block_iq3_xxs * x = (const block_iq3_xxs *) vx;
|
|
||||||
|
|
||||||
const int64_t tid = threadIdx.x;
|
|
||||||
const int64_t il = tid/8; // 0...3
|
|
||||||
const int64_t ib = tid%8; // 0...7
|
|
||||||
dst_t * y = yy + i*QK_K + 32*ib + 8*il;
|
|
||||||
const uint8_t * q3 = x[i].qs + 8*ib;
|
|
||||||
const uint16_t * gas = (const uint16_t *)(x[i].qs + QK_K/4) + 2*ib;
|
|
||||||
const uint8_t * grid1 = (const uint8_t *)(iq3xxs_grid + q3[2*il+0]);
|
|
||||||
const uint8_t * grid2 = (const uint8_t *)(iq3xxs_grid + q3[2*il+1]);
|
|
||||||
const uint32_t aux32 = gas[0] | (gas[1] << 16);
|
|
||||||
const float d = (float)x[i].d * (0.5f + (aux32 >> 28)) * 0.5f;
|
|
||||||
const uint8_t signs = ksigns_iq2xs[(aux32 >> 7*il) & 127];
|
|
||||||
for (int j = 0; j < 4; ++j) {
|
|
||||||
y[j+0] = ggml_cuda_cast<dst_t>(d * grid1[j] * (signs & kmask_iq2xs[j+0] ? -1.f : 1.f));
|
|
||||||
y[j+4] = ggml_cuda_cast<dst_t>(d * grid2[j] * (signs & kmask_iq2xs[j+4] ? -1.f : 1.f));
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
template<typename dst_t>
|
template<typename dst_t>
|
||||||
static __global__ void dequantize_block_iq3_s(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
static __global__ void dequantize_block_iq3_s(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
||||||
|
const int64_t i = blockIdx.x;
|
||||||
|
|
||||||
const int64_t i = blockIdx.x;
|
dequantize_iq3_s(vx, i, yy + i*QK_K, threadIdx.x);
|
||||||
const block_iq3_s * x = (const block_iq3_s *) vx;
|
|
||||||
|
|
||||||
const int64_t tid = threadIdx.x;
|
|
||||||
const int64_t il = tid/8; // 0...3
|
|
||||||
const int64_t ib = tid%8; // 0...7
|
|
||||||
dst_t * y = yy + i*QK_K + 32*ib + 8*il;
|
|
||||||
const uint8_t * qs = x[i].qs + 8*ib;
|
|
||||||
const uint8_t * grid1 = (const uint8_t *)(iq3s_grid + (qs[2*il+0] | ((x[i].qh[ib] << (8-2*il)) & 256)));
|
|
||||||
const uint8_t * grid2 = (const uint8_t *)(iq3s_grid + (qs[2*il+1] | ((x[i].qh[ib] << (7-2*il)) & 256)));
|
|
||||||
const float d = (float)x[i].d * (1 + 2*((x[i].scales[ib/2] >> 4*(ib%2)) & 0xf));
|
|
||||||
const uint8_t signs = x[i].signs[4*ib + il];
|
|
||||||
for (int j = 0; j < 4; ++j) {
|
|
||||||
y[j+0] = ggml_cuda_cast<dst_t>(d * grid1[j] * (signs & kmask_iq2xs[j+0] ? -1.f : 1.f));
|
|
||||||
y[j+4] = ggml_cuda_cast<dst_t>(d * grid2[j] * (signs & kmask_iq2xs[j+4] ? -1.f : 1.f));
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
template<typename dst_t>
|
template<typename dst_t>
|
||||||
static __global__ void dequantize_block_iq1_s(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
static __global__ void dequantize_block_iq1_s(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
||||||
|
const int64_t i = blockIdx.x;
|
||||||
|
|
||||||
const int64_t i = blockIdx.x;
|
dequantize_iq1_s(vx, i, yy + i*QK_K, threadIdx.x);
|
||||||
const block_iq1_s * x = (const block_iq1_s *) vx;
|
|
||||||
|
|
||||||
const int64_t tid = threadIdx.x;
|
|
||||||
const int64_t il = tid/8; // 0...3
|
|
||||||
const int64_t ib = tid%8; // 0...7
|
|
||||||
dst_t * y = yy + i*QK_K + 32*ib + 8*il;
|
|
||||||
const float delta = x[i].qh[ib] & 0x8000 ? -1 - IQ1S_DELTA : -1 + IQ1S_DELTA;
|
|
||||||
const float d = (float)x[i].d * (2*((x[i].qh[ib] >> 12) & 7) + 1);
|
|
||||||
uint32_t grid32[2]; const int8_t * q = (const int8_t *)grid32;
|
|
||||||
grid32[0] = iq1s_grid_gpu[x[i].qs[4*ib+il] | (((x[i].qh[ib] >> 3*il) & 7) << 8)];
|
|
||||||
grid32[1] = (grid32[0] >> 4) & 0x0f0f0f0f;
|
|
||||||
grid32[0] &= 0x0f0f0f0f;
|
|
||||||
for (int j = 0; j < 8; ++j) {
|
|
||||||
y[j] = ggml_cuda_cast<dst_t>(d * (q[j] + delta));
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
template<typename dst_t>
|
template<typename dst_t>
|
||||||
static __global__ void dequantize_block_iq1_m(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
static __global__ void dequantize_block_iq1_m(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
||||||
|
const int64_t i = blockIdx.x;
|
||||||
|
|
||||||
const int64_t i = blockIdx.x;
|
dequantize_iq1_m(vx, i, yy + i*QK_K, threadIdx.x);
|
||||||
const block_iq1_m * x = (const block_iq1_m *) vx;
|
|
||||||
|
|
||||||
const int64_t tid = threadIdx.x;
|
|
||||||
const int64_t il = tid/8; // 0...3
|
|
||||||
const int64_t ib = tid%8; // 0...7
|
|
||||||
dst_t * y = yy + i*QK_K + 32*ib + 8*il;
|
|
||||||
const uint16_t * sc = (const uint16_t *)x[i].scales;
|
|
||||||
iq1m_scale_t scale;
|
|
||||||
scale.u16 = (sc[0] >> 12) | ((sc[1] >> 8) & 0x00f0) | ((sc[2] >> 4) & 0x0f00) | (sc[3] & 0xf000);
|
|
||||||
const int64_t ib16 = 2*ib + il/2; // sc[ib16/4] >> 3*(ib16%4) -> sc[ib/2] >> 3*((2*ib+il/2)%4);
|
|
||||||
const float d = (float)scale.f16 * (2*((sc[ib16/4] >> 3*(ib16%4)) & 0x7) + 1);
|
|
||||||
const float delta = x[i].qh[2*ib+il/2] & (0x08 << 4*(il%2)) ? -1 - IQ1M_DELTA : -1 + IQ1M_DELTA;
|
|
||||||
uint32_t grid32[2]; const int8_t * q = (const int8_t *)grid32;
|
|
||||||
grid32[0] = iq1s_grid_gpu[x[i].qs[4*ib+il] | (((x[i].qh[2*ib+il/2] >> 4*(il%2)) & 7) << 8)];
|
|
||||||
grid32[1] = (grid32[0] >> 4) & 0x0f0f0f0f;
|
|
||||||
grid32[0] &= 0x0f0f0f0f;
|
|
||||||
for (int j = 0; j < 8; ++j) {
|
|
||||||
y[j] = ggml_cuda_cast<dst_t>(d * (q[j] + delta));
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
template<typename dst_t>
|
template<typename dst_t>
|
||||||
static __global__ void dequantize_block_iq4_nl(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
static __global__ void dequantize_block_iq4_nl(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
||||||
|
const int64_t i = blockIdx.x;
|
||||||
|
|
||||||
const int64_t i = blockIdx.x;
|
dequantize_iq4_nl(vx, i, yy + i*QK_K, threadIdx.x);
|
||||||
const block_iq4_nl * x = (const block_iq4_nl *) vx + i*(QK_K/QK4_NL);
|
|
||||||
|
|
||||||
const int64_t tid = threadIdx.x;
|
|
||||||
const int64_t il = tid/8; // 0...3
|
|
||||||
const int64_t ib = tid%8; // 0...7
|
|
||||||
dst_t * y = yy + i*QK_K + 32*ib + 4*il;
|
|
||||||
const uint8_t * q4 = x[ib].qs + 4*il;
|
|
||||||
const float d = (float)x[ib].d;
|
|
||||||
for (int j = 0; j < 4; ++j) {
|
|
||||||
y[j+ 0] = ggml_cuda_cast<dst_t>(d * kvalues_iq4nl[q4[j] & 0xf]);
|
|
||||||
y[j+16] = ggml_cuda_cast<dst_t>(d * kvalues_iq4nl[q4[j] >> 4]);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
template<typename dst_t>
|
template<typename dst_t>
|
||||||
static __global__ void dequantize_block_iq4_xs(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
static __global__ void dequantize_block_iq4_xs(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
||||||
const int64_t i = blockIdx.x;
|
const int64_t i = blockIdx.x;
|
||||||
const block_iq4_xs * x = (const block_iq4_xs *)vx;
|
|
||||||
|
|
||||||
const int64_t tid = threadIdx.x;
|
dequantize_iq4_xs(vx, i, yy + i*QK_K, threadIdx.x);
|
||||||
const int64_t il = tid/8; // 0...3
|
|
||||||
const int64_t ib = tid%8; // 0...7
|
|
||||||
dst_t * y = yy + i*QK_K + 32*ib + 4*il;
|
|
||||||
const uint8_t * q4 = x[i].qs + 16*ib + 4*il;
|
|
||||||
const float d = (float)x[i].d * ((((x[i].scales_l[ib/2] >> 4*(ib%2)) & 0xf) | (((x[i].scales_h >> 2*ib) & 3) << 4)) - 32);
|
|
||||||
for (int j = 0; j < 4; ++j) {
|
|
||||||
y[j+ 0] = ggml_cuda_cast<dst_t>(d * kvalues_iq4nl[q4[j] & 0xf]);
|
|
||||||
y[j+16] = ggml_cuda_cast<dst_t>(d * kvalues_iq4nl[q4[j] >> 4]);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
template<typename dst_t>
|
template<typename dst_t>
|
||||||
static __global__ void dequantize_block_mxfp4(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
static __global__ void dequantize_block_mxfp4(const void * __restrict__ vx, dst_t * __restrict__ yy) {
|
||||||
|
const int64_t i = blockIdx.x;
|
||||||
|
|
||||||
const int64_t i = blockIdx.x;
|
dequantize_mxfp4(vx, i, yy + i*QK_K, threadIdx.x);
|
||||||
const block_mxfp4 * x = (const block_mxfp4 *) vx + i*(QK_K/QK_MXFP4);
|
|
||||||
|
|
||||||
const int64_t tid = threadIdx.x;
|
|
||||||
const int64_t il = tid/8; // 0...3
|
|
||||||
const int64_t ib = tid%8; // 0...7
|
|
||||||
dst_t * y = yy + i*QK_K + 32*ib + 4*il;
|
|
||||||
const uint8_t * q4 = x[ib].qs + 4*il;
|
|
||||||
const float d = ggml_cuda_e8m0_to_fp32(x[ib].e);
|
|
||||||
for (int j = 0; j < 4; ++j) {
|
|
||||||
y[j+ 0] = ggml_cuda_cast<dst_t>(d * kvalues_mxfp4[q4[j] & 0xf]*0.5f);
|
|
||||||
y[j+16] = ggml_cuda_cast<dst_t>(d * kvalues_mxfp4[q4[j] >> 4]*0.5f);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
template <int qk, int qr, dequantize_kernel_t dequantize_kernel, typename dst_t>
|
template <int qk, int qr, dequantize_kernel_t dequantize_kernel, typename dst_t>
|
||||||
@@ -710,6 +459,8 @@ to_bf16_cuda_t ggml_get_to_bf16_cuda(ggml_type type) {
|
|||||||
switch (type) {
|
switch (type) {
|
||||||
case GGML_TYPE_Q1_0:
|
case GGML_TYPE_Q1_0:
|
||||||
return dequantize_block_cont_cuda<QK1_0, QR1_0, dequantize_q1_0>;
|
return dequantize_block_cont_cuda<QK1_0, QR1_0, dequantize_q1_0>;
|
||||||
|
case GGML_TYPE_Q2_0:
|
||||||
|
return dequantize_block_cont_cuda<QK2_0, QR2_0, dequantize_q2_0>;
|
||||||
case GGML_TYPE_Q4_0:
|
case GGML_TYPE_Q4_0:
|
||||||
return dequantize_row_q4_0_cuda;
|
return dequantize_row_q4_0_cuda;
|
||||||
case GGML_TYPE_Q4_1:
|
case GGML_TYPE_Q4_1:
|
||||||
@@ -765,6 +516,8 @@ to_fp16_cuda_t ggml_get_to_fp16_cuda(ggml_type type) {
|
|||||||
switch (type) {
|
switch (type) {
|
||||||
case GGML_TYPE_Q1_0:
|
case GGML_TYPE_Q1_0:
|
||||||
return dequantize_block_cont_cuda<QK1_0, QR1_0, dequantize_q1_0>;
|
return dequantize_block_cont_cuda<QK1_0, QR1_0, dequantize_q1_0>;
|
||||||
|
case GGML_TYPE_Q2_0:
|
||||||
|
return dequantize_block_cont_cuda<QK2_0, QR2_0, dequantize_q2_0>;
|
||||||
case GGML_TYPE_Q4_0:
|
case GGML_TYPE_Q4_0:
|
||||||
return dequantize_row_q4_0_cuda;
|
return dequantize_row_q4_0_cuda;
|
||||||
case GGML_TYPE_Q4_1:
|
case GGML_TYPE_Q4_1:
|
||||||
@@ -823,6 +576,8 @@ to_fp32_cuda_t ggml_get_to_fp32_cuda(ggml_type type) {
|
|||||||
switch (type) {
|
switch (type) {
|
||||||
case GGML_TYPE_Q1_0:
|
case GGML_TYPE_Q1_0:
|
||||||
return dequantize_block_cont_cuda<QK1_0, QR1_0, dequantize_q1_0>;
|
return dequantize_block_cont_cuda<QK1_0, QR1_0, dequantize_q1_0>;
|
||||||
|
case GGML_TYPE_Q2_0:
|
||||||
|
return dequantize_block_cont_cuda<QK2_0, QR2_0, dequantize_q2_0>;
|
||||||
case GGML_TYPE_Q4_0:
|
case GGML_TYPE_Q4_0:
|
||||||
return dequantize_row_q4_0_cuda;
|
return dequantize_row_q4_0_cuda;
|
||||||
case GGML_TYPE_Q4_1:
|
case GGML_TYPE_Q4_1:
|
||||||
@@ -880,6 +635,8 @@ to_fp16_nc_cuda_t ggml_get_to_fp16_nc_cuda(ggml_type type) {
|
|||||||
return convert_unary_cuda<float>;
|
return convert_unary_cuda<float>;
|
||||||
case GGML_TYPE_Q1_0:
|
case GGML_TYPE_Q1_0:
|
||||||
return dequantize_block_cuda<QK1_0, QR1_0, dequantize_q1_0>;
|
return dequantize_block_cuda<QK1_0, QR1_0, dequantize_q1_0>;
|
||||||
|
case GGML_TYPE_Q2_0:
|
||||||
|
return dequantize_block_cuda<QK2_0, QR2_0, dequantize_q2_0>;
|
||||||
case GGML_TYPE_Q4_0:
|
case GGML_TYPE_Q4_0:
|
||||||
return dequantize_block_cuda<QK4_0, QR4_0, dequantize_q4_0>;
|
return dequantize_block_cuda<QK4_0, QR4_0, dequantize_q4_0>;
|
||||||
case GGML_TYPE_Q4_1:
|
case GGML_TYPE_Q4_1:
|
||||||
@@ -903,6 +660,8 @@ to_bf16_nc_cuda_t ggml_get_to_bf16_nc_cuda(ggml_type type) {
|
|||||||
return convert_unary_cuda<float, nv_bfloat16>;
|
return convert_unary_cuda<float, nv_bfloat16>;
|
||||||
case GGML_TYPE_Q1_0:
|
case GGML_TYPE_Q1_0:
|
||||||
return dequantize_block_cuda<QK1_0, QR1_0, dequantize_q1_0>;
|
return dequantize_block_cuda<QK1_0, QR1_0, dequantize_q1_0>;
|
||||||
|
case GGML_TYPE_Q2_0:
|
||||||
|
return dequantize_block_cuda<QK2_0, QR2_0, dequantize_q2_0>;
|
||||||
case GGML_TYPE_Q4_0:
|
case GGML_TYPE_Q4_0:
|
||||||
return dequantize_block_cuda<QK4_0, QR4_0, dequantize_q4_0>;
|
return dequantize_block_cuda<QK4_0, QR4_0, dequantize_q4_0>;
|
||||||
case GGML_TYPE_Q4_1:
|
case GGML_TYPE_Q4_1:
|
||||||
@@ -926,6 +685,8 @@ to_fp32_nc_cuda_t ggml_get_to_fp32_nc_cuda(ggml_type type) {
|
|||||||
return convert_unary_cuda<half, float>;
|
return convert_unary_cuda<half, float>;
|
||||||
case GGML_TYPE_Q1_0:
|
case GGML_TYPE_Q1_0:
|
||||||
return dequantize_block_cuda<QK1_0, QR1_0, dequantize_q1_0>;
|
return dequantize_block_cuda<QK1_0, QR1_0, dequantize_q1_0>;
|
||||||
|
case GGML_TYPE_Q2_0:
|
||||||
|
return dequantize_block_cuda<QK2_0, QR2_0, dequantize_q2_0>;
|
||||||
case GGML_TYPE_Q4_0:
|
case GGML_TYPE_Q4_0:
|
||||||
return dequantize_block_cuda<QK4_0, QR4_0, dequantize_q4_0>;
|
return dequantize_block_cuda<QK4_0, QR4_0, dequantize_q4_0>;
|
||||||
case GGML_TYPE_Q4_1:
|
case GGML_TYPE_Q4_1:
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
#include "common.cuh"
|
#include "common.cuh"
|
||||||
|
#include "convert.cuh"
|
||||||
|
|
||||||
static __device__ __forceinline__ void dequantize_q1_0(const void * vx, const int64_t ib, const int iqs, float2 & v){
|
static __device__ __forceinline__ void dequantize_q1_0(const void * vx, const int64_t ib, const int iqs, float2 & v){
|
||||||
const block_q1_0 * x = (const block_q1_0 *) vx;
|
const block_q1_0 * x = (const block_q1_0 *) vx;
|
||||||
@@ -22,6 +23,26 @@ static __device__ __forceinline__ void dequantize_q1_0(const void * vx, const in
|
|||||||
v.y = (2*bit_1 - 1) * d;
|
v.y = (2*bit_1 - 1) * d;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static __device__ __forceinline__ void dequantize_q2_0(const void * vx, const int64_t ib, const int iqs, float2 & v){
|
||||||
|
const block_q2_0 * x = (const block_q2_0 *) vx;
|
||||||
|
|
||||||
|
const float d = x[ib].d;
|
||||||
|
|
||||||
|
// Q2_0: 2 bits per element, 4 elements per byte.
|
||||||
|
// Stored code c in {0,1,2,3} maps to symbol s = c - 1 in {-1, 0, +1, +2}.
|
||||||
|
const int byte_index_0 = iqs / 4;
|
||||||
|
const int bit_offset_0 = (iqs % 4) * 2;
|
||||||
|
|
||||||
|
const int byte_index_1 = (iqs + 1) / 4;
|
||||||
|
const int bit_offset_1 = ((iqs + 1) % 4) * 2;
|
||||||
|
|
||||||
|
const int c0 = (x[ib].qs[byte_index_0] >> bit_offset_0) & 0x3;
|
||||||
|
const int c1 = (x[ib].qs[byte_index_1] >> bit_offset_1) & 0x3;
|
||||||
|
|
||||||
|
v.x = (c0 - 1) * d;
|
||||||
|
v.y = (c1 - 1) * d;
|
||||||
|
}
|
||||||
|
|
||||||
static __device__ __forceinline__ void dequantize_q4_0(const void * vx, const int64_t ib, const int iqs, float2 & v){
|
static __device__ __forceinline__ void dequantize_q4_0(const void * vx, const int64_t ib, const int iqs, float2 & v){
|
||||||
const block_q4_0 * x = (const block_q4_0 *) vx;
|
const block_q4_0 * x = (const block_q4_0 *) vx;
|
||||||
|
|
||||||
@@ -97,3 +118,335 @@ static __device__ __forceinline__ void dequantize_q8_0(const void * vx, const in
|
|||||||
v.x *= d;
|
v.x *= d;
|
||||||
v.y *= d;
|
v.y *= d;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
//================================== k-quants
|
||||||
|
|
||||||
|
// Each call dequantizes one super-block of QK_K values into y using the
|
||||||
|
// thread layout of the caller: 32 threads for q4_K, 64 threads otherwise.
|
||||||
|
|
||||||
|
template<typename dst_t>
|
||||||
|
static __device__ __forceinline__ void dequantize_q2_K(const void * vx, const int64_t ib, dst_t * yy, const int tid) {
|
||||||
|
const block_q2_K * x = (const block_q2_K *) vx;
|
||||||
|
|
||||||
|
const int64_t n = tid/32;
|
||||||
|
const int64_t l = tid - 32*n;
|
||||||
|
const int64_t is = 8*n + l/16;
|
||||||
|
|
||||||
|
const uint8_t q = x[ib].qs[32*n + l];
|
||||||
|
dst_t * y = yy + 128*n;
|
||||||
|
|
||||||
|
float dall = __low2half(x[ib].dm);
|
||||||
|
float dmin = __high2half(x[ib].dm);
|
||||||
|
y[l+ 0] = ggml_cuda_cast<dst_t>(dall * (x[ib].scales[is+0] & 0xF) * ((q >> 0) & 3) - dmin * (x[ib].scales[is+0] >> 4));
|
||||||
|
y[l+32] = ggml_cuda_cast<dst_t>(dall * (x[ib].scales[is+2] & 0xF) * ((q >> 2) & 3) - dmin * (x[ib].scales[is+2] >> 4));
|
||||||
|
y[l+64] = ggml_cuda_cast<dst_t>(dall * (x[ib].scales[is+4] & 0xF) * ((q >> 4) & 3) - dmin * (x[ib].scales[is+4] >> 4));
|
||||||
|
y[l+96] = ggml_cuda_cast<dst_t>(dall * (x[ib].scales[is+6] & 0xF) * ((q >> 6) & 3) - dmin * (x[ib].scales[is+6] >> 4));
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename dst_t>
|
||||||
|
static __device__ __forceinline__ void dequantize_q3_K(const void * vx, const int64_t ib, dst_t * yy, const int tid) {
|
||||||
|
const block_q3_K * x = (const block_q3_K *) vx;
|
||||||
|
|
||||||
|
const int64_t r = tid/4;
|
||||||
|
const int64_t t = r/2;
|
||||||
|
const int64_t is0 = r%2;
|
||||||
|
const int64_t l0 = 16*is0 + 4*(tid%4);
|
||||||
|
const int64_t n = t / 4;
|
||||||
|
const int64_t j = t - 4*n;
|
||||||
|
|
||||||
|
uint8_t m = 1 << (4*n + j);
|
||||||
|
int64_t is = 8*n + 2*j + is0;
|
||||||
|
int shift = 2*j;
|
||||||
|
|
||||||
|
int8_t us = is < 4 ? (x[ib].scales[is-0] & 0xF) | (((x[ib].scales[is+8] >> 0) & 3) << 4) :
|
||||||
|
is < 8 ? (x[ib].scales[is-0] & 0xF) | (((x[ib].scales[is+4] >> 2) & 3) << 4) :
|
||||||
|
is < 12 ? (x[ib].scales[is-8] >> 4) | (((x[ib].scales[is+0] >> 4) & 3) << 4) :
|
||||||
|
(x[ib].scales[is-8] >> 4) | (((x[ib].scales[is-4] >> 6) & 3) << 4);
|
||||||
|
float d_all = x[ib].d;
|
||||||
|
float dl = d_all * (us - 32);
|
||||||
|
|
||||||
|
dst_t * y = yy + 128*n + 32*j;
|
||||||
|
const uint8_t * q = x[ib].qs + 32*n;
|
||||||
|
const uint8_t * hm = x[ib].hmask;
|
||||||
|
|
||||||
|
for (int l = l0; l < l0+4; ++l) {
|
||||||
|
y[l] = ggml_cuda_cast<dst_t>(dl * ((int8_t)((q[l] >> shift) & 3) - ((hm[l] & m) ? 0 : 4)));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline __device__ void get_scale_min_k4(int j, const uint8_t * q, uint8_t & d, uint8_t & m) {
|
||||||
|
if (j < 4) {
|
||||||
|
d = q[j] & 63; m = q[j + 4] & 63;
|
||||||
|
} else {
|
||||||
|
d = (q[j+4] & 0xF) | ((q[j-4] >> 6) << 4);
|
||||||
|
m = (q[j+4] >> 4) | ((q[j-0] >> 6) << 4);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename dst_t>
|
||||||
|
static __device__ __forceinline__ void dequantize_q4_K(const void * vx, const int64_t ib, dst_t * yy, const int tid) {
|
||||||
|
const block_q4_K * x = (const block_q4_K *) vx;
|
||||||
|
|
||||||
|
// assume 32 threads
|
||||||
|
const int64_t il = tid/8;
|
||||||
|
const int64_t ir = tid%8;
|
||||||
|
const int64_t is = 2*il;
|
||||||
|
const int64_t n = 4;
|
||||||
|
|
||||||
|
dst_t * y = yy + 64*il + n*ir;
|
||||||
|
|
||||||
|
const float dall = __low2half(x[ib].dm);
|
||||||
|
const float dmin = __high2half(x[ib].dm);
|
||||||
|
|
||||||
|
const uint8_t * q = x[ib].qs + 32*il + n*ir;
|
||||||
|
|
||||||
|
uint8_t sc, m;
|
||||||
|
get_scale_min_k4(is + 0, x[ib].scales, sc, m);
|
||||||
|
const float d1 = dall * sc; const float m1 = dmin * m;
|
||||||
|
get_scale_min_k4(is + 1, x[ib].scales, sc, m);
|
||||||
|
const float d2 = dall * sc; const float m2 = dmin * m;
|
||||||
|
for (int l = 0; l < n; ++l) {
|
||||||
|
y[l + 0] = ggml_cuda_cast<dst_t>(d1 * (q[l] & 0xF) - m1);
|
||||||
|
y[l +32] = ggml_cuda_cast<dst_t>(d2 * (q[l] >> 4) - m2);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename dst_t>
|
||||||
|
static __device__ __forceinline__ void dequantize_q5_K(const void * vx, const int64_t ib, dst_t * yy, const int tid) {
|
||||||
|
const block_q5_K * x = (const block_q5_K *) vx;
|
||||||
|
|
||||||
|
// assume 64 threads - this is very slightly better than the one below
|
||||||
|
const int64_t il = tid/16; // il is in 0...3
|
||||||
|
const int64_t ir = tid%16; // ir is in 0...15
|
||||||
|
const int64_t is = 2*il; // is is in 0...6
|
||||||
|
|
||||||
|
dst_t * y = yy + 64*il + 2*ir;
|
||||||
|
|
||||||
|
const float dall = __low2half(x[ib].dm);
|
||||||
|
const float dmin = __high2half(x[ib].dm);
|
||||||
|
|
||||||
|
const uint8_t * ql = x[ib].qs + 32*il + 2*ir;
|
||||||
|
const uint8_t * qh = x[ib].qh + 2*ir;
|
||||||
|
|
||||||
|
uint8_t sc, m;
|
||||||
|
get_scale_min_k4(is + 0, x[ib].scales, sc, m);
|
||||||
|
const float d1 = dall * sc; const float m1 = dmin * m;
|
||||||
|
get_scale_min_k4(is + 1, x[ib].scales, sc, m);
|
||||||
|
const float d2 = dall * sc; const float m2 = dmin * m;
|
||||||
|
|
||||||
|
uint8_t hm = 1 << (2*il);
|
||||||
|
y[ 0] = ggml_cuda_cast<dst_t>(d1 * ((ql[ 0] & 0xF) + (qh[ 0] & hm ? 16 : 0)) - m1);
|
||||||
|
y[ 1] = ggml_cuda_cast<dst_t>(d1 * ((ql[ 1] & 0xF) + (qh[ 1] & hm ? 16 : 0)) - m1);
|
||||||
|
hm <<= 1;
|
||||||
|
y[32] = ggml_cuda_cast<dst_t>(d2 * ((ql[ 0] >> 4) + (qh[ 0] & hm ? 16 : 0)) - m2);
|
||||||
|
y[33] = ggml_cuda_cast<dst_t>(d2 * ((ql[ 1] >> 4) + (qh[ 1] & hm ? 16 : 0)) - m2);
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename dst_t>
|
||||||
|
static __device__ __forceinline__ void dequantize_q6_K(const void * vx, const int64_t ib, dst_t * yy, const int tid) {
|
||||||
|
const block_q6_K * x = (const block_q6_K *) vx;
|
||||||
|
|
||||||
|
// assume 64 threads - this is very slightly better than the one below
|
||||||
|
const int64_t ip = tid/32; // ip is 0 or 1
|
||||||
|
const int64_t il = tid - 32*ip; // 0...32
|
||||||
|
const int64_t is = 8*ip + il/16;
|
||||||
|
|
||||||
|
dst_t * y = yy + 128*ip + il;
|
||||||
|
|
||||||
|
const float d = x[ib].d;
|
||||||
|
|
||||||
|
const uint8_t * ql = x[ib].ql + 64*ip + il;
|
||||||
|
const uint8_t qh = x[ib].qh[32*ip + il];
|
||||||
|
const int8_t * sc = x[ib].scales + is;
|
||||||
|
|
||||||
|
y[ 0] = ggml_cuda_cast<dst_t>(d * sc[0] * ((int8_t)((ql[ 0] & 0xF) | (((qh >> 0) & 3) << 4)) - 32));
|
||||||
|
y[32] = ggml_cuda_cast<dst_t>(d * sc[2] * ((int8_t)((ql[32] & 0xF) | (((qh >> 2) & 3) << 4)) - 32));
|
||||||
|
y[64] = ggml_cuda_cast<dst_t>(d * sc[4] * ((int8_t)((ql[ 0] >> 4) | (((qh >> 4) & 3) << 4)) - 32));
|
||||||
|
y[96] = ggml_cuda_cast<dst_t>(d * sc[6] * ((int8_t)((ql[32] >> 4) | (((qh >> 6) & 3) << 4)) - 32));
|
||||||
|
}
|
||||||
|
|
||||||
|
//================================== i-quants
|
||||||
|
|
||||||
|
// Each call dequantizes one super-block of QK_K values into y with 32
|
||||||
|
// threads; iq4_nl packs QK_K/QK4_NL sub-blocks per super-block.
|
||||||
|
|
||||||
|
template<typename dst_t>
|
||||||
|
static __device__ __forceinline__ void dequantize_iq2_xxs(const void * vx, const int64_t ibs, dst_t * yy, const int tid) {
|
||||||
|
|
||||||
|
const block_iq2_xxs * x = (const block_iq2_xxs *) vx;
|
||||||
|
|
||||||
|
const int64_t il = tid/8; // 0...3
|
||||||
|
const int64_t ib = tid%8; // 0...7
|
||||||
|
dst_t * y = yy + 32*ib + 8*il;
|
||||||
|
const uint16_t * q2 = x[ibs].qs + 4*ib;
|
||||||
|
const uint8_t * aux8 = (const uint8_t *)q2;
|
||||||
|
const uint8_t * grid = (const uint8_t *)(iq2xxs_grid + aux8[il]);
|
||||||
|
const uint32_t aux32 = q2[2] | (q2[3] << 16);
|
||||||
|
const float d = (float)x[ibs].d * (0.5f + (aux32 >> 28)) * 0.25f;
|
||||||
|
const uint8_t signs = ksigns_iq2xs[(aux32 >> 7*il) & 127];
|
||||||
|
for (int j = 0; j < 8; ++j) {
|
||||||
|
y[j] = ggml_cuda_cast<dst_t>(d * grid[j] * (signs & kmask_iq2xs[j] ? -1.f : 1.f));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename dst_t>
|
||||||
|
static __device__ __forceinline__ void dequantize_iq2_xs(const void * vx, const int64_t ibs, dst_t * yy, const int tid) {
|
||||||
|
|
||||||
|
const block_iq2_xs * x = (const block_iq2_xs *) vx;
|
||||||
|
|
||||||
|
const int64_t il = tid/8; // 0...3
|
||||||
|
const int64_t ib = tid%8; // 0...7
|
||||||
|
dst_t * y = yy + 32*ib + 8*il;
|
||||||
|
const uint16_t * q2 = x[ibs].qs + 4*ib;
|
||||||
|
const uint8_t * grid = (const uint8_t *)(iq2xs_grid + (q2[il] & 511));
|
||||||
|
const float d = (float)x[ibs].d * (0.5f + ((x[ibs].scales[ib] >> 4*(il/2)) & 0xf)) * 0.25f;
|
||||||
|
const uint8_t signs = ksigns_iq2xs[q2[il] >> 9];
|
||||||
|
for (int j = 0; j < 8; ++j) {
|
||||||
|
y[j] = ggml_cuda_cast<dst_t>(d * grid[j] * (signs & kmask_iq2xs[j] ? -1.f : 1.f));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename dst_t>
|
||||||
|
static __device__ __forceinline__ void dequantize_iq2_s(const void * vx, const int64_t ibs, dst_t * yy, const int tid) {
|
||||||
|
|
||||||
|
const block_iq2_s * x = (const block_iq2_s *) vx;
|
||||||
|
|
||||||
|
const int64_t il = tid/8; // 0...3
|
||||||
|
const int64_t ib = tid%8; // 0...7
|
||||||
|
dst_t * y = yy + 32*ib + 8*il;
|
||||||
|
const uint8_t * grid = (const uint8_t *)(iq2s_grid + (x[ibs].qs[4*ib+il] | ((x[ibs].qh[ib] << (8-2*il)) & 0x300)));
|
||||||
|
const float d = (float)x[ibs].d * (0.5f + ((x[ibs].scales[ib] >> 4*(il/2)) & 0xf)) * 0.25f;
|
||||||
|
const uint8_t signs = x[ibs].qs[QK_K/8+4*ib+il];
|
||||||
|
for (int j = 0; j < 8; ++j) {
|
||||||
|
y[j] = ggml_cuda_cast<dst_t>(d * grid[j] * (signs & kmask_iq2xs[j] ? -1.f : 1.f));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename dst_t>
|
||||||
|
static __device__ __forceinline__ void dequantize_iq3_xxs(const void * vx, const int64_t ibs, dst_t * yy, const int tid) {
|
||||||
|
|
||||||
|
const block_iq3_xxs * x = (const block_iq3_xxs *) vx;
|
||||||
|
|
||||||
|
const int64_t il = tid/8; // 0...3
|
||||||
|
const int64_t ib = tid%8; // 0...7
|
||||||
|
dst_t * y = yy + 32*ib + 8*il;
|
||||||
|
const uint8_t * q3 = x[ibs].qs + 8*ib;
|
||||||
|
const uint16_t * gas = (const uint16_t *)(x[ibs].qs + QK_K/4) + 2*ib;
|
||||||
|
const uint8_t * grid1 = (const uint8_t *)(iq3xxs_grid + q3[2*il+0]);
|
||||||
|
const uint8_t * grid2 = (const uint8_t *)(iq3xxs_grid + q3[2*il+1]);
|
||||||
|
const uint32_t aux32 = gas[0] | (gas[1] << 16);
|
||||||
|
const float d = (float)x[ibs].d * (0.5f + (aux32 >> 28)) * 0.5f;
|
||||||
|
const uint8_t signs = ksigns_iq2xs[(aux32 >> 7*il) & 127];
|
||||||
|
for (int j = 0; j < 4; ++j) {
|
||||||
|
y[j+0] = ggml_cuda_cast<dst_t>(d * grid1[j] * (signs & kmask_iq2xs[j+0] ? -1.f : 1.f));
|
||||||
|
y[j+4] = ggml_cuda_cast<dst_t>(d * grid2[j] * (signs & kmask_iq2xs[j+4] ? -1.f : 1.f));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename dst_t>
|
||||||
|
static __device__ __forceinline__ void dequantize_iq3_s(const void * vx, const int64_t ibs, dst_t * yy, const int tid) {
|
||||||
|
|
||||||
|
const block_iq3_s * x = (const block_iq3_s *) vx;
|
||||||
|
|
||||||
|
const int64_t il = tid/8; // 0...3
|
||||||
|
const int64_t ib = tid%8; // 0...7
|
||||||
|
dst_t * y = yy + 32*ib + 8*il;
|
||||||
|
const uint8_t * qs = x[ibs].qs + 8*ib;
|
||||||
|
const uint8_t * grid1 = (const uint8_t *)(iq3s_grid + (qs[2*il+0] | ((x[ibs].qh[ib] << (8-2*il)) & 256)));
|
||||||
|
const uint8_t * grid2 = (const uint8_t *)(iq3s_grid + (qs[2*il+1] | ((x[ibs].qh[ib] << (7-2*il)) & 256)));
|
||||||
|
const float d = (float)x[ibs].d * (1 + 2*((x[ibs].scales[ib/2] >> 4*(ib%2)) & 0xf));
|
||||||
|
const uint8_t signs = x[ibs].signs[4*ib + il];
|
||||||
|
for (int j = 0; j < 4; ++j) {
|
||||||
|
y[j+0] = ggml_cuda_cast<dst_t>(d * grid1[j] * (signs & kmask_iq2xs[j+0] ? -1.f : 1.f));
|
||||||
|
y[j+4] = ggml_cuda_cast<dst_t>(d * grid2[j] * (signs & kmask_iq2xs[j+4] ? -1.f : 1.f));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename dst_t>
|
||||||
|
static __device__ __forceinline__ void dequantize_iq1_s(const void * vx, const int64_t ibs, dst_t * yy, const int tid) {
|
||||||
|
|
||||||
|
const block_iq1_s * x = (const block_iq1_s *) vx;
|
||||||
|
|
||||||
|
const int64_t il = tid/8; // 0...3
|
||||||
|
const int64_t ib = tid%8; // 0...7
|
||||||
|
dst_t * y = yy + 32*ib + 8*il;
|
||||||
|
const float delta = x[ibs].qh[ib] & 0x8000 ? -1 - IQ1S_DELTA : -1 + IQ1S_DELTA;
|
||||||
|
const float d = (float)x[ibs].d * (2*((x[ibs].qh[ib] >> 12) & 7) + 1);
|
||||||
|
uint32_t grid32[2]; const int8_t * q = (const int8_t *)grid32;
|
||||||
|
grid32[0] = iq1s_grid_gpu[x[ibs].qs[4*ib+il] | (((x[ibs].qh[ib] >> 3*il) & 7) << 8)];
|
||||||
|
grid32[1] = (grid32[0] >> 4) & 0x0f0f0f0f;
|
||||||
|
grid32[0] &= 0x0f0f0f0f;
|
||||||
|
for (int j = 0; j < 8; ++j) {
|
||||||
|
y[j] = ggml_cuda_cast<dst_t>(d * (q[j] + delta));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename dst_t>
|
||||||
|
static __device__ __forceinline__ void dequantize_iq1_m(const void * vx, const int64_t ibs, dst_t * yy, const int tid) {
|
||||||
|
|
||||||
|
const block_iq1_m * x = (const block_iq1_m *) vx;
|
||||||
|
|
||||||
|
const int64_t il = tid/8; // 0...3
|
||||||
|
const int64_t ib = tid%8; // 0...7
|
||||||
|
dst_t * y = yy + 32*ib + 8*il;
|
||||||
|
const uint16_t * sc = (const uint16_t *)x[ibs].scales;
|
||||||
|
iq1m_scale_t scale;
|
||||||
|
scale.u16 = (sc[0] >> 12) | ((sc[1] >> 8) & 0x00f0) | ((sc[2] >> 4) & 0x0f00) | (sc[3] & 0xf000);
|
||||||
|
const int64_t ib16 = 2*ib + il/2; // sc[ib16/4] >> 3*(ib16%4) -> sc[ib/2] >> 3*((2*ib+il/2)%4);
|
||||||
|
const float d = (float)scale.f16 * (2*((sc[ib16/4] >> 3*(ib16%4)) & 0x7) + 1);
|
||||||
|
const float delta = x[ibs].qh[2*ib+il/2] & (0x08 << 4*(il%2)) ? -1 - IQ1M_DELTA : -1 + IQ1M_DELTA;
|
||||||
|
uint32_t grid32[2]; const int8_t * q = (const int8_t *)grid32;
|
||||||
|
grid32[0] = iq1s_grid_gpu[x[ibs].qs[4*ib+il] | (((x[ibs].qh[2*ib+il/2] >> 4*(il%2)) & 7) << 8)];
|
||||||
|
grid32[1] = (grid32[0] >> 4) & 0x0f0f0f0f;
|
||||||
|
grid32[0] &= 0x0f0f0f0f;
|
||||||
|
for (int j = 0; j < 8; ++j) {
|
||||||
|
y[j] = ggml_cuda_cast<dst_t>(d * (q[j] + delta));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename dst_t>
|
||||||
|
static __device__ __forceinline__ void dequantize_iq4_nl(const void * vx, const int64_t ibs, dst_t * yy, const int tid) {
|
||||||
|
|
||||||
|
const block_iq4_nl * x = (const block_iq4_nl *) vx + ibs*(QK_K/QK4_NL);
|
||||||
|
|
||||||
|
const int64_t il = tid/8; // 0...3
|
||||||
|
const int64_t ib = tid%8; // 0...7
|
||||||
|
dst_t * y = yy + 32*ib + 4*il;
|
||||||
|
const uint8_t * q4 = x[ib].qs + 4*il;
|
||||||
|
const float d = (float)x[ib].d;
|
||||||
|
for (int j = 0; j < 4; ++j) {
|
||||||
|
y[j+ 0] = ggml_cuda_cast<dst_t>(d * kvalues_iq4nl[q4[j] & 0xf]);
|
||||||
|
y[j+16] = ggml_cuda_cast<dst_t>(d * kvalues_iq4nl[q4[j] >> 4]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename dst_t>
|
||||||
|
static __device__ __forceinline__ void dequantize_iq4_xs(const void * vx, const int64_t ibs, dst_t * yy, const int tid) {
|
||||||
|
const block_iq4_xs * x = (const block_iq4_xs *)vx;
|
||||||
|
|
||||||
|
const int64_t il = tid/8; // 0...3
|
||||||
|
const int64_t ib = tid%8; // 0...7
|
||||||
|
dst_t * y = yy + 32*ib + 4*il;
|
||||||
|
const uint8_t * q4 = x[ibs].qs + 16*ib + 4*il;
|
||||||
|
const float d = (float)x[ibs].d * ((((x[ibs].scales_l[ib/2] >> 4*(ib%2)) & 0xf) | (((x[ibs].scales_h >> 2*ib) & 3) << 4)) - 32);
|
||||||
|
for (int j = 0; j < 4; ++j) {
|
||||||
|
y[j+ 0] = ggml_cuda_cast<dst_t>(d * kvalues_iq4nl[q4[j] & 0xf]);
|
||||||
|
y[j+16] = ggml_cuda_cast<dst_t>(d * kvalues_iq4nl[q4[j] >> 4]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename dst_t>
|
||||||
|
static __device__ __forceinline__ void dequantize_mxfp4(const void * vx, const int64_t ibs, dst_t * yy, const int tid) {
|
||||||
|
|
||||||
|
const block_mxfp4 * x = (const block_mxfp4 *) vx + ibs*(QK_K/QK_MXFP4);
|
||||||
|
|
||||||
|
const int64_t il = tid/8; // 0...3
|
||||||
|
const int64_t ib = tid%8; // 0...7
|
||||||
|
dst_t * y = yy + 32*ib + 4*il;
|
||||||
|
const uint8_t * q4 = x[ib].qs + 4*il;
|
||||||
|
const float d = ggml_cuda_e8m0_to_fp32(x[ib].e);
|
||||||
|
for (int j = 0; j < 4; ++j) {
|
||||||
|
y[j+ 0] = ggml_cuda_cast<dst_t>(d * kvalues_mxfp4[q4[j] & 0xf]*0.5f);
|
||||||
|
y[j+16] = ggml_cuda_cast<dst_t>(d * kvalues_mxfp4[q4[j] >> 4]*0.5f);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,294 @@
|
|||||||
|
#include "common.cuh"
|
||||||
|
#include "dsv4-hc.cuh"
|
||||||
|
|
||||||
|
|
||||||
|
static constexpr int DSV4_HC = 4;
|
||||||
|
|
||||||
|
|
||||||
|
static __device__ void dsv4_hc_comb_norm_cols(float * comb, float eps) {
|
||||||
|
for (int idst = 0; idst < DSV4_HC; ++idst) {
|
||||||
|
float sum = eps;
|
||||||
|
for (int isrc = 0; isrc < DSV4_HC; ++isrc) {
|
||||||
|
sum += comb[idst + DSV4_HC*isrc];
|
||||||
|
}
|
||||||
|
|
||||||
|
const float inv_sum = 1.0f / sum;
|
||||||
|
for (int isrc = 0; isrc < DSV4_HC; ++isrc) {
|
||||||
|
comb[idst + DSV4_HC*isrc] *= inv_sum;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static __device__ void dsv4_hc_comb_norm_rows(float * comb, float eps) {
|
||||||
|
for (int isrc = 0; isrc < DSV4_HC; ++isrc) {
|
||||||
|
float sum = eps;
|
||||||
|
for (int idst = 0; idst < DSV4_HC; ++idst) {
|
||||||
|
sum += comb[idst + DSV4_HC*isrc];
|
||||||
|
}
|
||||||
|
|
||||||
|
const float inv_sum = 1.0f / sum;
|
||||||
|
for (int idst = 0; idst < DSV4_HC; ++idst) {
|
||||||
|
comb[idst + DSV4_HC*isrc] *= inv_sum;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static __global__ void dsv4_hc_comb_f32(
|
||||||
|
const float * mixes,
|
||||||
|
const float * scale,
|
||||||
|
const float * base,
|
||||||
|
float * dst,
|
||||||
|
int64_t n_tokens,
|
||||||
|
int64_t sm0,
|
||||||
|
int64_t sm1,
|
||||||
|
int64_t ss0,
|
||||||
|
int64_t sb0,
|
||||||
|
int64_t sd0,
|
||||||
|
int64_t sd1,
|
||||||
|
int64_t sd2,
|
||||||
|
float eps,
|
||||||
|
int32_t n_iter) {
|
||||||
|
constexpr int comb_offset = 2*DSV4_HC;
|
||||||
|
|
||||||
|
ggml_cuda_pdl_lc();
|
||||||
|
const int64_t it = (int64_t) blockIdx.x * blockDim.x + threadIdx.x;
|
||||||
|
|
||||||
|
if (it >= n_tokens) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
ggml_cuda_pdl_sync();
|
||||||
|
|
||||||
|
const float scale_comb = scale[2*ss0];
|
||||||
|
float comb[DSV4_HC*DSV4_HC];
|
||||||
|
|
||||||
|
for (int isrc = 0; isrc < DSV4_HC; ++isrc) {
|
||||||
|
float max = -INFINITY;
|
||||||
|
for (int idst = 0; idst < DSV4_HC; ++idst) {
|
||||||
|
const int idx = idst + DSV4_HC*isrc;
|
||||||
|
const float v = mixes[(comb_offset + idx)*sm0 + it*sm1] * scale_comb + base[(comb_offset + idx)*sb0];
|
||||||
|
comb[idx] = v;
|
||||||
|
max = fmaxf(max, v);
|
||||||
|
}
|
||||||
|
|
||||||
|
float sum = 0.0f;
|
||||||
|
for (int idst = 0; idst < DSV4_HC; ++idst) {
|
||||||
|
const int idx = idst + DSV4_HC*isrc;
|
||||||
|
const float v = expf(comb[idx] - max);
|
||||||
|
comb[idx] = v;
|
||||||
|
sum += v;
|
||||||
|
}
|
||||||
|
|
||||||
|
const float inv_sum = 1.0f / sum;
|
||||||
|
for (int idst = 0; idst < DSV4_HC; ++idst) {
|
||||||
|
const int idx = idst + DSV4_HC*isrc;
|
||||||
|
comb[idx] = comb[idx] * inv_sum + eps;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
dsv4_hc_comb_norm_cols(comb, eps);
|
||||||
|
for (int32_t i = 1; i < n_iter; ++i) {
|
||||||
|
dsv4_hc_comb_norm_rows(comb, eps);
|
||||||
|
dsv4_hc_comb_norm_cols(comb, eps);
|
||||||
|
}
|
||||||
|
|
||||||
|
for (int isrc = 0; isrc < DSV4_HC; ++isrc) {
|
||||||
|
for (int idst = 0; idst < DSV4_HC; ++idst) {
|
||||||
|
const int idx = idst + DSV4_HC*isrc;
|
||||||
|
dst[idst*sd0 + isrc*sd1 + it*sd2] = comb[idx];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static __global__ void dsv4_hc_pre_f32(
|
||||||
|
const float * x,
|
||||||
|
const float * weights,
|
||||||
|
float * dst,
|
||||||
|
int64_t n_embd,
|
||||||
|
int64_t hc,
|
||||||
|
int64_t n_tokens,
|
||||||
|
int64_t sx0,
|
||||||
|
int64_t sx1,
|
||||||
|
int64_t sx2,
|
||||||
|
int64_t sw0,
|
||||||
|
int64_t sw1,
|
||||||
|
int64_t sd0,
|
||||||
|
int64_t sd1) {
|
||||||
|
ggml_cuda_pdl_lc();
|
||||||
|
const int64_t ir = (int64_t) blockIdx.x * blockDim.x + threadIdx.x;
|
||||||
|
const int64_t nr = n_embd * n_tokens;
|
||||||
|
|
||||||
|
if (ir >= nr) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
ggml_cuda_pdl_sync();
|
||||||
|
|
||||||
|
const int64_t i0 = ir % n_embd;
|
||||||
|
const int64_t it = ir / n_embd;
|
||||||
|
|
||||||
|
float sum = x[i0*sx0 + it*sx2] * weights[it*sw1];
|
||||||
|
for (int64_t ih = 1; ih < hc; ++ih) {
|
||||||
|
const float xv = x[i0*sx0 + ih*sx1 + it*sx2];
|
||||||
|
const float wv = weights[ih*sw0 + it*sw1];
|
||||||
|
sum += xv * wv;
|
||||||
|
}
|
||||||
|
|
||||||
|
dst[i0*sd0 + it*sd1] = sum;
|
||||||
|
}
|
||||||
|
|
||||||
|
static __global__ void dsv4_hc_post_f32(
|
||||||
|
const float * x,
|
||||||
|
const float * residual,
|
||||||
|
const float * post,
|
||||||
|
const float * comb,
|
||||||
|
float * dst,
|
||||||
|
int64_t n_embd,
|
||||||
|
int64_t hc,
|
||||||
|
int64_t n_tokens,
|
||||||
|
int64_t sx0,
|
||||||
|
int64_t sx1,
|
||||||
|
int64_t sr0,
|
||||||
|
int64_t sr1,
|
||||||
|
int64_t sr2,
|
||||||
|
int64_t sp0,
|
||||||
|
int64_t sp1,
|
||||||
|
int64_t sc0,
|
||||||
|
int64_t sc1,
|
||||||
|
int64_t sc2,
|
||||||
|
int64_t sd0,
|
||||||
|
int64_t sd1,
|
||||||
|
int64_t sd2) {
|
||||||
|
ggml_cuda_pdl_lc();
|
||||||
|
const int64_t ir = (int64_t) blockIdx.x * blockDim.x + threadIdx.x;
|
||||||
|
const int64_t nr = n_embd * hc * n_tokens;
|
||||||
|
|
||||||
|
if (ir >= nr) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
ggml_cuda_pdl_sync();
|
||||||
|
|
||||||
|
const int64_t i0 = ir % n_embd;
|
||||||
|
const int64_t idst = (ir / n_embd) % hc;
|
||||||
|
const int64_t it = ir / (n_embd * hc);
|
||||||
|
|
||||||
|
float sum = x[i0*sx0 + it*sx1] * post[idst*sp0 + it*sp1];
|
||||||
|
for (int64_t isrc = 0; isrc < hc; ++isrc) {
|
||||||
|
sum += residual[i0*sr0 + isrc*sr1 + it*sr2] * comb[idst*sc0 + isrc*sc1 + it*sc2];
|
||||||
|
}
|
||||||
|
|
||||||
|
dst[i0*sd0 + idst*sd1 + it*sd2] = sum;
|
||||||
|
}
|
||||||
|
|
||||||
|
void ggml_cuda_op_dsv4_hc_comb(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
|
||||||
|
const ggml_tensor * mixes = dst->src[0];
|
||||||
|
const ggml_tensor * scale = dst->src[1];
|
||||||
|
const ggml_tensor * base = dst->src[2];
|
||||||
|
|
||||||
|
GGML_ASSERT(mixes->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(scale->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(base->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(dst->type == GGML_TYPE_F32);
|
||||||
|
|
||||||
|
constexpr int64_t hc_mix_dim = (2 + DSV4_HC)*DSV4_HC;
|
||||||
|
|
||||||
|
GGML_ASSERT(mixes->ne[0] == hc_mix_dim);
|
||||||
|
GGML_ASSERT(dst->ne[0] == DSV4_HC);
|
||||||
|
GGML_ASSERT(dst->ne[1] == DSV4_HC);
|
||||||
|
GGML_ASSERT(dst->ne[2] == mixes->ne[1]);
|
||||||
|
GGML_ASSERT(scale->ne[0] >= 3);
|
||||||
|
GGML_ASSERT(base->ne[0] == hc_mix_dim);
|
||||||
|
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbm, mixes, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbs, scale, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbb, base, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbd, dst, nb);
|
||||||
|
|
||||||
|
const int64_t n_tokens = mixes->ne[1];
|
||||||
|
const float eps = ggml_get_op_params_f32(dst, 0);
|
||||||
|
const int32_t n_iter = ggml_get_op_params_i32(dst, 1);
|
||||||
|
|
||||||
|
const int block_size = 256;
|
||||||
|
const dim3 block_dims(block_size, 1, 1);
|
||||||
|
const dim3 grid_dims((n_tokens + block_size - 1) / block_size, 1, 1);
|
||||||
|
const ggml_cuda_kernel_launch_params launch_params = ggml_cuda_kernel_launch_params(grid_dims, block_dims, 0, ctx.stream());
|
||||||
|
|
||||||
|
ggml_cuda_kernel_launch(dsv4_hc_comb_f32, launch_params,
|
||||||
|
(const float *) mixes->data, (const float *) scale->data, (const float *) base->data, (float *) dst->data,
|
||||||
|
n_tokens,
|
||||||
|
nbm0 / sizeof(float), nbm1 / sizeof(float),
|
||||||
|
nbs0 / sizeof(float),
|
||||||
|
nbb0 / sizeof(float),
|
||||||
|
nbd0 / sizeof(float), nbd1 / sizeof(float), nbd2 / sizeof(float),
|
||||||
|
eps, n_iter);
|
||||||
|
}
|
||||||
|
|
||||||
|
void ggml_cuda_op_dsv4_hc_pre(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
|
||||||
|
const ggml_tensor * x = dst->src[0];
|
||||||
|
const ggml_tensor * weights = dst->src[1];
|
||||||
|
|
||||||
|
GGML_ASSERT(x->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(weights->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(dst->type == GGML_TYPE_F32);
|
||||||
|
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbx, x, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbw, weights, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbd, dst, nb);
|
||||||
|
|
||||||
|
const int64_t n_embd = x->ne[0];
|
||||||
|
const int64_t hc = x->ne[1];
|
||||||
|
const int64_t n_tokens = x->ne[2];
|
||||||
|
|
||||||
|
const int block_size = 256;
|
||||||
|
const int64_t nr = n_embd * n_tokens;
|
||||||
|
const dim3 block_dims(block_size, 1, 1);
|
||||||
|
const dim3 grid_dims((nr + block_size - 1) / block_size, 1, 1);
|
||||||
|
const ggml_cuda_kernel_launch_params launch_params = ggml_cuda_kernel_launch_params(grid_dims, block_dims, 0, ctx.stream());
|
||||||
|
|
||||||
|
ggml_cuda_kernel_launch(dsv4_hc_pre_f32, launch_params,
|
||||||
|
(const float *) x->data, (const float *) weights->data, (float *) dst->data,
|
||||||
|
n_embd, hc, n_tokens,
|
||||||
|
nbx0 / sizeof(float), nbx1 / sizeof(float), nbx2 / sizeof(float),
|
||||||
|
nbw0 / sizeof(float), nbw1 / sizeof(float),
|
||||||
|
nbd0 / sizeof(float), nbd1 / sizeof(float));
|
||||||
|
}
|
||||||
|
|
||||||
|
void ggml_cuda_op_dsv4_hc_post(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
|
||||||
|
const ggml_tensor * x = dst->src[0];
|
||||||
|
const ggml_tensor * residual = dst->src[1];
|
||||||
|
const ggml_tensor * post = dst->src[2];
|
||||||
|
const ggml_tensor * comb = dst->src[3];
|
||||||
|
|
||||||
|
GGML_ASSERT(x->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(residual->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(post->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(comb->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT(dst->type == GGML_TYPE_F32);
|
||||||
|
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbx, x, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbr, residual, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbp, post, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbc, comb, nb);
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbd, dst, nb);
|
||||||
|
|
||||||
|
const int64_t n_embd = x->ne[0];
|
||||||
|
const int64_t n_tokens = x->ne[1];
|
||||||
|
const int64_t hc = residual->ne[1];
|
||||||
|
|
||||||
|
const int block_size = 256;
|
||||||
|
const int64_t nr = n_embd * hc * n_tokens;
|
||||||
|
const dim3 block_dims(block_size, 1, 1);
|
||||||
|
const dim3 grid_dims((nr + block_size - 1) / block_size, 1, 1);
|
||||||
|
const ggml_cuda_kernel_launch_params launch_params = ggml_cuda_kernel_launch_params(grid_dims, block_dims, 0, ctx.stream());
|
||||||
|
|
||||||
|
ggml_cuda_kernel_launch(dsv4_hc_post_f32, launch_params,
|
||||||
|
(const float *) x->data, (const float *) residual->data,
|
||||||
|
(const float *) post->data, (const float *) comb->data, (float *) dst->data,
|
||||||
|
n_embd, hc, n_tokens,
|
||||||
|
nbx0 / sizeof(float), nbx1 / sizeof(float),
|
||||||
|
nbr0 / sizeof(float), nbr1 / sizeof(float), nbr2 / sizeof(float),
|
||||||
|
nbp0 / sizeof(float), nbp1 / sizeof(float),
|
||||||
|
nbc0 / sizeof(float), nbc1 / sizeof(float), nbc2 / sizeof(float),
|
||||||
|
nbd0 / sizeof(float), nbd1 / sizeof(float), nbd2 / sizeof(float));
|
||||||
|
}
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
#include "common.cuh"
|
||||||
|
#include "ggml.h"
|
||||||
|
|
||||||
|
void ggml_cuda_op_dsv4_hc_comb(ggml_backend_cuda_context & ctx, ggml_tensor * dst);
|
||||||
|
void ggml_cuda_op_dsv4_hc_pre(ggml_backend_cuda_context & ctx, ggml_tensor * dst);
|
||||||
|
void ggml_cuda_op_dsv4_hc_post(ggml_backend_cuda_context & ctx, ggml_tensor * dst);
|
||||||
@@ -1,6 +1,5 @@
|
|||||||
#include "common.cuh"
|
#include "common.cuh"
|
||||||
#include "fattn-tile.cuh"
|
#include "fattn-tile.cuh"
|
||||||
#include "fattn-wmma-f16.cuh"
|
|
||||||
|
|
||||||
void ggml_cuda_flash_attn_ext_tile(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
|
void ggml_cuda_flash_attn_ext_tile(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
|
||||||
const ggml_tensor * K = dst->src[1];
|
const ggml_tensor * K = dst->src[1];
|
||||||
|
|||||||
@@ -1,6 +1,5 @@
|
|||||||
#include "common.cuh"
|
#include "common.cuh"
|
||||||
#include "fattn-common.cuh"
|
#include "fattn-common.cuh"
|
||||||
#include "fattn-wmma-f16.cuh"
|
|
||||||
|
|
||||||
// nbatch_fa == number of KQ rows to process per iteration
|
// nbatch_fa == number of KQ rows to process per iteration
|
||||||
// nbatch_K == number of K columns to load in parallel for KQ calculation
|
// nbatch_K == number of K columns to load in parallel for KQ calculation
|
||||||
@@ -825,12 +824,7 @@ static __global__ void flash_attn_tile(
|
|||||||
|
|
||||||
// Skip unused kernel variants for faster compilation:
|
// Skip unused kernel variants for faster compilation:
|
||||||
|
|
||||||
if (
|
if ((use_logit_softcap && !(DV == 128 || DV == 256 || DV == 512))) {
|
||||||
#ifdef GGML_USE_WMMA_FATTN
|
|
||||||
(ncols2 != 1 && DV != 40 && DV != 72 && DV != 512) ||
|
|
||||||
#endif // GGML_USE_WMMA_FATTN
|
|
||||||
(use_logit_softcap && !(DV == 128 || DV == 256 || DV == 512))
|
|
||||||
) {
|
|
||||||
GGML_UNUSED_VARS(Q, K, V, mask, sinks, KV_max, dst, dst_meta, scale,
|
GGML_UNUSED_VARS(Q, K, V, mask, sinks, KV_max, dst, dst_meta, scale,
|
||||||
max_bias, m0, m1, n_head_log2, logit_softcap,
|
max_bias, m0, m1, n_head_log2, logit_softcap,
|
||||||
ne00, ne01, ne02, ne03,
|
ne00, ne01, ne02, ne03,
|
||||||
|
|||||||
@@ -1,705 +0,0 @@
|
|||||||
// Old and deprecated WMMA FlashAttention implementation.
|
|
||||||
// It is still needed for Volta since the memory layout of NVIDIA tensor cores changed with Turing.
|
|
||||||
// Long-term the WMMA code should be replaced with a dedicated Volta implementation.
|
|
||||||
|
|
||||||
#include "common.cuh"
|
|
||||||
#include "fattn-common.cuh"
|
|
||||||
#include "fattn-wmma-f16.cuh"
|
|
||||||
|
|
||||||
#ifdef GGML_USE_WMMA_FATTN
|
|
||||||
#if !defined(GGML_USE_HIP)
|
|
||||||
#include <mma.h>
|
|
||||||
#if defined(GGML_USE_MUSA)
|
|
||||||
namespace wmma = mtmusa::wmma;
|
|
||||||
#else // GGML_USE_MUSA
|
|
||||||
namespace wmma = nvcuda::wmma;
|
|
||||||
#endif // GGML_USE_MUSA
|
|
||||||
#elif defined(GGML_USE_HIP)
|
|
||||||
#include <rocwmma/rocwmma.hpp>
|
|
||||||
namespace wmma = rocwmma;
|
|
||||||
#endif // !defined(GGML_USE_HIP)
|
|
||||||
#endif // GGML_USE_WMMA_FATTN
|
|
||||||
|
|
||||||
// D == head size, VKQ_stride == num VKQ rows calculated in parallel:
|
|
||||||
template<int D, int ncols, int nwarps, int VKQ_stride, typename KQ_acc_t, bool use_logit_softcap>
|
|
||||||
__launch_bounds__(nwarps*ggml_cuda_get_physical_warp_size(), 1)
|
|
||||||
static __global__ void flash_attn_ext_f16(
|
|
||||||
const char * Q_ptr,
|
|
||||||
const char * K_ptr,
|
|
||||||
const char * V_ptr,
|
|
||||||
const char * mask_ptr,
|
|
||||||
const char * sinks_ptr,
|
|
||||||
const int * KV_max_ptr,
|
|
||||||
float * dst_ptr,
|
|
||||||
float2 * dst_meta_ptr,
|
|
||||||
const float scale,
|
|
||||||
const float max_bias,
|
|
||||||
const float m0,
|
|
||||||
const float m1,
|
|
||||||
const uint32_t n_head_log2,
|
|
||||||
const float logit_softcap,
|
|
||||||
const int32_t ne00, const uint3 ne01, const int32_t ne02, const int32_t ne03,
|
|
||||||
const int32_t nb01, const int32_t nb02, const int32_t nb03,
|
|
||||||
const int32_t ne10, const int32_t ne11, const int32_t ne12, const int32_t ne13,
|
|
||||||
const int32_t nb11, const int32_t nb12, const int64_t nb13,
|
|
||||||
const int32_t nb21, const int32_t nb22, const int64_t nb23,
|
|
||||||
const int32_t ne31, const int32_t ne32, const int32_t ne33,
|
|
||||||
const int32_t nb31, const int32_t nb32, const int64_t nb33) {
|
|
||||||
#if defined(FLASH_ATTN_AVAILABLE) && (defined(GGML_HIP_ROCWMMA_FATTN) && defined(GGML_USE_WMMA_FATTN))
|
|
||||||
const char * GGML_CUDA_RESTRICT Q = Q_ptr;
|
|
||||||
const char * GGML_CUDA_RESTRICT K = K_ptr;
|
|
||||||
const char * GGML_CUDA_RESTRICT V = V_ptr;
|
|
||||||
const char * GGML_CUDA_RESTRICT mask = mask_ptr;
|
|
||||||
const char * GGML_CUDA_RESTRICT sinks = sinks_ptr;
|
|
||||||
const int * GGML_CUDA_RESTRICT KV_max = KV_max_ptr;
|
|
||||||
float * GGML_CUDA_RESTRICT dst = dst_ptr;
|
|
||||||
float2 * GGML_CUDA_RESTRICT dst_meta = dst_meta_ptr;
|
|
||||||
// Skip unused kernel variants for faster compilation:
|
|
||||||
if (use_logit_softcap && !(D == 128 || D == 256)) {
|
|
||||||
NO_DEVICE_CODE;
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
//In this kernel Q, K, V are matrices while i, j, k are matrix indices.
|
|
||||||
|
|
||||||
constexpr int warp_size = ggml_cuda_get_physical_warp_size();
|
|
||||||
|
|
||||||
const int ic0 = ncols*blockIdx.x; // Index of the first Q/QKV column to work on.
|
|
||||||
|
|
||||||
static_assert(D <= FATTN_KQ_STRIDE, "D must be <= FATTN_KQ_STRIDE.");
|
|
||||||
static_assert(ncols == 8 || ncols % 16 == 0, "ncols must be 8 or a multiple of 16.");
|
|
||||||
constexpr int frag_m = ncols == 8 ? 32 : 16;
|
|
||||||
constexpr int frag_n = ncols == 8 ? 8 : 16;
|
|
||||||
static_assert(D % frag_m == 0, "If ncols == 8 then D % frag_m must be 0.");
|
|
||||||
#if defined(GGML_USE_HIP) && HIP_VERSION >= 60500000
|
|
||||||
typedef wmma::fragment<wmma::matrix_a, frag_m, frag_n, 16, _Float16, wmma::row_major> frag_a_K;
|
|
||||||
typedef wmma::fragment<wmma::matrix_a, frag_m, frag_n, 16, _Float16, wmma::col_major> frag_a_V;
|
|
||||||
typedef wmma::fragment<wmma::matrix_b, frag_m, frag_n, 16, _Float16, wmma::col_major> frag_b;
|
|
||||||
typedef wmma::fragment<wmma::accumulator, frag_m, frag_n, 16, KQ_acc_t> frag_c_KQ;
|
|
||||||
typedef wmma::fragment<wmma::accumulator, frag_m, frag_n, 16, _Float16> frag_c_VKQ;
|
|
||||||
#else
|
|
||||||
typedef wmma::fragment<wmma::matrix_a, frag_m, frag_n, 16, half, wmma::row_major> frag_a_K;
|
|
||||||
typedef wmma::fragment<wmma::matrix_a, frag_m, frag_n, 16, half, wmma::col_major> frag_a_V;
|
|
||||||
typedef wmma::fragment<wmma::matrix_b, frag_m, frag_n, 16, half, wmma::col_major> frag_b;
|
|
||||||
typedef wmma::fragment<wmma::accumulator, frag_m, frag_n, 16, KQ_acc_t> frag_c_KQ;
|
|
||||||
typedef wmma::fragment<wmma::accumulator, frag_m, frag_n, 16, half> frag_c_VKQ;
|
|
||||||
#endif
|
|
||||||
|
|
||||||
constexpr int KQ_stride_tc = nwarps*frag_m; // Number of KQ rows calculated in parallel.
|
|
||||||
constexpr int VKQ_ratio = KQ_stride_tc/VKQ_stride; // Number of parallel VKQ accumulators needed to keep all warps busy.
|
|
||||||
static_assert(VKQ_ratio <= nwarps, "VKQ_ratio must be <= nwarps.");
|
|
||||||
|
|
||||||
// Pad internal representation of KQ, KQV to reduce shared memory bank conflicts:
|
|
||||||
constexpr int D_padded = D + 8;
|
|
||||||
constexpr int kqs_padded = FATTN_KQ_STRIDE + 8;
|
|
||||||
constexpr int kqar = sizeof(KQ_acc_t)/sizeof(half);
|
|
||||||
|
|
||||||
ggml_cuda_pdl_sync();
|
|
||||||
const int sequence = blockIdx.z / ne02;
|
|
||||||
const int head = blockIdx.z - sequence*ne02;
|
|
||||||
const int gqa_ratio = ne02 / ne12; // With grouped query attention there are > 1 Q matrices per K, V matrix.
|
|
||||||
const float * Q_f = (const float *) (Q + nb03* sequence + nb02* head + nb01*ic0);
|
|
||||||
const half * K_h = (const half *) (K + nb13* sequence + nb12*(head / gqa_ratio));
|
|
||||||
const half * V_h = (const half *) (V + nb13* sequence + nb12*(head / gqa_ratio)); // K and V have same shape
|
|
||||||
const half * maskh = (const half *) (mask + nb33*(sequence % ne33) + nb31*ic0);
|
|
||||||
const half2 * mask2 = (const half2 *) maskh;
|
|
||||||
const float * sinksf = (const float *) sinks;
|
|
||||||
|
|
||||||
const int stride_Q = nb01 / sizeof(float);
|
|
||||||
const int stride_KV = nb11 / sizeof(half);
|
|
||||||
|
|
||||||
const float slopef = get_alibi_slope(max_bias, head, n_head_log2, m0, m1);
|
|
||||||
const half slopeh = __float2half(slopef);
|
|
||||||
const half2 slope2 = make_half2(slopef, slopef);
|
|
||||||
|
|
||||||
const half2 logit_softcap_2 = make_half2(logit_softcap, logit_softcap);
|
|
||||||
|
|
||||||
frag_b Q_b[D/16][ncols/frag_n];
|
|
||||||
|
|
||||||
// A single buffer for temporarily holding tiles of KQ and VKQ parts:
|
|
||||||
constexpr int mem_KQ = ncols*kqs_padded*kqar;
|
|
||||||
constexpr int mem_VKQ_parts = VKQ_ratio*ncols*D_padded;
|
|
||||||
__shared__ half KQ[mem_KQ >= mem_VKQ_parts ? mem_KQ : mem_VKQ_parts];
|
|
||||||
float * KQ_f = (float *) KQ;
|
|
||||||
half2 * KQ2 = (half2 *) KQ;
|
|
||||||
|
|
||||||
float KQ_rowsum_f[ncols/nwarps] = {0.0f};
|
|
||||||
float KQ_max_f[ncols/nwarps];
|
|
||||||
float KQ_max_scale_f[ncols/nwarps] = {0.0f};
|
|
||||||
|
|
||||||
#pragma unroll
|
|
||||||
for (int j = 0; j < ncols/nwarps; ++j) {
|
|
||||||
KQ_max_f[j] = -FLT_MAX/2.0f;
|
|
||||||
}
|
|
||||||
|
|
||||||
half2 KQ_rowsum_h2[ncols/nwarps] = {{0.0f, 0.0f}};
|
|
||||||
half2 KQ_max_h2[ncols/nwarps];
|
|
||||||
half2 KQ_max_scale_h2[ncols/nwarps] = {{0.0f, 0.0f}};
|
|
||||||
|
|
||||||
#pragma unroll
|
|
||||||
for (int j = 0; j < ncols/nwarps; ++j) {
|
|
||||||
KQ_max_h2[j] = make_half2(-HALF_MAX_HALF, -HALF_MAX_HALF);
|
|
||||||
}
|
|
||||||
|
|
||||||
__shared__ half VKQ[ncols*D_padded]; // Accumulator for final VKQ slice.
|
|
||||||
half2 * VKQ2 = (half2 *) VKQ;
|
|
||||||
|
|
||||||
#if defined(GGML_USE_HIP) && HIP_VERSION >= 60500000
|
|
||||||
const _Float16 * K_h_f16 = reinterpret_cast<const _Float16 *>(K_h);
|
|
||||||
const _Float16 * V_h_f16 = reinterpret_cast<const _Float16 *>(V_h);
|
|
||||||
_Float16 * KQ_f16 = reinterpret_cast<_Float16 *>(KQ);
|
|
||||||
_Float16 * VKQ_f16 = reinterpret_cast<_Float16 *>(VKQ);
|
|
||||||
#else
|
|
||||||
const half * K_h_f16 = K_h;
|
|
||||||
const half * V_h_f16 = V_h;
|
|
||||||
half * KQ_f16 = KQ;
|
|
||||||
half * VKQ_f16 = VKQ;
|
|
||||||
#endif
|
|
||||||
|
|
||||||
#pragma unroll
|
|
||||||
for (int j0 = 0; j0 < ncols; j0 += nwarps) {
|
|
||||||
const int j = j0 + threadIdx.y;
|
|
||||||
#pragma unroll
|
|
||||||
for (int i0 = 0; i0 < D/2; i0 += warp_size) {
|
|
||||||
const int i = i0 + threadIdx.x;
|
|
||||||
if (i0 + warp_size > D/2 && i >= D/2) {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
VKQ2[j*(D_padded/2) + i] = make_half2(0.0f, 0.0f);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Convert Q to half and apply scale, temporarily store in KQ:
|
|
||||||
#pragma unroll
|
|
||||||
for (int j0 = 0; j0 < ncols; j0 += nwarps) {
|
|
||||||
const int j = j0 + threadIdx.y;
|
|
||||||
#pragma unroll
|
|
||||||
for (int i0 = 0; i0 < D; i0 += warp_size) {
|
|
||||||
const int i = i0 + threadIdx.x;
|
|
||||||
if (i0 + warp_size > D && i >= D) {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
KQ[j*D_padded + i] = ic0 + j < int(ne01.z) ? Q_f[j*stride_Q + i] * scale : 0.0f;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
__syncthreads();
|
|
||||||
|
|
||||||
// Load Q into tensor core fragments/registers since it will be used frequently:
|
|
||||||
#pragma unroll
|
|
||||||
for (int i0 = 0; i0 < D; i0 += 16) {
|
|
||||||
#pragma unroll
|
|
||||||
for (int j0 = 0; j0 < ncols; j0 += frag_n) {
|
|
||||||
wmma::load_matrix_sync(Q_b[i0/16][j0/frag_n], KQ_f16 + j0*D_padded + i0, D_padded);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
__syncthreads();
|
|
||||||
|
|
||||||
// Iterate over ne11 == previous tokens:
|
|
||||||
const int k_VKQ_max = KV_max ? KV_max[sequence*gridDim.x + blockIdx.x] : ne11;
|
|
||||||
for (int k_VKQ_0 = blockIdx.y*FATTN_KQ_STRIDE; k_VKQ_0 < k_VKQ_max; k_VKQ_0 += gridDim.y*FATTN_KQ_STRIDE) {
|
|
||||||
// Calculate tile of KQ:
|
|
||||||
#pragma unroll
|
|
||||||
for (int i_KQ_0 = 0; i_KQ_0 < FATTN_KQ_STRIDE; i_KQ_0 += KQ_stride_tc) {
|
|
||||||
frag_c_KQ KQ_c[ncols/frag_n];
|
|
||||||
#pragma unroll
|
|
||||||
for (int j = 0; j < ncols/frag_n; ++j) {
|
|
||||||
wmma::fill_fragment(KQ_c[j], static_cast<KQ_acc_t>(0.0f));
|
|
||||||
}
|
|
||||||
#pragma unroll
|
|
||||||
for (int k_KQ_0 = 0; k_KQ_0 < D; k_KQ_0 += 16) {
|
|
||||||
frag_a_K K_a;
|
|
||||||
wmma::load_matrix_sync(K_a, K_h_f16 + int64_t(k_VKQ_0 + i_KQ_0 + frag_m*threadIdx.y)*stride_KV + k_KQ_0, stride_KV);
|
|
||||||
#pragma unroll
|
|
||||||
for (int j = 0; j < ncols/frag_n; ++j) {
|
|
||||||
wmma::mma_sync(KQ_c[j], K_a, Q_b[k_KQ_0/16][j], KQ_c[j]);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
#pragma unroll
|
|
||||||
for (int j0 = 0; j0 < ncols; j0 += frag_n) {
|
|
||||||
wmma::store_matrix_sync((KQ_acc_t *) KQ + j0*kqs_padded + i_KQ_0 + frag_m*threadIdx.y, KQ_c[j0/frag_n], kqs_padded, wmma::mem_col_major);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
__syncthreads();
|
|
||||||
|
|
||||||
// Calculate softmax for each KQ column using the current max. value.
|
|
||||||
// The divisor is stored in KQ_rowsum and will be applied at the end.
|
|
||||||
#pragma unroll
|
|
||||||
for (int j0 = 0; j0 < ncols; j0 += nwarps) {
|
|
||||||
const int j = j0 + threadIdx.y;
|
|
||||||
|
|
||||||
if (std::is_same<KQ_acc_t, float>::value) {
|
|
||||||
float KQ_f_tmp[FATTN_KQ_STRIDE / warp_size];
|
|
||||||
#pragma unroll
|
|
||||||
for (int k0 = 0; k0 < FATTN_KQ_STRIDE; k0 += warp_size) {
|
|
||||||
const int k = k0 + threadIdx.x;
|
|
||||||
|
|
||||||
KQ_f_tmp[k0/warp_size] = KQ_f[j*kqs_padded + k];
|
|
||||||
|
|
||||||
if (use_logit_softcap) {
|
|
||||||
KQ_f_tmp[k0/warp_size] = logit_softcap*tanhf(KQ_f_tmp[k0/warp_size]);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
float KQ_max_new = KQ_max_f[j0/nwarps];
|
|
||||||
#pragma unroll
|
|
||||||
for (int k0 = 0; k0 < FATTN_KQ_STRIDE; k0 += warp_size) {
|
|
||||||
const int k = k0 + threadIdx.x;
|
|
||||||
|
|
||||||
KQ_f_tmp[k0/warp_size] += mask && ic0 + j < int(ne01.z) ?
|
|
||||||
__half2float(slopeh*maskh[j*(nb31/sizeof(half)) + k_VKQ_0 + k]) : 0.0f;
|
|
||||||
KQ_max_new = max(KQ_max_new, KQ_f_tmp[k0/warp_size] + FATTN_KQ_MAX_OFFSET);
|
|
||||||
}
|
|
||||||
KQ_max_new = warp_reduce_max<warp_size>(KQ_max_new);
|
|
||||||
|
|
||||||
const float diff = KQ_max_f[j0/nwarps] - KQ_max_new;
|
|
||||||
KQ_max_scale_f[j0/nwarps] = expf(diff);
|
|
||||||
if (diff <= SOFTMAX_FTZ_THRESHOLD) {
|
|
||||||
KQ_max_scale_f[j0/nwarps] = 0.0f;
|
|
||||||
}
|
|
||||||
KQ_max_f[j0/nwarps] = KQ_max_new;
|
|
||||||
|
|
||||||
float KQ_rowsum_add = 0.0f;
|
|
||||||
#pragma unroll
|
|
||||||
for (int k0 = 0; k0 < FATTN_KQ_STRIDE; k0 += warp_size) {
|
|
||||||
const int k = k0 + threadIdx.x;
|
|
||||||
|
|
||||||
const float diff = KQ_f_tmp[k0/warp_size] - KQ_max_f[j0/nwarps];
|
|
||||||
KQ_f_tmp[k0/warp_size] = expf(diff);
|
|
||||||
if (diff <= SOFTMAX_FTZ_THRESHOLD) {
|
|
||||||
KQ_f_tmp[k0/warp_size] = 0.0f;
|
|
||||||
}
|
|
||||||
KQ_rowsum_add += KQ_f_tmp[k0/warp_size];
|
|
||||||
KQ[j*(kqar*kqs_padded) + k] = KQ_f_tmp[k0/warp_size];
|
|
||||||
}
|
|
||||||
KQ_rowsum_add = warp_reduce_sum<warp_size>(KQ_rowsum_add);
|
|
||||||
|
|
||||||
// Scale previous KQ_rowsum to account for a potential increase in KQ_max:
|
|
||||||
KQ_rowsum_f[j0/nwarps] = KQ_max_scale_f[j0/nwarps]*KQ_rowsum_f[j0/nwarps] + KQ_rowsum_add;
|
|
||||||
} else {
|
|
||||||
half2 KQ2_tmp[FATTN_KQ_STRIDE/(2*warp_size)];
|
|
||||||
#pragma unroll
|
|
||||||
for (int k0 = 0; k0 < FATTN_KQ_STRIDE/2; k0 += warp_size) {
|
|
||||||
const int k = k0 + threadIdx.x;
|
|
||||||
|
|
||||||
KQ2_tmp[k0/warp_size] = KQ2[j*(kqs_padded/2) + k];
|
|
||||||
|
|
||||||
if (use_logit_softcap) {
|
|
||||||
// There is no dedicated tangens hyperbolicus function for half2.
|
|
||||||
KQ2_tmp[k0/warp_size] = h2exp(KQ2_tmp[k0/warp_size]*make_half2(2.0f, 2.0f));
|
|
||||||
KQ2_tmp[k0/warp_size] = (KQ2_tmp[k0/warp_size] - make_half2(1.0f, 1.0f))
|
|
||||||
/(KQ2_tmp[k0/warp_size] + make_half2(1.0f, 1.0f));
|
|
||||||
|
|
||||||
KQ2_tmp[k0/warp_size] *= logit_softcap_2;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
half2 KQ_max_new = KQ_max_h2[j0/nwarps];
|
|
||||||
#pragma unroll
|
|
||||||
for (int k0 = 0; k0 < FATTN_KQ_STRIDE/2; k0 += warp_size) {
|
|
||||||
const int k = k0 + threadIdx.x;
|
|
||||||
|
|
||||||
KQ2_tmp[k0/warp_size] += mask && ic0 + j < int(ne01.z) ? slope2*mask2[(j*ne11 + k_VKQ_0)/2 + k] : make_half2(0.0f, 0.0f);
|
|
||||||
KQ_max_new = ggml_cuda_hmax2(KQ_max_new, KQ2_tmp[k0/warp_size]);
|
|
||||||
}
|
|
||||||
KQ_max_new = __half2half2(warp_reduce_max<warp_size>(ggml_cuda_hmax(__low2half(KQ_max_new), __high2half(KQ_max_new))));
|
|
||||||
const half2 diff = KQ_max_h2[j0/nwarps] - KQ_max_new;
|
|
||||||
KQ_max_scale_h2[j0/nwarps] = h2exp(diff);
|
|
||||||
const uint32_t ftz_mask = __hgt2_mask(diff, make_half2(SOFTMAX_FTZ_THRESHOLD, SOFTMAX_FTZ_THRESHOLD));
|
|
||||||
*((uint32_t *) &KQ_max_scale_h2[j0/nwarps]) &= ftz_mask;
|
|
||||||
KQ_max_h2[j0/nwarps] = KQ_max_new;
|
|
||||||
|
|
||||||
half2 KQ_rowsum_add = make_half2(0.0f, 0.0f);
|
|
||||||
#pragma unroll
|
|
||||||
for (int k0 = 0; k0 < FATTN_KQ_STRIDE/2; k0 += warp_size) {
|
|
||||||
const int k = k0 + threadIdx.x;
|
|
||||||
|
|
||||||
const half2 diff = KQ2_tmp[k0/warp_size] - KQ_max_h2[j0/nwarps];
|
|
||||||
KQ2_tmp[k0/warp_size] = h2exp(diff);
|
|
||||||
const uint32_t ftz_mask = __hgt2_mask(diff, make_half2(SOFTMAX_FTZ_THRESHOLD, SOFTMAX_FTZ_THRESHOLD));
|
|
||||||
*((uint32_t *) &KQ2_tmp[k0/warp_size]) &= ftz_mask;
|
|
||||||
KQ_rowsum_add += KQ2_tmp[k0/warp_size];
|
|
||||||
KQ2[j*(kqs_padded/2) + k] = KQ2_tmp[k0/warp_size];
|
|
||||||
}
|
|
||||||
KQ_rowsum_add = warp_reduce_sum<warp_size>(KQ_rowsum_add);
|
|
||||||
|
|
||||||
// Scale previous KQ_rowsum to account for a potential increase in KQ_max:
|
|
||||||
KQ_rowsum_h2[j0/nwarps] = KQ_max_scale_h2[j0/nwarps]*KQ_rowsum_h2[j0/nwarps] + KQ_rowsum_add;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
__syncthreads();
|
|
||||||
|
|
||||||
frag_b KQ_b[FATTN_KQ_STRIDE/(VKQ_ratio*16)][ncols/frag_n];
|
|
||||||
#pragma unroll
|
|
||||||
for (int j0 = 0; j0 < ncols; j0 += frag_n) {
|
|
||||||
#pragma unroll
|
|
||||||
for (int k0 = 0; k0 < FATTN_KQ_STRIDE; k0 += VKQ_ratio*16) {
|
|
||||||
const int k = k0 + (threadIdx.y % VKQ_ratio)*16;
|
|
||||||
wmma::load_matrix_sync(
|
|
||||||
KQ_b[k0/(VKQ_ratio*16)][j0/frag_n],
|
|
||||||
KQ_f16 + j0*(kqar*kqs_padded) + k,
|
|
||||||
kqar*kqs_padded);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
frag_c_VKQ VKQ_c[D/VKQ_stride][ncols/frag_n];
|
|
||||||
#pragma unroll
|
|
||||||
for (int i_VKQ_0 = 0; i_VKQ_0 < D; i_VKQ_0 += VKQ_stride) {
|
|
||||||
#pragma unroll
|
|
||||||
for (int j = 0; j < ncols/frag_n; ++j) {
|
|
||||||
wmma::fill_fragment(VKQ_c[i_VKQ_0/VKQ_stride][j], static_cast<half>(0.0f));
|
|
||||||
}
|
|
||||||
|
|
||||||
#pragma unroll
|
|
||||||
for (int k0 = 0; k0 < FATTN_KQ_STRIDE; k0 += VKQ_ratio*16) {
|
|
||||||
const int k = k0 + (threadIdx.y % VKQ_ratio)*16;
|
|
||||||
|
|
||||||
frag_a_V v_a;
|
|
||||||
wmma::load_matrix_sync(v_a, V_h_f16 + int64_t(k_VKQ_0 + k)*stride_KV + i_VKQ_0 + frag_m*(threadIdx.y/VKQ_ratio), stride_KV);
|
|
||||||
#pragma unroll
|
|
||||||
for (int j = 0; j < ncols/frag_n; ++j) {
|
|
||||||
wmma::mma_sync(VKQ_c[i_VKQ_0/VKQ_stride][j], v_a, KQ_b[k0/(VKQ_ratio*16)][j], VKQ_c[i_VKQ_0/VKQ_stride][j]);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
__syncthreads();
|
|
||||||
|
|
||||||
const int offset_k = (threadIdx.y % VKQ_ratio) * (ncols*D_padded);
|
|
||||||
#pragma unroll
|
|
||||||
for (int i_KQ_0 = 0; i_KQ_0 < D; i_KQ_0 += VKQ_stride) {
|
|
||||||
#pragma unroll
|
|
||||||
for (int j0 = 0; j0 < ncols; j0 += frag_n) {
|
|
||||||
wmma::store_matrix_sync(
|
|
||||||
KQ_f16 + offset_k + j0*D_padded + i_KQ_0 + frag_m*(threadIdx.y/VKQ_ratio),
|
|
||||||
VKQ_c[i_KQ_0/VKQ_stride][j0/frag_n],
|
|
||||||
D_padded, wmma::mem_col_major);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
__syncthreads();
|
|
||||||
|
|
||||||
#pragma unroll
|
|
||||||
for (int j0 = 0; j0 < ncols; j0 += nwarps) {
|
|
||||||
const int j = j0 + threadIdx.y;
|
|
||||||
|
|
||||||
half2 VKQ_scale;
|
|
||||||
if (std::is_same<KQ_acc_t, float>::value) {
|
|
||||||
VKQ_scale = make_half2(KQ_max_scale_f[j0/nwarps], KQ_max_scale_f[j0/nwarps]);
|
|
||||||
} else {
|
|
||||||
VKQ_scale = KQ_max_scale_h2[j0/nwarps];
|
|
||||||
}
|
|
||||||
|
|
||||||
#pragma unroll
|
|
||||||
for (int i0 = 0; i0 < D/2; i0 += warp_size) {
|
|
||||||
const int i = i0 + threadIdx.x;
|
|
||||||
if (i0 + warp_size > D/2 && i >= D/2) {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
|
|
||||||
half2 VKQ_add = make_half2(0.0f, 0.0f);
|
|
||||||
#pragma unroll
|
|
||||||
for (int l = 0; l < VKQ_ratio; ++l) {
|
|
||||||
VKQ_add += KQ2[l*(ncols*D_padded/2) + j*(D_padded/2) + i];
|
|
||||||
}
|
|
||||||
VKQ2[j*(D_padded/2) + i] = VKQ_scale*VKQ2[j*(D_padded/2) + i] + VKQ_add;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
__syncthreads();
|
|
||||||
}
|
|
||||||
|
|
||||||
// Apply attention sinks
|
|
||||||
if (sinksf && blockIdx.y == 0) {
|
|
||||||
const float sinkf = sinksf[head];
|
|
||||||
const half sinkh = __float2half(sinkf);
|
|
||||||
|
|
||||||
#pragma unroll
|
|
||||||
for (int j0 = 0; j0 < ncols; j0 += nwarps) {
|
|
||||||
const int j = j0 + threadIdx.y;
|
|
||||||
|
|
||||||
if (std::is_same<KQ_acc_t, float>::value) {
|
|
||||||
float kqmax_new = fmaxf(KQ_max_f[j0/nwarps], sinkf);
|
|
||||||
|
|
||||||
const float KQ_max_scale = expf(KQ_max_f[j0/nwarps] - kqmax_new);
|
|
||||||
KQ_max_f[j0/nwarps] = kqmax_new;
|
|
||||||
|
|
||||||
KQ_rowsum_f[j0/nwarps] = KQ_rowsum_f[j0/nwarps] * KQ_max_scale + expf(sinkf - KQ_max_f[j0/nwarps]);
|
|
||||||
|
|
||||||
const half2 scale_h2 = make_half2(KQ_max_scale, KQ_max_scale);
|
|
||||||
#pragma unroll
|
|
||||||
for (int i0 = 0; i0 < D/2; i0 += warp_size) {
|
|
||||||
const int i = i0 + threadIdx.x;
|
|
||||||
if (i0 + warp_size > D/2 && i >= D/2) break;
|
|
||||||
VKQ2[j*(D_padded/2) + i] *= scale_h2;
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
half kqmax_old = __low2half(KQ_max_h2[j0/nwarps]);
|
|
||||||
half kqmax_new = fmaxf(kqmax_old, sinkh);
|
|
||||||
KQ_max_h2[j0/nwarps] = __half2half2(kqmax_new);
|
|
||||||
|
|
||||||
const half KQ_max_scale_h = hexp(kqmax_old - kqmax_new);
|
|
||||||
const half2 KQ_max_scale = __half2half2(KQ_max_scale_h);
|
|
||||||
|
|
||||||
KQ_rowsum_h2[j0/nwarps] = KQ_rowsum_h2[j0/nwarps] * KQ_max_scale;
|
|
||||||
const half val = hexp(sinkh - kqmax_new);
|
|
||||||
KQ_rowsum_h2[j0/nwarps].x = __hadd(KQ_rowsum_h2[j0/nwarps].x, val);
|
|
||||||
|
|
||||||
#pragma unroll
|
|
||||||
for (int i0 = 0; i0 < D/2; i0 += warp_size) {
|
|
||||||
const int i = i0 + threadIdx.x;
|
|
||||||
if (i0 + warp_size > D/2 && i >= D/2) break;
|
|
||||||
VKQ2[j*(D_padded/2) + i] *= KQ_max_scale;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
__syncthreads();
|
|
||||||
}
|
|
||||||
#pragma unroll
|
|
||||||
for (int j0 = 0; j0 < ncols; j0 += nwarps) {
|
|
||||||
const int j_VKQ = j0 + threadIdx.y;
|
|
||||||
if (ic0 + j_VKQ >= int(ne01.z)) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
float KQ_rowsum_j;
|
|
||||||
if (std::is_same<KQ_acc_t, float>::value) {
|
|
||||||
KQ_rowsum_j = KQ_rowsum_f[j0/nwarps];
|
|
||||||
} else {
|
|
||||||
KQ_rowsum_j = __low2float(KQ_rowsum_h2[j0/nwarps]) + __high2float(KQ_rowsum_h2[j0/nwarps]);
|
|
||||||
}
|
|
||||||
|
|
||||||
const int j_dst_unrolled = ((sequence*int(ne01.z) + ic0 + j_VKQ)*ne02 + head)*gridDim.y + blockIdx.y;
|
|
||||||
|
|
||||||
#pragma unroll
|
|
||||||
for (int i0 = 0; i0 < D; i0 += warp_size) {
|
|
||||||
const int i = i0 + threadIdx.x;
|
|
||||||
if (i0 + warp_size > D && i >= D) {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
float dst_val = VKQ[j_VKQ*D_padded + i];
|
|
||||||
if (gridDim.y == 1) {
|
|
||||||
dst_val /= KQ_rowsum_j;
|
|
||||||
}
|
|
||||||
dst[j_dst_unrolled*D + i] = dst_val;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (gridDim.y == 1 || threadIdx.x != 0) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
float2 dst_meta_val;
|
|
||||||
if (std::is_same<KQ_acc_t, float>::value) {
|
|
||||||
dst_meta_val.x = KQ_max_f[j0/nwarps];
|
|
||||||
} else {
|
|
||||||
dst_meta_val.x = __low2float(KQ_max_h2[j0/nwarps]);
|
|
||||||
}
|
|
||||||
dst_meta_val.y = KQ_rowsum_j;
|
|
||||||
dst_meta[j_dst_unrolled] = dst_meta_val;
|
|
||||||
}
|
|
||||||
#else
|
|
||||||
GGML_UNUSED_VARS(Q_ptr, K_ptr, V_ptr, mask_ptr, sinks_ptr, KV_max_ptr, dst_ptr, dst_meta_ptr, scale,
|
|
||||||
max_bias, m0, m1, n_head_log2, logit_softcap,
|
|
||||||
ne00, ne01, ne02, ne03,
|
|
||||||
nb01, nb02, nb03,
|
|
||||||
ne10, ne11, ne12, ne13,
|
|
||||||
nb11, nb12, nb13,
|
|
||||||
nb21, nb22, nb23,
|
|
||||||
ne31, ne32, ne33,
|
|
||||||
nb31, nb32, nb33);
|
|
||||||
NO_DEVICE_CODE;
|
|
||||||
#endif // defined(FLASH_ATTN_AVAILABLE) && (defined(GGML_HIP_ROCWMMA_FATTN) && defined(GGML_USE_WMMA_FATTN))
|
|
||||||
}
|
|
||||||
|
|
||||||
constexpr int get_max_power_of_2(int x) {
|
|
||||||
return x % 2 == 0 ? 2*get_max_power_of_2(x/2) : 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
static_assert(get_max_power_of_2(1) == 1, "Test failed.");
|
|
||||||
static_assert(get_max_power_of_2(2) == 2, "Test failed.");
|
|
||||||
static_assert(get_max_power_of_2(4) == 4, "Test failed.");
|
|
||||||
static_assert(get_max_power_of_2(6) == 2, "Test failed.");
|
|
||||||
|
|
||||||
// Number of VKQ rows calculated in parallel:
|
|
||||||
constexpr int get_VKQ_stride(int D, int nwarps, int frag_m) {
|
|
||||||
return (get_max_power_of_2(D/frag_m) < nwarps ? get_max_power_of_2(D/frag_m) : nwarps)*frag_m;
|
|
||||||
}
|
|
||||||
|
|
||||||
static_assert(get_VKQ_stride(128, 1, 32) == 32, "Test failed.");
|
|
||||||
static_assert(get_VKQ_stride(128, 2, 32) == 64, "Test failed.");
|
|
||||||
static_assert(get_VKQ_stride(128, 4, 32) == 128, "Test failed.");
|
|
||||||
static_assert(get_VKQ_stride( 64, 1, 32) == 32, "Test failed.");
|
|
||||||
static_assert(get_VKQ_stride( 64, 2, 32) == 64, "Test failed.");
|
|
||||||
static_assert(get_VKQ_stride( 64, 4, 32) == 64, "Test failed.");
|
|
||||||
static_assert(get_VKQ_stride( 80, 1, 16) == 16, "Test failed.");
|
|
||||||
static_assert(get_VKQ_stride( 80, 2, 16) == 16, "Test failed.");
|
|
||||||
static_assert(get_VKQ_stride( 80, 4, 16) == 16, "Test failed.");
|
|
||||||
|
|
||||||
template <int D, int cols_per_block, typename KQ_acc_t>
|
|
||||||
void ggml_cuda_flash_attn_ext_wmma_f16_case(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
|
|
||||||
const ggml_tensor * KQV = dst;
|
|
||||||
|
|
||||||
constexpr int nwarps = 4;
|
|
||||||
|
|
||||||
constexpr int frag_m = cols_per_block == 8 && D % 32 == 0 ? 32 : 16;
|
|
||||||
const int warp_size = ggml_cuda_info().devices[ggml_cuda_get_device()].warp_size;
|
|
||||||
|
|
||||||
float logit_softcap;
|
|
||||||
memcpy(&logit_softcap, (const float *) KQV->op_params + 2, sizeof(float));
|
|
||||||
|
|
||||||
fattn_kernel_t fattn_kernel;
|
|
||||||
if (logit_softcap == 0.0f) {
|
|
||||||
constexpr bool use_logit_softcap = false;
|
|
||||||
fattn_kernel = flash_attn_ext_f16<
|
|
||||||
D, cols_per_block, nwarps, get_VKQ_stride(D, nwarps, frag_m), KQ_acc_t, use_logit_softcap>;
|
|
||||||
} else {
|
|
||||||
constexpr bool use_logit_softcap = true;
|
|
||||||
fattn_kernel = flash_attn_ext_f16<
|
|
||||||
D, cols_per_block, nwarps, get_VKQ_stride(D, nwarps, frag_m), KQ_acc_t, use_logit_softcap>;
|
|
||||||
}
|
|
||||||
launch_fattn<D, cols_per_block, 1>(ctx, dst, fattn_kernel, nwarps, 0, FATTN_KQ_STRIDE, true, true, false, warp_size);
|
|
||||||
}
|
|
||||||
|
|
||||||
void ggml_cuda_flash_attn_ext_wmma_f16(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
|
|
||||||
const ggml_tensor * KQV = dst;
|
|
||||||
const ggml_tensor * Q = dst->src[0];
|
|
||||||
|
|
||||||
const enum ggml_prec prec = ggml_flash_attn_ext_get_prec(KQV);
|
|
||||||
const int warp_size = ggml_cuda_info().devices[ctx.device].warp_size;
|
|
||||||
|
|
||||||
if (prec != GGML_PREC_DEFAULT) {
|
|
||||||
if (Q->ne[1] <= 32 || Q->ne[0] > 128) {
|
|
||||||
constexpr int cols_per_block = 16;
|
|
||||||
switch (Q->ne[0]) {
|
|
||||||
case 64:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case< 64, cols_per_block, float>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 80:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case< 80, cols_per_block, float>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 96:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case< 96, cols_per_block, float>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 112:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case<112, cols_per_block, float>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 128:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case<128, cols_per_block, float>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 256:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case<256, cols_per_block, float>(ctx, dst);
|
|
||||||
break;
|
|
||||||
default:
|
|
||||||
GGML_ABORT("fatal error");
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
constexpr int cols_per_block = 32;
|
|
||||||
switch (Q->ne[0]) {
|
|
||||||
case 64:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case< 64, cols_per_block, float>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 80:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case< 80, cols_per_block, float>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 96:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case< 96, cols_per_block, float>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 112:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case<112, cols_per_block, float>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 128:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case<128, cols_per_block, float>(ctx, dst);
|
|
||||||
break;
|
|
||||||
// case 256:
|
|
||||||
// ggml_cuda_flash_attn_ext_wmma_f16_case<256, cols_per_block, float>(ctx, dst);
|
|
||||||
// break;
|
|
||||||
default:
|
|
||||||
GGML_ABORT("fatal error");
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
#if !defined(GGML_USE_HIP)
|
|
||||||
if (Q->ne[1] <= 8 && Q->ne[0] % warp_size == 0) {
|
|
||||||
constexpr int cols_per_block = 8;
|
|
||||||
switch (Q->ne[0]) {
|
|
||||||
case 64:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case< 64, cols_per_block, half>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 96:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case< 96, cols_per_block, half>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 128:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case<128, cols_per_block, half>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 256:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case<256, cols_per_block, half>(ctx, dst);
|
|
||||||
break;
|
|
||||||
default:
|
|
||||||
GGML_ABORT("fatal error");
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
#endif // !defined(GGML_USE_HIP)
|
|
||||||
|
|
||||||
if (Q->ne[1] <= 32) {
|
|
||||||
constexpr int cols_per_block = 16;
|
|
||||||
switch (Q->ne[0]) {
|
|
||||||
case 64:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case< 64, cols_per_block, half>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 80:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case< 80, cols_per_block, half>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 96:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case< 96, cols_per_block, half>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 112:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case<112, cols_per_block, half>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 128:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case<128, cols_per_block, half>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 256:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case<256, cols_per_block, half>(ctx, dst);
|
|
||||||
break;
|
|
||||||
default:
|
|
||||||
GGML_ABORT("fatal error");
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
constexpr int cols_per_block = 32;
|
|
||||||
switch (Q->ne[0]) {
|
|
||||||
case 64:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case< 64, cols_per_block, half>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 80:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case< 80, cols_per_block, half>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 96:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case< 96, cols_per_block, half>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 112:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case<112, cols_per_block, half>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 128:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case<128, cols_per_block, half>(ctx, dst);
|
|
||||||
break;
|
|
||||||
case 256:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16_case<256, cols_per_block, half>(ctx, dst);
|
|
||||||
break;
|
|
||||||
default:
|
|
||||||
GGML_ABORT("fatal error");
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -1,51 +0,0 @@
|
|||||||
#pragma once
|
|
||||||
|
|
||||||
#include "common.cuh"
|
|
||||||
|
|
||||||
#if defined(GGML_USE_MUSA)
|
|
||||||
#define GGML_USE_WMMA_FATTN
|
|
||||||
#endif // defined(GGML_USE_MUSA)
|
|
||||||
|
|
||||||
#if defined(GGML_HIP_ROCWMMA_FATTN)
|
|
||||||
#if defined(CDNA) && (ROCWMMA_VERSION_MAJOR < 2 || ROCWMMA_VERSION_MINOR > 0 || ROCWMMA_VERSION_PATCH > 0)
|
|
||||||
#define GGML_USE_WMMA_FATTN
|
|
||||||
#elif defined(CDNA)
|
|
||||||
#warning "rocwmma fattn on CDNA is broken on rocwmma v2.0.0, expect degraded performance"
|
|
||||||
#endif // defined(CDNA) && (ROCWMMA_VERSION_MAJOR < 2 || ROCWMMA_VERSION_MINOR > 0 || ROCWMMA_VERSION_PATCH > 0)
|
|
||||||
#if defined(RDNA3)
|
|
||||||
#define GGML_USE_WMMA_FATTN
|
|
||||||
#endif // defined(RDNA3)
|
|
||||||
#if defined(RDNA4) && ROCWMMA_VERSION_MAJOR > 1
|
|
||||||
#define GGML_USE_WMMA_FATTN
|
|
||||||
#elif defined(RDNA4)
|
|
||||||
#warning "rocwmma fattn is not supported on RDNA4 on rocwmma < v2.0.0, expect degraded performance"
|
|
||||||
#endif // defined(RDNA4) && ROCWMMA_VERSION_MAJOR > 1
|
|
||||||
#endif // defined(GGML_HIP_ROCWMMA_FATTN)
|
|
||||||
|
|
||||||
// WMMA flash attention requires FP16 matrix instructions to be available for ggml code.
|
|
||||||
static bool ggml_cuda_should_use_wmma_fattn(const int cc) {
|
|
||||||
#if defined(GGML_USE_HIP) && !defined(GGML_HIP_ROCWMMA_FATTN)
|
|
||||||
return false;
|
|
||||||
#else
|
|
||||||
if ((GGML_CUDA_CC_IS_NVIDIA(cc) && ggml_cuda_highest_compiled_arch(cc) == GGML_CUDA_CC_VOLTA) ||
|
|
||||||
GGML_CUDA_CC_IS_RDNA3(cc) || GGML_CUDA_CC_IS_MTHREADS(cc)) {
|
|
||||||
return true;
|
|
||||||
} else if (GGML_CUDA_CC_IS_CDNA(cc)){
|
|
||||||
#if defined(GGML_HIP_ROCWMMA_FATTN) && (ROCWMMA_VERSION_MAJOR < 2 || ROCWMMA_VERSION_MINOR > 0 || ROCWMMA_VERSION_PATCH > 0)
|
|
||||||
return true;
|
|
||||||
#else
|
|
||||||
return false;
|
|
||||||
#endif // defined(GGML_HIP_ROCWMMA_FATTN) (ROCWMMA_VERSION_MAJOR < 2 || ROCWMMA_VERSION_MINOR > 0 || ROCWMMA_VERSION_PATCH > 0)
|
|
||||||
} else if (GGML_CUDA_CC_IS_RDNA4(cc)) {
|
|
||||||
#if defined(GGML_HIP_ROCWMMA_FATTN) && ROCWMMA_VERSION_MAJOR > 1
|
|
||||||
return true;
|
|
||||||
#else
|
|
||||||
return false;
|
|
||||||
#endif // defined(GGML_HIP_ROCWMMA_FATTN) && ROCWMMA_VERSION_MAJOR > 1
|
|
||||||
} else {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
#endif // defined(GGML_USE_HIP) && !defined(GGML_HIP_ROCWMMA_FATTN)
|
|
||||||
}
|
|
||||||
|
|
||||||
void ggml_cuda_flash_attn_ext_wmma_f16(ggml_backend_cuda_context & ctx, ggml_tensor * dst);
|
|
||||||
@@ -3,7 +3,6 @@
|
|||||||
#include "fattn-mma-f16.cuh"
|
#include "fattn-mma-f16.cuh"
|
||||||
#include "fattn-tile.cuh"
|
#include "fattn-tile.cuh"
|
||||||
#include "fattn-vec.cuh"
|
#include "fattn-vec.cuh"
|
||||||
#include "fattn-wmma-f16.cuh"
|
|
||||||
#include "fattn.cuh"
|
#include "fattn.cuh"
|
||||||
|
|
||||||
template <int DKQ, int DV, int ncols2>
|
template <int DKQ, int DV, int ncols2>
|
||||||
@@ -330,11 +329,10 @@ static void ggml_cuda_flash_attn_ext_vec(ggml_backend_cuda_context & ctx, ggml_t
|
|||||||
|
|
||||||
// Best FlashAttention kernel for a specific GPU:
|
// Best FlashAttention kernel for a specific GPU:
|
||||||
enum best_fattn_kernel {
|
enum best_fattn_kernel {
|
||||||
BEST_FATTN_KERNEL_NONE = 0,
|
BEST_FATTN_KERNEL_NONE = 0,
|
||||||
BEST_FATTN_KERNEL_TILE = 200,
|
BEST_FATTN_KERNEL_TILE = 200,
|
||||||
BEST_FATTN_KERNEL_VEC = 100,
|
BEST_FATTN_KERNEL_VEC = 100,
|
||||||
BEST_FATTN_KERNEL_WMMA_F16 = 300,
|
BEST_FATTN_KERNEL_MMA_F16 = 400,
|
||||||
BEST_FATTN_KERNEL_MMA_F16 = 400,
|
|
||||||
};
|
};
|
||||||
|
|
||||||
static bool ggml_cuda_fattn_kv_type_supported(ggml_type type) {
|
static bool ggml_cuda_fattn_kv_type_supported(ggml_type type) {
|
||||||
@@ -500,14 +498,6 @@ static best_fattn_kernel ggml_cuda_get_best_fattn_kernel(const int device, const
|
|||||||
return BEST_FATTN_KERNEL_MMA_F16;
|
return BEST_FATTN_KERNEL_MMA_F16;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Use the WMMA kernel if possible:
|
|
||||||
if (ggml_cuda_should_use_wmma_fattn(cc) && K->ne[1] % FATTN_KQ_STRIDE == 0 && Q->ne[0] != 40 && Q->ne[0] != 72 && Q->ne[0] != 192 && Q->ne[0] != 512 && Q->ne[0] != 576) {
|
|
||||||
if (can_use_vector_kernel && Q->ne[1] <= 2) {
|
|
||||||
return BEST_FATTN_KERNEL_VEC;
|
|
||||||
}
|
|
||||||
return BEST_FATTN_KERNEL_WMMA_F16;
|
|
||||||
}
|
|
||||||
|
|
||||||
// AMD MFMA needs a certain minimum batch size to outscale the tile kernel for large head sizes.
|
// AMD MFMA needs a certain minimum batch size to outscale the tile kernel for large head sizes.
|
||||||
if ((amd_mfma_available(cc) && Q->ne[0] <= 256) && Q->ne[0] != 40 && Q->ne[0] != 72) {
|
if ((amd_mfma_available(cc) && Q->ne[0] <= 256) && Q->ne[0] != 40 && Q->ne[0] != 72) {
|
||||||
if ((Q->ne[0] <= 64 && Q->ne[1] * gqa_ratio_eff > 8)) {
|
if ((Q->ne[0] <= 64 && Q->ne[1] * gqa_ratio_eff > 8)) {
|
||||||
@@ -559,7 +549,6 @@ size_t ggml_cuda_flash_attn_ext_get_alloc_size(int device, const ggml_tensor * d
|
|||||||
|
|
||||||
switch (kernel) {
|
switch (kernel) {
|
||||||
case BEST_FATTN_KERNEL_TILE:
|
case BEST_FATTN_KERNEL_TILE:
|
||||||
case BEST_FATTN_KERNEL_WMMA_F16:
|
|
||||||
case BEST_FATTN_KERNEL_MMA_F16:
|
case BEST_FATTN_KERNEL_MMA_F16:
|
||||||
need_f16_K = true;
|
need_f16_K = true;
|
||||||
need_f16_V = true;
|
need_f16_V = true;
|
||||||
@@ -589,9 +578,6 @@ void ggml_cuda_flash_attn_ext(ggml_backend_cuda_context & ctx, ggml_tensor * dst
|
|||||||
case BEST_FATTN_KERNEL_VEC:
|
case BEST_FATTN_KERNEL_VEC:
|
||||||
ggml_cuda_flash_attn_ext_vec(ctx, dst);
|
ggml_cuda_flash_attn_ext_vec(ctx, dst);
|
||||||
break;
|
break;
|
||||||
case BEST_FATTN_KERNEL_WMMA_F16:
|
|
||||||
ggml_cuda_flash_attn_ext_wmma_f16(ctx, dst);
|
|
||||||
break;
|
|
||||||
case BEST_FATTN_KERNEL_MMA_F16:
|
case BEST_FATTN_KERNEL_MMA_F16:
|
||||||
ggml_cuda_flash_attn_ext_mma_f16(ctx, dst);
|
ggml_cuda_flash_attn_ext_mma_f16(ctx, dst);
|
||||||
break;
|
break;
|
||||||
|
|||||||
+201
-22
@@ -40,6 +40,35 @@ static __global__ void k_get_rows(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
template<typename dst_t, dequantize_kq_t<dst_t> dequantize_kq>
|
||||||
|
static __global__ void k_get_rows_kq(
|
||||||
|
const void * __restrict__ src0, const int32_t * __restrict__ src1, dst_t * __restrict__ dst,
|
||||||
|
const int64_t ne00, /*const int64_t ne01, const int64_t ne02, const int64_t ne03,*/
|
||||||
|
/*const int64_t ne10,*/ const int64_t ne11, const uint3 ne12_fdv, /*const int64_t ne13,*/
|
||||||
|
/*const size_t s0,*/ const size_t s1, const size_t s2, const size_t s3,
|
||||||
|
/*const size_t nb00,*/ const size_t nb01, const size_t nb02, const size_t nb03,
|
||||||
|
const size_t s10, const size_t s11, const size_t s12/*, const size_t s13*/) {
|
||||||
|
|
||||||
|
ggml_cuda_pdl_sync();
|
||||||
|
const int64_t nsb = ne00/QK_K; // super-blocks per row
|
||||||
|
for (int64_t z = blockIdx.z; z < ne11*(int64_t)ne12_fdv.z; z += gridDim.z) {
|
||||||
|
// The x and y dimensions of the grid are swapped because the maximum allowed grid size for x is higher.
|
||||||
|
const int i10 = blockIdx.x;
|
||||||
|
const uint2 dm = fast_div_modulo((uint32_t)z, ne12_fdv);
|
||||||
|
const int i11 = dm.x;
|
||||||
|
const int i12 = dm.y;
|
||||||
|
|
||||||
|
const int i01 = src1[i10*s10 + i11*s11 + i12*s12];
|
||||||
|
|
||||||
|
dst_t * dst_row = dst + i10*s1 + i11*s2 + i12*s3;
|
||||||
|
const void * src0_row = (const char *) src0 + i01*nb01 + i11*nb02 + i12*nb03;
|
||||||
|
|
||||||
|
for (int64_t ib = blockIdx.y; ib < nsb; ib += gridDim.y) {
|
||||||
|
dequantize_kq(src0_row, ib, dst_row + ib*QK_K, threadIdx.x);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
template<typename src0_t, typename dst_t>
|
template<typename src0_t, typename dst_t>
|
||||||
static __global__ void k_get_rows_float(
|
static __global__ void k_get_rows_float(
|
||||||
const src0_t * src0_ptr, const int32_t * src1_ptr, dst_t * dst_ptr,
|
const src0_t * src0_ptr, const int32_t * src1_ptr, dst_t * dst_ptr,
|
||||||
@@ -55,27 +84,51 @@ static __global__ void k_get_rows_float(
|
|||||||
dst_t * GGML_CUDA_RESTRICT dst = dst_ptr;
|
dst_t * GGML_CUDA_RESTRICT dst = dst_ptr;
|
||||||
ggml_cuda_pdl_sync();
|
ggml_cuda_pdl_sync();
|
||||||
for (int64_t z = blockIdx.z; z < ne11*(int64_t)ne12_fdv.z; z += gridDim.z) {
|
for (int64_t z = blockIdx.z; z < ne11*(int64_t)ne12_fdv.z; z += gridDim.z) {
|
||||||
|
// The x and y dimensions of the grid are swapped because the maximum allowed grid size for x is higher.
|
||||||
|
const int i10 = blockIdx.x;
|
||||||
|
const uint2 dm = fast_div_modulo((uint32_t)z, ne12_fdv);
|
||||||
|
const int i11 = dm.x;
|
||||||
|
const int i12 = dm.y;
|
||||||
|
|
||||||
|
const int i01 = src1[i10*s10 + i11*s11 + i12*s12];
|
||||||
|
|
||||||
|
dst_t * GGML_CUDA_RESTRICT dst_row = dst + i10*s1 + i11*s2 + i12*s3;
|
||||||
|
const src0_t * GGML_CUDA_RESTRICT src0_row = (const src0_t *)((const char *) src0 + i01*nb01 + i11*nb02 + i12*nb03);
|
||||||
|
|
||||||
for (int64_t i00 = blockIdx.y*blockDim.x + threadIdx.x; i00 < ne00; i00 += gridDim.y*blockDim.x) {
|
for (int64_t i00 = blockIdx.y*blockDim.x + threadIdx.x; i00 < ne00; i00 += gridDim.y*blockDim.x) {
|
||||||
// The x and y dimensions of the grid are swapped because the maximum allowed grid size for x is higher.
|
|
||||||
const int i10 = blockIdx.x;
|
|
||||||
const uint2 dm = fast_div_modulo((uint32_t)z, ne12_fdv);
|
|
||||||
const int i11 = dm.x;
|
|
||||||
const int i12 = dm.y;
|
|
||||||
|
|
||||||
if (i00 >= ne00) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
const int i01 = src1[i10*s10 + i11*s11 + i12*s12];
|
|
||||||
|
|
||||||
dst_t * dst_row = dst + i10*s1 + i11*s2 + i12*s3;
|
|
||||||
const src0_t * src0_row = (const src0_t *)((const char *) src0 + i01*nb01 + i11*nb02 + i12*nb03);
|
|
||||||
|
|
||||||
dst_row[i00] = ggml_cuda_cast<dst_t>(src0_row[i00]);
|
dst_row[i00] = ggml_cuda_cast<dst_t>(src0_row[i00]);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
template<typename dst_t>
|
||||||
|
static __global__ void k_get_rows_float_vec(
|
||||||
|
const dst_t * src0_ptr, const int32_t * src1_ptr, dst_t * dst_ptr,
|
||||||
|
const int64_t ne00v,
|
||||||
|
const int64_t ne11, const uint3 ne12_fdv,
|
||||||
|
const size_t s1, const size_t s2, const size_t s3,
|
||||||
|
const size_t nb01, const size_t nb02, const size_t nb03,
|
||||||
|
const size_t s10, const size_t s11, const size_t s12) {
|
||||||
|
|
||||||
|
ggml_cuda_pdl_lc();
|
||||||
|
ggml_cuda_pdl_sync();
|
||||||
|
for (int64_t z = blockIdx.z; z < ne11*(int64_t)ne12_fdv.z; z += gridDim.z) {
|
||||||
|
const int i10 = blockIdx.x;
|
||||||
|
const uint2 dm = fast_div_modulo((uint32_t)z, ne12_fdv);
|
||||||
|
const int i11 = dm.x;
|
||||||
|
const int i12 = dm.y;
|
||||||
|
|
||||||
|
const int i01 = src1_ptr[i10*s10 + i11*s11 + i12*s12];
|
||||||
|
|
||||||
|
int4 * GGML_CUDA_RESTRICT dst_row = (int4 *) (dst_ptr + i10*s1 + i11*s2 + i12*s3);
|
||||||
|
const int4 * GGML_CUDA_RESTRICT src0_row = (const int4 *)((const char *) src0_ptr + i01*nb01 + i11*nb02 + i12*nb03);
|
||||||
|
|
||||||
|
for (int64_t i = blockIdx.y*blockDim.x + threadIdx.x; i < ne00v; i += gridDim.y*blockDim.x) {
|
||||||
|
dst_row[i] = src0_row[i];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
template<typename grad_t, typename dst_t>
|
template<typename grad_t, typename dst_t>
|
||||||
static __global__ void k_get_rows_back_float(
|
static __global__ void k_get_rows_back_float(
|
||||||
const grad_t * __restrict__ grad, const int32_t * __restrict__ rows, dst_t * __restrict__ dst,
|
const grad_t * __restrict__ grad, const int32_t * __restrict__ rows, dst_t * __restrict__ dst,
|
||||||
@@ -140,16 +193,18 @@ static void get_rows_cuda_q(
|
|||||||
s10, s11, s12/*, s13*/);
|
s10, s11, s12/*, s13*/);
|
||||||
}
|
}
|
||||||
|
|
||||||
template<typename src0_t, typename dst_t>
|
template<int block_dim, typename dst_t, dequantize_kq_t<dst_t> dequantize_kq>
|
||||||
static void get_rows_cuda_float(
|
static void get_rows_cuda_kq(
|
||||||
const src0_t * src0_d, const int32_t * src1_d, dst_t * dst_d,
|
const void * src0_d, const int32_t * src1_d, dst_t * dst_d,
|
||||||
const int64_t ne00, const size_t nb01, const size_t nb02, const size_t nb03,
|
const int64_t ne00, const size_t nb01, const size_t nb02, const size_t nb03,
|
||||||
const int64_t ne10, const int64_t ne11, const int64_t ne12, const size_t nb10, const size_t nb11, const size_t nb12,
|
const int64_t ne10, const int64_t ne11, const int64_t ne12, const size_t nb10, const size_t nb11, const size_t nb12,
|
||||||
const size_t nb1, const size_t nb2, const size_t nb3,
|
const size_t nb1, const size_t nb2, const size_t nb3,
|
||||||
cudaStream_t stream) {
|
cudaStream_t stream) {
|
||||||
const dim3 block_dims(CUDA_GET_ROWS_BLOCK_SIZE, 1, 1);
|
GGML_ASSERT(ne00 % QK_K == 0);
|
||||||
const int block_num_y = (ne00 + CUDA_GET_ROWS_BLOCK_SIZE - 1) / CUDA_GET_ROWS_BLOCK_SIZE;
|
const int64_t nsb = ne00/QK_K;
|
||||||
const dim3 block_nums(ne10, MIN(block_num_y, UINT16_MAX), MIN(ne11*ne12, UINT16_MAX));
|
|
||||||
|
const dim3 block_dims(block_dim, 1, 1);
|
||||||
|
const dim3 block_nums(ne10, MIN(nsb, UINT16_MAX), MIN(ne11*ne12, UINT16_MAX));
|
||||||
|
|
||||||
// strides in elements
|
// strides in elements
|
||||||
// const size_t s0 = nb0 / sizeof(dst_t);
|
// const size_t s0 = nb0 / sizeof(dst_t);
|
||||||
@@ -166,6 +221,67 @@ static void get_rows_cuda_float(
|
|||||||
GGML_ASSERT(ne11 <= std::numeric_limits<uint32_t>::max() / ne12);
|
GGML_ASSERT(ne11 <= std::numeric_limits<uint32_t>::max() / ne12);
|
||||||
const uint3 ne12_fdv = init_fastdiv_values(ne12);
|
const uint3 ne12_fdv = init_fastdiv_values(ne12);
|
||||||
|
|
||||||
|
k_get_rows_kq<dst_t, dequantize_kq><<<block_nums, block_dims, 0, stream>>>(
|
||||||
|
src0_d, src1_d, dst_d,
|
||||||
|
ne00, /*ne01, ne02, ne03,*/
|
||||||
|
/*ne10,*/ ne11, ne12_fdv, /*ne13,*/
|
||||||
|
/* s0,*/ s1, s2, s3,
|
||||||
|
/* nb00,*/ nb01, nb02, nb03,
|
||||||
|
s10, s11, s12/*, s13*/);
|
||||||
|
}
|
||||||
|
|
||||||
|
template<typename src0_t, typename dst_t>
|
||||||
|
static void get_rows_cuda_float(
|
||||||
|
const src0_t * src0_d, const int32_t * src1_d, dst_t * dst_d,
|
||||||
|
const int64_t ne00, const size_t nb01, const size_t nb02, const size_t nb03,
|
||||||
|
const int64_t ne10, const int64_t ne11, const int64_t ne12, const size_t nb10, const size_t nb11, const size_t nb12,
|
||||||
|
const size_t nb1, const size_t nb2, const size_t nb3,
|
||||||
|
cudaStream_t stream) {
|
||||||
|
const dim3 block_dims(CUDA_GET_ROWS_BLOCK_SIZE, 1, 1);
|
||||||
|
|
||||||
|
// strides in elements
|
||||||
|
// const size_t s0 = nb0 / sizeof(dst_t);
|
||||||
|
const size_t s1 = nb1 / sizeof(dst_t);
|
||||||
|
const size_t s2 = nb2 / sizeof(dst_t);
|
||||||
|
const size_t s3 = nb3 / sizeof(dst_t);
|
||||||
|
|
||||||
|
const size_t s10 = nb10 / sizeof(int32_t);
|
||||||
|
const size_t s11 = nb11 / sizeof(int32_t);
|
||||||
|
const size_t s12 = nb12 / sizeof(int32_t);
|
||||||
|
// const size_t s13 = nb13 / sizeof(int32_t);
|
||||||
|
|
||||||
|
GGML_ASSERT(ne12 > 0);
|
||||||
|
GGML_ASSERT(ne11 <= std::numeric_limits<uint32_t>::max() / ne12);
|
||||||
|
const uint3 ne12_fdv = init_fastdiv_values(ne12);
|
||||||
|
|
||||||
|
if constexpr (std::is_same<src0_t, dst_t>::value) {
|
||||||
|
constexpr int VEC = 16 / sizeof(dst_t);
|
||||||
|
const int64_t ne00v = ne00 / VEC;
|
||||||
|
const int64_t vec_block_num_y = (ne00v + CUDA_GET_ROWS_BLOCK_SIZE - 1) / CUDA_GET_ROWS_BLOCK_SIZE;
|
||||||
|
const bool enough_blocks = vec_block_num_y * ne10 * ne11 * ne12 >= 128;
|
||||||
|
const bool can_vec = VEC > 1 && enough_blocks &&
|
||||||
|
(ne00 % VEC == 0) &&
|
||||||
|
(nb01 % 16 == 0) && (nb02 % 16 == 0) && (nb03 % 16 == 0) &&
|
||||||
|
(nb1 % 16 == 0) && (nb2 % 16 == 0) && (nb3 % 16 == 0) &&
|
||||||
|
(((uintptr_t) src0_d) % 16 == 0) && (((uintptr_t) dst_d) % 16 == 0);
|
||||||
|
|
||||||
|
if (can_vec) {
|
||||||
|
const int block_num_y = vec_block_num_y;
|
||||||
|
const dim3 block_nums(ne10, MIN(block_num_y, UINT16_MAX), MIN(ne11*ne12, UINT16_MAX));
|
||||||
|
const ggml_cuda_kernel_launch_params launch_params = ggml_cuda_kernel_launch_params{block_nums, block_dims, 0, stream};
|
||||||
|
ggml_cuda_kernel_launch(k_get_rows_float_vec<dst_t>, launch_params,
|
||||||
|
(const dst_t *) src0_d, src1_d, dst_d,
|
||||||
|
ne00v, ne11, ne12_fdv,
|
||||||
|
s1, s2, s3,
|
||||||
|
nb01, nb02, nb03,
|
||||||
|
s10, s11, s12);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const int block_num_y = (ne00 + CUDA_GET_ROWS_BLOCK_SIZE - 1) / CUDA_GET_ROWS_BLOCK_SIZE;
|
||||||
|
const dim3 block_nums(ne10, MIN(block_num_y, UINT16_MAX), MIN(ne11*ne12, UINT16_MAX));
|
||||||
|
|
||||||
const ggml_cuda_kernel_launch_params launch_params = ggml_cuda_kernel_launch_params{block_nums, block_dims, 0, stream};
|
const ggml_cuda_kernel_launch_params launch_params = ggml_cuda_kernel_launch_params{block_nums, block_dims, 0, stream};
|
||||||
ggml_cuda_kernel_launch(k_get_rows_float<src0_t, dst_t>, launch_params,
|
ggml_cuda_kernel_launch(k_get_rows_float<src0_t, dst_t>, launch_params,
|
||||||
src0_d, src1_d, dst_d,
|
src0_d, src1_d, dst_d,
|
||||||
@@ -204,6 +320,10 @@ static void ggml_cuda_get_rows_switch_src0_type(
|
|||||||
get_rows_cuda_q<QK1_0, QR1_0, dequantize_q1_0>(src0_d, src1_d, dst_d,
|
get_rows_cuda_q<QK1_0, QR1_0, dequantize_q1_0>(src0_d, src1_d, dst_d,
|
||||||
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
break;
|
break;
|
||||||
|
case GGML_TYPE_Q2_0:
|
||||||
|
get_rows_cuda_q<QK2_0, QR2_0, dequantize_q2_0>(src0_d, src1_d, dst_d,
|
||||||
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
|
break;
|
||||||
case GGML_TYPE_Q4_0:
|
case GGML_TYPE_Q4_0:
|
||||||
get_rows_cuda_q<QK4_0, QR4_0, dequantize_q4_0>(src0_d, src1_d, dst_d,
|
get_rows_cuda_q<QK4_0, QR4_0, dequantize_q4_0>(src0_d, src1_d, dst_d,
|
||||||
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
@@ -224,8 +344,67 @@ static void ggml_cuda_get_rows_switch_src0_type(
|
|||||||
get_rows_cuda_q<QK8_0, QR8_0, dequantize_q8_0>(src0_d, src1_d, dst_d,
|
get_rows_cuda_q<QK8_0, QR8_0, dequantize_q8_0>(src0_d, src1_d, dst_d,
|
||||||
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
break;
|
break;
|
||||||
|
case GGML_TYPE_Q2_K:
|
||||||
|
get_rows_cuda_kq<64, dst_t, dequantize_q2_K<dst_t>>(src0_d, src1_d, dst_d,
|
||||||
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
|
break;
|
||||||
|
case GGML_TYPE_Q3_K:
|
||||||
|
get_rows_cuda_kq<64, dst_t, dequantize_q3_K<dst_t>>(src0_d, src1_d, dst_d,
|
||||||
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
|
break;
|
||||||
|
case GGML_TYPE_Q4_K:
|
||||||
|
get_rows_cuda_kq<32, dst_t, dequantize_q4_K<dst_t>>(src0_d, src1_d, dst_d,
|
||||||
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
|
break;
|
||||||
|
case GGML_TYPE_Q5_K:
|
||||||
|
get_rows_cuda_kq<64, dst_t, dequantize_q5_K<dst_t>>(src0_d, src1_d, dst_d,
|
||||||
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
|
break;
|
||||||
|
case GGML_TYPE_Q6_K:
|
||||||
|
get_rows_cuda_kq<64, dst_t, dequantize_q6_K<dst_t>>(src0_d, src1_d, dst_d,
|
||||||
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
|
break;
|
||||||
|
case GGML_TYPE_IQ2_XXS:
|
||||||
|
get_rows_cuda_kq<32, dst_t, dequantize_iq2_xxs<dst_t>>(src0_d, src1_d, dst_d,
|
||||||
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
|
break;
|
||||||
|
case GGML_TYPE_IQ2_XS:
|
||||||
|
get_rows_cuda_kq<32, dst_t, dequantize_iq2_xs<dst_t>>(src0_d, src1_d, dst_d,
|
||||||
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
|
break;
|
||||||
|
case GGML_TYPE_IQ2_S:
|
||||||
|
get_rows_cuda_kq<32, dst_t, dequantize_iq2_s<dst_t>>(src0_d, src1_d, dst_d,
|
||||||
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
|
break;
|
||||||
|
case GGML_TYPE_IQ3_XXS:
|
||||||
|
get_rows_cuda_kq<32, dst_t, dequantize_iq3_xxs<dst_t>>(src0_d, src1_d, dst_d,
|
||||||
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
|
break;
|
||||||
|
case GGML_TYPE_IQ3_S:
|
||||||
|
get_rows_cuda_kq<32, dst_t, dequantize_iq3_s<dst_t>>(src0_d, src1_d, dst_d,
|
||||||
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
|
break;
|
||||||
|
case GGML_TYPE_IQ1_S:
|
||||||
|
get_rows_cuda_kq<32, dst_t, dequantize_iq1_s<dst_t>>(src0_d, src1_d, dst_d,
|
||||||
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
|
break;
|
||||||
|
case GGML_TYPE_IQ1_M:
|
||||||
|
get_rows_cuda_kq<32, dst_t, dequantize_iq1_m<dst_t>>(src0_d, src1_d, dst_d,
|
||||||
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
|
break;
|
||||||
|
case GGML_TYPE_IQ4_NL:
|
||||||
|
get_rows_cuda_kq<32, dst_t, dequantize_iq4_nl<dst_t>>(src0_d, src1_d, dst_d,
|
||||||
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
|
break;
|
||||||
|
case GGML_TYPE_IQ4_XS:
|
||||||
|
get_rows_cuda_kq<32, dst_t, dequantize_iq4_xs<dst_t>>(src0_d, src1_d, dst_d,
|
||||||
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
|
break;
|
||||||
|
case GGML_TYPE_MXFP4:
|
||||||
|
get_rows_cuda_kq<32, dst_t, dequantize_mxfp4<dst_t>>(src0_d, src1_d, dst_d,
|
||||||
|
ne00, nb01, nb02, nb03, ne10, ne11, ne12, nb10, nb11, nb12, nb1, nb2, nb3, stream);
|
||||||
|
break;
|
||||||
default:
|
default:
|
||||||
// TODO: k-quants
|
|
||||||
GGML_ABORT("%s: unsupported src0 type: %s\n", __func__, ggml_type_name(src0_type));
|
GGML_ABORT("%s: unsupported src0 type: %s\n", __func__, ggml_type_name(src0_type));
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|||||||
+248
-56
@@ -58,6 +58,7 @@
|
|||||||
#include "ggml-cuda/wkv.cuh"
|
#include "ggml-cuda/wkv.cuh"
|
||||||
#include "ggml-cuda/gla.cuh"
|
#include "ggml-cuda/gla.cuh"
|
||||||
#include "ggml-cuda/gated_delta_net.cuh"
|
#include "ggml-cuda/gated_delta_net.cuh"
|
||||||
|
#include "ggml-cuda/dsv4-hc.cuh"
|
||||||
#include "ggml-cuda/set.cuh"
|
#include "ggml-cuda/set.cuh"
|
||||||
#include "ggml-cuda/set-rows.cuh"
|
#include "ggml-cuda/set-rows.cuh"
|
||||||
#include "ggml-cuda/pad_reflect_1d.cuh"
|
#include "ggml-cuda/pad_reflect_1d.cuh"
|
||||||
@@ -65,6 +66,7 @@
|
|||||||
#include "ggml-cuda/tri.cuh"
|
#include "ggml-cuda/tri.cuh"
|
||||||
#include "ggml-cuda/cumsum.cuh"
|
#include "ggml-cuda/cumsum.cuh"
|
||||||
#include "ggml-cuda/fill.cuh"
|
#include "ggml-cuda/fill.cuh"
|
||||||
|
#include "ggml-cuda/lightning-indexer.cuh"
|
||||||
#include "ggml.h"
|
#include "ggml.h"
|
||||||
|
|
||||||
#include <algorithm>
|
#include <algorithm>
|
||||||
@@ -104,17 +106,27 @@ void ggml_cuda_error(const char * stmt, const char * func, const char * file, in
|
|||||||
GGML_ABORT(GGML_CUDA_NAME " error");
|
GGML_ABORT(GGML_CUDA_NAME " error");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// map a (possibly virtual) device id to the physical CUDA device that backs it
|
||||||
|
static int ggml_cuda_get_physical_device(int device) {
|
||||||
|
const ggml_cuda_device_info & info = ggml_cuda_info();
|
||||||
|
GGML_ASSERT(device >= 0 && device < info.device_count);
|
||||||
|
return info.devices[device].physical_device;
|
||||||
|
}
|
||||||
|
|
||||||
// this is faster on Windows
|
// this is faster on Windows
|
||||||
// probably because the Windows CUDA libraries forget to make this check before invoking the drivers
|
// probably because the Windows CUDA libraries forget to make this check before invoking the drivers
|
||||||
void ggml_cuda_set_device(int device) {
|
void ggml_cuda_set_device(int device) {
|
||||||
|
// translate the (possibly virtual) device id to the physical CUDA device that backs it
|
||||||
|
const int physical_device = ggml_cuda_get_physical_device(device);
|
||||||
|
|
||||||
int current_device;
|
int current_device;
|
||||||
CUDA_CHECK(cudaGetDevice(¤t_device));
|
CUDA_CHECK(cudaGetDevice(¤t_device));
|
||||||
|
|
||||||
if (device == current_device) {
|
if (physical_device == current_device) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
CUDA_CHECK(cudaSetDevice(device));
|
CUDA_CHECK(cudaSetDevice(physical_device));
|
||||||
}
|
}
|
||||||
|
|
||||||
int ggml_cuda_get_device() {
|
int ggml_cuda_get_device() {
|
||||||
@@ -205,56 +217,102 @@ static int ggml_cuda_parse_id(char devName[]) {
|
|||||||
static ggml_cuda_device_info ggml_cuda_init() {
|
static ggml_cuda_device_info ggml_cuda_init() {
|
||||||
ggml_cuda_device_info info = {};
|
ggml_cuda_device_info info = {};
|
||||||
|
|
||||||
cudaError_t err = cudaGetDeviceCount(&info.device_count);
|
cudaError_t err = cudaGetDeviceCount(&info.physical_device_count);
|
||||||
if (err != cudaSuccess) {
|
if (err != cudaSuccess) {
|
||||||
GGML_LOG_ERROR("%s: failed to initialize " GGML_CUDA_NAME ": %s\n", __func__, cudaGetErrorString(err));
|
GGML_LOG_ERROR("%s: failed to initialize " GGML_CUDA_NAME ": %s\n", __func__, cudaGetErrorString(err));
|
||||||
return info;
|
return info;
|
||||||
}
|
}
|
||||||
|
|
||||||
GGML_ASSERT(info.device_count <= GGML_CUDA_MAX_DEVICES);
|
GGML_ASSERT(info.physical_device_count <= GGML_CUDA_MAX_DEVICES);
|
||||||
|
|
||||||
|
// by default expose exactly the physical devices; GGML_CUDA_DEVICES can request a different
|
||||||
|
// number of (virtual) devices to emulate multi-GPU systems on a machine with fewer GPUs
|
||||||
|
info.device_count = info.physical_device_count;
|
||||||
|
|
||||||
|
const char * devices_env = getenv("GGML_CUDA_DEVICES");
|
||||||
|
if (devices_env != nullptr && info.physical_device_count > 0) {
|
||||||
|
const int requested = atoi(devices_env);
|
||||||
|
if (requested > 0) {
|
||||||
|
info.device_count = requested;
|
||||||
|
} else {
|
||||||
|
GGML_LOG_WARN("%s: ignoring invalid GGML_CUDA_DEVICES=\"%s\"\n", __func__, devices_env);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (info.device_count > GGML_CUDA_MAX_DEVICES) {
|
||||||
|
GGML_LOG_WARN("%s: requested %d devices, clamping to GGML_CUDA_MAX_DEVICES=%d\n",
|
||||||
|
__func__, info.device_count, GGML_CUDA_MAX_DEVICES);
|
||||||
|
info.device_count = GGML_CUDA_MAX_DEVICES;
|
||||||
|
}
|
||||||
|
|
||||||
|
// map each (virtual) device to a backing physical device (round-robin), assign each its index
|
||||||
|
// among the (virtual) devices sharing that physical GPU, and store the per-physical share count
|
||||||
|
int physical_share_count[GGML_CUDA_MAX_DEVICES] = {};
|
||||||
|
GGML_ASSERT(info.device_count == 0 || info.physical_device_count > 0);
|
||||||
|
for (int id = 0; id < info.device_count; ++id) {
|
||||||
|
info.devices[id].physical_device = id % info.physical_device_count;
|
||||||
|
info.devices[id].virtual_index = physical_share_count[info.devices[id].physical_device]++;
|
||||||
|
}
|
||||||
|
|
||||||
int64_t total_vram = 0;
|
int64_t total_vram = 0;
|
||||||
for (int id = 0; id < info.device_count; ++id) {
|
for (int id = 0; id < info.physical_device_count; ++id) {
|
||||||
cudaDeviceProp prop;
|
cudaDeviceProp prop;
|
||||||
CUDA_CHECK(cudaGetDeviceProperties(&prop, id));
|
CUDA_CHECK(cudaGetDeviceProperties(&prop, id));
|
||||||
total_vram += prop.totalGlobalMem;
|
total_vram += prop.totalGlobalMem;
|
||||||
}
|
}
|
||||||
GGML_LOG_INFO("%s: found %d " GGML_CUDA_NAME " devices (Total VRAM: %zu MiB):\n",
|
GGML_LOG_INFO("%s: found %d " GGML_CUDA_NAME " devices (Total VRAM: %zu MiB):\n",
|
||||||
__func__, info.device_count, (size_t)(total_vram / (1024 * 1024)));
|
__func__, info.physical_device_count, (size_t)(total_vram / (1024 * 1024)));
|
||||||
|
if (info.device_count != info.physical_device_count) {
|
||||||
|
GGML_LOG_INFO("%s: emulating %d virtual device(s) on %d physical device(s) (GGML_CUDA_DEVICES)\n",
|
||||||
|
__func__, info.device_count, info.physical_device_count);
|
||||||
|
}
|
||||||
total_vram = 0;
|
total_vram = 0;
|
||||||
|
|
||||||
std::vector<std::pair<int, std::string>> turing_devices_without_mma;
|
std::vector<std::pair<int, std::string>> turing_devices_without_mma;
|
||||||
for (int id = 0; id < info.device_count; ++id) {
|
for (int id = 0; id < info.device_count; ++id) {
|
||||||
|
const int physical_id = info.devices[id].physical_device;
|
||||||
|
|
||||||
int device_vmm = 0;
|
int device_vmm = 0;
|
||||||
|
|
||||||
#if defined(GGML_USE_VMM)
|
#if defined(GGML_USE_VMM)
|
||||||
CUdevice device;
|
CUdevice device;
|
||||||
CU_CHECK(cuDeviceGet(&device, id));
|
CU_CHECK(cuDeviceGet(&device, physical_id));
|
||||||
CU_CHECK(cuDeviceGetAttribute(&device_vmm, CU_DEVICE_ATTRIBUTE_VIRTUAL_MEMORY_MANAGEMENT_SUPPORTED, device));
|
CU_CHECK(cuDeviceGetAttribute(&device_vmm, CU_DEVICE_ATTRIBUTE_VIRTUAL_MEMORY_MANAGEMENT_SUPPORTED, device));
|
||||||
|
|
||||||
if (device_vmm) {
|
if (device_vmm) {
|
||||||
CUmemAllocationProp alloc_prop = {};
|
CUmemAllocationProp alloc_prop = {};
|
||||||
alloc_prop.type = CU_MEM_ALLOCATION_TYPE_PINNED;
|
alloc_prop.type = CU_MEM_ALLOCATION_TYPE_PINNED;
|
||||||
alloc_prop.location.type = CU_MEM_LOCATION_TYPE_DEVICE;
|
alloc_prop.location.type = CU_MEM_LOCATION_TYPE_DEVICE;
|
||||||
alloc_prop.location.id = id;
|
alloc_prop.location.id = physical_id;
|
||||||
CU_CHECK(cuMemGetAllocationGranularity(&info.devices[id].vmm_granularity, &alloc_prop, CU_MEM_ALLOC_GRANULARITY_RECOMMENDED));
|
CU_CHECK(cuMemGetAllocationGranularity(&info.devices[id].vmm_granularity, &alloc_prop, CU_MEM_ALLOC_GRANULARITY_RECOMMENDED));
|
||||||
}
|
}
|
||||||
#endif // defined(GGML_USE_VMM)
|
#endif // defined(GGML_USE_VMM)
|
||||||
info.devices[id].vmm = !!device_vmm;
|
info.devices[id].vmm = !!device_vmm;
|
||||||
|
|
||||||
cudaDeviceProp prop;
|
cudaDeviceProp prop;
|
||||||
CUDA_CHECK(cudaGetDeviceProperties(&prop, id));
|
CUDA_CHECK(cudaGetDeviceProperties(&prop, physical_id));
|
||||||
|
|
||||||
|
// a virtual device owns only a share of its physical GPU's memory; report that share so the
|
||||||
|
// logged per-device VRAM sums to the physical total above.
|
||||||
|
GGML_ASSERT(physical_share_count[physical_id] > 0);
|
||||||
|
info.devices[id].physical_share_count = physical_share_count[physical_id];
|
||||||
|
const size_t device_vram = prop.totalGlobalMem / info.devices[id].physical_share_count;
|
||||||
|
const size_t device_vram_mib = device_vram / (1024 * 1024);
|
||||||
|
|
||||||
info.default_tensor_split[id] = total_vram;
|
info.default_tensor_split[id] = total_vram;
|
||||||
total_vram += prop.totalGlobalMem;
|
total_vram += device_vram;
|
||||||
|
#if defined(GGML_USE_HIP)
|
||||||
|
info.devices[id].integrated = prop.integrated;
|
||||||
|
#else
|
||||||
info.devices[id].integrated = false; // Temporarily disabled due to issues with corrupted output (e.g. #15034)
|
info.devices[id].integrated = false; // Temporarily disabled due to issues with corrupted output (e.g. #15034)
|
||||||
|
#endif
|
||||||
info.devices[id].nsm = prop.multiProcessorCount;
|
info.devices[id].nsm = prop.multiProcessorCount;
|
||||||
info.devices[id].smpb = prop.sharedMemPerBlock;
|
info.devices[id].smpb = prop.sharedMemPerBlock;
|
||||||
info.devices[id].warp_size = prop.warpSize;
|
info.devices[id].warp_size = prop.warpSize;
|
||||||
|
|
||||||
#ifndef GGML_USE_MUSA
|
#ifndef GGML_USE_MUSA
|
||||||
int supports_coop_launch = 0;
|
int supports_coop_launch = 0;
|
||||||
CUDA_CHECK(cudaDeviceGetAttribute(&supports_coop_launch, cudaDevAttrCooperativeLaunch, id));
|
CUDA_CHECK(cudaDeviceGetAttribute(&supports_coop_launch, cudaDevAttrCooperativeLaunch, physical_id));
|
||||||
info.devices[id].supports_cooperative_launch = !!supports_coop_launch;
|
info.devices[id].supports_cooperative_launch = !!supports_coop_launch;
|
||||||
#else
|
#else
|
||||||
info.devices[id].supports_cooperative_launch = false;
|
info.devices[id].supports_cooperative_launch = false;
|
||||||
@@ -277,7 +335,7 @@ static ggml_cuda_device_info ggml_cuda_init() {
|
|||||||
GGML_LOG_INFO(" Device %d: %s, %s (0x%x), VMM: %s, Wave Size: %d, VRAM: %zu MiB\n",
|
GGML_LOG_INFO(" Device %d: %s, %s (0x%x), VMM: %s, Wave Size: %d, VRAM: %zu MiB\n",
|
||||||
id, prop.name, prop.gcnArchName, info.devices[id].cc & 0xffff,
|
id, prop.name, prop.gcnArchName, info.devices[id].cc & 0xffff,
|
||||||
device_vmm ? "yes" : "no", prop.warpSize,
|
device_vmm ? "yes" : "no", prop.warpSize,
|
||||||
(size_t)(prop.totalGlobalMem / (1024 * 1024)));
|
device_vram_mib);
|
||||||
#elif defined(GGML_USE_MUSA)
|
#elif defined(GGML_USE_MUSA)
|
||||||
// FIXME: Ensure compatibility with varying warp sizes across different MUSA archs.
|
// FIXME: Ensure compatibility with varying warp sizes across different MUSA archs.
|
||||||
info.devices[id].warp_size = 32;
|
info.devices[id].warp_size = 32;
|
||||||
@@ -286,13 +344,13 @@ static ggml_cuda_device_info ggml_cuda_init() {
|
|||||||
info.devices[id].cc += prop.minor * 0x10;
|
info.devices[id].cc += prop.minor * 0x10;
|
||||||
GGML_LOG_INFO(" Device %d: %s, compute capability %d.%d, VMM: %s, VRAM: %zu MiB\n",
|
GGML_LOG_INFO(" Device %d: %s, compute capability %d.%d, VMM: %s, VRAM: %zu MiB\n",
|
||||||
id, prop.name, prop.major, prop.minor, device_vmm ? "yes" : "no",
|
id, prop.name, prop.major, prop.minor, device_vmm ? "yes" : "no",
|
||||||
(size_t)(prop.totalGlobalMem / (1024 * 1024)));
|
device_vram_mib);
|
||||||
#else
|
#else
|
||||||
info.devices[id].smpbo = prop.sharedMemPerBlockOptin;
|
info.devices[id].smpbo = prop.sharedMemPerBlockOptin;
|
||||||
info.devices[id].cc = 100*prop.major + 10*prop.minor;
|
info.devices[id].cc = 100*prop.major + 10*prop.minor;
|
||||||
GGML_LOG_INFO(" Device %d: %s, compute capability %d.%d, VMM: %s, VRAM: %zu MiB\n",
|
GGML_LOG_INFO(" Device %d: %s, compute capability %d.%d, VMM: %s, VRAM: %zu MiB\n",
|
||||||
id, prop.name, prop.major, prop.minor, device_vmm ? "yes" : "no",
|
id, prop.name, prop.major, prop.minor, device_vmm ? "yes" : "no",
|
||||||
(size_t)(prop.totalGlobalMem / (1024 * 1024)));
|
device_vram_mib);
|
||||||
std::string device_name(prop.name);
|
std::string device_name(prop.name);
|
||||||
if (device_name == "NVIDIA GeForce MX450") {
|
if (device_name == "NVIDIA GeForce MX450") {
|
||||||
turing_devices_without_mma.push_back({ id, device_name });
|
turing_devices_without_mma.push_back({ id, device_name });
|
||||||
@@ -307,7 +365,7 @@ static ggml_cuda_device_info ggml_cuda_init() {
|
|||||||
// TODO: Check for future drivers the default scheduling strategy and
|
// TODO: Check for future drivers the default scheduling strategy and
|
||||||
// remove this call again when cudaDeviceScheduleSpin is default.
|
// remove this call again when cudaDeviceScheduleSpin is default.
|
||||||
if (prop.major == 12 && prop.minor == 1) {
|
if (prop.major == 12 && prop.minor == 1) {
|
||||||
CUDA_CHECK(cudaSetDevice(id));
|
CUDA_CHECK(cudaSetDevice(physical_id));
|
||||||
CUDA_CHECK(cudaSetDeviceFlags(cudaDeviceScheduleSpin));
|
CUDA_CHECK(cudaSetDeviceFlags(cudaDeviceScheduleSpin));
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -332,9 +390,9 @@ static ggml_cuda_device_info ggml_cuda_init() {
|
|||||||
// CUBLAS_CHECK(cublasLoggerConfigure(1, 1, 0, nullptr));
|
// CUBLAS_CHECK(cublasLoggerConfigure(1, 1, 0, nullptr));
|
||||||
|
|
||||||
if (getenv("GGML_CUDA_P2P") != nullptr) {
|
if (getenv("GGML_CUDA_P2P") != nullptr) {
|
||||||
for (int id = 0; id < info.device_count; ++id) {
|
for (int id = 0; id < info.physical_device_count; ++id) {
|
||||||
ggml_cuda_set_device(id);
|
CUDA_CHECK(cudaSetDevice(id));
|
||||||
for (int id_other = 0; id_other < info.device_count; ++id_other) {
|
for (int id_other = 0; id_other < info.physical_device_count; ++id_other) {
|
||||||
if (id == id_other) {
|
if (id == id_other) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
@@ -479,6 +537,7 @@ struct ggml_cuda_pool_vmm : public ggml_cuda_pool {
|
|||||||
static const size_t CUDA_POOL_VMM_MAX_SIZE = 1ull << 35; // 32 GB
|
static const size_t CUDA_POOL_VMM_MAX_SIZE = 1ull << 35; // 32 GB
|
||||||
|
|
||||||
int device;
|
int device;
|
||||||
|
int physical_device;
|
||||||
CUdeviceptr pool_addr = 0;
|
CUdeviceptr pool_addr = 0;
|
||||||
size_t pool_used = 0;
|
size_t pool_used = 0;
|
||||||
size_t pool_size = 0;
|
size_t pool_size = 0;
|
||||||
@@ -489,6 +548,7 @@ struct ggml_cuda_pool_vmm : public ggml_cuda_pool {
|
|||||||
|
|
||||||
explicit ggml_cuda_pool_vmm(int device) :
|
explicit ggml_cuda_pool_vmm(int device) :
|
||||||
device(device),
|
device(device),
|
||||||
|
physical_device(ggml_cuda_get_physical_device(device)),
|
||||||
granularity(ggml_cuda_info().devices[device].vmm_granularity) {
|
granularity(ggml_cuda_info().devices[device].vmm_granularity) {
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -524,7 +584,7 @@ struct ggml_cuda_pool_vmm : public ggml_cuda_pool {
|
|||||||
CUmemAllocationProp prop = {};
|
CUmemAllocationProp prop = {};
|
||||||
prop.type = CU_MEM_ALLOCATION_TYPE_PINNED;
|
prop.type = CU_MEM_ALLOCATION_TYPE_PINNED;
|
||||||
prop.location.type = CU_MEM_LOCATION_TYPE_DEVICE;
|
prop.location.type = CU_MEM_LOCATION_TYPE_DEVICE;
|
||||||
prop.location.id = device;
|
prop.location.id = physical_device;
|
||||||
CUmemGenericAllocationHandle handle;
|
CUmemGenericAllocationHandle handle;
|
||||||
CU_CHECK(cuMemCreate(&handle, reserve_size, &prop, 0));
|
CU_CHECK(cuMemCreate(&handle, reserve_size, &prop, 0));
|
||||||
|
|
||||||
@@ -553,20 +613,28 @@ struct ggml_cuda_pool_vmm : public ggml_cuda_pool {
|
|||||||
// NCCL implicitly enables peer access (cudaDeviceEnablePeerAccess), and
|
// NCCL implicitly enables peer access (cudaDeviceEnablePeerAccess), and
|
||||||
// GGML_CUDA_P2P enables it explicitly. Unlike cudaMalloc buffers, VMM
|
// GGML_CUDA_P2P enables it explicitly. Unlike cudaMalloc buffers, VMM
|
||||||
// allocations do not become peer-accessible from that alone, so access
|
// allocations do not become peer-accessible from that alone, so access
|
||||||
// must be granted explicitly here.
|
// must be granted explicitly here. With virtual devices, grant access
|
||||||
|
// on the backing *physical* devices (deduplicated, since several
|
||||||
|
// virtual devices can map to the same physical GPU).
|
||||||
std::vector<CUmemAccessDesc> access_descs;
|
std::vector<CUmemAccessDesc> access_descs;
|
||||||
|
bool physical_seen[GGML_CUDA_MAX_DEVICES] = {};
|
||||||
const int device_count = ggml_cuda_info().device_count;
|
const int device_count = ggml_cuda_info().device_count;
|
||||||
for (int id = 0; id < device_count; ++id) {
|
for (int id = 0; id < device_count; ++id) {
|
||||||
if (id != device) {
|
const int id_physical = ggml_cuda_get_physical_device(id);
|
||||||
|
if (id_physical != physical_device) {
|
||||||
int can_access_peer = 0;
|
int can_access_peer = 0;
|
||||||
CUDA_CHECK(cudaDeviceCanAccessPeer(&can_access_peer, id, device));
|
CUDA_CHECK(cudaDeviceCanAccessPeer(&can_access_peer, id_physical, physical_device));
|
||||||
if (!can_access_peer) {
|
if (!can_access_peer) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if (physical_seen[id_physical]) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
physical_seen[id_physical] = true;
|
||||||
CUmemAccessDesc access = {};
|
CUmemAccessDesc access = {};
|
||||||
access.location.type = CU_MEM_LOCATION_TYPE_DEVICE;
|
access.location.type = CU_MEM_LOCATION_TYPE_DEVICE;
|
||||||
access.location.id = id;
|
access.location.id = id_physical;
|
||||||
access.flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE;
|
access.flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE;
|
||||||
access_descs.push_back(access);
|
access_descs.push_back(access);
|
||||||
}
|
}
|
||||||
@@ -575,7 +643,7 @@ struct ggml_cuda_pool_vmm : public ggml_cuda_pool {
|
|||||||
// set access for non P2P
|
// set access for non P2P
|
||||||
CUmemAccessDesc access = {};
|
CUmemAccessDesc access = {};
|
||||||
access.location.type = CU_MEM_LOCATION_TYPE_DEVICE;
|
access.location.type = CU_MEM_LOCATION_TYPE_DEVICE;
|
||||||
access.location.id = device;
|
access.location.id = physical_device;
|
||||||
access.flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE;
|
access.flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE;
|
||||||
CU_CHECK(cuMemSetAccess(start_ptr, reserve_size, &access, 1));
|
CU_CHECK(cuMemSetAccess(start_ptr, reserve_size, &access, 1));
|
||||||
}
|
}
|
||||||
@@ -751,13 +819,17 @@ static bool ggml_backend_cuda_buffer_cpy_tensor(ggml_backend_buffer_t buffer, co
|
|||||||
if (ggml_backend_buffer_is_cuda(src->buffer)) {
|
if (ggml_backend_buffer_is_cuda(src->buffer)) {
|
||||||
ggml_backend_cuda_buffer_context * src_ctx = (ggml_backend_cuda_buffer_context *)src->buffer->context;
|
ggml_backend_cuda_buffer_context * src_ctx = (ggml_backend_cuda_buffer_context *)src->buffer->context;
|
||||||
ggml_backend_cuda_buffer_context * dst_ctx = (ggml_backend_cuda_buffer_context *)dst->buffer->context;
|
ggml_backend_cuda_buffer_context * dst_ctx = (ggml_backend_cuda_buffer_context *)dst->buffer->context;
|
||||||
if (src_ctx->device == dst_ctx->device) {
|
// compare the backing physical devices: distinct virtual devices may share one physical GPU,
|
||||||
|
// in which case a same-device copy (not a peer copy) is required
|
||||||
|
const int src_physical = ggml_cuda_get_physical_device(src_ctx->device);
|
||||||
|
const int dst_physical = ggml_cuda_get_physical_device(dst_ctx->device);
|
||||||
|
if (src_physical == dst_physical) {
|
||||||
CUDA_CHECK(cudaMemcpyAsync(dst->data, src->data, ggml_nbytes(src), cudaMemcpyDeviceToDevice, cudaStreamPerThread));
|
CUDA_CHECK(cudaMemcpyAsync(dst->data, src->data, ggml_nbytes(src), cudaMemcpyDeviceToDevice, cudaStreamPerThread));
|
||||||
} else {
|
} else {
|
||||||
#ifdef GGML_CUDA_NO_PEER_COPY
|
#ifdef GGML_CUDA_NO_PEER_COPY
|
||||||
return false;
|
return false;
|
||||||
#else
|
#else
|
||||||
CUDA_CHECK(cudaMemcpyPeerAsync(dst->data, dst_ctx->device, src->data, src_ctx->device, ggml_nbytes(src), cudaStreamPerThread));
|
CUDA_CHECK(cudaMemcpyPeerAsync(dst->data, dst_physical, src->data, src_physical, ggml_nbytes(src), cudaStreamPerThread));
|
||||||
#endif
|
#endif
|
||||||
}
|
}
|
||||||
CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));
|
CUDA_CHECK(cudaStreamSynchronize(cudaStreamPerThread));
|
||||||
@@ -1099,6 +1171,15 @@ static void ggml_backend_cuda_comm_init_internal(ggml_backend_cuda_comm_context
|
|||||||
|
|
||||||
static void ggml_backend_cuda_comm_init_nccl(ggml_backend_cuda_comm_context * ret) {
|
static void ggml_backend_cuda_comm_init_nccl(ggml_backend_cuda_comm_context * ret) {
|
||||||
#ifdef GGML_USE_NCCL
|
#ifdef GGML_USE_NCCL
|
||||||
|
// Disabling NCCL path when CUDA virtual devices are in use since NCCL requires one distinct physical GPU per rank.
|
||||||
|
const ggml_cuda_device_info & info = ggml_cuda_info();
|
||||||
|
if (info.device_count > info.physical_device_count) {
|
||||||
|
GGML_LOG_WARN("NCCL disabled: virtual devices in use; "
|
||||||
|
"falling back to internal AllReduce\n");
|
||||||
|
ggml_backend_cuda_comm_init_internal(ret);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
const size_t n = ret->dev_ids.size();
|
const size_t n = ret->dev_ids.size();
|
||||||
ret->comms.resize(n);
|
ret->comms.resize(n);
|
||||||
ncclResult_t rc = ncclCommInitAll(ret->comms.data(), (int) n, ret->dev_ids.data());
|
ncclResult_t rc = ncclCommInitAll(ret->comms.data(), (int) n, ret->dev_ids.data());
|
||||||
@@ -2239,6 +2320,15 @@ static bool ggml_cuda_compute_forward(ggml_backend_cuda_context & ctx, struct gg
|
|||||||
case GGML_OP_GATED_DELTA_NET:
|
case GGML_OP_GATED_DELTA_NET:
|
||||||
ggml_cuda_op_gated_delta_net(ctx, dst);
|
ggml_cuda_op_gated_delta_net(ctx, dst);
|
||||||
break;
|
break;
|
||||||
|
case GGML_OP_DSV4_HC_COMB:
|
||||||
|
ggml_cuda_op_dsv4_hc_comb(ctx, dst);
|
||||||
|
break;
|
||||||
|
case GGML_OP_DSV4_HC_PRE:
|
||||||
|
ggml_cuda_op_dsv4_hc_pre(ctx, dst);
|
||||||
|
break;
|
||||||
|
case GGML_OP_DSV4_HC_POST:
|
||||||
|
ggml_cuda_op_dsv4_hc_post(ctx, dst);
|
||||||
|
break;
|
||||||
case GGML_OP_RWKV_WKV7:
|
case GGML_OP_RWKV_WKV7:
|
||||||
ggml_cuda_op_rwkv_wkv7(ctx, dst);
|
ggml_cuda_op_rwkv_wkv7(ctx, dst);
|
||||||
break;
|
break;
|
||||||
@@ -2257,6 +2347,9 @@ static bool ggml_cuda_compute_forward(ggml_backend_cuda_context & ctx, struct gg
|
|||||||
case GGML_OP_FILL:
|
case GGML_OP_FILL:
|
||||||
ggml_cuda_op_fill(ctx, dst);
|
ggml_cuda_op_fill(ctx, dst);
|
||||||
break;
|
break;
|
||||||
|
case GGML_OP_LIGHTNING_INDEXER:
|
||||||
|
ggml_cuda_lightning_indexer(ctx, dst);
|
||||||
|
break;
|
||||||
default:
|
default:
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
@@ -2355,13 +2448,17 @@ static bool ggml_backend_cuda_cpy_tensor_async(ggml_backend_t backend_src, ggml_
|
|||||||
|
|
||||||
if (backend_src != backend_dst) {
|
if (backend_src != backend_dst) {
|
||||||
// copy on src stream
|
// copy on src stream
|
||||||
if (cuda_ctx_src->device == cuda_ctx_dst->device) {
|
// compare the backing physical devices: distinct virtual devices may share one physical GPU,
|
||||||
|
// in which case a same-device copy (not a peer copy) is required
|
||||||
|
const int src_physical = ggml_cuda_get_physical_device(cuda_ctx_src->device);
|
||||||
|
const int dst_physical = ggml_cuda_get_physical_device(cuda_ctx_dst->device);
|
||||||
|
if (src_physical == dst_physical) {
|
||||||
CUDA_CHECK(cudaMemcpyAsync(dst->data, src->data, ggml_nbytes(dst), cudaMemcpyDeviceToDevice, cuda_ctx_src->stream()));
|
CUDA_CHECK(cudaMemcpyAsync(dst->data, src->data, ggml_nbytes(dst), cudaMemcpyDeviceToDevice, cuda_ctx_src->stream()));
|
||||||
} else {
|
} else {
|
||||||
#ifdef GGML_CUDA_NO_PEER_COPY
|
#ifdef GGML_CUDA_NO_PEER_COPY
|
||||||
return false;
|
return false;
|
||||||
#else
|
#else
|
||||||
CUDA_CHECK(cudaMemcpyPeerAsync(dst->data, cuda_ctx_dst->device, src->data, cuda_ctx_src->device, ggml_nbytes(dst), cuda_ctx_src->stream()));
|
CUDA_CHECK(cudaMemcpyPeerAsync(dst->data, dst_physical, src->data, src_physical, ggml_nbytes(dst), cuda_ctx_src->stream()));
|
||||||
#endif // GGML_CUDA_NO_PEER_COPY
|
#endif // GGML_CUDA_NO_PEER_COPY
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -2606,6 +2703,7 @@ static int ggml_cuda_try_gdn_cache_fusion(
|
|||||||
|
|
||||||
static bool ggml_cuda_topk_moe_fusion(const struct ggml_cgraph * cgraph, int node_idx, ggml_cuda_topk_moe_args & args) {
|
static bool ggml_cuda_topk_moe_fusion(const struct ggml_cgraph * cgraph, int node_idx, ggml_cuda_topk_moe_args & args) {
|
||||||
args.sigmoid = false;
|
args.sigmoid = false;
|
||||||
|
args.sqrt_softplus = false;
|
||||||
args.softmax = false;
|
args.softmax = false;
|
||||||
args.delayed_softmax = false;
|
args.delayed_softmax = false;
|
||||||
args.prob_bias = false;
|
args.prob_bias = false;
|
||||||
@@ -2619,10 +2717,17 @@ static bool ggml_cuda_topk_moe_fusion(const struct ggml_cgraph * cgraph, int nod
|
|||||||
}
|
}
|
||||||
|
|
||||||
if (nodes[node_idx]->op == GGML_OP_UNARY) {
|
if (nodes[node_idx]->op == GGML_OP_UNARY) {
|
||||||
if (ggml_get_unary_op(nodes[node_idx]) != GGML_UNARY_OP_SIGMOID) {
|
const ggml_unary_op unary_op = ggml_get_unary_op(nodes[node_idx]);
|
||||||
|
if (unary_op == GGML_UNARY_OP_SIGMOID) {
|
||||||
|
args.sigmoid = true;
|
||||||
|
} else if (unary_op == GGML_UNARY_OP_SOFTPLUS && node_idx + 1 < n_nodes &&
|
||||||
|
nodes[node_idx + 1]->op == GGML_OP_SQRT && nodes[node_idx + 1]->src[0] == nodes[node_idx]) {
|
||||||
|
// sqrt(softplus(x)) scoring (DeepSeek-V4)
|
||||||
|
args.sqrt_softplus = true;
|
||||||
|
node_idx++;
|
||||||
|
} else {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
args.sigmoid = true;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if (nodes[node_idx]->op == GGML_OP_ARGSORT) {
|
if (nodes[node_idx]->op == GGML_OP_ARGSORT) {
|
||||||
@@ -2631,7 +2736,7 @@ static bool ggml_cuda_topk_moe_fusion(const struct ggml_cgraph * cgraph, int nod
|
|||||||
|
|
||||||
node_idx++;
|
node_idx++;
|
||||||
|
|
||||||
if (args.sigmoid || args.softmax) {
|
if (args.sigmoid || args.sqrt_softplus || args.softmax) {
|
||||||
// SOFTMAX -> RESHAPE
|
// SOFTMAX -> RESHAPE
|
||||||
if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_RESHAPE ||
|
if (node_idx >= n_nodes || nodes[node_idx]->op != GGML_OP_RESHAPE ||
|
||||||
nodes[node_idx]->src[0] != nodes[node_idx - 1]) {
|
nodes[node_idx]->src[0] != nodes[node_idx - 1]) {
|
||||||
@@ -3075,21 +3180,27 @@ static int ggml_cuda_try_fuse(ggml_backend_cuda_context * cuda_ctx, ggml_cgraph
|
|||||||
const ggml_tensor * scale = nullptr;
|
const ggml_tensor * scale = nullptr;
|
||||||
|
|
||||||
if (!args.delayed_softmax) {
|
if (!args.delayed_softmax) {
|
||||||
ggml_op gating_op = args.sigmoid ? GGML_OP_UNARY : GGML_OP_SOFT_MAX;
|
int out_nodes[2]; // nodes which can't be elided
|
||||||
int out_nodes[2]; // nodes which can't be elided
|
|
||||||
|
if (args.sigmoid) {
|
||||||
|
ops.insert(ops.end(), { GGML_OP_UNARY });
|
||||||
|
} else if (args.sqrt_softplus) {
|
||||||
|
ops.insert(ops.end(), { GGML_OP_UNARY, GGML_OP_SQRT });
|
||||||
|
} else {
|
||||||
|
ops.insert(ops.end(), { GGML_OP_SOFT_MAX });
|
||||||
|
}
|
||||||
|
const int i_probs = i + (int) ops.size() - 1; // last node of the gating activation
|
||||||
|
|
||||||
if (args.prob_bias) {
|
if (args.prob_bias) {
|
||||||
bias = cgraph->nodes[i + 2]->src[1];
|
bias = cgraph->nodes[i_probs + 2]->src[1];
|
||||||
ops.insert(ops.end(), { gating_op, GGML_OP_RESHAPE, GGML_OP_ADD, GGML_OP_ARGSORT, GGML_OP_VIEW,
|
ops.insert(ops.end(), { GGML_OP_RESHAPE, GGML_OP_ADD, GGML_OP_ARGSORT, GGML_OP_VIEW,
|
||||||
GGML_OP_GET_ROWS });
|
GGML_OP_GET_ROWS });
|
||||||
out_nodes[0] = i + 4;
|
out_nodes[0] = i_probs + 4;
|
||||||
ids = cgraph->nodes[i + 4];
|
|
||||||
} else {
|
} else {
|
||||||
ops.insert(ops.end(),
|
ops.insert(ops.end(), { GGML_OP_RESHAPE, GGML_OP_ARGSORT, GGML_OP_VIEW, GGML_OP_GET_ROWS });
|
||||||
{ gating_op, GGML_OP_RESHAPE, GGML_OP_ARGSORT, GGML_OP_VIEW, GGML_OP_GET_ROWS });
|
out_nodes[0] = i_probs + 3;
|
||||||
out_nodes[0] = i + 3;
|
|
||||||
ids = cgraph->nodes[i + 3];
|
|
||||||
}
|
}
|
||||||
|
ids = cgraph->nodes[out_nodes[0]];
|
||||||
|
|
||||||
if (args.norm) {
|
if (args.norm) {
|
||||||
ops.insert(ops.end(),
|
ops.insert(ops.end(),
|
||||||
@@ -3974,7 +4085,7 @@ static bool ggml_cuda_graph_set_enabled(ggml_backend_cuda_context * cuda_ctx, co
|
|||||||
ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key);
|
ggml_cuda_graph * graph = cuda_ctx->cuda_graph(graph_key);
|
||||||
|
|
||||||
if (graph->graph == nullptr) {
|
if (graph->graph == nullptr) {
|
||||||
if (ggml_cuda_info().devices[cuda_ctx->device].cc < GGML_CUDA_CC_AMPERE) {
|
if (ggml_cuda_info().devices[cuda_ctx->device].cc < GGML_CUDA_CC_VOLTA) {
|
||||||
if (!graph->disable_due_to_gpu_arch) {
|
if (!graph->disable_due_to_gpu_arch) {
|
||||||
GGML_LOG_DEBUG("%s: disabling CUDA graphs due to GPU architecture\n", __func__);
|
GGML_LOG_DEBUG("%s: disabling CUDA graphs due to GPU architecture\n", __func__);
|
||||||
}
|
}
|
||||||
@@ -4346,16 +4457,38 @@ int ggml_backend_cuda_get_device_count() {
|
|||||||
return ggml_cuda_info().device_count;
|
return ggml_cuda_info().device_count;
|
||||||
}
|
}
|
||||||
|
|
||||||
void ggml_backend_cuda_get_device_description(int device, char * description, size_t description_size) {
|
static std::string ggml_cuda_device_description(int device) {
|
||||||
cudaDeviceProp prop;
|
cudaDeviceProp prop;
|
||||||
CUDA_CHECK(cudaGetDeviceProperties(&prop, device));
|
CUDA_CHECK(cudaGetDeviceProperties(&prop, ggml_cuda_get_physical_device(device)));
|
||||||
snprintf(description, description_size, "%s", prop.name);
|
|
||||||
|
const ggml_cuda_device_info & info = ggml_cuda_info();
|
||||||
|
std::string description = prop.name;
|
||||||
|
if (info.device_count > info.physical_device_count) {
|
||||||
|
description += " (physical device " + std::to_string(info.devices[device].physical_device) +
|
||||||
|
", virtual device " + std::to_string(info.devices[device].virtual_index) + ")";
|
||||||
|
}
|
||||||
|
return description;
|
||||||
|
}
|
||||||
|
|
||||||
|
void ggml_backend_cuda_get_device_description(int device, char * description, size_t description_size) {
|
||||||
|
snprintf(description, description_size, "%s", ggml_cuda_device_description(device).c_str());
|
||||||
|
}
|
||||||
|
|
||||||
|
static int ggml_cuda_physical_device_share_count(int device) {
|
||||||
|
const ggml_cuda_device_info & info = ggml_cuda_info();
|
||||||
|
GGML_ASSERT(device >= 0 && device < info.device_count);
|
||||||
|
return info.devices[device].physical_share_count;
|
||||||
}
|
}
|
||||||
|
|
||||||
void ggml_backend_cuda_get_device_memory(int device, size_t * free, size_t * total) {
|
void ggml_backend_cuda_get_device_memory(int device, size_t * free, size_t * total) {
|
||||||
ggml_cuda_set_device(device);
|
ggml_cuda_set_device(device);
|
||||||
|
|
||||||
CUDA_CHECK(cudaMemGetInfo(free, total));
|
CUDA_CHECK(cudaMemGetInfo(free, total));
|
||||||
|
|
||||||
|
// virtual devices sharing one physical GPU share its memory pool; split it between them
|
||||||
|
const int share_count = ggml_cuda_physical_device_share_count(device);
|
||||||
|
*free /= share_count;
|
||||||
|
*total /= share_count;
|
||||||
}
|
}
|
||||||
|
|
||||||
bool ggml_backend_cuda_register_host_buffer(void * buffer, size_t size) {
|
bool ggml_backend_cuda_register_host_buffer(void * buffer, size_t size) {
|
||||||
@@ -4493,13 +4626,20 @@ static bool ggml_backend_cuda_get_available_uma_memory(long * available_memory_k
|
|||||||
static void ggml_backend_cuda_device_get_memory(ggml_backend_dev_t dev, size_t * free, size_t * total) {
|
static void ggml_backend_cuda_device_get_memory(ggml_backend_dev_t dev, size_t * free, size_t * total) {
|
||||||
ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context;
|
ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *)dev->context;
|
||||||
ggml_cuda_set_device(ctx->device);
|
ggml_cuda_set_device(ctx->device);
|
||||||
CUDA_CHECK(cudaMemGetInfo(free, total));
|
cudaError_t err = cudaMemGetInfo(free, total);
|
||||||
|
if (err != cudaSuccess) {
|
||||||
|
(void)cudaGetLastError();
|
||||||
|
GGML_LOG_WARN("%s: cudaMemGetInfo failed (%s), returning 0/0\n", __func__, cudaGetErrorString(err));
|
||||||
|
*free = 0;
|
||||||
|
*total = 0;
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
// ref: https://github.com/ggml-org/llama.cpp/pull/17368
|
// ref: https://github.com/ggml-org/llama.cpp/pull/17368
|
||||||
#if defined(__linux__)
|
#if defined(__linux__)
|
||||||
// Check if this is a UMA (Unified Memory Architecture) system
|
// Check if this is a UMA (Unified Memory Architecture) system
|
||||||
cudaDeviceProp prop;
|
cudaDeviceProp prop;
|
||||||
CUDA_CHECK(cudaGetDeviceProperties(&prop, ctx->device));
|
CUDA_CHECK(cudaGetDeviceProperties(&prop, ggml_cuda_get_physical_device(ctx->device)));
|
||||||
|
|
||||||
// Check if UMA is explicitly enabled via environment variable
|
// Check if UMA is explicitly enabled via environment variable
|
||||||
bool uma_env = getenv("GGML_CUDA_ENABLE_UNIFIED_MEMORY") != nullptr;
|
bool uma_env = getenv("GGML_CUDA_ENABLE_UNIFIED_MEMORY") != nullptr;
|
||||||
@@ -4518,13 +4658,17 @@ static void ggml_backend_cuda_device_get_memory(ggml_backend_dev_t dev, size_t *
|
|||||||
}
|
}
|
||||||
#endif // defined(__linux__)
|
#endif // defined(__linux__)
|
||||||
|
|
||||||
|
// virtual devices sharing one physical GPU share its memory pool; split it between them
|
||||||
|
const int share_count = ggml_cuda_physical_device_share_count(ctx->device);
|
||||||
|
*free /= share_count;
|
||||||
|
*total /= share_count;
|
||||||
}
|
}
|
||||||
|
|
||||||
static enum ggml_backend_dev_type ggml_backend_cuda_device_get_type(ggml_backend_dev_t dev) {
|
static enum ggml_backend_dev_type ggml_backend_cuda_device_get_type(ggml_backend_dev_t dev) {
|
||||||
ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *) dev->context;
|
ggml_backend_cuda_device_context * ctx = (ggml_backend_cuda_device_context *) dev->context;
|
||||||
|
|
||||||
cudaDeviceProp prop;
|
cudaDeviceProp prop;
|
||||||
CUDA_CHECK(cudaGetDeviceProperties(&prop, ctx->device));
|
CUDA_CHECK(cudaGetDeviceProperties(&prop, ggml_cuda_get_physical_device(ctx->device)));
|
||||||
|
|
||||||
return prop.integrated
|
return prop.integrated
|
||||||
? GGML_BACKEND_DEVICE_TYPE_IGPU
|
? GGML_BACKEND_DEVICE_TYPE_IGPU
|
||||||
@@ -4658,6 +4802,7 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
|
|||||||
case GGML_TYPE_F32:
|
case GGML_TYPE_F32:
|
||||||
case GGML_TYPE_F16:
|
case GGML_TYPE_F16:
|
||||||
case GGML_TYPE_Q1_0:
|
case GGML_TYPE_Q1_0:
|
||||||
|
case GGML_TYPE_Q2_0:
|
||||||
case GGML_TYPE_Q4_0:
|
case GGML_TYPE_Q4_0:
|
||||||
case GGML_TYPE_Q4_1:
|
case GGML_TYPE_Q4_1:
|
||||||
case GGML_TYPE_Q5_0:
|
case GGML_TYPE_Q5_0:
|
||||||
@@ -4696,12 +4841,31 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
|
|||||||
case GGML_TYPE_BF16:
|
case GGML_TYPE_BF16:
|
||||||
case GGML_TYPE_I32:
|
case GGML_TYPE_I32:
|
||||||
case GGML_TYPE_Q1_0:
|
case GGML_TYPE_Q1_0:
|
||||||
|
case GGML_TYPE_Q2_0:
|
||||||
case GGML_TYPE_Q4_0:
|
case GGML_TYPE_Q4_0:
|
||||||
case GGML_TYPE_Q4_1:
|
case GGML_TYPE_Q4_1:
|
||||||
case GGML_TYPE_Q5_0:
|
case GGML_TYPE_Q5_0:
|
||||||
case GGML_TYPE_Q5_1:
|
case GGML_TYPE_Q5_1:
|
||||||
case GGML_TYPE_Q8_0:
|
case GGML_TYPE_Q8_0:
|
||||||
|
case GGML_TYPE_Q2_K:
|
||||||
|
case GGML_TYPE_Q3_K:
|
||||||
|
case GGML_TYPE_Q4_K:
|
||||||
|
case GGML_TYPE_Q5_K:
|
||||||
|
case GGML_TYPE_Q6_K:
|
||||||
|
case GGML_TYPE_IQ2_XXS:
|
||||||
|
case GGML_TYPE_IQ2_XS:
|
||||||
|
case GGML_TYPE_IQ2_S:
|
||||||
|
case GGML_TYPE_IQ3_XXS:
|
||||||
|
case GGML_TYPE_IQ3_S:
|
||||||
|
case GGML_TYPE_IQ1_S:
|
||||||
|
case GGML_TYPE_IQ1_M:
|
||||||
|
case GGML_TYPE_IQ4_XS:
|
||||||
return true;
|
return true;
|
||||||
|
case GGML_TYPE_IQ4_NL:
|
||||||
|
case GGML_TYPE_MXFP4:
|
||||||
|
// 32-value sub-blocks, the row size does not guarantee
|
||||||
|
// the QK_K super-blocks the get_rows kernel iterates on
|
||||||
|
return op->src[0]->ne[0] % QK_K == 0;
|
||||||
default:
|
default:
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
@@ -4809,13 +4973,23 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
|
|||||||
{
|
{
|
||||||
ggml_type src0_type = op->src[0]->type;
|
ggml_type src0_type = op->src[0]->type;
|
||||||
ggml_type src1_type = op->src[1]->type;
|
ggml_type src1_type = op->src[1]->type;
|
||||||
|
const int32_t dim = op->op_params[0];
|
||||||
return src0_type == src1_type &&
|
return src0_type == src1_type &&
|
||||||
src0_type == op->type &&
|
src0_type == op->type &&
|
||||||
(
|
(
|
||||||
(
|
(
|
||||||
ggml_is_quantized(src0_type) &&
|
ggml_is_quantized(src0_type) &&
|
||||||
ggml_is_contiguous(op->src[0]) &&
|
(
|
||||||
ggml_is_contiguous(op->src[1]) &&
|
(
|
||||||
|
dim == 3 &&
|
||||||
|
ggml_is_contiguous(op->src[0]) &&
|
||||||
|
ggml_is_contiguous(op->src[1])
|
||||||
|
) || (
|
||||||
|
dim != 3 &&
|
||||||
|
ggml_is_contiguous_to_3(op->src[0]) &&
|
||||||
|
ggml_is_contiguous_to_3(op->src[1])
|
||||||
|
)
|
||||||
|
) &&
|
||||||
op->src[0]->ne[0] % ggml_blck_size(src0_type) == 0 &&
|
op->src[0]->ne[0] % ggml_blck_size(src0_type) == 0 &&
|
||||||
op->src[1]->ne[0] % ggml_blck_size(src0_type) == 0
|
op->src[1]->ne[0] % ggml_blck_size(src0_type) == 0
|
||||||
) || (
|
) || (
|
||||||
@@ -4958,6 +5132,16 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
|
|||||||
#else
|
#else
|
||||||
return true;
|
return true;
|
||||||
#endif // GGML_USE_MUSA
|
#endif // GGML_USE_MUSA
|
||||||
|
case GGML_OP_DSV4_HC_COMB:
|
||||||
|
return op->src[0]->type == GGML_TYPE_F32 && op->src[1]->type == GGML_TYPE_F32 &&
|
||||||
|
op->src[2]->type == GGML_TYPE_F32 && op->type == GGML_TYPE_F32;
|
||||||
|
case GGML_OP_DSV4_HC_PRE:
|
||||||
|
return op->src[0]->type == GGML_TYPE_F32 && op->src[1]->type == GGML_TYPE_F32 &&
|
||||||
|
op->type == GGML_TYPE_F32;
|
||||||
|
case GGML_OP_DSV4_HC_POST:
|
||||||
|
return op->src[0]->type == GGML_TYPE_F32 && op->src[1]->type == GGML_TYPE_F32 &&
|
||||||
|
op->src[2]->type == GGML_TYPE_F32 && op->src[3]->type == GGML_TYPE_F32 &&
|
||||||
|
op->type == GGML_TYPE_F32;
|
||||||
case GGML_OP_FLASH_ATTN_EXT:
|
case GGML_OP_FLASH_ATTN_EXT:
|
||||||
return ggml_cuda_flash_attn_ext_supported(dev_ctx->device, op);
|
return ggml_cuda_flash_attn_ext_supported(dev_ctx->device, op);
|
||||||
case GGML_OP_CROSS_ENTROPY_LOSS:
|
case GGML_OP_CROSS_ENTROPY_LOSS:
|
||||||
@@ -4970,6 +5154,8 @@ static bool ggml_backend_cuda_device_supports_op(ggml_backend_dev_t dev, const g
|
|||||||
case GGML_OP_DIAG:
|
case GGML_OP_DIAG:
|
||||||
case GGML_OP_SOLVE_TRI:
|
case GGML_OP_SOLVE_TRI:
|
||||||
return true;
|
return true;
|
||||||
|
case GGML_OP_LIGHTNING_INDEXER:
|
||||||
|
return ggml_cuda_lightning_indexer_supported(dev_ctx->device, op);
|
||||||
|
|
||||||
default:
|
default:
|
||||||
return false;
|
return false;
|
||||||
@@ -5172,18 +5358,24 @@ ggml_backend_reg_t ggml_backend_cuda_reg() {
|
|||||||
ggml_backend_cuda_reg_context * ctx = new ggml_backend_cuda_reg_context;
|
ggml_backend_cuda_reg_context * ctx = new ggml_backend_cuda_reg_context;
|
||||||
const int min_batch_size = getenv("GGML_OP_OFFLOAD_MIN_BATCH") ? atoi(getenv("GGML_OP_OFFLOAD_MIN_BATCH")) : 32;
|
const int min_batch_size = getenv("GGML_OP_OFFLOAD_MIN_BATCH") ? atoi(getenv("GGML_OP_OFFLOAD_MIN_BATCH")) : 32;
|
||||||
|
|
||||||
for (int i = 0; i < ggml_cuda_info().device_count; i++) {
|
const ggml_cuda_device_info & info = ggml_cuda_info();
|
||||||
|
const bool virtual_devices = info.device_count > info.physical_device_count;
|
||||||
|
|
||||||
|
for (int i = 0; i < info.device_count; i++) {
|
||||||
|
const int physical_id = info.devices[i].physical_device;
|
||||||
|
|
||||||
ggml_backend_cuda_device_context * dev_ctx = new ggml_backend_cuda_device_context;
|
ggml_backend_cuda_device_context * dev_ctx = new ggml_backend_cuda_device_context;
|
||||||
dev_ctx->device = i;
|
dev_ctx->device = i;
|
||||||
dev_ctx->name = GGML_CUDA_NAME + std::to_string(i);
|
dev_ctx->name = GGML_CUDA_NAME + std::to_string(i);
|
||||||
|
dev_ctx->description = ggml_cuda_device_description(i);
|
||||||
cudaDeviceProp prop;
|
|
||||||
CUDA_CHECK(cudaGetDeviceProperties(&prop, i));
|
|
||||||
dev_ctx->description = prop.name;
|
|
||||||
|
|
||||||
char pci_bus_id[32] = {};
|
char pci_bus_id[32] = {};
|
||||||
CUDA_CHECK(cudaDeviceGetPCIBusId(pci_bus_id, sizeof(pci_bus_id), i));
|
CUDA_CHECK(cudaDeviceGetPCIBusId(pci_bus_id, sizeof(pci_bus_id), physical_id));
|
||||||
dev_ctx->pci_bus_id = pci_bus_id;
|
dev_ctx->pci_bus_id = pci_bus_id;
|
||||||
|
if (virtual_devices) {
|
||||||
|
// make the pci bus id unique for virtual devices
|
||||||
|
dev_ctx->pci_bus_id += "-v" + std::to_string(i);
|
||||||
|
}
|
||||||
for (char & c : dev_ctx->pci_bus_id) {
|
for (char & c : dev_ctx->pci_bus_id) {
|
||||||
c = std::tolower(c);
|
c = std::tolower(c);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,588 @@
|
|||||||
|
#include "common.cuh"
|
||||||
|
#include "lightning-indexer.cuh"
|
||||||
|
#include "fattn-common.cuh"
|
||||||
|
#include "convert.cuh"
|
||||||
|
|
||||||
|
#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
|
||||||
|
#if defined(TURING_MMA_AVAILABLE)
|
||||||
|
|
||||||
|
typedef union {
|
||||||
|
int2 i2;
|
||||||
|
half2 h2[2];
|
||||||
|
} half4;
|
||||||
|
|
||||||
|
// TODO add support for AMD cards via rocWMMA
|
||||||
|
#include <mma.h>
|
||||||
|
namespace wmma = nvcuda::wmma;
|
||||||
|
|
||||||
|
template <int WARPS_PER_BLOCK, int K_VECS_PER_BLOCK, int64_t N_EMBD, int64_t N_HEAD, ggml_type TYPE_K>
|
||||||
|
static __global__ void lightning_indexer_kernel_wmma(
|
||||||
|
const float * Q, const char * K, const float * W, const half * M, float * dst,
|
||||||
|
int64_t n_stream, int64_t n_batch, int64_t n_kv,
|
||||||
|
size_t nb1, size_t nb2, size_t nb3,
|
||||||
|
size_t nbq1, size_t nbq2, size_t nbq3,
|
||||||
|
size_t nbk1, size_t nbk2, size_t nbk3,
|
||||||
|
size_t nbw1, size_t nbw2, size_t nbw3,
|
||||||
|
size_t nbm1, size_t nbm2, size_t nbm3,
|
||||||
|
int64_t nem3
|
||||||
|
) {
|
||||||
|
|
||||||
|
constexpr int THREADS_PER_BLOCK = WARPS_PER_BLOCK * WARP_SIZE;
|
||||||
|
constexpr int HEADS_PER_INNER_LOOP = 8;
|
||||||
|
constexpr int K_EMBD_PER_INNER_LOOP = 16;
|
||||||
|
constexpr int N_EMBD_PADDED = N_EMBD + 8;
|
||||||
|
|
||||||
|
const int i_batch = blockIdx.y;
|
||||||
|
const int i_stream = blockIdx.z;
|
||||||
|
const int i_warp = threadIdx.y;
|
||||||
|
const int i_lane = threadIdx.x;
|
||||||
|
const int tid = i_warp * WARP_SIZE + i_lane;
|
||||||
|
|
||||||
|
// each block processes K_VECS_PER_BLOCK K vectors
|
||||||
|
const int start_kv = blockIdx.x * K_VECS_PER_BLOCK;
|
||||||
|
|
||||||
|
const char * q_base = (const char *) Q + i_batch*nbq2 + i_stream*nbq3;
|
||||||
|
const float * w_base = (const float *) ((const char *) W + i_batch*nbw1 + i_stream*nbw3);
|
||||||
|
|
||||||
|
// phase 1 - load weights and first Q tile to shared memory
|
||||||
|
|
||||||
|
__shared__ float w_shared[N_HEAD];
|
||||||
|
__shared__ int2 q_shared_h[HEADS_PER_INNER_LOOP][N_EMBD_PADDED / 4];
|
||||||
|
|
||||||
|
if (tid < N_HEAD) {
|
||||||
|
w_shared[tid] = w_base[tid];
|
||||||
|
}
|
||||||
|
|
||||||
|
// total number of half4 elements in HEADS_PER_INNER_LOOP x N_EMBD Q tile
|
||||||
|
constexpr int N_Q_TILE = HEADS_PER_INNER_LOOP * (N_EMBD / 4);
|
||||||
|
// number of registers needed in each thread to store Q tile in thread block
|
||||||
|
constexpr int N_Q_NEXT = (N_Q_TILE + THREADS_PER_BLOCK - 1) / THREADS_PER_BLOCK;
|
||||||
|
|
||||||
|
#pragma unroll
|
||||||
|
for (int i_q = tid; i_q < N_Q_TILE; i_q += THREADS_PER_BLOCK) {
|
||||||
|
const int i_head = i_q / (N_EMBD / 4);
|
||||||
|
const int i_embd = i_q % (N_EMBD / 4);
|
||||||
|
const float4 q = *(const float4 *) (q_base + i_head*nbq1 + i_embd*sizeof(float4));
|
||||||
|
half4 q_packed;
|
||||||
|
q_packed.h2[0] = __float22half2_rn(make_float2(q.x, q.y));
|
||||||
|
q_packed.h2[1] = __float22half2_rn(make_float2(q.z, q.w));
|
||||||
|
q_shared_h[i_head][i_embd] = q_packed.i2;
|
||||||
|
}
|
||||||
|
|
||||||
|
// phase 2 - load (and dequantize if needed) K to shared mem
|
||||||
|
|
||||||
|
__shared__ half2 k_shared_h[K_VECS_PER_BLOCK][N_EMBD_PADDED / 4][2];
|
||||||
|
|
||||||
|
constexpr int n_k = K_VECS_PER_BLOCK * (N_EMBD / 4);
|
||||||
|
|
||||||
|
if constexpr (TYPE_K == GGML_TYPE_F16) {
|
||||||
|
#pragma unroll
|
||||||
|
for (int i_k = tid; i_k < n_k; i_k += THREADS_PER_BLOCK) {
|
||||||
|
const int i_k_vec = i_k / (N_EMBD / 4);
|
||||||
|
const int i_embd = i_k % (N_EMBD / 4);
|
||||||
|
const int i_kv = start_kv + i_k_vec;
|
||||||
|
if (i_kv < n_kv) {
|
||||||
|
const int2 * k_base = (const int2 *) ((const char *) K + i_kv*nbk2 + i_stream*nbk3);
|
||||||
|
*(int2*) &k_shared_h[i_k_vec][i_embd] = k_base[i_embd];
|
||||||
|
} else {
|
||||||
|
*(int2*) &k_shared_h[i_k_vec][i_embd] = make_int2(0, 0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
constexpr dequantize_V_t dequantize_k = get_dequantize_V<TYPE_K, half, 4>();
|
||||||
|
#pragma unroll
|
||||||
|
for (int i_k = tid; i_k < n_k; i_k += THREADS_PER_BLOCK) {
|
||||||
|
const int i_k_vec = i_k / (N_EMBD / 4);
|
||||||
|
const int i_embd = i_k % (N_EMBD / 4);
|
||||||
|
const int i_kv = start_kv + i_k_vec;
|
||||||
|
if (i_kv < n_kv) {
|
||||||
|
const void * k_base = (const void *) ((const char *) K + i_kv*nbk2 + i_stream*nbk3);
|
||||||
|
dequantize_k(k_base, &k_shared_h[i_k_vec][i_embd][0], i_embd * 4);
|
||||||
|
} else {
|
||||||
|
*(int2*) &k_shared_h[i_k_vec][i_embd] = make_int2(0, 0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
__syncthreads();
|
||||||
|
|
||||||
|
// phase 3 - calculate lightning indexer scores
|
||||||
|
|
||||||
|
__shared__ float qk_shared[WARPS_PER_BLOCK][HEADS_PER_INNER_LOOP][K_VECS_PER_BLOCK];
|
||||||
|
|
||||||
|
// load K fragment
|
||||||
|
wmma::fragment<wmma::matrix_b, HEADS_PER_INNER_LOOP, K_VECS_PER_BLOCK, K_EMBD_PER_INNER_LOOP, half, wmma::col_major> frag_k;
|
||||||
|
wmma::load_matrix_sync(frag_k, (half*) &k_shared_h[0][i_warp * K_EMBD_PER_INNER_LOOP / 4], N_EMBD_PADDED);
|
||||||
|
|
||||||
|
float score_k = 0.0f;
|
||||||
|
|
||||||
|
for (int i_head_0 = 0; i_head_0 < N_HEAD; i_head_0 += HEADS_PER_INNER_LOOP) {
|
||||||
|
const int i_head_next = i_head_0 + HEADS_PER_INNER_LOOP;
|
||||||
|
|
||||||
|
// we don't use accumulator for anything, fill it with zeros
|
||||||
|
wmma::fragment<wmma::accumulator, HEADS_PER_INNER_LOOP, K_VECS_PER_BLOCK, K_EMBD_PER_INNER_LOOP, float> frag_acc;
|
||||||
|
wmma::fill_fragment(frag_acc, 0.0f);
|
||||||
|
|
||||||
|
// load Q fragment
|
||||||
|
wmma::fragment<wmma::matrix_a, HEADS_PER_INNER_LOOP, K_VECS_PER_BLOCK, K_EMBD_PER_INNER_LOOP, half, wmma::row_major> frag_q;
|
||||||
|
wmma::load_matrix_sync(frag_q, (half*) &q_shared_h[0][i_warp * K_EMBD_PER_INNER_LOOP / 4], N_EMBD_PADDED);
|
||||||
|
|
||||||
|
// preload next Q tile to registers during matrix multiplication
|
||||||
|
float4 q_next[N_Q_NEXT];
|
||||||
|
|
||||||
|
if (i_head_next < N_HEAD) {
|
||||||
|
#pragma unroll
|
||||||
|
for (int i_q = tid, i_q_next = 0; i_q < N_Q_TILE; i_q += THREADS_PER_BLOCK) {
|
||||||
|
const int i_head = i_head_next + i_q / (N_EMBD / 4);
|
||||||
|
const int i_embd = i_q % (N_EMBD / 4);
|
||||||
|
q_next[i_q_next++] = *(const float4 *) (q_base + i_head*nbq1 + i_embd*sizeof(float4));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// perform matrix multiplication
|
||||||
|
wmma::mma_sync(frag_acc, frag_q, frag_k, frag_acc);
|
||||||
|
wmma::store_matrix_sync((float*) &qk_shared[i_warp][0][0], frag_acc, K_VECS_PER_BLOCK, wmma::mem_row_major);
|
||||||
|
|
||||||
|
// make sure all threads finished using q_shared_h so we can store next tile
|
||||||
|
__syncthreads();
|
||||||
|
|
||||||
|
// write preloaded Q tile to shared memory
|
||||||
|
if (i_head_next < N_HEAD) {
|
||||||
|
#pragma unroll
|
||||||
|
for (int i_q = tid, i_q_next = 0; i_q < N_Q_TILE; i_q += THREADS_PER_BLOCK) {
|
||||||
|
const int i_head = i_q / (N_EMBD / 4);
|
||||||
|
const int i_embd = i_q % (N_EMBD / 4);
|
||||||
|
half4 q_packed;
|
||||||
|
q_packed.h2[0] = __float22half2_rn(make_float2(q_next[i_q_next].x, q_next[i_q_next].y));
|
||||||
|
q_packed.h2[1] = __float22half2_rn(make_float2(q_next[i_q_next].z, q_next[i_q_next].w));
|
||||||
|
q_shared_h[i_head][i_embd] = q_packed.i2;
|
||||||
|
++i_q_next;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// accumulate QK multiplication results from all block warps
|
||||||
|
// (there are 256 threads in block and 256 matmul outputs)
|
||||||
|
// TODO it will break if WARP_SIZE is not 32
|
||||||
|
const int h = tid / K_VECS_PER_BLOCK;
|
||||||
|
const int k = tid % K_VECS_PER_BLOCK;
|
||||||
|
const float w_val = w_shared[i_head_0 + h];
|
||||||
|
|
||||||
|
float sum = 0.0f;
|
||||||
|
#pragma unroll
|
||||||
|
for (int w = 0; w < WARPS_PER_BLOCK; ++w) {
|
||||||
|
sum += qk_shared[w][h][k];
|
||||||
|
}
|
||||||
|
|
||||||
|
// ReLU, weight
|
||||||
|
sum = sum > 0.0f ? sum : 0.0f;
|
||||||
|
sum *= w_val;
|
||||||
|
|
||||||
|
// wait until qk_shared[0] is no longer used
|
||||||
|
__syncthreads();
|
||||||
|
|
||||||
|
// reuse qk_shared[0] for storing partial results
|
||||||
|
qk_shared[0][h][k] = sum;
|
||||||
|
|
||||||
|
// wait until all threads write their results
|
||||||
|
__syncthreads();
|
||||||
|
|
||||||
|
// accumulate result over heads
|
||||||
|
if (tid < K_VECS_PER_BLOCK) {
|
||||||
|
#pragma unroll
|
||||||
|
for (int i_head = 0; i_head < HEADS_PER_INNER_LOOP; ++i_head) {
|
||||||
|
score_k += qk_shared[0][i_head][tid];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// make sure all threads finished using qk_shared
|
||||||
|
__syncthreads();
|
||||||
|
}
|
||||||
|
|
||||||
|
// phase 4 - store output to VRAM
|
||||||
|
|
||||||
|
if (tid < K_VECS_PER_BLOCK) {
|
||||||
|
const int i_kv = start_kv + tid;
|
||||||
|
if (i_kv < n_kv) {
|
||||||
|
const half * m_base = (const half *) ((const char *) M + i_batch*nbm1 + (i_stream%nem3)*nbm3);
|
||||||
|
float * dst_base = (float *) ((char *) dst + i_batch*nb1 + i_stream*nb3);
|
||||||
|
dst_base[i_kv] = score_k + __half2float(m_base[i_kv]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#else // defined(TURING_MMA_AVAILABLE)
|
||||||
|
|
||||||
|
template <int WARPS_PER_BLOCK, int K_VECS_PER_BLOCK, int64_t N_EMBD, int64_t N_HEAD, ggml_type TYPE_K>
|
||||||
|
static __global__ void lightning_indexer_kernel_wmma(
|
||||||
|
const float * Q, const char * K, const float * W, const half * M, float * dst,
|
||||||
|
int64_t n_stream, int64_t n_batch, int64_t n_kv,
|
||||||
|
size_t nb1, size_t nb2, size_t nb3,
|
||||||
|
size_t nbq1, size_t nbq2, size_t nbq3,
|
||||||
|
size_t nbk1, size_t nbk2, size_t nbk3,
|
||||||
|
size_t nbw1, size_t nbw2, size_t nbw3,
|
||||||
|
size_t nbm1, size_t nbm2, size_t nbm3,
|
||||||
|
int64_t nem3
|
||||||
|
) {
|
||||||
|
GGML_UNUSED_VARS(Q, K, W, M, dst,
|
||||||
|
n_stream, n_batch, n_kv,
|
||||||
|
nb1, nb2, nb3,
|
||||||
|
nbq1, nbq2, nbq3,
|
||||||
|
nbk1, nbk2, nbk3,
|
||||||
|
nbw1, nbw2, nbw3,
|
||||||
|
nem3);
|
||||||
|
NO_DEVICE_CODE;
|
||||||
|
}
|
||||||
|
|
||||||
|
#endif // defined(TURING_MMA_AVAILABLE)
|
||||||
|
#endif // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
|
||||||
|
|
||||||
|
// TODO there is one ugly assumption used in this kernel - that WARP_SIZE is equal to 32
|
||||||
|
// thanks to that one warp operating on float4 processes whole indexer K/Q vectors
|
||||||
|
// 32 * 4 = 128 (N_EMBD)
|
||||||
|
|
||||||
|
template <int WARPS_PER_BLOCK, int K_VECS_PER_BLOCK, int64_t N_EMBD, int64_t N_HEAD, ggml_type TYPE_K>
|
||||||
|
static __global__ void lightning_indexer_kernel_vec(
|
||||||
|
const float * Q, const char * K, const float * W, const half * M, float * dst,
|
||||||
|
int64_t n_stream, int64_t n_batch, int64_t n_kv,
|
||||||
|
size_t nb1, size_t nb2, size_t nb3,
|
||||||
|
size_t nbq1, size_t nbq2, size_t nbq3,
|
||||||
|
size_t nbk1, size_t nbk2, size_t nbk3,
|
||||||
|
size_t nbw1, size_t nbw2, size_t nbw3,
|
||||||
|
size_t nbm1, size_t nbm2, size_t nbm3,
|
||||||
|
int64_t nem3
|
||||||
|
) {
|
||||||
|
|
||||||
|
constexpr int K_VECS_PER_WARP = K_VECS_PER_BLOCK / WARPS_PER_BLOCK;
|
||||||
|
constexpr int THREADS_PER_BLOCK = WARPS_PER_BLOCK * WARP_SIZE;
|
||||||
|
|
||||||
|
const int i_batch = blockIdx.y;
|
||||||
|
const int i_stream = blockIdx.z;
|
||||||
|
const int i_warp = threadIdx.y;
|
||||||
|
const int i_lane = threadIdx.x;
|
||||||
|
const int tid = i_warp * WARP_SIZE + i_lane;
|
||||||
|
|
||||||
|
// each warp processes K_VECS_PER_WARP K vectors
|
||||||
|
const int start_kv_block = blockIdx.x * K_VECS_PER_BLOCK;
|
||||||
|
const int start_kv = start_kv_block + i_warp * K_VECS_PER_WARP;
|
||||||
|
|
||||||
|
const char * q_base = (const char *) Q + i_batch*nbq2 + i_stream*nbq3;
|
||||||
|
const float * w_base = (const float *) ((const char *) W + i_batch*nbw1 + i_stream*nbw3);
|
||||||
|
|
||||||
|
// phase 1 - load (and dequantize if needed) K to registers
|
||||||
|
|
||||||
|
float4 k_reg_f[K_VECS_PER_WARP];
|
||||||
|
|
||||||
|
if constexpr (TYPE_K == GGML_TYPE_F32) {
|
||||||
|
// direct copy of float4
|
||||||
|
#pragma unroll
|
||||||
|
for (int k = 0; k < K_VECS_PER_WARP; ++k) {
|
||||||
|
int i_kv = start_kv + k;
|
||||||
|
if (i_kv < n_kv) {
|
||||||
|
const float4 * k_base = (const float4 *) ((const char *) K + i_kv*nbk2 + i_stream*nbk3);
|
||||||
|
k_reg_f[k] = k_base[i_lane];
|
||||||
|
} else {
|
||||||
|
k_reg_f[k] = make_float4(0, 0, 0, 0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// dequantize remaining types to float
|
||||||
|
constexpr dequantize_V_t dequantize_k = get_dequantize_V<TYPE_K, float, 4>();
|
||||||
|
#pragma unroll
|
||||||
|
for (int k = 0; k < K_VECS_PER_WARP; ++k) {
|
||||||
|
int i_kv = start_kv + k;
|
||||||
|
if (i_kv < n_kv) {
|
||||||
|
const void * k_base = (const void *) ((const char *) K + i_kv*nbk2 + i_stream*nbk3);
|
||||||
|
dequantize_k(k_base, &k_reg_f[k], i_lane * 4);
|
||||||
|
} else {
|
||||||
|
k_reg_f[k] = make_float4(0, 0, 0, 0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
float score_k[K_VECS_PER_WARP] = { 0.0f };
|
||||||
|
|
||||||
|
// load weights and Q only for N_HEAD_INNER heads at once to reduce shared memory usage
|
||||||
|
constexpr int N_HEAD_INNER = N_HEAD / 4;
|
||||||
|
|
||||||
|
for (int i_head_0 = 0; i_head_0 < N_HEAD; i_head_0 += N_HEAD_INNER) {
|
||||||
|
// phase 2 - load weights and Q to shared memory
|
||||||
|
|
||||||
|
__shared__ float w_shared[N_HEAD_INNER];
|
||||||
|
__shared__ float4 q_shared_f[N_HEAD_INNER][N_EMBD / 4];
|
||||||
|
|
||||||
|
if (tid < N_HEAD_INNER) {
|
||||||
|
w_shared[tid] = w_base[i_head_0 + tid];
|
||||||
|
}
|
||||||
|
|
||||||
|
constexpr int n_q = N_HEAD_INNER * (N_EMBD / 4);
|
||||||
|
#pragma unroll
|
||||||
|
for (int i_q = tid; i_q < n_q; i_q += THREADS_PER_BLOCK) {
|
||||||
|
const int i_head_inner = i_q / (N_EMBD / 4);
|
||||||
|
const int i_head = i_head_0 + i_head_inner;
|
||||||
|
const int i_embd = i_q % (N_EMBD / 4);
|
||||||
|
q_shared_f[i_head_inner][i_embd] = *(const float4 *) (q_base + i_head*nbq1 + i_embd*sizeof(float4));
|
||||||
|
}
|
||||||
|
|
||||||
|
__syncthreads();
|
||||||
|
|
||||||
|
// phase 3 - calculate lightning indexer scores
|
||||||
|
|
||||||
|
for (int i_head_inner = 0; i_head_inner < N_HEAD_INNER; ++i_head_inner) {
|
||||||
|
const float w_val = w_shared[i_head_inner];
|
||||||
|
float qk[K_VECS_PER_WARP] = { 0.0f };
|
||||||
|
|
||||||
|
// dot product of floats
|
||||||
|
const float4 q_vec = q_shared_f[i_head_inner][i_lane];
|
||||||
|
|
||||||
|
#pragma unroll
|
||||||
|
for (int k = 0; k < K_VECS_PER_WARP; ++k) {
|
||||||
|
ggml_cuda_mad(qk[k], q_vec.x, k_reg_f[k].x);
|
||||||
|
ggml_cuda_mad(qk[k], q_vec.y, k_reg_f[k].y);
|
||||||
|
ggml_cuda_mad(qk[k], q_vec.z, k_reg_f[k].z);
|
||||||
|
ggml_cuda_mad(qk[k], q_vec.w, k_reg_f[k].w);
|
||||||
|
}
|
||||||
|
|
||||||
|
#pragma unroll
|
||||||
|
for (int k = 0; k < K_VECS_PER_WARP; ++k) {
|
||||||
|
float sum = warp_reduce_sum(qk[k]);
|
||||||
|
|
||||||
|
// ReLU, weight
|
||||||
|
if (i_lane == 0) {
|
||||||
|
sum = (sum > 0.0f) ? sum : 0.0f;
|
||||||
|
score_k[k] += sum * w_val;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
__syncthreads();
|
||||||
|
}
|
||||||
|
|
||||||
|
// phase 4 - store outputs to shared memory
|
||||||
|
|
||||||
|
__shared__ float dst_shared[K_VECS_PER_BLOCK];
|
||||||
|
|
||||||
|
if (i_lane == 0) {
|
||||||
|
#pragma unroll
|
||||||
|
for (int k = 0; k < K_VECS_PER_WARP; ++k) {
|
||||||
|
dst_shared[i_warp * K_VECS_PER_WARP + k] = score_k[k];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
__syncthreads();
|
||||||
|
|
||||||
|
// phase 5 - write from shared memory to VRAM in coalesced manner
|
||||||
|
|
||||||
|
if (tid < K_VECS_PER_BLOCK) {
|
||||||
|
int i_kv = start_kv_block + tid;
|
||||||
|
if (i_kv < n_kv) {
|
||||||
|
const half * m_base = (const half *) ((const char *) M + i_batch*nbm1 + (i_stream%nem3)*nbm3);
|
||||||
|
float * dst_base = (float *) ((char *) dst + i_batch*nb1 + i_stream*nb3);
|
||||||
|
dst_base[i_kv] = dst_shared[tid] + __half2float(m_base[i_kv]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#define LIGHTNING_INDEXER_CASE(lightning_indexer_kernel, n_embd, n_head, K, type_K) \
|
||||||
|
if (K->type == (type_K)) { \
|
||||||
|
lightning_indexer_kernel<WARPS_PER_BLOCK, K_VECS_PER_BLOCK, n_embd, n_head, type_K> \
|
||||||
|
<<<grid, block, 0, ctx.stream()>>>( \
|
||||||
|
q_d, k_d, w_d, m_d, dst_d, \
|
||||||
|
n_stream, n_batch, n_kv, \
|
||||||
|
nb1, nb2, nb3, \
|
||||||
|
nbq1, nbq2, nbq3, \
|
||||||
|
nbk1, nbk2, nbk3, \
|
||||||
|
nbw1, nbw2, nbw3, \
|
||||||
|
nbm1, nbm2, nbm3, \
|
||||||
|
nem3 \
|
||||||
|
); \
|
||||||
|
} else
|
||||||
|
|
||||||
|
void ggml_cuda_lightning_indexer(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
|
||||||
|
const ggml_tensor * q = dst->src[0];
|
||||||
|
const ggml_tensor * k = dst->src[1];
|
||||||
|
const ggml_tensor * w = dst->src[2]; // weights
|
||||||
|
const ggml_tensor * m = dst->src[3]; // mask
|
||||||
|
|
||||||
|
GGML_ASSERT(dst->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT( q->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT( w->type == GGML_TYPE_F32);
|
||||||
|
GGML_ASSERT( m->type == GGML_TYPE_F16);
|
||||||
|
|
||||||
|
GGML_TENSOR_LOCALS(int64_t, neq, q, ne)
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbq, q, nb)
|
||||||
|
GGML_TENSOR_LOCALS(int64_t, nek, k, ne)
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbk, k, nb)
|
||||||
|
GGML_TENSOR_LOCALS(int64_t, new, w, ne)
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbw, w, nb)
|
||||||
|
GGML_TENSOR_LOCALS(int64_t, nem, m, ne)
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbm, m, nb)
|
||||||
|
GGML_TENSOR_LOCALS(int64_t, ne, dst, ne)
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nb, dst, nb)
|
||||||
|
|
||||||
|
// input tensor rows must be contiguous
|
||||||
|
GGML_ASSERT(nbq0 == ggml_type_size(q->type));
|
||||||
|
GGML_ASSERT(nbk0 == ggml_type_size(k->type));
|
||||||
|
GGML_ASSERT(nbw0 == ggml_type_size(w->type));
|
||||||
|
GGML_ASSERT(nbm0 == ggml_type_size(m->type));
|
||||||
|
|
||||||
|
// dst cannot be transposed or permuted
|
||||||
|
GGML_ASSERT(nb0 == sizeof(float));
|
||||||
|
GGML_ASSERT(nb0 <= nb1);
|
||||||
|
GGML_ASSERT(nb1 <= nb2);
|
||||||
|
GGML_ASSERT(nb2 <= nb3);
|
||||||
|
|
||||||
|
const int n_embd = q->ne[0];
|
||||||
|
const int n_head = q->ne[1];
|
||||||
|
const int n_batch = q->ne[2];
|
||||||
|
const int n_stream = q->ne[3];
|
||||||
|
const int n_kv = k->ne[2];
|
||||||
|
|
||||||
|
const float * q_d = (const float *) q->data;
|
||||||
|
const char * k_d = (const char *) k->data;
|
||||||
|
const float * w_d = (const float *) w->data;
|
||||||
|
const half * m_d = (const half *) m->data;
|
||||||
|
float * dst_d = ( float *) dst->data;
|
||||||
|
|
||||||
|
const int device = ggml_cuda_get_device();
|
||||||
|
const int cc = ggml_cuda_info().devices[device].cc;
|
||||||
|
|
||||||
|
if (n_embd == 128 && n_head == 64) {
|
||||||
|
#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
|
||||||
|
if (GGML_CUDA_CC_IS_NVIDIA(cc) && turing_mma_available(cc) && k->type != GGML_TYPE_F32 && k->type != GGML_TYPE_BF16) {
|
||||||
|
// use wmma kernel
|
||||||
|
constexpr int K_VECS_PER_BLOCK = 32;
|
||||||
|
constexpr int WARPS_PER_BLOCK = 8;
|
||||||
|
|
||||||
|
dim3 block(32, WARPS_PER_BLOCK);
|
||||||
|
int num_kv_blocks = (n_kv + (K_VECS_PER_BLOCK) - 1) / (K_VECS_PER_BLOCK);
|
||||||
|
dim3 grid(num_kv_blocks, n_batch, n_stream);
|
||||||
|
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_wmma, 128, 64, k, GGML_TYPE_F16)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_wmma, 128, 64, k, GGML_TYPE_Q4_0)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_wmma, 128, 64, k, GGML_TYPE_Q4_1)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_wmma, 128, 64, k, GGML_TYPE_Q5_0)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_wmma, 128, 64, k, GGML_TYPE_Q5_1)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_wmma, 128, 64, k, GGML_TYPE_Q8_0)
|
||||||
|
GGML_ABORT("fatal error");
|
||||||
|
} else {
|
||||||
|
#else // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
|
||||||
|
{
|
||||||
|
#endif // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
|
||||||
|
// use vector kernel
|
||||||
|
constexpr int K_VECS_PER_WARP = 8;
|
||||||
|
constexpr int WARPS_PER_BLOCK = 8;
|
||||||
|
constexpr int K_VECS_PER_BLOCK = K_VECS_PER_WARP * WARPS_PER_BLOCK;
|
||||||
|
|
||||||
|
dim3 block(32, WARPS_PER_BLOCK);
|
||||||
|
int num_kv_blocks = (n_kv + (K_VECS_PER_BLOCK) - 1) / (K_VECS_PER_BLOCK);
|
||||||
|
dim3 grid(num_kv_blocks, n_batch, n_stream);
|
||||||
|
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_vec, 128, 64, k, GGML_TYPE_F16)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_vec, 128, 64, k, GGML_TYPE_Q4_0)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_vec, 128, 64, k, GGML_TYPE_Q4_1)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_vec, 128, 64, k, GGML_TYPE_Q5_0)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_vec, 128, 64, k, GGML_TYPE_Q5_1)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_vec, 128, 64, k, GGML_TYPE_Q8_0)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_vec, 128, 64, k, GGML_TYPE_BF16)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_vec, 128, 64, k, GGML_TYPE_F32)
|
||||||
|
GGML_ABORT("fatal error");
|
||||||
|
}
|
||||||
|
} else if (n_embd == 128 && n_head == 32) {
|
||||||
|
#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
|
||||||
|
if (GGML_CUDA_CC_IS_NVIDIA(cc) && turing_mma_available(cc) && k->type != GGML_TYPE_F32 && k->type != GGML_TYPE_BF16) {
|
||||||
|
// use wmma kernel
|
||||||
|
constexpr int K_VECS_PER_BLOCK = 32;
|
||||||
|
constexpr int WARPS_PER_BLOCK = 8;
|
||||||
|
|
||||||
|
dim3 block(32, WARPS_PER_BLOCK);
|
||||||
|
int num_kv_blocks = (n_kv + (K_VECS_PER_BLOCK) - 1) / (K_VECS_PER_BLOCK);
|
||||||
|
dim3 grid(num_kv_blocks, n_batch, n_stream);
|
||||||
|
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_wmma, 128, 32, k, GGML_TYPE_F16)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_wmma, 128, 32, k, GGML_TYPE_Q4_0)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_wmma, 128, 32, k, GGML_TYPE_Q4_1)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_wmma, 128, 32, k, GGML_TYPE_Q5_0)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_wmma, 128, 32, k, GGML_TYPE_Q5_1)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_wmma, 128, 32, k, GGML_TYPE_Q8_0)
|
||||||
|
GGML_ABORT("fatal error");
|
||||||
|
} else {
|
||||||
|
#else // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
|
||||||
|
{
|
||||||
|
#endif // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
|
||||||
|
// use vector kernel
|
||||||
|
constexpr int K_VECS_PER_WARP = 8;
|
||||||
|
constexpr int WARPS_PER_BLOCK = 8;
|
||||||
|
constexpr int K_VECS_PER_BLOCK = K_VECS_PER_WARP * WARPS_PER_BLOCK;
|
||||||
|
|
||||||
|
dim3 block(32, WARPS_PER_BLOCK);
|
||||||
|
int num_kv_blocks = (n_kv + (K_VECS_PER_BLOCK) - 1) / (K_VECS_PER_BLOCK);
|
||||||
|
dim3 grid(num_kv_blocks, n_batch, n_stream);
|
||||||
|
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_vec, 128, 32, k, GGML_TYPE_F16)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_vec, 128, 32, k, GGML_TYPE_Q4_0)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_vec, 128, 32, k, GGML_TYPE_Q4_1)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_vec, 128, 32, k, GGML_TYPE_Q5_0)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_vec, 128, 32, k, GGML_TYPE_Q5_1)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_vec, 128, 32, k, GGML_TYPE_Q8_0)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_vec, 128, 32, k, GGML_TYPE_BF16)
|
||||||
|
LIGHTNING_INDEXER_CASE(lightning_indexer_kernel_vec, 128, 32, k, GGML_TYPE_F32)
|
||||||
|
GGML_ABORT("fatal error");
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
GGML_ABORT("fatal error");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
bool ggml_cuda_lightning_indexer_supported(int device, const ggml_tensor * dst) {
|
||||||
|
GGML_UNUSED(device);
|
||||||
|
|
||||||
|
const ggml_tensor * q = dst->src[0];
|
||||||
|
const ggml_tensor * k = dst->src[1];
|
||||||
|
const ggml_tensor * w = dst->src[2]; // weights
|
||||||
|
const ggml_tensor * m = dst->src[3]; // mask
|
||||||
|
|
||||||
|
GGML_TENSOR_LOCALS(int64_t, neq, q, ne)
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbq, q, nb)
|
||||||
|
GGML_TENSOR_LOCALS(int64_t, nek, k, ne)
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbk, k, nb)
|
||||||
|
GGML_TENSOR_LOCALS(int64_t, new, w, ne)
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbw, w, nb)
|
||||||
|
GGML_TENSOR_LOCALS(int64_t, nem, m, ne)
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nbm, m, nb)
|
||||||
|
GGML_TENSOR_LOCALS(int64_t, ne, dst, ne)
|
||||||
|
GGML_TENSOR_LOCALS(size_t, nb, dst, nb)
|
||||||
|
|
||||||
|
if (neq0 != 128) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (neq1 != 64 && neq1 != 32) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
// alignment checks
|
||||||
|
for (const ggml_tensor * t : {q, k}) {
|
||||||
|
if (ggml_is_quantized(t->type)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
for (size_t i = 1; i < GGML_MAX_DIMS; ++i) {
|
||||||
|
if (t->nb[i] % 16 != 0) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
switch(k->type) {
|
||||||
|
case GGML_TYPE_F32:
|
||||||
|
case GGML_TYPE_BF16:
|
||||||
|
case GGML_TYPE_F16:
|
||||||
|
case GGML_TYPE_Q8_0:
|
||||||
|
case GGML_TYPE_Q5_1:
|
||||||
|
case GGML_TYPE_Q5_0:
|
||||||
|
case GGML_TYPE_Q4_1:
|
||||||
|
case GGML_TYPE_Q4_0:
|
||||||
|
return true;
|
||||||
|
default:
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
#include "common.cuh"
|
||||||
|
|
||||||
|
void ggml_cuda_lightning_indexer(ggml_backend_cuda_context & ctx, ggml_tensor * dst);
|
||||||
|
bool ggml_cuda_lightning_indexer_supported(int device, const ggml_tensor * dst);
|
||||||
@@ -85,7 +85,7 @@ void ggml_cuda_mul_mat_f(ggml_backend_cuda_context & ctx, const ggml_tensor * sr
|
|||||||
GGML_ASSERT(sis1 > 0);
|
GGML_ASSERT(sis1 > 0);
|
||||||
|
|
||||||
ggml_cuda_launch_mm_ids_helper(ids_d, ids_src_compact_dev.get(), ids_dst_compact_dev.get(), expert_bounds_dev.get(),
|
ggml_cuda_launch_mm_ids_helper(ids_d, ids_src_compact_dev.get(), ids_dst_compact_dev.get(), expert_bounds_dev.get(),
|
||||||
static_cast<int>(n_experts), static_cast<int>(n_tokens), static_cast<int>(n_expert_used), static_cast<int>(ne11), si1, sis1, ctx.stream());
|
static_cast<int>(n_experts), static_cast<int>(n_tokens), static_cast<int>(n_expert_used), static_cast<int>(ne11), si1, sis1, /*write_inverse =*/ false, ctx.stream());
|
||||||
CUDA_CHECK(cudaGetLastError());
|
CUDA_CHECK(cudaGetLastError());
|
||||||
|
|
||||||
ids_info.ids_src_compact = ids_src_compact_dev.get();
|
ids_info.ids_src_compact = ids_src_compact_dev.get();
|
||||||
|
|||||||
+18
-13
@@ -27,7 +27,7 @@ template <int n_expert_used_template>
|
|||||||
__launch_bounds__(ggml_cuda_get_physical_warp_size(), 1)
|
__launch_bounds__(ggml_cuda_get_physical_warp_size(), 1)
|
||||||
static __global__ void mm_ids_helper(
|
static __global__ void mm_ids_helper(
|
||||||
const int32_t * __restrict__ ids, int32_t * __restrict__ ids_src1, int32_t * __restrict__ ids_dst, int32_t * __restrict__ expert_bounds,
|
const int32_t * __restrict__ ids, int32_t * __restrict__ ids_src1, int32_t * __restrict__ ids_dst, int32_t * __restrict__ expert_bounds,
|
||||||
const int n_tokens, const int n_expert_used_var, const int nchannels_y, const int si1, const int sis1) {
|
const int n_tokens, const int n_expert_used_var, const int nchannels_y, const int si1, const int sis1, const bool write_inverse) {
|
||||||
constexpr int warp_size = ggml_cuda_get_physical_warp_size();
|
constexpr int warp_size = ggml_cuda_get_physical_warp_size();
|
||||||
const int n_expert_used = n_expert_used_template == 0 ? n_expert_used_var : n_expert_used_template;
|
const int n_expert_used = n_expert_used_template == 0 ? n_expert_used_var : n_expert_used_template;
|
||||||
const int expert = blockIdx.x;
|
const int expert = blockIdx.x;
|
||||||
@@ -98,8 +98,13 @@ static __global__ void mm_ids_helper(
|
|||||||
const mm_ids_helper_store store_it = store[itc];
|
const mm_ids_helper_store store_it = store[itc];
|
||||||
const int it = store_it.it();
|
const int it = store_it.it();
|
||||||
const int iex_used = store_it.iex_used();
|
const int iex_used = store_it.iex_used();
|
||||||
ids_src1[nex_prev + itc] = it*sis1 + iex_used % nchannels_y;
|
ids_dst[nex_prev + itc] = it*n_expert_used + iex_used;
|
||||||
ids_dst [nex_prev + itc] = it*n_expert_used + iex_used;
|
// ids_src1 holds the forward map, or the inverse map (token slot -> compact row) for quant dedup
|
||||||
|
if (write_inverse) {
|
||||||
|
ids_src1[it*n_expert_used + iex_used] = nex_prev + itc;
|
||||||
|
} else {
|
||||||
|
ids_src1[nex_prev + itc] = it*sis1 + iex_used % nchannels_y;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (threadIdx.x != 0) {
|
if (threadIdx.x != 0) {
|
||||||
@@ -118,7 +123,7 @@ static __global__ void mm_ids_helper(
|
|||||||
template <int n_expert_used_template>
|
template <int n_expert_used_template>
|
||||||
static void launch_mm_ids_helper(
|
static void launch_mm_ids_helper(
|
||||||
const int32_t * __restrict__ ids, int32_t * __restrict__ ids_src1, int32_t * __restrict__ ids_dst, int32_t * __restrict__ expert_bounds,
|
const int32_t * __restrict__ ids, int32_t * __restrict__ ids_src1, int32_t * __restrict__ ids_dst, int32_t * __restrict__ expert_bounds,
|
||||||
const int n_experts, const int n_tokens, const int n_expert_used_var, const int nchannels_y, const int si1, const int sis1, cudaStream_t stream) {
|
const int n_experts, const int n_tokens, const int n_expert_used_var, const int nchannels_y, const int si1, const int sis1, const bool write_inverse, cudaStream_t stream) {
|
||||||
GGML_ASSERT(n_tokens < (1 << 22) && "too few bits in mm_ids_helper_store");
|
GGML_ASSERT(n_tokens < (1 << 22) && "too few bits in mm_ids_helper_store");
|
||||||
GGML_ASSERT(n_expert_used_var < (1 << 10) && "too few bits in mm_ids_helper_store");
|
GGML_ASSERT(n_expert_used_var < (1 << 10) && "too few bits in mm_ids_helper_store");
|
||||||
|
|
||||||
@@ -132,33 +137,33 @@ static void launch_mm_ids_helper(
|
|||||||
const size_t nbytes_shared = n_tokens*sizeof(mm_ids_helper_store);
|
const size_t nbytes_shared = n_tokens*sizeof(mm_ids_helper_store);
|
||||||
GGML_ASSERT(nbytes_shared <= smpbo);
|
GGML_ASSERT(nbytes_shared <= smpbo);
|
||||||
mm_ids_helper<n_expert_used_template><<<num_blocks, block_size, nbytes_shared, stream>>>
|
mm_ids_helper<n_expert_used_template><<<num_blocks, block_size, nbytes_shared, stream>>>
|
||||||
(ids, ids_src1, ids_dst, expert_bounds, n_tokens, n_expert_used_var, nchannels_y, si1, sis1);
|
(ids, ids_src1, ids_dst, expert_bounds, n_tokens, n_expert_used_var, nchannels_y, si1, sis1, write_inverse);
|
||||||
}
|
}
|
||||||
|
|
||||||
void ggml_cuda_launch_mm_ids_helper(
|
void ggml_cuda_launch_mm_ids_helper(
|
||||||
const int32_t * __restrict__ ids, int32_t * __restrict__ ids_src1, int32_t * __restrict__ ids_dst, int32_t * __restrict__ expert_bounds,
|
const int32_t * __restrict__ ids, int32_t * __restrict__ ids_src1, int32_t * __restrict__ ids_dst, int32_t * __restrict__ expert_bounds,
|
||||||
const int n_experts, const int n_tokens, const int n_expert_used, const int nchannels_y, const int si1, const int sis1, cudaStream_t stream) {
|
const int n_experts, const int n_tokens, const int n_expert_used, const int nchannels_y, const int si1, const int sis1, const bool write_inverse, cudaStream_t stream) {
|
||||||
switch (n_expert_used) {
|
switch (n_expert_used) {
|
||||||
case 2:
|
case 2:
|
||||||
launch_mm_ids_helper< 2>(ids, ids_src1, ids_dst, expert_bounds, n_experts, n_tokens, n_expert_used, nchannels_y, si1, sis1, stream);
|
launch_mm_ids_helper< 2>(ids, ids_src1, ids_dst, expert_bounds, n_experts, n_tokens, n_expert_used, nchannels_y, si1, sis1, write_inverse, stream);
|
||||||
break;
|
break;
|
||||||
case 4:
|
case 4:
|
||||||
launch_mm_ids_helper< 4>(ids, ids_src1, ids_dst, expert_bounds, n_experts, n_tokens, n_expert_used, nchannels_y, si1, sis1, stream);
|
launch_mm_ids_helper< 4>(ids, ids_src1, ids_dst, expert_bounds, n_experts, n_tokens, n_expert_used, nchannels_y, si1, sis1, write_inverse, stream);
|
||||||
break;
|
break;
|
||||||
case 6:
|
case 6:
|
||||||
launch_mm_ids_helper< 6>(ids, ids_src1, ids_dst, expert_bounds, n_experts, n_tokens, n_expert_used, nchannels_y, si1, sis1, stream);
|
launch_mm_ids_helper< 6>(ids, ids_src1, ids_dst, expert_bounds, n_experts, n_tokens, n_expert_used, nchannels_y, si1, sis1, write_inverse, stream);
|
||||||
break;
|
break;
|
||||||
case 8:
|
case 8:
|
||||||
launch_mm_ids_helper< 8>(ids, ids_src1, ids_dst, expert_bounds, n_experts, n_tokens, n_expert_used, nchannels_y, si1, sis1, stream);
|
launch_mm_ids_helper< 8>(ids, ids_src1, ids_dst, expert_bounds, n_experts, n_tokens, n_expert_used, nchannels_y, si1, sis1, write_inverse, stream);
|
||||||
break;
|
break;
|
||||||
case 16:
|
case 16:
|
||||||
launch_mm_ids_helper<16>(ids, ids_src1, ids_dst, expert_bounds, n_experts, n_tokens, n_expert_used, nchannels_y, si1, sis1, stream);
|
launch_mm_ids_helper<16>(ids, ids_src1, ids_dst, expert_bounds, n_experts, n_tokens, n_expert_used, nchannels_y, si1, sis1, write_inverse, stream);
|
||||||
break;
|
break;
|
||||||
case 32:
|
case 32:
|
||||||
launch_mm_ids_helper<32>(ids, ids_src1, ids_dst, expert_bounds, n_experts, n_tokens, n_expert_used, nchannels_y, si1, sis1, stream);
|
launch_mm_ids_helper<32>(ids, ids_src1, ids_dst, expert_bounds, n_experts, n_tokens, n_expert_used, nchannels_y, si1, sis1, write_inverse, stream);
|
||||||
break;
|
break;
|
||||||
default:
|
default:
|
||||||
launch_mm_ids_helper< 0>(ids, ids_src1, ids_dst, expert_bounds, n_experts, n_tokens, n_expert_used, nchannels_y, si1, sis1, stream);
|
launch_mm_ids_helper< 0>(ids, ids_src1, ids_dst, expert_bounds, n_experts, n_tokens, n_expert_used, nchannels_y, si1, sis1, write_inverse, stream);
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,4 +2,4 @@
|
|||||||
|
|
||||||
void ggml_cuda_launch_mm_ids_helper(
|
void ggml_cuda_launch_mm_ids_helper(
|
||||||
const int32_t * ids, int32_t * ids_src1, int32_t * ids_dst, int32_t * expert_bounds,
|
const int32_t * ids, int32_t * ids_src1, int32_t * ids_dst, int32_t * expert_bounds,
|
||||||
int n_experts, int n_tokens, int n_expert_used, int nchannels_y, int si1, int sis1, cudaStream_t stream);
|
int n_experts, int n_tokens, int n_expert_used, int nchannels_y, int si1, int sis1, bool write_inverse, cudaStream_t stream);
|
||||||
|
|||||||
@@ -0,0 +1,383 @@
|
|||||||
|
static constexpr __host__ __device__ ggml_cuda_mmq_config ggml_cuda_mmq_get_config_ampere(ggml_type type, int J, bool fallback) {
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
return ggml_cuda_mmq_config(GGML_TYPE_COUNT, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, 256, false, true);
|
||||||
|
}
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
static constexpr __host__ __device__ ggml_cuda_mmq_config ggml_cuda_mmq_get_config_blackwell(ggml_type type, int J, bool fallback) {
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 1, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_FP4, MMQ_ITER_K_FP4, true, false);
|
||||||
|
|
||||||
|
return ggml_cuda_mmq_get_config_ampere(type, J, fallback);
|
||||||
|
}
|
||||||
@@ -0,0 +1,185 @@
|
|||||||
|
static constexpr __host__ __device__ ggml_cuda_mmq_config ggml_cuda_mmq_get_config_cdna(ggml_type type, int J, bool fallback) {
|
||||||
|
CASE(GGML_TYPE_Q1_0, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q2_0, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_0, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_1, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_0, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_1, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q8_0, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q2_K, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q3_K, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_K, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_K, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q6_K, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_MXFP4, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_NVFP4, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 512, 1, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 512, 1, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 512, 1, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, true, false);
|
||||||
|
|
||||||
|
return ggml_cuda_mmq_config(GGML_TYPE_COUNT, 512, 1, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, 256, false, true);
|
||||||
|
}
|
||||||
@@ -0,0 +1,273 @@
|
|||||||
|
static constexpr __host__ __device__ ggml_cuda_mmq_config ggml_cuda_mmq_get_config_pascal(ggml_type type, int J, bool fallback) {
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 64, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 64, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 64, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
return ggml_cuda_mmq_config(GGML_TYPE_COUNT, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, 256, false, true);
|
||||||
|
}
|
||||||
@@ -0,0 +1,273 @@
|
|||||||
|
static constexpr __host__ __device__ ggml_cuda_mmq_config ggml_cuda_mmq_get_config_rdna2(ggml_type type, int J, bool fallback) {
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 8, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 24, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 40, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
return ggml_cuda_mmq_config(GGML_TYPE_COUNT, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, 256, false, true);
|
||||||
|
}
|
||||||
@@ -0,0 +1,290 @@
|
|||||||
|
static constexpr __host__ __device__ ggml_cuda_mmq_config ggml_cuda_mmq_get_config_rdna3_5(ggml_type type, int J, bool fallback) {
|
||||||
|
CASE(GGML_TYPE_Q1_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_1, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_1, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q8_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q2_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q3_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q6_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_MXFP4, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_NVFP4, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
return ggml_cuda_mmq_config(GGML_TYPE_COUNT, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, 256, false, true);
|
||||||
|
}
|
||||||
@@ -0,0 +1,290 @@
|
|||||||
|
static constexpr __host__ __device__ ggml_cuda_mmq_config ggml_cuda_mmq_get_config_rdna3(ggml_type type, int J, bool fallback) {
|
||||||
|
CASE(GGML_TYPE_Q1_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_1, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_1, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q8_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q2_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q3_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q6_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_MXFP4, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_NVFP4, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
return ggml_cuda_mmq_config(GGML_TYPE_COUNT, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, 256, false, true);
|
||||||
|
}
|
||||||
@@ -0,0 +1,290 @@
|
|||||||
|
static constexpr __host__ __device__ ggml_cuda_mmq_config ggml_cuda_mmq_get_config_rdna4(ggml_type type, int J, bool fallback) {
|
||||||
|
CASE(GGML_TYPE_Q1_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q1_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_1, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_1, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_1, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_1, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q8_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q8_0, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q2_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q2_K, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q2_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q3_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q3_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q4_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q4_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q5_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q5_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_Q6_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_Q6_K, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q6_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ1_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XXS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_XS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ2_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q3_K, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_XXS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ3_S, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_XS, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_IQ4_NL, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_MXFP4, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_MXFP4, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_1, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
CASE(GGML_TYPE_NVFP4, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, true);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 128, 2, 64, 16, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 128, 2, 64, 32, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 128, 2, 64, 48, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 128, 2, 64, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 80, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 96, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 112, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
CASE(GGML_TYPE_NVFP4, 256, 2, 128, 128, GGML_CUDA_MMQ_SRAM_LAYOUT_NVFP4, MMQ_ITER_K, false, false);
|
||||||
|
|
||||||
|
return ggml_cuda_mmq_config(GGML_TYPE_COUNT, 256, 2, 128, 64, GGML_CUDA_MMQ_SRAM_LAYOUT_Q8_0, 256, false, true);
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+75
-70
@@ -3,11 +3,16 @@
|
|||||||
#include "quantize.cuh"
|
#include "quantize.cuh"
|
||||||
#include "mmid.cuh"
|
#include "mmid.cuh"
|
||||||
|
|
||||||
|
#include <cstdint>
|
||||||
|
|
||||||
static void ggml_cuda_mul_mat_q_switch_type(ggml_backend_cuda_context & ctx, const mmq_args & args, cudaStream_t stream) {
|
static void ggml_cuda_mul_mat_q_switch_type(ggml_backend_cuda_context & ctx, const mmq_args & args, cudaStream_t stream) {
|
||||||
switch (args.type_x) {
|
switch (args.type_x) {
|
||||||
case GGML_TYPE_Q1_0:
|
case GGML_TYPE_Q1_0:
|
||||||
mul_mat_q_case<GGML_TYPE_Q1_0>(ctx, args, stream);
|
mul_mat_q_case<GGML_TYPE_Q1_0>(ctx, args, stream);
|
||||||
break;
|
break;
|
||||||
|
case GGML_TYPE_Q2_0:
|
||||||
|
mul_mat_q_case<GGML_TYPE_Q2_0>(ctx, args, stream);
|
||||||
|
break;
|
||||||
case GGML_TYPE_Q4_0:
|
case GGML_TYPE_Q4_0:
|
||||||
mul_mat_q_case<GGML_TYPE_Q4_0>(ctx, args, stream);
|
mul_mat_q_case<GGML_TYPE_Q4_0>(ctx, args, stream);
|
||||||
break;
|
break;
|
||||||
@@ -23,12 +28,7 @@ static void ggml_cuda_mul_mat_q_switch_type(ggml_backend_cuda_context & ctx, con
|
|||||||
case GGML_TYPE_Q8_0:
|
case GGML_TYPE_Q8_0:
|
||||||
mul_mat_q_case<GGML_TYPE_Q8_0>(ctx, args, stream);
|
mul_mat_q_case<GGML_TYPE_Q8_0>(ctx, args, stream);
|
||||||
break;
|
break;
|
||||||
case GGML_TYPE_MXFP4:
|
// -----------------------------------------------------------------------
|
||||||
mul_mat_q_case<GGML_TYPE_MXFP4>(ctx, args, stream);
|
|
||||||
break;
|
|
||||||
case GGML_TYPE_NVFP4:
|
|
||||||
mul_mat_q_case<GGML_TYPE_NVFP4>(ctx, args, stream);
|
|
||||||
break;
|
|
||||||
case GGML_TYPE_Q2_K:
|
case GGML_TYPE_Q2_K:
|
||||||
mul_mat_q_case<GGML_TYPE_Q2_K>(ctx, args, stream);
|
mul_mat_q_case<GGML_TYPE_Q2_K>(ctx, args, stream);
|
||||||
break;
|
break;
|
||||||
@@ -44,6 +44,10 @@ static void ggml_cuda_mul_mat_q_switch_type(ggml_backend_cuda_context & ctx, con
|
|||||||
case GGML_TYPE_Q6_K:
|
case GGML_TYPE_Q6_K:
|
||||||
mul_mat_q_case<GGML_TYPE_Q6_K>(ctx, args, stream);
|
mul_mat_q_case<GGML_TYPE_Q6_K>(ctx, args, stream);
|
||||||
break;
|
break;
|
||||||
|
// -----------------------------------------------------------------------
|
||||||
|
case GGML_TYPE_IQ1_S:
|
||||||
|
mul_mat_q_case<GGML_TYPE_IQ1_S>(ctx, args, stream);
|
||||||
|
break;
|
||||||
case GGML_TYPE_IQ2_XXS:
|
case GGML_TYPE_IQ2_XXS:
|
||||||
mul_mat_q_case<GGML_TYPE_IQ2_XXS>(ctx, args, stream);
|
mul_mat_q_case<GGML_TYPE_IQ2_XXS>(ctx, args, stream);
|
||||||
break;
|
break;
|
||||||
@@ -59,15 +63,19 @@ static void ggml_cuda_mul_mat_q_switch_type(ggml_backend_cuda_context & ctx, con
|
|||||||
case GGML_TYPE_IQ3_S:
|
case GGML_TYPE_IQ3_S:
|
||||||
mul_mat_q_case<GGML_TYPE_IQ3_S>(ctx, args, stream);
|
mul_mat_q_case<GGML_TYPE_IQ3_S>(ctx, args, stream);
|
||||||
break;
|
break;
|
||||||
case GGML_TYPE_IQ1_S:
|
|
||||||
mul_mat_q_case<GGML_TYPE_IQ1_S>(ctx, args, stream);
|
|
||||||
break;
|
|
||||||
case GGML_TYPE_IQ4_XS:
|
case GGML_TYPE_IQ4_XS:
|
||||||
mul_mat_q_case<GGML_TYPE_IQ4_XS>(ctx, args, stream);
|
mul_mat_q_case<GGML_TYPE_IQ4_XS>(ctx, args, stream);
|
||||||
break;
|
break;
|
||||||
case GGML_TYPE_IQ4_NL:
|
case GGML_TYPE_IQ4_NL:
|
||||||
mul_mat_q_case<GGML_TYPE_IQ4_NL>(ctx, args, stream);
|
mul_mat_q_case<GGML_TYPE_IQ4_NL>(ctx, args, stream);
|
||||||
break;
|
break;
|
||||||
|
// -----------------------------------------------------------------------
|
||||||
|
case GGML_TYPE_MXFP4:
|
||||||
|
mul_mat_q_case<GGML_TYPE_MXFP4>(ctx, args, stream);
|
||||||
|
break;
|
||||||
|
case GGML_TYPE_NVFP4:
|
||||||
|
mul_mat_q_case<GGML_TYPE_NVFP4>(ctx, args, stream);
|
||||||
|
break;
|
||||||
default:
|
default:
|
||||||
GGML_ABORT("fatal error");
|
GGML_ABORT("fatal error");
|
||||||
break;
|
break;
|
||||||
@@ -118,24 +126,30 @@ void ggml_cuda_mul_mat_q(
|
|||||||
const int64_t s03 = src0->nb[3] / ts_src0;
|
const int64_t s03 = src0->nb[3] / ts_src0;
|
||||||
const int64_t s3 = dst->nb[3] / ts_dst;
|
const int64_t s3 = dst->nb[3] / ts_dst;
|
||||||
|
|
||||||
const bool use_stream_k = (GGML_CUDA_CC_IS_NVIDIA(cc) && ggml_cuda_highest_compiled_arch(cc) >= GGML_CUDA_CC_VOLTA)
|
const bool fallback = ne01 % 128 != 0;
|
||||||
|| GGML_CUDA_CC_IS_CDNA(cc);
|
|
||||||
|
|
||||||
// TODO: tighter pool buffer size vs q8 path
|
|
||||||
const bool use_native_fp4 = blackwell_mma_available(cc) && (src0->type == GGML_TYPE_MXFP4 || src0->type == GGML_TYPE_NVFP4);
|
const bool use_native_fp4 = blackwell_mma_available(cc) && (src0->type == GGML_TYPE_MXFP4 || src0->type == GGML_TYPE_NVFP4);
|
||||||
|
const size_t y_block_size = use_native_fp4 ? sizeof(block_fp4_mmq) : sizeof(block_q8_1_mmq);
|
||||||
|
const size_t y_values_per_block = use_native_fp4 ? QK_FP4_MMQ : QK8_1_MMQ;
|
||||||
|
|
||||||
if (!ids) {
|
if (!ids) {
|
||||||
const size_t nbytes_src1_q8_1 = ne13*ne12 * ne11*ne10_padded * sizeof(block_q8_1)/QK8_1 +
|
const size_t nbytes_src1_q8_1 = ne13*ne12 * ne11*ne10_padded * y_block_size/y_values_per_block +
|
||||||
get_mmq_x_max_host(cc)*sizeof(block_q8_1_mmq);
|
ggml_cuda_mmq_get_J_max(src0->type, fallback, cc, ne11) * sizeof(block_q8_1_mmq);
|
||||||
ggml_cuda_pool_alloc<char> src1_q8_1(ctx.pool(), nbytes_src1_q8_1);
|
ggml_cuda_pool_alloc<char> src1_q8_1(ctx.pool(), nbytes_src1_q8_1);
|
||||||
|
ggml_cuda_pool_alloc<float> src1_scale(ctx.pool());
|
||||||
|
if (src0->type == GGML_TYPE_NVFP4 && use_native_fp4) {
|
||||||
|
src1_scale.alloc(ne13*ne12*ne11);
|
||||||
|
}
|
||||||
|
|
||||||
{
|
{
|
||||||
const int64_t s11 = src1->nb[1] / ts_src1;
|
const int64_t s11 = src1->nb[1] / ts_src1;
|
||||||
const int64_t s12 = src1->nb[2] / ts_src1;
|
const int64_t s12 = src1->nb[2] / ts_src1;
|
||||||
const int64_t s13 = src1->nb[3] / ts_src1;
|
const int64_t s13 = src1->nb[3] / ts_src1;
|
||||||
if (use_native_fp4) {
|
if (use_native_fp4) {
|
||||||
|
static constexpr size_t align_float8 = 32;
|
||||||
|
const bool use_aligned_float8 = ggml_cuda_is_aligned(src1, align_float8);
|
||||||
static_assert(sizeof(block_fp4_mmq) == 4 * sizeof(block_q8_1));
|
static_assert(sizeof(block_fp4_mmq) == 4 * sizeof(block_q8_1));
|
||||||
quantize_mmq_fp4_cuda(src1_d, nullptr, src1_q8_1.get(), src0->type, ne10, s11, s12, s13, ne10_padded,
|
quantize_mmq_fp4_cuda(src1_d, nullptr, src1_q8_1.get(), src1_scale.ptr, src0->type, use_aligned_float8, ne10, s11, s12, s13, ne10_padded,
|
||||||
ne11, ne12, ne13, stream);
|
ne11, ne12, ne13, stream);
|
||||||
|
|
||||||
} else {
|
} else {
|
||||||
@@ -147,16 +161,17 @@ void ggml_cuda_mul_mat_q(
|
|||||||
|
|
||||||
// Stride depends on quantization format
|
// Stride depends on quantization format
|
||||||
const int64_t s12 = use_native_fp4 ?
|
const int64_t s12 = use_native_fp4 ?
|
||||||
ne11 * ne10_padded * sizeof(block_fp4_mmq) / (QK_K * sizeof(int)) : // block_fp4_mmq holds 256 values
|
ne11 * ne10_padded * sizeof(block_fp4_mmq) / (QK_FP4_MMQ * sizeof(int)) :
|
||||||
ne11 * ne10_padded * sizeof(block_q8_1) / (QK8_1 * sizeof(int));
|
ne11 * ne10_padded * sizeof(block_q8_1) / (QK8_1 * sizeof(int));
|
||||||
const int64_t s13 = ne12*s12;
|
const int64_t s13 = ne12*s12;
|
||||||
|
|
||||||
const mmq_args args = {
|
const mmq_args args = {
|
||||||
src0_d, src0->type, (const int *) src1_q8_1.ptr, nullptr, nullptr, dst_d,
|
src0_d, src0->type, (const int *) src1_q8_1.ptr, nullptr, nullptr, dst_d,
|
||||||
|
src0->type == GGML_TYPE_NVFP4 && use_native_fp4 ? src1_scale.ptr : nullptr,
|
||||||
ne00, ne01, ne1, s01, ne11, s1,
|
ne00, ne01, ne1, s01, ne11, s1,
|
||||||
ne02, ne12, s02, s12, s2,
|
ne02, ne12, s02, s12, s2,
|
||||||
ne03, ne13, s03, s13, s3,
|
ne03, ne13, s03, s13, s3,
|
||||||
use_stream_k, ne1};
|
ne1};
|
||||||
ggml_cuda_mul_mat_q_switch_type(ctx, args, stream);
|
ggml_cuda_mul_mat_q_switch_type(ctx, args, stream);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
@@ -173,19 +188,27 @@ void ggml_cuda_mul_mat_q(
|
|||||||
ggml_cuda_pool_alloc<int32_t> ids_dst(ctx.pool(), ne_get_rows);
|
ggml_cuda_pool_alloc<int32_t> ids_dst(ctx.pool(), ne_get_rows);
|
||||||
ggml_cuda_pool_alloc<int32_t> expert_bounds(ctx.pool(), ne02 + 1);
|
ggml_cuda_pool_alloc<int32_t> expert_bounds(ctx.pool(), ne02 + 1);
|
||||||
|
|
||||||
|
// gate/up activations are broadcast across experts (ne11 == 1): quantize each token once and
|
||||||
|
// scatter to its slots. ids_src1 then holds the inverse map (token slot -> compact row).
|
||||||
|
const bool dedup_bcast = ne11 == 1 && n_expert_used > 1;
|
||||||
|
|
||||||
{
|
{
|
||||||
GGML_ASSERT(ids->nb[0] == ggml_element_size(ids));
|
GGML_ASSERT(ids->nb[0] == ggml_element_size(ids));
|
||||||
const int si1 = ids->nb[1] / ggml_element_size(ids);
|
const int si1 = ids->nb[1] / ggml_element_size(ids);
|
||||||
const int sis1 = nb12 / nb11;
|
const int sis1 = nb12 / nb11;
|
||||||
|
|
||||||
ggml_cuda_launch_mm_ids_helper((const int32_t *) ids->data, ids_src1.get(), ids_dst.get(), expert_bounds.get(),
|
ggml_cuda_launch_mm_ids_helper((const int32_t *) ids->data, ids_src1.get(), ids_dst.get(), expert_bounds.get(),
|
||||||
ne02, ne12, n_expert_used, ne11, si1, sis1, stream);
|
ne02, ne12, n_expert_used, ne11, si1, sis1, /*write_inverse =*/ dedup_bcast, stream);
|
||||||
CUDA_CHECK(cudaGetLastError());
|
CUDA_CHECK(cudaGetLastError());
|
||||||
}
|
}
|
||||||
|
|
||||||
const size_t nbytes_src1_q8_1 = ne12*n_expert_used*ne10_padded * sizeof(block_q8_1)/QK8_1 +
|
const size_t nbytes_src1_q8_1 = ne12*n_expert_used*ne10_padded * y_block_size/y_values_per_block +
|
||||||
get_mmq_x_max_host(cc)*sizeof(block_q8_1_mmq);
|
ggml_cuda_mmq_get_J_max(src0->type, fallback, cc, ne11) * sizeof(block_q8_1_mmq);
|
||||||
ggml_cuda_pool_alloc<char> src1_q8_1(ctx.pool(), nbytes_src1_q8_1);
|
ggml_cuda_pool_alloc<char> src1_q8_1(ctx.pool(), nbytes_src1_q8_1);
|
||||||
|
ggml_cuda_pool_alloc<float> src1_scale(ctx.pool());
|
||||||
|
if (src0->type == GGML_TYPE_NVFP4 && use_native_fp4) {
|
||||||
|
src1_scale.alloc(ne12*n_expert_used);
|
||||||
|
}
|
||||||
|
|
||||||
const int64_t ne11_flat = ne12*n_expert_used;
|
const int64_t ne11_flat = ne12*n_expert_used;
|
||||||
const int64_t ne12_flat = 1;
|
const int64_t ne12_flat = 1;
|
||||||
@@ -197,8 +220,18 @@ void ggml_cuda_mul_mat_q(
|
|||||||
const int64_t s13 = src1->nb[3] / ts_src1;
|
const int64_t s13 = src1->nb[3] / ts_src1;
|
||||||
|
|
||||||
if (use_native_fp4) {
|
if (use_native_fp4) {
|
||||||
quantize_mmq_fp4_cuda(src1_d, ids_src1.get(), src1_q8_1.get(), src0->type, ne10, s11, s12, s13,
|
static constexpr size_t align_float8 = 32;
|
||||||
ne10_padded, ne11_flat, ne12_flat, ne13_flat, stream);
|
const bool use_aligned_float8 = ggml_cuda_is_aligned(src1, align_float8);
|
||||||
|
if (dedup_bcast) {
|
||||||
|
quantize_scatter_mmq_fp4_cuda(src1_d, ids_src1.get(), src1_q8_1.get(), src1_scale.ptr, src0->type, use_aligned_float8, ne10,
|
||||||
|
/*stride_token=*/s12, ne10_padded, ne12, ne11_flat, n_expert_used, stream);
|
||||||
|
} else {
|
||||||
|
quantize_mmq_fp4_cuda(src1_d, ids_src1.get(), src1_q8_1.get(), src1_scale.ptr, src0->type, use_aligned_float8, ne10, s11, s12, s13,
|
||||||
|
ne10_padded, ne11_flat, ne12_flat, ne13_flat, stream);
|
||||||
|
}
|
||||||
|
} else if (dedup_bcast) {
|
||||||
|
quantize_scatter_mmq_q8_1_cuda(src1_d, ids_src1.get(), src1_q8_1.get(), src0->type, ne10,
|
||||||
|
/*stride_token=*/s12, ne10_padded, ne12, ne11_flat, n_expert_used, stream);
|
||||||
} else {
|
} else {
|
||||||
quantize_mmq_q8_1_cuda(src1_d, ids_src1.get(), src1_q8_1.get(), src0->type, ne10, s11, s12, s13,
|
quantize_mmq_q8_1_cuda(src1_d, ids_src1.get(), src1_q8_1.get(), src0->type, ne10, s11, s12, s13,
|
||||||
ne10_padded, ne11_flat, ne12_flat, ne13_flat, stream);
|
ne10_padded, ne11_flat, ne12_flat, ne13_flat, stream);
|
||||||
@@ -206,64 +239,23 @@ void ggml_cuda_mul_mat_q(
|
|||||||
CUDA_CHECK(cudaGetLastError());
|
CUDA_CHECK(cudaGetLastError());
|
||||||
}
|
}
|
||||||
|
|
||||||
static_assert(QK_K == 8 * QK_MXFP4, "QK_K needs to be 8 * QK_MXFP4");
|
static_assert(QK_FP4_MMQ == 8 * QK_MXFP4, "QK_FP4_MMQ needs to be 8 * QK_MXFP4");
|
||||||
const int64_t s12 = use_native_fp4 ? ne11 * ne10_padded * sizeof(block_fp4_mmq) / (QK_K * sizeof(int)) :
|
const int64_t s12 = use_native_fp4 ? ne11 * ne10_padded * sizeof(block_fp4_mmq) / (QK_FP4_MMQ * sizeof(int)) :
|
||||||
ne11 * ne10_padded * sizeof(block_q8_1) / (QK8_1 * sizeof(int));
|
ne11 * ne10_padded * sizeof(block_q8_1) / (QK8_1 * sizeof(int));
|
||||||
const int64_t s13 = ne12*s12;
|
const int64_t s13 = ne12*s12;
|
||||||
|
|
||||||
// Note that ne02 is used instead of ne12 because the number of y channels determines the z dimension of the CUDA grid.
|
// Note that ne02 is used instead of ne12 because the number of y channels determines the z dimension of the CUDA grid.
|
||||||
const mmq_args args = {
|
const mmq_args args = {
|
||||||
src0_d, src0->type, (const int *) src1_q8_1.get(), ids_dst.get(), expert_bounds.get(), dst_d,
|
src0_d, src0->type, (const int *) src1_q8_1.get(), ids_dst.get(), expert_bounds.get(), dst_d,
|
||||||
|
src1_scale.ptr,
|
||||||
ne00, ne01, ne_get_rows, s01, ne_get_rows, s1,
|
ne00, ne01, ne_get_rows, s01, ne_get_rows, s1,
|
||||||
ne02, ne02, s02, s12, s2,
|
ne02, ne02, s02, s12, s2,
|
||||||
ne03, ne13, s03, s13, s3,
|
ne03, ne13, s03, s13, s3,
|
||||||
use_stream_k, ne12};
|
ne12};
|
||||||
|
|
||||||
ggml_cuda_mul_mat_q_switch_type(ctx, args, stream);
|
ggml_cuda_mul_mat_q_switch_type(ctx, args, stream);
|
||||||
}
|
}
|
||||||
|
|
||||||
void ggml_cuda_op_mul_mat_q(
|
|
||||||
ggml_backend_cuda_context & ctx,
|
|
||||||
const ggml_tensor * src0, const ggml_tensor * src1, ggml_tensor * dst, const char * src0_dd_i, const float * src1_ddf_i,
|
|
||||||
const char * src1_ddq_i, float * dst_dd_i, const int64_t row_low, const int64_t row_high, const int64_t src1_ncols,
|
|
||||||
const int64_t src1_padded_row_size, cudaStream_t stream) {
|
|
||||||
|
|
||||||
const int64_t ne00 = src0->ne[0];
|
|
||||||
|
|
||||||
const int64_t ne10 = src1->ne[0];
|
|
||||||
const int64_t ne11 = src1->ne[1];
|
|
||||||
GGML_ASSERT(ne10 % QK8_1 == 0);
|
|
||||||
|
|
||||||
const int64_t ne0 = dst->ne[0];
|
|
||||||
|
|
||||||
const int64_t row_diff = row_high - row_low;
|
|
||||||
const int64_t stride01 = ne00 / ggml_blck_size(src0->type);
|
|
||||||
|
|
||||||
const int id = ggml_cuda_get_device();
|
|
||||||
const int cc = ggml_cuda_info().devices[id].cc;
|
|
||||||
|
|
||||||
// the main device has a larger memory buffer to hold the results from all GPUs
|
|
||||||
// nrows_dst == nrows of the matrix that the kernel writes into
|
|
||||||
const int64_t nrows_dst = id == ctx.device ? ne0 : row_diff;
|
|
||||||
|
|
||||||
// The stream-k decomposition is only faster for recent NVIDIA GPUs.
|
|
||||||
// Also its fixup needs to allocate a temporary buffer in the memory pool.
|
|
||||||
// There are multiple parallel CUDA streams for src1_ncols != ne11 which would introduce a race condition for this buffer.
|
|
||||||
const bool use_stream_k = ((GGML_CUDA_CC_IS_NVIDIA(cc) && ggml_cuda_highest_compiled_arch(cc) >= GGML_CUDA_CC_VOLTA)
|
|
||||||
|| GGML_CUDA_CC_IS_CDNA(cc))
|
|
||||||
&& src1_ncols == ne11;
|
|
||||||
const mmq_args args = {
|
|
||||||
src0_dd_i, src0->type, (const int *) src1_ddq_i, nullptr, nullptr, dst_dd_i,
|
|
||||||
ne00, row_diff, src1_ncols, stride01, ne11, nrows_dst,
|
|
||||||
1, 1, 0, 0, 0,
|
|
||||||
1, 1, 0, 0, 0,
|
|
||||||
use_stream_k, src1_ncols};
|
|
||||||
|
|
||||||
ggml_cuda_mul_mat_q_switch_type(ctx, args, stream);
|
|
||||||
|
|
||||||
GGML_UNUSED_VARS(src1, dst, src1_ddf_i, src1_padded_row_size);
|
|
||||||
}
|
|
||||||
|
|
||||||
bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t n_experts) {
|
bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t n_experts) {
|
||||||
#ifdef GGML_CUDA_FORCE_CUBLAS
|
#ifdef GGML_CUDA_FORCE_CUBLAS
|
||||||
return false;
|
return false;
|
||||||
@@ -273,26 +265,30 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t
|
|||||||
|
|
||||||
switch (type) {
|
switch (type) {
|
||||||
case GGML_TYPE_Q1_0:
|
case GGML_TYPE_Q1_0:
|
||||||
|
case GGML_TYPE_Q2_0:
|
||||||
case GGML_TYPE_Q4_0:
|
case GGML_TYPE_Q4_0:
|
||||||
case GGML_TYPE_Q4_1:
|
case GGML_TYPE_Q4_1:
|
||||||
case GGML_TYPE_Q5_0:
|
case GGML_TYPE_Q5_0:
|
||||||
case GGML_TYPE_Q5_1:
|
case GGML_TYPE_Q5_1:
|
||||||
case GGML_TYPE_Q8_0:
|
case GGML_TYPE_Q8_0:
|
||||||
case GGML_TYPE_MXFP4:
|
// -------------------------------------------------
|
||||||
case GGML_TYPE_NVFP4:
|
|
||||||
case GGML_TYPE_Q2_K:
|
case GGML_TYPE_Q2_K:
|
||||||
case GGML_TYPE_Q3_K:
|
case GGML_TYPE_Q3_K:
|
||||||
case GGML_TYPE_Q4_K:
|
case GGML_TYPE_Q4_K:
|
||||||
case GGML_TYPE_Q5_K:
|
case GGML_TYPE_Q5_K:
|
||||||
case GGML_TYPE_Q6_K:
|
case GGML_TYPE_Q6_K:
|
||||||
|
// -------------------------------------------------
|
||||||
|
case GGML_TYPE_IQ1_S:
|
||||||
case GGML_TYPE_IQ2_XXS:
|
case GGML_TYPE_IQ2_XXS:
|
||||||
case GGML_TYPE_IQ2_XS:
|
case GGML_TYPE_IQ2_XS:
|
||||||
case GGML_TYPE_IQ2_S:
|
case GGML_TYPE_IQ2_S:
|
||||||
case GGML_TYPE_IQ3_XXS:
|
case GGML_TYPE_IQ3_XXS:
|
||||||
case GGML_TYPE_IQ3_S:
|
case GGML_TYPE_IQ3_S:
|
||||||
case GGML_TYPE_IQ1_S:
|
|
||||||
case GGML_TYPE_IQ4_XS:
|
case GGML_TYPE_IQ4_XS:
|
||||||
case GGML_TYPE_IQ4_NL:
|
case GGML_TYPE_IQ4_NL:
|
||||||
|
// -------------------------------------------------
|
||||||
|
case GGML_TYPE_MXFP4:
|
||||||
|
case GGML_TYPE_NVFP4:
|
||||||
mmq_supported = true;
|
mmq_supported = true;
|
||||||
break;
|
break;
|
||||||
default:
|
default:
|
||||||
@@ -304,6 +300,15 @@ bool ggml_cuda_should_use_mmq(enum ggml_type type, int cc, int64_t ne11, int64_t
|
|||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// MMQ tiles require at least 48 KiB per-block shared memory; fall back to BLAS otherwise.
|
||||||
|
{
|
||||||
|
const int id = ggml_cuda_get_device();
|
||||||
|
const size_t smpbo = ggml_cuda_info().devices[id].smpbo;
|
||||||
|
if (smpbo < 48 * 1024) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
if (turing_mma_available(cc)) {
|
if (turing_mma_available(cc)) {
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|||||||
+930
-3509
File diff suppressed because it is too large
Load Diff
@@ -10,6 +10,7 @@ typedef float (*vec_dot_q_cuda_t)(const void * __restrict__ vbq, const block_q8_
|
|||||||
static constexpr __device__ vec_dot_q_cuda_t get_vec_dot_q_cuda(ggml_type type) {
|
static constexpr __device__ vec_dot_q_cuda_t get_vec_dot_q_cuda(ggml_type type) {
|
||||||
switch (type) {
|
switch (type) {
|
||||||
case GGML_TYPE_Q1_0: return vec_dot_q1_0_q8_1;
|
case GGML_TYPE_Q1_0: return vec_dot_q1_0_q8_1;
|
||||||
|
case GGML_TYPE_Q2_0: return vec_dot_q2_0_q8_1;
|
||||||
case GGML_TYPE_Q4_0: return vec_dot_q4_0_q8_1;
|
case GGML_TYPE_Q4_0: return vec_dot_q4_0_q8_1;
|
||||||
case GGML_TYPE_Q4_1: return vec_dot_q4_1_q8_1;
|
case GGML_TYPE_Q4_1: return vec_dot_q4_1_q8_1;
|
||||||
case GGML_TYPE_Q5_0: return vec_dot_q5_0_q8_1;
|
case GGML_TYPE_Q5_0: return vec_dot_q5_0_q8_1;
|
||||||
@@ -38,6 +39,7 @@ static constexpr __device__ vec_dot_q_cuda_t get_vec_dot_q_cuda(ggml_type type)
|
|||||||
static constexpr __host__ __device__ int get_vdr_mmvq(ggml_type type) {
|
static constexpr __host__ __device__ int get_vdr_mmvq(ggml_type type) {
|
||||||
switch (type) {
|
switch (type) {
|
||||||
case GGML_TYPE_Q1_0: return VDR_Q1_0_Q8_1_MMVQ;
|
case GGML_TYPE_Q1_0: return VDR_Q1_0_Q8_1_MMVQ;
|
||||||
|
case GGML_TYPE_Q2_0: return VDR_Q2_0_Q8_1_MMVQ;
|
||||||
case GGML_TYPE_Q4_0: return VDR_Q4_0_Q8_1_MMVQ;
|
case GGML_TYPE_Q4_0: return VDR_Q4_0_Q8_1_MMVQ;
|
||||||
case GGML_TYPE_Q4_1: return VDR_Q4_1_Q8_1_MMVQ;
|
case GGML_TYPE_Q4_1: return VDR_Q4_1_Q8_1_MMVQ;
|
||||||
case GGML_TYPE_Q5_0: return VDR_Q5_0_Q8_1_MMVQ;
|
case GGML_TYPE_Q5_0: return VDR_Q5_0_Q8_1_MMVQ;
|
||||||
@@ -1010,6 +1012,12 @@ static void mul_mat_vec_q_switch_type(
|
|||||||
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
|
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
|
||||||
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, ids_stride, stream);
|
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, ids_stride, stream);
|
||||||
break;
|
break;
|
||||||
|
case GGML_TYPE_Q2_0:
|
||||||
|
mul_mat_vec_q_switch_ncols_dst<GGML_TYPE_Q2_0>
|
||||||
|
(vx, vy, ids, fusion, dst, ncols_x, nrows_x, ncols_dst, stride_row_x, stride_col_y, stride_col_dst,
|
||||||
|
nchannels_x, nchannels_y, nchannels_dst, stride_channel_x, stride_channel_y, stride_channel_dst,
|
||||||
|
nsamples_x, nsamples_dst, stride_sample_x, stride_sample_y, stride_sample_dst, ids_stride, stream);
|
||||||
|
break;
|
||||||
case GGML_TYPE_Q4_0:
|
case GGML_TYPE_Q4_0:
|
||||||
mul_mat_vec_q_switch_ncols_dst<GGML_TYPE_Q4_0>
|
mul_mat_vec_q_switch_ncols_dst<GGML_TYPE_Q4_0>
|
||||||
(vx, vy, ids, fusion, dst, ncols_x, nrows_x, ncols_dst, stride_row_x, stride_col_y, stride_col_dst,
|
(vx, vy, ids, fusion, dst, ncols_x, nrows_x, ncols_dst, stride_row_x, stride_col_y, stride_col_dst,
|
||||||
|
|||||||
+408
-161
@@ -1,6 +1,55 @@
|
|||||||
#include "quantize.cuh"
|
#include "quantize.cuh"
|
||||||
#include <cstdint>
|
#include <cstdint>
|
||||||
|
|
||||||
|
#if defined(BLACKWELL_MMA_AVAILABLE)
|
||||||
|
// this maps to 256-bit loads in PTX on supported devices,
|
||||||
|
// and otherwise falls back to 2 128-bit loads
|
||||||
|
struct __builtin_align__(32) float8 {
|
||||||
|
float x; float y; float z; float w;
|
||||||
|
float p; float q; float r; float s;
|
||||||
|
};
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#if CUDART_VERSION >= 12080
|
||||||
|
static __device__ __forceinline__ float nvfp4_native_scale_error(
|
||||||
|
const float vals[QK_NVFP4_SUB], const float inv_col_scale, const float inv_scale, const float scale) {
|
||||||
|
const float scale_dequant = 2.0f * scale;
|
||||||
|
float err = 0.0f;
|
||||||
|
|
||||||
|
#pragma unroll
|
||||||
|
for (int k = 0; k < QK_NVFP4_SUB; k += 4) {
|
||||||
|
const float v0 = vals[k + 0] * inv_col_scale;
|
||||||
|
const float v1 = vals[k + 1] * inv_col_scale;
|
||||||
|
const float v2 = vals[k + 2] * inv_col_scale;
|
||||||
|
const float v3 = vals[k + 3] * inv_col_scale;
|
||||||
|
|
||||||
|
const __nv_fp4x4_e2m1 q(make_float4(v0 * inv_scale, v1 * inv_scale, v2 * inv_scale, v3 * inv_scale));
|
||||||
|
const __nv_fp4x4_storage_t q_storage = q.__x;
|
||||||
|
const __nv_fp4x2_storage_t q_lo = static_cast<__nv_fp4x2_storage_t>(q_storage);
|
||||||
|
const __nv_fp4x2_storage_t q_hi = static_cast<__nv_fp4x2_storage_t>(q_storage >> 8U);
|
||||||
|
|
||||||
|
const __half2_raw hraw2_lo = __nv_cvt_fp4x2_to_halfraw2(q_lo, __NV_E2M1);
|
||||||
|
const __half2_raw hraw2_hi = __nv_cvt_fp4x2_to_halfraw2(q_hi, __NV_E2M1);
|
||||||
|
const __half2 h2_lo = static_cast<__half2>(hraw2_lo);
|
||||||
|
const __half2 h2_hi = static_cast<__half2>(hraw2_hi);
|
||||||
|
const float2 dq_lo = __half22float2(h2_lo);
|
||||||
|
const float2 dq_hi = __half22float2(h2_hi);
|
||||||
|
|
||||||
|
const float err0 = fabsf(v0) - fabsf(dq_lo.x) * scale_dequant;
|
||||||
|
const float err1 = fabsf(v1) - fabsf(dq_lo.y) * scale_dequant;
|
||||||
|
const float err2 = fabsf(v2) - fabsf(dq_hi.x) * scale_dequant;
|
||||||
|
const float err3 = fabsf(v3) - fabsf(dq_hi.y) * scale_dequant;
|
||||||
|
|
||||||
|
err = fmaf(err0, err0, err);
|
||||||
|
err = fmaf(err1, err1, err);
|
||||||
|
err = fmaf(err2, err2, err);
|
||||||
|
err = fmaf(err3, err3, err);
|
||||||
|
}
|
||||||
|
|
||||||
|
return err;
|
||||||
|
}
|
||||||
|
#endif // CUDART_VERSION >= 12080
|
||||||
|
|
||||||
__launch_bounds__(CUDA_QUANTIZE_BLOCK_SIZE, 1)
|
__launch_bounds__(CUDA_QUANTIZE_BLOCK_SIZE, 1)
|
||||||
static __global__ void quantize_q8_1(
|
static __global__ void quantize_q8_1(
|
||||||
const float * x_ptr, void * vy_ptr,
|
const float * x_ptr, void * vy_ptr,
|
||||||
@@ -74,97 +123,209 @@ __device__ __forceinline__ uint8_t compute_e8m0_scale(float amax) {
|
|||||||
return static_cast<uint8_t>(biased);
|
return static_cast<uint8_t>(biased);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// scatter: grid over tokens, quantize once, write to all the token's compact rows
|
||||||
|
template <bool scatter, bool use_aligned_float8>
|
||||||
static __global__ void quantize_mmq_nvfp4(
|
static __global__ void quantize_mmq_nvfp4(
|
||||||
const float * __restrict__ x, const int32_t * __restrict__ ids, void * __restrict__ vy,
|
const float * __restrict__ x, const int32_t * __restrict__ ids, void * __restrict__ vy, float * __restrict__ scale,
|
||||||
const int64_t ne00, const int64_t s01, const int64_t s02, const int64_t s03,
|
const int64_t ne00, const int64_t s01, const int64_t s02, const int64_t s03,
|
||||||
const int64_t ne0, const int64_t ne1, const int64_t ne2) {
|
const int64_t ne0, const int64_t ne1, const int64_t ne2, const int n_expert_used) {
|
||||||
#if defined(BLACKWELL_MMA_AVAILABLE)
|
#if defined(BLACKWELL_MMA_AVAILABLE)
|
||||||
|
|
||||||
const int64_t i0_base = ((int64_t) blockDim.x * blockIdx.y + threadIdx.x) * QK_NVFP4_SUB;
|
const int64_t blocks_per_col = (ne0 + QK_FP4_MMQ - 1) / QK_FP4_MMQ;
|
||||||
if (i0_base >= ne0) {
|
|
||||||
return;
|
int64_t base_idx;
|
||||||
|
if constexpr (scatter) {
|
||||||
|
base_idx = (int64_t) blockIdx.x * s02; // one physical row per token
|
||||||
|
} else {
|
||||||
|
const int64_t i2 = blockIdx.y % ne2;
|
||||||
|
const int64_t i3 = blockIdx.y / ne2;
|
||||||
|
const int64_t i01 = ids ? ids[blockIdx.x] : blockIdx.x;
|
||||||
|
base_idx = i3 * s03 + i2 * s02 + i01 * s01;
|
||||||
|
}
|
||||||
|
const float * __restrict__ x_row = x + base_idx;
|
||||||
|
|
||||||
|
float amax = 0.0f;
|
||||||
|
if constexpr (use_aligned_float8) {
|
||||||
|
for (int64_t i0 = 8 * threadIdx.x; i0 < ne00; i0 += 8 * blockDim.x) {
|
||||||
|
const float * x_base = x_row + i0;
|
||||||
|
const float8 v = reinterpret_cast<const float8 *>(x_base)[0];
|
||||||
|
amax = fmaxf(amax, fabsf(v.x));
|
||||||
|
amax = fmaxf(amax, fabsf(v.y));
|
||||||
|
amax = fmaxf(amax, fabsf(v.z));
|
||||||
|
amax = fmaxf(amax, fabsf(v.w));
|
||||||
|
amax = fmaxf(amax, fabsf(v.p));
|
||||||
|
amax = fmaxf(amax, fabsf(v.q));
|
||||||
|
amax = fmaxf(amax, fabsf(v.r));
|
||||||
|
amax = fmaxf(amax, fabsf(v.s));
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
for (int64_t i0 = threadIdx.x; i0 < ne00; i0 += blockDim.x) {
|
||||||
|
amax = fmaxf(amax, fabsf(x_row[i0]));
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
const int64_t i1 = blockIdx.x;
|
amax = warp_reduce_max<WARP_SIZE>(amax);
|
||||||
const int64_t i2 = blockIdx.z % ne2;
|
|
||||||
const int64_t i3 = blockIdx.z / ne2;
|
__shared__ float warp_amax[CUDA_QUANTIZE_BLOCK_SIZE_MMQ / WARP_SIZE];
|
||||||
const int64_t i01 = ids ? ids[i1] : i1;
|
const int lane = threadIdx.x % WARP_SIZE;
|
||||||
const int64_t k_block = i0_base / QK_K;
|
const int warp = threadIdx.x / WARP_SIZE;
|
||||||
const int64_t blocks_per_col = (ne0 + QK_K - 1) / QK_K;
|
|
||||||
if (k_block >= blocks_per_col) {
|
if (lane == 0) {
|
||||||
return;
|
warp_amax[warp] = amax;
|
||||||
}
|
}
|
||||||
|
__syncthreads();
|
||||||
|
|
||||||
const int64_t ib = blockIdx.z * ((int64_t) blocks_per_col * ne1) + k_block * ne1 + blockIdx.x;
|
if (warp == 0) {
|
||||||
block_fp4_mmq * y = (block_fp4_mmq *) vy;
|
amax = threadIdx.x < int(CUDA_QUANTIZE_BLOCK_SIZE_MMQ / WARP_SIZE) ? warp_amax[lane] : 0.0f;
|
||||||
block_fp4_mmq * yb = y + ib;
|
amax = warp_reduce_max<WARP_SIZE>(amax);
|
||||||
|
if (lane == 0) {
|
||||||
const int sub = (i0_base % QK_K) / QK_NVFP4_SUB;
|
warp_amax[0] = amax / (6.0f * 448.0f);
|
||||||
|
if constexpr (scatter) {
|
||||||
float vals_raw[QK_NVFP4_SUB];
|
|
||||||
float amax_raw = 0.0f;
|
|
||||||
const int64_t base_idx = i3 * s03 + i2 * s02 + i01 * s01;
|
|
||||||
#pragma unroll
|
#pragma unroll
|
||||||
for (int k = 0; k < QK_NVFP4_SUB; k++) {
|
for (int slot = 0; slot < n_expert_used; ++slot) {
|
||||||
const int64_t i00 = i0_base + k;
|
const int64_t i = ids[(int64_t) blockIdx.x * n_expert_used + slot];
|
||||||
if (i00 < ne00) {
|
scale[i] = warp_amax[0];
|
||||||
const float v = x[base_idx + i00];
|
}
|
||||||
vals_raw[k] = v;
|
} else {
|
||||||
amax_raw = fmaxf(amax_raw, fabsf(v));
|
scale[blockIdx.y * ne1 + blockIdx.x] = warp_amax[0];
|
||||||
} else {
|
}
|
||||||
vals_raw[k] = 0.0f;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
__syncthreads();
|
||||||
|
|
||||||
static constexpr int test_offsets[5] = { 0, -1, 1, -2, 2};
|
block_fp4_mmq * y = (block_fp4_mmq *) vy;
|
||||||
const int first_fp8_code = (int) ggml_cuda_fp32_to_ue4m3(amax_raw / 6.0f);
|
const int64_t n_subblocks = (ne0 + QK_NVFP4_SUB - 1) / QK_NVFP4_SUB;
|
||||||
|
|
||||||
float best_err = FLT_MAX;
|
for (int64_t isb = threadIdx.x; isb < n_subblocks; isb += blockDim.x) {
|
||||||
uint8_t fp8_code = 0;
|
const int64_t i0_base = isb * QK_NVFP4_SUB;
|
||||||
float subblock_scale = 0.0f;
|
const int64_t k_block = i0_base / QK_FP4_MMQ;
|
||||||
|
const int sub = (i0_base % QK_FP4_MMQ) / QK_NVFP4_SUB;
|
||||||
|
|
||||||
#pragma unroll // Check +/- 2 to find best code to reduce NVFP4 activation loss. Negligible overhead on Blackwell.
|
const float row_scale = warp_amax[0];
|
||||||
for (int i = 0; i < 5; i++) {
|
const float inv_col_scale = row_scale > 0.0f ? 1.0f / row_scale : 0.0f;
|
||||||
const int test_code = first_fp8_code + test_offsets[i];
|
|
||||||
if (test_code < 0 || test_code > 0x7e) {
|
float vals[QK_NVFP4_SUB];
|
||||||
continue;
|
if constexpr (use_aligned_float8) {
|
||||||
|
const float * x_base = x_row + i0_base;
|
||||||
|
const float8 v0 = i0_base + 7 < ne00 ? reinterpret_cast<const float8 *>(x_base)[0] : float8{0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f};
|
||||||
|
const float8 v1 = i0_base + 15 < ne00 ? reinterpret_cast<const float8 *>(x_base + 8)[0] : float8{0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f};
|
||||||
|
vals[0] = v0.x; vals[1] = v0.y; vals[2] = v0.z; vals[3] = v0.w;
|
||||||
|
vals[4] = v0.p; vals[5] = v0.q; vals[6] = v0.r; vals[7] = v0.s;
|
||||||
|
vals[8] = v1.x; vals[9] = v1.y; vals[10] = v1.z; vals[11] = v1.w;
|
||||||
|
vals[12] = v1.p; vals[13] = v1.q; vals[14] = v1.r; vals[15] = v1.s;
|
||||||
|
} else {
|
||||||
|
#pragma unroll
|
||||||
|
for (int k = 0; k < QK_NVFP4_SUB; ++k) {
|
||||||
|
const int64_t i00 = i0_base + k;
|
||||||
|
vals[k] = i00 < ne00 ? x_row[i00] : 0.0f;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
const uint8_t code = (uint8_t) test_code;
|
|
||||||
const float test_scale = ggml_cuda_ue4m3_to_fp32(code);
|
uint32_t q0 = 0;
|
||||||
const float test_inv_scale = test_scale > 0.0f ? 0.5f / test_scale : 0.0f;
|
uint32_t q1 = 0;
|
||||||
float cur_err = 0.0f;
|
|
||||||
|
float amax_sub = 0.0f;
|
||||||
#pragma unroll
|
#pragma unroll
|
||||||
for (int k = 0; k < QK_NVFP4_SUB; ++k) {
|
for (int k = 0; k < QK_NVFP4_SUB; ++k) {
|
||||||
const float v = vals_raw[k];
|
amax_sub = fmaxf(amax_sub, fabsf(vals[k] * inv_col_scale));
|
||||||
const uint8_t q = ggml_cuda_float_to_fp4_e2m1(v, test_inv_scale);
|
|
||||||
const float err_diff = fabsf(v) - fabsf(kvalues_mxfp4[q & 0x7]) * test_scale;
|
|
||||||
cur_err = fmaf(err_diff, err_diff, cur_err);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if (cur_err < best_err) {
|
static constexpr int test_offsets[5] = { 0, -1, 1, -2, 2 };
|
||||||
best_err = cur_err;
|
const int first_fp8_code = (int) ggml_cuda_fp32_to_ue4m3(amax_sub / 6.0f);
|
||||||
fp8_code = test_code;
|
|
||||||
subblock_scale = test_scale;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
const float inv_scale = subblock_scale > 0.0f ? 0.5f / subblock_scale : 0.0f;
|
uint8_t fp8_code = (uint8_t) first_fp8_code;
|
||||||
uint32_t q0 = 0;
|
float subblock_scale = ggml_cuda_ue4m3_to_fp32(fp8_code);
|
||||||
uint32_t q1 = 0;
|
float inv_scale_err = subblock_scale > 0.0f ? 0.5f / subblock_scale : 0.0f;
|
||||||
#pragma unroll // this is faster than the previous __nv_fp4x4_e2m1
|
#if CUDART_VERSION >= 12080
|
||||||
for (int k = 0; k < QK_NVFP4_SUB / 4; ++k) {
|
float best_err = nvfp4_native_scale_error(vals, inv_col_scale, inv_scale_err, subblock_scale);
|
||||||
q0 |= (uint32_t) ggml_cuda_float_to_fp4_e2m1(vals_raw[k + 0], inv_scale) << (8 * k);
|
|
||||||
q0 |= (uint32_t) ggml_cuda_float_to_fp4_e2m1(vals_raw[k + 8], inv_scale) << (8 * k + 4);
|
|
||||||
q1 |= (uint32_t) ggml_cuda_float_to_fp4_e2m1(vals_raw[k + 4], inv_scale) << (8 * k);
|
|
||||||
q1 |= (uint32_t) ggml_cuda_float_to_fp4_e2m1(vals_raw[k + 12], inv_scale) << (8 * k + 4);
|
|
||||||
}
|
|
||||||
|
|
||||||
uint32_t * yqs = reinterpret_cast<uint32_t *>(yb->qs);
|
|
||||||
yqs[2 * sub + 0] = q0;
|
|
||||||
yqs[2 * sub + 1] = q1;
|
|
||||||
reinterpret_cast<uint8_t *>(yb->d4)[sub] = fp8_code;
|
|
||||||
#else
|
#else
|
||||||
|
float best_err = 0.0f;
|
||||||
|
#pragma unroll
|
||||||
|
for (int k = 0; k < QK_NVFP4_SUB; ++k) {
|
||||||
|
const float v = vals[k] * inv_col_scale;
|
||||||
|
const uint8_t q = ggml_cuda_float_to_fp4_e2m1(v, inv_scale_err);
|
||||||
|
const float err_diff = fabsf(v) - fabsf(kvalues_fp4[q & 0x7]) * subblock_scale;
|
||||||
|
best_err = fmaf(err_diff, err_diff, best_err);
|
||||||
|
}
|
||||||
|
#endif // CUDART_VERSION >= 12080
|
||||||
|
|
||||||
|
#pragma unroll
|
||||||
|
for (int i = 1; i < 5; ++i) {
|
||||||
|
const int test_code = first_fp8_code + test_offsets[i];
|
||||||
|
if (test_code < 0 || test_code > 0x7e) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
const float test_scale = ggml_cuda_ue4m3_to_fp32((uint8_t) test_code);
|
||||||
|
const float test_inv_scale = test_scale > 0.0f ? 0.5f / test_scale : 0.0f;
|
||||||
|
#if CUDART_VERSION >= 12080
|
||||||
|
const float cur_err = nvfp4_native_scale_error(vals, inv_col_scale, test_inv_scale, test_scale);
|
||||||
|
#else
|
||||||
|
float cur_err = 0.0f;
|
||||||
|
#pragma unroll
|
||||||
|
for (int k = 0; k < QK_NVFP4_SUB; ++k) {
|
||||||
|
const float v = vals[k] * inv_col_scale;
|
||||||
|
const uint8_t q = ggml_cuda_float_to_fp4_e2m1(v, test_inv_scale);
|
||||||
|
const float err_diff = fabsf(v) - fabsf(kvalues_fp4[q & 0x7]) * test_scale;
|
||||||
|
cur_err = fmaf(err_diff, err_diff, cur_err);
|
||||||
|
}
|
||||||
|
#endif // CUDART_VERSION >= 12080
|
||||||
|
|
||||||
|
if (cur_err < best_err) {
|
||||||
|
best_err = cur_err;
|
||||||
|
fp8_code = (uint8_t) test_code;
|
||||||
|
subblock_scale = test_scale;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
#if CUDART_VERSION >= 12080
|
||||||
|
const float inv_scale = subblock_scale > 0.0f ? 0.5f / subblock_scale : 0.0f;
|
||||||
|
const float s = inv_col_scale * inv_scale;
|
||||||
|
|
||||||
|
__nv_fp4x4_e2m1 q0_lo(make_float4(vals[0] * s, vals[8] * s, vals[1] * s, vals[9] * s));
|
||||||
|
__nv_fp4x4_e2m1 q0_hi(make_float4(vals[2] * s, vals[10] * s, vals[3] * s, vals[11] * s));
|
||||||
|
__nv_fp4x4_e2m1 q1_lo(make_float4(vals[4] * s, vals[12] * s, vals[5] * s, vals[13] * s));
|
||||||
|
__nv_fp4x4_e2m1 q1_hi(make_float4(vals[6] * s, vals[14] * s, vals[7] * s, vals[15] * s));
|
||||||
|
|
||||||
|
const char2 q0_lo_c = *reinterpret_cast<char2 *>(&q0_lo);
|
||||||
|
const char2 q0_hi_c = *reinterpret_cast<char2 *>(&q0_hi);
|
||||||
|
const char2 q1_lo_c = *reinterpret_cast<char2 *>(&q1_lo);
|
||||||
|
const char2 q1_hi_c = *reinterpret_cast<char2 *>(&q1_hi);
|
||||||
|
|
||||||
|
q0 = uint32_t(uint8_t(q0_lo_c.x)) | (uint32_t(uint8_t(q0_lo_c.y)) << 8) |
|
||||||
|
(uint32_t(uint8_t(q0_hi_c.x)) << 16) | (uint32_t(uint8_t(q0_hi_c.y)) << 24);
|
||||||
|
q1 = uint32_t(uint8_t(q1_lo_c.x)) | (uint32_t(uint8_t(q1_lo_c.y)) << 8) |
|
||||||
|
(uint32_t(uint8_t(q1_hi_c.x)) << 16) | (uint32_t(uint8_t(q1_hi_c.y)) << 24);
|
||||||
|
#else
|
||||||
|
const float inv_scale = subblock_scale > 0.0f ? 0.5f / subblock_scale : 0.0f;
|
||||||
|
#pragma unroll
|
||||||
|
for (int k = 0; k < QK_NVFP4_SUB / 4; ++k) {
|
||||||
|
q0 |= uint32_t(ggml_cuda_float_to_fp4_e2m1(vals[k + 0] * inv_col_scale, inv_scale)) << (8 * k);
|
||||||
|
q0 |= uint32_t(ggml_cuda_float_to_fp4_e2m1(vals[k + 8] * inv_col_scale, inv_scale)) << (8 * k + 4);
|
||||||
|
q1 |= uint32_t(ggml_cuda_float_to_fp4_e2m1(vals[k + 4] * inv_col_scale, inv_scale)) << (8 * k);
|
||||||
|
q1 |= uint32_t(ggml_cuda_float_to_fp4_e2m1(vals[k + 12] * inv_col_scale, inv_scale)) << (8 * k + 4);
|
||||||
|
}
|
||||||
|
#endif // CUDART_VERSION >= 12080
|
||||||
|
|
||||||
|
if constexpr (scatter) {
|
||||||
|
#pragma unroll
|
||||||
|
for (int slot = 0; slot < n_expert_used; ++slot) {
|
||||||
|
const int64_t i = ids[(int64_t) blockIdx.x * n_expert_used + slot];
|
||||||
|
block_fp4_mmq * yb = y + (k_block * ne1 + i);
|
||||||
|
uint32_t * yqs = reinterpret_cast<uint32_t *>(yb->qs);
|
||||||
|
yqs[2 * sub + 0] = q0;
|
||||||
|
yqs[2 * sub + 1] = q1;
|
||||||
|
reinterpret_cast<uint8_t *>(yb->d4)[sub] = fp8_code;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
block_fp4_mmq * yb = y + (blockIdx.y * ((int64_t) blocks_per_col * ne1) + k_block * ne1 + blockIdx.x);
|
||||||
|
uint32_t * yqs = reinterpret_cast<uint32_t *>(yb->qs);
|
||||||
|
yqs[2 * sub + 0] = q0;
|
||||||
|
yqs[2 * sub + 1] = q1;
|
||||||
|
reinterpret_cast<uint8_t *>(yb->d4)[sub] = fp8_code;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
#else
|
||||||
|
GGML_UNUSED_VARS(x, ids, vy, scale, ne00, s01, s02, s03, ne0, ne1, ne2, n_expert_used);
|
||||||
NO_DEVICE_CODE; // This is for Blackwell NVFP4 activations only.
|
NO_DEVICE_CODE; // This is for Blackwell NVFP4 activations only.
|
||||||
#endif // defined(BLACKWELL_MMA_AVAILABLE)
|
#endif // defined(BLACKWELL_MMA_AVAILABLE)
|
||||||
|
|
||||||
@@ -172,6 +333,8 @@ static __global__ void quantize_mmq_nvfp4(
|
|||||||
|
|
||||||
// quantize values in the format mxfp4 is stored which is interleaved nibbles
|
// quantize values in the format mxfp4 is stored which is interleaved nibbles
|
||||||
// i.e. a block a0-a31 is represented as a0a16,a1a17 ...a15a31
|
// i.e. a block a0-a31 is represented as a0a16,a1a17 ...a15a31
|
||||||
|
// scatter: grid over tokens, quantize once, write to all the token's compact rows
|
||||||
|
template <bool scatter>
|
||||||
static __global__ void quantize_mmq_mxfp4(const float * __restrict__ x,
|
static __global__ void quantize_mmq_mxfp4(const float * __restrict__ x,
|
||||||
const int32_t * __restrict__ ids,
|
const int32_t * __restrict__ ids,
|
||||||
void * __restrict__ vy,
|
void * __restrict__ vy,
|
||||||
@@ -181,7 +344,8 @@ static __global__ void quantize_mmq_mxfp4(const float * __restrict__ x,
|
|||||||
const int64_t s03,
|
const int64_t s03,
|
||||||
const int64_t ne0,
|
const int64_t ne0,
|
||||||
const int ne1,
|
const int ne1,
|
||||||
const int ne2) {
|
const int ne2,
|
||||||
|
const int n_expert_used) {
|
||||||
constexpr int vals_per_scale = 32;
|
constexpr int vals_per_scale = 32;
|
||||||
constexpr int vals_per_warp = 2 * vals_per_scale; // Each warp processes 2 blocks of 32 = 64 values
|
constexpr int vals_per_warp = 2 * vals_per_scale; // Each warp processes 2 blocks of 32 = 64 values
|
||||||
|
|
||||||
@@ -196,30 +360,27 @@ static __global__ void quantize_mmq_mxfp4(const float * __restrict__ x,
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
const int64_t i1 = blockIdx.x;
|
const int64_t block_fp4_mmq_size = QK_FP4_MMQ;
|
||||||
const int64_t i2 = blockIdx.z % ne2;
|
const int64_t k_block = warp_start_offset / block_fp4_mmq_size;
|
||||||
const int64_t i3 = blockIdx.z / ne2;
|
|
||||||
|
|
||||||
ggml_cuda_pdl_sync();
|
|
||||||
const int64_t i01 = ids ? ids[i1] : i1;
|
|
||||||
const int64_t i02 = i2;
|
|
||||||
const int64_t i03 = i3;
|
|
||||||
|
|
||||||
block_fp4_mmq * y = (block_fp4_mmq *) vy;
|
|
||||||
|
|
||||||
const int64_t block_fp4_mmq_size = 8 * QK_MXFP4; // 256 values
|
|
||||||
const int64_t ib0 = blockIdx.z * ((int64_t) ne1 * (ne0 / block_fp4_mmq_size));
|
|
||||||
const int64_t ib = ib0 + (warp_start_offset / block_fp4_mmq_size) * ne1 + blockIdx.x;
|
|
||||||
const int64_t quad_idx_in_block = (warp_start_offset % block_fp4_mmq_size) / vals_per_warp;
|
const int64_t quad_idx_in_block = (warp_start_offset % block_fp4_mmq_size) / vals_per_warp;
|
||||||
|
|
||||||
const int group_id = lane_id_32 / 4;
|
const int group_id = lane_id_32 / 4;
|
||||||
const int lane_in_group = lane_id_32 % 4;
|
const int lane_in_group = lane_id_32 % 4;
|
||||||
const int base = group_id * 2;
|
const int base = group_id * 2;
|
||||||
char2 * yqs2 = (char2 *) y[ib].qs;
|
|
||||||
|
|
||||||
const int64_t base_pos = i03 * s03 + i02 * s02 + i01 * s01;
|
ggml_cuda_pdl_sync();
|
||||||
|
int64_t base_pos;
|
||||||
|
if constexpr (scatter) {
|
||||||
|
base_pos = (int64_t) blockIdx.x * s02; // one physical row per token
|
||||||
|
} else {
|
||||||
|
const int64_t i2 = blockIdx.z % ne2;
|
||||||
|
const int64_t i3 = blockIdx.z / ne2;
|
||||||
|
const int64_t i01 = ids ? ids[blockIdx.x] : blockIdx.x;
|
||||||
|
base_pos = i3 * s03 + i2 * s02 + i01 * s01;
|
||||||
|
}
|
||||||
|
|
||||||
uint8_t scales[2];
|
uint8_t scales[2];
|
||||||
|
char2 packed[2];
|
||||||
|
|
||||||
#pragma unroll
|
#pragma unroll
|
||||||
for (int b = 0; b < 2; ++b) {
|
for (int b = 0; b < 2; ++b) {
|
||||||
@@ -244,11 +405,8 @@ static __global__ void quantize_mmq_mxfp4(const float * __restrict__ x,
|
|||||||
const float val2 = __shfl_sync(0xFFFFFFFF, scaled_val, base + 1, WARP_SIZE);
|
const float val2 = __shfl_sync(0xFFFFFFFF, scaled_val, base + 1, WARP_SIZE);
|
||||||
const float val3 = __shfl_sync(0xFFFFFFFF, scaled_val, base + 17, WARP_SIZE);
|
const float val3 = __shfl_sync(0xFFFFFFFF, scaled_val, base + 17, WARP_SIZE);
|
||||||
|
|
||||||
if (lane_in_group == 0) {
|
__nv_fp4x4_e2m1 fp4_packed(make_float4(val0, val1, val2, val3));
|
||||||
__nv_fp4x4_e2m1 fp4_packed(make_float4(val0, val1, val2, val3));
|
packed[b] = *(char2 *) &fp4_packed;
|
||||||
|
|
||||||
yqs2[quad_idx_in_block * 16 + b * 8 + group_id] = *(char2 *) &fp4_packed;
|
|
||||||
}
|
|
||||||
#else
|
#else
|
||||||
// Fallback: manual FP4 conversion using LUT
|
// Fallback: manual FP4 conversion using LUT
|
||||||
const uint8_t q_val = ggml_cuda_float_to_fp4_e2m1(xi, inv_s);
|
const uint8_t q_val = ggml_cuda_float_to_fp4_e2m1(xi, inv_s);
|
||||||
@@ -258,26 +416,49 @@ static __global__ void quantize_mmq_mxfp4(const float * __restrict__ x,
|
|||||||
const uint8_t q_hi_0 = __shfl_sync(0xFFFFFFFF, q_val, base + 16, WARP_SIZE);
|
const uint8_t q_hi_0 = __shfl_sync(0xFFFFFFFF, q_val, base + 16, WARP_SIZE);
|
||||||
const uint8_t q_hi_1 = __shfl_sync(0xFFFFFFFF, q_val, base + 17, WARP_SIZE);
|
const uint8_t q_hi_1 = __shfl_sync(0xFFFFFFFF, q_val, base + 17, WARP_SIZE);
|
||||||
|
|
||||||
if (lane_in_group == 0) {
|
char2 q;
|
||||||
char2 q;
|
q.x = (q_hi_0 << 4) | q_lo_0;
|
||||||
q.x = (q_hi_0 << 4) | q_lo_0;
|
q.y = (q_hi_1 << 4) | q_lo_1;
|
||||||
q.y = (q_hi_1 << 4) | q_lo_1;
|
packed[b] = q;
|
||||||
yqs2[quad_idx_in_block * 16 + b * 8 + group_id] = q;
|
|
||||||
}
|
|
||||||
#endif // CUDART_VERSION >= 12080
|
#endif // CUDART_VERSION >= 12080
|
||||||
}
|
}
|
||||||
|
|
||||||
if (lane_id_32 == 0) {
|
block_fp4_mmq * y = (block_fp4_mmq *) vy;
|
||||||
// Store 2 scales packed into 1 uint32
|
if constexpr (scatter) {
|
||||||
y[ib].d4[quad_idx_in_block] = (scales[1] << 8) | scales[0];
|
#pragma unroll
|
||||||
|
for (int slot = 0; slot < n_expert_used; ++slot) {
|
||||||
|
const int64_t i = ids[(int64_t) blockIdx.x * n_expert_used + slot];
|
||||||
|
block_fp4_mmq * yb = y + (k_block * ne1 + i);
|
||||||
|
char2 * yqs2 = (char2 *) yb->qs;
|
||||||
|
if (lane_in_group == 0) {
|
||||||
|
yqs2[quad_idx_in_block * 16 + 0 * 8 + group_id] = packed[0];
|
||||||
|
yqs2[quad_idx_in_block * 16 + 1 * 8 + group_id] = packed[1];
|
||||||
|
}
|
||||||
|
if (lane_id_32 == 0) {
|
||||||
|
yb->d4[quad_idx_in_block] = (scales[1] << 8) | scales[0];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
const int64_t ib0 = blockIdx.z * ((int64_t) ne1 * (ne0 / block_fp4_mmq_size));
|
||||||
|
block_fp4_mmq * yb = y + (ib0 + k_block * ne1 + blockIdx.x);
|
||||||
|
char2 * yqs2 = (char2 *) yb->qs;
|
||||||
|
if (lane_in_group == 0) {
|
||||||
|
yqs2[quad_idx_in_block * 16 + 0 * 8 + group_id] = packed[0];
|
||||||
|
yqs2[quad_idx_in_block * 16 + 1 * 8 + group_id] = packed[1];
|
||||||
|
}
|
||||||
|
if (lane_id_32 == 0) {
|
||||||
|
yb->d4[quad_idx_in_block] = (scales[1] << 8) | scales[0];
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
GGML_UNUSED(n_expert_used);
|
||||||
}
|
}
|
||||||
|
|
||||||
template <mmq_q8_1_ds_layout ds_layout>
|
// scatter: grid over tokens, quantize once, write to all the token's compact rows
|
||||||
|
template <mmq_q8_1_ds_layout ds_layout, bool scatter>
|
||||||
static __global__ void quantize_mmq_q8_1(
|
static __global__ void quantize_mmq_q8_1(
|
||||||
const float * __restrict__ x, const int32_t * __restrict__ ids, void * __restrict__ vy,
|
const float * __restrict__ x, const int32_t * __restrict__ ids, void * __restrict__ vy,
|
||||||
const int64_t ne00, const int64_t s01, const int64_t s02, const int64_t s03,
|
const int64_t ne00, const int64_t s01, const int64_t s02, const int64_t s03,
|
||||||
const int64_t ne0, const int ne1, const int ne2) {
|
const int64_t ne0, const int ne1, const int ne2, const int n_expert_used) {
|
||||||
|
|
||||||
constexpr int vals_per_scale = ds_layout == MMQ_Q8_1_DS_LAYOUT_D2S6 ? 64 : 32;
|
constexpr int vals_per_scale = ds_layout == MMQ_Q8_1_DS_LAYOUT_D2S6 ? 64 : 32;
|
||||||
constexpr int vals_per_sum = ds_layout == MMQ_Q8_1_DS_LAYOUT_D2S6 ? 16 : 32;
|
constexpr int vals_per_sum = ds_layout == MMQ_Q8_1_DS_LAYOUT_D2S6 ? 16 : 32;
|
||||||
@@ -288,26 +469,27 @@ static __global__ void quantize_mmq_q8_1(
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
const int64_t i1 = blockIdx.x;
|
|
||||||
const int64_t i2 = blockIdx.z % ne2;
|
|
||||||
const int64_t i3 = blockIdx.z / ne2;
|
|
||||||
|
|
||||||
const int64_t i00 = i0;
|
const int64_t i00 = i0;
|
||||||
ggml_cuda_pdl_sync();
|
ggml_cuda_pdl_sync();
|
||||||
const int64_t i01 = ids ? ids[i1] : i1;
|
|
||||||
const int64_t i02 = i2;
|
int64_t base_idx;
|
||||||
const int64_t i03 = i3;
|
if constexpr (scatter) {
|
||||||
|
base_idx = (int64_t) blockIdx.x * s02; // one physical row per token
|
||||||
|
} else {
|
||||||
|
const int64_t i2 = blockIdx.z % ne2;
|
||||||
|
const int64_t i3 = blockIdx.z / ne2;
|
||||||
|
const int64_t i01 = ids ? ids[blockIdx.x] : blockIdx.x;
|
||||||
|
base_idx = i3*s03 + i2*s02 + i01*s01;
|
||||||
|
}
|
||||||
|
|
||||||
const float4 * x4 = (const float4 *) x;
|
const float4 * x4 = (const float4 *) x;
|
||||||
|
|
||||||
block_q8_1_mmq * y = (block_q8_1_mmq *) vy;
|
block_q8_1_mmq * y = (block_q8_1_mmq *) vy;
|
||||||
|
|
||||||
const int64_t ib0 = blockIdx.z*((int64_t)gridDim.x*gridDim.y*blockDim.x/QK8_1); // first block of channel
|
const int64_t k_block = i0 / QK8_1_MMQ; // column block in the channel
|
||||||
const int64_t ib = ib0 + (i0 / (4*QK8_1))*ne1 + blockIdx.x; // block index in channel
|
const int64_t iqs = i0 % QK8_1_MMQ; // quant index in block
|
||||||
const int64_t iqs = i0 % (4*QK8_1); // quant index in block
|
|
||||||
|
|
||||||
// Load 4 floats per thread and calculate max. abs. value between them:
|
// Load 4 floats per thread and calculate max. abs. value between them:
|
||||||
const float4 xi = i0 < ne00 ? x4[(i03*s03 + i02*s02 + i01*s01 + i00)/4] : make_float4(0.0f, 0.0f, 0.0f, 0.0f);
|
const float4 xi = i0 < ne00 ? x4[(base_idx + i00)/4] : make_float4(0.0f, 0.0f, 0.0f, 0.0f);
|
||||||
float amax = fabsf(xi.x);
|
float amax = fabsf(xi.x);
|
||||||
amax = fmaxf(amax, fabsf(xi.y));
|
amax = fmaxf(amax, fabsf(xi.y));
|
||||||
amax = fmaxf(amax, fabsf(xi.z));
|
amax = fmaxf(amax, fabsf(xi.z));
|
||||||
@@ -336,40 +518,41 @@ static __global__ void quantize_mmq_q8_1(
|
|||||||
q.y = roundf(xi.y*d_inv);
|
q.y = roundf(xi.y*d_inv);
|
||||||
q.z = roundf(xi.z*d_inv);
|
q.z = roundf(xi.z*d_inv);
|
||||||
q.w = roundf(xi.w*d_inv);
|
q.w = roundf(xi.w*d_inv);
|
||||||
|
|
||||||
// Write back 4 int8 values as a single 32 bit value for better memory bandwidth:
|
|
||||||
char4 * yqs4 = (char4 *) y[ib].qs;
|
|
||||||
yqs4[iqs/4] = q;
|
|
||||||
|
|
||||||
if (ds_layout == MMQ_Q8_1_DS_LAYOUT_D2S6) {
|
|
||||||
if (iqs % 16 != 0 || iqs >= 96) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
y[ib].d2s6[2 + iqs/16] = sum;
|
|
||||||
|
|
||||||
if (iqs % 64 != 0) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
const float d = 1.0f / d_inv;
|
|
||||||
|
|
||||||
y[ib].d2s6[iqs/64] = d;
|
|
||||||
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (iqs % 32 != 0) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
const float d = 1.0f / d_inv;
|
const float d = 1.0f / d_inv;
|
||||||
|
|
||||||
if (ds_layout == MMQ_Q8_1_DS_LAYOUT_DS4) {
|
// write the block once (normal) or to each of the token's compact rows (scatter)
|
||||||
y[ib].ds4[iqs/32] = make_half2(d, sum);
|
const int nwrite = scatter ? n_expert_used : 1;
|
||||||
} else {
|
#pragma unroll
|
||||||
y[ib].d4[iqs/32] = d;
|
for (int slot = 0; slot < nwrite; ++slot) {
|
||||||
|
int64_t ib;
|
||||||
|
if constexpr (scatter) {
|
||||||
|
const int64_t i = ids[(int64_t) blockIdx.x * n_expert_used + slot];
|
||||||
|
ib = k_block*ne1 + i;
|
||||||
|
} else {
|
||||||
|
const int64_t ib0 = blockIdx.z*((int64_t)gridDim.x*gridDim.y*blockDim.x/QK8_1); // first block of channel
|
||||||
|
ib = ib0 + k_block*ne1 + blockIdx.x;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Write back 4 int8 values as a single 32 bit value for better memory bandwidth:
|
||||||
|
char4 * yqs4 = (char4 *) y[ib].qs;
|
||||||
|
yqs4[iqs/4] = q;
|
||||||
|
|
||||||
|
if (ds_layout == MMQ_Q8_1_DS_LAYOUT_D2S6) {
|
||||||
|
if (iqs % 16 == 0 && iqs < 96) {
|
||||||
|
y[ib].d2s6[2 + iqs/16] = sum;
|
||||||
|
if (iqs % 64 == 0) {
|
||||||
|
y[ib].d2s6[iqs/64] = d;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else if (iqs % 32 == 0) {
|
||||||
|
if (ds_layout == MMQ_Q8_1_DS_LAYOUT_DS4) {
|
||||||
|
y[ib].ds4[iqs/32] = make_half2(d, sum);
|
||||||
|
} else {
|
||||||
|
y[ib].d4[iqs/32] = d;
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
GGML_UNUSED(n_expert_used);
|
||||||
}
|
}
|
||||||
|
|
||||||
void quantize_row_q8_1_cuda(
|
void quantize_row_q8_1_cuda(
|
||||||
@@ -394,7 +577,7 @@ void quantize_mmq_q8_1_cuda(
|
|||||||
const int64_t ne00, const int64_t s01, const int64_t s02, const int64_t s03,
|
const int64_t ne00, const int64_t s01, const int64_t s02, const int64_t s03,
|
||||||
const int64_t ne0, const int64_t ne1, const int64_t ne2, const int64_t ne3, cudaStream_t stream) {
|
const int64_t ne0, const int64_t ne1, const int64_t ne2, const int64_t ne3, cudaStream_t stream) {
|
||||||
GGML_ASSERT(ne00 % 4 == 0);
|
GGML_ASSERT(ne00 % 4 == 0);
|
||||||
GGML_ASSERT(ne0 % (4*QK8_1) == 0);
|
GGML_ASSERT(ne0 % QK8_1_MMQ == 0);
|
||||||
|
|
||||||
// ne1 tends to assume the highest values, therefore use it as the "x" dimension of the CUDA grid:
|
// ne1 tends to assume the highest values, therefore use it as the "x" dimension of the CUDA grid:
|
||||||
const int64_t block_num_y = (ne0 + 4*CUDA_QUANTIZE_BLOCK_SIZE_MMQ - 1) / (4*CUDA_QUANTIZE_BLOCK_SIZE_MMQ);
|
const int64_t block_num_y = (ne0 + 4*CUDA_QUANTIZE_BLOCK_SIZE_MMQ - 1) / (4*CUDA_QUANTIZE_BLOCK_SIZE_MMQ);
|
||||||
@@ -402,16 +585,16 @@ void quantize_mmq_q8_1_cuda(
|
|||||||
const dim3 block_size(CUDA_QUANTIZE_BLOCK_SIZE_MMQ, 1, 1);
|
const dim3 block_size(CUDA_QUANTIZE_BLOCK_SIZE_MMQ, 1, 1);
|
||||||
switch (mmq_get_q8_1_ds_layout(type_src0)) {
|
switch (mmq_get_q8_1_ds_layout(type_src0)) {
|
||||||
case MMQ_Q8_1_DS_LAYOUT_D4:
|
case MMQ_Q8_1_DS_LAYOUT_D4:
|
||||||
quantize_mmq_q8_1<MMQ_Q8_1_DS_LAYOUT_D4>
|
quantize_mmq_q8_1<MMQ_Q8_1_DS_LAYOUT_D4, false>
|
||||||
<<<num_blocks, block_size, 0, stream>>>(x, ids, vy, ne00, s01, s02, s03, ne0, ne1, ne2);
|
<<<num_blocks, block_size, 0, stream>>>(x, ids, vy, ne00, s01, s02, s03, ne0, ne1, ne2, /*n_expert_used=*/0);
|
||||||
break;
|
break;
|
||||||
case MMQ_Q8_1_DS_LAYOUT_DS4:
|
case MMQ_Q8_1_DS_LAYOUT_DS4:
|
||||||
quantize_mmq_q8_1<MMQ_Q8_1_DS_LAYOUT_DS4>
|
quantize_mmq_q8_1<MMQ_Q8_1_DS_LAYOUT_DS4, false>
|
||||||
<<<num_blocks, block_size, 0, stream>>>(x, ids, vy, ne00, s01, s02, s03, ne0, ne1, ne2);
|
<<<num_blocks, block_size, 0, stream>>>(x, ids, vy, ne00, s01, s02, s03, ne0, ne1, ne2, /*n_expert_used=*/0);
|
||||||
break;
|
break;
|
||||||
case MMQ_Q8_1_DS_LAYOUT_D2S6:
|
case MMQ_Q8_1_DS_LAYOUT_D2S6:
|
||||||
quantize_mmq_q8_1<MMQ_Q8_1_DS_LAYOUT_D2S6>
|
quantize_mmq_q8_1<MMQ_Q8_1_DS_LAYOUT_D2S6, false>
|
||||||
<<<num_blocks, block_size, 0, stream>>>(x, ids, vy, ne00, s01, s02, s03, ne0, ne1, ne2);
|
<<<num_blocks, block_size, 0, stream>>>(x, ids, vy, ne00, s01, s02, s03, ne0, ne1, ne2, /*n_expert_used=*/0);
|
||||||
break;
|
break;
|
||||||
default:
|
default:
|
||||||
GGML_ABORT("fatal error");
|
GGML_ABORT("fatal error");
|
||||||
@@ -419,21 +602,85 @@ void quantize_mmq_q8_1_cuda(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// scatter=true reuses the quant kernel: grid over tokens, ids = inverse map (token slot -> compact row)
|
||||||
|
void quantize_scatter_mmq_q8_1_cuda(
|
||||||
|
const float * x, const int32_t * ids_src1_inv, void * vy, const ggml_type type_src0,
|
||||||
|
const int64_t ne00, const int64_t stride_token, const int64_t ne0,
|
||||||
|
const int64_t n_tokens, const int64_t nrows_dst, const int n_expert_used, cudaStream_t stream) {
|
||||||
|
GGML_ASSERT(ne00 % 4 == 0);
|
||||||
|
GGML_ASSERT(ne0 % QK8_1_MMQ == 0);
|
||||||
|
|
||||||
|
const int64_t block_num_y = (ne0 + 4*CUDA_QUANTIZE_BLOCK_SIZE_MMQ - 1) / (4*CUDA_QUANTIZE_BLOCK_SIZE_MMQ);
|
||||||
|
const dim3 num_blocks(n_tokens, block_num_y, 1);
|
||||||
|
const dim3 block_size(CUDA_QUANTIZE_BLOCK_SIZE_MMQ, 1, 1);
|
||||||
|
switch (mmq_get_q8_1_ds_layout(type_src0)) {
|
||||||
|
case MMQ_Q8_1_DS_LAYOUT_D4:
|
||||||
|
quantize_mmq_q8_1<MMQ_Q8_1_DS_LAYOUT_D4, true><<<num_blocks, block_size, 0, stream>>>(
|
||||||
|
x, ids_src1_inv, vy, ne00, /*s01=*/0, /*s02=*/stride_token, /*s03=*/0, ne0, /*ne1=*/(int) nrows_dst, /*ne2=*/1, n_expert_used);
|
||||||
|
break;
|
||||||
|
case MMQ_Q8_1_DS_LAYOUT_DS4:
|
||||||
|
quantize_mmq_q8_1<MMQ_Q8_1_DS_LAYOUT_DS4, true><<<num_blocks, block_size, 0, stream>>>(
|
||||||
|
x, ids_src1_inv, vy, ne00, /*s01=*/0, /*s02=*/stride_token, /*s03=*/0, ne0, /*ne1=*/(int) nrows_dst, /*ne2=*/1, n_expert_used);
|
||||||
|
break;
|
||||||
|
case MMQ_Q8_1_DS_LAYOUT_D2S6:
|
||||||
|
quantize_mmq_q8_1<MMQ_Q8_1_DS_LAYOUT_D2S6, true><<<num_blocks, block_size, 0, stream>>>(
|
||||||
|
x, ids_src1_inv, vy, ne00, /*s01=*/0, /*s02=*/stride_token, /*s03=*/0, ne0, /*ne1=*/(int) nrows_dst, /*ne2=*/1, n_expert_used);
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
GGML_ABORT("fatal error");
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// scatter=true reuses the quant kernels: grid over tokens, ids = inverse map (token slot -> compact row)
|
||||||
|
void quantize_scatter_mmq_fp4_cuda(
|
||||||
|
const float * x, const int32_t * ids_src1_inv, void * vy, float * scale, const ggml_type type_src0, const bool use_aligned_float8,
|
||||||
|
const int64_t ne00, const int64_t stride_token, const int64_t ne0,
|
||||||
|
const int64_t n_tokens, const int64_t nrows_dst, const int n_expert_used, cudaStream_t stream) {
|
||||||
|
GGML_ASSERT(ne0 > 0);
|
||||||
|
if (type_src0 == GGML_TYPE_NVFP4) {
|
||||||
|
GGML_ASSERT(scale);
|
||||||
|
GGML_ASSERT(ne00 % QK_NVFP4 == 0);
|
||||||
|
const dim3 block_size(CUDA_QUANTIZE_BLOCK_SIZE_MMQ, 1, 1);
|
||||||
|
const dim3 num_blocks(n_tokens, 1, 1);
|
||||||
|
if (use_aligned_float8) {
|
||||||
|
quantize_mmq_nvfp4<true, true><<<num_blocks, block_size, 0, stream>>>(
|
||||||
|
x, ids_src1_inv, vy, scale, ne00, /*s01=*/0, /*s02=*/stride_token, /*s03=*/0, ne0, /*ne1=*/nrows_dst, /*ne2=*/1, n_expert_used);
|
||||||
|
} else {
|
||||||
|
quantize_mmq_nvfp4<true, false><<<num_blocks, block_size, 0, stream>>>(
|
||||||
|
x, ids_src1_inv, vy, scale, ne00, /*s01=*/0, /*s02=*/stride_token, /*s03=*/0, ne0, /*ne1=*/nrows_dst, /*ne2=*/1, n_expert_used);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
GGML_ASSERT(type_src0 == GGML_TYPE_MXFP4);
|
||||||
|
constexpr int nwarps = 8;
|
||||||
|
constexpr int vals_per_block = nwarps * 2 * QK_MXFP4;
|
||||||
|
const int64_t block_num_y = (ne0 + vals_per_block - 1) / vals_per_block;
|
||||||
|
const dim3 block_size(WARP_SIZE, nwarps, 1);
|
||||||
|
const dim3 num_blocks(n_tokens, block_num_y, 1);
|
||||||
|
quantize_mmq_mxfp4<true><<<num_blocks, block_size, 0, stream>>>(
|
||||||
|
x, ids_src1_inv, vy, ne00, /*s01=*/0, /*s02=*/stride_token, /*s03=*/0, ne0, /*ne1=*/(int) nrows_dst, /*ne2=*/1, n_expert_used);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
void quantize_mmq_fp4_cuda(
|
void quantize_mmq_fp4_cuda(
|
||||||
const float * x, const int32_t * ids, void * vy, const ggml_type type_src0,
|
const float * x, const int32_t * ids, void * vy, float * scale, const ggml_type type_src0, const bool use_aligned_float8,
|
||||||
const int64_t ne00, const int64_t s01, const int64_t s02, const int64_t s03,
|
const int64_t ne00, const int64_t s01, const int64_t s02, const int64_t s03,
|
||||||
const int64_t ne0, const int64_t ne1, const int64_t ne2, const int64_t ne3, cudaStream_t stream) {
|
const int64_t ne0, const int64_t ne1, const int64_t ne2, const int64_t ne3, cudaStream_t stream) {
|
||||||
GGML_ASSERT(type_src0 == GGML_TYPE_MXFP4 || type_src0 == GGML_TYPE_NVFP4);
|
GGML_ASSERT(type_src0 == GGML_TYPE_MXFP4 || type_src0 == GGML_TYPE_NVFP4);
|
||||||
GGML_ASSERT(ne0 > 0);
|
GGML_ASSERT(ne0 > 0);
|
||||||
|
|
||||||
if (type_src0 == GGML_TYPE_NVFP4) {
|
if (type_src0 == GGML_TYPE_NVFP4) {
|
||||||
|
GGML_ASSERT(scale);
|
||||||
GGML_ASSERT(ne00 % QK_NVFP4 == 0);
|
GGML_ASSERT(ne00 % QK_NVFP4 == 0);
|
||||||
constexpr int nvfp4_block_size = 128;
|
const dim3 block_size(CUDA_QUANTIZE_BLOCK_SIZE_MMQ, 1, 1);
|
||||||
const int64_t block_num_y = (ne0 + QK_NVFP4_SUB * nvfp4_block_size - 1) / (QK_NVFP4_SUB * nvfp4_block_size);
|
const dim3 num_blocks(ne1, ne2 * ne3, 1);
|
||||||
const dim3 block_size(nvfp4_block_size, 1, 1);
|
if (use_aligned_float8) {
|
||||||
const dim3 num_blocks(ne1, block_num_y, ne2 * ne3);
|
quantize_mmq_nvfp4<false, true><<<num_blocks, block_size, 0, stream>>>(
|
||||||
quantize_mmq_nvfp4<<<num_blocks, block_size, 0, stream>>>(
|
x, ids, vy, scale, ne00, s01, s02, s03, ne0, ne1, ne2, /*n_expert_used=*/0);
|
||||||
x, ids, vy, ne00, s01, s02, s03, ne0, ne1, ne2);
|
} else {
|
||||||
|
quantize_mmq_nvfp4<false, false><<<num_blocks, block_size, 0, stream>>>(
|
||||||
|
x, ids, vy, scale, ne00, s01, s02, s03, ne0, ne1, ne2, /*n_expert_used=*/0);
|
||||||
|
}
|
||||||
} else {
|
} else {
|
||||||
GGML_ASSERT(ne0 % (2 * QK_MXFP4) == 0);
|
GGML_ASSERT(ne0 % (2 * QK_MXFP4) == 0);
|
||||||
|
|
||||||
@@ -445,6 +692,6 @@ void quantize_mmq_fp4_cuda(
|
|||||||
const dim3 num_blocks(ne1, block_num_y, ne2 * ne3);
|
const dim3 num_blocks(ne1, block_num_y, ne2 * ne3);
|
||||||
const dim3 block_size(WARP_SIZE, nwarps, 1);
|
const dim3 block_size(WARP_SIZE, nwarps, 1);
|
||||||
|
|
||||||
quantize_mmq_mxfp4<<<num_blocks, block_size, 0, stream>>>(x, ids, vy, ne00, s01, s02, s03, ne0, ne1, ne2);
|
quantize_mmq_mxfp4<false><<<num_blocks, block_size, 0, stream>>>(x, ids, vy, ne00, s01, s02, s03, ne0, ne1, ne2, /*n_expert_used=*/0);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -29,7 +29,9 @@ void quantize_mmq_q8_1_cuda(
|
|||||||
void quantize_mmq_fp4_cuda(const float * x,
|
void quantize_mmq_fp4_cuda(const float * x,
|
||||||
const int32_t * ids,
|
const int32_t * ids,
|
||||||
void * vy,
|
void * vy,
|
||||||
|
float * scale,
|
||||||
ggml_type type_src0,
|
ggml_type type_src0,
|
||||||
|
bool use_aligned_float8,
|
||||||
int64_t ne00,
|
int64_t ne00,
|
||||||
int64_t s01,
|
int64_t s01,
|
||||||
int64_t s02,
|
int64_t s02,
|
||||||
@@ -39,3 +41,30 @@ void quantize_mmq_fp4_cuda(const float * x,
|
|||||||
int64_t ne2,
|
int64_t ne2,
|
||||||
int64_t ne3,
|
int64_t ne3,
|
||||||
cudaStream_t stream);
|
cudaStream_t stream);
|
||||||
|
|
||||||
|
// quantize each token once and scatter the block to its compact rows (via the inverse map)
|
||||||
|
void quantize_scatter_mmq_fp4_cuda(const float * x,
|
||||||
|
const int32_t * ids_src1_inv,
|
||||||
|
void * vy,
|
||||||
|
float * scale,
|
||||||
|
ggml_type type_src0,
|
||||||
|
bool use_aligned_float8,
|
||||||
|
int64_t ne00,
|
||||||
|
int64_t stride_token,
|
||||||
|
int64_t ne0,
|
||||||
|
int64_t n_tokens,
|
||||||
|
int64_t nrows_dst,
|
||||||
|
int n_expert_used,
|
||||||
|
cudaStream_t stream);
|
||||||
|
|
||||||
|
void quantize_scatter_mmq_q8_1_cuda(const float * x,
|
||||||
|
const int32_t * ids_src1_inv,
|
||||||
|
void * vy,
|
||||||
|
ggml_type type_src0,
|
||||||
|
int64_t ne00,
|
||||||
|
int64_t stride_token,
|
||||||
|
int64_t ne0,
|
||||||
|
int64_t n_tokens,
|
||||||
|
int64_t nrows_dst,
|
||||||
|
int n_expert_used,
|
||||||
|
cudaStream_t stream);
|
||||||
|
|||||||
@@ -9,6 +9,21 @@ using namespace cub;
|
|||||||
|
|
||||||
#include "ssm-scan.cuh"
|
#include "ssm-scan.cuh"
|
||||||
|
|
||||||
|
|
||||||
|
// Minimum number of tokens to use SSD (State Space Duality) matmul path instead of scan path.
|
||||||
|
// For n_tok <= this threshold, the scan kernel is used (lower overhead for short sequences).
|
||||||
|
#define SSM_SSD_MIN_TOKENS 128
|
||||||
|
|
||||||
|
// prepare_dt kernel dimensions: one block per (head, seq), each block handles DT_MAX_ITEMS items.
|
||||||
|
#define SSM_SSD_DT_BLOCK 256
|
||||||
|
#define SSM_SSD_DT_MAX_ITEMS 32
|
||||||
|
|
||||||
|
// Maximum tokens the SSD path supports, derived from the prepare_dt kernel block capacity.
|
||||||
|
#define SSM_SSD_MAX_TOKENS (SSM_SSD_DT_BLOCK * SSM_SSD_DT_MAX_ITEMS)
|
||||||
|
|
||||||
|
// Chunk size for chunked SSD. Caps matmul cost at O(chunk^2) per chunk.
|
||||||
|
#define SSM_SSD_CHUNK_SIZE 256
|
||||||
|
|
||||||
// We would like to keep pragma unroll for cases where L_template is not 0,
|
// We would like to keep pragma unroll for cases where L_template is not 0,
|
||||||
// so we suppress the clang transformation warning.
|
// so we suppress the clang transformation warning.
|
||||||
#ifdef __clang__
|
#ifdef __clang__
|
||||||
@@ -316,6 +331,429 @@ static void ssm_scan_f32_cuda(const float * src0, const float * src1, const floa
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
|
||||||
|
// ============================================================================
|
||||||
|
// SSD (State Space Duality) kernels for Mamba-2 prefill (n_tok > SSM_SSD_MIN_TOKENS)
|
||||||
|
//
|
||||||
|
// Instead of a sequential scan, SSD reformulates the output as:
|
||||||
|
// Y = (L (.) (C @ B^T)) @ (X * dt) + decay * C @ s_init
|
||||||
|
// where L is a causal decay mask derived from A and dt.
|
||||||
|
//
|
||||||
|
// This converts the O(T*N) sequential scan into parallel matmuls.
|
||||||
|
// ============================================================================
|
||||||
|
// Softplus(dt) and inclusive prefix sum per head using CUB BlockScan.
|
||||||
|
// Grid: (n_head, n_seqs)
|
||||||
|
template <int BLOCK_SIZE, int MAX_ITEMS>
|
||||||
|
__global__ void ssm_ssd_prepare_dt_kernel(
|
||||||
|
const float * __restrict__ dt_raw,
|
||||||
|
float * __restrict__ dt_sp_out,
|
||||||
|
float * __restrict__ cs_out,
|
||||||
|
const int n_head, const int n_tok,
|
||||||
|
const int dt_stride_tok, // elements between tokens in dt
|
||||||
|
const int dt_stride_seq) { // elements between sequences in dt
|
||||||
|
|
||||||
|
const int h = blockIdx.x;
|
||||||
|
const int s = blockIdx.y;
|
||||||
|
|
||||||
|
const float * dt_seq = dt_raw + s * dt_stride_seq;
|
||||||
|
|
||||||
|
float * dt_sp_seq = dt_sp_out + s * n_tok * n_head;
|
||||||
|
float * cs_seq = cs_out + s * n_tok * n_head;
|
||||||
|
|
||||||
|
const int items_per_thread = (n_tok + BLOCK_SIZE - 1) / BLOCK_SIZE;
|
||||||
|
|
||||||
|
// Phase 1: softplus with interleaved distribution (t = i*BLOCK_SIZE + threadIdx.x).
|
||||||
|
// Each warp reads BLOCK_SIZE consecutive tokens, giving coalesced dt_raw loads
|
||||||
|
// (stride n_head between threads vs. items_per_thread*n_head in blocked layout).
|
||||||
|
float local_vals[MAX_ITEMS];
|
||||||
|
for (int i = 0; i < items_per_thread; i++) {
|
||||||
|
const int t = i * BLOCK_SIZE + threadIdx.x;
|
||||||
|
if (t < n_tok) {
|
||||||
|
float val = dt_seq[h + t * dt_stride_tok];
|
||||||
|
float sp = (val <= 20.0f) ? log1pf(expf(val)) : val;
|
||||||
|
local_vals[i] = sp;
|
||||||
|
dt_sp_seq[t * n_head + h] = sp;
|
||||||
|
} else {
|
||||||
|
local_vals[i] = 0.0f;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Phase 2+3: per-step inclusive scan to build cs[] in token order.
|
||||||
|
// With interleaved distribution the per-thread total scan would not give token-order
|
||||||
|
// prefix sums, so we scan one BLOCK_SIZE slab at a time and carry a running total.
|
||||||
|
#ifdef USE_CUB
|
||||||
|
using BlockScan = cub::BlockScan<float, BLOCK_SIZE>;
|
||||||
|
__shared__ typename BlockScan::TempStorage scan_temp;
|
||||||
|
__shared__ float step_total;
|
||||||
|
|
||||||
|
float running = 0.0f;
|
||||||
|
for (int i = 0; i < items_per_thread; i++) {
|
||||||
|
float inclusive;
|
||||||
|
BlockScan(scan_temp).InclusiveSum(local_vals[i], inclusive);
|
||||||
|
const int t = i * BLOCK_SIZE + threadIdx.x;
|
||||||
|
if (t < n_tok) {
|
||||||
|
cs_seq[t * n_head + h] = running + inclusive;
|
||||||
|
}
|
||||||
|
if (threadIdx.x == BLOCK_SIZE - 1) {
|
||||||
|
step_total = inclusive;
|
||||||
|
}
|
||||||
|
__syncthreads();
|
||||||
|
running += step_total;
|
||||||
|
}
|
||||||
|
#else
|
||||||
|
// Fallback: sequential prefix scan in shared memory, one slab at a time.
|
||||||
|
__shared__ float sdata[BLOCK_SIZE];
|
||||||
|
float running = 0.0f;
|
||||||
|
for (int i = 0; i < items_per_thread; i++) {
|
||||||
|
const int t = i * BLOCK_SIZE + threadIdx.x;
|
||||||
|
sdata[threadIdx.x] = local_vals[i];
|
||||||
|
__syncthreads();
|
||||||
|
if (threadIdx.x == 0) {
|
||||||
|
for (int j = 1; j < BLOCK_SIZE; j++) {
|
||||||
|
sdata[j] += sdata[j - 1];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
__syncthreads();
|
||||||
|
if (t < n_tok) {
|
||||||
|
cs_seq[t * n_head + h] = running + sdata[threadIdx.x];
|
||||||
|
}
|
||||||
|
running += sdata[BLOCK_SIZE - 1];
|
||||||
|
__syncthreads();
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
}
|
||||||
|
|
||||||
|
// Prepare SSD matmul inputs for one chunk: X_dt, B_weighted, C_scaled.
|
||||||
|
// T_matmul controls precision for X_dt, B_weighted (float or half).
|
||||||
|
// C_scaled is always float (pairs with float s_cur in step 3c).
|
||||||
|
// Computation is always FP32; only the final store converts to T_matmul.
|
||||||
|
// Also materializes the causal M matrix = exp(A*(cs_out - cs_in)) * CB (fused with prep to save a launch).
|
||||||
|
// Grid: (ceil(max(C*head_dim, d_state*C, chunk_len^2) / BLOCK), n_head, n_seqs)
|
||||||
|
template <int BLOCK_SIZE, typename T_matmul>
|
||||||
|
__global__ void ssm_ssd_pre_matmul_kernel(
|
||||||
|
const float * __restrict__ cs, // {n_tok, n_head} cumulative dt sums
|
||||||
|
const float * __restrict__ dt_sp, // {n_tok, n_head} softplus(dt)
|
||||||
|
const float * __restrict__ A, // {1, n_head}
|
||||||
|
const float * __restrict__ x, // {head_dim, n_head, n_tok, n_seqs}
|
||||||
|
const float * __restrict__ B, // {d_state, n_group, n_tok, n_seqs}
|
||||||
|
const float * __restrict__ C_src, // {d_state, n_group, n_tok, n_seqs}
|
||||||
|
T_matmul * __restrict__ X_dt, // {head_dim, C, n_head} x * dt, d-fastest
|
||||||
|
T_matmul * __restrict__ B_weighted, // {d_state, C, n_head} B * decay_from_end
|
||||||
|
float * __restrict__ C_scaled, // {d_state, C, n_head} C * decay_to_pos (always float)
|
||||||
|
const float * __restrict__ CB, // {chunk_len, chunk_len, n_group, n_seqs}
|
||||||
|
half * __restrict__ M_out, // {chunk_len, chunk_len, n_head, n_seqs}
|
||||||
|
const int chunk_len, const int head_dim, const int n_head, const int n_group,
|
||||||
|
const int d_state, const int A_stride,
|
||||||
|
const int x_stride_tok, const int x_stride_seq,
|
||||||
|
const int B_stride_tok, const int B_stride_seq,
|
||||||
|
const int C_stride_tok, const int C_stride_seq,
|
||||||
|
const int chunk_offset,
|
||||||
|
const int n_tok_total) {
|
||||||
|
|
||||||
|
const int h = blockIdx.y;
|
||||||
|
const int s = blockIdx.z;
|
||||||
|
const int g = h / (n_head / n_group);
|
||||||
|
|
||||||
|
const float A_h = A[h * A_stride];
|
||||||
|
const int idx = blockIdx.x * BLOCK_SIZE + threadIdx.x;
|
||||||
|
|
||||||
|
const int cs_seq_off = s * n_tok_total * n_head;
|
||||||
|
const float cs_base = (chunk_offset > 0) ? cs[cs_seq_off + (chunk_offset - 1) * n_head + h] : 0.0f;
|
||||||
|
const float cs_last = cs[cs_seq_off + (chunk_offset + chunk_len - 1) * n_head + h] - cs_base;
|
||||||
|
|
||||||
|
// Prepare X_dt = x * dt, stored d-fastest for coalesced reads and writes.
|
||||||
|
const int n_xdt = chunk_len * head_dim;
|
||||||
|
if (idx < n_xdt) {
|
||||||
|
const int d = idx % head_dim;
|
||||||
|
const int t = idx / head_dim;
|
||||||
|
|
||||||
|
const float x_val = x[s * x_stride_seq + (chunk_offset + t) * x_stride_tok + d + h * head_dim];
|
||||||
|
const float dt_val = dt_sp[cs_seq_off + (chunk_offset + t) * n_head + h];
|
||||||
|
|
||||||
|
X_dt[d + t * head_dim + h * n_xdt + s * n_xdt * n_head] = (T_matmul)(x_val * dt_val);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Prepare B_weighted and C_scaled together: both share the same index space (d_state * chunk_len)
|
||||||
|
// and the same cs_t load, so merging halves the cs[] global memory traffic.
|
||||||
|
const int n_bw = d_state * chunk_len;
|
||||||
|
if (idx < n_bw) {
|
||||||
|
const int n = idx % d_state;
|
||||||
|
const int t = idx / d_state;
|
||||||
|
|
||||||
|
const float cs_t = cs[cs_seq_off + (chunk_offset + t) * n_head + h] - cs_base;
|
||||||
|
|
||||||
|
const float B_val = B[s * B_stride_seq + (chunk_offset + t) * B_stride_tok + g * d_state + n];
|
||||||
|
B_weighted[n + t * d_state + h * n_bw + s * n_bw * n_head] = (T_matmul)(B_val * __expf(A_h * (cs_last - cs_t)));
|
||||||
|
|
||||||
|
const float C_val = C_src[s * C_stride_seq + (chunk_offset + t) * C_stride_tok + g * d_state + n];
|
||||||
|
C_scaled[n + t * d_state + h * n_bw + s * n_bw * n_head] = C_val * __expf(A_h * cs_t);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Materialize M = exp(A*(cs_out - cs_in)) * CB with causal mask.
|
||||||
|
const int n_M = chunk_len * chunk_len;
|
||||||
|
if (idx < n_M) {
|
||||||
|
const int t_out = idx % chunk_len;
|
||||||
|
const int t_in = idx / chunk_len;
|
||||||
|
|
||||||
|
half val;
|
||||||
|
if (t_in <= t_out) {
|
||||||
|
const float cs_out = cs[cs_seq_off + (chunk_offset + t_out) * n_head + h] - cs_base;
|
||||||
|
const float cs_in = cs[cs_seq_off + (chunk_offset + t_in) * n_head + h] - cs_base;
|
||||||
|
const float decay = __expf(A_h * (cs_out - cs_in));
|
||||||
|
const float * CB_g = CB + (int64_t)s * chunk_len * chunk_len * n_group
|
||||||
|
+ (int64_t)g * chunk_len * chunk_len;
|
||||||
|
const float cb_val = CB_g[t_out + t_in * chunk_len];
|
||||||
|
val = __float2half(decay * cb_val);
|
||||||
|
} else {
|
||||||
|
val = __float2half(0.0f);
|
||||||
|
}
|
||||||
|
|
||||||
|
M_out[(int64_t)s * n_M * n_head + (int64_t)h * n_M + t_in * chunk_len + t_out] = val;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scale running state in-place: s_cur *= decay_total(chunk).
|
||||||
|
// Called BEFORE cuBLAS state update (beta=1) to fuse inter-chunk decay.
|
||||||
|
// Eliminates the s_old buffer and D2D memcpy vs the old approach of:
|
||||||
|
// memcpy(s_old, s_cur) -> cuBLAS(beta=0) -> s_cur += decay * s_old
|
||||||
|
// Grid: (ceil(d_state * head_dim / BLOCK), n_head, n_seqs)
|
||||||
|
template <int BLOCK_SIZE>
|
||||||
|
__global__ void ssm_ssd_scale_state_kernel(
|
||||||
|
float * __restrict__ s_cur, // {d_state, head_dim, n_head, n_seqs}
|
||||||
|
const float * __restrict__ cs, // {n_tok, n_head} cumulative dt sums
|
||||||
|
const float * __restrict__ A, // {1, n_head}
|
||||||
|
const int d_state, const int head_dim, const int n_head,
|
||||||
|
const int chunk_offset, const int chunk_len,
|
||||||
|
const int n_tok_total, const int A_stride) {
|
||||||
|
|
||||||
|
const int h = blockIdx.y;
|
||||||
|
const int s = blockIdx.z;
|
||||||
|
const int idx = blockIdx.x * BLOCK_SIZE + threadIdx.x;
|
||||||
|
const int state_per_head = d_state * head_dim;
|
||||||
|
if (idx >= state_per_head) return;
|
||||||
|
|
||||||
|
const float A_h = A[h * A_stride];
|
||||||
|
const int cs_seq_off = s * n_tok_total * n_head;
|
||||||
|
const float cs_base = (chunk_offset > 0) ? cs[cs_seq_off + (chunk_offset - 1) * n_head + h] : 0.0f;
|
||||||
|
const float cs_last = cs[cs_seq_off + (chunk_offset + chunk_len - 1) * n_head + h] - cs_base;
|
||||||
|
const float decay_total = __expf(A_h * cs_last);
|
||||||
|
|
||||||
|
const int off = s * state_per_head * n_head + h * state_per_head + idx;
|
||||||
|
s_cur[off] *= decay_total;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Copy initial state from src0[ids[s]] into s_cur for each sequence.
|
||||||
|
// Grid: (ceil(d_state * head_dim * n_head / BLOCK), n_seqs)
|
||||||
|
template <int BLOCK_SIZE>
|
||||||
|
__global__ void ssm_ssd_init_state_kernel(
|
||||||
|
const float * __restrict__ src0, // {d_state, head_dim, n_head, n_rs}
|
||||||
|
const int32_t * __restrict__ ids, // {n_seqs}
|
||||||
|
float * __restrict__ s_cur, // {d_state, head_dim, n_head, n_seqs}
|
||||||
|
const int state_size, // d_state * head_dim * n_head
|
||||||
|
const int64_t s0_stride_seq) { // elements between state rows
|
||||||
|
const int s = blockIdx.y;
|
||||||
|
const int idx = blockIdx.x * BLOCK_SIZE + threadIdx.x;
|
||||||
|
if (idx >= state_size) return;
|
||||||
|
|
||||||
|
const float * s_src = src0 + (int64_t)ids[s] * s0_stride_seq;
|
||||||
|
s_cur[s * state_size + idx] = s_src[idx];
|
||||||
|
}
|
||||||
|
|
||||||
|
// SSD (State Space Duality) dispatch for Mamba-2 prefill.
|
||||||
|
// Chunked matmuls: CB, materialize M + cuBLAS Y, S@C, B@X_dt.
|
||||||
|
// All strides are in elements (floats), not bytes.
|
||||||
|
static void ssm_scan_ssd_f32_cuda(
|
||||||
|
ggml_backend_cuda_context & ctx,
|
||||||
|
const float * src0_d, const float * src1_d, const float * src2_d, const float * src3_d,
|
||||||
|
const float * src4_d, const float * src5_d, const int32_t * src6_d, float * dst_d,
|
||||||
|
const int64_t s0_stride_seq, // state (src0) stride between seqs
|
||||||
|
const int x_stride_tok, const int x_stride_seq, // x (src1) strides
|
||||||
|
const int dt_stride_tok, const int dt_stride_seq, // dt (src2) strides
|
||||||
|
const int A_stride, // A (src3) stride between heads
|
||||||
|
const int B_stride_tok, const int B_stride_seq, // B (src4) strides
|
||||||
|
const int C_stride_tok, const int C_stride_seq, // C (src5) strides
|
||||||
|
const int64_t s_off, const int64_t d_state, const int64_t head_dim,
|
||||||
|
const int64_t n_head, const int64_t n_group, const int64_t n_tok, const int64_t n_seq) {
|
||||||
|
|
||||||
|
cudaStream_t stream = ctx.stream();
|
||||||
|
const int64_t d_inner = head_dim * n_head;
|
||||||
|
|
||||||
|
const int64_t chunk_size = SSM_SSD_CHUNK_SIZE;
|
||||||
|
const int64_t n_chunks = (n_tok + chunk_size - 1) / chunk_size;
|
||||||
|
|
||||||
|
const int64_t state_per_head = d_state * head_dim;
|
||||||
|
|
||||||
|
using matmul_t = half;
|
||||||
|
static constexpr cudaDataType_t matmul_dtype = CUDA_R_16F;
|
||||||
|
|
||||||
|
ggml_cuda_pool_alloc<float> dt_sp_buf(ctx.pool(), n_tok * n_head * n_seq);
|
||||||
|
ggml_cuda_pool_alloc<float> cs_buf(ctx.pool(), n_tok * n_head * n_seq);
|
||||||
|
ggml_cuda_pool_alloc<float> CB_buf(ctx.pool(), chunk_size * chunk_size * n_group * n_seq);
|
||||||
|
ggml_cuda_pool_alloc<matmul_t> X_dt_buf(ctx.pool(), chunk_size * head_dim * n_head * n_seq);
|
||||||
|
ggml_cuda_pool_alloc<matmul_t> B_w_buf(ctx.pool(), d_state * chunk_size * n_head * n_seq);
|
||||||
|
ggml_cuda_pool_alloc<float> C_s_buf(ctx.pool(), d_state * chunk_size * n_head * n_seq);
|
||||||
|
float * dt_sp = dt_sp_buf.get();
|
||||||
|
float * cs = cs_buf.get();
|
||||||
|
float * CB = CB_buf.get();
|
||||||
|
matmul_t * X_dt = X_dt_buf.get();
|
||||||
|
matmul_t * B_weighted = B_w_buf.get();
|
||||||
|
float * C_scaled = C_s_buf.get();
|
||||||
|
float * s_cur = (float *)((char *)dst_d + s_off); // write state directly to dst
|
||||||
|
|
||||||
|
// Step 1: softplus(dt) and parallel prefix sum over full sequence
|
||||||
|
{
|
||||||
|
dim3 grid(n_head, n_seq);
|
||||||
|
ssm_ssd_prepare_dt_kernel<SSM_SSD_DT_BLOCK, SSM_SSD_DT_MAX_ITEMS><<<grid, SSM_SSD_DT_BLOCK, 0, stream>>>(
|
||||||
|
src2_d, dt_sp, cs, n_head, n_tok, dt_stride_tok, dt_stride_seq);
|
||||||
|
CUDA_CHECK(cudaGetLastError());
|
||||||
|
}
|
||||||
|
|
||||||
|
// Step 2: initialize running state from src0[ids[s]]
|
||||||
|
{
|
||||||
|
constexpr int BLOCK = 256;
|
||||||
|
const int64_t state_size = d_state * head_dim * n_head;
|
||||||
|
dim3 grid((state_size + BLOCK - 1) / BLOCK, n_seq);
|
||||||
|
ssm_ssd_init_state_kernel<BLOCK><<<grid, BLOCK, 0, stream>>>(
|
||||||
|
src0_d, src6_d, s_cur, state_size, s0_stride_seq);
|
||||||
|
CUDA_CHECK(cudaGetLastError());
|
||||||
|
}
|
||||||
|
|
||||||
|
// Step 3: chunked SSD loop
|
||||||
|
// Per chunk: pre_matmul (incl. M) + 4 cuBLAS (CB, Y, S@C, state update) + scale_state
|
||||||
|
cublasHandle_t handle = ctx.cublas_handle();
|
||||||
|
CUBLAS_CHECK(cublasSetStream(handle, stream));
|
||||||
|
const float alpha_one = 1.0f;
|
||||||
|
const float beta_zero = 0.0f;
|
||||||
|
const float beta_one = 1.0f;
|
||||||
|
const int lda_C_src = C_stride_tok; // leading dim for C in CB = C^T @ B
|
||||||
|
const int ldb_B_src = B_stride_tok; // leading dim for B in CB = C^T @ B
|
||||||
|
|
||||||
|
// Scratch buffer for causal M matrix, reused across chunks (max size at chunk_size)
|
||||||
|
const int64_t n_M_max = chunk_size * chunk_size;
|
||||||
|
ggml_cuda_pool_alloc<half> M_buf(ctx.pool(), n_M_max * n_head * n_seq);
|
||||||
|
half * M_mat = M_buf.get();
|
||||||
|
|
||||||
|
for (int64_t k = 0; k < n_chunks; k++) {
|
||||||
|
const int64_t chunk_offset = k * chunk_size;
|
||||||
|
const int64_t chunk_len = (chunk_offset + chunk_size <= n_tok) ? chunk_size : (n_tok - chunk_offset);
|
||||||
|
|
||||||
|
// 3a: CB = C^T @ B per group
|
||||||
|
for (int64_t s = 0; s < n_seq; s++) {
|
||||||
|
const float * C_s = src5_d + s * C_stride_seq + chunk_offset * C_stride_tok;
|
||||||
|
const float * B_s = src4_d + s * B_stride_seq + chunk_offset * B_stride_tok;
|
||||||
|
float * CB_s = CB + s * chunk_len * chunk_len * n_group;
|
||||||
|
|
||||||
|
if (n_group == 1) {
|
||||||
|
CUBLAS_CHECK(cublasSgemm(handle, CUBLAS_OP_T, CUBLAS_OP_N,
|
||||||
|
chunk_len, chunk_len, d_state,
|
||||||
|
&alpha_one, C_s, lda_C_src, B_s, ldb_B_src,
|
||||||
|
&beta_zero, CB_s, (int)chunk_len));
|
||||||
|
} else {
|
||||||
|
CUBLAS_CHECK(cublasGemmStridedBatchedEx(handle, CUBLAS_OP_T, CUBLAS_OP_N,
|
||||||
|
chunk_len, chunk_len, d_state,
|
||||||
|
&alpha_one,
|
||||||
|
C_s, CUDA_R_32F, lda_C_src, d_state,
|
||||||
|
B_s, CUDA_R_32F, ldb_B_src, d_state,
|
||||||
|
&beta_zero,
|
||||||
|
CB_s, CUDA_R_32F, (int)chunk_len, (long long)(chunk_len * chunk_len),
|
||||||
|
n_group,
|
||||||
|
CUBLAS_COMPUTE_32F, CUBLAS_GEMM_DEFAULT));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 3b: prepare X_dt, B_weighted, C_scaled + materialize causal M matrix
|
||||||
|
const int64_t n_M = chunk_len * chunk_len;
|
||||||
|
{
|
||||||
|
constexpr int BLOCK = 256;
|
||||||
|
const int64_t n_xdt = chunk_len * head_dim;
|
||||||
|
const int64_t n_bw = d_state * chunk_len;
|
||||||
|
int64_t max_work = n_xdt;
|
||||||
|
if (n_bw > max_work) max_work = n_bw;
|
||||||
|
if (n_M > max_work) max_work = n_M;
|
||||||
|
dim3 grid((max_work + BLOCK - 1) / BLOCK, n_head, n_seq);
|
||||||
|
ssm_ssd_pre_matmul_kernel<BLOCK, matmul_t><<<grid, BLOCK, 0, stream>>>(
|
||||||
|
cs, dt_sp, src3_d, src1_d, src4_d, src5_d,
|
||||||
|
X_dt, B_weighted, C_scaled,
|
||||||
|
CB, M_mat,
|
||||||
|
chunk_len, head_dim, n_head, n_group, d_state, A_stride,
|
||||||
|
x_stride_tok, x_stride_seq, B_stride_tok, B_stride_seq, C_stride_tok, C_stride_seq,
|
||||||
|
chunk_offset, n_tok);
|
||||||
|
CUDA_CHECK(cudaGetLastError());
|
||||||
|
}
|
||||||
|
|
||||||
|
// 3c: dst = S_cur^T @ C_scaled (state contribution)
|
||||||
|
{
|
||||||
|
const int64_t stride_S = state_per_head;
|
||||||
|
const int64_t stride_Cs = d_state * chunk_len;
|
||||||
|
|
||||||
|
for (int64_t s = 0; s < n_seq; s++) {
|
||||||
|
float * dst_chunk = dst_d + s * d_inner * n_tok + chunk_offset * d_inner;
|
||||||
|
|
||||||
|
CUBLAS_CHECK(cublasGemmStridedBatchedEx(handle, CUBLAS_OP_T, CUBLAS_OP_N,
|
||||||
|
head_dim, chunk_len, d_state,
|
||||||
|
&alpha_one,
|
||||||
|
s_cur + s * stride_S * n_head, CUDA_R_32F, d_state, stride_S,
|
||||||
|
C_scaled + s * stride_Cs * n_head, CUDA_R_32F, d_state, stride_Cs,
|
||||||
|
&beta_zero,
|
||||||
|
dst_chunk, CUDA_R_32F, d_inner, head_dim,
|
||||||
|
n_head,
|
||||||
|
CUBLAS_COMPUTE_32F, CUBLAS_GEMM_DEFAULT));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 3d: dst += X_dt @ M^T (intra-chunk contribution, adds to 3c result)
|
||||||
|
// M is stored as M[t_out, t_in] (lower-triangular), transpose needed for Y = X @ M^T.
|
||||||
|
{
|
||||||
|
const int64_t stride_M = n_M;
|
||||||
|
const int64_t stride_X_h = (int64_t)chunk_len * head_dim;
|
||||||
|
|
||||||
|
for (int64_t s = 0; s < n_seq; s++) {
|
||||||
|
float * dst_chunk = dst_d + s * d_inner * n_tok + chunk_offset * d_inner;
|
||||||
|
CUBLAS_CHECK(cublasGemmStridedBatchedEx(handle, CUBLAS_OP_N, CUBLAS_OP_T,
|
||||||
|
head_dim, chunk_len, chunk_len,
|
||||||
|
&alpha_one,
|
||||||
|
X_dt + s * stride_X_h * n_head, matmul_dtype, head_dim, stride_X_h,
|
||||||
|
M_mat + s * stride_M * n_head, matmul_dtype, chunk_len, stride_M,
|
||||||
|
&beta_one,
|
||||||
|
dst_chunk, CUDA_R_32F, d_inner, head_dim,
|
||||||
|
n_head,
|
||||||
|
CUBLAS_COMPUTE_32F, CUBLAS_GEMM_DEFAULT));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 3e: s_cur = B_weighted @ X_dt^T + decay_total * s_cur_old (state update)
|
||||||
|
{
|
||||||
|
// Scale s_cur in-place by per-head decay_total BEFORE cuBLAS overwrites it
|
||||||
|
constexpr int BLOCK = 256;
|
||||||
|
dim3 grid((state_per_head + BLOCK - 1) / BLOCK, n_head, n_seq);
|
||||||
|
ssm_ssd_scale_state_kernel<BLOCK><<<grid, BLOCK, 0, stream>>>(
|
||||||
|
s_cur, cs, src3_d,
|
||||||
|
d_state, head_dim, n_head,
|
||||||
|
chunk_offset, chunk_len, n_tok, A_stride);
|
||||||
|
CUDA_CHECK(cudaGetLastError());
|
||||||
|
|
||||||
|
// cuBLAS with beta=1: s_cur = B_weighted @ X_dt^T + 1.0 * s_cur (already scaled)
|
||||||
|
const int64_t stride_Bw = d_state * chunk_len;
|
||||||
|
const int64_t stride_X = chunk_len * head_dim;
|
||||||
|
const int64_t stride_S = state_per_head;
|
||||||
|
|
||||||
|
for (int64_t s = 0; s < n_seq; s++) {
|
||||||
|
// X_dt is d-fastest {hd, C}, read as OP_T to get {C, hd}
|
||||||
|
CUBLAS_CHECK(cublasGemmStridedBatchedEx(handle, CUBLAS_OP_N, CUBLAS_OP_T,
|
||||||
|
d_state, head_dim, chunk_len,
|
||||||
|
&alpha_one,
|
||||||
|
B_weighted + s * stride_Bw * n_head, matmul_dtype, d_state, stride_Bw,
|
||||||
|
X_dt + s * stride_X * n_head, matmul_dtype, head_dim, stride_X,
|
||||||
|
&beta_one,
|
||||||
|
s_cur + s * stride_S * n_head, CUDA_R_32F, d_state, stride_S,
|
||||||
|
n_head,
|
||||||
|
CUBLAS_COMPUTE_32F, CUBLAS_GEMM_DEFAULT));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
#endif // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
|
||||||
|
|
||||||
void ggml_cuda_op_ssm_scan(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
|
void ggml_cuda_op_ssm_scan(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
|
||||||
const struct ggml_tensor * src0 = dst->src[0]; // s
|
const struct ggml_tensor * src0 = dst->src[0]; // s
|
||||||
const struct ggml_tensor * src1 = dst->src[1]; // x
|
const struct ggml_tensor * src1 = dst->src[1]; // x
|
||||||
@@ -357,6 +795,49 @@ void ggml_cuda_op_ssm_scan(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
|
|||||||
GGML_ASSERT(src6->type == GGML_TYPE_I32);
|
GGML_ASSERT(src6->type == GGML_TYPE_I32);
|
||||||
GGML_ASSERT(dst->type == GGML_TYPE_F32);
|
GGML_ASSERT(dst->type == GGML_TYPE_F32);
|
||||||
|
|
||||||
|
// Byte strides are narrowed to int for both scan and SSD paths.
|
||||||
|
GGML_ASSERT(src0->nb[2] <= (size_t)INT_MAX);
|
||||||
|
GGML_ASSERT(src0->nb[3] <= (size_t)INT_MAX);
|
||||||
|
GGML_ASSERT(src1->nb[2] <= (size_t)INT_MAX);
|
||||||
|
GGML_ASSERT(src1->nb[3] <= (size_t)INT_MAX);
|
||||||
|
GGML_ASSERT(src2->nb[1] <= (size_t)INT_MAX);
|
||||||
|
GGML_ASSERT(src2->nb[2] <= (size_t)INT_MAX);
|
||||||
|
GGML_ASSERT(src3->nb[1] <= (size_t)INT_MAX);
|
||||||
|
GGML_ASSERT(src4->nb[2] <= (size_t)INT_MAX);
|
||||||
|
GGML_ASSERT(src4->nb[3] <= (size_t)INT_MAX);
|
||||||
|
GGML_ASSERT(src5->nb[2] <= (size_t)INT_MAX);
|
||||||
|
GGML_ASSERT(src5->nb[3] <= (size_t)INT_MAX);
|
||||||
|
|
||||||
|
#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
|
||||||
|
// Mamba-2 with scalar A per head: use SSD matmul path for long sequences.
|
||||||
|
// Requires NVIDIA Turing+ otherwise fallback to scan.
|
||||||
|
const bool is_mamba2 = (src3->nb[1] == sizeof(float));
|
||||||
|
const int cc = ggml_cuda_info().devices[ggml_cuda_get_device()].cc;
|
||||||
|
const bool use_ssd = is_mamba2 && n_t > SSM_SSD_MIN_TOKENS
|
||||||
|
&& n_t <= SSM_SSD_MAX_TOKENS
|
||||||
|
&& GGML_CUDA_CC_IS_NVIDIA(cc)
|
||||||
|
&& cc >= GGML_CUDA_CC_TURING
|
||||||
|
&& nr % 8 == 0; // cuBLAS requires 8-element (16-byte) alignment
|
||||||
|
|
||||||
|
if (use_ssd) {
|
||||||
|
// ssm_ssd_init_state_kernel uses flat linear indexing within each sequence,
|
||||||
|
// so src0 must be fully contiguous across all inner dimensions.
|
||||||
|
// The scan path handles non-contiguous nb[2] via src0_nb2 but does not handle nb[1].
|
||||||
|
GGML_ASSERT(src0->nb[1] == nc * sizeof(float));
|
||||||
|
GGML_ASSERT(src0->nb[2] == nc * nr * sizeof(float));
|
||||||
|
|
||||||
|
ssm_scan_ssd_f32_cuda(ctx,
|
||||||
|
src0_d, src1_d, src2_d, src3_d, src4_d, src5_d, src6_d, dst_d,
|
||||||
|
(int64_t)(src0->nb[3] / sizeof(float)),
|
||||||
|
(int)(src1->nb[2] / sizeof(float)), (int)(src1->nb[3] / sizeof(float)),
|
||||||
|
(int)(src2->nb[1] / sizeof(float)), (int)(src2->nb[2] / sizeof(float)),
|
||||||
|
(int)(src3->nb[1] / sizeof(float)),
|
||||||
|
(int)(src4->nb[2] / sizeof(float)), (int)(src4->nb[3] / sizeof(float)),
|
||||||
|
(int)(src5->nb[2] / sizeof(float)), (int)(src5->nb[3] / sizeof(float)),
|
||||||
|
s_off, nc, nr, nh, ng, n_t, n_s);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
#endif
|
||||||
ssm_scan_f32_cuda(src0_d, src1_d, src2_d, src3_d, src4_d, src5_d, src6_d, dst_d,
|
ssm_scan_f32_cuda(src0_d, src1_d, src2_d, src3_d, src4_d, src5_d, src6_d, dst_d,
|
||||||
src0->nb[2], src0->nb[3], src1->nb[2], src1->nb[3], src2->nb[1], src2->nb[2],
|
src0->nb[2], src0->nb[3], src1->nb[2], src1->nb[3], src2->nb[1], src2->nb[2],
|
||||||
src3->nb[1], src4->nb[2], src4->nb[3], src5->nb[2], src5->nb[3],
|
src3->nb[1], src4->nb[2], src4->nb[3], src5->nb[2], src5->nb[3],
|
||||||
|
|||||||
@@ -36,6 +36,7 @@ SOURCE_FATTN_MMA_CASE = "DECL_FATTN_MMA_F16_CASE({head_size_kq}, {head_size_v},
|
|||||||
|
|
||||||
TYPES_MMQ = [
|
TYPES_MMQ = [
|
||||||
"GGML_TYPE_Q1_0",
|
"GGML_TYPE_Q1_0",
|
||||||
|
"GGML_TYPE_Q2_0",
|
||||||
"GGML_TYPE_Q4_0", "GGML_TYPE_Q4_1", "GGML_TYPE_Q5_0", "GGML_TYPE_Q5_1", "GGML_TYPE_Q8_0",
|
"GGML_TYPE_Q4_0", "GGML_TYPE_Q4_1", "GGML_TYPE_Q5_0", "GGML_TYPE_Q5_1", "GGML_TYPE_Q8_0",
|
||||||
"GGML_TYPE_Q2_K", "GGML_TYPE_Q3_K", "GGML_TYPE_Q4_K", "GGML_TYPE_Q5_K", "GGML_TYPE_Q6_K",
|
"GGML_TYPE_Q2_K", "GGML_TYPE_Q3_K", "GGML_TYPE_Q4_K", "GGML_TYPE_Q5_K", "GGML_TYPE_Q6_K",
|
||||||
"GGML_TYPE_IQ2_XXS", "GGML_TYPE_IQ2_XS", "GGML_TYPE_IQ2_S", "GGML_TYPE_IQ3_XXS", "GGML_TYPE_IQ3_S",
|
"GGML_TYPE_IQ2_XXS", "GGML_TYPE_IQ2_XS", "GGML_TYPE_IQ2_S", "GGML_TYPE_IQ3_XXS", "GGML_TYPE_IQ3_S",
|
||||||
|
|||||||
@@ -0,0 +1,5 @@
|
|||||||
|
// This file has been autogenerated by generate_cu_files.py, do not edit manually.
|
||||||
|
|
||||||
|
#include "../mmq.cuh"
|
||||||
|
|
||||||
|
DECL_MMQ_CASE(GGML_TYPE_Q2_0);
|
||||||
@@ -75,17 +75,26 @@ void ggml_cuda_op_top_k(ggml_backend_cuda_context & ctx, ggml_tensor * dst) {
|
|||||||
const int ncols_pad = next_power_of_2(ncols);
|
const int ncols_pad = next_power_of_2(ncols);
|
||||||
const size_t shared_mem = ncols_pad * sizeof(int);
|
const size_t shared_mem = ncols_pad * sizeof(int);
|
||||||
const size_t max_shared_mem = ggml_cuda_info().devices[ggml_cuda_get_device()].smpb;
|
const size_t max_shared_mem = ggml_cuda_info().devices[ggml_cuda_get_device()].smpb;
|
||||||
|
const bool use_bitonic = shared_mem <= max_shared_mem && ncols <= 1024;
|
||||||
|
const int chunk_nrows = argsort_f32_i32_cuda_cub_chunk_nrows(src0->nb[1], nrows);
|
||||||
|
|
||||||
ggml_cuda_pool_alloc<int> temp_dst_alloc(pool, ncols * nrows);
|
ggml_cuda_pool_alloc<int> temp_dst_alloc(pool, ncols * chunk_nrows);
|
||||||
int * tmp_dst = temp_dst_alloc.get();
|
int * tmp_dst = temp_dst_alloc.get();
|
||||||
|
|
||||||
if (shared_mem > max_shared_mem || ncols > 1024) {
|
for (int64_t i = 0; i < nrows; i += chunk_nrows) {
|
||||||
argsort_f32_i32_cuda_cub(pool, src0_d, tmp_dst, ncols, nrows, GGML_SORT_ORDER_DESC, stream);
|
int iter_nrows = std::min((int64_t) chunk_nrows, nrows - i);
|
||||||
} else {
|
|
||||||
argsort_f32_i32_cuda_bitonic(src0_d, tmp_dst, ncols, nrows, GGML_SORT_ORDER_DESC, stream);
|
if (use_bitonic) {
|
||||||
|
argsort_f32_i32_cuda_bitonic(src0_d, tmp_dst, ncols, iter_nrows, GGML_SORT_ORDER_DESC, stream);
|
||||||
|
} else {
|
||||||
|
argsort_f32_i32_cuda_cub(pool, src0_d, tmp_dst, ncols, iter_nrows, GGML_SORT_ORDER_DESC, stream);
|
||||||
|
}
|
||||||
|
CUDA_CHECK(cudaMemcpy2DAsync(dst_d, k * sizeof(int), tmp_dst, ncols * sizeof(int), k * sizeof(int), iter_nrows,
|
||||||
|
cudaMemcpyDeviceToDevice, stream));
|
||||||
|
|
||||||
|
src0_d += ncols * iter_nrows;
|
||||||
|
dst_d += k * iter_nrows;
|
||||||
}
|
}
|
||||||
CUDA_CHECK(cudaMemcpy2DAsync(dst_d, k * sizeof(int), tmp_dst, ncols * sizeof(int), k * sizeof(int), nrows,
|
|
||||||
cudaMemcpyDeviceToDevice, stream));
|
|
||||||
#else // GGML_CUDA_USE_CUB
|
#else // GGML_CUDA_USE_CUB
|
||||||
ggml_cuda_pool_alloc<int> temp_dst_alloc(pool, ncols * nrows);
|
ggml_cuda_pool_alloc<int> temp_dst_alloc(pool, ncols * nrows);
|
||||||
int * tmp_dst = temp_dst_alloc.get();
|
int * tmp_dst = temp_dst_alloc.get();
|
||||||
|
|||||||
@@ -8,6 +8,7 @@
|
|||||||
// Kernel config struct - passed by value to CUDA kernel
|
// Kernel config struct - passed by value to CUDA kernel
|
||||||
struct topk_moe_config {
|
struct topk_moe_config {
|
||||||
bool use_sigmoid;
|
bool use_sigmoid;
|
||||||
|
bool use_sqrt_softplus;
|
||||||
bool with_norm;
|
bool with_norm;
|
||||||
bool delayed_softmax;
|
bool delayed_softmax;
|
||||||
};
|
};
|
||||||
@@ -67,6 +68,16 @@ __device__ void sigmoid_warp_inplace(float (&vals)[experts_per_thread], const in
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
template <int experts_per_thread, bool use_limit>
|
||||||
|
__device__ void sqrt_softplus_warp_inplace(float (&vals)[experts_per_thread], const int limit, const int lane) {
|
||||||
|
#pragma unroll
|
||||||
|
for (int i = 0; i < experts_per_thread; i++) {
|
||||||
|
const int idx = lane + i * WARP_SIZE;
|
||||||
|
const bool active = !use_limit || (idx < limit);
|
||||||
|
vals[i] = active ? sqrtf(vals[i] > 20.0f ? vals[i] : logf(1.0f + expf(vals[i]))) : -INFINITY;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/*
|
/*
|
||||||
This kernel does the following:
|
This kernel does the following:
|
||||||
1. optionally softmax over the logits per token [n_experts, n_tokens]
|
1. optionally softmax over the logits per token [n_experts, n_tokens]
|
||||||
@@ -115,6 +126,8 @@ __launch_bounds__(4 * WARP_SIZE, 1) __global__ void topk_moe_cuda(const float *
|
|||||||
if (!config.delayed_softmax) {
|
if (!config.delayed_softmax) {
|
||||||
if (config.use_sigmoid) {
|
if (config.use_sigmoid) {
|
||||||
sigmoid_warp_inplace<experts_per_thread, false>(wt, n_experts, threadIdx.x);
|
sigmoid_warp_inplace<experts_per_thread, false>(wt, n_experts, threadIdx.x);
|
||||||
|
} else if (config.use_sqrt_softplus) {
|
||||||
|
sqrt_softplus_warp_inplace<experts_per_thread, false>(wt, n_experts, threadIdx.x);
|
||||||
} else {
|
} else {
|
||||||
softmax_warp_inplace<experts_per_thread, false>(wt, n_experts, threadIdx.x);
|
softmax_warp_inplace<experts_per_thread, false>(wt, n_experts, threadIdx.x);
|
||||||
}
|
}
|
||||||
@@ -364,9 +377,10 @@ void ggml_cuda_op_topk_moe(ggml_backend_cuda_context & ctx,
|
|||||||
}
|
}
|
||||||
|
|
||||||
topk_moe_config config;
|
topk_moe_config config;
|
||||||
config.use_sigmoid = args.sigmoid;
|
config.use_sigmoid = args.sigmoid;
|
||||||
config.with_norm = with_norm;
|
config.use_sqrt_softplus = args.sqrt_softplus;
|
||||||
config.delayed_softmax = args.delayed_softmax;
|
config.with_norm = with_norm;
|
||||||
|
config.delayed_softmax = args.delayed_softmax;
|
||||||
|
|
||||||
if (bias) {
|
if (bias) {
|
||||||
launch_topk_moe_cuda<true>(ctx, logits_d, weights_d, ids_d, bias_d, n_rows, n_experts, n_expert_used, clamp_val,
|
launch_topk_moe_cuda<true>(ctx, logits_d, weights_d, ids_d, bias_d, n_rows, n_experts, n_expert_used, clamp_val,
|
||||||
@@ -415,7 +429,7 @@ bool ggml_cuda_should_use_topk_moe(const ggml_tensor * gating_op,
|
|||||||
} else if (gating_op->op == GGML_OP_UNARY) {
|
} else if (gating_op->op == GGML_OP_UNARY) {
|
||||||
ggml_unary_op op = ggml_get_unary_op(gating_op);
|
ggml_unary_op op = ggml_get_unary_op(gating_op);
|
||||||
|
|
||||||
if (op != GGML_UNARY_OP_SIGMOID) {
|
if (op != GGML_UNARY_OP_SIGMOID && op != GGML_UNARY_OP_SOFTPLUS) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,6 +5,7 @@
|
|||||||
|
|
||||||
struct ggml_cuda_topk_moe_args {
|
struct ggml_cuda_topk_moe_args {
|
||||||
bool sigmoid{};
|
bool sigmoid{};
|
||||||
|
bool sqrt_softplus{};
|
||||||
bool softmax{};
|
bool softmax{};
|
||||||
bool delayed_softmax{};
|
bool delayed_softmax{};
|
||||||
bool prob_bias{};
|
bool prob_bias{};
|
||||||
|
|||||||
@@ -109,6 +109,9 @@ static __device__ __forceinline__ uint32_t unpack_ksigns(const uint8_t v) {
|
|||||||
#define VDR_Q1_0_Q8_1_MMVQ 1 // Process one 32-element chunk at a time for parallelism
|
#define VDR_Q1_0_Q8_1_MMVQ 1 // Process one 32-element chunk at a time for parallelism
|
||||||
#define VDR_Q1_0_Q8_1_MMQ 4 // Q1_0 has 128 bits (4 ints) per block
|
#define VDR_Q1_0_Q8_1_MMQ 4 // Q1_0 has 128 bits (4 ints) per block
|
||||||
|
|
||||||
|
#define VDR_Q2_0_Q8_1_MMVQ 1 // Process one 32-element chunk at a time for parallelism
|
||||||
|
#define VDR_Q2_0_Q8_1_MMQ 2 // Q2_0 group 64: 128 bits (4 ints) per block, 2 32-element chunks
|
||||||
|
|
||||||
#define VDR_Q4_0_Q8_1_MMVQ 2
|
#define VDR_Q4_0_Q8_1_MMVQ 2
|
||||||
#define VDR_Q4_0_Q8_1_MMQ 4
|
#define VDR_Q4_0_Q8_1_MMQ 4
|
||||||
|
|
||||||
@@ -681,27 +684,90 @@ static __device__ __forceinline__ float vec_dot_q1_0_q8_1(
|
|||||||
// Q8_1: 32 elements per block with individual scales
|
// Q8_1: 32 elements per block with individual scales
|
||||||
// iqs selects which of the 4 chunks of 32 elements to process (0-3)
|
// iqs selects which of the 4 chunks of 32 elements to process (0-3)
|
||||||
|
|
||||||
const float d1 = bq1_0->d;
|
const float d1 = bq1_0->d;
|
||||||
|
const int16_t * qs = (const int16_t *) bq1_0->qs + iqs * 2;
|
||||||
|
|
||||||
// Process only the chunk specified by iqs
|
// Process only the chunk specified by iqs
|
||||||
const block_q8_1 * bq8_1_chunk = bq8_1 + iqs;
|
const block_q8_1 * bq8_1_chunk = bq8_1 + iqs;
|
||||||
|
|
||||||
// Load 32 bits (4 bytes) for this chunk from Q1_0
|
int sumi = 0;
|
||||||
const int offset = iqs * 4;
|
#pragma unroll
|
||||||
const int v = bq1_0->qs[offset + 0] | (bq1_0->qs[offset + 1] << 8) |
|
for (int j = 0; j < 2; ++j) {
|
||||||
(bq1_0->qs[offset + 2] << 16) | (bq1_0->qs[offset + 3] << 24);
|
const int q = qs[j];
|
||||||
|
|
||||||
// Unpack 32 bits into 32 signed values (-1 or +1)
|
const int u0 = get_int_b4(bq8_1_chunk->qs, j*4+0);
|
||||||
|
const int u1 = get_int_b4(bq8_1_chunk->qs, j*4+1);
|
||||||
|
const int u2 = get_int_b4(bq8_1_chunk->qs, j*4+2);
|
||||||
|
const int u3 = get_int_b4(bq8_1_chunk->qs, j*4+3);
|
||||||
|
|
||||||
|
// unpack crumbs into nibble indices
|
||||||
|
const int n0 = __byte_perm(0x11100100, 0x11100100, q >> 0); // [0, 1, 4, 5] [ 8, 9, 12, 13]
|
||||||
|
const int n1 = __byte_perm(0x11100100, 0x11100100, q >> 2); // [2, 3, 6, 7] [10, 11, 14, 15]
|
||||||
|
// unpack nibbles into byte values
|
||||||
|
const int s0 = __byte_perm(0x01FF, 0x01FF, n0 >> 0);
|
||||||
|
const int s1 = __byte_perm(0x01FF, 0x01FF, n1 >> 0);
|
||||||
|
const int s2 = __byte_perm(0x01FF, 0x01FF, n0 >> 16);
|
||||||
|
const int s3 = __byte_perm(0x01FF, 0x01FF, n1 >> 16);
|
||||||
|
// unshuffle values
|
||||||
|
const int v0 = __byte_perm(s0, s1, 0x5410);
|
||||||
|
const int v1 = __byte_perm(s0, s1, 0x7632);
|
||||||
|
const int v2 = __byte_perm(s2, s3, 0x5410);
|
||||||
|
const int v3 = __byte_perm(s2, s3, 0x7632);
|
||||||
|
|
||||||
|
sumi = ggml_cuda_dp4a(v0, u0, sumi);
|
||||||
|
sumi = ggml_cuda_dp4a(v1, u1, sumi);
|
||||||
|
sumi = ggml_cuda_dp4a(v2, u2, sumi);
|
||||||
|
sumi = ggml_cuda_dp4a(v3, u3, sumi);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Apply Q1_0's single scale and this chunk's Q8_1 scale
|
||||||
|
const float d8 = __low2float(bq8_1_chunk->ds);
|
||||||
|
return d1 * d8 * sumi;
|
||||||
|
}
|
||||||
|
|
||||||
|
static __device__ __forceinline__ float vec_dot_q2_0_q8_1(
|
||||||
|
const void * __restrict__ vbq, const block_q8_1 * __restrict__ bq8_1, const int & kbx, const int & iqs) {
|
||||||
|
|
||||||
|
const block_q2_0 * bq2_0 = (const block_q2_0 *) vbq + kbx;
|
||||||
|
|
||||||
|
// Q2_0 (group 64): 64 elements with ONE scale, 2 bits per element (4 elements per byte)
|
||||||
|
// Q8_1: 32 elements per block with individual scales
|
||||||
|
// iqs selects which of the 2 chunks of 32 elements to process (0-1)
|
||||||
|
|
||||||
|
const float d2 = bq2_0->d;
|
||||||
|
|
||||||
|
// Process only the chunk specified by iqs
|
||||||
|
const block_q8_1 * bq8_1_chunk = bq8_1 + iqs;
|
||||||
|
|
||||||
|
// Load 64 bits (8 bytes) for this chunk from Q2_0: bytes [8*iqs, 8*iqs+8)
|
||||||
|
const int offset = iqs * 8;
|
||||||
|
const int v0 = bq2_0->qs[offset + 0] | (bq2_0->qs[offset + 1] << 8) |
|
||||||
|
(bq2_0->qs[offset + 2] << 16) | (bq2_0->qs[offset + 3] << 24);
|
||||||
|
const int v1 = bq2_0->qs[offset + 4] | (bq2_0->qs[offset + 5] << 8) |
|
||||||
|
(bq2_0->qs[offset + 6] << 16) | (bq2_0->qs[offset + 7] << 24);
|
||||||
|
|
||||||
|
// Unpack 32 2-bit codes into 8 int32s, each holding 4 signed int8 symbols in {-1,0,1,2}.
|
||||||
|
// Stored code c in {0,1,2,3} -> symbol s = c - 1.
|
||||||
int vi_bytes[8];
|
int vi_bytes[8];
|
||||||
#pragma unroll
|
#pragma unroll
|
||||||
for (int j = 0; j < 8; ++j) {
|
for (int j = 0; j < 4; ++j) {
|
||||||
const int shift = j * 4;
|
const int shift = j * 8;
|
||||||
const int bits4 = (v >> shift) & 0x0F;
|
const int codes = (v0 >> shift) & 0xFF;
|
||||||
const int b0 = (bits4 & 0x01) ? 1 : -1;
|
const int c0 = ((codes >> 0) & 0x3) - 1;
|
||||||
const int b1 = (bits4 & 0x02) ? 1 : -1;
|
const int c1 = ((codes >> 2) & 0x3) - 1;
|
||||||
const int b2 = (bits4 & 0x04) ? 1 : -1;
|
const int c2 = ((codes >> 4) & 0x3) - 1;
|
||||||
const int b3 = (bits4 & 0x08) ? 1 : -1;
|
const int c3 = ((codes >> 6) & 0x3) - 1;
|
||||||
vi_bytes[j] = (b0 & 0xFF) | ((b1 & 0xFF) << 8) | ((b2 & 0xFF) << 16) | ((b3 & 0xFF) << 24);
|
vi_bytes[j] = (c0 & 0xFF) | ((c1 & 0xFF) << 8) | ((c2 & 0xFF) << 16) | ((c3 & 0xFF) << 24);
|
||||||
|
}
|
||||||
|
#pragma unroll
|
||||||
|
for (int j = 0; j < 4; ++j) {
|
||||||
|
const int shift = j * 8;
|
||||||
|
const int codes = (v1 >> shift) & 0xFF;
|
||||||
|
const int c0 = ((codes >> 0) & 0x3) - 1;
|
||||||
|
const int c1 = ((codes >> 2) & 0x3) - 1;
|
||||||
|
const int c2 = ((codes >> 4) & 0x3) - 1;
|
||||||
|
const int c3 = ((codes >> 6) & 0x3) - 1;
|
||||||
|
vi_bytes[4 + j] = (c0 & 0xFF) | ((c1 & 0xFF) << 8) | ((c2 & 0xFF) << 16) | ((c3 & 0xFF) << 24);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Compute dot product for this 32-element chunk
|
// Compute dot product for this 32-element chunk
|
||||||
@@ -712,9 +778,9 @@ static __device__ __forceinline__ float vec_dot_q1_0_q8_1(
|
|||||||
sumi = ggml_cuda_dp4a(vi_bytes[j], u, sumi);
|
sumi = ggml_cuda_dp4a(vi_bytes[j], u, sumi);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Apply Q1_0's single scale and this chunk's Q8_1 scale
|
// Apply Q2_0's single scale and this chunk's Q8_1 scale
|
||||||
const float d8 = __low2float(bq8_1_chunk->ds);
|
const float d8 = __low2float(bq8_1_chunk->ds);
|
||||||
return d1 * d8 * sumi;
|
return d2 * d8 * sumi;
|
||||||
}
|
}
|
||||||
|
|
||||||
static __device__ __forceinline__ float vec_dot_q4_0_q8_1(
|
static __device__ __forceinline__ float vec_dot_q4_0_q8_1(
|
||||||
|
|||||||
Vendored
-4
@@ -6,10 +6,6 @@
|
|||||||
#include <hip/hip_fp16.h>
|
#include <hip/hip_fp16.h>
|
||||||
#include <hip/hip_bf16.h>
|
#include <hip/hip_bf16.h>
|
||||||
|
|
||||||
#if defined(GGML_HIP_ROCWMMA_FATTN)
|
|
||||||
#include <rocwmma/rocwmma-version.hpp>
|
|
||||||
#endif // defined(GGML_HIP_ROCWMMA_FATTN)
|
|
||||||
|
|
||||||
#ifdef GGML_USE_NCCL
|
#ifdef GGML_USE_NCCL
|
||||||
#include <rccl/rccl.h>
|
#include <rccl/rccl.h>
|
||||||
#endif // GGML_USE_NCCL
|
#endif // GGML_USE_NCCL
|
||||||
|
|||||||
@@ -0,0 +1,246 @@
|
|||||||
|
|
||||||
|
message(STATUS "Using ET backend")
|
||||||
|
|
||||||
|
# Configure ET platform path
|
||||||
|
if (DEFINED ENV{ET_PLATFORM})
|
||||||
|
set(ET_PLATFORM_PATH $ENV{ET_PLATFORM})
|
||||||
|
else()
|
||||||
|
set(ET_PLATFORM_PATH "/opt/et")
|
||||||
|
endif()
|
||||||
|
|
||||||
|
# Use sysemu for ET backend if compiled with `-DGGML_ET_SYSEMU=ON`
|
||||||
|
if (GGML_ET_SYSEMU)
|
||||||
|
message(STATUS "Using ET backend with sysemu instead of hardware")
|
||||||
|
else()
|
||||||
|
message(STATUS "Using ET backend with hardware device")
|
||||||
|
endif()
|
||||||
|
|
||||||
|
# Add ET platform CMake modules and config files to search paths
|
||||||
|
list(APPEND CMAKE_PREFIX_PATH ${ET_PLATFORM_PATH}/lib/cmake)
|
||||||
|
list(APPEND CMAKE_MODULE_PATH ${ET_PLATFORM_PATH}/lib/cmake)
|
||||||
|
include(aifoundry-utils/ProjectFunctions)
|
||||||
|
|
||||||
|
message(STATUS "Using ET Platform at ${ET_PLATFORM_PATH}")
|
||||||
|
|
||||||
|
find_package(runtime REQUIRED)
|
||||||
|
|
||||||
|
# Kernel list
|
||||||
|
set(KERNELS
|
||||||
|
el_map_f32
|
||||||
|
flash_attn_ext_f32
|
||||||
|
glu_f32
|
||||||
|
scale_f32
|
||||||
|
mul_mat_f32
|
||||||
|
mul_mat_f32_matrix_engine
|
||||||
|
mul_mat_id_f32
|
||||||
|
mul_mat_id_Q4_0
|
||||||
|
mul_mat_id_Q8_0
|
||||||
|
mul_mat_Q8_0
|
||||||
|
mul_mat_Q4_0
|
||||||
|
mul_mat_Q4_0_matrix_engine
|
||||||
|
mul_mat_f16
|
||||||
|
mul_mat_f16_matrix_engine
|
||||||
|
rope_f32
|
||||||
|
unary_f32
|
||||||
|
sqr_f32
|
||||||
|
clamp_f32
|
||||||
|
sum_rows_f32
|
||||||
|
mean_f32
|
||||||
|
cumsum_f32
|
||||||
|
norm_f32
|
||||||
|
l2_norm_f32
|
||||||
|
group_norm_f32
|
||||||
|
rms_norm_f32
|
||||||
|
rms_norm_mul_f32
|
||||||
|
softmax_f32
|
||||||
|
im2col
|
||||||
|
get_rows_f32
|
||||||
|
concat_f32
|
||||||
|
repeat_f32
|
||||||
|
rwkv_wkv6_f32
|
||||||
|
rwkv_wkv7_f32
|
||||||
|
gated_delta_net_f32
|
||||||
|
cont_f32
|
||||||
|
cont_f16
|
||||||
|
cpy_f32_f16
|
||||||
|
flash_attn_ext_f16_me
|
||||||
|
set_rows_f32
|
||||||
|
set_f32
|
||||||
|
fill_f32
|
||||||
|
pad_f32
|
||||||
|
diag_f32
|
||||||
|
tri_f32
|
||||||
|
solve_tri_f32
|
||||||
|
ssm_conv_f32
|
||||||
|
ssm_scan_f32
|
||||||
|
conv_2d_f32_me
|
||||||
|
memops
|
||||||
|
uberkernel
|
||||||
|
)
|
||||||
|
|
||||||
|
# Kernels that we support dispatch form Uberkernel
|
||||||
|
set(UBERKERNEL_SUPPORTED_KERNELS
|
||||||
|
el_map_f32
|
||||||
|
# unary_f32
|
||||||
|
# cpy_f32_f16
|
||||||
|
# cont_f32
|
||||||
|
# get_rows_f32
|
||||||
|
concat_f32
|
||||||
|
cont_f16
|
||||||
|
cumsum_f32
|
||||||
|
diag_f32
|
||||||
|
fill_f32
|
||||||
|
flash_attn_ext_f16_me
|
||||||
|
flash_attn_ext_f32
|
||||||
|
gated_delta_net_f32
|
||||||
|
glu_f32
|
||||||
|
group_norm_f32
|
||||||
|
im2col
|
||||||
|
l2_norm_f32
|
||||||
|
mul_mat_f16
|
||||||
|
mul_mat_f16_matrix_engine
|
||||||
|
mul_mat_f32
|
||||||
|
mul_mat_f32_matrix_engine
|
||||||
|
mul_mat_id_f32
|
||||||
|
mul_mat_Q4_0
|
||||||
|
mul_mat_Q8_0
|
||||||
|
norm_f32
|
||||||
|
pad_f32
|
||||||
|
repeat_f32
|
||||||
|
rms_norm_f32
|
||||||
|
rms_norm_mul_f32
|
||||||
|
rope_f32
|
||||||
|
rwkv_wkv6_f32
|
||||||
|
rwkv_wkv7_f32
|
||||||
|
scale_f32
|
||||||
|
set_f32
|
||||||
|
set_rows_f32
|
||||||
|
softmax_f32
|
||||||
|
solve_tri_f32
|
||||||
|
sqr_f32
|
||||||
|
# ssm_conv_f32
|
||||||
|
ssm_scan_f32
|
||||||
|
sum_rows_f32
|
||||||
|
tri_f32
|
||||||
|
)
|
||||||
|
|
||||||
|
set(UBERKERNEL_MAP_HPP ${CMAKE_CURRENT_BINARY_DIR}/et-kernels/ggml-et-uberkernel-kernel-map.h)
|
||||||
|
set(UBERKERNEL_MAP_CPP ${CMAKE_CURRENT_BINARY_DIR}/et-kernels/ggml-et-uberkernel-kernel-map.cpp)
|
||||||
|
|
||||||
|
set(UBERKERNEL_KERNELS_SORTED ${UBERKERNEL_SUPPORTED_KERNELS})
|
||||||
|
list(SORT UBERKERNEL_KERNELS_SORTED)
|
||||||
|
|
||||||
|
set(UBERKERNEL_ENUM_ENTRIES "")
|
||||||
|
set(UBERKERNEL_MAP_ENTRIES "")
|
||||||
|
set(_uk_idx 1)
|
||||||
|
foreach(KERNEL ${UBERKERNEL_KERNELS_SORTED})
|
||||||
|
string(TOUPPER ${KERNEL} _uk_upper)
|
||||||
|
string(APPEND UBERKERNEL_ENUM_ENTRIES
|
||||||
|
" GGML_ET_UBERKERNEL_KERNEL_${_uk_upper} = ${_uk_idx},\n")
|
||||||
|
string(APPEND UBERKERNEL_MAP_ENTRIES
|
||||||
|
" {\"${KERNEL}\", GGML_ET_UBERKERNEL_KERNEL_${_uk_upper}},\n")
|
||||||
|
math(EXPR _uk_idx "${_uk_idx} + 1")
|
||||||
|
endforeach()
|
||||||
|
|
||||||
|
configure_file(
|
||||||
|
${CMAKE_CURRENT_SOURCE_DIR}/cmake/ggml-et-uberkernel-kernel-map.h.in
|
||||||
|
${UBERKERNEL_MAP_HPP}
|
||||||
|
@ONLY)
|
||||||
|
configure_file(
|
||||||
|
${CMAKE_CURRENT_SOURCE_DIR}/cmake/ggml-et-uberkernel-kernel-map.cpp.in
|
||||||
|
${UBERKERNEL_MAP_CPP}
|
||||||
|
@ONLY)
|
||||||
|
|
||||||
|
add_custom_target(et-uberkernel-map
|
||||||
|
DEPENDS ${UBERKERNEL_MAP_HPP} ${UBERKERNEL_MAP_CPP}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Build ET kernels (cross-compiled in subdirectory scope)
|
||||||
|
add_subdirectory(et-kernels)
|
||||||
|
|
||||||
|
# Embed kernels into C++ source
|
||||||
|
set(EMBED_SCRIPT ${CMAKE_CURRENT_SOURCE_DIR}/cmake/embed_one_kernel.cmake)
|
||||||
|
set(EMBED_HPP ${CMAKE_CURRENT_BINARY_DIR}/et-kernels/ggml-et-kernels-embed.hpp)
|
||||||
|
set(EMBED_CPP ${CMAKE_CURRENT_BINARY_DIR}/et-kernels/ggml-et-kernels-embed.cpp)
|
||||||
|
set(EMBED_DIR ${CMAKE_CURRENT_BINARY_DIR}/et-kernels/embed)
|
||||||
|
file(MAKE_DIRECTORY ${EMBED_DIR})
|
||||||
|
|
||||||
|
set(EMBED_KERNEL_SOURCES)
|
||||||
|
set(EMBED_EXTERNS "")
|
||||||
|
set(EMBED_MAP_ENTRIES "")
|
||||||
|
|
||||||
|
foreach(KERNEL ${KERNELS})
|
||||||
|
set(ELF_PATH ${CMAKE_CURRENT_BINARY_DIR}/et-kernels/${KERNEL}.elf)
|
||||||
|
set(OUT_CPP ${EMBED_DIR}/${KERNEL}.cpp)
|
||||||
|
|
||||||
|
add_custom_command(
|
||||||
|
OUTPUT ${OUT_CPP}
|
||||||
|
COMMAND ${CMAKE_COMMAND}
|
||||||
|
-DELF_FILE=${ELF_PATH}
|
||||||
|
-DOUT_FILE=${OUT_CPP}
|
||||||
|
-DVAR_NAME=${KERNEL}
|
||||||
|
-P ${EMBED_SCRIPT}
|
||||||
|
DEPENDS ${KERNEL}.elf ${EMBED_SCRIPT}
|
||||||
|
COMMENT "Embedding ${KERNEL}.elf"
|
||||||
|
VERBATIM
|
||||||
|
)
|
||||||
|
list(APPEND EMBED_KERNEL_SOURCES ${OUT_CPP})
|
||||||
|
|
||||||
|
string(APPEND EMBED_EXTERNS
|
||||||
|
"extern unsigned char ${KERNEL}_data[];\n"
|
||||||
|
"extern const uint64_t ${KERNEL}_len;\n")
|
||||||
|
string(APPEND EMBED_MAP_ENTRIES
|
||||||
|
" {\"${KERNEL}\", {${KERNEL}_data, ${KERNEL}_len}},\n")
|
||||||
|
endforeach()
|
||||||
|
|
||||||
|
configure_file(
|
||||||
|
${CMAKE_CURRENT_SOURCE_DIR}/cmake/ggml-et-kernels-embed.hpp.in
|
||||||
|
${EMBED_HPP}
|
||||||
|
@ONLY)
|
||||||
|
configure_file(
|
||||||
|
${CMAKE_CURRENT_SOURCE_DIR}/cmake/ggml-et-kernels-embed.cpp.in
|
||||||
|
${EMBED_CPP}
|
||||||
|
@ONLY)
|
||||||
|
|
||||||
|
add_custom_target(et-kernels-embed ALL
|
||||||
|
DEPENDS ${EMBED_KERNEL_SOURCES} ${EMBED_HPP} ${EMBED_CPP} et-uberkernel-map
|
||||||
|
)
|
||||||
|
|
||||||
|
ggml_add_backend_library(ggml-et
|
||||||
|
ggml-et.cpp
|
||||||
|
ggml-et-kernels.cpp
|
||||||
|
ggml-et-memops.cpp
|
||||||
|
ggml-et-ops.cpp
|
||||||
|
ggml-et-cpu-compare.cpp
|
||||||
|
)
|
||||||
|
|
||||||
|
# Mark generated files as such
|
||||||
|
set_source_files_properties(
|
||||||
|
${EMBED_CPP}
|
||||||
|
${EMBED_HPP}
|
||||||
|
${EMBED_KERNEL_SOURCES}
|
||||||
|
${CMAKE_CURRENT_BINARY_DIR}/et-kernels/ggml-et-uberkernel-kernel-map.cpp
|
||||||
|
${CMAKE_CURRENT_BINARY_DIR}/et-kernels/ggml-et-uberkernel-kernel-map.h
|
||||||
|
PROPERTIES GENERATED TRUE
|
||||||
|
)
|
||||||
|
|
||||||
|
# Add embedded kernel sources
|
||||||
|
target_sources(ggml-et PRIVATE
|
||||||
|
${EMBED_CPP}
|
||||||
|
${EMBED_HPP}
|
||||||
|
${EMBED_KERNEL_SOURCES}
|
||||||
|
${CMAKE_CURRENT_BINARY_DIR}/et-kernels/ggml-et-uberkernel-kernel-map.cpp
|
||||||
|
${CMAKE_CURRENT_BINARY_DIR}/et-kernels/ggml-et-uberkernel-kernel-map.h
|
||||||
|
)
|
||||||
|
|
||||||
|
# Include directory for embedded headers
|
||||||
|
target_include_directories(ggml-et PRIVATE ${CMAKE_CURRENT_BINARY_DIR}/et-kernels)
|
||||||
|
|
||||||
|
target_link_libraries(ggml-et PRIVATE runtime::etrt_static deviceLayer::deviceLayer)
|
||||||
|
target_compile_definitions(ggml-et PRIVATE GGML_ET_UBERKERNEL_HOST_LOOKUP)
|
||||||
|
if (GGML_ET_SYSEMU)
|
||||||
|
target_compile_definitions(ggml-et PRIVATE GGML_ET_SYSEMU=1)
|
||||||
|
endif()
|
||||||
|
|
||||||
|
# Ensure kernels are built and embedded before the backend library
|
||||||
|
add_dependencies(ggml-et et-kernels-embed et-uberkernel-map)
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
# Inputs (via -D):
|
||||||
|
# ELF_FILE - path to source .elf
|
||||||
|
# OUT_FILE - path to output .cpp
|
||||||
|
# VAR_NAME - C symbol base name (kernel name)
|
||||||
|
|
||||||
|
file(READ "${ELF_FILE}" HEX HEX)
|
||||||
|
string(LENGTH "${HEX}" HEX_LEN)
|
||||||
|
math(EXPR SIZE "${HEX_LEN} / 2")
|
||||||
|
string(REGEX REPLACE "(..)" "0x\\1," BYTES "${HEX}")
|
||||||
|
|
||||||
|
file(WRITE "${OUT_FILE}"
|
||||||
|
"// Auto-generated by embed_one_kernel.cmake. Do not edit.\n"
|
||||||
|
"#include <cstdint>\n"
|
||||||
|
"unsigned char ${VAR_NAME}_data[${SIZE}] = { ${BYTES} };\n"
|
||||||
|
"extern const uint64_t ${VAR_NAME}_len = ${SIZE};\n")
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
// Auto-generated kernel embeddings. Do not edit.
|
||||||
|
#include "ggml-et-kernels-embed.hpp"
|
||||||
|
|
||||||
|
const std::unordered_map<std::string, std::pair<const unsigned char*, uint64_t>> ggml_et_embedded_kernels = {
|
||||||
|
@EMBED_MAP_ENTRIES@
|
||||||
|
};
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
// Auto-generated kernel embeddings. Do not edit.
|
||||||
|
#pragma once
|
||||||
|
|
||||||
|
#include <cstdint>
|
||||||
|
#include <unordered_map>
|
||||||
|
#include <string>
|
||||||
|
#include <utility>
|
||||||
|
|
||||||
|
@EMBED_EXTERNS@
|
||||||
|
|
||||||
|
// Kernel name -> (data, length) lookup map
|
||||||
|
extern const std::unordered_map<std::string, std::pair<const unsigned char*, uint64_t>> ggml_et_embedded_kernels;
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
// Auto-generated uberkernel kernel-id mapping. Do not edit.
|
||||||
|
#include "ggml-et-uberkernel-kernel-map.h"
|
||||||
|
|
||||||
|
#ifdef GGML_ET_UBERKERNEL_HOST_LOOKUP
|
||||||
|
#include <string>
|
||||||
|
#include <unordered_map>
|
||||||
|
|
||||||
|
uint16_t ggml_et_uberkernel_kernel_id_from_name(const char * kernel_name) {
|
||||||
|
if (kernel_name == nullptr) {
|
||||||
|
return GGML_ET_UBERKERNEL_KERNEL_INVALID;
|
||||||
|
}
|
||||||
|
static const std::unordered_map<std::string, uint16_t> kernel_id_map = {
|
||||||
|
@UBERKERNEL_MAP_ENTRIES@
|
||||||
|
};
|
||||||
|
auto it = kernel_id_map.find(std::string(kernel_name));
|
||||||
|
return it == kernel_id_map.end() ? GGML_ET_UBERKERNEL_KERNEL_INVALID : it->second;
|
||||||
|
}
|
||||||
|
#endif
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
// Auto-generated uberkernel kernel-id mapping. Do not edit.
|
||||||
|
#pragma once
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
enum ggml_et_uberkernel_kernel_id {
|
||||||
|
GGML_ET_UBERKERNEL_KERNEL_INVALID = 0,
|
||||||
|
@UBERKERNEL_ENUM_ENTRIES@
|
||||||
|
};
|
||||||
|
|
||||||
|
#ifdef GGML_ET_UBERKERNEL_HOST_LOOKUP
|
||||||
|
uint16_t ggml_et_uberkernel_kernel_id_from_name(const char * kernel_name);
|
||||||
|
#endif
|
||||||
@@ -0,0 +1,137 @@
|
|||||||
|
# ggml-et: Device kernels (cross-compiled within the main build)
|
||||||
|
#
|
||||||
|
# The RISC-V toolchain is set up in-scope so these targets use the
|
||||||
|
# cross-compiler while the rest of the build uses the host compiler.
|
||||||
|
# This keeps kernels in compile_commands.json for full IDE support.
|
||||||
|
|
||||||
|
# --- RISC-V toolchain setup (scoped to this directory) ---
|
||||||
|
set(TOOLCHAIN_DIR ${ET_PLATFORM_PATH})
|
||||||
|
include(${ET_PLATFORM_PATH}/lib/cmake/riscv64-ec-toolchain.cmake)
|
||||||
|
set(CMAKE_ADDR2LINE "${TOOLCHAIN_DIR}/bin/riscv64-unknown-elf-addr2line")
|
||||||
|
set(CMAKE_LINKER_TYPE LLD)
|
||||||
|
|
||||||
|
# Ensure kernels are built in this directory even if a global output directory is set
|
||||||
|
set(CMAKE_RUNTIME_OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
|
||||||
|
|
||||||
|
message(STATUS "ET kernels using RISC-V toolchain at: ${TOOLCHAIN_DIR}")
|
||||||
|
|
||||||
|
# DeviceUtils provides the add_riscv_executable macro
|
||||||
|
list(APPEND CMAKE_MODULE_PATH "${ET_PLATFORM_PATH}/lib/cmake/cmake-modules")
|
||||||
|
list(APPEND CMAKE_PREFIX_PATH "${ET_PLATFORM_PATH}/lib/cmake")
|
||||||
|
include(DeviceUtils)
|
||||||
|
|
||||||
|
find_package(et-common-libs REQUIRED)
|
||||||
|
find_package(esperantoTrace REQUIRED)
|
||||||
|
|
||||||
|
# --- Kernel configuration ---
|
||||||
|
if(NOT DEFINED ADDRESS)
|
||||||
|
set(ADDRESS "0x8005801000")
|
||||||
|
message(STATUS "ADDRESS not specified, using default: ${ADDRESS}")
|
||||||
|
endif()
|
||||||
|
|
||||||
|
set(LINKER_SCRIPT ${CMAKE_CURRENT_SOURCE_DIR}/src/linker.ld)
|
||||||
|
set(CHECK_SCRIPT ${CMAKE_CURRENT_SOURCE_DIR}/scripts/check_unimplemented_instructions.sh)
|
||||||
|
|
||||||
|
# Track address changes to trigger relinking
|
||||||
|
set(ADDRESS_FILE ${CMAKE_CURRENT_BINARY_DIR}/et_address.txt)
|
||||||
|
file(CONFIGURE OUTPUT ${ADDRESS_FILE} CONTENT "${ADDRESS}" @ONLY)
|
||||||
|
|
||||||
|
# KERNELS defined in upper CMakeLists.txt
|
||||||
|
foreach(KERNEL ${KERNELS})
|
||||||
|
add_riscv_executable(${KERNEL})
|
||||||
|
target_sources(${KERNEL}.elf PRIVATE
|
||||||
|
src/${KERNEL}.c
|
||||||
|
src/crt.S
|
||||||
|
)
|
||||||
|
target_include_directories(${KERNEL}.elf PRIVATE
|
||||||
|
${CMAKE_CURRENT_SOURCE_DIR}/src
|
||||||
|
${CMAKE_CURRENT_SOURCE_DIR}/..
|
||||||
|
${CMAKE_CURRENT_BINARY_DIR}
|
||||||
|
${CMAKE_SOURCE_DIR}/ggml/include
|
||||||
|
${CMAKE_SOURCE_DIR}/ggml/src
|
||||||
|
)
|
||||||
|
target_link_libraries(${KERNEL}.elf PRIVATE et-common-libs::cm-umode)
|
||||||
|
# C-only flags — must not apply to .S files
|
||||||
|
target_compile_options(${KERNEL}.elf PRIVATE
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-fno-zero-initialized-in-bss>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-ffreestanding>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-std=gnu99>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-ffat-lto-objects>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-mcmodel=medany>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-mabi=lp64f>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-march=rv64imf>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-ffunction-sections>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-fdata-sections>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-O3>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-g0>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-nostdlib>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-ffreestanding>
|
||||||
|
)
|
||||||
|
target_link_options(${KERNEL}.elf PRIVATE
|
||||||
|
-Wl,--defsym=BASE_ADDRESS=${ADDRESS}
|
||||||
|
-Wl,--entry=_start
|
||||||
|
)
|
||||||
|
# Append to LINK_DEPENDS (macro already sets it for the linker script)
|
||||||
|
set_property(TARGET ${KERNEL}.elf APPEND PROPERTY
|
||||||
|
LINK_DEPENDS "${ADDRESS_FILE}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Post-build: strip and check (fails build if check script fails)
|
||||||
|
add_custom_command(TARGET ${KERNEL}.elf POST_BUILD
|
||||||
|
COMMAND ${CMAKE_STRIP} --strip-debug $<TARGET_FILE:${KERNEL}.elf>
|
||||||
|
COMMAND ${CHECK_SCRIPT}
|
||||||
|
${CMAKE_OBJDUMP} ${CMAKE_ADDR2LINE} $<TARGET_FILE:${KERNEL}.elf>
|
||||||
|
DEPENDS ${CHECK_SCRIPT}
|
||||||
|
VERBATIM
|
||||||
|
)
|
||||||
|
endforeach()
|
||||||
|
|
||||||
|
add_dependencies(uberkernel.elf et-uberkernel-map)
|
||||||
|
|
||||||
|
# Each supported kernel is compiled in its own translation unit with
|
||||||
|
# -Dentry_point=<kernel>_entry
|
||||||
|
# so symbols and macros don't leak between kernels. The dispatcher
|
||||||
|
# (uberkernel.c) calls the renamed entries via extern declarations.
|
||||||
|
#
|
||||||
|
# HACK: we need to supresse _me kernels from setting up SCP themselves
|
||||||
|
set(_UBER_ME_KERNELS mul_mat_f16_matrix_engine mul_mat_f32_matrix_engine flash_attn_ext_f16_me)
|
||||||
|
|
||||||
|
foreach(UK_KERNEL ${UBERKERNEL_SUPPORTED_KERNELS})
|
||||||
|
set(_obj uber_${UK_KERNEL})
|
||||||
|
add_library(${_obj} OBJECT src/${UK_KERNEL}.c)
|
||||||
|
target_compile_definitions(${_obj} PRIVATE "entry_point=${UK_KERNEL}_entry" ET_UBERKERNEL)
|
||||||
|
target_include_directories(${_obj} PRIVATE
|
||||||
|
${CMAKE_CURRENT_SOURCE_DIR}/src
|
||||||
|
${CMAKE_CURRENT_SOURCE_DIR}/..
|
||||||
|
${CMAKE_CURRENT_BINARY_DIR}
|
||||||
|
${CMAKE_SOURCE_DIR}/ggml/include
|
||||||
|
${CMAKE_SOURCE_DIR}/ggml/src
|
||||||
|
)
|
||||||
|
target_link_libraries(${_obj} PRIVATE et-common-libs::cm-umode)
|
||||||
|
target_compile_options(${_obj} PRIVATE
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-fno-zero-initialized-in-bss>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-ffreestanding>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-std=gnu99>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-ffat-lto-objects>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-mcmodel=medany>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-mabi=lp64f>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-march=rv64imf>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-ffunction-sections>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-fdata-sections>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-O3>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-g0>
|
||||||
|
$<$<COMPILE_LANGUAGE:C>:-nostdlib>
|
||||||
|
)
|
||||||
|
# ME kernels: suppress setup_cache_scp() (called once by the dispatcher)
|
||||||
|
if(UK_KERNEL IN_LIST _UBER_ME_KERNELS)
|
||||||
|
target_compile_definitions(${_obj} PRIVATE UBERKERNEL_SUPPRESS_SCP_SETUP)
|
||||||
|
endif()
|
||||||
|
target_sources(uberkernel.elf PRIVATE $<TARGET_OBJECTS:${_obj}>)
|
||||||
|
endforeach()
|
||||||
|
|
||||||
|
# Print summary
|
||||||
|
message(STATUS "GGML ET Kernels configured:")
|
||||||
|
foreach(KERNEL ${KERNELS})
|
||||||
|
message(STATUS " - ${KERNEL}")
|
||||||
|
endforeach()
|
||||||
|
message(STATUS "Base address: ${ADDRESS}")
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
OBJDUMP=$1
|
||||||
|
ADDR2LINE=$2
|
||||||
|
TARGET_DEBUG=$3
|
||||||
|
TARGET_ASM=${TARGET_DEBUG}.S
|
||||||
|
BAD_INST_FILE=${TARGET_DEBUG}-BAD-INST.log
|
||||||
|
|
||||||
|
# grep expression to find unimplemented instructions
|
||||||
|
UNIMPLEMENTED_EXPR="fdiv.s\\|fsqrt.s\\|fcvt.l.s\\|fcvt.lu.s\\|fcvt.s.l\\|fcvt.s.lu\\|fdiv.pi\\|fdivu.pi\\|fremu.pi\\|frem.pi\\|fdiv.ps\\|fsqrt.ps\\|frsq.ps\\|fsin.ps"
|
||||||
|
|
||||||
|
# dump assembly into .S file
|
||||||
|
${OBJDUMP} -lwdSC ${TARGET_DEBUG} > ${TARGET_ASM}
|
||||||
|
|
||||||
|
# check with grep for unimplemented instructions
|
||||||
|
# Note: The exit status is 0 if selected lines are found, and 1 if not found.
|
||||||
|
grep ${UNIMPLEMENTED_EXPR} ${TARGET_ASM} > /dev/null
|
||||||
|
ret=$?
|
||||||
|
|
||||||
|
if [ ${ret} -eq 0 ]
|
||||||
|
then
|
||||||
|
# unimplemented instructions are found
|
||||||
|
echo -e "BUILD ERROR: Executable file ${TARGET_DEBUG} contains unimplemented instructions. Please review the lines of code listed in ${BAD_INST_FILE}"
|
||||||
|
echo -e "\t For further details, please read paragraph 3.4 of the ETSoC-1 Programmer's Reference Manual (PRM)"
|
||||||
|
|
||||||
|
# addr2line
|
||||||
|
grep ${UNIMPLEMENTED_EXPR} ${TARGET_ASM} | cut -d: -f 1 | ${ADDR2LINE} -i -e ${TARGET_DEBUG} > ${BAD_INST_FILE}
|
||||||
|
grep ${UNIMPLEMENTED_EXPR} ${TARGET_ASM} >> ${BAD_INST_FILE}
|
||||||
|
echo "------------------------------------------------------------"
|
||||||
|
cat ${BAD_INST_FILE}
|
||||||
|
echo "------------------------------------------------------------"
|
||||||
|
exit 1
|
||||||
|
|
||||||
|
else
|
||||||
|
rm -f ${BAD_INST_FILE}
|
||||||
|
fi
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
LOG="llama_bench_$(date +%Y%m%d_%H%M%S).log"
|
||||||
|
|
||||||
|
{
|
||||||
|
echo "===== START ====="
|
||||||
|
date
|
||||||
|
hostname
|
||||||
|
uname -a
|
||||||
|
echo "Command:"
|
||||||
|
echo "./build/bin/llama-bench -m ../../models/Llama-3.2-1B-Instruct-Q8_0.gguf -fa 0 -p 32,64,128,256,512 -n 32,64,128,256,512"
|
||||||
|
echo "================="
|
||||||
|
|
||||||
|
./build/bin/llama-bench \
|
||||||
|
-m ../../models/Llama-3.2-1B-Instruct-Q8_0.gguf \
|
||||||
|
-fa 0 \
|
||||||
|
-p 32,64,128,256,512 \
|
||||||
|
-n 32,64,128,256,512
|
||||||
|
|
||||||
|
echo "===== END ====="
|
||||||
|
date
|
||||||
|
} 2>&1 | tee "$LOG"
|
||||||
@@ -0,0 +1,997 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// ET Vectorized Block Operations Library
|
||||||
|
// Provides optimized block-level operations using ET hardware vector instructions
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#ifndef BLOCK_OPS_H
|
||||||
|
# define BLOCK_OPS_H
|
||||||
|
|
||||||
|
# include "math_fp.h"
|
||||||
|
# include "quants.h"
|
||||||
|
|
||||||
|
# include <stdint.h>
|
||||||
|
|
||||||
|
//******************************************************************************
|
||||||
|
// Block Dot Product Operations
|
||||||
|
//******************************************************************************
|
||||||
|
inline void __attribute__((always_inline)) excl_mode(uint64_t val) {
|
||||||
|
__asm__ __volatile__("csrw 0x7d3, %[csr_enc]\n" : : [csr_enc] "r"(val) : "x31");
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline float compute_block_dot_product_q4_0(const block_q4_0 * a_block, const float * b_col_start) {
|
||||||
|
// Set mask register to enable all 8 vector elements
|
||||||
|
unsigned long temp_mask;
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(temp_mask)); // Save current mask
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF"); // Enable all 8 elements
|
||||||
|
|
||||||
|
// Use f10 as accumulator, init to 0
|
||||||
|
__asm__ volatile("fbci.ps f10, 0" ::: "f10");
|
||||||
|
|
||||||
|
static const int32_t gather_pattern[8] = { 0, 1, 2, 3, 4, 5, 6, 7 };
|
||||||
|
__asm__ volatile("flw.ps f31, %[gather]\n" : : [gather] "m"(*(const int32_t (*)[8]) gather_pattern) : "f31");
|
||||||
|
|
||||||
|
// Process 32 elements in 2 chunks of 16 elements (8 bytes) each
|
||||||
|
for (int chunk = 0; chunk < 2; chunk++) {
|
||||||
|
int offset_a = chunk * 8;
|
||||||
|
int offset_b_low = chunk * 8; // Activations for lower nibbles
|
||||||
|
int offset_b_high = chunk * 8 + 16; // Activations for upper nibbles (16 elements later)
|
||||||
|
|
||||||
|
__asm__ volatile(
|
||||||
|
"fgb.ps f11, f31(%[a_ptr])\n" // Gather 8 bytes (16 packed q4_0 weights)
|
||||||
|
|
||||||
|
// 1. Extract & Multiply Lower Nibbles
|
||||||
|
"fandi.pi f12, f11, 15\n" // Mask lower 4 bits (x & 0xF)
|
||||||
|
"faddi.pi f12, f12, -8\n" // GGML offset to signed: (x & 0xF) - 8
|
||||||
|
"fcvt.ps.pw f12, f12, rne\n" // Convert INT32 to FP32
|
||||||
|
"flw.ps f13, 0(%[b_low])\n" // Load 8 B values (floats)
|
||||||
|
"fmadd.ps f10, f12, f13, f10, rne\n" // acc += A_low * B_low
|
||||||
|
|
||||||
|
// 2. Extract & Multiply Upper Nibbles
|
||||||
|
"fsrli.pi f14, f11, 4\n" // Shift upper 4 bits down
|
||||||
|
"fandi.pi f14, f14, 15\n" // Mask new lower 4 bits
|
||||||
|
"faddi.pi f14, f14, -8\n" // GGML offset to signed
|
||||||
|
"fcvt.ps.pw f14, f14, rne\n" // Convert INT32 to FP32
|
||||||
|
"flw.ps f15, 0(%[b_high])\n" // Load next 8 B values (floats)
|
||||||
|
"fmadd.ps f10, f14, f15, f10, rne\n" // acc += A_high * B_high
|
||||||
|
:
|
||||||
|
: [a_ptr] "r"(&a_block->qs[offset_a]), [b_low] "r"(&b_col_start[offset_b_low]),
|
||||||
|
[b_high] "r"(&b_col_start[offset_b_high])
|
||||||
|
// Note: f10 is explicitly NOT listed in the clobbers here to ensure the compiler
|
||||||
|
// preserves the running sum across C loop iterations safely.
|
||||||
|
: "f11", "f12", "f13", "f14", "f15");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Horizontal sum: reduce f10 into a single scalar
|
||||||
|
float final_sum;
|
||||||
|
__asm__ __volatile__(
|
||||||
|
// Pairwise sum within each 128-bit half
|
||||||
|
"fswizz.ps f1, f10, 0xB1 \n\t" // Swaps: e0<->e1 and e2<->e3
|
||||||
|
"fadd.ps f2, f10, f1, rne \n\t"
|
||||||
|
// Complete the sum for each 128-bit half
|
||||||
|
"fswizz.ps f3, f2, 0x4E \n\t" // Swaps: e0,e1 <-> e2,e3
|
||||||
|
"fadd.ps f4, f2, f3, rne \n\t"
|
||||||
|
// Sum across the two 128b halfs
|
||||||
|
"fmvz.x.ps t0, f4, 4 \n\t"
|
||||||
|
"fbcx.ps f5, t0 \n\t"
|
||||||
|
"fadd.ps %[vout], f4, f5, rne \n\t"
|
||||||
|
: [vout] "=f"(final_sum)::"t0", "f1", "f2", "f3", "f4", "f5", "f10");
|
||||||
|
|
||||||
|
// Restore original mask
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(temp_mask));
|
||||||
|
|
||||||
|
const float scale = fp16_to_fp32(a_block->d);
|
||||||
|
return final_sum * scale;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Compute dot product between dequantized q8_0 block and f32 column vector
|
||||||
|
// Vectorized: processes 8 elements at a time using ET vector instructions
|
||||||
|
// Block size: 32 int8 values (QK8_0)
|
||||||
|
static inline float compute_block_dot_product_q8_0(const block_q8_0 * a_block, const float * b_col_start) {
|
||||||
|
// Set mask register to enable all 8 vector elements
|
||||||
|
unsigned long temp_mask;
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(temp_mask)); // Save current mask
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF"); // Enable all 8 elements
|
||||||
|
__asm__ volatile("fbci.pi f10, 0" ::: "f10"); // Use f10 as accumulator, init to 0
|
||||||
|
|
||||||
|
static const int32_t gather_pattern[8] = { 0, 1, 2, 3, 4, 5, 6, 7 };
|
||||||
|
|
||||||
|
__asm__ volatile("flw.ps f31, %[gather]\n" : : [gather] "m"(*(const int32_t (*)[8]) gather_pattern) : "f31");
|
||||||
|
|
||||||
|
// Process 32 elements in 4 chunks of 8 elements each
|
||||||
|
for (int chunk = 0; chunk < 4; chunk++) {
|
||||||
|
int offset = chunk << 3; // chunk * 8
|
||||||
|
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f12, %[b_vec]\n" // Load 8 B values (floats)
|
||||||
|
"fgb.ps f11, f31(%[a_ptr])\n" // Gather 8 int8 bytes from A using pattern
|
||||||
|
"fcvt.ps.pw f11, f11\n" // Convert int8 vector to float vector
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n" // acc += a_vec * b_vec (8-wide)
|
||||||
|
:
|
||||||
|
: [a_ptr] "r"(&a_block->qs[offset]), [b_vec] "m"(*(const float (*)[8]) & b_col_start[offset]),
|
||||||
|
[scale] "m"(a_block->d)
|
||||||
|
: "f10", "f11", "f12");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Horizontal sum: reduce f10 into a single scalar
|
||||||
|
float final_sum;
|
||||||
|
__asm__ __volatile__(
|
||||||
|
// Pairwise sum within each 128-bit half
|
||||||
|
"fswizz.ps f1, f10, 0xB1 \n\t" // Swaps: e0<->e1 and e2<->e3
|
||||||
|
"fadd.ps f2, f10, f1, rne \n\t"
|
||||||
|
// Complete the sum for each 128-bit half
|
||||||
|
"fswizz.ps f3, f2, 0x4E \n\t" // Swaps: e0,e1 <-> e2,e3
|
||||||
|
"fadd.ps f4, f2, f3, rne \n\t"
|
||||||
|
// Sum across the two 128b halfs
|
||||||
|
"fmvz.x.ps t0, f4, 4 \n\t"
|
||||||
|
"fbcx.ps f5, t0 \n\t"
|
||||||
|
"fadd.ps %[vout], f4, f5, rne \n\t"
|
||||||
|
: [vout] "=f"(final_sum)::"t0", "f10", "f2", "f3", "f4", "f5");
|
||||||
|
|
||||||
|
// Restore original mask
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(temp_mask));
|
||||||
|
|
||||||
|
const float scale = fp16_to_fp32(a_block->d);
|
||||||
|
return final_sum * scale;
|
||||||
|
}
|
||||||
|
|
||||||
|
//******************************************************************************
|
||||||
|
// Split-phase Q8_0 dot product API
|
||||||
|
//
|
||||||
|
// q8_dot_begin(st) — save mask, set mask 0xFF
|
||||||
|
// q8_dot_reset() — zero vector accumulator f20
|
||||||
|
// q8_dot_tile(q, b, n) — accumulate n Q8_0 blocks into f20
|
||||||
|
// q8_dot_reduce() — horizontal sum of f20, return scalar float
|
||||||
|
// q8_dot_teardown(st) — restore original mask
|
||||||
|
//
|
||||||
|
// Register contract:
|
||||||
|
// f20 — row accumulator (persistent across tiles, reset per row)
|
||||||
|
// f31 — gather pattern (reloaded per q8_dot_tile call)
|
||||||
|
// f10-f12 — scratch within tile
|
||||||
|
// f15 — scale broadcast within tile
|
||||||
|
// f1-f5, t0 — scratch within reduce
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
static inline void __attribute__((always_inline)) q8_dot_reset(void) {
|
||||||
|
__asm__ volatile("fbci.pi f20, 0" ::: "f20");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Accumulate n_blocks Q8_0 blocks into f20.
|
||||||
|
// Uses fg32b.ps (fast gather with scalar pattern) for aligned chunks,
|
||||||
|
// falls back to fgb.ps for chunks crossing a 32-byte boundary.
|
||||||
|
static inline void __attribute__((always_inline)) q8_dot_tile(const block_q8_0 * q_row,
|
||||||
|
const float * b_col,
|
||||||
|
int64_t n_blocks) {
|
||||||
|
const int32_t gather_pattern[8] = { 0, 1, 2, 3, 4, 5, 6, 7 };
|
||||||
|
const uint64_t gather_0_to_7 = 0x398a418820ULL;
|
||||||
|
|
||||||
|
__asm__ volatile("flw.ps f31, %[g]\n" : : [g] "m"(*(const int32_t (*)[8]) gather_pattern) : "f31");
|
||||||
|
|
||||||
|
for (int64_t kb = 0; kb < n_blocks; kb++) {
|
||||||
|
const block_q8_0 * blk = q_row + kb;
|
||||||
|
const float * b_ptr = b_col + (kb << 5);
|
||||||
|
const uintptr_t qs_addr = (uintptr_t) blk->qs;
|
||||||
|
const uintptr_t qs_aligned = qs_addr & ~(uintptr_t) 31;
|
||||||
|
const uintptr_t qs_low = qs_addr & 31;
|
||||||
|
const int fast_chunks = (int) ((32 - qs_low) >> 3);
|
||||||
|
|
||||||
|
if (fast_chunks >= 3) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbci.pi f10, 0\n"
|
||||||
|
"flw.ps f12, %[bv0]\n"
|
||||||
|
"fg32b.ps f11, %[gi](%[ap0])\n"
|
||||||
|
"fcvt.ps.pw f11, f11\n"
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
"flw.ps f12, %[bv1]\n"
|
||||||
|
"fg32b.ps f11, %[gi](%[ap1])\n"
|
||||||
|
"fcvt.ps.pw f11, f11\n"
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
"flw.ps f12, %[bv2]\n"
|
||||||
|
"fg32b.ps f11, %[gi](%[ap2])\n"
|
||||||
|
"fcvt.ps.pw f11, f11\n"
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
"flw.ps f12, %[bv3]\n"
|
||||||
|
"fgb.ps f11, f31(%[ap3])\n"
|
||||||
|
"fcvt.ps.pw f11, f11\n"
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
:
|
||||||
|
: [gi] "r"(gather_0_to_7), [ap0] "r"(qs_addr), [ap1] "r"(qs_aligned | ((qs_addr + 8) & 31)),
|
||||||
|
[ap2] "r"(qs_aligned | ((qs_addr + 16) & 31)), [ap3] "r"(&blk->qs[24]),
|
||||||
|
[bv0] "m"(*(const float (*)[8]) & b_ptr[0]), [bv1] "m"(*(const float (*)[8]) & b_ptr[8]),
|
||||||
|
[bv2] "m"(*(const float (*)[8]) & b_ptr[16]), [bv3] "m"(*(const float (*)[8]) & b_ptr[24])
|
||||||
|
: "f10", "f11", "f12");
|
||||||
|
} else if (fast_chunks == 2) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbci.pi f10, 0\n"
|
||||||
|
"flw.ps f12, %[bv0]\n"
|
||||||
|
"fg32b.ps f11, %[gi](%[ap0])\n"
|
||||||
|
"fcvt.ps.pw f11, f11\n"
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
"flw.ps f12, %[bv1]\n"
|
||||||
|
"fg32b.ps f11, %[gi](%[ap1])\n"
|
||||||
|
"fcvt.ps.pw f11, f11\n"
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
"flw.ps f12, %[bv2]\n"
|
||||||
|
"fgb.ps f11, f31(%[ap2])\n"
|
||||||
|
"fcvt.ps.pw f11, f11\n"
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
"flw.ps f12, %[bv3]\n"
|
||||||
|
"fgb.ps f11, f31(%[ap3])\n"
|
||||||
|
"fcvt.ps.pw f11, f11\n"
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
:
|
||||||
|
: [gi] "r"(gather_0_to_7), [ap0] "r"(qs_addr), [ap1] "r"(qs_aligned | ((qs_addr + 8) & 31)),
|
||||||
|
[ap2] "r"(&blk->qs[16]), [ap3] "r"(&blk->qs[24]), [bv0] "m"(*(const float (*)[8]) & b_ptr[0]),
|
||||||
|
[bv1] "m"(*(const float (*)[8]) & b_ptr[8]), [bv2] "m"(*(const float (*)[8]) & b_ptr[16]),
|
||||||
|
[bv3] "m"(*(const float (*)[8]) & b_ptr[24])
|
||||||
|
: "f10", "f11", "f12");
|
||||||
|
} else if (fast_chunks == 1) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbci.pi f10, 0\n"
|
||||||
|
"flw.ps f12, %[bv0]\n"
|
||||||
|
"fg32b.ps f11, %[gi](%[ap0])\n"
|
||||||
|
"fcvt.ps.pw f11, f11\n"
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
"flw.ps f12, %[bv1]\n"
|
||||||
|
"fgb.ps f11, f31(%[ap1])\n"
|
||||||
|
"fcvt.ps.pw f11, f11\n"
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
"flw.ps f12, %[bv2]\n"
|
||||||
|
"fgb.ps f11, f31(%[ap2])\n"
|
||||||
|
"fcvt.ps.pw f11, f11\n"
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
"flw.ps f12, %[bv3]\n"
|
||||||
|
"fgb.ps f11, f31(%[ap3])\n"
|
||||||
|
"fcvt.ps.pw f11, f11\n"
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
:
|
||||||
|
: [gi] "r"(gather_0_to_7), [ap0] "r"(qs_addr), [ap1] "r"(&blk->qs[8]), [ap2] "r"(&blk->qs[16]),
|
||||||
|
[ap3] "r"(&blk->qs[24]), [bv0] "m"(*(const float (*)[8]) & b_ptr[0]),
|
||||||
|
[bv1] "m"(*(const float (*)[8]) & b_ptr[8]), [bv2] "m"(*(const float (*)[8]) & b_ptr[16]),
|
||||||
|
[bv3] "m"(*(const float (*)[8]) & b_ptr[24])
|
||||||
|
: "f10", "f11", "f12");
|
||||||
|
} else {
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbci.pi f10, 0\n"
|
||||||
|
"flw.ps f12, %[bv0]\n"
|
||||||
|
"fgb.ps f11, f31(%[ap0])\n"
|
||||||
|
"fcvt.ps.pw f11, f11\n"
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
"flw.ps f12, %[bv1]\n"
|
||||||
|
"fgb.ps f11, f31(%[ap1])\n"
|
||||||
|
"fcvt.ps.pw f11, f11\n"
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
"flw.ps f12, %[bv2]\n"
|
||||||
|
"fgb.ps f11, f31(%[ap2])\n"
|
||||||
|
"fcvt.ps.pw f11, f11\n"
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
"flw.ps f12, %[bv3]\n"
|
||||||
|
"fgb.ps f11, f31(%[ap3])\n"
|
||||||
|
"fcvt.ps.pw f11, f11\n"
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
:
|
||||||
|
: [ap0] "r"(&blk->qs[0]), [ap1] "r"(&blk->qs[8]), [ap2] "r"(&blk->qs[16]), [ap3] "r"(&blk->qs[24]),
|
||||||
|
[bv0] "m"(*(const float (*)[8]) & b_ptr[0]), [bv1] "m"(*(const float (*)[8]) & b_ptr[8]),
|
||||||
|
[bv2] "m"(*(const float (*)[8]) & b_ptr[16]), [bv3] "m"(*(const float (*)[8]) & b_ptr[24])
|
||||||
|
: "f10", "f11", "f12");
|
||||||
|
}
|
||||||
|
|
||||||
|
// f20 += f10 * broadcast(scale) — hardware fp16→fp32 via FCVT.PS.F16
|
||||||
|
uint32_t scale_raw = (uint32_t) blk->d;
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbcx.ps f15, %[sb]\n"
|
||||||
|
"fcvt.ps.f16 f15, f15\n"
|
||||||
|
"fmadd.ps f20, f10, f15, f20\n"
|
||||||
|
:
|
||||||
|
: [sb] "r"(scale_raw)
|
||||||
|
: "f15", "f20");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Horizontal sum of 8-element vector accumulator f20.
|
||||||
|
static inline float __attribute__((always_inline)) q8_dot_reduce(void) {
|
||||||
|
float result;
|
||||||
|
__asm__ __volatile__(
|
||||||
|
"fswizz.ps f1, f20, 0xB1 \n\t"
|
||||||
|
"fadd.ps f2, f20, f1, rne \n\t"
|
||||||
|
"fswizz.ps f3, f2, 0x4E \n\t"
|
||||||
|
"fadd.ps f4, f2, f3, rne \n\t"
|
||||||
|
"fmvz.x.ps t0, f4, 4 \n\t"
|
||||||
|
"fbcx.ps f5, t0 \n\t"
|
||||||
|
"fadd.ps %[vout], f4, f5, rne \n\t"
|
||||||
|
: [vout] "=f"(result)::"t0", "f1", "f2", "f3", "f4", "f5");
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Full-row dot product (convenience wrapper)
|
||||||
|
static inline float compute_row_dot_q8_0(const block_q8_0 * q_row, const float * b_col, int64_t K_blocks) {
|
||||||
|
unsigned long saved_mask;
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(saved_mask));
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF");
|
||||||
|
q8_dot_reset();
|
||||||
|
q8_dot_tile(q_row, b_col, K_blocks);
|
||||||
|
float result = q8_dot_reduce();
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(saved_mask));
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
//******************************************************************************
|
||||||
|
// Hoisted Q8_0 dot API
|
||||||
|
//
|
||||||
|
// q8_dot_begin/end save/restore the vector mask once around a long sequence of
|
||||||
|
// dot products, so the per-row mask shuffles are hoisted out of the inner
|
||||||
|
// loops. q8_dot_compute does a full-row dot (no mask handling). The _x2
|
||||||
|
// variant computes two rows together while reusing each loaded B chunk —
|
||||||
|
// only safe when both row pointers share the same 32-byte alignment phase
|
||||||
|
// (i.e. the Q8 row stride is a multiple of 32).
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
typedef struct {
|
||||||
|
unsigned long saved_mask;
|
||||||
|
} q8_dot_state;
|
||||||
|
|
||||||
|
static inline void q8_dot_begin(q8_dot_state * state) {
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(state->saved_mask));
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF");
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline void q8_dot_end(const q8_dot_state * state) {
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(state->saved_mask));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Equivalent to q8_dot_reset+tile+reduce, without touching the mask register.
|
||||||
|
// Caller is responsible for q8_dot_begin/end around the surrounding loop.
|
||||||
|
static inline float q8_dot_compute(const block_q8_0 * q_row, const float * b_col, int64_t K_blocks) {
|
||||||
|
q8_dot_reset();
|
||||||
|
q8_dot_tile(q_row, b_col, K_blocks);
|
||||||
|
return q8_dot_reduce();
|
||||||
|
}
|
||||||
|
|
||||||
|
// Compute two row dots together while reusing the same loaded B chunks.
|
||||||
|
//
|
||||||
|
// Safe when every row starts at the same 32-byte offset, i.e. the Q8 row stride
|
||||||
|
// is a multiple of 32. In that case the gather/alignment pattern is the same
|
||||||
|
// for both rows at a given `kb`, so one set of B vector loads feeds both row
|
||||||
|
// accumulators.
|
||||||
|
static inline void q8_dot_compute_x2_aligned(const block_q8_0 * q_row0,
|
||||||
|
const block_q8_0 * q_row1,
|
||||||
|
const float * b_col,
|
||||||
|
int64_t K_blocks,
|
||||||
|
float * out0,
|
||||||
|
float * out1) {
|
||||||
|
const int32_t gather_pattern[8] = { 0, 1, 2, 3, 4, 5, 6, 7 };
|
||||||
|
const uint64_t gather_0_to_7 = 0x398a418820ULL;
|
||||||
|
__asm__ volatile("flw.ps f31, %[g]\n" : : [g] "m"(*(const int32_t (*)[8]) gather_pattern) : "f31");
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbci.pi f20, 0\n"
|
||||||
|
"fbci.pi f21, 0\n" ::
|
||||||
|
: "f20", "f21");
|
||||||
|
|
||||||
|
for (int64_t kb = 0; kb < K_blocks; kb++) {
|
||||||
|
const block_q8_0 * blk0 = q_row0 + kb;
|
||||||
|
const block_q8_0 * blk1 = q_row1 + kb;
|
||||||
|
const float * b_ptr = b_col + (kb << 5);
|
||||||
|
|
||||||
|
const uintptr_t qs_addr0 = (uintptr_t) blk0->qs;
|
||||||
|
const uintptr_t qs_addr1 = (uintptr_t) blk1->qs;
|
||||||
|
const uintptr_t qs_aligned0 = qs_addr0 & ~(uintptr_t) 31;
|
||||||
|
const uintptr_t qs_aligned1 = qs_addr1 & ~(uintptr_t) 31;
|
||||||
|
const int fast_chunks = (int) ((32 - (qs_addr0 & 31)) >> 3);
|
||||||
|
|
||||||
|
if (fast_chunks >= 3) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbci.pi f10, 0\n"
|
||||||
|
"fbci.pi f11, 0\n"
|
||||||
|
|
||||||
|
"flw.ps f12, %[bv0]\n"
|
||||||
|
"fg32b.ps f16, %[gi](%[r0ap0])\n"
|
||||||
|
"fcvt.ps.pw f16, f16\n"
|
||||||
|
"fmadd.ps f10, f16, f12, f10\n"
|
||||||
|
"fg32b.ps f17, %[gi](%[r1ap0])\n"
|
||||||
|
"fcvt.ps.pw f17, f17\n"
|
||||||
|
"fmadd.ps f11, f17, f12, f11\n"
|
||||||
|
|
||||||
|
"flw.ps f13, %[bv1]\n"
|
||||||
|
"fg32b.ps f16, %[gi](%[r0ap1])\n"
|
||||||
|
"fcvt.ps.pw f16, f16\n"
|
||||||
|
"fmadd.ps f10, f16, f13, f10\n"
|
||||||
|
"fg32b.ps f17, %[gi](%[r1ap1])\n"
|
||||||
|
"fcvt.ps.pw f17, f17\n"
|
||||||
|
"fmadd.ps f11, f17, f13, f11\n"
|
||||||
|
|
||||||
|
"flw.ps f14, %[bv2]\n"
|
||||||
|
"fg32b.ps f16, %[gi](%[r0ap2])\n"
|
||||||
|
"fcvt.ps.pw f16, f16\n"
|
||||||
|
"fmadd.ps f10, f16, f14, f10\n"
|
||||||
|
"fg32b.ps f17, %[gi](%[r1ap2])\n"
|
||||||
|
"fcvt.ps.pw f17, f17\n"
|
||||||
|
"fmadd.ps f11, f17, f14, f11\n"
|
||||||
|
|
||||||
|
"flw.ps f15, %[bv3]\n"
|
||||||
|
"fgb.ps f16, f31(%[r0ap3])\n"
|
||||||
|
"fcvt.ps.pw f16, f16\n"
|
||||||
|
"fmadd.ps f10, f16, f15, f10\n"
|
||||||
|
"fgb.ps f17, f31(%[r1ap3])\n"
|
||||||
|
"fcvt.ps.pw f17, f17\n"
|
||||||
|
"fmadd.ps f11, f17, f15, f11\n"
|
||||||
|
:
|
||||||
|
: [gi] "r"(gather_0_to_7), [r0ap0] "r"(qs_addr0), [r0ap1] "r"(qs_aligned0 | ((qs_addr0 + 8) & 31)),
|
||||||
|
[r0ap2] "r"(qs_aligned0 | ((qs_addr0 + 16) & 31)), [r0ap3] "r"(&blk0->qs[24]), [r1ap0] "r"(qs_addr1),
|
||||||
|
[r1ap1] "r"(qs_aligned1 | ((qs_addr1 + 8) & 31)), [r1ap2] "r"(qs_aligned1 | ((qs_addr1 + 16) & 31)),
|
||||||
|
[r1ap3] "r"(&blk1->qs[24]), [bv0] "m"(*(const float (*)[8]) & b_ptr[0]),
|
||||||
|
[bv1] "m"(*(const float (*)[8]) & b_ptr[8]), [bv2] "m"(*(const float (*)[8]) & b_ptr[16]),
|
||||||
|
[bv3] "m"(*(const float (*)[8]) & b_ptr[24])
|
||||||
|
: "f10", "f11", "f12", "f13", "f14", "f15", "f16", "f17");
|
||||||
|
} else if (fast_chunks == 2) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbci.pi f10, 0\n"
|
||||||
|
"fbci.pi f11, 0\n"
|
||||||
|
|
||||||
|
"flw.ps f12, %[bv0]\n"
|
||||||
|
"fg32b.ps f16, %[gi](%[r0ap0])\n"
|
||||||
|
"fcvt.ps.pw f16, f16\n"
|
||||||
|
"fmadd.ps f10, f16, f12, f10\n"
|
||||||
|
"fg32b.ps f17, %[gi](%[r1ap0])\n"
|
||||||
|
"fcvt.ps.pw f17, f17\n"
|
||||||
|
"fmadd.ps f11, f17, f12, f11\n"
|
||||||
|
|
||||||
|
"flw.ps f13, %[bv1]\n"
|
||||||
|
"fg32b.ps f16, %[gi](%[r0ap1])\n"
|
||||||
|
"fcvt.ps.pw f16, f16\n"
|
||||||
|
"fmadd.ps f10, f16, f13, f10\n"
|
||||||
|
"fg32b.ps f17, %[gi](%[r1ap1])\n"
|
||||||
|
"fcvt.ps.pw f17, f17\n"
|
||||||
|
"fmadd.ps f11, f17, f13, f11\n"
|
||||||
|
|
||||||
|
"flw.ps f14, %[bv2]\n"
|
||||||
|
"fgb.ps f16, f31(%[r0ap2])\n"
|
||||||
|
"fcvt.ps.pw f16, f16\n"
|
||||||
|
"fmadd.ps f10, f16, f14, f10\n"
|
||||||
|
"fgb.ps f17, f31(%[r1ap2])\n"
|
||||||
|
"fcvt.ps.pw f17, f17\n"
|
||||||
|
"fmadd.ps f11, f17, f14, f11\n"
|
||||||
|
|
||||||
|
"flw.ps f15, %[bv3]\n"
|
||||||
|
"fgb.ps f16, f31(%[r0ap3])\n"
|
||||||
|
"fcvt.ps.pw f16, f16\n"
|
||||||
|
"fmadd.ps f10, f16, f15, f10\n"
|
||||||
|
"fgb.ps f17, f31(%[r1ap3])\n"
|
||||||
|
"fcvt.ps.pw f17, f17\n"
|
||||||
|
"fmadd.ps f11, f17, f15, f11\n"
|
||||||
|
:
|
||||||
|
: [gi] "r"(gather_0_to_7), [r0ap0] "r"(qs_addr0), [r0ap1] "r"(qs_aligned0 | ((qs_addr0 + 8) & 31)),
|
||||||
|
[r0ap2] "r"(&blk0->qs[16]), [r0ap3] "r"(&blk0->qs[24]), [r1ap0] "r"(qs_addr1),
|
||||||
|
[r1ap1] "r"(qs_aligned1 | ((qs_addr1 + 8) & 31)), [r1ap2] "r"(&blk1->qs[16]),
|
||||||
|
[r1ap3] "r"(&blk1->qs[24]), [bv0] "m"(*(const float (*)[8]) & b_ptr[0]),
|
||||||
|
[bv1] "m"(*(const float (*)[8]) & b_ptr[8]), [bv2] "m"(*(const float (*)[8]) & b_ptr[16]),
|
||||||
|
[bv3] "m"(*(const float (*)[8]) & b_ptr[24])
|
||||||
|
: "f10", "f11", "f12", "f13", "f14", "f15", "f16", "f17");
|
||||||
|
} else if (fast_chunks == 1) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbci.pi f10, 0\n"
|
||||||
|
"fbci.pi f11, 0\n"
|
||||||
|
|
||||||
|
"flw.ps f12, %[bv0]\n"
|
||||||
|
"fg32b.ps f16, %[gi](%[r0ap0])\n"
|
||||||
|
"fcvt.ps.pw f16, f16\n"
|
||||||
|
"fmadd.ps f10, f16, f12, f10\n"
|
||||||
|
"fg32b.ps f17, %[gi](%[r1ap0])\n"
|
||||||
|
"fcvt.ps.pw f17, f17\n"
|
||||||
|
"fmadd.ps f11, f17, f12, f11\n"
|
||||||
|
|
||||||
|
"flw.ps f13, %[bv1]\n"
|
||||||
|
"fgb.ps f16, f31(%[r0ap1])\n"
|
||||||
|
"fcvt.ps.pw f16, f16\n"
|
||||||
|
"fmadd.ps f10, f16, f13, f10\n"
|
||||||
|
"fgb.ps f17, f31(%[r1ap1])\n"
|
||||||
|
"fcvt.ps.pw f17, f17\n"
|
||||||
|
"fmadd.ps f11, f17, f13, f11\n"
|
||||||
|
|
||||||
|
"flw.ps f14, %[bv2]\n"
|
||||||
|
"fgb.ps f16, f31(%[r0ap2])\n"
|
||||||
|
"fcvt.ps.pw f16, f16\n"
|
||||||
|
"fmadd.ps f10, f16, f14, f10\n"
|
||||||
|
"fgb.ps f17, f31(%[r1ap2])\n"
|
||||||
|
"fcvt.ps.pw f17, f17\n"
|
||||||
|
"fmadd.ps f11, f17, f14, f11\n"
|
||||||
|
|
||||||
|
"flw.ps f15, %[bv3]\n"
|
||||||
|
"fgb.ps f16, f31(%[r0ap3])\n"
|
||||||
|
"fcvt.ps.pw f16, f16\n"
|
||||||
|
"fmadd.ps f10, f16, f15, f10\n"
|
||||||
|
"fgb.ps f17, f31(%[r1ap3])\n"
|
||||||
|
"fcvt.ps.pw f17, f17\n"
|
||||||
|
"fmadd.ps f11, f17, f15, f11\n"
|
||||||
|
:
|
||||||
|
: [gi] "r"(gather_0_to_7), [r0ap0] "r"(qs_addr0), [r0ap1] "r"(&blk0->qs[8]), [r0ap2] "r"(&blk0->qs[16]),
|
||||||
|
[r0ap3] "r"(&blk0->qs[24]), [r1ap0] "r"(qs_addr1), [r1ap1] "r"(&blk1->qs[8]),
|
||||||
|
[r1ap2] "r"(&blk1->qs[16]), [r1ap3] "r"(&blk1->qs[24]), [bv0] "m"(*(const float (*)[8]) & b_ptr[0]),
|
||||||
|
[bv1] "m"(*(const float (*)[8]) & b_ptr[8]), [bv2] "m"(*(const float (*)[8]) & b_ptr[16]),
|
||||||
|
[bv3] "m"(*(const float (*)[8]) & b_ptr[24])
|
||||||
|
: "f10", "f11", "f12", "f13", "f14", "f15", "f16", "f17");
|
||||||
|
} else {
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbci.pi f10, 0\n"
|
||||||
|
"fbci.pi f11, 0\n"
|
||||||
|
|
||||||
|
"flw.ps f12, %[bv0]\n"
|
||||||
|
"fgb.ps f16, f31(%[r0ap0])\n"
|
||||||
|
"fcvt.ps.pw f16, f16\n"
|
||||||
|
"fmadd.ps f10, f16, f12, f10\n"
|
||||||
|
"fgb.ps f17, f31(%[r1ap0])\n"
|
||||||
|
"fcvt.ps.pw f17, f17\n"
|
||||||
|
"fmadd.ps f11, f17, f12, f11\n"
|
||||||
|
|
||||||
|
"flw.ps f13, %[bv1]\n"
|
||||||
|
"fgb.ps f16, f31(%[r0ap1])\n"
|
||||||
|
"fcvt.ps.pw f16, f16\n"
|
||||||
|
"fmadd.ps f10, f16, f13, f10\n"
|
||||||
|
"fgb.ps f17, f31(%[r1ap1])\n"
|
||||||
|
"fcvt.ps.pw f17, f17\n"
|
||||||
|
"fmadd.ps f11, f17, f13, f11\n"
|
||||||
|
|
||||||
|
"flw.ps f14, %[bv2]\n"
|
||||||
|
"fgb.ps f16, f31(%[r0ap2])\n"
|
||||||
|
"fcvt.ps.pw f16, f16\n"
|
||||||
|
"fmadd.ps f10, f16, f14, f10\n"
|
||||||
|
"fgb.ps f17, f31(%[r1ap2])\n"
|
||||||
|
"fcvt.ps.pw f17, f17\n"
|
||||||
|
"fmadd.ps f11, f17, f14, f11\n"
|
||||||
|
|
||||||
|
"flw.ps f15, %[bv3]\n"
|
||||||
|
"fgb.ps f16, f31(%[r0ap3])\n"
|
||||||
|
"fcvt.ps.pw f16, f16\n"
|
||||||
|
"fmadd.ps f10, f16, f15, f10\n"
|
||||||
|
"fgb.ps f17, f31(%[r1ap3])\n"
|
||||||
|
"fcvt.ps.pw f17, f17\n"
|
||||||
|
"fmadd.ps f11, f17, f15, f11\n"
|
||||||
|
:
|
||||||
|
: [r0ap0] "r"(&blk0->qs[0]), [r0ap1] "r"(&blk0->qs[8]), [r0ap2] "r"(&blk0->qs[16]),
|
||||||
|
[r0ap3] "r"(&blk0->qs[24]), [r1ap0] "r"(&blk1->qs[0]), [r1ap1] "r"(&blk1->qs[8]),
|
||||||
|
[r1ap2] "r"(&blk1->qs[16]), [r1ap3] "r"(&blk1->qs[24]), [bv0] "m"(*(const float (*)[8]) & b_ptr[0]),
|
||||||
|
[bv1] "m"(*(const float (*)[8]) & b_ptr[8]), [bv2] "m"(*(const float (*)[8]) & b_ptr[16]),
|
||||||
|
[bv3] "m"(*(const float (*)[8]) & b_ptr[24])
|
||||||
|
: "f10", "f11", "f12", "f13", "f14", "f15", "f16", "f17");
|
||||||
|
}
|
||||||
|
|
||||||
|
const uint32_t scale_raw0 = (uint32_t) blk0->d;
|
||||||
|
const uint32_t scale_raw1 = (uint32_t) blk1->d;
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbcx.ps f24, %[s0]\n"
|
||||||
|
"fcvt.ps.f16 f24, f24\n"
|
||||||
|
"fmadd.ps f20, f10, f24, f20\n"
|
||||||
|
"fbcx.ps f25, %[s1]\n"
|
||||||
|
"fcvt.ps.f16 f25, f25\n"
|
||||||
|
"fmadd.ps f21, f11, f25, f21\n"
|
||||||
|
:
|
||||||
|
: [s0] "r"(scale_raw0), [s1] "r"(scale_raw1)
|
||||||
|
: "f20", "f21", "f24", "f25");
|
||||||
|
}
|
||||||
|
|
||||||
|
float result0;
|
||||||
|
float result1;
|
||||||
|
__asm__ __volatile__(
|
||||||
|
"fswizz.ps f1, f20, 0xB1 \n\t"
|
||||||
|
"fadd.ps f2, f20, f1, rne \n\t"
|
||||||
|
"fswizz.ps f3, f2, 0x4E \n\t"
|
||||||
|
"fadd.ps f4, f2, f3, rne \n\t"
|
||||||
|
"fmvz.x.ps t0, f4, 4 \n\t"
|
||||||
|
"fbcx.ps f5, t0 \n\t"
|
||||||
|
"fadd.ps %[vout], f4, f5, rne \n\t"
|
||||||
|
: [vout] "=f"(result0)::"t0", "f1", "f2", "f3", "f4", "f5");
|
||||||
|
__asm__ __volatile__(
|
||||||
|
"fswizz.ps f1, f21, 0xB1 \n\t"
|
||||||
|
"fadd.ps f2, f21, f1, rne \n\t"
|
||||||
|
"fswizz.ps f3, f2, 0x4E \n\t"
|
||||||
|
"fadd.ps f4, f2, f3, rne \n\t"
|
||||||
|
"fmvz.x.ps t0, f4, 4 \n\t"
|
||||||
|
"fbcx.ps f5, t0 \n\t"
|
||||||
|
"fadd.ps %[vout], f4, f5, rne \n\t"
|
||||||
|
: [vout] "=f"(result1)::"t0", "f1", "f2", "f3", "f4", "f5");
|
||||||
|
|
||||||
|
*out0 = result0;
|
||||||
|
*out1 = result1;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Compute dot product between f16 block and f32 column vector (NAIVE VERSION)
|
||||||
|
// Scalar implementation for debugging - no vectorization
|
||||||
|
// Block size: 32 f16 values (64 bytes = 1 cache line)
|
||||||
|
static inline float compute_block_dot_product_f16_naive(const uint16_t * a_block, const float * b_col_start) {
|
||||||
|
float acc_vec[8] __attribute__((aligned(32))) = { 0.0f };
|
||||||
|
// Byte offsets for 16-bit (half-word) elements
|
||||||
|
static const int32_t gather_pattern[8] = { 0, 2, 4, 6, 8, 10, 12, 14 };
|
||||||
|
unsigned long temp_mask;
|
||||||
|
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(temp_mask));
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF");
|
||||||
|
|
||||||
|
// Load the pattern once into f31 for the duration of all 4 chunks
|
||||||
|
__asm__ volatile("flw.ps f31, %[gather]\n" : : [gather] "m"(*(const int32_t (*)[8]) gather_pattern) : "f31");
|
||||||
|
|
||||||
|
for (int chunk = 0; chunk < 4; chunk++) {
|
||||||
|
// Correct pointers:
|
||||||
|
// a_block elements are 2 bytes, b_col elements are 4 bytes
|
||||||
|
const uint16_t * a_ptr = &a_block[chunk << 3]; // chunk * 8
|
||||||
|
const float * b_ptr = &b_col_start[chunk << 3]; // chunk * 8
|
||||||
|
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f10, %[acc]\n"
|
||||||
|
"fgh.ps f11, f31(%[a_p])\n" // Uses {0,2,4,6,8,10,12,14} byte offsets
|
||||||
|
"fcvt.ps.f16 f11, f11\n"
|
||||||
|
"flw.ps f12, (%[b_p])\n" // Standard vector load (32-bit floats)
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
"fsw.ps f10, %[result]\n"
|
||||||
|
|
||||||
|
: [result] "=m"(*(float (*)[8]) acc_vec)
|
||||||
|
: [acc] "m"(*(const float (*)[8]) acc_vec), [a_p] "r"(a_ptr), [b_p] "r"(b_ptr)
|
||||||
|
: "f10", "f11", "f12");
|
||||||
|
}
|
||||||
|
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(temp_mask));
|
||||||
|
|
||||||
|
return acc_vec[0] + acc_vec[1] + acc_vec[2] + acc_vec[3] + acc_vec[4] + acc_vec[5] + acc_vec[6] + acc_vec[7];
|
||||||
|
}
|
||||||
|
|
||||||
|
// Compute dot product between f16 block and f32 column vector
|
||||||
|
// SCALAR implementation for partial blocks
|
||||||
|
// Block size: up to 32 f16 values (can handle partial blocks for misaligned K)
|
||||||
|
static inline float compute_block_dot_product_f16_partial(const uint16_t * a_block,
|
||||||
|
const float * b_col_start,
|
||||||
|
int elements) {
|
||||||
|
// This matches compute_block_dot_product_f16_naive behavior
|
||||||
|
float sum = 0.0f;
|
||||||
|
|
||||||
|
for (int i = 0; i < elements; i++) {
|
||||||
|
float a_val = fp16_to_fp32(a_block[i]);
|
||||||
|
float b_val = b_col_start[i];
|
||||||
|
sum += a_val * b_val;
|
||||||
|
}
|
||||||
|
|
||||||
|
return sum;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Compute dot product between f16 block and f16 column vector
|
||||||
|
// Scalar implementation for generic non-matrix-engine fallback paths.
|
||||||
|
static inline float compute_block_dot_product_f16_f16_partial(const uint16_t * a_block,
|
||||||
|
const uint16_t * b_col_start,
|
||||||
|
int elements) {
|
||||||
|
float sum = 0.0f;
|
||||||
|
|
||||||
|
for (int i = 0; i < elements; i++) {
|
||||||
|
sum += fp16_to_fp32(a_block[i]) * fp16_to_fp32(b_col_start[i]);
|
||||||
|
}
|
||||||
|
|
||||||
|
return sum;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Compute dot product between f16 block and f32 column vector
|
||||||
|
// Vectorized: processes 8 elements at a time using ET vector instructions
|
||||||
|
// Block size: 32 f16 values (64 bytes = 1 cache line)
|
||||||
|
static inline float compute_block_dot_product_f16(const uint16_t * a_block, const float * b_col_start) {
|
||||||
|
return compute_block_dot_product_f16_partial(a_block, b_col_start, QK_F16);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Compute dot product between f32 block and f32 column vector
|
||||||
|
// Vectorized: processes 8 elements at a time using ET vector instructions
|
||||||
|
// Block size: up to 16 f32 values (can handle partial blocks for misaligned K)
|
||||||
|
static inline float compute_block_dot_product_f32_partial(const float * a_block,
|
||||||
|
const float * b_col_start,
|
||||||
|
int elements) {
|
||||||
|
float acc_vec[8] = { 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f, 0.0f }; // Accumulator vector
|
||||||
|
|
||||||
|
// Calculate how many full 8-element chunks we can process
|
||||||
|
int vec_end = (elements / 8) * 8;
|
||||||
|
|
||||||
|
if (vec_end > 0) {
|
||||||
|
// Set mask register to enable all 8 vector elements
|
||||||
|
unsigned long temp_mask;
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(temp_mask)); // Save current mask
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF"); // Enable all 8 elements
|
||||||
|
|
||||||
|
// Process full 8-element chunks
|
||||||
|
for (int i = 0; i < vec_end; i += 8) {
|
||||||
|
// Vectorized f32 multiply-accumulate
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f10, %[acc]\n" // Load current accumulator (8 floats)
|
||||||
|
"flw.ps f11, %[a_vec]\n" // Load 8 A values (f32)
|
||||||
|
"flw.ps f12, %[b_vec]\n" // Load 8 B values (f32)
|
||||||
|
"fmadd.ps f10, f11, f12, f10\n" // acc += a_vec * b_vec (8-wide)
|
||||||
|
"fsw.ps f10, %[result]\n" // Store back to accumulator
|
||||||
|
|
||||||
|
: [result] "=m"(*(float (*)[8]) acc_vec)
|
||||||
|
: [acc] "m"(*(const float (*)[8]) acc_vec), [a_vec] "m"(*(const float (*)[8])(a_block + i)),
|
||||||
|
[b_vec] "m"(*(const float (*)[8])(b_col_start + i))
|
||||||
|
: "f10", "f11", "f12");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Restore original mask
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(temp_mask));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Horizontal sum: reduce 8 accumulator elements to single scalar
|
||||||
|
float final_sum = 0.0f;
|
||||||
|
for (int i = 0; i < 8; i++) {
|
||||||
|
final_sum += acc_vec[i];
|
||||||
|
}
|
||||||
|
|
||||||
|
// Handle remaining elements (< 8) with scalar operations
|
||||||
|
for (int i = vec_end; i < elements; i++) {
|
||||||
|
final_sum += a_block[i] * b_col_start[i];
|
||||||
|
}
|
||||||
|
|
||||||
|
return final_sum;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Compute dot product between f32 block and f16 column vector
|
||||||
|
// Scalar implementation for generic non-matrix-engine fallback paths.
|
||||||
|
static inline float compute_block_dot_product_f32_f16_partial(const float * a_block,
|
||||||
|
const uint16_t * b_col_start,
|
||||||
|
int elements) {
|
||||||
|
float sum = 0.0f;
|
||||||
|
|
||||||
|
for (int i = 0; i < elements; i++) {
|
||||||
|
sum += a_block[i] * fp16_to_fp32(b_col_start[i]);
|
||||||
|
}
|
||||||
|
|
||||||
|
return sum;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Compute dot product between f32 block and f32 column vector
|
||||||
|
// Vectorized: processes 8 elements at a time using ET vector instructions
|
||||||
|
// Block size: 16 f32 values (64 bytes = 1 cache line)
|
||||||
|
static inline float compute_block_dot_product_f32(const float * a_block, const float * b_col_start) {
|
||||||
|
return compute_block_dot_product_f32_partial(a_block, b_col_start, QK_F32);
|
||||||
|
|
||||||
|
// float acc_vec[8];
|
||||||
|
// unsigned long old_mask;
|
||||||
|
// __asm__ volatile(
|
||||||
|
// // Save current mask
|
||||||
|
// "mova.x.m %[old_mask]\n"
|
||||||
|
// // Enable all 8 lanes
|
||||||
|
// "mov.m.x m0, x0, 0xFF\n"
|
||||||
|
|
||||||
|
// "flw.ps f11, %[a]\n"
|
||||||
|
// "flw.ps f12, %[b]\n"
|
||||||
|
// "fmadd.ps f10, f11, f12, f10\n"
|
||||||
|
// "fsw.ps f10, %[out]\n"
|
||||||
|
// "mova.m.x %[old_mask]\n"
|
||||||
|
|
||||||
|
// : [out] "=m" (*(float(*)[8])acc_vec),
|
||||||
|
// [old_mask] "=r"(old_mask)
|
||||||
|
// : [a] "m" (*(const float(*)[8])a_block),
|
||||||
|
// [b] "m" (*(const float(*)[8])b_col_start)
|
||||||
|
// : "f10", "f11", "f12"
|
||||||
|
// );
|
||||||
|
|
||||||
|
// // Horizontal reduction
|
||||||
|
// return acc_vec[0] + acc_vec[1] + acc_vec[2] + acc_vec[3] +
|
||||||
|
// acc_vec[4] + acc_vec[5] + acc_vec[6] + acc_vec[7];
|
||||||
|
}
|
||||||
|
|
||||||
|
#endif // BLOCK_OPS_H
|
||||||
|
|
||||||
|
static inline void __attribute__((always_inline)) q4_dot_reset(void) {
|
||||||
|
__asm__ volatile("fbci.pi f20, 0" ::: "f20");
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline void __attribute__((always_inline)) q4_dot_tile(const block_q4_0 * q_row,
|
||||||
|
const float * b_col,
|
||||||
|
int64_t n_blocks) {
|
||||||
|
const int32_t gather_pattern[8] = { 0, 1, 2, 3, 4, 5, 6, 7 };
|
||||||
|
__asm__ volatile("flw.ps f31, %[g]\n" : : [g] "m"(*(const int32_t (*)[8]) gather_pattern) : "f31");
|
||||||
|
|
||||||
|
for (int64_t kb = 0; kb < n_blocks; kb++) {
|
||||||
|
const block_q4_0 * blk = q_row + kb;
|
||||||
|
const float * b_ptr = b_col + (kb << 5);
|
||||||
|
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbci.pi f10, 0\n"
|
||||||
|
|
||||||
|
"fgb.ps f11, f31(%[a_ptr0])\n"
|
||||||
|
"fandi.pi f12, f11, 15\n"
|
||||||
|
"faddi.pi f12, f12, -8\n"
|
||||||
|
"fcvt.ps.pw f12, f12, rne\n"
|
||||||
|
"flw.ps f13, %[b_low0]\n"
|
||||||
|
"fmadd.ps f10, f12, f13, f10, rne\n"
|
||||||
|
|
||||||
|
"fsrli.pi f14, f11, 4\n"
|
||||||
|
"fandi.pi f14, f14, 15\n"
|
||||||
|
"faddi.pi f14, f14, -8\n"
|
||||||
|
"fcvt.ps.pw f14, f14, rne\n"
|
||||||
|
"flw.ps f15, %[b_high0]\n"
|
||||||
|
"fmadd.ps f10, f14, f15, f10, rne\n"
|
||||||
|
|
||||||
|
"fgb.ps f11, f31(%[a_ptr1])\n"
|
||||||
|
"fandi.pi f12, f11, 15\n"
|
||||||
|
"faddi.pi f12, f12, -8\n"
|
||||||
|
"fcvt.ps.pw f12, f12, rne\n"
|
||||||
|
"flw.ps f13, %[b_low1]\n"
|
||||||
|
"fmadd.ps f10, f12, f13, f10, rne\n"
|
||||||
|
|
||||||
|
"fsrli.pi f14, f11, 4\n"
|
||||||
|
"fandi.pi f14, f14, 15\n"
|
||||||
|
"faddi.pi f14, f14, -8\n"
|
||||||
|
"fcvt.ps.pw f14, f14, rne\n"
|
||||||
|
"flw.ps f15, %[b_high1]\n"
|
||||||
|
"fmadd.ps f10, f14, f15, f10, rne\n"
|
||||||
|
:
|
||||||
|
: [a_ptr0] "r"(&blk->qs[0]), [b_low0] "m"(*(const float (*)[8]) & b_ptr[0]),
|
||||||
|
[b_high0] "m"(*(const float (*)[8]) & b_ptr[16]), [a_ptr1] "r"(&blk->qs[8]),
|
||||||
|
[b_low1] "m"(*(const float (*)[8]) & b_ptr[8]), [b_high1] "m"(*(const float (*)[8]) & b_ptr[24])
|
||||||
|
: "f10", "f11", "f12", "f13", "f14", "f15");
|
||||||
|
|
||||||
|
uint32_t scale_raw = (uint32_t) blk->d;
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbcx.ps f15, %[sb]\n"
|
||||||
|
"fcvt.ps.f16 f15, f15\n"
|
||||||
|
"fmadd.ps f20, f10, f15, f20\n"
|
||||||
|
:
|
||||||
|
: [sb] "r"(scale_raw)
|
||||||
|
: "f15", "f20");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline float __attribute__((always_inline)) q4_dot_reduce(void) {
|
||||||
|
float result;
|
||||||
|
__asm__ __volatile__(
|
||||||
|
"fswizz.ps f1, f20, 0xB1 \n\t"
|
||||||
|
"fadd.ps f2, f20, f1, rne \n\t"
|
||||||
|
"fswizz.ps f3, f2, 0x4E \n\t"
|
||||||
|
"fadd.ps f4, f2, f3, rne \n\t"
|
||||||
|
"fmvz.x.ps t0, f4, 4 \n\t"
|
||||||
|
"fbcx.ps f5, t0 \n\t"
|
||||||
|
"fadd.ps %[vout], f4, f5, rne \n\t"
|
||||||
|
: [vout] "=f"(result)::"t0", "f1", "f2", "f3", "f4", "f5");
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline float compute_row_dot_q4_0(const block_q4_0 * q_row, const float * b_col, int64_t K_blocks) {
|
||||||
|
unsigned long saved_mask;
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(saved_mask));
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF");
|
||||||
|
q4_dot_reset();
|
||||||
|
q4_dot_tile(q_row, b_col, K_blocks);
|
||||||
|
float result = q4_dot_reduce();
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(saved_mask));
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
typedef struct {
|
||||||
|
unsigned long saved_mask;
|
||||||
|
} q4_dot_state;
|
||||||
|
|
||||||
|
static inline void q4_dot_begin(q4_dot_state * state) {
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(state->saved_mask));
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF");
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline void q4_dot_end(const q4_dot_state * state) {
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(state->saved_mask));
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline float q4_dot_compute(const block_q4_0 * q_row, const float * b_col, int64_t K_blocks) {
|
||||||
|
q4_dot_reset();
|
||||||
|
q4_dot_tile(q_row, b_col, K_blocks);
|
||||||
|
return q4_dot_reduce();
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline void q4_dot_compute_x2_aligned(const block_q4_0 * q_row0,
|
||||||
|
const block_q4_0 * q_row1,
|
||||||
|
const float * b_col,
|
||||||
|
int64_t K_blocks,
|
||||||
|
float * out0,
|
||||||
|
float * out1) {
|
||||||
|
const int32_t gather_pattern[8] = { 0, 1, 2, 3, 4, 5, 6, 7 };
|
||||||
|
__asm__ volatile("flw.ps f31, %[g]\n" : : [g] "m"(*(const int32_t (*)[8]) gather_pattern) : "f31");
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbci.pi f20, 0\n"
|
||||||
|
"fbci.pi f21, 0\n" ::
|
||||||
|
: "f20", "f21");
|
||||||
|
|
||||||
|
for (int64_t kb = 0; kb < K_blocks; kb++) {
|
||||||
|
const block_q4_0 * blk0 = q_row0 + kb;
|
||||||
|
const block_q4_0 * blk1 = q_row1 + kb;
|
||||||
|
const float * b_ptr = b_col + (kb << 5);
|
||||||
|
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbci.pi f10, 0\n"
|
||||||
|
"fbci.pi f16, 0\n"
|
||||||
|
|
||||||
|
"flw.ps f13, %[b_low0]\n"
|
||||||
|
"flw.ps f15, %[b_high0]\n"
|
||||||
|
|
||||||
|
"fgb.ps f11, f31(%[a_ptr0_0])\n"
|
||||||
|
"fgb.ps f17, f31(%[a_ptr1_0])\n"
|
||||||
|
|
||||||
|
"fandi.pi f12, f11, 15\n"
|
||||||
|
"faddi.pi f12, f12, -8\n"
|
||||||
|
"fcvt.ps.pw f12, f12, rne\n"
|
||||||
|
"fmadd.ps f10, f12, f13, f10, rne\n"
|
||||||
|
|
||||||
|
"fandi.pi f18, f17, 15\n"
|
||||||
|
"faddi.pi f18, f18, -8\n"
|
||||||
|
"fcvt.ps.pw f18, f18, rne\n"
|
||||||
|
"fmadd.ps f16, f18, f13, f16, rne\n"
|
||||||
|
|
||||||
|
"fsrli.pi f14, f11, 4\n"
|
||||||
|
"fandi.pi f14, f14, 15\n"
|
||||||
|
"faddi.pi f14, f14, -8\n"
|
||||||
|
"fcvt.ps.pw f14, f14, rne\n"
|
||||||
|
"fmadd.ps f10, f14, f15, f10, rne\n"
|
||||||
|
|
||||||
|
"fsrli.pi f19, f17, 4\n"
|
||||||
|
"fandi.pi f19, f19, 15\n"
|
||||||
|
"faddi.pi f19, f19, -8\n"
|
||||||
|
"fcvt.ps.pw f19, f19, rne\n"
|
||||||
|
"fmadd.ps f16, f19, f15, f16, rne\n"
|
||||||
|
|
||||||
|
"flw.ps f13, %[b_low1]\n"
|
||||||
|
"flw.ps f15, %[b_high1]\n"
|
||||||
|
|
||||||
|
"fgb.ps f11, f31(%[a_ptr0_1])\n"
|
||||||
|
"fgb.ps f17, f31(%[a_ptr1_1])\n"
|
||||||
|
|
||||||
|
"fandi.pi f12, f11, 15\n"
|
||||||
|
"faddi.pi f12, f12, -8\n"
|
||||||
|
"fcvt.ps.pw f12, f12, rne\n"
|
||||||
|
"fmadd.ps f10, f12, f13, f10, rne\n"
|
||||||
|
|
||||||
|
"fandi.pi f18, f17, 15\n"
|
||||||
|
"faddi.pi f18, f18, -8\n"
|
||||||
|
"fcvt.ps.pw f18, f18, rne\n"
|
||||||
|
"fmadd.ps f16, f18, f13, f16, rne\n"
|
||||||
|
|
||||||
|
"fsrli.pi f14, f11, 4\n"
|
||||||
|
"fandi.pi f14, f14, 15\n"
|
||||||
|
"faddi.pi f14, f14, -8\n"
|
||||||
|
"fcvt.ps.pw f14, f14, rne\n"
|
||||||
|
"fmadd.ps f10, f14, f15, f10, rne\n"
|
||||||
|
|
||||||
|
"fsrli.pi f19, f17, 4\n"
|
||||||
|
"fandi.pi f19, f19, 15\n"
|
||||||
|
"faddi.pi f19, f19, -8\n"
|
||||||
|
"fcvt.ps.pw f19, f19, rne\n"
|
||||||
|
"fmadd.ps f16, f19, f15, f16, rne\n"
|
||||||
|
:
|
||||||
|
: [a_ptr0_0] "r"(&blk0->qs[0]), [a_ptr0_1] "r"(&blk0->qs[8]), [a_ptr1_0] "r"(&blk1->qs[0]),
|
||||||
|
[a_ptr1_1] "r"(&blk1->qs[8]), [b_low0] "m"(*(const float (*)[8]) & b_ptr[0]),
|
||||||
|
[b_high0] "m"(*(const float (*)[8]) & b_ptr[16]), [b_low1] "m"(*(const float (*)[8]) & b_ptr[8]),
|
||||||
|
[b_high1] "m"(*(const float (*)[8]) & b_ptr[24])
|
||||||
|
: "f10", "f11", "f12", "f13", "f14", "f15", "f16", "f17", "f18", "f19");
|
||||||
|
|
||||||
|
const uint32_t scale_raw0 = (uint32_t) blk0->d;
|
||||||
|
const uint32_t scale_raw1 = (uint32_t) blk1->d;
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbcx.ps f24, %[s0]\n"
|
||||||
|
"fcvt.ps.f16 f24, f24\n"
|
||||||
|
"fmadd.ps f20, f10, f24, f20\n"
|
||||||
|
"fbcx.ps f25, %[s1]\n"
|
||||||
|
"fcvt.ps.f16 f25, f25\n"
|
||||||
|
"fmadd.ps f21, f16, f25, f21\n"
|
||||||
|
:
|
||||||
|
: [s0] "r"(scale_raw0), [s1] "r"(scale_raw1)
|
||||||
|
: "f20", "f21", "f24", "f25");
|
||||||
|
}
|
||||||
|
|
||||||
|
float result0, result1;
|
||||||
|
__asm__ __volatile__(
|
||||||
|
"fswizz.ps f1, f20, 0xB1 \n\t"
|
||||||
|
"fadd.ps f2, f20, f1, rne \n\t"
|
||||||
|
"fswizz.ps f3, f2, 0x4E \n\t"
|
||||||
|
"fadd.ps f4, f2, f3, rne \n\t"
|
||||||
|
"fmvz.x.ps t0, f4, 4 \n\t"
|
||||||
|
"fbcx.ps f5, t0 \n\t"
|
||||||
|
"fadd.ps %[vout], f4, f5, rne \n\t"
|
||||||
|
: [vout] "=f"(result0)::"t0", "f1", "f2", "f3", "f4", "f5");
|
||||||
|
__asm__ __volatile__(
|
||||||
|
"fswizz.ps f1, f21, 0xB1 \n\t"
|
||||||
|
"fadd.ps f2, f21, f1, rne \n\t"
|
||||||
|
"fswizz.ps f3, f2, 0x4E \n\t"
|
||||||
|
"fadd.ps f4, f2, f3, rne \n\t"
|
||||||
|
"fmvz.x.ps t0, f4, 4 \n\t"
|
||||||
|
"fbcx.ps f5, t0 \n\t"
|
||||||
|
"fadd.ps %[vout], f4, f5, rne \n\t"
|
||||||
|
: [vout] "=f"(result1)::"t0", "f1", "f2", "f3", "f4", "f5");
|
||||||
|
|
||||||
|
*out0 = result0;
|
||||||
|
*out1 = result1;
|
||||||
|
}
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// CLAMP F32 Kernel
|
||||||
|
// Element-wise: dst[i] = min(max(src0[i], min_val), max_val)
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "platform.h"
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
struct ggml_et_clamp_params {
|
||||||
|
struct ggml_tensor src0; // F32 input (contiguous)
|
||||||
|
struct ggml_tensor dst; // F32 output (contiguous; may alias src0.data)
|
||||||
|
float min_val;
|
||||||
|
float max_val;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Vectorized fmax/fmin clamp with scalar tail. n may be any non-negative int.
|
||||||
|
static inline void clamp_block_f32(float * dst, const float * src, float min_val, float max_val, int32_t n) {
|
||||||
|
int32_t i = 0;
|
||||||
|
const int32_t vec_end = (n / 8) * 8;
|
||||||
|
|
||||||
|
if (vec_end > 0) {
|
||||||
|
unsigned long temp_mask;
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(temp_mask));
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF");
|
||||||
|
|
||||||
|
for (; i < vec_end; i += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f10, %[s]\n"
|
||||||
|
"fbc.ps f11, %[mn]\n"
|
||||||
|
"fbc.ps f12, %[mx]\n"
|
||||||
|
"fmax.ps f13, f10, f11\n"
|
||||||
|
"fmin.ps f13, f13, f12\n"
|
||||||
|
"fsw.ps f13, %[d]\n"
|
||||||
|
: [d] "=m"(*(float (*)[8]) & dst[i])
|
||||||
|
: [s] "m"(*(const float (*)[8]) & src[i]), [mn] "m"(min_val), [mx] "m"(max_val)
|
||||||
|
: "f10", "f11", "f12", "f13");
|
||||||
|
}
|
||||||
|
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(temp_mask));
|
||||||
|
}
|
||||||
|
|
||||||
|
for (; i < n; i++) {
|
||||||
|
float v = src[i];
|
||||||
|
if (v < min_val) {
|
||||||
|
v = min_val;
|
||||||
|
}
|
||||||
|
if (v > max_val) {
|
||||||
|
v = max_val;
|
||||||
|
}
|
||||||
|
dst[i] = v;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_clamp_params * params, void * env) {
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
|
||||||
|
if (!kernel_env) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
int num_threads = get_num_threads(kernel_env->shire_mask);
|
||||||
|
|
||||||
|
if (thread_id < 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (params == 0 || ((uint64_t) params & 0x7) != 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct ggml_tensor * src0 = ¶ms->src0;
|
||||||
|
struct ggml_tensor * dst = ¶ms->dst;
|
||||||
|
|
||||||
|
if (src0->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
float * src0_data = (float *) src0->data;
|
||||||
|
float * dst_data = (float *) dst->data;
|
||||||
|
|
||||||
|
if (!src0_data || !dst_data) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t total_elements = src0->ne[0] * src0->ne[1] * src0->ne[2] * src0->ne[3];
|
||||||
|
if (total_elements <= 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const float min_val = params->min_val;
|
||||||
|
const float max_val = params->max_val;
|
||||||
|
|
||||||
|
// Distribute by cache lines (16 F32 elements). Each thread owns disjoint
|
||||||
|
// cache lines, so a partial trailing line is written by exactly one
|
||||||
|
// thread — safe under non-coherent caches.
|
||||||
|
const int64_t elems_per_cl = 16;
|
||||||
|
const int64_t total_cl = (total_elements + elems_per_cl - 1) / elems_per_cl;
|
||||||
|
|
||||||
|
const int64_t cl_per_thread = (total_cl + num_threads - 1) / num_threads;
|
||||||
|
const int64_t cl_start = (int64_t) thread_id * cl_per_thread;
|
||||||
|
int64_t cl_end = cl_start + cl_per_thread;
|
||||||
|
if (cl_end > total_cl) {
|
||||||
|
cl_end = total_cl;
|
||||||
|
}
|
||||||
|
if (cl_start >= total_cl) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t es = cl_start * elems_per_cl;
|
||||||
|
int64_t ee = cl_end * elems_per_cl;
|
||||||
|
if (ee > total_elements) {
|
||||||
|
ee = total_elements;
|
||||||
|
}
|
||||||
|
|
||||||
|
clamp_block_f32(dst_data + es, src0_data + es, min_val, max_val, (int32_t) (ee - es));
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,175 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// Concat F32 Kernel
|
||||||
|
// Concatenates two F32 tensors along a specified dimension.
|
||||||
|
// All copies are aligned to cacheline boundaries (64 bytes = 16 floats).
|
||||||
|
//
|
||||||
|
// For dim >= 1, entire rows are copied from src0 or src1 into dst.
|
||||||
|
// For dim == 0, use:
|
||||||
|
// - a fast vector path when both source row segments are cacheline-aligned
|
||||||
|
// - a scalar stride-aware path otherwise
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "platform.h"
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
#include <string.h>
|
||||||
|
|
||||||
|
struct ggml_et_concat_params {
|
||||||
|
struct ggml_tensor src0; // F32 input tensor 0
|
||||||
|
struct ggml_tensor src1; // F32 input tensor 1
|
||||||
|
struct ggml_tensor dst; // F32 output tensor
|
||||||
|
int32_t dim; // Concatenation dimension
|
||||||
|
};
|
||||||
|
|
||||||
|
// Copy n floats from src to dst using 8-wide vector loads/stores.
|
||||||
|
// n must be a multiple of 16 (cacheline-aligned).
|
||||||
|
static inline void copy_row_aligned(float * dst, const float * src, int32_t n) {
|
||||||
|
for (int32_t i = 0; i < n; i += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f11, %[src_vec]\n"
|
||||||
|
"fsw.ps f11, %[dst_vec]\n"
|
||||||
|
: [dst_vec] "=m"(*(float (*)[8]) & dst[i])
|
||||||
|
: [src_vec] "m"(*(const float (*)[8]) & src[i])
|
||||||
|
: "f11");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_concat_params * params, void * env) {
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
|
||||||
|
if (!kernel_env) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
int num_threads = get_num_threads(kernel_env->shire_mask);
|
||||||
|
|
||||||
|
if (thread_id < 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (params == 0 || ((uint64_t) params & 0x7) != 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct ggml_tensor * src0 = ¶ms->src0;
|
||||||
|
struct ggml_tensor * src1 = ¶ms->src1;
|
||||||
|
struct ggml_tensor * dst = ¶ms->dst;
|
||||||
|
int32_t dim = params->dim;
|
||||||
|
|
||||||
|
if (src0->type != GGML_TYPE_F32 || src1->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
float * src0_data = (float *) src0->data;
|
||||||
|
float * src1_data = (float *) src1->data;
|
||||||
|
float * dst_data = (float *) dst->data;
|
||||||
|
|
||||||
|
if (!src0_data || !src1_data || !dst_data) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t ne00 = src0->ne[0], ne01 = src0->ne[1], ne02 = src0->ne[2], ne03 = src0->ne[3];
|
||||||
|
const int64_t ne10 = src1->ne[0], ne11 = src1->ne[1], ne12 = src1->ne[2], ne13 = src1->ne[3];
|
||||||
|
const int64_t ne0 = dst->ne[0], ne1 = dst->ne[1], ne2 = dst->ne[2], ne3 = dst->ne[3];
|
||||||
|
|
||||||
|
// src strides in bytes
|
||||||
|
const size_t nb00 = src0->nb[0], nb01 = src0->nb[1], nb02 = src0->nb[2], nb03 = src0->nb[3];
|
||||||
|
const size_t nb10 = src1->nb[0], nb11 = src1->nb[1], nb12 = src1->nb[2], nb13 = src1->nb[3];
|
||||||
|
// dst strides in bytes
|
||||||
|
const size_t dnb1 = dst->nb[1], dnb2 = dst->nb[2], dnb3 = dst->nb[3];
|
||||||
|
|
||||||
|
// Total rows across all higher dimensions
|
||||||
|
const int64_t total_rows = ne1 * ne2 * ne3;
|
||||||
|
|
||||||
|
// Generic slow path for dim==0 when either source segment is not suitable for
|
||||||
|
// aligned vector copies. Threading is done by cacheline-aligned row groups,
|
||||||
|
// so writers do not share destination cache lines.
|
||||||
|
if (dim == 0 && (ne00 % 16 != 0 || ne10 % 16 != 0 || nb00 != sizeof(float) || nb10 != sizeof(float))) {
|
||||||
|
const int64_t rows_per_group = et_rows_per_cacheline_group(ne0, sizeof(float));
|
||||||
|
const int64_t total_groups = (total_rows + rows_per_group - 1) / rows_per_group;
|
||||||
|
|
||||||
|
for (int64_t grp = thread_id; grp < total_groups; grp += num_threads) {
|
||||||
|
const int64_t row_start = grp * rows_per_group;
|
||||||
|
int64_t row_end = row_start + rows_per_group;
|
||||||
|
if (row_end > total_rows) {
|
||||||
|
row_end = total_rows;
|
||||||
|
}
|
||||||
|
|
||||||
|
for (int64_t row = row_start; row < row_end; row++) {
|
||||||
|
int64_t i1 = row % ne1;
|
||||||
|
int64_t i2 = (row / ne1) % ne2;
|
||||||
|
int64_t i3 = row / (ne1 * ne2);
|
||||||
|
|
||||||
|
float * dst_row = (float *) ((char *) dst_data + i1 * dnb1 + i2 * dnb2 + i3 * dnb3);
|
||||||
|
|
||||||
|
const char * s0_base = (const char *) src0_data + i1 * nb01 + i2 * nb02 + i3 * nb03;
|
||||||
|
for (int64_t i0 = 0; i0 < ne00; i0++) {
|
||||||
|
dst_row[i0] = *(const float *) (s0_base + i0 * nb00);
|
||||||
|
}
|
||||||
|
|
||||||
|
const char * s1_base = (const char *) src1_data + i1 * nb11 + i2 * nb12 + i3 * nb13;
|
||||||
|
for (int64_t i0 = 0; i0 < ne10; i0++) {
|
||||||
|
dst_row[ne00 + i0] = *(const float *) (s1_base + i0 * nb10);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Standard path: ne0 % 16 == 0, aligned rows
|
||||||
|
for (int64_t row = thread_id; row < total_rows; row += num_threads) {
|
||||||
|
// Decompose linear row index into (i1, i2, i3)
|
||||||
|
int64_t i1 = row % ne1;
|
||||||
|
int64_t i2 = (row / ne1) % ne2;
|
||||||
|
int64_t i3 = row / (ne1 * ne2);
|
||||||
|
|
||||||
|
float * dst_row = (float *) ((char *) dst_data + i1 * dnb1 + i2 * dnb2 + i3 * dnb3);
|
||||||
|
|
||||||
|
if (dim == 0) {
|
||||||
|
// Concat along innermost dimension: [src0_row | src1_row]
|
||||||
|
// Both ne00 and ne10 are multiples of 16 (cacheline-aligned)
|
||||||
|
const float * s0_row = (const float *) ((const char *) src0_data + i1 * nb01 + i2 * nb02 + i3 * nb03);
|
||||||
|
const float * s1_row = (const float *) ((const char *) src1_data + i1 * nb11 + i2 * nb12 + i3 * nb13);
|
||||||
|
|
||||||
|
copy_row_aligned(dst_row, s0_row, (int32_t) ne00);
|
||||||
|
copy_row_aligned(dst_row + ne00, s1_row, (int32_t) ne10);
|
||||||
|
|
||||||
|
} else if (dim == 1) {
|
||||||
|
// Concat along dim 1: first ne01 rows from src0, rest from src1
|
||||||
|
if (i1 < ne01) {
|
||||||
|
const float * s0_row = (const float *) ((const char *) src0_data + i1 * nb01 + i2 * nb02 + i3 * nb03);
|
||||||
|
copy_row_aligned(dst_row, s0_row, (int32_t) ne0);
|
||||||
|
} else {
|
||||||
|
const float * s1_row =
|
||||||
|
(const float *) ((const char *) src1_data + (i1 - ne01) * nb11 + i2 * nb12 + i3 * nb13);
|
||||||
|
copy_row_aligned(dst_row, s1_row, (int32_t) ne0);
|
||||||
|
}
|
||||||
|
|
||||||
|
} else if (dim == 2) {
|
||||||
|
// Concat along dim 2: first ne02 slices from src0, rest from src1
|
||||||
|
if (i2 < ne02) {
|
||||||
|
const float * s0_row = (const float *) ((const char *) src0_data + i1 * nb01 + i2 * nb02 + i3 * nb03);
|
||||||
|
copy_row_aligned(dst_row, s0_row, (int32_t) ne0);
|
||||||
|
} else {
|
||||||
|
const float * s1_row =
|
||||||
|
(const float *) ((const char *) src1_data + i1 * nb11 + (i2 - ne02) * nb12 + i3 * nb13);
|
||||||
|
copy_row_aligned(dst_row, s1_row, (int32_t) ne0);
|
||||||
|
}
|
||||||
|
|
||||||
|
} else {
|
||||||
|
// dim == 3: first ne03 batches from src0, rest from src1
|
||||||
|
if (i3 < ne03) {
|
||||||
|
const float * s0_row = (const float *) ((const char *) src0_data + i1 * nb01 + i2 * nb02 + i3 * nb03);
|
||||||
|
copy_row_aligned(dst_row, s0_row, (int32_t) ne0);
|
||||||
|
} else {
|
||||||
|
const float * s1_row =
|
||||||
|
(const float *) ((const char *) src1_data + i1 * nb11 + i2 * nb12 + (i3 - ne03) * nb13);
|
||||||
|
copy_row_aligned(dst_row, s1_row, (int32_t) ne0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,107 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// Bare Metal CONT F16 Kernel
|
||||||
|
// Converts non-contiguous F16 tensors to contiguous memory layout
|
||||||
|
//
|
||||||
|
// Note: F16 is represented as uint16_t (IEEE 754 binary16 format)
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "platform.h"
|
||||||
|
|
||||||
|
#include <assert.h>
|
||||||
|
#include <stdbool.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
struct ggml_et_cont_params {
|
||||||
|
struct ggml_tensor src0; // F16 input tensor (non-contiguous)
|
||||||
|
struct ggml_tensor dst; // F16 output tensor (contiguous)
|
||||||
|
};
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_cont_params * params, void * env) {
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
|
||||||
|
if (!kernel_env) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
int num_threads = 2048; //get_num_threads(kernel_env->shire_mask);
|
||||||
|
|
||||||
|
if (thread_id < 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (params == 0 || ((uint64_t) params & 0x7) != 0) {
|
||||||
|
return -1; // Invalid pointer
|
||||||
|
}
|
||||||
|
|
||||||
|
struct ggml_tensor * src0 = ¶ms->src0; // Non-contiguous input
|
||||||
|
struct ggml_tensor * dst = ¶ms->dst; // Contiguous output
|
||||||
|
|
||||||
|
if (src0->type != GGML_TYPE_F16 || dst->type != GGML_TYPE_F16) {
|
||||||
|
return -1; // Unsupported type combination
|
||||||
|
}
|
||||||
|
|
||||||
|
uint16_t * src0_data = (uint16_t *) src0->data;
|
||||||
|
uint16_t * dst_data = (uint16_t *) dst->data;
|
||||||
|
|
||||||
|
if (!src0_data || !dst_data) {
|
||||||
|
return -1; // Null data pointer
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t src_elements = src0->ne[0] * src0->ne[1] * src0->ne[2] * src0->ne[3];
|
||||||
|
const int64_t dst_elements = dst->ne[0] * dst->ne[1] * dst->ne[2] * dst->ne[3];
|
||||||
|
if (src_elements != dst_elements) {
|
||||||
|
return -1; // Element count mismatch
|
||||||
|
}
|
||||||
|
|
||||||
|
// Source tensor dimensions and strides
|
||||||
|
const int64_t ne00 = src0->ne[0];
|
||||||
|
const int64_t ne01 = src0->ne[1];
|
||||||
|
const int64_t ne02 = src0->ne[2];
|
||||||
|
const int64_t ne03 = src0->ne[3];
|
||||||
|
|
||||||
|
const int64_t nb00 = src0->nb[0];
|
||||||
|
const int64_t nb01 = src0->nb[1];
|
||||||
|
const int64_t nb02 = src0->nb[2];
|
||||||
|
const int64_t nb03 = src0->nb[3];
|
||||||
|
|
||||||
|
// Parallelize by rows (dimension 1)
|
||||||
|
const int64_t total_rows = ne01;
|
||||||
|
const int64_t rows_per_thread = (total_rows + num_threads - 1) / num_threads;
|
||||||
|
const int64_t start_row = thread_id * rows_per_thread;
|
||||||
|
const int64_t end_row = (start_row + rows_per_thread < total_rows) ? (start_row + rows_per_thread) : total_rows;
|
||||||
|
|
||||||
|
if (start_row >= total_rows) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Iterate over source tensor dimensions
|
||||||
|
for (int64_t i03 = 0; i03 < ne03; i03++) {
|
||||||
|
for (int64_t i02 = 0; i02 < ne02; i02++) {
|
||||||
|
// Calculate base linear index for this (i03, i02) slice in destination
|
||||||
|
const int64_t dst_linear_base = i03 * ne02 * ne01 * ne00 + i02 * ne01 * ne00;
|
||||||
|
|
||||||
|
// Process this thread's assigned rows
|
||||||
|
for (int64_t i01 = start_row; i01 < end_row; i01++) {
|
||||||
|
// Linear index for start of this row in destination
|
||||||
|
const int64_t dst_linear_row_base = dst_linear_base + i01 * ne00;
|
||||||
|
|
||||||
|
// Inner loop over dimension 0
|
||||||
|
for (int64_t i00 = 0; i00 < ne00; i00++) {
|
||||||
|
// Source offset using non-contiguous strides
|
||||||
|
const int64_t src_offset_bytes = i00 * nb00 + i01 * nb01 + i02 * nb02 + i03 * nb03;
|
||||||
|
const uint16_t * src_ptr = (const uint16_t *) ((const char *) src0_data + src_offset_bytes);
|
||||||
|
|
||||||
|
// Destination linear index (contiguous layout)
|
||||||
|
const int64_t dst_linear_idx = dst_linear_row_base + i00;
|
||||||
|
|
||||||
|
// Use atomic store for thread safety
|
||||||
|
atomic_store_f16((volatile uint16_t *) &dst_data[dst_linear_idx], *src_ptr);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,248 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// Bare Metal CONT F32 Kernel
|
||||||
|
// Converts non-contiguous tensors to contiguous memory layout
|
||||||
|
//
|
||||||
|
// Fast path: src contiguous: flat vectorized copy by cache lines
|
||||||
|
// Aligned path: nb00==4 and ne00 % 16 == 0: distribute rows, no coherency issue
|
||||||
|
// Unaligned: nb00==4 and ne00 not aligned: distribute by cache lines,
|
||||||
|
// reverse-compute src coords, handle partial rows at boundaries
|
||||||
|
// Fallback: nb00 != 4: scalar per-element
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "platform.h"
|
||||||
|
|
||||||
|
#include <stdbool.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
struct ggml_et_cont_params {
|
||||||
|
struct ggml_tensor src0; // F32 input tensor (non-contiguous)
|
||||||
|
struct ggml_tensor dst; // F32 output tensor (contiguous)
|
||||||
|
};
|
||||||
|
|
||||||
|
// Vectorized copy with scalar tail
|
||||||
|
static inline void vec_copy_f32(float * dst, const float * src, int32_t n) {
|
||||||
|
int32_t i = 0;
|
||||||
|
const int32_t vec_end = (n / 8) * 8;
|
||||||
|
for (; i < vec_end; i += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f10, %[s]\n"
|
||||||
|
"fsw.ps f10, %[d]\n"
|
||||||
|
: [d] "=m"(*(float (*)[8]) & dst[i])
|
||||||
|
: [s] "m"(*(const float (*)[8]) & src[i])
|
||||||
|
: "f10");
|
||||||
|
}
|
||||||
|
for (; i < n; i++) {
|
||||||
|
dst[i] = src[i];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scalar copy
|
||||||
|
static inline void scalar_copy_f32(float * dst, const float * src, int32_t n) {
|
||||||
|
for (int32_t i = 0; i < n; i++) {
|
||||||
|
dst[i] = src[i];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// static inline size_t tensor_bytes(const struct ggml_tensor *t) {
|
||||||
|
// return (size_t)t->ne[0] * t->ne[1] * t->ne[2] * t->ne[3] * t->nb[0];
|
||||||
|
// }
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_cont_params * params, void * env) {
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
|
||||||
|
if (!kernel_env) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
int num_threads = get_num_threads(kernel_env->shire_mask);
|
||||||
|
|
||||||
|
if (thread_id < 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (params == 0 || ((uint64_t) params & 0x7) != 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct ggml_tensor * src0 = ¶ms->src0;
|
||||||
|
struct ggml_tensor * dst = ¶ms->dst;
|
||||||
|
|
||||||
|
if (src0->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
float * src0_data = (float *) src0->data;
|
||||||
|
float * dst_data = (float *) dst->data;
|
||||||
|
// evict_region_past_l2(src0_data, tensor_bytes(src0));
|
||||||
|
|
||||||
|
if (!src0_data || !dst_data) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t ne00 = src0->ne[0];
|
||||||
|
const int64_t ne01 = src0->ne[1];
|
||||||
|
const int64_t ne02 = src0->ne[2];
|
||||||
|
const int64_t ne03 = src0->ne[3];
|
||||||
|
|
||||||
|
const int64_t nb00 = src0->nb[0];
|
||||||
|
const int64_t nb01 = src0->nb[1];
|
||||||
|
const int64_t nb02 = src0->nb[2];
|
||||||
|
const int64_t nb03 = src0->nb[3];
|
||||||
|
|
||||||
|
const int64_t total_elements = ne00 * ne01 * ne02 * ne03;
|
||||||
|
|
||||||
|
if (total_elements == 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const bool src_contiguous = ggml_tensor_is_contiguous(src0, 4);
|
||||||
|
|
||||||
|
//==========================================================================
|
||||||
|
// Fast path: src is contiguous: flat vectorized copy by cache lines
|
||||||
|
//==========================================================================
|
||||||
|
if (src_contiguous) {
|
||||||
|
const int64_t elems_per_cl = 16;
|
||||||
|
const int64_t total_cl = (total_elements + elems_per_cl - 1) / elems_per_cl;
|
||||||
|
|
||||||
|
const int64_t cl_per_thread = (total_cl + num_threads - 1) / num_threads;
|
||||||
|
const int64_t cl_start = thread_id * cl_per_thread;
|
||||||
|
int64_t cl_end = cl_start + cl_per_thread;
|
||||||
|
if (cl_end > total_cl) {
|
||||||
|
cl_end = total_cl;
|
||||||
|
}
|
||||||
|
if (cl_start >= total_cl) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t es = cl_start * elems_per_cl;
|
||||||
|
int64_t ee = cl_end * elems_per_cl;
|
||||||
|
if (ee > total_elements) {
|
||||||
|
ee = total_elements;
|
||||||
|
}
|
||||||
|
|
||||||
|
vec_copy_f32(dst_data + es, src0_data + es, (int32_t) (ee - es));
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
//==========================================================================
|
||||||
|
// Non-contiguous paths: require nb00==4 (dim 0 contiguous in src)
|
||||||
|
//==========================================================================
|
||||||
|
if (nb00 != 4) {
|
||||||
|
// Fully non-contiguous scalar fallback — distribute by cache lines
|
||||||
|
const int64_t elems_per_cl = 16;
|
||||||
|
const int64_t total_cl = (total_elements + elems_per_cl - 1) / elems_per_cl;
|
||||||
|
|
||||||
|
const int64_t cl_per_thread = (total_cl + num_threads - 1) / num_threads;
|
||||||
|
const int64_t cl_start = thread_id * cl_per_thread;
|
||||||
|
int64_t cl_end = cl_start + cl_per_thread;
|
||||||
|
if (cl_end > total_cl) {
|
||||||
|
cl_end = total_cl;
|
||||||
|
}
|
||||||
|
if (cl_start >= total_cl) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t es = cl_start * elems_per_cl;
|
||||||
|
int64_t ee = cl_end * elems_per_cl;
|
||||||
|
if (ee > total_elements) {
|
||||||
|
ee = total_elements;
|
||||||
|
}
|
||||||
|
|
||||||
|
for (int64_t idx = es; idx < ee; idx++) {
|
||||||
|
const int64_t i00 = idx % ne00;
|
||||||
|
const int64_t rem1 = idx / ne00;
|
||||||
|
const int64_t i01 = rem1 % ne01;
|
||||||
|
const int64_t rem2 = rem1 / ne01;
|
||||||
|
const int64_t i02 = rem2 % ne02;
|
||||||
|
const int64_t i03 = rem2 / ne02;
|
||||||
|
|
||||||
|
const float * sp =
|
||||||
|
(const float *) ((const char *) src0_data + i00 * nb00 + i01 * nb01 + i02 * nb02 + i03 * nb03);
|
||||||
|
dst_data[idx] = *sp;
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
// nb00 == 4 from here: dim 0 is contiguous in src
|
||||||
|
|
||||||
|
//==========================================================================
|
||||||
|
// Aligned path: ne00 % 16 == 0: rows are cache-line aligned, distribute rows
|
||||||
|
//==========================================================================
|
||||||
|
if (ne00 % 16 == 0) {
|
||||||
|
const int64_t total_rows = ne01 * ne02 * ne03;
|
||||||
|
const int64_t rows_per_thread = (total_rows + num_threads - 1) / num_threads;
|
||||||
|
const int64_t start_row = thread_id * rows_per_thread;
|
||||||
|
const int64_t end_row = (start_row + rows_per_thread < total_rows) ? (start_row + rows_per_thread) : total_rows;
|
||||||
|
|
||||||
|
if (start_row >= total_rows) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
for (int64_t ir = start_row; ir < end_row; ir++) {
|
||||||
|
const int64_t i03 = ir / (ne02 * ne01);
|
||||||
|
const int64_t i02 = (ir - i03 * ne02 * ne01) / ne01;
|
||||||
|
const int64_t i01 = ir - i03 * ne02 * ne01 - i02 * ne01;
|
||||||
|
|
||||||
|
const float * src_row = (const float *) ((const char *) src0_data + i01 * nb01 + i02 * nb02 + i03 * nb03);
|
||||||
|
float * dst_row = dst_data + ir * ne00;
|
||||||
|
|
||||||
|
vec_copy_f32(dst_row, src_row, (int32_t) ne00);
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
//==========================================================================
|
||||||
|
// Unaligned path: ne00 % 16 != 0, nb00 == 4
|
||||||
|
// Distribute cache-line-aligned chunks of dst, handle partial rows at edges
|
||||||
|
//==========================================================================
|
||||||
|
{
|
||||||
|
const int64_t elems_per_cl = 16;
|
||||||
|
const int64_t total_cl = (total_elements + elems_per_cl - 1) / elems_per_cl;
|
||||||
|
|
||||||
|
const int64_t cl_per_thread = (total_cl + num_threads - 1) / num_threads;
|
||||||
|
const int64_t cl_start = thread_id * cl_per_thread;
|
||||||
|
int64_t cl_end = cl_start + cl_per_thread;
|
||||||
|
if (cl_end > total_cl) {
|
||||||
|
cl_end = total_cl;
|
||||||
|
}
|
||||||
|
if (cl_start >= total_cl) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t es = cl_start * elems_per_cl;
|
||||||
|
int64_t ee = cl_end * elems_per_cl;
|
||||||
|
if (ee > total_elements) {
|
||||||
|
ee = total_elements;
|
||||||
|
}
|
||||||
|
|
||||||
|
int64_t pos = es;
|
||||||
|
|
||||||
|
// Compute starting row coordinates
|
||||||
|
int64_t row_idx = pos / ne00;
|
||||||
|
int64_t col = pos % ne00;
|
||||||
|
|
||||||
|
while (pos < ee) {
|
||||||
|
// Decompose row_idx -> (i01, i02, i03)
|
||||||
|
const int64_t i03 = row_idx / (ne02 * ne01);
|
||||||
|
const int64_t i02 = (row_idx - i03 * ne02 * ne01) / ne01;
|
||||||
|
const int64_t i01 = row_idx - i03 * ne02 * ne01 - i02 * ne01;
|
||||||
|
|
||||||
|
const float * src_row = (const float *) ((const char *) src0_data + i01 * nb01 + i02 * nb02 + i03 * nb03);
|
||||||
|
|
||||||
|
// How many elements left in this row and in our chunk
|
||||||
|
int64_t row_remaining = ne00 - col;
|
||||||
|
int64_t chunk_remaining = ee - pos;
|
||||||
|
int32_t n = (int32_t) (row_remaining < chunk_remaining ? row_remaining : chunk_remaining);
|
||||||
|
|
||||||
|
vec_copy_f32(dst_data + pos, src_row + col, n);
|
||||||
|
|
||||||
|
pos += n;
|
||||||
|
col = 0; // subsequent rows start at column 0
|
||||||
|
row_idx++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,807 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// 2D F32 convolution on the ET-SoC-1 matrix engine (GGML CONV_2D layout).
|
||||||
|
//
|
||||||
|
// LAYOUT (matches GGML's standard CONV_2D, cwhn=false; wireable directly):
|
||||||
|
// src1 input : ne = [W, H, Cin, N=1] memory: input [n][cin][h][w]
|
||||||
|
// src0 filter: ne = [Kw, Kh, Cin, Cout] memory: filter[oc][ic][kh][kw]
|
||||||
|
// dst output: ne = [W, H, Cout, N=1] memory: output[n][oc][h][w]
|
||||||
|
//
|
||||||
|
// CONSTRAINTS (enforced at supports_op):
|
||||||
|
// F32 throughout, N == 1, Cin % 16 == 0, Cout % 16 == 0, positive
|
||||||
|
// stride/pad, dilation == 1. Tile/L2SCP limits are checked here.
|
||||||
|
//
|
||||||
|
// MEMORY MODEL:
|
||||||
|
// Each active shire uses its own 2 MB local L2 SCP:
|
||||||
|
// filter slice | pin buffer 0 | pin buffer 1? | output staging? | scratch
|
||||||
|
//
|
||||||
|
// The filter slice contains only the output-channel tiles (`mt`) consumed
|
||||||
|
// by this shire's tile assignment. That keeps hart-0's inner-loop
|
||||||
|
// tensor_loads local to the shire and avoids packing unused filter slabs.
|
||||||
|
//
|
||||||
|
// THREADING (multi-minion, multi-shire):
|
||||||
|
// PHASE 1 (per-shire filter pack): hart-1's pack this shire's filter
|
||||||
|
// slice into local L2 SCP. Work is slab-striped across the 32 minions.
|
||||||
|
//
|
||||||
|
// PHASE 2 (per-shire compute): hart-1's pack the input pin chunks while
|
||||||
|
// hart-0's run the matrix engine. Pin double-buffering hides the next
|
||||||
|
// chunk pack behind the current chunk's FMA pipeline when Cin does not
|
||||||
|
// fit in one local buffer.
|
||||||
|
//
|
||||||
|
// PERFORMANCE STRATEGIES:
|
||||||
|
// 1. Local filter slice: pack only the `mt` values this shire consumes;
|
||||||
|
// inner-loop tensor_loads stay shire-local.
|
||||||
|
// 2. Pin Cin streaming + chunk double-buffer: pack one
|
||||||
|
// chunk while computing the prior one.
|
||||||
|
// 3. TenC save/restore: f0..f31 IS the TenC accumulator;
|
||||||
|
// spill/refill via L2 SCP scratch lets each hart hold multiple
|
||||||
|
// partial accumulators across chunks.
|
||||||
|
// 4. OW%16 staging: for partial-tile output, write to a
|
||||||
|
// padded L2 SCP region then have one hart scalar-emit to DRAM.
|
||||||
|
//
|
||||||
|
// WHY THE FILTER PACK EXISTS:
|
||||||
|
// GGML's OIHW filter has stride Kh*Kw*4 between consecutive Cin elements
|
||||||
|
// (e.g. 36 bytes for 3x3) — usually NOT a multiple of 64, so plain
|
||||||
|
// tensor_load cannot gather it directly. The per-slab pack into a
|
||||||
|
// Cin-innermost form gives every per-tap slab a flat 64-byte row stride
|
||||||
|
// and enables tensor_load.
|
||||||
|
//
|
||||||
|
// Picking M=Cout, N=W means TenC's natural row stride matches NCHW
|
||||||
|
// output's per-channel stride (H*W*4) — the output store is a clean
|
||||||
|
// tensor_store with no transpose. The price is that conv_size/conv_ctrl
|
||||||
|
// no longer help with W boundaries (mask gates M, not N), so we handle
|
||||||
|
// boundaries up-front by zero-padding the input in L2SCP.
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "platform.h"
|
||||||
|
#include "tensor.h"
|
||||||
|
|
||||||
|
#include <etsoc/common/utils.h>
|
||||||
|
#include <stdbool.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
#define TILE 16 /* matrix engine native tile in M, K, N */
|
||||||
|
/* L1 SCP layout: A double-buffered, B single-buffered. Per the SDK doc
|
||||||
|
`dst_start` is a 6-bit field (max 63) but empirical testing shows the
|
||||||
|
physical L1 SCP per minion is 48 lines — writes to lines >= 48 corrupt.
|
||||||
|
So we get 3 × 16-line buffers max: A_0, A_1, B. Pick A as the
|
||||||
|
double-buffered operand (filter-slab loads, the longer of the two). */
|
||||||
|
#define LSCP_A_0 0 /* A buffer 0 at L1 SCP lines 0..15 */
|
||||||
|
#define LSCP_A_1 16 /* A buffer 1 at L1 SCP lines 16..31 */
|
||||||
|
#define LSCP_B 32 /* B (single buffer) at lines 32..47 */
|
||||||
|
#define N_MIN_PER_SHIRE 32 /* ET-SoC-1 geometry: 32 minions/shire */
|
||||||
|
#define N_SHIRES 32 /* default active shire count */
|
||||||
|
#define MAX_TILES_PER_HART 2 /* per-hart TenC slots (save/restore) */
|
||||||
|
#define MAX_DBL_BUFS 2 /* chunk pack buffers (double-buffered) */
|
||||||
|
|
||||||
|
/* Per-shire L2 SCP local budget. Per-shire SCP is 2 MB; we cap at
|
||||||
|
1984 KB to leave 64 KB headroom for per-hart TenC scratch (32 minions ×
|
||||||
|
2 slots × 1 KB), which lives at the tail of the SCP outside the pin
|
||||||
|
sizing budget. Bigger budget here means bigger feasible chunk_KT,
|
||||||
|
which means fewer chunks (each chunk costs 2 SHIRE barriers + ~30
|
||||||
|
TenC save/restore events per hart). */
|
||||||
|
#define LOCAL_BUDGET (1984 * 1024)
|
||||||
|
|
||||||
|
/* Cap on the per-shire filter region in local L2 SCP. The shire packs the
|
||||||
|
mt values it can consume under the current tile assignment, rather than
|
||||||
|
the whole Cout dimension. Reads in the inner loop are then fully
|
||||||
|
shire-local — no NoC fanout. */
|
||||||
|
#define LOCAL_FILTER_CAP (1024 * 1024) /* 1 MB / shire ceiling */
|
||||||
|
|
||||||
|
#define SLAB_BYTES ((uint64_t) TILE * TILE * sizeof(float)) /* 1024 */
|
||||||
|
#define SLAB_LINES ((SLAB_BYTES + 63) / 64) /* 16 */
|
||||||
|
|
||||||
|
/* Upper bound on the number of distinct mt values a single shire may pack.
|
||||||
|
This keeps the mt list stack-resident. Shapes that need more should fall
|
||||||
|
back until the filter-slice bookkeeping is made dynamic. */
|
||||||
|
#define MAX_MY_MT (N_MIN_PER_SHIRE * MAX_TILES_PER_HART)
|
||||||
|
|
||||||
|
typedef struct {
|
||||||
|
int mt;
|
||||||
|
int mt_idx;
|
||||||
|
int oh;
|
||||||
|
int ow_base;
|
||||||
|
} conv_tile_t;
|
||||||
|
|
||||||
|
static inline int ceil_div_i32(int x, int y) {
|
||||||
|
return (x + y - 1) / y;
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline int round_up_tile_i32(int x) {
|
||||||
|
return (x + TILE - 1) & ~(TILE - 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline int min_i32(int a, int b) {
|
||||||
|
return a < b ? a : b;
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline uint64_t min_u64(uint64_t a, uint64_t b) {
|
||||||
|
return a < b ? a : b;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ===== Vector helpers for hart-1 pack ============================
|
||||||
|
Both assume dst (and src for copy) are 32-byte aligned; n is in floats.
|
||||||
|
The 8-element tail is handled scalar. f30/f31 are scratch — clobbered
|
||||||
|
per-call via the asm clobber list. */
|
||||||
|
static inline void vec_zero_aligned(float * dst, int n) {
|
||||||
|
int i = 0;
|
||||||
|
const int n8 = n & ~7;
|
||||||
|
for (; i < n8; i += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"fsub.ps f31, f31, f31\n"
|
||||||
|
"fsw.ps f31, %[d]\n"
|
||||||
|
: [d] "=m"(*(float (*)[8]) & dst[i])
|
||||||
|
:
|
||||||
|
: "f31");
|
||||||
|
}
|
||||||
|
for (; i < n; ++i) {
|
||||||
|
dst[i] = 0.0f;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline void vec_copy_aligned(float * dst, const float * src, int n) {
|
||||||
|
int i = 0;
|
||||||
|
const int n8 = n & ~7;
|
||||||
|
for (; i < n8; i += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f30, %[s]\n"
|
||||||
|
"fsw.ps f30, %[d]\n"
|
||||||
|
: [d] "=m"(*(float (*)[8]) & dst[i])
|
||||||
|
: [s] "m"(*(const float (*)[8]) & src[i])
|
||||||
|
: "f30");
|
||||||
|
}
|
||||||
|
for (; i < n; ++i) {
|
||||||
|
dst[i] = src[i];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ===== TenC save/restore =========================================
|
||||||
|
The TenC accumulator IS the f0..f31 vector register file: row N occupies
|
||||||
|
f(2N) and f(2N+1) (two 8-fp32 vector regs per row). We save by
|
||||||
|
tensor_store-ing TILE rows × 64 bytes, and restore via 32 flw.ps after
|
||||||
|
forcing L1D to refetch from the L2SCP backing (tensor_store bypasses L1D
|
||||||
|
so the backing is always current). See feedback_tenc_save_restore.md. */
|
||||||
|
static inline void tenc_restore_from_scratch(uint64_t scr) {
|
||||||
|
FENCE;
|
||||||
|
evict_to_l2((const void *) scr, TILE, 64);
|
||||||
|
WAIT_CACHEOPS;
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f0, 0(%0)\n"
|
||||||
|
"flw.ps f1, 32(%0)\n"
|
||||||
|
"flw.ps f2, 64(%0)\n"
|
||||||
|
"flw.ps f3, 96(%0)\n"
|
||||||
|
"flw.ps f4, 128(%0)\n"
|
||||||
|
"flw.ps f5, 160(%0)\n"
|
||||||
|
"flw.ps f6, 192(%0)\n"
|
||||||
|
"flw.ps f7, 224(%0)\n"
|
||||||
|
"flw.ps f8, 256(%0)\n"
|
||||||
|
"flw.ps f9, 288(%0)\n"
|
||||||
|
"flw.ps f10, 320(%0)\n"
|
||||||
|
"flw.ps f11, 352(%0)\n"
|
||||||
|
"flw.ps f12, 384(%0)\n"
|
||||||
|
"flw.ps f13, 416(%0)\n"
|
||||||
|
"flw.ps f14, 448(%0)\n"
|
||||||
|
"flw.ps f15, 480(%0)\n"
|
||||||
|
"flw.ps f16, 512(%0)\n"
|
||||||
|
"flw.ps f17, 544(%0)\n"
|
||||||
|
"flw.ps f18, 576(%0)\n"
|
||||||
|
"flw.ps f19, 608(%0)\n"
|
||||||
|
"flw.ps f20, 640(%0)\n"
|
||||||
|
"flw.ps f21, 672(%0)\n"
|
||||||
|
"flw.ps f22, 704(%0)\n"
|
||||||
|
"flw.ps f23, 736(%0)\n"
|
||||||
|
"flw.ps f24, 768(%0)\n"
|
||||||
|
"flw.ps f25, 800(%0)\n"
|
||||||
|
"flw.ps f26, 832(%0)\n"
|
||||||
|
"flw.ps f27, 864(%0)\n"
|
||||||
|
"flw.ps f28, 896(%0)\n"
|
||||||
|
"flw.ps f29, 928(%0)\n"
|
||||||
|
"flw.ps f30, 960(%0)\n"
|
||||||
|
"flw.ps f31, 992(%0)\n"
|
||||||
|
:
|
||||||
|
: "r"(scr)
|
||||||
|
: "f0", "f1", "f2", "f3", "f4", "f5", "f6", "f7", "f8", "f9", "f10", "f11", "f12", "f13", "f14", "f15", "f16",
|
||||||
|
"f17", "f18", "f19", "f20", "f21", "f22", "f23", "f24", "f25", "f26", "f27", "f28", "f29", "f30", "f31",
|
||||||
|
"memory");
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ===== Pin pack context ==========================================
|
||||||
|
Loop-invariant state hart-1 needs to pack one Cin chunk's worth of
|
||||||
|
pin (Kw shifted, padded copies of input rows) into local L2 SCP. The
|
||||||
|
filter is not touched in this struct; it is packed into the per-shire
|
||||||
|
local slice before the per-chunk loop begins. */
|
||||||
|
typedef struct {
|
||||||
|
const float * in_base; /* DRAM input base [Cin][H][W] */
|
||||||
|
int Kw;
|
||||||
|
int chunk_KT; /* number of K_TILES (=16-wide) per chunk */
|
||||||
|
int H, W, Hp, Wp_a;
|
||||||
|
int pad_h, pad_w, s0;
|
||||||
|
int minion; /* this hart's minion id (0..31) */
|
||||||
|
uint64_t pin_copy_floats; /* per-_s pin plane size in floats */
|
||||||
|
uint64_t l2_pad_in_buf[MAX_DBL_BUFS];
|
||||||
|
uint64_t pin_chunk_bytes; /* one chunk pin buffer's total size */
|
||||||
|
} pin_ctx_t;
|
||||||
|
|
||||||
|
static inline int find_mt_idx(const int * my_mt, int n_my_mt, int mt) {
|
||||||
|
for (int j = 0; j < n_my_mt; ++j) {
|
||||||
|
if (my_mt[j] == mt) {
|
||||||
|
return j;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline conv_tile_t decode_tile(int t, int M_TILES, int w_tiles, const int * my_mt, int n_my_mt) {
|
||||||
|
conv_tile_t tile;
|
||||||
|
tile.mt = t % M_TILES;
|
||||||
|
t /= M_TILES;
|
||||||
|
const int wt = t % w_tiles;
|
||||||
|
t /= w_tiles;
|
||||||
|
tile.oh = t;
|
||||||
|
tile.ow_base = wt * TILE;
|
||||||
|
tile.mt_idx = find_mt_idx(my_mt, n_my_mt, tile.mt);
|
||||||
|
return tile;
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline uint64_t
|
||||||
|
filter_slab_addr(uint64_t l2_filter, int Kw, int K_TILES, int n_my_mt, int mt_idx, int kh, int kw, int kt_global) {
|
||||||
|
return l2_filter + (uint64_t) ((((kh * Kw + kw) * n_my_mt + mt_idx) * K_TILES + kt_global)) * SLAB_BYTES;
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline uint64_t pin_tile_addr(uint64_t l2_pad_in,
|
||||||
|
uint64_t pin_copy_bytes,
|
||||||
|
int ktc,
|
||||||
|
int kw,
|
||||||
|
int Hp,
|
||||||
|
int Wp_a,
|
||||||
|
int oh,
|
||||||
|
int ow_base,
|
||||||
|
int s1,
|
||||||
|
int kh) {
|
||||||
|
const int ir_pad = oh * s1 + kh;
|
||||||
|
return l2_pad_in + (uint64_t) kw * pin_copy_bytes +
|
||||||
|
(((uint64_t) (ktc * TILE) * Hp + ir_pad) * Wp_a + ow_base) * sizeof(float);
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline char * output_tile_addr(char * out_base,
|
||||||
|
const conv_tile_t * tile,
|
||||||
|
uint64_t out_chan_stride,
|
||||||
|
uint64_t out_row_stride) {
|
||||||
|
return out_base + (size_t) (tile->mt * TILE) * out_chan_stride + (size_t) tile->oh * out_row_stride +
|
||||||
|
(size_t) tile->ow_base * sizeof(float);
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline void flush_range_to_l2(const void * addr, uint64_t n_bytes) {
|
||||||
|
const uint64_t total_lines = (n_bytes + 63) / 64;
|
||||||
|
const char * fl_addr = (const char *) addr;
|
||||||
|
for (uint64_t done = 0; done < total_lines;) {
|
||||||
|
const uint64_t batch = min_u64(total_lines - done, 16);
|
||||||
|
flush_to_l2((const void *) (fl_addr + done * 64), batch, 64);
|
||||||
|
done += batch;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline void evict_range_past_l2(const void * addr, uint64_t n_bytes) {
|
||||||
|
const uint64_t total_lines = (n_bytes + 63) / 64;
|
||||||
|
const char * fl_addr = (const char *) addr;
|
||||||
|
for (uint64_t done = 0; done < total_lines;) {
|
||||||
|
const uint64_t batch = min_u64(total_lines - done, 16);
|
||||||
|
evict_past_l2((const void *) (fl_addr + done * 64), batch, 64);
|
||||||
|
done += batch;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* One matrix-engine tile for one Cin chunk. This is the main optimization
|
||||||
|
surface: A is double-buffered, B is single-buffered due to L1 SCP space. */
|
||||||
|
static inline void compute_tile_chunk(uint64_t l2_filter,
|
||||||
|
uint64_t l2_pad_in,
|
||||||
|
uint64_t pin_copy_bytes,
|
||||||
|
int Kh,
|
||||||
|
int Kw,
|
||||||
|
int K_TILES,
|
||||||
|
int chunk_KT,
|
||||||
|
int kt_base,
|
||||||
|
int n_my_mt,
|
||||||
|
int Hp,
|
||||||
|
int Wp_a,
|
||||||
|
int s1,
|
||||||
|
uint64_t a_row_stride,
|
||||||
|
uint64_t b_row_stride,
|
||||||
|
const conv_tile_t * tile,
|
||||||
|
bool first_fma_clears_tenc) {
|
||||||
|
const int n_iters = Kh * Kw * chunk_KT;
|
||||||
|
const uint64_t A_BUFS[2] = { LSCP_A_0, LSCP_A_1 };
|
||||||
|
|
||||||
|
const uint64_t a_addr0 = filter_slab_addr(l2_filter, Kw, K_TILES, n_my_mt, tile->mt_idx, 0, 0, kt_base);
|
||||||
|
tensor_load(false, false, A_BUFS[0], 0, 0, a_addr0, 0, (uint64_t) (TILE - 1), a_row_stride, 0);
|
||||||
|
|
||||||
|
for (int iter = 0; iter < n_iters; ++iter) {
|
||||||
|
const int ktc = iter % chunk_KT;
|
||||||
|
const int rem = iter / chunk_KT;
|
||||||
|
const int kw = rem % Kw;
|
||||||
|
const int kh = rem / Kw;
|
||||||
|
|
||||||
|
const uint64_t b_addr =
|
||||||
|
pin_tile_addr(l2_pad_in, pin_copy_bytes, ktc, kw, Hp, Wp_a, tile->oh, tile->ow_base, s1, kh);
|
||||||
|
tensor_load(false, false, LSCP_B, 0, 0, b_addr, 0, (uint64_t) (TILE - 1), b_row_stride, 1);
|
||||||
|
|
||||||
|
tensor_wait(TENSOR_LOAD_WAIT_0);
|
||||||
|
tensor_wait(TENSOR_LOAD_WAIT_1);
|
||||||
|
|
||||||
|
if (iter + 1 < n_iters) {
|
||||||
|
const int ktc_n = (iter + 1) % chunk_KT;
|
||||||
|
const int rem_n = (iter + 1) / chunk_KT;
|
||||||
|
const int kw_n = rem_n % Kw;
|
||||||
|
const int kh_n = rem_n / Kw;
|
||||||
|
const uint64_t a_addr_n =
|
||||||
|
filter_slab_addr(l2_filter, Kw, K_TILES, n_my_mt, tile->mt_idx, kh_n, kw_n, kt_base + ktc_n);
|
||||||
|
tensor_load(false, false, A_BUFS[(iter + 1) & 1], 0, 0, a_addr_n, 0, (uint64_t) (TILE - 1), a_row_stride,
|
||||||
|
0);
|
||||||
|
}
|
||||||
|
|
||||||
|
tensor_fma(false, 3, (uint64_t) (TILE - 1), (uint64_t) (TILE - 1), 0, false, false, false, false, LSCP_B,
|
||||||
|
A_BUFS[iter & 1], 0, first_fma_clears_tenc && (iter == 0));
|
||||||
|
tensor_wait(TENSOR_FMA_WAIT);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Pack only the slabs this shire's tiles actually consume, into local
|
||||||
|
L2 SCP. Slab layout in the filter buffer is [Kh][Kw][n_my_mt][K_TILES]
|
||||||
|
of TILE×TILE slabs (Cin-innermost form). Distributed across the 32
|
||||||
|
hart-1's of this shire by `slab % 32 == minion`.
|
||||||
|
|
||||||
|
This deliberately favors local inner-loop reads over global filter fanout.
|
||||||
|
Depending on tile shape, two shires may pack the same mt value; keep that
|
||||||
|
tradeoff visible when experimenting with shared-filter layouts. */
|
||||||
|
static void pack_filter_local_mt(const float * flt_base,
|
||||||
|
int Kh,
|
||||||
|
int Kw,
|
||||||
|
int Cin,
|
||||||
|
int K_TILES,
|
||||||
|
const int * my_mt,
|
||||||
|
int n_my_mt,
|
||||||
|
int minion,
|
||||||
|
uint64_t l2_filter_base) {
|
||||||
|
const int n_slabs = Kh * Kw * n_my_mt * K_TILES;
|
||||||
|
const size_t kstep = (size_t) Kh * Kw; /* Cin stride in floats */
|
||||||
|
|
||||||
|
for (int slab = minion; slab < n_slabs; slab += N_MIN_PER_SHIRE) {
|
||||||
|
int t = slab;
|
||||||
|
const int kt = t % K_TILES;
|
||||||
|
t /= K_TILES;
|
||||||
|
const int mt_idx = t % n_my_mt;
|
||||||
|
t /= n_my_mt;
|
||||||
|
const int kw = t % Kw;
|
||||||
|
t /= Kw;
|
||||||
|
const int kh = t;
|
||||||
|
const int mt = my_mt[mt_idx];
|
||||||
|
|
||||||
|
const uint64_t slab_offset = (uint64_t) slab * SLAB_BYTES;
|
||||||
|
float * cell = (float *) (l2_filter_base + slab_offset);
|
||||||
|
|
||||||
|
for (int oc_in = 0; oc_in < TILE; ++oc_in) {
|
||||||
|
const int oc = mt * TILE + oc_in;
|
||||||
|
const float * src = flt_base + (((size_t) oc * Cin + (size_t) kt * TILE) * Kh + kh) * Kw + kw;
|
||||||
|
float * row = cell + (size_t) oc_in * TILE;
|
||||||
|
float scratch[TILE] __attribute__((aligned(32)));
|
||||||
|
for (int ic_in = 0; ic_in < TILE; ++ic_in) {
|
||||||
|
scratch[ic_in] = src[(size_t) ic_in * kstep];
|
||||||
|
}
|
||||||
|
vec_copy_aligned(row, scratch, TILE);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Flush this hart's dirty L1D lines for the slabs it wrote. */
|
||||||
|
FENCE;
|
||||||
|
for (int slab = minion; slab < n_slabs; slab += N_MIN_PER_SHIRE) {
|
||||||
|
const uint64_t slab_offset = (uint64_t) slab * SLAB_BYTES;
|
||||||
|
flush_to_l2((const void *) (l2_filter_base + slab_offset), SLAB_LINES, 64);
|
||||||
|
}
|
||||||
|
WAIT_CACHEOPS;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Pack one Cin chunk of the input pin (Kw shifted padded copies) into the
|
||||||
|
buf_idx side of local L2SCP. Work distributed across the 32 hart-1's in
|
||||||
|
the shire by `plane % 32 == minion`. The final flush_to_l2 forces L1D
|
||||||
|
write-back so hart-0's tensor_load sees the freshly written bytes. */
|
||||||
|
static void pack_pin_chunk(const pin_ctx_t * ctx, int chunk_id, int buf_idx) {
|
||||||
|
const int kt_base = chunk_id * ctx->chunk_KT;
|
||||||
|
const int Kw = ctx->Kw;
|
||||||
|
const int chunk_KT = ctx->chunk_KT;
|
||||||
|
const int H = ctx->H, W = ctx->W, Hp = ctx->Hp, Wp_a = ctx->Wp_a;
|
||||||
|
const int pad_h = ctx->pad_h, pad_w = ctx->pad_w, s0 = ctx->s0;
|
||||||
|
const int minion = ctx->minion;
|
||||||
|
|
||||||
|
/* Pin pack: Kw shifted, padded copies of input rows. Bounds [vlo, vhi)
|
||||||
|
hoisted outside the row loop so the inner loop is three regions
|
||||||
|
(zero-prefix | bulk-copy | zero-suffix) with no per-element predicate. */
|
||||||
|
float * pin0 = (float *) ctx->l2_pad_in_buf[buf_idx];
|
||||||
|
const int chunk_Cin = chunk_KT * TILE;
|
||||||
|
const int n_pin_planes = Kw * chunk_Cin;
|
||||||
|
for (int p = minion; p < n_pin_planes; p += N_MIN_PER_SHIRE) {
|
||||||
|
const int s = p / chunk_Cin;
|
||||||
|
const int icc = p % chunk_Cin;
|
||||||
|
const int ic = kt_base * TILE + icc;
|
||||||
|
float * pin_s = pin0 + (size_t) s * ctx->pin_copy_floats;
|
||||||
|
|
||||||
|
const int offset = s - pad_w;
|
||||||
|
int vlo = 0;
|
||||||
|
while (vlo < Wp_a && (s0 * vlo + offset) < 0) {
|
||||||
|
vlo++;
|
||||||
|
}
|
||||||
|
int vhi = Wp_a;
|
||||||
|
while (vhi > vlo && (s0 * (vhi - 1) + offset) >= W) {
|
||||||
|
vhi--;
|
||||||
|
}
|
||||||
|
const bool aligned = (s0 == 1) && ((vlo & 7) == 0) && (((vlo + offset) & 7) == 0);
|
||||||
|
|
||||||
|
for (int r = 0; r < Hp; ++r) {
|
||||||
|
float * row = pin_s + ((size_t) icc * Hp + r) * Wp_a;
|
||||||
|
const int real_h = r - pad_h;
|
||||||
|
if (real_h < 0 || real_h >= H) {
|
||||||
|
vec_zero_aligned(row, Wp_a);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const float * src_row = ctx->in_base + ((size_t) ic * H + real_h) * W;
|
||||||
|
|
||||||
|
for (int cc = 0; cc < vlo; ++cc) {
|
||||||
|
row[cc] = 0.0f;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (aligned) {
|
||||||
|
vec_copy_aligned(row + vlo, src_row + vlo + offset, vhi - vlo);
|
||||||
|
} else if (s0 == 1) {
|
||||||
|
const float * csrc = src_row + vlo + offset;
|
||||||
|
const int n = vhi - vlo;
|
||||||
|
for (int cc = 0; cc < n; ++cc) {
|
||||||
|
row[vlo + cc] = csrc[cc];
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
for (int cc = vlo; cc < vhi; ++cc) {
|
||||||
|
row[cc] = src_row[s0 * cc + offset];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for (int cc = vhi; cc < Wp_a; ++cc) {
|
||||||
|
row[cc] = 0.0f;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Flush this buffer's L1D-dirty lines down to L2SCP backing. */
|
||||||
|
FENCE;
|
||||||
|
flush_range_to_l2((const void *) ctx->l2_pad_in_buf[buf_idx], ctx->pin_chunk_bytes);
|
||||||
|
WAIT_CACHEOPS;
|
||||||
|
}
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_binary_params * params, void * env) {
|
||||||
|
(void) env;
|
||||||
|
|
||||||
|
const int shire = get_shire_id();
|
||||||
|
const int hart_id = get_hart_id();
|
||||||
|
const int minion = (hart_id >> 1) & 0x1F;
|
||||||
|
const int hart1 = hart_id & 1;
|
||||||
|
|
||||||
|
const struct ggml_tensor * flt = ¶ms->src0; /* [Kw,Kh,Cin,Cout] */
|
||||||
|
const struct ggml_tensor * in = ¶ms->src1; /* [W, H, Cin,N=1 ] */
|
||||||
|
struct ggml_tensor * out = ¶ms->dst; /* [W, H, Cout,N=1] */
|
||||||
|
|
||||||
|
const int Kw = (int) flt->ne[0];
|
||||||
|
const int Kh = (int) flt->ne[1];
|
||||||
|
const int Cin = (int) flt->ne[2];
|
||||||
|
const int Cout = (int) flt->ne[3];
|
||||||
|
|
||||||
|
const int W = (int) in->ne[0];
|
||||||
|
const int H = (int) in->ne[1];
|
||||||
|
const int OW = (int) out->ne[0];
|
||||||
|
const int OH = (int) out->ne[1];
|
||||||
|
|
||||||
|
/* op_params layout (set by ggml_conv_2d):
|
||||||
|
[0]=s0 [1]=s1 [2]=p0 [3]=p1 [4]=d0 [5]=d1 */
|
||||||
|
const int s0 = out->op_params[0];
|
||||||
|
const int s1 = out->op_params[1];
|
||||||
|
const int pad_w = out->op_params[2];
|
||||||
|
const int pad_h = out->op_params[3];
|
||||||
|
|
||||||
|
if (Cin <= 0 || Cout <= 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
if (Cin % TILE != 0 || Cout % TILE != 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
if (W <= 0 || H <= 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
if (s0 <= 0 || s1 <= 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
if (in->ne[2] != Cin || in->ne[3] != 1) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
if (out->ne[2] != Cout || out->ne[3] != 1) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
if (!flt->data || !in->data || !out->data) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int K_TILES = Cin / TILE;
|
||||||
|
const int M_TILES = Cout / TILE;
|
||||||
|
|
||||||
|
const int Hp = H + 2 * pad_h;
|
||||||
|
const int Wp_a = round_up_tile_i32(OW);
|
||||||
|
const int OW_pad = Wp_a;
|
||||||
|
const bool need_stage = (OW % TILE != 0);
|
||||||
|
|
||||||
|
/* ===================== Tile assignment & active-shire selection =====
|
||||||
|
Computed up front because the per-shire mt set (and thus filter
|
||||||
|
region size) depends on n_active_shires. */
|
||||||
|
const int w_tiles = ceil_div_i32(OW, TILE);
|
||||||
|
const int total_tiles = OH * w_tiles * M_TILES;
|
||||||
|
const int n_active_shires = need_stage ? 1 : min_i32(total_tiles, N_SHIRES);
|
||||||
|
|
||||||
|
/* Inactive shires exit immediately. No global barrier — pack and
|
||||||
|
barriers are now per-shire, so unused shires don't need to vote. */
|
||||||
|
if (shire >= n_active_shires) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ===================== Determine this shire's mt set ================
|
||||||
|
Standard tile assignment: tile t is owned by
|
||||||
|
shire = t % n_active_shires
|
||||||
|
minion = (t / n_active_shires) % N_MIN_PER_SHIRE
|
||||||
|
slot = t / (n_active_shires * N_MIN_PER_SHIRE)
|
||||||
|
So the set of mt's this shire actually consumes is the set of
|
||||||
|
(t % M_TILES) for all t this shire owns. Enumerate all shire-owned
|
||||||
|
tiles, not just the first MAX_TILES_PER_HART slots; the one-chunk
|
||||||
|
path can process more tiles serially. */
|
||||||
|
int my_mt[MAX_MY_MT];
|
||||||
|
int n_my_mt = 0;
|
||||||
|
for (int t = shire; t < total_tiles; t += n_active_shires) {
|
||||||
|
const int mt = t % M_TILES;
|
||||||
|
bool found = false;
|
||||||
|
for (int j = 0; j < n_my_mt; ++j) {
|
||||||
|
if (my_mt[j] == mt) {
|
||||||
|
found = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!found) {
|
||||||
|
if (n_my_mt >= MAX_MY_MT) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
my_mt[n_my_mt++] = mt;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (n_my_mt == 0) {
|
||||||
|
return 0; /* no tiles for this shire */
|
||||||
|
}
|
||||||
|
|
||||||
|
const uint64_t filter_local_bytes = (uint64_t) Kh * Kw * n_my_mt * K_TILES * SLAB_BYTES;
|
||||||
|
if (filter_local_bytes > LOCAL_FILTER_CAP) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ===================== L2 SCP local layout =========================
|
||||||
|
filter (this shire's mt slice) | pin_buf[0] | pin_buf[1]?
|
||||||
|
| output_stage? | scratch (streaming) */
|
||||||
|
const uint64_t l2_base = (uint64_t) et_shire_l2scp_local(0);
|
||||||
|
const uint64_t l2_filter = l2_base;
|
||||||
|
|
||||||
|
/* Sizing for pin: budget = LOCAL_BUDGET - filter - output_stage. */
|
||||||
|
const int64_t output_stage_bytes_full = need_stage ? (int64_t) Cout * OH * OW_pad * (int64_t) sizeof(float) : 0;
|
||||||
|
const int64_t budget_for_chunks = (int64_t) LOCAL_BUDGET - (int64_t) filter_local_bytes - output_stage_bytes_full;
|
||||||
|
if (budget_for_chunks <= 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
const int64_t per_KT_pin_bytes = (int64_t) Kw * TILE * Hp * Wp_a * (int64_t) sizeof(float);
|
||||||
|
|
||||||
|
int chunk_KT;
|
||||||
|
int n_buffers;
|
||||||
|
if ((int64_t) K_TILES * per_KT_pin_bytes <= budget_for_chunks) {
|
||||||
|
chunk_KT = K_TILES;
|
||||||
|
n_buffers = 1;
|
||||||
|
} else {
|
||||||
|
chunk_KT = K_TILES;
|
||||||
|
while (chunk_KT > 1 && 2 * (int64_t) chunk_KT * per_KT_pin_bytes > budget_for_chunks) {
|
||||||
|
chunk_KT--;
|
||||||
|
}
|
||||||
|
while (chunk_KT > 1 && K_TILES % chunk_KT != 0) {
|
||||||
|
chunk_KT--;
|
||||||
|
}
|
||||||
|
n_buffers = (chunk_KT < K_TILES) ? 2 : 1;
|
||||||
|
if (chunk_KT < 1) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const int n_chunks = K_TILES / chunk_KT;
|
||||||
|
|
||||||
|
/* Streaming keeps partial sums in MAX_TILES_PER_HART scratch slots per
|
||||||
|
hart. The one-chunk path does not need scratch and can stream a longer
|
||||||
|
tile list serially, but multi-chunk shapes must fit this fixed slot
|
||||||
|
count until scratch scheduling is made more general. */
|
||||||
|
const int shire_tile_capacity = shire + MAX_TILES_PER_HART * n_active_shires * N_MIN_PER_SHIRE;
|
||||||
|
if (n_chunks > 1 && shire_tile_capacity < total_tiles) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const uint64_t pin_copy_floats = (uint64_t) chunk_KT * TILE * Hp * Wp_a;
|
||||||
|
const uint64_t pin_copy_bytes = pin_copy_floats * sizeof(float);
|
||||||
|
const uint64_t pin_chunk_bytes = (uint64_t) Kw * pin_copy_bytes;
|
||||||
|
|
||||||
|
const uint64_t l2_pin_base = l2_filter + filter_local_bytes;
|
||||||
|
const uint64_t l2_pin_buf[MAX_DBL_BUFS] = {
|
||||||
|
l2_pin_base,
|
||||||
|
l2_pin_base + pin_chunk_bytes,
|
||||||
|
};
|
||||||
|
|
||||||
|
const uint64_t l2_output_stage = need_stage ? l2_pin_base + (uint64_t) n_buffers * pin_chunk_bytes : 0;
|
||||||
|
|
||||||
|
const uint64_t scratch_per_hart = (uint64_t) MAX_TILES_PER_HART * (uint64_t) TILE * TILE * sizeof(float);
|
||||||
|
const uint64_t l2_scratch_base = need_stage ? l2_output_stage + (uint64_t) output_stage_bytes_full :
|
||||||
|
l2_pin_base + (uint64_t) n_buffers * pin_chunk_bytes;
|
||||||
|
|
||||||
|
/* ===================== PHASE 1: Filter pack (per-shire mt slice) ====
|
||||||
|
Hart-1's pack only this shire's mt slabs into local L2 SCP. The
|
||||||
|
SHIRE barrier below ensures the filter is in L2 SCP backing before
|
||||||
|
hart-0's first tensor_load. */
|
||||||
|
if (hart1) {
|
||||||
|
pack_filter_local_mt((const float *) flt->data, Kh, Kw, Cin, K_TILES, my_mt, n_my_mt, minion, l2_filter);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ===================== Hart 1: pin packer (per chunk) ==============
|
||||||
|
Double-buffered prefetch: pack chunk 0 synchronously, then per chunk c
|
||||||
|
signal "buf c ready", pack chunk c+1 into the alternate buffer
|
||||||
|
(overlaps hart-0's compute on c), signal "buf c done". */
|
||||||
|
if (hart1) {
|
||||||
|
const pin_ctx_t ctx = {
|
||||||
|
.in_base = (const float *) in->data,
|
||||||
|
.Kw = Kw,
|
||||||
|
.chunk_KT = chunk_KT,
|
||||||
|
.H = H,
|
||||||
|
.W = W,
|
||||||
|
.Hp = Hp,
|
||||||
|
.Wp_a = Wp_a,
|
||||||
|
.pad_h = pad_h,
|
||||||
|
.pad_w = pad_w,
|
||||||
|
.s0 = s0,
|
||||||
|
.minion = minion,
|
||||||
|
.pin_copy_floats = pin_copy_floats,
|
||||||
|
.l2_pad_in_buf = { l2_pin_buf[0], l2_pin_buf[1] },
|
||||||
|
.pin_chunk_bytes = pin_chunk_bytes,
|
||||||
|
};
|
||||||
|
|
||||||
|
pack_pin_chunk(&ctx, 0, 0); /* prologue */
|
||||||
|
|
||||||
|
for (int c = 0; c < n_chunks; ++c) {
|
||||||
|
et_barrier(ET_BARRIER_SHIRE); /* signal "buf c ready" */
|
||||||
|
if (n_buffers > 1 && c + 1 < n_chunks) {
|
||||||
|
pack_pin_chunk(&ctx, c + 1, (c + 1) & 1);
|
||||||
|
}
|
||||||
|
et_barrier(ET_BARRIER_SHIRE); /* wait "buf c done" */
|
||||||
|
}
|
||||||
|
|
||||||
|
if (need_stage) {
|
||||||
|
et_barrier(ET_BARRIER_SHIRE);
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ===================== Hart 0: matrix engine ======================
|
||||||
|
Two execution modes:
|
||||||
|
- n_chunks == 1: full Cin in one shot. Each hart processes a list
|
||||||
|
of tiles serially; TenC resets between tiles via first_pass=true.
|
||||||
|
- n_chunks > 1: streaming. Each hart owns up to MAX_TILES_PER_HART
|
||||||
|
tiles. For each chunk c, restore TenC from scratch[k] (skip on
|
||||||
|
c==0), accumulate this chunk's FMAs, then either save TenC back
|
||||||
|
to scratch[k] (c < last) or tensor_store directly (c == last). */
|
||||||
|
setup_cache_scp();
|
||||||
|
CLEAR_TENSOR_ERROR;
|
||||||
|
|
||||||
|
char * const out_base = need_stage ? (char *) l2_output_stage : (char *) out->data;
|
||||||
|
const int compute_OW = need_stage ? OW_pad : OW;
|
||||||
|
const uint64_t out_chan_stride = (uint64_t) OH * (uint64_t) compute_OW * sizeof(float);
|
||||||
|
const uint64_t out_row_stride = (uint64_t) compute_OW * sizeof(float);
|
||||||
|
|
||||||
|
const uint64_t a_row_stride = (uint64_t) TILE * sizeof(float); /* 64 */
|
||||||
|
const uint64_t b_row_stride = (uint64_t) Hp * (uint64_t) Wp_a * sizeof(float);
|
||||||
|
|
||||||
|
/* Tile assignment: shire-strided so small workloads spread across
|
||||||
|
shires before stacking minions in one shire. */
|
||||||
|
const int t_start = shire + minion * n_active_shires;
|
||||||
|
const int t_stride = n_active_shires * N_MIN_PER_SHIRE;
|
||||||
|
|
||||||
|
if (n_chunks == 1) {
|
||||||
|
et_barrier(ET_BARRIER_SHIRE); /* wait for the (only) pin chunk */
|
||||||
|
|
||||||
|
const uint64_t l2_pad_in = l2_pin_buf[0];
|
||||||
|
for (int t = t_start; t < total_tiles; t += t_stride) {
|
||||||
|
const conv_tile_t tile = decode_tile(t, M_TILES, w_tiles, my_mt, n_my_mt);
|
||||||
|
compute_tile_chunk(l2_filter, l2_pad_in, pin_copy_bytes, Kh, Kw, K_TILES, chunk_KT, 0, n_my_mt, Hp, Wp_a,
|
||||||
|
s1, a_row_stride, b_row_stride, &tile, /*first_fma_clears_tenc=*/true);
|
||||||
|
|
||||||
|
char * dst_addr = output_tile_addr(out_base, &tile, out_chan_stride, out_row_stride);
|
||||||
|
tensor_store(0, 0, 3, (uint64_t) (TILE - 1), (uint64_t) dst_addr, 0, out_chan_stride);
|
||||||
|
tensor_wait(TENSOR_STORE_WAIT);
|
||||||
|
}
|
||||||
|
|
||||||
|
et_barrier(ET_BARRIER_SHIRE); /* matches hart-1's second barrier */
|
||||||
|
|
||||||
|
} else {
|
||||||
|
/* Streaming path: each hart owns up to MAX_TILES_PER_HART tiles. */
|
||||||
|
int my_tiles[MAX_TILES_PER_HART];
|
||||||
|
int n_my_tiles = 0;
|
||||||
|
for (int slot = 0; slot < MAX_TILES_PER_HART; ++slot) {
|
||||||
|
const int t = t_start + slot * t_stride;
|
||||||
|
if (t < total_tiles) {
|
||||||
|
my_tiles[n_my_tiles++] = t;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
conv_tile_t tiles[MAX_TILES_PER_HART];
|
||||||
|
for (int k = 0; k < n_my_tiles; ++k) {
|
||||||
|
tiles[k] = decode_tile(my_tiles[k], M_TILES, w_tiles, my_mt, n_my_mt);
|
||||||
|
}
|
||||||
|
|
||||||
|
const uint64_t my_scratch_base = l2_scratch_base + (uint64_t) minion * scratch_per_hart;
|
||||||
|
|
||||||
|
for (int c = 0; c < n_chunks; ++c) {
|
||||||
|
et_barrier(ET_BARRIER_SHIRE); /* pin chunk c packed */
|
||||||
|
|
||||||
|
const int buf = c & 1;
|
||||||
|
const uint64_t l2_pad_in = l2_pin_buf[buf];
|
||||||
|
const int kt_base = c * chunk_KT;
|
||||||
|
|
||||||
|
for (int k = 0; k < n_my_tiles; ++k) {
|
||||||
|
const conv_tile_t * tile = &tiles[k];
|
||||||
|
const uint64_t scr = my_scratch_base + (uint64_t) k * (TILE * TILE * sizeof(float));
|
||||||
|
|
||||||
|
const bool first_pass_chunk = (c == 0);
|
||||||
|
if (!first_pass_chunk) {
|
||||||
|
tenc_restore_from_scratch(scr);
|
||||||
|
}
|
||||||
|
|
||||||
|
compute_tile_chunk(l2_filter, l2_pad_in, pin_copy_bytes, Kh, Kw, K_TILES, chunk_KT, kt_base, n_my_mt,
|
||||||
|
Hp, Wp_a, s1, a_row_stride, b_row_stride, tile, first_pass_chunk);
|
||||||
|
|
||||||
|
if (c == n_chunks - 1) {
|
||||||
|
char * dst_addr = output_tile_addr(out_base, tile, out_chan_stride, out_row_stride);
|
||||||
|
tensor_store(0, 0, 3, (uint64_t) (TILE - 1), (uint64_t) dst_addr, 0, out_chan_stride);
|
||||||
|
} else {
|
||||||
|
tensor_store(0, 0, 3, (uint64_t) (TILE - 1), (uint64_t) scr, 0, 64);
|
||||||
|
}
|
||||||
|
tensor_wait(TENSOR_STORE_WAIT);
|
||||||
|
}
|
||||||
|
|
||||||
|
et_barrier(ET_BARRIER_SHIRE); /* hart-0 done with chunk c */
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
FENCE;
|
||||||
|
|
||||||
|
/* ----------------------- DRAM emit phase ---------------------------
|
||||||
|
Only relevant when we staged into L2SCP because OW % 16 != 0. */
|
||||||
|
if (need_stage) {
|
||||||
|
et_barrier(ET_BARRIER_SHIRE);
|
||||||
|
|
||||||
|
if (minion == 0) {
|
||||||
|
const float * stage = (const float *) l2_output_stage;
|
||||||
|
float * dram = (float *) out->data;
|
||||||
|
for (int oc = 0; oc < Cout; ++oc) {
|
||||||
|
for (int oh2 = 0; oh2 < OH; ++oh2) {
|
||||||
|
const float * src = stage + ((size_t) oc * OH + oh2) * OW_pad;
|
||||||
|
float * dst = dram + ((size_t) oc * OH + oh2) * OW;
|
||||||
|
for (int ow2 = 0; ow2 < OW; ++ow2) {
|
||||||
|
dst[ow2] = src[ow2];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
FENCE;
|
||||||
|
const uint64_t total_bytes = (uint64_t) Cout * OH * OW * sizeof(float);
|
||||||
|
evict_range_past_l2((const void *) dram, total_bytes);
|
||||||
|
WAIT_CACHEOPS;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,110 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// CPY F32 -> F16 Kernel
|
||||||
|
// Copies F32 source tensor to F16 destination tensor (contiguous output).
|
||||||
|
// Source may have arbitrary strides; destination must be contiguous.
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "math_fp.h"
|
||||||
|
#include "platform.h"
|
||||||
|
|
||||||
|
#include <stdbool.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
struct ggml_et_cont_params {
|
||||||
|
struct ggml_tensor src0;
|
||||||
|
struct ggml_tensor dst;
|
||||||
|
};
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_cont_params * params, void * env) {
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
|
||||||
|
if (!kernel_env || !params) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
int num_threads = get_num_threads(kernel_env->shire_mask);
|
||||||
|
|
||||||
|
if (thread_id < 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct ggml_tensor * src0 = ¶ms->src0;
|
||||||
|
struct ggml_tensor * dst = ¶ms->dst;
|
||||||
|
|
||||||
|
if (src0->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F16) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const char * src_data = (const char *) src0->data;
|
||||||
|
uint16_t * dst_data = (uint16_t *) dst->data;
|
||||||
|
|
||||||
|
if (!src_data || !dst_data) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t ne00 = src0->ne[0];
|
||||||
|
const int64_t ne01 = src0->ne[1];
|
||||||
|
const int64_t ne02 = src0->ne[2];
|
||||||
|
const int64_t ne03 = src0->ne[3];
|
||||||
|
|
||||||
|
const int64_t nb00 = src0->nb[0];
|
||||||
|
const int64_t nb01 = src0->nb[1];
|
||||||
|
const int64_t nb02 = src0->nb[2];
|
||||||
|
const int64_t nb03 = src0->nb[3];
|
||||||
|
|
||||||
|
const int64_t total_elements = ne00 * ne01 * ne02 * ne03;
|
||||||
|
|
||||||
|
if (total_elements == 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check if src is contiguous F32
|
||||||
|
const bool src_contiguous =
|
||||||
|
(nb00 == 4 && nb01 == ne00 * 4 && nb02 == ne00 * ne01 * 4 && nb03 == ne00 * ne01 * ne02 * 4);
|
||||||
|
|
||||||
|
// Distribute by cache lines (16 F16 elements = 32 bytes = half cache line)
|
||||||
|
// Use 32 elements per chunk to keep output cache-line aligned
|
||||||
|
const int64_t elems_per_cl = 32;
|
||||||
|
const int64_t total_cl = (total_elements + elems_per_cl - 1) / elems_per_cl;
|
||||||
|
|
||||||
|
const int64_t cl_per_thread = (total_cl + num_threads - 1) / num_threads;
|
||||||
|
const int64_t cl_start = thread_id * cl_per_thread;
|
||||||
|
int64_t cl_end = cl_start + cl_per_thread;
|
||||||
|
if (cl_end > total_cl) {
|
||||||
|
cl_end = total_cl;
|
||||||
|
}
|
||||||
|
if (cl_start >= total_cl) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t es = cl_start * elems_per_cl;
|
||||||
|
int64_t ee = cl_end * elems_per_cl;
|
||||||
|
if (ee > total_elements) {
|
||||||
|
ee = total_elements;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (src_contiguous) {
|
||||||
|
// Fast path: src is contiguous F32
|
||||||
|
const float * src_f32 = (const float *) src_data;
|
||||||
|
for (int64_t i = es; i < ee; ++i) {
|
||||||
|
dst_data[i] = fp32_to_fp16(src_f32[i]);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// General path: stride-aware read
|
||||||
|
for (int64_t idx = es; idx < ee; ++idx) {
|
||||||
|
const int64_t i00 = idx % ne00;
|
||||||
|
const int64_t rem1 = idx / ne00;
|
||||||
|
const int64_t i01 = rem1 % ne01;
|
||||||
|
const int64_t rem2 = rem1 / ne01;
|
||||||
|
const int64_t i02 = rem2 % ne02;
|
||||||
|
const int64_t i03 = rem2 / ne02;
|
||||||
|
|
||||||
|
const float val = *(const float *) (src_data + i00 * nb00 + i01 * nb01 + i02 * nb02 + i03 * nb03);
|
||||||
|
dst_data[idx] = fp32_to_fp16(val);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
.section .text.init, "ax", @progbits
|
||||||
|
.global _start
|
||||||
|
_start:
|
||||||
|
# initialize global pointer
|
||||||
|
.option push
|
||||||
|
.option norelax
|
||||||
|
la gp, __global_pointer$
|
||||||
|
.option pop
|
||||||
|
# Firmware sets stack pointer before launch
|
||||||
|
# bss not allowed, no init
|
||||||
|
call entry_point
|
||||||
|
li a2, 0 /* KERNEL_RETURN_SUCCESS (0) */
|
||||||
|
mv a1, a0
|
||||||
|
li a0, 8 /* SYSCALL_RETURN_FROM_KERNEL (8) */
|
||||||
|
ecall
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// CUMSUM F32 Kernel
|
||||||
|
// Computes an inclusive prefix sum along dim 0 for each row in higher dims.
|
||||||
|
// First-pass implementation: scalar and row-contiguous input/output only.
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "platform.h"
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
struct ggml_et_cumsum_params {
|
||||||
|
struct ggml_tensor src0;
|
||||||
|
struct ggml_tensor dst;
|
||||||
|
};
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_cumsum_params * params, void * env) {
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
|
||||||
|
if (!kernel_env) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
int num_threads = get_num_threads(kernel_env->shire_mask);
|
||||||
|
|
||||||
|
if (thread_id < 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (params == 0 || ((uint64_t) params & 0x7) != 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct ggml_tensor * src0 = ¶ms->src0;
|
||||||
|
struct ggml_tensor * dst = ¶ms->dst;
|
||||||
|
|
||||||
|
if (src0->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
float * src0_data = (float *) src0->data;
|
||||||
|
float * dst_data = (float *) dst->data;
|
||||||
|
|
||||||
|
if (!src0_data || !dst_data) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t ne0 = src0->ne[0];
|
||||||
|
const int64_t ne1 = src0->ne[1];
|
||||||
|
const int64_t ne2 = src0->ne[2];
|
||||||
|
const int64_t ne3 = src0->ne[3];
|
||||||
|
|
||||||
|
const size_t snb0 = src0->nb[0];
|
||||||
|
const size_t snb1 = src0->nb[1];
|
||||||
|
const size_t snb2 = src0->nb[2];
|
||||||
|
const size_t snb3 = src0->nb[3];
|
||||||
|
|
||||||
|
const size_t dnb0 = dst->nb[0];
|
||||||
|
const size_t dnb1 = dst->nb[1];
|
||||||
|
const size_t dnb2 = dst->nb[2];
|
||||||
|
const size_t dnb3 = dst->nb[3];
|
||||||
|
|
||||||
|
if (snb0 != sizeof(float) || dnb0 != sizeof(float)) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t total_rows = ne1 * ne2 * ne3;
|
||||||
|
const int64_t rows_per_group = et_rows_per_cacheline_group(ne0, sizeof(float));
|
||||||
|
const int64_t total_groups = (total_rows + rows_per_group - 1) / rows_per_group;
|
||||||
|
|
||||||
|
for (int64_t grp = thread_id; grp < total_groups; grp += num_threads) {
|
||||||
|
const int64_t row_start = grp * rows_per_group;
|
||||||
|
int64_t row_end = row_start + rows_per_group;
|
||||||
|
if (row_end > total_rows) {
|
||||||
|
row_end = total_rows;
|
||||||
|
}
|
||||||
|
|
||||||
|
for (int64_t row = row_start; row < row_end; ++row) {
|
||||||
|
int64_t i1 = row % ne1;
|
||||||
|
int64_t i2 = (row / ne1) % ne2;
|
||||||
|
int64_t i3 = row / (ne1 * ne2);
|
||||||
|
|
||||||
|
const float * src_row = (const float *) ((const char *) src0_data + i1 * snb1 + i2 * snb2 + i3 * snb3);
|
||||||
|
float * dst_row = (float *) ((char *) dst_data + i1 * dnb1 + i2 * dnb2 + i3 * dnb3);
|
||||||
|
|
||||||
|
float acc = 0.0f;
|
||||||
|
for (int64_t i0 = 0; i0 < ne0; ++i0) {
|
||||||
|
acc += src_row[i0];
|
||||||
|
dst_row[i0] = acc;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// Diag F32 Kernel
|
||||||
|
// Creates a diagonal matrix from a 1D vector.
|
||||||
|
// dst[i][j] = (i == j) ? src0[i] : 0.0f
|
||||||
|
//
|
||||||
|
// src0: [N, 1, ne2, ne3] (1D vector per batch)
|
||||||
|
// dst: [N, N, ne2, ne3] (diagonal matrix per batch)
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "platform.h"
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
struct ggml_et_diag_params {
|
||||||
|
struct ggml_tensor src0; // F32 input vector
|
||||||
|
struct ggml_tensor dst; // F32 output diagonal matrix
|
||||||
|
};
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_diag_params * params, void * env) {
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
|
||||||
|
if (!kernel_env) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
int num_threads = get_num_threads(kernel_env->shire_mask);
|
||||||
|
|
||||||
|
if (thread_id < 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (params == 0 || ((uint64_t) params & 0x7) != 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct ggml_tensor * src0 = ¶ms->src0;
|
||||||
|
struct ggml_tensor * dst = ¶ms->dst;
|
||||||
|
|
||||||
|
if (src0->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
float * src0_data = (float *) src0->data;
|
||||||
|
float * dst_data = (float *) dst->data;
|
||||||
|
|
||||||
|
if (!src0_data || !dst_data) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t ne0 = dst->ne[0]; // N (row width = column count)
|
||||||
|
const int64_t ne1 = dst->ne[1]; // N (number of rows)
|
||||||
|
const int64_t ne2 = dst->ne[2];
|
||||||
|
const int64_t ne3 = dst->ne[3];
|
||||||
|
|
||||||
|
const size_t nb1 = dst->nb[1], nb2 = dst->nb[2], nb3 = dst->nb[3];
|
||||||
|
const size_t nb02 = src0->nb[2], nb03 = src0->nb[3];
|
||||||
|
|
||||||
|
// Total rows across all batches — parallelize over these
|
||||||
|
const int64_t total_rows = ne1 * ne2 * ne3;
|
||||||
|
|
||||||
|
// Prepare zero vector for SIMD zeroing
|
||||||
|
float zero = 0.0f;
|
||||||
|
__asm__ volatile("fbc.ps f10, %[z]\n" : : [z] "m"(zero) : "f10");
|
||||||
|
|
||||||
|
for (int64_t row = thread_id; row < total_rows; row += num_threads) {
|
||||||
|
int64_t i1 = row % ne1;
|
||||||
|
int64_t i2 = (row / ne1) % ne2;
|
||||||
|
int64_t i3 = row / (ne1 * ne2);
|
||||||
|
|
||||||
|
float * dst_row = (float *) ((char *) dst_data + i1 * nb1 + i2 * nb2 + i3 * nb3);
|
||||||
|
|
||||||
|
// Zero the entire row with SIMD
|
||||||
|
int64_t i0 = 0;
|
||||||
|
const int64_t vec_end = (ne0 / 8) * 8;
|
||||||
|
for (; i0 < vec_end; i0 += 8) {
|
||||||
|
__asm__ volatile("fsw.ps f10, %[d]\n" : [d] "=m"(*(float (*)[8]) & dst_row[i0])::"f10");
|
||||||
|
}
|
||||||
|
for (; i0 < ne0; i0++) {
|
||||||
|
dst_row[i0] = 0.0f;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Place the diagonal element: dst[i1][i1] = src0[i1]
|
||||||
|
const float * src_ptr = (const float *) ((const char *) src0_data + i2 * nb02 + i3 * nb03);
|
||||||
|
dst_row[i1] = src_ptr[i1];
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,377 @@
|
|||||||
|
// Element-wise operations: dst[i] = src0[i] op src1[i]
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "platform.h"
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
// Generic m0-gated element-wise block operation.
|
||||||
|
// The OP parameter selects the instruction: "fmul.ps", "fadd.ps", "fsub.ps".
|
||||||
|
#define DEFINE_BLOCK_OP(name, op_insn) \
|
||||||
|
static inline void name(float * dst_block, const float * src0_block, const float * src1_block, int elements) { \
|
||||||
|
const int32_t vec_end = (elements / 8) * 8; \
|
||||||
|
const int32_t tail = elements - vec_end; \
|
||||||
|
\
|
||||||
|
unsigned long temp_mask; \
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(temp_mask)); \
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF"); \
|
||||||
|
\
|
||||||
|
for (int32_t i = 0; i < vec_end; i += 8) { \
|
||||||
|
__asm__ volatile( \
|
||||||
|
"flw.ps f10, %[s0]\n" \
|
||||||
|
"flw.ps f11, %[s1]\n" op_insn \
|
||||||
|
" f12, f10, f11\n" \
|
||||||
|
"fsw.ps f12, %[d]\n" \
|
||||||
|
: [d] "=m"(*(float (*)[8]) & dst_block[i]) \
|
||||||
|
: [s0] "m"(*(const float (*)[8]) & src0_block[i]), [s1] "m"(*(const float (*)[8]) & src1_block[i]) \
|
||||||
|
: "f10", "f11", "f12"); \
|
||||||
|
} \
|
||||||
|
/* Deal with tail chunks */ \
|
||||||
|
if (tail > 0) { \
|
||||||
|
const unsigned long tail_m0 = (1ul << tail) - 1; \
|
||||||
|
__asm__ volatile( \
|
||||||
|
"mov.m.x m0, %[tm], 0\n" \
|
||||||
|
"flw.ps f10, 0(%[s0])\n" \
|
||||||
|
"flw.ps f11, 0(%[s1])\n" op_insn \
|
||||||
|
" f12, f10, f11\n" \
|
||||||
|
"fsw.ps f12, 0(%[d])\n" \
|
||||||
|
: \
|
||||||
|
: [s0] "r"(&src0_block[vec_end]), [s1] "r"(&src1_block[vec_end]), [d] "r"(&dst_block[vec_end]), \
|
||||||
|
[tm] "r"(tail_m0) \
|
||||||
|
: "f10", "f11", "f12", "memory"); \
|
||||||
|
} \
|
||||||
|
\
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(temp_mask)); \
|
||||||
|
}
|
||||||
|
|
||||||
|
DEFINE_BLOCK_OP(block_mul_cache_aligned, "fmul.ps")
|
||||||
|
DEFINE_BLOCK_OP(block_add_cache_aligned, "fadd.ps")
|
||||||
|
DEFINE_BLOCK_OP(block_sub_cache_aligned, "fsub.ps")
|
||||||
|
|
||||||
|
// Broadcast variants: src1 is a single scalar, broadcast to all 8 lanes.
|
||||||
|
#define DEFINE_BLOCK_OP_BROADCAST(name, op_insn) \
|
||||||
|
static inline void name(float * dst_block, const float * src0_block, float scalar, int elements) { \
|
||||||
|
const int32_t vec_end = (elements / 8) * 8; \
|
||||||
|
const int32_t tail = elements - vec_end; \
|
||||||
|
\
|
||||||
|
unsigned long temp_mask; \
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(temp_mask)); \
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF"); \
|
||||||
|
\
|
||||||
|
for (int32_t i = 0; i < vec_end; i += 8) { \
|
||||||
|
__asm__ volatile( \
|
||||||
|
"flw.ps f10, %[s0]\n" \
|
||||||
|
"fbc.ps f11, %[s]\n" op_insn \
|
||||||
|
" f12, f10, f11\n" \
|
||||||
|
"fsw.ps f12, %[d]\n" \
|
||||||
|
: [d] "=m"(*(float (*)[8]) & dst_block[i]) \
|
||||||
|
: [s0] "m"(*(const float (*)[8]) & src0_block[i]), [s] "m"(scalar) \
|
||||||
|
: "f10", "f11", "f12"); \
|
||||||
|
} \
|
||||||
|
\
|
||||||
|
if (tail > 0) { \
|
||||||
|
const unsigned long tail_m0 = (1ul << tail) - 1; \
|
||||||
|
__asm__ volatile( \
|
||||||
|
"mov.m.x m0, %[tm], 0\n" \
|
||||||
|
"flw.ps f10, 0(%[s0])\n" \
|
||||||
|
"fbc.ps f11, 0(%[ps])\n" op_insn \
|
||||||
|
" f12, f10, f11\n" \
|
||||||
|
"fsw.ps f12, 0(%[d])\n" \
|
||||||
|
: \
|
||||||
|
: [s0] "r"(&src0_block[vec_end]), [ps] "r"(&scalar), [d] "r"(&dst_block[vec_end]), [tm] "r"(tail_m0) \
|
||||||
|
: "f10", "f11", "f12", "memory"); \
|
||||||
|
} \
|
||||||
|
\
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(temp_mask)); \
|
||||||
|
}
|
||||||
|
|
||||||
|
DEFINE_BLOCK_OP_BROADCAST(block_mul_broadcast, "fmul.ps")
|
||||||
|
DEFINE_BLOCK_OP_BROADCAST(block_add_broadcast, "fadd.ps")
|
||||||
|
DEFINE_BLOCK_OP_BROADCAST(block_sub_broadcast, "fsub.ps")
|
||||||
|
|
||||||
|
static inline float scalar_el_map(float src0, float src1, enum ggml_op operation) {
|
||||||
|
switch (operation) {
|
||||||
|
case GGML_OP_MUL:
|
||||||
|
return src0 * src1;
|
||||||
|
case GGML_OP_ADD:
|
||||||
|
return src0 + src1;
|
||||||
|
case GGML_OP_SUB:
|
||||||
|
return src0 - src1;
|
||||||
|
default:
|
||||||
|
return 0.0f;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_binary_params * params, void * env) {
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
|
||||||
|
if (!kernel_env) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
int num_threads = get_num_threads(kernel_env->shire_mask);
|
||||||
|
|
||||||
|
if (thread_id < 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (params == 0 || ((uint64_t) params & 0x7) != 0) {
|
||||||
|
return -1; // Invalid pointer
|
||||||
|
}
|
||||||
|
|
||||||
|
struct ggml_tensor * src0 = ¶ms->src0;
|
||||||
|
struct ggml_tensor * src1 = ¶ms->src1;
|
||||||
|
struct ggml_tensor * dst = ¶ms->dst;
|
||||||
|
|
||||||
|
if (src0->type != GGML_TYPE_F32 || src1->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) {
|
||||||
|
return -1; // Unsupported type combination
|
||||||
|
}
|
||||||
|
|
||||||
|
float * src0_data = (float *) src0->data;
|
||||||
|
float * src1_data = (float *) src1->data;
|
||||||
|
float * dst_data = (float *) dst->data;
|
||||||
|
|
||||||
|
if (!src0_data || !src1_data || !dst_data) {
|
||||||
|
return -1; // Null data pointer
|
||||||
|
}
|
||||||
|
|
||||||
|
#ifdef ET_UBERKERNEL
|
||||||
|
// Consumer-side input eviction. Required because ET caches are
|
||||||
|
// incoherent across minions: if a previous kernel in this UK batch
|
||||||
|
// left stale lines for these addresses in this hart's L1, drop them
|
||||||
|
// so we read fresh from L3/DRAM (where the producer flushed its
|
||||||
|
// results). Standalone launches don't need this -- the host-side
|
||||||
|
// runtime boundary between kernel launches handles it.
|
||||||
|
const size_t src0_bytes = (size_t) src0->ne[0] * src0->ne[1] * src0->ne[2] * src0->ne[3] * src0->nb[0];
|
||||||
|
const size_t src1_bytes = (size_t) src1->ne[0] * src1->ne[1] * src1->ne[2] * src1->ne[3] * src1->nb[0];
|
||||||
|
evict_region_past_l2(src0_data, src0_bytes);
|
||||||
|
evict_region_past_l2(src1_data, src1_bytes);
|
||||||
|
WAIT_CACHEOPS;
|
||||||
|
FENCE;
|
||||||
|
et_barrier(ET_BARRIER_GLOBAL);
|
||||||
|
#endif
|
||||||
|
|
||||||
|
enum ggml_op operation = dst->op;
|
||||||
|
|
||||||
|
if (operation != GGML_OP_MUL && operation != GGML_OP_ADD && operation != GGML_OP_SUB) {
|
||||||
|
return -1; // Unsupported operation
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t ne0 = dst->ne[0], ne1 = dst->ne[1], ne2 = dst->ne[2], ne3 = dst->ne[3];
|
||||||
|
const int64_t ne00 = src0->ne[0], ne01 = src0->ne[1], ne02 = src0->ne[2], ne03 = src0->ne[3];
|
||||||
|
const int64_t ne10 = src1->ne[0], ne11 = src1->ne[1], ne12 = src1->ne[2], ne13 = src1->ne[3];
|
||||||
|
|
||||||
|
const size_t nb0 = dst->nb[0], nb1 = dst->nb[1], nb2 = dst->nb[2], nb3 = dst->nb[3];
|
||||||
|
const size_t nb00 = src0->nb[0], nb01 = src0->nb[1], nb02 = src0->nb[2], nb03 = src0->nb[3];
|
||||||
|
const size_t nb10 = src1->nb[0], nb11 = src1->nb[1], nb12 = src1->nb[2], nb13 = src1->nb[3];
|
||||||
|
|
||||||
|
const bool cache_aligned = (dst->ne[0] % 16 == 0);
|
||||||
|
|
||||||
|
// Fast path: no broadcasting, contiguous
|
||||||
|
const bool no_broadcast = (ne10 == ne0 && ne11 == ne1 && ne12 == ne2 && ne13 == ne3);
|
||||||
|
const bool all_contiguous =
|
||||||
|
(nb0 == 4 && nb00 == 4 && nb10 == 4 && nb1 == ne0 * 4 && nb01 == ne0 * 4 && nb11 == ne0 * 4);
|
||||||
|
|
||||||
|
if (no_broadcast && all_contiguous) {
|
||||||
|
const int64_t total_elements = ne0 * ne1 * ne2 * ne3;
|
||||||
|
const int64_t elements_per_cacheline = 16; // 64 bytes / 4 bytes
|
||||||
|
const int64_t total_cachelines = (total_elements + elements_per_cacheline - 1) / elements_per_cacheline;
|
||||||
|
|
||||||
|
const int64_t cl_per_thread = (total_cachelines + num_threads - 1) / num_threads;
|
||||||
|
const int64_t cl_start = thread_id * cl_per_thread;
|
||||||
|
int64_t cl_end = cl_start + cl_per_thread;
|
||||||
|
if (cl_end > total_cachelines) {
|
||||||
|
cl_end = total_cachelines;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (cl_start >= total_cachelines) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t elem_start = cl_start * elements_per_cacheline;
|
||||||
|
int64_t elem_end = cl_end * elements_per_cacheline;
|
||||||
|
if (elem_end > total_elements) {
|
||||||
|
elem_end = total_elements;
|
||||||
|
}
|
||||||
|
const int32_t count = (int32_t) (elem_end - elem_start);
|
||||||
|
|
||||||
|
switch (operation) {
|
||||||
|
case GGML_OP_MUL:
|
||||||
|
block_mul_cache_aligned(dst_data + elem_start, src0_data + elem_start, src1_data + elem_start, count);
|
||||||
|
break;
|
||||||
|
case GGML_OP_ADD:
|
||||||
|
block_add_cache_aligned(dst_data + elem_start, src0_data + elem_start, src1_data + elem_start, count);
|
||||||
|
break;
|
||||||
|
case GGML_OP_SUB:
|
||||||
|
block_sub_cache_aligned(dst_data + elem_start, src0_data + elem_start, src1_data + elem_start, count);
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
#ifdef ET_UBERKERNEL
|
||||||
|
// Producer-side flush: ET caches are incoherent across minions, so
|
||||||
|
// a consumer kernel running on a different minion can't see our
|
||||||
|
// dirty L1 lines via its own evict_region_past_l2. Push our writes
|
||||||
|
// all the way to DRAM so the next batched kernel reads fresh.
|
||||||
|
// Standalone launches don't need this -- the host runtime boundary
|
||||||
|
// between kernel launches handles cache writeback.
|
||||||
|
FENCE;
|
||||||
|
evict_region_past_l2(dst_data + elem_start, (size_t) count * sizeof(float));
|
||||||
|
WAIT_CACHEOPS;
|
||||||
|
FENCE;
|
||||||
|
#endif
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Slow path: broadcasting or non-contiguous
|
||||||
|
const int64_t total_rows = ne1 * ne2 * ne3;
|
||||||
|
|
||||||
|
int64_t start_row;
|
||||||
|
int64_t end_row;
|
||||||
|
|
||||||
|
if (cache_aligned) {
|
||||||
|
const int64_t rows_per_thread = (total_rows + num_threads - 1) / num_threads;
|
||||||
|
start_row = thread_id * rows_per_thread;
|
||||||
|
end_row = (start_row + rows_per_thread < total_rows) ? (start_row + rows_per_thread) : total_rows;
|
||||||
|
} else {
|
||||||
|
const int64_t rows_per_group = et_rows_per_cacheline_group(ne0, sizeof(float));
|
||||||
|
const int64_t total_groups = (total_rows + rows_per_group - 1) / rows_per_group;
|
||||||
|
|
||||||
|
if (thread_id >= total_groups) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t group_start = thread_id;
|
||||||
|
for (int64_t grp = group_start; grp < total_groups; grp += num_threads) {
|
||||||
|
const int64_t group_row_start = grp * rows_per_group;
|
||||||
|
int64_t group_row_end = group_row_start + rows_per_group;
|
||||||
|
if (group_row_end > total_rows) {
|
||||||
|
group_row_end = total_rows;
|
||||||
|
}
|
||||||
|
|
||||||
|
#ifdef ET_UBERKERNEL
|
||||||
|
// First row written by this group (used for producer-side evict).
|
||||||
|
const int64_t first_i03 = group_row_start / (ne2 * ne1);
|
||||||
|
const int64_t first_i02 = (group_row_start - first_i03 * ne2 * ne1) / ne1;
|
||||||
|
const int64_t first_i01 = (group_row_start - first_i03 * ne2 * ne1 - first_i02 * ne1);
|
||||||
|
char * group_dst_base = (char *) dst_data + first_i03 * nb3 + first_i02 * nb2 + first_i01 * nb1;
|
||||||
|
#endif
|
||||||
|
|
||||||
|
for (int64_t ir = group_row_start; ir < group_row_end; ir++) {
|
||||||
|
const int64_t i03 = ir / (ne2 * ne1);
|
||||||
|
const int64_t i02 = (ir - i03 * ne2 * ne1) / ne1;
|
||||||
|
const int64_t i01 = (ir - i03 * ne2 * ne1 - i02 * ne1);
|
||||||
|
|
||||||
|
const int64_t i13 = i03 % ne13;
|
||||||
|
const int64_t i12 = i02 % ne12;
|
||||||
|
const int64_t i11 = i01 % ne11;
|
||||||
|
|
||||||
|
float * dst_ptr = (float *) ((char *) dst_data + i03 * nb3 + i02 * nb2 + i01 * nb1);
|
||||||
|
const float * src0_ptr =
|
||||||
|
(const float *) ((const char *) src0_data + i03 * nb03 + i02 * nb02 + i01 * nb01);
|
||||||
|
const float * src1_ptr =
|
||||||
|
(const float *) ((const char *) src1_data + i13 * nb13 + i12 * nb12 + i11 * nb11);
|
||||||
|
|
||||||
|
if (ne10 == 1) {
|
||||||
|
const float scalar = src1_ptr[0];
|
||||||
|
for (int64_t i0 = 0; i0 < ne0; ++i0) {
|
||||||
|
dst_ptr[i0] = scalar_el_map(src0_ptr[i0], scalar, operation);
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
for (int64_t i0 = 0; i0 < ne0; ++i0) {
|
||||||
|
dst_ptr[i0] = scalar_el_map(src0_ptr[i0], src1_ptr[i0 % ne10], operation);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#ifdef ET_UBERKERNEL
|
||||||
|
// Producer-side flush for this group's rows. Group rows are
|
||||||
|
// contiguous because nb1 = ne0*4 in the cacheline-group layout.
|
||||||
|
// Only needed inside a UK batch; see comment in fast path.
|
||||||
|
const int64_t nrows = group_row_end - group_row_start;
|
||||||
|
if (nrows > 0) {
|
||||||
|
FENCE;
|
||||||
|
evict_region_past_l2(group_dst_base, (size_t) nrows * nb1);
|
||||||
|
WAIT_CACHEOPS;
|
||||||
|
FENCE;
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (start_row >= total_rows) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
for (int64_t ir = start_row; ir < end_row; ir++) {
|
||||||
|
// Convert flat row index to 3D coordinates
|
||||||
|
const int64_t i03 = ir / (ne2 * ne1);
|
||||||
|
const int64_t i02 = (ir - i03 * ne2 * ne1) / ne1;
|
||||||
|
const int64_t i01 = (ir - i03 * ne2 * ne1 - i02 * ne1);
|
||||||
|
|
||||||
|
// Handle broadcasting: src1 coordinates with modulo
|
||||||
|
const int64_t i13 = i03 % ne13;
|
||||||
|
const int64_t i12 = i02 % ne12;
|
||||||
|
const int64_t i11 = i01 % ne11;
|
||||||
|
|
||||||
|
// Calculate base pointers for this row using stride-based addressing
|
||||||
|
float * dst_ptr = (float *) ((char *) dst_data + i03 * nb3 + i02 * nb2 + i01 * nb1);
|
||||||
|
const float * src0_ptr = (const float *) ((const char *) src0_data + i03 * nb03 + i02 * nb02 + i01 * nb01);
|
||||||
|
const float * src1_ptr = (const float *) ((const char *) src1_data + i13 * nb13 + i12 * nb12 + i11 * nb11);
|
||||||
|
|
||||||
|
if (ne10 == 1) {
|
||||||
|
// Broadcast scalar: src1 has ne[0]=1, broadcast across entire row
|
||||||
|
float scalar = src1_ptr[0];
|
||||||
|
switch (operation) {
|
||||||
|
case GGML_OP_MUL:
|
||||||
|
block_mul_broadcast(dst_ptr, src0_ptr, scalar, (int) ne0);
|
||||||
|
break;
|
||||||
|
case GGML_OP_ADD:
|
||||||
|
block_add_broadcast(dst_ptr, src0_ptr, scalar, (int) ne0);
|
||||||
|
break;
|
||||||
|
case GGML_OP_SUB:
|
||||||
|
block_sub_broadcast(dst_ptr, src0_ptr, scalar, (int) ne0);
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// Broadcasting in dimension 0: src1 repeats across src0
|
||||||
|
const int64_t nr0 = ne0 / ne10;
|
||||||
|
|
||||||
|
for (int64_t r = 0; r < nr0; r++) {
|
||||||
|
const float * src0_block = src0_ptr + r * ne10;
|
||||||
|
float * dst_block = dst_ptr + r * ne10;
|
||||||
|
|
||||||
|
switch (operation) {
|
||||||
|
case GGML_OP_MUL:
|
||||||
|
block_mul_cache_aligned(dst_block, src0_block, src1_ptr, (int) ne10);
|
||||||
|
break;
|
||||||
|
case GGML_OP_ADD:
|
||||||
|
block_add_cache_aligned(dst_block, src0_block, src1_ptr, (int) ne10);
|
||||||
|
break;
|
||||||
|
case GGML_OP_SUB:
|
||||||
|
block_sub_cache_aligned(dst_block, src0_block, src1_ptr, (int) ne10);
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#ifdef ET_UBERKERNEL
|
||||||
|
// Producer-side flush for the cache-aligned slow path. Rows
|
||||||
|
// [start_row, end_row) are contiguous in dst because nb1 = ne0 * 4.
|
||||||
|
// Only needed inside a UK batch; see comment in fast path.
|
||||||
|
if (end_row > start_row) {
|
||||||
|
FENCE;
|
||||||
|
evict_region_past_l2((char *) dst_data + start_row * nb1, (size_t) (end_row - start_row) * nb1);
|
||||||
|
WAIT_CACHEOPS;
|
||||||
|
FENCE;
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,87 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// Fill F32 Kernel
|
||||||
|
// Fills entire tensor with a constant scalar value.
|
||||||
|
// dst[i] = c for all elements
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "platform.h"
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
struct ggml_et_fill_params {
|
||||||
|
struct ggml_tensor dst; // F32 output tensor (contiguous)
|
||||||
|
float c; // Constant value to fill
|
||||||
|
};
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_fill_params * params, void * env) {
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
|
||||||
|
if (!kernel_env) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
int num_threads = get_num_threads(kernel_env->shire_mask);
|
||||||
|
|
||||||
|
if (thread_id < 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (params == 0 || ((uint64_t) params & 0x7) != 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct ggml_tensor * dst = ¶ms->dst;
|
||||||
|
|
||||||
|
if (dst->type != GGML_TYPE_F32) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
float * dst_data = (float *) dst->data;
|
||||||
|
if (!dst_data) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t total_elements = dst->ne[0] * dst->ne[1] * dst->ne[2] * dst->ne[3];
|
||||||
|
|
||||||
|
if (total_elements == 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Distribute by cache lines (16 floats = 64 bytes)
|
||||||
|
const int64_t elems_per_cl = 16;
|
||||||
|
const int64_t total_cl = (total_elements + elems_per_cl - 1) / elems_per_cl;
|
||||||
|
const int64_t cl_per_thread = (total_cl + num_threads - 1) / num_threads;
|
||||||
|
const int64_t cl_start = thread_id * cl_per_thread;
|
||||||
|
int64_t cl_end = cl_start + cl_per_thread;
|
||||||
|
if (cl_end > total_cl) {
|
||||||
|
cl_end = total_cl;
|
||||||
|
}
|
||||||
|
if (cl_start >= total_cl) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t es = cl_start * elems_per_cl;
|
||||||
|
int64_t ee = cl_end * elems_per_cl;
|
||||||
|
if (ee > total_elements) {
|
||||||
|
ee = total_elements;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Broadcast constant to all SIMD lanes
|
||||||
|
float c = params->c;
|
||||||
|
__asm__ volatile("fbc.ps f10, %[v]\n" : : [v] "m"(c) : "f10");
|
||||||
|
|
||||||
|
// Vector fill (8-wide)
|
||||||
|
int64_t i = es;
|
||||||
|
const int64_t vec_end = es + ((ee - es) / 8) * 8;
|
||||||
|
for (; i < vec_end; i += 8) {
|
||||||
|
__asm__ volatile("fsw.ps f10, %[d]\n" : [d] "=m"(*(float (*)[8]) & dst_data[i])::"f10");
|
||||||
|
}
|
||||||
|
// Scalar tail
|
||||||
|
for (; i < ee; i++) {
|
||||||
|
dst_data[i] = c;
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,217 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// F32 Flash Attention for ET backend
|
||||||
|
//
|
||||||
|
// Supports:
|
||||||
|
// - arbitrary dk/dv (up to 128)
|
||||||
|
// - GQA (n_head_q can differ from n_head_kv)
|
||||||
|
// - mask (F16 or F32, causal pattern)
|
||||||
|
// - F16 or F32 K and V (with non-contiguous strides from KV cache permute)
|
||||||
|
//
|
||||||
|
// Limitations:
|
||||||
|
// - Q and dst must be F32
|
||||||
|
// - no sinks, ALiBi, logit softcap
|
||||||
|
//
|
||||||
|
// Parallelization strategy:
|
||||||
|
// - flatten [query position, head, outer batch] into independent rows
|
||||||
|
// - assign rows round-robin across ET threads
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "math_fp.h"
|
||||||
|
#include "platform.h"
|
||||||
|
|
||||||
|
#include <stdbool.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
struct ggml_et_flash_attn_ext_params {
|
||||||
|
struct ggml_tensor src0; // Q tensor (F32)
|
||||||
|
struct ggml_tensor src1; // K tensor (F16 or F32)
|
||||||
|
struct ggml_tensor src2; // V tensor (F16 or F32)
|
||||||
|
struct ggml_tensor mask; // mask tensor (F16 or F32), zeroed when absent
|
||||||
|
struct ggml_tensor dst; // Output tensor (F32)
|
||||||
|
float scale; // Scale factor applied to QK
|
||||||
|
int32_t has_mask; // nonzero if mask is present
|
||||||
|
};
|
||||||
|
|
||||||
|
// Maximum head dimension supported (128 covers all common LLMs).
|
||||||
|
#define FA_DV_MAX 128
|
||||||
|
|
||||||
|
// Read element d from a row, handling F16 or F32 type.
|
||||||
|
// row_base points to the start of the row (byte address).
|
||||||
|
// nb0 is the stride per element (2 for F16, 4 for F32).
|
||||||
|
static inline float read_kv_f32(const char * row_base, int64_t d, int64_t nb0, int type) {
|
||||||
|
if (type == GGML_TYPE_F32) {
|
||||||
|
return *(const float *) (row_base + d * nb0);
|
||||||
|
}
|
||||||
|
// F16
|
||||||
|
return fp16_to_fp32(*(const uint16_t *) (row_base + d * nb0));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Dot product of F32 query vector with a K row (F16 or F32).
|
||||||
|
static inline float dot_qk(const float * q, const char * k_row, int64_t dk, int64_t k_nb0, int k_type) {
|
||||||
|
float acc = 0.0f;
|
||||||
|
if (k_type == GGML_TYPE_F32) {
|
||||||
|
const float * kf = (const float *) k_row;
|
||||||
|
for (int64_t i = 0; i < dk; ++i) {
|
||||||
|
acc += q[i] * kf[i];
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// F16 stride-aware read
|
||||||
|
for (int64_t i = 0; i < dk; ++i) {
|
||||||
|
acc += q[i] * fp16_to_fp32(*(const uint16_t *) (k_row + i * k_nb0));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return acc;
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline float get_mask_val(const struct ggml_tensor * mask, int64_t iq1, int64_t ik1, int64_t iq2, int64_t iq3) {
|
||||||
|
// mask layout: [nk, nq, ne2, ne3] -> broadcast via modulo
|
||||||
|
const char * base = (const char *) mask->data + iq1 * mask->nb[1] + (iq2 % mask->ne[2]) * mask->nb[2] +
|
||||||
|
(iq3 % mask->ne[3]) * mask->nb[3];
|
||||||
|
|
||||||
|
if (mask->type == GGML_TYPE_F32) {
|
||||||
|
return *(const float *) (base + ik1 * mask->nb[0]);
|
||||||
|
}
|
||||||
|
// F16
|
||||||
|
return fp16_to_fp32(*(const uint16_t *) (base + ik1 * mask->nb[0]));
|
||||||
|
}
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_flash_attn_ext_params * params, void * env) {
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
|
||||||
|
if (!kernel_env || !params) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
const int num_threads = get_num_threads(kernel_env->shire_mask);
|
||||||
|
if (thread_id < 0 || num_threads <= 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct ggml_tensor * q = ¶ms->src0;
|
||||||
|
struct ggml_tensor * k = ¶ms->src1;
|
||||||
|
struct ggml_tensor * v = ¶ms->src2;
|
||||||
|
struct ggml_tensor * dst = ¶ms->dst;
|
||||||
|
const int32_t has_mask = params->has_mask;
|
||||||
|
struct ggml_tensor * mask = has_mask ? ¶ms->mask : (struct ggml_tensor *) 0;
|
||||||
|
|
||||||
|
const char * q_data = (const char *) q->data;
|
||||||
|
const char * k_data = (const char *) k->data;
|
||||||
|
const char * v_data = (const char *) v->data;
|
||||||
|
char * dst_data = (char *) dst->data;
|
||||||
|
|
||||||
|
const int k_type = k->type;
|
||||||
|
const int v_type = v->type;
|
||||||
|
const int64_t k_nb0 = k->nb[0];
|
||||||
|
const int64_t v_nb0 = v->nb[0];
|
||||||
|
|
||||||
|
const int64_t dk = q->ne[0]; // head dim for keys/queries
|
||||||
|
const int64_t nq = q->ne[1]; // number of query positions
|
||||||
|
const int64_t nhq = q->ne[2]; // number of query heads
|
||||||
|
const int64_t no = q->ne[3]; // outer batch
|
||||||
|
|
||||||
|
const int64_t nk = k->ne[1]; // number of key/value positions
|
||||||
|
const int64_t nhk = k->ne[2]; // number of kv heads
|
||||||
|
const int64_t dv = v->ne[0]; // head dim for values
|
||||||
|
|
||||||
|
if (dv > FA_DV_MAX) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
// GQA: query heads per kv head
|
||||||
|
const int64_t gqa_ratio = nhq / nhk;
|
||||||
|
|
||||||
|
const int64_t total_rows = nq * nhq * no;
|
||||||
|
const float scale = params->scale;
|
||||||
|
|
||||||
|
// When dv is a multiple of 16 (64 bytes = cache line), output rows are
|
||||||
|
// cache-line aligned and we can use fast normal stores. Otherwise we must
|
||||||
|
// use atomic stores to avoid cache-line sharing corruption.
|
||||||
|
const int use_fast_store = (dv % 16 == 0);
|
||||||
|
|
||||||
|
for (int64_t row = thread_id; row < total_rows; row += num_threads) {
|
||||||
|
const int64_t iq3 = row / (nhq * nq);
|
||||||
|
const int64_t rem = row % (nhq * nq);
|
||||||
|
const int64_t iq2 = rem / nq; // query head index
|
||||||
|
const int64_t iq1 = rem % nq; // query position
|
||||||
|
|
||||||
|
// Map query head -> kv head for GQA
|
||||||
|
const int64_t ik2 = iq2 / gqa_ratio;
|
||||||
|
|
||||||
|
// Q is always F32
|
||||||
|
const float * pq = (const float *) (q_data + iq1 * q->nb[1] + iq2 * q->nb[2] + iq3 * q->nb[3]);
|
||||||
|
|
||||||
|
// dst layout: [dv, nhq, nq, no]
|
||||||
|
float * out = (float *) (dst_data + iq2 * dst->nb[1] + iq1 * dst->nb[2] + iq3 * dst->nb[3]);
|
||||||
|
|
||||||
|
// Base byte offsets for K and V head+batch slice
|
||||||
|
const int64_t kv_base = ik2 * k->nb[2] + iq3 * k->nb[3];
|
||||||
|
const int64_t vv_base = ik2 * v->nb[2] + iq3 * v->nb[3];
|
||||||
|
|
||||||
|
float acc[FA_DV_MAX];
|
||||||
|
for (int64_t d = 0; d < dv; ++d) {
|
||||||
|
acc[d] = 0.0f;
|
||||||
|
}
|
||||||
|
|
||||||
|
float M = -3.402823466e+38f;
|
||||||
|
float S = 0.0f;
|
||||||
|
|
||||||
|
for (int64_t ik1 = 0; ik1 < nk; ++ik1) {
|
||||||
|
// If mask is present, check for -inf (skip masked positions)
|
||||||
|
float mask_val = 0.0f;
|
||||||
|
if (has_mask) {
|
||||||
|
mask_val = get_mask_val(mask, iq1, ik1, iq2, iq3);
|
||||||
|
// llama.cpp uses -inf for masked positions
|
||||||
|
if (mask_val == -3.402823466e+38f || mask_val != mask_val) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const char * pk = k_data + ik1 * k->nb[1] + kv_base;
|
||||||
|
const char * pv = v_data + ik1 * v->nb[1] + vv_base;
|
||||||
|
|
||||||
|
float s = dot_qk(pq, pk, dk, k_nb0, k_type) * scale + mask_val;
|
||||||
|
const float Mold = M;
|
||||||
|
|
||||||
|
float ms = 1.0f;
|
||||||
|
float vs = 1.0f;
|
||||||
|
if (s > M) {
|
||||||
|
M = s;
|
||||||
|
ms = et_expf(Mold - M);
|
||||||
|
for (int64_t d = 0; d < dv; ++d) {
|
||||||
|
acc[d] *= ms;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
vs = et_expf(s - M);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Accumulate weighted V
|
||||||
|
if (v_type == GGML_TYPE_F32) {
|
||||||
|
const float * pvf = (const float *) pv;
|
||||||
|
for (int64_t d = 0; d < dv; ++d) {
|
||||||
|
acc[d] += pvf[d] * vs;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
for (int64_t d = 0; d < dv; ++d) {
|
||||||
|
acc[d] += fp16_to_fp32(*(const uint16_t *) (pv + d * v_nb0)) * vs;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
S = S * ms + vs;
|
||||||
|
}
|
||||||
|
|
||||||
|
const float S_inv = S == 0.0f ? 0.0f : et_fdiv(1.0f, S);
|
||||||
|
if (use_fast_store) {
|
||||||
|
for (int64_t d = 0; d < dv; ++d) {
|
||||||
|
out[d] = acc[d] * S_inv;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
for (int64_t d = 0; d < dv; ++d) {
|
||||||
|
atomic_store_f32((volatile float *) &out[d], acc[d] * S_inv);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,346 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// Gated Delta Net F32 Kernel
|
||||||
|
//
|
||||||
|
// Implements the gated delta rule recurrence:
|
||||||
|
// For each head h, timestep t:
|
||||||
|
// 1. Gate decay: S *= exp(g) (scalar or per-element KDA)
|
||||||
|
// 2. Delta update: delta[j] = (v[j] - dot(S_row_j, k)) * beta
|
||||||
|
// 3. Outer product: S_row_j += k * delta[j]
|
||||||
|
// 4. Attention: attn[j] = dot(S_row_j, q) * scale
|
||||||
|
//
|
||||||
|
// State is stored transposed: s_out[j*S_v + i] = S[i][j]
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "math_fp.h"
|
||||||
|
#include "platform.h"
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
struct ggml_et_gated_delta_net_params {
|
||||||
|
struct ggml_tensor q; // [S_v, H_q, n_tokens, n_seqs_q]
|
||||||
|
struct ggml_tensor k; // [S_v, H_k, n_tokens, n_seqs_k]
|
||||||
|
struct ggml_tensor v; // [S_v, H, n_tokens, n_seqs]
|
||||||
|
struct ggml_tensor g; // [1 or S_v, H, n_tokens, n_seqs]
|
||||||
|
struct ggml_tensor beta; // [1, H, n_tokens, n_seqs]
|
||||||
|
struct ggml_tensor state_in; // [S_v*S_v*H, K, n_seqs]
|
||||||
|
struct ggml_tensor dst; // [S_v*H, n_tokens*n_seqs + S_v*n_seqs*K]
|
||||||
|
int32_t S_v; // head dimension
|
||||||
|
int32_t H; // number of value heads
|
||||||
|
int32_t H_q; // number of Q heads
|
||||||
|
int32_t H_k; // number of K heads
|
||||||
|
int32_t n_tokens; // total tokens
|
||||||
|
int32_t n_seqs; // number of sequences
|
||||||
|
int32_t n_seqs_q; // Q sequence count
|
||||||
|
int32_t n_seqs_k; // K sequence count
|
||||||
|
int32_t kda; // 1 if per-element gate, 0 if scalar
|
||||||
|
int32_t K; // snapshot slot count
|
||||||
|
float scale; // 1/sqrt(S_v)
|
||||||
|
};
|
||||||
|
|
||||||
|
static inline float hsum_f10(void) {
|
||||||
|
float result;
|
||||||
|
__asm__ __volatile__(
|
||||||
|
"fswizz.ps f1, f10, 0xB1 \n\t"
|
||||||
|
"fadd.ps f2, f10, f1, rne \n\t"
|
||||||
|
"fswizz.ps f3, f2, 0x4E \n\t"
|
||||||
|
"fadd.ps f4, f2, f3, rne \n\t"
|
||||||
|
"fmvz.x.ps t0, f4, 4 \n\t"
|
||||||
|
"fbcx.ps f5, t0 \n\t"
|
||||||
|
"fadd.ps %[vout], f4, f5, rne \n\t"
|
||||||
|
: [vout] "=f"(result)::"t0", "f1", "f2", "f3", "f4", "f5");
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_gated_delta_net_params * params, void * env) {
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
|
||||||
|
if (!kernel_env) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
int num_threads = get_num_threads(kernel_env->shire_mask);
|
||||||
|
|
||||||
|
if (thread_id < 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (params == 0 || ((uint64_t) params & 0x7) != 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const struct ggml_tensor * q_tsr = ¶ms->q;
|
||||||
|
const struct ggml_tensor * k_tsr = ¶ms->k;
|
||||||
|
const struct ggml_tensor * v_tsr = ¶ms->v;
|
||||||
|
const struct ggml_tensor * g_tsr = ¶ms->g;
|
||||||
|
const struct ggml_tensor * beta_tsr = ¶ms->beta;
|
||||||
|
const struct ggml_tensor * state_tsr = ¶ms->state_in;
|
||||||
|
const struct ggml_tensor * dst_tsr = ¶ms->dst;
|
||||||
|
|
||||||
|
const float * q = (const float *) q_tsr->data;
|
||||||
|
const float * k = (const float *) k_tsr->data;
|
||||||
|
const float * v = (const float *) v_tsr->data;
|
||||||
|
const float * g = (const float *) g_tsr->data;
|
||||||
|
const float * beta = (const float *) beta_tsr->data;
|
||||||
|
const float * state_in = (const float *) state_tsr->data;
|
||||||
|
float * dst_data = (float *) dst_tsr->data;
|
||||||
|
|
||||||
|
const int32_t S_v = params->S_v;
|
||||||
|
const int32_t H = params->H;
|
||||||
|
const int32_t H_q = params->H_q;
|
||||||
|
const int32_t H_k = params->H_k;
|
||||||
|
const int32_t n_tokens = params->n_tokens;
|
||||||
|
const int32_t n_seqs = params->n_seqs;
|
||||||
|
const int32_t n_seqs_q = params->n_seqs_q;
|
||||||
|
const int32_t n_seqs_k = params->n_seqs_k;
|
||||||
|
const int32_t kda = params->kda;
|
||||||
|
const int32_t K = params->K;
|
||||||
|
const float scale = params->scale;
|
||||||
|
|
||||||
|
if (!q || !k || !v || !g || !beta || !state_in || !dst_data) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Preserve the original contract for every tensor except q, k, and v, which may be
|
||||||
|
// row-contiguous with strided higher dimensions.
|
||||||
|
if (q_tsr->nb[0] != sizeof(float) || k_tsr->nb[0] != sizeof(float) || v_tsr->nb[0] != sizeof(float) ||
|
||||||
|
g_tsr->nb[0] != sizeof(float) || beta_tsr->nb[0] != sizeof(float) || state_tsr->nb[0] != sizeof(float) ||
|
||||||
|
dst_tsr->nb[0] != sizeof(float)) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int32_t attn_elems = S_v * H * n_tokens * n_seqs;
|
||||||
|
float * attn_out_base = dst_data;
|
||||||
|
float * state_out_base = dst_data + attn_elems;
|
||||||
|
|
||||||
|
const int32_t state_plane_floats = S_v * S_v * H * n_seqs;
|
||||||
|
|
||||||
|
const int32_t G0 = kda ? S_v : 1;
|
||||||
|
|
||||||
|
const size_t q_nb1 = q_tsr->nb[1];
|
||||||
|
const size_t q_nb2 = q_tsr->nb[2];
|
||||||
|
const size_t q_nb3 = q_tsr->nb[3];
|
||||||
|
const size_t k_nb1 = k_tsr->nb[1];
|
||||||
|
const size_t k_nb2 = k_tsr->nb[2];
|
||||||
|
const size_t k_nb3 = k_tsr->nb[3];
|
||||||
|
const size_t v_nb1 = v_tsr->nb[1];
|
||||||
|
const size_t v_nb2 = v_tsr->nb[2];
|
||||||
|
const size_t v_nb3 = v_tsr->nb[3];
|
||||||
|
const int32_t g_stride_h = G0;
|
||||||
|
const int32_t g_stride_t = G0 * H;
|
||||||
|
const int32_t g_stride_s = G0 * H * n_tokens;
|
||||||
|
const int32_t b_stride_t = H;
|
||||||
|
const int32_t b_stride_s = H * n_tokens;
|
||||||
|
|
||||||
|
float exp_g_buf[128];
|
||||||
|
|
||||||
|
// FP and SIMD share the same register file. Scalar FP needs the default
|
||||||
|
// mask; 8-wide .ps blocks need m0=255. Save once, toggle at boundaries.
|
||||||
|
unsigned long default_mask;
|
||||||
|
__asm__ volatile("mova.x.m %[ms]\n" : [ms] "=r"(default_mask));
|
||||||
|
|
||||||
|
// Parallelize over (j_block, head, seq). J_BLK must satisfy two separate
|
||||||
|
// cache-line alignment constraints at once:
|
||||||
|
// (a) State: J_BLK consecutive rows of s_out (each S_v floats) span an
|
||||||
|
// integer number of cache lines. For S_v * sizeof(float) >= 64 this
|
||||||
|
// is trivially any J_BLK >= 1.
|
||||||
|
// (b) Attention output: each j writes exactly one float into
|
||||||
|
// attn_ptr[j], which is densely packed. If J_BLK * sizeof(float) is
|
||||||
|
// less than a cache line, distinct threads will share a line and
|
||||||
|
// race on scalar stores — ET's L1 isn't coherent so we lose writes.
|
||||||
|
//
|
||||||
|
// (b) dominates: J_BLK must be at least ET_CACHE_LINE_SIZE_BYTES / 4 so
|
||||||
|
// that each thread owns a whole cache line of attn_ptr. That's 16 on
|
||||||
|
// ET-SoC-1, and it's also a whole number of state rows for every
|
||||||
|
// S_v >= 1, so (a) is automatically satisfied.
|
||||||
|
const int32_t J_BLK = ET_CACHE_LINE_SIZE_BYTES / (int32_t) sizeof(float);
|
||||||
|
const int32_t n_j_blocks = (S_v + J_BLK - 1) / J_BLK;
|
||||||
|
const int32_t total_work = n_j_blocks * H * n_seqs;
|
||||||
|
|
||||||
|
for (int32_t ir = thread_id; ir < total_work; ir += num_threads) {
|
||||||
|
const int32_t jb = ir % n_j_blocks;
|
||||||
|
const int32_t head = (ir / n_j_blocks) % H;
|
||||||
|
const int32_t seq = ir / (n_j_blocks * H);
|
||||||
|
|
||||||
|
const int32_t j_start = jb * J_BLK;
|
||||||
|
const int32_t j_end = (j_start + J_BLK < S_v) ? j_start + J_BLK : S_v;
|
||||||
|
|
||||||
|
const int32_t h_q = head % H_q;
|
||||||
|
const int32_t h_k = head % H_k;
|
||||||
|
const int32_t seq_q = (n_seqs_q == n_seqs) ? seq : (seq * n_seqs_q / n_seqs);
|
||||||
|
const int32_t seq_k = (n_seqs_k == n_seqs) ? seq : (seq * n_seqs_k / n_seqs);
|
||||||
|
|
||||||
|
const int32_t head_state_off = (seq * H + head) * S_v * S_v;
|
||||||
|
// Live RMW buffer = first snapshot plane (slot 0).
|
||||||
|
float * s_out = state_out_base + head_state_off;
|
||||||
|
// Input state: seq `seq`, head `head`.
|
||||||
|
const float * s_in = state_in + head_state_off;
|
||||||
|
|
||||||
|
// Skip the explicit s_in -> s_out copy. At t=0 pass A/B read through
|
||||||
|
// src_state = s_in; pass B writes the first new row to s_out. From
|
||||||
|
// t=1 onward src_state flips to s_out (read-modify-write in place).
|
||||||
|
const float * src_state = s_in;
|
||||||
|
|
||||||
|
const int32_t attn_stride_t = S_v * H;
|
||||||
|
float * attn_ptr = attn_out_base + (seq * n_tokens * H + head) * S_v;
|
||||||
|
|
||||||
|
const float zero = 0.0f;
|
||||||
|
|
||||||
|
for (int32_t t = 0; t < n_tokens; t++) {
|
||||||
|
const float * q_t = (const float *) ((const char *) q + seq_q * q_nb3 + t * q_nb2 + h_q * q_nb1);
|
||||||
|
const float * k_t = (const float *) ((const char *) k + seq_k * k_nb3 + t * k_nb2 + h_k * k_nb1);
|
||||||
|
const float * v_t = (const float *) ((const char *) v + seq * v_nb3 + t * v_nb2 + head * v_nb1);
|
||||||
|
const float * g_t = g + seq * g_stride_s + t * g_stride_t + head * g_stride_h;
|
||||||
|
const float beta_val = beta[seq * b_stride_s + t * b_stride_t + head];
|
||||||
|
|
||||||
|
// Precompute per-element gate for the kda path; scalar decay
|
||||||
|
// otherwise. Decay is fused into per-j pass A/B below, not
|
||||||
|
// applied to state in a separate pre-pass.
|
||||||
|
float decay = 0.0f; // only used when !kda
|
||||||
|
if (kda) {
|
||||||
|
const float log2e = 1.4426950408889634f;
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 255\n" :::);
|
||||||
|
__asm__ volatile("fbc.ps f20, %[l2e]\n" : : [l2e] "m"(log2e) : "f20");
|
||||||
|
for (int32_t i = 0; i < S_v; i += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f10, %[g_vec]\n"
|
||||||
|
"fmul.ps f10, f10, f20, rne\n"
|
||||||
|
"fexp.ps f10, f10\n"
|
||||||
|
"fsw.ps f10, %[out]\n"
|
||||||
|
: [out] "=m"(*(float (*)[8]) & exp_g_buf[i])
|
||||||
|
: [g_vec] "m"(*(const float (*)[8]) & g_t[i])
|
||||||
|
: "f10");
|
||||||
|
}
|
||||||
|
__asm__ volatile("mova.m.x %[ms]\n" : : [ms] "r"(default_mask));
|
||||||
|
} else {
|
||||||
|
decay = et_expf(g_t[0]);
|
||||||
|
}
|
||||||
|
|
||||||
|
for (int32_t j = j_start; j < j_end; j++) {
|
||||||
|
const float * src_row = src_state + j * S_v;
|
||||||
|
float * dst_row = s_out + j * S_v;
|
||||||
|
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 255\n" :::);
|
||||||
|
if (kda) {
|
||||||
|
__asm__ volatile("fbc.ps f10, %[z]\n" : : [z] "m"(zero) : "f10");
|
||||||
|
for (int32_t i = 0; i < S_v; i += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f11, %[s_vec]\n"
|
||||||
|
"flw.ps f12, %[g_vec]\n"
|
||||||
|
"flw.ps f13, %[k_vec]\n"
|
||||||
|
"fmul.ps f11, f11, f12\n" // row_dec = row * g
|
||||||
|
"fmadd.ps f10, f11, f13, f10\n" // acc += row_dec * k
|
||||||
|
:
|
||||||
|
: [s_vec] "m"(*(const float (*)[8]) & src_row[i]),
|
||||||
|
[g_vec] "m"(*(const float (*)[8]) & exp_g_buf[i]),
|
||||||
|
[k_vec] "m"(*(const float (*)[8]) & k_t[i])
|
||||||
|
: "f10", "f11", "f12", "f13");
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbc.ps f10, %[z]\n"
|
||||||
|
"fbc.ps f22, %[d]\n"
|
||||||
|
:
|
||||||
|
: [z] "m"(zero), [d] "m"(decay)
|
||||||
|
: "f10", "f22");
|
||||||
|
for (int32_t i = 0; i < S_v; i += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f11, %[s_vec]\n"
|
||||||
|
"flw.ps f13, %[k_vec]\n"
|
||||||
|
"fmul.ps f11, f11, f22\n" // row_dec = row * decay
|
||||||
|
"fmadd.ps f10, f11, f13, f10\n" // acc += row_dec * k
|
||||||
|
:
|
||||||
|
: [s_vec] "m"(*(const float (*)[8]) & src_row[i]), [k_vec] "m"(*(const float (*)[8]) &
|
||||||
|
k_t[i])
|
||||||
|
: "f10", "f11", "f13");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
float dot_sk = hsum_f10();
|
||||||
|
__asm__ volatile("mova.m.x %[ms]\n" : : [ms] "r"(default_mask));
|
||||||
|
|
||||||
|
float delta_j = (v_t[j] - dot_sk) * beta_val;
|
||||||
|
|
||||||
|
// -------- Pass B: decay + outer product + attn --------
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 255\n" :::);
|
||||||
|
if (kda) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbc.ps f10, %[z]\n"
|
||||||
|
"fbc.ps f21, %[dj]\n"
|
||||||
|
:
|
||||||
|
: [z] "m"(zero), [dj] "m"(delta_j)
|
||||||
|
: "f10", "f21");
|
||||||
|
for (int32_t i = 0; i < S_v; i += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f11, %[s_vec]\n"
|
||||||
|
"flw.ps f12, %[g_vec]\n"
|
||||||
|
"flw.ps f13, %[k_vec]\n"
|
||||||
|
"flw.ps f14, %[q_vec]\n"
|
||||||
|
"fmul.ps f11, f11, f12\n" // row_dec = row * g
|
||||||
|
"fmadd.ps f11, f13, f21, f11\n" // row_new = row_dec + k*delta_j
|
||||||
|
"fsw.ps f11, %[s_out]\n"
|
||||||
|
"fmadd.ps f10, f11, f14, f10\n" // attn_acc += row_new * q
|
||||||
|
: [s_out] "=m"(*(float (*)[8]) & dst_row[i])
|
||||||
|
: [s_vec] "m"(*(const float (*)[8]) & src_row[i]),
|
||||||
|
[g_vec] "m"(*(const float (*)[8]) & exp_g_buf[i]),
|
||||||
|
[k_vec] "m"(*(const float (*)[8]) & k_t[i]), [q_vec] "m"(*(const float (*)[8]) & q_t[i])
|
||||||
|
: "f10", "f11", "f12", "f13", "f14");
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbc.ps f10, %[z]\n"
|
||||||
|
"fbc.ps f21, %[dj]\n"
|
||||||
|
"fbc.ps f22, %[d]\n"
|
||||||
|
:
|
||||||
|
: [z] "m"(zero), [dj] "m"(delta_j), [d] "m"(decay)
|
||||||
|
: "f10", "f21", "f22");
|
||||||
|
for (int32_t i = 0; i < S_v; i += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f11, %[s_vec]\n"
|
||||||
|
"flw.ps f13, %[k_vec]\n"
|
||||||
|
"flw.ps f14, %[q_vec]\n"
|
||||||
|
"fmul.ps f11, f11, f22\n" // row_dec = row * decay
|
||||||
|
"fmadd.ps f11, f13, f21, f11\n" // row_new = row_dec + k*delta_j
|
||||||
|
"fsw.ps f11, %[s_out]\n"
|
||||||
|
"fmadd.ps f10, f11, f14, f10\n" // attn_acc += row_new * q
|
||||||
|
: [s_out] "=m"(*(float (*)[8]) & dst_row[i])
|
||||||
|
: [s_vec] "m"(*(const float (*)[8]) & src_row[i]),
|
||||||
|
[k_vec] "m"(*(const float (*)[8]) & k_t[i]), [q_vec] "m"(*(const float (*)[8]) & q_t[i])
|
||||||
|
: "f10", "f11", "f13", "f14");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
float attn_val = hsum_f10();
|
||||||
|
__asm__ volatile("mova.m.x %[ms]\n" : : [ms] "r"(default_mask));
|
||||||
|
|
||||||
|
attn_ptr[j] = attn_val * scale;
|
||||||
|
}
|
||||||
|
|
||||||
|
// n-way merge snapshot: live state lives in slot 0 (== s_out).
|
||||||
|
// Copies state to target snapshot slots [1, K-1] in reverse chronological order.
|
||||||
|
// target_slot == 0 is the live buffer itself => no copy.
|
||||||
|
// target_slot >= K (when n_tokens > K) => older slots are discarded.
|
||||||
|
if (K > 1) {
|
||||||
|
const int32_t target_slot = (n_tokens - 1) - t;
|
||||||
|
if (target_slot > 0 && target_slot < K) {
|
||||||
|
float * snap = state_out_base + target_slot * state_plane_floats + head_state_off;
|
||||||
|
for (int32_t j = j_start; j < j_end; j++) {
|
||||||
|
const float * src = s_out + j * S_v;
|
||||||
|
float * dst = snap + j * S_v;
|
||||||
|
for (int32_t i = 0; i < S_v; i++) {
|
||||||
|
dst[i] = src[i];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// After t=0, state lives in s_out; flip src_state so subsequent
|
||||||
|
// timesteps read-modify-write in place.
|
||||||
|
src_state = s_out;
|
||||||
|
attn_ptr += attn_stride_t;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,612 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// Bare Metal GET_ROWS F32 Kernel
|
||||||
|
// Extracts specific rows from a source tensor based on row indices
|
||||||
|
//
|
||||||
|
// Algorithm:
|
||||||
|
// 1. Read row indices from src1 (int32 tensor)
|
||||||
|
// 2. For each index, extract the corresponding row from src0
|
||||||
|
// 3. Copy the row data to the output tensor dst
|
||||||
|
// 4. Handle different input types: F32, Q8_0, Q4_0, and Q4_K (quantized)
|
||||||
|
//
|
||||||
|
// Operation: dst[i] = src0[indices[i]] for i = 0..num_indices
|
||||||
|
//
|
||||||
|
// Features supported:
|
||||||
|
// - F32 input data (direct copy)
|
||||||
|
// - Q4_0 quantized input data (dequantized to F32)
|
||||||
|
// - Q8_0 quantized input data (dequantized to F32)
|
||||||
|
// - Q4_K quantized input data (dequantized to F32)
|
||||||
|
// - Int32 row indices
|
||||||
|
// - Multi-dimensional tensor support
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "platform.h"
|
||||||
|
#include "quants.h"
|
||||||
|
|
||||||
|
#include <assert.h>
|
||||||
|
#include <stdbool.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
#define CACHE_LINE_SIZE_BYTES 64
|
||||||
|
|
||||||
|
struct ggml_et_get_rows_params {
|
||||||
|
struct ggml_tensor src0; // Data tensor (F32, Q4_0, Q8_0, or Q4_K)
|
||||||
|
struct ggml_tensor src1; // Row indices tensor (I32)
|
||||||
|
struct ggml_tensor dst; // Output tensor (F32)
|
||||||
|
};
|
||||||
|
|
||||||
|
#define CACHE_LINE_SIZE_BYTES 64
|
||||||
|
#define CACHE_ELEMENTS(elem_size) (CACHE_LINE_SIZE_BYTES / (elem_size))
|
||||||
|
|
||||||
|
// Copy a row of F32 data from source to destination
|
||||||
|
static void copy_f32_row(float * dst, const float * src, int64_t num_elements) {
|
||||||
|
// Simple memcpy for F32 data - no conversion needed
|
||||||
|
for (int64_t i = 0; i < num_elements; i++) {
|
||||||
|
dst[i] = src[i];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static void copy_f16_row(float * dst, const uint16_t * src, int64_t num_elements) {
|
||||||
|
for (int64_t i = 0; i < num_elements; i++) {
|
||||||
|
dst[i] = fp16_to_fp32(src[i]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Copy a row of F32 data from source to destination, aligned to cache line boundaries
|
||||||
|
// using FP32 load/store instructions. They don't perform data conversion so is fine.
|
||||||
|
// Requirement: n_bytes is a multiple of CACHE_LINE_SIZE (64 bytes)
|
||||||
|
static void copy_row_cache_align(float * dst, const float * src, int64_t n_bytes) {
|
||||||
|
int num_f32_elem = n_bytes / sizeof(float);
|
||||||
|
|
||||||
|
// Unrolled to do an entire cache line at a time
|
||||||
|
__asm__ volatile(
|
||||||
|
"1: \n\t"
|
||||||
|
// --- Process 64 Bytes (1 Cache Line) ---
|
||||||
|
// Load 256 bits (32 bytes) into f0 and the other into f1
|
||||||
|
"flq2 f0, 0(%[src]) \n\t"
|
||||||
|
"flq2 f1, 32(%[src]) \n\t"
|
||||||
|
|
||||||
|
// Store 256 bits (32 bytes) from f0 and f1
|
||||||
|
"fsq2 f0, 0(%[dst]) \n\t"
|
||||||
|
"fsq2 f1, 32(%[dst]) \n\t"
|
||||||
|
|
||||||
|
// Increment Pointers by 64 bytes
|
||||||
|
"addi %[src], %[src], 64 \n\t"
|
||||||
|
"addi %[dst], %[dst], 64 \n\t"
|
||||||
|
|
||||||
|
// Decrement count by 16 elements
|
||||||
|
"addi %[n], %[n], -16 \n\t"
|
||||||
|
|
||||||
|
// Loop if at least 16 elements remain
|
||||||
|
"bge %[n], %[stride_count], 1b \n\t"
|
||||||
|
|
||||||
|
: [dst] "+r"(dst), [src] "+r"(src), [n] "+r"(num_f32_elem)
|
||||||
|
: [stride_count] "r"(16L)
|
||||||
|
: "f0", "f1", "memory");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Copied from GGML: copy a row of Q4_0 data to F32 destination (with dequantization)
|
||||||
|
static void copy_q4_0_row(float * dst, const block_q4_0 * src_blocks, int64_t num_elements) {
|
||||||
|
const int64_t num_blocks = (num_elements + QK4_0 - 1) / QK4_0;
|
||||||
|
|
||||||
|
for (int64_t block_idx = 0; block_idx < num_blocks; block_idx++) {
|
||||||
|
const int64_t elements_in_block = (block_idx == num_blocks - 1) ? (num_elements - block_idx * QK4_0) : QK4_0;
|
||||||
|
|
||||||
|
float temp_buffer[QK4_0];
|
||||||
|
dequantize_q4_0_block(&src_blocks[block_idx], temp_buffer);
|
||||||
|
|
||||||
|
for (int64_t i = 0; i < elements_in_block; i++) {
|
||||||
|
dst[block_idx * QK4_0 + i] = temp_buffer[i];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Copy a row of Q8_0 data to F32 destination (with dequantization)
|
||||||
|
static void copy_q8_0_row(float * dst, const block_q8_0 * src_blocks, int64_t num_elements) {
|
||||||
|
// Number of Q8_0 blocks needed for this row
|
||||||
|
const int64_t num_blocks = (num_elements + QK8_0 - 1) / QK8_0; // Round up to handle partial blocks
|
||||||
|
|
||||||
|
for (int64_t block_idx = 0; block_idx < num_blocks; block_idx++) {
|
||||||
|
const int64_t elements_in_block =
|
||||||
|
(block_idx == num_blocks - 1) ? (num_elements - block_idx * QK8_0) : QK8_0; // Handle last partial block
|
||||||
|
|
||||||
|
// Dequantize the block
|
||||||
|
float temp_buffer[QK8_0];
|
||||||
|
dequantize_q8_0_block(&src_blocks[block_idx], temp_buffer);
|
||||||
|
|
||||||
|
// Copy dequantized values to destination
|
||||||
|
for (int64_t i = 0; i < elements_in_block; i++) {
|
||||||
|
dst[block_idx * QK8_0 + i] = temp_buffer[i];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Copy a row of Q4_K data to F32 destination (with dequantization)
|
||||||
|
static void copy_q4_K_row(float * dst, const block_q4_K * src_blocks, int64_t num_elements) {
|
||||||
|
const int64_t num_blocks = (num_elements + QK_K - 1) / QK_K;
|
||||||
|
|
||||||
|
for (int64_t block_idx = 0; block_idx < num_blocks; block_idx++) {
|
||||||
|
const int64_t elements_in_block = (block_idx == num_blocks - 1) ? (num_elements - block_idx * QK_K) : QK_K;
|
||||||
|
|
||||||
|
float temp_buffer[QK_K];
|
||||||
|
dequantize_q4_K_block(&src_blocks[block_idx], temp_buffer);
|
||||||
|
|
||||||
|
for (int64_t i = 0; i < elements_in_block; i++) {
|
||||||
|
dst[block_idx * QK_K + i] = temp_buffer[i];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static void dequantize_q8_0_block_cache_aligned(const block_q8_0 * block, float * dst) {
|
||||||
|
const int8_t * qs_ptr = block->qs;
|
||||||
|
|
||||||
|
uint64_t temp_mask;
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(temp_mask)); // Save current mask
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF"); // Enable all 8 elements
|
||||||
|
|
||||||
|
const int32_t __attribute__((aligned(32))) vec_indices[8] = { 0, 1, 2, 3, 4, 5, 6, 7 };
|
||||||
|
float scale = fp16_to_fp32(block->d);
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbcx.ps f0, %0 \n\t" // Broadcast integer scale to all lanes
|
||||||
|
"flq2 f1, 0(%1) \n\t" // Load gether indicies
|
||||||
|
::"r"(scale),
|
||||||
|
"r"(vec_indices)
|
||||||
|
: "f0", "f1");
|
||||||
|
|
||||||
|
for (int i = 0; i < 4; i++) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"fgb.ps f2, f1(%0) \n\t" // Loads 8 bytes from (qs_ptr + indices) and sign-extends to 32-bit int.
|
||||||
|
"fcvt.ps.pw f2, f2, rne \n\t" // Convert Int32 to Float32
|
||||||
|
"fmul.ps f2, f2, f0 \n\t" // f2 = f2 * f0 (scale)
|
||||||
|
"fsq2 f2, 0(%1) \n\t" // Store 256 bits (8 floats) to dst.
|
||||||
|
|
||||||
|
::"r"(qs_ptr),
|
||||||
|
"r"(dst)
|
||||||
|
: "f2", "memory");
|
||||||
|
|
||||||
|
// Advance pointers in C
|
||||||
|
qs_ptr += 8;
|
||||||
|
dst += 8;
|
||||||
|
}
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(temp_mask));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Copy a row of Q4_0 data to F32 destination (with dequantization), cache-aligned
|
||||||
|
static void copy_q4_0_row_cache_aligned(float * dst, const block_q4_0 * src_blocks, int64_t num_elements) {
|
||||||
|
const int64_t num_blocks = (num_elements + QK4_0 - 1) / QK4_0;
|
||||||
|
|
||||||
|
// Scatter byte offsets: even lanes -> dst[j], odd lanes -> dst[j + QK4_0/2]
|
||||||
|
// For 4 consecutive packed bytes producing [low0, high0, low1, high1, low2, high2, low3, high3]:
|
||||||
|
// low_i -> byte offset i*4 (positions 0,1,2,3 in first half)
|
||||||
|
// high_i -> byte offset (16+i)*4 (positions 16,17,18,19 in second half)
|
||||||
|
const int32_t __attribute__((aligned(32))) scatter_offsets[8] = { 0 * 4, 16 * 4, 1 * 4, 17 * 4,
|
||||||
|
2 * 4, 18 * 4, 3 * 4, 19 * 4 };
|
||||||
|
|
||||||
|
// Gather indices: each byte loaded twice for low/high nibble extraction
|
||||||
|
const int32_t __attribute__((aligned(32))) gather_indices[8] = { 0, 0, 1, 1, 2, 2, 3, 3 };
|
||||||
|
|
||||||
|
uint64_t temp_mask;
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(temp_mask)); // Save current mask
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF"); // Enable all 8 elements
|
||||||
|
|
||||||
|
// Load constant vectors once — shared across all blocks and iterations
|
||||||
|
__asm__ volatile(
|
||||||
|
"flq2 f4, 0(%0) \n\t" // f4 = scatter offsets
|
||||||
|
"flq2 f1, 0(%1) \n\t" // f1 = gather indices {0,0,1,1,2,2,3,3}
|
||||||
|
::"r"(scatter_offsets),
|
||||||
|
"r"(gather_indices)
|
||||||
|
: "f1", "f4");
|
||||||
|
|
||||||
|
for (int64_t block_idx = 0; block_idx < num_blocks; block_idx++) {
|
||||||
|
const block_q4_0 * block = &src_blocks[block_idx];
|
||||||
|
const uint8_t * qs = block->qs;
|
||||||
|
float * block_dst = dst + block_idx * QK4_0;
|
||||||
|
|
||||||
|
float scale = fp16_to_fp32(block->d);
|
||||||
|
float bias = -8.0f * scale;
|
||||||
|
|
||||||
|
// Per-block: broadcast scale and bias
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbcx.ps f0, %0 \n\t" // f0 = broadcast(scale)
|
||||||
|
"fbcx.ps f3, %1 \n\t" // f3 = broadcast(-8 * scale)
|
||||||
|
::"r"(scale),
|
||||||
|
"r"(bias)
|
||||||
|
: "f0", "f3");
|
||||||
|
|
||||||
|
// 4 iterations x 4 packed bytes = 16 bytes = full block -> 32 floats
|
||||||
|
for (int i = 0; i < 4; i++) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"fgb.ps f2, f1(%0) \n\t" // Gather: [b0,b0,b1,b1,b2,b2,b3,b3]
|
||||||
|
"mov.m.x m0, x0, 0xAA \n\t" // Odd lanes only (fills gather latency)
|
||||||
|
"fsrli.pi f2, f2, 4 \n\t" // Odd lanes: byte >> 4 (high nibble)
|
||||||
|
"mov.m.x m0, x0, 0xFF \n\t" // Restore full mask
|
||||||
|
"fslli.pi f2, f2, 28 \n\t" // Isolate low 4 bits: shift left 28
|
||||||
|
"fsrli.pi f2, f2, 28 \n\t" // then right 28 -> nibble in [3:0]
|
||||||
|
"fcvt.ps.pw f2, f2, rne \n\t" // Int32 -> Float32
|
||||||
|
"fmul.ps f2, f2, f0 \n\t" // * scale
|
||||||
|
"fadd.ps f2, f2, f3 \n\t" // + bias -> (nibble - 8) * scale
|
||||||
|
"fscw.ps f2, f4(%1) \n\t" // Scatter to GGML positions
|
||||||
|
|
||||||
|
::"r"(qs),
|
||||||
|
"r"(block_dst)
|
||||||
|
: "f2", "memory");
|
||||||
|
|
||||||
|
qs += 4; // 4 packed bytes consumed
|
||||||
|
block_dst += 4; // Advance base by 4 float positions
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(temp_mask)); // Restore mask
|
||||||
|
}
|
||||||
|
|
||||||
|
// Copy a row of Q8_0 data to F32 destination (with dequantization)
|
||||||
|
static void copy_q8_0_row_cache_aligned(float * dst, const block_q8_0 * src_blocks, int64_t num_elements) {
|
||||||
|
// Number of Q8_0 blocks needed for this row
|
||||||
|
const int64_t num_blocks = (num_elements + QK8_0 - 1) / QK8_0; // Round up to handle partial blocks
|
||||||
|
|
||||||
|
for (int64_t block_idx = 0; block_idx < num_blocks; block_idx++) {
|
||||||
|
const int64_t elements_in_block =
|
||||||
|
(block_idx == num_blocks - 1) ? (num_elements - block_idx * QK8_0) : QK8_0; // Handle last partial block
|
||||||
|
|
||||||
|
// Dequantize the block
|
||||||
|
float temp_buffer[QK8_0];
|
||||||
|
dequantize_q8_0_block_cache_aligned(&src_blocks[block_idx], temp_buffer);
|
||||||
|
|
||||||
|
// Copy dequantized values to destination
|
||||||
|
for (int64_t i = 0; i < elements_in_block; i++) {
|
||||||
|
dst[block_idx * QK8_0 + i] = temp_buffer[i];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Vectorized dequantization of a Q4_K super-block (256 elements) to F32
|
||||||
|
// Processes 8 groups of 32 elements, using ET SIMD for the inner loops.
|
||||||
|
// Output is sequential (no scatter needed unlike Q4_0).
|
||||||
|
static void copy_q4_K_row_cache_aligned(float * dst, const block_q4_K * src_blocks, int64_t num_elements) {
|
||||||
|
const int64_t num_blocks = (num_elements + QK_K - 1) / QK_K;
|
||||||
|
|
||||||
|
// Gather indices for sequential byte access: {0,1,2,3,4,5,6,7}
|
||||||
|
const int32_t __attribute__((aligned(32))) gather_indices[8] = { 0, 1, 2, 3, 4, 5, 6, 7 };
|
||||||
|
|
||||||
|
uint64_t temp_mask;
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(temp_mask)); // Save current mask
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF"); // Enable all 8 elements
|
||||||
|
|
||||||
|
// Load gather indices once — shared across all blocks
|
||||||
|
__asm__ volatile("flq2 f1, 0(%0) \n\t" // f1 = gather indices {0,1,2,3,4,5,6,7}
|
||||||
|
::"r"(gather_indices)
|
||||||
|
: "f1");
|
||||||
|
|
||||||
|
for (int64_t block_idx = 0; block_idx < num_blocks; block_idx++) {
|
||||||
|
const block_q4_K * block = &src_blocks[block_idx];
|
||||||
|
const uint8_t * qs = block->qs;
|
||||||
|
float * block_dst = dst + block_idx * QK_K;
|
||||||
|
|
||||||
|
const float d = fp16_to_fp32(block->d);
|
||||||
|
const float min = fp16_to_fp32(block->dmin);
|
||||||
|
|
||||||
|
int is = 0;
|
||||||
|
for (int j = 0; j < QK_K; j += 64) {
|
||||||
|
// Extract per-group scales and mins (scalar — only 8 pairs per super-block)
|
||||||
|
uint8_t sc, m;
|
||||||
|
get_scale_min_k4(is + 0, block->scales, &sc, &m);
|
||||||
|
const float d1 = d * sc;
|
||||||
|
const float neg_m1 = -(min * m);
|
||||||
|
get_scale_min_k4(is + 1, block->scales, &sc, &m);
|
||||||
|
const float d2 = d * sc;
|
||||||
|
const float neg_m2 = -(min * m);
|
||||||
|
|
||||||
|
// Low nibbles: 32 elements using d1, neg_m1
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbcx.ps f0, %0 \n\t" // f0 = broadcast(d1)
|
||||||
|
"fbcx.ps f3, %1 \n\t" // f3 = broadcast(-m1)
|
||||||
|
::"r"(d1),
|
||||||
|
"r"(neg_m1)
|
||||||
|
: "f0", "f3");
|
||||||
|
|
||||||
|
const uint8_t * qs_lo = qs;
|
||||||
|
float * dst_lo = block_dst + j;
|
||||||
|
for (int k = 0; k < 4; k++) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"fgb.ps f2, f1(%0) \n\t" // Gather 8 bytes, sign-extend to int32
|
||||||
|
"fandi.pi f2, f2, 0xF \n\t" // Mask low nibble (imm10=15)
|
||||||
|
"fcvt.ps.pw f2, f2, rne \n\t" // Int32 -> Float32
|
||||||
|
"fmadd.ps f2, f2, f0, f3\n\t" // d1 * nibble + (-m1)
|
||||||
|
"fsq2 f2, 0(%1) \n\t" // Store 8 floats
|
||||||
|
::"r"(qs_lo),
|
||||||
|
"r"(dst_lo)
|
||||||
|
: "f2", "memory");
|
||||||
|
qs_lo += 8;
|
||||||
|
dst_lo += 8;
|
||||||
|
}
|
||||||
|
|
||||||
|
// High nibbles: 32 elements using d2, neg_m2
|
||||||
|
__asm__ volatile(
|
||||||
|
"fbcx.ps f0, %0 \n\t" // f0 = broadcast(d2)
|
||||||
|
"fbcx.ps f3, %1 \n\t" // f3 = broadcast(-m2)
|
||||||
|
::"r"(d2),
|
||||||
|
"r"(neg_m2)
|
||||||
|
: "f0", "f3");
|
||||||
|
|
||||||
|
const uint8_t * qs_hi = qs;
|
||||||
|
float * dst_hi = block_dst + j + 32;
|
||||||
|
for (int k = 0; k < 4; k++) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"fgb.ps f2, f1(%0) \n\t" // Gather 8 bytes, sign-extend to int32
|
||||||
|
"fsrli.pi f2, f2, 4 \n\t" // Shift right 4: high nibble
|
||||||
|
"fandi.pi f2, f2, 0xF \n\t" // Mask to 4 bits (clean any sign-ext artifacts)
|
||||||
|
"fcvt.ps.pw f2, f2, rne \n\t" // Int32 -> Float32
|
||||||
|
"fmadd.ps f2, f2, f0, f3\n\t" // d2 * nibble + (-m2)
|
||||||
|
"fsq2 f2, 0(%1) \n\t" // Store 8 floats
|
||||||
|
::"r"(qs_hi),
|
||||||
|
"r"(dst_hi)
|
||||||
|
: "f2", "memory");
|
||||||
|
qs_hi += 8;
|
||||||
|
dst_hi += 8;
|
||||||
|
}
|
||||||
|
|
||||||
|
qs += 32; // Advance to next 32 packed bytes
|
||||||
|
is += 2;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(temp_mask)); // Restore mask
|
||||||
|
}
|
||||||
|
|
||||||
|
// Determine the number of F32 elements per work unit for a given source type.
|
||||||
|
// For F32: 1 cacheline (16 elements)
|
||||||
|
// For quantized types: 1 quant block
|
||||||
|
static int64_t get_elements_per_work_unit(int type) {
|
||||||
|
const int64_t elements_per_cacheline = CACHE_LINE_SIZE_BYTES / sizeof(float); // 16
|
||||||
|
switch (type) {
|
||||||
|
case GGML_TYPE_Q8_0:
|
||||||
|
return QK8_0; // 32 elements = 2 cachelines
|
||||||
|
case GGML_TYPE_Q4_0:
|
||||||
|
return QK4_0; // 32 elements = 2 cachelines
|
||||||
|
case GGML_TYPE_Q4_K:
|
||||||
|
return QK_K; // 256 elements = 16 cachelines
|
||||||
|
default:
|
||||||
|
return elements_per_cacheline; // 16 elements = 1 cacheline
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static int get_row_f32_mc_cacheline_aligned(struct ggml_et_get_rows_params * params, void * env) {
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
int num_threads = get_num_threads(kernel_env->shire_mask);
|
||||||
|
|
||||||
|
struct ggml_tensor * src0 = ¶ms->src0; // Data tensor
|
||||||
|
struct ggml_tensor * src1 = ¶ms->src1; // Row indices tensor (I32)
|
||||||
|
struct ggml_tensor * dst = ¶ms->dst; // Output tensor (F32)
|
||||||
|
|
||||||
|
const int64_t ne00 = src0->ne[0]; // Source columns (row width)
|
||||||
|
const int64_t ne01 = src0->ne[1]; // Source rows (total available rows)
|
||||||
|
const int64_t ne02 = src0->ne[2]; // Source batch dimension
|
||||||
|
const int64_t ne03 = src0->ne[3]; // Source outer batch dimension
|
||||||
|
|
||||||
|
const int64_t ne10 = src1->ne[0]; // Number of indices in dimension 0
|
||||||
|
const int64_t ne11 = src1->ne[1]; // Number of indices in dimension 1
|
||||||
|
const int64_t ne12 = src1->ne[2]; // Batch dimension for indices
|
||||||
|
const int64_t ne13 = src1->ne[3]; // Outer batch dimension for indices
|
||||||
|
|
||||||
|
const int64_t total_rows_to_extract = ne10 * ne11 * ne12 * ne13;
|
||||||
|
|
||||||
|
// Determine work unit size based on source type
|
||||||
|
const int64_t elements_per_wu = get_elements_per_work_unit(src0->type);
|
||||||
|
const int64_t wus_per_row = ne00 / elements_per_wu;
|
||||||
|
const int64_t total_wus = total_rows_to_extract * wus_per_row;
|
||||||
|
|
||||||
|
// Distribute work units across threads (contiguous ranges)
|
||||||
|
const int64_t wus_per_thread = (total_wus + num_threads - 1) / num_threads;
|
||||||
|
const int64_t wu_start = thread_id * wus_per_thread;
|
||||||
|
int64_t wu_end = wu_start + wus_per_thread;
|
||||||
|
if (wu_end > total_wus) {
|
||||||
|
wu_end = total_wus;
|
||||||
|
}
|
||||||
|
|
||||||
|
void * src0_data = src0->data;
|
||||||
|
int32_t * src1_data = (int32_t *) src1->data;
|
||||||
|
float * dst_data = (float *) dst->data;
|
||||||
|
|
||||||
|
int64_t wu = wu_start;
|
||||||
|
while (wu < wu_end) {
|
||||||
|
// Determine which row this work unit belongs to and offset within row
|
||||||
|
const int64_t row_idx = wu / wus_per_row;
|
||||||
|
const int64_t wu_in_row = wu % wus_per_row;
|
||||||
|
|
||||||
|
// How many work units to process in this row (batch contiguous WUs in same row)
|
||||||
|
int64_t wus_remaining_in_row = wus_per_row - wu_in_row;
|
||||||
|
int64_t wus_to_process = wu_end - wu;
|
||||||
|
if (wus_remaining_in_row < wus_to_process) {
|
||||||
|
wus_to_process = wus_remaining_in_row;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Calculate multi-dimensional index for this row
|
||||||
|
const int64_t i = row_idx;
|
||||||
|
const int64_t i13_idx = i / (ne12 * ne11 * ne10);
|
||||||
|
const int64_t i12_idx = (i - i13_idx * ne12 * ne11 * ne10) / (ne11 * ne10);
|
||||||
|
const int64_t i11_idx = (i - i13_idx * ne12 * ne11 * ne10 - i12_idx * ne11 * ne10) / ne10;
|
||||||
|
const int64_t i10_idx = i - i13_idx * ne12 * ne11 * ne10 - i12_idx * ne11 * ne10 - i11_idx * ne10;
|
||||||
|
|
||||||
|
// Get the row index from src1
|
||||||
|
const int64_t index_offset = i13_idx * ne12 * ne11 * ne10 + i12_idx * ne11 * ne10 + i11_idx * ne10 + i10_idx;
|
||||||
|
const int32_t row_index = src1_data[index_offset];
|
||||||
|
|
||||||
|
if (row_index < 0 || row_index >= ne01) {
|
||||||
|
return -1; // Index out of bounds
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t batch_offset =
|
||||||
|
i11_idx * ne01 * ne00 + i12_idx * ne02 * ne01 * ne00 + i13_idx * ne03 * ne02 * ne01 * ne00;
|
||||||
|
|
||||||
|
const int64_t elem_offset_in_row = wu_in_row * elements_per_wu;
|
||||||
|
const int64_t num_elements = wus_to_process * elements_per_wu;
|
||||||
|
|
||||||
|
float * dst_row = dst_data + row_idx * ne00 + elem_offset_in_row;
|
||||||
|
|
||||||
|
if (src0->type == GGML_TYPE_F32) {
|
||||||
|
// F32 source: direct copy of cacheline-aligned chunk
|
||||||
|
const float * src_row = (const float *) src0_data + row_index * ne00 + batch_offset + elem_offset_in_row;
|
||||||
|
copy_row_cache_align(dst_row, src_row, num_elements * sizeof(float));
|
||||||
|
} else if (src0->type == GGML_TYPE_F16) {
|
||||||
|
// F16 source: scalar conversion over a destination-aligned write chunk.
|
||||||
|
const uint16_t * src_row =
|
||||||
|
(const uint16_t *) src0_data + row_index * ne00 + batch_offset + elem_offset_in_row;
|
||||||
|
copy_f16_row(dst_row, src_row, num_elements);
|
||||||
|
} else if (src0->type == GGML_TYPE_Q8_0) {
|
||||||
|
// Q8_0 source: dequantize work-unit-aligned blocks
|
||||||
|
const int64_t blocks_per_row = (ne00 + QK8_0 - 1) / QK8_0;
|
||||||
|
const int64_t src_block_offset = (row_index * blocks_per_row) + (batch_offset / ne00) * blocks_per_row;
|
||||||
|
const int64_t block_start = elem_offset_in_row / QK8_0;
|
||||||
|
const block_q8_0 * src_blocks = (const block_q8_0 *) src0_data + src_block_offset + block_start;
|
||||||
|
copy_q8_0_row_cache_aligned(dst_row, src_blocks, num_elements);
|
||||||
|
} else if (src0->type == GGML_TYPE_Q4_0) {
|
||||||
|
// Q4_0 source: dequantize work-unit-aligned blocks
|
||||||
|
const int64_t blocks_per_row = (ne00 + QK4_0 - 1) / QK4_0;
|
||||||
|
const int64_t src_block_offset = (row_index * blocks_per_row) + (batch_offset / ne00) * blocks_per_row;
|
||||||
|
const int64_t block_start = elem_offset_in_row / QK4_0;
|
||||||
|
const block_q4_0 * src_blocks = (const block_q4_0 *) src0_data + src_block_offset + block_start;
|
||||||
|
copy_q4_0_row_cache_aligned(dst_row, src_blocks, num_elements);
|
||||||
|
} else if (src0->type == GGML_TYPE_Q4_K) {
|
||||||
|
// Q4_K source: dequantize work-unit-aligned blocks
|
||||||
|
const int64_t blocks_per_row = (ne00 + QK_K - 1) / QK_K;
|
||||||
|
const int64_t src_block_offset = (row_index * blocks_per_row) + (batch_offset / ne00) * blocks_per_row;
|
||||||
|
const int64_t block_start = elem_offset_in_row / QK_K;
|
||||||
|
const block_q4_K * src_blocks = (const block_q4_K *) src0_data + src_block_offset + block_start;
|
||||||
|
copy_q4_K_row_cache_aligned(dst_row, src_blocks, num_elements);
|
||||||
|
}
|
||||||
|
|
||||||
|
wu += wus_to_process;
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline size_t tensor_bytes(const struct ggml_tensor * t) {
|
||||||
|
return (size_t) t->ne[0] * t->ne[1] * t->ne[2] * t->ne[3] * t->nb[0];
|
||||||
|
}
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_get_rows_params * params, void * env) {
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
if (!kernel_env) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct ggml_tensor * src0 = ¶ms->src0; // Data tensor (F32, Q4_0, Q8_0, or Q4_K)
|
||||||
|
struct ggml_tensor * src1 = ¶ms->src1; // Row indices tensor (I32)
|
||||||
|
struct ggml_tensor * dst = ¶ms->dst; // Output tensor (F32)
|
||||||
|
|
||||||
|
// Fast path - we know how to deal with them multi-core
|
||||||
|
if ((src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_F16 || src0->type == GGML_TYPE_Q8_0 ||
|
||||||
|
src0->type == GGML_TYPE_Q4_0 || src0->type == GGML_TYPE_Q4_K) &&
|
||||||
|
src1->type == GGML_TYPE_I32 && dst->type == GGML_TYPE_F32 && dst->ne[0] % CACHE_ELEMENTS(sizeof(float)) == 0) {
|
||||||
|
return get_row_f32_mc_cacheline_aligned(params, env);
|
||||||
|
}
|
||||||
|
|
||||||
|
int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
if (thread_id < 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (thread_id != 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (params == 0 || ((uint64_t) params & 0x7) != 0) {
|
||||||
|
return -1; // Invalid pointer
|
||||||
|
}
|
||||||
|
|
||||||
|
if (dst->type != GGML_TYPE_F32 || src1->type != GGML_TYPE_I32) {
|
||||||
|
return -1; // Invalid output or index type
|
||||||
|
}
|
||||||
|
|
||||||
|
if (src0->type != GGML_TYPE_F32 && src0->type != GGML_TYPE_F16 && src0->type != GGML_TYPE_Q8_0 &&
|
||||||
|
src0->type != GGML_TYPE_Q4_0 && src0->type != GGML_TYPE_Q4_K) {
|
||||||
|
return -1; // Unsupported input type
|
||||||
|
}
|
||||||
|
|
||||||
|
void * src0_data = src0->data;
|
||||||
|
int32_t * src1_data = (int32_t *) src1->data;
|
||||||
|
float * dst_data = (float *) dst->data;
|
||||||
|
#ifdef ET_UBERKERNEL
|
||||||
|
evict_region_past_l2(src0_data, tensor_bytes(src0));
|
||||||
|
evict_region_past_l2(src1_data, tensor_bytes(src1));
|
||||||
|
evict_region_past_l2(dst_data, tensor_bytes(dst));
|
||||||
|
#endif
|
||||||
|
|
||||||
|
if (!src0_data || !src1_data || !dst_data) {
|
||||||
|
return -1; // Null data pointer
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t ne00 = src0->ne[0]; // Source columns (row width)
|
||||||
|
const int64_t ne01 = src0->ne[1]; // Source rows (total available rows)
|
||||||
|
const int64_t ne02 = src0->ne[2]; // Source batch dimension
|
||||||
|
const int64_t ne03 = src0->ne[3]; // Source outer batch dimension
|
||||||
|
|
||||||
|
const int64_t ne10 = src1->ne[0]; // Number of indices in dimension 0
|
||||||
|
const int64_t ne11 = src1->ne[1]; // Number of indices in dimension 1
|
||||||
|
const int64_t ne12 = src1->ne[2]; // Batch dimension for indices
|
||||||
|
const int64_t ne13 = src1->ne[3]; // Outer batch dimension for indices
|
||||||
|
|
||||||
|
const int64_t total_rows_to_extract = ne10 * ne11 * ne12 * ne13;
|
||||||
|
#ifdef ET_UBERKERNEL
|
||||||
|
et_barrier(ET_BARRIER_GLOBAL);
|
||||||
|
#endif
|
||||||
|
// Naive single-threaded implementation - process all rows sequentially
|
||||||
|
// XXX: Do we really need a single-threaded implementation?
|
||||||
|
for (int64_t i = 0; i < total_rows_to_extract; i++) {
|
||||||
|
// Calculate multi-dimensional index for the current output position
|
||||||
|
const int64_t i13_idx = i / (ne12 * ne11 * ne10);
|
||||||
|
const int64_t i12_idx = (i - i13_idx * ne12 * ne11 * ne10) / (ne11 * ne10);
|
||||||
|
const int64_t i11_idx = (i - i13_idx * ne12 * ne11 * ne10 - i12_idx * ne11 * ne10) / ne10;
|
||||||
|
const int64_t i10_idx = i - i13_idx * ne12 * ne11 * ne10 - i12_idx * ne11 * ne10 - i11_idx * ne10;
|
||||||
|
|
||||||
|
// Get the row index from src1
|
||||||
|
const int64_t index_offset = i13_idx * ne12 * ne11 * ne10 + i12_idx * ne11 * ne10 + i11_idx * ne10 + i10_idx;
|
||||||
|
const int32_t row_index = src1_data[index_offset];
|
||||||
|
|
||||||
|
if (row_index < 0 || row_index >= ne01) {
|
||||||
|
return -1; // Index out of bounds
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t batch_offset =
|
||||||
|
i11_idx * ne01 * ne00 + i12_idx * ne02 * ne01 * ne00 + i13_idx * ne03 * ne02 * ne01 * ne00;
|
||||||
|
|
||||||
|
const int64_t dst_offset = i;
|
||||||
|
|
||||||
|
if (src0->type == GGML_TYPE_F32) {
|
||||||
|
// F32 source: direct copy
|
||||||
|
const float * src_row = (const float *) src0_data + row_index * ne00 + batch_offset;
|
||||||
|
float * dst_row = dst_data + dst_offset * ne00;
|
||||||
|
copy_f32_row(dst_row, src_row, ne00);
|
||||||
|
} else if (src0->type == GGML_TYPE_F16) {
|
||||||
|
// F16 source: scalar conversion
|
||||||
|
const uint16_t * src_row = (const uint16_t *) src0_data + row_index * ne00 + batch_offset;
|
||||||
|
float * dst_row = dst_data + dst_offset * ne00;
|
||||||
|
copy_f16_row(dst_row, src_row, ne00);
|
||||||
|
} else if (src0->type == GGML_TYPE_Q8_0) {
|
||||||
|
// Q8_0 source: dequantize while copying
|
||||||
|
const int64_t blocks_per_row = (ne00 + QK8_0 - 1) / QK8_0;
|
||||||
|
const int64_t src_block_offset = (row_index * blocks_per_row) + (batch_offset / ne00) * blocks_per_row;
|
||||||
|
const block_q8_0 * src_blocks = (const block_q8_0 *) src0_data + src_block_offset;
|
||||||
|
float * dst_row = dst_data + dst_offset * ne00;
|
||||||
|
copy_q8_0_row(dst_row, src_blocks, ne00);
|
||||||
|
} else if (src0->type == GGML_TYPE_Q4_0) {
|
||||||
|
// Q4_0 source: dequantize while copying
|
||||||
|
const int64_t blocks_per_row = (ne00 + QK4_0 - 1) / QK4_0;
|
||||||
|
const int64_t src_block_offset = (row_index * blocks_per_row) + (batch_offset / ne00) * blocks_per_row;
|
||||||
|
const block_q4_0 * src_blocks = (const block_q4_0 *) src0_data + src_block_offset;
|
||||||
|
float * dst_row = dst_data + dst_offset * ne00;
|
||||||
|
copy_q4_0_row(dst_row, src_blocks, ne00);
|
||||||
|
} else if (src0->type == GGML_TYPE_Q4_K) {
|
||||||
|
// Q4_K source: dequantize while copying
|
||||||
|
const int64_t blocks_per_row = (ne00 + QK_K - 1) / QK_K;
|
||||||
|
const int64_t src_block_offset = (row_index * blocks_per_row) + (batch_offset / ne00) * blocks_per_row;
|
||||||
|
const block_q4_K * src_blocks = (const block_q4_K *) src0_data + src_block_offset;
|
||||||
|
float * dst_row = dst_data + dst_offset * ne00;
|
||||||
|
copy_q4_K_row(dst_row, src_blocks, ne00);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
// ET kernel entry-point parameter structs and tensor helpers.
|
||||||
|
|
||||||
|
#ifndef GGML_TENSOR_H
|
||||||
|
#define GGML_TENSOR_H
|
||||||
|
|
||||||
|
#include <stddef.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
#include "ggml.h"
|
||||||
|
|
||||||
|
struct ggml_et_binary_params {
|
||||||
|
struct ggml_tensor src0;
|
||||||
|
struct ggml_tensor src1;
|
||||||
|
struct ggml_tensor dst;
|
||||||
|
};
|
||||||
|
|
||||||
|
// bias.data == NULL -> unfused MUL_MAT; otherwise dst = mat_mul(...) + bias.
|
||||||
|
struct ggml_et_mm_q8_params {
|
||||||
|
struct ggml_tensor src0;
|
||||||
|
struct ggml_tensor src1;
|
||||||
|
struct ggml_tensor dst;
|
||||||
|
struct ggml_tensor bias;
|
||||||
|
};
|
||||||
|
|
||||||
|
struct ggml_et_mul_mat_id_params {
|
||||||
|
struct ggml_tensor src0; // [K, M, n_expert]
|
||||||
|
struct ggml_tensor src1; // [K, n_expert_used, batch]
|
||||||
|
struct ggml_tensor src2; // [n_expert_used, batch] (I32 expert indices)
|
||||||
|
struct ggml_tensor dst; // [M, n_expert_used, batch, 1]
|
||||||
|
};
|
||||||
|
|
||||||
|
// ne[i] == 1 axes are skipped: their stride is unobservable.
|
||||||
|
static inline int ggml_tensor_is_contiguous(const struct ggml_tensor * t, int type_size) {
|
||||||
|
int64_t expected = type_size;
|
||||||
|
for (int i = 0; i < GGML_MAX_DIMS; i++) {
|
||||||
|
if (t->ne[i] > 1 && (int64_t) t->nb[i] != expected) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
expected *= t->ne[i];
|
||||||
|
}
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
#endif // GGML_TENSOR_H
|
||||||
@@ -0,0 +1,551 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// GLU F32 Kernel (SwiGLU specifically)
|
||||||
|
// Gated Linear Unit: y[i] = silu(x[i]) * g[i] where silu(x) = x * sigmoid(x)
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "math_fp.h"
|
||||||
|
#include "platform.h"
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
// GLU kernel parameters structure (from ET backend ops)
|
||||||
|
struct ggml_et_glu_params {
|
||||||
|
struct ggml_tensor src0; // F32 input tensor A (or combined tensor if src1 is null)
|
||||||
|
struct ggml_tensor src1; // F32 input tensor B (null for single tensor mode)
|
||||||
|
struct ggml_tensor dst; // F32 output tensor (n/2 columns)
|
||||||
|
int32_t glu_op_type; // GLU operation type (REGLU=0, GEGLU=1, SWIGLU=2, etc.)
|
||||||
|
int32_t swapped; // Whether gate and value are swapped
|
||||||
|
float alpha; // SWIGLU_OAI: sigmoid scaling factor
|
||||||
|
float limit; // SWIGLU_OAI: clamp limit
|
||||||
|
};
|
||||||
|
|
||||||
|
// SiLU activation function: silu(x) = x * sigmoid(x) = x / (1 + exp(-x))
|
||||||
|
static inline float silu_f32(float x) {
|
||||||
|
// For numerical stability, use the mathematically equivalent form:
|
||||||
|
// silu(x) = x / (1 + exp(-x)) = x * sigmoid(x)
|
||||||
|
// For large negative x, exp(-x) -> inf, so silu(x) -> 0
|
||||||
|
// For large positive x, exp(-x) -> 0, so silu(x) -> x
|
||||||
|
|
||||||
|
if (x > 20.0f) {
|
||||||
|
// For x > 20, exp(-x) is negligible, silu(x) ~ x
|
||||||
|
return x;
|
||||||
|
} else if (x < -20.0f) {
|
||||||
|
// For x < -20, silu(x) ~ 0
|
||||||
|
return 0.0f;
|
||||||
|
} else {
|
||||||
|
// Use standard formula: silu(x) = x / (1 + exp(-x))
|
||||||
|
// Optimized using ET hardware division
|
||||||
|
float exp_neg_x = et_expf(-x);
|
||||||
|
float denominator = 1.0f + exp_neg_x;
|
||||||
|
return et_fdiv(x, denominator);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Vectorized GeGLU block processing (8 elements = 1 cache line, 64B aligned)
|
||||||
|
// gelu(x) = 0.5*x*(1 + tanh(z)) = x * (1 - 1/(exp(2z)+1))
|
||||||
|
// where z = sqrt(2/pi) * x * (1 + 0.044715*x^2)
|
||||||
|
// Reformulated to avoid inf*0 NaN: uses x * sigmoid(2z) identity
|
||||||
|
static inline void block_geglu(float * dst_block, const float * x_block, const float * g_block, int elements) {
|
||||||
|
unsigned long temp_mask;
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(temp_mask));
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF");
|
||||||
|
|
||||||
|
float one_const = 1.0f;
|
||||||
|
float coef_a_const = 0.044715f;
|
||||||
|
float sqrt2pi_const = 0.79788456080286535587989211986876f; // sqrt(2/pi)
|
||||||
|
float two_log2e_const = 2.8853900817779268f; // 2 * log2(e)
|
||||||
|
|
||||||
|
for (int32_t i = 0; i < elements; i += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
// Load inputs
|
||||||
|
"flw.ps f10, %[x_vec]\n" // f10 = x
|
||||||
|
"flw.ps f11, %[g_vec]\n" // f11 = g
|
||||||
|
|
||||||
|
// Broadcast constants
|
||||||
|
"fbc.ps f20, %[one_ptr]\n" // f20 = 1.0
|
||||||
|
"fbc.ps f22, %[coef_ptr]\n" // f22 = 0.044715
|
||||||
|
"fbc.ps f23, %[sqrt2pi_ptr]\n" // f23 = sqrt(2/pi)
|
||||||
|
"fbc.ps f24, %[two_log2e_ptr]\n" // f24 = 2*log2(e)
|
||||||
|
|
||||||
|
// inner = 1 + 0.044715 * x^2
|
||||||
|
"fmul.ps f12, f10, f10\n" // f12 = x^2
|
||||||
|
"fmadd.ps f13, f22, f12, f20\n" // f13 = 1 + 0.044715*x^2
|
||||||
|
|
||||||
|
// z = sqrt(2/pi) * x * inner
|
||||||
|
"fmul.ps f14, f23, f10\n" // f14 = sqrt(2/pi) * x
|
||||||
|
"fmul.ps f14, f14, f13\n" // f14 = z
|
||||||
|
|
||||||
|
// exp(2z) via fexp.ps: feed z * 2*log2(e) since fexp computes 2^input
|
||||||
|
"fmul.ps f15, f14, f24\n" // f15 = 2z * log2(e)
|
||||||
|
"fexp.ps f15, f15\n" // f15 = exp(2z)
|
||||||
|
|
||||||
|
// gelu(x) = x * (1 - 1/(exp(2z)+1)) [NaN-safe: no inf*0]
|
||||||
|
// exp(2z)->inf: rcp(inf)=0, 1-0=1, gelu=x
|
||||||
|
// exp(2z)->0: rcp(1)=1, 1-1=0, gelu=0
|
||||||
|
"fadd.ps f16, f15, f20\n" // f16 = exp(2z) + 1
|
||||||
|
"frcp.ps f16, f16\n" // f16 = 1/(exp(2z) + 1)
|
||||||
|
"fsub.ps f16, f20, f16\n" // f16 = 1 - 1/(exp(2z)+1)
|
||||||
|
"fmul.ps f16, f10, f16\n" // f16 = gelu(x)
|
||||||
|
|
||||||
|
// Final result
|
||||||
|
"fmul.ps f18, f16, f11\n" // f18 = gelu(x) * g
|
||||||
|
|
||||||
|
"fsw.ps f18, %[dst_out]\n"
|
||||||
|
|
||||||
|
: [dst_out] "=m"(*(float (*)[8]) & dst_block[i])
|
||||||
|
: [x_vec] "m"(*(const float (*)[8]) & x_block[i]), [g_vec] "m"(*(const float (*)[8]) & g_block[i]),
|
||||||
|
[one_ptr] "m"(one_const), [coef_ptr] "m"(coef_a_const), [sqrt2pi_ptr] "m"(sqrt2pi_const),
|
||||||
|
[two_log2e_ptr] "m"(two_log2e_const)
|
||||||
|
: "f10", "f11", "f12", "f13", "f14", "f15", "f16", "f18", "f20", "f22", "f23", "f24");
|
||||||
|
}
|
||||||
|
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(temp_mask));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Vectorized SwiGLU block processing (16 elements = 1 cache line)
|
||||||
|
static inline void block_swiglu(float * dst_block, const float * x_block, const float * g_block, int elements) {
|
||||||
|
// Process 8 elements at a time using vector instructions
|
||||||
|
int32_t vec_end = (elements / 8) * 8;
|
||||||
|
|
||||||
|
// Set mask register to enable all 8 vector elements
|
||||||
|
unsigned long temp_mask;
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(temp_mask)); // Save current mask
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF"); // Enable all 8 elements
|
||||||
|
|
||||||
|
// Constants for broadcasting
|
||||||
|
float zero_const = 0.0f;
|
||||||
|
float one_const = 1.0f;
|
||||||
|
float log2e_const = 1.4426950408889634f; // log2(e)
|
||||||
|
|
||||||
|
for (int32_t i = 0; i < vec_end; i += 8) {
|
||||||
|
// Vectorized SwiGLU: dst = silu(x) * g = (x / (1 + exp(-x))) * g
|
||||||
|
// Using ET hardware: exp, reciprocal, multiply operations
|
||||||
|
__asm__ volatile(
|
||||||
|
// Load input vectors
|
||||||
|
"flw.ps f10, %[x_vec]\n" // f10 = x[0..7]
|
||||||
|
"flw.ps f11, %[g_vec]\n" // f11 = g[0..7]
|
||||||
|
|
||||||
|
// Broadcast constants to vector registers
|
||||||
|
"fbc.ps f20, %[zero_ptr]\n" // f20 = broadcast(0.0f) to all 8 elements
|
||||||
|
"fbc.ps f21, %[one_ptr]\n" // f21 = broadcast(1.0f) to all 8 elements
|
||||||
|
|
||||||
|
// Compute -x (negate x by subtracting from zero)
|
||||||
|
"fsub.ps f12, f20, f10\n" // f12 = 0 - x = -x
|
||||||
|
|
||||||
|
// Convert to base-2 exponent: -x * log2(e) = -x * 1.44269504
|
||||||
|
// Load log2(e) constant
|
||||||
|
"fbc.ps f22, %[log2e_ptr]\n" // f22 = broadcast(1.44269504f)
|
||||||
|
"fmul.ps f13, f12, f22\n" // f13 = -x * log2(e)
|
||||||
|
|
||||||
|
// Compute 2^(-x * log2(e)) = exp(-x)
|
||||||
|
"fexp.ps f14, f13\n" // f14 = 2^(-x * log2(e)) = exp(-x)
|
||||||
|
|
||||||
|
// Compute 1 + exp(-x)
|
||||||
|
"fadd.ps f15, f14, f21\n" // f15 = exp(-x) + 1
|
||||||
|
|
||||||
|
// Compute 1 / (1 + exp(-x)) using reciprocal
|
||||||
|
"frcp.ps f16, f15\n" // f16 = 1 / (1 + exp(-x))
|
||||||
|
|
||||||
|
// Compute silu(x) = x * (1 / (1 + exp(-x)))
|
||||||
|
"fmul.ps f17, f10, f16\n" // f17 = x * (1 / (1 + exp(-x))) = silu(x)
|
||||||
|
|
||||||
|
// Compute final result: silu(x) * g
|
||||||
|
"fmul.ps f18, f17, f11\n" // f18 = silu(x) * g
|
||||||
|
|
||||||
|
// Store result
|
||||||
|
"fsw.ps f18, %[dst_out]\n" // Store 8 results to destination
|
||||||
|
|
||||||
|
: [dst_out] "=m"(*(float (*)[8]) & dst_block[i])
|
||||||
|
: [x_vec] "m"(*(const float (*)[8]) & x_block[i]), [g_vec] "m"(*(const float (*)[8]) & g_block[i]),
|
||||||
|
[zero_ptr] "m"(zero_const), // Memory reference to 0.0f for broadcasting
|
||||||
|
[one_ptr] "m"(one_const), // Memory reference to 1.0f for broadcasting
|
||||||
|
[log2e_ptr] "m"(log2e_const) // Memory reference to log2(e) for broadcasting
|
||||||
|
: "f10", "f11", "f12", "f13", "f14", "f15", "f16", "f17", "f18", "f20", "f21", "f22");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Restore original mask
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(temp_mask));
|
||||||
|
|
||||||
|
// Handle remaining elements (< 8) with scalar operations
|
||||||
|
for (int32_t i = vec_end; i < elements; i++) {
|
||||||
|
dst_block[i] = silu_f32(x_block[i]) * g_block[i];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Vectorized ReGLU block: dst = max(0, x) * g
|
||||||
|
static inline void block_reglu(float * dst_block, const float * x_block, const float * g_block, int elements) {
|
||||||
|
int32_t vec_end = (elements / 8) * 8;
|
||||||
|
|
||||||
|
unsigned long temp_mask;
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(temp_mask));
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF");
|
||||||
|
|
||||||
|
float zero_const = 0.0f;
|
||||||
|
|
||||||
|
for (int32_t i = 0; i < vec_end; i += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f10, %[x_vec]\n" // f10 = x
|
||||||
|
"flw.ps f11, %[g_vec]\n" // f11 = g
|
||||||
|
"fbc.ps f20, %[zero_ptr]\n" // f20 = 0.0
|
||||||
|
"fmax.ps f12, f10, f20\n" // f12 = max(x, 0)
|
||||||
|
"fmul.ps f13, f12, f11\n" // f13 = relu(x) * g
|
||||||
|
"fsw.ps f13, %[dst_out]\n"
|
||||||
|
: [dst_out] "=m"(*(float (*)[8]) & dst_block[i])
|
||||||
|
: [x_vec] "m"(*(const float (*)[8]) & x_block[i]), [g_vec] "m"(*(const float (*)[8]) & g_block[i]),
|
||||||
|
[zero_ptr] "m"(zero_const)
|
||||||
|
: "f10", "f11", "f12", "f13", "f20");
|
||||||
|
}
|
||||||
|
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(temp_mask));
|
||||||
|
|
||||||
|
for (int32_t i = vec_end; i < elements; i++) {
|
||||||
|
float xv = x_block[i];
|
||||||
|
dst_block[i] = (xv > 0.0f) ? xv * g_block[i] : 0.0f;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Vectorized GeGLU-Quick block: dst = x * sigmoid(1.702 * x) * g
|
||||||
|
// Using gelu_quick(x) = x / (1 + exp(-1.702*x))
|
||||||
|
static inline void block_geglu_quick(float * dst_block, const float * x_block, const float * g_block, int elements) {
|
||||||
|
int32_t vec_end = (elements / 8) * 8;
|
||||||
|
|
||||||
|
unsigned long temp_mask;
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(temp_mask));
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF");
|
||||||
|
|
||||||
|
float zero_const = 0.0f;
|
||||||
|
float one_const = 1.0f;
|
||||||
|
// -1.702 * log2(e), so that fexp.ps(x * neg_k_log2e) = exp(-1.702*x)
|
||||||
|
float neg_k_log2e_const = -1.702f * 1.4426950408889634f;
|
||||||
|
|
||||||
|
for (int32_t i = 0; i < vec_end; i += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f10, %[x_vec]\n" // f10 = x
|
||||||
|
"flw.ps f11, %[g_vec]\n" // f11 = g
|
||||||
|
"fbc.ps f20, %[zero_ptr]\n" // f20 = 0
|
||||||
|
"fbc.ps f21, %[one_ptr]\n" // f21 = 1
|
||||||
|
"fbc.ps f22, %[k_ptr]\n" // f22 = -1.702*log2(e)
|
||||||
|
"fmul.ps f13, f10, f22\n" // f13 = -1.702*x*log2(e)
|
||||||
|
"fexp.ps f14, f13\n" // f14 = exp(-1.702*x)
|
||||||
|
"fadd.ps f15, f14, f21\n" // f15 = 1 + exp(-1.702*x)
|
||||||
|
"frcp.ps f16, f15\n" // f16 = sigmoid(1.702*x)
|
||||||
|
"fmul.ps f17, f10, f16\n" // f17 = gelu_quick(x)
|
||||||
|
"fmul.ps f18, f17, f11\n" // f18 = gelu_quick(x) * g
|
||||||
|
"fsw.ps f18, %[dst_out]\n"
|
||||||
|
: [dst_out] "=m"(*(float (*)[8]) & dst_block[i])
|
||||||
|
: [x_vec] "m"(*(const float (*)[8]) & x_block[i]), [g_vec] "m"(*(const float (*)[8]) & g_block[i]),
|
||||||
|
[zero_ptr] "m"(zero_const), [one_ptr] "m"(one_const), [k_ptr] "m"(neg_k_log2e_const)
|
||||||
|
: "f10", "f11", "f13", "f14", "f15", "f16", "f17", "f18", "f20", "f21", "f22");
|
||||||
|
}
|
||||||
|
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(temp_mask));
|
||||||
|
|
||||||
|
for (int32_t i = vec_end; i < elements; i++) {
|
||||||
|
float xv = x_block[i];
|
||||||
|
// Reuse silu reciprocal path: sigmoid(1.702*x) = 1/(1+exp(-1.702*x))
|
||||||
|
float e = et_expf(-1.702f * xv);
|
||||||
|
dst_block[i] = et_fdiv(xv, 1.0f + e) * g_block[i];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Vectorized SwiGLU-OAI block (OpenAI gpt-oss variant):
|
||||||
|
// x_c = min(x, limit)
|
||||||
|
// y_c = clamp(g, -limit, limit)
|
||||||
|
// out = (x_c / (1 + exp(-alpha * x_c))) * (y_c + 1)
|
||||||
|
static inline void block_swiglu_oai(float * dst_block,
|
||||||
|
const float * x_block,
|
||||||
|
const float * g_block,
|
||||||
|
int elements,
|
||||||
|
float alpha,
|
||||||
|
float limit) {
|
||||||
|
int32_t vec_end = (elements / 8) * 8;
|
||||||
|
|
||||||
|
unsigned long temp_mask;
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(temp_mask));
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF");
|
||||||
|
|
||||||
|
float zero_const = 0.0f;
|
||||||
|
float one_const = 1.0f;
|
||||||
|
float limit_pos = limit;
|
||||||
|
float limit_neg = -limit;
|
||||||
|
// -alpha * log2(e): feed (x * neg_alpha_log2e) into fexp.ps to get exp(-alpha*x)
|
||||||
|
float neg_alpha_l2e = -alpha * 1.4426950408889634f;
|
||||||
|
|
||||||
|
for (int32_t i = 0; i < vec_end; i += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f10, %[x_vec]\n" // f10 = x raw
|
||||||
|
"flw.ps f11, %[g_vec]\n" // f11 = g raw
|
||||||
|
|
||||||
|
"fbc.ps f20, %[zero_ptr]\n" // f20 = 0
|
||||||
|
"fbc.ps f21, %[one_ptr]\n" // f21 = 1
|
||||||
|
"fbc.ps f23, %[lim_pos]\n" // f23 = +limit
|
||||||
|
"fbc.ps f24, %[lim_neg]\n" // f24 = -limit
|
||||||
|
"fbc.ps f25, %[k_ptr]\n" // f25 = -alpha*log2(e)
|
||||||
|
|
||||||
|
// x_c = min(x, +limit) (no lower bound on x per OAI spec)
|
||||||
|
"fmin.ps f12, f10, f23\n" // f12 = x_c
|
||||||
|
|
||||||
|
// y_c = clamp(g, -limit, +limit) = min(max(g, -limit), +limit)
|
||||||
|
"fmax.ps f13, f11, f24\n" // f13 = max(g, -limit)
|
||||||
|
"fmin.ps f13, f13, f23\n" // f13 = y_c
|
||||||
|
|
||||||
|
// sigmoid(alpha * x_c) = 1 / (1 + exp(-alpha * x_c))
|
||||||
|
"fmul.ps f14, f12, f25\n" // f14 = -alpha*x_c*log2(e)
|
||||||
|
"fexp.ps f15, f14\n" // f15 = exp(-alpha*x_c)
|
||||||
|
"fadd.ps f15, f15, f21\n" // f15 = 1 + exp(-alpha*x_c)
|
||||||
|
"frcp.ps f16, f15\n" // f16 = sigmoid(alpha*x_c)
|
||||||
|
|
||||||
|
// out_glu = x_c * sigmoid(alpha*x_c)
|
||||||
|
"fmul.ps f17, f12, f16\n" // f17 = swiglu_oai gate output
|
||||||
|
|
||||||
|
// dst = out_glu * (y_c + 1)
|
||||||
|
"fadd.ps f18, f13, f21\n" // f18 = y_c + 1
|
||||||
|
"fmul.ps f19, f17, f18\n" // f19 = final
|
||||||
|
"fsw.ps f19, %[dst_out]\n"
|
||||||
|
|
||||||
|
: [dst_out] "=m"(*(float (*)[8]) & dst_block[i])
|
||||||
|
: [x_vec] "m"(*(const float (*)[8]) & x_block[i]), [g_vec] "m"(*(const float (*)[8]) & g_block[i]),
|
||||||
|
[zero_ptr] "m"(zero_const), [one_ptr] "m"(one_const), [lim_pos] "m"(limit_pos), [lim_neg] "m"(limit_neg),
|
||||||
|
[k_ptr] "m"(neg_alpha_l2e)
|
||||||
|
: "f10", "f11", "f12", "f13", "f14", "f15", "f16", "f17", "f18", "f19", "f20", "f21", "f23", "f24", "f25");
|
||||||
|
}
|
||||||
|
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(temp_mask));
|
||||||
|
|
||||||
|
// Scalar tail (mirrors CPU reference exactly)
|
||||||
|
for (int32_t i = vec_end; i < elements; i++) {
|
||||||
|
float xv = x_block[i];
|
||||||
|
float yv = g_block[i];
|
||||||
|
if (xv > limit) {
|
||||||
|
xv = limit;
|
||||||
|
}
|
||||||
|
if (yv > limit) {
|
||||||
|
yv = limit;
|
||||||
|
}
|
||||||
|
if (yv < -limit) {
|
||||||
|
yv = -limit;
|
||||||
|
}
|
||||||
|
float e = et_expf(-alpha * xv);
|
||||||
|
float out_glu = et_fdiv(xv, 1.0f + e);
|
||||||
|
dst_block[i] = out_glu * (yv + 1.0f);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Scalar erf approximation (Abramowitz & Stegun 7.1.26, max error ~1.5e-7)
|
||||||
|
static inline float erf_approx(float x) {
|
||||||
|
const float a1 = 0.254829592f;
|
||||||
|
const float a2 = -0.284496736f;
|
||||||
|
const float a3 = 1.421413741f;
|
||||||
|
const float a4 = -1.453152027f;
|
||||||
|
const float a5 = 1.061405429f;
|
||||||
|
const float p = 0.3275911f;
|
||||||
|
|
||||||
|
float sign = (x < 0.0f) ? -1.0f : 1.0f;
|
||||||
|
float ax = (x < 0.0f) ? -x : x;
|
||||||
|
float t = et_fdiv(1.0f, 1.0f + p * ax);
|
||||||
|
float t2 = t * t;
|
||||||
|
float t3 = t2 * t;
|
||||||
|
float t4 = t3 * t;
|
||||||
|
float t5 = t4 * t;
|
||||||
|
float poly = a1 * t + a2 * t2 + a3 * t3 + a4 * t4 + a5 * t5;
|
||||||
|
float y = 1.0f - poly * et_expf(-ax * ax);
|
||||||
|
return sign * y;
|
||||||
|
}
|
||||||
|
|
||||||
|
// GeGLU-Erf block: dst = 0.5 * x * (1 + erf(x / sqrt(2))) * g
|
||||||
|
// Scalar implementation — variant is rarely used so we keep complexity low.
|
||||||
|
static inline void block_geglu_erf(float * dst_block, const float * x_block, const float * g_block, int elements) {
|
||||||
|
const float sqrt_2_inv = 0.70710678118654752440f;
|
||||||
|
for (int32_t i = 0; i < elements; i++) {
|
||||||
|
float xv = x_block[i];
|
||||||
|
dst_block[i] = 0.5f * xv * (1.0f + erf_approx(xv * sqrt_2_inv)) * g_block[i];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Main entry point for GLU kernel
|
||||||
|
int entry_point(struct ggml_et_glu_params * params, void * env) {
|
||||||
|
// Cast env to proper type
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
|
||||||
|
// Validate environment pointer
|
||||||
|
if (!kernel_env) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get thread info using shire mask from environment
|
||||||
|
int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
int num_threads = get_num_threads(kernel_env->shire_mask);
|
||||||
|
|
||||||
|
// Basic safety check on params
|
||||||
|
if (params == 0 || ((uint64_t) params & 0x7) != 0) {
|
||||||
|
return -1; // Invalid pointer
|
||||||
|
}
|
||||||
|
|
||||||
|
// Supported variants: SwiGLU, SwiGLU-OAI, GeGLU, GeGLU-Erf, GeGLU-Quick, ReGLU
|
||||||
|
switch (params->glu_op_type) {
|
||||||
|
case GGML_GLU_OP_SWIGLU:
|
||||||
|
case GGML_GLU_OP_SWIGLU_OAI:
|
||||||
|
case GGML_GLU_OP_GEGLU:
|
||||||
|
case GGML_GLU_OP_GEGLU_ERF:
|
||||||
|
case GGML_GLU_OP_GEGLU_QUICK:
|
||||||
|
case GGML_GLU_OP_REGLU:
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
return -1; // Unsupported GLU operation
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extract tensor references
|
||||||
|
struct ggml_tensor * src0 = ¶ms->src0;
|
||||||
|
struct ggml_tensor * src1 = params->src1.data ? ¶ms->src1 : 0;
|
||||||
|
struct ggml_tensor * dst = ¶ms->dst;
|
||||||
|
int32_t swapped = params->swapped;
|
||||||
|
|
||||||
|
// Validate tensor types (F32 only)
|
||||||
|
if (src0->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) {
|
||||||
|
return -1; // Unsupported type combination
|
||||||
|
}
|
||||||
|
|
||||||
|
if (src1 && src1->type != GGML_TYPE_F32) {
|
||||||
|
return -1; // Unsupported src1 type
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get data pointers
|
||||||
|
float * src0_data = (float *) src0->data;
|
||||||
|
float * src1_data = src1 ? (float *) src1->data : src0_data;
|
||||||
|
float * dst_data = (float *) dst->data;
|
||||||
|
|
||||||
|
// Validate data pointers
|
||||||
|
if (!src0_data || !dst_data) {
|
||||||
|
return -1; // Null data pointer
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get tensor dimensions
|
||||||
|
const int64_t nc = dst->ne[0]; // Output columns (input columns / 2)
|
||||||
|
const int64_t nr = dst->ne[1] * dst->ne[2] * dst->ne[3]; // Total rows
|
||||||
|
|
||||||
|
// Get strides
|
||||||
|
const size_t src0_stride = src0->nb[1]; // Stride between rows in src0
|
||||||
|
const size_t src1_stride = src1 ? src1->nb[1] : src0->nb[1]; // Stride between rows in src1
|
||||||
|
const size_t dst_stride = dst->nb[1]; // Stride between rows in dst
|
||||||
|
|
||||||
|
// Validate dimensions for split SwiGLU
|
||||||
|
if (src1) {
|
||||||
|
// Split tensor mode: src0 and src1 should have same shape as dst
|
||||||
|
if (src0->ne[0] != nc || src1->ne[0] != nc) {
|
||||||
|
return -1; // Dimension mismatch in split mode
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// Single tensor mode: src0 should have 2*nc columns
|
||||||
|
if (src0->ne[0] != 2 * nc) {
|
||||||
|
return -1; // Dimension mismatch in single tensor mode
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Calculate total elements for cache line distribution
|
||||||
|
const int64_t elements_per_cacheline = 16; // 64 bytes / 4 bytes per float
|
||||||
|
const int64_t total_elements = nr * nc;
|
||||||
|
const int64_t total_cachelines = (total_elements + elements_per_cacheline - 1) / elements_per_cacheline;
|
||||||
|
|
||||||
|
// Distribute cache lines across threads
|
||||||
|
int64_t cachelines_per_thread = (total_cachelines + num_threads - 1) / num_threads;
|
||||||
|
int64_t start_cacheline = thread_id * cachelines_per_thread;
|
||||||
|
int64_t end_cacheline = start_cacheline + cachelines_per_thread;
|
||||||
|
|
||||||
|
// Clamp end_cacheline to actual number of cache lines
|
||||||
|
if (end_cacheline > total_cachelines) {
|
||||||
|
end_cacheline = total_cachelines;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Thread should return if no work to do
|
||||||
|
if (start_cacheline >= total_cachelines) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Process cache lines assigned to this thread
|
||||||
|
for (int64_t cl = start_cacheline; cl < end_cacheline; cl++) {
|
||||||
|
// Map cache line back to element coordinates
|
||||||
|
int64_t global_element_start = cl * elements_per_cacheline;
|
||||||
|
int64_t row = global_element_start / nc;
|
||||||
|
int64_t col = global_element_start % nc;
|
||||||
|
|
||||||
|
// Skip if we're past the end of data
|
||||||
|
if (global_element_start >= total_elements) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Calculate how many elements to process in this cache line
|
||||||
|
int64_t elements_remaining = total_elements - global_element_start;
|
||||||
|
int elements_this_block =
|
||||||
|
(int) ((elements_remaining < elements_per_cacheline) ? elements_remaining : elements_per_cacheline);
|
||||||
|
|
||||||
|
// Process elements that span across rows
|
||||||
|
int64_t elements_processed = 0;
|
||||||
|
while (elements_processed < elements_this_block && row < nr) {
|
||||||
|
// Calculate elements to process in current row
|
||||||
|
int64_t elements_in_row = nc - col;
|
||||||
|
int64_t elements_to_process = elements_this_block - elements_processed;
|
||||||
|
if (elements_to_process > elements_in_row) {
|
||||||
|
elements_to_process = elements_in_row;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get pointers for current row and column range
|
||||||
|
float * dst_ptr = (float *) ((char *) dst_data + row * dst_stride) + col;
|
||||||
|
|
||||||
|
float * x_ptr;
|
||||||
|
float * g_ptr;
|
||||||
|
|
||||||
|
if (src1) {
|
||||||
|
// Split tensor mode
|
||||||
|
x_ptr = (float *) ((char *) src0_data + row * src0_stride) + col;
|
||||||
|
g_ptr = (float *) ((char *) src1_data + row * src1_stride) + col;
|
||||||
|
} else {
|
||||||
|
// Single tensor mode - src0 contains both x and g
|
||||||
|
float * src0_row = (float *) ((char *) src0_data + row * src0_stride);
|
||||||
|
if (swapped) {
|
||||||
|
g_ptr = src0_row + col; // First half is gate
|
||||||
|
x_ptr = src0_row + nc + col; // Second half is value
|
||||||
|
} else {
|
||||||
|
x_ptr = src0_row + col; // First half is value
|
||||||
|
g_ptr = src0_row + nc + col; // Second half is gate
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Process this segment
|
||||||
|
switch (params->glu_op_type) {
|
||||||
|
case GGML_GLU_OP_GEGLU:
|
||||||
|
block_geglu(dst_ptr, x_ptr, g_ptr, (int) elements_to_process);
|
||||||
|
break;
|
||||||
|
case GGML_GLU_OP_SWIGLU:
|
||||||
|
block_swiglu(dst_ptr, x_ptr, g_ptr, (int) elements_to_process);
|
||||||
|
break;
|
||||||
|
case GGML_GLU_OP_REGLU:
|
||||||
|
block_reglu(dst_ptr, x_ptr, g_ptr, (int) elements_to_process);
|
||||||
|
break;
|
||||||
|
case GGML_GLU_OP_GEGLU_QUICK:
|
||||||
|
block_geglu_quick(dst_ptr, x_ptr, g_ptr, (int) elements_to_process);
|
||||||
|
break;
|
||||||
|
case GGML_GLU_OP_GEGLU_ERF:
|
||||||
|
block_geglu_erf(dst_ptr, x_ptr, g_ptr, (int) elements_to_process);
|
||||||
|
break;
|
||||||
|
case GGML_GLU_OP_SWIGLU_OAI:
|
||||||
|
block_swiglu_oai(dst_ptr, x_ptr, g_ptr, (int) elements_to_process, params->alpha, params->limit);
|
||||||
|
break;
|
||||||
|
default:
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Update counters
|
||||||
|
elements_processed += elements_to_process;
|
||||||
|
col += elements_to_process;
|
||||||
|
|
||||||
|
// Move to next row if current row is complete
|
||||||
|
if (col >= nc) {
|
||||||
|
row++;
|
||||||
|
col = 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,171 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// GROUP_NORM F32 Kernel
|
||||||
|
// Baseline scalar implementation:
|
||||||
|
// normalize over (ne0 * ne1 * channels_in_group) for each (group, batch).
|
||||||
|
//
|
||||||
|
// Parallelization:
|
||||||
|
// - Work is partitioned across (group, batch) pairs.
|
||||||
|
// - For non-cache-aligned ne0, writes are emitted in row-groups so each thread's
|
||||||
|
// destination write footprint still spans an integer number of cache lines.
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "math_fp.h"
|
||||||
|
#include "platform.h"
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
struct ggml_et_group_norm_params {
|
||||||
|
struct ggml_tensor src0;
|
||||||
|
struct ggml_tensor dst;
|
||||||
|
int32_t n_groups;
|
||||||
|
float eps;
|
||||||
|
};
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_group_norm_params * params, void * env) {
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
|
||||||
|
if (!kernel_env) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
int num_threads = get_num_threads(kernel_env->shire_mask);
|
||||||
|
|
||||||
|
if (thread_id < 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (params == 0 || ((uint64_t) params & 0x7) != 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct ggml_tensor * src0 = ¶ms->src0;
|
||||||
|
struct ggml_tensor * dst = ¶ms->dst;
|
||||||
|
|
||||||
|
if (src0->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const float * src0_data = (const float *) src0->data;
|
||||||
|
float * dst_data = (float *) dst->data;
|
||||||
|
|
||||||
|
if (!src0_data || !dst_data) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int32_t n_groups = params->n_groups;
|
||||||
|
const float eps = params->eps;
|
||||||
|
|
||||||
|
if (n_groups <= 0 || eps < 0.0f) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t ne0 = dst->ne[0];
|
||||||
|
const int64_t ne1 = dst->ne[1];
|
||||||
|
const int64_t ne2 = dst->ne[2];
|
||||||
|
const int64_t ne3 = dst->ne[3];
|
||||||
|
|
||||||
|
if (src0->ne[0] != ne0 || src0->ne[1] != ne1 || src0->ne[2] != ne2 || src0->ne[3] != ne3) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t nb1 = dst->nb[1];
|
||||||
|
const int64_t nb2 = dst->nb[2];
|
||||||
|
const int64_t nb3 = dst->nb[3];
|
||||||
|
const int64_t nb01 = src0->nb[1];
|
||||||
|
const int64_t nb02 = src0->nb[2];
|
||||||
|
const int64_t nb03 = src0->nb[3];
|
||||||
|
|
||||||
|
const int64_t channels_per_group = (ne2 + n_groups - 1) / n_groups;
|
||||||
|
if (channels_per_group <= 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t active_groups = (ne2 + channels_per_group - 1) / channels_per_group;
|
||||||
|
const int64_t total_work = active_groups * ne3;
|
||||||
|
const int64_t rows_per_write_group = et_rows_per_cacheline_group(ne0, sizeof(float));
|
||||||
|
|
||||||
|
for (int64_t work = thread_id; work < total_work; work += num_threads) {
|
||||||
|
const int64_t i3 = work / active_groups;
|
||||||
|
const int64_t group_idx = work % active_groups;
|
||||||
|
|
||||||
|
const int64_t channel_start = group_idx * channels_per_group;
|
||||||
|
int64_t channel_end = channel_start + channels_per_group;
|
||||||
|
if (channel_end > ne2) {
|
||||||
|
channel_end = ne2;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t channel_count = channel_end - channel_start;
|
||||||
|
if (channel_count <= 0) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
float sum = 0.0f;
|
||||||
|
float denom = 0.0f;
|
||||||
|
for (int64_t i2 = channel_start; i2 < channel_end; ++i2) {
|
||||||
|
for (int64_t i1 = 0; i1 < ne1; ++i1) {
|
||||||
|
const float * src_row = (const float *) ((const char *) src0_data + i3 * nb03 + i2 * nb02 + i1 * nb01);
|
||||||
|
for (int64_t i0 = 0; i0 < ne0; ++i0) {
|
||||||
|
sum += src_row[i0];
|
||||||
|
denom += 1.0f;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const float mean = et_fdiv(sum, denom);
|
||||||
|
|
||||||
|
float var_sum = 0.0f;
|
||||||
|
for (int64_t i2 = channel_start; i2 < channel_end; ++i2) {
|
||||||
|
for (int64_t i1 = 0; i1 < ne1; ++i1) {
|
||||||
|
const float * src_row = (const float *) ((const char *) src0_data + i3 * nb03 + i2 * nb02 + i1 * nb01);
|
||||||
|
for (int64_t i0 = 0; i0 < ne0; ++i0) {
|
||||||
|
const float centered = src_row[i0] - mean;
|
||||||
|
var_sum += centered * centered;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const float variance = et_fdiv(var_sum, denom);
|
||||||
|
const float scale = et_fdiv(1.0f, et_sqrtf(variance + eps));
|
||||||
|
|
||||||
|
if (ne0 % 16 == 0) {
|
||||||
|
for (int64_t i2 = channel_start; i2 < channel_end; ++i2) {
|
||||||
|
for (int64_t i1 = 0; i1 < ne1; ++i1) {
|
||||||
|
const float * src_row =
|
||||||
|
(const float *) ((const char *) src0_data + i3 * nb03 + i2 * nb02 + i1 * nb01);
|
||||||
|
float * dst_row = (float *) ((char *) dst_data + i3 * nb3 + i2 * nb2 + i1 * nb1);
|
||||||
|
for (int64_t i0 = 0; i0 < ne0; ++i0) {
|
||||||
|
dst_row[i0] = (src_row[i0] - mean) * scale;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
const int64_t total_rows_in_group = channel_count * ne1;
|
||||||
|
const int64_t total_write_groups = (total_rows_in_group + rows_per_write_group - 1) / rows_per_write_group;
|
||||||
|
|
||||||
|
for (int64_t write_group = 0; write_group < total_write_groups; ++write_group) {
|
||||||
|
const int64_t row_start = write_group * rows_per_write_group;
|
||||||
|
int64_t row_end = row_start + rows_per_write_group;
|
||||||
|
if (row_end > total_rows_in_group) {
|
||||||
|
row_end = total_rows_in_group;
|
||||||
|
}
|
||||||
|
|
||||||
|
for (int64_t row = row_start; row < row_end; ++row) {
|
||||||
|
const int64_t local_i2 = row / ne1;
|
||||||
|
const int64_t i1 = row % ne1;
|
||||||
|
const int64_t i2 = channel_start + local_i2;
|
||||||
|
|
||||||
|
const float * src_row =
|
||||||
|
(const float *) ((const char *) src0_data + i3 * nb03 + i2 * nb02 + i1 * nb01);
|
||||||
|
float * dst_row = (float *) ((char *) dst_data + i3 * nb3 + i2 * nb2 + i1 * nb1);
|
||||||
|
for (int64_t i0 = 0; i0 < ne0; ++i0) {
|
||||||
|
dst_row[i0] = (src_row[i0] - mean) * scale;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,130 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// IM2COL Kernel
|
||||||
|
// Baseline scalar implementation for:
|
||||||
|
// src1: [N, IC, IH, IW] -> dst: [N, OH, OW, IC*KH*KW] (2D)
|
||||||
|
// src1: [N, IC, IW] -> dst: [N, 1, OW, IC* KW] (1D)
|
||||||
|
//
|
||||||
|
// Work is distributed by row-groups so threads own cache-line-aligned chunks of
|
||||||
|
// destination rows even when ne0 is not cache aligned.
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "math_fp.h"
|
||||||
|
#include "platform.h"
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
static inline void im2col_store_elem(void * dst_base, enum ggml_type dst_type, int64_t idx, float value) {
|
||||||
|
if (dst_type == GGML_TYPE_F32) {
|
||||||
|
((float *) dst_base)[idx] = value;
|
||||||
|
} else {
|
||||||
|
((uint16_t *) dst_base)[idx] = fp32_to_fp16(value);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
static inline float im2col_load_src_elem(const void * src_base, enum ggml_type src_type, int64_t idx) {
|
||||||
|
if (src_type == GGML_TYPE_F32) {
|
||||||
|
return ((const float *) src_base)[idx];
|
||||||
|
}
|
||||||
|
|
||||||
|
return fp16_to_fp32(((const uint16_t *) src_base)[idx]);
|
||||||
|
}
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_binary_params * params, void * env) {
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
|
||||||
|
if (!kernel_env || params == 0 || ((uint64_t) params & 0x7) != 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
int num_threads = get_num_threads(kernel_env->shire_mask);
|
||||||
|
|
||||||
|
if (thread_id < 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct ggml_tensor * src0 = ¶ms->src0;
|
||||||
|
struct ggml_tensor * src1 = ¶ms->src1;
|
||||||
|
struct ggml_tensor * dst = ¶ms->dst;
|
||||||
|
|
||||||
|
if (!src1->data || !dst->data) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!((dst->type == GGML_TYPE_F32 && src1->type == GGML_TYPE_F32) ||
|
||||||
|
(dst->type == GGML_TYPE_F16 && (src1->type == GGML_TYPE_F16 || src1->type == GGML_TYPE_F32)))) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int32_t s0 = ((const int32_t *) dst->op_params)[0];
|
||||||
|
const int32_t s1 = ((const int32_t *) dst->op_params)[1];
|
||||||
|
const int32_t p0 = ((const int32_t *) dst->op_params)[2];
|
||||||
|
const int32_t p1 = ((const int32_t *) dst->op_params)[3];
|
||||||
|
const int32_t d0 = ((const int32_t *) dst->op_params)[4];
|
||||||
|
const int32_t d1 = ((const int32_t *) dst->op_params)[5];
|
||||||
|
const int32_t is_2d = ((const int32_t *) dst->op_params)[6];
|
||||||
|
|
||||||
|
const int64_t N = is_2d ? src1->ne[3] : src1->ne[2];
|
||||||
|
const int64_t IC = is_2d ? src1->ne[2] : src1->ne[1];
|
||||||
|
const int64_t IH = is_2d ? src1->ne[1] : 1;
|
||||||
|
const int64_t IW = src1->ne[0];
|
||||||
|
|
||||||
|
const int64_t KH = is_2d ? src0->ne[1] : 1;
|
||||||
|
const int64_t KW = src0->ne[0];
|
||||||
|
|
||||||
|
const int64_t OH = is_2d ? dst->ne[2] : 1;
|
||||||
|
const int64_t OW = dst->ne[1];
|
||||||
|
const int64_t row_elems = dst->ne[0];
|
||||||
|
const int64_t total_rows = OW * OH * N;
|
||||||
|
|
||||||
|
const size_t src_batch_stride = is_2d ? src1->nb[3] : src1->nb[2];
|
||||||
|
const size_t src_channel_stride = is_2d ? src1->nb[2] : src1->nb[1];
|
||||||
|
|
||||||
|
const size_t dst_row_stride = dst->nb[1];
|
||||||
|
const size_t dst_plane_stride = is_2d ? dst->nb[2] : 0;
|
||||||
|
const size_t dst_batch_stride = is_2d ? dst->nb[3] : dst->nb[2];
|
||||||
|
|
||||||
|
const int64_t dst_elem_size = (dst->type == GGML_TYPE_F32) ? (int64_t) sizeof(float) : (int64_t) sizeof(uint16_t);
|
||||||
|
const int64_t rows_per_group = et_rows_per_cacheline_group(row_elems, dst_elem_size);
|
||||||
|
const int64_t total_groups = (total_rows + rows_per_group - 1) / rows_per_group;
|
||||||
|
|
||||||
|
for (int64_t grp = thread_id; grp < total_groups; grp += num_threads) {
|
||||||
|
const int64_t row_start = grp * rows_per_group;
|
||||||
|
int64_t row_end = row_start + rows_per_group;
|
||||||
|
if (row_end > total_rows) {
|
||||||
|
row_end = total_rows;
|
||||||
|
}
|
||||||
|
|
||||||
|
for (int64_t row = row_start; row < row_end; ++row) {
|
||||||
|
const int64_t in = row / (OH * OW);
|
||||||
|
const int64_t rem = row % (OH * OW);
|
||||||
|
const int64_t ioh = rem / OW;
|
||||||
|
const int64_t iow = rem % OW;
|
||||||
|
|
||||||
|
void * dst_row = (char *) dst->data + in * dst_batch_stride + ioh * dst_plane_stride + iow * dst_row_stride;
|
||||||
|
|
||||||
|
for (int64_t iic = 0; iic < IC; ++iic) {
|
||||||
|
const void * src_channel = (const char *) src1->data + in * src_batch_stride + iic * src_channel_stride;
|
||||||
|
|
||||||
|
for (int64_t ikh = 0; ikh < KH; ++ikh) {
|
||||||
|
for (int64_t ikw = 0; ikw < KW; ++ikw) {
|
||||||
|
const int64_t iiw = iow * s0 + ikw * d0 - p0;
|
||||||
|
const int64_t iih = ioh * s1 + ikh * d1 - p1;
|
||||||
|
const int64_t dst_idx = iic * (KH * KW) + ikh * KW + ikw;
|
||||||
|
|
||||||
|
if (iiw < 0 || iiw >= IW || iih < 0 || iih >= IH) {
|
||||||
|
im2col_store_elem(dst_row, dst->type, dst_idx, 0.0f);
|
||||||
|
} else {
|
||||||
|
const int64_t src_idx = iih * IW + iiw;
|
||||||
|
const float value = im2col_load_src_elem(src_channel, src1->type, src_idx);
|
||||||
|
im2col_store_elem(dst_row, dst->type, dst_idx, value);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,237 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// L2 Norm F32 Kernel (L2 Normalization)
|
||||||
|
// y[i] = x[i] / max(sqrt(sum(x^2)), eps)
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "math_fp.h"
|
||||||
|
#include "platform.h"
|
||||||
|
|
||||||
|
#include <assert.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
#include <string.h>
|
||||||
|
|
||||||
|
// L2 Norm kernel parameters structure
|
||||||
|
struct ggml_et_l2_norm_params {
|
||||||
|
struct ggml_tensor src0; // F32 input tensor
|
||||||
|
struct ggml_tensor dst; // F32 output tensor
|
||||||
|
float eps; // Epsilon parameter for numerical stability
|
||||||
|
};
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_l2_norm_params * params, void * env) {
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
|
||||||
|
if (!kernel_env) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
int num_threads = get_num_threads(kernel_env->shire_mask);
|
||||||
|
|
||||||
|
if (thread_id < 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (params == 0 || ((uint64_t) params & 0x7) != 0) {
|
||||||
|
return -1; // Invalid pointer
|
||||||
|
}
|
||||||
|
|
||||||
|
struct ggml_tensor * src0 = ¶ms->src0;
|
||||||
|
struct ggml_tensor * dst = ¶ms->dst;
|
||||||
|
float eps = params->eps;
|
||||||
|
|
||||||
|
if (src0->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) {
|
||||||
|
return -1; // Unsupported type combination
|
||||||
|
}
|
||||||
|
|
||||||
|
float * src0_data = (float *) src0->data;
|
||||||
|
float * dst_data = (float *) dst->data;
|
||||||
|
|
||||||
|
if (!src0_data || !dst_data) {
|
||||||
|
return -1; // Null data pointer
|
||||||
|
}
|
||||||
|
|
||||||
|
if (eps < 0.0f) {
|
||||||
|
return -1; // Invalid epsilon
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t ne0 = dst->ne[0];
|
||||||
|
const int64_t ne1 = dst->ne[1];
|
||||||
|
const int64_t ne2 = dst->ne[2];
|
||||||
|
const int64_t ne3 = dst->ne[3];
|
||||||
|
|
||||||
|
const size_t nb0 = dst->nb[0], nb1 = dst->nb[1], nb2 = dst->nb[2], nb3 = dst->nb[3];
|
||||||
|
const size_t nb00 = src0->nb[0], nb01 = src0->nb[1], nb02 = src0->nb[2], nb03 = src0->nb[3];
|
||||||
|
|
||||||
|
(void) nb0;
|
||||||
|
(void) nb00;
|
||||||
|
|
||||||
|
if (src0->ne[0] != ne0 || src0->ne[1] != ne1 || src0->ne[2] != ne2 || src0->ne[3] != ne3) {
|
||||||
|
return -1; // Shape mismatch
|
||||||
|
}
|
||||||
|
|
||||||
|
const int32_t total_rows = (int32_t) (ne1 * ne2 * ne3);
|
||||||
|
const int shire_threads = SOC_MINIONS_PER_SHIRE * NUM_HARTS_PER_MINION;
|
||||||
|
|
||||||
|
if (total_rows >= shire_threads) {
|
||||||
|
// Row-parallel: each thread processes whole rows
|
||||||
|
for (int64_t i3 = 0; i3 < ne3; i3++) {
|
||||||
|
for (int64_t i2 = 0; i2 < ne2; i2++) {
|
||||||
|
for (int64_t i1 = thread_id; i1 < ne1; i1 += num_threads) {
|
||||||
|
const float * src_ptr =
|
||||||
|
(const float *) ((const char *) src0_data + i3 * nb03 + i2 * nb02 + i1 * nb01);
|
||||||
|
float * dst_ptr = (float *) ((char *) dst_data + i3 * nb3 + i2 * nb2 + i1 * nb1);
|
||||||
|
|
||||||
|
float zero = 0.0f;
|
||||||
|
__asm__ volatile("fbc.ps f10, %[z]\n" : : [z] "m"(zero) : "f10");
|
||||||
|
|
||||||
|
for (int32_t i0 = 0; i0 < (int32_t) ne0; i0 += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f11, %[x_vec]\n"
|
||||||
|
"fmadd.ps f10, f11, f11, f10\n"
|
||||||
|
:
|
||||||
|
: [x_vec] "m"(*(const float (*)[8]) & src_ptr[i0])
|
||||||
|
: "f10", "f11");
|
||||||
|
}
|
||||||
|
|
||||||
|
float sum_sq;
|
||||||
|
__asm__ __volatile__(
|
||||||
|
"fswizz.ps f1, f10, 0xB1 \n\t"
|
||||||
|
"fadd.ps f2, f10, f1, rne \n\t"
|
||||||
|
"fswizz.ps f3, f2, 0x4E \n\t"
|
||||||
|
"fadd.ps f4, f2, f3, rne \n\t"
|
||||||
|
"fmvz.x.ps t0, f4, 4 \n\t"
|
||||||
|
"fbcx.ps f5, t0 \n\t"
|
||||||
|
"fadd.ps %[vout], f4, f5, rne \n\t"
|
||||||
|
: [vout] "=f"(sum_sq)::"t0", "f1", "f2", "f3", "f4", "f5");
|
||||||
|
|
||||||
|
float l2_norm = et_powf(sum_sq, 0.5f);
|
||||||
|
if (l2_norm < eps) {
|
||||||
|
l2_norm = eps;
|
||||||
|
}
|
||||||
|
const float scale = et_fdiv(1.0f, l2_norm);
|
||||||
|
|
||||||
|
for (int32_t i0 = 0; i0 < (int32_t) ne0; i0 += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f11, %[x_vec]\n"
|
||||||
|
"fbc.ps f12, %[scale_ptr]\n"
|
||||||
|
"fmul.ps f13, f11, f12\n"
|
||||||
|
"fsw.ps f13, %[result]\n"
|
||||||
|
: [result] "=m"(*(float (*)[8]) & dst_ptr[i0])
|
||||||
|
: [x_vec] "m"(*(const float (*)[8]) & src_ptr[i0]), [scale_ptr] "m"(scale)
|
||||||
|
: "f11", "f12", "f13");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// Intra-row: threads within each shire cooperate via L2 SCP
|
||||||
|
int shire_tid = thread_id % shire_threads;
|
||||||
|
int threads_per_row = shire_threads / total_rows;
|
||||||
|
int my_row = shire_tid / threads_per_row;
|
||||||
|
int local_tid = shire_tid % threads_per_row;
|
||||||
|
int group_base = my_row * threads_per_row;
|
||||||
|
|
||||||
|
if (my_row >= total_rows) {
|
||||||
|
FENCE;
|
||||||
|
et_barrier(ET_BARRIER_SHIRE);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
int64_t i1 = my_row % ne1;
|
||||||
|
int64_t i2 = (my_row / ne1) % ne2;
|
||||||
|
int64_t i3 = my_row / (ne1 * ne2);
|
||||||
|
|
||||||
|
const float * src_ptr = (const float *) ((const char *) src0_data + i3 * nb03 + i2 * nb02 + i1 * nb01);
|
||||||
|
float * dst_ptr = (float *) ((char *) dst_data + i3 * nb3 + i2 * nb2 + i1 * nb1);
|
||||||
|
|
||||||
|
const int32_t elems_per_cl = 16;
|
||||||
|
int32_t total_cls = ((int32_t) ne0 + elems_per_cl - 1) / elems_per_cl;
|
||||||
|
int32_t cls_per_thread = (total_cls + threads_per_row - 1) / threads_per_row;
|
||||||
|
int32_t my_start = local_tid * cls_per_thread * elems_per_cl;
|
||||||
|
int32_t my_end = my_start + cls_per_thread * elems_per_cl;
|
||||||
|
if (my_end > (int32_t) ne0) {
|
||||||
|
my_end = (int32_t) ne0;
|
||||||
|
}
|
||||||
|
if (my_start >= (int32_t) ne0) {
|
||||||
|
my_start = 0;
|
||||||
|
my_end = 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
unsigned long saved_mask;
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(saved_mask));
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF");
|
||||||
|
|
||||||
|
// Phase 1: partial sum of squares
|
||||||
|
__asm__ volatile("fbci.pi f10, 0" ::: "f10");
|
||||||
|
for (int32_t i0 = my_start; i0 < my_end; i0 += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f11, %[x_vec]\n"
|
||||||
|
"fmadd.ps f10, f11, f11, f10\n"
|
||||||
|
:
|
||||||
|
: [x_vec] "m"(*(const float (*)[8]) & src_ptr[i0])
|
||||||
|
: "f10", "f11");
|
||||||
|
}
|
||||||
|
|
||||||
|
float partial_sum;
|
||||||
|
__asm__ __volatile__(
|
||||||
|
"fswizz.ps f1, f10, 0xB1 \n\t"
|
||||||
|
"fadd.ps f2, f10, f1, rne \n\t"
|
||||||
|
"fswizz.ps f3, f2, 0x4E \n\t"
|
||||||
|
"fadd.ps f4, f2, f3, rne \n\t"
|
||||||
|
"fmvz.x.ps t0, f4, 4 \n\t"
|
||||||
|
"fbcx.ps f5, t0 \n\t"
|
||||||
|
"fadd.ps %[vout], f4, f5, rne \n\t"
|
||||||
|
: [vout] "=f"(partial_sum)::"t0", "f1", "f2", "f3", "f4", "f5");
|
||||||
|
|
||||||
|
// Phase 2: L2SCP exchange
|
||||||
|
volatile float * my_slot = (volatile float *) et_shire_l2scp_local((uint64_t) shire_tid * 64);
|
||||||
|
*my_slot = partial_sum;
|
||||||
|
FENCE;
|
||||||
|
evict_to_l2((const void *) my_slot, 1, 64);
|
||||||
|
WAIT_CACHEOPS;
|
||||||
|
|
||||||
|
et_barrier(ET_BARRIER_SHIRE);
|
||||||
|
|
||||||
|
// Phase 3: all threads reduce + apply scale to own chunk
|
||||||
|
int workers = threads_per_row < total_cls ? threads_per_row : total_cls;
|
||||||
|
|
||||||
|
for (int t = 0; t < workers; t++) {
|
||||||
|
volatile float * slot = (volatile float *) et_shire_l2scp_local((uint64_t) (group_base + t) * 64);
|
||||||
|
evict_to_l2((const void *) slot, 1, 64);
|
||||||
|
}
|
||||||
|
WAIT_CACHEOPS;
|
||||||
|
|
||||||
|
float total_sum_sq = 0.0f;
|
||||||
|
for (int t = 0; t < workers; t++) {
|
||||||
|
volatile float * slot = (volatile float *) et_shire_l2scp_local((uint64_t) (group_base + t) * 64);
|
||||||
|
total_sum_sq += *slot;
|
||||||
|
}
|
||||||
|
|
||||||
|
float l2_norm = et_powf(total_sum_sq, 0.5f);
|
||||||
|
if (l2_norm < eps) {
|
||||||
|
l2_norm = eps;
|
||||||
|
}
|
||||||
|
const float scale = et_fdiv(1.0f, l2_norm);
|
||||||
|
|
||||||
|
if (my_start < my_end) {
|
||||||
|
uint32_t scale_bits;
|
||||||
|
__asm__ volatile("fmv.x.s %0, %1" : "=r"(scale_bits) : "f"(scale));
|
||||||
|
__asm__ volatile("fbcx.ps f13, %[sb]\n" : : [sb] "r"(scale_bits) : "f13");
|
||||||
|
|
||||||
|
for (int32_t i0 = my_start; i0 < my_end; i0 += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f12, %[x_vec]\n"
|
||||||
|
"fmul.ps f14, f12, f13\n"
|
||||||
|
"fsw.ps f14, %[result]\n"
|
||||||
|
: [result] "=m"(*(float (*)[8]) & dst_ptr[i0])
|
||||||
|
: [x_vec] "m"(*(const float (*)[8]) & src_ptr[i0])
|
||||||
|
: "f12", "f14");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(saved_mask));
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
/*-------------------------------------------------------------------------
|
||||||
|
* Independent Linker Script for GGML Bare Metal Kernels
|
||||||
|
*-------------------------------------------------------------------------
|
||||||
|
*/
|
||||||
|
|
||||||
|
/* Define maximum number of harts (threads) - simplified for bare metal */
|
||||||
|
MAX_HARTS = 2112;
|
||||||
|
|
||||||
|
SECTIONS
|
||||||
|
{
|
||||||
|
/* Start at the base address passed by -Wl,--defsym=BASE_ADDRESS=... */
|
||||||
|
. = BASE_ADDRESS;
|
||||||
|
|
||||||
|
/* Export entry point symbol for runtime compatibility */
|
||||||
|
KERNEL_UMODE_ENTRY = BASE_ADDRESS;
|
||||||
|
|
||||||
|
/* Initialization section - must come first */
|
||||||
|
.text.init :
|
||||||
|
{
|
||||||
|
*(.text.init)
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Align to cache line boundary */
|
||||||
|
. = ALIGN(64);
|
||||||
|
|
||||||
|
/* Main text section for code */
|
||||||
|
.text : {
|
||||||
|
*(.text)
|
||||||
|
*(.text.*)
|
||||||
|
}
|
||||||
|
. = ALIGN(64);
|
||||||
|
|
||||||
|
/* Data section with global pointer setup */
|
||||||
|
.data :
|
||||||
|
{
|
||||||
|
_data_start = .;
|
||||||
|
. = ALIGN(64);
|
||||||
|
|
||||||
|
/* Small data section and global pointer */
|
||||||
|
*(.sdata .sdata.*)
|
||||||
|
PROVIDE( __global_pointer$ = . + 0x800 );
|
||||||
|
|
||||||
|
/* Regular data */
|
||||||
|
*(.data .data.*)
|
||||||
|
. = ALIGN(64);
|
||||||
|
_data_end = .;
|
||||||
|
}
|
||||||
|
. = ALIGN(64);
|
||||||
|
|
||||||
|
/* BSS section for uninitialized data */
|
||||||
|
.bss(NOLOAD) :
|
||||||
|
{
|
||||||
|
_bss_start = .;
|
||||||
|
*(.sbss*);
|
||||||
|
*(.bss*);
|
||||||
|
. = ALIGN(64);
|
||||||
|
_bss_end = .;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Thread Local Storage (TLS) sections */
|
||||||
|
. = ALIGN(64);
|
||||||
|
.tdata :
|
||||||
|
{
|
||||||
|
*(.tdata*)
|
||||||
|
. = ALIGN(64);
|
||||||
|
}
|
||||||
|
__tdata_start = ADDR(.tdata);
|
||||||
|
|
||||||
|
.tbss : {
|
||||||
|
__tbss_start = .;
|
||||||
|
*(.tbss*)
|
||||||
|
}
|
||||||
|
. = . + SIZEOF(.tbss);
|
||||||
|
. = ALIGN(64);
|
||||||
|
__tbss_end = .;
|
||||||
|
|
||||||
|
/* TLS allocation area for all harts */
|
||||||
|
.tls-alloc ALIGN(64) (NOLOAD) : {
|
||||||
|
__tls_alloc_start = .;
|
||||||
|
. = . + (ABSOLUTE(__tbss_end) - ABSOLUTE(__tdata_start)) * MAX_HARTS;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* End of kernel image */
|
||||||
|
_end = .;
|
||||||
|
}
|
||||||
@@ -0,0 +1,299 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// ET Floating Point Math Library
|
||||||
|
// Provides ET hardware-specific math functions, FP16 conversion, and trig functions
|
||||||
|
// for bare metal kernels
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#ifndef MATH_FP_H
|
||||||
|
#define MATH_FP_H
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
//******************************************************************************
|
||||||
|
// ET Hardware Math Functions
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
// ET hardware division function (uses FRCP.PS instruction)
|
||||||
|
static inline float et_fdiv(float a, float b) {
|
||||||
|
float d;
|
||||||
|
unsigned long temp;
|
||||||
|
|
||||||
|
__asm__ volatile(
|
||||||
|
"mova.x.m %[temp] \n\t"
|
||||||
|
"mov.m.x m0, x0, 1 \n\t"
|
||||||
|
"frcp.ps %[d], %[b] \n\t"
|
||||||
|
"fmul.s %[d], %[d], %[a] \n\t"
|
||||||
|
"mova.m.x %[temp] \n\t"
|
||||||
|
: [temp] "=&r"(temp), [d] "=&f"(d)
|
||||||
|
: [a] "f"(a), [b] "f"(b));
|
||||||
|
|
||||||
|
return d;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Power function using ET hardware vector instructions
|
||||||
|
// Implements pow(base, exp) = exp(exp * ln(base)) using FLOG.PS and FEXP.PS
|
||||||
|
static inline float et_powf(float base, float exp) {
|
||||||
|
// Handle special cases
|
||||||
|
if (base <= 0.0f) {
|
||||||
|
if (base == 0.0f) {
|
||||||
|
if (exp > 0.0f) {
|
||||||
|
return 0.0f;
|
||||||
|
}
|
||||||
|
|
||||||
|
// For exp <= 0, return +infinity (IEEE 754: sign=0, exp=0xFF, mantissa=0)
|
||||||
|
union {
|
||||||
|
float f;
|
||||||
|
uint32_t i;
|
||||||
|
} inf = { .i = 0x7F800000 };
|
||||||
|
|
||||||
|
return inf.f;
|
||||||
|
}
|
||||||
|
|
||||||
|
// For negative base, return NaN (IEEE 754: exp=0xFF, mantissa!=0)
|
||||||
|
union {
|
||||||
|
float f;
|
||||||
|
uint32_t i;
|
||||||
|
} nan = { .i = 0x7FC00000 };
|
||||||
|
|
||||||
|
return nan.f;
|
||||||
|
}
|
||||||
|
if (base == 1.0f) {
|
||||||
|
return 1.0f;
|
||||||
|
}
|
||||||
|
if (exp == 0.0f) {
|
||||||
|
return 1.0f;
|
||||||
|
}
|
||||||
|
if (exp == 1.0f) {
|
||||||
|
return base;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Use ET hardware instructions following DNN library pattern:
|
||||||
|
// pow(base, exp) = exp(exp * ln(base))
|
||||||
|
float result;
|
||||||
|
unsigned long temp;
|
||||||
|
|
||||||
|
__asm__ volatile(
|
||||||
|
"mova.x.m %[temp] \n\t" // Save current mask state
|
||||||
|
"mov.m.x m0, x0, 1 \n\t" // Set mask register m0 to enable element 0
|
||||||
|
"flog.ps %[result], %[base] \n\t" // result = ln(base)
|
||||||
|
"fmul.s %[result], %[result], %[exp]\n\t" // result = ln(base) * exp
|
||||||
|
"fexp.ps %[result], %[result] \n\t" // result = exp(ln(base) * exp) = base^exp
|
||||||
|
"mova.m.x %[temp] \n\t" // Restore mask state
|
||||||
|
: [temp] "=&r"(temp), [result] "=&f"(result)
|
||||||
|
: [base] "f"(base), [exp] "f"(exp));
|
||||||
|
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Natural logarithm.
|
||||||
|
static inline float et_logf(float x) {
|
||||||
|
// Handle special cases
|
||||||
|
if (x < 0.0f) {
|
||||||
|
// Return NaN for negative input
|
||||||
|
union {
|
||||||
|
float f;
|
||||||
|
uint32_t i;
|
||||||
|
} nan = { .i = 0x7FC00000 };
|
||||||
|
|
||||||
|
return nan.f;
|
||||||
|
}
|
||||||
|
if (x == 0.0f) {
|
||||||
|
// Return -infinity for log(0)
|
||||||
|
union {
|
||||||
|
float f;
|
||||||
|
uint32_t i;
|
||||||
|
} inf = { .i = 0xFF800000 };
|
||||||
|
|
||||||
|
return inf.f;
|
||||||
|
}
|
||||||
|
if (x == 1.0f) {
|
||||||
|
return 0.0f;
|
||||||
|
}
|
||||||
|
|
||||||
|
float log2_result;
|
||||||
|
unsigned long temp;
|
||||||
|
|
||||||
|
__asm__ volatile(
|
||||||
|
"mova.x.m %[temp] \n\t" // Save current mask state
|
||||||
|
"mov.m.x m0, x0, 1 \n\t" // Set mask register m0 to enable element 0
|
||||||
|
"flog.ps %[result], %[x] \n\t" // result = log2(x)
|
||||||
|
"mova.m.x %[temp] \n\t" // Restore mask state
|
||||||
|
: [temp] "=&r"(temp), [result] "=&f"(log2_result)
|
||||||
|
: [x] "f"(x));
|
||||||
|
|
||||||
|
// Convert log2 to natural log: ln(x) = log2(x) * ln(2)
|
||||||
|
const float ln2 = 0.69314718055994530942f;
|
||||||
|
return log2_result * ln2;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Square root function implemented as et_powf(x, 0.5)
|
||||||
|
static inline float et_sqrtf(float x) {
|
||||||
|
// Handle special cases
|
||||||
|
if (x < 0.0f) {
|
||||||
|
// Return NaN for negative input (IEEE 754: exp=0xFF, mantissa!=0)
|
||||||
|
union {
|
||||||
|
float f;
|
||||||
|
uint32_t i;
|
||||||
|
} nan = { .i = 0x7FC00000 };
|
||||||
|
|
||||||
|
return nan.f;
|
||||||
|
}
|
||||||
|
if (x == 0.0f) {
|
||||||
|
return 0.0f;
|
||||||
|
}
|
||||||
|
|
||||||
|
return et_powf(x, 0.5f);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Base-2 exponential: returns 2^x using the ET hardware FEXP.PS instruction.
|
||||||
|
// No base conversion, no special-case clamping — this is the raw hardware op
|
||||||
|
// with just the mask save/restore wrapper. Caller is responsible for ensuring
|
||||||
|
// x is in a range that produces a useful result (roughly [-126, 128] for fp32).
|
||||||
|
static inline float __attribute__((always_inline)) et_exp2f(float x) {
|
||||||
|
unsigned long old_mask;
|
||||||
|
float out;
|
||||||
|
__asm__ volatile(
|
||||||
|
"mova.x.m %[ms] \n\t"
|
||||||
|
"mov.m.x m0, x0, 1 \n\t"
|
||||||
|
"fexp.ps %[out], %[x] \n\t"
|
||||||
|
"mova.m.x %[ms] \n\t"
|
||||||
|
: [ms] "=&r"(old_mask), [out] "=&f"(out)
|
||||||
|
: [x] "f"(x));
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Exponential function using ET hardware FEXP.PS instruction
|
||||||
|
// Note: FEXP.PS computes 2^x, so we need to convert: exp(x) = 2^(x * log2(e))
|
||||||
|
static inline float et_expf(float x) {
|
||||||
|
// Handle special cases
|
||||||
|
if (x > 88.0f) {
|
||||||
|
// For x > 88, exp(x) would overflow, return +infinity
|
||||||
|
union {
|
||||||
|
float f;
|
||||||
|
uint32_t i;
|
||||||
|
} inf = { .i = 0x7F800000 };
|
||||||
|
|
||||||
|
return inf.f;
|
||||||
|
}
|
||||||
|
if (x < -87.0f) {
|
||||||
|
// For x < -87, exp(x) is essentially 0
|
||||||
|
return 0.0f;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Convert to base-2 exponent: x * log2(e)
|
||||||
|
const float log2e = 1.4426950408889634f; // log2(e)
|
||||||
|
float x_log2e = x * log2e;
|
||||||
|
|
||||||
|
// Use ET hardware instruction: fexp.ps computes 2^x
|
||||||
|
float result;
|
||||||
|
unsigned long temp;
|
||||||
|
|
||||||
|
__asm__ volatile(
|
||||||
|
"mova.x.m %[temp] \n\t" // Save current mask state
|
||||||
|
"mov.m.x m0, x0, 1 \n\t" // Set mask register m0 to enable element 0
|
||||||
|
"fexp.ps %[result], %[x_log2e] \n\t" // result = 2^(x * log2(e)) = exp(x)
|
||||||
|
"mova.m.x %[temp] \n\t" // Restore mask state
|
||||||
|
: [temp] "=&r"(temp), [result] "=&f"(result)
|
||||||
|
: [x_log2e] "f"(x_log2e));
|
||||||
|
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
//******************************************************************************
|
||||||
|
// Trigonometric Functions
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
// FSIN.PS
|
||||||
|
|
||||||
|
// Sine function using Taylor series
|
||||||
|
static inline float et_sinf(float x) {
|
||||||
|
const float pi = 3.14159265358979323846f;
|
||||||
|
const float two_pi = 6.28318530717958647693f;
|
||||||
|
const float pi_over_2 = 1.57079632679489661923f;
|
||||||
|
|
||||||
|
if (x > pi || x < -pi) {
|
||||||
|
float cycles = x * et_fdiv(1.0f, two_pi);
|
||||||
|
int n = (int) cycles;
|
||||||
|
if (x < 0.0f) {
|
||||||
|
n--; // Floor for negative
|
||||||
|
}
|
||||||
|
x = x - (float) n * two_pi;
|
||||||
|
}
|
||||||
|
|
||||||
|
// sin(x) = sin(π - x) for x in [π/2, π]
|
||||||
|
// sin(x) = -sin(-π - x) for x in [-π, -π/2]
|
||||||
|
int negate = 0;
|
||||||
|
if (x > pi_over_2) {
|
||||||
|
x = pi - x;
|
||||||
|
} else if (x < -pi_over_2) {
|
||||||
|
x = -pi - x;
|
||||||
|
negate = 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
// sin(x) ≈ x - x^3/3! + x^5/5! - x^7/7! + x^9/9! - x^11/11!
|
||||||
|
const float x2 = x * x;
|
||||||
|
const float x3 = x2 * x;
|
||||||
|
const float x5 = x3 * x2;
|
||||||
|
const float x7 = x5 * x2;
|
||||||
|
const float x9 = x7 * x2;
|
||||||
|
const float x11 = x9 * x2;
|
||||||
|
|
||||||
|
float result = x - x3 * et_fdiv(1.0f, 6.0f) // x^3/3!
|
||||||
|
+ x5 * et_fdiv(1.0f, 120.0f) // x^5/5!
|
||||||
|
- x7 * et_fdiv(1.0f, 5040.0f) // x^7/7!
|
||||||
|
+ x9 * et_fdiv(1.0f, 362880.0f) // x^9/9!
|
||||||
|
- x11 * et_fdiv(1.0f, 39916800.0f); // x^11/11!
|
||||||
|
|
||||||
|
return negate ? -result : result;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Cosine function using identity cos(x) = sin(x + π/2)
|
||||||
|
static inline float et_cosf(float x) {
|
||||||
|
const float pi_over_2 = 1.57079632679489661923f;
|
||||||
|
return et_sinf(x + pi_over_2);
|
||||||
|
}
|
||||||
|
|
||||||
|
//******************************************************************************
|
||||||
|
// FP16 <-> FP32 Conversion Functions
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
// Convert FP16 (IEEE 754 half precision) to FP32 (single precision)
|
||||||
|
// Uses ET hardware FCVT.PS.F16 instruction for accurate conversion
|
||||||
|
static inline float fp16_to_fp32(uint16_t h) {
|
||||||
|
float result;
|
||||||
|
unsigned long temp;
|
||||||
|
uint32_t raw = (uint32_t) h;
|
||||||
|
|
||||||
|
__asm__ volatile(
|
||||||
|
"mova.x.m %[temp] \n\t" // Save current mask state
|
||||||
|
"mov.m.x m0, x0, 1 \n\t" // Set mask register m0 to enable element 0
|
||||||
|
"fbcx.ps %[result], %[raw] \n\t" // Broadcast raw FP16 bits into vector register
|
||||||
|
"fcvt.ps.f16 %[result], %[result] \n\t" // Convert FP16 to FP32
|
||||||
|
"mova.m.x %[temp] \n\t" // Restore mask state
|
||||||
|
: [temp] "=&r"(temp), [result] "=&f"(result)
|
||||||
|
: [raw] "r"(raw));
|
||||||
|
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Convert FP32 (single precision) to FP16 (IEEE 754 half precision)
|
||||||
|
// Uses ET hardware FCVT.F16.PS instruction for accurate conversion
|
||||||
|
static inline uint16_t fp32_to_fp16(float f) {
|
||||||
|
float result_f;
|
||||||
|
unsigned long temp;
|
||||||
|
|
||||||
|
__asm__ volatile(
|
||||||
|
"mova.x.m %[temp] \n\t" // Save current mask state
|
||||||
|
"mov.m.x m0, x0, 1 \n\t" // Set mask register m0 to enable element 0
|
||||||
|
"fcvt.f16.ps %[result], %[f] \n\t" // Convert FP32 to FP16 (result in lower 16 bits)
|
||||||
|
"mova.m.x %[temp] \n\t" // Restore mask state
|
||||||
|
: [temp] "=&r"(temp), [result] "=&f"(result_f)
|
||||||
|
: [f] "f"(f));
|
||||||
|
|
||||||
|
// Extract lower 16 bits containing the FP16 value
|
||||||
|
// The instruction zero-extends to 32 bits, so upper 16 bits are 0
|
||||||
|
uint32_t result_bits = *(uint32_t *) &result_f;
|
||||||
|
return (uint16_t) result_bits;
|
||||||
|
}
|
||||||
|
|
||||||
|
#endif // MATH_FP_H
|
||||||
@@ -0,0 +1,220 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// MEAN F32 Kernel
|
||||||
|
// Row-wise mean reduction: dst[0, i1, i2, i3] = mean(src0[0..ne00-1, i1, i2, i3])
|
||||||
|
//
|
||||||
|
// Modes:
|
||||||
|
// - total_rows >= shire_threads: row-parallel, each thread handles whole rows.
|
||||||
|
// - total_rows < shire_threads: intra-row reduction within a shire. Threads
|
||||||
|
// within a shire cooperate via shire-local L2 SCP slots. All shires
|
||||||
|
// duplicate the work because L2 SCP is per-shire (no cross-shire coherency).
|
||||||
|
//
|
||||||
|
// ne00 may be any positive size and rows may have any 4-byte alignment. We
|
||||||
|
// take the 8-wide vector path only when the row pointer is 32B-aligned and
|
||||||
|
// fall back to scalar for the leftover tail (or for the entire row when the
|
||||||
|
// row start is not 32B-aligned).
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "math_fp.h"
|
||||||
|
#include "platform.h"
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
struct ggml_et_mean_params {
|
||||||
|
struct ggml_tensor src0; // F32 input [ne00, ne01, ne02, ne03]
|
||||||
|
struct ggml_tensor dst; // F32 output [1, ne01, ne02, ne03]
|
||||||
|
};
|
||||||
|
|
||||||
|
// Sum a contiguous F32 slice [base+i_lo, base+i_hi). Uses the 8-wide vector
|
||||||
|
// path only when `base + i_lo` is 32B-aligned; the tail (and the whole slice
|
||||||
|
// when misaligned) is summed with scalar fadd.s.
|
||||||
|
static inline float partial_sum_slice(const float * base, int32_t i_lo, int32_t i_hi) {
|
||||||
|
if (i_lo >= i_hi) {
|
||||||
|
return 0.0f;
|
||||||
|
}
|
||||||
|
|
||||||
|
const float * p = base + i_lo;
|
||||||
|
int32_t n = i_hi - i_lo;
|
||||||
|
float acc = 0.0f;
|
||||||
|
int32_t i = 0;
|
||||||
|
|
||||||
|
if (n >= 8 && (((uintptr_t) p) & 31) == 0) {
|
||||||
|
float zero = 0.0f;
|
||||||
|
__asm__ volatile("fbc.ps f10, %[z]\n" : : [z] "m"(zero) : "f10");
|
||||||
|
|
||||||
|
for (; i + 8 <= n; i += 8) {
|
||||||
|
__asm__ volatile(
|
||||||
|
"flw.ps f11, %[x]\n"
|
||||||
|
"fadd.ps f10, f10, f11\n"
|
||||||
|
:
|
||||||
|
: [x] "m"(*(const float (*)[8]) & p[i])
|
||||||
|
: "f10", "f11");
|
||||||
|
}
|
||||||
|
|
||||||
|
float vec_sum;
|
||||||
|
__asm__ __volatile__(
|
||||||
|
"fswizz.ps f1, f10, 0xB1 \n\t"
|
||||||
|
"fadd.ps f2, f10, f1, rne \n\t"
|
||||||
|
"fswizz.ps f3, f2, 0x4E \n\t"
|
||||||
|
"fadd.ps f4, f2, f3, rne \n\t"
|
||||||
|
"fmvz.x.ps t0, f4, 4 \n\t"
|
||||||
|
"fbcx.ps f5, t0 \n\t"
|
||||||
|
"fadd.ps %[vout], f4, f5, rne \n\t"
|
||||||
|
: [vout] "=f"(vec_sum)::"t0", "f1", "f2", "f3", "f4", "f5");
|
||||||
|
acc = vec_sum;
|
||||||
|
}
|
||||||
|
|
||||||
|
for (; i < n; i++) {
|
||||||
|
acc += p[i];
|
||||||
|
}
|
||||||
|
return acc;
|
||||||
|
}
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_mean_params * params, void * env) {
|
||||||
|
kernel_environment_t * kernel_env = (kernel_environment_t *) env;
|
||||||
|
|
||||||
|
if (!kernel_env) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
int thread_id = get_relative_thread_id(kernel_env->shire_mask);
|
||||||
|
int num_threads = get_num_threads(kernel_env->shire_mask);
|
||||||
|
|
||||||
|
if (thread_id < 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (params == 0 || ((uint64_t) params & 0x7) != 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct ggml_tensor * src0 = ¶ms->src0;
|
||||||
|
struct ggml_tensor * dst = ¶ms->dst;
|
||||||
|
|
||||||
|
if (src0->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
float * src0_data = (float *) src0->data;
|
||||||
|
float * dst_data = (float *) dst->data;
|
||||||
|
|
||||||
|
if (!src0_data || !dst_data) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int64_t ne00 = src0->ne[0];
|
||||||
|
const int64_t ne01 = src0->ne[1];
|
||||||
|
const int64_t ne02 = src0->ne[2];
|
||||||
|
const int64_t ne03 = src0->ne[3];
|
||||||
|
|
||||||
|
const size_t nb01 = src0->nb[1];
|
||||||
|
const size_t nb02 = src0->nb[2];
|
||||||
|
const size_t nb03 = src0->nb[3];
|
||||||
|
|
||||||
|
const size_t nb1 = dst->nb[1];
|
||||||
|
const size_t nb2 = dst->nb[2];
|
||||||
|
const size_t nb3 = dst->nb[3];
|
||||||
|
|
||||||
|
if (ne00 <= 0) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
const int32_t total_rows = (int32_t) (ne01 * ne02 * ne03);
|
||||||
|
const int shire_threads = SOC_MINIONS_PER_SHIRE * NUM_HARTS_PER_MINION;
|
||||||
|
const float inv_ne00 = et_fdiv(1.0f, (float) (int32_t) ne00);
|
||||||
|
|
||||||
|
// Row-parallel: each thread owns whole rows.
|
||||||
|
if (total_rows >= shire_threads) {
|
||||||
|
for (int64_t ir = thread_id; ir < total_rows; ir += num_threads) {
|
||||||
|
const int64_t i03 = ir / (ne02 * ne01);
|
||||||
|
const int64_t i02 = (ir - i03 * ne02 * ne01) / ne01;
|
||||||
|
const int64_t i01 = ir - i03 * ne02 * ne01 - i02 * ne01;
|
||||||
|
|
||||||
|
const float * src_row = (const float *) ((const char *) src0_data + i01 * nb01 + i02 * nb02 + i03 * nb03);
|
||||||
|
float * dst_ptr = (float *) ((char *) dst_data + i01 * nb1 + i02 * nb2 + i03 * nb3);
|
||||||
|
|
||||||
|
float row_sum = partial_sum_slice(src_row, 0, (int32_t) ne00);
|
||||||
|
atomic_store_f32(dst_ptr, row_sum * inv_ne00);
|
||||||
|
}
|
||||||
|
// Shire co-work
|
||||||
|
} else {
|
||||||
|
int shire_tid = thread_id % shire_threads;
|
||||||
|
int threads_per_row = shire_threads / total_rows;
|
||||||
|
int my_row = shire_tid / threads_per_row;
|
||||||
|
int local_tid = shire_tid % threads_per_row;
|
||||||
|
int group_base = my_row * threads_per_row;
|
||||||
|
|
||||||
|
if (my_row >= total_rows) {
|
||||||
|
FENCE;
|
||||||
|
et_barrier(ET_BARRIER_SHIRE);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
int64_t i1 = my_row % ne01;
|
||||||
|
int64_t i2 = (my_row / ne01) % ne02;
|
||||||
|
int64_t i3 = my_row / (ne01 * ne02);
|
||||||
|
|
||||||
|
const float * src_ptr = (const float *) ((const char *) src0_data + i3 * nb03 + i2 * nb02 + i1 * nb01);
|
||||||
|
float * dst_ptr = (float *) ((char *) dst_data + i3 * nb3 + i2 * nb2 + i1 * nb1);
|
||||||
|
|
||||||
|
// Chunk size in elements, rounded up to a multiple of 8 so that every
|
||||||
|
// thread's slice start stays 32B-aligned relative to src_ptr (which
|
||||||
|
// matters for the vector path inside partial_sum_slice).
|
||||||
|
int32_t chunk = ((int32_t) ne00 + threads_per_row - 1) / threads_per_row;
|
||||||
|
chunk = (chunk + 7) & ~7;
|
||||||
|
if (chunk < 8) {
|
||||||
|
chunk = 8;
|
||||||
|
}
|
||||||
|
|
||||||
|
int32_t my_start = local_tid * chunk;
|
||||||
|
int32_t my_end = my_start + chunk;
|
||||||
|
if (my_end > (int32_t) ne00) {
|
||||||
|
my_end = (int32_t) ne00;
|
||||||
|
}
|
||||||
|
if (my_start > (int32_t) ne00) {
|
||||||
|
my_start = my_end = (int32_t) ne00;
|
||||||
|
}
|
||||||
|
|
||||||
|
int workers = ((int32_t) ne00 + chunk - 1) / chunk;
|
||||||
|
if (workers > threads_per_row) {
|
||||||
|
workers = threads_per_row;
|
||||||
|
}
|
||||||
|
|
||||||
|
unsigned long saved_mask;
|
||||||
|
__asm__ volatile("mova.x.m %0" : "=r"(saved_mask));
|
||||||
|
__asm__ volatile("mov.m.x m0, x0, 0xFF");
|
||||||
|
|
||||||
|
float partial_sum = partial_sum_slice(src_ptr, my_start, my_end);
|
||||||
|
|
||||||
|
// Publish partial to shire-local L2 SCP slot (64B per slot, one per
|
||||||
|
// hart). evict_to_l2 is required on the WRITER because scalar stores
|
||||||
|
// land in L1D first; readers must also evict before reading.
|
||||||
|
volatile float * my_slot = (volatile float *) et_shire_l2scp_local((uint64_t) shire_tid * 64);
|
||||||
|
*my_slot = partial_sum;
|
||||||
|
FENCE;
|
||||||
|
evict_to_l2((const void *) my_slot, 1, 64);
|
||||||
|
WAIT_CACHEOPS;
|
||||||
|
|
||||||
|
et_barrier(ET_BARRIER_SHIRE);
|
||||||
|
|
||||||
|
if (local_tid == 0) {
|
||||||
|
// Reader-side evictions for every contributing peer slot.
|
||||||
|
for (int t = 0; t < workers; t++) {
|
||||||
|
volatile float * slot = (volatile float *) et_shire_l2scp_local((uint64_t) (group_base + t) * 64);
|
||||||
|
evict_to_l2((const void *) slot, 1, 64);
|
||||||
|
}
|
||||||
|
WAIT_CACHEOPS;
|
||||||
|
|
||||||
|
float total_sum = 0.0f;
|
||||||
|
for (int t = 0; t < workers; t++) {
|
||||||
|
volatile float * slot = (volatile float *) et_shire_l2scp_local((uint64_t) (group_base + t) * 64);
|
||||||
|
total_sum += *slot;
|
||||||
|
}
|
||||||
|
atomic_store_f32(dst_ptr, total_sum * inv_ne00);
|
||||||
|
}
|
||||||
|
|
||||||
|
__asm__ volatile("mova.m.x %0" ::"r"(saved_mask));
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,181 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// Memory Operations Kernel — tensor_store based memset
|
||||||
|
//
|
||||||
|
// Uses the tensor engine's store path (bypasses L1+L2 caches) to achieve hiher
|
||||||
|
// performance. Unrolled vector writes can write at ~25GB/s and tensor writes
|
||||||
|
// can so ~71 GB/s. Only even harts (hart 0 per minion) participate, as due to
|
||||||
|
// hardware design (only thye have matrix engine access and co-op stores seems
|
||||||
|
// slower)
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "platform.h"
|
||||||
|
#include "tensor.h"
|
||||||
|
|
||||||
|
#include <etsoc/common/utils.h>
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
// Operation identifiers for memops kernel
|
||||||
|
enum ggml_et_memop_type {
|
||||||
|
GGML_ET_MEMOP_MEMSET = 0,
|
||||||
|
};
|
||||||
|
|
||||||
|
// Memset operation parameters (must match host-side struct in ggml-et-memops.cpp)
|
||||||
|
struct memset_params {
|
||||||
|
uint32_t op_type;
|
||||||
|
uint32_t value;
|
||||||
|
void * dst_ptr;
|
||||||
|
size_t size;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Fill all 32 f-regs with a replicated byte pattern
|
||||||
|
static inline void __attribute__((always_inline)) fill_fregs(uint32_t fill32) {
|
||||||
|
register uint64_t val __asm__("a2") = fill32;
|
||||||
|
__asm__ __volatile__(
|
||||||
|
"fbcx.ps f0, %[v]\n\t"
|
||||||
|
"fbcx.ps f1, %[v]\n\t"
|
||||||
|
"fbcx.ps f2, %[v]\n\t"
|
||||||
|
"fbcx.ps f3, %[v]\n\t"
|
||||||
|
"fbcx.ps f4, %[v]\n\t"
|
||||||
|
"fbcx.ps f5, %[v]\n\t"
|
||||||
|
"fbcx.ps f6, %[v]\n\t"
|
||||||
|
"fbcx.ps f7, %[v]\n\t"
|
||||||
|
"fbcx.ps f8, %[v]\n\t"
|
||||||
|
"fbcx.ps f9, %[v]\n\t"
|
||||||
|
"fbcx.ps f10, %[v]\n\t"
|
||||||
|
"fbcx.ps f11, %[v]\n\t"
|
||||||
|
"fbcx.ps f12, %[v]\n\t"
|
||||||
|
"fbcx.ps f13, %[v]\n\t"
|
||||||
|
"fbcx.ps f14, %[v]\n\t"
|
||||||
|
"fbcx.ps f15, %[v]\n\t"
|
||||||
|
"fbcx.ps f16, %[v]\n\t"
|
||||||
|
"fbcx.ps f17, %[v]\n\t"
|
||||||
|
"fbcx.ps f18, %[v]\n\t"
|
||||||
|
"fbcx.ps f19, %[v]\n\t"
|
||||||
|
"fbcx.ps f20, %[v]\n\t"
|
||||||
|
"fbcx.ps f21, %[v]\n\t"
|
||||||
|
"fbcx.ps f22, %[v]\n\t"
|
||||||
|
"fbcx.ps f23, %[v]\n\t"
|
||||||
|
"fbcx.ps f24, %[v]\n\t"
|
||||||
|
"fbcx.ps f25, %[v]\n\t"
|
||||||
|
"fbcx.ps f26, %[v]\n\t"
|
||||||
|
"fbcx.ps f27, %[v]\n\t"
|
||||||
|
"fbcx.ps f28, %[v]\n\t"
|
||||||
|
"fbcx.ps f29, %[v]\n\t"
|
||||||
|
"fbcx.ps f30, %[v]\n\t"
|
||||||
|
"fbcx.ps f31, %[v]\n\t" ::[v] "r"(val)
|
||||||
|
: "f0", "f1", "f2", "f3", "f4", "f5", "f6", "f7", "f8", "f9", "f10", "f11", "f12", "f13", "f14", "f15", "f16",
|
||||||
|
"f17", "f18", "f19", "f20", "f21", "f22", "f23", "f24", "f25", "f26", "f27", "f28", "f29", "f30", "f31");
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fill a partial region [start, end) using tensor_store for 16-byte-aligned
|
||||||
|
// chunks and byte stores for any remainder < 16 bytes.
|
||||||
|
// Assumes f-regs are already loaded with the fill pattern.
|
||||||
|
static void memset_tail(uint8_t * start, uint8_t * end, uint8_t val) {
|
||||||
|
uint8_t * cur = start;
|
||||||
|
|
||||||
|
// Full 64-byte rows via tensor_store (up to 16 at a time = 1KB)
|
||||||
|
while (cur + 64 <= end) {
|
||||||
|
size_t rows = (end - cur) / 64;
|
||||||
|
if (rows > 16) {
|
||||||
|
rows = 16;
|
||||||
|
}
|
||||||
|
tensor_store(0, 0, 3, rows - 1, (uintptr_t) cur, 0, 64);
|
||||||
|
cur += rows * 64;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Remaining 16-byte aligned chunk (16, 32, or 48 bytes)
|
||||||
|
if (cur + 16 <= end) {
|
||||||
|
size_t cols = (end - cur) / 16;
|
||||||
|
tensor_store(0, 0, cols - 1, 0, (uintptr_t) cur, 0, 64);
|
||||||
|
cur += cols * 16;
|
||||||
|
}
|
||||||
|
|
||||||
|
tensor_wait(TENSOR_STORE_WAIT);
|
||||||
|
|
||||||
|
// Final < 16 bytes with byte stores
|
||||||
|
while (cur < end) {
|
||||||
|
*(volatile uint8_t *) cur = val;
|
||||||
|
cur++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#define ALIGN_UP(ptr, align) ((uint8_t *) (((uintptr_t) (ptr) + (align) - 1) & ~((uintptr_t) (align) - 1)))
|
||||||
|
|
||||||
|
int entry_point(struct memset_params * params, kernel_environment_t * env) {
|
||||||
|
uint64_t hart_id = get_hart_id();
|
||||||
|
|
||||||
|
// Only even harts have tensor engine access
|
||||||
|
if (hart_id & 1) {
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!params || ((uintptr_t) params & 0x7) != 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (params->op_type != GGML_ET_MEMOP_MEMSET) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
uint8_t * dst = (uint8_t *) params->dst_ptr;
|
||||||
|
size_t size = params->size;
|
||||||
|
|
||||||
|
if (!dst || size == 0) {
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Dynamic hart count from shire_mask
|
||||||
|
int num_even_harts = manual_popcountll(env->shire_mask) * SOC_MINIONS_PER_SHIRE;
|
||||||
|
|
||||||
|
// global_id: shire * 32 + minion (for even harts)
|
||||||
|
uint64_t global_id = ((hart_id >> 6) << 5) + ((hart_id >> 1) & 0x1F);
|
||||||
|
|
||||||
|
uint8_t val = params->value & 0xFF;
|
||||||
|
uint32_t fill32 = val | ((uint32_t) val << 8) | ((uint32_t) val << 16) | ((uint32_t) val << 24);
|
||||||
|
|
||||||
|
uint8_t * end = dst + size;
|
||||||
|
|
||||||
|
setup_cache_scp();
|
||||||
|
CLEAR_TENSOR_ERROR;
|
||||||
|
fill_fregs(fill32);
|
||||||
|
|
||||||
|
// Align to 16 bytes (tensor_store minimum alignment)
|
||||||
|
uint8_t * base = ALIGN_UP(dst, 16);
|
||||||
|
if (base > end) {
|
||||||
|
base = end;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Hart 0 handles head bytes before alignment
|
||||||
|
if (global_id == 0) {
|
||||||
|
volatile uint8_t * p = dst;
|
||||||
|
while (p < (volatile uint8_t *) base) {
|
||||||
|
*p++ = val;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Bulk: 1KB blocks distributed across all harts (base is already 16-byte aligned)
|
||||||
|
size_t aligned_size = end - base;
|
||||||
|
size_t total_blocks = aligned_size / 1024;
|
||||||
|
|
||||||
|
if (total_blocks > 0) {
|
||||||
|
size_t blocks_per_hart = total_blocks / num_even_harts;
|
||||||
|
size_t extra = total_blocks % num_even_harts;
|
||||||
|
size_t my_start = blocks_per_hart * global_id + (global_id < extra ? global_id : extra);
|
||||||
|
size_t my_count = blocks_per_hart + (global_id < extra ? 1 : 0);
|
||||||
|
|
||||||
|
uint8_t * addr = base + my_start * 1024;
|
||||||
|
for (size_t b = 0; b < my_count; b++) {
|
||||||
|
tensor_store(0, 0, 3, 15, (uintptr_t) addr, 0, 64);
|
||||||
|
addr += 1024;
|
||||||
|
}
|
||||||
|
tensor_wait(TENSOR_STORE_WAIT);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Hart 0 handles the tail after the last full 1KB block
|
||||||
|
if (global_id == 0) {
|
||||||
|
memset_tail(base + total_blocks * 1024, end, val);
|
||||||
|
}
|
||||||
|
|
||||||
|
FENCE;
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
@@ -0,0 +1,358 @@
|
|||||||
|
//******************************************************************************
|
||||||
|
// MUL_MAT Kernel
|
||||||
|
// Matrix multiplication: C[M,N] = A[M,K] * B[K,N]
|
||||||
|
//******************************************************************************
|
||||||
|
|
||||||
|
#include "block_ops.h"
|
||||||
|
#include "ggml_tensor.h"
|
||||||
|
#include "math_fp.h"
|
||||||
|
#include "platform.h"
|
||||||
|
#include "quants.h"
|
||||||
|
|
||||||
|
#include <stdint.h>
|
||||||
|
|
||||||
|
#define STRIDE_M 2048 /* 32 shires x 32 minions x 2 harts */
|
||||||
|
#define STRIDE_M_KSPLIT 1024 /* 32 shires x 32 minions (both harts share rows) */
|
||||||
|
#define KSPLIT_MIN_K_BLOCKS 256 /* K >= 8192 elements */
|
||||||
|
#define KSPLIT_SMALL_ROWS_K_BLOCKS 64 /* K >= 2048 elements for very small M */
|
||||||
|
#define KSPLIT_MAX_ROWS 8 /* max rows per minion for K-split */
|
||||||
|
#define TILE_KB 256 /* K-tile size in Q4_0 blocks (8192 elems, 32KB B data) */
|
||||||
|
#define KSPLIT_GROUP_ROWS 4
|
||||||
|
#define SIMPLE_X2_ROWS 2
|
||||||
|
|
||||||
|
int entry_point(struct ggml_et_binary_params * params, void * env) {
|
||||||
|
uint64_t hart_id = get_hart_id();
|
||||||
|
|
||||||
|
// Matrix dimensions
|
||||||
|
const int64_t K = params->src0.ne[0];
|
||||||
|
const int64_t M = params->src0.ne[1];
|
||||||
|
const int64_t N = params->src1.ne[1];
|
||||||
|
const int64_t ne02 = params->src0.ne[2];
|
||||||
|
const int64_t ne03 = params->src0.ne[3];
|
||||||
|
const int64_t ne12 = params->src1.ne[2];
|
||||||
|
const int64_t ne13 = params->src1.ne[3];
|
||||||
|
|
||||||
|
// Strides (in bytes)
|
||||||
|
const size_t nb01 = params->src0.nb[1];
|
||||||
|
const size_t nb02 = params->src0.nb[2];
|
||||||
|
const size_t nb03 = params->src0.nb[3];
|
||||||
|
|
||||||
|
const size_t nb11 = params->src1.nb[1];
|
||||||
|
const size_t nb12 = params->src1.nb[2];
|
||||||
|
const size_t nb13 = params->src1.nb[3];
|
||||||
|
|
||||||
|
const size_t nbd1 = params->dst.nb[1];
|
||||||
|
const size_t nbd2 = params->dst.nb[2];
|
||||||
|
const size_t nbd3 = params->dst.nb[3];
|
||||||
|
|
||||||
|
// Q4_0 block size is 32
|
||||||
|
const int64_t K_blocks = K / 32;
|
||||||
|
const int use_simple_x2 = ((nb01 & 31) == 0);
|
||||||
|
|
||||||
|
// Broadcasting ratios
|
||||||
|
const int64_t r2 = ne12 / ne02;
|
||||||
|
const int64_t r3 = ne13 / ne03;
|
||||||
|
|
||||||
|
// K-split decision
|
||||||
|
const int64_t minion_id = hart_id >> 1; /* 0..1023 global */
|
||||||
|
const int64_t local_minion = (hart_id >> 1) & 0x1F; /* 0..31 within shire */
|
||||||
|
const int is_hart1 = hart_id & 1;
|
||||||
|
const int64_t rows_per_minion = (M + STRIDE_M_KSPLIT - 1) / STRIDE_M_KSPLIT;
|
||||||
|
const int64_t k_half = K_blocks / 2;
|
||||||
|
const int use_ksplit_small_rows = (rows_per_minion <= 2) && (K_blocks >= KSPLIT_SMALL_ROWS_K_BLOCKS);
|
||||||
|
/*
|
||||||
|
* K-split when K is large enough to benefit, and either:
|
||||||
|
* - few rows (≤4): always safe, proven working
|
||||||
|
* - more rows (5-8): only if each hart's half fits in one tile,
|
||||||
|
* otherwise L1 thrashing from 2 harts × 8 rows kills performance
|
||||||
|
*
|
||||||
|
* Also allow K-split earlier for the low-M regime (≤2 rows/minion). In
|
||||||
|
* that case the simple row-striped path leaves half the machine idle, so
|
||||||
|
* using both harts on each row pays off even for moderate K.
|
||||||
|
*/
|
||||||
|
const int use_ksplit = ((K_blocks >= KSPLIT_MIN_K_BLOCKS) && (rows_per_minion <= KSPLIT_MAX_ROWS) &&
|
||||||
|
(rows_per_minion <= 4 || k_half <= TILE_KB)) ||
|
||||||
|
use_ksplit_small_rows;
|
||||||
|
const int use_ksplit_group = !use_ksplit && (K_blocks >= KSPLIT_MIN_K_BLOCKS) && (rows_per_minion > 4) &&
|
||||||
|
(rows_per_minion <= KSPLIT_MAX_ROWS);
|
||||||
|
|
||||||
|
if (use_ksplit) {
|
||||||
|
/* Each hart processes half the K dimension */
|
||||||
|
const int64_t k_start = is_hart1 ? k_half : 0;
|
||||||
|
const int64_t k_len = is_hart1 ? (K_blocks - k_half) : k_half;
|
||||||
|
|
||||||
|
/* One cache-line-aligned L2SCP slot per minion for exchange */
|
||||||
|
volatile float * l2scp_slot = (volatile float *) et_shire_l2scp_local(local_minion * 64);
|
||||||
|
|
||||||
|
for (int64_t i3 = 0; i3 < ne13; i3++) {
|
||||||
|
const int64_t i03 = i3 / r3;
|
||||||
|
const char * src0_ptr3 = (const char *) params->src0.data + i03 * nb03;
|
||||||
|
const char * src1_ptr3 = (const char *) params->src1.data + i3 * nb13;
|
||||||
|
char * dst_ptr3 = (char *) params->dst.data + i3 * nbd3;
|
||||||
|
|
||||||
|
for (int64_t i2 = 0; i2 < ne12; i2++) {
|
||||||
|
const int64_t i02 = i2 / r2;
|
||||||
|
const char * src0_ptr2 = src0_ptr3 + i02 * nb02;
|
||||||
|
const char * src1_ptr2 = src1_ptr3 + i2 * nb12;
|
||||||
|
char * dst_ptr2 = dst_ptr3 + i2 * nbd2;
|
||||||
|
|
||||||
|
for (int64_t n = 0; n < N; n++) {
|
||||||
|
const float * b_col_base = (const float *) (src1_ptr2 + n * nb11);
|
||||||
|
|
||||||
|
for (int64_t m = minion_id; m < M; m += STRIDE_M_KSPLIT) {
|
||||||
|
const block_q4_0 * q_row = (const block_q4_0 *) (src0_ptr2 + m * nb01);
|
||||||
|
|
||||||
|
float partial = compute_row_dot_q4_0(q_row + k_start, b_col_base + k_start * 32, k_len);
|
||||||
|
|
||||||
|
if (is_hart1) {
|
||||||
|
*l2scp_slot = partial;
|
||||||
|
FENCE;
|
||||||
|
flush_to_l2((const void *) l2scp_slot, 1, 64);
|
||||||
|
WAIT_CACHEOPS;
|
||||||
|
et_sem_post(ET_BARRIER_MINION);
|
||||||
|
et_sem_wait(ET_BARRIER_MINION);
|
||||||
|
} else {
|
||||||
|
et_sem_wait(ET_BARRIER_MINION);
|
||||||
|
float other = *l2scp_slot;
|
||||||
|
et_sem_post(ET_BARRIER_MINION);
|
||||||
|
|
||||||
|
float * dst_entry = (float *) (dst_ptr2 + n * nbd1 + m * sizeof(float));
|
||||||
|
atomic_store_f32((volatile float *) dst_entry, partial + other);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else if (use_ksplit_group) {
|
||||||
|
/*
|
||||||
|
* Grouped K-split for the 5-8 rows/minion regime.
|
||||||
|
*
|
||||||
|
* Both harts process the same 4-row group, each on half of K, and
|
||||||
|
* exchange 4 partial sums once per group instead of once per row.
|
||||||
|
* This keeps the K-split bandwidth benefit while cutting semaphore
|
||||||
|
* traffic by 4x relative to the old per-row exchange.
|
||||||
|
*/
|
||||||
|
const int64_t k_start = is_hart1 ? k_half : 0;
|
||||||
|
const int64_t k_len = is_hart1 ? (K_blocks - k_half) : k_half;
|
||||||
|
volatile float * l2scp_slot = (volatile float *) et_shire_l2scp_local(local_minion * 64);
|
||||||
|
|
||||||
|
for (int64_t i3 = 0; i3 < ne13; i3++) {
|
||||||
|
const int64_t i03 = i3 / r3;
|
||||||
|
const char * src0_ptr3 = (const char *) params->src0.data + i03 * nb03;
|
||||||
|
const char * src1_ptr3 = (const char *) params->src1.data + i3 * nb13;
|
||||||
|
char * dst_ptr3 = (char *) params->dst.data + i3 * nbd3;
|
||||||
|
|
||||||
|
for (int64_t i2 = 0; i2 < ne12; i2++) {
|
||||||
|
const int64_t i02 = i2 / r2;
|
||||||
|
const char * src0_ptr2 = src0_ptr3 + i02 * nb02;
|
||||||
|
const char * src1_ptr2 = src1_ptr3 + i2 * nb12;
|
||||||
|
char * dst_ptr2 = dst_ptr3 + i2 * nbd2;
|
||||||
|
|
||||||
|
for (int64_t n = 0; n < N; n++) {
|
||||||
|
const float * b_col_base = (const float *) (src1_ptr2 + n * nb11);
|
||||||
|
|
||||||
|
for (int64_t m_base = minion_id; m_base < M; m_base += STRIDE_M_KSPLIT * KSPLIT_GROUP_ROWS) {
|
||||||
|
const int64_t m0 = m_base;
|
||||||
|
const int64_t m1 = m0 + STRIDE_M_KSPLIT;
|
||||||
|
const int64_t m2 = m1 + STRIDE_M_KSPLIT;
|
||||||
|
const int64_t m3 = m2 + STRIDE_M_KSPLIT;
|
||||||
|
|
||||||
|
float s0 = 0.0f, s1 = 0.0f, s2 = 0.0f, s3 = 0.0f;
|
||||||
|
|
||||||
|
for (int64_t kb = 0; kb < K_blocks; kb += TILE_KB) {
|
||||||
|
int64_t tile_len = k_len - kb;
|
||||||
|
if (tile_len > TILE_KB) {
|
||||||
|
tile_len = TILE_KB;
|
||||||
|
}
|
||||||
|
if (tile_len <= 0) {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
const float * b_tile = b_col_base + (k_start + kb) * 32;
|
||||||
|
const int64_t row_kb = k_start + kb;
|
||||||
|
|
||||||
|
if (m0 < M) {
|
||||||
|
s0 += compute_row_dot_q4_0((const block_q4_0 *) (src0_ptr2 + m0 * nb01) + row_kb,
|
||||||
|
b_tile, tile_len);
|
||||||
|
}
|
||||||
|
if (m1 < M) {
|
||||||
|
s1 += compute_row_dot_q4_0((const block_q4_0 *) (src0_ptr2 + m1 * nb01) + row_kb,
|
||||||
|
b_tile, tile_len);
|
||||||
|
}
|
||||||
|
if (m2 < M) {
|
||||||
|
s2 += compute_row_dot_q4_0((const block_q4_0 *) (src0_ptr2 + m2 * nb01) + row_kb,
|
||||||
|
b_tile, tile_len);
|
||||||
|
}
|
||||||
|
if (m3 < M) {
|
||||||
|
s3 += compute_row_dot_q4_0((const block_q4_0 *) (src0_ptr2 + m3 * nb01) + row_kb,
|
||||||
|
b_tile, tile_len);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (is_hart1) {
|
||||||
|
l2scp_slot[0] = s0;
|
||||||
|
l2scp_slot[1] = s1;
|
||||||
|
l2scp_slot[2] = s2;
|
||||||
|
l2scp_slot[3] = s3;
|
||||||
|
FENCE;
|
||||||
|
flush_to_l2((const void *) l2scp_slot, 1, 64);
|
||||||
|
WAIT_CACHEOPS;
|
||||||
|
et_sem_post(ET_BARRIER_MINION);
|
||||||
|
et_sem_wait(ET_BARRIER_MINION);
|
||||||
|
} else {
|
||||||
|
et_sem_wait(ET_BARRIER_MINION);
|
||||||
|
const float p0 = l2scp_slot[0];
|
||||||
|
const float p1 = l2scp_slot[1];
|
||||||
|
const float p2 = l2scp_slot[2];
|
||||||
|
const float p3 = l2scp_slot[3];
|
||||||
|
et_sem_post(ET_BARRIER_MINION);
|
||||||
|
|
||||||
|
float * c_base = (float *) (dst_ptr2 + n * nbd1);
|
||||||
|
if (m0 < M) {
|
||||||
|
atomic_store_f32((volatile float *) (c_base + m0), s0 + p0);
|
||||||
|
}
|
||||||
|
if (m1 < M) {
|
||||||
|
atomic_store_f32((volatile float *) (c_base + m1), s1 + p1);
|
||||||
|
}
|
||||||
|
if (m2 < M) {
|
||||||
|
atomic_store_f32((volatile float *) (c_base + m2), s2 + p2);
|
||||||
|
}
|
||||||
|
if (m3 < M) {
|
||||||
|
atomic_store_f32((volatile float *) (c_base + m3), s3 + p3);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else if (K_blocks > TILE_KB) {
|
||||||
|
/*
|
||||||
|
* Tile-outer with scalar row groups: process up to 4 rows per
|
||||||
|
* hart sharing each B tile before advancing to the next tile.
|
||||||
|
* Uses scalar float variables (not an array) to accumulate across
|
||||||
|
* tiles — avoids the flw/fadd.s/fsw stack ops that corrupt vector
|
||||||
|
* register state on ET-SoC-1's MMX-style shared FP file.
|
||||||
|
*/
|
||||||
|
for (int64_t i3 = 0; i3 < ne13; i3++) {
|
||||||
|
const int64_t i03 = i3 / r3;
|
||||||
|
const char * src0_ptr3 = (const char *) params->src0.data + i03 * nb03;
|
||||||
|
const char * src1_ptr3 = (const char *) params->src1.data + i3 * nb13;
|
||||||
|
char * dst_ptr3 = (char *) params->dst.data + i3 * nbd3;
|
||||||
|
|
||||||
|
for (int64_t i2 = 0; i2 < ne12; i2++) {
|
||||||
|
const int64_t i02 = i2 / r2;
|
||||||
|
const char * src0_ptr2 = src0_ptr3 + i02 * nb02;
|
||||||
|
const char * src1_ptr2 = src1_ptr3 + i2 * nb12;
|
||||||
|
char * dst_ptr2 = dst_ptr3 + i2 * nbd2;
|
||||||
|
|
||||||
|
for (int64_t n = 0; n < N; n++) {
|
||||||
|
const float * b_col_base = (const float *) (src1_ptr2 + n * nb11);
|
||||||
|
|
||||||
|
for (int64_t m0 = hart_id; m0 < M; m0 += STRIDE_M * 4) {
|
||||||
|
const int64_t m1 = m0 + STRIDE_M;
|
||||||
|
const int64_t m2 = m0 + STRIDE_M * 2;
|
||||||
|
const int64_t m3 = m0 + STRIDE_M * 3;
|
||||||
|
|
||||||
|
float s0 = 0.0f, s1 = 0.0f, s2 = 0.0f, s3 = 0.0f;
|
||||||
|
|
||||||
|
for (int64_t kb = 0; kb < K_blocks; kb += TILE_KB) {
|
||||||
|
int64_t tile_len = K_blocks - kb;
|
||||||
|
if (tile_len > TILE_KB) {
|
||||||
|
tile_len = TILE_KB;
|
||||||
|
}
|
||||||
|
const float * b_tile = b_col_base + kb * 32;
|
||||||
|
|
||||||
|
s0 += compute_row_dot_q4_0((const block_q4_0 *) (src0_ptr2 + m0 * nb01) + kb, b_tile,
|
||||||
|
tile_len);
|
||||||
|
if (m1 < M) {
|
||||||
|
s1 += compute_row_dot_q4_0((const block_q4_0 *) (src0_ptr2 + m1 * nb01) + kb, b_tile,
|
||||||
|
tile_len);
|
||||||
|
}
|
||||||
|
if (m2 < M) {
|
||||||
|
s2 += compute_row_dot_q4_0((const block_q4_0 *) (src0_ptr2 + m2 * nb01) + kb, b_tile,
|
||||||
|
tile_len);
|
||||||
|
}
|
||||||
|
if (m3 < M) {
|
||||||
|
s3 += compute_row_dot_q4_0((const block_q4_0 *) (src0_ptr2 + m3 * nb01) + kb, b_tile,
|
||||||
|
tile_len);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
float * dst_base = (float *) (dst_ptr2 + n * nbd1);
|
||||||
|
atomic_store_f32((volatile float *) (dst_base + m0), s0);
|
||||||
|
if (m1 < M) {
|
||||||
|
atomic_store_f32((volatile float *) (dst_base + m1), s1);
|
||||||
|
}
|
||||||
|
if (m2 < M) {
|
||||||
|
atomic_store_f32((volatile float *) (dst_base + m2), s2);
|
||||||
|
}
|
||||||
|
if (m3 < M) {
|
||||||
|
atomic_store_f32((volatile float *) (dst_base + m3), s3);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
/*
|
||||||
|
* Simple path for small K.
|
||||||
|
*
|
||||||
|
* When `nb01` is 32-byte aligned, every row has the same block-alignment
|
||||||
|
* pattern. That lets us compute two rows together and reuse each loaded
|
||||||
|
* B chunk across both rows instead of reloading it in a second dot call.
|
||||||
|
*/
|
||||||
|
for (int64_t i3 = 0; i3 < ne13; i3++) {
|
||||||
|
const int64_t i03 = i3 / r3;
|
||||||
|
const char * src0_ptr3 = (const char *) params->src0.data + i03 * nb03;
|
||||||
|
const char * src1_ptr3 = (const char *) params->src1.data + i3 * nb13;
|
||||||
|
char * dst_ptr3 = (char *) params->dst.data + i3 * nbd3;
|
||||||
|
|
||||||
|
for (int64_t i2 = 0; i2 < ne12; i2++) {
|
||||||
|
const int64_t i02 = i2 / r2;
|
||||||
|
const char * src0_ptr2 = src0_ptr3 + i02 * nb02;
|
||||||
|
const char * src1_ptr2 = src1_ptr3 + i2 * nb12;
|
||||||
|
char * dst_ptr2 = dst_ptr3 + i2 * nbd2;
|
||||||
|
|
||||||
|
for (int64_t n = 0; n < N; n++) {
|
||||||
|
const float * b_col_base = (const float *) (src1_ptr2 + n * nb11);
|
||||||
|
q4_dot_state q4_state;
|
||||||
|
q4_dot_begin(&q4_state);
|
||||||
|
|
||||||
|
if (use_simple_x2) {
|
||||||
|
for (int64_t m0 = hart_id; m0 < M; m0 += STRIDE_M * SIMPLE_X2_ROWS) {
|
||||||
|
const int64_t m1 = m0 + STRIDE_M;
|
||||||
|
const block_q4_0 * q_row0 = (const block_q4_0 *) (src0_ptr2 + m0 * nb01);
|
||||||
|
|
||||||
|
if (m1 < M) {
|
||||||
|
const block_q4_0 * q_row1 = (const block_q4_0 *) (src0_ptr2 + m1 * nb01);
|
||||||
|
float s0, s1;
|
||||||
|
q4_dot_compute_x2_aligned(q_row0, q_row1, b_col_base, K_blocks, &s0, &s1);
|
||||||
|
|
||||||
|
float * dst0 = (float *) (dst_ptr2 + n * nbd1 + m0 * sizeof(float));
|
||||||
|
float * dst1 = (float *) (dst_ptr2 + n * nbd1 + m1 * sizeof(float));
|
||||||
|
atomic_store_f32((volatile float *) dst0, s0);
|
||||||
|
atomic_store_f32((volatile float *) dst1, s1);
|
||||||
|
} else {
|
||||||
|
float sum = q4_dot_compute(q_row0, b_col_base, K_blocks);
|
||||||
|
float * dst = (float *) (dst_ptr2 + n * nbd1 + m0 * sizeof(float));
|
||||||
|
atomic_store_f32((volatile float *) dst, sum);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
for (int64_t m = hart_id; m < M; m += STRIDE_M) {
|
||||||
|
const block_q4_0 * q_row = (const block_q4_0 *) (src0_ptr2 + m * nb01);
|
||||||
|
|
||||||
|
float sum = q4_dot_compute(q_row, b_col_base, K_blocks);
|
||||||
|
|
||||||
|
float * dst_entry = (float *) (dst_ptr2 + n * nbd1 + m * sizeof(float));
|
||||||
|
atomic_store_f32((volatile float *) dst_entry, sum);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
q4_dot_end(&q4_state);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user