練營】Qwen3-8B 推理優(yōu)化實(shí)踐筆記（2~3）)
1 簡介見【CANN訓(xùn)練營】Qwen3-8B 推理優(yōu)化實(shí)踐食用指南-CSDN博客2 Baseline跑通notebook地址gitcode.com/cann/cann-learning-hub/tutorials/llm_inference/qwen3_8b/02_baseline_inference.ipynb2.1 環(huán)境與工作區(qū)準(zhǔn)備import json import os import subprocess import sys from pathlib import Path print([環(huán)境準(zhǔn)備] 開始, flushTrue) os.environ.setdefault(QWEN3_8B_DOWNLOAD_BACKEND, modelscope) os.environ[TOKENIZERS_PARALLELISM] false os.environ.setdefault(ASCEND_RT_VISIBLE_DEVICES, 0) def locate_repo_root(): repo_roots [] for env_name in (GITCODE_REPO_ROOT, CANN_LEARNING_HUB_ROOT, REPO_ROOT): if os.environ.get(env_name): repo_roots.append(Path(os.environ[env_name])) try: cwd Path.cwd() lineage [cwd, *cwd.parents] repo_roots.extend(lineage) repo_roots.extend(base / cann-learning-hub for base in lineage) for base in lineage: try: repo_roots.extend(base.glob(*/cann-learning-hub)) except OSError: pass except FileNotFoundError: pass repo_roots.append(Path(/opt/atomgit/cann-learning-hub)) seen set() for repo_root in repo_roots: key str(repo_root) if key in seen: continue seen.add(key) if (repo_root / tutorials/llm_inference/qwen3_8b/src).exists(): return repo_root raise FileNotFoundError(Cannot locate cann-learning-hub repository root. Open this notebook inside cann-learning-hub or set CANN_LEARNING_HUB_ROOT to the repository root.) REPO_ROOT locate_repo_root() os.chdir(REPO_ROOT) TUTORIAL_DIR REPO_ROOT / tutorials / llm_inference / qwen3_8b SRC_DIR TUTORIAL_DIR / src SOURCE_DIR REPO_ROOT / Sources / model_inference_optimization / qwen3_8b SOURCE_DIR.mkdir(parentsTrue, exist_okTrue) if str(SRC_DIR) not in sys.path: sys.path.insert(0, str(SRC_DIR)) from inference_scripts.environment_check import find_unmet_requirements, read_pinned_requirements if not os.environ.get(ASCEND_TOOLKIT_HOME): raise EnvironmentError(ASCEND_TOOLKIT_HOME is not set. Please activate the CANN environment before running this notebook.) cann_script Path(os.environ[ASCEND_TOOLKIT_HOME]) / set_env.sh env_cmd fsource {cann_script} env env subprocess.check_output(fbash -lc {env_cmd}, shellTrue, textTrue, cwdREPO_ROOT) for line in env.splitlines(): if in line: os.environ.__setitem__(*line.split(, 1)) REQUIREMENTS_FILE TUTORIAL_DIR / requirements.txt pinned_requirements read_pinned_requirements(REQUIREMENTS_FILE) unmet_requirements find_unmet_requirements(pinned_requirements) if unmet_requirements: print([依賴檢查] 以下依賴缺失或版本不匹配) for name, actual, expected in unmet_requirements: print(f - {name}: 當(dāng)前{actual or 未安裝}, 要求{expected}) requirements_to_install [f{name}{expected} for name, _, expected in unmet_requirements] subprocess.run( [ sys.executable, -m, pip, install, -i, https://pypi.tuna.tsinghua.edu.cn/simple, --extra-index-url, https://download.pytorch.org/whl/cpu, --trusted-host, pypi.tuna.tsinghua.edu.cn, *requirements_to_install, ], checkTrue, ) remaining find_unmet_requirements(pinned_requirements) if remaining: details , .join(f{name}: {actual or 未安裝} ! {expected} for name, actual, expected in remaining) raise RuntimeError(f依賴安裝后仍不滿足統(tǒng)一版本{details}) print([依賴檢查] 安裝完成統(tǒng)一環(huán)境已滿足) else: print([依賴檢查] 所有依賴版本均滿足跳過安裝) RECIPE_ROOT SRC_DIR / inference_scripts / recipe_qwen3_8b RECIPE_CONFIG_DIR RECIPE_ROOT / models / qwen / config BASELINE_YAML_TEMPLATE RECIPE_CONFIG_DIR / qwen3_8b_1tp.yaml PROFILE_YAML_TEMPLATE RECIPE_CONFIG_DIR / qwen3_8b_1tp_profile.yaml OPTIMIZED_YAML_TEMPLATE RECIPE_CONFIG_DIR / qwen3_8b_1tp_add_rmsnorm.yaml INFER_SH RECIPE_ROOT / executor / scripts / infer.sh MODEL_ID os.environ.get(QWEN3_8B_MODEL_PATH) or Qwen/Qwen3-8B MODEL_SOURCE_HINT local_path if os.environ.get(QWEN3_8B_MODEL_PATH) else os.environ[QWEN3_8B_DOWNLOAD_BACKEND] print(倉庫目錄 , REPO_ROOT) print(教程目錄 , TUTORIAL_DIR) print(運(yùn)行目錄 , SOURCE_DIR) print(recipes 目錄 , RECIPE_ROOT) print(YAML 目錄 , RECIPE_CONFIG_DIR) print(推理入口 , INFER_SH) print(可見 NPU , os.environ.get(ASCEND_RT_VISIBLE_DEVICES)) print([環(huán)境準(zhǔn)備] 完成, flushTrue)這段代碼是整個(gè)教程的環(huán)境準(zhǔn)備階段核心目標(biāo)是把當(dāng)前 Notebook 進(jìn)程配置成“可以在昇騰 NPU 上跑 Qwen3-8B 推理”的完整環(huán)境。執(zhí)行對應(yīng)單元格即可。這部分代碼完成了環(huán)境準(zhǔn)備工作先通過多策略定位并切換到教程倉庫根目錄然后設(shè)置模型下載后端、tokenizer并行開關(guān)和可見NPU卡號(hào)等基礎(chǔ)環(huán)境變量接著通過執(zhí)行CANN的set_env.sh腳本把昇騰工具鏈所需的全部環(huán)境變量導(dǎo)入當(dāng)前Notebook進(jìn)程隨后讀取requirements.txt并檢查當(dāng)前Python環(huán)境的依賴版本若有缺失或不匹配則自動(dòng)用清華鏡像源安裝補(bǔ)齊最后定義recipes推理框架、YAML配置文件、推理入口腳本和模型ID等路徑常量并打印環(huán)境摘要供確認(rèn)為后續(xù)在昇騰NPU上運(yùn)行Qwen3-8B推理和Profiling采集做好了完整的運(yùn)行環(huán)境準(zhǔn)備。2.2 查看YAML并執(zhí)行Baseline推理2.2.1準(zhǔn)備Baseline YAML把 Baseline YAML 模板復(fù)制一份到運(yùn)行目錄填入真實(shí)模型路徑并打印關(guān)鍵配置供確認(rèn)。import os import time from inference_scripts.recipe_workflow import ( collect_recipe_metrics, find_recipe_case_dir, prepare_runtime_yaml, print_yaml_focus, resolve_model_path, ) from inference_scripts.inference_qwen3_8b import ensure_recipe_log_ok BASELINE_WORK_DIR SOURCE_DIR / run_outputs / baseline_demo BASELINE_YAML SOURCE_DIR / recipe_yaml / baseline_demo.yaml BASELINE_WORK_DIR.mkdir(parentsTrue, exist_okTrue) print([Baseline 推理] 開始, flushTrue) resolved_model_path, model_source resolve_model_path(str(MODEL_ID), quiet_model_ioFalse) prepare_runtime_yaml(BASELINE_YAML_TEMPLATE, BASELINE_YAML, model_pathresolved_model_path) print([YAML 準(zhǔn)備] Baseline 配置, flushTrue) print(參考 YAML , BASELINE_YAML_TEMPLATE) print(運(yùn)行 YAML , BASELINE_YAML) print_yaml_focus(BASELINE_YAML) baseline_log_dir BASELINE_WORK_DIR / recipe_logs baseline_log_dir.mkdir(parentsTrue, exist_okTrue) os.environ[QWEN3_RECIPE_ROOT] str(RECIPE_ROOT) os.environ[QWEN3_RUN_YAML] str(BASELINE_YAML) os.environ[QWEN3_RUN_LOG] str(baseline_log_dir / baseline_demo_infer_stdout.log) os.environ[QWEN3_RUN_START_TIME] str(time.perf_counter())什么是YAMLYAMLYAML Aint Markup Language是一種人類可讀的配置文件格式用縮進(jìn)和鍵值對來表示數(shù)據(jù)就像“簡化版的 JSON但不用寫括號(hào)和逗號(hào)”。為什么需要 YAML這套教程里YAML 的作用是用文本描述“怎么跑模型推理”而不需要改 Python 代碼。比如需要改模型路徑、是否開 Profiler、是否開 RMSNorm 融合、生成長度這些內(nèi)容只需要改 YAML 文件不用動(dòng)任何 Python 代碼。YAML 長什么樣運(yùn)行YAML目錄文件地址在cann-learning-hub/Sources/model_inference_optimization/qwen3_8b/recipe_yaml/baseline_demo.yaml上一部分打印的內(nèi)容作為Baseline配置這份YAML的核心特征是關(guān)閉了各類性能優(yōu)化如 Profiler、RMSNorm 融合、緩存編譯等用于作為后續(xù)優(yōu)化對比的“原始基準(zhǔn)”。YAML文件的主要字段及解釋1 model_config模型與推理核心配置字段當(dāng)前值含義model_nameqwen3_8b模型名稱標(biāo)識(shí)model_path/home/developer/.cache/...模型權(quán)重實(shí)際路徑由代碼自動(dòng)填入dtypebfloat16計(jì)算精度BF16兼顧性能與精度exe_modeeager執(zhí)行模式即時(shí)執(zhí)行非圖模式enable_profilerfalse?關(guān)鍵關(guān)閉性能剖析Baseline 不采集數(shù)據(jù)with_ckpttrue啟用檢查點(diǎn)enable_cache_compilefalse?關(guān)閉緩存編譯enable_static_kernelfalse?關(guān)閉靜態(tài)內(nèi)核custom_params子字段自定義優(yōu)化開關(guān)見下custom_params自定義優(yōu)化Baseline 全部關(guān)閉enable_npu_rmsnorm:false關(guān)閉 NPU 原生 RMSNormenable_add_rmsnorm:false關(guān)閉 AddRMSNorm 融合這是第4章要優(yōu)化的核心enable_prefetch:false關(guān)閉預(yù)取prefetch_size:18874368預(yù)取大小雖關(guān)閉但保留默認(rèn)值2data_config數(shù)據(jù)配置dataset:default使用默認(rèn)數(shù)據(jù)集input_truncated_len:256輸入截?cái)嚅L度最多 256 token3parallel_config并行配置全部為1表示單卡運(yùn)行無張量并行/專家并行world_size: 1attn_tp_size: 1注意力張量并行moe_tp_size: 1MoE 張量并行embed_tp_size: 1嵌入層并行l(wèi)mhead_tp_size: 1輸出頭并行4scheduler_config調(diào)度與生成配置max_new_tokens:64最多生成 64 個(gè)新 tokenbatch_size:1單批次推理enable_thinking:false關(guān)閉思考模式Qwen3 特有2.2.2 使用 recipes 命令啟動(dòng)推理%%bash set -euo pipefail # 激活虛擬環(huán)境如果有 source /mnt/workspace/gitCode/cann/cann-learning-hub/tutorials/llm_inference/qwen3_8b/venv/bin/activate # recipes 原生啟動(dòng)命令 echo recipes 啟動(dòng)命令: echo cd ${QWEN3_RECIPE_ROOT} echo bash executor/scripts/infer.sh --model qwen --mode offline --yaml ${QWEN3_RUN_YAML} cd ${QWEN3_RECIPE_ROOT} bash executor/scripts/infer.sh --model qwen --mode offline --yaml ${QWEN3_RUN_YAML} 21 | tee ${QWEN3_RUN_LOG}這部分是 Baseline 推理的實(shí)際啟動(dòng)命令核心作用是通過 recipes 框架在昇騰 NPU 上真正跑一次 Qwen3-8B 推理。注意這部分需要在原代碼開頭加一行激活虛擬環(huán)境的代碼如果有recipes 是什么recipes是昇騰 CANN 生態(tài)里的一套“大模型推理樣例框架”它不是模型本身也不是 PyTorch而是怎么在 NPU 上把某個(gè)大模型跑起來的工程封裝。infer.sh 是什么qwen3_8b/src/inference_scripts/recipe_qwen3_8b/executor/scripts/infer.sh 是 recipes 的統(tǒng)一推理入口腳本。主要任務(wù)解析參數(shù) → 加載環(huán)境 → 定位 YAML → 校驗(yàn)配置 → 調(diào)用launch函數(shù)真正在 NPU 上跑推理。2.2.3 讀取recipes運(yùn)行結(jié)果case_dir find_recipe_case_dir(RECIPE_ROOT, BASELINE_YAML) if case_dir is None: raise RuntimeError(f未找到 recipes 結(jié)果目錄{BASELINE_YAML.stem}) recipe_log_path case_dir / log_0.log ensure_recipe_log_ok(recipe_log_path) run_info { stdout_path: os.environ[QWEN3_RUN_LOG], stderr_path: os.environ[QWEN3_RUN_LOG], wall_time_s: time.perf_counter() - float(os.environ[QWEN3_RUN_START_TIME]), recipe_case_dir: str(case_dir), recipe_log_path: str(recipe_log_path), } baseline_metrics collect_recipe_metrics( RECIPE_ROOT, BASELINE_WORK_DIR, baseline_demo, BASELINE_YAML, run_info, model_idstr(MODEL_ID), resolved_model_pathresolved_model_path, model_sourcemodel_source, optimization_modenone, max_new_tokens64, max_input_tokens256, enable_thinkingFalse, ) print([Baseline 推理] 完成, flushTrue)這部分代碼是Baseline推理的收尾階段核心作用是定位推理結(jié)果目錄檢查日志是否正常 收集并保存推理指標(biāo)為后續(xù)分析和對比做準(zhǔn)備。參數(shù)含義參數(shù)值含義optimization_modenone本次是 Baseline無優(yōu)化max_new_tokens64生成長度限制max_input_tokens256輸入長度限制enable_thinkingFalse關(guān)閉思考模式2.3 查看結(jié)果print([結(jié)果查看] 開始, flushTrue) def status_cn(status): return {ok: 通過, warning: 告警, failed: 失敗}.get(status, status) def yes_no(value): if value is None: return 未記錄 return 是 if value else 否 def load_metrics(work_dir, tag): metrics_path Path(work_dir) / metrics / f{tag}_summary.json metrics json.loads(metrics_path.read_text(encodingutf-8)) print(指標(biāo)文件 , metrics_path) print(recipes YAML , metrics.get(recipe_yaml_path)) print(recipes 結(jié)果目錄 , metrics.get(recipe_case_dir)) print(生成文件 , metrics.get(outputs_path)) print(生成 token 數(shù) , metrics.get(generated_tokens)) print(總耗時(shí)(s) , metrics.get(total_time_s)) print(吞吐(tokens/s) , metrics.get(tokens_per_second)) rmsnorm_status metrics.get(rmsnorm_status) or {} print(RMSNorm 融合 , yes_no(rmsnorm_status.get(rmsnorm_fused))) print(AddRMSNorm 融合 , yes_no(rmsnorm_status.get(add_rmsnorm_fused))) sanity metrics.get(generation_text_sanity) if sanity: print(吐字檢查 , status_cn(sanity.get(status)), f警告{sanity.get(num_with_warnings)}, f失敗{sanity.get(num_failed)}) return metrics baseline_metrics load_metrics(BASELINE_WORK_DIR, baseline_demo) print() print(生成文本:) for line in Path(baseline_metrics[outputs_path]).read_text(encodingutf-8).splitlines(): if line.strip(): print(json.loads(line).get(generated_text, )) print([結(jié)果查看] 完成, flushTrue)這段代碼讀取并展示剛才那次推理的完整指標(biāo)和生成文本便于直觀確認(rèn) Baseline 狀態(tài)。運(yùn)行結(jié)果[結(jié)果查看] 開始 指標(biāo)文件 /mnt/workspace/gitCode/cann/cann-learning-hub/Sources/model_inference_optimization/qwen3_8b/run_outputs/baseline_demo/metrics/baseline_demo_summary.json recipes YAML /mnt/workspace/gitCode/cann/cann-learning-hub/Sources/model_inference_optimization/qwen3_8b/recipe_yaml/baseline_demo.yaml recipes 結(jié)果目錄 /mnt/workspace/gitCode/cann/cann-learning-hub/tutorials/llm_inference/qwen3_8b/src/inference_scripts/recipe_qwen3_8b/models/qwen/res/20261013/qwen3_8b_baseline_demo 生成文件 /mnt/workspace/gitCode/cann/cann-learning-hub/Sources/model_inference_optimization/qwen3_8b/run_outputs/baseline_demo/outputs/baseline_demo_generations.jsonl 生成 token 數(shù) 38 總耗時(shí)(s) 5.3463899999999995 吞吐(tokens/s) 7.10759970746616 RMSNorm 融合 否 AddRMSNorm 融合 否 吐字檢查 通過 警告0 失敗0 生成文本: The output of an **attention function** is a **weighted sum of the value vectors**, where the weights are determined by the **similarity between the query vector and each key vector**. [結(jié)果查看] 完成打印內(nèi)容實(shí)際值含義指標(biāo)文件路徑指標(biāo) JSON 的位置recipes YAML運(yùn)行 YAML 路徑用的是哪個(gè)配置recipes 結(jié)果目錄推理產(chǎn)物目錄日志、輸出等存放位置生成文件outputs 路徑生成文本的 JSONL 文件生成 token 數(shù)38實(shí)際生成了 38 個(gè) token總耗時(shí)(s)5.35推理總耗時(shí) 5.35 秒吞吐(tokens/s)7.11每秒生成 7.11 個(gè) tokenRMSNorm 融合否確認(rèn)關(guān)閉AddRMSNorm 融合否確認(rèn)關(guān)閉吐字檢查通過無告警、無失敗3 Profiling分析這部分在Baseline的基礎(chǔ)上打開torch_npu.profiler。torch_npu.profiler 是 PyTorch 在昇騰 NPU 上的性能剖析器它底層會(huì)調(diào)用 CANN / Ascend Profiling? 能力把三層數(shù)據(jù)對起來1Host 側(cè)Python 調(diào)用、PyTorch 算子下發(fā)、CPU 活動(dòng)2CANN 側(cè)算子編譯、任務(wù)調(diào)度、內(nèi)存拷貝、流同步3NPU 側(cè)AI Core / AI CPU / Vector 核上的 kernel 執(zhí)行時(shí)間、耗時(shí)、等待時(shí)間它的目的不是“讓模型更快”而是先看清楚 Baseline 推理里時(shí)間到底耗在哪些算子上了。準(zhǔn)備工作與運(yùn)行環(huán)境工作區(qū)準(zhǔn)備的流程和baseline一致。準(zhǔn)備YAML的時(shí)候準(zhǔn)備的是Profiling YAML在baseline的基礎(chǔ)上打開Profiling。其余運(yùn)行流程與baseline一致。Profiling YAML中model_config.enable_profiler 設(shè)為true其余推理參數(shù)保持和 Baseline 一致。YAML 文件 /mnt/workspace/gitCode/cann/cann-learning-hub/Sources/model_inference_optimization/qwen3_8b/recipe_yaml/baseline_profile.yaml model_path /home/developer/.cache/modelscope/models/Qwen--Qwen3-8B/snapshots/master enable_profiler True # 注意這里的enable_profiler即yaml里打開了profiler enable_npu_rmsnorm False enable_add_rmsnorm False enable_prefetch False prefetch_size None dataset default input_truncated_len 256 max_new_tokens 64 batch_size 1結(jié)果分析[Profiler 產(chǎn)物] 開始 指標(biāo)文件 /mnt/workspace/gitCode/cann/cann-learning-hub/Sources/model_inference_optimization/qwen3_8b/run_outputs/baseline_profile/metrics/baseline_profile_summary.json 生成 token 數(shù) 38 Profiler 開啟 是 Profile 目錄 /mnt/workspace/gitCode/cann/cann-learning-hub/tutorials/llm_inference/qwen3_8b/src/inference_scripts/recipe_qwen3_8b/models/qwen/res/20261013/qwen3_8b_baseline_profile/prof prof/prefill /mnt/workspace/gitCode/cann/cann-learning-hub/tutorials/llm_inference/qwen3_8b/src/inference_scripts/recipe_qwen3_8b/models/qwen/res/20261013/qwen3_8b_baseline_profile/prof/prefill 一級目錄 7f27e7db10fd4ffda8f0b031bf276588_11042_20261013190353199_ascend_pt kernel_details.csv 已生成 trace_view.json 已生成 prof/decode /mnt/workspace/gitCode/cann/cann-learning-hub/tutorials/llm_inference/qwen3_8b/src/inference_scripts/recipe_qwen3_8b/models/qwen/res/20261013/qwen3_8b_baseline_profile/prof/decode 一級目錄 7f27e7db10fd4ffda8f0b031bf276588_11042_20261013190359433_ascend_pt kernel_details.csv 已生成 trace_view.json 已生成 吐字檢查 通過 警告0 失敗0 Profiler 摘要文件 /mnt/workspace/gitCode/cann/cann-learning-hub/Sources/model_inference_optimization/qwen3_8b/run_outputs/baseline_profile/profiler_summary/baseline_profile_summary.txt Profiler summary from kernel_details source: /mnt/workspace/gitCode/cann/cann-learning-hub/tutorials/llm_inference/qwen3_8b/src/inference_scripts/recipe_qwen3_8b/models/qwen/res/20261013/qwen3_8b_baseline_profile/prof/decode/7f27e7db10fd4ffda8f0b031bf276588_11042_20261013190359433_ascend_pt/ASCEND_PROFILER_OUTPUT/kernel_details.csv OP Type Count Total(us) Ratio(%) ---------------------------------------------------------------------------------------- aclnnMatmulWeightNz_MatMulCommon_MatMulV2 435 36531.740 68.653 aclnnMul_MulAiCore_Mul 978 3579.760 6.727 aclnnPowTensorScalar_PowsAiCore_Pows 435 2855.780 5.367 FusedInferAttentionScore 108 2413.320 4.535 aclnnMean_ReduceMeanAiCore_ReduceMean 435 2318.040 4.356 ... aclnnInplaceOne_OnesLikeAiCore_OnesLike 3 35.100 0.066 aclnnEmbedding_GatherV2AiCore_GatherV2 3 31.100 0.058 ... 僅顯示前 24 行共 35 行 [Profiler 產(chǎn)物] 完成[分析結(jié)論] 開始 摘要類型 kernel_details 生成 token 數(shù) 38 Baseline RMSNorm 融合 否 Baseline AddRMSNorm 融合 否 Top 算子: - aclnnMatmulWeightNz_MatMulCommon_MatMulV2: 次數(shù)435總耗時(shí)(us)36531.740占比68.653% - aclnnMul_MulAiCore_Mul: 次數(shù)978總耗時(shí)(us)3579.760占比6.727% - aclnnPowTensorScalar_PowsAiCore_Pows: 次數(shù)435總耗時(shí)(us)2855.780占比5.367% - FusedInferAttentionScore: 次數(shù)108總耗時(shí)(us)2413.320占比4.535% - aclnnMean_ReduceMeanAiCore_ReduceMean: 次數(shù)435總耗時(shí)(us)2318.040占比4.356% - aclnnInplaceCopy_CastAiCore_Cast: 次數(shù)873總耗時(shí)(us)1377.980占比2.590% - aclnnScatterNdUpdate_ScatterNdUpdateAiCore_ScatterNdUpdate: 次數(shù)216總耗時(shí)(us)1090.060占比2.049% - aclnnAdds_AddAiCore_Add: 次數(shù)441總耗時(shí)(us)625.340占比1.175% - aclnnRsqrt_RsqrtAiCore_Rsqrt: 次數(shù)435總耗時(shí)(us)533.560占比1.003% - aclnnAdd_AddAiCore_Add: 次數(shù)219總耗時(shí)(us)473.720占比0.890% 結(jié)論: - Baseline YAML 中 Dense RMSNorm 兩個(gè)融合開關(guān)均為關(guān)閉因此本章看到的是融合前的執(zhí)行鏈路。 - 摘要中沒有直接出現(xiàn)融合后的 RMSNorm 算子名說明 Baseline 仍按小算子鏈路執(zhí)行。 - 結(jié)合 QwenRMSNorm 的計(jì)算式摘要中的歸一化相關(guān)小算子包括aclnnAdd_CastAiCore_Cast、aclnnArgMax_CastAiCore_Cast、aclnnGather_CastAiCore_Cast、aclnnInplaceCopy_CastAiCore_Cast、aclnnMax_ReduceMaxAiCore_ReduceMax、aclnnMean_ReduceMeanAiCore_ReduceMean、aclnnPowTensorScalar_PowsAiCore_Pows、aclnnRemainderTensorScalar_BroadcastToAiCore_BroadcastTo。 - 摘要中也可以看到殘差 Add 相關(guān)算子aclnnAdd_AddAiCore_Add、aclnnAdd_CastAiCore_Cast、aclnnAdds_AddAiCore_Add。 - 下一章保持 recipes 的 YAML infer.sh 工作流只切換 Dense RMSNorm NPU 融合開關(guān)對比精度、吐字健康和吞吐變化。 分析文件 /mnt/workspace/gitCode/cann/cann-learning-hub/Sources/model_inference_optimization/qwen3_8b/run_outputs/profiling_analysis_summary.json [分析結(jié)論] 完成算子耗時(shí)排名排名算子次數(shù)耗時(shí)(us)占比屬于什么1MatMul?43536531.7468.65%?線性層QKV/O/FFN2Mul?9783579.766.73%RMSNorm 小算子?3Pow?4352855.785.37%RMSNorm 小算子?4FusedInferAttentionScore?1082413.324.54%注意力融合算子5ReduceMean?4352318.044.36%RMSNorm 小算子?6Cast8731377.982.59%類型轉(zhuǎn)換7ScatterNdUpdate2161090.062.05%KV Cache 更新8Adds441625.341.18%殘差 Add?9Rsqrt?435533.561.00%RMSNorm 小算子?10Add219473.720.89%殘差 Add?可以看出MatMul矩陣乘法耗時(shí)最長其余部分RMSNorm 被拆成了 8 個(gè)小算子殘差 Add 也是小算子沒有融合。Profiling 分析證實(shí)了 RMSNorm 未融合時(shí)被拆成 8 個(gè)小算子Pow/ReduceMean/Rsqrt/Mul 等導(dǎo)致launch 次數(shù)多、訪存多、耗時(shí)高是明確的性能瓶頸。下一章將打開enable_add_rmsnorm融合開關(guān)用 A/B 對比驗(yàn)證優(yōu)化效果。trace_view.json 時(shí)間線圖譜trace_view.json時(shí)間線圖譜呈現(xiàn)了 Qwen3-8B 在昇騰 NPU 上Prefill?與Decode?兩個(gè)階段的執(zhí)行特征。Prefill對應(yīng)prof/prefill即處理輸入提示詞的階段。時(shí)間跨度與標(biāo)記整體約 150ms50ms-200ms標(biāo)記為ProfilerStep#0末尾帶有Dec? 標(biāo)記暗示向 Decode 階段的過渡。算子形態(tài)NPUStream 46/49上的色塊相對“大塊”雖然密集但多為一次性大粒度計(jì)算如首輪的 QKV 大矩陣乘。重疊與頻率CPUPython/CANN與 NPU 有明顯的調(diào)度交互。底部Overlap Analysis中 Computing 與 Free 交替由于僅執(zhí)行一次整體耗時(shí)較短未暴露出小算子的反復(fù)累加效應(yīng)。Decode時(shí)間跨度與標(biāo)記整體約 350ms100ms-450ms包含#3、#4、#5連續(xù)多步每步對應(yīng)一個(gè)或幾個(gè)Token 的生成。極度密集的細(xì)碎條紋這是最核心的視覺特征。NPU 時(shí)間線上布滿高頻、細(xì)碎的彩色條紋直觀對應(yīng)了前文數(shù)據(jù)中435次 MatMul? 以及伴隨的Pow/ReduceMean/Rsqrt/MulRMSNorm 未融合等小算子。多流并行Stream 46 與 Stream 49 交錯(cuò)CANN 線程持續(xù)調(diào)度AI Core 頻率滿載淺藍(lán)色長條說明計(jì)算單元持續(xù)忙碌。對比維度PrefillDecode性能含義執(zhí)行次數(shù)?單次Step#0多次迭代Step#3/4/5Decode 重復(fù)放大了局部開銷算子粒度?相對大塊計(jì)算密集極度細(xì)碎訪存/調(diào)度密集?Decode 中小算子 Launch 開銷顯著時(shí)間線密度?密集但無高頻鋸齒滿屏細(xì)碎條紋印證 RMSNorm 拆分成 4~8 個(gè)小算子耗時(shí)占比?較短~150ms較長~350ms端到端吞吐受 Decode 階段主導(dǎo)圖譜直觀揭示了Decode階段因 RMSNorm 未融合產(chǎn)生大量小算子調(diào)度細(xì)碎條紋是端到端性能的可優(yōu)化瓶頸Prefill 則相對健康。后續(xù) 【CANN訓(xùn)練營】Qwen3-8B 推理優(yōu)化實(shí)踐筆記（2~3）
1 簡介見【CANN訓(xùn)練營】Qwen3-8B 推理優(yōu)化實(shí)踐食用指南-CSDN博客2 Baseline跑通notebook地址gitcode.com/cann/cann-learning-hub/tutorials/llm_inference/qwen3_8b/02_baseline_inference.ipynb2.1 環(huán)境與工作區(qū)準(zhǔn)備import json import os import subprocess import sys from pathlib import Path print([環(huán)境準(zhǔn)備] 開始, flushTrue) os.environ.setdefault(QWEN3_8B_DOWNLOAD_BACKEND, modelscope) os.environ[TOKENIZERS_PARALLELISM] false os.environ.setdefault(ASCEND_RT_VISIBLE_DEVICES, 0) def locate_repo_root(): repo_roots [] for env_name in (GITCODE_REPO_ROOT, CANN_LEARNING_HUB_ROOT, REPO_ROOT): if os.environ.get(env_name): repo_roots.append(Path(os.environ[env_name])) try: cwd Path.cwd() lineage [cwd, *cwd.parents] repo_roots.extend(lineage) repo_roots.extend(base / cann-learning-hub for base in lineage) for base in lineage: try: repo_roots.extend(base.glob(*/cann-learning-hub)) except OSError: pass except FileNotFoundError: pass repo_roots.append(Path(/opt/atomgit/cann-learning-hub)) seen set() for repo_root in repo_roots: key str(repo_root) if key in seen: continue seen.add(key) if (repo_root / tutorials/llm_inference/qwen3_8b/src).exists(): return repo_root raise FileNotFoundError(Cannot locate cann-learning-hub repository root. Open this notebook inside cann-learning-hub or set CANN_LEARNING_HUB_ROOT to the repository root.) REPO_ROOT locate_repo_root() os.chdir(REPO_ROOT) TUTORIAL_DIR REPO_ROOT / tutorials / llm_inference / qwen3_8b SRC_DIR TUTORIAL_DIR / src SOURCE_DIR REPO_ROOT / Sources / model_inference_optimization / qwen3_8b SOURCE_DIR.mkdir(parentsTrue, exist_okTrue) if str(SRC_DIR) not in sys.path: sys.path.insert(0, str(SRC_DIR)) from inference_scripts.environment_check import find_unmet_requirements, read_pinned_requirements if not os.environ.get(ASCEND_TOOLKIT_HOME): raise EnvironmentError(ASCEND_TOOLKIT_HOME is not set. Please activate the CANN environment before running this notebook.) cann_script Path(os.environ[ASCEND_TOOLKIT_HOME]) / set_env.sh env_cmd fsource {cann_script} env env subprocess.check_output(fbash -lc {env_cmd}, shellTrue, textTrue, cwdREPO_ROOT) for line in env.splitlines(): if in line: os.environ.__setitem__(*line.split(, 1)) REQUIREMENTS_FILE TUTORIAL_DIR / requirements.txt pinned_requirements read_pinned_requirements(REQUIREMENTS_FILE) unmet_requirements find_unmet_requirements(pinned_requirements) if unmet_requirements: print([依賴檢查] 以下依賴缺失或版本不匹配) for name, actual, expected in unmet_requirements: print(f - {name}: 當(dāng)前{actual or 未安裝}, 要求{expected}) requirements_to_install [f{name}{expected} for name, _, expected in unmet_requirements] subprocess.run( [ sys.executable, -m, pip, install, -i, https://pypi.tuna.tsinghua.edu.cn/simple, --extra-index-url, https://download.pytorch.org/whl/cpu, --trusted-host, pypi.tuna.tsinghua.edu.cn, *requirements_to_install, ], checkTrue, ) remaining find_unmet_requirements(pinned_requirements) if remaining: details , .join(f{name}: {actual or 未安裝} ! {expected} for name, actual, expected in remaining) raise RuntimeError(f依賴安裝后仍不滿足統(tǒng)一版本{details}) print([依賴檢查] 安裝完成統(tǒng)一環(huán)境已滿足) else: print([依賴檢查] 所有依賴版本均滿足跳過安裝) RECIPE_ROOT SRC_DIR / inference_scripts / recipe_qwen3_8b RECIPE_CONFIG_DIR RECIPE_ROOT / models / qwen / config BASELINE_YAML_TEMPLATE RECIPE_CONFIG_DIR / qwen3_8b_1tp.yaml PROFILE_YAML_TEMPLATE RECIPE_CONFIG_DIR / qwen3_8b_1tp_profile.yaml OPTIMIZED_YAML_TEMPLATE RECIPE_CONFIG_DIR / qwen3_8b_1tp_add_rmsnorm.yaml INFER_SH RECIPE_ROOT / executor / scripts / infer.sh MODEL_ID os.environ.get(QWEN3_8B_MODEL_PATH) or Qwen/Qwen3-8B MODEL_SOURCE_HINT local_path if os.environ.get(QWEN3_8B_MODEL_PATH) else os.environ[QWEN3_8B_DOWNLOAD_BACKEND] print(倉庫目錄 , REPO_ROOT) print(教程目錄 , TUTORIAL_DIR) print(運(yùn)行目錄 , SOURCE_DIR) print(recipes 目錄 , RECIPE_ROOT) print(YAML 目錄 , RECIPE_CONFIG_DIR) print(推理入口 , INFER_SH) print(可見 NPU , os.environ.get(ASCEND_RT_VISIBLE_DEVICES)) print([環(huán)境準(zhǔn)備] 完成, flushTrue)這段代碼是整個(gè)教程的環(huán)境準(zhǔn)備階段核心目標(biāo)是把當(dāng)前 Notebook 進(jìn)程配置成“可以在昇騰 NPU 上跑 Qwen3-8B 推理”的完整環(huán)境。執(zhí)行對應(yīng)單元格即可。這部分代碼完成了環(huán)境準(zhǔn)備工作先通過多策略定位并切換到教程倉庫根目錄然后設(shè)置模型下載后端、tokenizer并行開關(guān)和可見NPU卡號(hào)等基礎(chǔ)環(huán)境變量接著通過執(zhí)行CANN的set_env.sh腳本把昇騰工具鏈所需的全部環(huán)境變量導(dǎo)入當(dāng)前Notebook進(jìn)程隨后讀取requirements.txt并檢查當(dāng)前Python環(huán)境的依賴版本若有缺失或不匹配則自動(dòng)用清華鏡像源安裝補(bǔ)齊最后定義recipes推理框架、YAML配置文件、推理入口腳本和模型ID等路徑常量并打印環(huán)境摘要供確認(rèn)為后續(xù)在昇騰NPU上運(yùn)行Qwen3-8B推理和Profiling采集做好了完整的運(yùn)行環(huán)境準(zhǔn)備。2.2 查看YAML并執(zhí)行Baseline推理2.2.1準(zhǔn)備Baseline YAML把 Baseline YAML 模板復(fù)制一份到運(yùn)行目錄填入真實(shí)模型路徑并打印關(guān)鍵配置供確認(rèn)。import os import time from inference_scripts.recipe_workflow import ( collect_recipe_metrics, find_recipe_case_dir, prepare_runtime_yaml, print_yaml_focus, resolve_model_path, ) from inference_scripts.inference_qwen3_8b import ensure_recipe_log_ok BASELINE_WORK_DIR SOURCE_DIR / run_outputs / baseline_demo BASELINE_YAML SOURCE_DIR / recipe_yaml / baseline_demo.yaml BASELINE_WORK_DIR.mkdir(parentsTrue, exist_okTrue) print([Baseline 推理] 開始, flushTrue) resolved_model_path, model_source resolve_model_path(str(MODEL_ID), quiet_model_ioFalse) prepare_runtime_yaml(BASELINE_YAML_TEMPLATE, BASELINE_YAML, model_pathresolved_model_path) print([YAML 準(zhǔn)備] Baseline 配置, flushTrue) print(參考 YAML , BASELINE_YAML_TEMPLATE) print(運(yùn)行 YAML , BASELINE_YAML) print_yaml_focus(BASELINE_YAML) baseline_log_dir BASELINE_WORK_DIR / recipe_logs baseline_log_dir.mkdir(parentsTrue, exist_okTrue) os.environ[QWEN3_RECIPE_ROOT] str(RECIPE_ROOT) os.environ[QWEN3_RUN_YAML] str(BASELINE_YAML) os.environ[QWEN3_RUN_LOG] str(baseline_log_dir / baseline_demo_infer_stdout.log) os.environ[QWEN3_RUN_START_TIME] str(time.perf_counter())什么是YAMLYAMLYAML Aint Markup Language是一種人類可讀的配置文件格式用縮進(jìn)和鍵值對來表示數(shù)據(jù)就像“簡化版的 JSON但不用寫括號(hào)和逗號(hào)”。為什么需要 YAML這套教程里YAML 的作用是用文本描述“怎么跑模型推理”而不需要改 Python 代碼。比如需要改模型路徑、是否開 Profiler、是否開 RMSNorm 融合、生成長度這些內(nèi)容只需要改 YAML 文件不用動(dòng)任何 Python 代碼。YAML 長什么樣運(yùn)行YAML目錄文件地址在cann-learning-hub/Sources/model_inference_optimization/qwen3_8b/recipe_yaml/baseline_demo.yaml上一部分打印的內(nèi)容作為Baseline配置這份YAML的核心特征是關(guān)閉了各類性能優(yōu)化如 Profiler、RMSNorm 融合、緩存編譯等用于作為后續(xù)優(yōu)化對比的“原始基準(zhǔn)”。YAML文件的主要字段及解釋1 model_config模型與推理核心配置字段當(dāng)前值含義model_nameqwen3_8b模型名稱標(biāo)識(shí)model_path/home/developer/.cache/...模型權(quán)重實(shí)際路徑由代碼自動(dòng)填入dtypebfloat16計(jì)算精度BF16兼顧性能與精度exe_modeeager執(zhí)行模式即時(shí)執(zhí)行非圖模式enable_profilerfalse?關(guān)鍵關(guān)閉性能剖析Baseline 不采集數(shù)據(jù)with_ckpttrue啟用檢查點(diǎn)enable_cache_compilefalse?關(guān)閉緩存編譯enable_static_kernelfalse?關(guān)閉靜態(tài)內(nèi)核custom_params子字段自定義優(yōu)化開關(guān)見下custom_params自定義優(yōu)化Baseline 全部關(guān)閉enable_npu_rmsnorm:false關(guān)閉 NPU 原生 RMSNormenable_add_rmsnorm:false關(guān)閉 AddRMSNorm 融合這是第4章要優(yōu)化的核心enable_prefetch:false關(guān)閉預(yù)取prefetch_size:18874368預(yù)取大小雖關(guān)閉但保留默認(rèn)值2data_config數(shù)據(jù)配置dataset:default使用默認(rèn)數(shù)據(jù)集input_truncated_len:256輸入截?cái)嚅L度最多 256 token3parallel_config并行配置全部為1表示單卡運(yùn)行無張量并行/專家并行world_size: 1attn_tp_size: 1注意力張量并行moe_tp_size: 1MoE 張量并行embed_tp_size: 1嵌入層并行l(wèi)mhead_tp_size: 1輸出頭并行4scheduler_config調(diào)度與生成配置max_new_tokens:64最多生成 64 個(gè)新 tokenbatch_size:1單批次推理enable_thinking:false關(guān)閉思考模式Qwen3 特有2.2.2 使用 recipes 命令啟動(dòng)推理%%bash set -euo pipefail # 激活虛擬環(huán)境如果有 source /mnt/workspace/gitCode/cann/cann-learning-hub/tutorials/llm_inference/qwen3_8b/venv/bin/activate # recipes 原生啟動(dòng)命令 echo recipes 啟動(dòng)命令: echo cd ${QWEN3_RECIPE_ROOT} echo bash executor/scripts/infer.sh --model qwen --mode offline --yaml ${QWEN3_RUN_YAML} cd ${QWEN3_RECIPE_ROOT} bash executor/scripts/infer.sh --model qwen --mode offline --yaml ${QWEN3_RUN_YAML} 21 | tee ${QWEN3_RUN_LOG}這部分是 Baseline 推理的實(shí)際啟動(dòng)命令核心作用是通過 recipes 框架在昇騰 NPU 上真正跑一次 Qwen3-8B 推理。注意這部分需要在原代碼開頭加一行激活虛擬環(huán)境的代碼如果有recipes 是什么recipes是昇騰 CANN 生態(tài)里的一套“大模型推理樣例框架”它不是模型本身也不是 PyTorch而是怎么在 NPU 上把某個(gè)大模型跑起來的工程封裝。infer.sh 是什么qwen3_8b/src/inference_scripts/recipe_qwen3_8b/executor/scripts/infer.sh 是 recipes 的統(tǒng)一推理入口腳本。主要任務(wù)解析參數(shù) → 加載環(huán)境 → 定位 YAML → 校驗(yàn)配置 → 調(diào)用launch函數(shù)真正在 NPU 上跑推理。2.2.3 讀取recipes運(yùn)行結(jié)果case_dir find_recipe_case_dir(RECIPE_ROOT, BASELINE_YAML) if case_dir is None: raise RuntimeError(f未找到 recipes 結(jié)果目錄{BASELINE_YAML.stem}) recipe_log_path case_dir / log_0.log ensure_recipe_log_ok(recipe_log_path) run_info { stdout_path: os.environ[QWEN3_RUN_LOG], stderr_path: os.environ[QWEN3_RUN_LOG], wall_time_s: time.perf_counter() - float(os.environ[QWEN3_RUN_START_TIME]), recipe_case_dir: str(case_dir), recipe_log_path: str(recipe_log_path), } baseline_metrics collect_recipe_metrics( RECIPE_ROOT, BASELINE_WORK_DIR, baseline_demo, BASELINE_YAML, run_info, model_idstr(MODEL_ID), resolved_model_pathresolved_model_path, model_sourcemodel_source, optimization_modenone, max_new_tokens64, max_input_tokens256, enable_thinkingFalse, ) print([Baseline 推理] 完成, flushTrue)這部分代碼是Baseline推理的收尾階段核心作用是定位推理結(jié)果目錄檢查日志是否正常 收集并保存推理指標(biāo)為后續(xù)分析和對比做準(zhǔn)備。參數(shù)含義參數(shù)值含義optimization_modenone本次是 Baseline無優(yōu)化max_new_tokens64生成長度限制max_input_tokens256輸入長度限制enable_thinkingFalse關(guān)閉思考模式2.3 查看結(jié)果print([結(jié)果查看] 開始, flushTrue) def status_cn(status): return {ok: 通過, warning: 告警, failed: 失敗}.get(status, status) def yes_no(value): if value is None: return 未記錄 return 是 if value else 否 def load_metrics(work_dir, tag): metrics_path Path(work_dir) / metrics / f{tag}_summary.json metrics json.loads(metrics_path.read_text(encodingutf-8)) print(指標(biāo)文件 , metrics_path) print(recipes YAML , metrics.get(recipe_yaml_path)) print(recipes 結(jié)果目錄 , metrics.get(recipe_case_dir)) print(生成文件 , metrics.get(outputs_path)) print(生成 token 數(shù) , metrics.get(generated_tokens)) print(總耗時(shí)(s) , metrics.get(total_time_s)) print(吞吐(tokens/s) , metrics.get(tokens_per_second)) rmsnorm_status metrics.get(rmsnorm_status) or {} print(RMSNorm 融合 , yes_no(rmsnorm_status.get(rmsnorm_fused))) print(AddRMSNorm 融合 , yes_no(rmsnorm_status.get(add_rmsnorm_fused))) sanity metrics.get(generation_text_sanity) if sanity: print(吐字檢查 , status_cn(sanity.get(status)), f警告{sanity.get(num_with_warnings)}, f失敗{sanity.get(num_failed)}) return metrics baseline_metrics load_metrics(BASELINE_WORK_DIR, baseline_demo) print() print(生成文本:) for line in Path(baseline_metrics[outputs_path]).read_text(encodingutf-8).splitlines(): if line.strip(): print(json.loads(line).get(generated_text, )) print([結(jié)果查看] 完成, flushTrue)這段代碼讀取并展示剛才那次推理的完整指標(biāo)和生成文本便于直觀確認(rèn) Baseline 狀態(tài)。運(yùn)行結(jié)果[結(jié)果查看] 開始 指標(biāo)文件 /mnt/workspace/gitCode/cann/cann-learning-hub/Sources/model_inference_optimization/qwen3_8b/run_outputs/baseline_demo/metrics/baseline_demo_summary.json recipes YAML /mnt/workspace/gitCode/cann/cann-learning-hub/Sources/model_inference_optimization/qwen3_8b/recipe_yaml/baseline_demo.yaml recipes 結(jié)果目錄 /mnt/workspace/gitCode/cann/cann-learning-hub/tutorials/llm_inference/qwen3_8b/src/inference_scripts/recipe_qwen3_8b/models/qwen/res/20261013/qwen3_8b_baseline_demo 生成文件 /mnt/workspace/gitCode/cann/cann-learning-hub/Sources/model_inference_optimization/qwen3_8b/run_outputs/baseline_demo/outputs/baseline_demo_generations.jsonl 生成 token 數(shù) 38 總耗時(shí)(s) 5.3463899999999995 吞吐(tokens/s) 7.10759970746616 RMSNorm 融合 否 AddRMSNorm 融合 否 吐字檢查 通過 警告0 失敗0 生成文本: The output of an **attention function** is a **weighted sum of the value vectors**, where the weights are determined by the **similarity between the query vector and each key vector**. [結(jié)果查看] 完成打印內(nèi)容實(shí)際值含義指標(biāo)文件路徑指標(biāo) JSON 的位置recipes YAML運(yùn)行 YAML 路徑用的是哪個(gè)配置recipes 結(jié)果目錄推理產(chǎn)物目錄日志、輸出等存放位置生成文件outputs 路徑生成文本的 JSONL 文件生成 token 數(shù)38實(shí)際生成了 38 個(gè) token總耗時(shí)(s)5.35推理總耗時(shí) 5.35 秒吞吐(tokens/s)7.11每秒生成 7.11 個(gè) tokenRMSNorm 融合否確認(rèn)關(guān)閉AddRMSNorm 融合否確認(rèn)關(guān)閉吐字檢查通過無告警、無失敗3 Profiling分析這部分在Baseline的基礎(chǔ)上打開torch_npu.profiler。torch_npu.profiler 是 PyTorch 在昇騰 NPU 上的性能剖析器它底層會(huì)調(diào)用 CANN / Ascend Profiling? 能力把三層數(shù)據(jù)對起來1Host 側(cè)Python 調(diào)用、PyTorch 算子下發(fā)、CPU 活動(dòng)2CANN 側(cè)算子編譯、任務(wù)調(diào)度、內(nèi)存拷貝、流同步3NPU 側(cè)AI Core / AI CPU / Vector 核上的 kernel 執(zhí)行時(shí)間、耗時(shí)、等待時(shí)間它的目的不是“讓模型更快”而是先看清楚 Baseline 推理里時(shí)間到底耗在哪些算子上了。準(zhǔn)備工作與運(yùn)行環(huán)境工作區(qū)準(zhǔn)備的流程和baseline一致。準(zhǔn)備YAML的時(shí)候準(zhǔn)備的是Profiling YAML在baseline的基礎(chǔ)上打開Profiling。其余運(yùn)行流程與baseline一致。Profiling YAML中model_config.enable_profiler 設(shè)為true其余推理參數(shù)保持和 Baseline 一致。YAML 文件 /mnt/workspace/gitCode/cann/cann-learning-hub/Sources/model_inference_optimization/qwen3_8b/recipe_yaml/baseline_profile.yaml model_path /home/developer/.cache/modelscope/models/Qwen--Qwen3-8B/snapshots/master enable_profiler True # 注意這里的enable_profiler即yaml里打開了profiler enable_npu_rmsnorm False enable_add_rmsnorm False enable_prefetch False prefetch_size None dataset default input_truncated_len 256 max_new_tokens 64 batch_size 1結(jié)果分析[Profiler 產(chǎn)物] 開始 指標(biāo)文件 /mnt/workspace/gitCode/cann/cann-learning-hub/Sources/model_inference_optimization/qwen3_8b/run_outputs/baseline_profile/metrics/baseline_profile_summary.json 生成 token 數(shù) 38 Profiler 開啟 是 Profile 目錄 /mnt/workspace/gitCode/cann/cann-learning-hub/tutorials/llm_inference/qwen3_8b/src/inference_scripts/recipe_qwen3_8b/models/qwen/res/20261013/qwen3_8b_baseline_profile/prof prof/prefill /mnt/workspace/gitCode/cann/cann-learning-hub/tutorials/llm_inference/qwen3_8b/src/inference_scripts/recipe_qwen3_8b/models/qwen/res/20261013/qwen3_8b_baseline_profile/prof/prefill 一級目錄 7f27e7db10fd4ffda8f0b031bf276588_11042_20261013190353199_ascend_pt kernel_details.csv 已生成 trace_view.json 已生成 prof/decode /mnt/workspace/gitCode/cann/cann-learning-hub/tutorials/llm_inference/qwen3_8b/src/inference_scripts/recipe_qwen3_8b/models/qwen/res/20261013/qwen3_8b_baseline_profile/prof/decode 一級目錄 7f27e7db10fd4ffda8f0b031bf276588_11042_20261013190359433_ascend_pt kernel_details.csv 已生成 trace_view.json 已生成 吐字檢查 通過 警告0 失敗0 Profiler 摘要文件 /mnt/workspace/gitCode/cann/cann-learning-hub/Sources/model_inference_optimization/qwen3_8b/run_outputs/baseline_profile/profiler_summary/baseline_profile_summary.txt Profiler summary from kernel_details source: /mnt/workspace/gitCode/cann/cann-learning-hub/tutorials/llm_inference/qwen3_8b/src/inference_scripts/recipe_qwen3_8b/models/qwen/res/20261013/qwen3_8b_baseline_profile/prof/decode/7f27e7db10fd4ffda8f0b031bf276588_11042_20261013190359433_ascend_pt/ASCEND_PROFILER_OUTPUT/kernel_details.csv OP Type Count Total(us) Ratio(%) ---------------------------------------------------------------------------------------- aclnnMatmulWeightNz_MatMulCommon_MatMulV2 435 36531.740 68.653 aclnnMul_MulAiCore_Mul 978 3579.760 6.727 aclnnPowTensorScalar_PowsAiCore_Pows 435 2855.780 5.367 FusedInferAttentionScore 108 2413.320 4.535 aclnnMean_ReduceMeanAiCore_ReduceMean 435 2318.040 4.356 ... aclnnInplaceOne_OnesLikeAiCore_OnesLike 3 35.100 0.066 aclnnEmbedding_GatherV2AiCore_GatherV2 3 31.100 0.058 ... 僅顯示前 24 行共 35 行 [Profiler 產(chǎn)物] 完成[分析結(jié)論] 開始 摘要類型 kernel_details 生成 token 數(shù) 38 Baseline RMSNorm 融合 否 Baseline AddRMSNorm 融合 否 Top 算子: - aclnnMatmulWeightNz_MatMulCommon_MatMulV2: 次數(shù)435總耗時(shí)(us)36531.740占比68.653% - aclnnMul_MulAiCore_Mul: 次數(shù)978總耗時(shí)(us)3579.760占比6.727% - aclnnPowTensorScalar_PowsAiCore_Pows: 次數(shù)435總耗時(shí)(us)2855.780占比5.367% - FusedInferAttentionScore: 次數(shù)108總耗時(shí)(us)2413.320占比4.535% - aclnnMean_ReduceMeanAiCore_ReduceMean: 次數(shù)435總耗時(shí)(us)2318.040占比4.356% - aclnnInplaceCopy_CastAiCore_Cast: 次數(shù)873總耗時(shí)(us)1377.980占比2.590% - aclnnScatterNdUpdate_ScatterNdUpdateAiCore_ScatterNdUpdate: 次數(shù)216總耗時(shí)(us)1090.060占比2.049% - aclnnAdds_AddAiCore_Add: 次數(shù)441總耗時(shí)(us)625.340占比1.175% - aclnnRsqrt_RsqrtAiCore_Rsqrt: 次數(shù)435總耗時(shí)(us)533.560占比1.003% - aclnnAdd_AddAiCore_Add: 次數(shù)219總耗時(shí)(us)473.720占比0.890% 結(jié)論: - Baseline YAML 中 Dense RMSNorm 兩個(gè)融合開關(guān)均為關(guān)閉因此本章看到的是融合前的執(zhí)行鏈路。 - 摘要中沒有直接出現(xiàn)融合后的 RMSNorm 算子名說明 Baseline 仍按小算子鏈路執(zhí)行。 - 結(jié)合 QwenRMSNorm 的計(jì)算式摘要中的歸一化相關(guān)小算子包括aclnnAdd_CastAiCore_Cast、aclnnArgMax_CastAiCore_Cast、aclnnGather_CastAiCore_Cast、aclnnInplaceCopy_CastAiCore_Cast、aclnnMax_ReduceMaxAiCore_ReduceMax、aclnnMean_ReduceMeanAiCore_ReduceMean、aclnnPowTensorScalar_PowsAiCore_Pows、aclnnRemainderTensorScalar_BroadcastToAiCore_BroadcastTo。 - 摘要中也可以看到殘差 Add 相關(guān)算子aclnnAdd_AddAiCore_Add、aclnnAdd_CastAiCore_Cast、aclnnAdds_AddAiCore_Add。 - 下一章保持 recipes 的 YAML infer.sh 工作流只切換 Dense RMSNorm NPU 融合開關(guān)對比精度、吐字健康和吞吐變化。 分析文件 /mnt/workspace/gitCode/cann/cann-learning-hub/Sources/model_inference_optimization/qwen3_8b/run_outputs/profiling_analysis_summary.json [分析結(jié)論] 完成算子耗時(shí)排名排名算子次數(shù)耗時(shí)(us)占比屬于什么1MatMul?43536531.7468.65%?線性層QKV/O/FFN2Mul?9783579.766.73%RMSNorm 小算子?3Pow?4352855.785.37%RMSNorm 小算子?4FusedInferAttentionScore?1082413.324.54%注意力融合算子5ReduceMean?4352318.044.36%RMSNorm 小算子?6Cast8731377.982.59%類型轉(zhuǎn)換7ScatterNdUpdate2161090.062.05%KV Cache 更新8Adds441625.341.18%殘差 Add?9Rsqrt?435533.561.00%RMSNorm 小算子?10Add219473.720.89%殘差 Add?可以看出MatMul矩陣乘法耗時(shí)最長其余部分RMSNorm 被拆成了 8 個(gè)小算子殘差 Add 也是小算子沒有融合。Profiling 分析證實(shí)了 RMSNorm 未融合時(shí)被拆成 8 個(gè)小算子Pow/ReduceMean/Rsqrt/Mul 等導(dǎo)致launch 次數(shù)多、訪存多、耗時(shí)高是明確的性能瓶頸。下一章將打開enable_add_rmsnorm融合開關(guān)用 A/B 對比驗(yàn)證優(yōu)化效果。trace_view.json 時(shí)間線圖譜trace_view.json時(shí)間線圖譜呈現(xiàn)了 Qwen3-8B 在昇騰 NPU 上Prefill?與Decode?兩個(gè)階段的執(zhí)行特征。Prefill對應(yīng)prof/prefill即處理輸入提示詞的階段。時(shí)間跨度與標(biāo)記整體約 150ms50ms-200ms標(biāo)記為ProfilerStep#0末尾帶有Dec? 標(biāo)記暗示向 Decode 階段的過渡。算子形態(tài)NPUStream 46/49上的色塊相對“大塊”雖然密集但多為一次性大粒度計(jì)算如首輪的 QKV 大矩陣乘。重疊與頻率CPUPython/CANN與 NPU 有明顯的調(diào)度交互。底部Overlap Analysis中 Computing 與 Free 交替由于僅執(zhí)行一次整體耗時(shí)較短未暴露出小算子的反復(fù)累加效應(yīng)。Decode時(shí)間跨度與標(biāo)記整體約 350ms100ms-450ms包含#3、#4、#5連續(xù)多步每步對應(yīng)一個(gè)或幾個(gè)Token 的生成。極度密集的細(xì)碎條紋這是最核心的視覺特征。NPU 時(shí)間線上布滿高頻、細(xì)碎的彩色條紋直觀對應(yīng)了前文數(shù)據(jù)中435次 MatMul? 以及伴隨的Pow/ReduceMean/Rsqrt/MulRMSNorm 未融合等小算子。多流并行Stream 46 與 Stream 49 交錯(cuò)CANN 線程持續(xù)調(diào)度AI Core 頻率滿載淺藍(lán)色長條說明計(jì)算單元持續(xù)忙碌。對比維度PrefillDecode性能含義執(zhí)行次數(shù)?單次Step#0多次迭代Step#3/4/5Decode 重復(fù)放大了局部開銷算子粒度?相對大塊計(jì)算密集極度細(xì)碎訪存/調(diào)度密集?Decode 中小算子 Launch 開銷顯著時(shí)間線密度?密集但無高頻鋸齒滿屏細(xì)碎條紋印證 RMSNorm 拆分成 4~8 個(gè)小算子耗時(shí)占比?較短~150ms較長~350ms端到端吞吐受 Decode 階段主導(dǎo)圖譜直觀揭示了Decode階段因 RMSNorm 未融合產(chǎn)生大量小算子調(diào)度細(xì)碎條紋是端到端性能的可優(yōu)化瓶頸Prefill 則相對健康。后續(xù)