話實(shí)戰(zhàn)指南：Transformers、ModelScope 與 Streamlit Web Demo)
大模型人工智能基礎(chǔ)模型AI Agent【免費(fèi)下載鏈接】InternLMOfficial release of InternLM series (InternLM, InternLM2, InternLM2.5, InternLM3).項(xiàng)目地址https://gitcode.com/gh_mirrors/in/InternLM點(diǎn)擊查看免費(fèi)下載本指南基于 InternLM 官方倉(cāng)庫(kù)的 chat/README.md系統(tǒng)講解 InternLM3-8B-Instruct 的三種使用方式通過 HuggingFace Transformers 加載推理、通過 ModelScope 加載推理、以及通過 Streamlit 前端 Web Demo 進(jìn)行交互式對(duì)話。讀完本文你將掌握模型加載時(shí)的精度選擇與顯存優(yōu)化技巧、基于 ChatML 模板的多輪對(duì)話組裝方式、量化推理8-bit/4-bit的完整配置以及 Web Demo 中「普通回答 / 深度思考Deep Thinking」雙推理模式與對(duì)比功能的源碼級(jí)實(shí)現(xiàn)原理。為什么選擇這三條推理路徑InternLM3-Instruct 系列當(dāng)前倉(cāng)庫(kù)針對(duì)internlm/internlm3-8b-instruct是上海人工智能實(shí)驗(yàn)室開源的可商用對(duì)話模型。倉(cāng)庫(kù)將其推理入口整理為三條互補(bǔ)的路徑覆蓋從「腳本內(nèi)聯(lián)推理」到「瀏覽器可視化對(duì)話」再到「服務(wù)化部署」的完整場(chǎng)景Transformers 路徑面向 Python 開發(fā)環(huán)境模型權(quán)重、分詞器均可由 HuggingFace 生態(tài)的標(biāo)準(zhǔn) API 加載適合在自有代碼中集成推理邏輯ModelScope 路徑面向國(guó)內(nèi)網(wǎng)絡(luò)環(huán)境與 ModelScope 生態(tài)通過snapshot_download將模型權(quán)重先落地為本地目錄再加載Web Demo 路徑基于 Streamlit 的前端界面開箱即用地演示多輪對(duì)話并支持推理模式切換與結(jié)果對(duì)比。倉(cāng)庫(kù)根目錄的 requirements.txt 給出了基礎(chǔ)依賴transformers4.38、streamlit、sentencepiece而 Web Demo 章節(jié)要求transformers4.48實(shí)際使用時(shí)請(qǐng)以更高版本為準(zhǔn)。通過 Transformers 加載 InternLM3-8B-Instructchat/README.md提供的最小可用代碼如下我們先逐段拆解其關(guān)鍵點(diǎn)import torch from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer AutoTokenizer.from_pretrained(internlm/internlm3-8b-instruct, trust_remote_codeTrue) # Set torch_dtypetorch.float16 to load model in float16, otherwise it will be loaded as float32 and might cause OOM Error. model AutoModelForCausalLM.from_pretrained(internlm/internlm3-8b-instruct, trust_remote_codeTrue, torch_dtypetorch.float16) # (Optional) If on low resource devices, you can load model in 4-bit or 8-bit to further save GPU memory via bitsandbytes. # InternLM3 8B in 4bit will cost nearly 8GB GPU memory. # pip install -U bitsandbytes # 8-bit: model AutoModelForCausalLM.from_pretrained(internlm/internlm3-8b-instruct, device_mapauto, trust_remote_codeTrue, load_in_8bitTrue) # 4-bit: model AutoModelForCausalLM.from_pretrained(internlm/internlm3-8b-instruct, device_mapauto, trust_remote_codeTrue, load_in_4bitTrue) model model.eval() messages [ {role: system, content: You are an AI assistant whose name is InternLM.}, {role: user, content: Please tell me five scenic spots in Shanghai}, ] tokenized_chat tokenizer.apply_chat_template(messages, tokenizeTrue, add_generation_promptTrue, return_tensorspt) generated_ids model.generate(tokenized_chat, max_new_tokens512) generated_ids [ output_ids[len(input_ids):] for input_ids, output_ids in zip(tokenized_chat, generated_ids) ] response tokenizer.batch_decode(generated_ids)[0]參數(shù)要點(diǎn)與顯存控制trust_remote_codeTrue是必需的InternLM3 的建模代碼modeling_internlm3.py與分詞器邏輯隨權(quán)重倉(cāng)庫(kù)發(fā)布必須顯式授權(quán)加載遠(yuǎn)程代碼否則from_pretrained會(huì)直接報(bào)錯(cuò)。torch_dtypetorch.float16用于控制加載精度不指定時(shí) Transformers 默認(rèn)按 float32 加載8B 參數(shù)模型會(huì)成倍占用顯存極易觸發(fā) OOM顯存充足也可改用torch.bfloat16。倉(cāng)庫(kù)測(cè)試用例 tests/test_hf_model.py 中對(duì) InternLM2.5 系列模型的加載方式與此完全一致torch_dtypetorch.float16, trust_remote_codeTrue可視為官方驗(yàn)證過的標(biāo)準(zhǔn)寫法。低顯存設(shè)備可疊加 bitsandbytes 量化8-bit 與 4-bit 兩種量化加載方式是可選分支需要在加載前pip install -U bitsandbytes并配合device_mapauto自動(dòng)切分權(quán)重。官方注明 InternLM3 8B 以 4-bit 加載時(shí)顯存占用約為 8GB適合單卡消費(fèi)級(jí) GPU 運(yùn)行。model.eval()切換到推理模式關(guān)閉 Dropout 等訓(xùn)練期行為。對(duì)話模板與生成截?cái)鄑okenizer.apply_chat_template(messages, tokenizeTrue, add_generation_promptTrue, return_tensorspt)是組裝對(duì)話的關(guān)鍵調(diào)用它將messages列表按 InternLM 的 ChatML 對(duì)話格式序列化為 token并追加生成提示符。關(guān)于該格式的詳細(xì)規(guī)范見倉(cāng)庫(kù)的 chat/chat_format.md。generated_ids中既包含輸入側(cè) token 也包含新生成的 token因此代碼用切片output_ids[len(input_ids):]去掉輸入前綴只解碼新生成的部分。max_new_tokens512限制本次生成的新 token 數(shù)而非總長(zhǎng)度。通過 ModelScope 加載 InternLM3-8B-Instruct面向國(guó)內(nèi)環(huán)境或 ModelScope 生態(tài)用戶chat/README.md給出了等價(jià)的加載方式。核心差異在于先通過snapshot_download將模型權(quán)重同步到本地目錄model_dir后續(xù)代碼與 Transformers 路徑幾乎一致import torch from modelscope import snapshot_download, AutoTokenizer, AutoModelForCausalLM model_dir snapshot_download(Shanghai_AI_Laboratory/internlm3-8b-instruct) tokenizer AutoTokenizer.from_pretrained(model_dir,trust_remote_codeTrue) # Set torch_dtypetorch.float16 to load model in float16, otherwise it will be loaded as float32 and might cause OOM Error. model AutoModelForCausalLM.from_pretrained(model_dir, trust_remote_codeTrue, torch_dtypetorch.float16) # (Optional) If on low resource devices, you can load model in 4-bit or 8-bit to further save GPU memory via bitsandbytes. # InternLM3 8B in 4bit will cost nearly 8GB GPU memory. # pip install -U bitsandbytes # 8-bit: model AutoModelForCausalLM.from_pretrained(model_dir, device_mapauto, trust_remote_codeTrue, load_in_8bitTrue) # 4-bit: model AutoModelForCausalLM.from_pretrained(model_dir, device_mapauto, trust_remote_codeTrue, load_in_4bitTrue) model model.eval() messages [ {role: system, content: You are an AI assistant whose name is InternLM.}, {role: user, content: Please tell me five scenic spots in Shanghai}, ] tokenized_chat tokenizer.apply_chat_template(messages, tokenizeTrue, add_generation_promptTrue, return_tensorspt) generated_ids model.generate(tokenized_chat, max_new_tokens512) generated_ids [ output_ids[len(input_ids):] for input_ids, output_ids in zip(tokenized_chat, generated_ids) ] response tokenizer.batch_decode(generated_ids)[0]注意兩處差異模型 ID 使用 ModelScope 側(cè)的命名空間Shanghai_AI_Laboratory/internlm3-8b-instructAutoTokenizer、AutoModelForCausalLM從modelscope包導(dǎo)入而非transformers。精度、量化、apply_chat_template等其余語(yǔ)義完全相同。通過 Web Demo 進(jìn)行交互式對(duì)話chat/README.md的對(duì)話章節(jié)給出的啟動(dòng)方式極為簡(jiǎn)潔pip install streamlit pip install transformers4.48 streamlit run ./chat/web_demo.py其中./chat/web_demo.py即倉(cāng)庫(kù)根目錄下的 chat/web_demo.py。腳本頭部注釋提醒務(wù)必使用streamlit run啟動(dòng)直接用python chat/web_demo.py可能導(dǎo)致未知問題如需對(duì)外訪問可追加--server.address0.0.0.0 --server.port 7860。核心源碼模型加載與雙推理模式從源碼看該 Demo 不只是簡(jiǎn)單聊天框還實(shí)現(xiàn)了「Normal Response」與「Deep Thinking」兩種推理模式并支持對(duì)比輸出模型加載load_modelchat/web_demo.py 將internlm/internlm3-8b-instruct以torch.bfloat16精度加載到 CUDA并用st.cache_resource緩存避免頁(yè)面刷新時(shí)反復(fù)加載權(quán)重。生成參數(shù)prepare_generation_config側(cè)邊欄提供Max Length滑桿 8~32768默認(rèn) 32768、Top P默認(rèn) 0.8、Temperature默認(rèn) 0.7三個(gè)滑桿以及推理模式單選。默認(rèn)生成配置類GenerationConfigchat/web_demo.py設(shè)定max_length32768、top_p0.8、temperature0.8、do_sampleTrue、repetition_penalty1.005。Deep Thinking 模式開啟后系統(tǒng)提示會(huì)追加一段面向數(shù)學(xué)競(jìng)賽的「深度思考」指令chat/web_demo.py引導(dǎo)模型先充分理解問題、多角度分析、系統(tǒng)化推理、嚴(yán)格論證并反復(fù)驗(yàn)證最終以\boxed{}給出結(jié)論?；卮饡?huì)以紅色「Deep Thinking」前綴標(biāo)識(shí)普通回答則以藍(lán)色「Normal Response」前綴標(biāo)識(shí)postprocess函數(shù)。對(duì)比功能每條機(jī)器人回復(fù)旁都有compare開關(guān)開啟后會(huì)在雙欄中并行渲染普通回答與深度思考回答便于直接對(duì)照兩種推理模式的質(zhì)量差異——這正是 README 中「supports switching between different inference modes and comparing their responses」對(duì)應(yīng)的源碼實(shí)現(xiàn)chat/web_demo.py。對(duì)話歷史組裝combine_history以s|im_start|system\n{meta_instruction}|im_end|\n開頭逐輪按|im_start|user\n...與|im_start|assistant\n...拼接最后以當(dāng)前用戶問題的 assistant 起始符收尾。流式生成generate_interactive逐 token 前向計(jì)算經(jīng) logits processor 處理后按do_sample決定采樣或貪心并在生成時(shí)通過additional_eos_token_id92542顯式指定對(duì)話結(jié)束符即|im_end|詳見下文對(duì)話格式實(shí)現(xiàn)邊生成邊渲染的效果。對(duì)話格式與特殊 TokenChatML 擴(kuò)展Web Demo 中的|im_start|/|im_end|結(jié)構(gòu)來自 InternLM 系列沿用的 ChatML 風(fēng)格對(duì)話格式倉(cāng)庫(kù)文檔 chat/chat_format.md 對(duì)其有完整規(guī)范常規(guī)對(duì)話包含system、user、assistant三個(gè)角色為支持智能體應(yīng)用還引入了environment角色與工具調(diào)用相關(guān) token。該格式的核心要點(diǎn)每輪對(duì)話以|im_start|role開頭、以|im_end|結(jié)尾role可取system、user、assistant、environment。詞表中維護(hù)了 6 個(gè)特殊 token 映射|im_start|對(duì)話開始符→ token ID92543|im_end|對(duì)話結(jié)束符→ token ID92542|action_start|調(diào)用外部工具開始符→ token ID92541|action_end|調(diào)用外部工具結(jié)束符→ token ID92540|interpreter|代碼解釋器→ token ID92539|plugin|外部插件/常規(guī)工具→ token ID92538工具調(diào)用場(chǎng)景下模型會(huì)流式輸出「思考文字 |action_start||plugin| JSON 格式調(diào)用參數(shù) |action_end|」環(huán)境返回結(jié)果則以|im_start|environment name|plugin|開頭。代碼解釋器場(chǎng)景使用|interpreter|與 markdown 代碼塊支持?jǐn)?shù)據(jù)分析、復(fù)雜計(jì)算、文件操作等完整示例見 chat/chat_format.md。理解這套格式有助于排查 Web Demo 中生成截?cái)唷⒔Y(jié)束符缺失等問題——additional_eos_token_id92542正是因?yàn)槠胀?EOS token 無法可靠標(biāo)識(shí)多輪對(duì)話的輪次邊界。進(jìn)階以 LMDeploy 做服務(wù)化推理若需更高吞吐的批量推理或 OpenAI 兼容的 HTTP 服務(wù)倉(cāng)庫(kù) chat/lmdeploy.md 提供了基于 LMDeploy 的補(bǔ)充路徑面向 InternLM2.5 系列Python 3.8pip install lmdeploy0.2.1離線批處理只需四行代碼from lmdeploy import pipeline pipe pipeline(internlm/internlm2_5-7b-chat) response pipe([Hi, pls intro yourself, Shanghai is]) print(response)長(zhǎng)文本場(chǎng)景可借助 dynamic ntk 將上下文外推到 200Kfrom lmdeploy import pipeline, TurbomindEngineConfig engine_config TurbomindEngineConfig(session_len200000, rope_scaling_factor2.0) pipe pipeline(internlm/internlm2_5-7b-chat, backend_engineengine_config) gen_config GenerationConfig(top_p0.8, top_k40, temperature0.8, max_new_tokens1024) response pipe(prompt, gen_configgen_config) print(response)一鍵部署服務(wù)并測(cè)試lmdeploy serve api_server internlm/internlm2_5-7b-chat lmdeploy serve api_client http://0.0.0.0:23333api_server默認(rèn)監(jiān)聽 23333 端口提供的 RESTful API 兼容 OpenAI 接口也可通過 Swagger UIhttp://0.0.0.0:23333在線試用。倉(cāng)庫(kù)測(cè)試 tests/test_hf_model.py 還演示了以TurbomindEngineConfig(model_formatawq)加載 AWQ 量化版模型internlm/internlm2-chat-20b-4bits做批量推理的寫法可作為低顯存服務(wù)化的參考?？焖衮?yàn)證清單加載時(shí)務(wù)必帶trust_remote_codeTrue并顯式指定torch_dtype否則可能 OOM低顯存環(huán)境按需選擇 8-bit/4-bit 量化4-bit 下 InternLM3 8B 顯存占用約 8GB對(duì)話組裝使用tokenizer.apply_chat_template(...)生成后需切片去除輸入前綴再解碼Web Demo 一律用streamlit run ./chat/web_demo.py啟動(dòng)transformers4.48、streamlit缺一不可多輪對(duì)話邊界由|im_end|token 92542標(biāo)識(shí)Web Demo 中已將其作為附加 EOS token 傳入生成函數(shù)生產(chǎn)級(jí)推理可遷移至 chat/lmdeploy.md 描述的 LMDeploy pipeline / api_server 方案。以上推理流程與參數(shù)均以本倉(cāng)庫(kù) chat/README.md 及其對(duì)應(yīng)源碼chat/web_demo.py、chat/chat_format.md、tests/test_hf_model.py為準(zhǔn)可直接復(fù)制運(yùn)行并在此基礎(chǔ)上二次開發(fā)。贊分享大模型人工智能基礎(chǔ)模型AI Agent【免費(fèi)下載鏈接】InternLMOfficial release of InternLM series (InternLM, InternLM2, InternLM2.5, InternLM3).項(xiàng)目地址https://gitcode.com/gh_mirrors/in/InternLM點(diǎn)擊查看免費(fèi)下載相關(guān)推薦InternLM3-8B-Instruct 對(duì)話推理實(shí)戰(zhàn)Transformers 與 ModelScope 加載、Streamlit 網(wǎng)頁(yè)對(duì)話與推理模式切換InternLM3 8B Instruct 對(duì)話推理實(shí)戰(zhàn)Transformers 與 ModelScope 加載、Streamlit 網(wǎng)頁(yè)對(duì)話與推理模式切換大模型人工智能基礎(chǔ)模型AI AgentInternLM-Chat-7B 對(duì)話 Web Demo 部署指南基于 Streamlit 的本地 Web 對(duì)話全流程實(shí)戰(zhàn)InternLM Chat 7B 對(duì)話 Web Demo 部署指南基于 Streamlit 的本地 Web 對(duì)話全流程實(shí)戰(zhàn) 本篇指南以 Datawhale「開大模型人工智能教程本地部署微調(diào)Datawhale Self-LLM 實(shí)戰(zhàn)基于 transformers 與 peft 的 InternLM3-8B-Instruct LoRA 微調(diào)教程Datawhale Self LLM 實(shí)戰(zhàn)基于 transformers 與 peft 的 InternLM3 8B Instruct LoRA 微調(diào)教程 本大模型人工智能教程本地部署微調(diào)上一篇PowerSploit 域滲透?jìng)刹焓褂?Get-DomainGPOUserLocalGroupMapping 通過 GPO 關(guān)聯(lián)枚舉用戶本地組成員關(guān)系下一篇IQInvision 攝像頭 Telnet 默認(rèn)憑據(jù)檢測(cè)routersploit 字典攻擊模塊實(shí)戰(zhàn)指南創(chuàng)作聲明：本文部分內(nèi)容由AI輔助生成（AIGC），僅供參考 InternLM3-Instruct 推理與對(duì)話實(shí)戰(zhàn)指南：Transformers、ModelScope 與 Streamlit Web Demo
大模型人工智能基礎(chǔ)模型AI Agent【免費(fèi)下載鏈接】InternLMOfficial release of InternLM series (InternLM, InternLM2, InternLM2.5, InternLM3).項(xiàng)目地址https://gitcode.com/gh_mirrors/in/InternLM點(diǎn)擊查看免費(fèi)下載本指南基于 InternLM 官方倉(cāng)庫(kù)的 chat/README.md系統(tǒng)講解 InternLM3-8B-Instruct 的三種使用方式通過 HuggingFace Transformers 加載推理、通過 ModelScope 加載推理、以及通過 Streamlit 前端 Web Demo 進(jìn)行交互式對(duì)話。讀完本文你將掌握模型加載時(shí)的精度選擇與顯存優(yōu)化技巧、基于 ChatML 模板的多輪對(duì)話組裝方式、量化推理8-bit/4-bit的完整配置以及 Web Demo 中「普通回答 / 深度思考Deep Thinking」雙推理模式與對(duì)比功能的源碼級(jí)實(shí)現(xiàn)原理。為什么選擇這三條推理路徑InternLM3-Instruct 系列當(dāng)前倉(cāng)庫(kù)針對(duì)internlm/internlm3-8b-instruct是上海人工智能實(shí)驗(yàn)室開源的可商用對(duì)話模型。倉(cāng)庫(kù)將其推理入口整理為三條互補(bǔ)的路徑覆蓋從「腳本內(nèi)聯(lián)推理」到「瀏覽器可視化對(duì)話」再到「服務(wù)化部署」的完整場(chǎng)景Transformers 路徑面向 Python 開發(fā)環(huán)境模型權(quán)重、分詞器均可由 HuggingFace 生態(tài)的標(biāo)準(zhǔn) API 加載適合在自有代碼中集成推理邏輯ModelScope 路徑面向國(guó)內(nèi)網(wǎng)絡(luò)環(huán)境與 ModelScope 生態(tài)通過snapshot_download將模型權(quán)重先落地為本地目錄再加載Web Demo 路徑基于 Streamlit 的前端界面開箱即用地演示多輪對(duì)話并支持推理模式切換與結(jié)果對(duì)比。倉(cāng)庫(kù)根目錄的 requirements.txt 給出了基礎(chǔ)依賴transformers4.38、streamlit、sentencepiece而 Web Demo 章節(jié)要求transformers4.48實(shí)際使用時(shí)請(qǐng)以更高版本為準(zhǔn)。通過 Transformers 加載 InternLM3-8B-Instructchat/README.md提供的最小可用代碼如下我們先逐段拆解其關(guān)鍵點(diǎn)import torch from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer AutoTokenizer.from_pretrained(internlm/internlm3-8b-instruct, trust_remote_codeTrue) # Set torch_dtypetorch.float16 to load model in float16, otherwise it will be loaded as float32 and might cause OOM Error. model AutoModelForCausalLM.from_pretrained(internlm/internlm3-8b-instruct, trust_remote_codeTrue, torch_dtypetorch.float16) # (Optional) If on low resource devices, you can load model in 4-bit or 8-bit to further save GPU memory via bitsandbytes. # InternLM3 8B in 4bit will cost nearly 8GB GPU memory. # pip install -U bitsandbytes # 8-bit: model AutoModelForCausalLM.from_pretrained(internlm/internlm3-8b-instruct, device_mapauto, trust_remote_codeTrue, load_in_8bitTrue) # 4-bit: model AutoModelForCausalLM.from_pretrained(internlm/internlm3-8b-instruct, device_mapauto, trust_remote_codeTrue, load_in_4bitTrue) model model.eval() messages [ {role: system, content: You are an AI assistant whose name is InternLM.}, {role: user, content: Please tell me five scenic spots in Shanghai}, ] tokenized_chat tokenizer.apply_chat_template(messages, tokenizeTrue, add_generation_promptTrue, return_tensorspt) generated_ids model.generate(tokenized_chat, max_new_tokens512) generated_ids [ output_ids[len(input_ids):] for input_ids, output_ids in zip(tokenized_chat, generated_ids) ] response tokenizer.batch_decode(generated_ids)[0]參數(shù)要點(diǎn)與顯存控制trust_remote_codeTrue是必需的InternLM3 的建模代碼modeling_internlm3.py與分詞器邏輯隨權(quán)重倉(cāng)庫(kù)發(fā)布必須顯式授權(quán)加載遠(yuǎn)程代碼否則from_pretrained會(huì)直接報(bào)錯(cuò)。torch_dtypetorch.float16用于控制加載精度不指定時(shí) Transformers 默認(rèn)按 float32 加載8B 參數(shù)模型會(huì)成倍占用顯存極易觸發(fā) OOM顯存充足也可改用torch.bfloat16。倉(cāng)庫(kù)測(cè)試用例 tests/test_hf_model.py 中對(duì) InternLM2.5 系列模型的加載方式與此完全一致torch_dtypetorch.float16, trust_remote_codeTrue可視為官方驗(yàn)證過的標(biāo)準(zhǔn)寫法。低顯存設(shè)備可疊加 bitsandbytes 量化8-bit 與 4-bit 兩種量化加載方式是可選分支需要在加載前pip install -U bitsandbytes并配合device_mapauto自動(dòng)切分權(quán)重。官方注明 InternLM3 8B 以 4-bit 加載時(shí)顯存占用約為 8GB適合單卡消費(fèi)級(jí) GPU 運(yùn)行。model.eval()切換到推理模式關(guān)閉 Dropout 等訓(xùn)練期行為。對(duì)話模板與生成截?cái)鄑okenizer.apply_chat_template(messages, tokenizeTrue, add_generation_promptTrue, return_tensorspt)是組裝對(duì)話的關(guān)鍵調(diào)用它將messages列表按 InternLM 的 ChatML 對(duì)話格式序列化為 token并追加生成提示符。關(guān)于該格式的詳細(xì)規(guī)范見倉(cāng)庫(kù)的 chat/chat_format.md。generated_ids中既包含輸入側(cè) token 也包含新生成的 token因此代碼用切片output_ids[len(input_ids):]去掉輸入前綴只解碼新生成的部分。max_new_tokens512限制本次生成的新 token 數(shù)而非總長(zhǎng)度。通過 ModelScope 加載 InternLM3-8B-Instruct面向國(guó)內(nèi)環(huán)境或 ModelScope 生態(tài)用戶chat/README.md給出了等價(jià)的加載方式。核心差異在于先通過snapshot_download將模型權(quán)重同步到本地目錄model_dir后續(xù)代碼與 Transformers 路徑幾乎一致import torch from modelscope import snapshot_download, AutoTokenizer, AutoModelForCausalLM model_dir snapshot_download(Shanghai_AI_Laboratory/internlm3-8b-instruct) tokenizer AutoTokenizer.from_pretrained(model_dir,trust_remote_codeTrue) # Set torch_dtypetorch.float16 to load model in float16, otherwise it will be loaded as float32 and might cause OOM Error. model AutoModelForCausalLM.from_pretrained(model_dir, trust_remote_codeTrue, torch_dtypetorch.float16) # (Optional) If on low resource devices, you can load model in 4-bit or 8-bit to further save GPU memory via bitsandbytes. # InternLM3 8B in 4bit will cost nearly 8GB GPU memory. # pip install -U bitsandbytes # 8-bit: model AutoModelForCausalLM.from_pretrained(model_dir, device_mapauto, trust_remote_codeTrue, load_in_8bitTrue) # 4-bit: model AutoModelForCausalLM.from_pretrained(model_dir, device_mapauto, trust_remote_codeTrue, load_in_4bitTrue) model model.eval() messages [ {role: system, content: You are an AI assistant whose name is InternLM.}, {role: user, content: Please tell me five scenic spots in Shanghai}, ] tokenized_chat tokenizer.apply_chat_template(messages, tokenizeTrue, add_generation_promptTrue, return_tensorspt) generated_ids model.generate(tokenized_chat, max_new_tokens512) generated_ids [ output_ids[len(input_ids):] for input_ids, output_ids in zip(tokenized_chat, generated_ids) ] response tokenizer.batch_decode(generated_ids)[0]注意兩處差異模型 ID 使用 ModelScope 側(cè)的命名空間Shanghai_AI_Laboratory/internlm3-8b-instructAutoTokenizer、AutoModelForCausalLM從modelscope包導(dǎo)入而非transformers。精度、量化、apply_chat_template等其余語(yǔ)義完全相同。通過 Web Demo 進(jìn)行交互式對(duì)話chat/README.md的對(duì)話章節(jié)給出的啟動(dòng)方式極為簡(jiǎn)潔pip install streamlit pip install transformers4.48 streamlit run ./chat/web_demo.py其中./chat/web_demo.py即倉(cāng)庫(kù)根目錄下的 chat/web_demo.py。腳本頭部注釋提醒務(wù)必使用streamlit run啟動(dòng)直接用python chat/web_demo.py可能導(dǎo)致未知問題如需對(duì)外訪問可追加--server.address0.0.0.0 --server.port 7860。核心源碼模型加載與雙推理模式從源碼看該 Demo 不只是簡(jiǎn)單聊天框還實(shí)現(xiàn)了「Normal Response」與「Deep Thinking」兩種推理模式并支持對(duì)比輸出模型加載load_modelchat/web_demo.py 將internlm/internlm3-8b-instruct以torch.bfloat16精度加載到 CUDA并用st.cache_resource緩存避免頁(yè)面刷新時(shí)反復(fù)加載權(quán)重。生成參數(shù)prepare_generation_config側(cè)邊欄提供Max Length滑桿 8~32768默認(rèn) 32768、Top P默認(rèn) 0.8、Temperature默認(rèn) 0.7三個(gè)滑桿以及推理模式單選。默認(rèn)生成配置類GenerationConfigchat/web_demo.py設(shè)定max_length32768、top_p0.8、temperature0.8、do_sampleTrue、repetition_penalty1.005。Deep Thinking 模式開啟后系統(tǒng)提示會(huì)追加一段面向數(shù)學(xué)競(jìng)賽的「深度思考」指令chat/web_demo.py引導(dǎo)模型先充分理解問題、多角度分析、系統(tǒng)化推理、嚴(yán)格論證并反復(fù)驗(yàn)證最終以\boxed{}給出結(jié)論?；卮饡?huì)以紅色「Deep Thinking」前綴標(biāo)識(shí)普通回答則以藍(lán)色「Normal Response」前綴標(biāo)識(shí)postprocess函數(shù)。對(duì)比功能每條機(jī)器人回復(fù)旁都有compare開關(guān)開啟后會(huì)在雙欄中并行渲染普通回答與深度思考回答便于直接對(duì)照兩種推理模式的質(zhì)量差異——這正是 README 中「supports switching between different inference modes and comparing their responses」對(duì)應(yīng)的源碼實(shí)現(xiàn)chat/web_demo.py。對(duì)話歷史組裝combine_history以s|im_start|system\n{meta_instruction}|im_end|\n開頭逐輪按|im_start|user\n...與|im_start|assistant\n...拼接最后以當(dāng)前用戶問題的 assistant 起始符收尾。流式生成generate_interactive逐 token 前向計(jì)算經(jīng) logits processor 處理后按do_sample決定采樣或貪心并在生成時(shí)通過additional_eos_token_id92542顯式指定對(duì)話結(jié)束符即|im_end|詳見下文對(duì)話格式實(shí)現(xiàn)邊生成邊渲染的效果。對(duì)話格式與特殊 TokenChatML 擴(kuò)展Web Demo 中的|im_start|/|im_end|結(jié)構(gòu)來自 InternLM 系列沿用的 ChatML 風(fēng)格對(duì)話格式倉(cāng)庫(kù)文檔 chat/chat_format.md 對(duì)其有完整規(guī)范常規(guī)對(duì)話包含system、user、assistant三個(gè)角色為支持智能體應(yīng)用還引入了environment角色與工具調(diào)用相關(guān) token。該格式的核心要點(diǎn)每輪對(duì)話以|im_start|role開頭、以|im_end|結(jié)尾role可取system、user、assistant、environment。詞表中維護(hù)了 6 個(gè)特殊 token 映射|im_start|對(duì)話開始符→ token ID92543|im_end|對(duì)話結(jié)束符→ token ID92542|action_start|調(diào)用外部工具開始符→ token ID92541|action_end|調(diào)用外部工具結(jié)束符→ token ID92540|interpreter|代碼解釋器→ token ID92539|plugin|外部插件/常規(guī)工具→ token ID92538工具調(diào)用場(chǎng)景下模型會(huì)流式輸出「思考文字 |action_start||plugin| JSON 格式調(diào)用參數(shù) |action_end|」環(huán)境返回結(jié)果則以|im_start|environment name|plugin|開頭。代碼解釋器場(chǎng)景使用|interpreter|與 markdown 代碼塊支持?jǐn)?shù)據(jù)分析、復(fù)雜計(jì)算、文件操作等完整示例見 chat/chat_format.md。理解這套格式有助于排查 Web Demo 中生成截?cái)唷⒔Y(jié)束符缺失等問題——additional_eos_token_id92542正是因?yàn)槠胀?EOS token 無法可靠標(biāo)識(shí)多輪對(duì)話的輪次邊界。進(jìn)階以 LMDeploy 做服務(wù)化推理若需更高吞吐的批量推理或 OpenAI 兼容的 HTTP 服務(wù)倉(cāng)庫(kù) chat/lmdeploy.md 提供了基于 LMDeploy 的補(bǔ)充路徑面向 InternLM2.5 系列Python 3.8pip install lmdeploy0.2.1離線批處理只需四行代碼from lmdeploy import pipeline pipe pipeline(internlm/internlm2_5-7b-chat) response pipe([Hi, pls intro yourself, Shanghai is]) print(response)長(zhǎng)文本場(chǎng)景可借助 dynamic ntk 將上下文外推到 200Kfrom lmdeploy import pipeline, TurbomindEngineConfig engine_config TurbomindEngineConfig(session_len200000, rope_scaling_factor2.0) pipe pipeline(internlm/internlm2_5-7b-chat, backend_engineengine_config) gen_config GenerationConfig(top_p0.8, top_k40, temperature0.8, max_new_tokens1024) response pipe(prompt, gen_configgen_config) print(response)一鍵部署服務(wù)并測(cè)試lmdeploy serve api_server internlm/internlm2_5-7b-chat lmdeploy serve api_client http://0.0.0.0:23333api_server默認(rèn)監(jiān)聽 23333 端口提供的 RESTful API 兼容 OpenAI 接口也可通過 Swagger UIhttp://0.0.0.0:23333在線試用。倉(cāng)庫(kù)測(cè)試 tests/test_hf_model.py 還演示了以TurbomindEngineConfig(model_formatawq)加載 AWQ 量化版模型internlm/internlm2-chat-20b-4bits做批量推理的寫法可作為低顯存服務(wù)化的參考?？焖衮?yàn)證清單加載時(shí)務(wù)必帶trust_remote_codeTrue并顯式指定torch_dtype否則可能 OOM低顯存環(huán)境按需選擇 8-bit/4-bit 量化4-bit 下 InternLM3 8B 顯存占用約 8GB對(duì)話組裝使用tokenizer.apply_chat_template(...)生成后需切片去除輸入前綴再解碼Web Demo 一律用streamlit run ./chat/web_demo.py啟動(dòng)transformers4.48、streamlit缺一不可多輪對(duì)話邊界由|im_end|token 92542標(biāo)識(shí)Web Demo 中已將其作為附加 EOS token 傳入生成函數(shù)生產(chǎn)級(jí)推理可遷移至 chat/lmdeploy.md 描述的 LMDeploy pipeline / api_server 方案。以上推理流程與參數(shù)均以本倉(cāng)庫(kù) chat/README.md 及其對(duì)應(yīng)源碼chat/web_demo.py、chat/chat_format.md、tests/test_hf_model.py為準(zhǔn)可直接復(fù)制運(yùn)行并在此基礎(chǔ)上二次開發(fā)。贊分享大模型人工智能基礎(chǔ)模型AI Agent【免費(fèi)下載鏈接】InternLMOfficial release of InternLM series (InternLM, InternLM2, InternLM2.5, InternLM3).項(xiàng)目地址https://gitcode.com/gh_mirrors/in/InternLM點(diǎn)擊查看免費(fèi)下載相關(guān)推薦InternLM3-8B-Instruct 對(duì)話推理實(shí)戰(zhàn)Transformers 與 ModelScope 加載、Streamlit 網(wǎng)頁(yè)對(duì)話與推理模式切換InternLM3 8B Instruct 對(duì)話推理實(shí)戰(zhàn)Transformers 與 ModelScope 加載、Streamlit 網(wǎng)頁(yè)對(duì)話與推理模式切換大模型人工智能基礎(chǔ)模型AI AgentInternLM-Chat-7B 對(duì)話 Web Demo 部署指南基于 Streamlit 的本地 Web 對(duì)話全流程實(shí)戰(zhàn)InternLM Chat 7B 對(duì)話 Web Demo 部署指南基于 Streamlit 的本地 Web 對(duì)話全流程實(shí)戰(zhàn) 本篇指南以 Datawhale「開大模型人工智能教程本地部署微調(diào)Datawhale Self-LLM 實(shí)戰(zhàn)基于 transformers 與 peft 的 InternLM3-8B-Instruct LoRA 微調(diào)教程Datawhale Self LLM 實(shí)戰(zhàn)基于 transformers 與 peft 的 InternLM3 8B Instruct LoRA 微調(diào)教程 本大模型人工智能教程本地部署微調(diào)上一篇PowerSploit 域滲透?jìng)刹焓褂?Get-DomainGPOUserLocalGroupMapping 通過 GPO 關(guān)聯(lián)枚舉用戶本地組成員關(guān)系下一篇IQInvision 攝像頭 Telnet 默認(rèn)憑據(jù)檢測(cè)routersploit 字典攻擊模塊實(shí)戰(zhàn)指南創(chuàng)作聲明：本文部分內(nèi)容由AI輔助生成（AIGC），僅供參考