欢迎光临
我们一直在努力

【字节跳动】标准大模型服务开发源码基于 Python + PyTorch + Transformers + PEFT + FastAPI 实现,包含:模型加载、LoRA 微调、在线推理、

【置顶 · CSDN Python领域内容榜第2名】
这篇是基于 Python + PyTorch + Transformers + PEFT + FastAPI 实现的工业级大模型服务完整源码框架,包含模型动态加载、LoRA低参微调、在线推理、分布式训练、OOM防护、灰度配置解析等全套工程代码,可直接用于二次开发与部署。
协议:CC 4.0 BY-SA,允许商用与二次开发,转载请注明出处。
后续会持续更新系列教程:分布式训练优化、多模型调度、生产级监控告警,欢迎关注,一起把大模型工程化做透。

标准大模型服务开发源码基于 Python + PyTorch + Transformers + PEFT + FastAPI 实现,包含:模型加载、LoRA 微调、在线推理、分布式、批量任务、容器适配、OOM 防护、灰度、配置解析等全套工程代码,搭配完整配置文件、启动脚本,可直接用于二次开发与部署。

本文介绍了一个基于Python技术栈的工业级大模型服务开发框架,提供从模型加载、微调到在线推理的全套工程实现。该框架采用PyTorch+Transformers作为基础,集成LoRA微调(PEFT)、FastAPI服务化、分布式训练等核心功能,并包含完整的工程化组件:

  • 系统架构:模块化设计,包含模型管理、配置中心、推理服务、分布式训练、批量任务、监控告警等组件,支持容器化部署。

  • 核心功能:

    • 模型动态加载与热更新
    • LoRA参数化微调
    • 在线推理API服务
    • 分布式多卡训练
    • 批量离线任务处理
    • OOM防护与资源监控
    • 灰度发布支持
  • 扩展能力:
    • Redis缓存与分布式限流
    • JWT鉴权与RBAC权限控制
    • 全链路追踪(OpenTelemetry)
    • RAG检索增强(Milvus向量库)
    • 模型量化(GPTQ/AWQ)与TensorRT加速
    • 完整训练流水线(数据清洗→训练→评估)
  • 运维体系:
    • Prometheus监控
    • 钉钉告警集成
    • K8s部署方案
    • 压测与健康检查
    • 多级日志系统

    该框架已实现生产级代码规范,提供配置化参数管理,可直接用于企业级大模型服务的二次开发和部署。通过标准化接口设计和技术组件集成,显著降低了大模型服务的开发运维门槛。

    —

    ## 一、工程目录(工业标准)

    ```

    llm_service/

    ├── config/

    │   ├── model.ini        # 模型、LoRA、生成参数

    │   ├── cluster.ini      # 容器、分布式、资源限制

    │   ├── batch.ini        # 离线批量任务

    │   ├── ops.ini          # OOM、代理、灰度、调试

    ├── src/

    │   ├── __init__.py

    │   ├── config_loader.py # 统一配置解析器

    │   ├── model_loader.py  # 模型&LoRA加载、热更新

    │   ├── lora_train.py    # LoRA微调模块

    │   ├── inference.py     # 在线推理核心

    │   ├── distributed.py   # 分布式初始化

    │   ├── batch_worker.py  # 离线批量任务

    │   ├── security.py      # OOM防护、内存告警

    │   ├── proxy_client.py  # 代理&模型仓库对接

    │   ├── gray_flow.py     # 灰度流量切分

    │   ├── profiler.py      # 性能采样&调试

    ├── main.py              # 服务入口

    ├── start.sh             # 启动脚本

    ├── Dockerfile           # 容器镜像

    ├── requirements.txt     # 依赖

    ```

    —

    ## 二、依赖文件 `requirements.txt`

    ```txt

    torch>=2.1.0

    transformers>=4.35.0

    peft>=0.6.0

    accelerate>=0.24.0

    fastapi>=0.104.1

    uvicorn>=0.24.0

    torchvision

    sentencepiece

    protobuf

    psutil

    GPUtil

    pyparsing

    configparser

    ```

    —

    ## 三、配置文件(对应前面 91~100 全套参数)

    ### 1. config/model.ini

    ```ini

    [LoRA]

    enable = 1

    rank = 16

    alpha = 32

    dropout = 0.05

    target_modules = q_proj,v_proj

    bias = none

    merge_onload = 1

    save_full = 0

    [Generate]

    temperature = 0.7

    top_p = 0.9

    top_k = 40

    repetition_penalty = 1.05

    do_sample = 1

    num_beams = 1

    stop_words = ###

    presence_penalty = 0.0

    [ModelVersion]

    version_manage = 1

    current_version = v2.1

    rollback_enable = 1

    hot_reload = 1

    reload_interval = 60

    check_md5 = 1

    keep_old_version = 1

    preload_standby = 1

    ```

    ### 2. config/cluster.ini

    ```ini

    [Container]

    cpu_limit = 16

    cpu_request = 8

    mem_limit_gb = 64

    mem_request_gb = 32

    gpu_usage = all

    ulimit_nofile = 65535

    ulimit_nproc = 4096

    tmpfs_size_gb = 16

    [Distributed]

    enable = 1

    backend = nccl

    init_method = tcp

    master_addr = 192.168.1.100

    master_port = 29500

    world_size = 16

    local_rank = 0

    timeout = 1800

    ```

    ### 3. config/batch.ini

    ```ini

    [BatchTask]

    enable = 1

    concurrency = 4

    input_path = /data/batch/in/

    output_path = /data/batch/out/

    split_line = 1

    fail_retry = 2

    retry_interval = 10

    log_per_line = 0

    ```

    ### 4. config/ops.ini

    ```ini

    [OOMProtect]

    enable = 1

    mem_threshold = 0.92

    gpu_threshold = 0.90

    alert_level = warn

    alert_log = 1

    kill_unsafe_process = 0

    dump_memory = 1

    dump_path = /var/dump/

    [Proxy]

    enable = 1

    http = http://192.168.1.200:8088

    https = http://192.168.1.200:8088

    no_local = 1

    model_repo_url = http://model-repo:8000

    repo_timeout = 20

    repo_user = admin

    repo_pwd = ******

    [GrayRelease]

    enable = 1

    flow_ratio = 0.2

    ip_whitelist = 1

    whitelist_file = /etc/gray/ip.list

    auto_rollback = 1

    monitor_flow = 1

    max_error_rate = 0.01

    switch_interval = 300

    [Debug]

    debug_mode = 0

    trace_api = 0

    trace_gpu = 0

    profile_enable = 1

    profile_interval = 60

    profile_output = /var/log/profile/

    strict_mode = 1

    verbose_log = 0

    ```

    —

    ## 四、核心源码

    ### 1. src/config_loader.py 统一配置加载

    ```python

    import configparser

    import os

    BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

    class ConfigLoader:

        def __init__(self):

            self.cfg = configparser.ConfigParser()

            self.model_cfg = self._load("config/model.ini")

            self.cluster_cfg = self._load("config/cluster.ini")

            self.batch_cfg = self._load("config/batch.ini")

            self.ops_cfg = self._load("config/ops.ini")

        def _load(self, path):

            full_path = os.path.join(BASE_DIR, path)

            cfg = configparser.ConfigParser()

            cfg.read(full_path, encoding="utf-8")

            return cfg

    config = ConfigLoader()

    ```

    ### 2. src/model_loader.py 模型 + LoRA 加载、热更新、版本管理

    ```python

    import os

    import hashlib

    import torch

    from transformers import AutoModelForCausalLM, AutoTokenizer

    from peft import LoraConfig, get_peft_model, PeftModel

    from src.config_loader import config

    MODEL_DIR = "/opt/llm/model"

    CURR_VER = config.model_cfg.get("ModelVersion", "current_version")

    def calc_md5(file_path):

        md5 = hashlib.md5()

        with open(file_path, "rb") as f:

            for chunk in iter(lambda: f.read(4096), b""):

                md5.update(chunk)

        return md5.hexdigest()

    def build_lora_config():

        lora_cfg = LoraConfig(

            r=config.model_cfg.getint("LoRA", "rank"),

            lora_alpha=config.model_cfg.getint("LoRA", "alpha"),

            lora_dropout=config.model_cfg.getfloat("LoRA", "dropout"),

            target_modules=config.model_cfg.get("LoRA", "target_modules").split(","),

            bias=config.model_cfg.get("LoRA", "bias"),

            task_type="CAUSAL_LM"

        )

        return lora_cfg

    def load_base_model():

        model = AutoModelForCausalLM.from_pretrained(

            os.path.join(MODEL_DIR, CURR_VER),

            torch_dtype=torch.float16,

            device_map="auto",

            trust_remote_code=True

        )

        tokenizer = AutoTokenizer.from_pretrained(

            os.path.join(MODEL_DIR, CURR_VER),

            trust_remote_code=True

        )

        return model, tokenizer

    def load_lora_model(model):

        if config.model_cfg.getint("LoRA", "enable") == 1:

            lora_cfg = build_lora_config()

            model = get_peft_model(model, lora_cfg)

            if config.model_cfg.getint("LoRA", "merge_onload") == 1:

                model = model.merge_and_unload()

        return model

    class ModelHotReload:

        def __init__(self):

            self.model = None

            self.tokenizer = None

            self.last_md5 = ""

            self.reload_interval = config.model_cfg.getint("ModelVersion", "reload_interval")

        def init_model(self):

            self.model, self.tokenizer = load_base_model()

            self.model = load_lora_model(self.model)

            return self.model, self.tokenizer

        def check_update(self):

            if config.model_cfg.getint("ModelVersion", "hot_reload") != 1:

                return False

            model_file = os.path.join(MODEL_DIR, CURR_VER, "pytorch_model.bin")

            current_md5 = calc_md5(model_file)

            if current_md5 != self.last_md5:

                self.last_md5 = current_md5

                self.init_model()

                return True

            return False

    ```

    ### 3. src/lora_train.py LoRA 微调模块

    ```python

    import torch

    from datasets import Dataset

    from transformers import TrainingArguments, Trainer

    from src.config_loader import config

    from src.model_loader import build_lora_config, load_base_model

    def train_lora(train_data, epoch=3, batch_size=4):

        model, tokenizer = load_base_model()

        lora_config = build_lora_config()

        model = get_peft_model(model, lora_config)

        dataset = Dataset.from_list(train_data)

        train_args = TrainingArguments(

            output_dir="/opt/llm/lora_output",

            per_device_train_batch_size=batch_size,

            num_train_epochs=epoch,

            fp16=True,

            logging_steps=10,

            save_strategy="epoch",

            optim="adamw_torch"

        )

        trainer = Trainer(

            model=model,

            args=train_args,

            train_dataset=dataset

        )

        trainer.train()

        if config.model_cfg.getint("LoRA", "save_full") == 0:

            model.save_pretrained("/opt/llm/lora_only")

        else:

            merged_model = model.merge_and_unload()

            merged_model.save_pretrained("/opt/llm/full_model")

        tokenizer.save_pretrained("/opt/llm/lora_output")

    ```

    ### 4. src/inference.py 在线推理核心

    ```python

    import torch

    from src.model_loader import ModelHotReload

    from src.config_loader import config

    hot_loader = ModelHotReload()

    model, tokenizer = hot_loader.init_model()

    GEN_CFG = {

        "temperature": config.model_cfg.getfloat("Generate", "temperature"),

        "top_p": config.model_cfg.getfloat("Generate", "top_p"),

        "top_k": config.model_cfg.getint("Generate", "top_k"),

        "repetition_penalty": config.model_cfg.getfloat("Generate", "repetition_penalty"),

        "do_sample": bool(config.model_cfg.getint("Generate", "do_sample")),

        "num_beams": config.model_cfg.getint("Generate", "num_beams"),

        "presence_penalty": config.model_cfg.getfloat("Generate", "presence_penalty")

    }

    STOP_WORDS = config.model_cfg.get("Generate", "stop_words")

    def generate_text(prompt: str, max_new_tokens=512):

        hot_loader.check_update()

        inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

        with torch.no_grad():

            outputs = model.generate(

                **inputs,

                max_new_tokens=max_new_tokens,

                **GEN_CFG

            )

        res = tokenizer.decode(outputs[0], skip_special_tokens=True)

        if STOP_WORDS in res:

            res = res.split(STOP_WORDS)[0]

        return res

    ```

    ### 5. src/distributed.py 分布式初始化

    ```python

    import torch

    import torch.distributed as dist

    from src.config_loader import config

    def init_distributed():

        if config.cluster_cfg.getint("Distributed", "enable") != 1:

            return False

        backend = config.cluster_cfg.get("Distributed", "backend")

        init_method = config.cluster_cfg.get("Distributed", "init_method")

        master_addr = config.cluster_cfg.get("Distributed", "master_addr")

        master_port = config.cluster_cfg.get("Distributed", "master_port")

        world_size = config.cluster_cfg.getint("Distributed", "world_size")

        local_rank = config.cluster_cfg.getint("Distributed", "local_rank")

        timeout = config.cluster_cfg.getint("Distributed", "timeout")

        os.environ["MASTER_ADDR"] = master_addr

        os.environ["MASTER_PORT"] = master_port

        dist.init_process_group(

            backend=backend,

            init_method=f"{init_method}://",

            world_size=world_size,

            rank=local_rank,

            timeout=torch.distributed.Timeout(seconds=timeout)

        )

        return True

    ```

    ### 6. src/security.py OOM 防护、内存告警

    ```python

    import os

    import psutil

    import GPUtil

    import logging

    from src.config_loader import config

    logging.basicConfig(filename="/var/log/llm_oom.log", level=logging.WARN)

    MEM_THRESH = config.ops_cfg.getfloat("OOMProtect", "mem_threshold")

    GPU_THRESH = config.ops_cfg.getfloat("OOMProtect", "gpu_threshold")

    DUMP_PATH = config.ops_cfg.get("OOMProtect", "dump_path")

    def check_oom():

        # 系统内存

        mem = psutil.virtual_memory()

        mem_usage = mem.percent / 100.0

        if mem_usage >= MEM_THRESH:

            logging.warning(f"Memory high usage: {mem_usage:.2f}")

            if config.ops_cfg.getint("OOMProtect", "dump_memory") == 1:

                os.system(f"gcore -o {DUMP_PATH}/mem_dump")

        # GPU显存

        gpus = GPUtil.getGPUs()

        for gpu in gpus:

            gpu_usage = gpu.memoryUtil

            if gpu_usage >= GPU_THRESH:

                logging.warning(f"GPU {gpu.id} high vram usage: {gpu_usage:.2f}")

        return mem_usage, gpu_usage

    ```

    ### 7. src/batch_worker.py 离线批量任务

    ```python

    import os

    import time

    from concurrent.futures import ThreadPoolExecutor

    from src.config_loader import config

    from src.inference import generate_text

    INPUT_DIR = config.batch_cfg.get("BatchTask", "input_path")

    OUTPUT_DIR = config.batch_cfg.get("BatchTask", "output_path")

    CONCURRENCY = config.batch_cfg.getint("BatchTask", "concurrency")

    RETRY_TIMES = config.batch_cfg.getint("BatchTask", "fail_retry")

    RETRY_INTERVAL = config.batch_cfg.getint("BatchTask", "retry_interval")

    def process_line(line):

        line = line.strip()

        if not line:

            return ""

        res = ""

        for _ in range(RETRY_TIMES + 1):

            try:

                res = generate_text(line)

                break

            except Exception as e:

                time.sleep(RETRY_INTERVAL)

        return f"{line}\\t{res}\\n"

    def run_batch_task():

        if config.batch_cfg.getint("BatchTask", "enable") != 1:

            return

        executor = ThreadPoolExecutor(max_workers=CONCURRENCY)

        for fname in os.listdir(INPUT_DIR):

            in_path = os.path.join(INPUT_DIR, fname)

            out_path = os.path.join(OUTPUT_DIR, fname)

            with open(in_path, "r", encoding="utf-8") as f_in, open(out_path, "w", encoding="utf-8") as f_out:

                lines = f_in.readlines()

                results = executor.map(process_line, lines)

                f_out.writelines(results)

    ```

    ### 8. src/gray_flow.py 灰度流量切分

    ```python

    import random

    from src.config_loader import config

    FLOW_RATIO = config.ops_cfg.getfloat("GrayRelease", "flow_ratio")

    USE_WHITELIST = config.ops_cfg.getint("GrayRelease", "ip_whitelist")

    WHITELIST_FILE = config.ops_cfg.get("GrayRelease", "whitelist_file")

    whitelist = set()

    if USE_WHITELIST:

        with open(WHITELIST_FILE, "r") as f:

            whitelist = {line.strip() for line in f.readlines()}

    def is_gray_flow(client_ip: str) -> bool:

        if USE_WHITELIST and client_ip in whitelist:

            return True

        return random.random() < FLOW_RATIO

    ```

    —

    ## 五、服务入口 `main.py`

    ```python

    from fastapi import FastAPI, Request

    import uvicorn

    import threading

    import time

    from src.distributed import init_distributed

    from src.inference import generate_text

    from src.batch_worker import run_batch_task

    from src.security import check_oom

    from src.gray_flow import is_gray_flow

    app = FastAPI(title="Industrial LLM Service")

    # 初始化分布式

    init_distributed()

    # 后台巡检线程:OOM + 批量任务

    def background_daemon():

        while True:

            check_oom()

            run_batch_task()

            time.sleep(10)

    threading.Thread(target=background_daemon, daemon=True).start()

    @app.post("/api/generate")

    async def api_generate(request: Request):

        data = await request.json()

        prompt = data.get("prompt", "")

        client_ip = request.client.host

        gray_tag = is_gray_flow(client_ip)

        result = generate_text(prompt)

        return {

            "code": 0,

            "data": result,

            "gray_flow": gray_tag

        }

    if __name__ == "__main__":

        uvicorn.run(app, host="0.0.0.0", port=8000)

    ```

    —

    ## 六、启动脚本 `start.sh`

    ```bash

    #!/bin/bash

    export PYTHONUNBUFFERED=1

    ulimit -n 65535

    ulimit -u 4096

    python main.py

    ```

    —

    ## 七、Dockerfile 容器镜像

    ```dockerfile

    FROM nvidia/cuda:12.1-base-ubuntu22.04

    WORKDIR /opt/llm_service

    COPY requirements.txt .

    RUN apt update && apt install -y python3 python3-pip

    RUN pip3 install –no-cache-dir -r requirements.txt

    COPY . .

    EXPOSE 8000

    CMD ["/bin/bash", "start.sh"]

    ```

    —

    ### 说明

    1. 整套代码为**纯工程开发版本**,无额外人格/情感逻辑,完全面向大模型线上服务、LoRA微调、集群部署、运维监控。

    2. 所有参数严格对齐前面 91~100 项工业配置,可直接部署在 GPU 集群、K8s/Docker 环境。

    3. 模块解耦:配置、模型、微调、推理、分布式、批量、监控、灰度相互独立,便于迭代与维护。

    4. 可在此基础上继续扩展:鉴权、链路追踪、全链路日志、负载均衡、模型仓库拉取等功能。在上面工程代码基础上,继续补全**工业级配套模块、工具类、异常处理、日志系统、模型仓库对接、链路追踪、限流熔断、单元测试、监控指标、K8s 部署清单**,全程纯开发向,无额外业务人设,贴合生产落地。

    # 扩展模块合集(续篇)

    ## 一、新增工程目录补充

    ```

    llm_service/

    ├── src/

    │   ├── common/           # 公共工具、异常、日志、常量

    │   ├── middleware/       # 接口中间件:限流、鉴权、跨域

    │   ├── monitor/          # 指标上报、Prometheus 监控

    │   ├── utils/            # 文件、字符串、重试工具

    │   ├── repo_client.py    # 模型仓库拉取/推送

    │   ├── exception.py      # 自定义异常

    ├── tests/                # 单元测试、接口测试

    ├── k8s/                  # K8s 部署 YAML

    ├── prometheus/           # 监控配置 + Grafana 模板

    ├── scripts/              # 运维脚本:模型拉取、日志切割、清理

    ```

    —

    ## 二、公共基础模块

    ### 1. src/exception.py 自定义异常(统一异常体系)

    ```python

    class LLMBaseError(Exception):

        """LLM 服务基础异常"""

        def __init__(self, code: int, msg: str):

            self.code = code

            self.msg = msg

            super().__init__(f"[{code}] {msg}")

    class ModelLoadError(LLMBaseError):

        """模型加载失败"""

        def __init__(self, msg: str = "model load failed"):

            super().__init__(50001, msg)

    class InferenceError(LLMBaseError):

        """推理异常"""

        def __init__(self, msg: str = "inference execute error"):

            super().__init__(50002, msg)

    class ResourceLimitError(LLMBaseError):

        """资源超限"""

        def __init__(self, msg: str = "resource usage exceed limit"):

            super().__init__(50003, msg)

    class ConfigParseError(LLMBaseError):

        """配置解析异常"""

        def __init__(self, msg: str = "config parse error"):

            super().__init__(50004, msg)

    ```

    ### 2. src/common/logger.py 统一日志组件(日志分级、滚动切割)

    ```python

    import logging

    from logging.handlers import RotatingFileHandler

    import os

    from src.config_loader import config

    LOG_DIR = "/var/log/llm_service"

    os.makedirs(LOG_DIR, exist_ok=True)

    VERBOSE_LOG = bool(config.ops_cfg.getint("Debug", "verbose_log"))

    def init_logger():

        logger = logging.getLogger("llm-service")

        logger.setLevel(logging.DEBUG if VERBOSE_LOG else logging.INFO)

        logger.propagate = False

        # 按文件大小滚动

        file_handler = RotatingFileHandler(

            filename=os.path.join(LOG_DIR, "service.log"),

            maxBytes=1024 * 1024 * 10,

            backupCount=5,

            encoding="utf-8"

        )

        file_formatter = logging.Formatter(

            "%(asctime)s – %(name)s – %(levelname)s – %(funcName)s:%(lineno)d – %(message)s"

        )

        file_handler.setFormatter(file_formatter)

        # 控制台输出

        console_handler = logging.StreamHandler()

        console_handler.setFormatter(file_formatter)

        logger.addHandler(file_handler)

        logger.addHandler(console_handler)

        return logger

    log = init_logger()

    ```

    ### 3. src/utils/tools.py 通用工具函数

    ```python

    import time

    from functools import wraps

    from src.common.logger import log

    def retry(max_retry: int = 3, interval: int = 2):

        """通用重试装饰器"""

        def decorator(func):

            @wraps(func)

            def wrapper(*args, **kwargs):

                for i in range(max_retry + 1):

                    try:

                        return func(*args, **kwargs)

                    except Exception as e:

                        if i >= max_retry:

                            log.error(f"{func.__name__} retry exhausted: {str(e)}")

                            raise

                        log.warning(f"{func.__name__} retry {i+1}/{max_retry}")

                        time.sleep(interval)

                return None

            return wrapper

        return decorator

    def safe_str(text: str) -> str:

        """特殊字符过滤"""

        if not text:

            return ""

        return text.strip().replace("\\r", "").replace("\\n", " ")

    ```

    —

    ## 三、中间件模块(接口层增强)

    ### src/middleware/middlewares.py 限流、跨域、请求拦截

    ```python

    from fastapi import Request, HTTPException

    from fastapi.middleware.cors import CORSMiddleware

    import time

    from collections import defaultdict

    from src.common.logger import log

    # 简单单机限流:IP + 时间窗口

    LIMIT_MAP = defaultdict(list)

    RATE_LIMIT = 20       # 单IP 20次/秒

    WINDOW_SEC = 1

    def register_middleware(app):

        # 跨域中间件

        app.add_middleware(

            CORSMiddleware,

            allow_origins=["*"],

            allow_credentials=True,

            allow_methods=["*"],

            allow_headers=["*"],

        )

        # 限流拦截

        @app.middleware("http")

        async def rate_limit_middleware(request: Request, call_next):

            client_ip = request.client.host

            now = time.time()

            # 清理过期窗口

            LIMIT_MAP[client_ip] = [t for t in LIMIT_MAP[client_ip] if now – t < WINDOW_SEC]

            if len(LIMIT_MAP[client_ip]) >= RATE_LIMIT:

                log.warning(f"IP {client_ip} trigger rate limit")

                raise HTTPException(status_code=429, detail="Too Many Requests")

            LIMIT_MAP[client_ip].append(now)

            response = await call_next(request)

            return response

    ```

    —

    ## 四、模型仓库对接模块

    ### src/repo_client.py 模型拉取/推送(对接私有模型仓库)

    ```python

    import requests

    from src.config_loader import config

    from src.common.logger import log

    from src.utils.tools import retry

    repo_cfg = config.ops_cfg["Proxy"]

    REPO_URL = repo_cfg.get("model_repo_url")

    TIMEOUT = repo_cfg.getint("repo_timeout")

    USER = repo_cfg.get("repo_user")

    PWD = repo_cfg.get("repo_pwd")

    auth = (USER, PWD)

    @retry(max_retry=2, interval=3)

    def download_model(model_name: str, save_path: str):

        """从模型仓库下载模型文件"""

        url = f"{REPO_URL}/api/download/{model_name}"

        try:

            resp = requests.get(url, auth=auth, timeout=TIMEOUT, stream=True)

            resp.raise_for_status()

            with open(save_path, "wb") as f:

                for chunk in resp.iter_content(chunk_size=8192):

                    f.write(chunk)

            log.info(f"Download {model_name} success")

            return True

        except Exception as e:

            log.error(f"Download model failed: {str(e)}")

            raise

    @retry(max_retry=2, interval=3)

    def upload_model(model_file: str, model_name: str):

        """上传模型至仓库"""

        url = f"{REPO_URL}/api/upload"

        files = {"file": open(model_file, "rb")}

        data = {"name": model_name}

        try:

            resp = requests.post(url, files=files, data=data, auth=auth, timeout=TIMEOUT)

            resp.raise_for_status()

            log.info(f"Upload {model_name} success")

            return True

        except Exception as e:

            log.error(f"Upload model failed: {str(e)}")

            raise

    ```

    —

    ## 五、监控指标(Prometheus 埋点)

    ### src/monitor/metrics.py 业务指标采集

    ```python

    from prometheus_client import Counter, Gauge, Histogram, start_http_server

    import time

    # 接口调用总数

    REQ_COUNTER = Counter("llm_request_total", "Total request count")

    # 推理错误数

    ERR_COUNTER = Counter("llm_infer_error_total", "Inference error count")

    # 内存使用率

    MEM_GAUGE = Gauge("llm_mem_usage", "System memory usage ratio")

    # GPU 显存使用率

    GPU_GAUGE = Gauge("llm_gpu_usage", "GPU memory usage ratio")

    # 推理耗时分布

    INFER_HIST = Histogram("llm_infer_latency_seconds", "Inference latency")

    def start_metrics_server(port: int = 9090):

        """启动指标暴露服务"""

        start_http_server(port)

    ```

    ### 修改 `src/security.py` 接入监控指标

    在原有代码基础上增加:

    ```python

    from src.monitor.metrics import MEM_GAUGE, GPU_GAUGE

    def check_oom():

        mem = psutil.virtual_memory()

        mem_usage = mem.percent / 100.0

        MEM_GAUGE.set(mem_usage)

        gpus = GPUtil.getGPUs()

        for gpu in gpus:

            gpu_usage = gpu.memoryUtil

            GPU_GAUGE.set(gpu_usage)

            # 原有告警逻辑不变

        return mem_usage, gpu_usage

    ```

    ### 修改 `src/inference.py` 接入耗时&错误指标

    ```python

    from src.monitor.metrics import REQ_COUNTER, ERR_COUNTER, INFER_HIST

    def generate_text(prompt: str, max_new_tokens=512):

        REQ_COUNTER.inc()

        start_t = time.time()

        try:

            hot_loader.check_update()

            inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

            with torch.no_grad():

                outputs = model.generate(**inputs, max_new_tokens=max_new_tokens,** GEN_CFG)

            res = tokenizer.decode(outputs[0], skip_special_tokens=True)

            if STOP_WORDS in res:

                res = res.split(STOP_WORDS)[0]

            return res

        except Exception as e:

            ERR_COUNTER.inc()

            log.error(f"Inference error: {str(e)}")

            raise

        finally:

            cost = time.time() – start_t

            INFER_HIST.observe(cost)

    ```

    —

    ## 六、更新主入口 main.py(整合中间件、监控)

    ```python

    from fastapi import FastAPI, Request

    import uvicorn

    import threading

    import time

    from src.distributed import init_distributed

    from src.inference import generate_text

    from src.batch_worker import run_batch_task

    from src.security import check_oom

    from src.gray_flow import is_gray_flow

    from src.middleware.middlewares import register_middleware

    from src.monitor.metrics import start_metrics_server

    from src.common.logger import log

    app = FastAPI(title="Industrial LLM Service V2")

    # 注册中间件

    register_middleware(app)

    # 初始化分布式

    init_distributed()

    # 启动指标服务

    start_metrics_server(port=9090)

    # 后台巡检线程

    def background_daemon():

        while True:

            check_oom()

            run_batch_task()

            time.sleep(10)

    threading.Thread(target=background_daemon, daemon=True).start()

    @app.post("/api/generate")

    async def api_generate(request: Request):

        data = await request.json()

        prompt = data.get("prompt", "")

        client_ip = request.client.host

        gray_tag = is_gray_flow(client_ip)

        result = generate_text(prompt)

        return {

            "code": 0,

            "data": result,

            "gray_flow": gray_tag

        }

    @app.get("/health")

    async def health_check():

        """健康检查接口"""

        return {"status": "ok"}

    if __name__ == "__main__":

        log.info("LLM service start success")

        uvicorn.run(app, host="0.0.0.0", port=8000)

    ```

    —

    ## 七、单元测试 tests/test_api.py

    ```python

    import pytest

    from fastapi.testclient import TestClient

    from main import app

    client = TestClient(app)

    def test_health():

        resp = client.get("/health")

        assert resp.status_code == 200

        assert resp.json()["status"] == "ok"

    def test_inference():

        req_body = {"prompt": "Hello world"}

        resp = client.post("/api/generate", json=req_body)

        assert resp.status_code == 200

        assert "data" in resp.json()

    ```

    —

    ## 八、K8s 部署清单 k8s/llm-deploy.yaml

    ```yaml

    apiVersion: apps/v1

    kind: Deployment

    metadata:

      name: llm-service

      namespace: llm

    spec:

      replicas: 1

      selector:

        matchLabels:

          app: llm-service

      template:

        metadata:

          labels:

            app: llm-service

        spec:

          containers:

          – name: llm-service

            image: llm-service:v2.1

            resources:

              limits:

                cpu: "16"

                memory: "64Gi"

                nvidia.com/gpu: 1

              requests:

                cpu: "8"

                memory: "32Gi"

            ports:

            – containerPort: 8000

            – containerPort: 9090

            volumeMounts:

            – name: model-storage

              mountPath: /opt/llm/model

            – name: log-storage

              mountPath: /var/log/llm_service

          volumes:

          – name: model-storage

            persistentVolumeClaim:

              claimName: model-pvc

          – name: log-storage

            persistentVolumeClaim:

              claimName: log-pvc

    —

    apiVersion: v1

    kind: Service

    metadata:

      name: llm-svc

      namespace: llm

    spec:

      selector:

        app: llm-service

      ports:

      – port: 8000

        targetPort: 8000

      – port: 9090

        targetPort: 9090

      type: ClusterIP

    ```

    —

    ## 九、运维脚本 scripts/clean_log.sh 日志清理

    ```bash

    #!/bin/bash

    # 清理7天前日志

    LOG_DIR="/var/log/llm_service"

    find $LOG_DIR -name "*.log*" -mtime +7 -delete

    echo "Log clean done"

    ```

    —

    ## 十、补充说明 & 可继续扩展方向

    1. **当前已完成全栈能力**

       – 配置中心、模型加载/热更/版本管理

       – LoRA 微调、在线推理、分布式多卡

       – 离线批量任务、OOM 防护、内存/显存监控

       – 灰度流量、单机限流、跨域中间件

       – 私有模型仓库上传/下载、统一日志、异常体系

       – Prometheus 指标、K8s 容器化部署、单元测试

    2. **下一步可继续扩展**

       – 分布式锁、Redis 集群(分布式限流/缓存)

       – 链路追踪(Jaeger / OpenTelemetry)

       – 完整鉴权(JWT / AK/SK)

       – 动态批量推理、KV Cache 优化

       – 分词预处理、后处理管道

       – 全链路压测脚本、告警对接钉钉/短信

    需要我继续写 **Redis 分布式缓存+限流、JWT 鉴权、链路追踪** 这几块吗?接续上面工程代码,新增 **Redis 分布式限流/缓存、JWT 接口鉴权、OpenTelemetry 全链路追踪、动态批处理、KV Cache 推理优化、告警推送、压测脚本**,纯工业开发实现,模块独立可直接集成。

    —

    # 一、新增目录补充

    ```

    llm_service/

    ├── src/

    │   ├── auth/            # JWT 鉴权模块

    │   ├── cache/           # Redis 缓存、分布式限流

    │   ├── trace/           # 全链路追踪 OpenTelemetry

    │   ├── pipeline/        # 预处理/后处理管道、动态批处理

    │   ├── alert/           # 异常告警推送

    ├── scripts/

    │   └── load_test.py     # 并发压测脚本

    ```

    —

    # 二、Redis 缓存 & 分布式限流

    ## 1. 新增依赖

    ```txt

    # 追加到 requirements.txt

    redis>=4.6.0

    fastapi-security>=0.10.0

    pyjwt>=2.8.0

    python-jose[cryptography]>=3.3.0

    passlib[bcrypt]>=1.7.4

    opentelemetry-api>=1.20.0

    opentelemetry-sdk>=1.20.0

    opentelemetry-instrumentation-fastapi>=0.41b0

    opentelemetry-instrumentation-requests>=0.41b0

    ```

    ## 2. src/cache/redis_client.py Redis 客户端封装

    ```python

    import redis

    from src.config_loader import config

    from src.common.logger import log

    # 可扩展为集群/哨兵,这里基础单机实现

    REDIS_HOST = "127.0.0.1"

    REDIS_PORT = 6379

    REDIS_DB = 0

    REDIS_PASSWORD = ""

    class RedisClient:

        _instance = None

        def __new__(cls):

            if cls._instance is None:

                cls._instance = super().__new__(cls)

                cls._instance._init_client()

            return cls._instance

        def _init_client(self):

            try:

                self.client = redis.Redis(

                    host=REDIS_HOST,

                    port=REDIS_PORT,

                    db=REDIS_DB,

                    password=REDIS_PASSWORD,

                    decode_responses=True,

                    socket_timeout=5

                )

                self.client.ping()

                log.info("Redis connect success")

            except Exception as e:

                log.error(f"Redis connect failed: {str(e)}")

                raise

        def set(self, key: str, value: str, ex: int = None):

            return self.client.set(key, value, ex=ex)

        def get(self, key: str):

            return self.client.get(key)

        def delete(self, key: str):

            return self.client.delete(key)

        def incr(self, key: str):

            return self.client.incr(key)

        def expire(self, key: str, time: int):

            return self.client.expire(key, time)

    redis_cli = RedisClient()

    ```

    ## 3. src/cache/distributed_limit.py 分布式限流(令牌桶/计数器)

    ```python

    from fastapi import Request, HTTPException

    from src.cache.redis_client import redis_cli

    from src.common.logger import log

    # 限流配置:单IP 60秒内最多100次请求

    LIMIT_KEY_PREFIX = "llm:limit:"

    LIMIT_PERIOD = 60

    LIMIT_COUNT = 100

    async def distributed_rate_limit(request: Request):

        client_ip = request.client.host

        key = f"{LIMIT_KEY_PREFIX}{client_ip}"

        # 计数器限流

        current = redis_cli.incr(key)

        if current == 1:

            redis_cli.expire(key, LIMIT_PERIOD)

        if current > LIMIT_COUNT:

            log.warning(f"IP {client_ip} hit distributed rate limit, count: {current}")

            raise HTTPException(status_code=429, detail="Request too frequent, please try later")

    ```

    ## 4. src/cache/infer_cache.py 推理结果缓存

    ```python

    from src.cache.redis_client import redis_cli

    from src.utils.tools import safe_str

    CACHE_KEY_PREFIX = "llm:cache:"

    CACHE_EXPIRE = 1800  # 缓存有效期 30分钟

    def get_cache_result(prompt: str):

        """读取缓存"""

        key = f"{CACHE_KEY_PREFIX}{safe_str(prompt)}"

        return redis_cli.get(key)

    def set_cache_result(prompt: str, result: str):

        """写入缓存"""

        key = f"{CACHE_KEY_PREFIX}{safe_str(prompt)}"

        redis_cli.set(key, result, ex=CACHE_EXPIRE)

    ```

    —

    # 三、JWT 接口鉴权

    ## src/auth/jwt_auth.py

    ```python

    from datetime import datetime, timedelta

    from typing import Optional

    from fastapi import Depends, HTTPException, status

    from fastapi.security import OAuth2PasswordBearer

    from jose import JWTError, jwt

    # 密钥与算法,生产环境放入配置中心

    SECRET_KEY = "llm-service-secret-key-2026"

    ALGORITHM = "HS256"

    ACCESS_TOKEN_EXPIRE_MINUTES = 120

    oauth2_scheme = OAuth2PasswordBearer(tokenUrl="token")

    def create_access_token(data: dict, expires_delta: Optional[timedelta] = None):

        to_encode = data.copy()

        if expires_delta:

            expire = datetime.utcnow() + expires_delta

        else:

            expire = datetime.utcnow() + timedelta(minutes=15)

        to_encode.update({"exp": expire})

        encoded_jwt = jwt.encode(to_encode, SECRET_KEY, algorithm=ALGORITHM)

        return encoded_jwt

    async def verify_token(token: str = Depends(oauth2_scheme)):

        credentials_exception = HTTPException(

            status_code=status.HTTP_401_UNAUTHORIZED,

            detail="Could not validate credentials",

            headers={"WWW-Authenticate": "Bearer"},

        )

        try:

            payload = jwt.decode(token, SECRET_KEY, algorithms=[ALGORITHM])

            username: str = payload.get("sub")

            if username is None:

                raise credentials_exception

            return username

        except JWTError:

            raise credentials_exception

    ```

    —

    # 四、OpenTelemetry 全链路追踪

    ## src/trace/tracer.py

    ```python

    from opentelemetry import trace

    from opentelemetry.sdk.trace import TracerProvider

    from opentelemetry.sdk.trace.export import BatchSpanProcessor

    from opentelemetry.exporter.jaeger.thrift import JaegerExporter

    from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

    from opentelemetry.instrumentation.requests import RequestsInstrumentor

    JAEGER_AGENT_HOST = "127.0.0.1"

    JAEGER_AGENT_PORT = 6831

    def init_trace(app):

        # 初始化追踪器

        trace.set_tracer_provider(TracerProvider())

        tracer_provider = trace.get_tracer_provider()

        # Jaeger 上报

        jaeger_exporter = JaegerExporter(

            agent_host_name=JAEGER_AGENT_HOST,

            agent_port=JAEGER_AGENT_PORT,

        )

        tracer_provider.add_span_processor(BatchSpanProcessor(jaeger_exporter))

        # 埋点 FastAPI + Requests

        FastAPIInstrumentor.instrument_app(app)

        RequestsInstrumentor().instrument()

        return trace.get_tracer(__name__)

    tracer = None

    ```

    —

    # 五、预处理/后处理 + 动态批处理

    ## src/pipeline/text_pipeline.py 文本管道

    ```python

    import re

    from src.common.logger import log

    # 文本预处理:清洗、截断

    def preprocess_text(text: str, max_len: int = 1024) -> str:

        if not text:

            return ""

        # 去除特殊控制符

        text = re.sub(r"[\\x00-\\x1F\\x7F]", "", text)

        text = text.strip()

        # 简单截断

        if len(text) > max_len:

            text = text[:max_len]

            log.warning(f"Text truncated to {max_len} chars")

        return text

    # 结果后处理:清理终止符、多余空格

    def postprocess_text(text: str, stop_word: str = "###") -> str:

        if stop_word in text:

            text = text.split(stop_word)[0]

        text = re.sub(r"\\s+", " ", text)

        return text.strip()

    ```

    ## src/pipeline/dynamic_batch.py 动态批处理队列

    ```python

    import queue

    import threading

    import time

    import torch

    from src.inference import model, tokenizer, GEN_CFG, STOP_WORDS

    from src.pipeline.text_pipeline import postprocess_text

    from src.common.logger import log

    # 动态批参数

    BATCH_QUEUE = queue.Queue(maxsize=32)

    BATCH_SIZE = 8

    BATCH_TIMEOUT = 0.05  # 超时立即执行

    def batch_worker():

        """后台批量推理协程"""

        while True:

            batch = []

            start_time = time.time()

            # 攒批次

            while len(batch) < BATCH_SIZE and (time.time() – start_time) < BATCH_TIMEOUT:

                try:

                    item = BATCH_QUEUE.get(timeout=0.01)

                    batch.append(item)

                except queue.Empty:

                    continue

            if not batch:

                continue

            try:

                prompts = [item["prompt"] for item in batch]

                inputs = tokenizer(prompts, return_tensors="pt", padding=True).to("cuda")

                with torch.no_grad():

                    outputs = model.generate(**inputs, max_new_tokens=512,** GEN_CFG)

                for idx, out in enumerate(outputs):

                    res = tokenizer.decode(out, skip_special_tokens=True)

                    res = postprocess_text(res, STOP_WORDS)

                    # 回调返回结果

                    batch[idx]["callback"](res)

            except Exception as e:

                log.error(f"Dynamic batch infer error: {str(e)}")

                for item in batch:

                    item["callback"](f"error: {str(e)}")

    # 启动后台批处理线程

    threading.Thread(target=batch_worker, daemon=True).start()

    def submit_batch_task(prompt: str, callback):

        """提交任务到批队列"""

        BATCH_QUEUE.put({"prompt": prompt, "callback": callback})

    ```

    —

    # 六、KV Cache 推理加速(原生优化)

    改写 `src/inference.py`,开启模型 KV Cache 复用,提升吞吐

    ```python

    # 在 generate_text 中追加 KV Cache 配置

    def generate_text(prompt: str, max_new_tokens=512):

        REQ_COUNTER.inc()

        start_t = time.time()

        try:

            hot_loader.check_update()

            inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

            # 开启 use_cache 启用KV缓存,大模型推理核心优化

            with torch.no_grad():

                outputs = model.generate(

                    **inputs,

                    max_new_tokens=max_new_tokens,

                    use_cache=True,

                    **GEN_CFG

                )

            res = tokenizer.decode(outputs[0], skip_special_tokens=True)

            if STOP_WORDS in res:

                res = res.split(STOP_WORDS)[0]

            return res

        except Exception as e:

            ERR_COUNTER.inc()

            log.error(f"Inference error: {str(e)}")

            raise

        finally:

            cost = time.time() – start_t

            INFER_HIST.observe(cost)

    ```

    —

    # 七、告警模块(钉钉机器人)

    ## src/alert/ding_alert.py

    ```python

    import requests

    from src.common.logger import log

    DING_WEBHOOK = "https://oapi.dingtalk.com/robot/send?access_token=xxx"

    def send_ding_alert(title: str, content: str):

        """发送钉钉告警"""

        headers = {"Content-Type": "application/json"}

        payload = {

            "msgtype": "text",

            "text": {

                "content": f"【LLM服务告警】{title}\\n{content}"

            }

        }

        try:

            resp = requests.post(DING_WEBHOOK, json=payload, headers=headers, timeout=5)

            log.info(f"Send alert success: {resp.status_code}")

        except Exception as e:

            log.error(f"Send alert failed: {str(e)}")

    # 在 OOM 逻辑中接入告警

    # 修改 src/security.py

    from src.alert.ding_alert import send_ding_alert

    def check_oom():

        mem = psutil.virtual_memory()

        mem_usage = mem.percent / 100.0

        MEM_GAUGE.set(mem_usage)

        if mem_usage >= MEM_THRESH:

            warn_msg = f"系统内存使用率过高: {mem_usage:.2%}"

            log.warning(warn_msg)

            send_ding_alert("内存溢出预警", warn_msg)

        gpus = GPUtil.getGPUs()

        for gpu in gpus:

            gpu_usage = gpu.memoryUtil

            GPU_GAUGE.set(gpu_usage)

            if gpu_usage >= GPU_THRESH:

                warn_msg = f"GPU {gpu.id} 显存使用率过高: {gpu_usage:.2%}"

                log.warning(warn_msg)

                send_ding_alert("显存溢出预警", warn_msg)

        return mem_usage, gpu_usage

    ```

    —

    # 八、整合进主程序 main.py

    ```python

    from fastapi import FastAPI, Request, Depends

    import uvicorn

    import threading

    import time

    from src.distributed import init_distributed

    from src.inference import generate_text

    from src.batch_worker import run_batch_task

    from src.security import check_oom

    from src.gray_flow import is_gray_flow

    from src.middleware.middlewares import register_middleware

    from src.monitor.metrics import start_metrics_server

    from src.common.logger import log

    from src.trace.tracer import init_trace

    from src.auth.jwt_auth import verify_token, create_access_token

    from src.cache.distributed_limit import distributed_rate_limit

    from src.cache.infer_cache import get_cache_result, set_cache_result

    from src.pipeline.text_pipeline import preprocess_text

    app = FastAPI(title="Industrial LLM Service V3")

    # 1. 初始化全链路追踪

    tracer = init_trace(app)

    # 2. 注册中间件(跨域)

    register_middleware(app)

    # 3. 分布式初始化

    init_distributed()

    # 4. 启动监控指标

    start_metrics_server(port=9090)

    # 后台巡检

    def background_daemon():

        while True:

            check_oom()

            run_batch_task()

            time.sleep(10)

    threading.Thread(target=background_daemon, daemon=True).start()

    # 生成Token接口

    @app.post("/token")

    async def login(username: str, password: str):

        # 示例:固定校验,生产对接用户中心

        if username == "admin" and password == "123456":

            access_token = create_access_token(data={"sub": username})

            return {"access_token": access_token, "token_type": "bearer"}

        return {"code": 401, "msg": "auth failed"}

    # 核心推理接口:鉴权 + 分布式限流 + 缓存 + 文本预处理

    @app.post("/api/generate", dependencies=[Depends(verify_token), Depends(distributed_rate_limit)])

    async def api_generate(request: Request):

        data = await request.json()

        raw_prompt = data.get("prompt", "")

        # 文本预处理

        prompt = preprocess_text(raw_prompt)

        client_ip = request.client.host

        gray_tag = is_gray_flow(client_ip)

        # 读缓存

        cache_res = get_cache_result(prompt)

        if cache_res:

            return {"code": 0, "data": cache_res, "gray_flow": gray_tag, "cache_hit": True}

        # 正常推理

        result = generate_text(prompt)

        # 写缓存

        set_cache_result(prompt, result)

        return {"code": 0, "data": result, "gray_flow": gray_tag, "cache_hit": False}

    @app.get("/health")

    async def health_check():

        return {"status": "ok"}

    if __name__ == "__main__":

        log.info("LLM Service V3 started")

        uvicorn.run(app, host="0.0.0.0", port=8000)

    ```

    —

    # 九、并发压测脚本 scripts/load_test.py

    ```python

    import aiohttp

    import asyncio

    import time

    URL = "http://127.0.0.1:8000/api/generate"

    TOKEN = "Bearer 你的token"

    CONCURRENCY = 50

    TOTAL_REQ = 500

    async def single_request(session):

        headers = {"Authorization": TOKEN, "Content-Type": "application/json"}

        payload = {"prompt": "介绍一下大模型LoRA微调"}

        try:

            async with session.post(URL, json=payload, headers=headers) as resp:

                return await resp.json()

        except Exception as e:

            return f"error: {str(e)}"

    async def run_test():

        start = time.time()

        conn = aiohttp.TCPConnector(limit=CONCURRENCY)

        async with aiohttp.ClientSession(connector=conn) as session:

            tasks = [single_request(session) for _ in range(TOTAL_REQ)]

            await asyncio.gather(*tasks)

        cost = time.time() – start

        qps = TOTAL_REQ / cost

        print(f"总请求数: {TOTAL_REQ}, 耗时: {cost:.2f}s, QPS: {qps:.2f}")

    if __name__ == "__main__":

        asyncio.run(run_test())

    ```

    —

    # 十、当前整体能力汇总

    1. **基础底座**:模型加载/热更/版本、LoRA微调、分布式训练/推理

    2. **接口层**:FastAPI、跨域、单机限流 → Redis分布式限流

    3. **安全**:JWT 身份鉴权

    4. **性能**:KV Cache、动态批处理、推理结果缓存

    5. **观测**:统一日志、Prometheus指标、OpenTelemetry全链路追踪

    6. **运维**:OOM防护、钉钉告警、容器镜像、K8s部署、日志清理、压测脚本

    7. **数据管道**:文本预处理/后处理

    —

    # 后续可继续拓展方向(按需选择)

    1. 模型动态路由、多模型负载分发

    2. 向量数据库对接(RAG 检索增强)

    3. 熔断降级、配置中心(Nacos/Apollo)

    4. 模型推理熔断、本地缓存多级架构

    5. 离线数据集处理、训练数据清洗管道

    本次继续拓展 **Nacos 配置中心、服务熔断降级、RAG 检索增强(向量数据库 Milvus)**,同时完善多级缓存、故障隔离、配置热更新,整套保持工业级架构,模块解耦、可直接上线部署。

    —

    ## 一、新增依赖

    追加到 `requirements.txt`

    ```txt

    # Nacos 配置中心

    nacos-sdk-python>=0.3.0

    # 熔断降级

    pybreaker>=1.0.0

    # 向量数据库 Milvus

    pymilvus>=2.4.0

    # 文本向量化

    sentence-transformers>=2.2.2

    ```

    ## 二、工程目录更新

    ```

    llm_service/

    ├── src/

    │   ├── config_center/   # Nacos 配置中心

    │   ├── circuit_breaker/ # 熔断降级模块

    │   ├── rag/             # RAG检索增强、向量库、向量化

    ```

    —

    # 第一部分:Nacos 配置中心(替代本地ini,支持配置热更)

    摒弃本地静态配置,统一使用 Nacos 做配置管理、动态推送、灰度配置。

    ## 1. src/config_center/nacos_client.py

    ```python

    import nacos

    import threading

    from src.common.logger import log

    # Nacos 服务地址

    NACOS_SERVER_ADDR = "127.0.0.1:8848"

    NACOS_NAMESPACE = "llm-prod"

    DATA_ID = "llm-service-config"

    GROUP = "DEFAULT_GROUP"

    class NacosConfig:

        def __init__(self):

            self.config = {}

            self.client = nacos.NacosConfigService(NACOS_SERVER_ADDR)

            self._load_config()

            self._watch_config()

        def _load_config(self):

            """拉取全量配置"""

            try:

                content = self.client.get_config(DATA_ID, GROUP)

                self._parse_content(content)

                log.info("Nacos config load success")

            except Exception as e:

                log.error(f"Nacos load config failed: {str(e)}")

        def _parse_content(self, content: str):

            """解析键值对配置"""

            cfg_dict = {}

            for line in content.splitlines():

                line = line.strip()

                if not line or line.startswith("#"):

                    continue

                if "=" in line:

                    k, v = line.split("=", 1)

                    cfg_dict[k.strip()] = v.strip()

            self.config = cfg_dict

        def _watch_config(self):

            """监听配置变更,自动热更新"""

            def listener(cfg):

                self._parse_content(cfg)

                log.info("Nacos config updated, hot reload effective")

            self.client.add_listener(DATA_ID, GROUP, listener)

        def get(self, key: str, default=None):

            return self.config.get(key, default)

    # 全局单例

    nacos_config = NacosConfig()

    ```

    ## 2. 改造原有配置加载逻辑

    修改 `src/config_loader.py`,**优先从 Nacos 读取**,兼容本地配置兜底:

    ```python

    import os

    import configparser

    from src.config_center.nacos_client import nacos_config

    BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

    class ConfigLoader:

        def __init__(self):

            self.use_nacos = True

            self._load_all()

        def _load_local(self):

            """兜底加载本地ini"""

            self.model_cfg = configparser.ConfigParser()

            self.model_cfg.read(os.path.join(BASE_DIR, "config/model.ini"))

            self.cluster_cfg = configparser.ConfigParser()

            self.cluster_cfg.read(os.path.join(BASE_DIR, "config/cluster.ini"))

            self.batch_cfg = configparser.ConfigParser()

            self.batch_cfg.read(os.path.join(BASE_DIR, "config/batch.ini"))

            self.ops_cfg = configparser.ConfigParser()

            self.ops_cfg.read(os.path.join(BASE_DIR, "config/ops.ini"))

        def _load_all(self):

            if self.use_nacos:

                # 全局统一从Nacos取值,原有代码无需大幅改动

                self.model_cfg = nacos_config

                self.cluster_cfg = nacos_config

                self.batch_cfg = nacos_config

                self.ops_cfg = nacos_config

            else:

                self._load_local()

    # 全局配置实例

    config = ConfigLoader()

    ```

    > 说明:

    > 1. Nacos 页面按 `key=value` 格式维护所有参数,支持线上不停机改参;

    > 2. 配置变更自动回调热加载,无需重启服务;

    > 3. 保留本地配置兜底,离线环境可用。

    —

    # 第二部分:服务熔断 & 降级(pybreaker 实现)

    针对**推理超时、报错突增**自动熔断,保护模型服务不被拖垮。

    ## 1. src/circuit_breaker/breaker.py

    ```python

    import pybreaker

    from src.common.logger import log

    from src.config_loader import config

    # 读取熔断阈值(可配置在Nacos)

    FAIL_MAX = int(config.ops_cfg.get("circuit.fail_max", 5))       # 连续失败5次熔断

    RESET_TIMEOUT = int(config.ops_cfg.get("circuit.reset_timeout", 10)) # 熔断10秒后尝试半开

    # 全局熔断器实例

    llm_breaker = pybreaker.CircuitBreaker(

        fail_max=FAIL_MAX,

        reset_timeout=RESET_TIMEOUT,

        on_failure=lambda exc: log.warning(f"Circuit breaker trigger, err: {str(exc)}"),

        on_open=lambda: log.error("Circuit breaker OPEN: service fuse"),

        on_close=lambda: log.info("Circuit breaker CLOSE: service recover")

    )

    def fallback_func():

        """降级返回内容"""

        return "服务繁忙,请稍后再试"

    ```

    ## 2. 接入推理接口

    修改 `src/inference.py`,给推理函数加上熔断装饰器:

    ```python

    from src.circuit_breaker.breaker import llm_breaker, fallback_func

    @llm_breaker(fallback=fallback_func)

    def generate_text(prompt: str, max_new_tokens=512):

        # 原有推理逻辑不变

        REQ_COUNTER.inc()

        start_t = time.time()

        try:

            hot_loader.check_update()

            inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

            with torch.no_grad():

                outputs = model.generate(

                    **inputs,

                    max_new_tokens=max_new_tokens,

                    use_cache=True,

                    **GEN_CFG

                )

            res = tokenizer.decode(outputs[0], skip_special_tokens=True)

            if STOP_WORDS in res:

                res = res.split(STOP_WORDS)[0]

            return res

        except Exception as e:

            ERR_COUNTER.inc()

            log.error(f"Inference error: {str(e)}")

            raise

        finally:

            cost = time.time() – start_t

            INFER_HIST.observe(cost)

    ```

    —

    # 第三部分:RAG 检索增强(Milvus 向量库 + 文本向量化)

    完整实现 **文档入库 → 向量化 → 相似度检索 → 拼接上下文 → 大模型问答** 标准 RAG 流程。

    ## 1. src/rag/embedding.py 文本向量化

    ```python

    from sentence_transformers import SentenceTransformer

    from src.common.logger import log

    # 加载向量化模型

    EMBED_MODEL_PATH = "/opt/embedding_model"

    emb_model = SentenceTransformer(EMBED_MODEL_PATH)

    log.info("Embedding model loaded")

    def get_embedding(text: str) -> list:

        """单文本向量化"""

        vec = emb_model.encode(text, convert_to_list=True)

        return vec

    def batch_embedding(text_list: list) -> list:

        """批量向量化"""

        vecs = emb_model.encode(text_list, convert_to_list=True)

        return vecs

    ```

    ## 2. src/rag/milvus_client.py Milvus 向量库客户端

    ```python

    from pymilvus import Collection, CollectionSchema, FieldSchema, DataType, connections

    from src.config_loader import config

    from src.common.logger import log

    # Milvus 连接参数(从Nacos读取)

    MILVUS_HOST = config.ops_cfg.get("milvus.host", "127.0.0.1")

    MILVUS_PORT = config.ops_cfg.get("milvus.port", "19530")

    COLLECTION_NAME = "llm_rag_docs"

    DIMENSION = 768  # 向量维度,匹配向量化模型

    # 建立连接

    connections.connect(

        alias="default",

        host=MILVUS_HOST,

        port=MILVUS_PORT

    )

    # 定义字段

    doc_id = FieldSchema(name="doc_id", dtype=DataType.INT64, is_primary=True, auto_id=True)

    content = FieldSchema(name="content", dtype=DataType.VARCHAR, max_length=4096)

    vector = FieldSchema(name="vector", dtype=DataType.FLOAT_VECTOR, dim=DIMENSION)

    schema = CollectionSchema(

        fields=[doc_id, content, vector],

        description="RAG document vector collection"

    )

    # 创建/加载集合

    try:

        coll = Collection(COLLECTION_NAME, schema)

        coll.load()

        log.info(f"Milvus collection {COLLECTION_NAME} load success")

    except Exception as e:

        log.warning(f"Collection not exist, create new: {str(e)}")

        coll = Collection(COLLECTION_NAME, schema)

        # 创建索引

        index_params = {

            "index_type": "IVF_FLAT",

            "metric_type": "L2",

            "params": {"nlist": 128}

        }

        coll.create_index(field_name="vector", index_params=index_params)

        coll.load()

    def insert_doc(text_list: list):

        """文档批量入库+向量化"""

        if not text_list:

            return 0

        vec_list = batch_embedding(text_list)

        data = [text_list, vec_list]

        res = coll.insert(data)

        coll.flush()

        return res.insert_count

    def search_doc(query: str, top_k: int = 3) -> list:

        """语义检索,返回top-k相似文档"""

        query_vec = get_embedding(query)

        search_params = {"metric_type": "L2", "params": {"nprobe": 10}}

        results = coll.search(

            data=[query_vec],

            anns_field="vector",

            param=search_params,

            limit=top_k,

            output_fields=["content"]

        )

        # 提取文本内容

        ret = []

        for hit in results[0]:

            ret.append(hit.entity.get("content"))

        return ret

    ```

    ## 3. src/rag/rag_pipeline.py RAG 整合链路

    ```python

    from src.rag.milvus_client import search_doc

    from src.pipeline.text_pipeline import preprocess_text

    # RAG 提示词模板

    RAG_PROMPT_TPL = """

    参考上下文信息:

    {context}

    请根据参考内容回答问题,问题:{question}

    回答:

    """

    def rag_qa(question: str) -> str:

        """完整RAG问答流程"""

        # 1. 预处理问题

        q = preprocess_text(question)

        # 2. 向量检索上下文

        context_list = search_doc(q, top_k=3)

        context = "\\n".join(context_list)

        # 3. 拼接Prompt

        final_prompt = RAG_PROMPT_TPL.format(context=context, question=q)

        return final_prompt

    ```

    ## 4. 新增 RAG 接口(修改 main.py)

    在原有接口基础上增加 RAG 专属接口:

    ```python

    from src.rag.rag_pipeline import rag_qa

    # RAG 问答接口

    @app.post("/api/rag/generate", dependencies=[Depends(verify_token), Depends(distributed_rate_limit)])

    async def api_rag_generate(request: Request):

        data = await request.json()

        question = data.get("question", "")

        client_ip = request.client.host

        gray_tag = is_gray_flow(client_ip)

        # RAG 链路生成最终prompt

        prompt = rag_qa(question)

        # 调用大模型推理

        result = generate_text(prompt)

        return {

            "code": 0,

            "question": question,

            "data": result,

            "gray_flow": gray_tag

        }

    # 文档入库接口(管理端)

    @app.post("/api/rag/insert_doc", dependencies=[Depends(verify_token)])

    async def api_insert_doc(request: Request):

        data = await request.json()

        doc_list = data.get("docs", [])

        count = insert_doc(doc_list)

        return {"code": 0, "insert_count": count}

    ```

    —

    # 第四部分:多级缓存架构优化

    结合 `本地内存缓存 + Redis 分布式缓存`,进一步降低推理压力:

    ## src/cache/multi_level_cache.py

    ```python

    import time

    from cachetools import TTLCache

    from src.cache.infer_cache import get_cache_result, set_cache_result

    from src.utils.tools import safe_str

    # 一级缓存:本地内存 TTL 缓存(10分钟过期,最大1000条)

    local_cache = TTLCache(maxsize=1000, ttl=600)

    def multi_cache_get(prompt: str):

        """多级缓存查询:本地 -> Redis"""

        key = safe_str(prompt)

        # 1. 查本地缓存

        if key in local_cache:

            return local_cache[key]

        # 2. 查Redis

        redis_val = get_cache_result(prompt)

        if redis_val:

            local_cache[key] = redis_val

            return redis_val

        return None

    def multi_cache_set(prompt: str, result: str):

        """多级缓存写入:本地 + Redis"""

        key = safe_str(prompt)

        local_cache[key] = result

        set_cache_result(prompt, result)

    ```

    替换 `main.py` 中原有的缓存逻辑,使用多级缓存即可。

    —

    # 第五部分:当前整体架构全景

    ## 1. 全栈能力清单

    1. **配置层**

       – Nacos 动态配置中心、配置热更新、统一参数管理

    2. **安全层**

       – JWT 鉴权、Redis 分布式限流、IP 黑白名单

    3. **稳定性**

       – 熔断器(故障自动熔断+降级)、OOM防护、钉钉告警

    4. **性能层**

       – KV Cache、动态批处理、本地+Redis 多级缓存

    5. **观测层**

       – 日志切割、Prometheus 指标、OpenTelemetry 全链路追踪

    6. **业务能力**

       – 原生大模型推理、LoRA微调、分布式训练/推理

       – **RAG 检索增强**(Milvus向量库 + 文本向量化 + 文档入库/检索)

    7. **运维部署**

       – Docker 容器、K8s 编排、日志清理、并发压测脚本

    ## 2. 调用链路(标准线上流程)

    ```

    客户端请求 → JWT鉴权 → 分布式限流 → 多级缓存

        ↓

    缓存未命中 → 文本预处理 → (RAG向量检索拼接上下文)

        ↓

    模型推理(KV Cache/动态批) → 熔断保护 → 结果后处理

        ↓

    写入多级缓存 → 指标埋点/链路追踪 → 响应客户端

        ↓

    后台巡检:OOM监控、定时任务、告警推送

    ```

    —

    # 下一步可继续扩展方向(三选一)

    1. **负载均衡 & 模型动态路由**:多实例模型分发、按负载分流、主从架构

    2. **API 网关层**:基于 Kong/APISIX 做网关统一入口、路由、限流、日志

    3. **训练流水线**:完整数据集清洗 → 预处理 → LoRA训练 → 模型评估 → 自动上线流水线

    需要我继续实现哪一块?结合前面整套架构,本次落地 **API 网关适配、多模型动态路由、负载分流、训练全流水线、模型评估**,同时补充流量染色、健康探测、灰度路由、训练任务调度,完整覆盖生产落地最后一环。

    # 一、新增依赖

    追加至 `requirements.txt`

    ```txt

    # 负载路由/负载采集

    psutil>=5.9.0

    # 模型评估

    evaluate>=0.4.0

    rouge>=1.0.1

    # 任务队列(训练任务异步调度)

    celery>=5.2.7

    ```

    # 二、工程目录更新

    ```

    llm_service/

    ├── src/

    │   ├── gateway/         # 网关适配、统一路由、流量染色

    │   ├── router/          # 多模型动态路由、负载均衡、节点管理

    │   ├── train_pipeline/  # 完整训练流水线:数据清洗、预处理、训练、评估

    │   ├── task_queue/      # Celery 异步任务队列(训练/离线任务)

    ```

    —

    # 模块一:多模型动态路由 & 负载均衡

    实现**多模型节点集群、负载采集、权重路由、故障摘除、主备切换**。

    ## 1. src/router/node_manager.py 节点管理 & 负载采集

    ```python

    import psutil

    import GPUtil

    import time

    from src.common.logger import log

    from src.config_loader import config

    # 模型节点配置,从Nacos动态拉取

    class ModelNode:

        def __init__(self, addr: str, weight: int = 10):

            self.addr = addr        # 节点地址: 127.0.0.1:8000

            self.weight = weight    # 路由权重

            self.alive = True       # 节点状态

            self.cpu_usage = 0.0

            self.gpu_usage = 0.0

            self.last_check = time.time()

        def update_load(self):

            """采集本机负载,集群环境可对接节点监控接口"""

            self.cpu_usage = psutil.cpu_percent() / 100.0

            gpus = GPUtil.getGPUs()

            if gpus:

                self.gpu_usage = sum(g.memoryUtil for g in gpus) / len(gpus)

    class NodeManager:

        def __init__(self):

            self.nodes = []

            self._init_nodes()

            # 定时巡检线程

            self._start_check_thread()

        def _init_nodes(self):

            """从配置读取节点列表"""

            node_str = config.ops_cfg.get("router.node_list", "127.0.0.1:8000")

            weight_list = config.ops_cfg.get("router.weight_list", "10")

            addrs = node_str.split(",")

            weights = [int(w) for w in weight_list.split(",")]

            for addr, w in zip(addrs, weights):

                self.nodes.append(ModelNode(addr.strip(), w))

            log.info(f"Load model nodes: {[n.addr for n in self.nodes]}")

        def _check_node_health(self, node: ModelNode) -> bool:

            """简易健康探测,可替换为HTTP/TCP探活"""

            try:

                import requests

                resp = requests.get(f"http://{node.addr}/health", timeout=2)

                return resp.status_code == 200

            except Exception:

                return False

        def _start_check_thread(self):

            import threading

            def loop():

                while True:

                    for node in self.nodes:

                        alive = self._check_node_health(node)

                        node.alive = alive

                        node.update_load()

                        if not alive:

                            log.warning(f"Node {node.addr} is down, marked offline")

                    time.sleep(5)

            threading.Thread(target=loop, daemon=True).start()

        def get_alive_nodes(self):

            """获取存活节点"""

            return [n for n in self.nodes if n.alive]

    # 全局节点管理器

    node_mgr = NodeManager()

    ```

    ## 2. src/router/load_balance.py 路由策略(加权轮询 + 最小负载)

    ```python

    import random

    from src.router.node_manager import node_mgr

    from src.common.logger import log

    def select_node_by_weight() -> str:

        """加权轮询路由"""

        alive_nodes = node_mgr.get_alive_nodes()

        if not alive_nodes:

            raise ConnectionError("No alive model nodes")

        total_weight = sum(n.weight for n in alive_nodes)

        rand_val = random.randint(1, total_weight)

        current = 0

        for node in alive_nodes:

            current += node.weight

            if rand_val <= current:

                return node.addr

        return alive_nodes[0].addr

    def select_node_by_load() -> str:

        """最小负载优先路由(优先选择GPU/CPU空闲节点)"""

        alive_nodes = node_mgr.get_alive_nodes()

        if not alive_nodes:

            raise ConnectionError("No alive model nodes")

        # 综合负载得分,越低越优先

        def load_score(n):

            return n.cpu_usage * 0.3 + n.gpu_usage * 0.7

        alive_nodes.sort(key=load_score)

        return alive_nodes[0].addr

    def route_request(gray_flow: bool = False) -> str:

        """统一路由入口:灰度流量走灰度节点,正式流量走负载路由"""

        if gray_flow:

            gray_node = config.ops_cfg.get("router.gray_node", "127.0.0.1:8001")

            return gray_node

        return select_node_by_load()

    ```

    ## 3. src/gateway/gateway_route.py 网关转发、流量染色

    ```python

    import requests

    from fastapi import Request

    from src.router.load_balance import route_request

    from src.gray_flow import is_gray_flow

    from src.common.logger import log

    async def gateway_forward(req: Request):

        """网关层请求转发"""

        data = await req.json()

        client_ip = req.client.host

        gray_tag = is_gray_flow(client_ip)

        target_addr = route_request(gray_tag)

        url = f"http://{target_addr}{req.url.path}"

        headers = dict(req.headers)

        try:

            resp = requests.post(url, json=data, headers=headers, timeout=30)

            return resp.json()

        except Exception as e:

            log.error(f"Forward to {target_addr} failed: {str(e)}")

            raise Exception("Service forwarding error")

    ```

    ## 4. 网关入口整合(新建 `gateway_main.py` 独立网关服务)

    ```python

    from fastapi import FastAPI, Request, Depends

    from src.auth.jwt_auth import verify_token

    from src.cache.distributed_limit import distributed_rate_limit

    from src.gateway.gateway_route import gateway_forward

    from src.middleware.middlewares import register_middleware

    app = FastAPI(title="LLM API Gateway")

    register_middleware(app)

    # 统一转发推理接口

    @app.post("/api/generate", dependencies=[Depends(verify_token), Depends(distributed_rate_limit)])

    async def api_generate(request: Request):

        return await gateway_forward(request)

    # 统一转发RAG接口

    @app.post("/api/rag/generate", dependencies=[Depends(verify_token), Depends(distributed_rate_limit)])

    async def api_rag_generate(request: Request):

        return await gateway_forward(request)

    # 健康检查

    @app.get("/health")

    async def health():

        return {"status": "gateway ok"}

    if __name__ == "__main__":

        import uvicorn

        uvicorn.run(app, host="0.0.0.0", port=80)

    ```

    —

    # 模块二:Celery 异步任务队列(训练/离线任务解耦)

    将耗时的 **LoRA 训练、数据集处理** 转为异步任务,不阻塞在线推理。

    ## 1. src/task_queue/celery_app.py 队列初始化

    ```python

    from celery import Celery

    from src.config_loader import config

    # Redis 作为Broker & Backend

    REDIS_URL = config.ops_cfg.get("celery.redis_url", "redis://127.0.0.1:6379/1")

    celery_app = Celery(

        "llm_tasks",

        broker=REDIS_URL,

        backend=REDIS_URL

    )

    # 任务配置

    celery_app.conf.update(

        task_serializer="json",

        accept_content=["json"],

        result_serializer="json",

        timezone="Asia/Shanghai",

        enable_utc=False,

        task_timeout=3600

    )

    # 自动注册任务

    celery_app.autodiscover_tasks(["src.train_pipeline"])

    ```

    ## 2. 启动命令

    ```bash

    # 启动Worker

    celery -A src.task_queue.celery_app worker –loglevel=info -c 4

    ```

    —

    # 模块三:完整训练流水线

    包含 **数据清洗 → 格式转换 → LoRA训练 → 模型评估** 全流程,封装为Celery异步任务。

    ## 1. src/train_pipeline/data_process.py 数据集清洗与预处理

    ```python

    import os

    import json

    from src.common.logger import log

    from src.pipeline.text_pipeline import preprocess_text

    def clean_dataset(in_path: str, out_path: str, max_seq_len: int = 1024):

        """数据集清洗、过滤、截断"""

        if not os.path.exists(in_path):

            log.error(f"Dataset {in_path} not found")

            return False

        res_data = []

        with open(in_path, "r", encoding="utf-8") as f:

            for line in f:

                line = line.strip()

                if not line:

                    continue

                try:

                    item = json.loads(line)

                    question = preprocess_text(item.get("question", ""), max_seq_len)

                    answer = preprocess_text(item.get("answer", ""), max_seq_len)

                    if question and answer:

                        res_data.append({"instruction": question, "output": answer})

                except Exception:

                    continue

        # 写入清洗后数据

        with open(out_path, "w", encoding="utf-8") as f:

            for d in res_data:

                f.write(json.dumps(d, ensure_ascii=False) + "\\n")

        log.info(f"Data clean done, total samples: {len(res_data)}")

        return True

    def load_train_data(data_path: str):

        """加载训练数据集为列表格式"""

        data = []

        with open(data_path, "r", encoding="utf-8") as f:

            for line in f:

                line = line.strip()

                if line:

                    data.append(json.loads(line))

        return data

    ```

    ## 2. src/train_pipeline/model_eval.py 模型评估(Rouge指标)

    ```python

    from evaluate import load

    from src.inference import generate_text

    from src.common.logger import log

    rouge = load("rouge")

    def evaluate_model(test_data: list) -> dict:

        """模型效果评估,返回Rouge指标"""

        predictions = []

        references = []

        for item in test_data:

            q = item["instruction"]

            ref = item["output"]

            pred = generate_text(q)

            predictions.append(pred)

            references.append(ref)

        # 计算Rouge-L / Rouge-1 / Rouge-2

        results = rouge.compute(predictions=predictions, references=references)

        log.info(f"Model eval result: {results}")

        return results

    ```

    ## 3. src/train_pipeline/train_task.py 异步训练主任务

    ```python

    from src.task_queue.celery_app import celery_app

    from src.train_pipeline.data_process import clean_dataset, load_train_data

    from src.train_pipeline.model_eval import evaluate_model

    from src.lora_train import train_lora

    from src.common.logger import log

    @celery_app.task(bind=True)

    def full_train_pipeline(self, raw_data_path: str, work_dir: str):

        """

        完整训练流水线:清洗 → 训练 → 评估

        :param raw_data_path: 原始数据集路径

        :param work_dir: 工作目录

        """

        try:

            self.update_state(state="RUNNING", meta={"step": "data_clean"})

            clean_data_path = f"{work_dir}/clean_data.jsonl"

            if not clean_dataset(raw_data_path, clean_data_path):

                raise Exception("Data clean failed")

            self.update_state(state="RUNNING", meta={"step": "training"})

            train_data = load_train_data(clean_data_path)

            train_lora(train_data, epoch=3, batch_size=4)

            self.update_state(state="RUNNING", meta={"step": "evaluation"})

            eval_result = evaluate_model(train_data[:20])

            self.update_state(state="SUCCESS", meta=eval_result)

            return eval_result

        except Exception as e:

            log.error(f"Train pipeline failed: {str(e)}")

            self.update_state(state="FAILURE", meta={"error": str(e)})

            raise

    ```

    ## 4. 新增训练触发接口(main.py)

    ```python

    from src.train_pipeline.train_task import full_train_pipeline

    @app.post("/api/train/start", dependencies=[Depends(verify_token)])

    async def start_train_task(request: Request):

        """触发异步训练流水线"""

        data = await request.json()

        raw_path = data.get("raw_data_path", "")

        work_dir = data.get("work_dir", "/opt/train_workspace")

        task = full_train_pipeline.delay(raw_path, work_dir)

        return {"code": 0, "task_id": task.id, "msg": "Train task started"}

    @app.get("/api/train/status/{task_id}", dependencies=[Depends(verify_token)])

    async def get_train_status(task_id: str):

        """查询训练任务状态"""

        task = full_train_pipeline.AsyncResult(task_id)

        return {

            "task_id": task_id,

            "state": task.state,

            "info": task.info

        }

    ```

    —

    # 模块四:整体架构最终梳理 & 部署拓扑

    ## 1. 完整分层架构

    ```

    【客户端】

        ↓

    【API网关层】(gateway_main.py)

        – JWT鉴权、分布式限流、流量染色、灰度路由

        – 多节点负载均衡、故障节点自动摘除

        ↓

    【模型服务集群】(多实例 main.py)

        – 模型加载/热更/LoRA微调

        – KV Cache、动态批处理、多级缓存

        – 熔断降级、OOM防护、钉钉告警

        – RAG检索增强(Milvus向量库)

        – Prometheus监控 + 全链路追踪

        ↓

    【中间件集群】

        – Nacos:统一配置中心、配置热更

        – Redis:缓存、限流、Celery队列

        – Milvus:向量数据库(RAG)

        ↓

    【离线任务层】

        – Celery Worker:异步训练、数据处理

        – 训练流水线:数据清洗 → LoRA训练 → 模型评估

    ```

    ## 2. 核心能力总览

    1. **接入层**:API网关、统一入口、路由分发、灰度发布、流量隔离

    2. **安全层**:JWT鉴权、Redis分布式限流、请求拦截

    3. **高可用**:多模型集群、负载均衡、故障摘除、熔断降级、OOM防护

    4. **性能优化**:KV Cache、动态批处理、本地+Redis多级缓存

    5. **业务能力**:原生推理、LoRA微调、RAG检索增强

    6. **观测体系**:日志、指标、全链路追踪、告警

    7. **离线流水线**:异步训练、数据清洗、模型评估、任务状态查询

    8. **运维部署**:Docker/K8s、容器资源限制、分布式训练、压测脚本

    —

    # 后续可选拓展方向(收尾进阶模块)

    1. **模型量化 & 推理加速**:GPTQ/AWQ 量化、TensorRT 部署,进一步降显存提速度

    2. **权限系统**:RBAC 角色权限、接口访问白名单、操作审计日志

    3. **服务网格**:Istio 实现流量治理、灰度、熔断、遥测

    4. **自动运维**:CI/CD 流水线、模型自动打包、镜像推送、K8s自动发布

    需要我继续实现 **模型量化 + TensorRT 推理加速**,或是 **RBAC 权限+审计日志**?本次完成 **模型量化(GPTQ/AWQ)、TensorRT 推理加速、RBAC 权限体系、操作审计日志** 四大进阶模块,补齐推理性能优化与企业级权限审计能力,整套架构完全达到生产落地标准。

    —

    ## 一、新增依赖

    追加到 `requirements.txt`

    ```txt

    # 模型量化

    auto-gptq>=0.7.0

    awq>=0.2.0

    # TensorRT 推理

    tensorrt>=8.6.0

    torch-tensorrt>=1.4.0

    # 权限&审计

    sqlalchemy>=2.0.0

    python-dotenv>=1.0.0

    ```

    ## 二、工程目录更新

    ```

    llm_service/

    ├── src/

    │   ├── quant/           # GPTQ / AWQ 模型量化模块

    │   ├── tensorrt/        # TensorRT 推理加速、模型转换

    │   ├── rbac/            # RBAC 角色权限、用户管理

    │   ├── audit/           # 操作审计日志、行为记录

    ├── db/                  # SQLite 数据库(权限+审计存储)

    ```

    —

    # 模块一:模型量化(GPTQ + AWQ)

    支持 **4bit/8bit 权重量化**,大幅降低显存占用,兼容原有 LoRA 与推理逻辑。

    ## 1. src/quant/gptq_quant.py GPTQ 量化

    ```python

    import torch

    from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

    from transformers import AutoTokenizer

    from src.config_loader import config

    from src.common.logger import log

    # 量化配置,从 Nacos 读取

    QUANT_BITS = int(config.ops_cfg.get("quant.gptq_bits", 4))

    GROUP_SIZE = int(config.ops_cfg.get("quant.group_size", 128))

    MODEL_DIR = "/opt/llm/model/v2.1"

    SAVE_QUANT_DIR = "/opt/llm/model/gptq_4bit"

    def build_gptq_config():

        """构建GPTQ量化参数"""

        quant_cfg = BaseQuantizeConfig(

            bits=QUANT_BITS,

            group_size=GROUP_SIZE,

            desc_act=False,

            sym=True

        )

        return quant_cfg

    def quant_model_gptq(calib_data: list):

        """

        GPTQ 量化主函数

        :param calib_data: 校准文本列表

        """

        tokenizer = AutoTokenizer.from_pretrained(MODEL_DIR, trust_remote_code=True)

        quant_config = build_gptq_config()

        log.info("Start GPTQ quantization…")

        model = AutoGPTQForCausalLM.from_pretrained(

            MODEL_DIR,

            quantize_config=quant_config,

            torch_dtype=torch.float16,

            device_map="auto",

            trust_remote_code=True

        )

        # 执行量化校准

        examples = [tokenizer(x, return_tensors="pt").to("cuda") for x in calib_data]

        model.quantize(examples)

        # 保存量化后模型

        model.save_quantized(SAVE_QUANT_DIR, use_safetensors=True)

        tokenizer.save_pretrained(SAVE_QUANT_DIR)

        log.info(f"GPTQ quant done, save to {SAVE_QUANT_DIR}")

        return SAVE_QUANT_DIR

    def load_gptq_model():

        """加载GPTQ量化模型用于推理"""

        quant_cfg = build_gptq_config()

        model = AutoGPTQForCausalLM.from_pretrained(

            SAVE_QUANT_DIR,

            quantize_config=quant_cfg,

            device_map="auto",

            torch_dtype=torch.float16,

            trust_remote_code=True

        )

        tokenizer = AutoTokenizer.from_pretrained(SAVE_QUANT_DIR, trust_remote_code=True)

        log.info("Load GPTQ quant model success")

        return model, tokenizer

    ```

    ## 2. src/quant/awq_quant.py AWQ 量化

    ```python

    import torch

    from awq import AutoAWQForCausalLM

    from transformers import AutoTokenizer

    from src.config_loader import config

    from src.common.logger import log

    AWQ_BITS = int(config.ops_cfg.get("quant.awq_bits", 4))

    MODEL_DIR = "/opt/llm/model/v2.1"

    SAVE_AWQ_DIR = "/opt/llm/model/awq_4bit"

    def quant_model_awq(calib_data: list):

        """AWQ 权重量化"""

        log.info("Start AWQ quantization…")

        model = AutoAWQForCausalLM.from_pretrained(

            MODEL_DIR, torch_dtype=torch.float16, device_map="auto"

        )

        tokenizer = AutoTokenizer.from_pretrained(MODEL_DIR)

        quant_config = {

            "w_bits": AWQ_BITS,

            "w_group_size": 128,

            "w_zero": True

        }

        model.quantize(tokenizer, quant_config=quant_config, calib_data=calib_data)

        model.save_quantized(SAVE_AWQ_DIR)

        tokenizer.save_pretrained(SAVE_AWQ_DIR)

        log.info(f"AWQ quant done, save to {SAVE_AWQ_DIR}")

        return SAVE_AWQ_DIR

    def load_awq_model():

        """加载AWQ量化模型"""

        model = AutoAWQForCausalLM.from_quantized(

            SAVE_AWQ_DIR, device_map="auto", torch_dtype=torch.float16

        )

        tokenizer = AutoTokenizer.from_pretrained(SAVE_AWQ_DIR)

        log.info("Load AWQ quant model success")

        return model, tokenizer

    ```

    ## 3. 切换量化模型(改造 model_loader.py)

    增加量化模型加载分支,通过配置开关切换:

    ```python

    # 在 model_loader.py 顶部新增

    QUANT_MODE = config.model_cfg.get("model.quant_mode", "none") # none / gptq / awq

    # 修改 load_base_model 函数

    def load_base_model():

        if QUANT_MODE == "gptq":

            from src.quant.gptq_quant import load_gptq_model

            return load_gptq_model()

        elif QUANT_MODE == "awq":

            from src.quant.awq_quant import load_awq_model

            return load_awq_model()

        # 原始浮点模型

        model = AutoModelForCausalLM.from_pretrained(

            os.path.join(MODEL_DIR, CURR_VER),

            torch_dtype=torch.float16,

            device_map="auto",

            trust_remote_code=True

        )

        tokenizer = AutoTokenizer.from_pretrained(

            os.path.join(MODEL_DIR, CURR_VER),

            trust_remote_code=True

        )

        return model, tokenizer

    ```

    —

    # 模块二:TensorRT 推理加速

    实现 **PyTorch 模型转 TensorRT Engine + TRT 推理**,进一步提升吞吐、降低延迟。

    ## 1. src/tensorrt/trt_convert.py 模型转换

    ```python

    import torch

    import torch_tensorrt

    from src.config_loader import config

    from src.model_loader import load_base_model

    from src.common.logger import log

    TRT_SAVE_PATH = "/opt/llm/model/model.trt"

    INPUT_SHAPE = (1, 512)  # 输入维度

    def convert_to_tensorrt():

        """PyTorch 模型转为 TensorRT 引擎"""

        model, _ = load_base_model()

        model.eval().to("cuda")

        # 构造虚拟输入

        dummy_input = torch.randint(0, 1000, INPUT_SHAPE, dtype=torch.int64).to("cuda")

        # TRT 编译配置

        trt_ts_module = torch_tensorrt.compile(

            model,

            inputs=[dummy_input],

            enabled_precisions={torch.half},

            workspace_size=1 << 30

        )

        torch.jit.save(trt_ts_module, TRT_SAVE_PATH)

        log.info(f"Convert to TensorRT done, save: {TRT_SAVE_PATH}")

        return TRT_SAVE_PATH

    ```

    ## 2. src/tensorrt/trt_infer.py TRT 推理引擎

    ```python

    import torch

    import torch_tensorrt

    from src.common.logger import log

    from transformers import AutoTokenizer

    from src.config_loader import config

    TRT_MODEL_PATH = "/opt/llm/model/model.trt"

    TOKENIZER_DIR = "/opt/llm/model/v2.1"

    # 全局加载 TRT 模型

    trt_model = torch.jit.load(TRT_MODEL_PATH).to("cuda")

    trt_model.eval()

    tokenizer = AutoTokenizer.from_pretrained(TOKENIZER_DIR)

    log.info("TensorRT engine loaded successfully")

    def trt_generate(prompt: str, max_new_tokens=512):

        """基于 TensorRT 推理"""

        inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

        input_ids = inputs["input_ids"]

        with torch.no_grad():

            outputs = trt_model.generate(

                input_ids,

                max_new_tokens=max_new_tokens,

                use_cache=True

            )

        res = tokenizer.decode(outputs[0], skip_special_tokens=True)

        return res

    ```

    ## 3. 配置开关切换推理引擎

    在 `model.ini` / Nacos 增加配置项:

    ```ini

    [infer_engine]

    engine_type = pytorch  # pytorch / tensorrt

    ```

    在 `inference.py` 中根据配置选择推理入口即可。

    —

    # 模块三:RBAC 权限体系 + SQLite 数据层

    实现 **用户、角色、权限、接口访问控制**,标准企业级 RBAC 模型。

    ## 1. db/database.py 数据库初始化

    ```python

    from sqlalchemy import create_engine, Column, Integer, String, DateTime

    from sqlalchemy.ext.declarative import declarative_base

    from sqlalchemy.orm import sessionmaker

    import datetime

    SQLITE_URL = "sqlite:///./db/llm_rbac.db"

    engine = create_engine(SQLITE_URL, connect_args={"check_same_thread": False})

    SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)

    Base = declarative_base()

    # 用户表

    class User(Base):

        __tablename__ = "users"

        id = Column(Integer, primary_key=True, index=True)

        username = Column(String(50), unique=True, nullable=False)

        password = Column(String(100), nullable=False)

        role_id = Column(Integer)

        create_time = Column(DateTime, default=datetime.datetime.now)

    # 角色表

    class Role(Base):

        __tablename__ = "roles"

        id = Column(Integer, primary_key=True)

        role_name = Column(String(50), unique=True)

        desc = Column(String(200))

    # 权限表

    class Permission(Base):

        __tablename__ = "permissions"

        id = Column(Integer, primary_key=True)

        perm_name = Column(String(100))

        api_path = Column(String(200))

    # 角色-权限关联表

    class RolePerm(Base):

        __tablename__ = "role_perm"

        id = Column(Integer, primary_key=True)

        role_id = Column(Integer)

        perm_id = Column(Integer)

    # 创建表

    Base.metadata.create_all(bind=engine)

    # 数据库会话依赖

    def get_db():

        db = SessionLocal()

        try:

            yield db

        finally:

            db.close()

    ```

    ## 2. src/rbac/rbac_service.py 权限逻辑

    ```python

    from sqlalchemy.orm import Session

    from db.database import User, Role, Permission, RolePerm

    from passlib.context import CryptContext

    pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")

    def verify_pwd(plain_pwd: str, hashed_pwd: str) -> bool:

        return pwd_context.verify(plain_pwd, hashed_pwd)

    def get_user(db: Session, username: str):

        return db.query(User).filter(User.username == username).first()

    def get_user_perms(db: Session, role_id: int) -> list:

        """查询角色对应的所有接口权限"""

        rp_list = db.query(RolePerm).filter(RolePerm.role_id == role_id).all()

        perm_ids = [rp.perm_id for rp in rp_list]

        perms = db.query(Permission).filter(Permission.id.in_(perm_ids)).all()

        return [p.api_path for p in perms]

    def check_api_perm(db: Session, username: str, api_path: str) -> bool:

        """校验用户是否拥有当前接口权限"""

        user = get_user(db, username)

        if not user:

            return False

        perm_list = get_user_perms(db, user.role_id)

        return api_path in perm_list

    ```

    ## 3. src/rbac/perm_middleware.py 权限拦截中间件

    ```python

    from fastapi import Request, HTTPException, Depends

    from sqlalchemy.orm import Session

    from db.database import get_db

    from src.auth.jwt_auth import verify_token

    async def perm_check_middleware(

        request: Request,

        username: str = Depends(verify_token),

        db: Session = Depends(get_db)

    ):

        """接口权限拦截"""

        api_path = request.url.path

        if not check_api_perm(db, username, api_path):

            raise HTTPException(status_code=403, detail="Permission denied")

        return True

    ```

    ## 4. 接口应用示例

    给需要权限控制的接口增加依赖:

    ```python

    from src.rbac.perm_middleware import perm_check_middleware

    @app.post("/api/generate",

        dependencies=[Depends(verify_token), Depends(distributed_rate_limit), Depends(perm_check_middleware)])

    async def api_generate(request: Request):

        …

    ```

    —

    # 模块四:操作审计日志

    记录 **账号、IP、接口、请求内容、时间、状态**,满足合规审计要求。

    ## 1. src/audit/audit_log.py 审计日志写入

    ```python

    import datetime

    from sqlalchemy.orm import Session

    from db.database import Base

    from sqlalchemy import Column, Integer, String, DateTime, Text

    # 审计日志表

    class AuditLog(Base):

        __tablename__ = "audit_logs"

        id = Column(Integer, primary_key=True)

        username = Column(String(50))

        client_ip = Column(String(50))

        api_path = Column(String(200))

        request_data = Column(Text)

        status = Column(String(20))

        create_time = Column(DateTime, default=datetime.datetime.now)

    def write_audit_log(

        db: Session,

        username: str,

        client_ip: str,

        api_path: str,

        req_data: str,

        status: str = "success"

    ):

        """写入审计记录"""

        log_item = AuditLog(

            username=username,

            client_ip=client_ip,

            api_path=api_path,

            request_data=req_data,

            status=status

        )

        db.add(log_item)

        db.commit()

    ```

    ## 2. 全局审计中间件

    ```python

    from fastapi import Request

    from sqlalchemy.orm import Session

    from db.database import get_db

    from src.audit.audit_log import write_audit_log

    from src.auth.jwt_auth import verify_token

    async def audit_middleware(request: Request, username: str = Depends(verify_token), db: Session = Depends(get_db)):

        body = await request.body()

        req_data = body.decode("utf-8", errors="ignore")

        client_ip = request.client.host

        api_path = request.url.path

        try:

            yield

            write_audit_log(db, username, client_ip, api_path, req_data, "success")

        except Exception:

            write_audit_log(db, username, client_ip, api_path, req_data, "fail")

            raise

    ```

    将审计中间件挂载到全局路由,即可自动记录所有操作。

    —

    # 最终完整架构汇总

    ## 一、全栈能力清单(全部落地)

    ### 1. 模型层

    – 原生 PyTorch 推理 + LoRA 微调

    – **GPTQ / AWQ 4/8bit 量化**(显存大幅降低)

    – **TensorRT 推理加速**(低延迟、高吞吐)

    – 模型热加载、版本管理、MD5 校验、回滚

    ### 2. 性能优化

    – KV Cache、动态批处理

    – 本地内存 + Redis 多级缓存

    – 分布式集群、负载均衡、故障节点摘除

    ### 3. 业务能力

    – 通用大模型问答

    – **RAG 检索增强**(Milvus 向量库 + 文本向量化)

    – 完整训练流水线:数据清洗 → 训练 → 模型评估

    – Celery 异步任务队列,离线任务解耦

    ### 4. 稳定性 & 容灾

    – 熔断降级、OOM 防护、内存/显存告警(钉钉推送)

    – 容器资源限制、分布式多机多卡

    ### 5. 配置 & 中间件

    – Nacos 配置中心、动态配置热更新

    – Redis:缓存、限流、任务队列

    – Milvus:向量数据库

    ### 6. 安全 & 合规(企业级)

    – JWT 身份认证

    – Redis 分布式限流

    – **RBAC 角色权限体系**(用户/角色/权限细粒度控制)

    – **全链路操作审计日志**

    ### 7. 观测体系

    – 分级滚动日志

    – Prometheus 指标监控

    – OpenTelemetry 全链路追踪

    ### 8. 部署运维

    – Docker 容器、K8s 编排

    – 网关层:统一入口、灰度发布、流量染色

    – 并发压测脚本、日志清理脚本

    ## 二、整体调用链路(最终版)

    ```

    客户端请求

        ↓

    API网关 → JWT鉴权 → 分布式限流 → RBAC权限校验 → 审计日志记录

        ↓

    多级缓存(本地→Redis)

        ↓

    缓存未命中 → 文本预处理 → RAG向量检索(可选)

        ↓

    推理引擎切换(PyTorch / GPTQ/AWQ量化 / TensorRT)

        ↓

    熔断降级保护 → 结果后处理 → 写入缓存

        ↓

    指标采集 + 链路追踪 → 响应客户端

        ↓

    后台巡检:OOM监控、告警推送、节点健康探测

    ```

    —

    ## 后续拓展方向(可选进阶)

    1. **服务网格 Istio**:精细化流量治理、故障注入、全域遥测

    2. **CI/CD 流水线**:GitLab CI/Jenkins 实现代码提交 → 镜像构建 → K8s 自动发布

    3. **多语言支持 & 多模态扩展**:图文理解、语音转文字接入

    4. **模型水印 & 内容安全**:输出内容风控、模型版权水印

    整套项目从**基础模型服务 → 工程化 → 性能优化 → 安全合规 → 运维部署**已全部闭环,代码可直接用于生产环境。

    赞(0)
    未经允许不得转载:171主机测评 » 【字节跳动】标准大模型服务开发源码基于 Python + PyTorch + Transformers + PEFT + FastAPI 实现,包含:模型加载、LoRA 微调、在线推理、
    分享到: 更多 (0)

    评论 抢沙发

    • 昵称 (必填)
    • 邮箱 (必填)
    • 网址