Complete Guide to Local LLM Deployment: Building Your Own Agent Workflow for Token Freedom (Newbie-Friendly)

This article aims to be as easy to understand as possible, allowing any novice player to easily master local model deployment and build their own workflow.

This year, domestic open-source large models have been very active. DeepSeek, Qwen, Kimi, GLM, MiniMax... new models are appearing one after another, constantly opening their weights, and starting to go back and forth with closed-source models from the US.

But the more models there are, the more I care about a specific question: can these models not just stay in web pages and APIs, but truly be installed into our own machines, connecting to local files, tools, and workflows?

Although the unit price of API tokens is generally decreasing, the cost remains high when encountering high-frequency calls and long texts. Coupled with issues like privacy, networking, and data control, local deployment is becoming the choice for more and more developers and enterprises.

So this time, I want to choose a model with a challenging size and representative of single-machine deployment to run through the complete local deployment process.

The protagonist finally selected is Ling-3.0-flash, open-sourced by Ant Bailing—a Mixture of Experts (MoE) model with a total parameter count of 124B. I happen to have an NVIDIA DGX Spark on hand, which can just run it.

Without further ado, let's start the main show.

01 Terminology Primer

Before we start, let's introduce some model-related terms to get everyone up to speed ✌🏻

Model Precision

The same model often provides different precision or quantized versions, which directly affect the model size and running threshold.

This time we are using the official Ling INT4 version, which measures about 71.75GB.

Dense or MoE

Note that 5.1B is only the number of parameters activated per inference; the full 124B weights still need to be loaded into memory.

Inference Engine

The inference engine is responsible for loading weights, managing context and concurrency, and providing interfaces. It is the tool for running the model, not the model itself.

We chose vLLM this time because the current official adaptation supports Ling-3.0-flash's INT4 weights and MTP speculative decoding.

02 Model Installation and Deployment

First, let's introduce the installation environment: I am using an NVIDIA DGX Spark, equipped with a GB10 chip and 121.6GB unified memory, running Ubuntu 24.04 on ARM64 architecture. The deployment is Ling-3.0-flash-INT4, and the subsequent throughput tests will use the local Qwen 3.8-27B as a reference.

The official provides a basic version and different precision versions like FP8, FP4, and INT4. You can choose according to your hardware.

There is also an 8B Ling-3.0-tiny and its FP8, INT4 versions. Users with a regular Mac or a single 4090 can prioritize trying the low-precision versions of Tiny.

Step 1: Download the model. I used the official INT4 version this time. Download entries 👇🏻

• Hugging Face: Ling-3.0-flash-int4
• ModelScope: Ling-3.0-flash-int4

If accessing Hugging Face is inconvenient, you can download manually from ModelScope. After downloading, there are 24 safetensors shards, totaling about 71.75GB. You also need to leave space for the inference engine and KV Cache during runtime.

Step 2: Prepare the environment. The BailingMoeV3 architecture of Ling-3.0-flash is quite new. I tried using GGUF with llama.cpp, but it errored with unknown model architecture: 'bailingmoe3'. So this time, I used the officially adapted vLLM branch directly:

1pip install uv
2uv venv ~/my_ling_env
3source ~/my_ling_env/bin/activate
4
5git clone -b ling_3_0 https://github.com/inclusionAI/vllm-ling-v3.git
6cd vllm-ling-v3
7VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto

Step 3: Start the inference service. Replace the model path with your locally downloaded INT4 weights:

1vllm serve /path/to/Ling-3.0-flash-int4 \
2  --served-model-name ling-int4 \
3  --host 127.0.0.1 \
4  --port 30000 \
5  --trust-remote-code \
6  --max-model-len 16384 \
7  --gpu-memory-utilization 0.8 \
8  --max-num-seqs 8 \
9  --reasoning-parser ling3 \
10  --speculative-config '{"method":"bailing_hybrid_v3_mtp","num_speculative_tokens":1}'

⚠️

Please replace the paths in the command with the actual paths on your machine.

Here, the context limit is set to 16K, concurrency to 8, and MTP speculative decoding is enabled.

Note: MTP (Multi-Token Prediction) allows the model to try predicting multiple tokens at once. Correctly predicted parts can be adopted directly, reducing the calculation rounds for generating tokens one by one and increasing output speed.

Step 4: Verify the service. Once started, send a simple request:

1curl -s http://127.0.0.1:30000/v1/chat/completions \
2  -H "Content-Type: application/json" \
3  -d '{"model":"ling-int4",
4       "messages":[{"role":"user","content":"Hello, please introduce yourself in one sentence."}],
5       "stream":true}'

The terminal starts returning content in a stream, indicating that this 124B model is running locally.

03 Testing Model Performance and Capability

1. Token Throughput When the model first started, I ran a round of tests using vLLM's built-in benchmark. All 20 requests were completed, with an output throughput of 84.34 tok/s, total token throughput of 133.10 tok/s, and an MTP acceptance rate of 67.35%.

Original log output 👇🏻

1============ Serving Benchmark Result ============
2Successful requests:                     20
3Failed requests:                         0
4Request rate configured (RPS):           2.00
5Benchmark duration (s):                  60.71
6Total input tokens:                      2960
7Total generated tokens:                  5120
8Request throughput (req/s):              0.33
9Output token throughput (tok/s):         84.34
10Peak output token throughput (tok/s):    61.00
11Peak concurrent requests:                20.00
12Total token throughput (tok/s):          133.10
13---------------Time to First Token----------------
14Mean TTFT (ms):                          19692.98
15Median TTFT (ms):                        18877.99
16P99 TTFT (ms):                           40039.02
17-----Time per Output Token (excl. 1st token)------
18Mean TPOT (ms):                          44.61
19Median TPOT (ms):                        44.09
20P99 TPOT (ms):                           49.68
21---------------Inter-token Latency----------------
22Mean ITL (ms):                           74.09
23Median ITL (ms):                         72.39
24P99 ITL (ms):                            280.71
25---------------Speculative Decoding---------------
26Acceptance rate (%):                     67.35
27Acceptance length:                       1.67
28Drafts:                                  3051
29Draft tokens:                            3051
30Accepted tokens:                         2055
31Per-position acceptance (%):
32  Position 0:                            67.35
33==================================================

Subsequently, I performed concurrency tests on Ling and the local Qwen 3.8-27B. At 8 concurrency, Ling's aggregate throughput was 141.38 tok/s, while Qwen 3.8 was 32.61 tok/s, a difference of about 4.34 times in this round.

🔥🔥🔥Ling-3.0-Flash Vs Qwen3.8-27B Stress Test Comparison

Ling-3.0-flash

Qwen 3.8 -27B

Some might wonder: why is Ling's single concurrency only 34.77 tok/s, but at 8 concurrency it becomes 141.38 tok/s? Technical friends might also ask if this single concurrency score is slow compared to official data.

Here we need to explain the specific testing method and the speed differences caused by different evaluation methods:

1. Testing Method and Real End-to-End Link: This test uses a local OpenAI-compatible interface, stress-testing via HTTP streaming requests, not an offline inference test detached from the service framework. Each Ling request uses a long text prompt of about 150 tokens, generating up to 512 tokens. Timing starts from the client's HTTP request until the streaming response is complete, thus including local HTTP calls, service scheduling, Tokenizer processing, Prefill, token-by-token decoding, and streaming return.
2. Single-stream Output Rate vs. Machine Aggregate Throughput: Single-stream speed (real user experience): At $c=1$, the end-to-end output rate is about 35.34 tok/s (real single conversation tests at 38+ tok/s), equivalent to over 35 Chinese characters per second, which is extremely fast visually; Aggregate throughput (total output under concurrency): As concurrency increases, vLLM uses Continuous Batching to combine multiple requests into GPU calculations, fully utilizing Blackwell's unified memory bandwidth. At 8 concurrency, the machine's aggregate throughput soared to 141.38 tok/s.

As for why it differs from some official benchmarks, the key is the testing criteria. Model precision, inference engine, context length, input/output token count, concurrency scale, and whether offline inference or HTTP services are used all affect the final result. Only when these conditions are basically consistent are the numbers suitable for direct comparison.

Simply put: Speed varies due to evaluation methods—if tested in extremely high concurrency (like 32/64) or a pure computing environment without network protocols, the total throughput numbers would look higher; in our daily single-person conversations or coding production calls, Ling's single-stream 35+ tok/s and sub-220ms first-token latency already feel extremely smooth and lag-free.

At single concurrency, Ling's average single-stream speed is about 35.34 tok/s; at 8 concurrency, the aggregate throughput reaches 141.38 tok/s. Qwen3.8's aggregate throughput at 8 concurrency is 32.61 tok/s. In this round, Ling's advantage is mainly reflected in output speed and concurrent throughput, with noticeably faster replies in actual use.

2. Real Capabilities

Benchmarks only reflect part of the performance; actual usability depends on the model's performance on specific problems. I chose three directions for simple testing.

(1) Logical Reasoning

I prepared a variation of the chicken-and-rabbit problem, a classic car wash problem, and a car wash machine problem, mainly to see if it can accurately understand conditions rather than applying a familiar-looking answer. Among them, the chicken-and-rabbit problem included 4 mechanical birds with three legs to break conventional patterns.

(2) Safety Boundaries: Next, I tested its reaction to high-risk operations: whether it executes directly or identifies risks, confirms with the user, and provides safer alternatives.

(3) Long Text

Finally, a round of long text testing. I hid key information in a long context to see if it could accurately find and answer, while observing output speed and stability under long text.

04 From API to TUI, then to Tool Calling

We've completed model deployment and capability testing, but for ordinary users, curl is more suitable for verifying interfaces than daily use. To talk to a local model for a long time, a more convenient interaction interface is needed.

So I first made a simple TUI, a chat interface running in the terminal. It's not complex software; I had AI write a Python script to encapsulate local interface calls, streaming output, and conversation history, then started it with one command:

1python3 ling-3.0-chat.py

This way, I don't have to write curl every time. Open the terminal and chat directly; answers stream in, and you can see TPS, TTFT, and token counts. I completed the previous capability tests in this interface.

However, at this point, the TUI is just a text chat tool without the ability to call tools. The large model is like a "brain" responsible for thinking, but it has no "hands and feet" to read files, execute commands, or know the system's current state.

To let it call system capabilities, we need to add tool calling. Simply put, operations like command execution, file reading, and writing are encapsulated as tools. The model first judges what information it needs, then initiates a tool use; the Python script executes it and hands the result back to the model for further processing.

For example, initially, when I asked it to check system GPU info, it didn't know the real usage and could only tell me how to check. After adding tool calling, it could execute system commands itself and organize the query results directly in the TUI.

Based on the same principle, you can continue to connect Web search, internal enterprise interfaces, databases, etc. Specific tools can be expanded according to business needs.

05 Connecting to a Visual Interface

For personal use, a TUI with tool calling is actually quite enough. To go further and put the model into a more complete Agent workbench, you can connect to an agent framework like Harness.

This time I chose Liang Sheng's DeepSeek Harness, which doesn't just add a chat page but also provides context management, workspaces, tool calling, permission control, and task planning, with a built-in Web UI. It uses an "everything is a plugin" architecture, allowing for future functional expansion.

Note that DeepSeek Harness is still in the developer preview stage and updates quickly, which may lead to incompatible changes. There are many other open-source Agent frameworks; you can choose according to your needs.

The connection process isn't complex: the core is adding a custom model service and pointing the address to the local interface provided by vLLM. Besides configuring in the Web UI, you can also modify the configuration file as shown:

1llm-pi-ai:
2  providers:
3    ling:
4      displayName: "Ling-3.0-flash (124B)"
5      api: openai-completions
6      baseURL: http://127.0.0.1:30000/v1
7      apiKeyEnv: OPENAI_API_KEY
8      models:
9        - id: ling-int4
10          name: Ling-3.0-flash (124B MoE)

After starting the Web UI, open the default address in your browser:

1http://localhost:3080

This way, daily chatting, history, and model switching can all be done in DeepSeek Harness. Harness itself provides file operations, command execution, and task planning, which can be further extended via plugins. For specific usage and third-party plugins, interested friends can search for them.

Note that tools I wrote in the Python TUI won't automatically migrate. To use them in Harness, they need to be re-integrated according to its plugin mechanism. Below is the actual integration effect I made for Ling; you can watch the recording.

06 Building an AI Workflow

At this point, the single-model link from deployment to interface and tool calling for Ling-3.0-flash is fully established.

However, in real projects, we usually don't use just one model. Different models excel at different things; combining them is often better than having one model handle everything.

Ling-3.0-flash's advantage is text processing and generation speed, but it doesn't support native multimodal input. If a task requires understanding images or videos, you can connect a multimodal model like Qwen3.8-27B; if you need to generate video, you can connect the recently open-sourced MiniMax H3. Each model handles what it's best at, passing results to the next.

For example, to build a video generation workflow, Ling can first understand requirements, write scripts, and split storyboards, then the multimodal model checks reference materials and visual consistency, and finally, the video model generates it. After generation, another round of visual inspection can be done to modify prompts and regenerate based on results:

1Requirements and Materials
2→ Ling generates script and storyboards
3→ Multimodal model checks materials and visual requirements
4→ MiniMax H3 generates video
5→ Multimodal model checks visuals and continuity
6→ Ling adjusts prompts based on feedback
7→ Human completes final review

The video below shows the actual effect of prompts generated by Ling-3.0-flash and then handed to MiniMax H3 for generation.

Once this process is fixed, you only need to change requirements and materials for repeated use. However, running multiple large models locally simultaneously has very high VRAM and memory requirements. Ling INT4 is about 72GB, Qwen3.8-27B BF16 is about 51.77GB, plus KV Cache and video models; it's hard to keep them all resident on this Spark.

Before actual deployment, be sure to calculate how much VRAM or unified memory each model needs. When resources are insufficient, you can switch models step-by-step, choose quantized versions, or split models across multiple devices. Multiple models no longer run independently but collaborate around the same task—this is a truly usable multi-model AI workflow.

07 Final Thoughts

From model deployment, TUI, and tool calling to multi-model workflows, the entire link is complete. This article aims to share a repeatable method rather than a fixed configuration.

Open-source models will continue to update. In the future, whether you change models, quantization versions, or inference engines, the path from downloading weights and starting services to connecting tools and workflows won't change much.

If conditions allow, I still recommend everyone deploy local models personally:

1. Data Control: Files, conversations, and business data stay on your machine or intranet, with directory and command permissions restricted by you.
2. Suitable for High-Frequency Use: No need to calculate token costs per use, reducing dependence on network and third-party services.
3. Easy Customization: Models, quantization precision, inference engines, and tools can all be adjusted, and internal interfaces and databases can be connected.

As open-source models get stronger, local deployment will become the choice for more people. You can build tools and workflows based on your task needs, equipment conditions, and budget. I hope everyone can have their own local Agent in the future, without staring at token consumption every time, truly achieving their own "Token Freedom"!

Related Resources

• Hugging Face: Ling-3.0-flash INT4 Weights
• ModelScope: Ling-3.0-flash INT4 Weights
• Official vLLM Adaptation Repository

📚 Historical Article Summary

1. Hermes Agent Practical Guide: From X Anxiety to Automatic Accumulation
2. Programmer's Anti-Hair Loss Guide
3. Connecting Hermes to iMessage
4. Connecting Hermes to X Premium
5. Hermes Agent Complete Guide
6. Hermes Agent Introductory Guide: Auxiliary Models
7. Hermes Agent Introductory Guide
8. Hermes Agent Advanced Guide
9. Hermes Agent Incomplete Guide
10. Complete Guide to Claude Pro Subscription in Nigeria
11. Tutorial for Registering Apple ID in Nigeria
12. Half-Price ChatGPT Plus Subscription in Turkey
13. US Apple ID Registration
14. Claude/ChatGPT/Gemini Alipay Subscription
15. Complete Tutorial for Local LLM Deployment on Mac
16. IP Quality Check
17. Why Doubao Doesn't Recommend Your Brand
18. How to Explain to Your Grandma That What Doubao Said Isn't True

If this was helpful, please follow + bookmark + forward 👏🏻

Follow @Lonely__MH for continuous updates on beginner-friendly tutorials and AI tool insights.