DeepSeek R1 and Reasoning Models: Architecture, RL & Local Deployment

The open-source artificial intelligence landscape underwent an earthquake with the arrival of DeepSeek R1. For years, frontier reasoning capabilities—specifically test-time compute scaling, recursive backtracking, and long-horizon chain-of-thought (CoT) problem solving—remained locked behind proprietary, black-box APIs like OpenAI's o1 and o3. DeepSeek R1 shattered this barrier by matching and often exceeding frontier reasoning benchmarks while making its model weights, architectural methodology, and distillation datasets fully accessible to the global engineering community.

Unlike standard autoregressive transformer models that generate immediate token predictions, reasoning models introduce an explicit deliberation phase before synthesizing their final answer. Inside this internal deliberation workspace, the model explores alternative analytical hypotheses, verifies intermediate algebraic steps, backtracks upon hitting dead ends, and autonomously corrects its own reasoning flaws. In this architectural guide, we dissect the core mechanics of DeepSeek R1: Group Relative Policy Optimization (GRPO), pure reinforcement learning without supervised warmups, knowledge distillation into compact edge models, and low-latency self-hosted deployment using vLLM and Ollama.

1. The Core Paradigm Shift: Test-Time Compute vs Pre-Training Scaling

Historically, advancing large language model capabilities relied almost entirely on pre-training scaling laws: increasing total training parameter count, expanding dataset tokens, and consuming exponential GPU-cluster compute hours. While effective, pre-training hits steep diminishing returns, bounded by web data saturation and prohibitive multi-million-dollar training budgets.

Reasoning models like DeepSeek R1 unlock a complementary dimension: inference-time compute scaling. Instead of forcing the transformer to produce token t_{i} based solely on static parameter weights, the system generates thousands of hidden reasoning tokens enclosed within a <think> ... </think> envelope. This gives the model test-time computational budget to simulate possibilities, check constraints, and evaluate logic trees before committing to a public answer.

2. DeepSeek R1-Zero vs R1: The Power of Pure RL

The most fascinating breakthrough published by DeepSeek AI was the creation of DeepSeek R1-Zero. Traditional instruction-tuning requires hundreds of thousands of meticulously human-annotated Supervised Fine-Tuning (SFT) reasoning chains. DeepSeek demonstrated that reasoning can emerge entirely through pure reinforcement learning applied directly to a base foundation model without prior SFT warmups.

During R1-Zero training, the model received binary reward feedback solely on verifiable tasks (like mathematical proofs and competitive programming unit tests):

• Accuracy Rewards: Does the final answer match the ground truth mathematically, or does the generated Python code pass all unit tests in a sandboxed compiler?
• Format Rewards: Did the model place its internal deliberation inside the designated <think> tokens and its final answer after the closing tag?

Remarkably, as RL steps progressed, R1-Zero autonomously developed self-reflection, step verification, and long chain-of-thought exploration without human demonstration. However, R1-Zero suffered from language mixing (switching between English and Chinese mid-thought) and poor readability. The production DeepSeek R1 pipeline solved this by adding a modest cold-start dataset of structured reasoning data, followed by multi-stage reinforcement learning and supervised distillation.

3. Group Relative Policy Optimization (GRPO) Explained

In standard Reinforcement Learning from Human Feedback (RLHF), algorithms like Proximal Policy Optimization (PPO) require maintaining a separate Critic Model (value network) identical in parameter size to the actor model to estimate expected value states. For a 671-billion-parameter Mixture-of-Experts (MoE) model, running a concurrent critic network exhausts petabytes of GPU VRAM.

DeepSeek invented Group Relative Policy Optimization (GRPO). Instead of training a heavy critic network, GRPO samples a group of candidate outputs {o_1, o_2, ..., o_G} from the old policy for each prompt q. The reward r_i for each candidate is normalized against the group's mean and standard deviation:

import numpy as np

def calculate_grpo_advantages(rewards: list[float], eps: float = 1e-8) -> np.ndarray:
    """
    Computes Group Relative Policy Optimization (GRPO) advantages.
    Eliminates the separate critic network by normalizing rewards across a candidate group.
    """
    rewards_arr = np.array(rewards, dtype=np.float32)
    group_mean = np.mean(rewards_arr)
    group_std = np.std(rewards_arr)
    
    # Normalized relative advantage per generation candidate
    advantages = (rewards_arr - group_mean) / (group_std + eps)
    return advantages

# Example: 4 candidate reasoning paths generated by the model
candidate_rewards = [0.85, 0.20, 0.95, 0.40]
adv = calculate_grpo_advantages(candidate_rewards)

for i, (r, a) in enumerate(zip(candidate_rewards, adv)):
    print(f"Candidate {i+1}: Raw Reward = {r:.2f} | Relative GRPO Advantage = {a:+.3f}")

This formulation slashes memory overhead dramatically, enabling massive-scale RL training across distributed cluster nodes without allocating memory to an idle critic network.

4. Knowledge Distillation: Powering 1.5B to 70B Models

While the full DeepSeek R1 model runs on a massive 671B parameter Mixture-of-Experts architecture (activating 37B parameters per token), DeepSeek utilized 800,000 curated R1 reasoning chains to distill reasoning behavior directly into compact dense architectures, including Llama-3 and Qwen-2.5:

5. Production Deployment: Running DeepSeek R1 with vLLM & Ollama

For high-throughput enterprise API backends, vLLM provides state-of-the-art PagedAttention, continuous batching, and tensor parallelism. Here is how to orchestrate a high-performance production server with Docker:

# Run DeepSeek-R1-Distill-Qwen-14B with FP8 / AWQ quantization on NVIDIA GPU
docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model deepseek-ai/DeepSeek-R1-Distill-Qwen-14B \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.92 \
  --enforce-eager

For local developer workstations running macOS or Windows with consumer GPUs, Ollama allows instant command-line execution:

# Pull and run the 14-billion distilled reasoning model
ollama run deepseek-r1:14b

# Query via standard OpenAI-compatible curl endpoint
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-r1:14b",
    "messages": [
      {"role": "user", "content": "Prove that the sum of the first n odd integers is n^2."}
    ]
  }'

6. Frequently Asked Questions (FAQ)