> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/QwenLM/Qwen3-VL/llms.txt
> Use this file to discover all available pages before exploring further.

# Training Script

> Complete training script reference with all parameters for Qwen3-VL fine-tuning

## Complete Training Script

Here's the complete training script with full parameter documentation:

```bash launch_training.sh theme={null}
#!/bin/bash
# Complete QwenVL Training Launch Script with Full Parameter Documentation

# ======================
# Distributed Configuration
# ======================
MASTER_ADDR="127.0.0.1"                     # [Required] Master node IP for multi-GPU training
MASTER_PORT=$(shuf -i 20000-29999 -n 1)     # Random port to avoid conflicts
NPROC_PER_NODE=$(nvidia-smi --list-gpus | wc -l)  # Automatically detects available GPUs

# ======================
# Path Configuration
# ======================
MODEL_PATH="/path/to/Qwen2.5-VL-3B-Instruct"  # [ModelArguments] Pretrained model path
OUTPUT_DIR="./checkpoints"                   # Directory for saving checkpoints
CACHE_DIR="./cache"                          # [TrainingArguments] Cache directory for models

# ======================
# Model Configuration
# ======================
DATASETS="your_dataset%100"                  # [DataArguments] Dataset with sampling rate

# ======================
# Training Hyperparameters
# ======================
torchrun --nproc_per_node=$NPROC_PER_NODE \
         --master_addr=$MASTER_ADDR \
         --master_port=$MASTER_PORT \
         qwenvl/train/train_qwen.py \
         # Core Arguments
         --model_name_or_path $MODEL_PATH \  # [ModelArguments] Model identifier
         --tune_mm_llm True \                # [TrainingArguments] Train LLM or not
         --tune_mm_vision False \            # [TrainingArguments] Train VIT or not
         --tune_mm_mlp False \               # [TrainingArguments] Train MLP or not
         --dataset_use $DATASETS \           # [DataArguments] Dataset specification
         --output_dir $OUTPUT_DIR \          # Output directory for checkpoints
         --cache_dir $CACHE_DIR \            # [TrainingArguments] Model cache location
         
         # Precision & Memory
         --bf16 \                            # Use bfloat16 precision (Ampere+ GPUs)
         --per_device_train_batch_size 4 \   # Batch size per GPU
         --gradient_accumulation_steps 4 \   # Effective batch size multiplier
         
         # Learning Rate Configuration
         --learning_rate 2e-7 \              # Base learning rate
         --mm_projector_lr 1e-5 \            # [TrainingArguments] Projector-specific LR
         --vision_tower_lr 1e-6 \            # [TrainingArguments] Vision encoder LR
         --optim adamw_torch \               # [TrainingArguments] Optimizer selection
         
         # Sequence Configuration
         --model_max_length 4096 \           # [TrainingArguments] Max sequence length
         --data_flatten True \               # [DataArguments] Concatenate batch sequences
         --data_packing True \               # [DataArguments] Using packing data
         
         # Image Processing
         --max_pixels 576\*28\*28 \               # [DataArguments] Max image pixels (H*W) for image
         --min_pixels 16\*28\*28 \                # [DataArguments] Min image pixels for image
         # Video Processing
         --video_fps 2 \                          # [DataArguments] video fps
         --video_max_frames 8 \                   # [DataArguments] Max frames per video
         --video_min_frames 4 \                   # [DataArguments] Min frames per video
         --video_max_pixels 1664\*28\*28 \        # [DataArguments] Max pixels per video
         --video_min_pixels 256\*28\*28 \         # [DataArguments] Min pixels per video
         
         # Training Schedule
         --num_train_epochs 3 \              # Total training epochs
         --warmup_ratio 0.03 \               # LR warmup proportion
         --lr_scheduler_type "cosine" \      # Learning rate schedule
         --weight_decay 0.01 \               # L2 regularization strength
         
         # Logging & Checkpoints
         --logging_steps 10 \               # Log metrics interval
         --save_steps 500 \                 # Checkpoint save interval
         --save_total_limit 3 \             # Max checkpoints to keep

         # Lora Config
         --lora_enable True \                 # [TrainingArguments] Enable LoRA
         --lora_r 8 \                         # [TrainingArguments] LoRA r
         --lora_alpha 16 \                    # [TrainingArguments] LoRA alpha 
         --lora_dropout 0.0 \                # [TrainingArguments] LoRA dropout

         # Advanced Options
         --deepspeed zero3.json \           # DeepSpeed configuration
```

## Parameter Categories

The script accepts arguments in three main categories:

### Model Arguments

| Parameter | Description | Default |
| - | - | - |
| `--model_name_or_path` | Path or identifier for pretrained model | Required |

### Training Arguments

#### Component Training Flags

| Parameter | Description | Recommended |
| - | - | - |
| `--tune_mm_llm` | Whether to train the language model | `True` |
| `--tune_mm_vision` | Whether to train the vision encoder | `False` (for mixed image/video) |
| `--tune_mm_mlp` | Whether to train the MLP projector | `False` |

> [!WARNING]
>
> When training with both image and video data, set `--tune_mm_vision False` to avoid instability.

#### Precision & Memory

| Parameter | Description | Value |
| - | - | - |
| `--bf16` | Use bfloat16 precision (requires Ampere+ GPUs) | Flag |
| `--per_device_train_batch_size` | Batch size per GPU | `4` |
| `--gradient_accumulation_steps` | Gradient accumulation steps | `4` |
| `--cache_dir` | Cache directory for models | `./cache` |

#### Learning Rate Configuration

| Parameter | Description | Range |
| - | - | - |
| `--learning_rate` | Base learning rate for the model | `1e-6` to `2e-7` |
| `--mm_projector_lr` | Learning rate for multimodal projector | `1e-5` |
| `--vision_tower_lr` | Learning rate for vision encoder | `1e-6` |
| `--optim` | Optimizer type | `adamw_torch` |

> [!TIP]
>
> The suggested learning rate range is from `1e-6` to `2e-7`. Start with `2e-7` for stable training.

#### Sequence Configuration

| Parameter | Description | Value |
| - | - | - |
| `--model_max_length` | Maximum sequence length | `4096` |

### Data Arguments

#### Dataset Selection

| Parameter | Description | Example |
| - | - | - |
| `--dataset_use` | Dataset names with sampling rates | `"my_dataset%100"` |

#### Data Processing

| Parameter | Description | Default |
| - | - | - |
| `--data_flatten` | Concatenate batch sequences into one | `True` |
| `--data_packing` | Use packed data (requires preprocessing) | `True` |

> [!NOTE]
>
> * `data_flatten=True` means data in a batch are concatenated into one sequence
> * `data_packing=True` requires preprocessing with `tools/pack_data.py`

#### Image Processing

| Parameter | Description | Value |
| - | - | - |
| `--max_pixels` | Maximum image pixels (H×W) | `576*28*28` |
| `--min_pixels` | Minimum image pixels | `16*28*28` |

#### Video Processing

| Parameter | Description | Value |
| - | - | - |
| `--video_fps` | Video frames per second | `2` |
| `--video_max_frames` | Maximum frames per video | `8` |
| `--video_min_frames` | Minimum frames per video | `4` |
| `--video_max_pixels` | Maximum pixels per video | `1664*28*28` |
| `--video_min_pixels` | Minimum pixels per video | `256*28*28` |

> [!WARNING]
>
> **Training resolution is critical** for model performance. Ensure `--max_pixels` and `--min_pixels` are properly set for your use case.

#### Training Schedule

| Parameter | Description | Value |
| - | - | - |
| `--num_train_epochs` | Total training epochs | `3` |
| `--warmup_ratio` | Learning rate warmup proportion | `0.03` |
| `--lr_scheduler_type` | Learning rate schedule | `cosine` |
| `--weight_decay` | L2 regularization strength | `0.01` |

#### Logging & Checkpoints

| Parameter | Description | Value |
| - | - | - |
| `--logging_steps` | Interval for logging metrics | `10` |
| `--save_steps` | Interval for saving checkpoints | `500` |
| `--save_total_limit` | Maximum checkpoints to keep | `3` |

## Advanced Options

### DeepSpeed Configuration

```bash theme={null}
--deepspeed zero3.json
```

Provide a DeepSpeed configuration file for distributed training optimization.

> [!WARNING]
>
> The Qwen3VL MoE model does **not support** DeepSpeed with ZeRO-3. Additionally, Hugging Face's official implementation does not include support for load balancing loss currently.

### Flash Attention

To enable Flash Attention 2, add the following to your model's `config.json`:

```json config.json theme={null}
{
  "_attn_implementation": "flash_attention_2",
  ...
}
```

## Hardware Requirements

### Training Qwen2.5-VL-3B

Minimum requirements:

* 4x GPUs with 24GB VRAM (e.g., RTX 3090, RTX 4090)
* With DeepSpeed ZeRO-3 and gradient checkpointing

### Training Qwen2.5-VL-32B

Recommended configuration:

* 8x 80GB GPUs (e.g., A100, H100)
* Refer to `scripts/sft_32b.sh` for configuration

## Example Usage

### Basic Training

```bash theme={null}
bash launch_training.sh
```

### Single GPU Training

```bash theme={null}
NPROC_PER_NODE=1 bash launch_training.sh
```

### Multi-node Training

On the master node:

```bash theme={null}
MASTER_ADDR="192.168.1.1" bash launch_training.sh
```

On worker nodes:

```bash theme={null}
MASTER_ADDR="192.168.1.1" bash launch_training.sh
```

## Monitoring Training

Monitor your training progress:

```bash theme={null}
# View logs
tail -f checkpoints/training.log

# Monitor GPU usage
watch -n 1 nvidia-smi
```

## Troubleshooting

> **Out of Memory (OOM) Errors**
>
> Try these solutions:
>
> 1. Reduce `--per_device_train_batch_size`
> 2. Increase `--gradient_accumulation_steps`
> 3. Reduce `--model_max_length`
> 4. Enable gradient checkpointing in DeepSpeed config
> 5. Use DeepSpeed ZeRO-3 for larger models

> **Slow Training Speed**
>
> Optimize performance:
>
> 1. Enable Flash Attention 2 in config.json
> 2. Use `--data_packing True` with preprocessed data
> 3. Ensure `--bf16` is enabled on Ampere+ GPUs
> 4. Check if GPU utilization is at 100%
> 5. Increase batch size if memory allows

> **Training Instability**
>
> If you see NaN losses or diverging training:
>
> 1. Lower the learning rate (try `1e-7` or `5e-8`)
> 2. Set `--tune_mm_vision False` when using mixed image/video data
> 3. Increase warmup ratio to `0.05` or `0.1`
> 4. Check data for corrupted images or invalid annotations
> 5. Reduce `--max_pixels` if processing very large images


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.