inference/README.md

Inference

The evaluation example uses the diagnostic prompts and settings in docs/evaluation_prompts_and_settings.md. It keeps system and user messages separate and loads the released checkpoint with the native Transformers Qwen2_5_VLForConditionalGeneration class in bfloat16. Evaluation, chat, and API entry points share the full-precision loader. Training components remain under train/ and are not imported by inference.

Install

Use the repository environment with a CUDA-enabled PyTorch build:

conda env create -f environment.yml
conda activate skingpt-r1
python -m pip install flash-attn --no-build-isolation

FlashAttention-2 requires a compatible GPU, CUDA toolkit, and PyTorch build. Use --attn-implementation sdpa when selecting the PyTorch attention implementation. This changes the attention setting recorded with the run. See the Qwen2.5-VL documentation for attention and image processing requirements.

Single-image example

Run from the repository root. Point --model-path to the local checkpoint directory containing the actual weight shards and processor files.

CUDA_VISIBLE_DEVICES=0 python -m inference.evaluation.run_inference --model-path ./checkpoint --image /path/to/lesion.jpg --mode ddi --output outputs/lesion.json

The system message is:

You are a dermatology assistant. Analyze the supplied skin-lesion image, produce a concise hierarchical clinical rationale based on observable image evidence, consider plausible differential diagnoses, and state the final diagnosis. Do not invent findings that are not supported by the image.

The user message contains the image and prompts/ddi_user.txt. No reference diagnosis or candidate list is supplied in this mode.

Candidate-label classification

This mode has no system message. The supplied JSON file defines the candidate labels and their order.

CUDA_VISIBLE_DEVICES=0 python -m inference.evaluation.run_inference --model-path ./checkpoint --image /path/to/lesion.jpg --mode classification --labels prompts/labels_160case.json --output outputs/classification.json

The vocabulary file provides the 23 candidate diagnoses used in the 160-case comparison. The original candidate order varied by case. Supply the intended ordering for a specific evaluation request.

Runtime controls

Control	Default
Precision	bfloat16
Attention	FlashAttention-2
TF32	Enabled for CUDA matrix multiplication
Padding	Left
Image pixels	3136 to 1003520
Batch size	16 images per worker
Maximum new tokens	4096
Sampling	Temperature 0.7, top-p 0.9
Repetition penalty	1.0
Seed	42
Cache	Enabled

Repeat --image to supply multiple images. The example processes them in batches on one visible GPU. --batch-size, --max-new-tokens, --seed, and --attn-implementation override their configuration values. --greedy disables sampling. The output includes messages, effective settings, generation configuration, model configuration digest, and runtime versions. It records full clinical model outputs locally under the path you choose.

Inspect a request before loading the model:

python -m inference.evaluation.run_inference --image /path/to/lesion.jpg --mode ddi --dry-run

Dry runs use the standard library and do not load weights or require a GPU. Actual inference requires local weight tensors and fails explicitly if a shard is absent or is only a Git LFS pointer. This entry point does not download checkpoints. Transformers may report unused checkpoint tensors outside the inference architecture. Missing or incompatible inference weights cause loading to fail.

Other interfaces

The full_precision/ directory provides interactive chat and FastAPI examples using the same model loader. Their interactive prompts and generation defaults are separate from the evaluation configuration. The default model path is ./checkpoint.

Interface	Command from the repository root
Full-precision chat	bash inference/full_precision/run_chat.sh --image /path/to/lesion.jpg
Full-precision API	bash inference/full_precision/run_api.sh

The API endpoints include /v1/upload/{state_id}, /v1/predict/{state_id}, /v1/reset/{state_id}, /diagnose/stream, and /health. The default API port is 5900. See each entry point for its request schema.