references/token_usage.md

Token Usage Guide for Hugging Face Jobs

⚠️ CRITICAL: Proper token usage is essential for any job that interacts with the Hugging Face Hub.

Overview

Hugging Face tokens are authentication credentials that allow your jobs to interact with the Hub. They're required for:

• Pushing models/datasets to Hub
• Accessing private repositories
• Creating new repositories
• Using Hub APIs programmatically
• Any authenticated Hub operations

Token Types

Read Token

• Permissions: Download models/datasets, read private repos
• Use case: Jobs that only need to download/read content
• Creation: https://huggingface.co/settings/tokens

Write Token

• Permissions: Push models/datasets, create repos, modify content
• Use case: Jobs that need to upload results (most common)
• Creation: https://huggingface.co/settings/tokens
• ⚠️ Required for: Pushing models, datasets, or any uploads

Organization Token

• Permissions: Act on behalf of an organization
• Use case: Jobs running under organization namespace
• Creation: Organization settings → Tokens

Providing Tokens to Jobs

Method 1: Automatic Token (Recommended) ⭐

hf_jobs("uv", {
    "script": "your_script.py",
    "secrets": {"HF_TOKEN": "$HF_TOKEN"}  # ✅ Automatic replacement
})

How it works:

1. $HF_TOKEN is a placeholder that gets replaced with your actual token
2. Uses the token from your logged-in session (hf auth login)
3. Token is encrypted server-side when passed as a secret
4. Most secure and convenient method

Benefits:

• ✅ No token exposure in code
• ✅ Uses your current login session
• ✅ Automatically updated if you re-login
• ✅ Works seamlessly with MCP tools
• ✅ Token encrypted server-side

Requirements:

• Must be logged in: hf auth login or hf_whoami() works
• Token must have required permissions

Method 2: Explicit Token (Not Recommended)

hf_jobs("uv", {
    "script": "your_script.py",
    "secrets": {"HF_TOKEN": "hf_abc123..."}  # ⚠️ Hardcoded token
})

When to use:

• Only if automatic token doesn't work
• Testing with a specific token
• Organization tokens (use with caution)

Security concerns:

• ❌ Token visible in code/logs
• ❌ Must manually update if token rotates
• ❌ Risk of token exposure
• ❌ Not recommended for production

Method 3: Environment Variable (Less Secure)

hf_jobs("uv", {
    "script": "your_script.py",
    "env": {"HF_TOKEN": "hf_abc123..."}  # ⚠️ Less secure than secrets
})

Difference from secrets:

• env variables are visible in job logs
• secrets are encrypted server-side
• Always prefer secrets for tokens

When to use:

• Only for non-sensitive configuration
• Never use for tokens (use secrets instead)

Using Tokens in Scripts

Accessing Tokens

Tokens passed via secrets are available as environment variables in your script:

import os

# Get token from environment
token = os.environ.get("HF_TOKEN")

# Verify token exists
if not token:
    raise ValueError("HF_TOKEN not found in environment!")

Using with Hugging Face Hub

Option 1: Explicit token parameter

from huggingface_hub import HfApi

api = HfApi(token=os.environ.get("HF_TOKEN"))
api.upload_file(...)

Option 2: Auto-detection (Recommended)

from huggingface_hub import HfApi

# Automatically uses HF_TOKEN env var
api = HfApi()  # ✅ Simpler, uses token from environment
api.upload_file(...)

Option 3: With transformers/datasets

from transformers import AutoModel
from datasets import load_dataset

# Auto-detects HF_TOKEN from environment
model = AutoModel.from_pretrained("username/model")
dataset = load_dataset("username/dataset")

# For push operations, token is auto-detected
model.push_to_hub("username/new-model")
dataset.push_to_hub("username/new-dataset")

Complete Example

# /// script
# dependencies = ["huggingface-hub", "datasets"]
# ///

import os
from huggingface_hub import HfApi
from datasets import Dataset

# Verify token is available
assert "HF_TOKEN" in os.environ, "HF_TOKEN required for Hub operations!"

# Use token for Hub operations
api = HfApi()  # Auto-detects HF_TOKEN

# Create and push dataset
data = {"text": ["Hello", "World"]}
dataset = Dataset.from_dict(data)

# Push to Hub (token auto-detected)
dataset.push_to_hub("username/my-dataset")

print("✅ Dataset pushed successfully!")

Token Verification

Check Authentication Locally

from huggingface_hub import whoami

try:
    user_info = whoami()
    print(f"✅ Logged in as: {user_info['name']}")
except Exception as e:
    print(f"❌ Not authenticated: {e}")

Verify Token in Job

import os

# Check token exists
if "HF_TOKEN" not in os.environ:
    raise ValueError("HF_TOKEN not found in environment!")

token = os.environ["HF_TOKEN"]

# Verify token format (should start with "hf_")
if not token.startswith("hf_"):
    raise ValueError(f"Invalid token format: {token[:10]}...")

# Test token works
from huggingface_hub import whoami
try:
    user_info = whoami(token=token)
    print(f"✅ Token valid for user: {user_info['name']}")
except Exception as e:
    raise ValueError(f"Token validation failed: {e}")

Common Token Issues

Error: 401 Unauthorized

Symptoms:

401 Client Error: Unauthorized for url: https://huggingface.co/api/...

Causes:

1. Token missing from job
2. Token invalid or expired
3. Token not passed correctly

Solutions:

1. Add secrets={"HF_TOKEN": "$HF_TOKEN"} to job config
2. Verify hf_whoami() works locally
3. Re-login: hf auth login
4. Check token hasn't expired

Verification:

# In your script
import os
assert "HF_TOKEN" in os.environ, "HF_TOKEN missing!"

Error: 403 Forbidden

Symptoms:

403 Client Error: Forbidden for url: https://huggingface.co/api/...

Causes:

1. Token lacks required permissions (read-only token used for write)
2. No access to private repository
3. Organization permissions insufficient

Solutions:

1. Ensure token has write permissions
2. Check token type at https://huggingface.co/settings/tokens
3. Verify access to target repository
4. Use organization token if needed

Check token permissions:

from huggingface_hub import whoami

user_info = whoami()
print(f"User: {user_info['name']}")
print(f"Type: {user_info.get('type', 'user')}")

Error: Token not found in environment

Symptoms:

KeyError: 'HF_TOKEN'
ValueError: HF_TOKEN not found

Causes:

1. secrets not passed in job config
2. Wrong key name (should be HF_TOKEN)
3. Using env instead of secrets

Solutions:

1. Use secrets={"HF_TOKEN": "$HF_TOKEN"} (not env)
2. Verify key name is exactly HF_TOKEN
3. Check job config syntax

Correct configuration:

# ✅ Correct
hf_jobs("uv", {
    "script": "...",
    "secrets": {"HF_TOKEN": "$HF_TOKEN"}
})

# ❌ Wrong - using env instead of secrets
hf_jobs("uv", {
    "script": "...",
    "env": {"HF_TOKEN": "$HF_TOKEN"}  # Less secure
})

# ❌ Wrong - wrong key name
hf_jobs("uv", {
    "script": "...",
    "secrets": {"TOKEN": "$HF_TOKEN"}  # Wrong key
})

Error: Repository access denied

Symptoms:

403 Client Error: Forbidden
Repository not found or access denied

Causes:

1. Token doesn't have access to private repo
2. Repository doesn't exist and can't be created
3. Wrong namespace

Solutions:

1. Use token from account with access
2. Verify repo visibility (public vs private)
3. Check namespace matches token owner
4. Create repo first if needed

Check repository access:

from huggingface_hub import HfApi

api = HfApi()
try:
    repo_info = api.repo_info("username/repo-name")
    print(f"✅ Access granted: {repo_info.id}")
except Exception as e:
    print(f"❌ Access denied: {e}")

Token Security Best Practices

1. Never Commit Tokens

❌ Bad:

# Never do this!
token = "hf_abc123xyz..."
api = HfApi(token=token)

✅ Good:

# Use environment variable
token = os.environ.get("HF_TOKEN")
api = HfApi(token=token)

2. Use Secrets, Not Environment Variables

❌ Bad:

hf_jobs("uv", {
    "script": "...",
    "env": {"HF_TOKEN": "$HF_TOKEN"}  # Visible in logs
})

✅ Good:

hf_jobs("uv", {
    "script": "...",
    "secrets": {"HF_TOKEN": "$HF_TOKEN"}  # Encrypted server-side
})

3. Use Automatic Token Replacement

❌ Bad:

hf_jobs("uv", {
    "script": "...",
    "secrets": {"HF_TOKEN": "hf_abc123..."}  # Hardcoded
})

✅ Good:

hf_jobs("uv", {
    "script": "...",
    "secrets": {"HF_TOKEN": "$HF_TOKEN"}  # Automatic
})

4. Rotate Tokens Regularly

• Generate new tokens periodically
• Revoke old tokens
• Update job configurations
• Monitor token usage

5. Use Minimal Permissions

• Create tokens with only needed permissions
• Use read tokens when write isn't needed
• Don't use admin tokens for regular jobs

6. Don't Share Tokens

• Each user should use their own token
• Don't commit tokens to repositories
• Don't share tokens in logs or messages

7. Monitor Token Usage

• Check token activity in Hub settings
• Review job logs for token issues
• Set up alerts for unauthorized access

Token Workflow Examples

Example 1: Push Model to Hub

hf_jobs("uv", {
    "script": """
# /// script
# dependencies = ["transformers"]
# ///

import os
from transformers import AutoModel, AutoTokenizer

# Verify token
assert "HF_TOKEN" in os.environ, "HF_TOKEN required!"

# Load and process model
model = AutoModel.from_pretrained("base-model")
# ... process model ...

# Push to Hub (token auto-detected)
model.push_to_hub("username/my-model")
print("✅ Model pushed!")
""",
    "flavor": "a10g-large",
    "timeout": "2h",
    "secrets": {"HF_TOKEN": "$HF_TOKEN"}  # ✅ Token provided
})

Example 2: Access Private Dataset

hf_jobs("uv", {
    "script": """
# /// script
# dependencies = ["datasets"]
# ///

import os
from datasets import load_dataset

# Verify token
assert "HF_TOKEN" in os.environ, "HF_TOKEN required!"

# Load private dataset (token auto-detected)
dataset = load_dataset("private-org/private-dataset")
print(f"✅ Loaded {len(dataset)} examples")
""",
    "flavor": "cpu-basic",
    "timeout": "30m",
    "secrets": {"HF_TOKEN": "$HF_TOKEN"}  # ✅ Token provided
})

Example 3: Create and Push Dataset

hf_jobs("uv", {
    "script": """
# /// script
# dependencies = ["datasets", "huggingface-hub"]
# ///

import os
from datasets import Dataset
from huggingface_hub import HfApi

# Verify token
assert "HF_TOKEN" in os.environ, "HF_TOKEN required!"

# Create dataset
data = {"text": ["Sample 1", "Sample 2"]}
dataset = Dataset.from_dict(data)

# Push to Hub
api = HfApi()  # Auto-detects HF_TOKEN
dataset.push_to_hub("username/my-dataset")
print("✅ Dataset pushed!")
""",
    "flavor": "cpu-basic",
    "timeout": "30m",
    "secrets": {"HF_TOKEN": "$HF_TOKEN"}  # ✅ Token provided
})

Quick Reference

Token Checklist

Before submitting a job that uses Hub:

• Job includes secrets={"HF_TOKEN": "$HF_TOKEN"}
• Script checks for token: assert "HF_TOKEN" in os.environ
• Token has required permissions (read/write)
• User is logged in: hf_whoami() works
• Token not hardcoded in script
• Using secrets not env for token

Common Patterns

Pattern 1: Auto-detect token

from huggingface_hub import HfApi
api = HfApi()  # Uses HF_TOKEN from environment

Pattern 2: Explicit token

import os
from huggingface_hub import HfApi
api = HfApi(token=os.environ.get("HF_TOKEN"))

Pattern 3: Verify token

import os
assert "HF_TOKEN" in os.environ, "HF_TOKEN required!"

Key Takeaways

1. Always use secrets={"HF_TOKEN": "$HF_TOKEN"} for Hub operations
2. Never hardcode tokens in scripts or job configs
3. Verify token exists in script before Hub operations
4. Use auto-detection when possible (HfApi() without token parameter)
5. Check permissions - ensure token has required access
6. Monitor token usage - review activity regularly
7. Rotate tokens - generate new tokens periodically