Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
Paper • 2608.03796 • Published • 20
SomosNLP
non-profit
AI & ML interests
Democratizar el PLN en español e incentivar su aplicación para generar impacto social 💛
Recent Activity
View all activity
Iker
submitted a
paper to Daily Papers about 2 months ago
Iker
authored 5
papers 2 months ago
Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
Paper • 2608.03796 • Published • 20
Refusal Steering: Fine-grained Control over LLM Refusal Behaviour for Sensitive Topics
Paper • 2512.16602 • Published
Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque
Paper • 2506.07597 • Published
GuideX: Guided Synthetic Data Generation for Zero-Shot Information Extraction
Paper • 2506.00649 • Published • 4
Cross-Lingual Transfer for Low-Resource Natural Language Processing
Paper • 2502.02722 • Published
maximorulli
authored a
paper 3 months ago
maximorulli
submitted a
paper to Daily Papers 3 months ago
osanseviero
authored a
paper 3 months ago
osanseviero
submitted a
paper to Daily Papers 3 months ago
haritzpuerto
authored a
paper 4 months ago
haritzpuerto
submitted a
paper to Daily Papers 4 months ago
alvarobartt
posted an update 5 months ago
Post
2165
Open agents on AWS SageMaker AI with open models from the Hugging Face Hub!
> Deploy an open model from the Hugging Face Hub on SageMaker AI
> Connect the deployed model to Strands Agents
> Add built-in and custom tools for tool calling
> Expose external capabilities through MCP integration
> Bonus: talk to your agent and visualize traces with Gradio
https://alvarobartt.com/agents-on-aws-sagemaker
> Deploy an open model from the Hugging Face Hub on SageMaker AI
> Connect the deployed model to Strands Agents
> Add built-in and custom tools for tool calling
> Expose external capabilities through MCP integration
> Bonus: talk to your agent and visualize traces with Gradio
https://alvarobartt.com/agents-on-aws-sagemaker
alvarobartt
posted an update 5 months ago
Post
4685
Latest
TL; DR MoEs can be misleading to reason about from active parameters alone, since each token only activates a subset of experts, while the serving setup still needs to account for the full resident memory footprint.
🧠
🏗️ Dense models usually load and use most weights every forward pass, while MoEs load many experts but only route each token to a few of them
⚡ Active params isn't the same as memory footprint, especially for sparse architectures
📦 Runtime memory is about what is used per request/token, while loading memory also includes the expert weights that need to be resident
📚 KV cache can still dominate depending on context length, batch size, and concurrency
🔀 Expert Parallelism (EP) helps shard experts across accelerators when expert weights dominate
🚀 Data Parallelism (DP) + EP is often a good fit for throughput-oriented MoE serving
Check the repository at https://github.com/alvarobartt/hf-mem
hf-mem release added a breakdown of Mixture-of-Experts (MoE) memory usage!TL; DR MoEs can be misleading to reason about from active parameters alone, since each token only activates a subset of experts, while the serving setup still needs to account for the full resident memory footprint.
🧠
hf-mem now splits MoE memory into base model weights, routed experts, and KV cache🏗️ Dense models usually load and use most weights every forward pass, while MoEs load many experts but only route each token to a few of them
⚡ Active params isn't the same as memory footprint, especially for sparse architectures
📦 Runtime memory is about what is used per request/token, while loading memory also includes the expert weights that need to be resident
📚 KV cache can still dominate depending on context length, batch size, and concurrency
🔀 Expert Parallelism (EP) helps shard experts across accelerators when expert weights dominate
🚀 Data Parallelism (DP) + EP is often a good fit for throughput-oriented MoE serving
Check the repository at https://github.com/alvarobartt/hf-mem
alvarobartt
posted an update 7 months ago
Post
4607
Learn how to deploy Microsoft Research VibeVoice ASR on Microsoft Azure Foundry with Hugging Face to generate rich audio transcriptions with Who, When, and What! 💥
> 🕒 60-minute single-pass processing, no chunking or stitching
> 👤 Customized hotwords to guide recognition on domain-specific content
> 📝 Rich transcription: joint ASR + diarization + timestamping in one pass
> 🌍 50+ languages with automatic detection and code-switching support
> 🤗 Deployed on Microsoft Foundry via an OpenAI-compatible Chat Completions API
https://huggingface.co/docs/microsoft-azure/foundry/examples/deploy-vibevoice-asr
> 🕒 60-minute single-pass processing, no chunking or stitching
> 👤 Customized hotwords to guide recognition on domain-specific content
> 📝 Rich transcription: joint ASR + diarization + timestamping in one pass
> 🌍 50+ languages with automatic detection and code-switching support
> 🤗 Deployed on Microsoft Foundry via an OpenAI-compatible Chat Completions API
https://huggingface.co/docs/microsoft-azure/foundry/examples/deploy-vibevoice-asr
haritzpuerto
authored a
paper 7 months ago
mariagrandury
authored 2
papers 8 months ago
lewtun
submitted 2
papers to Daily Papers 8 months ago