Mercity/figure-skating-transformer-bilstm

Figure Skating Transformer-BiLSTM Action Classifier

A two-stage temporal model for classifying figure-skating actions (jumps, spins, sequences) from pose-skeleton time series, with no conv backbone: a linear projection + sinusoidal positional encoding + nn.TransformerEncoder (self-attention, global context) feeding a bidirectional LSTM decoder (re-imposes local temporal order before pooling).

This repo holds two checkpoints trained on the same architecture and data split, differing only in label granularity:

file	label space	classes
model_transformer_bilstm_fine.pt	fine-grained	28 (jump rotation-count preserved, e.g. 3Lutz, 2Axel)
model_transformer_bilstm_coarse.pt	coarse	11 (jump rotation-count collapsed, e.g. Lutz, Axel)

Not transformers-compatible in the HF sense — this is a plain PyTorch nn.Module + state_dict checkpoint (custom architecture, not a Hub AutoModel). Model source code (SkatingTransformerBiLSTMClassifier in the project's model_transformer_bilstm.py, which depends on model.py and model_transformers.py) is not included in this repo — you'll need the project source to reconstruct the class before loading the state dict.

How to load

from huggingface_hub import hf_hub_download
import torch

repo_id = "Mercity/figure-skating-transformer-bilstm"
ckpt_path = hf_hub_download(repo_id, "model_transformer_bilstm_fine.pt")
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)

# reconstruct with the project's SkatingTransformerBiLSTMClassifier (model_transformer_bilstm.py):
# model = SkatingTransformerBiLSTMClassifier(ckpt["in_features"], ckpt["num_classes"])
# model.load_state_dict(ckpt["state_dict"])
# model.eval()

# ckpt also carries the standardization stats used at train time and the taxonomy:
mu, sd = ckpt["feature_mean"], ckpt["feature_std"]     # normalize inputs with (x - mu) / sd
taxonomy = ckpt["taxonomy"]                            # {0: "3Toeloop", 1: "3Loop", ...}

Swap model_transformer_bilstm_fine.pt for model_transformer_bilstm_coarse.pt to load the 11-class variant.

Config

Stage 1 (encoder)	Linear projection to d_model=384 + sinusoidal positional encoding + nn.TransformerEncoder, 3 layers, nhead=8, dim_feedforward=1536, dropout=0.1, GELU — no conv backbone, raw 94-dim features projected directly
Stage 2 (decoder)	2-layer BiLSTM, hidden=192/direction (→384), dropout 0.2, sweeping the encoder output
Head	Dense(384→1024,p0.5) → Dense(1024→512,p0.4) → Dense(512→256,p0.3) → LayerNorm(256) → Linear(256→num_classes)
Optimizer	Adam, lr=5e-4, weight_decay=1e-4
Batch size	64
Grad clip	1.0
Class weight cap	5.0
Early stop	patience 20 on val macro-F1, 100-epoch cap
Seed	42
Params	8.20M (both)

Data

Trained on skeleton-extracted figure-skating clips (YOLO11n-pose, COCO-17 joints, imgsz=384) from an internal Mercity dataset. Stratified train/val/test split, seed 42. Not included in this repo (private internal dataset).

Results (test split)

Fine-grained (28-class)

Early-stopped epoch 64 (best val macro-F1 0.449) — notably unstable val loss/F1 trajectory during training.

Overall: accuracy 0.7297 · precision (macro/weighted) 0.3764 / 0.7598 · recall 0.4114 / 0.7297 · F1 0.3615 / 0.7341

class	precision	recall	f1	support
3Toeloop	0.409	0.237	0.300	38
3Loop	0.833	0.476	0.606	21
2Axel	0.532	0.767	0.629	43
CamelSpin	0.821	0.880	0.849	125
SitSpin	0.969	0.932	0.950	132
UprightSpin	0.974	0.694	0.811	108
2Salchow	0.300	0.600	0.400	5
2Toeloop	0.526	0.625	0.571	32
3Salchow	0.333	0.286	0.308	21
3Axel	0.647	0.733	0.688	15
3Flip	0.302	0.528	0.384	36
3Lutz	0.312	0.122	0.175	41
NoBasic	0.750	0.484	0.588	31
2Lutz	0.333	0.286	0.308	7
4Salchow	0.500	0.333	0.400	3
4Flip	0.000	0.000	0.000	1
4Toeloop	0.000	0.000	0.000	6
4Lutz	0.000	0.000	0.000	1
4Loop	0.000	0.000	0.000	1
2Flip	0.125	0.250	0.167	8
2Loop	0.600	0.375	0.462	8
1Axel	0.000	0.000	0.000	3
1Loop	0.250	1.000	0.400	1
1Salchow	0.000	0.000	0.000	1
1Toeloop	0.000	0.000	0.000	1
1Flip	0.000	0.000	0.000	2
1Lutz	0.125	1.000	0.222	1
Sequence	0.897	0.912	0.905	307
macro avg	0.376	0.411	0.361	999
weighted avg	0.760	0.730	0.734	999

Coarse (11-class)

Early-stopped epoch 34 (best val macro-F1 0.793)

Overall: accuracy 0.8679 · precision (macro/weighted) 0.7996 / 0.8741 · recall 0.8081 / 0.8679 · F1 0.7995 / 0.8683

class	precision	recall	f1	support
Axel	0.902	0.902	0.902	61
Toeloop	0.780	0.597	0.676	77
Salchow	0.714	0.667	0.690	30
Loop	0.771	0.871	0.818	31
Flip	0.610	0.532	0.568	47
Lutz	0.534	0.780	0.634	50
CamelSpin	1.000	0.928	0.963	125
SitSpin	0.933	0.955	0.944	132
UprightSpin	0.877	0.861	0.869	108
Sequence	0.951	0.958	0.955	307
NoBasic	0.722	0.839	0.776	31
macro avg	0.800	0.808	0.800	999
weighted avg	0.874	0.868	0.868	999

Notable failure modes (from confusion matrix analysis, coarse label space)

• Weakest architecture of the family, especially fine-grained — 73.0% acc / 0.361 macro-F1 is the worst fine-grained result measured across every architecture tried; early-stops fastest on coarse (epoch 34) too.
• Flip → Lutz collapse is severe: 18/47 Flip test samples (38%) predicted as Lutz — the single worst individual confusion measured across every architecture and label space tested.
• Unique failure mode not seen elsewhere: UprightSpin → Sequence leakage (8/108, 7.4%).
• No conv backbone to establish local temporal structure before self-attention appears to hurt more than it helps at this data scale.

License

Internal/proprietary — not licensed for external use. Contact the Mercity team for access terms.