HSTU¶
HSTU uses non-normalized SiLU attention, an update gate, and relative attention bias for sequential recommendation. In GenRec it is a strong discriminative counterpart to SASRec—not a semantic-ID generator.
| Field | Current repository |
|---|---|
| Paper | Actions Speak Louder than Words, ICML 2024 |
| Official code | facebookresearch/generative-recommenders |
| Representation/output | unique item ID / full-catalog logits |
| Data | Amazon and MovieLens-1M adapters |
| Status | small-scale dense-tensor HSTU-inspired baseline, not Meta's production stack |
How it differs from SASRec¶
| Component | SASRec | HSTU here |
|---|---|---|
| attention activation | normalized softmax | non-normalized SiLU |
| projections | Q, K, V | U, V, Q, K |
| aggregation | softmax(QK) @ V |
SiLU(QK + RAB) / max_seq_len @ V |
| gate | none | Norm(A) * U, optional concatenated gate |
| temporal signal | position embedding | relative position and optional time-difference bias |
For input X, one layer computes
The repository adds a SiLU FFN residual after this transformation. With concat_ua=True, the projection consumes [U,A,U*A]. Scaling uses the configured maximum sequence length, not the longest sequence in the current batch.
RelativePositionBias uses exact small-distance and logarithmic large-distance buckets. TemporalBias buckets absolute Unix-time differences logarithmically. Timestamp padding is 0 but is also key-masked.
Objectives and tensor contract¶
logits, loss = model(
input_ids, # [B,L]
timestamps=None, # [B,L]
targets=None, # [B,L]
sampled_softmax=False,
...,
)
Full CE returns [B,L,V+1] logits. During sampled-softmax training logits are None. The model's score_l2_norm and score_temperature keep training and evaluation scoring consistent.
The sampled-softmax config creates a positive-plus-negatives classification problem. Meta's sampling_strategy="local" means the item embedding table is local rather than sharded: every positive independently draws 128 uniform catalog IDs with replacement. It is not in-batch sampling. Accidental positives are masked, query/items are L2-normalized, and temperature is 0.05. in_batch is a separate supported sampler; global preserves GenRec's older shared, deduplicated negative set and must not be mixed with the corrected local results.
Data and evaluation¶
The Amazon adapter follows the same leave-one-out structure as SASRec and retains timestamps. dataset_type="ml1m" selects MovieLens1MHSTUDataset. Evaluation scores the full catalog at the last valid position, excludes padding ID 0, and does not mask history items.
| Layer | Source | Symbols |
|---|---|---|
| model | hstu.py |
HSTU, HSTULayer, position/time bias |
| Amazon data | amazon_hstu.py |
dataset and collators |
| MovieLens | ml1m_hstu.py |
MovieLens1MHSTUDataset |
| training | hstu_trainer.py |
train, evaluate |
| configs | config/hstu/ |
CE and sampled-softmax variants |
Run¶
# full-vocabulary CE with temporal bias
python genrec/trainers/hstu_trainer.py config/hstu/amazon.gin --split beauty
# Meta local-table sampled softmax
python genrec/trainers/hstu_trainer.py config/hstu/amazon_ss.gin --split beauty
# temporal-bias ablation
python genrec/trainers/hstu_trainer.py config/hstu/amazon.gin \
--split beauty --gin 'train.use_temporal_bias=False'
# separate global-temporal experiment
python genrec/trainers/hstu_trainer.py config/hstu/amazon.gin \
--split beauty \
--gin 'AmazonHSTUDataset.evaluation_protocol="global_temporal"' \
--gin 'train.save_dir_root="out/hstu/amazon/beauty/global_temporal/s42"'
Disabling temporal bias does not disable relative position bias; bind train.use_position_bias=False as well when that is the intended ablation.
Reproduction boundary¶
Meta's work targets long jagged sequences, specialized kernels, many action/features, and industrial scale. This implementation uses dense [B,H,L,L] attention, item ID plus timestamp, small academic configurations, and an extra FFN. It is suitable for testing the SiLU-attention/U-gate/temporal-RAB ideas under a common benchmark, not for validating production throughput or scaling claims.
See trainer parameters, dataset schemas, and implementation status.