LCRec: adapting a causal LLM with collaborative semantic IDs¶
| Field | Current repository |
|---|---|
| Paper | Adapting Large Language Models by Integrating Collaborative Semantics, ICDE 2024 |
| Task | next-item generative retrieval plus text/SID alignment tasks |
| Item representation | main config: 5×256; paper-alignment config: 4×256 plus USM |
| Backbone | Hugging Face causal LM; current Gin selects local Qwen3-1.7B |
| Training | response-only causal LM, full-parameter by default |
| Decoding | position-specific codebook masks with KV-cached beam search |
| Headline | item Recall/NDCG after training-only SID resolution and deduplication |
| Checkpoint | Hugging Face model/tokenizer directory; no optimizer/scheduler/epoch |
The wrapper uses AutoModelForCausalLM and AutoTokenizer; it is not hard-coded to Qwen. Python's Qwen2.5 fallback and old docstrings are not the current experiment setting. The repository's reference/LC-Rec/ is reference code, while this implementation is an independent GenRec adaptation.
Two stages and vocabulary¶
Stage 1 trains LCRec's dedicated RQ-VAE. The legacy main recipe uses input 768, latent 64, hidden 512/256/128 and five 256-code layers. rqvae_paper_recipe.gin follows the reference indexer geometry with latent 32, hidden 2048/1024/512/256/128/64 and four 256-code layers. It also matches the released run.sh behavior: first-batch K-Means on 1,024 items, Xavier-normal MLP weights, and mean-reduced reconstruction/per-layer VQ losses. Stage 2 trains the causal LM on user sequences. The catalog order, Sentence-T5-XL embedding space, and RQ-VAE geometry must remain identical.
For each configured layer, LCRec.add_codebook_tokens() adds position-specific special tokens such as <C0_0>. A complete SID contains five tokens in the legacy recipe or four in the paper recipe. Model input/output embeddings are resized, so the saved tokenizer is a required checkpoint artifact.
uniform_semantic_mapping=True performs the official-style iterative Sinkhorn reassignment on collision groups in the final codebook. True duplicates or a fixed point may still collide, and the adapter reports the resulting unique count instead of silently treating tuples as item identities.
Tasks and objective¶
AmazonLCRecDataset implements six tasks:
| Task | Mapping | Enabled in current main config? |
|---|---|---|
seqrec |
history SIDs to next SID | Yes |
item2index |
item text to SID | Yes |
index2item |
SID to item text | Yes |
fusionseqrec |
history SIDs to item text | No |
itemsearch |
synthetic query/history to SID | No |
preferenceobtain |
history to category-preference text | No |
The optional tasks use local metadata heuristics and are not equivalent to the paper's data generation simply because all six names exist. The legacy main config enables three tasks; lcrec_paper_recipe.gin enables all six. Validation/test datasets generate seqrec only.
The collator concatenates prompt + response + EOS and sets prompt/padding labels to -100:
train.max_length=512 controls actual LM truncation. Dataset max_text_len is currently stored but does not truncate LCRec metadata.
Generation and evaluation¶
The class has no LCRec.generate() method. Its production trainer uses:
sem_ids, scores = model.generate_constrained_beam(
input_ids=input_ids,
attention_mask=attention_mask,
position_mask=helper.position_mask,
code_map=helper.code_map,
num_codebooks=5,
beam_width=10,
topk=10,
)
generate_topk is an older per-example alternative. ConstrainedDecodingHelper allows only <Cj_*> at code position j and EOS after the final code. This guarantees positional token type, not membership in the catalog; it is not a catalog PrefixTrie.
The item mapper resolves each SID using only training-prefix popularity, breaks ties by smaller item ID, drops absent tuples, and removes duplicate items. Item-level Recall/NDCG@⅕/10 drives validation-only model selection. The trainer saves checkpoint_best, reloads that state on every rank, then evaluates test exactly once. Pre-resolution SID tuple metrics remain diagnostics under results.json.extra.
| Layer | Source | Contract |
|---|---|---|
| wrapper | lcrec.py |
HF forward and custom beam methods |
| data | amazon_lcrec.py |
task/prompt/response/target dictionaries |
| collator/helper/trainer | lcrec_trainer.py |
response mask, code-position mask, Accelerate loop |
| resolver | metrics.py |
SID candidates to deduplicated items |
Run and artifacts¶
python -m genrec.trainers.rqvae_trainer \
config/lcrec/amazon/rqvae.gin --split beauty
python -m genrec.trainers.lcrec_trainer \
config/lcrec/amazon/lcrec.gin --split beauty
# Paper-structure adaptation: 4-code indexer, USM, six tasks, history 20,
# beam 20, and Qwen3-4B as the locally available LLaMA-7B substitute.
python -m genrec.trainers.rqvae_trainer \
config/lcrec/amazon/rqvae_paper_recipe.gin --split beauty
python -m genrec.trainers.lcrec_trainer \
config/lcrec/amazon/lcrec_paper_recipe.gin --split beauty
Stage 2 currently reads out/lcrec/amazon/<split>/rqvae/checkpoint_epoch_1999.pt. The RQ-VAE run continues past that checkpoint, so an alternative epoch requires an explicit path override.
There is no lcrec_debug.gin. For a smoke run after real model/tokenizer assets exist, override epochs/sample limits on the standard config.
The current main config uses Qwen3-1.7B, 10 epochs, batch 16 with accumulation 4, learning rate 2e-5, 1% warmup, 50 history items, 512 LM tokens, full-parameter bf16 training, beam 10, and per-epoch save/eval.
Output contains checkpoint_epoch_<zero-based epoch>/, checkpoint_best/, checkpoint_final/, and results.json. checkpoint_best is selected only on validation Recall@10; results.json contains its one-shot test metrics. Final still means last. eval_only=True currently evaluates validation only and does not write a new results artifact. LoRA without PEFT only warns and falls back, so verify trainable-parameter counts.
Reproduction boundary¶
The October matrix recipe uses four codebooks, USM, six tasks, history 20, four epochs, micro-batch 4 × accumulation 32 (effective batch 128), learning rate 5e-5, and beam 20. It substitutes Sentence-T5-XL and Qwen3-4B and uses Amazon-2014 Beauty/Sports/Toys, so it is an adaptation rather than the original paper setup. All 18 item-level cells are complete as of 2026-10-22: LOO R@10 is 0.0729 ± 0.0004 / 0.0399 ± 0.0005 / 0.0705 ± 0.0041. Temporal rows and provenance are in BENCHMARKS.md. Evaluation prompt templates are still sampled rather than frozen across passes. Historical SID-tuple results remain excluded.
See RQ-VAE, trainer API, and evaluation architecture.