Adding a custom dataset¶
The repository has no BaseRecommenderDataset, dataset factory, processor
registry, or dataclass configuration layer. Extend the closest concrete adapter,
reuse the shared split protocol and collate contract, and bind the new class to a
trainer through Gin when that trainer exposes a dataset argument.
Choose the closest integration point¶
| Goal | Best reference |
|---|---|
| One embedding per item / tokenizer input | AmazonItemDataset |
| Item-ID next-item baseline | AmazonSASRecDataset |
| Timestamp-aware HSTU | AmazonHSTUDataset |
| SID encoder-decoder | AmazonSeqDataset |
| LLM prompt, SFT, or RL | AmazonOneRecSFTDataset, AmazonOneRecRLDataset |
| Parallel OPQ prediction | AmazonRPGDataset |
| Dense item-to-item retrieval | AmazonNoteLLMDataset |
| Lexical IDs and multiple passages | AmazonGRAMDataset |
Do not hide genuine differences in ID bases, padding, text inputs, and sample augmentation behind a new universal abstraction.
Minimal raw schema¶
Normalize the source into two logical tables:
interactions: user_key, item_key, timestamp[, rating/action]
items: item_key, title[, description, brand, category, ...]
Required invariants:
(user, item, timestamp)parses deterministically.- Equal timestamps have a stable tie-break.
- The item mapping is identical across splits and the embedding catalog.
- Every validation/test target maps to the catalog.
- Source revision, filtering, and checksums are recorded.
Item-ID sequence skeleton¶
Reuse select_protocol_sequences rather than implementing another leave-one-out
split. A SASRec-like adapter reserves 0 for padding and maps real items from 1:
from collections import defaultdict
from torch.utils.data import Dataset
from genrec.data.evaluation_protocol import select_protocol_sequences
class MySASRecDataset(Dataset):
def __init__(
self,
root,
split,
train_test_split="train",
max_seq_len=50,
min_seq_len=5,
evaluation_protocol="leave_one_out",
temporal_train_ratio=0.8,
temporal_valid_ratio=0.1,
):
rows = load_my_interactions(root, split)
self.item_id_mapping = {}
user_sequences = defaultdict(list)
for row in rows:
if row.item_key not in self.item_id_mapping:
self.item_id_mapping[row.item_key] = len(self.item_id_mapping) + 1
item_id = self.item_id_mapping[row.item_key]
user_sequences[row.user_key].append((row.timestamp, item_id))
selected, self.protocol_info = select_protocol_sequences(
user_sequences,
train_test_split,
protocol=evaluation_protocol,
min_seq_len=min_seq_len,
temporal_train_ratio=temporal_train_ratio,
temporal_valid_ratio=temporal_valid_ratio,
)
self.evaluation_protocol = evaluation_protocol
self.num_items = len(self.item_id_mapping)
self.sequences = [[item for _, item in seq] for _, seq in selected]
self.samples = build_samples_like_amazon_sasrec(
self.sequences, train_test_split, max_seq_len
)
def __len__(self):
return len(self.samples)
def __getitem__(self, index):
return self.samples[index] # {"history": [...], "target": int}
If the returned schema is identical, reuse sasrec_collate_fn and
sasrec_eval_collate_fn from genrec.data.amazon_sasrec.
Trainer binding¶
Trainers such as TIGER, OneRec, and GRAM expose the dataset class as a Gin parameter:
import my_package.my_dataset
train.dataset = @MyTigerDataset
MyTigerDataset.root = "dataset/my_data"
MyTigerDataset.split = "category_a"
The current SASRec and HSTU trainers instantiate AmazonSASRecDataset and
AmazonHSTUDataset inside train. Supporting another adapter therefore
requires adding a configurable dataset parameter or a structurally equivalent
trainer. Update train, validation, and test construction together.
Item-tokenizer adapters¶
RQ-VAE and RQ-KMeans expect a fixed-width float vector from each item. A custom adapter should expose:
Persist the item_key <-> row_index mapping with the array. An anonymous
[N, D] matrix cannot be safely joined back to user sequences.
Extra requirements for generative adapters¶
Keep the real target item even when the model predicts a semantic ID:
from genrec.data.schemas import SeqData
return SeqData(
user_id=user_id,
item_ids=flat_history_codes,
target_ids=target_sid,
target_item=target_item_id,
)
At evaluation time, construct SemanticIdItemMapper from training
interactions only, resolve each SID to one item, remove duplicate items, and
score against target_item. Validation/test target popularity must not enter
the resolver.
Tests and provenance¶
At minimum, test:
- identical item mappings across splits;
- padding and ID-base conventions;
- leave-one-out or global-temporal target boundaries;
- catalog coverage of every target;
- collate shapes, masks, and padding direction;
- embedding/SID row alignment;
- full-catalog item-level evaluation.
Useful references are tests/test_evaluation_protocol.py,
tests/test_honest_metrics.py, and tests/test_notellm_full_catalog.py.
Store a manifest with the source URL/revision, raw hashes, filtering rule, mapping rule, user/item/interaction counts, timestamp unit, and preparation time. Commit preparation code, schemas, checksums, and a small synthetic fixture; do not commit restricted raw data.
Before considering the adapter complete, document its Gin binding, command, output directory, data license, and supported-model status.