Appendix B — Datasets & Benchmarks
Ann reads two papers. Both list LightGCN (a standard model that learns from the graph of who interacted with what) on a dataset called “Amazon-Book”; one reports Recall@20 (the share of a user’s held-out items found in the top 20 recommendations) of 0.0411 (He et al., 2020), the other 0.1411 (Ren et al., 2024). Neither is a typing error: the name does not say which file was used, or how it was cut into a training part and a test part. This appendix teaches the question to ask before any number is read: which data, and which split?
Purpose: after this appendix you can read “we tested on Amazon-Book” and say which file that is, how it was split, and whether its number may stand beside another paper’s. Assumes: the metrics of the Evaluation Metrics chapter, Recall@K and the normalized discounted cumulative gain (NDCG@K), and arithmetic with fractions; most other terms are defined where they first appear, and §8 repeats the main ones. This is a lookup appendix, not a chapter to read in order: §1 to §3 give the datasets and the split rules, §4 maps the field, §5 and §7 are cautions, and §6 is how the field gets its data today.
1. The datasets you will actually meet
Bob has a new recommender and asks the obvious question: which dataset do I download? Every paper answers with a name, and the name hides a choice, because one name can stand for several files (§2, §4). Academic recommendation runs on a surprisingly small set of public logs (a log is a file with one row per user action). Group them by family: the family suggests the usual task, and so the loss (the training objective) and the metric. Star ratings suggest predicting the number itself, with a squared-error loss and the root-mean-squared error (RMSE) of Evaluation Metrics; clicks suggest ranking, with a ranking loss and Recall@K. The family only suggests: as §4 shows, the protocol is ultimately fixed by the research community, not by the dataset.
- Rating prediction (explicit stars): MovieLens, Netflix, Book-Crossing.
- Implicit top-\(N\) / graph collaborative filtering (CF) (clicks, check-ins, buys; top-\(N\) means recommending each user the \(N\) best items): Gowalla, Yelp, Amazon, Steam, LastFM (music listening logs). (Yelp and Amazon ship explicit stars, but the graph-CF trio binarizes them: every observed pair becomes a 1, and the neural graph collaborative filtering (NGCF) paper states no rating threshold. Some later papers threshold first, see §4.)
- Click-through rate (CTR) / news (a binary click, often with rich side-features, i.e. extra columns about users or items): Criteo (a display-advertising click log), MIND (the Microsoft News Dataset), Taobao (Alibaba’s e-commerce behavior log: click / add-to-cart / favorite / buy, released through the Tianchi platform). The row label is an impression: one showing of an item, or of a list of items, to a user, together with the click or no-click that followed. If a page shows Bob five films and he clicks the second, that is five impressions (item by item): one click and four no-clicks. The question the model answers is was this impression clicked?
- Sequential / next-item (the same logs, but read as time-ordered sequences per user): MovieLens-1M, Amazon category subsets, Steam. This is a way of using a dataset, not a separate family of data; §4 shows it has its own protocol.
For each, the four properties that actually change your modeling choice are: feedback (explicit ratings vs. implicit 0/1), timestamps? (needed for a time-based split, §3), side-features? (they enable content models and factorization machines (FM), which score items from their features, and cold-start models, which must serve a new user or item with no history, such as a user who joined today; §3 shows a cold item), and license.
| Dataset | Domain | Scale (approx.) | Feedback | Time | Side-features | Status & license |
|---|---|---|---|---|---|---|
| MovieLens (100K–32M) | movies | 100K–32M ratings | explicit 0.5–5 (whole stars 1–5 in 100K/1M only) | yes | genres, tags, tag-genome; demographics (100K/1M only) | Active — GroupLens (the University of Minnesota lab that publishes MovieLens); research, non-commercial |
| Netflix Prize | movies | 100M ratings | explicit 1–5 | yes | title/year only | Withdrawn (de-anonymization); copies circulate — cite as history (§7) |
| Amazon Reviews | e-commerce | 233M (’18) → 571M (’23) | explicit 1–5 + text | yes | metadata, images; per category | Active — research; on HuggingFace (a public dataset/model hosting hub) |
| Yelp Open Dataset | local business | ~7M reviews | explicit 1–5 + text | yes | attributes, social graph, check-ins | Restricted (rolling: the owner replaces the files from time to time) — academic only, revocable |
| Gowalla | check-ins | 6.4M raw check-ins; ~1M in the 10-core benchmark snapshot (every user and item has \(\ge 10\) interactions; §3) | implicit | yes | geo, friendship graph | Active — SNAP (Stanford’s network-dataset collection), public; no stated terms (§7) |
| Steam | games | ~7.8M reviews | implicit + reviews | yes | genres, price, playtime | Active — research |
| LastFM (LFM, short for Last.fm: the releases LFM-1b and LFM-2b) | music | 1B–2B plays | implicit | yes | demographics, genre | Withdrawn — both no longer distributed; LFM-1b’s page cites license issues (§7) |
| MIND (the Microsoft News Dataset) | news | 1M users, >15M impr. | implicit click | yes | title, abstract, body, entities | Restricted — Microsoft Research license, research only |
| Criteo | display ads | ~46M rows | implicit click | — | 13 numeric + 26 categorical (hashed) | Active — public (Kaggle, a data-science competition site) |
| Book-Crossing | books | 1.15M ratings | explicit 1–10 + implicit | no | age, location, metadata | Active — public (GroupLens); no stated terms (§7) |
Reading Table B.1:
- Status. Active means downloadable today; Restricted, gated (academic-only, revocable, or by signed agreement); Superseded, replaced by a newer release of the same family; Frozen, a benchmark snapshot that now lives in a code repository rather than with the vendor; Withdrawn, officially pulled (keep it for history, do not build on it).
- Time. yes means the rows carry usable timestamps, no that there is no time information, and — that the rows are delivered in time order but carry no timestamp you can read (the owners’ description of Criteo, not re-checked here).
- LastFM. LFM-1b holds more than one billion play events and LFM-2b more than two billion; both are no longer distributed (§7).
Which one do I pick? Rules of thumb from practice, not tested here. First test of a rating or next-item idea: MovieLens-1M, small and frozen, with timestamps. Comparing with published graph-CF numbers: the trio of §2, on its frozen files. Click prediction: Criteo; news: MIND. New work on reviews: Amazon Reviews 2023. Before any of them, read the Status cell: a Withdrawn set is history, not a benchmark.
Inside the families: which variant carries what. Several entries above are not one dataset but a family of releases, and a beginner can lose hours to the wrong one (MovieLens alone has the seven below, plus a 32M release, which the note under the table covers). The tables below show, per family, what data each variant actually gives you.
| MovieLens variant | Ratings | Rating range | Tags | Tag-genome | Demographics | Best for |
|---|---|---|---|---|---|---|
| 100K (1998) | 100K | 1–5 | — | — | yes | smallest frozen set; demographics |
| 1M (2003) | 1M | 1–5 | — | — | yes | the classic mid-scale CF benchmark |
| 10M (2009) | 10M | 0.5–5 | yes | — | — | first release with free-text tags |
| 20M (2015) | 20M | 0.5–5 | yes | yes | — | the classic tag-genome benchmark |
| 25M (2019) | 25M | 0.5–5 | yes | yes | — | larger frozen genome set |
| latest-small | ~100K | 0.5–5 | yes | — | — | quick dev / teaching — not for results |
| latest-full | ~33M | 0.5–5 | yes | yes | — | dev only; changes over time |
Ratings are whole-star 1–5 in 100K/1M, half-star 0.5–5 from 10M on. User \(\times\) item counts grow with size: 943 \(\times\) 1.7K at 100K, 6,040 \(\times\) 3,706 rated movies at 1M, up to 162,541 \(\times\) 62,423 at 25M (the 25M README’s counts; the GroupLens page rounds them to 162,000 and 62,000). GroupLens also ships a 32M frozen release (collected 10/2023, released 5/2024; 200,948 users \(\times\) 87,585 movies; its page does not state whether it carries the tag genome, so it has no row here); that page lists 32M under “recommended for new research” and 25M under “older datasets”.
| Amazon Reviews version | Curators | Reviews | Categories | Adds over prior | Status |
|---|---|---|---|---|---|
| 2014 | McAuley et al. | 142.8M raw (83.7M once duplicate items are removed) | 24 | ratings, text, descriptions, price, brand, sales rank, image features, also-bought graph | Superseded |
| 2018 | Ni, Li & McAuley | 233.1M | 29 | \(+\) 90M reviews (to Oct 2018), 5 categories, transaction metadata (color / size / package), buyer photos, bullet-point descriptions, technical-details tables | Active |
| 2023 | Hou et al. (McAuley Lab) | 571.5M | 33 | \(+\) reviews through Sep 2023 (54.5M users, 48.2M items); richer item metadata; HuggingFace-hosted | Active — current |
Three cautions apply to the table: what 2018 added comes first, then the Amazon-Book snapshot, then the way the name fails to identify the file.
- What 2018 added. Price, brand and image features were already in the 2014 release, and the 2018 page says so: it keeps them “as in the previous version” and adds volume plus finer metadata.
- Amazon-Book is a snapshot. The graph-CF “Amazon-Book” benchmark is a 10-core subset of Amazon’s Books reviews (52,643 users, 91,599 items, 2,984,108 interactions) frozen in the code repositories of neural graph collaborative filtering (NGCF) and LightGCN, the two standard graph-CF models (§2). NGCF names no release year and cites He & McAuley (2016) for the reviews; the 2014 release is the likely source, an inference and not NGCF’s statement. It is a benchmark artifact, not the live category, which is far larger.
- The name does not identify the file. NGCF writes Amazon-book in its experiments text but Amazon-Book in its statistics table, and papers using exactly the name often mean different files: §4 sets three beside this one, the large-language-model recommender RLMRec’s 11,000 users, the knowledge-graph attention network (KGAT)’s 70,679 and the contrastive graph model LightGCL’s 78,578, against 52,643. Always check the user and item counts.
Three more cautions, about the families in the table below:
- Frozen artifacts. Three rows of the table below are evaluation artifacts, not live datasets: the graph-CF trio’s Yelp2018, Amazon-Book and Gowalla snapshots (§2). They live only in the benchmark repositories (Gowalla and Amazon-Book in NGCF’s and LightGCN’s, Yelp2018 in LightGCN’s alone), so do not try to re-download them from the vendor.
- Near-identical names. The frozen 52.6K-user Amazon-Book artifact and the live Amazon Reviews Books category, orders of magnitude larger, are two different objects; the spelling will not tell them apart, so check the user and item counts.
- MIND-small is a 50K-user sample for quick iteration (no test split). Book-Crossing’s implicit feedback is coded 0: an interaction without a grade, not a low score.
| Other family · variant | Scale | Carries | Status |
|---|---|---|---|
| Yelp Open Dataset (rolling, as above) | ~7M reviews · 150K businesses · 11 metropolitan areas (Yelp’s own page publishes no user count) | text + stars, business attributes, friend graph, check-ins, tips | Restricted — academic only, revocable |
| Yelp2018 (snapshot) | 31,668 users · 38,048 items · 1,561,406 (10-core) | interactions only | Frozen — in the LightGCN repositories; the NGCF repository ships only Gowalla and Amazon-Book (this revised copy is LightGCN’s and the latest NGCF arXiv version’s, not NGCF’s first one, §4) |
| Amazon-Book (graph-CF snapshot) | 52,643 users · 91,599 items · 2,984,108 (10-core, Amazon Books) | interactions only | Frozen — in the NGCF / LightGCN repositories |
| Gowalla (graph-CF snapshot) | 29,858 users · 40,981 items · 1,027,370 (10-core) | interactions only | Frozen — in the NGCF / LightGCN repositories |
| MIND-large (news) | 1M users · ~160K news · >15M impressions | title, abstract, body, category, entities | Restricted — Microsoft Research license |
| MIND-small | 50K sampled users (train + dev, i.e. validation) | a subset of MIND-large | Restricted — same license |
| Book-Crossing | 1.15M ratings · 279K users · 271K books | explicit 1–10 + implicit (0); age/location | Active — GroupLens |
| Goodreads (from UCSD, the University of California, San Diego) | ~229M interactions · 876K users · 2.36M books | reviews, shelves (a user’s to-read / currently-reading / read lists) | Restricted — academic |
| Amazon Reviews – Books category (live; not the Amazon-Book snapshot above) | = the whole “Books” category of the release you pick | (see the Amazon table) | Active |
In one line: pick the family that matches your task, then the variant, and never trust a dataset name without its counts.
2. The standard graph-CF trio: Gowalla, Yelp2018, Amazon-Book
Cara opens three papers on recommenders built on the user–item graph (graph-CF, defined just below) and finds the same three dataset names in all of them. Is that laziness or design? The graph chapters (From Graphs to LightGCN, Self-Supervised & Contrastive Learning), and most graph collaborative-filtering (graph-CF) papers, report on the same three datasets: Gowalla, Yelp2018, and Amazon-Book. This is not a coincidence, and knowing why saves a beginner real confusion:
They are implicit / one-class — only positive events are recorded, and a missing user–item pair means “unknown”, never “disliked”. Graph CF models the user–item interaction graph; explicit star ratings are discarded (NGCF states no rating threshold; §4 shows later papers that filter low ratings first) and every observed pair is a binary edge. All three are pure interaction logs, not rating-prediction tasks.
(One-class data is exactly why §4 must argue about negative sampling, scoring the held-out item against a few randomly drawn items the user never touched and treating them as non-relevant, and about full-catalog ranking, scoring it against every item the user has not touched: there are no recorded negatives to rank against, so the protocol has to invent them.)
They span three domains (check-ins, local business, books), but, as the arithmetic below shows, not a range of densities. Density, interactions divided by users \(\times\) items, works out as \[ \underbrace{\tfrac{1{,}027{,}370}{29{,}858\times40{,}981}}_{\text{Gowalla}}\!\approx 0.084\%,\quad \underbrace{\tfrac{1{,}561{,}406}{31{,}668\times38{,}048}}_{\text{Yelp2018}}\!\approx 0.130\%,\quad \underbrace{\tfrac{2{,}984{,}108}{52{,}643\times91{,}599}}_{\text{Amazon-Book}}\!\approx 0.062\%. \]
So Amazon-Book is the sparsest and Yelp2018 the densest, and the densest is only about \(2.1\times\) the sparsest (\(0.130/0.062\)). They do not span density regimes: all three have \(99.87\%\) sparsity (the unfilled share, \(1-\) density) or more.
Whatever else the trio varies, it does not vary this, and the trio’s density is not the density of recommender data in general: MovieLens-1M, for one, is about \(4.5\%\) dense (\(1{,}000{,}209/(6{,}040\times3{,}706)\)).
The preprocessing is frozen, inside one lineage. NGCF (the paper LightGCN built on), LightGCN and the papers that reuse their files share one 10-core-filtered, fixed train/test split, with one exception (Yelp2018 is a revised file in LightGCN and in NGCF’s latest arXiv version, but not in NGCF’s first one, §4). A main reason the trio dominates is probably this shared split and not any statistical superiority: it lets those papers quote each other’s numbers as directly comparable (a claim from practice, not tested here). The lineage has limits, and §4 shows where it breaks: KGAT, SimGCL (a contrastive graph-CF model), LightGCL and RLMRec keep the names but not all of the files.
Why this matters (and its catch). Comparability is the whole point, but the frozen split is itself a known hazard (§3). NGCF’s own paper says it: “we randomly select 80% of historical interactions of each user to constitute the training set, and treat the remaining as the test set.” That is a random per-user split, not a time-based one (§4) — so every model trained on it inherits the same future-leak. Comparable is not the same as correct.
In one line: the trio is the shared yardstick of one lineage of graph-CF papers: comparable on its frozen files, but cut at random, so not leak-free.
3. Splitting without leaking the future (the keystone)
Ann trains a model on her shop’s click log, tests it on the same log, and gets a wonderful score. A colleague asks: did any test click happen before a training click? If so, the model has seen the future. A model is graded by holding out some interactions as a test set, and how you choose them decides whether the score is honest. Three protocols dominate, and telling them apart is the single most important idea in this appendix. This book uses these three names and no others — “temporal split” on its own is ambiguous in the literature and is never used bare here, because two of the three are temporal.
- Random split — hold out a random fraction of the interactions (papers differ: the fraction is taken per user or globally); the variant called \(k\)-fold cuts the data into \(k\) parts, tests on each part in turn, trains on the rest and averages. Simple, and wrong for any data with time: it can put a user’s later interactions in train and an earlier one in test, so the model trains on the future to predict the past. The score can inflate (the finding of Ji et al., 2023, §9, not tested here; the box after the worked example shows the mechanism).
- Leave-one-out (LOO) — for each user, sort by time and hold out one interaction: the chronologically last one unless the paper says otherwise, and often the second-last as a validation item (a held-out check used to tune settings and to decide when to stop training, kept separate from the final test). Some papers write “leave-one-out” for a random held-out interaction — check which one a paper means. Holding out the last one grades the model the way it is used: predict the next thing from everything before it. It is the standard for next-item models, which compare themselves with published leave-one-out numbers (Sequential & Session-Based Recommendation §6), though not the strictest rule. It is leak-free within a user — but not across them: one user’s held-out interaction may predate another user’s training data, so the model can learn from the future of the population.
- Global temporal split — one cut-off timestamp for everybody: train on everything before it, test on everything after. This is the strictest of the three and the only one that is leak-free across users; it is the one Evaluation Metrics §11a calls the most honest, and Implementation Choices §5 records that the choice can reorder a leaderboard (Ji et al., 2023). Its price is shown below.
Both of the last two are temporal; only the global one is leak-free across users. That is why this appendix never writes bare “temporal” for a per-user hold-out.
The rule this book uses. Evaluation Metrics §11a says: prefer the global temporal split whenever timestamps exist, and leave-one-out is still the standard split for next-item (sequential) models. This appendix adds three things of its own, which that section does not say: the price of the global cut (lost test coverage, shown below); leave-one-out also when one global cut would leave most users untestable; and never a random split on data with time, except to compare with an earlier paper, and then label the number as leaky.
Almost always, a k-core filter runs first: repeatedly drop users and items with fewer than \(k\) interactions until none remain. There is no universal \(k\): the graph-CF papers (NGCF, LightGCN and their successors) use \(k=10\), while SASRec (a Transformer next-item model, §4) discards “users and items with fewer than 5 related actions” and BERT4Rec (its bidirectional relative) keeps “users with at least five feedbacks” — so \(k=5\) in the sequential literature. We use \(k=2\) to keep the example tiny. This removes most of the noisiest cold cases from the data — though, as the example below shows, not necessarily from the training set.
Worked example — k-core then leave-one-out, by hand. Four users over five items, with timestamps \(t\), as the table below lists them.
| User | Interactions (item@time) |
|---|---|
| U1 | A@1, B@3, C@4 |
| U2 | A@1, B@2 |
| U3 | A@1, C@2, D@3 |
| U4 | E@1 |
Step 1 — 2-core filter. Count interactions: users U1=3, U2=2, U3=3, U4=1; items A=3, B=2, C=2, D=1, E=1. Drop everything below \(2\): out goes U4 (only one interaction) and items D and E. Re-check what remains — U1=3, U2=2, U3=2; A=3, B=2, C=2 — all \(\ge 2\), so we stop (the filter is iterative; here one pass suffices).
Why “iterative” is not decoration. Change one thing — suppose U3 had interacted only with C@2 and D@3. Pass 1 still drops U4, D and E, which leaves U3 holding just C: pass 2 drops U3 for having one interaction, which leaves C with only U1: pass 3 drops item C, and only pass 4 finds nothing left to drop. An item that passed the first check — item C, held by exactly \(k=2\) users — is gone by the end. A single threshold applied once cannot do this; the cascade stops only when every surviving user and every surviving item has \(\ge k\).
Step 2 — leave-one-out. For each surviving user, hold out the chronologically last interaction: \[ \text{TEST}=\{\text{U1}\!:\!\text{C@4},\ \text{U2}\!:\!\text{B@2},\ \text{U3}\!:\!\text{C@2}\},\qquad \text{TRAIN}=\{\text{U1}\!:\!\text{A@1, B@3},\ \text{U2}\!:\!\text{A@1},\ \text{U3}\!:\!\text{A@1}\}. \]
Note the subtlety that trips people up: U3’s chronologically last interaction was D@3 — but D was removed by the k-core filter, so U3’s held-out test item is the next-latest survivor, C@2. Filtering happens before splitting; the order matters.
D@3, was filtered out before the split, so its test item is the next-latest survivor, C@2 — filtering precedes splitting. Read the clocks across the rows and the cross-user leak is visible: U1’s training item B@3 is later than U2’s test item B@2.
Three things this tiny example is now big enough to show.
(a) The filter’s guarantee does not survive the split. Look at the training set again. Only items A and B appear in it — yet C is the test target for two of the three users (U1’s C@4 and U3’s C@2). The 2-core guarantee held on the full data, not on the training set: holding out each user’s last interaction turned a surviving item cold again (an item with no training history). U2 and U3 keep one training interaction each. Ask the obvious question — how can a model rank C when it never saw C? — and the answer is that a model that has only the interactions cannot do better than guess (a content model has side-features).
Real pipelines (a claim from practice, not tested here) use a large \(k\) (10), which makes such items rarer but does not remove them. They also drop test items that never appear in training, or report those cases separately; whichever a paper did, check.
(b) Leave-one-out leaks across users. Compare the clocks. The latest training event is U1’s B@3, at \(t=3\). The earliest test events are U2’s B@2 and U3’s C@2, at \(t=2\). So the model is shown an interaction from \(t=3\) and then asked to predict what U2 and U3 did at \(t=2\) — and worse, the very item it must predict for U2, item B, it has already watched U1 adopt afterwards. Nothing is leaked inside any one user’s own history; the leak is across the population.
(c) The global temporal split fixes that — and here is its price. Put one cut-off at \(t=2.5\) for everybody. Then
\[ \text{TRAIN}=\{\text{U1}\!:\!\text{A@1},\ \text{U2}\!:\!\text{A@1, B@2},\ \text{U3}\!:\!\text{A@1, C@2}\}, \qquad \text{TEST}=\{\text{U1}\!:\!\text{B@3, C@4}\}. \]
Every training event now precedes every test event, so the cross-user leak in (b) is gone. But U2 and U3 have nothing left to be tested on — the whole evaluation rests on U1. That is the real trade a global cut makes: honesty about time, paid for in test coverage (and, on real data, in whole cohorts of users who never appear in the test set).
Why not just split randomly? Because recommendation is a forecast. A random split that trains on U1’s C@4 and tests on U1’s A@1 asks the model to predict U1’s past after showing it U1’s future — information it will never have when it is actually used. (That is a future leak, not an answer leak: the model is not handed A, it is handed later behaviour, such as U1 liking C@4, that carries information about U1’s earlier choices.) An honest evaluation should use a time-based hold-out — and note that the graph-CF numbers the graph chapters quote from He et al. (2020) do not: they were measured on NGCF’s random per-user split (§2, §4). Read them as comparable, not as leak-free.
In one line: filter first, then split by time: one global cut when the test coverage allows, leave-one-out for next-item models, and a random split only as a labelled, leaky comparison.
4. The benchmark map: which model, which dataset, which protocol
Bob is handed a results table from a paper and a skeptic’s first question: can your number sit beside these? This section is the map that answers it: for each research community, the datasets it runs on, the models usually compared on them, and the protocol they are compared under. The protocol is often one a later benchmark paper re-ran (FuxiCTR, an open benchmark for click-through-rate prediction models; the MIND dataset paper; the full-ranking re-scoring in later sequential papers), not the one each model’s own paper used. Read it alongside the Large Language Models and Recommendation chapter, which surveys large language models (LLMs) and the recommenders built on them. It answers, in one place, the question a practitioner or a skeptical reader asks first: which model, on which dataset, under which protocol?
Figure B.2, at the end of this section, draws the choices as a tree.
One idea organizes the whole map, and it is the opposite of what a beginner expects: the protocol is fixed by the research community, not by the dataset. MovieLens-1M alone is evaluated three incompatible ways — as root-mean-squared-error (RMSE) rating prediction (the Netflix-Prize lineage), as leave-one-out ranking against 99 sampled negatives (the NCF — neural collaborative filtering — lineage), and as leave-one-out next-item prediction against 100 sampled negatives (the SASRec / BERT4Rec lineage). A number is therefore comparable to another only when the regime (a research community’s package of datasets, models and one protocol), the split, and the ranking scope (how many candidate items the held-out one is ranked against) all match, the concrete form of the reproducibility warning §5 sharpens.
The metric acronyms, in one line, so the table below reads: the root-mean-squared error (RMSE) of a predicted star rating (lower is better); the mean absolute error (MAE), the same idea without the squaring; the mean average precision (MAP) (Evaluation Metrics §6); the hit rate (HR) at \(K\), written HR@\(K\) — was the held-out item in the top \(K\)?; Recall@\(K\), the share of a user’s held-out items found in the top \(K\); the normalized discounted cumulative gain (NDCG) at \(K\), which also rewards putting them near the top; the mean reciprocal rank (MRR), the average of \(1/(\text{rank of the held-out item})\); the area under the receiver-operating-characteristic curve (AUC), the chance that a random clicked impression outscores a random unclicked one; and LogLoss, the binary cross-entropy of a predicted click probability. Their full definitions, and the sampled-vs-full argument, live in Evaluation Metrics; here they are only labels.
Before the table: why “ranking scope” is a column at all. It is the one entry a beginner skips and a careful reader never does, so work it once on §3’s own example. Put U1’s test item C in a catalog of 1,000 items. U1 trained on A and B, so those two are excluded and 998 candidates are ranked: C and 997 negatives. Suppose the model puts C 50th — so 49 candidates outscore it.
- Full ranking (all 998): C is 50th, which is not in the top 10, so \(\text{HR@10}=0\).
- Sampled (the legacy protocol): C is ranked against only 99 negatives drawn at random from the 997. On average just \(99\times 49/997 \approx 4.87\) of the 49 better candidates get drawn, so C lands about 6th, and \(\text{HR@10}=1\) with probability \(\approx 0.982\): the chance that 9 or fewer of the 49 better candidates are among the 99 drawn (a hypergeometric tail: the sum counts every way a random draw of 99 from the 997 can contain \(k\) of the 49 better items, for \(k=0\) to 9, as a share of all possible draws of 99; \(\sum_{k=0}^{9}\binom{49}{k}\binom{948}{99-k}\big/\binom{997}{99}\), too long to add by hand, so summed by program).
Same model, same test item, same held-out data: the sampled score is \(1\) with probability \(0.982\) and the full score is \(0\). That is why two numbers cannot be compared until their Ranking scope cells match: a gain measured under one scope says little about the other.
A key to the names in the benchmark maps, so none of them arrives bare. It is a lookup list: skim it now, and return to it when a name in the two benchmark-map tables below is unfamiliar.
- matrix factorization (MF), the rating-prediction workhorse;
- SVD++ and timeSVD++, Koren’s extensions of the singular value decomposition (SVD) line, adding implicit feedback and, in timeSVD++, drift over time;
- RBM-CF, a collaborative filter built on a restricted Boltzmann machine (RBM);
- BellKor, the Netflix-Prize-winning blend of matrix-factorization and neighborhood models;
- neural collaborative filtering (NCF), which replaces the dot product with a learned network, and NeuMF (short for neural matrix factorization), its authors’ own model;
- GRU4Rec, a recurrent session model that reads the sequence one step at a time;
- Caser, a convolutional sequence model;
- SASRec and BERT4Rec, Transformer sequence models (they weigh every earlier item by attention), one-directional and bidirectional;
- S3-Rec, a self-supervised sequential model pre-trained on item attributes;
- CL4SRec, a contrastive sequential model that augments each user’s sequence and learns to pull two views of the same sequence together;
- DuoRec, a contrastive sequential model aimed at representation degeneration, item vectors collapsing into a narrow region;
- FEARec, a sequential model that filters the sequence in the frequency domain, as one filters a signal, before attending;
- neural graph collaborative filtering (NGCF) and LightGCN, the two standard graph-convolution collaborative filters;
- disentangled graph collaborative filtering (DGCF) and disentangled contrastive collaborative filtering (DCCF), graph-CF variants whose embeddings are disentangled, split into parts meant to capture different reasons for an interaction, plus GCCF, a simplified graph-convolution collaborative filter, and AutoCF, an automated self-supervised collaborative filter of the same family;
- self-supervised graph learning (SGL), SimGCL and LightGCL, contrastive graph-CF models;
- the knowledge-graph attention network (KGAT);
- for click-through rate, Wide & Deep, deep factorization machine (DeepFM), the deep & cross network (DCN), xDeepFM, AutoInt, a network that learns feature interactions by self-attention, and the deep interest network (DIN), which attends over the user’s behavior history (it has no table row; see the bullet after Table B.5);
- for news, neural recommenders built on a deep knowledge-aware network (DKN), neural personalized attention (NPA), neural attentive multi-view learning (NAML), long- and short-term user representations (LSTUR) and neural news recommendation with multi-head self-attention (NRMS);
- RLMRec, which attaches large-language-model-written user and item profiles to a collaborative backbone (the base model the method is added to).
Datasets named only here. Pinterest: implicit “pin” events on images. Avazu: a mobile-advertising click log. Alibaba-iFashion: clicks on fashion outfits. Taobao: the Alibaba behavior log of §1 (no table row; cite its Tianchi page). Amazon Games / Sports / Toys / Beauty: single-category subsets of Amazon Reviews. ML-20M: MovieLens-20M (papers abbreviate MovieLens as ML). Note that “Yelp” in the sequential and LLM-augmented rows is each paper’s own subset of the live Yelp Open Dataset, not the frozen Yelp2018 snapshot of §2.
Two notations in the table below. A ratio such as 8:1:1 (train : validation : test) means that 80%, 10% and 10% of the interactions go to the three parts, and 3:1:1 means 60%, 20% and 20%. “10-core” means every user and item has at least 10 interactions (§1).
| Regime (community) | Typical datasets | Most-cited models (year) | Split | Ranking scope | Headline metrics |
|---|---|---|---|---|---|
| Rating prediction | Netflix Prize; MovieLens 100K/1M/10M | MF / SVD++ (’08), timeSVD++ (’09), RBM-CF (’07), BellKor (’09) | Netflix: each user’s most recent ratings held out; MovieLens: random or \(k\)-fold over ratings (what the papers cited here did; the ratings carry timestamps, so a time split is possible, §3) | predict the star value | RMSE (MAE) |
| Implicit top-\(N\), sampled | MovieLens-1M; Pinterest | NeuMF / NCF (’17) | leave-one-out (each user’s “latest interaction”, per the NCF paper) | vs. 99 sampled negatives | HR@10, NDCG@10 |
| Sequential, sampled | MovieLens-1M; Amazon Beauty/Games; Steam (BERT4Rec adds ML-20M); Amazon Sports/Toys, Yelp | GRU4Rec (’16), Caser (’18), SASRec (’18), BERT4Rec (’19), S3-Rec (’20) | leave-one-out (last by time) | vs. 100 sampled negatives (S3-Rec: 99) | HR@10, NDCG@10, MRR |
| Sequential, full | Amazon Beauty / Sports; Yelp; MovieLens-1M — CL4SRec reuses S3-Rec’s preprocessing | CL4SRec (’22), DuoRec (’22), FEARec (’23); and DuoRec’s own tables re-score GRU4Rec / SASRec / BERT4Rec / S3-Rec | leave-one-out (last by time) | full ranking (all items) | HR / NDCG @\(K\), MRR (the \(K\) values vary by paper) |
| Graph CF (the trio, §2) | Gowalla, Yelp2018, Amazon-Book (SGL swaps Gowalla for iFashion; KGAT uses its own Amazon-book / Yelp2018 files plus Last-FM) | NGCF (’19), LightGCN (’20), DGCF (’20), SGL (’21), SimGCL (’22), LightGCL (’23) | NGCF-lineage files: random per-user 80/20, 10-core (§3; SGL restates it as 7:1:2; SimGCL and LightGCL use their own splits or files, see below) | full ranking (all items) | Recall@20, NDCG@20 |
| CTR prediction | Criteo, Avazu | Wide&Deep (’16), DeepFM (’17), DCN (’17), xDeepFM (’18), AutoInt (’19) | random 8:1:1 (train : validation : test) | per-impression binary click | AUC, LogLoss |
| News | MIND | DKN (’18), NPA (’19), NAML (’19), LSTUR (’19), NRMS (’19) | impression / time-ordered | within each impression | AUC, MRR, NDCG@5/10 |
| LLM-augmented CF | RLMRec’s own subsets of Amazon-book, Yelp and Steam — not the trio’s files | RLMRec (’24) over GCCF / LightGCN / SGL / SimGCL / DCCF / AutoCF backbones, all re-trained | 3:1:1 (train : validation : test) on those subsets, after \(k\)-core; the paper does not say whether the cut is random | full ranking (all items) | Recall / NDCG @5,10,20 |
What the rows are and are not: they record how these models are compared today, which for several is not how their own papers evaluated.
- GRU4Rec was introduced for session-based next-click data (RecSys Challenge 2015 click-streams, Recall@20 / MRR@20) and appears in the sequential row because later work re-ran it per user.
- DIN needs a user-behavior sequence, which Criteo and Avazu do not have; its own paper uses Amazon Electronics, MovieLens-20M and an Alibaba advertising log, so it is not in the CTR row.
- The five news models all predate MIND (2020) and were re-run on it by the MIND paper.
- Caser’s own paper splits each user’s sequence 70/10/20 and reports Precision, Recall and mean average precision (MAP).
A few precise points the table compresses, each worth knowing before you trust a leaderboard:
- The sequential shift (the two sequential rows). Early next-item models ranked the held-out item against a small sample of negatives and reported HR@10 / NDCG@10: SASRec (Kang & McAuley, 2018) “randomly sample[s] 100 negative items” and ranks “these 101 items”; BERT4Rec (Sun et al., 2019) pairs each test item with “100 randomly sampled negative items” drawn “according to their popularity”; S3-Rec (Zhou et al., 2020), despite being the newest of the three, pairs the test item with “99 randomly sampled negative items”, which is why it sits in the sampled row, not the full one.
- Why the field moved to full ranking. Krichene & Rendle (2020) showed sampled metrics can reverse which model looks better (Evaluation Metrics §11b). After that, CL4SRec (Xie et al., 2022) “evaluate[s] each method on the whole item set without sampling”, and DuoRec (Qiu et al., 2022) does the same, as reported there and not tested here: it ranks “over the whole item set” and re-scores GRU4Rec, Caser, SASRec, BERT4Rec and S3-Rec that way in its own tables. One model, two very different headline numbers: always check which.
- NCF vs. sequential negatives. The NCF paper (He et al., 2017) says it samples 100 items the user has not touched and ranks the test item among them; its released evaluation files carry exactly 99 negative item IDs per test pair (a 100-item list with the test item), and the 99 of Table B.5 means these files. The SASRec / BERT4Rec protocol uses 100 (a 101-item list). The one-item difference is real, and it is exactly the kind of detail that makes two “leave-one-out HR@10” numbers quietly non-comparable.
- The trio’s split is random, not time-based (§2, §3). NGCF (Wang et al., 2019) released a fixed per-user 80/20 split (10% of the training part held out for validation, 10-core filtered). LightGCN reuses it for two of the three: it says it requested the datasets “including train/test splits” from NGCF’s authors, and that Gowalla and Amazon-Book “are exactly the same as the NGCF paper used”. The third is the exception, in the paper’s own words (reported there, not tested here): Yelp2018 “is a revised version”; “the previous version did not filter out cold-start items in the testing set”, the authors shared only the revised one, and so LightGCN “re-run[s] NGCF on the Yelp2018 data”.
- Yelp2018 exists in two NGCF versions. NGCF’s first arXiv version (May 2019) lists Yelp2018 as 31,831 users, 40,841 items and 1,666,869 interactions. Its latest version (July 2020, after LightGCN) says in a footnote that the earlier test set did not filter out cold-start items, so it reran all methods on a revised file, now marked with a star after its name, with LightGCN’s counts of Table B.4. That revised file is the one this appendix’s Yelp2018 row describes. Numbers from NGCF’s earlier version (the ACM-published one, by the later version’s first-page note) and LightGCN’s sit on different files on Yelp2018.
- NGCF’s later version changed its NDCG, on all three datasets. A second footnote says the earlier NDCG implementation differed slightly from the standard definition, so the authors re-implemented it and reran all methods. NGCF-3 (the three-layer model, the paper’s headline row) NDCG@20 fell from 0.2237 to 0.1327 on Gowalla, 0.0926 to 0.0477 on Yelp2018 and 0.0630 to 0.0261 on Amazon-Book, while recall barely moved on the two datasets whose files did not change (Gowalla 0.1547 to 0.1569, Amazon-Book 0.0344 to 0.0337). Quote any NGCF number, recall or NDCG, only with the version named.
- SGL’s 7:1:2 is the same protocol in rounder arithmetic. SGL (Wu et al., 2021) says it follows NGCF’s strategy, and NGCF’s 80/20 with a tenth of the training part held back is 72:8:20, i.e. 7.2:0.8:2. SGL’s own statistics table repeats Yelp2018 and Amazon-Book at exactly LightGCN’s counts, so on those two datasets the numbers are comparable. What SGL changes is the third dataset: out goes Gowalla, in comes Alibaba-iFashion, which SGL samples itself (300,000 users \(\times\) 81,614 items \(\times\) 1,607,813 interactions).
- Drift sets in later. SimGCL (Yu et al., 2022) reuses those same two datasets but cuts its own 7:1:2 split and takes as its third Douban-Book, a 1-5 star book-rating set it binarizes itself; LightGCL (Cai et al., 2023) keeps the names while running different files, its Amazon-book being 78,578 \(\times\) 77,801 \(\times\) 2,240,156. Check the user / item counts and the split ratio before quoting one paper against another. And because the cut is random, none of the NGCF lineage tests generalization over time: here “comparable” still is not “leak-free.”
- Dataset lineage, kept straight (same name, different file). Alibaba-iFashion enters this list with SGL (2021), not the original trio. Last-FM belongs to KGAT’s knowledge-graph trio, a regime in which the model also sees a graph of facts linking items to entities (song \(\to\) artist \(\to\) genre), and that trio is not the NGCF files despite sharing two names.
- KGAT’s files, in numbers. KGAT’s (Wang et al., 2019) Amazon-book is 70,679 users \(\times\) 24,915 items \(\times\) 847,733 interactions against the trio’s 52,643 \(\times\) 91,599 \(\times\) 2,984,108: \(1.3\times\) the users but \(0.27\times\) the items and \(0.28\times\) the interactions. Its Yelp2018 is 45,919 \(\times\) 45,538 \(\times\) 1,185,068 against 31,668 \(\times\) 38,048 \(\times\) 1,561,406. KGAT’s numbers therefore cannot be read against LightGCN’s, and neither can its Last-FM subset be read as LFM-1b or LFM-2b (§7).
- The LLM-augmented row does not reuse the trio’s files (last row). This is the trap the rest of §4 warns about, and the most recent regime walks straight into it. RLMRec (Ren et al., 2024) needs a language-model-written profile for every user and every item, so it builds its own much smaller subsets. Its statistics table gives Amazon-book as 11,000 users \(\times\) 9,332 items \(\times\) 120,464 interactions (about \(4\%\) of the trio’s Amazon-Book interactions), Yelp as \(11{,}091\times11{,}010\times166{,}620\) and Steam as \(23{,}310\times5{,}237\times316{,}190\). It filters ratings below 3 on the first two, applies a \(k\)-core, then splits “into training, validation, and testing sets using a 3:1:1 ratio” and re-trains every backbone itself. The paper does not say whether that cut is random or by time; its released code would settle it (not checked here).
- A contrast, with each number’s owner. Ren et al. (2024, Table 1) report Recall@20 \(=0.1411\) for LightGCN on RLMRec’s own Amazon-book, and He et al. (2020, Table 4) report \(0.0411\) for LightGCN on the trio’s Amazon-Book (reported there, not tested here). The files differ (11,000 users against 52,643), so the two numbers describe two benchmarks and not a disagreement about one model. RLMRec’s lifts (its gains over the base models) are readable only against the backbones it re-ran itself, never against LightGCN’s published trio numbers.
The protocol anchors, the papers whose numbers later work quotes itself against, are, by regime: rating prediction, the Netflix Prize and Koren’s matrix-factorization line; the sampled top-\(N\) regime, NCF (He et al., 2017); early sequential, SASRec (2018) and BERT4Rec (2019); the shift to full ranking, Krichene & Rendle (2020); graph CF, NGCF (2019, which released the split) and LightGCN (2020); CTR, Wide & Deep (2016) onward, standardized by the FuxiCTR open benchmark (2021); news, the MIND dataset paper (Wu et al., 2020). Full citations for these anchors are in §9, which also lists most of the other models named above (BellKor has no citation here); the rest are cited in the chapters that introduce them (the chapters listed under Table B.5).
| Dataset | Feedback | Used by (regime) | Representative models | Protocol (split · scope) | Headline metric |
|---|---|---|---|---|---|
| MovieLens-1M | explicit \(\to\) implicit (ratings binarized to 1) | rating · NCF · sequential | MF, NeuMF, SASRec, BERT4Rec | random ratings or LOO · sampled | RMSE or HR@10 / NDCG@10 |
| MovieLens-20M | explicit \(\to\) implicit | BERT4Rec | BERT4Rec | LOO · 100 sampled | HR@10 / NDCG@10 |
| Netflix Prize | explicit | rating prediction (history) | MF, SVD++, BellKor | a fixed hold-out of each user’s most recent ratings — a per-user cut on time, like leave-one-out but with about nine items — split into Probe (public validation), Quiz (leaderboard) and Test (final, hidden) · — | RMSE |
| Amazon-Book | implicit (binarized stars) | graph CF (trio); LLM-augmented CF (RLMRec, on its own subset) | NGCF, LightGCN, SGL, RLMRec | random 80/20, 10-core · full (SGL restates 7:1:2; RLMRec: 3:1:1 on its own subset) | Recall@20 / NDCG@20 |
| Amazon Beauty / Sports / Toys | implicit (binarized stars) | sequential | SASRec, BERT4Rec, S3-Rec, CL4SRec | LOO · sampled \(\to\) full | HR@10 / NDCG@10 |
| Yelp2018 | implicit (binarized stars) | graph CF (trio) | NGCF, LightGCN, SimGCL | random 80/20, 10-core · full (SimGCL: its own 7:1:2) | Recall@20 / NDCG@20 |
| Gowalla | implicit | graph CF (trio); point-of-interest (POI) | NGCF, LightGCN, DGCF | random 80/20, 10-core · full | Recall@20 / NDCG@20 |
| Steam | implicit | sequential; LLM-augmented CF (RLMRec, own subset) | SASRec, BERT4Rec, RLMRec | LOO · sampled \(\to\) full (RLMRec: 3:1:1 · full) | HR@10 / NDCG@10 (RLMRec: Recall / NDCG @5,10,20) |
| LastFM / Last-FM | implicit | knowledge-graph CF — the model also sees a graph of facts linking items to entities (song \(\to\) artist \(\to\) genre) | KGAT | random 80/20, 10-core · full — on KGAT’s own file, not the trio’s or LFM-1b (§7) | Recall@20 / NDCG@20 |
| MIND | implicit click | news | NRMS, NAML, LSTUR | impression / time · within-impression | AUC / MRR / NDCG@5,10 |
| Criteo / Avazu | implicit click | CTR | DeepFM, DCN, xDeepFM | random 8:1:1 (train : validation : test) · per-impression | AUC / LogLoss |
How to use the two tables. Go down the benchmark map when you are choosing how to run an experiment: pick the regime that matches your task and inherit its split, scope and metrics, so your numbers join an existing conversation. Where that inherited protocol is random or sampled, also report a time-based, full-ranking result (a global temporal split when timestamps allow, else leave-one-out, §3), because comparable is not the same as correct (§2, §3); it costs one extra run and answers the skeptic’s first question (a claim from practice, not tested here). Go into the dataset index when you are reading a paper or holding a dataset: look up what its community does, and you will know whether a headline number is full-ranking or sampled, time-based or random, before you decide how much to trust it.
In one line: the protocol belongs to the community, not the dataset, so before you set two numbers side by side check that the regime, the split and the ranking scope all match.
5. Two evaluation traps the data side creates
Cara’s model finds the held-out film in the top 3 for half of her users (a hit rate, HR@3, of \(0.5\), §4) when each user’s last interaction is held out (leave-one-out, §3), and she wonders whether that is good. It depends on what a lazy baseline scores on the same data. Beyond the split, two well-documented pitfalls live on the data side, each with a companion bullet (the popularity-aware views, what the critique teaches); both are reasons to distrust a headline number.
- Popularity bias in evaluation. A few blockbusters collect most test interactions (the long tail of From Graphs to LightGCN §13), so a recommender that simply pushes popular items scores well on Recall/NDCG without personalizing at all. Put a number on it: under leave-one-out every user has exactly one held-out item, so if the three most popular items account for half of all held-out items, then a list that shows those same three to everyone, with no personalization in it, scores at least \(\text{HR@3}=0.5\) (a user who already has one of the three gets another item in its place, which can only add hits). Whatever your model scores, that is the floor it has to beat; if Cara’s data were like this, her \(0.5\) would sit exactly on it.
- Popularity-aware views. Report them alongside the headline metric. Coverage is the share of the catalog the system ever recommends: if 40 of 1,000 items ever appear in anyone’s list, coverage is 4%. Tail-Recall is Recall computed only over long-tail items, i.e. excluding the few blockbusters: if 200 held-out items are in the tail and the lists find 30 of them, it is \(30/200=0.15\) (made-up numbers).
- The reproducibility critique. Ferrari Dacrema, Cremonesi & Jannach (“Are we really making much progress?”, RecSys 2019) examined 18 neural-recommendation algorithms from top conferences and report that only 7 could be reproduced with reasonable effort, and that 6 of those were often beaten by comparably simple heuristics based on nearest-neighbor or graph-based techniques (reported there, not tested here). A typical such baseline is item-kNN: recommend the items most similar, by who interacted with both, to the ones a user already liked: if Ann and Cara both watched films A and C, a user who liked A is offered C.
- What the critique teaches. A strong, tuned baseline is mandatory, and a gain is meaningless without identical splits, released code, and — for top-\(N\) ranking tasks — full-ranking metrics (the sampled-vs-full caution of Evaluation Metrics). Rating prediction, CTR and news are scored differently (§4), so “full ranking” is a requirement of the ranking regimes, not of every number. Implementation Choices §5 turns this finding into the operational checklist — five pitfalls (three of them documented in the literature) and the tuple rule: quote a number only together with its whole tuple, which there has eight fields. The five that this appendix covers are model, dataset, split, ranking scope and metric@\(K\); the other three (the \(k\)-core filter, which §3 here only illustrates, the training sampler and the tuning budget) are left to Implementation Choices.
In one line: a headline number can hide a popularity shortcut or an untuned baseline, so ask what the lazy baseline scores and whether the baselines were tuned.
6. How the field gets data today
Bob would rather not write a file parser and a splitter for every dataset he tries. He rarely has to: the work has moved into toolkits and metadata standards.
- Curated catalogs. Julian McAuley’s Recommender Systems Datasets page (at UCSD) is a widely used index (the author’s reading, not tested here); several GitHub “awesome-datasets” lists mirror it.
- Benchmark frameworks ship datasets in a unified format and run dozens of models on them under one protocol — the modern way to “present” a dataset: RecBole (release v1.2.1 of 23 February 2025, whose README lists 94 models and 44 datasets, though its own comparison table lists 43; and an atomic format, one plain-text file each for interactions,
.inter, user features,.user, and item features,.item), Cornac (multimodal / side-information focus), Elliot (reproducibility-first: one configuration file drives models \(+\) hyper-parameter optimization (HPO, searching the settings automatically) \(+\) metrics \(+\) significance tests, which check that a gain is larger than chance), Microsoft Recommenders (notebooks for an end-to-end pipeline, from data preparation to deployment), and LensKit (a Python toolkit for offline recommender experiments). - Hosting & metadata — metadata here meaning the machine-readable description that travels with a dataset: what its columns are, where it came from, how it may be used. HuggingFace Datasets hosts Amazon-Reviews-2023 with standard splits (its project page links there); Croissant (from MLCommons, the industry–academic machine-learning benchmarking consortium, 2024) is a machine-readable metadata standard.
- Croissant at NeurIPS, 2024 to 2026. NeurIPS tightened it in steps. Its 2024 Datasets & Benchmarks call encouraged it (“We encourage the authors to use Croissant format”) and listed a Croissant record URL among the supplementary items; the 2025 call put the Croissant file under its “required” dataset and code items, worded “should use the Croissant machine-readable format”, and the organizers’ post of 10 March 2025 made proper hosting with a Croissant description a requirement.
- Croissant at NeurIPS in 2026. The 2026 call of the renamed Evaluations and Datasets track says dataset authors “must use the Croissant machine-readable format”, but only “request[s]” both the core and the Responsible Artificial Intelligence (RAI) fields. The organizers’ post of 4 May 2026 is stricter: all dataset submissions “must now include” the RAI metadata in the Croissant file, and those without it “will be flagged during review”. The track’s hosting page also says the RAI metadata “must be included”. The sources differ in force on the RAI fields.
In practice (advice from practice, not tested here). Don’t hand-roll loading and splitting. Pick a framework (RecBole or Elliot for research comparability; Microsoft Recommenders for an end-to-end pipeline), use its built-in time-based split (a global temporal split if you can afford the lost test coverage, else leave-one-out, §3) and report the framework \(+\) version so others can reproduce you. If you must publish a dataset, attach a Croissant record.
In one line: use a framework’s loader and its time-based split, and name the framework and its version.
7. Active, deprecated, and a licensing caution
Ann finds a famous dataset on a mirror site. May she train on it? Famous datasets mostly split into two groups, and a from-zero reader should know both: the active ones you actually build on, and the deprecated / withdrawn ones that shaped the field but are no longer safe, or even possible, to use. The withdrawn ones are still worth knowing by name (you will meet them in older papers); they are never worth training on.
- Netflix Prize (2006): a famous early rating benchmark, withdrawn after researchers de-anonymized its users (Narayanan & Shmatikov, 2008). A federal privacy lawsuit and an inquiry by the Federal Trade Commission of the United States followed, and Netflix cancelled a planned second contest in March 2010 (reported by the Electronic Privacy Information Center, 2010). There is no official download; unofficial copies circulate, but cite it as history, do not train on it. For §3: this benchmark already held out each user’s most recent ratings (Bennett & Lanning, 2007: nine per user, or half of the ratings for users with fewer than 18), so the time-based idea is not new.
- LastFM LFM-2b / LFM-1b: both large listening-history sets are no longer officially distributed. The LFM-1b page says “The dataset is not available for download anymore due to license issues”; LFM-2b is also no longer distributed (check its page). The origin papers are Schedl (2016) for LFM-1b and Schedl et al. (2022) for LFM-2b. The §1 table tags both Withdrawn; treat any circulating copy as restricted and check current terms. Do not confuse either with the small Last-FM file that ships with KGAT’s code (23,566 users \(\times\) 48,123 items) and is what knowledge-graph papers actually run on (§4); cite that one as KGAT’s, and read its terms before building on it.
- Yahoo! Webscope (e.g. the R3 music-ratings set): still available, but only by signed agreement (academic use; the terms sit behind the access gate and were not re-read for this appendix). R3 remains the classic missing-not-at-random benchmark (the ratings are missing because of what users chose to rate, not by chance): it pairs users’ organic ratings with a randomly-surveyed set, so it recurs in debiasing work (methods that correct for users rating mostly what they chose to see) despite the access gate.
The datasets you can still obtain (MovieLens, Gowalla, Yelp, Amazon, MIND, Steam, Book-Crossing, Goodreads) are all free to get, but they do not all grant the same thing, and the only trustworthy source for what each grants is that dataset’s own page (read on the verification date unless noted). Do not read “free” as “open”. Always read the specific license, and re-read it, because these change, before any commercial use.
- MovieLens may be used “for any research purposes” but not “for any commercial or revenue-bearing purposes without first obtaining permission”, and asks to be cited.
- Yelp, MIND and Goodreads each gate the data to educational, research or academic use: the Yelp landing page says the data is “intended for educational use”, and MIND ships under Microsoft Research License Terms you must accept first; the Goodreads page says “academic use only”.
- Yelp’s Data Agreement is shown at download, and its text could not be re-read for this appendix. As read in June 2026 it grants a license that is “royalty-free, non-exclusive, revocable, non-sublicensable, non-transferable” and “solely for academic use”, running for a twelve-month term that either side may end at any time; treat the quotation as June 2026 and re-read it before relying on it.
- Amazon and Steam: the pages ask only that you cite the right paper.
- Gowalla and Book-Crossing: SNAP’s Gowalla page and GroupLens’s Book-Crossing page state no terms at all, which is not the same as granting them.
Only some of these are Active in §1’s sense (just download it); Yelp, MIND and Goodreads are tagged Restricted there because they are gated, revocable, or both.
In one line: before you train on a dataset, check its Status and read its own license page, because free to download does not mean free to use.
8. Glossary
| Term | Plain meaning |
|---|---|
| Explicit / implicit feedback | Stated ratings (e.g. 1–5 stars, but 1–10 in Book-Crossing and 0.5–5 in MovieLens 10M and later) vs. behavioral \(0/1\) (click/buy); the latter is positives-only. |
| Density | Interactions \(\div\) (users \(\times\) items) — the share of the user–item grid that is filled. The graph-CF trio sits near \(0.1\%\) (§2), but MovieLens-1M is about \(4.5\%\) (\(1{,}000{,}209/(6{,}040\times3{,}706)\)) and Netflix about \(1.2\%\) (\(100{,}480{,}507/(480{,}189\times17{,}770)\)), so check. |
| Sparsity | \(1-\) density: the share of the user–item grid that is empty (§2). |
| k-core filter | Iteratively drop users/items with \(<k\) interactions until none remain; stabilizes the data. The guarantee holds before the split, not after it (§3). |
| Random split | Hold out a random fraction of interactions, per user or globally (say which) — leaks the future for time-ordered data. |
| Leave-one-out (LOO) | Hold out one interaction per user — the chronologically last unless stated otherwise, often the second-last for validation. Leak-free within a user, not across users (§3). Some papers use the name for a random held-out interaction: check. |
| Global temporal split | One cut-off timestamp for everybody: train before it, test after. The only one of the three that is leak-free across users; its price is that some users get no test item at all (§3). |
| Frozen split | A train/test partition released as files and reused unchanged by later papers of one lineage (the trio’s defining feature, §2), which makes their numbers mutually quotable; not every graph-CF paper reuses it (§4). |
| Full ranking vs. sampled negatives | Score the held-out item against every item the user has not seen, or against a handful (99–100) drawn at random (negative sampling; a negative is an item the user did not interact with, treated as non-relevant). The two give different numbers for the same model (§4). |
| Graph-CF trio | Gowalla / Yelp2018 / Amazon-Book — the frozen-split benchmark of graph-CF papers. |
| Popularity bias (in eval) | Popular items dominate the test set, so pushing them inflates accuracy metrics. |
| Benchmark framework | RecBole / Cornac / Elliot / Microsoft Recommenders / LensKit — unified datasets + models + protocol. |
| Croissant | MLCommons machine-readable dataset-metadata standard (2024); NeurIPS encouraged a record in 2024, worded it “should use” in 2025 and “must use” in 2026, with the Responsible Artificial Intelligence (RAI) fields only requested in the 2026 call (the organizers’ post of 4 May 2026 requires them, §6). |
| Impression | One showing of an item, or of a list of items, to a user, with the click or no-click that followed (§1). |
| Cold-start | A new user or item with no interaction history, so a model has nothing to learn it from (§1, §3). |
| \(k\)-fold | Cut the data into \(k\) parts; test on each in turn, train on the rest, average (§3). A random scheme. |
| Item-kNN | Recommend the items most similar, by who interacted with both, to those the user already liked; a strong simple baseline (§5). |
| Debiasing | Methods that correct for users rating mostly what they chose to see (§7). |
| Ranking scope | How many candidate items the held-out one is ranked against: all of them (full ranking) or a sample. Printed in Table B.5 as its own column (§4). |
| HR@\(K\), MRR, AUC | Hit rate: was the held-out item in the top \(K\); mean reciprocal rank: the average of \(1/\text{rank}\); area under the receiver-operating-characteristic (ROC) curve: the chance a clicked impression outscores an unclicked one. Labels here; definitions in Evaluation Metrics (§4). |
| Backbone, lift | The base model a method is added to, and the gain over it (§4, RLMRec). |
| Regime | A research community’s shared package of datasets + models + one evaluation protocol. Numbers are comparable only inside a regime, and only when split, preprocessing and ranking scope match as well (§4). |
9. References
Ordered alphabetically. These are the protocol anchors of §4, the papers whose numbers later work quotes itself against, the papers of the models whose splits or file counts §4 quotes, the source papers of the datasets §1 and §2 lean on hardest, and the reproducibility references. Each carries a short note on what it anchors. It is not a source list for every dataset named in this appendix: for the rest (Criteo, Avazu, Taobao, Pinterest, Alibaba-iFashion, Yahoo! R3, and the benchmark frameworks other than RecBole, which is listed below), cite the dataset’s or project’s own page; BellKor (§4) is named without a citation.
An, M., Wu, F., Wu, C., Zhang, K., Liu, Z., & Xie, X. (2019). Neural news recommendation with long- and short-term user representations (LSTUR). In Proceedings of ACL (pp. 336–345). https://doi.org/10.18653/v1/P19-1033 (a news-regime model, re-run on MIND, §4)
Bennett, J., & Lanning, S. (2007). The Netflix Prize. In Proceedings of KDD Cup and Workshop 2007 (the rating-prediction benchmark — and the source for its probe / quiz / test sets being drawn from each user’s most recent ratings, §4, §7)
Cai, X., Huang, C., Xia, L., & Ren, X. (2023). LightGCL: Simple yet effective graph contrastive learning for recommendation. In Proceedings of ICLR. arXiv:2302.08191 (its own Amazon-book file, 78,578 users, §4)
Cheng, H.-T., Koc, L., Harmsen, J., et al. (2016). Wide & Deep learning for recommender systems. In Proceedings of the 1st Workshop on Deep Learning for Recommender Systems (DLRS @ RecSys). arXiv:1606.07792 (the CTR wide-and-deep anchor)
Cho, E., Myers, S. A., & Leskovec, J. (2011). Friendship and mobility: User movement in location-based social networks. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) (pp. 1082–1090). https://doi.org/10.1145/2020408.2020579 (the original Gowalla check-in data, as distributed by SNAP; NGCF cites Liang et al. below for it — §1, §2)
Du, X., Yuan, H., Zhao, P., Qu, J., Zhuang, F., Liu, G., & Sheng, V. S. (2023). Frequency enhanced hybrid attention network for sequential recommendation (FEARec). In Proceedings of SIGIR. arXiv:2304.09184 (a full-ranking sequential model, §4)
Electronic Privacy Information Center (2010). Netflix cancels contest over privacy concerns. https://archive.epic.org/2010/03/netflix-cancels-contest-over-p.html (the lawsuit and the cancelled second contest, §7)
Ferrari Dacrema, M., Cremonesi, P., & Jannach, D. (2019). Are we really making much progress? A worrying analysis of recent neural recommendation approaches. In Proceedings of the 13th ACM Conference on Recommender Systems (RecSys). https://doi.org/10.1145/3298689.3347058 (tuned classical baselines beat most neural recommenders — §5)
Harper, F. M., & Konstan, J. A. (2015). The MovieLens datasets: History and context. ACM Transactions on Interactive Intelligent Systems, 5(4). https://doi.org/10.1145/2827872 (the MovieLens family — §1)
He, R., & McAuley, J. (2016). Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th International Conference on World Wide Web (WWW) (pp. 507–517). https://doi.org/10.1145/2872427.2883037 (the source NGCF cites for its Amazon-Book reviews, §1)
He, X., Deng, K., Wang, X., Li, Y., Zhang, Y., & Wang, M. (2020). LightGCN: Simplifying and powering graph convolution network for recommendation. In Proceedings of SIGIR. arXiv:2002.02126 (reuses NGCF’s split for Gowalla and Amazon-Book, with a revised Yelp2018 that NGCF’s later arXiv version also adopts; the de-facto graph-CF backbone)
He, X., Liao, L., Zhang, H., Nie, L., Hu, X., & Chua, T.-S. (2017). Neural collaborative filtering (NCF / NeuMF). In Proceedings of WWW. arXiv:1708.05031 (the leave-one-out + 99-sampled-negatives top-\(N\) protocol)
Hidasi, B., Karatzoglou, A., Baltrunas, L., & Tikk, D. (2016). Session-based recommendations with recurrent neural networks (GRU4Rec). In Proceedings of ICLR. arXiv:1511.06939 (session click-streams, Recall@20 / MRR@20, §4)
Hou, Y., Li, J., Fu, X., He, Z., Yan, A., Chen, X., & McAuley, J. (2026). Bridging language and items for retrieval and recommendation: Benchmarking LLMs as semantic encoders. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers (pp. 3251–3265). https://doi.org/10.18653/v1/2026.acl-long.147 (the Amazon Reviews 2023 release — 571.5M reviews, 33 categories, as its dataset page gives them; the release was first described in the 2024 preprint arXiv:2403.03952v1, whose title lacked the subtitle and had six authors)
Ji, Y., Sun, A., Zhang, J., & Li, C. (2023). A critical study on data leakage in recommender system offline evaluation. ACM Transactions on Information Systems, 41(3), 1–27. https://doi.org/10.1145/3569930 (splits that ignore the global timeline let a model see items from after the test point — the case for a global temporal split, §3)
Kang, W.-C., & McAuley, J. (2018). Self-attentive sequential recommendation (SASRec). In Proceedings of ICDM. arXiv:1808.09781 (leave-one-out + 100 sampled negatives; also introduced the Steam dataset)
Koren, Y. (2008). Factorization meets the neighborhood: A multifaceted collaborative filtering model. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) (pp. 426–434). https://doi.org/10.1145/1401890.1401944 (SVD++, §4)
Koren, Y. (2009). Collaborative filtering with temporal dynamics. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) (pp. 447–456). https://doi.org/10.1145/1557019.1557072 (timeSVD++, §4)
Koren, Y., Bell, R., & Volinsky, C. (2009). Matrix factorization techniques for recommender systems. Computer, 42(8), 30–37. https://doi.org/10.1109/MC.2009.263 (the rating-prediction MF line behind the Netflix Prize)
Krichene, W., & Rendle, S. (2020). On sampled metrics for item recommendation. In Proceedings of KDD. https://doi.org/10.1145/3394486.3403226 (the case against sampled metrics — why the field moved to full ranking)
Liang, D., Charlin, L., McInerney, J., & Blei, D. M. (2016). Modeling user exposure in recommendation. In Proceedings of the 25th International Conference on World Wide Web (WWW) (pp. 951–961). https://doi.org/10.1145/2872427.2883090 (the source NGCF cites for its Gowalla data, §1)
McAuley, J., Targett, C., Shi, Q., & van den Hengel, A. (2015). Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (pp. 43–52). https://doi.org/10.1145/2766462.2767755 (the Amazon Reviews 2014 release, the likely source of the trio’s Amazon-Book; NGCF names no year, §1, §2)
MLCommons (2024). Croissant: A metadata format for ML-ready datasets. MLCommons Croissant working group. https://docs.mlcommons.org/croissant/ (the dataset-metadata standard — §6)
Narayanan, A., & Shmatikov, V. (2008). Robust de-anonymization of large sparse datasets. In IEEE Symposium on Security and Privacy (the Netflix Prize de-anonymization — §7). https://doi.org/10.1109/SP.2008.33
Ni, J., Li, J., & McAuley, J. (2019). Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In Proceedings of EMNLP-IJCNLP (pp. 188–197). https://aclanthology.org/D19-1018/ (the Amazon Reviews 2018 release — 233.1M reviews)
Qiu, R., Huang, Z., Yin, H., & Wang, Z. (2022). Contrastive learning for representation degeneration problem in sequential recommendation (DuoRec). In Proceedings of WSDM (pp. 813–823). https://doi.org/10.1145/3488560.3498433 (a full-ranking sequential model, §4)
Ren, X., Wei, W., Xia, L., Su, L., Cheng, S., Wang, J., Yin, D., & Huang, C. (2024). Representation learning with large language models for recommendation (RLMRec). In Proceedings of the ACM Web Conference (WWW) (pp. 3464–3475). https://doi.org/10.1145/3589334.3645458 (the LLM-augmented row, and its own subsets of Amazon-book / Yelp / Steam with a 3:1:1 split, §4; Table 1 reports Recall and NDCG at 5, 10 and 20)
Salakhutdinov, R., Mnih, A., & Hinton, G. (2007). Restricted Boltzmann machines for collaborative filtering. In Proceedings of the 24th International Conference on Machine Learning (ICML) (pp. 791–798). https://doi.org/10.1145/1273496.1273596 (RBM-CF, §4)
Schedl, M. (2016). The LFM-1b dataset for music retrieval and recommendation. In Proceedings of the 2016 ACM International Conference on Multimedia Retrieval (ICMR) (pp. 103–110). https://doi.org/10.1145/2911996.2912004 (the LFM-1b release, §7)
Schedl, M., Brandl, S., Lesota, O., Parada-Cabaleiro, E., Penz, D., & Rekabsaz, N. (2022). LFM-2b: A dataset of enriched music listening events for recommender systems research and fairness analysis. In Proceedings of the ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR) (pp. 337–341). https://doi.org/10.1145/3498366.3505791 (the LFM-2b release, §7)
Sun, F., Liu, J., Wu, J., Pei, C., Lin, X., Ou, W., & Jiang, P. (2019). BERT4Rec: Sequential recommendation with bidirectional encoder representations from Transformer. In Proceedings of CIKM. arXiv:1904.06690 (leave-one-out + 100 popularity-sampled negatives)
Tang, J., & Wang, K. (2018). Personalized top-N sequential recommendation via convolutional sequence embedding (Caser). In Proceedings of WSDM (pp. 565–573). https://doi.org/10.1145/3159652.3159656 (its own 70/10/20 split, §4)
Wan, M., & McAuley, J. (2018). Item recommendation on monotonic behavior chains. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys) (pp. 86–94). https://doi.org/10.1145/3240323.3240369 (the UCSD Goodreads dataset)
Wang, H., Zhang, F., Xie, X., & Guo, M. (2018). DKN: Deep knowledge-aware network for news recommendation. In Proceedings of the 2018 World Wide Web Conference (WWW) (pp. 1835–1844). https://doi.org/10.1145/3178876.3186175 (a news-regime model, re-run on MIND, §4)
Wang, X., He, X., Cao, Y., Liu, M., & Chua, T.-S. (2019). KGAT: Knowledge graph attention network for recommendation. In Proceedings of KDD. arXiv:1905.07854 (its own knowledge-graph trio: Amazon-book / Last-FM / Yelp2018 — the same names as, but different files from, the NGCF trio, §4)
Wang, X., He, X., Wang, M., Feng, F., & Chua, T.-S. (2019). Neural graph collaborative filtering (NGCF). In Proceedings of SIGIR. arXiv:1905.08108 (released the random per-user 80/20 split that LightGCN reuses; the latest arXiv version, July 2020, replaces the Yelp2018 file, re-implements NDCG and reruns all methods, so quote any NGCF number with the version named; later graph-CF papers keep the dataset names but often re-split)
Wu, C., Wu, F., An, M., Huang, J., Huang, Y., & Xie, X. (2019). Neural news recommendation with attentive multi-view learning (NAML). In Proceedings of IJCAI (pp. 3863–3869). https://doi.org/10.24963/ijcai.2019/536 (a news-regime model, re-run on MIND, §4)
Wu, C., Wu, F., An, M., Huang, J., Huang, Y., & Xie, X. (2019). NPA: Neural news recommendation with personalized attention. In Proceedings of KDD (pp. 2576–2584). https://doi.org/10.1145/3292500.3330665 (a news-regime model, re-run on MIND, §4)
Wu, C., Wu, F., Ge, S., Qi, T., Huang, Y., & Xie, X. (2019). Neural news recommendation with multi-head self-attention (NRMS). In Proceedings of EMNLP-IJCNLP (pp. 6388–6393). https://doi.org/10.18653/v1/D19-1671 (a news-regime model, re-run on MIND, §4)
Wu, F., Qiao, Y., Chen, J.-H., et al. (2020). MIND: A large-scale dataset for news recommendation. In Proceedings of ACL. https://aclanthology.org/2020.acl-main.331 (the news-recommendation impression-based protocol)
Wu, J., Wang, X., Feng, F., He, X., Chen, L., Lian, J., & Xie, X. (2021). Self-supervised graph learning for recommendation (SGL). In Proceedings of SIGIR. arXiv:2010.10783 (Yelp2018, Amazon-Book and Alibaba-iFashion — no Gowalla; its much-quoted 7:1:2 is NGCF’s split restated, not a new one, §4)
Xie, X., Sun, F., Liu, Z., Wu, S., Gao, J., Zhang, J., Ding, B., & Cui, B. (2022). Contrastive learning for sequential recommendation (CL4SRec). In Proceedings of the IEEE International Conference on Data Engineering (ICDE) (pp. 1259–1273). https://doi.org/10.1109/ICDE53745.2022.00099 (full ranking over the whole item set; reuses S3-Rec’s preprocessing, §4)
Yu, J., Yin, H., Xia, X., Chen, T., Cui, L., & Nguyen, Q. V. H. (2022). Are graph augmentations necessary? Simple graph contrastive learning for recommendation (SimGCL). In Proceedings of SIGIR (pp. 1294–1303). https://doi.org/10.1145/3477495.3531937 (its own 7:1:2 split and Douban-Book, §4)
Zhao, W. X., et al. (2021). RecBole: Towards a unified, comprehensive and efficient framework for recommendation algorithms. In Proceedings of CIKM. arXiv:2011.01731 (the unified benchmark framework — §6)
Zhou, G., Zhu, X., Song, C., Fan, Y., Zhu, H., Ma, X., Yan, Y., Jin, J., Li, H., & Gai, K. (2018). Deep interest network for click-through rate prediction (DIN). In Proceedings of KDD (pp. 1059–1068). https://doi.org/10.1145/3219819.3219823 (Amazon Electronics, MovieLens-20M and an Alibaba log, not Criteo or Avazu, §4)
Zhou, K., Wang, H., Zhao, W. X., Zhu, Y., Wang, S., Zhang, F., Wang, Z., & Wen, J.-R. (2020). S3-Rec: Self-supervised learning for sequential recommendation with mutual information maximization. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management (CIKM) (pp. 1893–1902). https://doi.org/10.1145/3340531.3411954 (leave-one-out + 99 sampled negatives — the sampled row, not the full one, §4)
Zhu, J., Liu, J., Yang, S., Zhang, Q., & He, X. (2021). Open benchmarking for click-through rate prediction (FuxiCTR). In Proceedings of CIKM. arXiv:2009.05794 (standardized the Criteo / Avazu 8:1:1 split + AUC / LogLoss)
Ziegler, C.-N., McNee, S. M., Konstan, J. A., & Lausen, G. (2005). Improving recommendation lists through topic diversification. In Proceedings of the 14th International Conference on World Wide Web (WWW) (pp. 22–32). https://doi.org/10.1145/1060745.1060754 (the paper behind Book-Crossing; the data was crawled 2004)
Online sources and scale figures verified 17 October 2026, except the Yelp Data Agreement wording (shown at download, quoted as read in June 2026) and the Yahoo! Webscope terms, which were not re-read.