- Recommendations without a citation graph
For the 02807 group project, five students (Group 36) built a paper-recommendation system that never looks at citations: embed every abstract, connect papers whose embeddings sit close enough together, and let graph communities stand in for research topics. Query time becomes a graph lookup: the report's phrase for it was trading time complexity for memory complexity.
- The pipeline in one line each
Stream the arXiv metadata snapshot (Kaggle, CC0) → filter to recent papers with substantial abstracts (year >= 2024, >= 200 words; 148,477 papers) → embed each abstract with Google's Universal Sentence Encoder into 512 dimensions → build an HNSW index for fast approximate nearest neighbours → threshold the k-NN distances at tau to get edges → run Louvain for communities → recommend a clicked paper's nearest neighbours, tagged 'significant' when they share its community.
- The tau decision
tau came from the k-NN distance histogram: pick candidate percentiles (keep 5-25% of neighbour links), then let modularity judge. The catch the project hit: modularity is highest at tau=0.19 (Q=0.934), but that graph is 133,006 communities, most of them a single paper. The report chose tau=0.27 instead: slightly lower Q, vastly more useful communities. The slider above lets you feel that trade-off on live data.
- Click a paper
Recommendation is the HNSW lookup replayed: the five nearest neighbours by cosine distance, each tagged with whether it landed in the same Louvain community ('significant', in the project's vocabulary). Every node links to its real paper on arxiv.org.
- The weekly from-scratch algorithms
02807's exercises built the same machinery by hand. Three of David's are live here: A-priori frequent-pair mining with support and lift on 14,963 real grocery baskets; Girvan-Newman community detection by repeatedly cutting the highest-betweenness edge of the karate-club graph; and the spectral view of the same graph: Laplacian eigenvalues, the spectral gap, and the Fiedler vector's sign cut.
- Rebuilt for this page (2026-09-01)
This page re-runs the group's real pipeline code path at build on a 2,500-paper stratified subsample (the same Kaggle snapshot, grown to 240,061 filtered papers by now), embedded with the same USE v4 model. Two disclosed substitutions: exact brute-force k-NN replaces hnswlib at this scale (same cosine metric, zero approximation), and node positions come from a build-time t-SNE (the original had no 2-D layout). The TS ports (Louvain, modularity, tau-from-percentile, A-priori, betweenness, a Jacobi eigensolver) were written with AI coding tools and are fixture-tested against the Python/NetworkX pipeline. The arxiv_semantic_graph package is the group's shared work; the story here doesn't assign parts.