scrapr.site

A researcher’s score is the relevance of their single strongest publication. Titles provide a concise signal; abstracts carry more topical information.

The 40–60 weighting reflects a practical finding: abstracts contain more topical signal than titles. Exact keyword matches then act as a modest multiplier—strong enough to preserve literal intent without replacing the semantic comparison.

Reconstructing OpenAlex abstracts

OpenAlex stores abstracts as inverted indexes rather than plaintext. To read one, Scrapr flattens every word-position pair, sorts by position, and joins the words.

Input: { "machine": [0, 5], "learning": [1, 6], "is": [2],
         "great": [3, 7], "and": [4] }

01  Flatten → [(machine,0), (learning,1), (is,2), (great,3),
               (and,4), (machine,5), (learning,6), (great,7)]
02  Sort by position
03  Join → "machine learning is great and machine learning great"

This runs on every work during discovery, potentially hundreds of times per search, so its performance matters.

The client gathers a CV, institution, and research interest. The server resolves identity, widens the query, retrieves candidates, verifies current affiliation, and only then pays the cost of semantic ranking.

async function getInstitutionId(
  collegeName: string
): Promise<string | null> {
  const variations = getCollegeVariations(collegeName);

  for (const name of variations) {
    const response = await fetch(
      `${OPENALEX_INSTITUTIONS_API}?search=${name}&per-page=1`
    );
    const data = await response.json();
    if (data.results?.[0]) return data.results[0].id;
  }

  return null;
}

getCollegeVariations() handles the asymmetry between names like “MIT” and “Stanford University” by trying raw, expanded, and stripped forms. Common shorthand such as “CMU” works; highly informal abbreviations remain an unresolved edge case.

Query expansion turns a narrow term such as “optogenetics” into five to eight related academic terms. Three retrieval paths run against OpenAlex and accumulate candidates into one shared author map.

The hot inner loop

addAuthorsFromWork() runs for every work returned by every strategy. It uses a cheap keyword score to choose which publication represents a researcher before the full embedding stage.

// For each authorship on a work:
// 1. Skip the work when its title matches an exclusion keyword
// 2. Score paper relevance with the fast keyword method
// 3. Match institution lineage or a directly discovered author ID
// 4. Apply the optional lab-name filter
// 5. Upsert the author; keep the more relevant publication
// 6. Add up to three recent works, deduplicated by title

Affiliation is a correctness gate

A work-level affiliation only proves where an author was when a paper was published. Scrapr checks last-known institutions so a former postdoc does not continue to appear in a university’s results years after leaving.

const atInstitution = (author.last_known_institutions || []).some(
  (institution: Institution) =>
    institution?.id === institutionId ||
    institution?.lineage?.includes(institutionId)
);

The lineage condition preserves legitimate matches when a department is a child of the university—for example, an EECS department nested beneath MIT.

The precision layer

After retrieval can produce more than 75 candidates, relevancy.ts re-scores each researcher using dense vector similarity. Embeddings are cached by the first 200 characters of input.

async function getEmbedding(text: string) {
  const cacheKey = text.slice(0, 200);
  if (embeddingCache.has(cacheKey)) {
    return embeddingCache.get(cacheKey)!;
  }

  const response = await client.embeddings.create({
    model: "text-embedding-3-small",
    input: text,
    encoding_format: "float",
  });

  const vector = response.data[0].embedding;
  embeddingCache.set(cacheKey, vector);
  return vector;
}

function cosineSimilarity(a: number[], b: number[]) {
  let dot = 0, magA = 0, magB = 0;

  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    magA += a[i] * a[i];
    magB += b[i] * b[i];
  }

  if (magA === 0 || magB === 0) return 0;
  return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}

Each vector occupies roughly 6 KiB. For 75 researchers, title and abstract comparison requires at most 150 network calls before cache hits; the query embedding is computed once. A first-200-character cache collision is possible, though unlikely for academic text.

Every fallback protects a different property: availability, relevance quality, affiliation correctness, or persistence.

checkRateLimit("search")
  → allowed? YES → write incremented counter → do work
  → allowed? NO  → return 429

A search is observable from institution resolution to the final ranked list. The terminal below replays the structured log sequence from a representative MIT query.

The critical path is the embedding phase. At roughly 100–300 ms per call, 150 title-and-abstract requests in five-way parallel batches take about six seconds in the best case.