How I used Claude to match 200 Clinical Trials to 700 PubMed Papers¶

Claude Code is a powerful general-purpose coding agent that can design and execute multi-stage data pipelines. However, it's commonly tripped up by messy real-world data operations problems: I have two tables to merge, with no common key, and matching requires a deep understanding of the subject matter.

But with a tool purpose-built for large scale data operations like merging, we get shockingly better results.

This notebook compares what Claude Code does versus what the everyrow SDK can do.

You can also scroll down to see how to reproduce these results yourself.

Approach 1: Claude Code¶

When given a table of clinical trials and a table of PubMed papers, then tasked with finding which paper(s) report results for which trial, Claude Code independently devised and executed a multi-stage strategy:

Phase 1: TF-IDF pre-filtering. Built TF-IDF text representations for all trials and papers, computed cosine similarity, and selected the top-15 candidate papers per trial. This narrowed the 200 x 700 = 140,000 possible pairs down to ~3,000 candidates.

Phase 2: Direct NCT ID matching. Searched paper abstracts for explicit NCT ID mentions using regex. Found 8 papers that directly cite their trial’s identifier.

Phase 3: 8 parallel LLM subagents. Split the 200 trials into 8 batches of 25. Each subagent received its batch of trials plus TF-IDF candidate papers, and assessed whether candidates were genuine matches by checking alignment of interventions, conditions, study design, sponsors, endpoints, and institutions.

Claude Code Results: 200 trials + 700 papers

Claude Code achieved perfect precision: every match it reported was correct. But it missed over 40% of the true matches. The conservative TF-IDF pre-filtering and high-confidence thresholds meant many genuine but harder-to-detect links were never surfaced.

Approach 2: everyrow SDK¶

When given the same task with everyrow, the entire merge is a single function call. Behind the scenes, merge() orchestrates hundreds of LLM agents that build an understanding of each trial’s key attributes, search through the paper pool for semantic matches, and verify candidates with detailed reasoning.

Since multiple papers can report results from the same trial (e.g., primary results and follow-up analyses), this is a many-to-one mapping: papers (left) map to trials (right). Getting the table orientation right matters: the left table is the "many" side.

Comparing Results¶

How Performance Scales¶

We ran the same experiment at a smaller scale (200 trials + 200 papers) to see how both approaches respond to growing data.

At 200 + 200 papers, Claude Code independently chose a similar strategy: regex NCT ID extraction (3 matches), followed by 8 parallel subagents using keyword-based search (Grep) on a papers text file. Each agent handled 25 trials across 200 papers.

The key finding: everyrow dynamically scales its resources to match the problem, maintaining accuracy as datasets grow. Claude Code allocated exactly 8 subagents regardless of whether the paper pool contained 200 or 700 papers. Each agent handled 25 trials, the same workload whether it was searching through 200 or 700 candidates. As the dataset grows, each agent's search space expands but its compute budget doesn't.

A pattern emerges:

• everyrow's F1 held steady (85.7% to 87.2%) as the paper pool grew by a factor of 3.5. It allocated proportionally more resources to handle the larger search space.
• Claude Code's F1 degraded (78.8% to 74.5%). It used the same 8 subagents and TF-IDF top-15 filtering regardless of dataset size, so each agent's search became less thorough.
• Claude Code's cost decreased slightly ($15 to $12). This is the signature of a fixed-budget approach: the same compute is spent whether the problem is small or large.
• everyrow's cost scaled with the problem ($6 to $18). The extra spend went directly toward maintaining quality at scale.

At 200 + 200, everyrow was cheaper, faster, and more accurate. At 200 + 700, the accuracy gap widened. Extrapolating to larger datasets (thousands of trials and papers), we would expect the gap to grow further: Claude Code's fixed 8-agent budget would be spread even thinner, while everyrow would continue to scale its orchestration, thus nicely complementing Claude Code.

Key Takeaways¶

1. Specialized orchestration beats general-purpose agent planning for data operations at scale. everyrow's merge() is purpose-built to decompose a large matching problem into hundreds of parallel agent tasks, with intelligent candidate selection and verification. Claude Code is remarkably clever: it independently invented a TF-IDF + parallel-subagent pipeline. But a general-purpose coding agent can't match a system designed specifically for this class of problem.

2. The recall gap is the key differentiator. Claude Code achieved perfect precision (100%). Every match it reported was correct. But it only found ~59% of the true links. everyrow's higher F1 comes from substantially better recall: it surfaces matches that a fixed-budget approach misses.

3. Fixed compute means quality suffers as the scale grows. Claude Code used 8 subagents for both the 200-paper and 700-paper runs. This is a natural consequence of how coding agents plan: they estimate a reasonable level of parallelism and stick with it. As the dataset grows, each agent's workload increases but the total compute stays constant. EveryRow, by contrast, scales its agent count to the problem.

4. Cost is about what you get for your money. At the 200 + 700 scale, everyrow cost ~$18 and achieved 87.2% F1. Claude Code cost ~$12 and achieved 74.5% F1. The relevant metric isn't raw cost but cost per unit of quality. The ~$6 in savings from Claude Code comes at the price of missing 40%+ of the true matches.

5. Claude Code can use the everyrow SDK. Claude Code is an excellent coding agent already. Everyrow doesn't replace Claude Code. Rather, it complements Claude Code, allowing it to be even more capable!

Reproduce It Yourself¶

This dataset (200 trials + 700 papers) was sized to fit within everyrow's free-tier credits (~$20). To reproduce:

1. Run the cells below to execute the everyrow merge and score it
2. To test Claude Code, give it the trials_200.csv and papers_700.csv files and ask it to match papers to trials
3. Save Claude Code's predictions as a CSV with nct_id and pmid columns, then score them with the code below.

The Task¶

ClinicalTrials.gov maintains structured metadata for clinical trials: conditions studied, interventions tested, outcomes measured, sponsors, and timelines. When a trial's results are published, the publication is linked to the trial record.

PubMed papers describe the same studies in natural language: titles and abstracts discuss the intervention, patient population, endpoints, and findings.

The matching problem: given a trial's structured metadata and a paper's title + abstract, determine whether the paper reports results from that trial. This requires:

• Recognizing that a brand name in a trial record (e.g., "PGL4001") maps to a generic name in a paper (e.g., "ulipristal acetate")
• Matching disease terminology across ontologies (e.g., "uterine myomas" vs. "uterine fibroids")
• Distinguishing studies with similar interventions but different populations or designs
• Linking sponsor organizations, outcome measures, and study timelines across formats

Dataset¶

The evaluation dataset was constructed as follows:

1. Trials with known links: We queried the ClinicalTrials.gov API for completed trials that have RESULT or DERIVED reference types pointing to PubMed IDs. These known links serve as gold labels.

2. PubMed papers: We scanned the PubMed 2019 baseline JSONL (~20 GB, ~19M papers) to extract:

   • Gold papers: those whose PMIDs match the trial references
   • Distractor papers: reservoir-sampled English-language papers with abstracts >= 100 characters, at a 10:1 distractor-to-gold ratio

3. Subsampling: To keep this example reproducible within everyrow's free-tier credits, we subsampled to 200 trials and 700 papers (64 gold papers + 636 distractors).

A note on gold labels: gold_labels_200.csv contains 64 (nct_id, pmid) pairs. All 64 gold PMIDs are present in papers_700.csv, so all pairs are achievable. If you use a smaller paper subset, you must filter the gold labels to only pairs whose PMID is in your paper set. Otherwise, you'll penalize recall for matches that are impossible to find. The score_from_csv helper below handles this automatically.

Merge Using everyrow¶

Scoring a Merge¶

We evaluate merge quality using standard information retrieval metrics on the set of predicted (nct_id, pmid) pairs:

• Precision = correct pairs / predicted pairs. In other words, of the matches we found, how many are real?
• Recall = correct pairs / gold pairs. In other words, of the real matches, how many did we find?
• F1 = harmonic mean of precision and recall

A system with perfect precision but low recall is too conservative: it only reports matches it's certain about, but misses many real links. In practice, missed links are harder to recover than false positives (which can be reviewed), so recall matters.

Head-to-Head: 200 Trials × 700 Papers¶