MIS 752 · Lab 8 · mikey-atwood
Answer the Survivor: what a reranker changes, measured
If you need support now, RAINN’s national hotline is 800-656-4673. A seven-page synthetic survivor guide generated inside the notebook, chunked, embedded and indexed; twenty rough candidates narrowed to three by a cross-encoder; and the answer with its source page cited. Both models run locally on a free CPU. Every figure in the guide is an invented placeholder, not guidance. $0. Measured in Google Colab; the numbers are this run's and will vary.
The 30-second read
THE DOCUMENT - written and rendered inside this notebook
source : PDF (reportlab wrote it, pypdf read it back)
pages : 7 chunks: 24 median chunk 287 chars
extraction : 100.0% of the authored sentences survived the PDF round trip
chunker used : YOUR chunk_pages() -- it passed the contract
THE TWO STAGES - both local, both $0
bi-encoder : sentence-transformers/all-MiniLM-L6-v2
cross-encoder : cross-encoder/ms-marco-MiniLM-L6-v2
index backend : faiss-cpu 1.15.1 IndexFlatIP (exact inner product)
pool -> prompt : 20 candidates retrieved, 3 promoted
cost per question: 23 ms retrieve + 818 ms rerank
WHAT THE RERANKER CHANGED - 5 answerable questions, scored against written gold judgements
answer was in the pool at all : 1.00 <- the ceiling
precision@3 vector -> reranked : 0.33 -> 0.33 (+0.00)
MRR vector -> reranked : 1.00 -> 1.00 (+0.00)
hit@3 vector -> reranked : 1.00 -> 1.00 (+0.00)
reranked order differs from the vector order on 5 of 6 questions
THE ANSWERS - nvidia/nemotron-3-ultra-550b-a55b:free, 18 scored calls of 18, all :free
total cost : $0.000000
ungrounded full_credit 0.00 · coverage 0.33 · cited 0% · unsafe 3 · doses stated on the unanswerable question 1
vector full_credit 1.00 · coverage 1.00 · cited 83% · unsafe 0 · doses stated on the unanswerable question 0
reranked full_credit 1.00 · coverage 1.00 · cited 83% · unsafe 0 · doses stated on the unanswerable question 0
unanswerable question: refused in 2 of 3 answered condition(s), invented a timeline in 1
What did the reranker actually change?
One dot per stage per question, on the same axis: the rank of the first passage that actually answers the question, inside the pool of 20 candidates. Lower is better, so a dot moving LEFT is a promotion. The number at the right of each row is the move. A dot parked at the far right never surfaced at all — and no reranker can rescue a passage stage 1 did not return. Rows are sorted by the size of the move.
How big should a chunk be?
The whole pipeline re-measured at 4 chunk sizes, pool fixed at 20. The red dashed squares are stage 1 alone; the solid blue circles are what the cross-encoder made of the same pool. The number above each pair is what the reranker added there, so the curve you are really reading is does the reranker help more when chunks are small and noisy, or large and vague. Chunk counts are labelled underneath. the pool of 20 held the answer on 100% of questions at EVERY chunk size, so the ceiling is flat at 1.00 -- it is in the table below, not on the chart, because a flat line shows nothing.
Side by side: report_not_required
No question changed rank in this run, so this one is shown to explain why stage 1 was already right. Gold answer is on page(s) 2. Left is what raw vector search surfaced; right is what the cross-encoder promoted from the same pool. page 6 covers mandatory reporting and shares the vocabulary. Only page 2 says the exam does not depend on a report.
1. page 2 · p2c2 · cosine 0.587 <== ANSWERS IT
The exam does not require a police report. A survivor can have the exam and decide later, or never, whether to report. Medical care and the forensic exam are offered together at designated hospitals, but a survivor can accept care without the evidence collection, or the other way round. A survivor c...
2. page 6 · p6c2 · cosine 0.562
If the patient is an adult, hospital staff make a report to police only with the patient's consent. An adult who comes in for care or an exam is not reported without being asked, and can change their mind about reporting later. Certain people at a school or a workplace may have their own duty to rep...
3. page 4 · p4c3 · cosine 0.484
If they already have, the exam is still worth having, and the examiner will note what changed. A survivor who is past the window can still get medical care and support from an advocate. Only the evidence collection is affected.
1. page 2 · p2c2 · logit +4.11, was vector rank 1 <== ANSWERS IT
The exam does not require a police report. A survivor can have the exam and decide later, or never, whether to report. Medical care and the forensic exam are offered together at designated hospitals, but a survivor can accept care without the evidence collection, or the other way round. A survivor c...
2. page 6 · p6c2 · logit +3.90, was vector rank 2
If the patient is an adult, hospital staff make a report to police only with the patient's consent. An adult who comes in for care or an exam is not reported without being asked, and can change their mind about reporting later. Certain people at a school or a workplace may have their own duty to rep...
3. page 2 · p2c1 · logit +1.60, was vector rank 14
A survivor can choose any combination of the following: get medical care, have a forensic exam, talk to an advocate, report to the police, or do nothing for now. These are separate choices. Making one does not commit anyone to another. A forensic exam is a medical visit in which a trained nurse exam...
Three conditions, one model, one system prompt
The only thing that changed between the three columns is which passages — if any — reached the model. Cost is $0 because every model here is on OpenRouter's free tier; it lives in the table rather than on an axis, because a cost axis that reads $0.000000 three times over shows nothing.
n full_credit coverage cited_rate unsafe unanswerable_doses mean_seconds total_cost_usd
condition
ungrounded 6 0.0 0.333 0.000 3 1 7.852 0.0
vector 6 1.0 1.000 0.833 0 0 2.920 0.0
reranked 6 1.0 1.000 0.833 0 0 5.797 0.0
The question the document cannot answer
How long will it take the district attorney to decide whether to file charges, and will my case go to trial?
[ungrounded] passages shown: [-] -> INVENTED A TIMELINE
I don't have the specific guide passages in front of me, so I'm answering from general knowledge (moderate confidence).
**Timeline for charging decisions:** This varies significantly by jurisdiction, case complexity, evidence availability, and the prosecutor's caseload. It can take anywhere from a few weeks to several months (sometimes longer). Some offices have specific policies or statutory deadlines, but there's no universal timeframe.
**Whether your case goes to trial:** Most criminal case
[vector] passages shown: [1,4,1] -> REFUSED
NOT IN DOCUMENT
[reranked] passages shown: [1,4,5] -> REFUSED
NOT IN DOCUMENT
Every question, before and after
question voice answerable gold_pages pool_rank vector_rank reranked_rank vector_pages reranked_pages precision_vector precision_reranked rr_vector rr_reranked hit3_vector hit3_reranked gold_in_pool vector_ms rerank_ms
report_not_required survivor True 2 1.0 1.0 1.0 2,6,4 2,6,2 0.3333 0.3333 1.0 1.0 1 1 1 21.1 781.7
exam_cost advocate True 3 1.0 1.0 1.0 3,2,2 3,3,2 0.3333 0.3333 1.0 1.0 1 1 1 20.3 779.3
collection_window advocate True 4 1.0 1.0 1.0 4,1,4 4,4,4 0.3333 0.3333 1.0 1.0 1 1 1 24.9 866.7
kit_storage survivor True 5 1.0 1.0 1.0 5,4,5 5,4,2 0.3333 0.3333 1.0 1.0 1 1 1 22.6 815.5
staff_reporting survivor True 6 1.0 1.0 1.0 6,6,2 6,6,2 0.3333 0.3333 1.0 1.0 1 1 1 22.8 819.6
prosecution_timeline survivor False - NaN NaN NaN 1,4,1 1,4,5 0.0000 0.0000 0.0 0.0 0 0 0 24.9 842.0
How this was measured
The reference document is generated inside the notebook: seven authored pages rendered to a real PDF with reportlab, then read back with pypdf (PDF (reportlab wrote it, pypdf read it back)). 100.0% of the authored sentences survived the round trip, and that number is measured at run time rather than assumed.
Chunking splits each page separately on sentence boundaries, so no chunk can span two pages and every chunk carries the page number it came from. That produced 24 chunks (median 287 characters). Passages were embedded with sentence-transformers/all-MiniLM-L6-v2 and stored in faiss-cpu 1.15.1 IndexFlatIP (exact inner product); the query is embedded with the same model and scored by cosine.
Stage 1 returns 20 candidates. Stage 2 re-scores each (question, passage) PAIR with cross-encoder/ms-marco-MiniLM-L6-v2 and promotes 3. The two score scales are kept in separate columns: cosine is bounded and comparable across queries, the cross-encoder logit is neither.
Retrieval is scored against six questions with written judgements (5 (page, phrase) pairs). A chunk counts as a hit only when its PAGE matches and the gold PHRASE appears in its text, so retrieving the right page with the wrong passage does not score. One of the six is deliberately unanswerable and is scored separately.
Generation used nvidia/nemotron-3-ultra-550b-a55b:free through OpenRouter at temperature 0, 18 scored calls of 18 attempted, total measured cost $0.000000. Every score comes from a rule printed in the notebook: must_include synonym groups, must_not_include banned claims, and a citation regex over the model's own output.
This is a seven-page synthetic survivor guide with a six-question gold set: a demonstration of method, not a validation of any survivor information. Every figure in it (the 72- and 120-hour windows, the 24-month storage period, the cost and consent rules) is an invented placeholder, so nothing here is legal or medical guidance and nothing in it should be acted on. It contains no real survivor data. Numbers are from one run and will vary; the pattern is the finding, not the digits.