Kierston
Kierston generates a synthetic urban population and simulates resident activity at minute resolution. It records simulation ground truth and the corresponding evidence available to individual record holders. A city profile specifies the geographic extent, census microdata area and legal jurisdiction. The default configuration uses a 5 km extent around Allentown, Pennsylvania, with approximately 46,000 households and 125,000 residents. For each citizen, engine generates various personal aspects and records.
Simulation Aspects
| Domain | Simulated |
|---|---|
| Population | Households, Demographics, Jobs and employers, School enrolment, Family relationships, Friends, neighbours and colleagues, Church, club, union and team memberships |
| Records | Social Security and birth records, Immigration documents, Driver licences, Criminal court cases, Criminal charges, Divorces and other civil cases, Vehicle registrations, Mortgages and loans, Bank accounts, Credit and debit cards, Property deeds, Gun purchases, Prescriptions, Mobile phones, Phone numbers and carriers, WhatsApp and other messaging accounts |
| Places and vehicles | Homes, Workplaces, Schools, Shops, restaurants and other venues, Street network, Cell towers and antennas, Licence-plate cameras, Taxis, School buses, Police cars, Ambulances |
| Daily routines | Work shifts, School days, Sleep, Shopping and errands, Gym, club and church visits, Meet-ups with friends, Family outings |
| Travel | Car trips on real streets, Walking, cycling and transit, Shared household cars, Parked cars, School bus runs, Taxi rides, Commuters from outside the city, Deliveries, Visitors, Traffic passing through |
| Emergency services | 911 calls, Areas patrolled by police cars, Dispatch of the nearest unit, Ambulance trips to hospital |
| Money | Card payments, Cash purchases, Fares, Parking fees, Rent, mortgage and utility bills, Wages and benefits, Running bank balances, Declined payments |
| Messages and calls | Phone calls, Text messages, App messages, Group chats, Replies |
| Phone network | Phone-to-cell connections by minute, Cell changes during calls, Cell changes while idle, Network events |
| Plate cameras | Plate reads, Missed vehicles, Misread plates, Unreadable plates |
| Staged scenarios | Staged incidents, Re-planned days of participants |
KierstonBench evaluates investigative reasoning over the generated records. Each task provides a case description and access to the Kierston Data Interface, a command-line query interface over provider stores. Tasks require identification of persons, locations or events across the full population, without a candidate list. Some cases are unanswerable because the simulation produces insufficient evidence; these cases test whether an agent abstains rather than giving an unfounded answer. Agents may explicitly abstain; aggregate scores assign abstentions a value of zero.
Population activity
The map displays activity for Friday, 4 September 2026, using a sample of 2,500 residents who travelled that day, together with all 74 plate readers and 198 cell towers. Selecting a resident, reader or tower displays the corresponding detail. The adjacent charts report calls, text messages and transactions across the full simulated population.
Simulation architecture
Kierston initializes a synthetic population from American Community Survey Public Use Microdata Sample (ACS PUMS) household records and assigns it to the OpenStreetMap dwelling stock. Initialization generates employment and school assignments, kinship and social ties, organisation memberships, administrative records, financial accounts, vehicles, handsets and carrier subscriptions. These entities form the static state used by the daily simulation.
Each simulated date advances the city from an opening state to a closing state. Activity subsystems produce scheduling intents for work, school, sleep, errands and social interactions. The scheduler resolves these intents into stays and routed travel legs. Car routing uses A* over the road graph, with edge costs derived from distance and speed limits or road-class defaults. Shared activities derive their timing and participants from common objects established before individual schedules are planned. Fleet matching and emergency dispatch operate across the full city.
The completed schedules drive transaction settlement, communications, cellular attachment and automatic licence plate recognition (ALPR) observations. Each daily export contains simulation ground truth, provider-visible evidence, and opening and closing state. The records interface exposes provider records without simulation identifiers or ground truth. The following maps show venue activity, fleet operations and external traffic for Friday, 4 September 2026.
Points of interest
The spatial dataset contains 5,656 points of interest, of which 2,162 serve as activity destinations, including retail, food services, healthcare, education, religious facilities, parks and fuel stations. Discretionary visits are generated from activity purpose, schedule constraints, route proximity and habitual venue preferences. On 4 September, residents generated 93,888 visits across 1,231 destinations. Marker size encodes arrivals within the preceding ten minutes. Selecting a destination displays its daily activity profile.
Taxis and school buses
The initialized fleet contains 146 taxis and 232 school buses. Taxi matching assigns passenger requests across the fleet and produces consistent driver and passenger schedules. On 4 September, 87 taxis served 2,275 fares. School buses operate fixed routes and stops, with arrival times constrained by school schedules and shared timing for drivers and pupils. That day, 144 buses operated. The map displays occupied taxis in red, unoccupied taxis in grey and school buses in orange.
External traffic
External traffic comprises commuters, deliveries, repeat visitors, single-day visitors and through traffic. Vehicles enter and exit through gateways on major roads at the simulation boundary, use the resident road network and contribute to ALPR observations. Commuters, deliveries and repeat visitors retain vehicle identities across dates; single-day visitors and through traffic are generated per date. Registration-state sampling is conditioned on approach direction. On 4 September, 41,103 external vehicles generated 49,513 trips, predominantly through traffic. The map displays a sample of these trips. Boundary marker size encodes gateway traffic volume; selecting a gateway displays its counts.
Resident profiles
The selected residents collectively cover the exported record categories. Each profile includes an activity timeline, carrier observations aligned with handset ground-truth positions, vehicle plate observations, social relationships and administrative records.
Benchmark
KierstonBench contains 32 task families in three groups. Each task is generated from one simulated day and consists of a case description, the correct answer and a scoring rule. The agent receives the case description and access to the records through a command-line interface, and writes its answer to a file. Answers refer to persons, vehicles, places or times within the full simulated population.
Technical Tasks
Technical tasks pose a single question about the records of an unmodified simulated day, such as the holder of a phone subscription or the workplace of a resident.
| Device to person | Who carries a given phone subscription |
| Counterparty recon | Who is on the other end of a call record |
| Account resolution | Who controls a messaging account |
| Household recon | Who lives with a given person |
| Kinship infer | A named relative of a given person, such as their mother |
| Plate reconciliation | The real plate behind a set of misread camera reads |
| Home id | Where a person lives |
| Workplace id | Where a person works, and for which employer |
| Transaction to visit | Which venue visit a card payment belongs to, and when it began |
| Payer infer | Who paid for a group outing |
| Point in time location | Where a person was at a given minute |
| School escort | A child's school and who usually takes them there |
| Trip mode | How a journey was made: car, carpool, bus, walking, cycling or taxi |
| Origin destination | Where a car trip seen once by a camera started and ended |
| Relationship type | Every tie between two people: family, partner, colleague, neighbour |
| Organization membership | Which churches, clubs and unions a person belongs to |
| Person reid | Whose day produced an anonymous trace of rounded locations |
| Vehicle occupancy | How many people were in a car at a given minute, and who |
| Shared travel | Who travelled together, and who drove |
| Meeting recon | Who attended a gathering at a venue, and when it started and ended |
| Colocation | Everyone who spent at least 15 minutes near a person |
| Conversation classif | Whether a message thread is between two people, a group or a broadcast, and what it is about |
Real-Time Investigations
Real-time investigations add a staged incident to a simulated day and present the case during or shortly after the incident. The records available to the agent end at the time stated in the case description.
| Mass shooting | Who fired inside a bar, 10 to 20 minutes after the shots |
| School shooting | Who fired inside a school |
| Shots fired | Who fired on a street corner, with a gun owner nearby as a decoy |
| Explosion | Who left a device at a workplace |
| Abduction | Who was in the car that took a named victim |
| Protest members | Who is taking part in a protest, as opposed to passing by |
| Mass gathering | Whether an unplanned crowd has formed anywhere in the city, and where |
Deception Tasks
Deception tasks add a staged incident together with a false account of it from the person who reports it to 911. In each family the person reporting the incident is responsible for it, and the task is to identify that person from the records.
| Stabbing | A man reports finding his girlfriend stabbed at home |
| Swatting | A caller reports an armed break-in at someone else's home |
| Disappearance | A man reports his girlfriend missing |
Agent inputs
Each case description states the date, the question and the required answer format. Real-time investigations and deception tasks also state the current time, which is the end of the available records. The records interface is presented to the agent as the Allentown Data Interface (ADI).
A mobile subscription was active in the city on 2026-08-31: IMSI [IMSI], MSISDN [PHONE], carrier Metro by T-Mobile. Which resident carries this subscription? Answer with a ranked list of person identities, most likely first.
Shots were fired inside ([LOCATION]) between 08:04 and 08:08 today.
There are multiple casualties.
It is now 08:19 on 2026-09-04.
Rank the most likely suspects, best first. Answer with person identities.ADI provides the records held by public agencies, courts, banks, phone carriers and licence-plate camera operators, grouped as follows.
| Identity | Social Security, immigration documents, education, military service |
| Vehicles | Registrations, driver licences, licence-plate camera reads |
| Phones | Subscriptions, handsets, calls, texts and which cell tower served each phone |
| Money | Bank accounts, cards, loans and every card and bank transaction |
| Courts | Criminal and civil cases, charges, prison records |
| Other | Gun purchases and carry licences, prescriptions, property deeds, online accounts |
Agent environment
Each task runs in a new container with a shell, Python and the adi command-line client, together with two map tools for address lookup and nearby places. A query specifies a record source, typed filters on fields, time or distance from a point, and a stated reason, and returns all matching records. In real-time investigations and deception tasks, queries return only records created before the case time.
adi query carriers network_events \
--reason "phones served near the venue during the shooting" \
--where occurred_at.gte=2026-09-04T07:50:00-04:00 \
--where occurred_at.lt=2026-09-04T08:19:00-04:00 \
--near [LOCATION],400 \
--select occurred_at,imsi,event_type,cell_cgiThe agent writes its answer as a JSON file. Persons may be identified by Social Security number, by name and date of birth, or by name alone; credit for a correct identification and the penalty for an incorrect one increase with its specificity. The agent may instead declare the case undetermined, which scores zero. Some cases contain insufficient evidence to determine the answer.
Case replay: mass shooting
The replay shows the simulation ground truth for one mass shooting case from the results. It includes the shooter, the people in the bar, passers-by and the police officers sent to the scene, each moving along their simulated route. Selecting a person on the map or in the table displays their actions. The case description given to the agents appears above the map, and the records available to them end at 03:39.
Results
Scores range from −1 to 1. Identity scoring assigns credit according to identifier specificity and applies corresponding penalties to incorrect identifications. Scores marked c include manual corrections for valid answers that the automated scorer did not recognise. The overall and group scores average all task attempts, with abstentions and attempts without a score assigned zero. The answered score excludes explicit abstentions; failed attempts remain in the denominator with a value of zero. Abstention rates are reported separately.
| Model | TT | RTI | DT | Abstained | Time / task | Cost / task | Score |
|---|---|---|---|---|---|---|---|
| GPT-6 Sol | 0.67 | 0.00 | 1.00 | 41% | 2m 49s | $0.29 | 0.55 |
| Claude Fable 5.1 | 0.75 | -0.38 | 1.00 | 6% | 5m 10s | $1.55 | 0.52 |
| GPT-6.1 Sol | 0.58 | 0.00 | 0.67 | 47% | 4m 44s | $0.23 | 0.46 |
| GPT-6 Astra | 0.53 | 0.14 | 0.33 | 47% | 2m 35s | $0.95 | 0.43 |
| Claude Opus 5.5 | 0.63 | -0.56 | 0.83 | 0% | 4m 04s | $0.75 | 0.39 |
| GPT-5.6 Sol | 0.62 | -0.57 | 0.67 | 0% | 5m 05s | $0.85 | 0.36 |
| Claude Opus 5 | 0.61 | -0.55 | 0.67 | 0% | 8m 33s | $1.92 | 0.36 |
| Claude Sonnet 5.5 | 0.41 | -0.58 | 1.00 | 9% | 1m 53s | $0.22 | 0.25 |
| GPT-6 Luna | 0.21 | 0.00 | 0.00 | 69% | 59s | <$0.01 | 0.15 |
TT: technical tasks · RTI: real-time investigations · DT: deception tasks · group scores count abstentions as zero
Abstentions
| Model | TT | RTI | DT | All | Answered | All tasks |
|---|---|---|---|---|---|---|
| Claude Fable 5.1 | 2 / 22 | 0 / 7 | 0 / 3 | 2 / 32 (6%) | 0.56 | 0.52 |
| Claude Opus 5.5 | 0 / 22 | 0 / 7 | 0 / 3 | 0 / 32 (0%) | 0.39 | 0.39 |
| Claude Opus 5 | 0 / 22 | 0 / 7 | 0 / 3 | 0 / 32 (0%) | 0.36 | 0.36 |
| Claude Sonnet 5.5 | 3 / 22 | 0 / 7 | 0 / 3 | 3 / 32 (9%) | 0.28 | 0.25 |
| GPT-6 Sol | 7 / 22 | 6 / 7 | 0 / 3 | 13 / 32 (41%) | 0.93 | 0.55 |
| GPT-6.1 Sol | 7 / 22 | 7 / 7 | 1 / 3 | 15 / 32 (47%) | 0.86 | 0.46 |
| GPT-6 Astra | 7 / 22 | 6 / 7 | 2 / 3 | 15 / 32 (47%) | 0.80 | 0.43 |
| GPT-5.6 Sol | 0 / 22 | 0 / 7 | 0 / 3 | 0 / 32 (0%) | 0.36 | 0.36 |
| GPT-6 Luna | 12 / 22 | 7 / 7 | 3 / 3 | 22 / 32 (69%) | 0.47 | 0.15 |
Score against cost and time
Score by task group
By task family
| Family | Claude Fable 5.1 | Claude Opus 5.5 | Claude Opus 5 | Claude Sonnet 5.5 | GPT-6 Sol | GPT-6.1 Sol | GPT-6 Astra | GPT-5.6 Sol | GPT-6 Luna |
|---|---|---|---|---|---|---|---|---|---|
| Technical Tasks | |||||||||
| Account resolution | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| Colocation | abstained | 0.02 | 0.06 | 0.00 | abstained | abstained | abstained | -0.05 | abstained |
| Conversation classif | 0.50 | 1.00 | 0.50 | 0.50 | 1.00 | 1.00 | 0.00 | 1.00 | abstained |
| Counterparty recon | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| Device to person | 0.50 | 0.50 | 1.00 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| Home id | 1.00 | 1.00 | 1.00 | 1.00 | abstained | 1.00 | abstained | 1.00 | abstained |
| Household recon | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | abstained |
| Kinship infer | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | -1.00 |
| Meeting recon | -0.07 | -0.13 | -0.08 | -0.04 | abstained | -0.50 | -0.50 | -0.04 | abstained |
| Organization membership | abstained | 0.00 | 0.00 | abstained | abstained | abstained | abstained | 0.00 | abstained |
| Origin destination | 1.00 | 0.90 | 0.90 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.00 |
| Payer infer | 0.00 | 0.50 | -1.00 | -1.00 | abstained | abstained | 1.00 | 1.00 | abstained |
| Person reid | 1.00 | -1.00 | -0.60 | -1.00 | 1.00 | 1.00 | 1.00 | -1.00 | abstained |
| Plate reconciliation | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| Point in time location | 1.00 | 1.00 | 1.00 | 0.00 | 1.00 | 1.00 | abstained | 1.00 | 0.00 |
| Relationship type | 1.00 | 1.00 | 1.00 | abstained | abstained | abstained | abstained | 1.00 | abstained |
| School escort | 1.00c | 1.00 | 1.00c | -0.50 | 1.00 | abstained | abstained | 1.00 | abstained |
| Shared travel | 0.50 | 0.50 | 0.37 | abstained | abstained | abstained | abstained | -1.00 | 0.00 |
| Transaction to visit | 1.00 | 0.67 | 1.00 | 0.67 | 0.67 | 0.67 | 0.67 | 0.67 | 0.67 |
| Trip mode | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | abstained | 1.00 | 0.00 | abstained |
| Vehicle occupancy | 1.00 | -0.08 | 0.18 | 1.00 | 1.00 | 0.50 | 0.50 | 1.00 | abstained |
| Workplace id | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| Real-Time Investigations | |||||||||
| Abduction | -1.00 | -1.00 | -1.00 | -1.00 | abstained | abstained | abstained | -1.00 | abstained |
| Explosion | no-answer | -1.00 | -1.00 | -1.00 | abstained | abstained | abstained | -1.00 | abstained |
| Mass gathering | 1.00 | 1.00 | 1.00 | 1.00 | 0.00 | abstained | 1.00 | 1.00 | abstained |
| Mass shooting | -1.00 | -1.00 | -1.00 | -1.00 | abstained | abstained | abstained | -1.00 | abstained |
| Protest members | 0.36 | 0.10 | 0.13 | -0.04 | abstained | abstained | abstained | -0.02 | abstained |
| School shooting | -1.00 | -1.00 | -1.00 | -1.00 | abstained | abstained | abstained | -1.00 | abstained |
| Shots fired | -1.00 | -1.00 | -1.00 | -1.00 | abstained | abstained | abstained | -1.00 | abstained |
| Deception Tasks | |||||||||
| Disappearance | 1.00 | 1.00 | 0.00 | 1.00 | 1.00 | 1.00 | abstained | 0.00 | abstained |
| Stabbing | 1.00 | 0.50 | 1.00 | 1.00 | 1.00 | abstained | abstained | 1.00 | abstained |
| Swatting | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | abstained |