Ballot Image Audit Report

San Francisco, California, 2024 General Election

AuditEngine version:'v2.32.X'

Run Date: 2026-11-10 21:56:29.053735 Copyright (c) Citizens' Oversight

Return to Master Report Index

Summary

This report gives the results of an audit of the ballot images of the San Francisco, California, 2024 General Election. Citizens' Oversight Projects (COPs) performed the audit with AuditEngine, its ballot image auditing system.

AuditEngine reads each ballot image on its own. It then compares its reading with the cast vote record (CVR), the voting system's record of each ballot, one ballot at a time. In this report, CVR always means cast vote record, never Conditional Voter Registration.

Comparing two independent readings can expose errors in either one that would be very hard to find otherwise. Voting systems are usually accurate. Still, a mistake, a software update, a change in the election definition or malicious activity can change a result.

The report is meant for election officials and for the public. It shows how far the official result can be checked from the ballot images.

This report is produced from a fixed template. Its text was written and reviewed in advance. The numbers are filled in from the audit's own tables, which are published in the Reconciliation Report. Terms such as Non-Variant Sheets and Contest Variants are defined there.

Key findings

• AuditEngine and the CVR agree in 99.6% of the 3,878,114 Ballot-Contests audited, with no variation of any kind.
• In 1 contests, the disagreements and overvotes are large compared with the margin, even leaving out those that involve a write-in. These need a closer look, ballot by ballot.
• In 2,299 Ballot-Contests, the voting system first read the vote differently from AuditEngine. Election staff then adjudicated them, and changed the CVR to agree with AuditEngine.
• AuditEngine read the printed text of the 12,617 sheets printed by ballot marking devices (BMDs), not their barcodes. The voter can read the text, but not the barcode.
• The contest with the most Contest Variants compared with its margin is MEMBER, BOARD OF EDUCATION. It has 2,537 Contest Variants and a margin of 248 votes, with 177 Disagreed.

About this election

San Francisco is a consolidated city and county in California. Its Department of Elections runs its elections, and uses a voting system from Dominion Voting Systems. Voters may mark a paper ballot by hand, or use a ballot marking device (BMD), which prints a sheet with the voter's choices. Ballots cast by mail and in person are scanned, and the scanners save an image of each sheet.

These facts come from public sources, not from the audit.

• Population: San Francisco had 873,965 residents in the 2020 census. Its 11 supervisorial districts were redrawn in 2022 for about 79,545 residents each. Sources: SF Standard, Nichi Bei.
• Registered voters: 522,265 citywide. 412,231 San Francisco voters took part. Source: the Department of Elections certification of the election.
• President, citywide: Kamala D. Harris 323,719 votes (80.3%), Donald J. Trump 62,594 (15.5%), of 402,997. Source: the Statement of Vote, summed over all 510 precincts.

The table gives the profile of the ballot images provided and, for a partial audit, of the part audited.

The map shows the precincts of the jurisdiction. The precincts audited are colored. The boundaries come from a public source, given in the job settings.

What the ballots look like

The figure shows both sides of each sheet of a hand-marked ballot, and a sheet printed by a BMD. The sheets shown are from ballots of one precinct. The ballot images do not say which sheets belong to the same voter's ballot. Each sheet is labeled Audited or Not audited. Click an image to see it full size.

Sheet 1 Audited 00001_00008_000001
BMD sheet Audited 00001_01322_000011

This audit was a demonstration of AuditEngine on part of the election. To choose a subset of the ballots for this pilot study, we used a 'profile' to reduce the number of images processed. The results apply only to those ballot sheets selected.

• The audit covered sheet 1 of every ballot cast in San Francisco, in all 11 supervisorial districts. It includes ballots cast by mail, in person and on supplemental cards.
• Hand-marked paper ballots: the audit read sheet 1 of each ballot. It holds the contests at the top of the ballot: President, both United States Senate contests, Representative in Congress, State Senate, State Assembly, Board of Education, Community College Board and BART. Sheets 2 to 4 hold the ranked-choice contests, such as Mayor, and the measures. They were not included.
• Ballots printed by ballot marking devices: the audit read every contest, since they are all included on one sheet.
• Sheet 1 of every ballot gives citywide totals for its contests. These compare directly with the certified results of the whole city.

The audit processed 412,351 ballot sheet images, with one image for each side of that sheet in one image file. 120 sheets had no CVR record, and 0 CVR records had no sheet image. These cannot be compared, and are left out of the results below.

12,617 of the sheets were printed by ballot marking devices (BMDs). AuditEngine read the printed text of each one, not its barcode.

What the audit found

AuditEngine compared 412,187 Sheets, holding 3,878,114 Ballot-Contests. A Ballot-Contest is a contest provided on one ballot.

• 401,598 Sheets (97.4%) are Non-Variant Sheets. On these, AuditEngine and the CVR agree in every contest, with no write-ins, overvotes or ambiguous marks.
• 3,864,112 Ballot-Contests (99.6%) agree, with no variation of any kind.

• 14,002 Ballot-Contests (0.4%) are Contest Variants. These have a write-in, an overvote or a mark flagged gray as ambiguous, or AuditEngine and the CVR disagree.

• 44 Sheets are Not Processed. AuditEngine could not read them, so they are not compared.

• 882 Sheets are Ballot Variants, such as a sheet that is blank.

The charts show the Sheets and the Ballot-Contests, as in the Reconciliation Report.

Contest Variants by kind

Contest Variants are the Ballot-Contests set aside for a closer look. Most of them are not errors. A write-in, an overvote or a mark flagged gray as ambiguous makes a Ballot-Contest a Contest Variant, even when AuditEngine and the CVR agree. A disagreement between AuditEngine and the CVR makes one too. Of the 14,002 Contest Variants:

• 5,635 (40.2%) have a write-in.
• 4,064 (29.0%) are overvoted, with more votes than the contest allows.
• 4,303 (30.7%) have neither, and are called Other. Of these:
  • 1,308 (9.3%) agree with the CVR, and AuditEngine flagged a mark gray.
  • 2,299 (16.4%) agree with the CVR in another way, such as after adjudication.
  • 696 (5.0%) disagree with the CVR.

AuditEngine flags a mark gray when it is faint or unclear. A gray flag is not a disagreement with the CVR. It is a hint that a review could help, if the contest is close.

The chart shows them by kind, as in the Contest Variants Breakdown of the Reconciliation Report.

The Other Contest Variants

Other is 30.7% of the Contest Variants, so the table below breaks it down by pattern. Each row gives what AuditEngine read, the CVR original record, and the CVR modified record if the Ballot-Contest was adjudicated. Votes A is the set of votes AuditEngine read. Votes B and Votes C are different sets. Undervote means fewer votes than the contest allows, often none. The Comparison column says whether AuditEngine and the CVR agree. AuditEngine is compared with the CVR modified record if there is one, else with the original. The Groups column names each pattern's groups in the Reconciliation Report.

The CVR has an 'original' evaluation of each ballot, and if adjudicated, the CVR has a 'modified' record after adjudication. In adjudication, election staff review ballots and may change how a contest is counted. AuditEngine compared its reading with both. In 2,299 Ballot-Contests, adjudication changed the original evaluation to agree with AuditEngine.

Contest outcomes

A disagreement can at most move one vote from the winner to the runner-up, which changes the margin by two. An overvote that AuditEngine and the CVR agree on is not a disagreement. Still, a review could find the voter's intent in it, a vote that can narrow the margin by one. So a contest is compared with its margin by twice its Disagreed Ballot-Contests, plus its agreed overvotes. In 1 of the contests audited, this is equal to or more than the margin of victory. In every other contest, the disagreements and overvotes found could not change the outcome.

The margins are among the ballots audited. Since the audit covered part of the election, they are not the margins of the whole contest.

Major contests

The table below always lists the major contests on the ballot, such as President, United States Senator, United States Representative and Governor. Each contest links to its section of the Reconciliation Report.

The table below lists the contests where the Disagreed Ballot-Contests that do not involve a write-in, and the agreed overvotes, could narrow the margin the most, compared with it. Each contest links to its section of the Reconciliation Report.

Ranked-choice contests were compared rank by rank on the ballots audited. Generating ranked-choice outcomes requires all the ballots. However, we can check the results for each rank.

RCV contests appeared only on the BMD ballots processed, while in hand-marked ballots, they were on sheet 2, which was not included.

Batches scanned twice

The audit compares the votes on the ballots of each batch with those of every other batch. 9 pairs of batches hold the same ballots. Each pair is one set of paper scanned twice. A reviewer compared the two images of sample ballots in each pair, and confirmed it. The Repeated Batch Report lists the pairs.

In 6 of these pairs, both batches have cast vote records. So 335 ballots are counted twice in the CVR. The audit counts each of these papers once. The Reconciliation Report shows the audit's totals with the repeats taken out.

In 3 pairs, one batch has no cast vote records. Its images are extra, but not counted.

Other findings

• Mapping the ovals. The ovals on San Francisco's hand-marked ballots do not show in the ballot images. This made it hard to find them automatically. Mapping them by hand in TargetMapper was fairly easy. There are many ballot styles, so only sheet 1, pages 1 and 2, was mapped.
• Batches scanned twice, by sheet. The ballot has four sheets, and the CVR has a record for each.
• In the four pairs from tabulator 35, only sheet 1 was scanned again. That is 298 ballots. Their sheets 2 to 4 are counted once.
• In the other two pairs counted twice, 37 ballots were scanned again whole, all four sheets.
• A batch scanned twice is still one stack of paper, so an audit that draws ballots from the paper finds each one matched. A check of the ballot manifest against the CVR, batch by batch, can show an extra batch. It does not show which batch it repeats. Pairing the images does.

Conclusion

AuditEngine read each ballot sheet on its own and compared its reading with the CVR. 99.6% of the Ballot-Contests agreed, with no variation of any kind. In 1 contests, the disagreements and overvotes are large compared with the margin, even leaving out the disagreements that involve a write-in. These need a closer look, ballot by ballot, before their outcomes can be confirmed.

These results apply to the ballot sheets audited. An audit of every ballot sheet of an election can be done the same way, given the ballot images and the CVR.