New — The State of AI Hiring 2026. Every number sourced→
Ethics & Compliance

How to Conduct an AI Bias Audit

How to conduct an AI bias audit the NYC Local Law 144 way: who can audit, which data to use, the calculations, what to publish, and real cost ranges.

How to conduct an AI bias audit, with a card contrasting the independent Local Law 144 audit and an internal self-check

TL;DR: How to conduct an AI bias audit: there are two separate jobs. The legal audit under NYC Local Law 144 can only be performed by an independent auditor, and an internal self-check cannot substitute for it. The legal audit runs through nine steps: scope, calculation, auditor, data, dataset, calculations, interpretation, publication and notice, and repeat. The statistics are the small part; data preparation is the long pole, per provider commentary.

General information, not legal advice. Have counsel confirm how this applies to you.

The nine steps, in order:

  1. Decide whether the tool is in scope
  2. Pick the calculation
  3. Choose and vet an independent auditor
  4. Choose historical, pooled or test data
  5. Assemble the dataset
  6. Run the three tables
  7. Interpret the results
  8. Publish the summary and notify candidates
  9. Repeat

Updated 21 October 2026. First publication. The cost table is refreshed quarterly, and the page is revised on any change to the DCWP rule or FAQ and whenever new written auditor quotes arrive.

Key takeaways

  • An employer cannot satisfy Local Law 144 with an internal check. The rule requires an independent auditor, defined by three tests in 6 RCNY § 5-300 (DCWP Notice of Adoption, April 2023).
  • The employer, not the vendor, is responsible for the audit, though a vendor may commission one (DCWP FAQ V.1–V.2).
  • DCWP "has not set a specific requirement for statistical significance" (FAQ III.7), so DCWP sets no statistical-significance threshold.
  • Imputed or inferred demographic data cannot be used. Test data is allowed only where historical data is insufficient for a statistically significant audit.
  • One provider publishes a first-tool price of $5,500 with a five-business-day turnaround from clean data (Bias Audit USA, fetched 15 October 2026). In our reading, few other auditors list prices openly, so ask for a written quote. A dated table is below.

An employer that screens candidates in New York City with an automated tool needs a bias audit within a year before it uses the tool, and the audit has to come from someone independent of both the employer and the vendor. Whether an in-house check will do is the first question, and for the legal audit the answer is no. The rules on data, publication and notice are also specific.

This page covers the audit required by the Local Law 144 requirements, and only that audit. It does not cover general AI audits, security reviews or model-risk exercises. What follows is the process, step by step, with the rule cited at each point and the cost laid out with dates and sources.

The term "bias audit" covers two different jobs. The bias audit definition page explains what an audit measures; this page covers how it is done. The two jobs are:

The nine audit steps grouped into four stages: decide, data, run, publish, with a banner stating an independent auditor is required

  • The Local Law 144 audit. An independent auditor performs it (6 RCNY § 5-300; DCWP FAQ II.1 and IV.2, 29 June 2023).
  • An internal pre-audit or ongoing self-check. The employer or the vendor runs it. It can be useful for catching data problems early. It does not satisfy the first job.

So can you run a bias audit in-house for Local Law 144? Not for the legal audit. The table below sets out the nine steps, what each produces, and where the rule says so.

StepWhat you produceRule cite
1. ScopeA decision, with counsel, on whether the tool is coveredDCWP FAQ I.2–I.6
2. CalculationThe metric that applies: selection rate or scoring rate6 RCNY § 5-301(b), (c)
3. AuditorAn independent auditor, vetted against three tests§ 5-300; FAQ IV.1–IV.2, V.1–V.2
4. DataA choice among historical, pooled and test data§ 5-302; FAQ III
5. DatasetPer-candidate records, with an "unknown" value kept§ 5-301(b)(5), (c)(5); FAQ III.3, III.6
6. RunThree tables, with the optional 2% exclusion§ 5-301
7. InterpretA reading of the results; the law requires no specific actionFAQ II.2; 29 CFR § 1607.4(D)
8. PublishA posted summary and distribution date; candidate notice§ 5-303; Admin. Code § 20-871(b); FAQ VI
9. RepeatA new audit no more than one year before use§ 5-301(a)

The table summarizes the rule; it is not rule text.

Steps 1–3: Decide the scope, pick the calculation, choose an auditor

The first three steps are decisions, not computations. They fix what is being audited, how it will be measured and who may do it.

Step 1: Decide whether the tool is in scope

DCWP's FAQ sets out the scope tests (FAQ I.2–I.6). The tool uses machine learning, statistical modeling, data analytics or artificial intelligence. It helps make employment decisions, meaning it "substantially assists or replaces discretionary decision-making." It applies to candidates who applied for a specific position. And the "in the city" tests are met. Whether any specific product is in scope is a question for counsel, and this page does not answer it.

Scope tests differ outside New York City. The state-by-state AI hiring rules tracker shows which states add audit or notice duties.

Step 2: Pick the calculation

The calculation depends on what the tool does (6 RCNY § 5-301(b), (c)):

  • A tool that selects or classifies candidates: compute the selection rate and the impact ratio for each category.
  • A tool that scores candidates: take the median score of the full sample, compute each category's scoring rate (the share scoring above the median), then the impact ratio.
  • A tool that classifies candidates into groups: repeat the calculation for each group.

Step 3: Choose and vet an independent auditor

The rule defines independence through three tests. The auditor must not have been involved in using, developing or distributing the tool. The auditor must have no employment relationship with the employer or the vendor during the audit. And the auditor must have no direct or material indirect financial interest in either (§ 5-300; FAQ IV.1–IV.2). DCWP keeps no approved list of auditors.

A vendor may commission an audit of its own tool, but "employers and employment agencies are ultimately responsible" (FAQ V.1–V.2). That applies to employers who buy from an AI-interview vendor as much as to anyone else; the guide to choosing AI interview software covers the buying decision itself.

Vetting matters. Fisher Phillips attorneys Usama Kahf and David J. Walton (20 August 2025) group the questions to ask under five headings and offer a free questionnaire. They write that "a poor-quality audit could do more harm than good." The checklist below is our own, mapped to the rule's three independence tests and to those five question groups.

Auditor vetting checklist

CheckQuestion to put to the auditorBasis
Independence 1Have you been involved in using, developing or distributing this tool?§ 5-300
Independence 2Do you have an employment relationship with us or the vendor during the audit?§ 5-300
Independence 3Do you have a direct or material indirect financial interest in either?§ 5-300
Background and expertiseWho will perform the audit, and what prior audits of hiring tools can you describe?Fisher Phillips group
Methodologies and toolsWhich calculations will you run, and how do you handle the 2% exclusion and the "unknown" category?Fisher Phillips group
Data security and privacyWhere will candidate-level data be stored, who can access it, and when is it deleted?Fisher Phillips group
Post-audit supportWhat do you provide for the published summary, and who answers questions afterward?Fisher Phillips group
Industry and regulatory experienceWhat Local Law 144 audits have you completed?Fisher Phillips group

The wording of the questions is ours. Fisher Phillips publishes its own questionnaire.

Steps 4–5: Choose the data and assemble the dataset

Data preparation is the long pole. The rule is specific about which data may be used, and the dataset must be in a form the auditor can calculate from.

Step 4: Choose historical, pooled or test data

Three kinds of data are in play (6 RCNY § 5-302; DCWP FAQ III):

Historical, pooled and test data conditions, plus the three worked cases from 6 RCNY 5-302 for first-time, six-month and three-year users

  • Historical data from the employer's own use of the tool is the default.
  • Pooled data from several employers using the same tool is allowed. An employer can rely on it only if it supplied its own historical data to the auditor or has never used the tool.
  • Test data is allowed only if historical data is insufficient for a statistically significant audit. The summary must explain why test data was used and should explain how it was sourced or developed.

DCWP sets no requirements for test data and no requirement for statistical significance (FAQ III.7–III.8). The rule's own worked cases (§ 5-302, Examples 1–3) show how this plays out:

  • An employer using the tool for the first time may rely on pooled or test data.
  • An employer that has used the tool for six months needs an updated audit once 12 months have passed since the audit it first relied on.
  • An employer with three years of data "may not rely on a bias audit conducted using test data."

The three-year example is the one to take to counsel first.

Step 5: Assemble the dataset

Collect the same fields for every candidate the tool assessed. The template below is our data-template design, derived from § 5-301 and § 5-302. It is not rule text.

FieldWhat to send the auditorBasis
Candidate IDA stable identifier, unique per candidateOur design
Tool outputAdvanced or not (selection tools), or the score (scoring tools)§ 5-301(b), (c)
Decision stageThe stage at which the tool's output was usedOur design
DateThe date of the tool's outputOur design
Job familyThe role grouping the candidate applied forOur design
SexSelf-identified, using the rule's categories, plus "unknown"§ 5-301(b)(5), (c)(5)
Race/ethnicitySelf-identified, using the rule's categories, plus "unknown"§ 5-301(b)(5), (c)(5)

The column list is a proposal for an auditor data request, not a DCWP form.

The "unknown" value is counted and reported, not dropped. If the data limits the time window or region covered, the summary should explain why (FAQ III.3). Imputed or inferred demographics cannot be used (FAQ III.6).

What if you do not collect demographic data? The rule bars imputed or inferred demographics, so an auditor cannot fill the gap by estimating. Whether alternatives exist is a question for counsel and the auditor.

Steps 6–7: Run the calculations and interpret them without a pass mark

The calculations are short once the dataset is clean. The interpretation is where the rule says the least.

Step 6: Run the three tables

The auditor produces three tables: sex, race/ethnicity, and sex by race/ethnicity. The auditor may exclude a category under 2% of the data, with a written justification and the excluded category's applicant count and rate in the summary (§ 5-301(d)). Ask the auditor and counsel whether the 2% test applies table by table.

The arithmetic is covered on the impact ratio and four-fifths rule page. A self-check is not a bias audit.

Step 7: Interpret the results

The law requires no specific action based on the results; anti-discrimination law may still require action (DCWP FAQ II.2). The EEOC's four-fifths rule of thumb (29 CFR § 1607.4(D)) is a screening reference, with its own caveats on statistical and practical significance and on small numbers. What to do with a given result is a question for counsel.

Two points are worth correcting. DCWP "has not set a specific requirement for statistical significance," so DCWP sets no statistical-significance threshold. And 0.80 is not a Local Law 144 test; the law requires the ratio to be calculated, not to meet a mark.

The same numbers have a legal context beyond the audit. The documented AI hiring bias cases show where selection-rate questions have been raised in practice.

Steps 8–9: Publish the summary, notify candidates and repeat

Step 8: Publish the summary and notify candidates

The employer posts the summary and its distribution date on the employment section of its website. A labeled link is allowed. The summary stays posted for at least six months after the tool's latest use (§ 5-303). Candidates who are NYC residents receive notice at least 10 business days before the tool is used (Admin. Code § 20-871(b); FAQ VI).

For a finished example of the format, see Pfizer's Bias Audit Summary of the HiredScore Grade System, published May 2026. It is shown as a format example only.

Step 9: Repeat

The audit must be done no more than one year before the tool is used (§ 5-301(a)). Whether a material change to the model calls for a new audit sooner is a decision for counsel and the auditor.

This is where the law's duties end. What an employer does with the numbers sits under anti-discrimination law, not under Local Law 144.

How much does a bias audit cost, and how long does it take?

There is no official tariff, and in our reading few auditors publish prices, so written quotes matter. The figures that exist are below, each with its basis, its date and a reliability note.

SourceFigureBasis and dateReliability
Bias Audit USA$5,500 first tool; $1,500 each for tools 2–3; $1,200 for tools 4–7; $900 for tool 8 and beyond; renewal $4,500; re-audit after a model change $3,950; 5 business days from clean dataPublished price list, fetched 15 Oct 2026Verified for this provider only
Sprad (Jürgen Ulbrich, 20 Aug 2026)"roughly between $5,000 and $50,000"Self-described as a vendor-published reference, not an official tariff or a guaranteed market averageLow
SiftFirst provider list (last reviewed 1 Aug 2026)Low-to-mid four figures for one tool on clean data at boutiques; "well into five figures" for multiple tools, messy data or a custom methodEditorial; no quotes shownLow to medium; direction only

List prices start in the low thousands for a single tool on clean data and rise to five figures or more for multiple tools, messy data or large audit firms. Turnaround follows the same pattern: one provider states five business days from clean data, and SiftFirst's provider list says the main variable is data preparation, not statistics. In practice, the work in Steps 4 and 5 sets the schedule.

This table is refreshed quarterly. Treat every figure as a starting point and request written quotes from at least two auditors; confirm each auditor is independent of the vendor whose tool is being audited.

FAQ

Can you do a bias audit in-house for Local Law 144?

Not for the legal audit. Local Law 144 requires an independent auditor, so an in-house check cannot satisfy it. An internal check is a different exercise and can still help prepare the data.

Can historical data be replaced with test data?

Only if historical data is insufficient for a statistically significant audit. The summary must then explain why and how the test data was generated. An employer with three years of data may not rely on a test-data audit.

What if we do not collect demographic data?

Imputed or inferred demographics cannot be used. Whether alternatives exist is a question for counsel and the auditor.

Can a vendor's audit cover me?

The employer remains responsible for the audit, even when a vendor commissions one. Whether an employer may rely on a vendor's audit depends on the data rules in Step 4.

How long does a bias audit take?

One provider states five business days from clean data. Data preparation usually takes longer, according to SiftFirst's provider list.

How often must a bias audit be repeated?

The audit must be done no more than one year before the tool is used. For the audit's contents, see what a bias audit tests.

Is there a minimum sample size?

DCWP has not set one. It "has not set a specific requirement for statistical significance."

What is the 2% exclusion?

The auditor may exclude a category that represents less than 2% of the data, with a written justification. The summary must give the excluded category's applicant count and rate. The calculation is shown on the impact ratio page.

Next step

See an evidence pack in a demo to see the candidate-level evidence a ZeaHire evidence pack contains. Your independent auditor decides what data the audit needs.

ComplianceAI hiring law
Was this useful?

Comments

No comments yet — start the conversation.

Leave a comment

Your email is never published. Comments appear once a moderator approves them.

Replying to
Plain text only. Be kind and stay on topic.

By commenting you agree to our privacy policy.