GitHub added a Copilot metrics impact dashboard for enterprise administrators and organization owners this week, aimed at answering a question most teams adopting AI coding tools have struggled to answer with anything better than a guess: is this actually helping?
Up to now, the metrics most orgs had access to were adoption proxies — seats activated, suggestions shown, suggestions accepted. Those numbers tell you whether people are using the tool. They don't tell you whether the team is shipping faster, whether code quality held up, or whether the tool is quietly encouraging developers to accept plausible-looking suggestions they haven't fully read. A dashboard aimed at "impact" rather than "usage" is a meaningful shift in what's being measured, even before you look at the specific charts it ships with.
Why usage metrics were the wrong thing to optimize
Acceptance rate in particular is a metric that's easy to game without anyone intending to. A developer under time pressure who accepts more suggestions without fully reading them will show up as a heavier, more "successful" Copilot user than one who reads carefully and rejects half of what's offered — even if the second developer is producing more reliable code. If an org ties any part of performance review or team comparison to acceptance rate, it creates a quiet incentive to rubber-stamp AI output.
This is a solvable problem, but it requires measuring outcomes instead of activity. GitHub's new dashboard reportedly moves in that direction, but the specific signals it surfaces matter less than the org actually deciding, ahead of time, what "impact" is supposed to mean.
Outcome metrics worth tracking regardless of which dashboard you use
- PR cycle time, split by whether a meaningful portion of the diff was AI-assisted. If cycle time isn't improving for AI-assisted PRs relative to your baseline, the tool isn't delivering the speed benefit it's marketed on for your specific codebase.
- Revert and hotfix rate on AI-assisted PRs vs. others. This is the metric most likely to surface a problem quietly — a team that's shipping faster but reverting more isn't actually ahead.
- Review comment density. More back-and-forth on AI-assisted PRs than on manually written ones is a signal the generated code needs more human correction than it appears to on first read.
- Test coverage delta on AI-assisted changes. Coding agents are good at producing code that looks complete; whether it shipped with adequate test coverage is a separate question worth checking explicitly.
If your org isn't using GitHub's built-in dashboard, these are trackable with the gh CLI and PR labels:
# Tag PRs as AI-assisted at open time, then compare cycle time later
gh pr list --state merged --label "ai-assisted" \
--json number,createdAt,mergedAt \
--jq '.[] | (((.mergedAt | fromdateiso8601) - (.createdAt | fromdateiso8601)) / 3600)' \
| awk '{ sum += $1; n++ } END { print "avg hours to merge:", sum/n }'Bucketing PRs this way and comparing against a non-AI-assisted baseline gives you a rough version of the same signal the dashboard is trying to surface, without waiting for the built-in tooling to reach your plan tier.
What we're recommending to clients rolling this out
- Pick 2–3 outcome metrics before turning the dashboard on org-wide, and write down what "good" looks like for each. A dashboard full of numbers without a prior definition of success mostly produces post-hoc storytelling.
- Don't attach acceptance rate to individual performance reviews. It measures engagement with the tool, not code quality, and using it as a performance signal creates exactly the rubber-stamping incentive described above.
- Review monthly, not per-sprint. Cycle time and revert rate are noisy at the individual-PR level; the signal shows up over weeks, not days.
- Segment by team and by codebase area, not just org-wide. A legacy service with poor test coverage will show different AI-assisted outcomes than a well-tested greenfield service, and averaging across both hides the more useful information.
Takeaway
A dashboard is only as useful as the questions it's pointed at. GitHub building impact metrics into the product is a good sign that the industry is past the "seats activated" phase of AI coding tool adoption — but the org still has to decide what impact means before the numbers are worth reading as more than activity theater.