In October, the 3,000 free CI minutes our GitHub organization gets each month lasted about 27 hours. A team of mostly AI coding agents opens a lot of pull requests, and every push to a pull request ran a pipeline. Minutes are billed, so we stopped guessing and measured.

This is the whole accounting, not just the happy ending: what we changed, the change that made things worse before it made them better, one idea we tested and dropped, and the real numbers from GitHub and from our new runner provider, normalised by how many pull requests we opened.

1. The numbers that drove the decisions

  • About 30% of runs were wasted. They were either cancelled by a newer push or failed on something we could have caught locally.

  • Setup dominated the bill. Each job bills its own minutes, rounded up, and repeats checkout, dependency install and, for test jobs, booting a local database stack. With four test shards, setup was about two thirds of every shard. A run billed roughly 32 minutes for about 6.5 minutes of wall clock.

  • One trigger did nothing useful. We re-ran tests on every merge to our staging branch. That was 57 of 300 runs in three days, around 765 billed minutes a day, and it gated nothing: deploys did not wait for it, and the pull request had already passed the same changes.

2. The full accounting

All figures are for one repository. Minutes come from GitHub's billing report, plus the Ubicloud invoice from October 17 (all of this repository's CI runs there; the only GitHub-hosted minutes left in that window were 42). Pull request and run counts come from the GitHub API. Cost is billed minutes times the rate: $0.006 per minute on GitHub-hosted runners, $0.002 on Ubicloud.

Window

What changed

PRs opened

PRs per day

CI runs

Billed min

Min per PR

Cost

Cost per day

Cost per PR

Aug 28 – Sep 7 (11 days)

Baseline: one runner, full suite.

63

5.7

211

2,348

37

$14

$1.30

$0.22

Sep 8 – 15 (8 days)

Parallel tests; prose-only changes skipped CI.

90

11.2

332

4,495

50

$27

$3.40

$0.30

Sep 16 – 27 (12 days)

Run only the tests a PR can reach.

91

7.6

427

5,603

62

$34

$2.80

$0.37

Sep 28 – Oct 1 (4 days)

Full suite gates promotion; staging merges re-test; four shards. On Oct 1 the local database stack also went from 6 to 12 images.

78

19.5

384

9,821

126

$59

$14.75

$0.76

Oct 2 – 16 (15 days)

Two shards, one checks job, a cost guard, local pre-push checks. Staging re-runs still on.

225

15.0

901

20,957

93

$126

$8.40

$0.56

Oct 17 – 29 (13 days)

Staging re-runs removed; every job on Ubicloud.

152

11.7

589

11,943

79

$23.89

$1.84

$0.16

Read it honestly, because the table says more than the headline:

  • The first three changes did not cut the bill; they raised it. Parallel tests and test selection cost less per run than the alternative, but each run also got heavier, so cost per pull request climbed from $0.22 to $0.37.

  • The explosion was a ramp-up and our own change at the same time. Between September 16 and October 1, pull requests per day went from 7.6 to 19.5 (2.6 times), and cost per pull request doubled from $0.37 to $0.76, so daily spend went from $2.80 to $14.75 (5 times). About half of that was volume. The other half was our change: making the full suite gate promotion was right for safety, but adding a four-shard run on every staging merge, then doubling the local database images, made every pull request twice as expensive.

  • The corrections worked, and they worked at a lower volume of work, not because of it. From the peak to today, billed minutes per pull request fell from 126 to 79 (about 37% fewer), and cost per pull request fell 79%. Most of the dollar drop came from the runner rate.

  • Per-run weight is the open problem. A run billed about 11 minutes at baseline and about 20 today. We moved cost from "many wasted runs" to "fewer, heavier runs", and the wasted share did not move: 29% to 31% of runs were cancelled in the first and last windows alike.

3. What we changed, in the end

  1. Run on pull requests only. Nothing runs on a push to a long-lived branch. Staging pushes went from 167 runs in the Oct 2 – 16 window to 1 after the change. The pull request that promotes staging to production runs the whole suite before the merge, so that is the gate that matters.

  2. Cancel superseded runs. One run per pull request: a new push cancels the old run.

  3. Run only the tests a change can reach. We use the test runner's own dependency graph, plus a short list of files that read the tree from disk or spawn subprocesses (nothing imports those, so the graph cannot see them).

  4. Fewer, fatter jobs. Four test shards became two, and lint, type check and build share one job. Wall clock stayed within a minute or two; billed minutes per run dropped to roughly two thirds.

  5. Never skip the required check. It is tempting to add path filters so docs-only changes skip CI. But a required status check that never runs sits pending forever and blocks the merge. We run the pipeline on every pull request and let the test selection make small changes cheap. (Our first attempt, in September, did skip prose-only changes. It had to be walked back for exactly this reason.)

  6. A guard on the bill itself. A script fails the build when someone adds a job, grows the shard matrix or enables another local service, until the cost allowance is raised in the same pull request. The cost of a change is now part of its review. (We wrote it after the image doubling; it would have caught that.)

  7. Fewer pushes. The cheapest CI run is the one you do not start. Contributors, human and agent, now run a local script that mirrors CI's test selection and environment before each push, and batch review fixes into one push.

4. The idea we tested and dropped

About 29% of runs were cancelled by a newer push, at roughly 17 billed minutes each. The obvious fix is a debounce: wait a minute or two before starting the heavy jobs, so a quick follow-up push cancels a run that has barely begun.

We measured it. Most re-pushes land 4 to 8 minutes into a run (median 5.5), the cancel itself lags the new push by 15 to 95 seconds, and the debounce job's own minutes on every completed run cost more than it saved. Net result: a loss of 114 to 184 minutes a day at the 60 to 120 second settings we modelled. We dropped it and went after the pushes instead.

5. Changing the runner

The last lever was the price per minute. On October 17 we moved every CI job to a managed runner service (Ubicloud) with the same 2 vCPU and Ubuntu 24.04 image family as GitHub's standard runner.

The move is one repository variable: each job runs on vars.CI_RUNNER or falls back to GitHub's runner when it is unset. Switching back is a settings change, not a pull request. Jobs that hold secrets stay on GitHub's own runners.

Runner (2 vCPU)

Rate per minute

GitHub-hosted

$0.006

Ubicloud

$0.002

That is a third of the price, and it applies to every minute left after the trimming.

6. The actual bill since the switch

Twelve days of real usage from the Ubicloud invoice, October 17 to 29 (no usage was recorded on the 25th):

Date

Minutes

Cost at $0.002/min

Oct 17

828

$1.66

Oct 18

1,579

$3.16

Oct 19

2,774

$5.55

Oct 20

1,832

$3.66

Oct 21

683

$1.37

Oct 22

1,857

$3.71

Oct 23

1,108

$2.22

Oct 24

13

$0.03

Oct 26

645

$1.29

Oct 27

362

$0.72

Oct 28

202

$0.40

Oct 29

18

$0.04

Total

11,901

$23.80

A $2.50 runner credit brings the invoice to $21.30.

7. Before and after, per pull request

Comparing the last window with the one just before it (same repository, same team, similar working pattern):

Oct 2 – 16

Oct 17 – 29

Change

Pull requests per day

15.0

11.7

−22%

Billed minutes per pull request

93

79

−15%

Cost per pull request

$0.56

$0.16

−72%

The volume-adjusted version: the 152 pull requests we opened since October 17 would have cost about $85 at the October 2 – 16 price per pull request, and cost $23.89. Against the September 28 peak ($0.76 per pull request) the drop is 79%. So the claim is "about three quarters cheaper per pull request than where we were", and about 3.5 times cheaper in the like-for-like comparison. It is only 29% cheaper than the original baseline of $0.22 per pull request, because we gave back some of the saving in heavier runs.

What the numbers do and do not show

  • The rate saving is exact: the same minutes at a third of the price. Minutes alone fell 15% per pull request; the rest of the 72% is the rate.

  • Pull request volume fell 22% per day in the last window, so part of the total drop (from $126 to $23.89) is a quieter stretch, not engineering. Per-pull-request figures and the $85 comparison remove that.

  • Minutes per run are billed minutes in the repository divided by CI runs, so they include a little time from other workflows. Treat them as approximate.

  • Cost uses list rates and ignores GitHub's monthly free minutes and Ubicloud's $2.50 credit.

  • Managed runners are not a free lunch. They need Docker for the database-backed tests, and anything holding a secret deserves a deliberate decision about where it runs.

The short version

  1. Measure which minutes are wasted before you change anything, and keep measuring after each change. One of ours doubled the cost per pull request.

  2. Normalise by activity. Total minutes and total dollars will lie to you in both directions.

  3. Stop running CI on events that gate nothing.

  4. Cancel superseded runs, and select tests by what the diff can reach.

  5. Fewer, bigger jobs beat many small ones when setup dominates, but watch per-run weight creep.

  6. Put a budget guard in CI so the bill gets reviewed along with the code.

  7. Measure ideas before shipping them. Our debounce looked great on paper and lost on the numbers.

  8. Then change the runner: the same work at a third of the rate.