CI 通過後 macOS Installer 仍可能失敗：Controller 遷移實例的分階段驗收

This is the English edition of an article first published in Japanese on Zenn: https://zenn.dev/asagiman/articles/macos-installer-actual-acceptance

Even after CI turns green, the macOS Installer can stop. This happens at the final real-machine acceptance step even when development relies on AI agents. The tests have passed and no problem has been found in the built package. Yet a product cannot ship until the user's Mac accepts the installation and the authorized work actually runs.

During the migration of the Controller that Netsujo is developing, the internal records show that, within the two days of October 16 and 17, 2026 alone, we ran the real macOS Installer eleven times. Four runs succeeded (three generation-update stages and one limited diagnostic without authentication) and seven failed (three generation-update stages and four stages involving the administrator's own authentication). All seven failures passed preinstall and then stopped in postinstall, and the operating system recorded a failed installation. In one of the authentication runs, part of the changes had already been applied to the machine when the run failed. There were also real-machine failures and successes before and after these two days. On October 30, 2026, another installation stage succeeded on the real machine and passed readback, but a normal authenticated AI job had not yet been accepted.

Below is a way to record evidence so that the next decision can be made without losing the proof of stages that did succeed, separated into five boundaries: source (including CI), package, OS installation, authorization, and a real AI job. It ends with a check you can run quickly with a disposable test package.

The two-day real-machine results are based on internal records from October 16–17, 2026. The October 30 result and the current status were confirmed by checking the same internal records on November 1, 2026. As of November 1, 2026, the overall Controller migration and acceptance of a normal authenticated job are still in progress. Credentials, internal recovery procedures, secrets, and confidential configuration values are excluded.

What happened: a built package stopped in postinstall

CI runs tests against a specified source, environment, and input. It can find many problems that are reproducible on the development side in advance: types, unit tests, static checks, and package generation.

When the macOS Installer is involved, the existing OS state, pre- and post-install scripts, the actual user and permissions, the administrator's own authentication, and migration state from earlier generations all affect the result. Some of these conditions cannot be fully reproduced in a CI environment.

In the Controller recovery work, we used the real installation logs to separate the progress of preinstall from the later postinstall failure. In the failed runs, postinstall returned an error, and the OS recorded the installation failure together with a PackageKit error (PKInstallErrorDomain Code=112). A separate diagnostic that succeeded is valid only for the diagnostic's own scope; the failed package stays on hold and needs its own acceptance.

The first thing a developer needs to establish is which candidate, which stage, and which recorded outcome were involved. Fixing on a single cause too early risks treating failures from different generations as the same phenomenon.

Sorted by type, the records showed that different candidates succeeded at different stages

The table below organizes the observations by type, within what can be published. The rows are not in chronological order. "Observed evidence" shows what an actual record exists for; "Supported conclusion" shows what that record supports.

The real logs tell us the terminal state of each stage. Even when an authentication-related process fails, an OS denial or nearby warning that appears in the log at the same time cannot simply be taken as the root cause. Establishing causality requires a safe reproduction that isolates the failure conditions.

We therefore adopted a rule: a stage that succeeded for one candidate is never reused as a success for another candidate. The evidence of the successful diagnostic is preserved, and a new package requires new acceptance evidence.

First, identify "the same subject"

In long-running development, conversations, PRs, CI, release files, and host logs are each updated at different times. Even after a commit moves on following review, a package built from an older revision can remain on disk. When this happens during incident response, it becomes easy to attach past check results to the current candidate by mistake.

So we pin the identity of the subject being accepted.

For example, a package's SHA-256 can confirm file identity: it checks that the file a reviewer approved matches the file a user is about to install. Even when the hash matches, the postinstall terminal status and authentication success must be confirmed separately.

When reporting a single failure, save at least "the candidate identifier, the stage executed, the observed terminal status, and the remaining unknowns" in the same record.

Why a simple retry is dangerous

An installer may update some files or state before it fails. In our records, too, there was a run that failed after part of its changes had been applied. A rerun then starts from a different initial state. Even if only the second attempt succeeds, the cause of the first failure can be lost.

Privileged operations and authentication add a question of authorized scope. If an operation run under a one-time approval fails, treat that approval as used and stop. If a developer removes a protection through an environment variable or a different execution path, the result may look like a technical success, but the legitimate permission model breaks.

After a failure, our practice is to review the preserved logs and remaining state, organize the source, and re-confirm the target file and authorization before rerunning the same operation. The eleven runs above are records of different packages; none of them is a rerun of the same package.

What to check after a failed stage

• Which terminal state the OS recorded: success, failure, or interruption.
• Whether any changes were applied before the failure, and how far to preserve the parts that cannot be confirmed.
• Whether the next run would make additional changes to existing data or managed state.
• Whether the existing approval was consumed by the run, and who decides the next authorization.
• Whether a local reproduction environment exists where the same input can be tested safely.

Checking these five points lets you explain why the next operation is needed, even during an urgent recovery. Retrying without checking adds one more partial state of unknown cause.

Put a safe preflight first to make failure cheap

Some operations change the environment's state the moment they succeed. Separating the conditions that can be inspected read-only before execution reduces failures that happen after irreversible changes.

As a general design, consider the following order.

1. Fix the input and the subject: identify the acceptance candidate, the package to run, and the authorized scope.
2. Static checks: inspect composition, dependencies, format, and hash.
3. Non-destructive environment checks: confirm the required OS conditions and readable state within the authorized scope.
4. Recovery assessment: evaluate in advance what state remains after an interrupted run and how to reconcile it.
5. Authorized change: execute according to an approval that matches the subject and the operation.
6. Real-machine readback: compare the OS terminal record with the durable result.
7. Normal job acceptance: confirm authentication and roles, shutdown, output persistence, and independent review.

This order is a design proposal. We do not claim that the current Controller implementation follows it. Acceptance of a normal job remained incomplete as of November 1, 2026.

What RED→GREEN should confirm

What follows is a generalized approach, not a result carried out on the Controller. Once you find a failure condition, first pin a local counterexample. For example, remove an environmental condition that the package-building logic assumes, using a safe fixture, and confirm that the preflight stops correctly. That is the RED state.

Next, add the minimum condition check or recovery and confirm that the same fixture turns GREEN. Loosening a protection mechanism for testing at this point can weaken the production permission guarantee. Keep the operations that must be refused refused, and give a reason only to the operations that are needed. Testing only the happy path is not enough; include the conditions that must fall explicitly into a hold.

This table generalizes the refusal and stop conditions to aim for. When applying it to a real product, build tests that fit the assets being protected and the OS specification.

Keep acceptance as a record per stage

Bringing AI into the development process increases the number of times an agent reports "implementation complete" or "tests passed". If the granularity of those reports is vague, a source fix gets taken as a successful start of operations.

We treat each stage as an independent unit of acceptance. The following is a generalized design example. It was constructed to explain the stages and is separate from the actual state names of any product. There are six state names, and they map to the five boundaries from the introduction as follows: SOURCE_REVIEWED and CI_ACCEPTED are "source", PACKAGE_VERIFIED is "package", OS_INSTALL_ACCEPTED is "OS installation", and AUTHORIZED_REAL_JOB_ACCEPTED covers "authorization" and "a real AI job". The final INDEPENDENT_OPERATIONAL_REVIEW is not a boundary but an independent check of the whole.

SOURCE_REVIEWED
      |
      v
CI_ACCEPTED
      |
      v
PACKAGE_VERIFIED
      |
      v
OS_INSTALL_ACCEPTED
      |
      v
AUTHORIZED_REAL_JOB_ACCEPTED
      |
      v
INDEPENDENT_OPERATIONAL_REVIEW

For each stage, record the subject identifier, the observed result, who authorized it, and when it ran. If a stage fails, mark it HOLD and keep the evidence of the earlier stages that succeeded. When the candidate changes, re-examine the stages affected by the change. Keep unknowns visible as unknowns until an authorized observation resolves them.

Example acceptance record

{
  "candidate": "example-source-revision",
  "packageDigest": "example-package-digest",
  "check": "os_install",
  "observed": "failed",
  "partialState": "requires_reconciliation",
  "authorization": "consumed",
  "realJobAccepted": false,
  "nextGate": "independent_review_before_new_approval"
}

This is fictional data for explanation. It contains no real identifiers or credentials. The record keeps failure and uncertainty as values and does not promote them to success. Any mechanical judgment also needs a design that never promotes an ambiguous value to success.

What to ask an independent review for

An independent review confirms what each piece of evidence actually proves. A review of source changes and tests looks at reproduction of the problem, the changed code and its impact, and boundary conditions including abnormal cases. A review of the package and the real machine looks at the identity of the target bytes, the actual macOS Installer result, and whether anything was partially applied. A review of security and operations looks at the permission boundary and validity of approval, and at output persistence, shutdown, and independent acceptance of the formal job.

The role of an independent review is less about running more tests and more about verifying assumptions the implementer may have missed. For operations that change permissions and OS state in particular, confirm that the reviewed source matches the actual subject of the operation that is later approved.

A minimal check you can run yourself

You can observe what does not become "success" when postinstall fails, using a test package that contains nothing confidential. The steps below were run and confirmed on macOS 26.6.2 on November 1, 2026. They use no administrator privileges and target the user's home directory. The package is empty, and no files are installed.

1. Create and enter a temporary directory, then create scripts/postinstall and make it fail on purpose.

work=$(mktemp -d) && cd "$work" && mkdir -p scripts
printf '#!/bin/sh\necho "demo postinstall: failing on purpose" >&2\nexit 1\n' > scripts/postinstall
chmod +x scripts/postinstall

1. Build the package and record its hash.

pkgbuild --nopayload --scripts ./scripts --identifier com.example.demo --version 1.0 ./demo.pkg
shasum -a 256 demo.pkg

1. Install into the user domain and record the exit status and the log.

installer -pkg demo.pkg -target CurrentUserHomeDirectory; echo "exit=$?"
grep 'com.example.demo\|demo.pkg' /var/log/install.log | tail -5

In our run, installer printed "The install failed" and ended with exit=1, and /var/log/install.log recorded PKInstallErrorDomain Code=112 (the explanatory text in parentheses depends on the display language; if you cannot find the log line, search for Code=112). If you cannot read the log, the exit status alone is enough to identify the failure. For comparison, rebuilding with exit 0 instead of exit 1 gave exit=0 and "The install was successful". An empty package does not appear in the list of installed packages (pkgutil --pkgs) even when it succeeds, so its absence from that list cannot be used as evidence of failure.

1. In the same shell as step 1, delete the temporary directory. /var/log/install.log keeps a record of the run, which is an ordinary system log entry.

[ -f "$work/demo.pkg" ] && cd / && rm -rf "$work"

What this confirms is the gap that a package whose identity is confirmed by its hash can still fail OS installation. One observation is complete when you write "observed": "failed" in the acceptance record above and inspect the remaining state before any rerun.

Where to start when applying this to another product

First, write down the scope your usual CI protects. Then list the execution environments outside CI. For the macOS Installer, that is the OS installation result; for a web service, the deployment target and the anonymous production page; for an AI agent, legitimate authentication and the completion of a real task.

You do not need to build a large management platform from the start. Linking the source SHA, check results, release file, and real-environment acceptance result for a single change into one record is enough to make it easier to see where the work stopped. Next, add the person responsible for each stage and its stop conditions.

When a check fails, look at that record before starting any additional run. Reuse valid results for subjects that have not changed, and safely isolate only the parts whose cause remains.

Summary: the last thing to confirm is whether the work actually ran

The Controller migration confirmed that failures can remain in the OS installation and authentication stages even after CI succeeds. The fact that the unauthenticated diagnostic succeeded is kept as valid evidence, while the authenticated stages and the real job need additional acceptance. Work that is still in progress as of November 1, 2026 is not treated as a completed result.

Completing real operations requires evidence that the machine accepted the specified package, started the work with legitimate authority, and finished output persistence, shutdown, and review. Even when AI raises development speed, responsibility for this final confirmation stays with the development team.

Go one step beyond source and CI acceptance: include real-environment and real-use confirmation in the plan, with separate evidence for each stage.
That is the lesson that generalizes from these failures and the recovery process.