A Build Artifact Changed
The hash at a build output path is not the one from last week and nobody owns the change. How to establish what the new file is, where it went, and when.
Somebody noticed that the binary your pipeline publishes has a different hash than the one from last Thursday. There is no commit that explains it, no release ticket, and no build in the log for that day. Two engineers have already said it was probably a rebuild. Nobody is willing to say that in writing.
This is the software supply chain question pointed at your own pipeline rather than a vendor's, and it is uncomfortable for a specific reason: your build system is trusted infrastructure, so nothing was watching it the way something watches an endpoint. There is no alert to look back at. What you have instead is the file itself, kept from whenever it appeared, along with every earlier object that occupied the same path, and the ability to ask what the new one resembles.
Be precise about what this page produces. Stairwell can establish that a file was present, on which machines, at which paths, from when. It can establish what the file resembles, across your own private files and the global malware corpus. It cannot establish that the artifact ran. Stairwell is not inline, is not an endpoint detection product, and a sighting is a record of presence rather than of execution. Presence and timing, stated as presence and timing, is a strong finding here and it is what the pipeline owners can act on. An execution claim you cannot support is what gets your whole report discounted.
Easy Mode: once you have established the artifact is not one of yours, hand its hash to Backstory rather than scoping it by hand. Until then the question is a comparison, and the comparison is the work below.
What will I know by the end?
Every distinct object that has occupied that build path and in what order, when the new one first appeared and on which machines, how it compares to the build it replaced, what it resembles anywhere in the corpus, and whether its arrival lines up with a build that actually happened.
Why this works
Three ideas carry this one, and the first is why the investigation is possible at all.
The path has a history, not just a current occupant. Stairwell keeps every distinct object that has lived at that location, so the artifact's real release history is recoverable even though your pipeline logged nothing and nobody filed a ticket. One hash is one build; the ordered sequence is the finding.
Two builds of the same source resemble each other. A recompile changes detail everywhere while keeping the file's arrangement, so the honest comparison is not "same hash or different hash", which is always different, but "does this look like a build of that program". A file that keeps the arrangement and gains a block is the shape that should worry you.
Presence is not execution. A sighting proves the artifact was on a machine at a path from a date. That is a strong finding here and it is what pipeline owners can act on. An execution claim you cannot support is what gets the whole report discounted.
If you want the instruments themselves rather than this scenario, Investigate & Hunt covers each one on its own.
Step 1: What is the history of that path?
Search the path rather than the hash. One hash is one build; the path is the sequence, and the sequence is the finding.
pc:`C:\BuildAgent\out` few:\.exe
Path contains, filename ends with. Use backticks around anything with backslashes so every character is taken literally, and add the artifact's own filename when the path holds more than one thing you care about. See StairQL for the prefixes and CEL Query Language for the explicit form.
If the artifact is a Portable Executable, its own metadata is a better handle than a path, because it follows the file wherever it was copied to:
company:"Your Company" product:"Your Product" version:2.0.0+
What comes back is every distinct object Stairwell holds for you that has lived at that path or carried that identity, not only the copy currently deployed. Add the First seen column and sort by it. That ordered list is the artifact's history as it actually happened, including the builds nobody wrote a ticket for.
Read the shape before you read any single row. A steady cadence of one new object per release, each present on the same handful of machines, is a pipeline behaving. An extra object between two releases is the thing you were told to look into. Two objects with the same version metadata and different hashes is the finding that ends most of these investigations, because a version number is written by the build and a hash is not.
Step 2: When did the new object first appear, and where?
Open it and read its sightings. This is the factual core of the answer and the part that will be quoted.
For each machine that reported the file you get the asset, the path, the filename it used there, when it was first seen on that machine, and when it was last seen. Sort by first seen and you have the artifact's distribution in the order it happened: the build agent, then the artifact store, then the deployment share, then whatever ran it.
Three things to read off that table.
The earliest first-seen is your candidate timestamp for the change. Not a proof, because a machine reports what it observes and coverage is not uniform, but it is the tightest bound you have and it is usually tight enough to compare against a build log.
The first machine matters more than the count. An artifact whose earliest sighting is the build agent behaved like a build output. An artifact whose earliest sighting is a deployment share, a laptop, or a jump host did not come out of the pipeline, and that single fact reframes the entire question.
Presence is the claim. The file was on that machine at that path from that date. Whether it was executed is not on this screen. For an artifact that reached a deployment share and was never run, presence is the honest answer, and it is still the answer that tells the pipeline owners what happened.
Compare last seen against each asset's last check-in before you tell anyone the artifact is gone from a machine.
Step 3: How does the new object compare to the build it replaced?
Put them side by side, starting with the object panel header on each. Five comparisons, cheapest first.
- Prevalence. A build artifact of yours should be rare globally and confined locally to the machines your pipeline touches. Rare in both places is expected. Common globally is a red flag worth explaining, because it means the file is not yours, and the usual explanation is that somebody replaced your artifact with something off the shelf.
- Signature. If your pipeline signs, an unsigned or invalidly signed object at that path is close to conclusive on its own. If it is signed correctly, that tells you the file is authentic to your signing key and nothing about whether the content is what you meant to sign. Authenticity is not correctness.
- File metadata. Company, product, internal name, original filename, version, and the imported libraries. A legitimate rebuild changes very little here. Something inserted usually changes something, and an added import is often the tell.
- Size and file type. Compared against the previous three or four builds rather than against one, so ordinary drift does not read as a signal.
- The binary image. Two builds of the same source look alike at a glance and the difference in a modified one is often visible without reading a byte. This is the fastest qualitative comparison on the page and it is worth doing before any of the slower work.
Decision point. If every comparison lands where a rebuild would land, the likely answer is a rebuild that nobody logged, and the remaining work is a process problem rather than a security incident. Record it that way and hand it to whoever owns the pipeline. If any comparison is off, keep going.
Step 4: What does the new object resemble?
Open Variants. This is the step that distinguishes "a build we cannot account for" from "a build with something in it", and neither the path nor the hash was ever going to tell you.
Variant Discovery compares across the entire corpus, both your own private files and the global malware corpus, and returns the files that resemble this one, ranked, each with its own verdict, prevalence, and opinion. Two very different results are possible and they mean opposite things.
- It resembles your own previous builds and little else. That is what a rebuild of the same source looks like, and the earlier builds in that cluster are yours, which you can confirm from their sightings. This is the reassuring outcome, and it is reassuring in a way that a matching version string is not.
- It resembles something in the global malware corpus. Then your question has been answered and the investigation changes shape immediately. Go to An EDR Alert Just Fired and work the resemblance as a lead: what is it, where else is it, what landed with it.
There is a third outcome worth naming because it catches people out. A cluster of common, high-prevalence files usually means the similarity is being carried by a shared runtime, a packer, or a statically linked library rather than by anything your build did. Sort for the rare end of the list and read that.
Variant Discovery runs automatically for threat reports and Backstory investigations, and on demand when you ask for it from the UI or the API, which is what you are doing here.
Step 5: What is inside it?
If the artifact is an installer, an archive, or a bundle, look at the Related files section on the Summary tab. When Stairwell unpacks a container, the files inside become objects in their own right, each remembering the object it came out of and each with its own verdict, prevalence, and sightings.
For a supply chain question this is usually where the answer is. The container is signed and versioned and looks correct, because that is the point of attacking a build system. Work down the children and take the one that is rare, or that carries a bad verdict, or that has no business being in your product. A child recorded this way is credited to the machine that reported the container, so you keep the asset attribution even though no machine reported that file directly.
Run AI Triage on anything you cannot account for. It gives you a summary, a threat type, the indicators inside the file, and its reasoning. Read the confidence before the likelihood, and keep the indicator list.
From here on, an unexplained child is your pivot rather than the artifact. Take it back through Step 4, because the child's variants are what tell you whether this is a one-off or part of something with a history.
Step 6: Does the timing line up with a build that happened?
This is the decision point that resolves "nobody owns the change", and it is the one place where the answer comes from correlating Stairwell's dates against a system Stairwell does not hold.
Take the earliest first-seen from Step 2 and put it next to your build system's own record: the build log, the commit history, the release calendar, the change tickets. Three outcomes, and they route to three different places.
- A build ran, and the artifact appeared when it finished. Unlogged but real. The gap is in your release process, and the finding is that an unticketed build reached production. Hand it to the pipeline owners.
- No build ran, and the artifact appeared anyway. State it exactly this way: a new object appeared at the build output path on this date, on these machines, with no corresponding build. That sentence is the finding, it is supported entirely by presence and timing, and it is enough to justify treating the build system as in scope.
- A build ran, and the artifact appeared before it. Look hardest at this one. It means the object reached your fleet by a route other than the pipeline, and Step 2's first-machine reading will usually tell you which route.
Whichever way it lands, extend the sightings work outward. Every machine that holds the object is in scope for a look, and if the artifact reached hosts your pipeline has no business touching, that distribution is itself part of the finding.
Step 7: What else landed on the build agent?
Only once Step 6 has established the artifact is not accounted for. Until then this step will bury you in ordinary build output.
Run Run-to-Ground from the artifact's hash. Right-click it and choose Run to ground under Workflows. It expands across the artifact's variants, finds which of your assets held any of them, and reports the rare files that landed on those machines within 24 hours either side of arrival.
On a build agent that window is unusually informative. Build machines have a narrow, repetitive file diet: the same toolchain, the same dependencies, the same outputs, over and over. Anything rare that arrived next to an unexplained artifact stands out against that regularity far more sharply than it would on a laptop.
Run it again on whatever it surfaces. Each new component is a fresh starting hash, and each run anchors its window to when that file landed on that machine, so a tool staged on the agent a week before the artifact appeared opens a window a week earlier. That recursion is what moves the finding from "an unexplained binary appeared" to "here is when the build system was first touched", which is the sentence the pipeline owners actually need.
If the result is clean, say so plainly. An unlogged build that brought nothing with it is a release process problem rather than a security incident, and distinguishing the two is most of the value of this page.
Step 8: How do I record this so it holds?
- Set opinions on both sides. Trusted on your confirmed legitimate builds, which stops your own product inflating every threat report's match count for the next year. Malicious on anything you established, with a comment naming what you found and how.
- Write your own threat report for the hashes you confirmed as bad, plus any infrastructure they referenced. It keeps matching backward across what you already hold and forward across everything that arrives next, which is the coverage you want if the build system was reached once. See Write It Up.
- Keep the Step 1 query. It is one line, it fits in a runbook, and running it before each release turns this incident into a standing check. The list of objects that have occupied a build output path is a question worth asking on a schedule rather than after somebody notices.
- Configure a trigger on your report so a re-arrival reaches a person rather than waiting to be found.
- Write the presence and timing sentences carefully. This report will be read by engineers who will check it. Presence with dates, distribution with asset names, resemblance with the corpus it was compared against, and no execution claim anywhere. See Write It Up for how to draft it.
What are the limits of this answer?
- Presence and timing, not execution. Stated at the top and worth restating here, because it is the single most likely sentence in your report to be overstated.
- First-seen is bounded by coverage. The earliest sighting is the earliest observation, not necessarily the moment of creation. A build agent without a forwarder shifts your timeline, and knowing which machines report is part of the answer. See Assets.
- Only what is collected. Roughly 65 executable and script formats by default, tunable by your team. A modified configuration file, a template, or a script type you have not added is outside the default set.
- Resemblance is not attribution. Variant Discovery tells you a file belongs with a family. Who built it and how it reached your pipeline are conclusions you argue for from the sightings, not results the platform returns.
What should I read next?
- A Vendor in Your Supply Chain Got Breached, the same problem when the compromised build system belongs to somebody else.
- Variants, for what Step 4 is comparing against and why the two corpora matter.
- What Are Hilbert Curves?, for the visual comparison in Step 3.
- Write It Up, for stating presence and timing in a way that survives review.
Updated 4 days ago