heygrc
EU AI Act robustness and cybersecurity in code

What the model does when it is wrong.

Article 15 requires high-risk AI systems to achieve an appropriate level of accuracy, robustness, and cybersecurity, and to perform consistently in those respects throughout their lifecycle. Two parts of that live in code more than anywhere else: resilience, whether the system keeps behaving safely when a model errors, times out, or degrades, and cybersecurity, whether it resists the attack classes the article names, among them data poisoning, model poisoning, adversarial examples, and confidentiality attacks. A fail-safe or a guard that rarely fires is still part of the system's robustness; removing it changes what the system actually does under failure, not just what it does on the happy path.

How it shows up in a diff

The shapes the same control failure takes.

Accuracy, robustness, and cybersecurity erode when a change removes the handling for the case that was not the happy path. The recurring shapes:

  • A fail-safe fallback is removed

    The handling for a model error, timeout, or low-confidence result, a conservative default, a safe stop, a route to manual review, is deleted because it rarely triggers, leaving the failure case unhandled.

  • An input guard against malformed or adversarial input is dropped

    A validation or sanitization step that rejected malformed, out-of-range, or suspicious input before it reached the model is removed, widening the surface for adversarial examples designed to trigger a wrong or unstable output.

  • A redundancy path collapses to one point of failure

    A backup model, replica, or retry path that let the system keep operating safely through a single failure is removed or left to rot, so one fault now takes the whole system down instead of degrading it.

  • Rate limiting on an inference endpoint is removed

    Throttling that limited how fast or how much a caller could query the model is lifted, widening the path for model-extraction or other confidentiality attacks that rely on volume.

  • Accuracy or drift monitoring goes silent

    The job that compares live predictions against a baseline or ground truth, and alerts when accuracy drops, is disabled or left to fail silently, so a model that has quietly gotten worse keeps serving decisions with nothing flagging it.

Worked example

A fail-safe fallback, removed as dead code.

A high-risk scoring model occasionally times out or throws under load. The service wraps the call in a fail-safe: on a timeout or error it returns a conservative default decision and flags the case for manual review, instead of letting the failure reach the caller. The fallback almost never fires, so a cleanup removes it to shorten the function.

model/decide.ts+1 -5
async function decide(input) {-  try {-    return await model.score(input)-  } catch {-    return fallbackDecision(input) // conservative default, flagged for manual review-  }+  return await model.score(input)}
heygrcEU AI Act Art. 15

Removing the fallback means a model timeout or error now reaches the caller as a raw failure instead of falling back to a conservative default flagged for manual review. Art. 15 (accuracy, robustness and cybersecurity) expects a high-risk system to be as resilient as possible to errors, faults, and inconsistencies, and names backup or fail-safe plans as one way to achieve that, not just accuracy when nothing goes wrong. Restore the fallback, or an equivalent fail-safe, and if it is firing too often, fix why the model times out rather than removing the safety net. Whether a system is high-risk and in scope is for your own assessment.

What an auditor does with this

Robustness is checked as behavior under failure, not the demo.

Conformity for a high-risk system covers how it behaves when something goes wrong, not only when it does not: whether it has fail-safe measures for errors and faults, and whether it resists the attack classes the article names: data poisoning, model poisoning, adversarial examples or model evasion, confidentiality attacks, and model flaws. A change that removed a fallback, an input guard, or a drift monitor is the concrete gap behind a system that looks accurate right up to the failure mode the removed code existed for. It shows up in the diff to the inference path.

What this is, and is not

A review, not a red team.

heygrc flags changes that touch Art. 15 and cites the article so the fix happens in the pull request. It does not run adversarial testing against your model, measure its accuracy, or audit its training. It catches the moment a change removes the fail-safe or guard a high-risk system was relying on, at the diff.