Assessing an open source component
The problem
Article 13(5) requires due diligence when integrating third-party components, and that diligence must be documented to be enforceable. The grid that follows from it has nine criteria: known vulnerabilities, activity, governance, upstream security policy, historical responsiveness, licence, upstream SBOM, artefact signing, depth.
Applied by hand it is impeccable across ten components and unworkable across three thousand. Which is the order of magnitude of a real dependency graph. Diligence that does not get done is not diligence.
The grid therefore has to be made computable.
OpenSSF Scorecard
Scorecard is an open tool from the OpenSSF that runs a series of automated checks on a repository and produces, for each, a score out of 10 together with a risk level that weights it into an aggregate score.
Its contribution is not to replace judgement: it is to make objective and repeatable what was previously an impression.
The eighteen checks
| Family | Checks |
|---|---|
| Security practices | Vulnerabilities · Dependency-Update-Tool · Maintained · Security-Policy · License · CII-Best-Practices · CI-Tests · Fuzzing · SAST |
| Source risk | Binary-Artifacts · Branch-Protection · Dangerous-Workflow · Code-Review · Contributors |
| Build risk | Pinned-Dependencies · Token-Permissions · Packaging · Signed-Releases |
What each of your criteria becomes
| Your criterion | Matching Scorecard check | Automatable |
|---|---|---|
| Known vulnerabilities | Vulnerabilities (via OSV) |
Yes |
| Project activity | Maintained |
Yes |
| Governance, lone maintainer | Contributors |
Partly — it counts organisations, not fragility |
| Upstream security policy | Security-Policy |
Yes |
| Historical responsiveness | — | No |
| Licence | License (presence only) |
Partly |
| Upstream SBOM | — | No |
| Artefact signing | Signed-Releases |
Yes |
| Dependency depth | — | No, that is the SBOM’s job |
Six of the nine become measurable, and Scorecard adds nine the grid did not have — branch protection, code review, token permissions, dependency pinning, dangerous workflow patterns, static analysis, tests, fuzzing, and non-auditable binary artefacts.
What Scorecard does not tell you
Essential, and worth writing into the policy to avoid a false sense of security:
- nothing about licence compatibility with your context of use — see Open source licensing;
- nothing about reachability: a well-scored component may carry a vulnerability exploitable in your product, and a poorly scored one may be harmless;
- nothing about upstream support duration, which nevertheless constrains your support period;
- nothing about code quality or functional fit;
- a high score would have ruled out none of the incidents documented in Reference incidents — those projects were active, reviewed and widely adopted.
Scorecard measures hygiene signals, not the absence of malice. That is useful, bounded, and worth saying.
S2C2F: the ingestion framework
Where Scorecard scores a component, the Secure Supply Chain Consumption Framework (S2C2F), also stewarded by the OpenSSF, describes how to consume open source: a set of tool-agnostic practices, with a maturity model and a mapping to other supply chain specifications.
Its guiding principle is the one you already apply without naming it: do not consume directly from a public registry, but go through controlled, inventoried and verified ingestion — see Locking and updating dependencies.
Its value to you is twofold: it gives a shared vocabulary with your suppliers and customers, and it supplies a maturity scale usable in the self-assessment.
The threshold policy
A score with no threshold is not a policy. What must be written down:
| Situation | Proposed rule |
|---|---|
| New component, score ≥ 7 | Adopt with no further review |
| New component, score 5 to 7 | Adoption possible, with documented review of the failing checks |
| New component, score < 5 | Refuse by default, or a reasoned, dated waiver |
Signed-Releases failing on a critical component |
Hash verification mandatory |
Maintained failing |
Treat as an abandoned component |
| Already-integrated component whose score drops | Alert, re-examine, no automatic removal |
The last row is the one that gets forgotten: diligence is not an entry check, it is monitoring. A project can lose its maintainer three years after adoption.
Where it runs
| Moment | What happens |
|---|---|
| Adoption request for a component | Score computed, attached to the ticket, threshold applied |
| Build | Threshold check on direct dependencies, warning below |
| Periodically | Recomputation on critical components, alert on decline |
| Annual review | Full re-examination, starting with components whose score has degraded |
Continuous computation across the whole transitive graph is expensive and of limited use: restricting periodic monitoring to direct dependencies and critical transitives — those whose replacement would take months — captures most of the benefit.
The evidence to keep
| Artefact | Why |
|---|---|
| Timestamped score per component and version | Shows the state of knowledge at the time of the decision |
| Completed diligence grid, with the non-automatable criteria | Covers what the tool does not measure |
| Versioned threshold policy | Shows the decision follows a rule, not a preference |
| Waiver register, with expiry | Shows exceptions are controlled |
| Re-examination log | Shows diligence is continuous |
It is that set, and not the score alone, that constitutes the documented diligence Article 13(5) expects.
The trap to avoid. A score archived without the decision it supported demonstrates nothing. What an authority examines is not the number: it is that a written rule was applied, and that departures from it were reasoned.