What counts as a significant change for a penetration test?

PCI DSS, NIST, DORA and NIS2 all tie security testing to change, and each leaves the judgement of what counts as significant to the organisation. This post sets out what each standard says and who decides. It then shows how to judge a change by what it touched and how to record the decision so an assessor can follow it. It stops at the point where a change needs testing. Choosing the kind of testing is covered in targeted testing or a full assessment.

What do the standards say about significant change?

Each standard below ties testing or review to change and leaves the definition of "significant" to the organisation. PCI DSS comes closest, with a list of activities that must at least be evaluated. DORA, NIS2 and NIST work through the organisation's own risk assessment. SOC 2 and ISO/IEC 27001 ask for controlled change management and set no testing interval.

PCI DSS is the only standard here with a list. The Council's FAQ 1317 names the activities the standard includes under the term. They include new or replaced hardware and software in the cardholder data environment, and changes to how account data flows or is stored. Changes to the environment's boundary, its supporting infrastructure and its service providers are on the list too. Each "must be considered and evaluated" to decide whether a change is significant for that entity.

The Council's Penetration Testing Guidance says "a significant change is not prescribed by PCI DSS" (section 2.6). A change that could affect network security or allow access to cardholder data may be considered significant. The guidance dates from 2017 and uses the older numbers, 11.3.1 and 11.3.2.

DORA uses "major change" in Article 8(3) and never defines it. NIS2's implementing rules apply to named providers, including cloud, data centre, managed service and managed security service providers. NIST SP 800-115 dates from 2008 and has not been replaced.

In every case your organisation decides, and the assessor checks how. An assessor reads your definition, then looks at the changes that happened to see whether you applied it. For PCI DSS 6.5.2, the expected testing includes examining the documentation for significant changes. Customers apply their own rules too, which how long a penetration test stays valid covers.

Judge a change by what it touched in the tested areas

A change matters for testing when it alters something the last penetration test examined or relied on. Its size says little. Changed lines, files or tickets measure the work involved, and they say nothing about which tested behaviour moved.

The tested areas are the pages and APIs in scope and the controls the tester relied on. Sign-in, session handling, authorisation, input handling, security headers and third-party code on sensitive pages are the usual ones. NIST SP 800-53 starts from the controls too. After a change, control CM-4(2) calls for verifying that the impacted controls work as intended.

Take two changes to a customer portal in one week. Marketing rewrites the home page and adds 40 news pages, changing several thousand lines. A release adds one script source to the content security policy on the sign-in page, changing one line. The first sits outside the tested areas. The second weakens a control the tester relied on to protect the page that handles passwords.

A one-line change to the content security policy on a sign-in page can matter more than a redesign of every marketing page.

Size matters in one way. Small changes to the tested areas add up, and after enough of them the assessment describes a different application. Count them against the assessment, and treat the total as a change in its own right.

Vikax compares production with a sealed record of what the last assessment covered. Changes outside the tested areas show as Minor change, and changes inside them as Significant change. That state shares the standards' phrase and answers a narrower question, whether the change touched what the assessment covered. Whether it is significant under your definition stays your team's decision. Our post on assessment freshness explains the six states.

Which changes usually count as significant?

The changes that usually count alter how people sign in, where data goes, which routes exist or which code runs on sensitive pages. Weakened security headers and platform moves belong with them. Each one changes a control or an input the last test examined.

Many significant changes never show on a page. Network segmentation, internal firewall rules, identity and access management and database moves are invisible to a visitor. PCI DSS requirement 11.4.5 asks for segmentation testing "after any changes to segmentation controls/methods", so any such change counts. Evidence for these changes has to come from change records and the cloud inventory.

Vikax checks applications from the outside, so it sees the first group and none of the second. It reads the pages, headers, scripts, API responses and documents your team chooses to monitor. It names the kind of each change, such as a removed security header or a new API route. Platform moves show up through the headers and routes they change. Our post on what changes first after a penetration test covers which tend to move soonest.

Which changes usually do not count?

Changes that leave the tested areas alone usually do not count. Content edits, routine noise, routine patches and rollbacks are common examples. Each one still needs a recorded reason when someone sets it aside, so the record shows it was seen.

• Content outside the tested pages. New articles, copy, images and prices on pages the test did not rely on.
• Routine noise. Timestamps, session IDs and tracking parameters change on every visit.
• Routine patches. Patches that change no behaviour in a tested area. NIST names significant patches as a trigger, which is a different case.
• Changes outside the scope. These count once the scope itself stops holding. A new subdomain that takes payments changes the scope.
• A rollback. Production returns to the version that was tested.

The word usually matters. A marketing page that adds a sign-up form now collects personal data. A pricing page that embeds a payment widget brings in third-party code. Judge each change by what it touched, whichever team made it.

Vikax filters routine noise such as timestamps and session IDs before it compares, and keeps the stored evidence exactly as observed.

Write the definition down before you need it

A written definition names the tested areas, the kinds of change that trigger a decision, who decides, how quickly and when the definition is reviewed. Write it before the next release. A definition written after a disputed change reads like a justification.

The standards point the same way. NIST SP 800-53 control CM-3 asks organisations to "determine and document the types of changes" under change control. NIS2's implementing rules ask for change procedures to be reviewed at planned intervals and after significant changes to operations or risks (Annex 6.4.4).

The paragraph below is a starting point to adapt with your assessor or QSA. It is not legal or compliance advice. Your team sets the values in square brackets.

Significant change for penetration testing. A change to an application in scope is significant when it alters any of these in the areas the last test covered:

• sign-in or session handling;
• authorisation;
• where a form sends data;
• API routes or response fields;
• a security header or content security policy;
• third-party code on sign-in or payment pages.

The activities PCI DSS lists in FAQ 1317 are always evaluated, whether or not they touch a tested area. [The application owner and the security lead] decide within [five working days] of the change reaching production, and record the reason and the evidence. This definition is reviewed every [12 months] and after any significant incident.

In Vikax, each monitor can carry qualifying-change rules, which decide which differences reach your team for review. A rule can match text or patterns, compare numbers or count changed lines. Rules are versioned, and your team can try a new version against stored evidence before relying on it. Because each rule belongs to one monitor, the sign-in page can have a strict rule and the news pages a loose one. Your team still decides what is significant.

Record who decided and why

A significance decision needs a record made at the time. It holds the change, the evidence before and after, the release and its change reference, and who decided, when and why. A record rebuilt from memory before an audit carries far less weight.

DORA asks for part of this record. Its risk-management standards require the potential impact of each change on existing ICT security measures to be identified (Article 17(1)(h)). Change tickets hold another part, though CDN rules, firewall policy, tag manager updates and vendor scripts often reach production without one. Our post on change management evidence covers those changes and the emergency changes recorded after the fact.

In Vikax the record builds up during review. Vikax groups changes into releases for each application and environment. Your team confirms the releases it planned and records its own change reference, such as CR-10482. Releases nobody confirms stay open for review, and every confirmation or dismissal needs a written reason. Decisions are recorded against the evidence and cannot be edited or deleted. The audit log is kept for at least 400 days.

Vikax stores the change reference as your team wrote it and does not check it against a ticketing system. It does not decide whether a change is significant. It keeps your team's decision with its reason.

An export in PDF, HTML, JSON or ZIP gives an assessor the record for any change or period. Vikax records a fingerprint of each export file, so whoever holds it can check that it was not altered. The same exports answer audit and customer questions. A testing provider sees a client's record only with access the client approves.

Once the record says a change needs testing, the scope comes next. Our post on targeted testing or a full assessment sets out how to choose it.