Change management evidence: checking approved changes against production

Change management evidence is the record that each change to a production system was justified, assessed, approved, tested and written down. Most of it sits in a ticketing or ITSM system, and it describes what a team intended to ship. Production shows what actually shipped. This post covers what ISO/IEC 27001, SOC 2, PCI DSS and DORA expect, and which changes bypass the process. It then shows how to check approved changes against production, and what an auditor can use as evidence.

What does change management have to prove?

Change management has to prove five things about every change to a production system. It had a reason and an assessment of its security impact. Someone with authority approved it, and it was tested and recorded. Each framework also expects a route for emergencies, written up after the change. The wording differs between frameworks, and the evidence an auditor asks for is much the same.

DORA puts it in one line. Article 9(4)(e) of DORA requires ICT change management so that all changes to ICT systems are "recorded, tested, assessed, approved, implemented and verified in a controlled manner". The DORA risk-management RTS lists what those procedures must contain in Article 17(1).

ISO/IEC 27001 and the AICPA Trust Services Criteria are licensed texts, so the table paraphrases them.

Why does an approved change record leave a gap?

A change record shows what a team intended to ship and who approved it. Production shows what actually shipped. The two drift apart when a change ships differently from its description, and when a change reaches production with no record at all. An approval is evidence that someone reviewed the plan. It says nothing about whether production now matches that plan.

The UK Financial Conduct Authority has published data on how often change goes wrong. Its multi-firm review, Implementing technology change (5 February 2021), analysed over 1 million production changes made by 23 firms in 2019. Of nearly 1,000 material incidents reported to the FCA that year, 17% were attributed to change activity. Change advisory boards approved over 90% of the major changes they reviewed, and in some firms they rejected none during 2019. The FCA said this raised questions about how well those boards work as an assurance mechanism.

Audits test change management by sampling. An ITGC or SOC 2 auditor takes the population of changes for the period and tests a sample for approval and testing. When the population comes from the ticketing system, a change that never had a ticket can never be sampled. The test can then pass on an incomplete population.

When the population comes from the ticketing system, a change that never had a ticket can never be sampled.

PCI DSS v4.0.1 builds the other direction into its testing procedures. Testing procedure 6.5.1.b tells the assessor to examine recent changes to system components and "trace those changes back to related change control documentation". That test starts from the system and works back to the records. Our post on what happens to a penetration test report the day after delivery describes the same gap between tickets and production after a test.

Which production changes bypass the change process?

The changes that most often bypass the change process sit outside the application's own release pipeline. Edge rules, vendor scripts, tag managers, feature flags, upgrade side effects and emergency fixes can each change what customers reach without a change record.

The FCA found that "most firms in this review did not effectively track third-party changes". In its data, third-party incidents accounted for 18% of all incidents, and 22% of those came from third-party change activity. Firms also told the FCA that their suppliers often do not tell them about changes.

How do you check approved changes against production?

You check approved changes against production by comparing two lists for the same period and applications. One list holds the approved changes from your change system. The other holds the releases observed in production. Each release should match an approved change or carry a reason, and each approved change should appear in production or carry an explanation.

Matching the two lists produces four outcomes.

• Approved and observed. Confirm the release under its change reference.
• Approved but not observed. Check that the change was deployed. Some changes, such as a database index or an internal service, cannot be seen from outside at all, and the record should say so.
• Observed but not approved. Review the change, find its owner and record why it happened. Decide whether it needs testing, and raise a change record afterwards if your process requires one.
• Neither. Nothing to reconcile.

The observed list has to come from somewhere other than the change system. Vikax checks each application from outside on a schedule your team sets, from every 5 minutes to every 7 days. It reads pages, scripts, response headers, the structure of API responses and the text of PDF documents. It groups changes that land within a few minutes of each other into one release for that application and environment. Each environment sets that window, between 5 and 15 minutes.

In the console, your team confirms the releases it expected and records its own change reference on each one, such as CR-10482. It can also dismiss a release. A release nobody confirms stays open for review. Every confirmation or dismissal needs a written reason and goes into the audit log, and none of them alters the changes underneath.

Vikax stores the reference exactly as your team wrote it. It does not connect to ticketing or deployment tools, so it never checks the reference against a ticket. A confirmation records that someone expected the release and named its change. The approval itself stays in your change system. A release in Vikax is a grouping of changes seen from outside, and it does not prove that a deployment ran. The platform page shows releases, change review and exports in the console.

How should emergency and unplanned changes be recorded?

Emergency and unplanned changes need the same record as planned ones, completed after the change. The record says what changed, when, who made it, why the normal procedure was skipped and whether the change needs testing. For emergency changes, the DORA RTS and the NIS2 Implementing Regulation both require this record in writing.

Point (g) of Article 17(1) in the DORA RTS asks for "procedures to document, re-evaluate, assess, and approve emergency changes after their implementation". The NIS2 Implementing Regulation (EU) 2024/2690 applies to cloud, CDN, managed service and managed security service providers, among others. Many banks depend on these providers. Point 6.4.3 of its Annex requires them to document the result of an emergency change and "the explanation for why the procedures could not be followed".

The FCA counted over 33,000 emergency changes in 2019, between 3 and 5% of all changes in its sample. Some firms classed almost 20% of their changes as emergencies.

An unplanned change found in production needs the same record, opened from the other end. An emergency record starts with the person who made the change. An observed change starts with the evidence, and the team has to find the person. If the change touched an area the last penetration test covered, record whether it needs testing.

PCI DSS leaves it to each organisation to evaluate whether such a change is significant under 6.5.2. Our post on what counts as a significant change sets out how to decide and record it. The post on targeted testing or a full assessment covers the choice of testing. In Vikax, the testing decision stays with the change it concerns, and nobody can edit or delete it afterwards.

What can an auditor use as evidence, and where does it stop?

An auditor can use a complete list of the production releases in the period. Each release is tied to a change reference or a written reason, with who confirmed it and when. That list has to come from a record nobody can edit afterwards. Otherwise the auditor is testing a document the team had time to tidy.

For each period, the evidence should hold four things.

The record also shows how far production has moved since the last penetration test, which bears on how long that test stays valid. A testing provider with access the client has approved works from the same record, as described on the page for testing providers.

Outside evidence has limits, and the auditor should hear them from you. It cannot show who approved a change. It cannot see databases, internal services, network devices or the firewall rules themselves, only their effect on what a visitor receives. Vikax reads what the server returns and does not run pages in a browser. A script that a tag manager adds at runtime can fall outside its view. A confirmed release says that the change was expected. Whether it was secure is a question for testing.

Production evidence gives the auditor a second population of changes, collected independently of the change system and the pipeline records. It sits beside both and replaces neither. Vikax for security teams shows that record across every application a team answers for.