Change Management: How to verify population completeness

Subversion - Image from

Subversion – Image from

As a brand-new auditor I remember receiving giant text or excel files full of data I couldn’t decipher. Inside the file would be hundreds (or more likely thousands) of line items. I was too naive and not confident enough to question what the client had provided – so I accepted it without asking a single question about how it was generated.

After working in the field for a few years and speaking with countless other auditors, I have come to realize this problem is rampant in the IT audit field – especially in the case of testing change management.

If you have ever been audited (or are an auditor yourself), you have probably produced or requested “a complete population of changes to XYZ application from date to date“.

The Auditor’s plan here is to choose a small sample of changes from the total population for auditing purposes. By choosing a small sample from a complete population, the auditor can statistically make reasonable assumptions about the population as a whole without burdening the auditee with requests for potentially hundreds or thousands of changes.

The key here is that the population has to be accurate and complete.

What happens if the population is corrupt? What happens if the population only includes changes that followed the change management process, but ignores those that skipped it (Like those that weren’t entered into the ticketing system!)? And how can an auditor ensure that the population is complete?

Verifying Population Completeness

Here are a few tools and questions that should help assure the auditor that the population of changes is complete.

The Query

1. Built-in or Custom Query: Understand if the query is built-in to the application (as in part of the application’s features) or custom-built by the developer. As an auditor we have greater confidence in built-in features than those developed by a individual as an add-on (it is typically more difficult to ensure quality and accuracy of the query since it hasn’t been as well vetted).

If the query was custom-built by an in-house developer you should spend a few minutes walking through the query structure (even if you don’t fully understand the complexities) to get a sense for the developer’s thought process. Often, problems with the query will quickly surface – especially if the only time the company uses the query is for the audit (which we see pretty often). It’s a red flag if the developer doesn’t seem confident or familiar with their own work.

2. Query Parameters: Whether the query is built-in or custom-built, you should always inspect the parameters. Ensure that the query includes all applications in scope as well as the date range in question.


Parameters for a built-in query.

Population Sources

1. From the Ticketing System: The first place most developers go to grab a population of changes is usually the change ticketing system. This makes sense because the ticketing system (hypothetically) has the most detailed information about a given change and is by far the most user-friendly source to procure information (especially when it comes to audit trails).

Unfortunately, as an auditor you DO NOT want to rely on the ticketing system for your population of changes.

Changes from the ticketing system are inherently flawed from an audit perspective because anything entered in the ticketing system should have (hypothetically) followed the change management process. (This is sometimes called testing the affirmative – or only testing those that are likely to pass.) What the ticketing system ignores are those changes that bypassed the change management process and are thus, more likely to have high risks and audit findings attached to them.

The ticketing system is a great place to start, but only the first step in gathering population details. The next two options should help validate what you are seeing in the ticketing system.

2. From the source code development tool: Most source code management tools have a method to generate all changes implemented to production (often called “change commits”). Tools like Subversion, JenkinsGit, and various add-ons are worth researching as they are commonly utilized to track deployments. This is a highly effective way to choose changes because the list of implementations is comprehensive and not easily manipulated.

Challenges: If you use the Source Code Management Tool (SCMT) as a source of your population, be aware there are several challenges including:

  • Tracing a change in the SCMT (which is often in an unorganized/non-descriptive form) back to the change management documentation in the ticketing system (which is usually more organized).
  • Filtering the changes in the SCMT to the appropriate scope (filtering for the appropriate apps and dates can be difficult).
  • If your scope includes multiple applications there could be more than one SCMT in which you have to familiarize yourself to effectively inspect.
Inspect the change commit log in the version control software.

Inspect the change commit log in the version control software. This example from Subversion.

3. Modification date on live executable files: Another great place you should check for changes is the modification dates attached to the application executable files. This information is typically available (i.e. in Microsoft Windows) in the folder where the files are housed in the “Date modified” column.

Challenges: There are certain limitations on utilizing the Executable files for change populations. For one, often the executable file typically only shows the last date modified, not all modifications. This pretty much limits the usefulness of this method to verifying when a clients says that there have been no changes during the review period, but ineffective for pulling a full population.

However, I recommend checking the last modification date as be an easy way to verify that the information you see from the SCMT and the ticketing system are in-line.

Modify date on executable files indicate possible change history.

Hopefully these suggestions add a few tools to your auditing toolkit. Are we missing anything?

10 thoughts on “Change Management: How to verify population completeness

  • Great blog.
    Is there a difference between Change Management and Configuration management? I don’t think so, but often I find management will treat them differently (assuming there is less risk in Config) and therefore the supporting documentation is limited. Beyond Config and ini files I often find it hard to nail down where those details are stored in various systems. Documentation is better for large ERPs but so is the complexity.

  • @John – I’ll jump in with some thoughts.
    Depends on what you are referring to. Configuration changes should go through the Change Management process to ensure they have been tested, reviewed, schedule, etc. But Configuration Management is a separate process for recording and tracking all of the system configurations to ensure they meet standards and haven’t been modified intentionally or by mistake. I have seen Configuration Management tied to system build and deployment processes as well as compliance and vulnerability scanning. Systems are built according to spec with certain configurations set (e.g. no log on as root, logging turned on, etc) and stored in a CMDB. Scanners (we used BindView) can also be employed to verify those configurations have been set and remain.

    Another great post Christian. Change Management for me has been the most difficult area to audit for this very reason. Working backward from the system files has worked best for me. I’ve also reviewed modifications against an established change window (i.e. You should only find changes during the Wednesday night change window). But if they have a weak Change Management process/policy and access to the production environment is open, it can be very difficult to confirm changes are valid and appropriate.

  • Great post, Christian. Covers the basics with a very practical focus.

    (assuming you agree with the premise) A good next topic may be DevOps and how an automated code-commit solution can be both lower cost for the business (less manpower) AND compliance (less testing).

    • @Chris – I like the idea. I have seen that work for a lot of places – especially places that do a lot of development. It would be interesting to think through what type of companies would benefit (and perhaps not benefit) from a DevOps type solution.

  • like you said, more often than not, there’s really no other way but to use the ticketing system because clients don’t have any other way to extract a complete list of system-generated program changes.

    that being said, additional procedures can be performed to validate completeness and accuracy of the change testing population (or at least, obtain reasonable assurance over the completeness and validity).

    i still believe that it would be better to at least perform testing in spite of the challenge over the change population instead of instantly concluding that the control is ineffective and not testing the change management at all just because you can’t rely on the population.

    1. test access controls to ensure that access to migrate changes to production is only limited to authorized personnel

    2. ensure that there is a rigorous business process in place where all changes are appropriately documented in the ticketing system before being promoted to production.

    3. confirm that the people responsible for promotion check to ensure that change has been logged in the ticketing system and tested/approved for promotion

    4. confirm that there’s a strong change review board type controls in place to review all changes

    • @L – I think you are right that testing changes from the ticketing system is better than not testing anything at all. Unfortunately, that method leaves a lot of risks on the table though.

      By only testing changes from the ticketing system the assumption is that all changes were documented, but you could be missing fraudulent changes, changes that didn’t follow the change management process, and accidents – which is where the real value of the audit comes in.

      Of course if 1-4 are in place that reduces the risk; but if 1-4 are in place (and working thoroughly) you would also think it would be possible to confirm the population outside the ticketing system (in the source code management tool, for example).

      • While I agree that testing only tickets is not the best route, if you find issues in the tickets where changes were not tested, signoffs didn’t occur, etc., it is more likely that not all changes are managed.

        In that case, you already know you have a big problem. You just don’t know how BIG it is.

Leave a Reply