CI/CD Integration with CodeScene’s Delta Analysis

CodeScene identifies and prioritizes technical debt, while at the same time uncovering and measuring social factors of the organization behind the system. The earlier you can react to any potential finding, the better. That’s why CodeScene offers integration points that let you incorporate the analysis results into your build pipeline.

The purpose of a Delta Analysis is to:

  • Prioritize code reviews based on the risk of the commits.
  • Specify quality gates for the goals specified on identified hotspots using CodeScene’s Intelligent Notes.
  • Specify quality gates that trigger in case the Code Health of a hotspot declines.
  • Get early warnings such a complexity trend increases and detect the absence of expected change coupling.

Select a Delta Analysis Strategy

CodeScene supports two different strategies for the delta analysis:

1. Full Scan: CodeScene supervises the code health of all files in a change set or pull request. We do make an exception for really, really large pull requests; if a change set exceeeds the hardcoded limit of 30 files, then CodeScene automatically reverts to a Hotspot Scan in order to maintain quick response times. A Full Scan always fetches the latest version of a file from the repository’s main branch and use that version as a reference.

2. Hotspot Scan: With this strategy, CodeScene limits the delta analysis to files that are identified as hotspots. This means that the most relevant modules are always checked. Non-hotspots will not be analysed for code health decline (use the Full Scan for that). The Hotspot Scan is quicker than the Full Scan.

By default, CodeScene’s uses the Full Scan which is the recommended strategy since it covers all modified content. The Hotspot Scan is faster and intended for CI/CD gates in a build pipeline that have to execute quickly while still supervising the specified hotspot goals and quality gates.

You change strategy in the project configuration:

Configure a delta analysis strategy for your project.

Fig. 170 Configure a delta analysis strategy for your project.

What are the Pre-Requisites for a Delta Analysis?

A Delta Analysis is always relative to a full analysis. CodeScene will use the latest completed analysis as a baseline for the Delta Analysis.

This is why we recommend that you configure your analysis to run at least once a day. On projects with more contributors and high commit frequencies you want to schedule CodeScene to run a full analysis each hour. We’d say that any project with more than 10 commits per day should run the analysis frequently.

Integrate CodeScene in your Continuous Integration Pipeline

Connect using CodeScene’s REST API

CodeScene provides a REST API that lets you integrate the analysis results in a continuous integration pipelinei and/or as robot comments in a code review tool like Gerrit.

CodeScene’s REST API provides a special type of analysis called a Delta Analysis. A Delta Analysis is fast, it usually just takes a few seconds to run, and is used to get early feedback on a pull request or range of new commits.

Connect to external webhooks

CodeScene’s delta analysis can be connected to some common repository hosting services using webhooks. All that is required is API credentials (API Token/App Password) authorized to modify webhooks and pull requests.

CodeScene will install webhooks at the remote service that will trigger a Delta Analysis when a pull request is created or updated. The results of the Delta Analysis will be visible as a comment on the pull request.

Meet the Delta Analysis

A Delta Analysis is triggered by a pull request, a range of commits or a single commit; You decide through the API. Each time you trigger a Delta Analysis, CodeScene calculates the following information:

1. Delivery risk of the suggested change set. The risk classification is relative to the risk profile for your codebase as described in Risk Analysis. Use this information to prioritize code reviews and to decide upon delivery risks.

2. Status of Quality Gates. There are two independent – but related – quality gates. The first and most important one ensures that none of the goals you have specified via Intelligent Notes (see Manage Hotspots and Technical Debt with Goals) are violated. The second quality gate lets you catch code that declines in Code Health directly in the CI/CD pipeline.

3. Early detection of Complexity Trend Warnings. CodeScene already provides an early warning in case the code complexity starts to rise in a Hotspot. Now the delta analysis can catch such complexity patterns based on the changes in a pull request. That provides a great opportunity to refactor the code before delivering it to your main branch.

4. Suggests absent change patterns. This analysis identifies change sets where an expected temporal coupling is absent. If a cluster of files have changed together for a long time they are intimately related. This warning fires when such a temporal change pattern is broken. Please note that this may be good - we’ve refactored something - but it may also be a sign of omission and a potential bug. As a consequence, this warning is based on a self-correcting algorithm; If you keep ignoring the warning it will go away automatically as the temporal coupling decreases below the thresholds.

A Delta Analysis gives you early warnings, actionable quality gates, and detects high risk changes

Fig. 171 A Delta Analysis gives you early warnings, actionable quality gates, and detects high risk changes (example from Jenkins).

The screenshot in Fig. 171 shows an example of a delta analysis result. This information is consumed and integrated via the REST API for delta analyses that we’ll discuss soon.

Please note that future releases of CodeScene will expand the Delta Analysis capabilities. Our plan is to provide even more detailed information that helps you get the most out of your time.

Use a Delta Analysis to Save Time in Code Reviews

The main advantage of a delta analysis if that it lets you react to potential problems early. But there’s a potentially large saving at the other end of the spectrum too; Instead of treating all pull requests as equals, CodeScene’s risk classification lets you prioritize your code reviews and focus your time where (and when) it’s likely to be needed the most. Code reviewer fatigue is a real thing, so let’s use our review efforts wisely.

Code review tools like Gerrit lets you select a label. For example, you specify a label that either allows or blocks the change. In addition you may select a label as an opinion (+1 and -1 in Gerrit).

When you integrate CodeScene with Gerrit, it’s our recommendation that you map CodeScene’s risk classification to an automated +1 or -1. For example, all commits below the risk category 3 may be +1, which indicates to the reviewers how much time they need to spend on this review.

In addition, the delta analyses lets you auto-detect files that seem to degrade in quality through issues introduced in the current commit or pull request. This is done by calculating code biomarkers (see Explore your Code’s Biomarkers), which are then supervised for their trend as shown in Fig. 172.

Degrading biomarkers

Fig. 172 A Delta Analysis detects degrading Code Health.

CodeScene’s delta analysis works in the other direction too; It’s not only about spotting problems. If you enable the quality gates and work pro-actively with Intelligent Notes in CodeScene, then you can allocate less time on the features with 1) low risk, and 2) passing gates, as shown in Fig. 173.

Quality gates are OK

Fig. 173 Spend less time on reviewing code with low risk and passing quality gates.

Keep up the Good

CodeScene also detects hotspots that improve their code health. This information is included as a positive reinforcement intended to show the effect of the current changeset on the overall code health. Fig. 174 shows an example where a hotspot is successfully refactored – an occasion to celebrate!

Code improvements detected in the delta analysis

Fig. 174 Measure code improvements as part of the CI/CD pipeline to reinforce a positive trend.

Notify the Code Owners on Failed Quality Gates

As discussed in Knowledge Distribution, CodeScene parses and includes code ownership information when present. If you have this feature enabled, then CodeScene will include a mention of the code owners should a quality gate fail. You see an example in Fig. 175.

CodeScene notifies code owners when a quality gate fails.

Fig. 175 CodeScene notifies code owners when a quality gate fails.

The REST API for Delta Analyses

CodeScene lets you create a special Bot user role intended to consume the REST API. Login as administrator and create a Bot user for each of your integration points as illustrated in Fig. 176

Configure a Bot user

Fig. 176 Configure a Bot user for each of your integration points.

You trigger a Delta Analysis by POSTing a request to the REST endpoint specified in your analysis configuration as illustrated in Fig. 177.

REST API endpoint

Fig. 177 Your analysis configuration specifies the REST endpoint to trigger a delta analysis.

The payload of the POST request specifies two required fields:

  1. commits: This is a JSON array containing one or more commits. CodeScene will run a delta analysis on all these commits by considering them as a single unit of work.
  2. repository: Specifies the Git repository where the commits that you want to analyse are located. You need to specify the repository name since an analysis project may contain multiple Git repositories.

Other optional parameters:

  • coupling_threshold_percent: Specifies minimal temporal coupling for the “Absence of Expected Change” warning. Default is 80 (%).
  • use_biomarkers: Instructs CodeScene to look for degrading code health, and enables reporting of the quality gates state. Note that this requires that the biomarkers are enabled for the analysis project.

Let’s say that we have created an analysis project by specifying a Git remote:

In this case, the repository payload parameter is PhpSpreadsheet (strip the .git extension). If we want to simulate a delta analysis of the commit designated by the hash 99e5a8e919e1f7b83371a8a586fd6d7875f63583 we issue the following request:

curl -X POST -d '{"commits": ["149f9e6"], "repository": "PhpSpreadsheet"}' http://localhost:3003/projects/64/delta-analysis -u 'CodeReview:MyPassword' -H "content-type: application/json"

You can also specify a custom temporal coupling threshold:

curl -X POST -d '{"commits": ["149f9e6"], "repository": "PhpSpreadsheet", "coupling_threshold_percent": 50}' http://localhost:3003/projects/64/delta-analysis -u 'CodeReview:MyPassword' -H "content-type: application/json"

Finally, you can enable biomarkers to detect potential code quality problems early:

curl -X POST -d '{"commits": ["149f9e6"], "repository": "PhpSpreadsheet", "use_biomarkers": true}' http://localhost:3003/projects/64/delta-analysis -u 'CodeReview:MyPassword' -H "content-type: application/json"

The example assumes that 1) CodeScene runs on localhost and 2) we have configured a user named CodeReview.

Notes to Windows Users: The curl syntax above won’t work on Windows unless you escape the payload properly. We recommend that you use Fiddler instead of curl if you want to test the API on Windows.

Once you’ve issued the POST request above, CodeScene’s Delta Analysis will respond with the following JSON document:

           "description":"The change is low risk as it touches less code than your typical change set",
           "improvements":[" improves its code health from 5.6 -> 6.2"],

The parameters in the response carry the following meaning:

  • version: This is the version of the REST API and will change in future versions of CodeScene.
  • url: This URL points to the Delta Analysis resource in CodeScene. You can fetch it with an HTTP GET request at any time and it will return the same result document.
  • view: Points to the page in CodeScene that contains the graphical representation of the result as illustrated in Fig. 171.
  • result: This JSON object contains four fields:
    • risk is the risk classification of the commit(s), range 1-10.
    • description is a human readable interpretation of the risk calculated by CodeScene’s machine learning algorithms.
    • improvements is a positive reinforcement that shows code that improves its health.
    • quality-gates specifies the state of the two gates that can, optionally, be checked and enforced in the requesting build pipeline.
    • warnings specify any early warnings like Complexity Trend increases. In this case it’s a low risk commit without any early warning.

Now, let’s look at a more complex result. In this case a new developer has made a modification to one of the top Hotspots on a separate branch. The Delta Analysis reports the following results:

           "description":"The change is high risk as it is a more wide-spread
                          change (1232 lines of code in 32 files) than your
                          typical change patterns.
                          The risk increases as the author has somewhat lower
                          experience in this repository.",
             {"category":"Modifies Hotspot",
             {"category":"Absence of Expected Change Pattern",

We see that CodeScene reports a high risk of 10. We also note that CodeScene calls our attention to the modified Hotspot. We use this information to review the change more carefully. Finally we note that CodeScene detects the absence of an expected change pattern. In this codebase, the LinkTagHelper.cs is usually changed together with the ScriptTagHelper.cs file. Since that wasn’t the case here, CodeScene informs us about the omission so that we can investigate it and catch a potential bug early.

Finally, let’s see how a failed code quality gate looks:

           "description":"The change is high risk as it is a more wide-spread
                          change (1232 lines of code in 32 files) than your
                          typical change patterns.
                          The risk increases as the author has somewhat lower
                          experience in this repository.",
           "code-owners-for-quality-gates":["@TheTester" "@TheMicroManager"],
             {"category":"Degrades in Code Health",
              "details":["ViewComponentResultTest.cs degrades from a Code Health of 10.0 -> 9.0"
                         "ControllerActionInvokerTest.cs degrades from a Code Health of 5.0 -> 4.7"]},
             {"category":"Violates Goals",
              "details": ["Hotspots marked supervise, ControllerActionInvokerTest.cs, degrades from a Code Health of 5.0 -> 4.67"]}]}}

The failed quality gates are indicated in the quality-gates field. We also note that CodeScene includes the code-owners-for-quality-gates field, which specifies the two owners – as read from a CODEOWNERS file – that are responsible for the code that failed the quality gates.

Delta Analysis in Offline Mode

Delta analysis is triggered via an API call and thus requires authentication. Since CodeScene checks the license on a remote license server with every authentication request, the delta analysis API call will fail if CodeScene can’t reach the license server. If you don’t have an Internet connection or you don’t want to let CodeScene access the Internet, you need to specify offline-mode parameter:

curl -X POST -d '{"commits": ["149f9e6"], "repository": "PhpSpreadsheet"}' http://localhost:3003/projects/64/delta-analysis?offline-mode=offline -u 'CodeReview:MyPassword' -H "content-type: application/json"

Please note that there is a new offline-mode=offline parameter in the query string.

If you always run CodeScene in offline mode, you can also turn on Global Offline Mode in the configuration. With global offline mode, you don’t have to append offline-mode=offline parameter to your delta analysis API URLs.

Read more about the limitations and usage in Offline Mode documentation.