CI/CD Integration with CodeScene’s Delta Analysis

CodeScene identifies and prioritizes technical debt, while at the same time uncovering and measuring social factors of the organization behind the system. The earlier you can react to any potential finding, the better. That’s why CodeScene offers integration points that let you incorporate the analysis results into your build pipeline.

The purpose of a Delta Analysis is to:

  • Prioritize code reviews based on the risk of the commits.

  • Use quality gates for the goals specified on identified hotspots inside CodeScene.

  • Specify quality gates that trigger in case the Code Health of a hotspot declines.

  • Get early warnings when new complex code is written, and detect the absence of expected changes that might indicate bugs.

In particular, it’s during the delta analysis that you identify any violations of the goals you have specified.

Example on the review recommendations: CodeScene notifies reviewers when critical code with potential security implications is modified.

Fig. 167 Example on the review recommendations: CodeScene notifies reviewers when critical code with potential security implications is modified.

Select a Delta Analysis Strategy

CodeScene has two different strategies for the delta analysis:

  1. Full Scan: The Full Scan will supervise the code health of all files in a change set or pull request. CodeScene will always use the Full Scan for PR integrations to cover all modified content, but for delta analysis requested over the API there is a threshold; if a change set exceeeds the default limit of 50 files, then CodeScene automatically reverts to a Hotspot Scan in order to maintain quick response times. You can override this threshold via environment variables:

    • env variable: DELTA_ANALYSIS__LIMIT_DELTA_SCAN_TO_MAX_FILES

    • system property: delta.analysis_limit.delta.scan.to.max.files

  2. Hotspot Scan: With this strategy, CodeScene limits the delta analysis to files that are identified as hotspots. This means that the most relevant modules are always checked. Non-hotspots will not be analysed for code health decline.

Integrate CodeScene in your Continuous Integration Pipeline

Connect automatically to external webhooks managed by CodeScene

CodeScene’s delta analysis can be connected to the common Git repository hosting services using webhooks. CodeScene has integrations for GitHub, BitBucket, Azure DevOps, etc. out of the box as defined in Integrate CodeScene in your CI/CD Pipeline.

CodeScene will install webhooks at the remote service that will trigger a Delta Analysis when a pull request is created or updated. The results of the Delta Analysis will be visible as a comment on the pull request.

Connect using CodeScene’s REST API

CodeScene provides a REST API that lets you integrate the analysis results in a continuous integration pipelinei and/or as robot comments in a code review tool like Gerrit.

The REST API for custom integrations is described below.

Delta Analysis: Use Cases and Quality Gates

A Delta Analysis is triggered by a pull request, a range of commits or a single commit; You decide through the API. Each time you trigger a Delta Analysis, CodeScene calculates the following information in order of importance:

  1. Hotspot Goals: Quality Gate. CodeScene ensures that none of the goals you have specified on a hotspot (see Manage Hotspots and Technical Debt with Goals) are violated.

  2. Code Health Decline: Quality Gate. This quality gate is triggered if any code declines in code health.

  3. Low Code Health in New Code. This quality gate is triggered when a new module has a code health below a pre-defined threshold. This ensure that new code doesn’t start with preventable code quality issues or reckless technical debt.

  4. Detects absent change patterns. If a cluster of files have changed together for a long time they are intimately related. This warning fires when such a temporal change pattern is broken. Please note that this may be good - we’ve refactored something - but it may also be a sign of omission and a potential bug. As a consequence, this warning is based on a self-correcting algorithm; If you keep ignoring the warning it will go away automatically as the temporal coupling decreases below the thresholds.

  5. Positive Reinforcements. CodeScene also detects hotspots that improve their code health. This information is included as a positive reinforcement intended to show the effect of the current change set on the overall well-being of the codebase.

  6. Recommended Review Level. When enabled, CodeScene posts a recommendation to perform a detailed code review on any PRs that has a higher probability for risks.The recommended review level is relative to the risk profile for your codebase as described below. Use this information to prioritize code reviews and to decide upon delivery risks.

A Delta Analysis gives you early warnings, actionable quality gates, and detects high risk changes

Fig. 168 An example on CodeScene’s quality gates where Details contain actionable feedback (example from CodeScene’s GitHub integration).

Use a Delta Analysis to Save Time in Code Reviews

The main advantage of a delta analysis if that it lets you react to potential problems early. But there’s a potentially large saving at the other end of the spectrum too; Instead of treating all pull requests as equals, CodeScene’s risk classification lets you prioritize your code reviews and focus your time where (and when) it’s likely to be needed the most. Code reviewer fatigue is a real thing, so let’s use our review efforts wisely.

Code review tools like Gerrit lets you select a label. For example, you specify a label that either allows or blocks the change. In addition you may select a label as an opinion (+1 and -1 in Gerrit).

When you integrate CodeScene with Gerrit, it’s our recommendation that you map CodeScene’s risk classification to an automated +1 or -1. For example, all commits below the risk category 3 may be +1, which indicates to the reviewers how much time they need to spend on this review.

In addition, the delta analyses lets you auto-detect files that seem to degrade in quality through issues introduced in the current commit or pull request. This is done by calculating code health (see Code Health identifies factors known to impact Maintenance costs and Delivery Risks), which are then supervised for their trend as shown in Fig. 169.

Degrading code health

Fig. 169 A Delta Analysis detects degrading Code Health.

CodeScene’s delta analysis works in the other direction too; It’s not only about spotting problems. If you enable the quality gates and work pro-actively with Intelligent Notes in CodeScene, then you can allocate less time on the features with 1) low risk, and 2) passing gates, as shown in Fig. 170.

Quality gates are OK

Fig. 170 Spend less time on reviewing code with low risk and passing quality gates.

Keep up the Good

CodeScene also detects hotspots that improve their code health. This information is included as a positive reinforcement intended to show the effect of the current changeset on the overall code health. Fig. 171 shows an example where a hotspot is successfully refactored – an occasion to celebrate!

Code improvements detected in the delta analysis

Fig. 171 Measure code improvements as part of the CI/CD pipeline to reinforce a positive trend.

Notify the Code Owners on Failed Quality Gates

As discussed in Knowledge Distribution, CodeScene parses and includes code ownership information when present. If you have this feature enabled, then CodeScene will include a mention of the code owners should a quality gate fail. You see an example in Fig. 172.

CodeScene notifies code owners when a quality gate fails.

Fig. 172 CodeScene notifies code owners when a quality gate fails.

The REST API for Delta Analyses

CodeScene provides out of the box integrations with code review platforms like GitHub, BitBucket, GitLab, etc. In addition, CodeScene also provides an open REST API that lets you write custom integrations. The API is described below.

CodeScene lets you create a special Bot user role intended to consume the REST API. Login as administrator and create a Bot user for each of your integration points as illustrated in Fig. 173

Configure a Bot user

Fig. 173 Configure a Bot user for each of your integration points.

You trigger a Delta Analysis by POSTing a request to the REST endpoint specified in your analysis configuration as illustrated in Fig. 174.

REST API endpoint

Fig. 174 Your analysis configuration specifies the REST endpoint to trigger a delta analysis.

The payload of the POST request specifies two required fields:

  1. commits: This is a JSON array containing one or more commits. CodeScene will run a delta analysis on all these commits by considering them as a single unit of work.

  2. repository: Specifies the Git repository where the commits that you want to analyse are located. You need to specify the repository name since an analysis project may contain multiple Git repositories.

Other optional parameters:

  • coupling_threshold_percent: Specifies minimal temporal coupling for the “Absence of Expected Change” warning. Default is 80 (%).

  • use_biomarkers: Instructs CodeScene to look for degrading code health, and enables reporting of the quality gates state.

Let’s say that we have created an analysis project by specifying a Git remote:

In this case, the repository payload parameter is PhpSpreadsheet (strip the .git extension). If we want to simulate a delta analysis of the commit designated by the hash 99e5a8e919e1f7b83371a8a586fd6d7875f63583 we issue the following request:

curl -X POST -d '{"commits": ["149f9e6"], "repository": "PhpSpreadsheet"}' http://localhost:3003/projects/64/delta-analysis -u 'CodeReview:MyPassword' -H "content-type: application/json"

You can also specify a custom temporal coupling threshold:

curl -X POST -d '{"commits": ["149f9e6"], "repository": "PhpSpreadsheet", "coupling_threshold_percent": 50}' http://localhost:3003/projects/64/delta-analysis -u 'CodeReview:MyPassword' -H "content-type: application/json"

Finally, you can enable biomarkers to detect potential code quality problems early:

curl -X POST -d '{"commits": ["149f9e6"], "repository": "PhpSpreadsheet", "use_biomarkers": true}' http://localhost:3003/projects/64/delta-analysis -u 'CodeReview:MyPassword' -H "content-type: application/json"

The example assumes that 1) CodeScene runs on localhost and 2) we have configured a user named CodeReview.

Notes to Windows Users: The curl syntax above won’t work on Windows unless you escape the payload properly. We recommend that you use Fiddler instead of curl if you want to test the API on Windows.

Once you’ve issued the POST request above, CodeScene’s Delta Analysis will respond with the following JSON document:

{"version":"1",
 "url":"/projects/64/delta-analysis/75686456d695d60d99a7cd73302f83606c8a8efc",
 "view":"/64/delta-analysis/view/75686456d695d60d99a7cd73302f83606c8a8efc",
 "result":{"recommended-review-level":"Lightweight sanity check",
           "improvements":["Connector.java improves its code health from 5.6 -> 6.2"],
           "new-files-info": {
               "summary": "1 new file where the code health is below the threshold for new code, 8.1",
               "warnings": [
                 {
                   "name": "my-repo\/src\/a_new_file.c",
                   "review": [
                     {
                       "title": "Bumpy Road Ahead",
                       "description": "The code is complex to read due to its nesting with multiple logical blocks. The most complex function is putmsg with 3 logical blocks. A bumpy road like putmsg indicates a lack of encapsulation. Consider to extract smaller, cohesive functions from the bumpy functions.",
                       "indication": 3
                     },
                   ]
                 }
               ]
             },
           "quality-gates":{"degrades-in-code-health":true,
                            "violates-goal":false},
           "warnings":[]}}

The parameters in the response carry the following meaning:

  • version: This is the version of the REST API and will change in future versions of CodeScene.

  • url: This URL points to the Delta Analysis resource in CodeScene. You can fetch it with an HTTP GET request at any time and it will return the same result document.

  • view: Points to the page in CodeScene that contains the graphical representation of the result as illustrated in Fig. 168.

  • result: This JSON object contains four fields:
    • recommended-review-level is based on the risk profile of the repository, and the riskier the change, the more time we recommend spending on the review.

    • improvements is a positive reinforcement that shows code that improves its health.

    • new-files-info is an _optional_ field that’s included if any new files are introduced in the change set. New content with a low code health fails the quality gate and CodeScene presents the issues.

    • quality-gates specifies the state of the two gates that can, optionally, be checked and enforced in the requesting build pipeline.

    • warnings specify any early warnings like Complexity Trend increases. In this case it’s a low risk commit without any early warning.

Deprecated Fields

The following fields are included in the response, but will be removed from future CodeScene versions:

  • risk: use the recommended-review-level instead.

  • description: use the recommended-review-level instead.

Quality Gates in the REST API

Now, let’s look at a more complex result. In this case a new developer has made a modification to one of the top Hotspots on a separate branch. The Delta Analysis reports the following results:

{"version":"1",
 "url":"/projects/2/delta-analysis/43cc8a146cc0957f2fcb4b09ae3dee71d5a5cf2e",
 "view":"/2/delta-analysis/view/43cc8a146cc0957f2fcb4b09ae3dee71d5a5cf2e",
 "result":{"recommended-review-level":"Detailed: the change is high risk as it is a more wide-spread
                                       change than 98% of all your commits.
                                       The risk increases as the author has somewhat lower
                                       experience in this repository.",
           "improvements":[],
           "quality-gates":{"degrades-in-code-health":false,
                            "violates-goal":false},
           "warnings":[
             {"category":"Modifies Hotspot",
              "details":["mvc/src/Microsoft.AspNetCore.Mvc.TagHelpers/LinkTagHelper.cs"]},
             {"category":"Absence of Expected Change Pattern",
              "details":["mvc/src/Microsoft.AspNetCore.Mvc.TagHelpers/ScriptTagHelper.cs"]}]}}

We see that CodeScene recommends a detailed review due to the increased defect risk. We also note that CodeScene calls our attention to the modified Hotspot. We use this information to review the change more carefully. Finally we note that CodeScene detects the absence of an expected change pattern. In this codebase, the LinkTagHelper.cs is usually changed together with the ScriptTagHelper.cs file. Since that wasn’t the case here, CodeScene informs us about the omission so that we can investigate it and catch a potential bug early.

Finally, let’s see how a failed code quality gate looks:

{"version":"1",
 "url":"/projects/2/delta-analysis/43cc8a146cc0957f2fcb4b09ae3dee71d5a5cf2e",
 "view":"/2/delta-analysis/view/43cc8a146cc0957f2fcb4b09ae3dee71d5a5cf2e",
 "result":{"recommended-review-level":"Lightweight sanity check",
           "improvements":[],
           "quality-gates":{"degrades-in-code-health":true,
                            "violates-goal":true},
           "code-owners-for-quality-gates":["@TheTester" "@TheMicroManager"],
           "warnings":[
             {"category":"Degrades in Code Health",
              "details":["ViewComponentResultTest.cs degrades from a Code Health of 10.0 -> 9.0"
                         "ControllerActionInvokerTest.cs degrades from a Code Health of 5.0 -> 4.7"]},
             {"category":"Violates Goals",
              "details": ["Hotspots marked supervise, ControllerActionInvokerTest.cs, degrades from a Code Health of 5.0 -> 4.67"]}]}}

The failed quality gates are indicated in the quality-gates field. We also note that CodeScene includes the code-owners-for-quality-gates field, which specifies the two owners – as read from a CODEOWNERS file – that are responsible for the code that failed the quality gates.

Delta Analysis in Offline Mode

Delta analysis is triggered via an API call and thus requires authentication. Since CodeScene checks the license on a remote license server with every authentication request, the delta analysis API call will fail if CodeScene can’t reach the license server. If you don’t have an Internet connection or you don’t want to let CodeScene access the Internet, you need to specify offline-mode parameter:

curl -X POST -d '{"commits": ["149f9e6"], "repository": "PhpSpreadsheet"}' http://localhost:3003/projects/64/delta-analysis?offline-mode=offline -u 'CodeReview:MyPassword' -H "content-type: application/json"

Please note that there is a new offline-mode=offline parameter in the query string.

If you always run CodeScene in offline mode, you can also turn on Global Offline Mode in the configuration. With global offline mode, you don’t have to append offline-mode=offline parameter to your delta analysis API URLs.

Read more about the limitations and usage in Offline Mode documentation.