How to set up SonarQube code duplication by folder or group of projects and not for the whole Solution in a single Azure DevOps repository?

Question

In our product stack, we have an intricate projects structure.

There is a main app that is responsible for core functions (routing, auth, etc.) and there are submodules. Submodules each consist of two Projects.

The solution structure is following:

UI for web stuff (js, cshtml) and Application for business logic (C#).

We have three submodules (total six projects (3modules*2projects) that run under the main app)

We have introduced SonarQube analysis, and for code duplications, by default, all six projects were scanned and it gave quite a few false positives. The modules are not intended to be coupled and anything overlapping is intentional. In SonarQube one can exclude or include things by file folder etc, I couldn't find a setting how to run it per set of things.

Our azure pipeline relevant bits are following:

- variables:
  buildConfiguration: 'Release'
  SolutionPath: 'src/Solution.sln'

steps:
- task: NuGetToolInstaller@1
  inputs:
    versionSpec: '5.11.0'
    checkLatest: true

- task: DotNetCoreCLI@2
  displayName: "Restoring Company.Department.UI Solution"
  inputs:
    command: 'restore'
    projects: '$(SolutionPath)'
    feedsToUse: 'config'
    nugetConfigPath: 'src/nuget.config'

- task: PowerShell@2
  displayName: Scan for NuGet vulnerabilities
  enabled: true
  inputs:
    targetType: inline
    script: >-
        $solutionPath = '$(SolutionPath)'

        Write-Host $solutionPath

        ($output = dotnet list "$solutionPath" package --vulnerable)

        $output

        $errors = $output | Select-String '>'

        if ($errors.Count -gt 0) {
            foreach ($err in $errors) {
                Write-Host "##vso[task.logissue type=error]Found vulnerable NuGet package $err"
            }

            exit 1
        }

        exit 0

- task: UseNode@1
  displayName: Use Node 16.x
  inputs:
    version: 16.x
    checkLatest: true

- task: SonarQubePrepare@5
  inputs:
    SonarQube: 'SonarQube'
    scannerMode: 'MSBuild'
    projectKey: 'R-Department_Department-Frontend'
    extraProperties: "# Additional properties that will be passed to the scanner, \n# Put one key=value per line, example:\n# sonar.exclusions=**/*.bin\nsonar.cs.vscoveragexml.reportsPaths=$(Agent.TempDirectory)/**/*.coveragexml\nsonar.coverage.exclusions=**/*.js,**/*.cshtml"

- task: DotNetCoreCLI@2
  displayName: "Building"
  inputs:
    command: 'build'
    projects: $(SolutionPath)
    arguments: --configuration $(buildConfiguration)

- task: DotNetCoreCLI@2
  displayName: "Running Tests"
  inputs:
      command: test
      projects: >-
        **/*Test*.csproj

        !**\*TestAdapter.dll;

        !**\obj\**
      arguments: --configuration $(buildConfiguration) --no-build --collect "Code Coverage"

- task: PowerShell@2
  displayName: Create coveragexml file
  inputs:
    targetType: inline
    script: >-
        Get-ChildItem -Recurse -Filter "*.coverage" | % {

        $outfile = "$([System.IO.Path]::GetFileNameWithoutExtension($_.FullName)).coveragexml"

        $output = [System.IO.Path]::Combine([System.IO.Path]::GetDirectoryName($_.FullName), $outfile)

        "Analyse '$($_.Name)' with output '$outfile'..."

        . $env:USERPROFILE\.nuget\packages\microsoft.codecoverage\15.8.0\build\netstandard1.0\CodeCoverage\CodeCoverage.exe analyze /output:$output $_.FullName

        }

        "Done"

- task: SonarQubeAnalyze@5
  displayName: Run Code Analysis

- task: SonarQubePublish@5
  displayName: Publish Quality Gate Result

Update: What have we tried so far was to set up four SonarQube projects, that point so same folder on local. Then run four times

...
- task: SonarQubePrepare@5
 ...
- task: SonarQubeAnalyze@5
- task: SonarQubePublish@5
...

Three runs while filtering out all the other sub-project folders, and shared one Forth run filtering out the sub-projects and running it for the Shared one.

Which worked on local

However in our development infrastructure, we use Azure DevOps and there we cannot set up multiple SonarQube projects pointing to same repository.

Running with same SonarQube project id four times just overwrites the previous run and you get only results for whichever is last.

I am sure there has to be a way to configure code duplication to run per folder or per group of projects.

How to set up to SonarQube check for code duplication by folder or per group of projects and not for the whole Solution in a single Azure DevOps repository?

Answer 1

One solution is to create something like following:

Where first time we exclude everything but the Project1 (o) Second we exclude everything but Project2 (c) Last we exclude everything but project3 (m) And then we exclude all projects and run scan for what is left apart from Projects 1, 2 and 3

Then set up in SonarCube projects accordingly.

trigger: none
pr: [branch-name]

pool:
  vmImage: 'aa-latest'
  name: 'NAT'
  
variables:
  buildConfiguration: 'Release'
  SolutionPath: 'src/XXX.YYY.Frontend.sln'

steps:
- task: NuGetToolInstaller@1
  inputs:
    versionSpec: '5.11.0'
    checkLatest: true

- task: DotNetCoreCLI@2
  displayName: "Restoring XXX.YYY.UI Solution"
  inputs:
    command: 'restore'
    projects: '$(SolutionPath)'
    feedsToUse: 'config'
    nugetConfigPath: 'src/nuget.config'

- task: PowerShell@2
  displayName: Scan for NuGet vulnerabilities
  enabled: true
  inputs:
    targetType: inline
    script: >-
        $solutionPath = '$(SolutionPath)'

        Write-Host $solutionPath

        ($output = dotnet list "$solutionPath" package --vulnerable)

        $output

        $errors = $output | Select-String '>'

        if ($errors.Count -gt 0) {
            foreach ($err in $errors) {
                Write-Host "##vso[task.logissue type=error]Found vulnerable NuGet package $err"
            }

            exit 1
        }

        exit 0

- task: UseNode@1
  displayName: Use Node 16.x
  inputs:
    version: 16.x
    checkLatest: true

- task: SonarQubePrepare@5
  inputs:
    SonarQube: 'SonarQube'
    scannerMode: 'MSBuild'
    projectKey: 'RBS-YYY_YYY-Frontend'
    extraProperties: "# Additional properties that will be passed to the scanner, \n# Put one key=value per line, example:\n# sonar.exclusions=**/*.bin\nsonar.cs.vscoveragexml.reportsPaths=$(Agent.TempDirectory)/**/*.coveragexml\nsonar.coverage.exclusions=**/*.js,**/*.cshtml\nsonar.exclusions=**/MJ/m/**,**/MJ/c/**,**/MJ/o/**\nsonar.issue.ignore.multicriteria=c,m,o,shared\nsonar.issue.ignore.multicriteria.c.ruleKey=*\nsonar.issue.ignore.multicriteria.c.resourceKey=**/MJ/c/**\nsonar.issue.ignore.multicriteria.m.ruleKey=*\nsonar.issue.ignore.multicriteria.m.resourceKey=**/MJ/m/**\nsonar.issue.ignore.multicriteria.o.ruleKey=*\nsonar.issue.ignore.multicriteria.o.resourceKey=**/MJ/o/**\nsonar.issue.ignore.multicriteria.shared.ruleKey=*\nsonar.issue.ignore.multicriteria.shared.resourceKey=**/XXX.YYY.Shared.Application/**\nsonar.test.exclusions=**/*.Tests/**"

- task: DotNetCoreCLI@2
  displayName: "Building"
  inputs:
    command: 'build'
    projects: $(SolutionPath)
    arguments: --configuration $(buildConfiguration)

- task: DotNetCoreCLI@2
  displayName: "Running Tests"
  inputs:
      command: test
      projects: >-
        **/*Test*.csproj

        !**\*TestAdapter.dll;

        !**\obj\**
      arguments: --configuration $(buildConfiguration) --no-build --collect "Code Coverage"

- task: PowerShell@2
  displayName: Create coveragexml file
  inputs:
    targetType: inline
    script: >-
        Get-ChildItem -Recurse -Filter "*.coverage" | % {

        $outfile = "$([System.IO.Path]::GetFileNameWithoutExtension($_.FullName)).coveragexml"

        $output = [System.IO.Path]::Combine([System.IO.Path]::GetDirectoryName($_.FullName), $outfile)

        "Analyse '$($_.Name)' with output '$outfile'..."

        . $env:USERPROFILE\.nuget\packages\microsoft.codecoverage\15.8.0\build\netstandard1.0\CodeCoverage\CodeCoverage.exe analyze /output:$output $_.FullName

        }

        "Done"

- task: SonarQubeAnalyze@5
  displayName: Run Code Analysis

- task: SonarQubePublish@5
  displayName: Publish Quality Gate Result

- task: SonarQubePrepare@5
  displayName: "SonarQube Prepare (m)"
  inputs:
    SonarQube: 'SonarQube'
    scannerMode: 'MSBuild'
    projectKey: 'YYY-Frontend-m'
    extraProperties: "# Additional properties that will be passed to the scanner, \n# Put one key=value per line, example:\n# sonar.exclusions=**/*.bin\nsonar.cs.vscoveragexml.reportsPaths=$(Agent.TempDirectory)/**/*.coveragexml\nsonar.coverage.exclusions=**/*.js,**/*.cshtml\nsonar.exclusions=**/MJ/c/**,**/MJ/o/**\nsonar.issue.ignore.multicriteria=c,o,shared\nsonar.issue.ignore.multicriteria.c.ruleKey=*\nsonar.issue.ignore.multicriteria.c.resourceKey=**/MJ/c/**\nsonar.issue.ignore.multicriteria.o.ruleKey=*\nsonar.issue.ignore.multicriteria.o.resourceKey=**/MJ/o/**\nsonar.issue.ignore.multicriteria.shared.ruleKey=*\nsonar.issue.ignore.multicriteria.shared.resourceKey=**/XXX.YYY.Shared.Application/**\nsonar.test.exclusions=**/*.Tests/**"

- task: DotNetCoreCLI@2
  displayName: "Building (m)"
  inputs:
    command: 'build'
    projects: $(SolutionPath)
    arguments: --configuration $(buildConfiguration)

- task: SonarQubeAnalyze@5
  displayName: Run Code Analysis (m)

- task: SonarQubePublish@5
  displayName: Publish Quality Gate Result (m)

- task: SonarQubePrepare@5
  displayName: "SonarQube Prepare (cs)"
  inputs:
    SonarQube: 'SonarQube'
    scannerMode: 'MSBuild'
    projectKey: 'YYY-Frontend-cs'
    extraProperties: "# Additional properties that will be passed to the scanner, \n# Put one key=value per line, example:\n# sonar.exclusions=**/*.bin\nsonar.cs.vscoveragexml.reportsPaths=$(Agent.TempDirectory)/**/*.coveragexml\nsonar.coverage.exclusions=**/*.js,**/*.cshtml\nsonar.exclusions=**/MJ/m/**,**/MJ/o/**\nsonar.issue.ignore.multicriteria=m,o,shared\nsonar.issue.ignore.multicriteria.m.ruleKey=*\nsonar.issue.ignore.multicriteria.m.resourceKey=**/MJ/m/**\nsonar.issue.ignore.multicriteria.o.ruleKey=*\nsonar.issue.ignore.multicriteria.o.resourceKey=**/MJ/o/**\nsonar.issue.ignore.multicriteria.shared.ruleKey=*\nsonar.issue.ignore.multicriteria.shared.resourceKey=**/XXX.YYY.Shared.Application/**\nsonar.test.exclusions=**/*.Tests/**"

- task: DotNetCoreCLI@2
  displayName: "Building (cs)"
  inputs:
    command: 'build'
    projects: $(SolutionPath)
    arguments: --configuration $(buildConfiguration)

- task: SonarQubeAnalyze@5
  displayName: Run Code Analysis (cs)

- task: SonarQubePublish@5
  displayName: Publish Quality Gate Result (cs)

- task: SonarQubePrepare@5
  displayName: "SonarQube Prepare (o)"
  inputs:
    SonarQube: 'SonarQube'
    scannerMode: 'MSBuild'
    projectKey: 'YYY-Frontend-o'
    extraProperties: "# Additional properties that will be passed to the scanner, \n# Put one key=value per line, example:\n# sonar.exclusions=**/*.bin\nsonar.cs.vscoveragexml.reportsPaths=$(Agent.TempDirectory)/**/*.coveragexml\nsonar.coverage.exclusions=**/*.js,**/*.cshtml\nsonar.exclusions=**/MJ/c/**,**/MJ/m/**\nsonar.issue.ignore.multicriteria=m,c,shared\nsonar.issue.ignore.multicriteria.m.ruleKey=*\nsonar.issue.ignore.multicriteria.m.resourceKey=**/MJ/m/**\nsonar.issue.ignore.multicriteria.c.ruleKey=*\nsonar.issue.ignore.multicriteria.c.resourceKey=**/MJ/c/**\nsonar.issue.ignore.multicriteria.shared.ruleKey=*\nsonar.issue.ignore.multicriteria.shared.resourceKey=**/XXX.YYY.Shared.Application/**\nsonar.test.exclusions=**/*.Tests/**"

- task: DotNetCoreCLI@2
  displayName: "Building (o)"
  inputs:
    command: 'build'
    projects: $(SolutionPath)
    arguments: --configuration $(buildConfiguration)

- task: SonarQubeAnalyze@5
  displayName: Run Code Analysis (o)

- task: SonarQubePublish@5
  displayName: Publish Quality Gate Result (o)