[Toolchain] Static Code Analysis

[Lessaire]

Spoiler: anyway...

After having a wonderfully colorful conversation about the importance of maintaining a rich standard of pre-flight signal-to-noise-ratio checks before opening a GitHub issue, I have now learned that VCG is dead last among CERN's listed SCA tools for C++. The shame I feel for not already knowing this is immense.

So, despite my repeated urging that creating this thread would be a good idea and they should do so, it seems I have to create it myself...

The idea of running Static Code Analysis tools on the server codebase has merit and is worthy of investigation. But there is enough noise on the Github already. So I make this thread as a place where the development of a strategy for how best to approach this can be hashed out, so that such analyses may be incubated into consumable units of execution before becoming yet another ping. Lord knows we all likely have enough of those in our lives already.

So, reducing the scope to the immediately useful options to accomplish this:

Tools:

Professional services that offer useful trials or free access for Open Source projects:

• Coverity Scan is free, but requires sign up by a maintainer, and so I believe this would fall squarely into @Don Daniello's lap.
• PVS-Studio now offers a free one year license to open source projects. Sign up is here, and I believe anyone that has collaborator status can be the one to do it. This is not like their free personal license and no code alterations are prerequisite.
• Embold, formerly Acellere Gamma, is free for open source projects. Needs to be done by a maintainer, here.
• CppDepend offers a 14-day trial, and their EULA does not prohibit the dissemination of it's output to team members.

Community editions:

• SonarQube

Open source tools with focus on C/C++:

• cppcheck
• Clang Analyzer
• Clang Tidy
• Tentacle of Zuckerberg

Tools written in Visual Basic:

• VCG

Spoiler: Considered, but found unsuitable

Frama-C: It's lovely, mature, and powerful. And chokes pretty hard once you start using advanced C++ features. It's meant for pure C, and that will become evident very quickly here.

Parasoft: A shame their OSS requirements are so high, I've never seen it's equal elsewhere.
PC-Lint by Gimpel: I believe the terms make it unsuitable as they require the evaluator not share the findings unless a license is purchased.
Klocwork: They offer trial licenses but in my experience they are thoroughly dedicated to corporate-type customers
Axivion: Bauhaus started out as a university project but is thoroughly commercialized and I've heard they'd rather the world forgot it's academic past.
Veracode: Sees open source merely as a commercially exploitable asset.
Grammatech: They support open source as an idea, support disclosure of vulns to oss libs when their tools find such as a matter of course in the service of licensed customers, have released parts of their toolchain as open source, but as far as I know do not offer free trials to OSS devs. Their purview tends to be military customers.
Kiuwan: Supports opensource as a concept. And supports using their products on opensource projects of paying corporate entities.

Parsing:

With an array of tools covered, we are left with the biggest hurdle of how to deal with the prolific outputs of such tools. SASTs are famous in general for how much noise they can make, and the sheer number of products on the market is a sure indicator of just how complicated reducing analysis to useful information can be.

Ericssons CodeChecker natively supports the Clang tools and can import reports from both cppcheck and Infer. Seems like it could be a useful place to start.

I've seen a number of strategies deployed in corporate settings, they tend to be obvious, like writing parsers to unify output format and then sort issues by the occurence rate, or require the one element missing here: paying people money to spend hours doing a tedious thankless job.

And this is where I invite input from those with any vested interest in this idea.

 

[Don Daniello]

We've been using Coverity for the past decade. The reports can't be publicly accessible but developers have access to it.

 

[Lessaire]

Aye, I thought it would be odd if there was such an obvious gap in the usual development lifecycle map. It's a shame the person who prompted this query couldn't be bothered to make this thread themselves, even after I suggested they should. I guess just consider this my exercise of due diligence after such an exchange.