Static analysis and code coverage are tools - not goals

Dec 28, 2020

I like structure in my code and over a period of 5-10 years, tooling in PHP has improved immensely to improve the structure and readability of code.

xDebug code coverage, PHPStan and the Laravel specific LaraStan can all help with identifying troublesome places in the code that are worty of a critical eye, but it can be appealing to use them as metrics. I want to warn against this practice.

Consider this Laravel code:

public function destroy(Content $content)
    if(!$content->delete()) {
        abort(503, 'Delete error');
    return $this->response->noContent();

A PHPUnit test that tests the happy path - delete content, it completes and a 204 No Content response is returned - will result in code coverage marking the abort() call as untested.

PHPStan/LaraStan will say that there are cyclical dependencies (although not an alarming number in this example)

Both tools will alert us that something should be done about this code, but what? If we focus only on the feedback and if we have a metric or goal of 100% code coverage and no warnings from ***Stan, we could be inclined to use the Laravel abort_unless function like this:

public function destroy(Content $content)
    abort_unless($content->delete(), 503, 'Delete error');
    return $this->response->noContent();

Gone is the cyclical dependency and we have full code coverage - no matter the input and processing, all lines are executed - but are we actually testing all cases? We are not.

In order to have full logic coverage in our tests, we need a test to ensure that we get the correct 503 Internal Server Error if the content could not be deleted.

Therefore, I always use code coverage and static analysis as guidelines for thinking things through rather than hard goals for developers to be measured on.

Goodhart's law generalized by Marilyn Strathern: "When a measure becomes a target, it ceases to be a good measure."

by Jan Keller

Long time developer, architect and CTO with a real love for the backend. On this blog, I give out insight gathered through working with Vue, Nuxt, Laravel and more, when building and maintaining our SaaS.

Catch me on Twitter and Github

Restructuring a Laravel application using Domains, Actions and more

In coding tutorials, much is often left out in order not to be too verbose, to keep the complexity down or because it feels over engineering to follow through with the examples. I will give a full structural suggestion based on my experience that can be applied directly, regardless of the project size.

Jan Keller Jun 8, 2022

Mutation Testing - the missing link

Mutation testing can give your tests that extra confidence level not achieved simply by measuring code test coverage.

Jan Keller Dec 15, 2021

TDD without tests first approach

TDD is synonymous with writing tests first and implementations later. I want to challenge that approach as a catch-all do-all approach.

Jan Keller Dec 15, 2021

Lessons learned migrating to Laravel Vapor

When moving to Laravel Vapor, there are things to consider. In this article, I name the obstacles I have experienced in a recent project.

Jan Keller Jun 4, 2021