I've recently added two packages (and their dependencies) to Debian and thought I'd cover a bit more about why.

Black

black, the uncompromising Python code formatter, has arrived in Debian unstable and testing.

black is being adopted by the LAVA Software Community Project in a gradual way and the new CI will be checking that files which have been formatted by black stay formatted by black in merge requests.

There are endless ways to format Python code and pycodestyle and pylint are often too noisy to use without long lists of ignored errors and warnings. Black takes the stress out of maintaining a large Python codebase as long as a few simple steps are taken:

  • Changes due to black are not functional changes. A merge request applying black to a source code file must not include functional changes. Just the change done by black. This makes code review manageable.
  • Changes made by black are recorded and once made, CI is used to ensure that there are no regressions.
  • Black is only run on files which are not currently being changed in existing merge requests. This is a simple sanity provision, rebasing functional changes after running black is not fun.

Consistent formatting goes a long way to helping humans spot problematic code.

See https://black.readthedocs.io/en/stable/ or apt-get install python-black-doc for a version which doesn't "call home".

Radon

So much for code formatting, that's nice and all but what can matter more is an overview of the complexity of the codebase.

We're experimenting with running radon as part of our CI to get a CodeClimate report which GitLab should be able to understand.

https://docs.codeclimate.com/docs/workflow

https://git.metabarcoding.org/help/ci/examples/code_climate.md

(Take a bow http://vincentsanders.blogspot.com/2018/09/all-i-wanted-to-do-is-check-error-code.html - Vince gave me the idea by mentioning his use of Cyclomatic Complexity.)

What we're hoping to achieve here is a failed CI test if the complexity of critical elements increases and a positive indication if the code complexity of areas which are currently known to be complex can be reduced without losing functionality.

Initially, just having the data is a bonus. The first try at CodeClimate support took the best part of an hour to scan our code repository. radon took 3 seconds.

See https://radon.readthedocs.io/en/latest/ or apt-get install python-radon-doc for a version which doesn't "call home".

(It would be really nice for upstreams to understand that putting badges in their sphinx documentation templates makes things harder to distribute fairly. Fine, have a nice web UI for your own page but remove the badges from the pages in the released tarballs, e.g. with a sphinx build time option.)

One more mention - bandit

I had nothing to do with introducing this to Debian but I am very grateful that it exists in Debian. bandit is proving to be very useful in our CI, providing SAST reports in GitLab. As with many tools of it's kind, it is noisy at first. However, with a few judicious changes and the use of the # nosec comment to rule out scanning of things like unit tests which deliberately tried to be insecure, we have substantially reduced the number of reports produced with bandit.

Having the tools available is so important to actually fixing problems before the software gets released.