Free software activities in May 2020

Here is my monthly update covering what I have been doing in the free software world during May 2020 (previous month):

• Opened a pull request against the kitty shell to set a default socket timeout when retrieving remote items via the icat command-line tool. (#659)

• Opened a pull request to make the documentation for the Wand Python/ImageMagick graphics library to build in reproducible manner. [...]

• Fixed an issue in my tickle-me-email library that implements Gettings Things Done (GTD)-like behaviours in IMAP inboxes to prevent a traceback when adding text attachments that were not valid UTF-8. ...]

• The Open Source Initiative held their twice-annual multi-day 'face-to-face' board meeting — this time held virtually — and participated in the accompanying conversations on strategy, tactical and governance issues, as well as the usual discussions regarding licensing and policy (minutes pending). I also attended the regular monthly meeting for Software in the Public Interest (minutes).

• For my django-slack library (which provides a convenient library between projects using the Django and the Slack chat platform) I reviewed and merged a pull request from Cleiton de Lima to fix an issue due to the force_text method was deprecated favour of force_str. [...]

• Attended more of OpenUK's Teabreak Tuesday and Future Leaders' Training events as well did more work as part of being on the judging panel of the OpenUK Awards. Nominations are open through 15th June in five different categories.

• Opened a pull request for the Redis key-value database to correct a "conflicts" spelling error. [...]

In Lintian, the static analysis tool for Debian packages:

• New features:

  • Check for packages that use ${misc:Pre-Depends) in the Depends field. (#961290)
  • Check for packages installing icon cache files directly under /usr/share/icons/hicolor as they will invariably clash with other packages. (#959855)
  • Check for Homepage fields in debian/control that point to known directory listing pages. (#960366)
  • Update data/fields/perl-provides. [...]

• Bug fixes:

  • Mark that dh-sequence-gnome provides the services of dh_gnome and dh_gnome_clean. (#958500)
  • Accommodate commented-out lines when checking for no-dh-sequencer. (#960485)

• Reporting/output:

  • Clarify error message when the file to check does not exist. (#960970)
  • Mark the debian-rules-uses-as-needed-linker-flag tag as being pedantic & experimental until at least the release of Debian bullseye. (#959696)

• Code improvements:

  • Replace Copyright (C) with the Unicode copyright symbol for consistency [...] and update my copyright years [...].
  • Factor out matching Homepage fields to data/fields/bad-homepages. [...]
  • Various alterations for the continuous integration pipeline. [...][...]

Reproducible builds

One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes.

The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

The initiative is proud to be a member project of the Software Freedom Conservancy, a not-for-profit 501(c)(3) charity focused on ethical technology and user freedom.

Conservancy acts as a corporate umbrella allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter.

• Opened a pull request to make the documentation for the Wand Python/ImageMagick graphics library to build in reproducible manner. [...]

• In Debian:

  • Submitted 16 patches to fix specific reproducibility issues in ataqv, bowtie2, briquolo, cryptominisat, elinks, mistral, nulib2, process-cpp, properties-cpp, python-watcherclient, tree-puzzle, tree-puzzle, tree-puzzle, vows, wand & wolfssl.

  • Kept isdebianreproducibleyet.com up to date. [...]

• Categorised a large number of packages and issues in the Reproducible Builds "notes" repository.

• In disorderfs, our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues, I replaced the term "dirents" in place of "directory entries" in human-readable output/log messages [...] and used the astyle source code formatter with the default settings to the main disorderfs.cpp file [...].

• Drafted, published and publicised our monthly report.

Elsewhere in our tooling, I made the following changes to diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues, including preparing and uploading versions 142, 143, 144, 145 and 146 to Debian:

• Comparison improvements:

  • Improve fuzzy matching of JSON files as file now supports recognising JSON data. (#106)
  • Refactor .changes and .buildinfo handling to show all details (including the GnuPG header and footer components) even when referenced files are not present. (#122)
  • Use our BuildinfoFile comparator (etc.) regardless of whether the associated files (such as the orig.tar.gz and the .deb) are present. [...]
  • Include GnuPG signature data when comparing .buildinfo, .changes, etc. [...]
  • Add support for printing Android APK signatures via apksigner(1). (#121)
  • Identify "iOS App Zip archive data" as .zip files. (#116)
  • Add support for Apple Xcode .mobilepovision files. (#113)

• Bug fixes:

  • Don't print a traceback if we pass a single, missing argument to diffoscope (eg. a JSON diff to re-load). [...]
  • Correct differences typo in the ApkFile handler. (#127)

• Output improvements:

  • Never emit the same id="foo" anchor reference twice in the HTML output, otherwise identically-named parts will not be able to linked to via a #foo anchor. (#120)
  • Never emit an empty "id" anchor either; it is not possible to link to #. [...]
  • Don't pretty-print the output when using the --json presenter; it will usually be too complicated to be readable by the human anyway. [...]
  • Use the SHA256 over MD5 hash when generating page names for the HTML directory-style presenter. (#124)

• Reporting improvements:

  • Clarify the message when we truncate the number of lines to standard error [...] and reduce the number of maximum lines printed to 25 as usually the error is obvious by then [...].
  • Print the amount of free space that we have available in our temporary directory as a debugging message. [...]
  • Clarify Command […] failed with exit code messages to remove duplicate exited with exit but also to note that diffoscope is interpreting this as an error. [...]
  • Don't leak the full path of the temporary directory in Command […] exited with 1 messages. (#126)
  • Clarify the warning message when we cannot import the debian Python module. [...]
  • Don't repeat stderr from {} if both commands emit the same output. [...]
  • Clarify that an external command emits for both files, otherwise it can look like we are repeating itself when, in reality, it is being run twice. [...]

• Testsuite improvements:

  • Prevent apksigner test failures due to lack of binfmt_misc, eg. on Salsa CI and elsewhere. [...]
  • Drop .travis.yml as we use Salsa instead. [...]

• Dockerfile improvements:

  • Add a .dockerignore file to whitelist files we actually need in our container. (#105)
  • Use ARG instead of ENV when setting up the DEBIAN_FRONTEND environment variable at runtime. (#103)
  • Run as a non-root user in container. (#102)
  • Install/remove the build-essential during build so we can install the recommended packages from Git. [...]

• Codebase improvements:

  • Bump the officially required version of Python from 3.5 to 3.6. (#117)
  • Drop the (default) shell=False keyword argument to subprocess.Popen so that the potentially-unsafe shell=True is more obvious. [...]
  • Perform string normalisation in Black [...] and include the Black output in the assertion failure too [...].
  • Inline MissingFile's special handling of deb822 to prevent leaking through abstract layers. [...][...]
  • Allow a bare try/except block when cleaning up temporary files with respect to the flake8 quality assurance tool. [...]
  • Rename in_dsc_path to dsc_in_same_dir to clarify the use of this variable. [...]
  • Abstract out the duplicated parts of the debian_fallback class [...] and add descriptions for the file types. [...]
  • Various commenting and internal documentation improvements. [...][...]
  • Rename the Openssl command class to OpenSSLPKCS7 to accommodate other command names with this prefix. [...]

• Misc:

  • Rename the --debugger command-line argument to --pdb. [...]
  • Normalise filesystem stat(2) "birth times" (ie. st_birthtime) in the same way we do with the stat(1) command's Access: and Change: times to fix a nondeterministic build failure in GNU Guix. (#74)
  • Ignore case when ordering our file format descriptions. [...]
  • Drop, add and tidy various module imports. [...][...][...][...]

I also performed a huge overhaul of diffoscope's website:

• Add a completely new design. [...][...]
• Add a separate, canonical page for every new release. [...][...][...]
• Generate a 'latest release' section and display that with the corresponding date on the homepage. [...]
• Add an RSS feed of our releases [...][...][...][...][...][...] and add to Planet Debian [...].
• Dynamically generate our contributor list [...] and supported file formats [...] from the main Git repository.
• Use Jekyll's absolute_url and relative_url where possible [...][...] and move a number of configuration variables to _config.yml [...][...].

Lastly, I made a large number of changes to the main Reproducible Builds website and documentation:

• Rename the "Who" page to "Projects". [...]
• Ensure that Jekyll enters the _docs subdirectory to find the _docs/index.md file after an internal move. (#27)
• Wrap ltmain.sh etc. in preformatted quotes. [...]
• Wrap the SOURCE_DATE_EPOCH Python examples onto more lines to prevent visual overflow on the page. [...]
• Correct a "preferred" spelling error. [...]

Debian LTS

This month I contributed 17¼ hours to Debian Long Term Support (LTS) and 9¼ hours on its sister Extended LTS project.

• Investigated and triaged freerdp, keystone, nginx, tcpreplay & thunderbird, as well as tended to the general upkeep of the dla-needed.txt and ela-needed.txt files, adding various notes, references, attributions and citations.

• Frontdesk duties including responding to user/developer questions, reviewing others' packages, participating in mailing list discussions as well as attending our second regular IRC contributor meeting.

• Issued DLA 2201-1 to prevent a Denial of Service (DoS) vulnerability the ntp network time protocol server/client. ntp allowed an "off-path" attacker to block unauthenticated synchronisation via a server mode packet with a spoofed source IP address because transmissions were rescheduled even if a packet lacked a valid "origin timestamp".

• Issued DLA 2203-1 for the SQLite database to prevent a denial of service attack. In the event of a semantic error in an aggregate query, SQLite did not return early from the resetAccumulator() function which would lead to a crash via a segmentation fault.

• Issued DLA 2204-1 for the Mailman mailing list manager to prevent an arbitrary content injection attack.

• Issued DLA 2211-1 in order to prevent an XML external entity vulnerability in log4net, a logging API for the ECMA Common Language Infrastructure (CLI), sometimes referred to as "Mono". This type of attack occurs when XML input containing a reference to an internet-faced entity is processed by a weakly configured XML parser.

• Prepared and issued ELA-229-1 and DLA 2217-1 for the Apache Tomcat Java server to prevent a remote code execution exploit.

You can find out more about the two projects via the following video:

Debian

I filed the following bug reports in Debian this month:

• apksigner: Uses en-dashes (U+2013) in manpage over two hyphens. (#960778)

• devscripts: dd-list -nou results in "unknown option: […]". (#960891)

• node-redis: autopkgtest regressions against Redis 6.x. (#960105)

I also filed a number of bugs against packages that are not compatible with Django 3.x, (organised around a single master bug) including django-taggit, sorl-thumbnail, django-simple-captcha, django-cas-server, django-cors-headers, python-django-csp, django-pipeline, python-django-jsonfield, python-django-contact-form, django-model-utils, django-fsm, django-modeltranslation, django-oauth-toolkit, libthumbor, python-django-extensions, python-django-imagekit, python-django-navtag, python-django-tagging, djangorestframework, django-haystack, django-taggit, django-simple-captcha, python-django-registration, python-django-pyscss, python-django-compressor, python-django-crispy-forms & python-django-mptt,

Lastly, I made the following uploads to Debian:

• python-django (3.0.6-1) — New upstream bugfix release.

• redis:

  • 6.0.0-1 — New upstream "General Availability" release.
  • 6.0.0-2 — Mark an autopkgtest as being flaky for now.
  • 6.0.1-1, 6.0.1-2 & 6.0.1-2~bpo10+1— New upstream "General Availability" releases.
  • 6.0.3-1 & 6.0.3-1~bpo10+1 — New upstream "General Availability" releases.

• memcached (1.6.6-1) — New upstream release, fixing a build failure on s390x and similar architectures. (#960459)

• bfs (1.7-1) — New upstream release & move to debhelper-compat level 13.

I also sponsored an upload for adminer (4.7.7-1), also uploading it to buster-backports.

You can subscribe to new posts via email or RSS.