Free software activities in October 2019

  • 31 October, 2019

Here is my monthly update covering what I have been doing in the free software world during October 2019 (previous month):

  • Made some changes to my tickle-me-email library which implements Gettings Things Done-like behaviours in IMAP inboxes including ensuring attached files have their "basename" path as the filename metadata, not the full/absolute one passed to the program [...].

  • As part of my duties of being on the board of directors of the Open Source Initiative and Software in the Public Interest I attended their respective monthly meeting and participated in various licensing and other discussions occurring on the internet, as well as the usual internal discussions regarding logistics and policy etc.

  • Opened pull requests to make the build reproducible in:

    • SPIRV-Tools, part of the Khronos 3D graphics processing libraries etc. to ensure a timestamp does not vary with the build timezone. [...]

    • The "stacked" Git stgit tool. [...]

    • The traitlets Python type-checking/enforcement library to make sure that traitlet.Set values are returned in a sorted order. [...]

    • The flask microframework for building Python web applications to make the documentation build reproducibly. [...]

    • The ROS Robot Operating System code generation library for Python to ensure that generated struct constructs are reproducible. [...]

    • khard, a commandline address book utility. [...]

  • Even more hacking on the Lintian static analysis tool for Debian packages:


Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws almost all software is distributed pre-compiled to end users.

The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

The initiative is proud to be a member project of the Software Freedom Conservancy, a not-for-profit 501(c)(3) charity focused on ethical technology and user freedom.

Conservancy acts as a corporate umbrella, allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter.

This month, I:

  • I spent some more time working on our website this month too, including:

    • Improving the formatting of our reports. [...]
    • Adding some missing space. [...]
    • Tidying the new "Testing framework" links. [...]
    • Updating the monthly report template. [...]
  • strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. This month, I dropped the bug_803503.zip test fixture as it is no longer compatible with the latest version of Perl's Archive::Zip. (#940973)


diffoscope

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, I made the following changes:

  • Disassembling and reporting on files related to the R (programming language):

    • Expose an .rdb file's absolute paths in the semantic/human-readable output, not hidden deep in a hexdump. [...]
    • Rework and refactor the handling of .rdb files with respect to locating the parallel .rdx prior to inspecting the file to ensure that we do not add files to the user's filesystem in the case of directly comparing two .rdb files or — worse — overwriting a file in is place. [...]
    • Query the container for the full path of the parallel .rdx file to the .rdb file as well as looking in the same directory. This ensures that comparing two Debian packages shows any varying path. [...]
    • Correct the matching of .rds files by also detecting newer versions of this file format. [...]
    • Don't read the site and user environment when comparing .rdx, .rdb or .rds files by using Rscript's --vanilla option. [...][...]
    • Ensure all object names are displayed, including ones beginning with a fullstop (.) [...] and sort package fields when dumping data from .rdb files [...].
    • Mask/hide standard error when processing .rdb files [...] and don't include useless/misleading NULL when dumping data from them. [...]
    • Format package contents as foo = bar rather than using ugly and misleading brackets, etc. [...] and include the object's type [...].
    • Don't pass our long script to parse .rdb files via the command line; use standard input instead. [...]
    • Call thedeparse function to ensure that we do not error out and revert to a binary diff when processing .rdb files with internal "vector" types; they do not automatically coerce to strings. [...]
    • Other misc/cosmetic changes. [...][...][...]
  • Output/logging:

    • When printing an error from a command, format the command for the user. [...]
    • Truncate very long command lines when displaying them as an external source of data. [...]
    • When formatting command lines ensure newlines and other metacharacters appear escaped as \n, etc. [...][...]
    • When displaying the standard error from commands, ensure we use the escaped version. [...]
    • Use "exit code" over "return code" terminology when referring to UNIX error codes in displayed differences. [...]
  • Internal API:

    • Add ability to pass bytestring input to external commands. [...]
    • Split out command-line formatting into a separate utility function. [...]
    • Add support for easily masking the standard error of commands. [...][...]
    • To match the libarchive container, raise a KeyError exception if we request an invalid member from a directory. [...]
    • Correct string representation output in the traceback when we cannot locate a specific item in a container. [[...] (https://salsa.debian.org/reproducible-builds/diffoscope/commit/2478e9c)]
  • Misc:

    • Move build-dependency on python-argcomplete to its Python 3 equivalent to facilitate Python 2.x removal. (#942967)
    • Track and report on missing Python modules. (#72)
    • Move from deprecated $ADTTMP to $AUTOPKGTEST_TMP in the autopkgtests. [...]
    • Truncate the tcpdump expected diff to 8KB (from ~600KB). [...]
    • Try and ensure that new test data files are generated dynamically, ie. at least no new ones are added without "good" reasons. [...]
    • Drop unused BASE_DIR global in the tests. [...]


Debian

I filed two patches against the r-base package for not respecting the nocheck and nodoc build profiles respectfully (#942867 & #942870) as well as filing a bug against python3-pluggy for missing a dependency on python3-importlib-metadata (#943320).

Uploads


Debian LTS

This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project.

You can find out more about the Debian LTS project via the following video:


FTP Team

As a Debian FTP assistant I ACCEPTed 25 packages: backintime, celery-batches, eslint, golang-github-containers-image, gtk-d, jsbundle-web-interfaces, networkx, node-eslint-plugin-eslint-plugin, node-eslint-plugin-node, node-eslint-scope, node-eslint-visitor-keys, node-esquery, node-file-entry-cache, node-flatted, node-functional-red-black-tree, node-ignore, node-leche, node-mock-fs, node-proxyquire, numpy, openvswitch, puppet-module-voxpupuli-collectd, pyrsistent, python-dbussy & z3.

I additionally filed 5 RC bugs against packages that had potentially-incomplete debian/copyright files against backintime, celery-batches, networkx, openvswitch & z3.