Free software activities in April 2020

Here is my monthly update covering what I have been doing in the free software world during April 2020 (previous month's report). Looking it over prior to publishing, I am surprised how much I got done this month — I felt that I was not only failing to do all the extra things I had planned, but I was doing far less than normal. But let us go easy on ourselves; nobody is nailing this.

• Made some small changes to my tickle-me-email library which implements Gettings Things Done (GTD)-like behaviours in IMAP inboxes in order to decode various headers correctly [...] and correct the counting logic in the send-later command's message limit. [...]

• Worked with @dormando with a architecture-specific problem in the Memcached caching system to fix grossly incorrect behaviour on big-endian architectures. [...]

  

• As part of my duties of being on the board of directors of the Open Source Initiative and Software in the Public Interest I attended their respective monthly meetings and participated in various licensing and other discussions occurring on the internet, as well as the usual internal discussions regarding logistics, licensing, policy, liaising with the ClearlyDefined project and so on. In particular, I on-boarded the Ganeti project to SPI.

• Reviewed and merged a contribution to my django-cache-toolbox caching library for Django web applications to fix a nested traceback issue. (#20)

• Attended a number of OpenUK's Teabreak Tuesday and Future Leaders' Training events as well did more work as part of being on the judging panel of the OpenUK Awards. Nominations are open until 15th June in five different categories. I also attended a virtual open source 'hallway track' organised by Stormy Peters and plan to attend the next meeting in early May.

In addition, I did more hacking on the Lintian static analysis tool for Debian packages:

• New features and improvements:

  • Check for debian/rules files that specify -Wl,--as-needed as this is now the default from bullseye onwards. (#956146)
  • Warn about automatically-generated debug packages that ship files other than .debug. (#958945)
  • Warn about packages that specify --with=systemd with a Debhelper compatibility level of 10 or higher. (#949844)
  • Detect $* as using the Debhelper sequencer. (#930679)
  • Check for override dh_install (ie. with a space) in debian/rules; in 99% of cases this will be an omission of the underscore. [...]

• Bug fixes:

  • Allow python3-all-dev and python3-all-dbg to satisfy the check for packages that use py3versions with the -s argument. (#955799, #956134)
  • Build-Depends-Arch and Build-Depends-Indep do not imply each other, so don't warn about "duplicate" dependencies in this case. (#956368)
  • Ignore build profiles when checking packages for py3versions -s without the corresponding Build-Depends. (#958794)
  • Remove the pre-depends-directly-on-multiarch-support tag; any package pre-depending on the multiarch-support will not be installable in the bullseye distribution. (#798762)
  • Mark mailing-list-obsolete-in-debian-infrastructure as being experimental. (#958182, #958666)
  • Do not warn about empty dh_dwz-generated "multi-files". (#955752)
  • Don't warn about package-relation-with-self if we have specified a required architecture. (#956227)

• Misc:

  • Move Lintian itself to compatibility level 13 so it does not emit package-uses-old-debhelper-compat-version when run against itself. [...]
  • Drop the .travis.yml file as we are using Salsa now. [...]

Reproducible builds

One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes.

The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

The initiative is proud to be a member project of the Software Freedom Conservancy, a not-for-profit 501(c)(3) charity focused on ethical technology and user freedom.

Conservancy acts as a corporate umbrella allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter.

• I submitted 14 patches to fix specific reproducibility issues in gmap, gpick, herbstluftwm, libcamera, libctl, libctl, minetest-mod-xdecor, netgen-lvs, nickle, nmrpflash, node-mqtt, sprai, xxhash & yaz.

• Submitted the following patches to fix reproducibility-related toolchain issues within Debian:

  • dh-cargo: Please make the output reproducible. (#958301)

  • rspamd: Please make paths.lua files reproducible. (#956120)

• Wrote a 20-page funding report to the Open Technology Fund -- whilst the Reproducible Builds project has submitted monthly reports to the otf-active mailing list this final report described in detail the status of each objective, our overall lessons and our future plans.

• The Reproducible Builds project also operates a Jenkins-based testing framework that powers tests.reproducible-builds.org. I made the following changes:

  • Print the build environment prior to executing a build. [...]
  • Drop a misleading disorderfs-debug prefix in log output when we change non-disorderfs things in the file and, as it happens, do not run disorderfs at all. [...]
  • The CSS for the package report pages added a margin to all a HTML elements under li ones, which was causing a comma/bullet spacing issue. [...]
  • Tidy the copy in the project links sidebar. [...]

• Continued collaborative work on an academic paper to be published within the next few months.

• Kept isdebianreproducibleyet.com up to date. [...]

• strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. In April, I made the following changes:

  • Add deprecation plans to all handlers documenting how (or if) they could be disabled and hopefully eventually removed, etc. (#3)
  • Normalise *.sym files as Java archives. (#15)
  • Add support for custom .zip filename filtering and exclude two patterns of files generated by Maven projects in "fork" mode. (#13)

• disorderfs is our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues. This month I fixed a long-standing issue by not drop UNIX groups in FUSE multi-user mode when we are not root. (#1)

• Categorised a large number of packages and issues in the Reproducible Builds "notes" repository.

• Drafted, published and publicised our monthly report.

Elsewhere in our tooling, I made the following changes to diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues, including preparing and uploading versions 139, 140, 141 and 142 to Debian:

• Comparison improvements:

  • Dalvik .dex files can also serve as APK containers so restrict the narrower identification of .dex files to files ending with this extension and widen the identification of APK files to when file(1) discovers a Dalvik file. (#28)
  • Add support for Hierarchical Data Format (HD5) files. (#95)
  • Add support for .p7c and .p7b certificates. (#94)
  • Strip paths from the output of zipinfo(1) warnings. (#97)
  • Don't uselessly include the JSON "similarity" percentage if it is "0.0%". [...]
  • Render multi-line difference comments in a way to show indentation. (#101)

• Testsuite improvements:

  • Add pdftotext as a requirement to run the PDF test_metadata text. (#99)
  • apktool 2.5.0 changed the handling of output of XML schemas so update and restrict the corresponding test to match. (#96)
  • Explicitly list python3-h5py in debian/tests/control.in to ensure that we have this module installed during a test run to generate the fixtures in these tests. [...]
  • Correct parsing of ./setup.py test --pytest-args arguments. [...]

• Misc:

  • Capitalise "Ordering differences only" in text comparison comments. [...]
  • Improve documentation of FILE_TYPE_HEADER_PREFIX and FALLBACK_FILE_TYPE_HEADER_PREFIX to highlight that only the first 16 bytes are used. [...]

Lastly, I made a large number of changes to our website and documentation in the following categories:

• Community engagement improvements:

  • Update instructions to register for Salsa on our Contribute page now that the signup process has been overhauled. [...]
  • Make it clearer that joining the rb-general mailing list is probably a first step for contributors to take. [...]
  • Make our full contact information easier to find in the footer (#19) and improve text layout using bullets to separate sections [...].

• Accessibility:

  • To improve accessibility, make all links underlined. (#12)
  • Use an enhanced foreground/background contrast ratio of 7.04:1. (#11)

• General improvements:

  • Add a new Academic publications page. (#22)
  • Add Trezor to our list of affiliated projects. (#26)
  • Add the JVM page to the documentation index (#17) and tidy the page itself a little [...].
  • Add a GNU Libtool pointer to the Archive metadata documentation page. [...]

• Internals:

  • Move to using jekyll-redirect-from over manual redirect pages [...][...] and add a redirect from /docs/buildinfo/ to /docs/recording/. (#23)
  • Limit the website self-check to not scan generated files [...] and remove the "old layout" checker now that I have migrated all them [...].
  • Move the news archive under the /news/ namespace [...] and improve formatting of archived news links [...].
  • Various improvements to the draft template generation. [...][...][...][...]

Debian LTS

This month I have contributed 18 hours to Debian Long Term Support (LTS) and 7¾ hours on its sister Extended LTS project.

• Frontdesk duties, responding to user/developer questions, reviewing others' packages, participating in mailing list discussions, etc.

• Issued DLA 2171-1 for the Ceph distributed storage and file system to prevent a header-splitting vulnerability (CVE-2020-1760).

• It was discovered that there was a path-traversal issue in the Apache Shiro Java security framework (CVE-2020-1957) where a specially-crafted request could cause an authentication bypass. I therefore issued DLA 2181-1 to address this.

• Issued ELA-224-1 for the ntp Network Time Protocol server to prevent a denial of service vulnerability (CVE-2020-11868).

• Issued ELA-225-1 for dom4j, a library for working with various XML formats on the Java platform to address an XML external external entity vulnerability (CVE-2020-10683). This type of attack occurs when XML input containing a reference to an internet-faced entity is processed by a weakly configured XML parser. This attack may lead to the disclosure of confidential data, denial of service, server side request forgery as well as other system impacts.

• Issued ELA-224-2 to update the fix to ntp in ELA-224-2 to add further protection that was not present in the previous update.

• Investigated and triaged bluez [...], dom4 [...], inetutils [...], openldap [...], otrs2 [...], qemu [...][...][...], samba [...][...], varnish [...], xcftools [...], etc.

• Attended our first regular IRC contributor meeting.

You can find out more about the project via the following video:

Debian

I only filed three bugs in April, including one against snapshot.debian.org to report that a Content-Type HTTP header is missing when downloading .deb files (#956471) and to report build failures in the macs & ruby-enumerable-statistics packages:

• Redis:

  • 6.0~rc3-1 — New upstream beta release.
  • 6.0~rc4-1 — New upstream beta release and use the newly-packaged liblzf-dev package over the local version. (#958321)
  • 5.0.7-3 — Fix build failures with GCC 10. (#957751)
  • 5.0.7-4 — Use the newly-packaged liblzf-dev package over the bundled version. (#958321)
  • 5.0.7-5 — Ensure that the daemon is running prior to running the autopkgtests.
  • 5.0.7-6 — Perform a "no change" sourceful upload to permit migration to the testing distribution.
  • 5.0.7-7 — Add a sleep to ensure that the server has started before running the tests.

• Django:

  • 2.2.12-1 — New upstream release.
  • 3.0.5-1 — New upstream release.

• Memcached:

  • 1.6.3-1 — New upstream release.
  • 1.6.5-1 — New upstream release.
  • 1.6.5-2 — Fix a failing autopkgtest that assumes a particular string is in the first kilobyte of the response which was no longer the case due to the addition of new configuration options.

• Redisearch:

  • 1.2.2-1 — New upstream release.
  • 1.2.2-2 — Ensure that the server is started before running autopkgtests.
  • 1.2.2-3 — Sleep to ensure the server has really started before running the autopkgtests.

• OnionShare (2.2-2) — Replace python-nautilus in the build-dependencies with python3-nautilus. (#943137)

• installation-birthday (15) — Correct the (confusing) internal logic of --force from being visible to users. (#895686)

• txtorcon (20.0.0-1) — New upstream release and refresh packaging.

• python-hiredis (1.0.1-1) — New upstream release and refresh packaging.

• onioncircuits (0.6-3) — Mark a non-deterministic autopkgtest as "flaky" for now to ease migration (#930448) and use DEB_VERSION_UPSTREAM_REVISION over manually parsing dpkg-parsechangelog in debian/rules.

You can subscribe to new posts via email or RSS.