Free software activities in July 2019

Here is my monthly update covering what I have been doing in the free software world during July 2019 (previous month):

• Most of my time this month was expended attending DebConf19 in Curitiba, Brazil where I presented in three sessions.

  The first was titled "Reproducible Builds — aiming for bullseye" which discussed the current status of Reproducible Builds within Debian and our goals for the next release followed by "Bits from the Lintian maintainers" in which I talked about the past, present and future of the Lintian static-analysis tool for Debian packages. I finished my speaking engagements by appearing on a panel about the Debian Long Term Support (LTS) project subtitled "The good, the bad and the better" which served as a venue to describe how LTS is , how it can be supported it and to receive feedback.

  I naturally attended a large number of sessions, side-events and ad-hoc BoF sessions too. DebConf was also preceeded by DebCamp, a week where Developers can focus on their Debian-related projects, tasks or problems.

• Whilst I was in Curitiba, I managed to attend a "release party" for Debian 10 buster which took place at the Festa do frango e polenta festival at the Bosque São Cristóvão, situated within an traditionally Italian suburb of the city.

• Whilst we did not have our regular monthly meeting this month, as part of my duties of being on the board of directors of the Open Source Initiative I participated in various licensing discussions occurring in/around the internet, etc. I did further work as the ClearlyDefined project's official representative on the Board due to a colleague no longer being able to commit to the position.

• I opened a pull request for the Flameshot screenshot tool to add a keyboard shortcut for uploading to the Imgur image hosting service [...]. Thanks to Jonathan Carter for originally pointing me towards this utility via his Debian Package of the Day video series.

• Andrej Shadura forwarded a long-standing patch of mine to the "dash" shell to the upstream mailing list.

• After merging and reviewing a number of pull requests I made some updates and released a version my django-dumpslow project, an add-on to the Django web-development framework that, taking inspiration from the mysqdumpslow tool, logs requests that take a long time to execute and provides an tool to summarise the resulting data.

• Moved away from using the ambiguous "BSD" license specifier in my django-slack library that provides a convenient library between projects using the Django and the Slack chat platform.

• Even more hacking on the Lintian static analysis tool for Debian packages, including:

  • New features:

    • Reflect the release of Debian 10 buster, updating dates of oldstable etc. [...]
    • Check that if a package that ships a /etc/sv/foo directory then /etc/sv/foo/run exists and is executable. (#931426)
    • Add 4.4.0 as a known Standards-Version. [...]
    • Bump the "old" and "ancient" version numbers for Python 3 checks re. #903399. [...]
    • Bump the recommended debhelper level to 12 (see #918809). [...]

  • Bug fixes:

    • Correct multiple "gobject-introspection" typos. (#933394)
    • Correct the generate-tag-summary to point to the new path for tag files [...] and actually ship this directory in the package [...].
    • Correct the exit code if the specified profile does not exist. (#932215)
    • Don't emit latest-debian-changelog-entry-without-new-version for uploads to (eg.) buster-security to avoid false-positives when performing "no-change" uploads with a ~debXuY suffix. [...]
    • Fix mistake||correction delimiter for a number of entries in spelling/corrections-case. Thanks, Francois Gouget. (#931446)
    • Update regular expression matches for lines in debian/rules correctly identify calls starting with environment modifications (eg. LC_ALL=C.UTF-8). (#932128)
    • Don't emit command-in-sbin-has-manpage-in-incorrect-section for symlinks as they are invariably provided for legacy/compatibility reasons. (#931951)
    • Override dh_dwz in a test to avoid the tests hanging. (#931632)

  • Reporting/misc:

    • Upgrade package-uses-vendor-specific-patch-series to E: severity following the release of buster (re. #904302). [...]
    • Downgrade severity/certainty of package-supports-alternative-init-but-no-init.d-script for the time being. (#931889)
    • Downgrade command-in-sbin-has-manpage-in-incorrect-section to pedantic severity for the time being. (#570998)
    • Expand the long description of duplicate-files to suggest how to remove them. Thanks to Rebecca Palmer for the patch. (#932411)
    • Drop the deprecated --fail-on-warnings option scheduled for removal in buster to help towards fixing #709932. [...]
    • Tidy package long description. [...]

Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws almost all software is distributed pre-compiled to end users. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

The initiative is proud to be a member project of the Software Freedom Conservancy, a not-for-profit 501(c)(3) charity focused on ethical technology and user freedom. Conservancy acts as a corporate umbrella, allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter.

This month:

• As part of the aforementioned DebConf19 in Curitiba, Brazil I presented in a session titled "Reproducible Builds - aiming for bullseye" and had almost-countless discussions regarding Reproducible Builds in and around Debian on the questions of tooling, infrastructure and our next steps as a project.

• In Debian:

  • Submitted a patch for liblopsub to make the output of the lopsubgen tool reproducible (#931854). I also submitted 7 patches to fix specific reproducibility issues in calendar, ninja-build, node-d3-selection, python-manilaclient, python-os-faults, snakemake & sysvinit.

  • I filed two bugs against the jenkins.debian.org metapackage to drop the test jobs for both strip-nondeterminism (#932366) and reprotest (#932374).

  • Kept isdebianreproducibleyet.com up to date [...] and updated our blog entry on Planet Debian [...].

• Categorised a huge number of packages and issues in our "package notes" repository.

• Drafted, published and publicised our monthly report.

I spent significant amount of time working on our website this month, including:

• Split out our non-fiscal sponsors with a description [...] and make them non-display three-in-a-row [...].
• Correct references to "1&1 IONOS" (née Profitbricks). [...]
• Lets not promote yet more ambiguity in our environment names! [...]
• Recreate the badge image, saving the .svg alongside it. [...]
• Update our fiscal sponsors. [...][...][...]
• Tidy the weekly reports section on the news page [...], fixup the typography on the documentation page [...] and make all headlines stand out a bit more [...].
• Drop some old CSS files and fonts. [...]
• Tidy news page a bit. [...]
• Fixup a number of issues in the report template and previous reports. [...][...][...][...][...][...]

I also made the following changes to our tooling:

diffoscope

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues.

• Add support for Java .jmod modules (#60). However, not all versions of file(1) support detection of these files yet, so we perform a manual comparison instead [...].
• If a command fails to execute but does not print anything to standard error, try and include the first line of standard output in the message we include in the difference. This was motivated by readelf(1) returning its error messages on standard output. [#59) [...]
• Add general support for file(1) 5.37 (#57) but also adjust the code to not fail in tests when, eg, we do not have sufficiently newer or older version of file(1) (#931881).
• Factor out the ability to ignore the exit codes of zipinfo and zipinfo -v in the presence of non-standard headers. [...] but only override the exit code from our special-cased calls to zipinfo(1) if they are 1 or 2 to avoid potentially masking real errors [...].
• Cease ignoring test failures in stable-backports. [...]
• Add missing textual DESCRIPTION headers for .zip and "Mozilla"-optimised .zip files. [...]
• Merge two overlapping environment variables into a single DIFFOSCOPE_FAIL_TESTS_ON_MISSING_TOOLS. [...]
• Update some reporting:
  • Re-add "return code" noun to "Command foo exited with X" error messages. [...]
  • Use repr(..)-style output when printing DIFFOSCOPE_TESTS_FAIL_ON_MISSING_TOOLS in skipped test rationale text. [...]
  • Skip the extra newline in Output:\nfoo. [...]
• Add some explicit return values to appease Pylint, etc. [...]
• Also include the python3-tlsh in the Debian test dependencies. [...]
• Released and uploaded releasing versions 116, 117, 118, 119 & 120. [...][...][...][...][...]

strip-nondeterminism

strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build.

• Support OpenJDK ".jmod" files. [...]
• Identify data files from the COmmon Data Access (CODA) framework as being .zip files. [...]
• Pass --no-sandbox if necessary to bypass seccomp-enabled version of file(1) which was causing a huge number of regressions in our testing framework.
• Don't just run the tests but build the Debian package instead using Salsa's centralised scripts so that we get code coverage, Lintian, autopkgtests, etc. [...][...]
• Update tests:
  • Don't build release Git tags on salsa.debian.org. [...]
  • Merge the debian branch into the master branch to simplify testing and deployment [...] and update debian/gbp.conf to match [...].
• Drop misleading and outdated MANIFEST and MANIFEST.SKIP files as they are not used by our release process. [...]

Debian LTS

This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS (ELTS) project.

• Frontdesk duties, responding to user/developer questions, reviewing others' packages, etc.

• As part of attending DebConf19 in Curitiba, Brazil I prepared and took part in a panel about Debian LTS subtitled "The good, the bad and the better" which served as a venue to describe how LTS is done, how it can be supported it and to receive feedback. I also helped provide feedback on the flyer which was distributed as well as an to-be-released video advertising the merits of the project.

• Investigated and triaged golang-go.crypto, libonig, otrs2, php5, python3.4, redis, ruby-mini-magick, ruby-openid & zeromq3 for the LTS project and asterisk, exiv2, libonig, libspring-security-2.0-java, libxslt, nss, otrs2, redis & rustc for ELTS.

• Issued DLA 1842-1 for the python-django web development framework fixing an issue where Django did not correct identify HTTP connections when a reverse proxy was connected via HTTPS.

• Issued DLA 1844-1 for the lemonldap-ng single sign-on system to resolve an XML external entity vulnerability which may have led to the disclosure of confidential data, denial of service, server side request forgery, port scanning, etc.

• Issued DLA 1847-1 to fix multiple cross-site scripting vulnerabilities in the Squid caching proxy server.

• Issued DLA 1850-1 for the Redis key-value database to fix two heap buffer overflows in the "Hyperloglog" functionality. database.

• Issued DLA 1855-1 for the exiv2 tool to manipulate images containing (for example) embedded EXIF metadata to fix an integer overflow vulnerability.

• Assisted the security team by preparing, testing and uploading packages for DSA-4476 for python-django and DSA-4480 for redis.

Uploads

• python-django:

  • 1:1.11.22-1 — New upstream security release
  • 2:2.2.3-1 — New upstream security release
  • 2:2.2.3-2 — Upload version from Debian experimental to unstable after the release of Debian buster.
  • 2:2.2.3-3 — python3-mysqlclient 1.3.13 or newer is now required; add a Breaks on versions older than this (#931592)) and drop "Python 3 version" suffix from the package description as we only have this variant now.
  • 2:2.2.3-4 — Fix doc-base references to refer to the new locations of the documentation. (#931652)
  • 2:2.2.3-5 — Drop Pre-Depends on a version of dpkg that is satisfied in very old versions of Debian.

• bfs:

  • 1.5-1 — New upstream release.
  • 1.5-2 — Add missing acl package to build-dependencies.

• python-daiquiri (1.5.0-2) and python-redis (3.2.1-3) in order to move these packages under the umbrella of the Debian Python Modules Team.

I also made "sourceful" uploads to unstable to ensure migration to testing after recent changes that prevent maintainer-supplied packages entering bullseye for bfs (1.5-3), redis (5:5.0.5-2), lastpass-cli (1.3.3-2), python-daiquiri (1.5.0-3) and I finally performed a sponsored upload of elpy (1.29.1+40.gb929013-1).

FTP Team

As a Debian FTP assistant I ACCEPTed 19 packages: aiorwlock, bolt, caja-mediainfo, cflow, cwidget, dgit, fonts-smc-gayathri, gmt, gnuastro, guile-gcrypt, guile-sqlite3, guile-ssh, hepmc3, intel-gmmlib, iptables, mescc-tools, nyacc, python-pdal & scheme-bytestructures. I additionally filed a bug against scheme-bytestructures for having a seemingly-incomplete debian/copyright file. (#932466)

You can subscribe to new posts via email or RSS.