Free software activities in August 2019

Here is my monthly update covering most of what I have been doing in the free software world during August 2019 (previous month):

• I was extremely proud to be announce I will be joining the folks at Software in the Public Interest to assist its many umbrella projects and free software in General. You can read the official announcement here.

• Migrated my Tickle Me Email tool that implements Gettings Things Done-like behaviours in any IMAP inbox to Python 3 as Python 2.x will not be maintained past 2020. [...]

• As part of my duties of being on the board of directors of the Open Source Initiative I attended our monthy meeting and participated in various licensing discussions occurring on the internet, as well as the usual discussions happening internally etc.

• I attended this year's Debian Party in Cambridge, UK. Better known as the "OMGWTFBBQ", I had a great time despite the remarkable heat. A special thanks to Steve for hosting the event and all others who helpedand organise this, as well as Mythic Beasts, Collabora and Codethink for sponsoring the event. For my part I made some souvenir beer mats commemorating the event, offering them gratis with a nudge towards becoming a supporter of the Software Freedom Conservancy:

• Opened pull requests to make the build reproducible for Mozilla's Bleach [...] and the re2c regular expression library [...].

Tails

For the Tails privacy-oriented operating system, I was made a number of updates as part of the pkg-privacy-tools team in Debian:

• onionshare:

  • Package new upstream version 2.1. [...]
  • Correct spelling, format and syntax errors in manpage.
  • Update debian/copyright; socks.py no longer in upstream.
  • Misc updates:
    • Drop "ancient" X-Python3-Version specifier (satisfied in oldoldstable).
    • Move to debhelper compatibility level 12 and use the debhelper-compat virtual package, dropping debian/compat.
  • debian/watch: Ignore dev releases and move to version 4 format.

• monkeysphere:

  • Prevent a FTBFS by updating the tests to accommodate an updated GnuPG in stretch now producing a different output. (#934034).

  • I also filed a "proposed update" to actually update the package in the stretch distribution. (#934775)

• onioncircuits: Update continuous integration tests to the Python 3.x version of Dogtail. (#935174)

• seahorse-nautilus: (Almost) no-change upload to unstable to ensure migration to the testing distribution as binaries were uploaded with previous 3.11.92-3 release. [...]

• obfs4proxy: Move to using the debian-compat virtual package, level 12. [...]

Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws almost all software is distributed pre-compiled to end users.

The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

The initiative is proud to be a member project of the Software Freedom Conservancy, a not-for-profit 501(c)(3) charity focused on ethical technology and user freedom.

Conservancy acts as a corporate umbrella, allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter.

This month:

• Filed upstream pull requests for Mozilla Bleach and the re2c regular expression compiler.

• In Debian:

  • The first "package sets" — ie. defined subsets of the entire archive — have become 100% reproducible including as the so-called "essential" set for the bullseye distribution on the amd64 and the armhf architectures due to patches I sent for bash, readline and other low-level libraries and tools.

  • Submitted patches to fix reproducibility-related toolchain issues within Debian for gdbm and node-package-preamble.

  • I also submitted 17 patches to fix specific reproducibility issues in bash, ecbuild, haskell-devscripts, libchamplain, litl, musescore-snapshot, node-autoprefixer, norsnet, omnidb, python-bleach, python-etcd3gw, re2c, readline, rust-fs-extra, scons, superlu-dist & virulencefinder.

  • Kept isdebianreproducibleyet.com up to date. [...]

• Vagrant Cascadian suggested and subsequently implemented that we additionally test a varying build directory of different string lengths (eg. /path/to/123 vs /path/to/123456 but we also vary the number of directory components within this, eg. /path/to/dir vs. /path/to/parent/subdir. Curiously, whilst it was a priori believed that was rather unlikely to yield differences, I managed to identify approximately twenty packages that are affected by this issue.

• Requested that the "molior" build tool prefers to use the term "repeatable build" in order to avoid confusion over the term "reproducible."

• I spent more time working on our website this month, including updating the documentation for the SOURCE_DATE_EPOCH environment variable. [...]

• Categorised a large number of packages and issues in the Reproducible Builds "notes" repository and ran the black source code reformatter over the edit-notes script that is used here [...].

• Drafted, published and publicised our monthly report.

• For our testing framework I ensured that the size the log for the second build in HTML pages was also correctly formatted (eg. "12KB" vs "12345"). [...]

I also made the following changes to our tooling:

diffoscope

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues.

Improvements:

• Don't fallback to an unhelpful raw hexdump when, for example, readelf(1) reports an minor issue in a section in an ELF binary. For example, when the .frames section is of the NOBITS type its contents are apparently "unreliable" and thus readelf(1) returns 1. (#58, #931962)
• Include either standard error or standard output (not just the latter) when an external command fails. [...]

Bug fixes:

• Skip calls to unsquashfs when we are neither root nor running under fakeroot. (#63)
• Ensure that all of our artificially-created subprocess.CalledProcessError instances have output instances that are bytes objects, not str. [...]
• Correct a reference to parser.diff; diff in this context is a Python function in the module. [...]
• Avoid a possible traceback caused by a str/bytes type confusion when handling the output of failing external commands. [...]

Testsuite improvements:

• Test for 4.4 in the output of squashfs -version, even though the Debian package version is 1:4.3+git190823-1. [...]
• Apply a patch from László Böszörményi to update the squashfs test output and additionally bump the required version for the test itself. (#62 & #935684)
• Add the wabt Debian package to the test-dependencies so that we run the WebAssembly tests on our continuous integration platform, etc. [...]

Improve debugging:

• Add the containing module name to the (eg.) Using StaticLibFile for ... debugging messages. [...]
• Strip off trailing "original size modulo 2^32 671" (etc.) from gzip compressed data as this is just a symptom of the contents itself changing that will be reflected elsewhere. (#61)
• Avoid a lack of space between "... with return code 1" and "Standard output". [...]
• Improve debugging output when instantantiating our Comparator object types. [...]
• Add a literal "eg." to the comment on stripping "original size modulo..." text to emphasise that the actual numbers are not fixed. [...]

Internal code improvements:

• No need to parse the section group from the class name; we can pass it via type built-in kwargs argument. [...]
• Add support to Difference.from_command_exc and friends to ignore specific returncodes from the called program and treat them as "no" difference. [...]
• Simplify parsing of optional command_args argument to Difference.from_command_exc. [...]
• Set long_description_content_type to text/x-rst to appease the PyPI.org linter. [...]
• Reposition a comment regarding an exception within the indented block to match Python code convention. [...]

strip-nondeterminism

strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build.

• Add support for enabling and disabling specific normalizers via the command line. (#10)
• Drop accidentally-committed warning emitted on every fixture-based test. [...]
• Reintroduce the .ar normalizer [...] but disable it by default so that it can be enabled with --normalizers=+ar or similar. (#3)
• In verbose mode, print the normalizers that strip-nondeterminism will apply. [...]

Debian

Lintian

More hacking on the Lintian static analysis tool for Debian packages, including uploading versions 2.17.0, 2.18.0 and 2.19.0:

New features:

• Split out building the test packages when running the tests on Salsa into a new continuous integration stage in order that we can cache the testing packages that can be time-consuming to build. (#930487)
• Ignore all initscripts provided by the initscripts binary package instead of whitelisting issues individually or requiring overrides; these will never be regular-style initscripts. (#933383)
• Also check (eg.) ${source:Upstream:Version} for the version-substvar-for-external-package tag not just ${source:Version}. (#935138)
• Update the name of the node-html5shiv library for the privacy-breach-uses-embedded-file-html5shiv and embedded-javascript-library tags. (#935083)

Bug fixes:

• Avoid a large number of false-positives for package-does-not-use-debhelper-or-cdbs by actually taking into account whether the package uses CDBS or not. (#933902, #933903)
• Remove "busses" as a false-positive plural of "bus". (#934452)
• Don't emit the latest-debian-changelog-entry-without-new-version warning for uploads for the buster distribution. (#934982)
• Avoid package-does-not-use-debhelper-or-cdbs false-positives for packages using the pkg-kde-tools CDBS routines. (#933901)
• Ignore Python scripts that are shipped under /usr/lib/python3/dist-packages as these files aren't supposed to be called as scripts. (#933167)
• Don't emit package-name-doesnt-match-sonames for udebs. (#934748)
• Don't emit the changelog-file-missing-explicit-entry tag for uploads to the security archives (Re. #934026). [...]

Other:

• Recommend that maintainers do not override latest-debian-changelog-entry-changed-to-native . [...]
• Avoid DRY violation in pkg-js-autotest-test-is-missing and pkg-js-autotest-test-is-empty. [...]
• Remove test-only qualifier from python3-docutils build-dependency as we now need it to build the package. [...]

Debian LTS

This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project.

• Frontdesk duties, responding to user/developer questions, reviewing others' packages, participating in mailing list discussions, etc.

• Investigated and triaged cent, clamav, enigmail, freeradius, ghostscript, libcrypto++, musl, open-cobol, pango1.0, php5, python-django, python-werkzeug, radare2, salt, subversion, suricata, u-boot, xtrlock & yara.

• Updated our lts-cve-triage.py script to correct undefined reference to colored when standard output is not a terminal [...] and address a number of flake8 issues [...].

• Worked on a number of interations towards a comprehensive patch to xtrlock to address an issue whereby multitouch events (such as on a tablet or many modern laptops) are not correct locked. Whilst originally filed by a user as #830726 whilst triaging issues for this package I was able to reproduce it. I thus requested and was granted my first CVE number (CVE-2016-10894) and hope to upload a patched version early next month.

• Issued DLA 1896-1 for to fix a remote arbitrary code vulnerability in commons-beanutils, a set of tools and utilities for manipulating JavaBeans.

• Issued DLA 1872-1 for the Django web development framework correcting two denial of service vulnerabilities and requiring a backport of upstream's patch series. I also fixed these issues in the buster distribution as well as an SQL injection possibility and potential memory exhaustion issues.

You can find out more about the project in the following video:

Debian uploads

• python-django (2:2.2.4-1) — New upstream security release.

• xtrlock:

  • 2.9:

    • Apply a patch from Helmut Grohne to move away from the xmkmf, a deprecated tool to create a Makefile from an IMakefile. (#902648)
    • Add an example systemd .service file to lock screen on suspend/sleep.
    • Ensure that patchlevel.h header file always matches the version in debian/changelog.
    • Move to "minimal" Debhelper style.
    • Add Vcs-Git and Vcs-Browser metadata; packaging has been added to Salsa

  • 2.1: Ensure that our patchlevel.h version check works with binary NMUs. (#935697)

• I also sponsored an upload of version 4.7.2-1 of adminer.

FTP Team

As a Debian FTP assistant I ACCEPTed 28 packages: bitshuffle, golang-github-abdullin-seq, golang-github-centurylinkcloud-clc-sdk, golang-github-cnf-structhash, golang-github-deanthompson-ginpprof, golang-github-ensighten-udnssdk, golang-github-gin-contrib-cors, golang-github-gin-contrib-gzip, golang-github-gin-contrib-static, golang-github-hansrodtang-randomcolor, golang-github-jarcoal-httpmock, golang-github-mcuadros-go-gin-prometheus, golang-github-mitchellh-go-linereader, golang-github-nesv-go-dynect, golang-github-sethvargo-go-fastly, golang-github-terra-farm-udnssdk, golang-github-yourbasic-graph, golang-github-ziutek-mymysql, golang-gopkg-go-playground-colors.v1, gulkan, kdeplasma-applets-xrdesktop, libcds, libinputsynth, openvr, parfive, transip, znc & znc-push.

You can subscribe to new posts via email or RSS.