Free software activities in February 2017

  • 28 February, 2017

Here is my monthly update covering what I have been doing in the free software world (previous month):

  • Submitted a number of pull requests to the Django web development framework:
    • Add a --mode=unified option to the "diffsettings" management command. (#8113)
    • Fix a crash in setup_test_environment() if ALLOWED_HOSTS is a tuple. (#8101)
    • Use Python 3 "shebangs" now that the master branch is Python 3 only. (#8105)
    • URL namespacing warning should consider nested namespaces. (#8102)
  • Created an experimental patch against the Python interpreter in order to find reproducibility-related assumptions in dict handling in arbitrary Python code. (#29431)
  • Filed two issues against dh-virtualenv, a tool to package Python virtualenv environments in Debian packages:
    • Fix "upgrage-pip" typo in usage documentation. (#195)
    • Missing DH_UPGRADE_SETUPTOOLS equivalent for dh_virtualenv (#196)
  • Fixed a large number of spelling corrections in Samba, a free-software re-implementation of the Windows networking protocols.
  • Reviewed and merged a pull request by @jheld for django-slack (my library to easily post messages to the Slack group-messaging utility) to support per-message backends and channels. (#63)
  • Created a pull request for django-two-factor-auth, a complete Two-Factor Authentication (2FA) framework for projects using the Django web development framework to drop use of the @lazy_property decorator to ensure compatibility with Django 1.11. (#195)
  • Filed, triaged and eventually merged a change from @evgeni to fix an autopkgtest-related issue in travis.debian.net, my hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds on every code change). (#41)
  • Submitted a pull request against social-core — a library to allow Python applications to authenticate against third-party web services such as Facebook, Twitter, etc. — to use the more-readable X if Y else Z construction over Y and X or Z. (#44)
  • Filed an issue against freezegun (a tool to make it easier to write Python tests involving times) to report that dateutils was missing from requirements.txt. (#173)
  • Submitted a pull request against the Hypothesis "QuickCheck"-like testing framework to make the build reproducible. (#440)
  • Fixed an issue reported by @davidak in trydiffoscope (a web-based version of the diffoscope in-depth and content-aware diff utility) where the maximum upload size was incorrectly calculated. (#22)
  • Created a pull request for the Mars Simulation Project to remove some embedded timestamps from the changelog.gz and mars-sim.1.gz files in order to make the build reproducible. (#24)
  • Filed a bug against the cpio archiving utility to report that the testsuite fails when run in the UTC +1300 timezone. (Thread)
  • Submitted a pull request against the "pnmixer" system-tray volume mixer in order to make the build reproducible. (#153)
  • Sent a patch to Testfixtures (a collection of helpers and mock objects that are useful when writing Python unit tests or doctests) to make the build reproducible. (#56)
  • Created a pull request for the "Cloud" Sphinx documentation theme in order to make the output reproducible. (#22)

Reproducible builds


Whilst anyone can inspect the source code of free software for malicious flaws, most software is distributed pre-compiled to end users.

The motivation behind the Reproducible Builds effort is to permit verification that no flaws have been introduced — either maliciously or accidentally — during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

(I have been awarded a grant from the Core Infrastructure Initiative to fund my work in this area.)

This month I:


I also made the following changes to our tooling:

diffoscope

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues.

  • New features:
    • Add a machine-readable JSON output format. (Closes: #850791).
    • Add an --exclude option. (Closes: #854783).
    • Show results from debugging packages last. (Closes: #820427).
    • Extract archive members using an auto-incrementing integer avoiding the need to sanitise filenames. (Closes: #854723).
    • Apply --max-report-size to --text output. (Closes: #851147).
    • Specify <html lang="en"> in the HTML output. (re. #849411).
  • Bug fixes:
    • Fix errors when comparing directories with non-directories. (Closes: #835641).
    • Device and RPM fallback comparisons require xxd. (Closes: #854593).
    • Fix tests that call xxd on Debian Jessie due to change of output format. (Closes: #855239).
    • Add missing Recommends for comparators. (Closes: #854655).
    • Importing submodules (ie. parent.child) will attempt to import parent. (Closes: #854670).
    • Correct logic of module_exists ensuring we correctly skip the debian.deb822 tests when python3-debian is not installed. (Closes: #854745).
    • Clean all temporary files in the signal handler thread instead of attempting to pass the exception back to the main thread. (Closes: #852013).
    • Fix behaviour of setting report maximums to zero (ie. no limit).
  • Optimisations:
    • Don't uselessly run xxd(1) on non-directories.
    • No need to track libarchive directory locations.
    • Optimise create_limited_print_func.
  • Tests:
    • When comparing two empty directories, ensure that the mtime of the directory is consistent to avoid non-deterministic failures.
    • Ensure we can at least import the "deb_fallback" and "rpm_fallback" modules.
    • Add test for symlink differing in destination.
    • Add tests for --progress, --status-fd and profiling output options as well as the Deb{Changes,Buildinfo,Dsc} and RPM fallback comparisons.
    • Add get_data and @skip_unless_module_exists test helpers.
    • Mark impossible-to-reach code to improve test coverage.

buildinfo.debian.net

buildinfo.debian.net is my experiment into how to process, store and distribute .buildinfo files after the Debian archive software has processed them.

  • Drop raw_text fields now as we've moved these to Amazon S3.
  • Drop storage of Installed-Build-Depends and subsequently-orphaned Binary package instances to recover diskspace.

strip-nondeterminism

strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build.

  • Print log entry when fixing a file. (Closes: #777239).
  • Run our entire testsuite in autopkgtests, not just the first test. (Closes: #852517).
  • Don't test for stat(2)'s blksize and block attributes. (Closes: #854937).
  • Use error() from Dh_Lib.pm over "manual" die().


Debian


Debian LTS


This month I have been paid to work 13 hours on Debian Long Term Support (LTS). In that time I did the following:

  • "Frontdesk" duties, triaging CVEs, etc.
  • Issued DLA 817-1 for libphp-phpmailer, correcting a local file disclosure vulnerability where insufficient parsing of HTML messages could potentially be used by attacker to read a local file.
  • Issued DLA 826-1 for wireshark which fixes a denial of service vulnerability in wireshark, where a malformed NATO Ground Moving Target Indicator Format ("STANAG 4607") capture file could cause a memory exhausion/infinite loop.

Uploads

  • python-django (1:1.11~beta1-1) — New upstream beta release.
  • redis (3:3.2.8-1) — New upstream release.
  • gunicorn (19.6.0-11) — Use ${misc:Pre-Depends} to populate Pre-Depends for dpkg-maintscript-helper.
  • dh-virtualenv (1.0-1~bpo8+1) — Upload to jessie-backports.

I sponsored the following uploads:

I also performed the following QA uploads:

  • dh-kpatches (0.99.36+nmu4) — Make kernel kernel builds reproducible.

Finally, I made the following non-maintainer uploads:

  • cpio (2.12+dfsg-3) — Remove rmt.8.gz to prevent a piuparts error.
  • dot-forward (1:0.71-2.2) — Correct a FTBFS; we don't install anything to /usr/sbin, so use GNU Make's $(wildcard ..) over the shell's own * expansion.


FTP Team


As a Debian FTP assistant I ACCEPTed 116 packages: autobahn-cpp, automat, bglibs, bitlbee, bmusb, bullet, case, certspotter, checkit-tiff, dash-el, dash-functional-el, debian-reference, el-x, elisp-bug-hunter, emacs-git-messenger, emacs-which-key, examl, genwqe-user, giac, golang-github-cloudflare-cfssl, golang-github-docker-goamz, golang-github-docker-libnetwork, golang-github-go-openapi-spec, golang-github-google-certificate-transparency, golang-github-karlseguin-ccache, golang-github-karlseguin-expect, golang-github-nebulouslabs-bolt, gpiozero, gsequencer, jel, libconfig-mvp-slicer-perl, libcrush, libdist-zilla-config-slicer-perl, libdist-zilla-role-pluginbundle-pluginremover-perl, libevent, libfunction-parameters-perl, libopenshot, libpod-weaver-section-generatesection-perl, libpodofo, libprelude, libprotocol-http2-perl, libscout, libsmali-1-java, libtest-abortable-perl, linux, linux-grsec, linux-signed, lockdown, lrslib, lua-curses, lua-torch-cutorch, mariadb-10.1, mini-buildd, mkchromecast, mocker-el, node-arr-exclude, node-brorand, node-buffer-xor, node-caller, node-duplexer3, node-ieee754, node-is-finite, node-lowercase-keys, node-minimalistic-assert, node-os-browserify, node-p-finally, node-parse-ms, node-plur, node-prepend-http, node-safe-buffer, node-text-table, node-time-zone, node-tty-browserify, node-widest-line, npd6, openoverlayrouter, pandoc-citeproc-preamble, pydenticon, pyicloud, pyroute2, pytest-qt, pytest-xvfb, python-biomaj3, python-canonicaljson, python-cgcloud, python-gffutils, python-h5netcdf, python-imageio, python-kaptan, python-libtmux, python-pybedtools, python-pyflow, python-scrapy, python-scrapy-djangoitem, python-signedjson, python-unpaddedbase64, python-xarray, qcumber, r-cran-urltools, radiant, repo, rmlint, ruby-googleauth, ruby-os, shutilwhich, sia, six, slimit, sphinx-celery, subuser, swarmkit, tmuxp, tpm2-tools, vine, wala & x265.

I additionally filed 8 RC bugs against packages that had incomplete debian/copyright files against: checkit-tiff, dash-el, dash-functional-el, libcrush, libopenshot, mkchromecast, pytest-qt & x265.