Toolchain issues

I submitted the following patches to fix reproducibility-related toolchain issues:

• dh-python: Please make builds reproducible with a "mismatched" kernel and userland
• dh-lua: Please make the substvars reproducible
• filepp: Please make the output (and build) reproducible
• javatools: Please make the Recommends substvars reproducible
• Perl:
  • Please make the output of ExtUtils::MM_Unix reproducible
  • Please make the output of ExtUtils::Command::MM reproducible
  • Please make the output of libmodule-build-withxspp-perl reproducible

My work in the Reproducible Builds project was also covered in our weekly reports. (#67, #68, #69, #70).

Diffoscope

diffoscope is our "diff on steroids" that will not only recursively unpack archives but will transform binary formats into human-readable forms in order to compare them:

• Added a command-line interface to the try.diffoscope.org web service.
• Added a JSON comparator.
• In the HTML output, highlight lines when hovering to make it easier to visually track.
• Ensure that we pass str types to our Difference class, otherwise we can't be sure we can render them later.
• Testsuite improvements:
  • Generate test coverage reports.
  • Add tests for Haskell and GitIndex comparators.
  • Completely refactored all of the comparator tests, extracting out commonly-used routines.
  • Confirm rendering of text and HTML presenters when checking non-existing files.
  • Dropped a squashfs test as it was simply too unreliable and/or has too many requirements to satisfy.
• A large number of miscellaneous cleanups, including:
  • Reworking the comparator setup/preference internals by dynamically importing classes via a single list.
  • Split exceptions out into dedicated diffoscope.exc module.
  • Tidying the PROVIDERS dict in diffoscope/__init__.py.
  • Use html.escape over xml.sax.saxutils.escape, cgi.escape, etc.
  • Removing hard-coding of manual page targets names in debian/rules.
  • Specify all string format arguments as logging function parameters, not using interpolation.
  • Tidying imports, correcting indentation levels and drop unnecessary whitespace.

disorderfs

disorderfs is our FUSE filesystem that deliberately introduces nondeterminism in system calls such as readdir(3).

• Added a testsuite to prevent regressions. (f124965)
• Added a --sort-dirents=yes|no option for forcing deterministic ordering. (2aae325)

Other

• Improved strip-nondeterminism, our tool to remove specific nondeterministic information after a build:
  • Match more styles of Java .properties files.
  • Remove hyphen from "non-determinism" and "non-deterministic" throughout package for consistency.
• Improvements to our testing infrastucture:
  • Improve the top-level navigation so that we can always get back to "home" of a package.
  • Give expandable elements cursor: pointer CSS styling to highlight they are clickable.
  • Drop various trailing underlined whitespaces after links.
  • Explicitly log that build was successful or not.
  • Various code-quality improvements, including prefering str.format over concatentation.
• Miscellaneous updates to our filter-packages internal tool:
  • Add --random=N and --url options.
  • Add support for --show=comments.
  • Correct ordering so that --show-version runs after --filter-ftbfs.
  • Rename --show-ftbfs to --filter-ftbfs and --show-version to --show=version.
• Created a proof-of-concept reproducible-utils package to contain commonly-used snippets aimed at developers wishing to make their packages reproducible.

I also submitted 92 patches to fix specific reproducibility issues in advi, amora-server, apt-cacher-ng, ara, argyll, audiotools, bam, bedtools, binutils-m68hc1x, botan1.10, broccoli, congress, cookiecutter, dacs, dapl, dateutils, ddd, dicom3tools, dispcalgui, dnssec-trigger, echoping, eekboek, emacspeak, eyed3, fdroidserver, flashrom, fntsample, forkstat, gkrellm, gkrellm, gnunet-gtk, handbrake, hardinfo, ircd-irc2, ircd-ircu, jack-audio-connection-kit, jpy, kxmlgui, libbson, libdc0, libdevel-cover-perl, libfm, libpam-ldap, libquvi, librep, lilyterm, mozvoikko, mp4h, mp4v2, myghty, n2n, nagios-nrpe, nikwi, nmh, nsnake, openhackware, pd-pdstring, phpab, phpdox, phpldapadmin, pixelmed-codec, pleiades, pybit, pygtksourceview, pyicu, python-attrs, python-gflags, quvi, radare2, rc, rest2web, roaraudio, rt-extension-customfieldsonupdate, ruby-compass, ruby-pg, sheepdog, tf5, ttf-tiresias, ttf-tiresias, tuxpaint, tuxpaint-config, twitter-bootstrap3, udpcast, uhub, valknut, varnish, vips, vit, wims, winswitch, wmweather+ & xshisen.

Patches contributed

• Lintian:
  • Warn about Python packages that ship .coverage information
  • Check for libjs-* binary package name outside of web section
  • Warn if filenames contain wildcard characters)
  • Correct false positives in matching typo targets with extra whitespace
• devscripts: Add an "--extra-packages" option to install arbitrary extra packages
• dh-python: Emit invokation and command-line arguments of "after" and "before" commands by default
• debsources: Retain line context on 404s by appending the hash for specific links
• python-debian: Improve message when passing incorrect type to iter_paragraphs
• autopkgtest: Does not run pyflakes3 tests
• python3-debian: Include examples in package
• snapshot.debian.org: Correct link to API page in sidebar
• dh-python: Misparses d/control and copies build profiles into substvars
• auto-apt-proxy: FTBFS: In POSIX sh, 'local' is undefined

I also submitted 22 patches to fix typos in debian/rules files against ctsim, f2c, fonts-elusive-icons, ifrit, ldapscripts, libss7, libvmime, link-grammar, menulibre, mit-scheme, mugshot, nlopt, nunit, proftpd-mod-autohost, proftpd-mod-clamav, rabbyt, radvd, ruby-image-science, snmpsim, speech-tools, varscan & whatmaps.

Debian LTS

This month I have been paid to work 15 hours on Debian Long Term Support (LTS). In that time I did the following:

• "Frontdesk" duties, triaging CVEs, etc.
• Authored the patch & issued DLA 596-1 for extplorer, a web-based file manager, fixing an archive traversal exploit.
• Issued DLA 598-1 for suckless-tools, fixing a segmentation fault in the slock screen locking tool.
• Issued DLA 599-1 for cracklib2, a pro-active password checker library, fixing a stack-based buffer overflow when parsing large GECOS fields.
• Improved the find-work internal tool adding optional colour highlighting and migrating it to Python 3.
• Wrote an lts-missing-uploads tool to find mistakes where there was no correponding package in the archive after an announcement.
• Added optional colour highlighting to the lts-cve-triage tool.

Uploads

• redis 2:3.2.3-1 — New upstream release, move to the DEP-5 debian/copyright format, ensure that we are running as root in LSB initscripts and add a README.Source regarding our local copies of redis.conf and sentinel.conf.
• python-django:
  • 1:1.10-1 — New upstream release.
  • 1:1.10-2 — Fix test failures due to mishandled upstream translation updates.

• gunicorn:
  • 19.6.0-2 — Reload logrotate in the postrotate action to avoid processes writing to the old files and move to DEP-5 debian/copyright format.
  • 19.6.0-3 — Drop our /usr/sbin/gunicorn{,3}-debian and related Debian-specific machinery to be more like upstream.
  • 19.6.0-4 — Drop "template" systemd .service files and point towards examples and documentation instead.

• adminer:
  • 4.2.5-1 — Take over package maintenance, completely overhauling the packaging with a new upstream version, move to virtual-mysql-server to support MariaDB, updating package names of dependencies and fix the outdated Apache configuration.
  • 4.2.5-2 — Correct the php5 package names.

Bugs filed (without patches)

• python3-debian: Can't parse input of bytes under Python 3
• devscripts: Please warn if DEBUILD_DPKG_BUILDPACKAGE_OPTS contains "-i -I"
• gbrowse: Ships a predictable OpenID constumer secret
• Popcon, etc:
  • python-popcon: Must not crash on invalid data
  • popularity-contest: all-popcon-results.txt contains invalid data
• congress: CongressParser.py not generated during build
• libfuzzer-3.8-dev:
  • "Illegal hardware instruction"
  • Please add "Suggests" for (some) version of Clang
  • Please provide a programmatic method of finding libFuzzer.a
• ceph: Maintainer fields points to a moderated mailing list
• nsntrace: Please package a new upstream snapshot and please upload to jessie-backports.
• safecat: Homepage field refers to missing URL
• foxeye: Please make the the package autoreconfable

RC bugs

I filed 3 RC bugs with patches:

• gnome-packagekit: Missing Build-Depends on 'sp'
• python-popcon: Assumes all-popcon-results.txt.gz file is UTF-8
• python3-popcon: Missing Depends on 'python3-xdg'

I additionally filed 8 RC bugs for packages that access the internet during build against autopkgtest, golang-github-xenolf-lego, pam-python, pexpect, python-certbot, python-glanceclient, python-pykka & python-tornado.

I also filed 74 FTBFS bugs against airlift-airline, airlift-slice, alter-sequence-alignment, apktool, atril, auto-apt-proxy, bookkeeper, bristol, btfs, caja-extensions, ccbuild, cinder, clustalo, colorhug-client, cpp-netlib, dimbl, edk2, elasticsearch, ganv, git-remote-hg, golang-codegangsta-cli, golang-goyaml, gr-radar, imagevis3d, jacktrip, jalv, kdepim, kiriki, konversation, libabw, libcereal, libdancer-plugin-database-perl, libdist-zilla-plugins-cjm-perl, libfreemarker-java, libgraph-writer-dsm-perl, libmail-gnupg-perl, libminc, libsmi, linthesia, lv2-c++-tools, lvtk, mate-power-manager, mcmcpack, mopidy-podcast, nageru, nfstrace, nova, nurpawiki, open-gram, php-crypt-gpg, picmi, projectl, pygpgme, python-apt, python-django-bootstrap-form, python-django-navtag, python-oslo.config, qmmp, qsapecng, r-cran-sem, rocs, ruby-mini-magick, seahorse-nautilus, shiro, snap, tcpcopy, tiledarray, triggerhappy, ucto, urdfdom, vmmlib, yara-python, yi & z3.

FTP Team

As a Debian FTP assistant I ACCEPTed 90 packages: android-platform-external-jsilver, android-platform-frameworks-data-binding, camlpdf, consolation, dfwinreg, diffoscope, django-restricted-resource, django-testproject, django-testscenarios, gitlab-ci-multi-runner, gnome-shell-extension-taskbar, golang-github-flynn-archive-go-shlex, golang-github-jamesclonk-vultr, golang-github-weppos-dnsimple-go, golang-golang-x-time, google-android-ndk-installer, haskell-expiring-cache-map, haskell-hclip, haskell-hdbc-session, haskell-microlens-ghc, haskell-names-th, haskell-persistable-record, haskell-should-not-typecheck, haskell-soap, haskell-soap-tls, haskell-th-reify-compat, haskell-with-location, haskell-wreq, kbtin, libclipboard-perl, libgtk3-simplelist-perl, libjs-jquery-selectize.js, liblemon, libplack-middleware-header-perl, libreoffice, libreswan, libtest-deep-json-perl, libtest-timer-perl, linux, linux-signed, live-tasks, llvm-toolchain-3.8, llvm-toolchain-snapshot, lua-luv, lua-torch-image, lua-torch-nn, magic-wormhole, mini-buildd, ncbi-vdb, node-ast-util, node-es6-module-transpiler, node-es6-promise, node-inline-source-map, node-number-is-nan, node-object-assign, nvidia-graphics-drivers, openhft-chronicle-bytes, openhft-chronicle-core, openhft-chronicle-network, openhft-chronicle-threads, openhft-chronicle-wire, pycodestyle, python-aptly, python-atomicwrites, python-click-log, python-django-casclient, python-git-os-job, python-hypothesis, python-nosehtmloutput, python-overpy, python-parsel, python-prov, python-py, python-schema, python-tackerclient, python-tornado, pyvo, r-cran-cairo, r-cran-mi, r-cran-rcppgsl, r-cran-sem, ruby-curses, ruby-fog-rackspace, ruby-mixlib-archive, ruby-tzinfo-data, salt-formula-swift, scapy3k, self-destructing-cookies, trollius-redis & websploit.