[Jenkins-infra] More ci.jenkins.io outages and problems

[R. Tyler Croy]

Last night we had more outages with ci.jenkins.io due to issues with Jenkins
Pipeline.

Jesse helpfully pointed me to this issue:
    https://issues.jenkins-ci.org/browse/JENKINS-47158

Which, when I responded to the disk usage alert on the / partition, was being
streamed to disk on an infinite loop. I did a hard shut down of Jenkins (no
"Prepare for Shutdown"), turned off Jenkins, and restarted the Docker daemon.
This allowed for some disk space to be reclaimed on the root partition, since a
bunch of garbage was in the union file system for the `jenkins` container.

Upon restart, the Jenkins process inexplicably stopped spewing that exception
on an infinite loop. If it does so again, I have asked olblak to shut off
Jenkins and then I will escalate the issue to Kohsuke :)

This morning, I responded to a disk usage alert for the /var/lib/jenkins
partition, which is a couple hundred gigabytes, and was down to the last 800
megabytes. I filed this issue:
    https://issues.jenkins-ci.org/browse/JENKINS-48452

And ran a Groovy script which I have added to the runbooks which deletes all be
the five most recent Pipeline runs in the Plugins GitHub Org Folder. This
reclaimed about 48GB of space.



Long story short, my tolerance for systemic deficiencies in the operability of
Jenkins Pipeline has reached an all-time low. James Dumay, product manager for
Pipeline at CloudBees, and I will be discussing the challenges facing Pipelines
from a Sysadmin standpoint next week, and hopefully will get some of the
long-overdue-features implemented and some long-overdue-bugs fixed.



Cheers
- R. Tyler Croy

------------------------------------------------------
     Code: <https://github.com/rtyler>
  Chatter: <https://twitter.com/agentdero>
     xmpp: rtyler at jabber.org

  % gpg --keyserver keys.gnupg.net --recv-key 1426C7DC3F51E16F
------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <http://lists.jenkins-ci.org/pipermail/jenkins-infra/attachments/20171208/4f020295/attachment.asc>