[Jenkins-infra] cucumber outage

Arnaud Héritier aheritier at gmail.com
Wed Jan 6 13:19:05 UTC 2016


Hi,

  This morning I received some alerts about a full disk but there was no
detail from which server.
  This afternoon ldap crashed, the wiki was unavailable and I found that
the full disk was / on cucumber

aheritier at cucumber:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             895G  850G     0 100% /
none                  3.9G  216K  3.9G   1% /dev
none                  3.9G     0  3.9G   0% /dev/shm
none                  3.9G   75M  3.8G   2% /var/run
none                  3.9G     0  3.9G   0% /var/lock
none                  3.9G     0  3.9G   0% /lib/init/rw

Strangely Used < Size

I stopped jenkins service and remove 3Gb of logs because it was recording
many exceptions about the no space left on device

WARNING: exception while collecting data: java.io.IOException: No space
left on device
java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:345)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.util.zip.DeflaterOutputStream.flush(DeflaterOutputStream.java:282)
at
net.bull.javamelody.FilterServletOutputStream.flush(FilterServletOutputStream.java:65)
at
java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1822)
at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:718)
at java.io.ObjectOutputStream.close(ObjectOutputStream.java:739)
at net.bull.javamelody.CounterStorage.writeToFile(CounterStorage.java:83)
at net.bull.javamelody.Counter.writeToFile(Counter.java:881)
at
net.bull.javamelody.Collector.collectCounterRequestsAndErrorsData(Collector.java:694)
at net.bull.javamelody.Collector.collectCounterData(Collector.java:673)
at net.bull.javamelody.Collector.collect(Collector.java:334)
at net.bull.javamelody.Collector.collectWithoutErrors(Collector.java:306)
at
net.bull.javamelody.Collector.collectLocalContextWithoutErrors(Collector.java:295)
at
net.bull.javamelody.FilterContext$CollectTimerTask.run(FilterContext.java:56)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

I restarted LDAP and confluence

I don't know where to do more cleanup on cucumber

cheers

-- 
-----
Arnaud Héritier
http://aheritier.net
Mail/GTalk: aheritier AT gmail DOT com
Twitter/Skype : aheritier
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.jenkins-ci.org/pipermail/jenkins-infra/attachments/20160106/8b5fe8bf/attachment.html>


More information about the Jenkins-infra mailing list