[Jenkins-infra] cucumber outage
Arnaud Héritier
aheritier at gmail.com
Wed Jan 6 13:19:05 UTC 2016
Hi,
This morning I received some alerts about a full disk but there was no
detail from which server.
This afternoon ldap crashed, the wiki was unavailable and I found that
the full disk was / on cucumber
aheritier at cucumber:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 895G 850G 0 100% /
none 3.9G 216K 3.9G 1% /dev
none 3.9G 0 3.9G 0% /dev/shm
none 3.9G 75M 3.8G 2% /var/run
none 3.9G 0 3.9G 0% /var/lock
none 3.9G 0 3.9G 0% /lib/init/rw
Strangely Used < Size
I stopped jenkins service and remove 3Gb of logs because it was recording
many exceptions about the no space left on device
WARNING: exception while collecting data: java.io.IOException: No space
left on device
java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:345)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.util.zip.DeflaterOutputStream.flush(DeflaterOutputStream.java:282)
at
net.bull.javamelody.FilterServletOutputStream.flush(FilterServletOutputStream.java:65)
at
java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1822)
at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:718)
at java.io.ObjectOutputStream.close(ObjectOutputStream.java:739)
at net.bull.javamelody.CounterStorage.writeToFile(CounterStorage.java:83)
at net.bull.javamelody.Counter.writeToFile(Counter.java:881)
at
net.bull.javamelody.Collector.collectCounterRequestsAndErrorsData(Collector.java:694)
at net.bull.javamelody.Collector.collectCounterData(Collector.java:673)
at net.bull.javamelody.Collector.collect(Collector.java:334)
at net.bull.javamelody.Collector.collectWithoutErrors(Collector.java:306)
at
net.bull.javamelody.Collector.collectLocalContextWithoutErrors(Collector.java:295)
at
net.bull.javamelody.FilterContext$CollectTimerTask.run(FilterContext.java:56)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
I restarted LDAP and confluence
I don't know where to do more cleanup on cucumber
cheers
--
-----
Arnaud Héritier
http://aheritier.net
Mail/GTalk: aheritier AT gmail DOT com
Twitter/Skype : aheritier
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.jenkins-ci.org/pipermail/jenkins-infra/attachments/20160106/8b5fe8bf/attachment.html>
More information about the Jenkins-infra
mailing list