[Jenkins-infra] 504 downloading from pkg.jenkins-ci.org

[Arnaud Héritier]

It seems that we have always a lot of 502/503 errors for
pkg.jenkins-ci.orgtoday (reported by a coworker of me)

The access log is huge

1.1G 2014-05-13 07:43 access.log.20140508000000

There is no rotation of logs ?
I tried to find 503 without success thus I suppose that the problem is with
haproxy that doesn't succeed to talk with apache
I found a 502 for example on apache side :

"82.125.195.8" - - [13/May/2014:06:11:33 -0400] "GET /debian-stable/binary/
HTTP/1.1" 403 481 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"
502 481

In syslog we find many

kernel: [25582120.640081] possible SYN flooding on port 8888. Sending
cookies.
Is it a real attack ? Or haproxy which is badly configured sends to many
things to apache on 8888 ?
On internet we can find various articles about the kernel tuning to avoid
this but I didn't test them by myself.

CPU and memory are "fine" and load average not high (~1,2)


cheers

arnaud




On Fri, Apr 4, 2014 at 6:53 PM, Kohsuke Kawaguchi
<kkawaguchi at cloudbees.com>wrote:

>
> Tyler and I speculated several possibilities. I'll login to the box in the
> morning and see if I can figure out what's going on.
>
>
> On 04/03/2014 08:30 PM, R. Tyler Croy wrote:
>
>> On Wed, 2 Apr 2014 15:07:55 +0000
>> "Konrad, Martin" <konrad at frib.msu.edu> wrote:
>>
>>  Hi,
>>> I am experiencing problems downloading Debian packages from
>>> pkg.jenkins-ci.org:
>>>
>>
>>
>> I'm hoping Kohsuke can weigh in on this thread. There's something
>> that's running on cucumber that's obliterating the machine ~5am PST.
>>
>> I've now seen reports of outages for the past 3 days during this early
>> morning time frame and they all seem to be very transient.
>>
>> The infra_drupalcron job fails during the same period too :/
>>
>>
>> WTF is running on J-on-J?
>>
>>
>
> --
> Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/
> Try Jenkins Enterprise, our professional version of Jenkins
>
> _______________________________________________
> Jenkins-infra mailing list
> Jenkins-infra at lists.jenkins-ci.org
> http://lists.jenkins-ci.org/mailman/listinfo/jenkins-infra
>



-- 
-----
Arnaud Héritier
http://aheritier.net
Mail/GTalk: aheritier AT gmail DOT com
Twitter/Skype : aheritier
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.jenkins-ci.org/pipermail/jenkins-infra/attachments/20140513/3b92e1ad/attachment.html>