<div dir="ltr">I wonder if there's any way to track who this IP address belongs to. Port 22 responds with "SSH-2.0-NetScreen", which I assume is a <a href="http://www.juniper.net/us/en/products-services/security/netscreen/">Juniper NetScreen</a>. The IP address <a href="http://64.125.71.142.ipaddress.com/">claims to be in San Francisco</a>.</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">2014-06-02 16:52 GMT-07:00 Kohsuke Kawaguchi <span dir="ltr"><<a href="mailto:kkawaguchi@cloudbees.com" target="_blank">kkawaguchi@cloudbees.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
More progress.<br>
<br>
When the last overage happened we started using mod_logio to record total bytes in/out per request [1]. So I looked at which virtual host is eating bandwidth, and the result was quite surprising.<br>
<br>
Looking at the # of bytes transferred outbound per virtual host per week, I get this.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
ci 489,290,628,016<br>
main 12,203,528,309<br>
maven 7,378,839,415<br>
pkg 8,105,888,873<br>
svn 937,561,148<br>
updates 17,900,766,883<br>
</blockquote>
<br>
So <a href="http://ci.jenkins-ci.org" target="_blank">ci.jenkins-ci.org</a> has served 490GB/week (!) of data. That's clearly too much.<br>
<br>
So I looked at its access log and noticed that <a href="tel:64.125.71.142" value="+16412571142" target="_blank">64.125.71.142</a> is downloading /job/jenkins_rc_branch/<u></u>lastSuccessfulBuild/artifact/<u></u>war/target/jenkins.war over and over.<br>
<br>
This file is about 68MB in size, and a download happens every minute. That's about 95GB/day, or 2.8TB/month if extrapolated.<br>
<br>
THIS GUY SINGLEHANDEDLY CONSUMED MORE THAN HALF THE BANDWIDTH!<br>
<br>
<br>
<br>
The actual download counts per week is as follows:<br>
<br>
Apr,03 10165<br>
Apr,10 10171<br>
Apr,17 7426<br>
Apr,24 10180<br>
May,1 11203<br>
May,8 10226<br>
May,15 8975<br>
May,22 10140<br>
<br>
There are occasional gaps (perhaps <a href="http://ci.jenkins-ci.org" target="_blank">ci.jenkins-ci.org</a> was down, or there's some pause in what he does), but still clearly this amounts to a ridiculous amount of network transfer.<br>
<br>
I'm going to ban this IP in apache config to stop this immediately.<br>
<br>
<br>
Going forward, we need a better approach to this than waiting for the overage to slap our face before noticing the problem. For that I think we need to have some monitoring in place to alert us in case of sudden traffic increase.<br>
<br>
Also, as you see in the log above, this didn't start in May, so it doesn't explain the sudden increase in the traffic of May. So we've got still more analysis to do.<br>
<br>
<br>
<br>
<br>
<br>
[1] <a href="http://httpd.apache.org/docs/2.2/mod/mod_logio.html" target="_blank">http://httpd.apache.org/docs/<u></u>2.2/mod/mod_logio.html</a><div class="HOEnZb"><div class="h5"><br>
<br>
On 06/02/2014 03:39 PM, Kohsuke Kawaguchi wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I looked at output from haproxy<br>
<a href="http://kohsuke.org/private/20140602/haproxy-stats.png" target="_blank">http://kohsuke.org/private/<u></u>20140602/haproxy-stats.png</a>, and this shows a<br>
large amount of activities under "maven", which is<br>
<a href="http://maven.jenkins-ci.org/" target="_blank">http://maven.jenkins-ci.org/</a> that acts as a reverse proxy to<br>
<a href="http://repo.jenkins-ci.org" target="_blank">repo.jenkins-ci.org</a>.<br>
<br>
If you look under "bytes out", of 6.8TB that has been served for the<br>
duration of the haproxy uptime, 1TB is from <a href="http://maven.jenkins-ci.org" target="_blank">maven.jenkins-ci.org</a>.<br>
<br>
This is surprising because all the download traffic for Maven repository<br>
should be served through <a href="http://repo.jenkins-ci.org/" target="_blank">http://repo.jenkins-ci.org/</a><br>
<br>
I need to look into this a bit more.<br>
<br>
<br>
OTOH, note that this shows only the cumulative value. I recorded the<br>
value 2.5 hours later, and the delta for maven was 169MB (extrapolates<br>
to 50GB/month) whereas delta for overall is 9.6GB (extrapolates to<br>
2.7TB/month.) So it's nowhere near big enough to explain the<br>
1.45TB/month usage spike in May.<br>
<br>
In addition, the traffic to <a href="http://maven.jenkins-ci.org" target="_blank">maven.jenkins-ci.org</a> is reverse-proxied to<br>
<a href="http://repo.jenkins-ci.org" target="_blank">repo.jenkins-ci.org</a>, and so if this accounts for the traffic increase,<br>
it should show up in the corresponding increase in the RX side. There's<br>
no such spike in the RX side.<br>
<br>
I'll continue digging...<br>
<br>
<br>
On 06/02/2014 11:15 AM, Kohsuke Kawaguchi wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Tyler got a surprisingly bill for the overage charge for cucumber, which<br>
runs "<a href="http://jenkins-ci.org" target="_blank">jenkins-ci.org</a>", "<a href="http://mirrors.jenkins-ci.org" target="_blank">mirrors.jenkins-ci.org</a>" and a number of other<br>
virtual hosts.<br>
<br>
I think I've set up vnstat when it happened the last time to track<br>
utlization. Here is the vnstat output.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
kohsuke@cucumber:~$ vnstat -m<br>
<br>
eth0 / monthly<br>
<br>
month rx | tx | total | avg. rate<br>
------------------------+-----<u></u>--------+-------------+-------<u></u>--------<br>
Jul '13 338.35 GiB | 3.21 TiB | 3.54 TiB | 11.36 Mbit/s<br>
Aug '13 338.36 GiB | 2.86 TiB | 3.19 TiB | 10.23 Mbit/s<br>
Sep '13 354.39 GiB | 3.52 TiB | 3.87 TiB | 12.82 Mbit/s<br>
Oct '13 395.09 GiB | 4.20 TiB | 4.59 TiB | 14.72 Mbit/s<br>
Nov '13 449.73 GiB | 3.51 TiB | 3.94 TiB | 13.07 Mbit/s<br>
Dec '13 562.26 GiB | 3.68 TiB | 4.23 TiB | 13.56 Mbit/s<br>
Jan '14 672.19 GiB | 3.91 TiB | 4.56 TiB | 14.64 Mbit/s<br>
Feb '14 370.69 GiB | 3.13 TiB | 3.49 TiB | 12.39 Mbit/s<br>
Mar '14 351.83 GiB | 3.33 TiB | 3.67 TiB | 11.77 Mbit/s<br>
Apr '14 362.76 GiB | 3.39 TiB | 3.74 TiB | 12.40 Mbit/s<br>
May '14 401.56 GiB | 4.80 TiB | 5.19 TiB | 16.65 Mbit/s<br>
Jun '14 20.11 GiB | 241.83 GiB | 261.94 GiB | 16.07 Mbit/s<br>
------------------------+-----<u></u>--------+-------------+-------<u></u>--------<br>
estimated 381.25 GiB | 4.48 TiB | 4.85 TiB |<br>
</blockquote>
<br>
As you see, the outbound traffic jumped in May.<br>
Here's the daily output, and I think it means that new trend is<br>
continuing in June so far as I can tell.<br>
<br>
In other words, we need to act on it ASAP to avoid another overage for June.<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
kohsuke@cucumber:~$ vnstat -d<br>
<br>
eth0 / daily<br>
<br>
day rx | tx | total | avg. rate<br>
------------------------+-----<u></u>--------+-------------+-------<u></u>--------<br>
05/04/14 11.61 GiB | 151.50 GiB | 163.11 GiB | 15.84 Mbit/s<br>
05/05/14 18.37 GiB | 167.82 GiB | 186.18 GiB | 18.08 Mbit/s<br>
05/06/14 12.54 GiB | 176.53 GiB | 189.07 GiB | 18.36 Mbit/s<br>
05/07/14 13.07 GiB | 169.65 GiB | 182.73 GiB | 17.74 Mbit/s<br>
05/08/14 12.49 GiB | 152.46 GiB | 164.95 GiB | 16.01 Mbit/s<br>
05/09/14 13.54 GiB | 167.91 GiB | 181.45 GiB | 17.62 Mbit/s<br>
05/10/14 10.72 GiB | 149.50 GiB | 160.22 GiB | 15.56 Mbit/s<br>
05/11/14 15.25 GiB | 141.75 GiB | 157.01 GiB | 15.24 Mbit/s<br>
05/12/14 16.06 GiB | 168.05 GiB | 184.11 GiB | 17.88 Mbit/s<br>
05/13/14 13.25 GiB | 144.43 GiB | 157.68 GiB | 15.31 Mbit/s<br>
05/14/14 18.24 GiB | 160.37 GiB | 178.61 GiB | 17.34 Mbit/s<br>
05/15/14 14.30 GiB | 154.11 GiB | 168.41 GiB | 16.35 Mbit/s<br>
05/16/14 12.99 GiB | 153.21 GiB | 166.20 GiB | 16.14 Mbit/s<br>
05/17/14 9.87 GiB | 127.64 GiB | 137.51 GiB | 13.35 Mbit/s<br>
05/18/14 11.43 GiB | 186.48 GiB | 197.91 GiB | 19.22 Mbit/s<br>
05/19/14 14.08 GiB | 171.26 GiB | 185.35 GiB | 18.00 Mbit/s<br>
05/20/14 13.47 GiB | 149.67 GiB | 163.14 GiB | 15.84 Mbit/s<br>
05/21/14 12.67 GiB | 150.21 GiB | 162.89 GiB | 15.81 Mbit/s<br>
05/22/14 13.43 GiB | 168.17 GiB | 181.61 GiB | 17.63 Mbit/s<br>
05/23/14 14.06 GiB | 165.47 GiB | 179.53 GiB | 17.43 Mbit/s<br>
05/24/14 9.75 GiB | 132.61 GiB | 142.36 GiB | 13.82 Mbit/s<br>
05/25/14 10.10 GiB | 131.50 GiB | 141.60 GiB | 13.75 Mbit/s<br>
05/26/14 13.81 GiB | 180.48 GiB | 194.28 GiB | 18.86 Mbit/s<br>
05/27/14 14.51 GiB | 187.94 GiB | 202.45 GiB | 19.66 Mbit/s<br>
05/28/14 12.76 GiB | 163.84 GiB | 176.60 GiB | 17.15 Mbit/s<br>
05/29/14 11.95 GiB | 157.68 GiB | 169.63 GiB | 16.47 Mbit/s<br>
05/30/14 11.62 GiB | 142.46 GiB | 154.08 GiB | 14.96 Mbit/s<br>
05/31/14 9.97 GiB | 136.38 GiB | 146.35 GiB | 14.21 Mbit/s<br>
06/01/14 11.49 GiB | 142.08 GiB | 153.56 GiB | 14.91 Mbit/s<br>
06/02/14 8.62 GiB | 99.75 GiB | 108.37 GiB | 18.07 Mbit/s<br>
------------------------+-----<u></u>--------+-------------+-------<u></u>--------<br>
estimated 14.82 GiB | 171.41 GiB | 186.22 GiB |<br>
</blockquote>
<br>
I'm trying to get vnstat print out more daily data going back to April<br>
and March, but I notice that the infra rehaul was April 29-May 2, so I'm<br>
suspecting we'd changed something during that period to add more load.<br>
In particular, I remember my shrinking disk consumption on OSUOSL<br>
mirrors by removing old releases. I wonder if this somehow resulted in<br>
the traffic increase.<br>
<br>
<br>
<br>
</blockquote>
<br>
<br>
</blockquote>
<br>
<br>
-- <br>
Kohsuke Kawaguchi | CloudBees, Inc. | <a href="http://cloudbees.com/" target="_blank">http://cloudbees.com/</a><br>
Try Jenkins Enterprise, our professional version of Jenkins<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Kohsuke Kawaguchi
</div>