[Jenkins-infra] Cucumber bandwidth situation

[Kohsuke Kawaguchi]

Tyler got a surprisingly bill for the overage charge for cucumber, which 
runs "jenkins-ci.org", "mirrors.jenkins-ci.org" and a number of other 
virtual hosts.

I think I've set up vnstat when it happened the last time to track 
utlization. Here is the vnstat output.

> kohsuke at cucumber:~$ vnstat -m
>
>  eth0  /  monthly
>
>        month        rx      |     tx      |    total    |   avg. rate
>     ------------------------+-------------+-------------+---------------
>       Jul '13    338.35 GiB |    3.21 TiB |    3.54 TiB |   11.36 Mbit/s
>       Aug '13    338.36 GiB |    2.86 TiB |    3.19 TiB |   10.23 Mbit/s
>       Sep '13    354.39 GiB |    3.52 TiB |    3.87 TiB |   12.82 Mbit/s
>       Oct '13    395.09 GiB |    4.20 TiB |    4.59 TiB |   14.72 Mbit/s
>       Nov '13    449.73 GiB |    3.51 TiB |    3.94 TiB |   13.07 Mbit/s
>       Dec '13    562.26 GiB |    3.68 TiB |    4.23 TiB |   13.56 Mbit/s
>       Jan '14    672.19 GiB |    3.91 TiB |    4.56 TiB |   14.64 Mbit/s
>       Feb '14    370.69 GiB |    3.13 TiB |    3.49 TiB |   12.39 Mbit/s
>       Mar '14    351.83 GiB |    3.33 TiB |    3.67 TiB |   11.77 Mbit/s
>       Apr '14    362.76 GiB |    3.39 TiB |    3.74 TiB |   12.40 Mbit/s
>       May '14    401.56 GiB |    4.80 TiB |    5.19 TiB |   16.65 Mbit/s
>       Jun '14     20.11 GiB |  241.83 GiB |  261.94 GiB |   16.07 Mbit/s
>     ------------------------+-------------+-------------+---------------
>     estimated    381.25 GiB |    4.48 TiB |    4.85 TiB |

As you see, the outbound traffic jumped in May.
Here's the daily output, and I think it means that new trend is 
continuing in June so far as I can tell.

In other words, we need to act on it ASAP to avoid another overage for June.


> kohsuke at cucumber:~$ vnstat -d
>
>  eth0  /  daily
>
>          day         rx      |     tx      |    total    |   avg. rate
>      ------------------------+-------------+-------------+---------------
>       05/04/14     11.61 GiB |  151.50 GiB |  163.11 GiB |   15.84 Mbit/s
>       05/05/14     18.37 GiB |  167.82 GiB |  186.18 GiB |   18.08 Mbit/s
>       05/06/14     12.54 GiB |  176.53 GiB |  189.07 GiB |   18.36 Mbit/s
>       05/07/14     13.07 GiB |  169.65 GiB |  182.73 GiB |   17.74 Mbit/s
>       05/08/14     12.49 GiB |  152.46 GiB |  164.95 GiB |   16.01 Mbit/s
>       05/09/14     13.54 GiB |  167.91 GiB |  181.45 GiB |   17.62 Mbit/s
>       05/10/14     10.72 GiB |  149.50 GiB |  160.22 GiB |   15.56 Mbit/s
>       05/11/14     15.25 GiB |  141.75 GiB |  157.01 GiB |   15.24 Mbit/s
>       05/12/14     16.06 GiB |  168.05 GiB |  184.11 GiB |   17.88 Mbit/s
>       05/13/14     13.25 GiB |  144.43 GiB |  157.68 GiB |   15.31 Mbit/s
>       05/14/14     18.24 GiB |  160.37 GiB |  178.61 GiB |   17.34 Mbit/s
>       05/15/14     14.30 GiB |  154.11 GiB |  168.41 GiB |   16.35 Mbit/s
>       05/16/14     12.99 GiB |  153.21 GiB |  166.20 GiB |   16.14 Mbit/s
>       05/17/14      9.87 GiB |  127.64 GiB |  137.51 GiB |   13.35 Mbit/s
>       05/18/14     11.43 GiB |  186.48 GiB |  197.91 GiB |   19.22 Mbit/s
>       05/19/14     14.08 GiB |  171.26 GiB |  185.35 GiB |   18.00 Mbit/s
>       05/20/14     13.47 GiB |  149.67 GiB |  163.14 GiB |   15.84 Mbit/s
>       05/21/14     12.67 GiB |  150.21 GiB |  162.89 GiB |   15.81 Mbit/s
>       05/22/14     13.43 GiB |  168.17 GiB |  181.61 GiB |   17.63 Mbit/s
>       05/23/14     14.06 GiB |  165.47 GiB |  179.53 GiB |   17.43 Mbit/s
>       05/24/14      9.75 GiB |  132.61 GiB |  142.36 GiB |   13.82 Mbit/s
>       05/25/14     10.10 GiB |  131.50 GiB |  141.60 GiB |   13.75 Mbit/s
>       05/26/14     13.81 GiB |  180.48 GiB |  194.28 GiB |   18.86 Mbit/s
>       05/27/14     14.51 GiB |  187.94 GiB |  202.45 GiB |   19.66 Mbit/s
>       05/28/14     12.76 GiB |  163.84 GiB |  176.60 GiB |   17.15 Mbit/s
>       05/29/14     11.95 GiB |  157.68 GiB |  169.63 GiB |   16.47 Mbit/s
>       05/30/14     11.62 GiB |  142.46 GiB |  154.08 GiB |   14.96 Mbit/s
>       05/31/14      9.97 GiB |  136.38 GiB |  146.35 GiB |   14.21 Mbit/s
>       06/01/14     11.49 GiB |  142.08 GiB |  153.56 GiB |   14.91 Mbit/s
>       06/02/14      8.62 GiB |   99.75 GiB |  108.37 GiB |   18.07 Mbit/s
>      ------------------------+-------------+-------------+---------------
>      estimated     14.82 GiB |  171.41 GiB |  186.22 GiB |

I'm trying to get vnstat print out more daily data going back to April 
and March, but I notice that the infra rehaul was April 29-May 2, so I'm 
suspecting we'd changed something during that period to add more load. 
In particular, I remember my shrinking disk consumption on OSUOSL 
mirrors by removing old releases. I wonder if this somehow resulted in 
the traffic increase.



-- 
Kohsuke Kawaguchi                          http://kohsuke.org/