[Jenkins-infra] INFRA-896

Fri Aug 19 18:08:33 UTC 2016

The updated stats
<http://stats.jenkins.io/jenkins-stats/svg/total-jenkins.svg> look more
reasonable this time around, which is good, but the total installation
count still came short of what I thought it'd be when I saw the json.gz
data size.

So here is a couple of graphs
<https://docs.google.com/spreadsheets/d/1b2kQeLPCcbQISRSirvZYDqsm4U-ybOp1G6KMV2dFt50/edit#gid=0>
to contrast json.gz file size against installation count.

We know the dip in the graph early this year was caused by a bug in
Jenkins, but while the json.gz file size bounced back from that loss, the
installation count hasn't. I think we need to understand this better.

On Thu, Aug 18, 2016 at 4:35 PM Kohsuke Kawaguchi <kk at kohsuke.org> wrote:

> infra-statistics didn't pick up any *.fudged.gz because of the restrictive
> regexp patterns. Another run is in progress now
> <https://ci.jenkins.io/job/Infrastructure/job/statistics/107/console>
>
> -rw-r--r-- 1 jenkins jenkins 105921928 Jun  1 17:25 201605.json.gz
> -rw-rw-r-- 1 jenkins jenkins 114012823 Aug 18 23:13 201606.json.gz
> -rw-rw-r-- 1 jenkins jenkins 113340387 Aug 18 23:24 201607.json.gz
>
> At least it had produced June and July data. There's this suspiciously
> large 7.6% jump from 05->06, then 0.7% loss in 06->07 which doesn't align
> well with typical seasonal cycles, but we'll see. It's almost as if June
> eat some of the data from July...
>
>
>
> On Thu, Aug 18, 2016 at 12:43 PM Kohsuke Kawaguchi <kk at kohsuke.org> wrote:
>
>> Re-anonymization is complete and /srv/usage/usage-stats is up to date. To
>> summarize data that should be there,
>>
>> Data up to 2016/05:
>>
>>    - access.log.YYYYMMDD*.gz from cucumber
>>
>> 2016/06 & 2016/07 & 2016/08:
>>
>>    - access.log.YYYYMMDD*.fudged.gz: which is log files from
>>    usage.jenkins.io processed from weekly (in its original form) to daily
>>    - acecss_nonssl.log.YYYYMMDD*.fudged.gz: like above, but for non-SSL.
>>    - cucumber.log.YYYYMMDD*.gz: logs from cucumber
>>
>> 2016/09 and onward:
>>
>>    - access.log.YYYYMMDD*.gz: logs from usage.jenkins.io. No
>>    reprocessing.
>>    - acecss_nonssl.log.YYYYMMDD*.gz: like above, but for non-SSL.
>>
>> infra-statistics run #102
>> <https://ci.jenkins.io/job/Infrastructure/job/statistics/102/console> is
>> going on that regenerates June & July numbers for stats.jenkins.io. From
>> long files I verified that it is correctly picking up new data sets, and
>> botched files are removed.
>>
>>
>>
>> On Thu, Aug 18, 2016 at 10:03 AM Andrew Bayer <andrew.bayer at gmail.com>
>> wrote:
>>
>>> Okiedokie.
>>>
>>> A.
>>>
>>> On Thu, Aug 18, 2016 at 10:02 AM, Kohsuke Kawaguchi <kk at kohsuke.org>
>>> wrote:
>>>
>>>> That and 201606.json.gz, yes. So far I have only gotten to "anonymized
>>>> data <https://wiki.jenkins-ci.org/display/JENKINS/Usage+Statistics>"
>>>> in our taxonomy. The stats are still untouched.
>>>>
>>>> On Thu, Aug 18, 2016 at 10:01 AM Andrew Bayer <andrew.bayer at gmail.com>
>>>> wrote:
>>>>
>>>>> Also note that I think we have to delete the 201607.json.gz file to
>>>>> get it to rebuild.
>>>>>
>>>>> A.
>>>>>
>>>>> On Thu, Aug 18, 2016 at 9:59 AM, Kohsuke Kawaguchi <kk at kohsuke.org>
>>>>> wrote:
>>>>>
>>>>>> Andrew was right. This saga continues.
>>>>>>
>>>>>> I let the reprocessing happen over night, and this morning, I noticed
>>>>>> that the new data size is much smaller. I digged this further and
>>>>>> discovered that logs from old usage.jenkins-ci.org (cucumber) and
>>>>>> new usage.jenkins.io (which also now owns CNAME usage.jenkins-ci.org)
>>>>>> are overwriting each other in interesting ways because they have the same
>>>>>> 'access.log.YYYYMMDD*.gz' file names.
>>>>>>
>>>>>> Looking at the record, I believe usage.jenkins.io was created on
>>>>>> July 2nd, and apache access logs from cucumber was moved over to new node
>>>>>> at that time. The following two log files are from cucumber, and the rest
>>>>>> of the log files on usage.jenkins.io is new.
>>>>>>
>>>>>> root at usage:/srv/usage/INFRA-896# ls -la input/
>>>>>> -rw-r--r-- 1 root root  88611424 Aug 18 00:52 access.log.20160601000000.gz
>>>>>> -rw-r--r-- 1 root root  88365458 Aug 18 00:52 access.log.20160602000000.1.gz
>>>>>>
>>>>>> Cucumber continues to receive traffic until July 19th, at which point
>>>>>> I think usage.jenkins-ci.org CNAME moved over from cucumber to
>>>>>> usage.jenkins.io.
>>>>>>
>>>>>> root at cucumber:/var/log/apache2/usage.jenkins-ci.org#-rw-r--r--  1 root root 44119899 2016-07-18 19:59 access.log.20160718000000.gz
>>>>>> -rw-r--r--  1 root root 32642209 2016-07-19 19:30 access.log.20160719000000.gz
>>>>>> -rw-r--r--  1 root root    65562 2016-07-20 17:57 access.log.20160720000000.gz
>>>>>> -rw-r--r--  1 root root    23084 2016-07-21 17:19 access.log.20160721000000.gz
>>>>>> -rw-r--r--  1 root root    15763 2016-07-22 18:17 access.log.20160722000000.gz
>>>>>> -rw-r--r--  1 root root     1268 2016-07-23 10:11 access.log.20160723000000.gz
>>>>>> -rw-r--r--  1 root root     3280 2016-07-24 09:04 access.log.20160724000000.gz
>>>>>>
>>>>>> To avoid overwriting, I'm going to take cucumber logs from 6/3 to
>>>>>> 7/19 and rename it to cucumber.log.*.gz and put that on the processing
>>>>>> pipeline.
>>>>>>
>>>>>> On Wed, Aug 17, 2016 at 6:55 PM Kohsuke Kawaguchi <kk at kohsuke.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Fudging was done on usage.jenkins.io. I left the script and the
>>>>>>> record of it here <https://github.com/jenkins-infra/INFRA-896> for
>>>>>>> the sanity checking and re-processing later.
>>>>>>>
>>>>>>> jenkins-infra/infra-statistics:Jenkinsfile  did indeed require
>>>>>>> updates. I've fixed that well. I'll re-run the pipeline against this new
>>>>>>> data set and let's see what stats.jenkins.io would say.
>>>>>>>
>>>>>>> Andrew seemed to think there are some additional problems. So this
>>>>>>> might not be the end of it.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 17, 2016 at 4:11 PM R. Tyler Croy <tyler at monkeypox.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> (replies inline)
>>>>>>>>
>>>>>>>> On Wed, 17 Aug 2016, Kohsuke Kawaguchi wrote:
>>>>>>>>
>>>>>>>> > Probably during the migration to usage.jenkins.io, the access
>>>>>>>> log file
>>>>>>>> > split has changed from daily to weekly, which broke the rest of
>>>>>>>> the log
>>>>>>>> > processing pipelines.
>>>>>>>> >
>>>>>>>> > Andrew reported this problem independently yesteday in INFRA-896
>>>>>>>> > <https://issues.jenkins-ci.org/browse/INFRA-896>, so I'm fixing
>>>>>>>> that here
>>>>>>>> > <https://github.com/jenkins-infra/jenkins-infra/pull/558>. I'm
>>>>>>>> also going
>>>>>>>> > to retroactively fudge log files during the past 2 months to fix
>>>>>>>> stuff back
>>>>>>>> > in normal.
>>>>>>>> >
>>>>>>>> > Tyler suggested that we should rather fix the log processing
>>>>>>>> pipeline so
>>>>>>>> > that it doesn't make this assumption, but there are many and
>>>>>>>> that's harder
>>>>>>>> > to do.
>>>>>>>> >
>>>>>>>> > I'll make sure to save the originals just in case I mess up
>>>>>>>> fudging.
>>>>>>>>
>>>>>>>>
>>>>>>>> On what host are you intending to fudge the log files? The
>>>>>>>> originals are on
>>>>>>>> usage.jenkins.io, but they are rsynced around to a Jenkins agent
>>>>>>>> under
>>>>>>>> ci.jenkins.io which I *think* doesn't do an `rsync --delete` so we
>>>>>>>> might end pu
>>>>>>>> double-counting.
>>>>>>>>
>>>>>>>> Check the Jenkinsfile in jenkins-infra/infra-statistics for details
>>>>>>>> on that.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> - R. Tyler Croy
>>>>>>>>
>>>>>>>> ------------------------------------------------------
>>>>>>>>      Code: <https://github.com/rtyler>
>>>>>>>>   Chatter: <https://twitter.com/agentdero>
>>>>>>>>
>>>>>>>>   % gpg --keyserver keys.gnupg.net --recv-key 1426C7DC3F51E16F
>>>>>>>> ------------------------------------------------------
>>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Jenkins-infra mailing list
>>>>>> Jenkins-infra at lists.jenkins-ci.org
>>>>>> http://lists.jenkins-ci.org/mailman/listinfo/jenkins-infra
>>>>>>
>>>>>>
>>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.jenkins-ci.org/pipermail/jenkins-infra/attachments/20160819/b18dc0a6/attachment.html>