[Jenkins-infra] INFRA-896

Thu Aug 18 23:35:30 UTC 2016

infra-statistics didn't pick up any *.fudged.gz because of the restrictive
regexp patterns. Another run is in progress now
<https://ci.jenkins.io/job/Infrastructure/job/statistics/107/console>

-rw-r--r-- 1 jenkins jenkins 105921928 Jun  1 17:25 201605.json.gz
-rw-rw-r-- 1 jenkins jenkins 114012823 Aug 18 23:13 201606.json.gz
-rw-rw-r-- 1 jenkins jenkins 113340387 Aug 18 23:24 201607.json.gz

At least it had produced June and July data. There's this suspiciously
large 7.6% jump from 05->06, then 0.7% loss in 06->07 which doesn't align
well with typical seasonal cycles, but we'll see. It's almost as if June
eat some of the data from July...

On Thu, Aug 18, 2016 at 12:43 PM Kohsuke Kawaguchi <kk at kohsuke.org> wrote:

> Re-anonymization is complete and /srv/usage/usage-stats is up to date. To
> summarize data that should be there,
>
> Data up to 2016/05:
>
>    - access.log.YYYYMMDD*.gz from cucumber
>
> 2016/06 & 2016/07 & 2016/08:
>
>    - access.log.YYYYMMDD*.fudged.gz: which is log files from
>    usage.jenkins.io processed from weekly (in its original form) to daily
>    - acecss_nonssl.log.YYYYMMDD*.fudged.gz: like above, but for non-SSL.
>    - cucumber.log.YYYYMMDD*.gz: logs from cucumber
>
> 2016/09 and onward:
>
>    - access.log.YYYYMMDD*.gz: logs from usage.jenkins.io. No reprocessing.
>    - acecss_nonssl.log.YYYYMMDD*.gz: like above, but for non-SSL.
>
> infra-statistics run #102
> <https://ci.jenkins.io/job/Infrastructure/job/statistics/102/console> is
> going on that regenerates June & July numbers for stats.jenkins.io. From
> long files I verified that it is correctly picking up new data sets, and
> botched files are removed.
>
>
>
> On Thu, Aug 18, 2016 at 10:03 AM Andrew Bayer <andrew.bayer at gmail.com>
> wrote:
>
>> Okiedokie.
>>
>> A.
>>
>> On Thu, Aug 18, 2016 at 10:02 AM, Kohsuke Kawaguchi <kk at kohsuke.org>
>> wrote:
>>
>>> That and 201606.json.gz, yes. So far I have only gotten to "anonymized
>>> data <https://wiki.jenkins-ci.org/display/JENKINS/Usage+Statistics>" in
>>> our taxonomy. The stats are still untouched.
>>>
>>> On Thu, Aug 18, 2016 at 10:01 AM Andrew Bayer <andrew.bayer at gmail.com>
>>> wrote:
>>>
>>>> Also note that I think we have to delete the 201607.json.gz file to get
>>>> it to rebuild.
>>>>
>>>> A.
>>>>
>>>> On Thu, Aug 18, 2016 at 9:59 AM, Kohsuke Kawaguchi <kk at kohsuke.org>
>>>> wrote:
>>>>
>>>>> Andrew was right. This saga continues.
>>>>>
>>>>> I let the reprocessing happen over night, and this morning, I noticed
>>>>> that the new data size is much smaller. I digged this further and
>>>>> discovered that logs from old usage.jenkins-ci.org (cucumber) and new
>>>>> usage.jenkins.io (which also now owns CNAME usage.jenkins-ci.org) are
>>>>> overwriting each other in interesting ways because they have the same
>>>>> 'access.log.YYYYMMDD*.gz' file names.
>>>>>
>>>>> Looking at the record, I believe usage.jenkins.io was created on July
>>>>> 2nd, and apache access logs from cucumber was moved over to new node at
>>>>> that time. The following two log files are from cucumber, and the rest of
>>>>> the log files on usage.jenkins.io is new.
>>>>>
>>>>> root at usage:/srv/usage/INFRA-896# ls -la input/
>>>>> -rw-r--r-- 1 root root  88611424 Aug 18 00:52 access.log.20160601000000.gz
>>>>> -rw-r--r-- 1 root root  88365458 Aug 18 00:52 access.log.20160602000000.1.gz
>>>>>
>>>>> Cucumber continues to receive traffic until July 19th, at which point
>>>>> I think usage.jenkins-ci.org CNAME moved over from cucumber to
>>>>> usage.jenkins.io.
>>>>>
>>>>> root at cucumber:/var/log/apache2/usage.jenkins-ci.org#-rw-r--r--  1 root root 44119899 2016-07-18 19:59 access.log.20160718000000.gz
>>>>> -rw-r--r--  1 root root 32642209 2016-07-19 19:30 access.log.20160719000000.gz
>>>>> -rw-r--r--  1 root root    65562 2016-07-20 17:57 access.log.20160720000000.gz
>>>>> -rw-r--r--  1 root root    23084 2016-07-21 17:19 access.log.20160721000000.gz
>>>>> -rw-r--r--  1 root root    15763 2016-07-22 18:17 access.log.20160722000000.gz
>>>>> -rw-r--r--  1 root root     1268 2016-07-23 10:11 access.log.20160723000000.gz
>>>>> -rw-r--r--  1 root root     3280 2016-07-24 09:04 access.log.20160724000000.gz
>>>>>
>>>>> To avoid overwriting, I'm going to take cucumber logs from 6/3 to 7/19
>>>>> and rename it to cucumber.log.*.gz and put that on the processing pipeline.
>>>>>
>>>>> On Wed, Aug 17, 2016 at 6:55 PM Kohsuke Kawaguchi <kk at kohsuke.org>
>>>>> wrote:
>>>>>
>>>>>> Fudging was done on usage.jenkins.io. I left the script and the
>>>>>> record of it here <https://github.com/jenkins-infra/INFRA-896> for
>>>>>> the sanity checking and re-processing later.
>>>>>>
>>>>>> jenkins-infra/infra-statistics:Jenkinsfile  did indeed require
>>>>>> updates. I've fixed that well. I'll re-run the pipeline against this new
>>>>>> data set and let's see what stats.jenkins.io would say.
>>>>>>
>>>>>> Andrew seemed to think there are some additional problems. So this
>>>>>> might not be the end of it.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 17, 2016 at 4:11 PM R. Tyler Croy <tyler at monkeypox.org>
>>>>>> wrote:
>>>>>>
>>>>>>> (replies inline)
>>>>>>>
>>>>>>> On Wed, 17 Aug 2016, Kohsuke Kawaguchi wrote:
>>>>>>>
>>>>>>> > Probably during the migration to usage.jenkins.io, the access log
>>>>>>> file
>>>>>>> > split has changed from daily to weekly, which broke the rest of
>>>>>>> the log
>>>>>>> > processing pipelines.
>>>>>>> >
>>>>>>> > Andrew reported this problem independently yesteday in INFRA-896
>>>>>>> > <https://issues.jenkins-ci.org/browse/INFRA-896>, so I'm fixing
>>>>>>> that here
>>>>>>> > <https://github.com/jenkins-infra/jenkins-infra/pull/558>. I'm
>>>>>>> also going
>>>>>>> > to retroactively fudge log files during the past 2 months to fix
>>>>>>> stuff back
>>>>>>> > in normal.
>>>>>>> >
>>>>>>> > Tyler suggested that we should rather fix the log processing
>>>>>>> pipeline so
>>>>>>> > that it doesn't make this assumption, but there are many and
>>>>>>> that's harder
>>>>>>> > to do.
>>>>>>> >
>>>>>>> > I'll make sure to save the originals just in case I mess up
>>>>>>> fudging.
>>>>>>>
>>>>>>>
>>>>>>> On what host are you intending to fudge the log files? The originals
>>>>>>> are on
>>>>>>> usage.jenkins.io, but they are rsynced around to a Jenkins agent
>>>>>>> under
>>>>>>> ci.jenkins.io which I *think* doesn't do an `rsync --delete` so we
>>>>>>> might end pu
>>>>>>> double-counting.
>>>>>>>
>>>>>>> Check the Jenkinsfile in jenkins-infra/infra-statistics for details
>>>>>>> on that.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> - R. Tyler Croy
>>>>>>>
>>>>>>> ------------------------------------------------------
>>>>>>>      Code: <https://github.com/rtyler>
>>>>>>>   Chatter: <https://twitter.com/agentdero>
>>>>>>>
>>>>>>>   % gpg --keyserver keys.gnupg.net --recv-key 1426C7DC3F51E16F
>>>>>>> ------------------------------------------------------
>>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Jenkins-infra mailing list
>>>>> Jenkins-infra at lists.jenkins-ci.org
>>>>> http://lists.jenkins-ci.org/mailman/listinfo/jenkins-infra
>>>>>
>>>>>
>>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.jenkins-ci.org/pipermail/jenkins-infra/attachments/20160818/5942170b/attachment-0001.html>