[Jenkins-infra] [#73645] One more, hopefully the last, request

[Daniel Beck]

> On 16. Dec 2017, at 01:46, support at jfrog.com wrote:
> 
>   In your request log we saw direst the api was called without specifying any repos. Note that indexing is resource intensive. Calculating and indexing for a repository may be a resource-intensive operation, especially for a large local repository or if the repository is a virtual one containing other underlying repositories.
> 
>   Therefore, we recommend that you do not include repositories that do not require indexing for a periodic index calculation. Since you poke the indexing API every 15 minutes against all your repo. If your repo has increased in size to a point where each indexing requests took more than 15 mins, you may see the 500 error. Can you refine your choice of repo in your scheduled api call?

Hi Paul,

I hope you don't mind me chiming in. I'm doing much of the work with Artifactory in the Jenkins project, and discovered this problem last week.

I don't see the unrestricted indexing requests you're referring to. There are requests logged like this one:

> 20171216011801|285|REQUEST|(IP address)|(user name)|POST|/api/maven|HTTP/1.1|200|0

The request is done by this command, explicitly specifying a single repository:

> $ curl --fail -X POST -H 'Content-Length: 0' -u $USERNAME:$PASSWORD "https://repo.jenkins-ci.org/api/maven?repos=releases&force=1"

The Artifactory log confirms that it only indexes one actual local repo:

> 2017-12-16 01:18:01,321 [http-nio-8083-exec-248] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:115) - Activating indexer for repo '[releases]' manually
> 2017-12-16 01:18:01,326 [art-exec-108213] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:136) - Starting Maven indexing
> 2017-12-16 01:18:01,326 [art-exec-108213] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:157) - Starting non virtual repositories indexing...
> 2017-12-16 01:18:01,327 [art-exec-108213] [INFO ] (o.a.m.i.MavenIndexerServiceImpl:159) - Non virtual repositories to index: [releases]

Are you seeing other requests than these?

Notably, this restriction to a single local repo is a recent optimization attempt (from Wednesday/Thursday) that I implemented in the hope of speeding up indexing again. We used to request indexing on multiple explicitly specified repositories before Wednesday, and had done the same for at least the several months when indexing was finished much quicker that it does now. The indexing duration went from ~15 minutes to 1.5 hours last week without any changes on our side that we're aware of.

It would be great if you could take another look what's going on here. As KK previously wrote, this impacts our ability to deliver security fixes in a timely manner.

Thanks!
Daniel