[Jenkins-infra] [#73645] One more, hopefully the last, request

[Daniel Beck]

Hi Loren,

Thanks for your response! Please see my answers below.


> On 21. Dec 2017, at 07:35, support at jfrog.com wrote:
> 
> As we noticed that the "releases" local repository is in the list of a virtual repository, it's reindex will trigger the reindex/merge of the virtual repository as well. We checked with our DevOps team and they can confirm that we did not make any changes on our end, so it is likely that this is a configuration change.

Note that roughly during the time when the increase of indexing time happened (between Dec 11 14:00 UTC, when we saw it was still around 15 minutes, and Dec 13 22:00 UTC, when we noticed the increase to 1.5 hrs), you resolved ticket #73540. Might be a coincidence, but we cannot know from our end. #72947 was also going on during that time, but your responses indicate it was acted on only later.


> Please let us know if your team made any major changes to your instance, for example has this repo been included in more virtuals than before, and if having the repository not be in a virtual resolves the timing.

We've basically always (several years at least) had the 'public' virtual repo that includes both 'snapshots' and 'releases'. That's the only virtual repo we have 'releases' in. Due to how our POMs are structured, we are unable to remove 'releases' from 'public' as an experiment, as that would break hundreds of sub-projects within the Jenkins project relying on this structure:
https://github.com/jenkinsci/plugin-pom/blob/03b71039fd98fa63769613109ca9a1d1e3b1b7b3/pom.xml#L758...L782

Regarding other recent changes, other than regularly upload artifacts, we're just doing the following, and neither seems relevant here:
- We create private local repos for every security update we publish, and stage artifacts there using Maven. On release day (and December 11 was one of those), we just copy over the contents of these repos (entire repo to prevent unix copying semantics from messing things up), and, because this overrides maven-metadata.xml files, update the maven metadata of specific affected groupIds via `/api/maven/calculateMetadata`. Each of those usually gets a permission target that includes one group, and perhaps a handful of users.
- We are (and have been for more than a year) programmatically defining and updating most of the permissions on Artifactory at a fine-grained (per-artifact) level. You can see this in hundreds of permission targets with the 'generated' name prefix.

None of these are entirely new either. We have added 15 staging repos in the last six months or so, with only a handful of files in each. We add perhaps 10-20 permission targets per month, but we've been at easily more than 1000 permission targets on this Artifactory for months without problems.

I am unaware of further notable changes, or actual configuration changes, early last week. I also wouldn't know what sort of changes we could have done, Artifactory just works, so all we do is the above.


> We also noted a number of errors pertaining to “Error occurred while processing the filtered resource 'releases:com/amcbridge/build-configurator/1.0.6.0/build-configurator-1.0.6.0.hpi': Token manager error: freemarker.core.TokenMgrError: Lexical error at line 24, column 124.  Encountered: \"\\f\" (12), after : \"\" in template“
> 
> We believe this is because the file has the artifact filter property checked, but it does not appear to be a text file of sorts, hence it throws the error. You should be able to remove these errors by unchecking the box.

I removed the flag from the file. I think it's likely that it was set in error as the file was uploaded, in April.


> We also see alot of requests for SNAPSHOTs to a release repository, please let us know if this is intended, as the snapshot precedes the release repository in the virtual repository list.

Are you referring to log messages that look like the following?

> Sending HTTP error code 409: The repository 'releases' rejected the resolution of an artifact


Is this something we can control? The Artifactory is public, and we cannot control how everyone uses it. For example, roughly half of those requests are with the same 'com/taboola' groupId prefix, and we host none of those artifacts as far as I can tell -- only our repo1-cache has some. So someone may have set up their Maven POM or similar to query our Artifactory for their own stuff, and this is the result…?


> We have also added in a debug logger for your instance to see further logging for the indexer, please let us know the full timestamp when you attempt to run the indexer again so that we can see the logging for it. 

We send indexing requests via `curl` every 15 minutes at minute 3/18/33/48. Since this problem started, some of those started to get rejected, but with short windows of up to 15 minutes, indexing is always running. The recent one at 8:48 AM UTC did not get rejected:

> 20171221084801|289|REQUEST|(IP)|(username)|POST|/api/maven|HTTP/1.1|200|0


Thanks again for looking into this.


Daniel