[Jenkins-infra] (fwd) RE: Your Azure Account - Terms of Use Reminder - Jenkins

Mon Aug 6 20:40:34 UTC 2018

Hello,
A short update.

Currently we don't have enough information to take any further decision.Tyler is still waiting for news from Azure. The main problem that we
have, is all the activities on ci.jenkins.io required a lot of
resources so we can't tell at the moment if the load on the infra was
legitimated or not.We agreed to keep this discussion public as long as we don't provide
sensible information.
We discussed about different approach that could explain the current
situation or help in the futur to detect/prevent such cases. Feel free
to add any ideas that you may have.
* We found nothing obvious from jenkins logs, datadog, or azure
  activities* We could review all projects from jenkinsci that do not use
  'buildPlugin'* We could apply a quarantine to agents inside the Azure network but it
  sounds hard as those agents must be able to reach different services
  outside Azure like github, maven repo, etc.* We could keep agents longer (stopped), for example one week to
  simplify investigation.* Was it possible that the windows machine was corrupted during it start
  either by downloading a corrupted maven artifact, running an old
  application version, or targeting a corrupted web service?* Should we set an alarm on cpu/memory usage, it sounds hard to
  differentiate with legitimated load.
If you have any idea, suggestion, feel free to add your input.

Cheers

---
-> gpg --keyserver keys.gnupg.net --recv-key 52210D3D
---

On Mon, Aug 6, 2018, at 2:49 PM, Arnaud Héritier wrote:
> Couldn't we ask to M$ to have a more detailled timeline of when they
> are discovering this issue (last activity occurred 8/1/2018 ...) and
> then we look at the jenkins build history to find if there is a
> project building at that times ?> For example this job (and branches) ran on August 1st :
> https://ci.jenkins.io/job/Plugins/job/azure-app-service-plugin/> And it is launching a windows agent on Azure labelled win2012-ABCDEF
> https://ci.jenkins.io/job/Plugins/job/azure-vm-agents-plugin/job/master/107/consoleFull> 
> On Mon, Aug 6, 2018 at 1:42 PM Oleg Nenashev
> <o.v.nenashev at gmail.com> wrote:>> Sorry, 9:30PM UTC is too late in my TZ, I will unlikely be able
>> to join.>> Could you please send out summary notes after the meeting?
>> 
>> Thanks in advance,
>> 
>> On Mon, Aug 6, 2018 at 1:19 PM, Olblak <me at olblak.com> wrote:
>>> __
>>> The main problem here is that kind of machine have short live and I
>>> don't think we can know which project was build on that particular
>>> machine. For auditing purpose, it would be nice to have a log file
>>> with such information.>>> 
>>> 
>>> > * there's some repo(s) which are invoking Maven and using it to
>>> >   run>>> 
>>> Or for instance gradlew https://git.io/fN1TZ
>>> Unfortunately they are different ways to run arbitrary scripts on
>>> those agents and I don't think it will be possible to avoid that.>>> 
>>> A solution that come to my mind would be to isolate those machine as
>>> much as possible to forbid outgoing/ingoing connections as much as
>>> possible.>>> 
>>> We'll probably discuss about this during the infra meeting today so
>>> feel free to join  https://jenkins.io/event-calendar/>>> 
>>> 
>>> 
>>> ---
>>> -> gpg --keyserver keys.gnupg.net --recv-key 52210D3D
>>> ---
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Mon, Aug 6, 2018, at 10:44 AM, Oleg Nenashev wrote:
>>>> Likely, it looks like an urgent issue so we should get all
>>>> available folks working on it. Added the security team to CC.>>>> 
>>>> I have tried to investigate the issue by looking at build
>>>> histories, etc.>>>> I was not really successful, unfortunately I cannot help much on
>>>> the server side with my current permission level.>>>> 
>>>> It would help if Azure support provides more info about type of the
>>>> cryptominer running there and exact timestamps. It would allow
>>>> narrowing down the scope and going through plugins to find
>>>> malicious dependencies.>>>> 
>>>> BR, Oleg
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Mon, Aug 6, 2018 at 10:27 AM, Olblak <me at olblak.com> wrote:
>>>>> Shouldn't we move this discussion to security?
>>>>> 
>>>>> ---
>>>>> -> gpg --keyserver keys.gnupg.net --recv-key 52210D3D
>>>>> ---
>>>>> 
>>>>> 
>>>>> On Mon, Aug 6, 2018, at 12:20 AM, R. Tyler Croy wrote:
>>>>> > (replies inline)
>>>>> > 
>>>>> > On Sun, 05 Aug 2018, Oleg Nenashev wrote:
>>>>> > 
>>>>> > > Is any help needed with this case?
>>>>> > > Or are you handling it together with Olivier?
>>>>> > 
>>>>> > 
>>>>> > Olivier and I haven't had a chance to really discuss it, so any
>>>>> > thoughts you>>>>> > might have would be valuable.
>>>>> > 
>>>>> > 
>>>>> > My current theories are:
>>>>> > 
>>>>> > * there's some repo(s) which are invoking Maven and using it to
>>>>> >   run>>>>> > miners perhaps.
>>>>> > * The dynamic agents are coming online and somehow being
>>>>> >   infected rather>>>>> >   quickly, somehow, with nonsense.
>>>>> > 
>>>>> > 
>>>>> > 
>>>>> > _______________________________________________
>>>>>  > Jenkins-infra mailing list Jenkins-infra at lists.jenkins-ci.org
>>>>>  > http://lists.jenkins-ci.org/mailman/listinfo/jenkins-infra
>>>>> > Email had 1 attachment:>>>>> > + signature.asc
>>>>> >   1k (application/pgp-signature)
>>>>> 
>>>>> _______________________________________________
>>>>> Jenkins-infra mailing list
>>>>> Jenkins-infra at lists.jenkins-ci.org
>>>>> http://lists.jenkins-ci.org/mailman/listinfo/jenkins-infra
>>> 
>> 
>> _______________________________________________
>>  Jenkins-infra mailing list
>> Jenkins-infra at lists.jenkins-ci.org
>> http://lists.jenkins-ci.org/mailman/listinfo/jenkins-infra
> 
> 
> -- 
> -----
> Arnaud Héritier
> http://aheritier.net
> Mail/GTalk: aheritier AT gmail DOT com
> Twitter/Skype : aheritier

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.jenkins-ci.org/pipermail/jenkins-infra/attachments/20180806/50bfb2a7/attachment-0001.html>