[Jenkins-infra] (fwd) RE: Your Azure Account - Terms of Use Reminder - Jenkins

Tue Aug 7 16:53:29 UTC 2018

Could be overkill, but what about anomaly detection in the logs? Some
simple machine learning can help detect security breaches.

On Mon, Aug 6, 2018 at 3:40 PM Olblak <me at olblak.com> wrote:

> Hello,
> A short update.
>
> Currently we don't have enough information to take any further decision.
> Tyler is still waiting for news from Azure. The main problem that we have,
> is all the activities on ci.jenkins.io required a lot of resources so we
> can't tell at the moment if the load on the infra was legitimated or not.
> We agreed to keep this discussion public as long as we don't provide
> sensible information.
>
> We discussed about different approach that could explain the current
> situation or help in the futur to detect/prevent such cases. Feel free to
> add any ideas that you may have.
>
> * We found nothing obvious from jenkins logs, datadog, or azure activities
> * We could review all projects from jenkinsci that do not use 'buildPlugin'
> * We could apply a quarantine to agents inside the Azure network but it
> sounds hard as those agents must be able to reach different services
> outside Azure like github, maven repo, etc.
> * We could keep agents longer (stopped), for example one week to simplify
> investigation.
> * Was it possible that the windows machine was corrupted during it start
> either by downloading a corrupted maven artifact, running an old
> application version, or targeting a corrupted web service?
> * Should we set an alarm on cpu/memory usage, it sounds hard to
> differentiate with legitimated load.
>
> If you have any idea, suggestion, feel free to add your input.
>
> Cheers
>
>
> ---
> -> gpg --keyserver keys.gnupg.net --recv-key 52210D3D
> ---
>
>
>
>
> On Mon, Aug 6, 2018, at 2:49 PM, Arnaud Héritier wrote:
>
> Couldn't we ask to M$ to have a more detailled timeline of when they are
> discovering this issue (last activity occurred 8/1/2018 ...) and then we
> look at the jenkins build history to find if there is a project building at
> that times ?
> For example this job (and branches) ran on August 1st :
> https://ci.jenkins.io/job/Plugins/job/azure-app-service-plugin/
> And it is launching a windows agent on Azure labelled win2012-ABCDEF
>
> https://ci.jenkins.io/job/Plugins/job/azure-vm-agents-plugin/job/master/107/consoleFull
>
> On Mon, Aug 6, 2018 at 1:42 PM Oleg Nenashev <o.v.nenashev at gmail.com>
> wrote:
>
> Sorry, 9:30PM UTC is too late in my TZ, I will unlikely be able to join.
> Could you please send out summary notes after the meeting?
>
> Thanks in advance,
>
> On Mon, Aug 6, 2018 at 1:19 PM, Olblak <me at olblak.com> wrote:
>
>
> The main problem here is that kind of machine have short live and I don't
> think we can know which project was build on that particular machine. For
> auditing purpose, it would be nice to have a log file with such information.
>
>
> > * there's some repo(s) which are invoking Maven and using it to run
>
> Or for instance gradlew https://git.io/fN1TZ
> Unfortunately they are different ways to run arbitrary scripts on those
> agents and I don't think it will be possible to avoid that.
>
> A solution that come to my mind would be to isolate those machine as much
> as possible to forbid outgoing/ingoing connections as much as possible.
>
> We'll probably discuss about this during the infra meeting today so feel
> free to join  https://jenkins.io/event-calendar/
>
>
>
> ---
> -> gpg --keyserver keys.gnupg.net --recv-key 52210D3D
> ---
>
>
>
>
>
> On Mon, Aug 6, 2018, at 10:44 AM, Oleg Nenashev wrote:
>
> Likely, it looks like an urgent issue so we should get all available folks
> working on it. Added the security team to CC.
>
> I have tried to investigate the issue by looking at build histories, etc.
> I was not really successful, unfortunately I cannot help much on the
> server side with my current permission level.
>
> It would help if Azure support provides more info about type of the
> cryptominer running there and exact timestamps. It would allow narrowing
> down the scope and going through plugins to find malicious dependencies.
>
> BR, Oleg
>
>
>
>
> On Mon, Aug 6, 2018 at 10:27 AM, Olblak <me at olblak.com> wrote:
>
> Shouldn't we move this discussion to security?
>
> ---
> -> gpg --keyserver keys.gnupg.net --recv-key 52210D3D
> ---
>
>
> On Mon, Aug 6, 2018, at 12:20 AM, R. Tyler Croy wrote:
> > (replies inline)
> >
> > On Sun, 05 Aug 2018, Oleg Nenashev wrote:
> >
> > > Is any help needed with this case?
> > > Or are you handling it together with Olivier?
> >
> >
> > Olivier and I haven't had a chance to really discuss it, so any thoughts
> you
> > might have would be valuable.
> >
> >
> > My current theories are:
> >
> > * there's some repo(s) which are invoking Maven and using it to run
> > miners perhaps.
> > * The dynamic agents are coming online and somehow being infected rather
> >   quickly, somehow, with nonsense.
> >
> >
> >
> > _______________________________________________
> > Jenkins-infra mailing list
> > Jenkins-infra at lists.jenkins-ci.org
> > http://lists.jenkins-ci.org/mailman/listinfo/jenkins-infra
> > Email had 1 attachment:
> > + signature.asc
> >   1k (application/pgp-signature)
>
> _______________________________________________
> Jenkins-infra mailing list
> Jenkins-infra at lists.jenkins-ci.org
> http://lists.jenkins-ci.org/mailman/listinfo/jenkins-infra
>
>
>
> _______________________________________________
> Jenkins-infra mailing list
> Jenkins-infra at lists.jenkins-ci.org
> http://lists.jenkins-ci.org/mailman/listinfo/jenkins-infra
>
>
>
> --
> -----
> Arnaud Héritier
> http://aheritier.net
> Mail/GTalk: aheritier AT gmail DOT com
> Twitter/Skype : aheritier
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Jenkins CERT" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jenkinsci-cert+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Matt Sicker
Software Engineer, CloudBees
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.jenkins-ci.org/pipermail/jenkins-infra/attachments/20180807/8836171d/attachment-0001.html>