[Jenkins-infra] Jenkins.io migration on Azure

Ben Walding bwalding at cloudbees.com
Wed Jul 5 23:38:15 UTC 2017


Olivier,

I have had a look through the IEP and here some of my comments

*URL*
jenkins.io vs. www.jenkins.io - not a big deal (as jenkins.io doesn't have
cookies at present) - but having your website on the no-www means cookies
on the apex will propagate across all sub-domains.

*DNS*
Not mentioned - but since you can't have an apex (jenkins.io) pointing at a
CNAME, then covering off how that is handled by Azure would be useful for
non-Azure people. You may need to migrate DNS to handle this "the Azure
way" - an additional consideration.

*Container Architecture / Content Location*
Baking the content inside a docker container raises many pain points, so
keeping content on AFS/blob store makes sense.

*Storage*
I had a quick look at IEP-006 - it didn't mention Azure File Storage (AFS)
being the storage system being considered.

My only concern would be that AFS may not be able handle the high I/O
(small files, lots of them) effectively - compared with local SSD storage.
We had issues when testing the Amazon equivalent (EFS) - where the
characteristics of EFS need to be well understood to make it perform
properly.
*However - given jenkins.io <http://jenkins.io> is static - an aggressive
CDN will eliminate nearly all IO at the web-servers except after cache
invalidation.*

Given that jenkins.io is on a 30 minute refresh cycle, some of the
advantages of AFS would be that it means you don't need to redeploy the
content.

*CDN*
I would strongly recommend using a CDN - it's the perfect use-case.  The
only issue is cache invalidation (one of the 3 hardest things in computer
science) - if a CDN is used then I think updating the IEP with the
high-level CDN approach would be useful.

*Success Criteria*
I think having some success criteria would be useful - "Updates are
published and cache invalidated with 30 minutes", "Response time of X",
"Cluster handles failures and recovery without operator intervention" etc.

*Blob Storage + CDN vs Container + "File Store"*
(you need to keep existing URL structure - changing URLs for this project
is a non-starter)

Biggest loss with blob store + CDN is that your development cycle is quite
different to container + file-store. We've found this change in development
cycle makes rapid development a lot slower / or you end up with a container
version anyway (and it's subtly different - which introduces deployment
'weirdness')

The other advantage of "container on k8s architecture" is that it then
looks like every other application - big win for maintenance /
documentation at Ops level.

*HTTP => HTTPS*

I would suggest that HSTS headers are added to the HTTPS site. Initially
with a short expiry.

*Load Balancer*

It's unclear from IEP-004 / IEP-006 how a request is routed in from the
internet (or CDN) to a particular container.

*Load Testing*

Load test the solution to determine how long it takes to "update" after
cache invalidation and that the server can respond to quickly under heavy
load during cache invalidation.

Get some real-world stats from jenkins.io - especially around events like
Jenkins World etc.

*Development*
Ensure that there is a streamlined development environment for rapid
prototyping of content for non-k8s savvy users (e.g. people who just want
to do "docker-compose up" and start editing pages)


*Overall Comment*
I think your proposal is sound - my preference (if implementing /
supporting) would be for a solution like:

Aggressive CDN => Azure load balancer => cluster load balancer[3] =>
cluster of k8s servers[2] => Docker containers[1] => Azure File Storage
backend

[1] Docker containers would be relatively static and just serve up content
[2] I only quickly read through IEP-004 (k8s) so I'm not sure on the
details of what you're doing here (for web load balancing etc) - but we
have iterated through this at CloudBees on the AWS ECS platform (much the
same).
[3] I'm a bit hazy on how the cluster level load balancing works vs. Azure
load balancer - we went through many iterations of this on AWS ECS to get a
robust routing solution (note: AWS ALBs cat do a lot more heavy lifting
(routing) now than when we started out)



Cheers,

Ben








On 5 July 2017 at 23:05, Olblak <me at olblak.com> wrote:

> Hi,
>
> I am planning to migrate jenkins.io on Azure.
> I wrote an IEP <https://github.com/jenkins-infra/iep/pull/10> document
> about it and I would be interested by any feedback or concerns
>
>  To make a long story short, I plan to mount Azure File Storage into
> several docker containers and deploy them on the k8s cluster.
>
>
>
>
> _______________________________________________
> Jenkins-infra mailing list
> Jenkins-infra at lists.jenkins-ci.org
> http://lists.jenkins-ci.org/mailman/listinfo/jenkins-infra
>
>


-- 
Ben Walding
bwalding at cloudbees.com
Operations Manager

*CloudBees, Inc.*
www.cloudbees.com
<https://www.cloudbees.com/>
*software at the speed of ideas*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.jenkins-ci.org/pipermail/jenkins-infra/attachments/20170706/f73ef0bd/attachment.html>


More information about the Jenkins-infra mailing list