[Jenkins-infra] Procedure for upgrading our Kubernetes cluster (Azure Container Service)

[Vernin Olivier]

Even if I had preferred the upgrade solution, it's easier and more
reliable to deploy a new cluster as upgrading is still a manual
procedure that changes depending of the version.

If we deploy a new cluster (almost is already automated), the only
downtime will be while the DNS record 'nginx.azure.jenkins.io' is
updated to the new cluster IP

Redeployment Steps:
    1. Deploy a second k8s cluster.
    2. Configure the new cluster.
    3. Validate new k8s cluster
    4. Update nginx.azure.jenkins.io
    5. Remove old cluster


On 06/15/2017 05:19 PM, R. Tyler Croy wrote:
> (replies inline)
>
> On Thu, 15 Jun 2017, Olblak wrote:
>
>> My point here is that we are running one version behind what ACS deploy
>> now.
>> -> https://github.com/Azure/ACS/blob/master/kubernetes-status.md
>>
>> It's not yet urgent but we have to think about "how to upgrade the
>> cluster".
>> And meanwhile, we all know what happen when we delay to many upgrades.
>>
>> Apparently upgrades are not yet supported by Azure which means that we
>> gonna have to do it manually and by 'ourself'.
>>
>> So Right now we have two possibilities:
>>   - We upgrade the cluster 'now'
>>   - We delay a little bit and we hope that the procedure will be
>>   simplified by Azure
>
> Regardless of the timing, we still *must* upgrade the Kubernetes cluster at
> some point. My goal for this thread was more to discuss *how* we would perform
> the upgrade, and the impact it would have.
>
> The way I see it, there are two paths we can follow, please correct me if there
> are more:
>
> 1. Deploy a new cluster, migrate all existing workloads over
> 2. Manually upgrade the existing cluster
>
>
> The details of the upgrade path is really what I would like us to discuss here.
>
> - R. Tyler Croy
>
> ------------------------------------------------------
>      Code: <https://github.com/rtyler>
>   Chatter: <https://twitter.com/agentdero>
>      xmpp: rtyler at jabber.org
>
>   % gpg --keyserver keys.gnupg.net --recv-key 1426C7DC3F51E16F
> ------------------------------------------------------