[Jenkins-infra] Jenkins infrastructure vpn

Olblak me at olblak.com
Mon Mar 18 08:31:14 UTC 2019 

Hello,

Some news about the jenkins vpn.

As suggested in this thread:
* I merged openvpn docker repository and certificate management repository into one single repository in order to easily build a new docker image each time a configuration change.
-> jenkins-infra/openvpn <https://github.com/jenkins-infra/openvpn>
* I defined one single vpn network with different permission depending on your source IP
* I wrote a basic golang tool to generate certificate request, signing certificate and assign a fixed IP in the correct network.

Depending on the network you are supposed to be, default or admin, you don't have access to the same things endpoints.

Admin network member can reach "public_dmz", "public_data", "public_app" subnetwork.
Default network member can only access network "public-app" on ports 80/443
This file <https://github.com/jenkins-infra/openvpn/blob/master/config.yaml> is used to generate client config and config files are placed in the ccd <https://github.com/jenkins-infra/openvpn/tree/master/cert/ccd> directory .
The big picture is the vpn network is configured with CIDR 10.8.0.0/23 and IPs from 10.8.0.0/24 are assign to default and 10.8.1.1/24 to admin.
The vpn machine apply different firewall rules depending on the source IP as defined here <https://github.com/jenkins-infra/jenkins-infra/blob/045e7f4e89628ebeae011e8a12d209437b498aae/dist/profile/manifests/openvpn.pp#L104-L64>

I am now looking for reviewers and beta testers. :-)

Everything is explained here <https://github.com/jenkins-infra/openvpn#howto-get-client-access>, I hope it's clear enough
While I am not planning to work on this service at the moment, unless for big concerns.
I think it would be important for a next iteration to automate signing request and regularly test data consistencies


---
-> gpg --keyserver keys.gnupg.net --recv-key 52210D3D
---




On Thu, Feb 21, 2019, at 3:03 PM, Baptiste Mathus wrote:
> Oh yeah, my bad, I thought you were talking about some source IP filtering, like we can for instance do to restrict access to AWS EC2 instances, etc. Ignore me then :). Thanks!
> 
> Le jeu. 21 févr. 2019 à 09:02, Adam Papai <apapai at cloudbees.com> a écrit :
>> Hey Baptiste,
>> 
>> On VPN you can define a static IP pool for the users (internal IP address). So you don't need a 3 VPN servers, but need 1 with static IP mapping + iptables. Each user will have a known static internal IP, so you can completely restrict who can access the servers via iptables. 
>> 
>> 
>> 2019年2月21日(木) 16:53 Baptiste Mathus <ml at batmat.net>:
>>> 
>>> 
>>> Le jeu. 21 févr. 2019 à 02:52, Adam Papai <apapai at cloudbees.com> a écrit :
>>>> Hey,
>>>> 
>>>> What comes into my mind:
>>>> 
>>>> 1. Use 1 VPN with a static user/IP mapping. So you could use iptables to let people access specific servers based on IP mapping.
>>> 
>>> IIUC what you mean, wouldn't this be possibly impractical for people who may connect from various locations, and hence change IPs regularly if not often?
>>> Just to illustrate, I regularly work from home connection, co-working space, phone tethering (even from home when my ISP goes AWOL) and I suspect I get dynamic IPs in that case.
>>> 
>>> Thanks
>>> 
>>> 
>>>> 2. Use Signed SSH certs to let users / groups log in to a specific server. You could create temporary (1min - 5 year) signed certs which combined with the SSH key could be "the private key".
>>>> You can easily configure the servers which "roles" aka "Principals" you want to login to that server. And as I mentioned before, you could limit the access via IP if the IP mapping is static.
>>>> 
>>>> I'm happy to explain how we implemented this approach internally. 
>>>> 
>>>> 
>>>> 2019年2月20日(水) 23:25 Olblak <me at olblak.com>:
>>>>> __
>>>>> Thanks Tyler for your review, you highlighted different topics that I wasn't able to decide by myself :D
>>>>>> 
>>>>>> What I'm not understanding right now is how you would solve for "group
>>>>>> membership" here? Let's take Daniel Beck for example, I would expect him to be
>>>>>> in all three "groups". Does that meant he has to have three different sets of
>>>>>> keys and three different sets of .ovpn configurations? Would he then
>>>>>> effectively only be able to work in one of these silos at once?
>>>>> 
>>>>> I am paraphrasing your questions.
>>>>> 
>>>>> 1. Do we need different vpn?
>>>>> 2. If we have different vpn, do we need different client certificates
>>>>> 
>>>>> _Do we need different vpn?_
>>>>> 
>>>>> They are two different reasons which could lead to the different vpn approach.
>>>>> 1. We want a clear distinction about who can access what and if that distinction exist, how many networks it implies.
>>>>> 2. The network you can reach depends on the number of network interface your vpn gateway has which is limited by the size of the machine.
>>>>> 
>>>>> So Let's take the following example:
>>>>> 
>>>>> Security network would have two network interfaces, one in the public dmz in order to be reachable from the wild and a second one in the "security" network (that doesn't exist today)
>>>>> This means that anyone who can connect to the security vpn can ping any machines in one of those two networks
>>>>> 
>>>>> Same approach for "release" vpn, but excepted that we don't want to allow the same group of people than those who work on security.
>>>>> 
>>>>> But let's consider a third "admin" vpn which would have three different interfaces.
>>>>> First network interface connected to dmz, second connected to security network and third interface connected to the release network.
>>>>> So the admin vpn overrides security and release and so admins don't need to use three different vpn.
>>>>> 
>>>>> The technical limitation is: Bigger is a machine size (and so the cost) and bigger is the number of network interface that you can attach to it and the paradox here is more a vpn has network interface and more we want to be strict with the number of people who can use it and more expensive will be the machine that doesn "nothing" excepted routing.
>>>>> 
>>>>> Because my initial though was to have only one vpn network, I deployed the current vpn machine with accesses to 3 different networks, public dmz, public data and public app as defined here https://github.com/jenkins-infra/azure/blob/72b2c7538e40b2920dfb6a8eef475e8c8f719d99/plans/machine_vpn.tf#L22 and that machine can go up to 4 network interfaces.
>>>>> 
>>>>> Then I had a discussion with Daniel and he mentions that +-30 people should have access to cert.ci.jenkins.io inside the vpn.
>>>>> And I don't have a strong opinion if it's critical or not if those people "see" other services like trusted, puppet console, ...
>>>>> 
>>>>> Because we only have cert.ci.jenkins.io running in the vpn at the moment, it's still an easy change to go to the multiple vpn approach
>>>>> 
>>>>> 2) Do we need different certificates
>>>>> 
>>>>> Regarding multiple clients certificates, I see two advantages, the maintainer of a specific network can be responsible to validate demands and if something goes wrong with ldap authentication we can still rely on the client certificates to handle authorization.
>>>>> The counterside is, it increases the management complexity and I am not planning to manage it.
>>>>> 
>>>>> > Basically, I don't think we should be more complicated than our respective
>>>>> > corporate VPN setups unless we have a really really really damn good reason for
>>>>> > it.
>>>>> 
>>>>> This is the thing that I try to figure out
>>>>> So unless someone has a good reason , I'll go with the single network and if someone needs more later, he can find inspiration to provision and configure additional vpn gateways.
>>>>> 
>>>>> 
>>>>> ---
>>>>> -> gpg --keyserver keys.gnupg.net --recv-key 52210D3D
>>>>> ---
>>>>> 
>>>>> On Tue, Feb 19, 2019, at 6:06 PM, R. Tyler Croy wrote:
>>>>> > (replies inline)
>>>>> > 
>>>>> > On Tue, 19 Feb 2019, Olblak wrote:
>>>>> > 
>>>>> > > Initially I though to have only one vpn network for simplicity but in the end I don't think it would be a biggest effort to maintain multiple vpn, even though I still have multiple changes that I need to do to reflect this.
>>>>> > > 
>>>>> > > To give you a big picture before you go deeper:
>>>>> > > 
>>>>> > > The first part is the machine provisioning <https://github.com/jenkins-infra/azure/blob/master/plans/machine_vpn.tf>, basically we use Terraform to provision a virtual machine inside the correct network with as many network interface as we need to access the different sub network and then we use puppet to configure <https://github.com/jenkins-infra/jenkins-infra/blob/staging/dist/profile/manifests/openvpn.pp> that machine. 
>>>>> > > 
>>>>> > > The Docker Image <https://github.com/jenkins-infra/openvpn> used for the vpn service is design to use two level of authentication, one with client certificates and then we use Ldap group membership to ensure that the right person has access to the right network. Another advantage, if one of the two authentication method is compromised, we can still rely on the other method as an additional protection.
>>>>> > > 
>>>>> > > Client certificate are managed from this repository <https://github.com/olblak/jenkins-vpn-keys>, the main idea is they are two kind of person, "client" who create a private key and a signing certificate request and then submit PR with their csr. Then administrators can sign csr once the PR is merged on master. The ca private key used to sign is encrypted with sops and store on the git repository so we have everything needed in one place. This approach allow us to have multiple administrators for different vpn network with great visibility on who can access what.
>>>>> > 
>>>>> > 
>>>>> > Overall this looks like really great work olblak, thanks for driving it.
>>>>> > 
>>>>> > This key signing dance is something I've seen at multiple companies, so seems
>>>>> > reasonable.
>>>>> > 
>>>>> > 
>>>>> > What I'm not understanding right now is how you would solve for "group
>>>>> > membership" here? Let's take Daniel Beck for example, I would expect him to be
>>>>> > in all three "groups". Does that meant he has to have three different sets of
>>>>> > keys and three different sets of .ovpn configurations? Would he then
>>>>> > effectively only be able to work in one of these silos at once?
>>>>> > 
>>>>> > 
>>>>> > I'm not sure that multiple VPN networks is actually necessary since we will
>>>>> > also have application level access controls. For example, evne with cert.ci and
>>>>> > trusted.ci, there will still be application-layer authentication and
>>>>> > authorization. I am personally comfortable with the trust expectation that
>>>>> > everybody with VPN access is more trusted than the public internet, and will
>>>>> > not act maliciously agaisnt internal services which they might not otherwise
>>>>> > have access to.
>>>>> > 
>>>>> > Basically, I don't think we should be more complicated than our respective
>>>>> > corporate VPN setups unless we have a really really really damn good reason for
>>>>> > it.
>>>>> > 
>>>>> > 
>>>>> > 
>>>>> > Cheers
>>>>> > 
>>>>> > --
>>>>> > GitHub: https://github.com/rtyler
>>>>> > 
>>>>> > GPG Key ID: 0F2298A980EE31ACCA0A7825E5C92681BEF6CEA2
>>>>> >
>>>>> _______________________________________________
>>>>>  Jenkins-infra mailing list
>>>>> Jenkins-infra at lists.jenkins-ci.org
>>>>> http://lists.jenkins-ci.org/mailman/listinfo/jenkins-infra
>>>> 
>>>> 
>>>> -- 
>>>> *Adam Papai*
>>>> Senior Operations Engineer
>>>> CloudBees, Inc.

>>>> CloudBees-Logo.png <https://www.cloudbees.com/>

>>>> 
>>>> P: +8170-4092-4762*
*E: apapai at cloudbees.com

>>>> Skype: woohskype
>>>> Twitter: @wooh_tw <https://twitter.com/wooh_tw>

>>>> _______________________________________________
>>>>  Jenkins-infra mailing list
>>>> Jenkins-infra at lists.jenkins-ci.org
>>>> http://lists.jenkins-ci.org/mailman/listinfo/jenkins-infra
>> 
>> 
>> -- 
>> *Adam Papai*
>> Senior Operations Engineer
>> CloudBees, Inc.

>> CloudBees-Logo.png <https://www.cloudbees.com/>

>> 
>> P: +8170-4092-4762*
*E: apapai at cloudbees.com

>> Skype: woohskype
>> Twitter: @wooh_tw <https://twitter.com/wooh_tw>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.jenkins-ci.org/pipermail/jenkins-infra/attachments/20190318/a5fd2e7c/attachment-0001.html>

More information about the Jenkins-infra mailing list