Automatic DNS and SSL on Kubernetes with LetsEncrypt – Part 1

TL;DR Howto make your Kubernetes cluster super awesome by adding two pods which automatically handle public DNS registration and SSL certs for any deployment you choose! Reduces the complexity of deployments and reduces manual extra tasks.

Overview

Kubernetes Ingress controllers provide developers an API for creating HTTP/HTTPS (L7) proxies in front of your applications, something that historically we’ve done ourselves; Either inside our application pod’s with our apps, or more likley, as a separate set of pods infront of our application, strung together with Kubernetes Service’s (L4).

Without Ingress Controller

Public kubernetes service flow without ingress controllers

With Ingress Controller

Ingress controller simplifying k8s services

Technically, there is still a Service in the background to track membership, but it’s not in the “path of traffic” as it is in the first diagram.

Whats more, Ingress controllers are pluggable, a single Kubernetes API to developers, but any L7 load balancer in reality, be it Nginx, GCE, Treifik, or Hardware… Excellent.

However, there are some things Ingress controllers *DONT* do for us, and that is what I want to tackle today…

  1. Registering our ingress loadbalancer in public DNS with a useful domain name.
  2. Automatically getting SSL/TLS certificates for our domain and configuring them on our Ingress load balancer.

With these two additions developers can deploy their application to K8’s, and automatically have it accessible and TLS secured.. Perfect!

DNS First

DNS is fairly simple, yet a google search for this topic makes it sound anything but. Lots of different suggestions, github issues, half-started projects.

All we want it something to listen for new Ingress Controllers, find the public IP given to the new Ingress loadbalancer and update DNS with the apps DNS name and loadbalancer IP.

After some research, code exists to do exactly what we want, it’s called ‘dns-controller’ and it’s now part of the ‘kops’ codebase from the cluster-ops SIG. It currently updates AWS Route53, but thats fine, as it’s what i’m using anyway.

https://github.com/kubernetes/kops/tree/master/dns-controller

However, the documentation is slim and unless you’re using KOPS, it’s not packaged in a useful way. Thankfully, someone has already extracted the dns-controller pieces and packaged them in a docker container for us.

The security guy in me points out: If you’re looking at anything more than testing, i’d strongly recommend packaging the DNS-Controller code yourself so you know 100% whats in it.

DNS – Here’s how to deploy (1/2)

Create the following deployment.yaml manifest

This pulls down our pre-packaged dns-controller code and runs it on our cluster. By default i’ve placed this in the kube-system namespace.

The code needs to change AWS Route53 DNS entries *duh*, so it also needs AWS Credentials.

(I recommend using AWS IAM to create a user with ONLY the access to the Route53 zone you need this app to control. Don’t give it your developer keys, anyone in your K8’s cluster could potentially read them)

When we’ve got our credentials, create a secret with your AWS credentials file in it as follows..

we’ve The path to your AWS credentials file will differ. If you don’t have a credentials file, it’s a simple format as shown below.

Now deploy your dns-controller into K8’s with  kubectl create -f deployment.yaml

You can query the applications logs to see it working, by default it will try to update any DNS domain it finds configured for an Ingress controller with a matching zone in Route53.

Example log output:

You will see errors here if it cannot find your AWS credentials (check your secret) and that the credentials are valid!

Using our new automated DNS service.

Right! How do we use it? This is an ingress controller without automatic DNS..

And this is one WITH our new automatic DNS registration..

Simply add the annotation  dns.alpha.kubernetes.io/external: "true" to any ingress controller and our new dns-controller will try to add the domain listed under  - host: app.ourdomain.com  to DNS with the public IP of the Ingress controller.

Try it out! My cluster is on GCE (a GKE cluster), we’re using the google load balancers. I’m noticing they take around ~60 seconds to get a public IP assigned, so DNS can take ~90-120 seconds to be populated. That said, I don’t need to re-deploy my ingress controllers with my software deployments, so this is acceptable for me.

In the next section, we’ll configure automatic SSL certificate generation and configuration for our GCE load balancers!

Go read part two now, CLICK HERE.

Comments or suggestions? Please find me on twitter @mattdashj

OpenStack infrastructure automation with Terraform – Part 2

TL;DR: Second of a two post series looking at automation of an openstack project with Terraform, using the new Terraform OpenStack Provider.

With the Openstack provider for Terraform being close to accepted into the Terraform release, it’s time to unleash it’s power on the Cisco Openstack-based Cloud..

In this post, we will:

  • Write a terraform ‘.TF’ file to describe our desired deployment state including;
    • Neutron networks/subnets
    • Neutron gateways
    • Keypairs and Security Groups
    • Virtual machines and Volumes
    • Virtual IP’s
    • Load balancers (LBaaS).
  • Have terraform deploy, modify and rip down our infrastructure.

If you don’t have the terraform openstack beta provider available, you’ll want to read Part 1 of this series.

Terraform Intro

Terraform “provides a common configuration to launch infrastructure“. From IaaS instances and virtual networks to DNS entries and e-mail configuration.

The idea being that a single Terraform deployment file can leverage multiple providers to describe your entire application infrastructure in one deployment tool; even if your DNS, LB and Compute resources come from three different providers.

Support for different infrastructure types is supported by provider modules, it’s the Openstack provider we’re focused on testing here.

If you’re not sure why you want to use Terraform, you’re probably best getting off here and having a look around Terraform.io first!

Terraform Configuration Files

Terraform configuration files describe your desired infrastructure state, built up of multiple resources, using one or more providers.

Configuration files are a custom, but easy to read format with a .TF extension. (They can also be written in JSON for machine generated content.)

Generally, a configuration file will hold necessary parameters for any providers needed, followed by a number of resources from those providers.

Below is a simple example with one provider (Openstack) and one resource (an SSH public key to be uploaded to our Openstack tenant)

Save the above as  demo1.tf and replace the following placeholders with your own Openstack environment login details.

Now run $terraform plan  in the same directory as your demo1.tf  file. Terraform will tell you what it’s going to do (add/remove/update resources), based on checking the current state of the infrastructure:

Terraform checks, the keypair doesn’t already exist on our openstack provider, so a new resource is going to be created if we apply our infrastructure… good!

Terraform Apply!

Success! At this point you can check Openstack to confirm our new keypair exists in the IaaS:

 

Terraform State

Future deployments of this infrastructure will check the state first, running $terraform plan  again shows no changes, as our single resource already exists in Openstack.

That’s basic terraform deployment covered using the openstack provider.

Adding More Resources

The resource we deployed above was ‘ openstack_compute_keypair_v2 ‘. Resource types are named by the author of a given plugin! not centrally by terraform (which means TF config files are not re-usable between differing provider infrastructures).

Realistically this just means you need to read the doc of the provider(s) you choose to use.

Here are some openstack provider resource types we’ll use for the next demo:

“openstack_compute_keypair_v2”
“openstack_compute_secgroup_v2”
“openstack_networking_network_v2” 
“openstack_networking_subnet_v2”
“openstack_networking_router_v2”
“openstack_networking_router_interface_v2”
“openstack_compute_floatingip_v2”
“openstack_compute_instance_v2”
“openstack_lb_monitor_v1”
“openstack_lb_pool_v1”
“openstack_lb_vip_v1”

If you are familiar with Openstack, then their purpose should be clear!

The following Terraform configuration will build on our existing configuration to:

  • Upload a keypair
  • Create a security group
    • SSH and HTTPS in, plus all TCP in from other VM’s in same group.
  • Create a new Quantum network and Subnet
  • Create a new Quantum router with an external gateway
  • Assign the network to the router (router interface)
  • Request two floating IP’s into our Openstack project
  • Spin up three instances of CentOS7 based on an existing image in glance
    • With sample metadata provided in our .tf configuration file
    • Assigned to the security group terraform created
    • Using the keypair terraform created
    • Assigned to the network terraform created
      • Assigned static IP’s 100-103
    • The first two instances will be bound to the two floating IP’s
  • Create a Load Balancer Pool, Monitor and VIP.

Before we go ahead and $terraform plan ; $terraform apply  this configuration.. A couple of notes.

Terraform Instance References / Variables

This configuration introduces a lot of resources, each resource may have a set of required and optional fields.

Some of these fields require the UUID/ID of other openstack resources, but as we haven’t created any of the infrastructure yet via  $terraform apply , we can’t be expected to know the UUID of objects that don’t yet exist.

Terraform allows you to reference other resources in the configuration file by their terraform resource name, terraform will then order the creation of resources and dynamically fill in the required information when needed.

For example. In the following resource section, we need the ID of an Openstack Neutron network in order to create a subnet under it. The ID of the network is not known, as it doesn’t yet exist. So instead a reference to our named instance of the the openstack_network_v2 resource,   tf_network  is used and from that resource we want the ID passing to the subnet resource hence the .id  at the end.

Regions

You will notice each resource has a region=""  field. This is a required field in the openstack terraform provider module for every resource (try deleting it, $terraform plan  will error).

If your openstack target is not region aware/enabled, then you must set the region to null in this way.

Environment specific knowledge

Even with dynamic referencing of ID’s explained above, you are still not going to be able to copy, paste, save and $terraform apply , as there are references in the configuration specific to my openstack environment, just like username, password and openstack API URL in demo1, in demo2 you will need to provide the following in your copy of the configuration:

  • Your own keypair public key
  • The ID of your environment’s ‘external gateway’ network for binding your Neutron router too.
  • The pool name(s) to request floating IP’s from.
  • The Name/ID of a glance image to boot the instances from.
  • The Flavour name(s) of your environment’s instances.

I have placed a sanitised version of the configuration file in a gist, with these locations clearly marked by <<USER_INPUT_NEEDED>> to make the above items easier to find/edit.

http://goo.gl/B3x1o4

Creating the Infrastructure 

With your edits to the configuration done:

Terraform Apply! (for the final time in this post!)

Enjoy your new infrastructure!

We can also confirm these items really do exist in openstack:

Destroying Infrastructure

$terraform destroy  will destroy your infrastructure. I find this often needs running twice, as certain objects (subnets, security groups etc) are still in use when terraform tries to delete them.

This could simply be our terraform API calls being quicker than the state update within openstack, there is a bug open with the openstack terraform provider.

First Run:

Second Run: Remaining resources are now removed.

Thats all for now boys and girls!

Enjoy your weekend.

 

 

 

UCS vMedia Configuration and Boot Order

Just a quick note on Cisco UCS vMedia.

If you have configured a remote CD/DVD from a remote ISO and UCS manager is showing the image is ‘mounted’ but your server is still stuck in a PXE/Netboot loop…

It may be helpful to know that your regular boot order policy in your service profile doesn’t apply here.

AKA. If you have ‘CD/DVD’ in your Boot Order.
This still wont automatically boot into a vMedia CD/DVD.

UCS System manager boot priority list
UCS System manager boot priority list

 

Solution
You’ll need to F6 from the KVM console on server boot, there you will see an option for booting from CIMC vMedia DVD.

B Series UCS F6 Boot Options
B Series UCS F6 Boot Options

This will get you where you need to be!

Also. For those that don’t know. You can check the status of your vMedia mount under Equipment > Server > Inventory > CIMC.

Scroll down and you’ll see something like below.

UCS System Manager vMedia inventory

 

Matt

Cisco IOS Policy-Map stops working after Class-Map changes

Hi all,

Came accross this issue the other day and since i’m currently stuck on a train between London Euston and Manchester, with what could be called a passable excuse of an internet connection, I’ll take a moment to document it, hope it saves someone else some head-scratching.

Issue: If using a Policy-Map based service policy within IOS to filter traffic on an interface and the undelying class-map is edited with any ‘match protocol http <more>‘ statements, the policy-map stops processing traffic, effectivley turning itself off for that interface.

Consider the following, two interfaces;

Vlan 1: 192.168.0.1/24

Loopback 99: 192.168.100.1/24

We the create a simple class-map to match ICMP traffic and use this in a policy-map with a match action of ‘drop’.

Screenshot showing class-map and policy-map

We now assign this to the output of the loopback99 interface, with the following command;

conf t
interface loopback 99
service-policy output TEST_POLICY_1
exit
exit

This should now block ICMP traffic (such as an echo/ping) to the interface IP;

We can also see the policy-map status for the interface, showing packets flowing through the assigned service policy and that drops are occuring;

Here is the issue, when we add/change any match criteria in the class-map (TEST_CLASS_1) relating to HTTP, the policy map stops working.

I also I added a ‘match protocol smtp’  before this, but you’ll just have to trust me that the policy-map continued working after that, only the addition of HTTP inspection caused a failure.

 

And now our traffic fails to pass through the policy-map, allowing ICMP which should be dropped;

The only workaround I have found is to remove the service policy from the interface and then re-add it after a class-map change, this restores correct functionality;

The following further output (too large for a screenshot) shows that while ‘broken’, the traffic was not even hitting the policy-map (as can be seen through traffic counters);

APCI877#sh policy-map interface loopback99
Loopback99


Service-policy output: TEST_POLICY_1
Class-map: TEST_CLASS_1 (match-any)
20 packets, 2000 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: protocol icmp
20 packets, 2000 bytes
5 minute rate 0 bps
Match: protocol smtp
0 packets, 0 bytes
5 minute rate 0 bps
Match: protocol http host "bob.com"
0 packets, 0 bytes
5 minute rate 0 bps
drop


Class-map: class-default (match-any)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: any
APCI877#ping 192.168.100.1 source vlan 1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.100.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.0.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/2/4 ms
APCI877#ping 192.168.100.1 source vlan 1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.100.1, timeout is 2 seconds:
Packet sent with a source address of 192.168.0.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
APCI877#sh policy-map interface loopback99
Loopback99
Service-policy output: TEST_POLICY_1
Class-map: TEST_CLASS_1 (match-any)
20 packets, 2000 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: protocol icmp
20 packets, 2000 bytes
5 minute rate 0 bps
Match: protocol smtp
0 packets, 0 bytes
5 minute rate 0 bps
Match: protocol http host "bob.com"
0 packets, 0 bytes
5 minute rate 0 bps
drop

Class-map: class-default (match-any)
0 packets, 0 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: any

I have has a search via google and looked around the cisco bug toolkit, but couldnt see anything exactly matching this behaviour. I will be testing on the latest IOS 15.1 at some point (when i’m infront of a router that has a little more flash/ram).

Any suggestions, comments or blaringly obvious known cisco bugs i’ve missed are welcome on this one!

Matt