Weave Net

Integrating Kubernetes via the Addon

The following topics are discussed:

Installation

Before installing Weave Net, you should make sure the following ports are not blocked by your firewall: TCP 6783 and UDP 6783/6784. For more details, see the FAQ.

Weave Net can be installed onto your CNI-enabled Kubernetes cluster with a single command:

$ kubectl apply -f https://reweave.azurewebsites.net/k8s/v1.29/net.yaml

Replace the v1.29 with the version of Kubernetes running on your cluster.

Important: this configuration won’t enable encryption by default. If your data plane traffic isn’t secured that could allow malicious actors to access your pod network. Read on to see the alternatives.

After a few seconds, a Weave Net pod should be running on each Node and any further pods you create will be automatically attached to the Weave network.

Note: This command requires Kubernetes 1.4 or later, and we recommend your master node has at least two CPU cores.

CNI, the Container Network Interface, is a proposed standard for configuring network interfaces for Linux containers.

If you do not already have a CNI-enabled cluster, you can bootstrap one easily with kubeadm.

Alternatively, you can configure CNI yourself

Weave net depends on the portmap standard CNI plugin to support hostport functionality. Please ensure that portmap CNI plugin is installed (either by cluster installers like kubeadm or manually if you have configured CNI yourself) in /opt/cni/bin directory.

Note: If using the Weave CNI Plugin from a prior full install of Weave Net with your cluster, you must first uninstall it before applying the Weave-kube addon. Shut down Kubernetes, and on all nodes perform the following:

Then relaunch Kubernetes and install the addon as described above.

Installing on GKE

Please note that you must grant the user the ability to create roles in Kubernetes before launching Weave Net. This is a prerequisite to use use role-based access control on GKE. Please see the GKE instructions.

Installing on EKS

EKS by default installs amazon-vpc-cni-k8s CNI. Please follow below steps to use Weave-net as CNI

Please note that while pods can connect to the Kubernetes API server for your cluser, API server will not be able to connect to the pods as API server nodes are not connected to Weave Net (they run on network managed by EKS).

Upgrading the Daemon Sets

The DaemonSet definition specifies Rolling Updates, so when you apply a new version Kubernetes will automatically restart the Weave Net pods one by one.

CPU and Memory Requirements

Kubernetes manages resources on each node, and only schedules pods to run on nodes that have enough free resources.

In the example manifests we request 10% of a CPU, which will be enough for small installations, but you should monitor how much it uses in your production clusters and adjust the requests to suit.

We do not recommend that you set a CPU or memory limit unless you are very experienced in such matters, because the implementation of limits in the Linux kernel sometimes behaves in a surprising way.

On a 1-node single-CPU cluster you may find Weave Net does not install at all, because other Kubernetes components already take 95% of the CPU. The best way to resolve this issue is to use machines with at least two CPU cores.

Pod Eviction

If a node runs out of CPU, memory or disk, Kubernetes may decide to evict one or more pods. It may choose to evict the Weave Net pod, which will disrupt pod network operations.

You can reduce the chance of eviction by changing the DaemonSet to have a much bigger request, and a limit of the same value.

This causes Kubernetes to apply a “guaranteed” rather than a “burstable” policy. However a similar request for disk space can not be made, and so please be aware of this issue and monitor your resources to ensure that they stay below 100%.

You can see when pods have been evicted via the kubectl get events command

LASTSEEN   COUNT     NAME          KIND    TYPE      REASON     SOURCE            MESSAGE
1m         1         mypod-09vkd   Pod     Warning   Evicted    kubelet, node-1   The node was low on resource: memory.

or kubectl get pods

NAME                READY     STATUS    RESTARTS   AGE       IP          NODE
mypod-09vkd         0/1       Evicted   0          1h        <none>      node-1

If you see this in your cluster, consider some of the above steps to reduce disruption.

Features

Pod Network

Weave Net provides a network to connect all pods together, implementing the Kubernetes model.

Kubernetes uses the Container Network Interface (CNI) to join pods onto Weave Net.

Kubernetes implements many network features itself on top of the pod network. This includes Services, Service Discovery via DNS and Ingress into the cluster. WeaveDNS is disabled when using the Kubernetes addon.

Network Policy

Kubernetes Network Policies let you securely isolate pods from each other based on namespaces and labels. For more information on configuring network policies in Kubernetes see the walkthrough and the NetworkPolicy API object definition

Note: as of version 1.9 of Weave Net, the Network Policy Controller allows all multicast traffic. Since a single multicast address may be used by multiple pods, we cannot implement rules to isolate them individually. You can turn this behaviour off (block all multicast traffic) by adding --allow-mcast=false as an argument to weave-npc in the YAML configuration.

Note: Since ingress traffic is masqueraded, it makes sense to use ipBlock selector in an ingress rule only when limiting access to a Service annotated with externalTrafficPolicy=Local or between Pods when podIP is used to access a Pod.

Troubleshooting

The first thing to check is whether Weave Net is up and running. The kubectl apply command you used to install it only requests that it be downloaded and started; if anything goes wrong at startup, those details will only be visible in the logs of the container(s).

To check what is running:

$ kubectl get pods -n kube-system -l name=weave-net
NAME              READY  STATUS   RESTARTS  AGE
weave-net-1jkl6   2/2    Running  0         1d
weave-net-bskbv   2/2    Running  0         1d
weave-net-m4x1b   2/2    Running  0         1d

You should see one line for each node in your cluster; each line should have STATUS “Running”, and READY should be 2 out of 2. If you see a STATUS like “Error” or “CrashLoopBackoff”, look in the logs of the container with that status.

Reading the logs

Pick one of the pods from the list output by kubectl get pods and ask for the logs like this:

$ kubectl logs -n kube-system weave-net-1jkl6 weave

For easier viewing, pipe the output into a file, especially if it is long.

By default log level of weave container is set to info level. If you wish to see more detailed logs you can set the desired log level for the --log-level flag through the EXTRA_ARGS environment variable for the weave container in the weave-net daemon set. Add environment variable as below.

      containers:
      - command:
        - /home/weave/launch.sh
        name: weave
        env:
        - name: EXTRA_ARGS
          value: --log-level=debug

You may also set the --log-level flag to warning or error if you prefer to only log exceptional conditions.

Many Kubernetes network issues occur at a higher level than Weave Net. The Kubernetes Service Debugging Guide has a detailed step-by-step guide.

Once it is up and running, the status of Weave Net can be checked by running its CLI commands. This can be done in various ways:

1. Install the weave script and run:

$ weave status
        Version: 2.0.1 (up to date; next check at 2017/07/10 13:49:29)

        Service: router
       Protocol: weave 1..2
           Name: 42:8e:e8:c4:52:1b(host-0)
     Encryption: disabled
  PeerDiscovery: enabled
        Targets: 3
    Connections: 3 (2 established, 1 failed)
          Peers: 3 (with 6 established connections)
 TrustedSubnets: none

        Service: ipam
         Status: ready
          Range: 10.32.0.0/12
  DefaultSubnet: 10.32.0.0/12

2. If you don’t want to install additional software onto your hosts, run via kubectl commands, which produce the exact same outcome as the previous example:

### Identify the Weave Net pods:

$ kubectl get pods -n kube-system -l name=weave-net -o wide
NAME              READY  STATUS   RESTARTS  AGE  IP           NODE
weave-net-1jkl6   2/2    Running  0         1d   10.128.0.4   host-0
weave-net-bskbv   2/2    Running  0         1d   10.128.0.5   host-1
weave-net-m4x1b   2/2    Running  0         1d   10.128.0.6   host-2

The above shows all Weave Net pods available in your cluster. You can see Kubernetes has deployed one Weave Net pod per host, in order to interconnect all hosts.

You then need to:

$ kubectl exec -n kube-system weave-net-1jkl6 -c weave -- /home/weave/weave --local status

        Version: 2.0.1 (up to date; next check at 2017/07/10 13:49:29)

        Service: router
       Protocol: weave 1..2
           Name: 42:8e:e8:c4:52:1b(host-0)
     Encryption: disabled
  PeerDiscovery: enabled
        Targets: 3
    Connections: 3 (2 established, 1 failed)
          Peers: 3 (with 6 established connections)
 TrustedSubnets: none

        Service: ipam
         Status: ready
          Range: 10.32.0.0/12
  DefaultSubnet: 10.32.0.0/12

Troubleshooting Blocked Connections

If you suspect that legitimate traffic is being blocked by the Weave Network Policy Controller, the first thing to do is check the weave-npc container’s logs.

To do this, first you have to find the name of the Weave Net pod running on the relevant host:

$ kubectl get pods -n kube-system -o wide | grep weave-net
weave-net-08y45                  2/2       Running   0          1m        10.128.0.2   host1
weave-net-2zuhy                  2/2       Running   0          1m        10.128.0.4   host3
weave-net-oai50                  2/2       Running   0          1m        10.128.0.3   host2

Select the relevant container, for example, if you want to look at host2 then pick weave-net-oai50 and run:

$ kubectl logs <weave-pod-name-as-above> -n kube-system weave-npc

When the Weave Network Policy Controller blocks a connection, it logs the following details about it:

as per the below example:

TCP connection from 10.32.0.7:56648 to 10.32.0.11:80 blocked by Weave NPC.
UDP connection from 10.32.0.7:56648 to 10.32.0.11:80 blocked by Weave NPC.

Things to watch out for

The following documents a problem in Weave Net before version 2.8.2. The problem does not exist any longer. The documentation is being kept as is, for now.

### Troubleshooting FailedCreatePodSandBox errors

If your Kubernetes cluster uses the containerd runtime (versions 1.6.0 through 1.6.4), Weave Net will not be able to allocate IP addresses to pods. Your pods, except the ones that use HostNetworking, will be stuck at ContainerCreating status.

You can examine any pod so affected by running kubectl describe, for example:

 $ kubectl describe pod -n kube-system coredns-78fcd69978-dbxs9

The events section will show repeated errors like the following:

   Warning  FailedCreatePodSandBox  3m6s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "09a23f79c96333b9f54e12df54e817837c8021cbaa32bdfeefbe2a1fb215d9ef": plugin type="weave-net" name="weave" failed (add): unable to allocate IP address: Post "http://127.0.0.1:6784/ip/09a23f79c96333b9f54e12df54e817837c8021cbaa32bdfeefbe2a1fb215d9ef": dial tcp 127.0.0.1:6784: connect: connection refused

You can verify that you are running an affected version of containerd by using the following:

 $ kubectl get nodes -o wide

 NAME     STATUS   ROLES                  AGE   VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                         KERNEL-VERSION    CONTAINER-RUNTIME
 host-1   Ready    control-plane,master   13m   v1.22.10   172.21.107.129   <none>        Debian GNU/Linux 11 (bullseye)   5.10.0-14-amd64   containerd://1.6.4

The last column shows the container runtime and version.

Alternatively, you can run:

 $ containerd -v
 containerd containerd.io 1.6.4 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16

The problem can be solved by upgrading containerd to v1.6.5 or above. For example, on Debian Linux, using the docker official repositories, you can use:

 sudo apt install containerd.io=1.6.6-1

The problem occurs because of a behaviour change in cni v1.1.0, which caused a regression issue in Weave. It was corrected in cni v1.1.1. Containerd 1.6.5 onwards uses cni 1.1.1 and above.

Changing Configuration Options

Manually editing the YAML file

You can manually edit the YAML file downloaded from our releases page,

For example,

      containers:
        - name: weave
          env:
            - name: IPALLOC_RANGE
              value: 10.0.0.0/16

The list of variables you can set is:

Securing the Setup

You should set the environment variable WEAVE_PASSWORD as stated in the previous section to enable the data plane encryption; this is a recommended option in case you cannot be sure about the security of the fabric between your nodes.

A different option is to use trusted-subnets and whitelist only the subnets that host your k8s nodes. Mind that depending on your circumstances that might allow a malicious container running in your cluster to access the weave dataplane, still.

Read on the Securing Connections Across Untrusted Networks document to see the alternatives.

To improve security drop CAP_NET_RAW from pod capabilities: by default pods can forge packets from anywhere on the network, which enables attacks such as DNS spoofing.

     securityContext:
       capabilities:
         drop: ["NET_RAW"]