Notes to self, 2023

2023-01-27 - windows openvpn / unexpected default route

The other day, I was looking into as VPN client issue. The user could connect, he would get his routes pushed, but he would then proceed to use the VPN for all traffic instead of just the routes we pushed.

We did not push a default route, because this VPN server exposed a small internal network only. Any regular internet surfing should be done directly. So, when I looked at a tcpdump I was baffled when I saw that DNS lookups were attempted through the OpenVPN tunnel:

12:50:45.992684 IP 10.8.8.11.51953 > 8.8.8.8.53:
  51928+ A? kv601.prod.do.dsp.mp.microsoft.com. (52)

The server in question runs OpenVPN. The client that exhibited this behaviour was OpenVPN Connect v3 for Windows, with the following peer info, according to the server logs:

peer info: IV_VER=3.git::d3f8b18b
peer info: IV_PLAT=win
peer info: IV_NCP=2
peer info: IV_TCPNL=1
peer info: IV_PROTO=30
peer info: IV_CIPHERS=AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305
peer info: IV_AUTO_SESS=1
peer info: IV_GUI_VER=OCWindows_3.3.6-2752
peer info: IV_SSO=webauth,openurl,crtext

There are not many user settings in OpenVPN Connect, and there was no “Send all traffic over this tunnel” option to uncheck.

We had recently forced 8.8.8.8 as DNS on the users (to solve a different issue) with the following rules:

push "dhcp-option DNS 8.8.8.8"
push "dhcp-option DNS 8.8.4.4"
push "dhcp-option DOMAIN-ROUTE one-specific-domain.tld"

This rule was supposed to force lookups for *.one-specific-domain.tld to go through the aforementioned Google DNS servers. Maybe the VPN client secretly adds a route for this under the assumption that if you want a specific DNS server for VPN, it should be routed through the VPN as well.

This was easy enough to test. I allowed traffic to 8.8.8.8 and 8.8.4.4 to go through the VPN.

Did that fix the problem? Well, no. DNS resolving worked for the user, and now actual (non-DNS) traffic would be attempted through the VPN as well:

13:02:14.618777 IP 10.8.8.11.52040 > 23.2.214.66.443:
  Flags [S], seq 932856193, win 64240,
  options [mss 1289,nop,wscale 8,nop,nop,sackOK], length 0

What is up with this? A route print on the Windows side showed nothing out of the ordinary:

============================================================
Active Routes:
Network Destination      Netmask      Gateway      Interface
        0.0.0.0          0.0.0.0  192.168.1.1  192.168.1.100  <- default
    10.55.55.55  255.255.255.255     10.8.8.1      10.8.8.11  <- vpn
       10.8.8.0    255.255.255.0     On-link       10.8.8.11  <- vpn
      10.8.8.11  255.255.255.255     On-link       10.8.8.11  <- vpn
     10.8.8.255  255.255.255.255     On-link       10.8.8.11  <- vpn
      127.0.0.0        255.0.0.0     On-link       127.0.0.1
      127.0.0.1  255.255.255.255     On-link       127.0.0.1
127.255.255.255  255.255.255.255     On-link       127.0.0.1
    192.168.1.0    255.255.255.0     On-link   192.168.1.100
  192.168.1.100  255.255.255.255     On-link   192.168.1.100
  192.168.1.255  255.255.255.255     On-link   192.168.1.100
123.123.123.123  255.255.255.255  192.168.1.1  192.168.1.100  <- vpn-server
      224.0.0.0        240.0.0.0     On-link       127.0.0.1
      224.0.0.0        240.0.0.0     On-link   192.168.1.100
      224.0.0.0        240.0.0.0     On-link       10.8.8.11  <- vpn
255.255.255.255  255.255.255.255     On-link       127.0.0.1
255.255.255.255  255.255.255.255     On-link   192.168.1.100
255.255.255.255  255.255.255.255     On-link       10.8.8.11  <- vpn
============================================================

Ignoring broadcast and multicast addresses, only 10.55.55.55 and 10.8.8.* should go through the VPN interface. The default route 0.0.0.0 is clearly marked to go through the regular internet via the 192.168.1.1 gateway. This does not explain at all why traffic to 8.8.8.8 or 23.2.214.66 goes to the VPN.

In a last ditch attempt to fix things, I tried what happens if we did push 8.8.8.8 and 8.8.4.4 as routes that should go through the VPN:

push "route 8.8.8.8 255.255.255.255 vpn_gateway"
push "route 8.8.4.4 255.255.255.255 vpn_gateway"

Lo and behold! Things started working properly. Traffic to 10.55.55.55 (and to the nameservers) now goes through the tunnel, but traffic to the rest of the internet properly takes the default route.

I cannot explain why OpenVPN Connect on Windows would not use the routes it prints. Maybe there is a “Use default gateway on remote network” setting somewhere that got enabled when it received a DNS server IP that was not pushed over the same tunnel. One would think that this would be visible on the routing table though. If anyone reading this can explain this phenomenon, please drop me a line.

2023-01-17 - django 1.8 / python 3.10

After upgrading a machine to Ubuntu/Jammy there was an old Django 1.8 project that refused to run with the newer Python 3.10.

...
  File "django/db/models/sql/query.py", line 11, in <module>
    from collections import Iterator, Mapping, OrderedDict
ImportError: cannot import name 'Iterator' from 'collections' (/usr/lib/python3.10/collections/__init__.py)

This was relatively straight forward to fix, by using the following patch. Some parts were stolen from a stackoverflow response by Elias Prado.

--- a/django/core/paginator.py  2023-01-11 14:09:04.915505171 +0100
+++ b/django/core/paginator.py  2023-01-11 14:09:29.407130151 +0100
@@ -1,4 +1,4 @@
-import collections
+from collections.abc import Sequence
 from math import ceil
 
 from django.utils import six
@@ -103,7 +103,7 @@ class Paginator(object):
 QuerySetPaginator = Paginator   # For backwards-compatibility.
 
 
-class Page(collections.Sequence):
+class Page(Sequence):
 
     def __init__(self, object_list, number, paginator):
         self.object_list = object_list
--- a/django/db/migrations/writer.py  2023-01-11 14:13:07.507799080 +0100
+++ b/django/db/migrations/writer.py  2023-01-11 14:14:36.978436145 +0100
@@ -1,6 +1,6 @@
 from __future__ import unicode_literals
 
-import collections
+from collections.abc import Iterable
 import datetime
 import decimal
 import math
@@ -434,7 +434,7 @@ class MigrationWriter(object):
                     % (value.__name__, module_name, get_docs_version()))
             return "%s.%s" % (module_name, value.__name__), {"import %s" % module_name}
         # Other iterables
-        elif isinstance(value, collections.Iterable):
+        elif isinstance(value, Iterable):
             imports = set()
             strings = []
             for item in value:
--- a/django/db/models/base.py  2023-01-11 14:17:13.471982572 +0100
+++ b/django/db/models/base.py  2023-01-11 14:19:38.337720520 +0100
@@ -80,7 +80,12 @@ class ModelBase(type):
 
         # Create the class.
         module = attrs.pop('__module__')
-        new_class = super_new(cls, name, bases, {'__module__': module})
+        new_attrs = {'__module__': module}
+        classcell = attrs.pop('__classcell__', None)
+        if classcell is not None:
+            new_attrs['__classcell__'] = classcell
+        new_class = super_new(cls, name, bases, new_attrs)
+
         attr_meta = attrs.pop('Meta', None)
         abstract = getattr(attr_meta, 'abstract', False)
         if not attr_meta:
--- a/django/db/models/fields/__init__.py 2023-01-11 14:12:50.780054102 +0100
+++ b/django/db/models/fields/__init__.py 2023-01-11 14:14:02.290964344 +0100
@@ -1,7 +1,7 @@
 # -*- coding: utf-8 -*-
 from __future__ import unicode_literals
 
-import collections
+from collections.abc import Iterable, Iterator
 import copy
 import datetime
 import decimal
@@ -417,7 +417,7 @@ class Field(RegisterLookupMixin):
         for name, default in possibles.items():
             value = getattr(self, attr_overrides.get(name, name))
             # Unroll anything iterable for choices into a concrete list
-            if name == "choices" and isinstance(value, collections.Iterable):
+            if name == "choices" and isinstance(value, Iterable):
                 value = list(value)
             # Do correct kind of comparison
             if name in equals_comparison:
@@ -852,7 +852,7 @@ class Field(RegisterLookupMixin):
         return smart_text(self._get_val_from_obj(obj))
 
     def _get_choices(self):
-        if isinstance(self._choices, collections.Iterator):
+        if isinstance(self._choices, Iterator):
             choices, self._choices = tee(self._choices)
             return choices
         else:
--- a/django/db/models/sql/query.py 2023-01-11 14:07:45.900716653 +0100
+++ b/django/db/models/sql/query.py 2023-01-11 14:08:08.724366450 +0100
@@ -8,7 +8,8 @@ all about the internals of models in ord
 """
 import copy
 import warnings
-from collections import Iterator, Mapping, OrderedDict
+from collections import OrderedDict
+from collections.abc import Iterator, Mapping
 from itertools import chain, count, product
 from string import ascii_uppercase
 
--- a/django/db/models/sql/where.py 2023-01-11 14:13:01.595889201 +0100
+++ b/django/db/models/sql/where.py 2023-01-11 14:14:25.322613605 +0100
@@ -2,7 +2,7 @@
 Code to manage the creation and SQL rendering of 'where' constraints.
 """
 
-import collections
+from collections.abc import Iterator
 import datetime
 import warnings
 from itertools import repeat
@@ -59,7 +59,7 @@ class WhereNode(tree.Node):
         if not isinstance(data, (list, tuple)):
             return data
         obj, lookup_type, value = data
-        if isinstance(value, collections.Iterator):
+        if isinstance(value, Iterator):
             # Consume any generators immediately, so that we can determine
             # emptiness and transform any non-empty values correctly.
             value = list(value)

And to avoid the following warnings, the Django included six can be patched.

<frozen importlib._bootstrap>:914:
  ImportWarning: _SixMetaPathImporter.find_spec() not found;
  falling back to find_module()
<frozen importlib._bootstrap>:671:
  ImportWarning: _SixMetaPathImporter.exec_module() not found;
  falling back to load_module()

These changes were taken from six 1.16:

--- a/django/utils/six.py  2023-01-17 11:08:00.267645405 +0100
+++ b/django/utils/six.py  2023-01-17 11:12:13.993813451 +0100
@@ -71,6 +71,7 @@ else:
             MAXSIZE = int((1 << 63) - 1)
         del X
 
+from importlib.util import spec_from_loader
 
 def _add_doc(func, doc):
     """Add documentation to a function."""
@@ -186,6 +187,11 @@ class _SixMetaPathImporter(object):
             return self
         return None
 
+    def find_spec(self, fullname, path, target=None):
+        if fullname in self.known_modules:
+            return spec_from_loader(fullname, self)
+        return None
+
     def __get_module(self, fullname):
         try:
             return self.known_modules[fullname]
@@ -223,6 +229,12 @@ class _SixMetaPathImporter(object):
         return None
     get_source = get_code  # same as get_code
 
+    def create_module(self, spec):
+        return self.load_module(spec.name)
+
+    def exec_module(self, module):
+        pass
+
 _importer = _SixMetaPathImporter(__name__)
 
 
@@ -679,11 +691,15 @@ if PY3:
     exec_ = getattr(moves.builtins, "exec")
 
     def reraise(tp, value, tb=None):
-        if value is None:
-            value = tp()
-        if value.__traceback__ is not tb:
-            raise value.with_traceback(tb)
-        raise value
+        try:
+            if value is None:
+                value = tp()
+            if value.__traceback__ is not tb:
+                raise value.with_traceback(tb)
+            raise value
+        finally:
+            value = None
+            tb = None
 
 else:
     def exec_(_code_, _globs_=None, _locs_=None):
@@ -699,19 +715,19 @@ else:
         exec("""exec _code_ in _globs_, _locs_""")
 
     exec_("""def reraise(tp, value, tb=None):
-    raise tp, value, tb
+    try:
+        raise tp, value, tb
+    finally:
+        tb = None
 """)
 
 
-if sys.version_info[:2] == (3, 2):
-    exec_("""def raise_from(value, from_value):
-    if from_value is None:
-        raise value
-    raise value from from_value
-""")
-elif sys.version_info[:2] > (3, 2):
+if sys.version_info[:2] > (3,):
     exec_("""def raise_from(value, from_value):
-    raise value from from_value
+    try:
+        raise value from from_value
+    finally:
+        value = None
 """)
 else:
     def raise_from(value, from_value):
@@ -788,11 +804,10 @@ _add_doc(reraise, """Reraise an exceptio
 if sys.version_info[0:2] < (3, 4):
     def wraps(wrapped, assigned=functools.WRAPPER_ASSIGNMENTS,
               updated=functools.WRAPPER_UPDATES):
-        def wrapper(f):
-            f = functools.wraps(wrapped, assigned, updated)(f)
-            f.__wrapped__ = wrapped
-            return f
-        return wrapper
+        return functools.partial(_update_wrapper, wrapped=wrapped,
+                                 assigned=assigned, updated=updated)
+    wraps.__doc__ = functools.wraps.__doc__
+
 else:
     wraps = functools.wraps
 
@@ -802,10 +817,22 @@ def with_metaclass(meta, *bases):
     # This requires a bit of explanation: the basic idea is to make a dummy
     # metaclass for one level of class instantiation that replaces itself with
     # the actual metaclass.
-    class metaclass(meta):
+    class metaclass(type):
 
         def __new__(cls, name, this_bases, d):
-            return meta(name, bases, d)
+            if sys.version_info[:2] >= (3, 7):
+                # This version introduced PEP 560 that requires a bit
+                # of extra care (we mimic what is done by __build_class__).
+                resolved_bases = types.resolve_bases(bases)
+                if resolved_bases is not bases:
+                    d['__orig_bases__'] = bases
+            else:
+                resolved_bases = bases
+            return meta(name, resolved_bases, d)
+
+        @classmethod
+        def __prepare__(cls, name, this_bases):
+            return meta.__prepare__(name, bases)
     return type.__new__(metaclass, 'temporary_class', (), {})
 
 
@@ -821,6 +848,8 @@ def add_metaclass(metaclass):
                 orig_vars.pop(slots_var)
         orig_vars.pop('__dict__', None)
         orig_vars.pop('__weakref__', None)
+        if hasattr(cls, '__qualname__'):
+            orig_vars['__qualname__'] = cls.__qualname__
         return metaclass(cls.__name__, cls.__bases__, orig_vars)
     return wrapper
 

Patching is done using patch -p1 — you should know how.

2023-01-10 - sysctl / modules / load order / nf_conntrack

Recently we ran into an issue where connections were unexpectedly aborted. Connections from a NAT-ed client (a K8S pod) to a server would suddently get an old packet (according to the sequence number) in the middle of the data. This triggered the Linux NAT-box to issue an RST. Setting the kernel flag to mitigate this behaviour required some knowledge of module load order during boot.

Spurious retransmits causing connection teardown

To start off: we observed traffic from a pod to a server get disconnected. We enabling debug logging on the Kubernetes host where the pod resides. After enabling modprobe nf_log_ipv4 and net.netfilter.nf_conntrack_log_invalid=255, we saw this:

kernel: nf_ct_proto_6: SEQ is under the lower bound (already ACKed data retransmitted)
  IN= OUT= SRC=10.x.x.x DST=10.x.x.x LEN=1480 TOS=0x00 PREC=0x00 TTL=61 ID=53534 DF
  PROTO=TCP SPT=6379 DPT=26110 SEQ=4213094653 ACK=3402842193 WINDOW=509 RES=0x00
  ACK PSH URGP=0 OPT (0101080A084C76F30D12DCAA) 

In the middle of a sequence of several packets of data from the server, an apparently unrelated packet — it had data, but not intended for this stream — but with the same source/destination tuples and yet a sequence number that was more than 80K too low. (Wireshark flags this packet as invalid with a TCP Spurious Retransmission message.)

This triggered a reset (RST) by the Linux connection tracking module. And that in turned caused (unexpected) RSTs from the server.

POD <-> NAT <-> SRV
--------------------
            <-- TCP seq 2000000 ack 5555 len 1400
    <-- TCP seq 2000000 ack 5555 len 1400

            <-- TCP seq 1200000 ack 5555 len 1234 (seq is _way_ off)
            --> TCP RST seq 5555 len 0

            <-- TCP seq 2001400 ack 5555 len 1000
    <-- TCP seq 2001400 ack 5555 len 1000

(Made up numbers in the above table, but they illustrate the problem.)

At this point, the non-rejected traffic still got forwarded back to the pod. Its ACKs back to the server were now however rejected by the server with an RST of its own — that end of the connection thinks it was tore down already after all.

kernel: nf_ct_proto_6: invalid rst
  IN= OUT= SRC=10.x.x.x DST=10.x.x.x LEN=40 TOS=0x00 PREC=0x00
  TTL=61 ID=0 DF PROTO=TCP SPT=6379 DPT=26110 SEQ=4213164625
  ACK=0 WINDOW=0 RES=0x00 RST URGP=0 

The next packet (sequence 2001400 in the above example), was fine though. So if we could convince the Linux kernel to ignore the packet with the unexpected sequence number, we'd be good to go.

Luckily there is such a flag: net.netfilter.nf_conntrack_tcp_be_liberal=1

While this does not explain the root cause, setting said flag mitigates the problem. The kernel now ignores all spurious retransmits. So, we placed net.netfilter.nf_conntrack_tcp_be_liberal = 1 in /etc/sysctl.conf and were done with it.

... or so we thought. Because after a reboot, the flag was unset again.

sysctl.conf not picked up?

That's odd. Setting kernel parameters during boot should be done in sysctl.conf (or sysctl.d). Why did it not get picked up?

The cause turned out to be this: this particular setting is not built-in. It belongs to a module; the nf_conntrack module. And that module is not necessarily loaded before sysctl settings are applied.

nf_conntrack was loaded on demand, and not in a particular well-defined order. Luckily, loading modules through /etc/modules-load.d is well defined, as you can see:

# systemd-analyze critical-chain systemd-sysctl.service
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

systemd-sysctl.service +44ms
└─systemd-modules-load.service @314ms +90ms
  └─systemd-journald.socket @242ms
    └─system.slice @220ms
      └─-.slice @220ms

Indeed, it sysctl settings are applied after systemd-modules-load.service:

# systemctl show systemd-sysctl.service | grep ^After
After=systemd-journald.socket system.slice systemd-modules-load.service

So, we can use the kernel module loading to ensure that the conntrack module is loaded before we attempt to set its parameters:

# cat /etc/modules-load.d/modules.conf 
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
nf_conntrack

And that works. Now all settings are properly set during boot.

As for the spurious retransmissions: the last hasn't been said about that...