Notes to self, 2023
2023-01-27 - windows openvpn / unexpected default route
The other day, I was looking into as VPN client issue. The user could connect, he would get his routes pushed, but he would then proceed to use the VPN for all traffic instead of just the routes we pushed.
We did not push a default route, because this VPN server exposed a small internal network only. Any regular internet surfing should be done directly. So, when I looked at a tcpdump I was baffled when I saw that DNS lookups were attempted through the OpenVPN tunnel:
12:50:45.992684 IP 10.8.8.11.51953 > 8.8.8.8.53: 51928+ A? kv601.prod.do.dsp.mp.microsoft.com. (52)
The server in question runs OpenVPN. The client that exhibited this behaviour was OpenVPN Connect v3 for Windows, with the following peer info, according to the server logs:
peer info: IV_VER=3.git::d3f8b18b peer info: IV_PLAT=win peer info: IV_NCP=2 peer info: IV_TCPNL=1 peer info: IV_PROTO=30 peer info: IV_CIPHERS=AES-256-GCM:AES-128-GCM:CHACHA20-POLY1305 peer info: IV_AUTO_SESS=1 peer info: IV_GUI_VER=OCWindows_3.3.6-2752 peer info: IV_SSO=webauth,openurl,crtext
There are not many user settings in OpenVPN Connect, and there was no “Send all traffic over this tunnel” option to uncheck.
We had recently forced 8.8.8.8
as DNS on the
users (to solve a different issue) with the following rules:
push "dhcp-option DNS 8.8.8.8" push "dhcp-option DNS 8.8.4.4" push "dhcp-option DOMAIN-ROUTE one-specific-domain.tld"
This rule was supposed to force lookups for *.one-specific-domain.tld to go through the aforementioned Google DNS servers. Maybe the VPN client secretly adds a route for this under the assumption that if you want a specific DNS server for VPN, it should be routed through the VPN as well.
This was easy enough to test. I allowed traffic to
8.8.8.8
and 8.8.4.4
to go through the
VPN.
Did that fix the problem? Well, no. DNS resolving worked for the user, and now actual (non-DNS) traffic would be attempted through the VPN as well:
13:02:14.618777 IP 10.8.8.11.52040 > 23.2.214.66.443: Flags [S], seq 932856193, win 64240, options [mss 1289,nop,wscale 8,nop,nop,sackOK], length 0
What is up with this? A route print
on the
Windows side showed nothing out of the ordinary:
============================================================ Active Routes: Network Destination Netmask Gateway Interface 0.0.0.0 0.0.0.0 192.168.1.1 192.168.1.100 <- default 10.55.55.55 255.255.255.255 10.8.8.1 10.8.8.11 <- vpn 10.8.8.0 255.255.255.0 On-link 10.8.8.11 <- vpn 10.8.8.11 255.255.255.255 On-link 10.8.8.11 <- vpn 10.8.8.255 255.255.255.255 On-link 10.8.8.11 <- vpn 127.0.0.0 255.0.0.0 On-link 127.0.0.1 127.0.0.1 255.255.255.255 On-link 127.0.0.1 127.255.255.255 255.255.255.255 On-link 127.0.0.1 192.168.1.0 255.255.255.0 On-link 192.168.1.100 192.168.1.100 255.255.255.255 On-link 192.168.1.100 192.168.1.255 255.255.255.255 On-link 192.168.1.100 123.123.123.123 255.255.255.255 192.168.1.1 192.168.1.100 <- vpn-server 224.0.0.0 240.0.0.0 On-link 127.0.0.1 224.0.0.0 240.0.0.0 On-link 192.168.1.100 224.0.0.0 240.0.0.0 On-link 10.8.8.11 <- vpn 255.255.255.255 255.255.255.255 On-link 127.0.0.1 255.255.255.255 255.255.255.255 On-link 192.168.1.100 255.255.255.255 255.255.255.255 On-link 10.8.8.11 <- vpn ============================================================
Ignoring broadcast and multicast addresses, only
10.55.55.55
and 10.8.8.*
should go through the VPN interface.
The default route 0.0.0.0
is clearly marked to
go through the regular internet via the
192.168.1.1
gateway.
This does not explain at all why traffic to 8.8.8.8
or
23.2.214.66
goes to the VPN.
In a last ditch attempt to fix things, I tried what happens if we
did push 8.8.8.8
and 8.8.4.4
as
routes that should go through the VPN:
push "route 8.8.8.8 255.255.255.255 vpn_gateway" push "route 8.8.4.4 255.255.255.255 vpn_gateway"
Lo and behold! Things started working properly. Traffic to
10.55.55.55
(and to the nameservers) now goes through the
tunnel, but traffic to the rest of the internet properly takes the default
route.
I cannot explain why OpenVPN Connect on Windows would not use the routes it prints. Maybe there is a “Use default gateway on remote network” setting somewhere that got enabled when it received a DNS server IP that was not pushed over the same tunnel. One would think that this would be visible on the routing table though. If anyone reading this can explain this phenomenon, please drop me a line.
2023-01-17 - django 1.8 / python 3.10
After upgrading a machine to Ubuntu/Jammy there was an old Django 1.8 project that refused to run with the newer Python 3.10.
... File "django/db/models/sql/query.py", line 11, in <module> from collections import Iterator, Mapping, OrderedDict ImportError: cannot import name 'Iterator' from 'collections' (/usr/lib/python3.10/collections/__init__.py)
This was relatively straight forward to fix, by using the following patch. Some parts were stolen from a stackoverflow response by Elias Prado.
--- a/django/core/paginator.py 2023-01-11 14:09:04.915505171 +0100 +++ b/django/core/paginator.py 2023-01-11 14:09:29.407130151 +0100 @@ -1,4 +1,4 @@ -import collections +from collections.abc import Sequence from math import ceil from django.utils import six @@ -103,7 +103,7 @@ class Paginator(object): QuerySetPaginator = Paginator # For backwards-compatibility. -class Page(collections.Sequence): +class Page(Sequence): def __init__(self, object_list, number, paginator): self.object_list = object_list --- a/django/db/migrations/writer.py 2023-01-11 14:13:07.507799080 +0100 +++ b/django/db/migrations/writer.py 2023-01-11 14:14:36.978436145 +0100 @@ -1,6 +1,6 @@ from __future__ import unicode_literals -import collections +from collections.abc import Iterable import datetime import decimal import math @@ -434,7 +434,7 @@ class MigrationWriter(object): % (value.__name__, module_name, get_docs_version())) return "%s.%s" % (module_name, value.__name__), {"import %s" % module_name} # Other iterables - elif isinstance(value, collections.Iterable): + elif isinstance(value, Iterable): imports = set() strings = [] for item in value: --- a/django/db/models/base.py 2023-01-11 14:17:13.471982572 +0100 +++ b/django/db/models/base.py 2023-01-11 14:19:38.337720520 +0100 @@ -80,7 +80,12 @@ class ModelBase(type): # Create the class. module = attrs.pop('__module__') - new_class = super_new(cls, name, bases, {'__module__': module}) + new_attrs = {'__module__': module} + classcell = attrs.pop('__classcell__', None) + if classcell is not None: + new_attrs['__classcell__'] = classcell + new_class = super_new(cls, name, bases, new_attrs) + attr_meta = attrs.pop('Meta', None) abstract = getattr(attr_meta, 'abstract', False) if not attr_meta: --- a/django/db/models/fields/__init__.py 2023-01-11 14:12:50.780054102 +0100 +++ b/django/db/models/fields/__init__.py 2023-01-11 14:14:02.290964344 +0100 @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- from __future__ import unicode_literals -import collections +from collections.abc import Iterable, Iterator import copy import datetime import decimal @@ -417,7 +417,7 @@ class Field(RegisterLookupMixin): for name, default in possibles.items(): value = getattr(self, attr_overrides.get(name, name)) # Unroll anything iterable for choices into a concrete list - if name == "choices" and isinstance(value, collections.Iterable): + if name == "choices" and isinstance(value, Iterable): value = list(value) # Do correct kind of comparison if name in equals_comparison: @@ -852,7 +852,7 @@ class Field(RegisterLookupMixin): return smart_text(self._get_val_from_obj(obj)) def _get_choices(self): - if isinstance(self._choices, collections.Iterator): + if isinstance(self._choices, Iterator): choices, self._choices = tee(self._choices) return choices else: --- a/django/db/models/sql/query.py 2023-01-11 14:07:45.900716653 +0100 +++ b/django/db/models/sql/query.py 2023-01-11 14:08:08.724366450 +0100 @@ -8,7 +8,8 @@ all about the internals of models in ord """ import copy import warnings -from collections import Iterator, Mapping, OrderedDict +from collections import OrderedDict +from collections.abc import Iterator, Mapping from itertools import chain, count, product from string import ascii_uppercase --- a/django/db/models/sql/where.py 2023-01-11 14:13:01.595889201 +0100 +++ b/django/db/models/sql/where.py 2023-01-11 14:14:25.322613605 +0100 @@ -2,7 +2,7 @@ Code to manage the creation and SQL rendering of 'where' constraints. """ -import collections +from collections.abc import Iterator import datetime import warnings from itertools import repeat @@ -59,7 +59,7 @@ class WhereNode(tree.Node): if not isinstance(data, (list, tuple)): return data obj, lookup_type, value = data - if isinstance(value, collections.Iterator): + if isinstance(value, Iterator): # Consume any generators immediately, so that we can determine # emptiness and transform any non-empty values correctly. value = list(value)
And to avoid the following warnings, the Django included six can be patched.
<frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module() <frozen importlib._bootstrap>:671: ImportWarning: _SixMetaPathImporter.exec_module() not found; falling back to load_module()
These changes were taken from six 1.16:
--- a/django/utils/six.py 2023-01-17 11:08:00.267645405 +0100 +++ b/django/utils/six.py 2023-01-17 11:12:13.993813451 +0100 @@ -71,6 +71,7 @@ else: MAXSIZE = int((1 << 63) - 1) del X +from importlib.util import spec_from_loader def _add_doc(func, doc): """Add documentation to a function.""" @@ -186,6 +187,11 @@ class _SixMetaPathImporter(object): return self return None + def find_spec(self, fullname, path, target=None): + if fullname in self.known_modules: + return spec_from_loader(fullname, self) + return None + def __get_module(self, fullname): try: return self.known_modules[fullname] @@ -223,6 +229,12 @@ class _SixMetaPathImporter(object): return None get_source = get_code # same as get_code + def create_module(self, spec): + return self.load_module(spec.name) + + def exec_module(self, module): + pass + _importer = _SixMetaPathImporter(__name__) @@ -679,11 +691,15 @@ if PY3: exec_ = getattr(moves.builtins, "exec") def reraise(tp, value, tb=None): - if value is None: - value = tp() - if value.__traceback__ is not tb: - raise value.with_traceback(tb) - raise value + try: + if value is None: + value = tp() + if value.__traceback__ is not tb: + raise value.with_traceback(tb) + raise value + finally: + value = None + tb = None else: def exec_(_code_, _globs_=None, _locs_=None): @@ -699,19 +715,19 @@ else: exec("""exec _code_ in _globs_, _locs_""") exec_("""def reraise(tp, value, tb=None): - raise tp, value, tb + try: + raise tp, value, tb + finally: + tb = None """) -if sys.version_info[:2] == (3, 2): - exec_("""def raise_from(value, from_value): - if from_value is None: - raise value - raise value from from_value -""") -elif sys.version_info[:2] > (3, 2): +if sys.version_info[:2] > (3,): exec_("""def raise_from(value, from_value): - raise value from from_value + try: + raise value from from_value + finally: + value = None """) else: def raise_from(value, from_value): @@ -788,11 +804,10 @@ _add_doc(reraise, """Reraise an exceptio if sys.version_info[0:2] < (3, 4): def wraps(wrapped, assigned=functools.WRAPPER_ASSIGNMENTS, updated=functools.WRAPPER_UPDATES): - def wrapper(f): - f = functools.wraps(wrapped, assigned, updated)(f) - f.__wrapped__ = wrapped - return f - return wrapper + return functools.partial(_update_wrapper, wrapped=wrapped, + assigned=assigned, updated=updated) + wraps.__doc__ = functools.wraps.__doc__ + else: wraps = functools.wraps @@ -802,10 +817,22 @@ def with_metaclass(meta, *bases): # This requires a bit of explanation: the basic idea is to make a dummy # metaclass for one level of class instantiation that replaces itself with # the actual metaclass. - class metaclass(meta): + class metaclass(type): def __new__(cls, name, this_bases, d): - return meta(name, bases, d) + if sys.version_info[:2] >= (3, 7): + # This version introduced PEP 560 that requires a bit + # of extra care (we mimic what is done by __build_class__). + resolved_bases = types.resolve_bases(bases) + if resolved_bases is not bases: + d['__orig_bases__'] = bases + else: + resolved_bases = bases + return meta(name, resolved_bases, d) + + @classmethod + def __prepare__(cls, name, this_bases): + return meta.__prepare__(name, bases) return type.__new__(metaclass, 'temporary_class', (), {}) @@ -821,6 +848,8 @@ def add_metaclass(metaclass): orig_vars.pop(slots_var) orig_vars.pop('__dict__', None) orig_vars.pop('__weakref__', None) + if hasattr(cls, '__qualname__'): + orig_vars['__qualname__'] = cls.__qualname__ return metaclass(cls.__name__, cls.__bases__, orig_vars) return wrapper
Patching is done using patch -p1
— you should know how.
2023-01-10 - sysctl / modules / load order / nf_conntrack
Recently we ran into an issue where connections were unexpectedly aborted. Connections from a NAT-ed client (a K8S pod) to a server would suddently get an old packet (according to the sequence number) in the middle of the data. This triggered the Linux NAT-box to issue an RST. Setting the kernel flag to mitigate this behaviour required some knowledge of module load order during boot.
Spurious retransmits causing connection teardown
To start off: we observed traffic from a pod to a server get disconnected.
We enabling debug logging on the Kubernetes host where the pod resides.
After enabling modprobe nf_log_ipv4
and
net.netfilter.nf_conntrack_log_invalid=255
, we saw this:
kernel: nf_ct_proto_6: SEQ is under the lower bound (already ACKed data retransmitted) IN= OUT= SRC=10.x.x.x DST=10.x.x.x LEN=1480 TOS=0x00 PREC=0x00 TTL=61 ID=53534 DF PROTO=TCP SPT=6379 DPT=26110 SEQ=4213094653 ACK=3402842193 WINDOW=509 RES=0x00 ACK PSH URGP=0 OPT (0101080A084C76F30D12DCAA)
In the middle of a sequence of several packets of data from the
server, an apparently unrelated packet — it had data, but not
intended for this stream — but with the same
source/destination tuples and yet a sequence number that was more than 80K too low.
(Wireshark flags this packet as invalid with a TCP
Spurious Retransmission
message.)
This triggered a reset (RST) by the Linux connection tracking module. And that in turned caused (unexpected) RSTs from the server.
POD <-> NAT <-> SRV -------------------- <-- TCP seq 2000000 ack 5555 len 1400 <-- TCP seq 2000000 ack 5555 len 1400 <-- TCP seq 1200000 ack 5555 len 1234 (seq is _way_ off) --> TCP RST seq 5555 len 0 <-- TCP seq 2001400 ack 5555 len 1000 <-- TCP seq 2001400 ack 5555 len 1000
(Made up numbers in the above table, but they illustrate the problem.)
At this point, the non-rejected traffic still got forwarded back to the pod. Its ACKs back to the server were now however rejected by the server with an RST of its own — that end of the connection thinks it was tore down already after all.
kernel: nf_ct_proto_6: invalid rst IN= OUT= SRC=10.x.x.x DST=10.x.x.x LEN=40 TOS=0x00 PREC=0x00 TTL=61 ID=0 DF PROTO=TCP SPT=6379 DPT=26110 SEQ=4213164625 ACK=0 WINDOW=0 RES=0x00 RST URGP=0
The next packet (sequence 2001400 in the above example), was fine though. So if we could convince the Linux kernel to ignore the packet with the unexpected sequence number, we'd be good to go.
Luckily there is such a flag: net.netfilter.nf_conntrack_tcp_be_liberal=1
While this does not explain the root cause, setting
said flag mitigates the problem. The kernel now ignores all spurious retransmits.
So, we placed net.netfilter.nf_conntrack_tcp_be_liberal =
1
in /etc/sysctl.conf
and were done with it.
... or so we thought. Because after a reboot, the flag was unset again.
sysctl.conf not picked up?
That's odd. Setting kernel parameters during boot should be done
in sysctl.conf
(or sysctl.d
). Why did it not get picked up?
The cause turned out to be this: this particular setting is not
built-in. It belongs to a module; the nf_conntrack
module.
And that module is not necessarily loaded before sysctl
settings are applied.
nf_conntrack
was loaded on demand, and not in a
particular well-defined order. Luckily, loading modules through
/etc/modules-load.d
is well defined, as you can see:
# systemd-analyze critical-chain systemd-sysctl.service The time when unit became active or started is printed after the "@" character. The time the unit took to start is printed after the "+" character. systemd-sysctl.service +44ms └─systemd-modules-load.service @314ms +90ms └─systemd-journald.socket @242ms └─system.slice @220ms └─-.slice @220ms
Indeed, it sysctl settings are applied after
systemd-modules-load.service
:
# systemctl show systemd-sysctl.service | grep ^After After=systemd-journald.socket system.slice systemd-modules-load.service
So, we can use the kernel module loading to ensure that the conntrack module is loaded before we attempt to set its parameters:
# cat /etc/modules-load.d/modules.conf # /etc/modules: kernel modules to load at boot time. # # This file contains the names of kernel modules that should be loaded # at boot time, one per line. Lines beginning with "#" are ignored. nf_conntrack
And that works. Now all settings are properly set during boot.
As for the spurious retransmissions: the last hasn't been said about that...