I solved it all by myself!
The culprit was my Realtek 2.5Gb/s USB NIC.
If you are having a similar problem have a look at the output of dmesg | grep enx
If you see something like this:
[ 2154.282777] r8152 2-1.3:1.0 enx6c1ff719fa24: Tx status -71
[ 2154.286261] r8152 2-1.3:1.0 enx6c1ff719fa24: Tx status -71
[ 2154.289741] r8152 2-1.3:1.0 enx6c1ff719fa24: Tx status -71
[ 2154.293400] r8152 2-1.3:1.0 enx6c1ff719fa24: Tx status -71
[ 2154.592577] r8152 2-1.3:1.0 enx6c1ff719fa24: Promiscuous mode enabled
[ 2165.865585] r8152 2-1.3:1.0 enx6c1ff719fa24: Promiscuous mode enabled
Then chances are it’s the same problem.
Not much you can do about it except recompile the kernel with a more recent driver version. I changed to a 1 Gb/s NIC, it was easier.