~fhusson

Connection to gitlab.com closed by remote host - SSH and IPQoS

The problem

Under Manjaro, I can’t clone a reposity from Gitlab with git.

$ git clone git@gitlab.com/fhusson/xxxx.git
Cloning into 'xxxx'...
Connection to gitlab.com closed by remote host.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights

My ssh key are ok :) and I have the access rights !

TL;DR : The solution with IPQoS

In the file /etc/ssh/ssh_config add the following configuration :

Host *
    IPQoS reliability

Full debug story

Chapter 1 : The fix

If I try directly with ssh :

$ ssh -T git@gitlab.com
Connection to gitlab.com closed by remote host.

Searching similar problem with Google I have found this article that suggest using the ssh option IPQoS=0

$ ssh -o IPQoS=0 -T git@gitlab.com
Welcome to GitLab, @fhusson!

and it works !

\o/

In the OpenSSH article of the Arch Wiki I have found where to set it globally

In the file /etc/ssh/ssh_config add the following configuration :

Host *
    IPQoS reliability

Chapter 2 : Manjaro vs Ubuntu (Debian inside)

Ok but now it works but why did I have this problem with Manjaro and not with Ubuntu ?

If I run the ssh command with more verbose log (-vvv) I have this interesting line : debug3: ssh_packet_set_tos: set IP_TOS 0x20

I have done the same test in Ubuntu and I have this line : debug3: ssh_packet_set_tos: set IP_TOS 0x08

To be sure under Manjaro I tested IPQoS=0x08 and it works too and with IPQoS=0x20 if fail.

Ok next why did I have a different default value for the IPQoS in Ubuntu and in Manjaro.

With man ssh_config for the IPQoS option we have :

  • In Manjaro : The default is af21 (Low-Latency Data) for interactive sessions and cs1 (Lower Effort) for non-interactive sessions.
  • In Ubuntu : The default is lowdelay for interactive sessions and throughput for non-interactive sessions.

I have found this mail/patch from ‘Job Snijders’ where we learn about this change in OpenSSH and here is the link to the commit message

Package version for Manjaro is openssh 8.4p1-2 and ubuntu is openssh-client (1:8.3p1-1)

The change were made in the OpenSSH version 7.8 and we can see in the ubuntu changelog

openssh (1:7.9p1-10) unstable; urgency=medium

  • Temporarily revert IPQoS defaults to pre-7.8 values until issues with “iptables -m tos” and VMware have been fixed (closes: #923879, #926229; LP: #1822370).

– Colin Watson cjwatson@debian.org Mon, 08 Apr 2019 11:13:04 +0100

In the debian/patches/revert-ipqos-defaults.patch we have :

-		options->ip_qos_bulk = IPTOS_DSCP_CS1;
+		options->ip_qos_bulk = IPTOS_THROUGHPUT;

and in the openssh/defines.h we have

/*
 * Definitions for IP type of service (ip_tos)
 */
IPTOS_THROUGHPUT 0x08

/*
 * Definitions for DiffServ Codepoints as per RFC2474
 */
IPTOS_DSCP_CS1   0x20

More information from the debian patch :

The IPQoS default changes have some unfortunate interactions with iptables (see https://bugs.debian.org/923880) and VMware, so I’m temporarily reverting them until those have been fixed. Bug-Debian: https://bugs.debian.org/923879 Bug-Debian: https://bugs.debian.org/926229 Bug-Ubuntu: https://bugs.launchpad.net/bugs/1822370 Last-Update: 2019-04-08

Chapter 3 : More tests

I still need to understand why ‘cs1’ is not working.

  • I have changed my network links with no success :
    • My Computer -> My router with OpenWrt -> The Internet provider box : Failed
    • My Computer -> The Internet provider box : Failed
    • My Computer -> My router with OpenWrt : Failed
  • I have the same error for github.
  • Test with my mobile network over USB : Success

Chapter 4 : To be continued

The next step is to try to open a ticket at my internet provider ^_^

Discuss on Twitter