FreeBSD Password Login Timeout Error

When the online WebShell executes password login to FreeBSD 13.0 64-bit, it reports a timeout error, which prompted me to start investigating. Here’s the analysis:

Troubleshooting

1. The direct cause of the error is SSH login timeout. After examining the SSH2 module and its specific call logic, I found two possibilities that could cause timeout:

   1. When executing keyboard-interactive login, if the user never provides input, it will timeout. If the network is disconnected, the error would be ECONNRESET
   2. If the SSH handshake server never responds, it will trigger a timeout

2. According to the current client login logic, although the client tries multiple login methods in rotation, keyboard-interactive login is enabled in the current logic, but it can only be triggered when public key login fails

3. Testing in an environment consistent with the online program reproduces the issue, and the production environment consistently reproduces it. Local terminals like iTerm2 direct login do not reproduce this problem.

4. Trying to use the ssh command directly on the production server

   Specific timing tests revealed that when testing direct command-line ssh connections on the production server, after 20s+, it reports ssh_exchange_identification: read: Connection reset by peer. Comparing this with the default timeout mentioned above, this makes sense.

   1. Production Environment
   
   2. Test Environment

   

Based on this, it’s basically confirmed that it’s related to environment differences

Analysis

After understanding the current production and test environments, one obvious difference is that the test environment machines have direct internet access, while production does not. Using this obvious difference as an entry point, I finally consulted system/network experts and figured out the problem:

1. Packet capturing of the SSH handshake process in production/test environments revealed that in the production environment, ACK packets during TCP handshake do not carry timestamps, while the pre-production environment does

2. FreeBSD is strictly designed according to the RFC1323 PAWS mechanism. When both SYN and SYN+ACK steps contain timestamp options, the PAWS mechanism is enabled. If the third ACK packet doesn’t contain a timestamp, it’s directly dropped

3. FreeBSD systems have timestamp checking enabled by default, while other systems like Ubuntu/CentOS don’t have it enabled, or don’t have this setting

   You can view related configurations using the following command:

   sudo sysctl -a | grep net|grep rfc
   

4. In FreeBSD 12.3, net.inet.tcp.tolerate_missing_ts defaults to 1, while in version 13 it defaults to 0

Solution

1. FreeBSD-side modification

   # Temporary modification
   sysctl -w net.inet.tcp.tolerate_missing_ts=1
   
   # Permanent modification
   sysctl.conf 
   net.inet.tcp.tolerate_missing_ts=1
   
   /etc/rc.d/sysctl reload
   

   This setting ignores timestamp validation after configuration

2. Client-side modification (current WebShell client side)

   net.ipv4.tcp_wan_timestamps=1, and also ensure net.ipv4.tcp_timestamps=1

   This carries timestamps even on non-public networks

Ultimately, we chose solution 1. Solution 2 was rejected because the current service is deployed in containers and doesn’t have this setting. It would require modifying the host machine and changing network configurations, which poses high risks.

Testing the above solution confirmed that the problem was resolved.

Supplement

Command Line Scripts

The terminal SSH connections mentioned above can be used to debug the login authentication process. Here are the scripts:

ssh -p 22 -v root@ip

ssh -p 22 -v -i privateKey  root@ip

Packet Capture Tools

1. tcpdump
2. Analysis tool: wireshark

SSH Server Login Authentication Configuration

Whether keyboard-interactive login will be attempted depends not only on client control but also on whether the server has it enabled. Different systems have different default support for login methods. Ubuntu/CentOS systems have keyboard-interactive disabled by default, while FreeBSD has it enabled.

Configuration documentation: https://www.freebsd.org/cgi/man.cgi?query=ssh_config

ChallengeResponseAuthentication yes

Final Thoughts

The problem is now resolved. In addition to the SSH handshake failing due to timestamp issues, it’s also important to note that SSH login itself will attempt multiple authentication methods.

Related Issues

• https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=250499
• https://github.com/mscdex/ssh2/issues/393
• 关于 TCP 协议中的 timestamp 问题