Ssh Debugging

[Mick Farmer]

Dear GLLUGers,

I've recently upgraded to FTTC from Plusnet and now get 20+
Mbps download speeds.

The downside appears to be that my ssh link to my work
machine goes down five or six times a day. So, I'm trying
to run sshd in debug mode on my work machine. However, this
is what I get.

[***@penguin log]# /usr/sbin/sshd -d -p 1234
debug1: sshd version OpenSSH_3.9p1
debug1: private host key: #0 type 0 RSA1
debug1: read PEM private key done: type RSA
debug1: private host key: #1 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: private host key: #2 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
debug1: rexec_argv[2]='-p'
debug1: rexec_argv[3]='1234'
debug1: Bind to port 1234 on ::.
Server listening on :: port 1234.
debug1: Bind to port 1234 on 0.0.0.0.
Bind to port 1234 on 0.0.0.0 failed: Address already in use.
Generating 768 bit RSA key.
RSA key generation complete.

Presuably this means that the server is using a default IP
address of 0.0.0.0. How can I overcome this? Or is there
another error waiting in the wings?

Your help is appreciated.

Regards,

Mick /"\
\ /
Linux Registered X ASCII Ribbon Campaign
User #287765 / \ Against HTML Mail
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[Dimitrios Siganos]

Hi Mick,

There is nothing unsual about a server binding to IP address "0.0.0.0"
The zero IP address, in that context, simply means "any". In other
words, the server is happy to receive incoming requests from any interface.

The real error is this:
Bind to port 1234 on 0.0.0.0 failed: Address already in use.

It means that another program is already using TCP port 1234. When it
says address, it means the combination of port and address.

To find out who holds port 1234, do:
netstat -tanp|grep 1234

Regards,
Dimitrios Siganos

On 25/02/12 20:44, Mick Farmer wrote:
> Dear GLLUGers,
>
> I've recently upgraded to FTTC from Plusnet and now get 20+
> Mbps download speeds.
>
> The downside appears to be that my ssh link to my work
> machine goes down five or six times a day. So, I'm trying
> to run sshd in debug mode on my work machine. However, this
> is what I get.
>
> [***@penguin log]# /usr/sbin/sshd -d -p 1234
> debug1: sshd version OpenSSH_3.9p1
> debug1: private host key: #0 type 0 RSA1
> debug1: read PEM private key done: type RSA
> debug1: private host key: #1 type 1 RSA
> debug1: read PEM private key done: type DSA
> debug1: private host key: #2 type 2 DSA
> debug1: rexec_argv[0]='/usr/sbin/sshd'
> debug1: rexec_argv[1]='-d'
> debug1: rexec_argv[2]='-p'
> debug1: rexec_argv[3]='1234'
> debug1: Bind to port 1234 on ::.
> Server listening on :: port 1234.
> debug1: Bind to port 1234 on 0.0.0.0.
> Bind to port 1234 on 0.0.0.0 failed: Address already in use.
> Generating 768 bit RSA key.
> RSA key generation complete.
>
> Presuably this means that the server is using a default IP
> address of 0.0.0.0. How can I overcome this? Or is there
> another error waiting in the wings?
>
> Your help is appreciated.
>
> Regards,
>
> Mick /"\
> \ /
> Linux Registered X ASCII Ribbon Campaign
> User #287765 / \ Against HTML Mail
> --
> Gllug mailing list - ***@gllug.org.uk
> http://lists.gllug.org.uk/mailman/listinfo/gllug

--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[Mick Farmer]

Dear Dimitrios,

Thanks for your reply, but no success, even as root.

I also tried port 5678 and got the same result.

Regards,

Mick /"\
\ /
Linux Registered X ASCII Ribbon Campaign
User #287765 / \ Against HTML Mail
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[Dimitrios Siganos]

Hi Mick,

What's the output of (as root):

netstat -tanp

Regards,
Dimitris

On 25/02/12 21:16, Mick Farmer wrote:
> Dear Dimitrios,
>
> Thanks for your reply, but no success, even as root.
>
> I also tried port 5678 and got the same result.
>
> Regards,
>
> Mick /"\
> \ /
> Linux Registered X ASCII Ribbon Campaign
> User #287765 / \ Against HTML Mail
> --
> Gllug mailing list - ***@gllug.org.uk
> http://lists.gllug.org.uk/mailman/listinfo/gllug

--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[Dimitrios Siganos]

On 25/02/12 21:16, Mick Farmer wrote:
> Dear Dimitrios,
>
> Thanks for your reply, but no success, even as root.
>
> I also tried port 5678 and got the same result.
>
> Regards,
>
> Mick /"\
> \ /
> Linux Registered X ASCII Ribbon Campaign
> User #287765 / \ Against HTML Mail
> --
> Gllug mailing list - ***@gllug.org.uk
> http://lists.gllug.org.uk/mailman/listinfo/gllug

Hi Mick,

After a lot of emails there is still no clear clue as to what is happening.

Iain is right that the logs show that it successfully binds to IPv6 but
fails to bind to IPv4.

At this stage we need to understand your network setup. I suggest you
run the following commands (on both client and server) and post their
output:

1) ip addr

2) ip link

3) ip route

4) netstat -atnp

Another thing we can do is to take ssh out of the equation and see if
basic networking works.

Try on the server:
nc -l 1234

On the client:
nc <server_addr> 1234

Whatever you type on one terminal should show up on the other (after
pressing enter).

That will tell us if you have an ssh related problem or a more basic
networking problem.

Regards,
Dimitrios Siganos
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[Mick Farmer]

Dear Dimitrios,

Absolutely nothing.

[***@lapwing ~]# netstat -tanp|grep 1234
[***@lapwing ~]#

Regards,

Mick /"\
\ /
Linux Registered X ASCII Ribbon Campaign
User #287765 / \ Against HTML Mail
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[Iain Gray]

>>debug1: Bind to port 1234 on ::.
>>Server listening on :: port 1234.
>>debug1: Bind to port 1234 on 0.0.0.0.
>>Bind to port 1234 on 0.0.0.0 failed: Address already in use.

Looks to me like it binds on the IPV6 address then fails on the IPV4

Can you try using -4 in your cmd
sshd -d -4 -p 1234

Just to rule out some weirdness you have going on with your IPV6

Iain

On 25 February 2012 23:31, Mick Farmer <***@dcs.bbk.ac.uk> wrote:

> Dear Dimitrios,
>
> Absolutely nothing.
>
> [***@lapwing ~]# netstat -tanp|grep 1234
> [***@lapwing ~]#
>
> Regards,
>
> Mick /"\
> \ /
> Linux Registered X ASCII Ribbon Campaign
> User #287765 / \ Against HTML Mail
> --
> Gllug mailing list - ***@gllug.org.uk
> http://lists.gllug.org.uk/mailman/listinfo/gllug
>



--
--
www.theblight.org

[Ian Northeast]

Mick Farmer wrote:
> Dear Dimitrios,
>
> Absolutely nothing.
>
> [***@lapwing ~]# netstat -tanp|grep 1234
> [***@lapwing ~]#

Mick, are you running this on the right machine? The server was
"penguin" originally.

Also you might find "lsof -i :1234" useful.

Regards, Ian
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[Mick Farmer]

Dear Ian and others,

Some progress. On my server, penguin, the command

/usr/sbin/sshd -d -4 -p 1234

kicks off without error. However, on my client, lapwing,
the command

/usr/sbin/ssh -4 -p 1234 ***@penguin.dcs.bbk.ac.uk

elicits no response whatsoever.

Regards,

Mick /"\
\ /
Linux Registered X ASCII Ribbon Campaign
User #287765 / \ Against HTML Mail
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[Ian Northeast]

Mick Farmer wrote:
> Dear Ian and others,
>
> Some progress. On my server, penguin, the command
>
> /usr/sbin/sshd -d -4 -p 1234
>
> kicks off without error. However, on my client, lapwing,
> the command
>
> /usr/sbin/ssh -4 -p 1234 ***@penguin.dcs.bbk.ac.uk
>
> elicits no response whatsoever.
>
>
Hi Mick,

Do you mean it hangs, or returns without output?

In the first case, that is normally a firewall issue. In the latter,
strace may help.

ssh is normally in /usr/bin not /usr/sbin.

Regards, Ian
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[JLMS]

On 26 February 2012 23:55, Mick Farmer <***@dcs.bbk.ac.uk> wrote:
> Dear Ian and others,
>
> Some progress.  On my server, penguin, the command
>
>     /usr/sbin/sshd -d -4 -p 1234
>
> kicks off without error.  However, on my client, lapwing,
> the command
>
>    /usr/sbin/ssh -4 -p 1234 ***@penguin.dcs.bbk.ac.uk
>
> elicits no response whatsoever.
>
> Regards,
>
> Mick               /"\

not even in verbose mode? ("-v" switch...) , normally that should
give you some pointers about where it is getting stucked.
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[James Courtier-Dutton]

On 25 February 2012 20:44, Mick Farmer <***@dcs.bbk.ac.uk> wrote:
> Dear GLLUGers,
>
> I've recently upgraded to FTTC from Plusnet and now get 20+
> Mbps download speeds.
>
> The downside appears to be that my ssh link to my work
> machine goes down five or six times a day.  So, I'm trying
> to run sshd in debug mode on my work machine.  However, this
> is what I get.
>
> [***@penguin log]# /usr/sbin/sshd -d -p 1234
> debug1: sshd version OpenSSH_3.9p1
> debug1: private host key: #0 type 0 RSA1
> debug1: read PEM private key done: type RSA
> debug1: private host key: #1 type 1 RSA
> debug1: read PEM private key done: type DSA
> debug1: private host key: #2 type 2 DSA
> debug1: rexec_argv[0]='/usr/sbin/sshd'
> debug1: rexec_argv[1]='-d'
> debug1: rexec_argv[2]='-p'
> debug1: rexec_argv[3]='1234'
> debug1: Bind to port 1234 on ::.
> Server listening on :: port 1234.
> debug1: Bind to port 1234 on 0.0.0.0.
> Bind to port 1234 on 0.0.0.0 failed: Address already in use.
> Generating 768 bit RSA key.
> RSA key generation complete.
>
> Presuably this means that the server is using a default IP
> address of 0.0.0.0.  How can I overcome this?  Or is there
> another error waiting in the wings?
>
> Your help is appreciated.
>

I would do a packet sniff at each end of the link, and see what
happens when the existing ssh session stops working. I assume by "ssh
link goes down", you mean a running ssh session gets cut off.
You might need to turn on some ssh keepalives in order to keep
firewalls and NAT boxes between the endstations happy.
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[Mick Farmer]

Dear Ian,

On further investigation, both sshd and ssh hang without
doing anything interesting.

Regards,

Mick /"\
\ /
Linux Registered X ASCII Ribbon Campaign
User #287765 / \ Against HTML Mail
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[Mick Farmer]

Dear Dimitrios,

I tried your suggestion. Both the server and the client
program started, but no diagnostic output.

At both the server and the client I could type a command,
but no output appeared in either direction.

Regards,

Mick /"\
\ /
Linux Registered X ASCII Ribbon Campaign
User #287765 / \ Against HTML Mail
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[Dimitrios Siganos]

On 27/02/12 23:45, Mick Farmer wrote:
> Dear Dimitrios,
>
> I tried your suggestion. Both the server and the client
> program started, but no diagnostic output.
>
> At both the server and the client I could type a command,
> but no output appeared in either direction.
>
> Regards,
>
> Mick /"\
> \ /
> Linux Registered X ASCII Ribbon Campaign
> User #287765 / \ Against HTML Mail
> --
> Gllug mailing list - ***@gllug.org.uk
> http://lists.gllug.org.uk/mailman/listinfo/gllug

Hi Mick,

You really do need to go into a lot more detail. What exactly did you
try? Can you elaborate?

What about the output from the commands?
1) ip addr
2) ip link
3) ip route
4) netstat -atnp

I don't want to antagonise you but you are not helping us help you.

Dimitris

--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[JLMS]

On 25 February 2012 20:44, Mick Farmer <***@dcs.bbk.ac.uk> wrote:
> Dear GLLUGers,
>
> I've recently upgraded to FTTC from Plusnet and now get 20+
> Mbps download speeds.
>
> The downside appears to be that my ssh link to my work
> machine goes down five or six times a day.  So, I'm trying
> to run sshd in debug mode on my work machine.  However, this
> is what I get.
>
> [***@penguin log]# /usr/sbin/sshd -d -p 1234
> debug1: sshd version OpenSSH_3.9p1
> debug1: private host key: #0 type 0 RSA1
> debug1: read PEM private key done: type RSA
> debug1: private host key: #1 type 1 RSA
> debug1: read PEM private key done: type DSA
> debug1: private host key: #2 type 2 DSA
> debug1: rexec_argv[0]='/usr/sbin/sshd'
> debug1: rexec_argv[1]='-d'
> debug1: rexec_argv[2]='-p'
> debug1: rexec_argv[3]='1234'
> debug1: Bind to port 1234 on ::.
> Server listening on :: port 1234.
> debug1: Bind to port 1234 on 0.0.0.0.
> Bind to port 1234 on 0.0.0.0 failed: Address already in use.
> Generating 768 bit RSA key.
> RSA key generation complete.
>
> Presuably this means that the server is using a default IP
> address of 0.0.0.0.  How can I overcome this?  Or is there
> another error waiting in the wings?
>
> Your help is appreciated.
>
> Regards,
>
> Mick               /"\

I ran the command above on my server and the same IPv4 address
(0.0.0.0) is used, successfully

Did you take the message at its word? Could it be that unknown to you
there is something else in port 1234? (some quarters report that
starting sshd for both IPv6 & IPv4 confuses things, it seems disabling
IPv6 works in some instances where this is doable)
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[JLMS]

On 27 February 2012 23:51, JLMS <***@googlemail.com> wrote:
> On 25 February 2012 20:44, Mick Farmer <***@dcs.bbk.ac.uk> wrote:
>> Dear GLLUGers,
>>
>> I've recently upgraded to FTTC from Plusnet and now get 20+
>> Mbps download speeds.
>>
>> The downside appears to be that my ssh link to my work
>> machine goes down five or six times a day.  So, I'm trying
>> to run sshd in debug mode on my work machine.  However, this
>> is what I get.
>>
>> [***@penguin log]# /usr/sbin/sshd -d -p 1234
>> debug1: sshd version OpenSSH_3.9p1
>> debug1: private host key: #0 type 0 RSA1
>> debug1: read PEM private key done: type RSA
>> debug1: private host key: #1 type 1 RSA
>> debug1: read PEM private key done: type DSA
>> debug1: private host key: #2 type 2 DSA
>> debug1: rexec_argv[0]='/usr/sbin/sshd'
>> debug1: rexec_argv[1]='-d'
>> debug1: rexec_argv[2]='-p'
>> debug1: rexec_argv[3]='1234'
>> debug1: Bind to port 1234 on ::.
>> Server listening on :: port 1234.
>> debug1: Bind to port 1234 on 0.0.0.0.
>> Bind to port 1234 on 0.0.0.0 failed: Address already in use.
>> Generating 768 bit RSA key.
>> RSA key generation complete.
>>
>> Presuably this means that the server is using a default IP
>> address of 0.0.0.0.  How can I overcome this?  Or is there
>> another error waiting in the wings?
>>
>> Your help is appreciated.
>>
>> Regards,
>>
>> Mick               /"\
>
> I ran the command above on my server and the same IPv4 address
> (0.0.0.0) is used, successfully
>
> Did you take the message at its word? Could it be that unknown to you
> there is something else in port 1234? (some quarters report that
> starting sshd for both IPv6 & IPv4 confuses things, it seems disabling
> IPv6 works in some instances where this is doable)

Forgot to add a link to a discussion of this matter which seems
useful: http://blog.wains.be/2006/03/30/howto-fix-bind-to-port-22-on-0000-failed-address-already-in-use-error/
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[t.clarke]

Re ssh bind to port 1234 problem;

I have only taken a brief glance at this thread, but it seems to me that if
the server is saying it cannot bind to the port, there are only really three
possibilities:-

1 - most likely - another process is bound to the port; the netstat -p command
I believe will show which processes are bound to which ports.

2 - the server is giving a misleading message; I would imagine it can be placed
in full debug mode to capture messages as to exactly what it is doing (maybe
possible to trace the system calls?)

3 - something wrong with the kernel's IP stack (highly unlikely!)

Tim
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[James Courtier-Dutton]

On 28 February 2012 07:45, t.clarke <***@seacon.co.uk> wrote:
> Re ssh bind to port 1234 problem;
>
> I have only taken a brief glance at this thread,  but it seems to me that if
> the server is saying it cannot bind to the port, there are only really three
> possibilities:-
>
> 1 - most likely - another process is bound to the port;  the netstat -p command
> I believe will show which processes are bound to which ports.
>
> 2 - the server is giving a misleading message; I would imagine it can be placed
> in full debug mode to capture messages as to exactly what it is doing (maybe
> possible to trace the system calls?)
>
> 3 - something wrong with the kernel's IP stack  (highly unlikely!)
>

But, I don't think this is a configuration problem at either end.
The OP said the ssh link goes down 5-6 times a day, so it works most
of the time.
The only thing that has changed is the WAN link.
The binding problem is most likely a red herring.
My guess is that the real problem is most likely to be due to some
deep packet inspection/policy being applied by the ISP.
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug

[Andrew Farnsworth]

On Tue, Feb 28, 2012 at 10:23 AM, James Courtier-Dutton <
***@gmail.com> wrote:

> On 28 February 2012 07:45, t.clarke <***@seacon.co.uk> wrote:
> > Re ssh bind to port 1234 problem;
> >
> > I have only taken a brief glance at this thread, but it seems to me
> that if
> > the server is saying it cannot bind to the port, there are only really
> three
> > possibilities:-
> >
> > 1 - most likely - another process is bound to the port; the netstat -p
> command
> > I believe will show which processes are bound to which ports.
> >
> > 2 - the server is giving a misleading message; I would imagine it can be
> placed
> > in full debug mode to capture messages as to exactly what it is doing
> (maybe
> > possible to trace the system calls?)
> >
> > 3 - something wrong with the kernel's IP stack (highly unlikely!)
> >
>
> But, I don't think this is a configuration problem at either end.
> The OP said the ssh link goes down 5-6 times a day, so it works most
> of the time.
> The only thing that has changed is the WAN link.
> The binding problem is most likely a red herring.
> My guess is that the real problem is most likely to be due to some
> deep packet inspection/policy being applied by the ISP.


It could also just be a much simpler "drop all connections after XXX
minutes" configuration in the ISP (or middle tier ISP). I would try
logging around this and see if all the disconnect times are approximately
the same number of minutes after the connect occurs. If this is the case
you can either raise it with the ISP to get it fixed (good luck), switch
ISPs, or code around it.

Andy

[John Winters]

On Tue, 28 Feb 2012 11:05:03 +0100, Andrew Farnsworth
<***@stonedoor.com> wrote:
[snip]
>
> It could also just be a much simpler "drop all connections after XXX
> minutes" configuration in the ISP (or middle tier ISP). I would try
> logging around this and see if all the disconnect times are
approximately
> the same number of minutes after the connect occurs. If this is the
case
> you can either raise it with the ISP to get it fixed (good luck), switch
> ISPs, or code around it.

It could also be worth setting:

ServerAliveInterval 30

in /etc/ssh/ssh_config on the client, making it specific to the affected
server. I find an increasing number of firewalls drop apparently unused
connections after comparatively little time.

John
--
Gllug mailing list - ***@gllug.org.uk
http://lists.gllug.org.uk/mailman/listinfo/gllug