Checked up on system throughput.
WAN2 (8Mbps ADSL) 3703
WAN3 (2Mbps SDSL) 0534/1308/1564
WAN4 (8Mbps ADSL) 5306

Is WAN3 Intermittant?

Tue 25th Nov, had message that some users were complaining. Checked system remotely, WAN2, 3 & 4 all seemed to be passing a good amount of data. Arranged to meet up with Lee Vobes to show me the problem.

He explained all complaints seem to be linked to DSLAM 10. (IP address 10.25.1.19). I could do telnet management OK. Could see from lights users were connected.

Went up to one of the rooms with complaint. PC was not getting DHCP. Re booted DSLAM 10, re tested user port, now getting DHCP OK.

BT have connected 8Mb ADSL to new phone line. I went in and installed a Draytek router.
It is installed as a router not a bridge like the other two ADSL lines.

20th Nov, CX reported a problem. WAN1 had been disconnected due to dogey ADSL line. The Draytek router showed WAN3 and WAN4 had been automatically disconnected. The test to check the line was up was to ping www.bbc.co.uk
It has stopped responding to pings. Changed pings to check www.batchworth.org lines came back up.

Yesterday, Lee reported phone calls from 5 people saying there was a problem with their Internet connection. I went into CX and saw the ADSL light on WAN1 was green then red, then green.

I took WAN1 out of the Load sharing group.

WAN1 ADSL connected at G.DMT not ADSL2+
Rx Connection speed 3520Kbps (WAN2 7101Kbps)
Tx Connection speed 832Kbps (WAN2 1057Kbps)
ATU-R SNR margin 12.0 dB (WAN2 9.0 dB)
ATU-C SNR margin 7 dB (WAN2 12 dB)
ATU-R Attenuation 27.5 dB (WAN2 20.5 dB)
ATU-C Attenuation 41.5 dB (WAN2 31.5 dB)

Could log on to Draytek via WAN2 and WAN3 but not WAN1. Can log into WAN1 Router, it says it is connected. Restarted WAN1 Router.
Reconnected to ADSL OK (7101Kbps and 1041Kbps), but still can can't Log into Draytek. I think this is no return path screw up because of Be strange subnet arrangement.

WAN2 (set to 80%) passing 2600 packets, WAN3 (set to 20%) passing 2,200
pings to www.bbc.co.uk between 40 - 260mSec and timeouts.
at 19.30 WAN2 3300, WAN3 2,800

WAN1 failed, set system to run on WAN2 only.

Checked system OK.
WAN1 passing 800 packets, WAN2 passing 550 packets every 10 seconds.
50/50 single digit/double digit some 3 digit ping times to www.bbc.co.uk

Q went into CX to meet BT engineer, line fault on cable to exchange, they swapped the pair at the exchange and and at the CX end, line started working. Back up to 7Mb down / 1Mb up.

WAN1 passing 1000 packets, WAN2 passing 300 packets.

Checked by going onto Google and following links.

Checked with two lots of pings to www.bbc.co.uk mainly single digit.

Mick ordering adsl from BT to add to the loadbalancer.

System OK, LB set to run 20% on WAN3 SDSL, 80% on WAN1 ADSL.
WAN1 4100 packets, WAN3 400 packets
Ping times to www.bbc.co.uk sometimes single digit, sometimes timed out.

Finished insulating overhead 4" Heating pipes that heat up the room where all the DSLAMs are situated. Equipment running in very high temperatures are more likely to fail.

Hopefully this will eliminate the need to build partitions to section of a part of the room and air condition it.

Mon - Lee called to say the system had been down over the weekend. I could log onto WAN1 OK, but could not log onto WAN2 (I got onto the terminal server connected to its Async port OK but it said PO=0 busy). I set load balancer to use just WAN1.

Later Graham phoned to say system down again. I could still log onto WAN1 OK, it reported the ADSL connection was OK, I rebooted it anyway as I had just had a similar problem at home where my router needed rebooting as it had a good DSL link, but was not getting to DNS.

On site I plugged direct into the WAN2 AR440S async port, it spewed out loads of:
Info (1045274): This device is locked temporarily (login-lock). (This probably indicates attempted hacker activity).
After this it allowed me to login as manager. The receive rate was 2Mbps the Tx rate 352K.

I rebooted the WAN2 AR440S router, it was them going up and down, but max connection rate 2Mb.
I plugged into the AR440S used as a Terminal server, this also spewed out Info (1045274): This device is locked temporarily (login lock).
Found that line 020 8741 0055 had no dial time. Connection to ADSL must have been crosstalk.

By 17:00 WAN1 was doing 1900 packets every 10 seconds.
Ping times to www.bbc.co.uk were generally single digit.

System OK, WAN1 passing 2000 packets, WAN2 passing 1400 packets every 10 seconds
ping times to www.bbc.co.uk about half single digit, half double digit.

System OK, WAN1 passing 2700 packets, WAN2 passing 1800 packets.
Mainly single figure ping times to www.bbc.co.uk

System OK, WAN1 passing 2300 packets, WAN2 passing 1100 packets.

System OK, WAN1 passing 1400 packets, WAN2 passing 2300 packets.
About equal single digit and double digit ping times.

Installed two AR440S Business Class ADSL routers.
WAN1 passing 4200 packets, WAN2 passing 1600 packets every 10 seconds

System OK, WAN1 passing 2,200 packets, WAN2 passing 1000 packets.
Mainly single digit ping times to www.bbc.co.uk

Router had failed, used Remote power cycle to restore connection.

Checked the system, (running ping tests to Router Real IP addresses) both WAN ports showing connected on Loadbalancer, but Could not Load Google inbox.
Used Remote Power Cycle Could then load Google inbox.

Looked like WAN1 failed, reset it with remote power cycle.
Set Loadbalancer to ping real IP address of Router as check for port being up.
WAN1 now passing 200 packets, WAN2 now passing 500 packets.

WAN1 failed, Blogspot failed to load. Used remote power reset, WAN1 router came back OK.
WAN1 passing 1300 packets, WAN2 passing 1100 packets.

Checked OK

WAN1 passing 900 packets, WAN2 passing 1300 packets every 10 seconds

WAN1 passing 2000 packets, WAN2 passing 1800 packets every 10 seconds.
Mainly single digit ping times to www.bbc.co.uk

WAN1 passing 1300 packets, WAN2 passing 1000 packets every 10 seconds

WAN1 passing 1000 packets, WAN2 passing 1800 packets every 10 sec.

WAN1 passing 1800 packets, WAN2 passing 2600 packets every 10 sec.

WAN1 passing 1000 packets, WAN2 passing 2800 packets every 10 seconds.

WAN1 passing 1000 packets every 10 seconds, WAN2 passing 800 packets every 10 seconds.

WAN1 passing 200 packets, WAN2 passing 1200 packets every 10 seconds
Some 1mS ping times, some double digit to www.bbc.co.uk

WAN1 passing 2300 packets, WAN2 passing 1200 packets

WAN1 passing 300 packets, WAN2 passing 1500 packets every 10 seconds.

WAN1 passing 500 packets, WAN2 passing 200 packets every 10 seconds.
ping times mainly single digit.

WAN1 passing 2000 packets, WAN2 passing 2000 packets

WAN1 passing 900 packets, WAN2 passing 900 packets
mainly single digit ping times

WAN1 passing 1500 packets, WAN2 passing 1000 packets every 10 sec
Mainly single digit poing times to www.bbc.co.uk

WAN1 passing 1000 packets, WAN2 passing 4000 packets every 10 sec

WAN1 4000 packets, WAN2 2000 packets every 10 seconds
about equal one and two digit ping times to www.bbc.co.uk

WAN1 1100 packets, WAN2 2000 packets in 10 seconds

System OK
WAN1 passing 2000 packets, WAN2 passing 3000 packets

WAN1 failed, restarted with remote power cycle OK.
WAN1 600 passing packets, WAN2 passing 800 packets
Mainly single digit ping times.

WAN1 failed, WAN2 passing 3000 packets. Used remote power cycle to restart.
WAN1 passing 2000 packets, WAN2 passing 2000 packets.
Mainly two digit ping times, some 3 digit

WAN1 3000 packets, WAN2 passing 3000 packets every 10 seconds
few more double digit ping times than single digit

System failed on WAN1.
Could ping www.bbc.co.uk OK, but not ping www.batchworth.org simultaneously.
Did remote power cycle, system returned to norm.

WAN1 passing 1200 packets, WAN2 passing 3000 packets every 10 seconds.
More single digit ping times than two digit ping times.

2ooo packets passed WAN1, 2000 packets passed WAN2 every 10 seconds.
mainly single digit ping times to www.bbc.co.uk

System OK

600 packets WAN1, 800 packets WAN2 every 10 seconds.
Mainly single digit ping times

WAN1 2000 packets, WAN2 2500 packets in 10 seconds
Mainly one digit ping times to www.bbc.co.uk

WAN1 and WAN2 each passing 2000 packets every 10 seconds.
Mainly single digit ping times

System OK, WAN1 passing 5000 packets, WAN2 passing 4000 packets every 10 sec.
Every ping request timed out to www.bbc.co.uk
A few seconds later, pings were 50 50 single digit and double digit
WAN1 still passing 5000 packets, WAN2 passing 3500 packets every 10 sec

Sytem OK, WAN1 2000 packets, WAN2 2000 packets every 10 seconds

3000 packets WAN1, 1800 packets WAN2

3000 packets passed WAN1, 4000 packets passed WAN2 in 10 seconds
Double digit ping times, mainly below 50mSec

WAN1 1500 packets , WAN2 1000 packets every 10 sec
Low teen ping timesto www.bbc.co.uk

System OK, WAN1 and WAN2 passing 2000 packets every 10 seconds.
About 50 50 single digit / double digit ping times to www.bbc.co.uk

WAN1 passing 1000 packets, WAN2 passing 2000 packets every 10 sec
ping times mainly single digit to www.bbc.co.uk

WAN1 and WAN2 both passing 3000 packets every 10 seconds.
ping times 50% single digit, 50% double digit to www.bbc.co.uk

1000 packets WAN1, 2000 packets WAN2
mainly single digit ping times

800 packets passed WAN1, 1400 packets passed WAN2 every 10 seconds.
About 50% pings single digit, 50% double digit to www.bbc.co.uk

WAN1 2000 packets every 10 sec, WAN2 4000 packets every 10 sec.
Pings to www.bbc.co.uk, a few more single digit than double digit.

1000 Packets WAN1, 3000 packets WAN2
Mainly single Digit Ping times

800 Packets per 10 sec WAN1, 1000 packets per 10 sec WAN2
Still some sub 1mSec ping times even via VPN.
Found a DSLAM 10.25.1.16 was not pingable

System OK
WAN1 500 passing packets, WAN2 1000 packets per 10 seconds
50/50 1 digit /2 digit ping times to www.bbc.co.uk

System OK.
So little traffic running this morning I went to site to see why. Everything seems OK.
Only 500 packets WAN1 and 500 packets passing WAN 2 every 10 sec.
pings (from site, not via vpn) mainly sub 1 mSec.

System OK, WAN1 800 packets, WAN2 2000 packets every 10 seconds.
Mainly 2 digit ping times, occasional three digit to www.bbc.co.uk

Checked system, WAN2 failed. WAN1 passing 4000 packets every 10 seconds.
pings to www.bbc.co.uk looked OK, mainly two digit. Was having trouble bringing up blog.
Used remote power cycle, WAN2 resumed service.
WAN1 now passing 1500 packets, WAN2 passing 2000 packets every 10 seconds.

System OK, WAN1 1000 packets evey 10 sec, WAN2 2500 packets every 10 sec.
Mainly two digit ping times to www.bbc.co.uk, some three digit, no timeouts.

System OK, WAN1 & WAN2 running about 1500 packets every 10 sec
Mainly two digit, occasional 3 digit ping times to www.bbc.co.uk
Not had a WAN1 router failure since reloading its operating software 8 days ago (9th June)
we were getting a WaN1 failure typically every day.

System OK, WAN1 passing about 1200 packets WAN2 passing 2000 packets.
Some single digit ping times, mainly double digit ping times to www.bbc.co.uk

System OK, about 2500 packets every 10 sec on WAN1 & WAN2.
Mainly low two digit ping times, some 3 digit.

System OK, WAN1 100 packets every 10 sec, WAN2 1000 packets every 10 sec.
Two digit ping times to www.bbc.co.uk

System OK, 2500 packets WAN1, 2500 packets WAN2 every 10 seconds.
Mainly 2 digit ping times to www.bbc.co.uk

System OK, WAN1 2000, WAN2 2500 packets every 10 sec
2 digit ping times to www.bbc.co.uk

System OK, WAN1 1500 packets every 10 sec, WAN2 3000.
Low two digit ping times to www.bbc.co.uk

System OK, WAN1 and WAN2 passing about 1400 packets every 10 seconds.
About 50/50 single digit ping times / double digit ping times.

System OK, WAN1 2000 packets every 10 sec, WAN2 passing 1800 packets every 10 sec.
Mainly single digit ping times to www.bbc.co.uk

System OK, WAN1 1500, WAN2 2000 packets every 10 sec

System OK, WAN1 3000 packets, WAN2 2000 packets every 10 seconds

13.52 running only about 1000 packets every 10 seconds on each WAN1 & WAN2

System running OK, 2000 packets every 10 seconds on WAN1 and WAN2

System Running OK, 3000 packets every 10 sec WAN1 and WAN3

Running OK 3000 packets every 10 sec on WAN1 and WAN2

I requested replacement router from Be, but they say it must fail whilst running the latest code before they will replace it. It is running latest code. Be said we have to re load the latest code in case the latest code was slightly corrupt. Sigh! Went to site today and did the slightly pointless excersize of reloading the latest code on WAN1 router.

System running OK, 1500 packets on WAN1 and WAN2

I checked the blog for router failures:
25th May WAN1 fail 22.48
26th May WAN1 fail 12.oo
27th May WAN1 fail 20.20
2nd June unspecified 14.00
8th June WAN1 13.50
9th June WAN1 22.45
I had thought the failures were on both routers - this indicates it could be a problem with hardware of just one router. I called Be they requested I update the firmware before they replace the hardware.

Checked at 22.45, WAN1 failed, remote power cycle restored it to operation.
WAN1 and WAN2 now passing 2000 packets every 10 sec.
As at 20.43 ping times largely single digit.

WAN1 and WAN2 passing 2000 packets per second each.

WAN1 passing 2000 packets per second, WAN2 passing 1000 packets per second.

System working well. WAN1 passing 1500 packets every 10 sec, WAN2 passing 2000 packets every 10 sec. Loadbalancer set to share load 50/50.

Checked system Sunday at 13.50, WAN1 had failed successfully reset with remote power cycle, rebooted Draytek to make it easier to see if stats on WAN1 and WAN2 balance.

System checked at 11.20am. WAN1: 4,000 packets every 10 seconds. WAN2: 2,000 packets every 10 seconds.

Checked system 12.30pm Big demand on both lines, 6000 packets per line. Ping times to www.bbc.co.uk from Draytek console running between 10-30mSec with occasional unresolved domain names
I guess customers getting service a bit lumpy but OK.
WAN1 111474422 packets service up 1day 20 hours.
WAN2 119152798 packets, service up 1 day 20 hours.

Graham phoned Quent around 2pm to report problem on the network. I talked him through the remote power cycle. System restarted OK. I checked the system at about 3.30pm OK, checked the system at 11.30pm OK.

System looks good, mainly single digit ping times.

Looks good

System looked good all day.

22.04 was watching system with reasonably low ping times when suddenly all pings timing out.
Checked Draytek, it was doing 8000 packets every 10 sec on one line and 5000 packets every 10 sec on the other. Definatly need policy based routing!

Checked at 20.25
Longest ping times I've ever seen on the system mainly 50-80mS, but not too many ping timeouts. Draytek reports about 3000 packets every 10 seconds on both "Be" lines. I think this must be a user downloading a big file.

Checked OK 08.00
Checked OK 10.00
Checked OK 11.51

Checked system throughout the day, all running well. Mainly single digit mS times in ping (Loadbalancing over both Be lines).

Checked system 20.20
WAN1 had stopped, WAN2 still going, (Load balancer made users experience impossible) used remote power cycle, System running OK.

09.00 Tue checked system, WAN1 (only WAN line) had failed. It restarted with remote power cycle.
10.30 arrived on site, discovered WAN2 switched off and disconnected.
Had to log into WAN2 router and reconfigure it, System now loadbalancing WAN1 and WAN2.

Checked service most of the day - all OK, by 21.30 line so busy some pings failing, not too bad generally get 20 or 30 pings through before you get a failure.

Checked Mon 07.30 OK
Checked Mon 12.00 WAN1 failed, remote power cycle restored it.

Checked OK 20.01 watched off and on over the evening.
Checked 22.48 WAN1 line failed, used remote power cycle, returned back to 4000 packets per second.

Checked OK 18.14 (WAN1 only) 4000 packets every 10 seconds. Ping responses mainly single digit - ie customer experience good.

Checked OK at 22.30
Checked OK at 12.18

Checked system 19.00, it had been using just one "Be" line (other one would not return with remote power cycle). The one working "Be" line had failed, remotely power cycled it - it returned.
(re booted Loadbalancer). System back working on one line.

Sat morning one "Be" line doing about 4000 packets every 10 seconds. Some timeouts.

Checked system Fri night, Spitfire line failed under load, one "Be" line down. Ran system on one "Be" line. When checking to www.bbc.co.uk there were ping timeouts every 4-5 seconds, so users would have been getting a slow service.
There are two failure modes of "Be" routers, one where the router comes back after power cycling - one where it does not. At the moment we have the latter. Neither are acceptable.

I think the failures are down to the routers. I rang "Be" tech support to check if either failure mode is known / common by tech support, thinking they might be able to indicate whether it is worth replacing the "Be" router with the same model (this never normally happens) or with a more reliable brand of router (this happens whenever the line is under a lot of stress).

One thing occurs to me that if the BBCi player becomes popular amongst Charing cross residents we will be have to switch back to some sort of policy based routing.

Checked system again this morning seemed to be running OK.

I understand there were some problems this last week. When a "Be" Internet line went down and would not reset.
Kugan left the system load balancing between a 16Mb ADSL and an SDSL.
Both the Be lines and he spitfire line are proving troublesome.
I checked the system and had to reset it again.

System running OK, about 2000 packets every 10 sec on WAN1 same on WAN2.

Checked system, WAN2 running at about 2000 packets every 10 seconds. WAN1 not passing packets. Rebooted Loadbalancer with WAN1 WAN2 fixed 50% 50% rather than auto weighting.
WAN1 still not passing packets. Rebooted WAN1 WAN2 Routers, system running normally.

System checked OK, running at about 1000 packets every 10 seconds for WAN1 & WAN2

Checked system, looks OK, running about 1000 packets every 10 seconds out of WAN1 & WAN2

Graham phoned to say user reported system running slowly. He has checked Internet access to web pages, he said it seemed fast.

Quent checked the system, WAN1 and WAN2 running about 1000 packets every 10 seconds. Looks fine.

Rebooted Loadbalancer because of imbalance between WAN1 and WAN2, set auto weighting.
System still putting more traffic down WAN1 than WAN2.
Not rebooted WAN Routers this moring.

07.55 checked the system, Internet seemed to respond OK, but as last night much more traffic on WAN1 than WAN2.

Checked System at about quater past midnight. Stats shows data passing on WAN2 but not on WAN1. Since the Loadbalancer can not detect failure of WAN1 this would have given poor to no service to the users. Number of packets passed by the interface showed that the system must have been down for several hours. Reset the system using the new Router reset device. System back up again about 25 past midnight.

Fitted device so can remotely reboot WAN1 & WAN2.
Checked Loadbalancer by logging in and doing ping to www.bbc.co.uk
All three WAN ports failed ping test.
Had to reboot Loadbalancer to set it back to balancing between WAN1 and WAN2.
Load balancer started pinging out on WAN1 (as expected) failed to ping out on WAN2 (as expected because Be lines both in same subnet).
Loadbalancer would not ping out on WAN3 for a few minutes. Eventually WAN3 pings restarted without rebooting the router. It could be too much load on WAN3 made it fail.

Checked System, no packets out of WAN1 or WAN2, rebooted Loadbalancer to use WAN3 only.
Logged in again. WAN 3 now passing 2000 packets every 10 sec.
Could not ping from WAN1,2 or 3. Could be that WAN3 is too slow to respond in time as so busy?

WAN1 Rx passing approx 1,400 packets every 10 seconds. Up time 3 days 12 hrs. 43,620,505
WAN3 Rx passing approx 2,500 packets every 10 seconds. Up time 3 days 12 hrs 47,096,512
Tx and Rx fairly well matched on each line.
Only been loadbalancing between WAN1 and WAN2 (16Mb lines)

WAN1 Rx passing approx 1,400 packets every 10 seconds. Up time 2 days 22 hrs. 36,184,505
WAN3 Rx passing approx 2,500 packets every 10 seconds. Up time 2 days 22 hrs 39,251,666
Tx and Rx fairly well matched on each line.
Only been loadbalancing between WAN1 and WAN2 (16Mb lines)

System working well.

System working well

System working well

System working well.

Charing Cross Staff Internet checked all day yesterday and this morning, all seem to be working, and working fairly fast.