The Roving Network Engineer's blog..

EIGRP Feasible Successor routes

2010-07-28T21:03:00.000+01:00

I always forget what these means in-between exams (I simply don't use EIGRP anywhere), but it is quite important to understand EIGRPs computation logic. The logic is this - when looking at possible 'backup' routes to hold ready for use, it's important to avoid the possibility of considering a path which has come from a neighbor which is going to send the packet back to you in the end - i.e. a routing loop. Without the 'global view' of link state protocols - you need to have a way to determine what inbound advertisements are 'safe' and which ones are 'maybe not safe'.

If you've a backup route which you know is safe, you can switch over really quickly in the event of the loss of your primary link. If you don't - you want to take a little time and be sure that you're not going to introduce a loop. You need to wait for the network to re-converge. If you've your head around that - the rest is just terminology.

First key term is 'Feasible Distance'. This is basically the metric of the current 'best' route (i.e. the live one), including any metric elements that were added by the local router. The 'whole cost' if you like. Let's make up a number and say it's 10,000. 'Reported Distance' of an inbound route advertisement is the metric that the neighbor is passing to us, before we add on any metric ourselves.

There are two possibilities for the Reported Distance, it's either more or less than our current 'best' metric - the Feasible Distance. If it's more, say 20,000, then it's possible that that neighbor is actually advertising our own advertisement back to us - i.e. a loop. Maybe it's not, but until the network has re-converged, we have to consider the possibility. If the RD is less, then it is not possible that it's going back via us. This cannot be a loop, and therefore it's safe to quickly switch to this link.

Feasible Successors are quite simply an alternative path to the live route (the 'Successor'), which has a lower reported distance than the current feasible distance - i.e. backup routes which we know are not a loop. If a router looses it's main route, and has a feasible successor, it simply promotes it to the 'live' route, and sends out updates to it's neighbors to tell them the metric has changed.

If it doesn't have a feasible successor, then the router has no quick backup path, marks the route as 'active' (a counter intuitive tern which means the router is actively trying to work out how to get to the route in question), and starts querying it's neighbors to see if any of them know an way to get to the destination. Either it'll get an answer back with a loop free route, which will be installed as the new successor (i.e live route), or it'll give up and remove the route from the routing table..

Why EIGRP hello and hold timers don't have to match

2010-07-28T17:20:00.000+01:00

Lets take a simple topology. Two routers, R1 and R2, on a common LAN. Assume basic EIGRP is up. One morning you decide to start 'optimizing' the protocol by adjusting timers, but get half way through it and get bored and go for tea :

(both)

router eigrp 100
network 150.150.0.0

R1#sh run int f0/0
!
interface FastEthernet0/0
ip address 150.150.36.3 255.255.255.128
!! default EIGRP timers of hello (5) and holdtime (15) apply
!
end

R2#sh run int f0/0
!
interface FastEthernet0/0
ip address 150.150.36.126 255.255.255.0
ip hello-interval eigrp 100 1
ip hold-time eigrp 100 3
end

R1's hellos are only going to be sent out every 5 seconds, which is longer than R2's hold time of 3 seconds, so it's going to break right? No, actually it isn't. In EIGRP, when a router exchanges hellos with a neighbor, it looks at the timers in the inbound hello, and expects packets at that rate.

So in the scenario above, R1 knows to expect packets every second from R2, and applies a hold time of 3 seconds, even though it's sending it's own hellos out ever 5 seconds. Vice versa, even though R2's interface is configured with a hold-time of only 3 seconds, it knows to expect hellos from R1 every 5, and to apply a hold time of 15.

I don't know if this is a good thing, as it basically makes EIGRP very forgiving of bad configuration, but I've no doubt that if you look hard enough, you'll find neighbor relationships relying on this behavior. Fun eh!

How resilient does your DC network need to be?

2010-07-26T10:00:00.001+01:00

We had a great conversation this weekend on packetpushers about how resilient you should make your datacentre switching. The guts of the conversation is on your core chassis devices, do you need to put in multiple supervisors/PSUs/ETC.

Now, the real answer to these kinds of questions is always a varient on 'it depends on the requirements'.
Ethan Banks was making the point that there are networks where a two second loss is a big deal. I guess we're primarily talking about trading houses/banks/call centres etc here, and in those places it's a fair point.

For pretty much anyone else, the first question I'd ask is 'How much money is it worth to avoid a 3 second loss once a year'. Why those numbers? Assuming you're doing something vaguely sensible with Rapid Spanning tree and/or your L3 routing protocol of choice, you should expect an unplanned device loss to be recovered in that time. Why once a year? If you're losing devices more often than that, you probably have an environmental issue you need to fix first.

So what's the significance of the 3 second drop? Well, of course you need to test in your environment, but you should expect TCP connections to stay up (or you can tweak your TCP stacks to ensure they will). SMB transfers may drop, VOIP calls will drop. Storage calls will fail. These things should all recover*. I'm sure you can think of a few more things.

* Yeah I know the VOIP call won't 'recover', but you can call them back. As long as it's not a regular even, this is usually an acceptable risk. As for the rest, test your applications, and see what will recover, and what dies horribly. The 'dies horribly' list is a set of applications which are the drivers for 'hitless redundancy'. Make sure it's made clear to the business that these are the apps which are responsible for the extra cost.
So once we have our understanding of consequence, we can take our techie hat off, and we need to start thinking like a business person. What is the consequence of the occasional drop worth to the business. Throw it out to your internal customers - probably some of them will jump up and down and tell you they can't tolerate any loss - it must be up all the time. That's fine. Make them come up with a number - what will this failure scenario cost. Compare this cost to the price from Cisco/Juniper/Whoever for the extra kit, and this is your business case, one way or the other.

One useful trick (and it depends on how internal financing works in your company) is you budget to build your network to a certain 'reasonable' level of resiliency - and if any particular application owner/customer needs more, then they pay for the extra. It's not just about being a smartypants, but making people understand that these extra uptime percentage points get expensive. There is a consequence to the company's finances for demanding them. Often, it might turn out to be a lot cheaper to re-engineer the application to learn to recover from a network failure.

The problem with net neutrality..

2010-07-19T20:45:00.000+01:00

I've always considered myself in the 'yaa boo ISP opposition to net neutrality is bad' camp, but over the weekend I was speaking to a friend of mine who is a network engineer at a medium sized UK ISP, who all but hissed when I mentioned it.

Why all the fuss? The best way to look at it (and the thing that ISP's are worried about) is not so much the stuff that people are doing today, but the changes that are on the horizon to how media gets delivered. Do any of you remember the old days, when kids were respectful, policemen kept order, the summers where always hot, and music and books where things you bought in shops? God knows how people got them into their kindles, but hey.

To the ISP's these things are not huge deals. An eBook is a few hundred kilobytes, an album is say 18-25Mb. And the key point I guess is that they are one off transactions. A typical user is only going to download them occasionally.

TV and video are next. At the moment we go to the video store, or maybe occasionally rent videos from iTunes or whoever, but primarily we still watch live TV via traditional methods. The ISP business in the UK got a real wakeup call when the BBC launched their iPlayer application, which allowed people to ignore shedules, and watch their tv whenever they wanted. Unlike PVR type systems, the content is delivered in demand from the cloud, rather than the user storing the media locally.

This hurt their networks really badly. But what really scares them is the question of what happens when this becomes the mainstream. Fast forward to a few weeks ago when google announce Google TV. A set top box which allows people to scour the Internet for tv content (no doubt with the help of YouTube acting as a CDN). Moderatly scary. Then the fact that they've tied up with Sony to build it into their TVs.. Scary. Also the story around the campfire is that Sony can hit the switch on however many million PS3's are out there and turn them into into GoogleTV set top boxes. Oh and Microsoft can do something similar with the Xbox. Terrifying.

The Internet as its designed today (a unicast based, far away datacentre centric network) simply couldn't cope. It would die horribly. It simply can't do that huge sustained volume. And this is why they want to be able to limit it. They have to.

Now in reality, this is a when not if scenario. So in reality, someone (IETF?) needs to start planning for this, because it's perfectly possible to do, it just takes a different design than we have right now for how traffic gets for A to B across the Internet. Unlike when I say 'someone should do it' at work, i doubt it'll end up being me. However I do suspect CDNs will become extremely important.

The main conclusion I came to over the weekend was to buy some Akami stock! I know this will end up with big nasty corporate telephone companies controlling what we get access to, rather than the current system where Rupert Murdoch does, but hey, whoever said speech was free right...

Location:Dublin, Ireland

Why ACS replication fails through a firewall..

2010-05-07T20:54:00.001+01:00

Ever tried putting a Cisco ACS server on each side of an ASA, and getting them to replicate? By default it doesn't work, and it's bugged me for ages as to why. I finally discovered why today. It's one of those 'once it's explained it's flippin obvious' ones..

I am assuming of course that NAT and ACL entries are correct!

Why you get a problem

ACS uses port TCP2000 for replication traffic. Anyone think what else uses TCP2000? SCCP (Skinny) of course! And guess what is a default inspection on the ASA :

policy-map global_policy
class inspection_default
  inspect dns migrated_dns_map_1
  inspect ftp
  inspect h323 h225
  inspect h323 ras
  inspect http
  inspect netbios
  inspect rsh
  inspect rtsp
  inspect skinny
  inspect esmtp
  inspect sqlnet
  inspect sunrpc
  inspect tftp
  inspect sip
  inspect xdmcp
!

How to resolve the issue

By default, the ASA will inspect the traffic as if it's SCCP, will see that it's not valid SCCP traffic, and quietly drop it. You can stop this behaviour in two ways:

1) Disable the inspection completely if you're not using Cisco IPT.
2) Remove it from the default inspection list, and set up a separate class to match the traffic you DO want to inspect for SCCP, and inspect only it..

Example

Lets say two ACS servers, 1.1.1.1 and 2.2.2.2 need to replicate, and you do use SCCP on the network :

access-list NOT-ACS extended deny tcp host 1.1.1.1 host 2.2.2.2
access-list NOT-ACS extended permit ip any any
!

class-map NOT-ACS
match access-list NOT-ACS
!

policy-map global_policy
class inspection_default
  inspect dns migrated_dns_map_1
  inspect ftp
  inspect h323 h225
  inspect h323 ras
  inspect http
  inspect netbios
  inspect rsh
  inspect rtsp
  inspect esmtp
  inspect sqlnet
  inspect sunrpc
  inspect tftp
  inspect sip
  inspect xdmcp
class NOT-ACS
  inspect skinny
!

Your ACS devices will replicate, and your SCCP still gets inspected..

Thanks to my old buddy Frank Gannon for spotting this one!

(updated to correct config error spotted by Robert)

Which is the worlds best firewall? Windows Firewall of course..

2010-06-16T22:28:00.002+01:00

Over the years I've often been asked 'Which is the best firewall?' My answer is usually the same - it's the one you know best.. Customers often think I'm just being a smart arse, but I'm not. So I thought I'd lay out the argument here..

How do Companies buy firewalls?

There have been lots of religious wars over firewall types. Who can forget the proxy vs statefull inspection of the nineties and early noughies. Vendors still forward the whole 'ours is best because of feature X' arguments.

As a result of these wars, many organisations now use a formulaic approach to vendor and product selection. You write down a list of the features you want, compare the products, and bingo, there is the selection.

It sounds logical and sensible as a method, although you could argue that it's main purpose is to assist in the justification of why the purchase was made after the fact.. Most people (subconsciously maybe) tilt the requirements towards the product they actually want, and vendors have become expert in planting requirements into peoples heads. But still you can say 'No, we didn't buy from them because my brother is the salesman, it's because it's the only firewall to protect against Mosaic vulnerabilities'..

How well do we know our firewalls?

Firewalls are complex beasts, and while most people learn quickly enough how to make a policy change, or set up a simple NAT, regardless of brand. However really understanding how it's processing the packet, the order and types of checks that are going on, and how to troubleshoot each aspect of the inspection logic is a rarer thing. For example, I would say that I know the PIX/ASA firewalls pretty well. I can set up complex features and troubleshoot them well. However give me a checkpoint box, and I can do a lot, but quickly get lost on advanced troubleshooting tasks.

This is hugely important, as the moment things start to go wrong with a firewall deployment, people quickly start to compromise their lofty security ideals to keep their traffic working. If we don't really fully understand exactly what that knob does, other than make the traffic work when we turn it off, we end up in a position where the firewall configuration is suboptimal - and we might not even know!

That's not to say we can make every feature work on our firewall of choice, but we probably have the experience and product knowledge to understand the consequence of turning off a feature. If I'm working on an ASA, I'm pretty confident that I can stand over my work and say 'Yes, this will protect exactly what you've asked me to protect'. Do it with a product I don't know so well, and I'm not so sure how it will perform.

So which is best, Cisco ASA or XP firewall?

I tweeted about this recently, and someone sent me (in jest) a comment 'so XP firewall is as good as a pix then?'.. Well, this could be the logical conclusion of the argument.

Let's say as an organisation you have a windows admin who is so good he can change the computer case colour by group policy, and is a passable cisco admin. A really good windows admin can close down windows so tight that you can be plugged into the same LAN as it and you will get nowhere (and no, I don't mean by powering it off).

So what's the best use of his time? If he spends his time making windows bulletproof, using all the tools he knows, he will have more effect than poking at a PIX he doesn't really understand..

Am I seriously recommending windows firewall over a PIX? No, I'm not, because I'm a Cisco guy. But my windows admin buddy probably will, and that's the point. The tools we know how to use, are the tools that we will have the most effect with.

Of course, as any security person will tell you, a layered approach is essential, but when you select your products for each layer, make sure it's the product which you can make fly. Otherwise, all you've bought is a bunch of problems and caveats you haven't seen before..

Location:Dublin, Ireland

ASA new features in 8.3

2010-06-12T21:14:00.000+01:00

Cisco quietly released the latest version of ASA code (8.3 which i must have missed the release of) and while checking through the new feature releases I noticed two key changes.

The first one is the option to use Globally Applied access lists. This is a big deal, as cisco firewall policies have always been interface based. It's one of the barriers that administrators of other devices find when starting to use PIX/ASA/IOS firewalls. The last couple of years have seen a couple of changes that indicated a move in that direction. We had zone based firewall (now the recommended way to firewall on IOS), where controls are applied between zones rather than interfaces. Now ASA (not PIX btw) move to global access lists.

Is it better? That's going to turn into a religious argument I suspect. Some people like the idea that you have one policy where rules are configured, and one place only. You can argue that this simplifies configuration, especially on devices that have multiple occasional administrators, as you don't need to look in multiple places to find where rules are configured.

Others (probably people like me who are very used to the PIX way of doing things) would argue that per-interface configuration means shorter more specific access lists per interface, meaning you've less to look through to find a rule - you just need to know what you're doing to make sure you look in the right place.. Time will tell what become the prevalent method, and it'll be interesting to see whether Cisco optimize performance towards one method or another (like ZBF being the 'optimised' method on ISR routers)..

The second big change is the 'simplification' of NAT.. I've not finished reading up on this yet, but the bit I don't like is the bit in the release notes where it says that legacy configuration will be automatically upgraded. That sounds like something which will need a lot of testing before we're all confident in making live upgrades.. For me, I'd rather see an option to use either method, just for a version or two.

Location:Fingrean Rd,,United Kingdom

Will the CCIE lab become more accesible - and is this a good thing?

2010-05-21T22:57:00.001+01:00

Rumours are flying that CCIE lab prices are about to drop. We've already seen the lab move out of their static locations, and the interface used by candidates means there is no need to be in the next room to the kit.

Taking the lab

In my day (an unimaginable 19 months ago) taking the CCIE lab meant checking the lab booking tool daily to try to get lab slots, waiting months then for test day, the best bit of €2k of spend (including travel/hotel/etc). You can look at this in two ways. The fact it was hard and expensive to get a slot meant you made very sure you were ready before going. Conversely, you couldn't just study until you felt ready, then go..

So what's changed

In reality the change is more approach than technology. Over the last year, Cisco have had 'mobile' test centers which meant they could expand the amount of testing they could do, without the limitation of the space in their offices. Accessing routers remotely is of course not a new thing. It makes no difference if you're in the next room or 1000 miles from your rack. Most CCIE candidates use remote labs as part of their training, so there is nothing new to them.

How far could they scale this

Well, they could really increase the capacity enormously. There are four things they need to scale up to deliver more tests :

1) Racks - This one is pretty easy.
2) Seats and screens - either using their relationship with prometric, or by opening up in Cisco offices around the world, this would not be a major problem.
3) Marking - this would need a little time to get trusted people with the right skills up to speed. Still, I'm sure they can do it easily enough with a little planning.
4) Proctors - now this is the challenge.

Proctors

For those who haven't done it, the CCIE lab is different to the rest of their tests (in many ways) in that a proctor is there to help you understand what's being asked. Wording can often be unclear - I know when I was in the lab I was up with the proctor constantly clarifying things. This is probably the hardest thing to scale. Options that jump to mind would include :

1) On-site proctors - if you've a permanent site, why not just hire another proctor?
2) Access via telepresence - you could have a 'farm' of proctors all available remotely. This would have a big advantage that if you have people doing voice, security, R&S in the same room - each can speak to a proctor who knows their track. That can be a big problem with the current system.. This is probably more cost effective as a proctor covering a small test site might not be overworked.
3) Mobile proctors - covering a number of sites making sure that each site is covered when it's running tests. More appropriate to part time sites, or sites which run different tracks on different days..

Either way, the issue is quite solvable.

Where does this leave us?

Well, if Cisco scaled up the ability to deliver tests, then there would probably be more cost effective. Darby Weaver is predicting the $1000 lab this year, and as well as the poor economy, this more efficient delivery method may be a driver for that.

Is it a good thing? One thing comes to mind. If it's easy and quick to get a lab (as it is with a 'normal' cisco exam), is this going to encourage people to 'go and give it a go' rather than putting time/money/effort into training/studying? Would more cheap attendances lead to increased braindumping? Is it going to mean candidates aren't going to put the same study effort in before going to the lab, and just hope to get lucky?

Or will it just mean the well prepared student can get a lab date when it suits them. Hopefully, it's the latter.

Has IT certification become nothing more than a racket?

2010-05-20T16:49:00.001+01:00

I've a funny relationship with 'authorized training' and certification. In one way, it is great that I know that within a reasonable period of time, I'll be able to attend a course which will give me a good grounding on 'how to drive' a particular technology. In Cisco land, the breadth of subjects that they cover is very impressive, they've made a real effort here. the certification process (especially the CCIE level exams in my opinion) drive us to better knowledge and to be better engineers.

On the other side - it is generally just an intro. My personal experience of these courses has generally been underwhelming, and I've hoped to come out with a lot better understanding than I actually did.

Who goes down the route of certification?

Speaking to people on these courses, there are typically three kinds of people who attend them :

1) End users of the equipment, who are buying an and as part of the budget are going on the training. They generally don't care about doing the exam.
2) People doing certifications for their own career (more applicable to the CCNA/CCNP/CCIE specific courses). Often very dedicated to get as much out of the experience as they can - although you can get braindump-bunnies too..
3) Partner employee's who have to get the certification asap as their employer is going to be audited by Cisco soon.

Audit time

Let's ignore the first two for the moment. Anyone who's ever worked for a Cisco partner knows audit time. It's when senior management work out that if they get two people with certification X, they'll get an extra percentage point or two from Cisco on everything they buy. It's big money actually, and can be the difference between winning and losing business, so is important. So if you make the mistake of walking near them at the time they have this revelation, you will volunteer to go and do the certification.

This leads to a lot of cheating. You see nobody cares how well the engineer knows the technology. That's another days problem. The key is to make sure that the certifications are in place by audit day. Quick is better than good. I don't like this (and let's be clear, it's not the way I do things!), but it's the way it is. It's a rotten part of the certification business, always has been and probably always will be.

how bad is it?

However this week I heard a story that made me realize how bad things have actually got. And I have to say, after all these years, it still shocked me. A friend of mine has recently attended a Cisco course, provided by one of the big UK authorized providers. The instructor started by handing out the official course guide, and a photocopy of the testking for the exam, and told the attendees to start reading through the questions during the evenings and come to him with any questions. Then he started going through the course notes.

Now as I've said before, I don't approve of the whole braindump thing. It's pointless, it devalues the work of those who make the effort to actually go and learn the technology, but mostly it turns the whole certification business into nothing more than a racket. But I didn't think we'd got to the point where there was such acceptance of the fact that most people on the course are just going to then just going to braindump the exam. Surely most of us want to do it properly right?

Summary

We have to decide what we want from a certification process. The problem is the partners who need certified staff, the training company, the testing firms, and Cisco themselves all do just fine out of the current system. It's us as engineers who are devalued and cheapened by a process which is meant to make us better at what we do, but actually just makes us rats in a process. If we don't approve, then expose the cheats, do it properly ourselves, and lets take the high ground and be better people as well as better engineers..

Troubleshooting: Dan's mistake of the week..

2010-05-19T00:11:00.001+01:00

I've spent time over the last two weeks pulling my hair out as to why new traffic on an existing site to site VPN didn't work. Finally got it today, and reminded myself an important lesson in the process.

Scenario

Take a simple site-site internet VPN between two sites (well there are lots, but this about a VPN between two sites), and two networks in each site. London has the network 10.15.1.0/24 and 10.15.2.0/24. Dublin has 10.35.130.0/24 and 10.35.139.0/24. Each site has a pair of ASAs (up to date code).

Summary of the configuration here (it's not the real one for obvious reasons, and cut down to the key bits, but is accurate for the sake of the article) :

Dublin-PIX :

interface e0/0
nameif outside
security-level 0
ip address 35.35.35.35 255.255.255.0
!
interface e0/1
nameif inside
security-level 100
ip address 10.35.139.254 255.255.255.0
!
interface e0/1
nameif dmz
security-level 50
ip address 10.35.130.254 255.255.255.0
!
crypto map Dublin 24 match address Dublin-London
crypto map Dublin 24 set peer 15.15.15.15
crypto map Dublin 24 set transform-set AES-256VPN
crypto map Dublin interface outside
!
access-list Dublin-London ex per ip 10.35.130.0 255.255.252.0 10.15.0.0 255.255.0.0
access-list Dublin-London ex per ip 10.35.139.0 255.255.255.0 10.15.0.0 255.255.0.0
!
access-list NONAT extended per ip 10.35.130.0 255.255.252.0 10.0.0.0 255.0.0.0
access-list NONAT extended per ip 10.35.139.0 255.255.255.0 10.0.0.0 255.255.255.0
!
nat (inside) 0 access-list NONAT
nat (inside) 1 10.35.139.0 255.255.255.0
nat (dmz) 0 access-list NONAT
nat (dmz) 1 10.35.130.0 255.255.255.0
!
global (outside) 1 interface

London-PIX :

interface e0/0
nameif outside
security-level 0
ip address 15.15.15.15 255.255.255.0
!
interface e0/1
nameif inside
security-level 100
ip address 10.15.1.254 255.255.255.0
!
interface e0/1
nameif dmz
security-level 50
ip address 10.15.2.254 255.255.255.0
!
crypto map London 14 match address London-Dublin
crypto map London 14 set peer 35.35.35.35
crypto map London 14 set transform-set AES-256VPN
crypto map London interface outside
!
access-list London-Dublin extended permit ip 10.15.0.0 255.255.0.0 10.35.130.0 255.255.252.0
access-list London-Dublin extended permit ip 10.15.0.0 255.255.0.0 10.35.139.0 255.255.255.0
!
access-list NONAT extended permit ip 10.15.0.0 255.255.0.0 10.0.0.0 255.0.0.0
!
nat (inside) 0 access-list NONAT
nat (inside) 1 10.15.1.0 255.255.255.0
nat (dmz) 0 access-list NONAT
nat (dmz) 1 10.15.2.0 255.255.255.0
!
global (outside) 1 interface

Symptoms

The problem is that while traffic from 10.35.130.0/24 could get to machines in 10.15.1.0/24, traffic from 10.35.139.0/24 consistently could not. Running a packet-trace (at both ends) showed that it should work, and packets where leaving the 10.35.0.0 site, arriving at 10.15.0.0 site, the response packets never made it back. A capture on the inside interface in site 15 showed the server did respond. Rule access-lists are all correct..

To keep it simple - it's nothing to do with rules or the servers. I has probably never worked, but this is the first traffic to go between these two networks.

Simply - the packets from Dublin->London pass, but packets from London->Dublin don't.

What it could be

Once I've a clear definition of the problem, the next thing is to rule in or out the most likely causes.

The first thing that jumped to mind (and a common cause of issues with these symptoms) was that the NAT exemption wasn't set up correctly. Unless traffic is NAT exempted, it will NAT behind the interface IP, which will put it outside the VPN interesting traffic, and we would get these symptoms. However the packet-trace I did showed that the traffic was 'Allowed' to NAT exempt, so it wasn't that. I was actually still fairly convinced it could be, but eventually moved on.

Second thought was 'could it be badly written ACLs for the VPN definition', or even some incorrect routing. After lots of staring, answer was nope, none of them.

As sherlock homes (he's a British policeman) once said, 'when you've ruled out the probable, then whatever remains, however improbable, must be the answer'. So we're into the unlikely stuff. I spent hours looking for odd traffic handling quirks of the ASA. Tried a few things. Rebooted them. I was getting nowhere.

So what was it?

Have you worked it out yet (I'll be impressed with you if you have)?

I think you cross a line when you decide it isn't a probable cause. You decide it's going be something odd, then common sense leaves you and you start looking for crazy stuff. Don't get me wrong, sometimes it can be a bug, or a feature you don't understand properly, or something else crazy. But usually, it's something simple.

I showed you the same parts of the configuration that I focused on - the sections relevant to the connection in question. The sharp eyed amongst you will have noticed the sequence numbers in the crypto map may point to other entries prior to this one. Such as :

London :

crypto map London 9 match address London-Cork
crypto map London 9 set peer 9.9.9.9
crypto map London 9 set transform-set AES-256VPN
!
access-list London-SITE9 extended per ip 10.15.0.0 255.255.0.0 10.35.136.0 255.255.252.0

You see, the second octect in the IP scheme denotes country code (353 is ireland - shortened to 35), and for some reason in the distant past someone decided to use non contiguous addresses (I can only assume satan was whispering in their ear). When the (older) Cork connection was set up (which has 136,137,138 ranges) they simplified it by using 10.35.136.0 255.255.252.0. As this is a lower sequence number in the crypto map, it gets hit first, and the traffic never gets down to the ACL we want it to hit.

As with so many of these things, once you work it out, it's obvious and you're dumb. So where did I go wrong? My biggest mistake was immediately deciding (see the second sentence of this article) that the issue could only be to do with the configuration of this particular site-site VPN (i.e. sequence 14 of the crypto map), or something crazy and probably global. I didn't look at the rest of the crypto map at all. Why would I?

Dan's approach to troubleshooting

When you've spent many years troubleshooting networks, you learn not to beat yourself up over silly mistakes like this. You learn from them and move on. I'll certainly not make this mistake again. However to help you with the issues you've never encountered before, troubleshooting depends on being logical and methodical. It's so incredibly important, and normally something I take very seriously.

In this case, I wasn't as methodical or as logical as I should have been, and that's the real lesson I need to take away with me.

How do you interview for technical people?

2010-05-14T17:15:00.001+01:00

Having been through a rather grilling interview process myself this week, I'm curious how people out there interview for technical staff, and how successful or not they find certain techniques to be?

The classical method of interviewing (either in person or over the phone), I've always found to be quite useless. It does have a 'weeding' effect of removing the completely clueless, but that's all. The problem is when you ask people to describe protocols/systems/methods etc you're testing their booksmart skills, not their real world ones.

Practical Tests

I've always loved the idea of a practical test, where some kit is set up in a 'mini lab' type setup, and a number of trouble tickets are given to the candidate. This will answer the 'can they troubleshoot' question! However it's quite time consuming to set up and grade - if you've ever tried to write these kinds of tests they actually take quite a lot of time to get clear and fair.

As an alternative - there is whiteboarding a problem, where the candidate is given a theoretical issue, which they need to talk through their troubleshooting techniques for. It's simpler to do for the interviewer, but can be quite daunting for people who are not used to public presentations.

Whiteboarding

In general, making someone answer an open question (e.g. show me on the board how OSPF works) will really test the depth of their knowledge - beyond being able to regurgitate the books they've read. However, again you have the problems of nerves and people unused to standing up and presenting. If you're not looking for 'public facing' staff, are you ruling out people based on a skill you don't need? Speaking to recruiters, this kind of method can rule out a lot of people!

Comments

I'm really interested to hear your comments on this one. What have you done (or had done to you!) over the years, and what do you think worked and didn't work!

ASA using the sdi protocol to authenticate against replica RSA servers

2009-04-11T11:42:00.001+01:00

I found an interesting thing the other day which is not immediately obvious from either Cisco or RSA's documentation. We're all used to using RADIUS as a AAA method, and I would usually use it when talking to an RSA server, but you can also use the RSA protocol itself (check version).

In a single RSA server environment, this is well documented ( links), but there is a caveat when using replicas. Let's say we have two locations, an ASA in each and an RSA server in each. What you'd expect to do is configure two aaa-server host on each asa, with the local one first right?

Turns out that's not quite how it works. When you configure the aaa-server host what you are actually telling the asa is 'go talk to this RSA server, and download information about the RSA topology'. This MUST be the primary RSA server.

The first time an authentication request is sent, the ASA will connect to the RSA sever, agree a shared secret, and download the topology. From then on, even though you only have the one server configured, the asa will be aware of both the primary and replica.

I haven't yet found a way to control which RSA server the ASA uses - I suspect you can't - but I did test failover and the firewall will authenticate against the replica (even though it isn't configured) when the primary fails..

If you specifically want the site B firewall to use the site B RSA server (the replica) I suspect you need to use RADIUS and configure both servers in the order you want to use them.

Why the CCIE program is more useful than the certification itself.

2010-04-27T07:20:00.001+01:00

About 3 years ago, when I'd first decided to go down the route of getting my CCIE, I remember speaking a colleague who had done his a year or two before, and asking him was it worth the effort. He told me 'regardless of having the certification, the process will make you a better engineer'. When I asked him what he meant by that, he told me 'you'll see', then went off the pub for the afternoon leaving the hard work to his minion (me).

Now, as my first recert is getting close, I get what he meant. There's an awful lot of rubbish talked about the CCIE, and how guru-like CCIE's are begged on a daily basis to move to higher and higher paid jobs.. I can't deny, it is a door opener when looking for a gig, and sometimes clients specify they want a CCIE for a given role, but door opener is all that the certification is. You don't get an easy interview as a result - in fact when you say you're a CCIE, people try harder to catch you out, and you have to be better than ever to get and keep your credibility.

The reason the CCIE teaches you to be a better engineer, is not because it gives you god-like knowledge of the IOS (you do have to learn the IOS backwards, but this soon starts to slip and become out of date unless you keep it up), but it teaches you four things :

1) Discipline - when you're studying for the CCIE, you have to be able to come home from a tough day, go to your lab, and study for hours, day, after day, for an average of 12-18 months. You give up on your home life, lose weekends, don't see your buddies, and so on. It's hard - really hard, but you have to have the discipline to sit down, and get on with it. This stands to you afterwards, for example, before I did my CCIE, I couldn't work at home - I got nothing done. Now I work at home 4 days a week and get twice as much done as when I go into the office.

2) The use of the DocCD (sorry, I'm old fashioned). I don't remember every timer default, or how to configure every feature of the IOS. But I can find out pretty damned quick, and I never use google. You have to know the documentation site backwards to do the lab. During preparation, you learn how to navigate the documentation site quickly and effectively, and learn where everything is. In the real world, this is really useful every single day. No more searching google and hoping the answer isn't on experts-exchange.

3) Attention to detail - you can't make mistakes in the CCIE lab. One little thing can cost you the exam (the way the marking works makes a mistake on the underlying infrastructure mean that lots of questions which depend on it fail). So you have to be perfect every time. Personally, this was a bigger challenge than learning the IOS - when I did mock labs it was the dumb mistakes that got me every time. You have to work on that to a point where your planning and coding is perfect every time.

4) Troubleshooting - this is the big difference between the CCIE lab and all other Cisco exams. It's not enough to know the theory. You need to be able to set up a really complex network, spot the problems, and troubleshoot them quickly and effectively when the little trap that was left for you trips up your configuration. During your preparation, you spend hours and hours every week setting up scenarios to fail, so you can troubleshoot them. This is invaluable.

I would always recommend this process to anyone who hasn't done it, and wants to be the best engineer they can be to do the CCIE. Not because you learn lots of cool features (although you do), and not because you'll get a better paid job (hopefully you will), but because it teaches you how to work better, faster, and more reliable in the real world. This is why CCIE's are still so sought after.

One last point - don't be tempted with the shallow promises of braindump sites. By giving yourself this 'help', you're not giving yourself the main learning experience, you're only passing the exam. When you're interviewed, people will know. When you go to a job, you'll get found out..

DNSSec - and why the Internet probably won't break today.

2010-04-14T17:34:00.001+01:00

If you're like me someone (quite possibly pointy haired) probably came running up to your desk in the last day or two telling you how the internet is about to come crashing down around us (yes, Cisco did promise it'd never be the same!) due to the deployment of DNSSec which is happening at the moment, and asked what you were doing about it. Articles like this really don't help.

DNSSec

DNSSec is something we all need to start looking at, and it is doing to deliver a lot of security benefits. It's also something which we need to actively test and probably change things within our infrastructures to make work. For a good overview of DNSSec, start your reading at wikipedia.

As with all DNS related subjects, there are two separate issues. Firstly, what we do with the domains we own - i.e. when do we start signing them - simple answer, not until the next tier up has (the current work is on the root servers, when that's complete the TLD's will be done, then you can start looking at your domains!). Secondly, how does it affect how we resolve other people's domains. This is where the current publicity is focused, and where we will probably have to do a little work soon.

In a simple scenario, where you have stub resolvers in your organization (i.e. your DNS servers point out to your ISP for recursive queries), there is little you need to change in the short term. This is mainly down to the fact that the main difference will be an extra flag set in the response to say 'this is Authenticated Data'.

If however, you do recursive lookups yourself (i.e. your DNS servers don't have a forwarder set, they just go out the root servers and start doing a full recursive lookup) then there is a change. Put simply, it will be asking for more data in the response from the various servers it talks to, using an extended version of DNS called EDNS. When the whole system is filled with signed DNS records, you'll be getting answers back which contain records larger then 512bytes (and probably fragmented packets). Most firewalls will block these large UDP DNS packets by default, as it has become accepted security practice to do so.

It's important to note that some firewalls will know the difference between EDNS, and will not apply the restriction to EDNS. For example, in earlier versions of PIX (6.3.2 and below), you had to manually configure the DNS fixup to permit DNS packets with the longer length :

fixup protocol dns maximum-length 4096

in more recent versions, it would be covered by :

policy-map type inspect dns preset_dns_map
parameters
message-length maximum 4096

but here's the thing, it's EDNS not DNS, so it doesn't actually affect it. However, all firewalls are different in their support for EDNS, you do need to check this on your firewall/router to be sure that it's supported.

Testing

A good way to tell if your firewall/router is blocking these larger DNS packets, is to use the test listed at OARC. Here are two tests, the first one running to a non-EDNS capable DNS server (OpenDNS), and the other running to a EDNS capable one (Verizon). In both cases, they are running through an ASA (8.2.2) with default DNS message length of 512 :

dhughes@dynamips:~$ dig @208.67.222.222 +short rs.dns-oarc.net txt
rst.x476.rs.dns-oarc.net.
rst.x485.x476.rs.dns-oarc.net.
rst.x490.x485.x476.rs.dns-oarc.net.
"208.69.34.6 DNS reply size limit is at least 490"
"208.69.34.6 lacks EDNS, defaults to 512"
"Tested at 2010-04-14 15:32:28 UTC

Here - EDNS is not available.

dhughes@dynamips:~$ dig @158.43.128.1 +short rs.dns-oarc.net txt
rst.x3827.rs.dns-oarc.net.
rst.x3837.x3827.rs.dns-oarc.net.
rst.x3843.x3837.x3827.rs.dns-oarc.net.
"62.189.58.236 sent EDNS buffer size 4096"
"Tested at 2010-04-14 16:10:18 UTC"
"62.189.58.236 DNS reply size limit is at least 3843"

Here is a response which is considerably larger than 512. This means we can get the traffic through our firewall.

Summary

The Internet isn't broken. You should check to be sure that your firewall will pass the EDNS traffic ASAP, and if not, make the appropriate changes. You can tell the pointy haired boss who's sweating in the corner, that you've fixed the internet, and it'll all be fine...

Making changes to remote systems..

2010-04-14T13:06:00.000+01:00

Every time we change something on the network, we take a risk. We may well have tested it in the lab/pre-prod environment, but there's always some risk. The nature of the change determines how much risk. It's why networks are most stable when the admin is on holiday.

If it's on a device that you are standing next to, you know that if it goes wrong, you can always recover it manually. But what do you do if it's remote? Now with cisco routers and switches - as long as it's something in the config, you can usually get back by rebooting*, and the startup-config will be reloaded. But it's a little bit embarrassing to have to call the office and get someone to do this for you. Also, if you're following a sensible change control process, you've made the change out of hours when there's no-one there to do this.

A really useful command here is 'reload in ' - where x is a number of minutes. If you follow this procedure - you'll always know you can get back to where you were :

1) write mem the old, known working, config

2) enter the command 'reload in 20' (or whatever interval you choose)

3) make the change.

console#telnet 192.168.1.250

Trying 192.168.1.250 ... Open

User Access Verification

Username: cisco

Password:

ADMIN-SW#wr mem

Building configuration...

[OK]

ADMIN-SW#reload in 5

Reload scheduled in 5 minutes by cisco on vty0 (192.168.1.150)

Proceed with reload? [confirm]

ADMIN-SW#

***

*** --- SHUTDOWN in 0:05:00 ---

***

ADMIN-SW#conf t

Enter configuration commands, one per line. End with CNTL/Z.

ADMIN-SW(config)#int f0/4

ADMIN-SW(config-if)#sw acc vlan 2

Now, either the change works, and you maintain your access. Depending on the nature of the change, reconnect a new session to be sure. Then, when you're happy that it's not going to break again, enter 'reload cancel', and the scheduled reboot will be cancelled. If for some reason the change fails, then after 20 minutes, the device will reboot, and you can go back to it and work out why.

In this case, the change worked and I can reconnect :

console#telnet 192.168.1.250

Trying 192.168.1.250 ... Open

User Access Verification

Username: cisco

Password:

Reload scheduled in 4 minutes and 30 seconds by cisco on vty0 (192.168.1.150)

ADMIN-SW#relo

ADMIN-SW#reload cancel

ADMIN-SW#

***

*** --- SHUTDOWN ABORTED ---

***

ADMIN-SW#

If that hadn't worked (or if I hadn't gone back in to cancel it!) then after the five minutes the switch would reboot, and come back in its previous state. It's still an outage of course, and you still should have tested your change better, but at least you're able to recover, regroup, and try again.

I ALWAYS do this before a change on a remote device, even if I can't imagine any way that this change can break anything. Why - because I've been bitten too many times by devices far far away.

*ASA/PIX in FO mode is different. I'll post on it another day.

How to create a strong yet memorable password.

2010-04-07T17:14:00.001+01:00

We're often told 'make sure you use a good password'. When we change our passwords at work we're often forced to add random characters into it to make it more complicated. This can have a detrimental effect on how how easy it is to remember. For a rant on the importance of passwords being memorable see this. This post is the second of a short series covering various areas of password management and related issues, and is going to cover a simple method for devising strong yet memorable passwords.

So how do we come up with a password which is memorable, and really hard to guess. We need to be sure it's a reasonable length, contains a mix both small and capital letters, numbers (ideally) and non alpha-numberic characters to make it had to guess. Unfortunately passwords like lk^7*Sn7@'h& are hard to guess, but also hard to remember..

We tend to remember things we associate images, sounds, feelings, taste, touch etc with, rather than lists of characters. Lines in songs, poems, phrases with films, headlines in newspapers stick with us because of an image we form in our minds, which we find easy to recall.

Take the time at the end of 'Back to the Future' when the Doc Brown comes back in the now flying DeLorian, and tells Marty that they need to go to the future. When told the road isn't long enough to get to 88Mph, who can forget the Doc dropping down his sunglasses and saying 'Roads, where we're going, we don't need roads!'.

Memorable right? If you weren't a fan of 'back to the future', take a song lyric you like, or line from a film/poem/court judgement you enjoyed, and write it down capitalized and punctuated.

Roads, where we're going, we don't need roads!

Lets take the first character from each word, and the punctuation, and we get :

R,wwg,wdnr!

So memorable that you will never need to write it down, and almost impossible to guess. If your system also want's a number, then either substitute, or add one.. So we could make it R,wwg,wdnr1 .

I've used this system for years, and don't have to write down my password to remember it, and have never had a problem with 'password to simple' complaints from the operating system.

Comments, as always, are welcome!

BGP Route selection (the brief version)

2010-03-29T22:35:00.000+01:00

If you're preparing for an exam right now, you probably need to know the full version of this - located here..

However in the real world there are ( in cisco land anyway ) 4 parameters that you typically use. There are two separate 'sets' - attributes which govern how traffic leaves your AS (i.e. routes which are members of other AS's - or outbound traffic); and attributes which govern how traffic enters your AS (i.e. they belong to your AS - inbound traffic).

Lets look at outbound traffic first. The two parameters we can change to alter this are weight and local preference. Weight is Cisco proprietary, and is local to the router (it is not exchanged with other devices). Local preference on the other hand is exchanged by routers within an AS (subject to normal iBGP rules on split horizon). The path with the lowest weight (best) will be selected over the path with the highest (best) local preference.

For inbound traffic you also have two options, MED (AKA metric), and path. MED has a few limitations - firstly - and most importantly - this attribute can be passed to (and within) a directly neighboring AS, but will be passed no further.

Let's take and example - you are connecting to the internet via two separate ISPs, and you want to control which ISP your inbound traffic is to come down, using the MED attribute. You set a MED of 100 on ISP A, and 200 on ISP B (lowest wins), so all the traffic will come down ISP A right? Wrong. ISP A will have a path with a MED of 100, and ISP B will have a path of 200, but they won't pass the MED to each other to compare. When the paths get advertised out to the rest of the internet, the MED will be stripped off, and the rest of the world will pay no attention to it.

What will actually determine* how this traffic enters your network is the AS path. Go back to our example above - both ISP A and B will have two paths to get to your network. One will your peering with them. It will have a path length (i.e. the number of AS's in the path) of 1 (your AS). Let's pretend ISP A and ISP B peer with each other directly and exchange all paths - so they will have a second path to get to you via the other ISP. This will have a path length of 2. Path length works on shortest path wins, so they will always send traffic down the direct peering. Their peers (and backbones) will see both paths, and depending on how they connect, will favour one over the other. Typically, lets say ISP A is a well peered tier 1 ISP, it will probably have a shorter path to most destinations than ISP B, who lets say is a small local ISP, which is often going to have longer paths to get to most destinations. This by itself will bring most (but not all) traffic down ISP A.

If you want to force a particular path, you can perform what's called 'AS Path pre-pending' (or sometimes path 'stuffing'). Simply put, we repeatedly prepend our own AS to the path that we want to make 'worst' - lets say that's ISP B. If our AS is 1234, then we send a path of <1234,1234,1234,1234,1234,1234> to ISP B, and a path of <1234> to ISP A. This will generally make A preferential to all but the most crazily connected peers. Even traffic generated from within ISP B will now travel over the peering to A and down that link (path length of 2 rather than 6).

MED does have a use though - let's say you have two connections to the same ISP, and want to choose one link (and we'll assume the ISP does nothing clever at their end) over the other. MED will work well, as it propagate the metric to the ISP's backbone, and within that backbone, control which path is seen as best.

For enterprise networks, this is a common use, although it's often can be a waste of time, as the ISP will often assume that everything you tell them is wrong, strip all attributes from your advertisements, and control this themselves. Having done the job, and knowing how many customers screw this up, it's not as crazy as it may seem. If you want to do this, make sure you have the conversation with the ISP about it first, and make sure they're not going to strip your attributes. In such a scenario - I'd generally still use path - it's harder to overwrite, and given the choice, best path will take preference over best MED.

Comments and corrections welcome!

* it can be a lot more complicated than this - ISP backbone teams spent half their time making sure traffic enters/leaves their network the way they want it to - i.e. the cheapest way. It's a form of warfare. Let's pretend for the sake of this example that it's simple.

AAA Part 1 - Password Management - myths and a good rant.

2010-04-07T11:18:00.001+01:00

Password management is a personal bugbear of mine. I think it's generally done really badly, and the security industry has adopted a number of policies which are universally accepted as best practice, without much thought as to why. I call them the myths of password management.

This post will be the first of a short series covering various areas of AAA, and is going to cover the myths around password quality and rotation.

The Myth of password rotation

I've always argued that there are two really important objectives when it comes to passwords :

1) No-one can find out your password.
2) You can remember the password.

Over simple? Not really. When you boil it down, this is what you need from a password.

Security managers tend to focus on the first objective, without much attention being given to the second. I'd argue that both are equally important. Let's take a scenario (which is amazingly common) where policy is set that passwords must be changed every 12 weeks. The first question I'd have is why?

What does this actually achieve? If the password is not guessable, then constant rotation achieves nothing positive. I'd won't go as far as saying it should never be changed (although it is the logical conclusion of my argument), but if objective 1 has been achieved, then there is no need for constant password rotation.

In fact, the one thing that is achieved by over-regular password changes, is we will fail to satisfy objective 2. A place I recently worked make me change passwords on about 20 different systems every 2 months. Most of those systems, I didn't log onto very often. As a result, I often had two or three revisions of my password in play at any one time. I couldn't remember which password to use in which place. And after one has been deprecated a while, I don't remember it at all. Which means I need to document it somewhere. Now we've broken objective 1.

I use a system (I'll do a separate post on how) for creating passwords which makes them unguessable(tm) - (but very memorable). There are plenty of others too. Any password can of course be broken by a brute force attack, but assuming any kind of sensible lock-out mechanism is applied, I'm happy to say that no-one knows my password. If someone logs into a system with my credentials - it had to be me. No excuses. The term often used for this is non-repudiation - in other words I can't deny it was me.

Once my password gets written down, even in some kind of encrypted password management tool, we can now only go as far as I'm fairly sure no-one knows my password. Other people have access to the tool (anything stored on my 'domain member' PC is readable by admins), there may be a flaw in the password management tool. You see the difference - we've gone from definitely, to fairly sure. Objective 1 is now no longer satisfied, and there is no longer non-repudiation.

To put it very simply, if the password is well formulated, and your systems have a sensible lock out mechanism, then constantly changing the password is not going to have any positive affect on your security. Conversely, constantly changing passwords make people write them down, this means they are no longer unguessable. This has made your security worse.

Myth - you need at least 8 characters in your password

This one really annoys me. Lets say a password using a combination of letters, capitals, numbers and non-alphanumberic characters (based on my Mac keyboard) gives you something like 108 to the power of X, where X is the number of characters in your password (in other words, there are 108 characters that 'could' make up each character) as the number of combinations that you could use for your password.

A 5 character password has a 1:14.7bn chance of being guessed, a 6 character password has a 1:1.5tn chance of being guessed, a 7 character password has a 1:171tn chance of getting guessed, and an 8 character password has an 1:18.5 sextillion (thousand trillion) chance of being guessed.

As long as you use a sensible way of coming up with passwords, and lock out an account after a sensible number of failed attempts - then the difference between 14.7 billion (5 characters) and 18.5 sextillion (8 characters) is irrelevant. To put it in context, the chance of winning the UK lottery jackpot is an easily guessable 1:14 million. In reality the odds will be lower than that as most people use more letters than anything else, and you'll get a couple of goes before the account locks, but you're not going to guess it.

'As long as you have good passwords and a sensible lock out policy....'

You've probably noticed that written a few times. If you've ever been in a meeting with me you've probably heard me chant it like a mantra. The more astute amongst you are hiding in the long grass waiting to pounce with something like 'ahh, but good security comes from layering good practices in case one fails - like the system that doesn't have the lockout policy set'.

Yes, that's true. But here's the thing, if you have an accessible system that does not lock out properly, and an attacker decides to run a brute force attack to guess the password, then it makes no difference if the password was changed yesterday or 6 months ago. They're either going to get it or they're not. You're into attack detection and mitigation now.

Which leads us predictably onto the question of 'how do you make users make their passwords complicated?'. It is the big question of all this - and focusing effort here, rather than on satisfying myths, is how you will achieve improved security. And of course, if I had a magic answer for this, I'd be so stinking rich I wouldn't be wasting good beach time on writing this post.

The most cost effective (best benefit for your buck) is always education, combined with the tools available within your NOS to enforce complexity. Teach people how to create good passwords - so that when windows tells them to make it complicated, they know how to do so easily. Teach them why it's important. Point out that if someone logs in as them from the outside and does something bad because they had a crappy password, they will get blamed and fired. Once a year, a compulsory 1 hour session from IT security for all staff will take less time, and achieve more, than making them change their password all the time.

If you have the need and the budget, and have the infrastructure and skills to support it, then of course OTP systems and/or biometrics will give you a great solution. It's complicated and expensive, but fulfills the requirements really well and takes the role away from the user.

Comments/flames welcome :-)

F5 LTM and tcp timouts

2010-03-25T15:33:00.000Z

One of the dangers of being from a pure cisco background is assumption. You treat all devices as if they have the same defaults as 'normal' Cisco devices. I think I'm pretty good at avoiding this, but it gets us all sometimes.

As we all know, when you run long lived TCP connections through application aware devices, you need to ensure the connection is used. The classic problem is Oracle SQLnet through firewalls, where oracle sets up pools of TCP connections for later use, so that when it gets a burst of traffic it has the connections set up and doesn't need to waste precious milliseconds on TCP handshakes.

The problem comes, if they remain unused for longer than the TCP session timeout value of the firewall (typically 60 minutes), where the firewall silently drops the connections as being dead, but the client and server think they're still up. Next time one side or the other decides to send a little traffic on one, you get a 'broken socket' error.

This is normal behaviour, and needs to be taken into consideration whenever you're building systems/applications which connect through firewalls or load balancers.

Now, lets be clear. The answer to this issue is always setting a TCP keepalive. Send a packet every minute, and you will never have a problem. Ever. Really, do this. In fact, given you work with such a sensible bunch who will immediately implement this, there's no need to read on.

Back to F5. Interesting fact of the day, is when you use the F5 LTM for load balancing TCP connections, the default timeout is only 5 minutes - i.e. a TCP connection which does not send a packet for 301 seconds gets dropped. That's not that long, unlike the 60 minutes (3600 seconds) I have in my head from Cisco land. So the whole 'using a TCP keepalive' becomes even more important. Really, you are a crazy person if you don't use a keepalive in this situation. There's really no point reading on. You're not a crazy person, nor is your co-worker who's setting up the app right?

Still here? OK - there's two ways to know you have this issue - you'll see this as RST packets being sent back to the client from the F5 (which do not come from the 'real' server) when they send traffic on a timed out connection, and also you can see the current connection timers using the 'b conn client' command :

[dan@ltm01:Active] ~ # b conn client | grep tcp

CONN client 10.1.4.90:1873 server 10.31.8.10:https any protocol tcp age 216 - client: 10.1.4.90:1873
CONN client 10.1.4.90:1876 server 10.31.8.10:https any protocol tcp age 217 - client: 10.1.4.90:1876
CONN client 10.1.4.90:1877 server 10.31.8.10:https any protocol tcp age 238 - client: 10.1.4.90:1877

You can see even more detail using 'b conn client show all' :

[dan@ltm01:Active] ~ # b conn client 10.1.4.90 show all
VIRTUAL 10.31.8.10:https <-> NODE any6:any TYPE any
   CLIENTSIDE 10.1.4.90:1927 <-> 10.31.8.10:https
   (pkts,bits) in = (71, 16867) out = (122, 139083)
   SERVERSIDE any6:any <-> any6:any
   (pkts,bits) in = (0, 0) out = (0, 0)
   PROTOCOL tcp UNIT 1 IDLE 83 (300) LASTHOP 8 00:19:a9:f7:c0:00

So you can now see what the actual timeout value is for the this connection (83 seconds used from a 300 second timer in this case). This is particularly hand as it shows you if your 'fix' has actually taken.

If you do have a crazy person setting up your application, here's how you can be a network hero and 'fix the F5'. Write an iRule with the following content :

when SERVER_CONNECTED {
IP::idle_timeout 3600
}

and apply to the virtual server. This simply ups the timeout to 1 hr (obviously you can adjust the time to suit your environment). You can actually be quite granular, and set different values for different protocols, check the always useful http://devcentral.f5.com site for more detail, or see this excellent post..

To see the change :

[dan@ltm01:Active] ~ # b conn client 10.1.4.90 show all

VIRTUAL 10.31.8.10:https <-> NODE any6:any TYPE any

CLIENTSIDE 10.1.4.90:1943 <-> 10.31.8.10:https

(pkts,bits) in = (83, 18157) out = (165, 188758)

SERVERSIDE any6:any <-> any6:any

(pkts,bits) in = (0, 0) out = (0, 0)

PROTOCOL tcp UNIT 1 IDLE 4 (3600) LASTHOP 8 00:19:a9:f7:c0:00

Last point on this, as with most iRules, simply applying it to the virtual server doesn't immediately effect current connections. Because the rule starts with 'when SERVER_CONNECTED' - it'll be invoked when a new TCP connection is set up, and the F5 makes the backend connection to the server. You could probably fiddle with this to find other ways to tune when it's started.

Peer Neighbor route

2010-03-22T11:18:00.000Z

This is a useful feature for PPP links where the two ends are not on the same subnet. The most common scenario would be in the use of IP Unnumbered links (although the feature comes from the dial environment). In the scenario of two routers connected via a PPP link with nothing but a loopback (/32) and unnumbered link between them :

R4# sh ip int brief
Serial1/1 155.1.254.4 YES TFTP up up
Loopback10 155.1.254.4 YES NVRAM up up

R5#sh ip int brief
Serial1/1 155.1.254.5 YES TFTP up up
Loopback10 155.1.254.5 YES NVRAM up up

you would expect only one /32 route in each routing table, but :

R5# show ip route
155.1.0.0/32 is subnetted, 2 subnets
C 155.1.254.4 is directly connected, Serial1/1
C 155.1.254.5 is directly connected, Loopback10

R4#sh ip route
155.1.0.0/32 is subnetted, 2 subnets
C 155.1.254.4 is directly connected, Loopback10
C 155.1.254.5 is directly connected, Serial1/1

we’ve actually learnt the route from the other end via PPP. It only learns the route of the other end of the link – not remote connected networks. This behaviour is enabled by default, but can be disabled using R5(config-if)#no peer neighbor-route.

This can be debugged using deb ppp negotiation.

Back to the grind

2010-03-16T21:29:00.000Z

I need to re-certify my CCIE by November. Back to the study again.. Plan so far is read up on my old notes, learn the new subjects, and give it a go and see how I get on. Apparently the new 4.0 exam is bit of a toughie, but hey, I've 6 months..

Let you know how I get on..

F5 Virtual Appliance

2010-03-16T21:17:00.000Z

I've been slowly coming around to the F5 way of thinking for a while now. It took a while - mainly because I'm not a coder in any way - and while you slowly start to realize how amazing iRules are the more you use them, to a non-coder it often seems like an awfully complicated way of do some simple things. But you get used to it.

But F5 has two really good things which Cisco could learn from. Firstly the deventral.f5.com site. It's a community support forum - and while you can get help for all kinds of things F5 related, it's really about iRules. While most vendors have some equivalent, the great thing with devcentral is F5 encourage their finest geeks to actively post on it. You get really good answers, really quickly. I've been really impressed by it.

Second (and new) is they've done exactly what everyone is asking Cisco and Juniper to do - and released a performance crippled VM of their product. So you can lab it, play with it, learn the product well. I'm over the moon about this. Now the real test is what they do with the licensing. Currently it's a 90 day trial, which is renewable. You don't need special contracts or be a customer. You can just have one. I understand they intend to release a commercial version for long term dev/staging environments.

I love this. It means I can learn the product, mess with it, get to know it backwards. That is a good thing for me, for my employer, and for F5.

This is the opposite of what Cisco are doing. With the new restrictions on IOS licensing, dynamips/GNS3 will slowly become less useful, and I'm going to find it harder to learn the intricacies of their product.

I really do wonder at the complete disconnect that Cisco seems to have with their user base. It's kinda scary. They need to halve the number of marketing people they have, and start talking to their community. Learn a little from F5. They have the right idea on some things..

DSL line stats

2009-11-26T11:27:00.001Z

If you've ever had problems with DSL connectivity, here's a great resource for interpreting the output from the line stats: http://www.kitz.co.uk/adsl/linestats.htm

Backing up an older PIX

2009-11-24T17:25:00.000Z

If you've a PIX running older (6.x) code, and want to get a clean backup of the config (including keys) then here's a useful little command pairing. Firstly set up the location of the TFTP server :

pix(config)#tftp-server

so for example to perform the crazy action of an unencrypted 'accross the internet' backup :

pix(config)#tftp-server outside 11.11.11.11 /pix-config

Then to backup :

pix# write net
Building configuration...
TFTP write '/pix-config' at 11.11.11.11 on interface 0
[OK]

or to merge the current config with a backed up one (i.e. restore to a bare bones device) :

pix# configure net

nice eh!

Domain name

2009-11-03T17:20:00.003Z

The sharper of you will note I've finally got around to setting up my domain name to point to this blog. So http://blog.olorin.co.uk we now are..

Special points to anyone who knows what olorin is without asking google...