"Where do you think you're putting that!" : Or alternatively "What's the lab at home look like, Beards?"

Imagine the scene...I'm moving in with the Peachy goddess (in whose presence all things are possible), and she's eyeing up the armful of teal boxes that are heading towards the office....

I know I should have done it while she was out... the washing up needed doing anyway... ;)

When I started on this journey of discovery around all things Netflow I decided that looking at real traffic was the only way to get a good perspective on what you might expect in a network (you don't know until you look!)

So having a router at home seemed the logical answer...a few clicks later there's a couple of Cisco 881 routers on the way! 

Then the decision is where to put it...I've been both ways on this...infront of the work virtual office router (so it sees the encrypted traffic going back to Corporate, and the TVs, smarthub stuff, etc.) or behind/alongside the CVO router (with my own lab of stuff behind it).

At the moment, I'm going with the C881 in parallel to the CVO and TVs etc. but with the other lab stuff behind it...

So what's in the lab side of the network?

Well early on I decided to make it office/home friendly I wouldn't go for anything noisy or dragging too much power...

Enter the Gigabyte BRIX range of cube sized, low power, SSD friendly PCs...think Intel NUC but just a beefier box...virtually no fan noise and powerful enough to run Fedora 20 for the PMACCT netflow collector and Splunk, and another one for the Windows Server with LiveAction's LiveNX 8.3.1 running in HyperV)

From time to time there's a ISR 4321 router, a Meraki wireless AP (for client app testing from various platforms), a couple of Raspberry Pi's for fun and a smartTV...

UNTIL recently...

Again the Peachy goddess is all seeing so the large box containing a Cisco DNAC, SG250X switch and wireless sensor didn't slip through the net! Probably gonna be a Fabric in a Box Cat 9300 at some point just to make it toasty/noisy from time to time in the office too!

At a minimum, I'd say put a small router infront of most of your home traffic...it's very insightful...I always put NAT and ZBFW on the router to ensure some level of security (never got around to configuring the ASA525...) but a cautionary note recently was around router throughput...it's an obvious thing but when we got our place we slogged through with an AT&T DSL connection for months!!!! I call them the years in the wasteland!...Then along came my Spectrum Business connection...a blistering 400Mbps down and 25Mbps up...of course the ISR4321 can't cope with that topping out at 50Mbps (or 100Mbps with the performance license running)...for the moment the C881 sitting on the side with just the lab behind it is enough and sees enough traffic to gain insights! I'll talk more about collectors and what to do about the traffic at a later date...that's where the real journey begins...

Back to the lab...Earplugs anyone?

Beards out   ? ; {)

Where to start with NBAR and Netflow: The meaningful middle...

With both NBAR and Netflow it's easy to get tied up in their history and evolution...but you didn't come here for that! You don't wanna know the esoteric uses of either capability too (at least not just yet).

So really you wanna start with some basic things that work pretty much out of the box and have the biggest impact.

So let's start in the middle...

NBARv2

NBARv2 is the Cisco feature that allows application recognition (using deep packet inspection through hardware) on a series of routers, WLCs and more recently switches.

So let's turn it on on my lab's C881 router.. for a router we're probably gonna focus on application traffic that consumes our WAN link's bandwidth. On my router this is the FastEthernet4 interface.

c881#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
c881(config)#int fa4                   
c881(config-if)#ip nbar protocol-discovery
c881(config-if)#exit
c881(config)#exit
c881#

This looks pretty straightforward but what is it doing?

Well under the hood the router is now using that hardware DPI feature to identify traffic going through that interface based on port numbers, HTTP headers and bi-directional signatures of more than 1000 different types of application traffic!

So to uncover this information you can simply run a command on the router to inspect what the router is forwarding...

 c881#sho ip nbar protocol-discovery interface fa4


 FastEthernet4

 Last clearing of "show ip nbar protocol-discovery" counters 3w1d


                              Input                    Output                
                              -----                    ------                
 Protocol                     Packet Count             Packet Count          
                              Byte Count               Byte Count            
                              30sec Bit Rate (bps)     30sec Bit Rate (bps)  
                              30sec Max Bit Rate (bps) 30sec Max Bit Rate (bps)
 ---------------------------- ------------------------ ------------------------
 binary-over-http             1751733                  435519                
                              2641958509               25586931              
                              0                        0                     
                              44533000                 400000                
 ssl                          400302                   296256                
                              375207665                101664657             
                              0                        0                     
                              24984000                 8217000               
 http                         33917                    16395                 
                              39392103                 2773633               
        
                              0                        0                     
                              4205000                  49000                 
 ms-wbt                       108460                   152204                
                              7946739                  148796581             
                              0                        0                     
                              57000                    3643000               
 google-services              5348                     2123                  
                              6582783                  193631                
                              0                        0                     
                              1108000                  15000                 
 google-docs                  3435                     644                   
                              4983105                  40454                 
                              0                        0                     
                              1057000                  10000                 
 bing                         7397                     4489                  
                              6839813                  1162902               
                              0                        0                     
                              362000                   23000                 
 ms-update                    10867                    5783                  
                              11230441                 2293489               
                              0                        0                     
                              299000                   45000                 
 dhcp                         26961                    3                     
        
                              11312762                 1053                  
                              0                        0                     
                              310000                   0                     
 speedtest                    2058                     848                   
                              2753421                  67484                 
                              0                        0                     
                              286000                   8000                  
 vnc                          347584                   430334                
                              21993854                 53980354              
                              0                        0                     
                              26000                    180000                
 ms-live-accounts             5318                     4032                  
                              3992467                  1684374               
                              0                        0                     
                              28000                    14000                 
 dns                          190508                   298844                
                              19587995                 25522124              
                              0                        0                     
                              15000                    8000                  
 unknown                      163508                   54227                 
                              16367578                 3638805               
                              0                        0                     
                              7000                     12000                 
        
 yahoo-accounts               49                       28                    
                              61053                    3916                  
                              0                        0                     
                              15000                    2000                  
 ssh                          9360                     6188                  
                              800132                   857610                
                              1000                     0                     
                              4000                     8000                  
 godaddy                      120                      97                    
                              59727                    11535                 
                              0                        0                     
                              8000                     2000                  
 pocket                       31                       23                    
                              30668                    2633                  
                              0                        0                     
                              8000                     1000                  
 windows-azure                4388                     3157                  
                              4304189                  281545                
                              0                        0                     
                              6000                     0                     
 twitter                      78                       104                   
                              49814                    14157                 
                              0                        0                     
        
                              4000                     2000                  
 netbios-ns                   32100                    600                   
                              3053262                  65856                 
                              0                        0                     
                              3000                     2000                  
 outbrain                     21                       23                    
                              15657                    3095                  
                              0                        0                     
                              4000                     1000                  
 facebook                     37                       37                    
                              12248                    4372                  
                              0                        0                     
                              3000                     2000                  
 skype                        2117                     1356                  
                              1633662                  214231                
                              0                        0                     
                              4000                     0                     
 ntp                          5107                     5178                  
                              459630                   466020                
                              0                        0                     
                              2000                     2000                  
 icmp                         6013                     8                     
                              643498                   560                   
        
                              0                        0                     
                              4000                     0                     
 ms-office-365                15                       17                    
                              11722                    2309                  
                              0                        0                     
                              3000                     1000                  
 youtube                      23                       23                    
                              10088                    2528                  
                              0                        0                     
                              3000                     1000                  
 amazon-web-services          31                       28                    
                              9580                     2681                  
                              0                        0                     
                              3000                     1000                  
 addthis                      18                       16                    
                              8910                     2927                  
                              0                        0                     
                              3000                     1000                  
 internet-audio-streaming     2736                     2211                  
                              1671636                  262136                
                              0                        0                     
                              2000                     0                     
 ping                         48942                    48942                 
        
                              2938526                  2938622               
                              0                        0                     
                              0                        0                     
 snmp                         1                        0                     
                              80                       0                     
                              0                        0                     
                              0                        0                     
 Total                        3168583                  1769737               
                              3185923317               372543205             
                              1000                     0                     
                              77356000                 12650000              
c881#

What's useful here and how can I use this information?

Well the counters here start when NBAR is turned on and runs for as long as the router is up...so this is a running total of packets and bytes (not so useful).

But the Bit Rate (that's the current traffic going through) and the Max Bit Rate values are extremely useful...let's focus on the Top 5 consumers of bandwidth on our WAN interface...

c881#sho ip nbar protocol-discovery interface fa4 stats max-bit-rate top-n 5


 FastEthernet4

 Last clearing of "show ip nbar protocol-discovery" counters 3w1d


                              Input                    Output                 
                              -----                    ------                 
 Protocol                     30sec Max Bit Rate (bps) 30sec Max Bit Rate (bps)
 ---------------------------- ------------------------ ------------------------
 binary-over-http             44533000                 400000                 
 ssl                          24984000                 8217000                
 http                         4205000                  49000                  
 ms-wbt                       57000                    3643000                
 google-services              1108000                  15000                  
 Total                        77356000                 12650000               
c881#

44Mbps peak download via HTTP (yeah I've been downloading some ISO images recently)...and an 8.2Mbps upload from my lab Linux box...

We've just uncovered some of the top applications within our network! And what has it cost us? Virtually nothing!

⚙Is there an impact on the CPU of running NBARv2 on my router? Yes (even for my c881 it only adds maybe 3-5% CPU load, less in larger routers!)...so test it in the lab to feel confident, but I've had my customers test this and then successfully deploy this across thousands of ISR G2, ISR 4K and ASR1K platforms without incident. 

The benefit of being able to gain visibility to a customers real applications is invaluable!

🚩One customer turned on NBARv2 on a sample set of production routers and was horrified at what they saw...unauthorized video streaming/sharing consuming all their precious WAN bandwidth at it's peak! No wonder their production processes were running slow!

 Now this is good but how do I look into the actual flows of application traffic?

Netflow

Or in our case Flexible Netflow!

Forget the original Netflow standards and go straight to Netflow v9 or IPFIX where NBAR can paste in the application ID into those flow records it sends to the Netflow collector...

Let's set up our FNF configs (we'll talk about Netflow collector choices later...)

First with Netflow v9 or IPFIX we define the fields we want to match or collect in our Netflow records

Here's my standard Netflow record configuration:

!
flow record FLOWREC
 match ipv4 dscp
 match ipv4 protocol
 match ipv4 source address
 match ipv4 destination address
 match transport source-port
 match transport destination-port
 match interface input
 match flow direction
 match application name
 collect routing forwarding-status
 collect ipv4 ttl
 collect transport tcp window-size
 collect transport tcp flags
 collect interface output
 collect counter bytes
 collect counter packets
 collect timestamp sys-uptime first
 collect timestamp sys-uptime last
!

You don't strictly need them all but I've seen uses for most of these fields... but let's pick out my fav!

match application name - this one puts the NBARv2 application ID information into the Netflow records (note it's a number and not really the name but I'll get back to this...)

Now we need to say where we are going to send our Netflow records (i.e our Netflow collector system)...we'll see that even without a Netflow collector we can still inspect the Netflow cache when troubleshooting problems on a customer's network

!
flow exporter FLOWEXP
 destination 192.168.2.52
 source Vlan200
 transport udp 2055
 export-protocol ipfix
 option interface-table
 option application-table
!

Alongside the destination IP address of the Netflow collector we also have to specify the UDP port that the Netflow Collector will be listening on (for the true nerds you may wanna test that you can receive UDP traffic on that port on the server using the 'nc' or 'netcat' command...geek out folks!)

The rest makes sense but what are the option config lines all about?

Well remember we said the Netflow records themselves only send the NBAR application ID (a weird looking number such as 03:000021). The option lines tell our router to send a couple of lookup tables every 10 minutes...The important one for us is the application-table which is sent as a burst of custom Netflow records that give names to the application IDs (in our example 03:000021 would translate to 'ssh'). The Netflow collector stores these tables so it can lookup and translate the application IDs into readable names.

The second table we're sending every 10 minutes is a lookup table of the interface-table (again a translation table of ifIndex entries for inbound or outbound interface IDs into the actual interface names we know and love!)

The last piece that ties the flow record and the flow exporter together is the 'flow monitor'...

!
flow monitor FLOWMON
 exporter FLOWEXP
 record FLOWREC
!        
And lastly we have to decide which interface we will apply this 'flow monitor' to (in our case, and in most cases, to the same WAN interface where we turned on NBARv2):

!
interface FastEthernet4
 ip nbar protocol-discovery
 ip flow monitor FLOWMON input
 ip flow monitor FLOWMON output
!

That's the basics of a very useful set of features that allow you to collect application information from a router!

And if you wanted to see what's in the flow cache while troubleshooting an issue you could simply use the 'show flow mon FLOWMON cache' command to uncover what is happening right now!

c881#sho flow mon FLOWMON cache format csv | sec ssh 
IPV4 SRC ADDR,IPV4 DST ADDR,TRNS SRC PORT,TRNS DST PORT,INTF INPUT,FLOW DIRN,IP DSCP,IP PROT,APP NAME,ip fwd status,tcp window size,tcp flags,intf output,bytes,pkts,time first,time last,http referer,ip ttl
x.x.x.181,10.0.1.2,11429,22,Fa4,Input,0x00,6,port ssh,Consume,65535,0x18,Null,20584,255,20:12:58.458,20:13:43.998,,46
c881#

And we see the format of the flow record information it will send to the Netflow collector...

That's enough to whet your appetite for further dabblings with uncovering the traffic within your network! Stay tuned

Beards out   ? : {)




 

Humourous and somewhat factual: Beards' disclaimer...part geek/part fool!

The intent of my blog is simply to share some of my experiences with various geeky features, tools and devices...and have a little fun along the way.

I'll not mention names of companies, customers or identify situations apart from in the vaguest terms (unless it's something funny! Joking!!!)

I'll try not to express my own opinions on things, products or software and simply 'stick to the facts Ma'am!'

No animals were harmed in the making of this blog...