Status: March 2023 - Note on progress, issues identified and work arounds or IOS XE replacement outlined
Today's task is to migrate Internet connection from Cisco ISR 2901 to ISR 4331 Router.
The reason is that:
- ISR 2901 is now over 10 years in service and EOL
- the interface on the ISR 2901 for ISP (via nbn) is via FastEthernet (HWIC-2FE Module) and I need to upgrade this to 1GBE
- Can now get a used ISR 4331 at price not much more than an additional 4 Port 1GBE Ethernet Switch Module (HWIC-4ESW)
- New router means new IOS which has benefits from throughput, features and bug fix (security) perspective
While the router has been humming away for over ten years, I would have thought that many others had completed simillar upgrade, so was a little surprised that there was not much in terms of specific information on the migration process.
My existing ISR 2901 includes: SEC & UC licenses and I briefly used DATA license to do some smart routing to handle fail over to alternate service. So it is running Zone Based Firewall & Ciso Unified Communications (SIP Proxy).
The IOS configuration consists of:
- Main configuration - mostly device neutral things like: hostname, domain, time and dns server details, dhcp pools, ip services to run
- Interface Additions - Lookbacks, Virtual Templates, Switch Interface vlan partitioning
- Main IP configuration - device neutral: Interface & vlan IP adddresss for IP Subnets, routing and nats
- Miscellaneous optional configuation - logging, netflow, ip performance and behaviour configurations
- Security/Firewall configuration - by far the largest part of the configuration with all the access lists, security zones, policy rules etc
- Unified Communications configuration - sip agents, dail plans etc
So process to migrate is:
- Break existing monolithic IOS config into sections, so these can be managed seperately
- Define the mapping of between ISR2901 Interfaces and vlans and new ISR 4331 Interfaces
- Establish initial admin IP connectivity (ssh, tftp)
- Starting with a clean configuation on ISR 4331 add the following into this configuration in this order: Main, Interface Additions, Security/Firewall, Main IP, Misc, Unified Comms
Doing things in this order helps avoid configuration dependency issues when you move the configuration into new router.
Differences to Manage
In moving from 2901 to 4331 the Interface naming changes, so first part of change is to create an interfaces mapping sheet. This has all the source interfaces on the 2901 and mapping of these to the interfaces / vlans on the 4331.
As virtually all of the configuration with exception of route definitions is via IP addresss, the physical interface name changes only impacts a small part of the Main IP configuration.
The hard part of the migration is due to changes in the IOS commands/features across the two routers. This requires digging into the details of change.
The "Issue #" items cover the impacts of the IOS changes.
NOTE #1: IOS XE only supports Zone-Based Firewall (i.e. define policys based on traffic classes and zones and then enrolling interfaces into zones) and not Context-Based Access Control Firewall (CBAC FW) (i.e. define access control lists and traffic types and attaching control lists to interfaces). So if you have very old CBAC FW on your IOS Router, then you will need to start from scratch with IOS XE.
NOTE #2: While IOS XE does not support CBAC FW, it still uses CBAC based mechanisms to define extended access lists and policy maps. So ZBFW has superset of features, but does not use access lists on interfaces, instead it enrolls interfaces into zones.
Issue #1 - IOS Change - ip virtual-reassembly
On my ISR 2901 15.5 "ip virtual-reassembly" definitions can be specified as being for in bound interfaces:
- "ip virtual-reassembly in" - to allow application level assembly the router need to hold and reassemble the message to do protocol compliance check.
In IOS XE 16.6.4 on ISR 4331 the specification of the direction "in" is not supported so need to change these configurations to:
- "ip virtual-reassembly" - so the "in" is redundant and now removed.
Issue #2 - IOS XE does not support "user" defined Port and Application Mapping (PAM)
The Cisco firewall uses a combination of "Context-Based Access Controls" (CBAC) and "Zone Base Firewall" (ZBFW) as it primary way of managing security configuration.
The CBAC uses port to application mapping to allow specification of security policy based on application traffic. To do this it uses IP port definitions i.e.: http == port 80, ssh == port 22, https == port 443 etc.
On IOS 15.5 these application mappings can be extented to include user defined mappings (with convention that all user mappings have a name prefix "user-").
On IOS XE this support was dropped, which is not typical with IOS which generally maintains high level of backward compatability with older versions.
As you cannot create your own definitions, the only workaround is the "hijack" some unused "system definitions". Here is the list of available applications (or services - as in "/etc/services" file..), from IOS XE 16.6.4:
#ip port-map ? 802-11-iapp IEEE 802.11 WLANs WG IAPP ace-svr ACE Server/Propagation aol America-Online Instant Messenger appleqtc Apple QuickTime bgp Border Gateway Protocol biff Bliff mail notification bootpc Bootstrap Protocol Client bootps Bootstrap Protocol Server cddbp CD Database Protocol cifs CIFS cisco-fna Cisco FNATIVE cisco-net-mgmt cisco-net-mgmt cisco-svcs cisco license/perf/GDP/X.25/ident svcs cisco-sys Cisco SYSMAINT cisco-tdp Cisco TDP cisco-tna Cisco TNATIVE citrix Citrix IMA/ADMIN/RTMP citriximaclient Citrix IMA Client clp Cisco Line Protocol creativepartnr Creative Partnr creativeserver Creative Server cuseeme CUSeeMe Protocol daytime Daytime (RFC 867) dbase dBASE Unix dbcontrol_agent Oracle dbControl Agent po ddns-v3 Dynamic DNS Version 3 dhcp-failover DHCP Failover discard Discard port dns Domain Name Server dnsix DNSIX Securit Attribute Token Map echo Echo port entrust-svc-handler Entrust KM/Admin Service Handler entrust-svcs Entrust sps/aaas/aams exec Remote Process Execution fcip-port FCIP finger Finger ftp File Transfer Protocol ftps FTP over TLS/SSL gdoi GDOI giop Oracle GIOP/SSL gopher Gopher gtpv0 GPRS Tunneling Protocol Version 0 gtpv1 GPRS Tunneling Protocol Version 1 h225ras H225 RAS over Unicast h225rasMcast H225 RAS over Multicast h323 H.323 Protocol (e.g, MS NetMeeting, Inte h323callsigalt h323 Call Signal Alternate hp-alarm-mgr HP Performance data alarm manager hp-collector HP Performance data collector hp-managed-node HP Performance data managed node hsrp Hot Standby Router Protocol http Hypertext Transfer Protocol https Secure Hypertext Transfer Protocol ica ica (Citrix) icabrowser icabrowser (Citrix) ident Authentication Service igmpv3lite IGMP over UDP for SSM imap Internet Message Access Protocol imap3 Interactive Mail Access Protocol 3 imaps IMAP over TLS/SSL ipass IPASS ipsec-msft Microsoft IPsec NAT-T ipx IPX irc Internet Relay Chat Protocol irc-serv IRC-SERV ircs IRC over TLS/SSL ircu IRCU isakmp ISAKMP iscsi iSCSI iscsi-target iSCSI port kazaa KAZAA kerberos Kerberos kermit kermit l2tp L2TP/L2F ldap Lightweight Directory Access Protocol ldap-admin LDAP admin server port ldaps LDAP over TLS/SSL login Remote login lotusmtap Lotus Mail Tracking Agent Protocol lotusnote Lotus Note mgcp Media Gateway Control Protocol microsoft-ds Microsoft-DS ms-cluster-net MS Cluster Net ms-dotnetster Microsoft .NETster Port ms-sna Microsoft SNA Server/Base ms-sql Microsoft SQL ms-sql-m Microsoft SQL Monitor msexch-routing Microsoft Exchange Routing msnmsgr MSN Instant Messenger msrpc Microsoft Remote Procedure Call mysql MySQL n2h2server N2H2 Filter Service Port ncp NCP (Novell) net8-cman Oracle Net8 Cman/Admin netbios-dgm NETBIOS Datagram Service netbios-ns NETBIOS Name Service netbios-ssn NETBIOS Session Service netshow Microsoft NetShow netstat Variant of systat nfs Network File System nntp Network News Transport Protocol ntp Network Time Protocol oem-agent OEM Agent (Oracle) oracle Oracle oracle-em-vp Oracle EM/VP oraclenames Oracle Names orasrv Oracle SQL*Net v1/v2 pcanywheredata pcANYWHEREdata pcanywherestat pcANYWHEREstat pop3 Post Office Protocol - Version 3 pop3s POP3 over TLS/SSL pptp PPTP pwdgen Password Generator Protocol qmtp Quick Mail Transfer Protocol r-winsock remote-winsock radius RADIUS & Accounting rdb-dbs-disp Oracle RDB realmedia RealNetwork's Realmedia Protocol realsecure ISS Real Secure Console Service Port router Local Routing Process rsvd RSVD rsvp-encap RSVP ENCAPSULATION-1/2 rsvp_tunnel RSVP Tunnel rtc-pm-port Oracle RTC-PM port rtelnet Remote Telnet Service rtsp Real Time Streaming Protocol send SEND shell Remote command sip Session Initiation Protocol sip-tls SIP-TLS skinny Skinny Client Control Protocol sms SMS RCINFO/XFER/CHAT smtp Simple Mail Transfer Protocol snmp Simple Network Management Protocol snmptrap SNMP Trap socks Socks sql-net SQL-NET sqlserv SQL Services sqlsrv SQL Service ssh SSH Remote Login Protocol sshell SSLshell ssp State Sync Protocol streamworks StreamWorks Protocol stun cisco STUN sunrpc SUN Remote Procedure Call syslog SysLog Service syslog-conn Reliable Syslog Service tacacs Login Host Protocol (TACACS) tacacs-ds TACACS-Database Service tarantella Tarantella telnet Telnet telnets Telnet over TLS/SSL tftp Trivial File Transfer Protocol time Time timed Time server tr-rsrb cisco RSRB ttc Oracle TTC/SSL uucp UUCPD/UUCP-RLOGIN vdolive VDOLive Protocol vqp VQP webster Network Disctionary who Who's service wins Microsoft WINS x11 X Window System xdmcp XDM Control Protocol ymsgr Yahoo! Instant Messenger
For my "custom" services I had combination of: service running over SSL/TLS, are http variations (WebDAV, CalDAV, CardDAV) or relate to XMPP (Jabber).
For the SSL based protocols the router cannot do inspection of contents as the packets are encrypted and so best candidate for hijacking is old unsed SSL based protocol.
For http variations the best "hijacking" candidate would be any of the "standard" systems defined ones that is based on http (I could not find such a candidate).
Here is an example of my approach for XMPP traffic:
! ! XMPP custom portmap: ! XMPP uses port 5222 / 5223 for user traffic & ! XMPP uses port 5269 for XMPP <-> XMPP Server Federation traffic ! ip port-map user-xmpp-5222 port tcp 5222 description xmpp 5222 ip port-map user-xmpp-5223 port tcp 5223 description xmpp 5223 ip port-map user-xmpp-fed port tcp 5269 description xmpp federation ! ! To replace these need to: ! 1. Create an standard access list (ie < 100) ! 2. Choice you application to hijack ! 3. Create port-map items using hijacked applicaton ! ! Looking at details of this the firewall access configuration found the public IP address of the XMPP server so the following access list was defined: ! access-list 10 remark XMPP port-map user-xmpp-5223 user-xmpp-fed substitute access-list 10 permit XXX.XXX.XXX.35 ! ! In this case for XMPP I decided to hijack the Lotus Notes and IPX PAM as ! there is no Lotus Notes server or IPX on my network and Lotus Notes includes two applications: lotusnote, lotusmtap with IPX for notifications ! ! So to ensure that the hijack configuration does not open any unneeded ports, I disabled the fault PAM and added new ones: ! no ip port-map lotusnote no ip port-map lotusmtap no ip port-map ipx ip port-map lotusnote port tcp 5222 list 10 description xmpp 5222 ip port-map ipx port tcp 5223 list 10 description xmpp 5223 ip port-map lotusmtap port tcp 5269 list 10 description xmpp federation 5269 ! ! This creates a PAM that only applies against the hosts defined in the access list ->
This work around approach was applied to all the custom defined PAMs.
The corresponding ZBFW inspection and policy rules have to then refer to the "hijacked" application name and not the original "user-<APP>" one.
- lotusnote, lotusmtap - for XMPP
- qmtp - for SMTP over SSL (smtps) (as smtp is used for mail transfer)
- pop3s - for CalDAV & CardDAV over SSL (as all mail collection is via imap)
- oem-agent (Oracle) - for Apple Push Notifications
- ttc (Oracle) - for Harbor Registry https. which is over SSL
As this issue is pretty fundemental I posted work around in Cisco Community.
Issue #3 - IOS XE does not support "ip inspect"
The "ip inspect" command from IOS 15.5 was used to configure the application inspection.
This is now done using "class-map type inspect <Protocol> .." and "appfw".
Here are examples for http, im & imap:
! ! Inspection policy for http ! which come originally from CCP (Cisco Configuration Proffessional) ! hence "cpp-" ! policy-map type inspect http ccp-action-app-http class type inspect http ccp-http-blockparam log allow class type inspect http ccp-app-httpmethods log allow class type inspect http ccp-http-allowparam log allow ! ! For ios xe this becames ! Noting that: Most http traffic will be via https ! and maintaining original ccp config ! appfw policy-name CCP_MEDIUM application http strict-http action allow alarm request-method rfc default action allow alarm request-method extension default action allow alarm ! ! And add the follow to replace the original policy-map ! class-map type inspect match-any HTTP-PROTOCOL match protocol http ! ! Then use put the following into your zone policy's ! policy-map type inspect ZONE-TO-ZONE-POLICY ... ... class type inspect HTTP-PROTOCOL inspect ... ... class class-default ! ! Same pattern applies to other cases ! ! Inspection policy for im ! policy-map type inspect im ccp-action-app-im class type inspect aol ccp-app-aol class type inspect msnmsgr ccp-app-msn class type inspect ymsgr ccp-app-yahoo class type inspect aol ccp-app-aol-otherservices class type inspect msnmsgr ccp-app-msn-otherservices class type inspect ymsgr ccp-app-yahoo-otherservices ! ! For ios xe this becomes ! appfw policy-name CCP_MEDIUM application im aol ... ... application im msn ... ... application im yahoo ... ... ! ! class-map type inspect match-any IM-PROTOCOLS match protocol ymsgr match protocol aol match protocol mnsmsgr ! ! As per HTTP-PROTOCOL above add IM-PROTOCOLS to your zone policies ! ! ! Inspection policy for imap ! policy-map type inspect imap ccp-action-imap class type inspect imap ccp-app-imap log ! ! For ios xe this becames ! just inspect with new IOS XE
My configuration was initially created using Cisco Configuration Professional (CCP) and there are a lot of ZBFW configration rules, for application level inspection, in this. Now most of the traffic traversing the internet is running over SSL/TLS so FW is really only doing TCP level validation. So some of the more detailed application level configuration (http payload checking for example), is unlikely to be used. In particular there should not no IMAP or POP3 exposed on network, rather these should be via IMAPS & POP3S. So take this into account when reviewng the security configuration.
Issue #4 - IOS XE does not support "parameter-map type protocol-info"
This configuration information should be moved to "appfw" for example:
! ! Old parameter0map type protocol ... ! parameter-map type protocol-info msn-servers server name messenger.hotmail.com server name gateway.messenger.hotmail.com server name webmessenger.msn.com ! ! Same configuration via appfw ! appfw policy-name CCP_MEDIUM application im msn service default action allow alarm service text-chat action allow alarm server permit name messenger.hotmail.com server permit name gateway.messenger.hotmail.com server permit name webmessenger.msn.com audit-trail on !
Issue #5 - Out of Order Packets and Packet Reassembly
With IOS there is a global configuration for management of "out of order" packets: "parameter-map type 000 global". This is no longer supported with IOS XE.
The alternative to this is to configure "ip virtual-reassembly" on each interface (this configure this for ingress & egress).
See "Issue #6 - ip flow configuration" for example.
Issue #6 - ip flow confguration
With IOS ip flow can be configured on a per interface basis via "ip flow in" and "ip flow out".
With IOS XE this is configured on a per interface basis via "ip flow monitor application-mon input" & "ip flow monitor application-mon output"
Here is example:
! interface GigabitEthernet0/0/0 description $ETH-GBE000$FW_IN$$ETH-WAN$ ip flow monitor application-mon input ip flow monitor application-mon output ip address XXX.XXX.XXX.XXX 255.255.255.0 ip nat outside ip nbar protocol-discovery zone-member security IN negotiation auto ip virtual-reassembly !
Issue #7 - ip flow-top-talkers
What IOS XE Version ?
When doing your migration from 2901 to 4331 you will need to select your IOS XE version. For 4300 the options are 15.X, 16.X & 17.X releases. In general it is best to select the most "mature" version (i.e. the one with least security bugs), which has support for the hardware & features you are after. With 4300 there have been some significant differences that need to be factored into the selection: Smart Licensing (which change from the hardware embedded license to an online one) and SD-WAN which is a move away from traditional IOS configuration managed approach to an cloud automated management solution for connectivity management.
To complicate things a little the IOS XE release used a versioning convention that has the release of the Linux OS 3.x and the release of the the IOS service on top of this (15.X). With the 16.X release the 3.X prefix was removed and the IOS XE release followed the same pattern as other IOS related release, so 15 -> 16 -> 17. In addition the 16.X release took on release names: Everest, Fuji, Amsterdam, Gilbralter etc.
As I started with IOS 15.5 base configuration, my initial migration was to 15.5 IOS XE as I thought this would ensure configuration compatability. This was not the case.
To get support for some additional hardware modules, I moved to 16.6 only to find that there were critical security vulnerabilities in this release which meant moving up to 16.9 or 16.12 or even to 17.X series.
However a move to 16.12 means you must shift to Smart Licensing and the last release that does not mandate Smart Licensing is 16.9.8.
Using the "CISCO Software Checker" tool you can do a lookup on particular versions to see the list of know defects. The version within 16.X series with least number if known defects is final 16.12.9.
The current (April 2023) Cisco recommended release are: 17.6.5 & 17.3.5.
To make your choice you should:
- Review release notes for routing feature and HW support
- Review the Software Checker reports for security vulnerabilities and so an assessment against these for the your deployment, as while there might be vulnerabilities they only apply under certain conditions. Example are: "virtual reassembly" defect occures if you use large (jumbo) MTU settings, bugs which only show up when using DHCPv6, defects with certain MPLS configuration or defects which only affect particular HW).
- Decide whether you want to stay on conventional or more to Smart Licensing model.
The migration of configuration is now complete. Before pulling out the old ISR 2901 and putting in ISR 4331 I will do "Greenbone Scan" (see blog "OpenVAS / Greenbone Vulnerability Management (GVM) / Greenbone Source Edition on AWS") to check that config is at least as secure as current ISR 2901.
Not unexpectly the security configuration took the most effort and getting this fully transitioned to ISR 4331 required creation of 16 config updates to address issues which mostly related to ZBFW configuration.
Unified Communications configuration by comparison only took 2 loads to move across, including PVDM transcode configurations.
Main IP took a few rounds as initially I did this before doing full security configuration, but defining interfaces needs to go in order of: Main (including NetFlow definitions), Addition Interface (to create loopback and virtual template definitions), then Security/Firewall and only then Main IP definitions , which has references to Netflow and Scecurity/Firewall definitions.
Most the effort was spent finding IOS XE equivalents to changed / removed IOS commands. So lots of links to Cisco and other documentation below in the "References & Links" section.
Due to critical defect in 16.6 (Everest) (see cert notice below) my final configuration deployment was with 16.9 (Fuji), which required no further update to migrated configuration.
Happy IOS to IOS XE migrations.
References & Links:
- Cisco 4000 Series ISRs Software Configuration Guide - Using Cisco IOS XE Software - the initial software guide
- Cisco Configuring Application to Port Mapping - very old IOS PAM configuration information and this blog has a reasonable summary "Port to Application Mapping - Classic Firewall"
- PAM with IOS XE Issues - this was been raised as issue on Cisco Community a few times, but no clear resolution or workaround outlined: "IP Port map custom protocol in IOS XE" & "ISR 4331 cannot add ip port-map user define port"
- PAM with IOS XE Workaround - the workaround posted in Cisco Community..
- CBAC Configuration Tutorial - this tutorial provides configuration example with real world context
- "Inspection on cisco router ISR4431" - covers removal of "ip inspect" in ISR 4000 IOS XE routers
- Cisco IOS XE Zone Based Firewall Changes - there are changes to ZBFW with IOS XE which are covered in: "Cisco - Security Configuration Guide Zone-Based Firewall - Cisco IOS XE Release 3S" and the following IOS XE security configuration updates
- Cisco IOS XE 16.1 Zone-Base Firewall Updates - "Security Configuration Guide: Zone-Based Policy Firewall, Cisco IOS XE Everest 16.6.1"
- Cisco IOS XE 17 Zone Based Firewall Updates - "Security Configuration Guide - Zone-Based Policy Firewall, Cisco IOS XE 17"
- Classic IOS ZBFW - "Understand the Zone-Based Policy Firewall Design" covers classic IOS but is generally applicable to IOS XE as well
- "HTTP Services Configuration Guide, Cisco IOS Release 15M&T" - provides details on http inspection service and its configuration
- "firewall messages" due to ccp (Cisco Configuration Professional) generated config - Cisco Configuration Professional (now out of support) generated my initial configuration and there are still remants of this, here is useful post for other who may have these remnant.
- Auscert - ESB-2020.3297 - Security Bulletin for IOS XE ZBFW with dropped packets logging enabled
- Cisco Software Checker - provides an easy way to seeing what defects and vulnerabilities exists with a given software release.
- "Greenbone Vulnerability Management" - test your config before deploying using Greenbone with local install on KVM/QEMU and via AWS
Opening picture is from: "Finding the Way - Waterfowl have amazing navigational abilities that guide their migrations"
Closing Duck Picture - is uncreditted but pops up in lots of article on good public behaviour