Cisco IOS 15.5 (ISR 2901) to IOS XE 16.6 / 16.9 (ISR 4331) Migration
Status: March 2023 - Note on progress, issues identified and work arounds or IOS XE replacement outlined
Today's task is to migrate Internet connection from Cisco ISR 2901 to ISR 4331 Router.
The reason is that:
- ISR 2901 is now over 10 years in service and EOL
- the interface on the ISR 2901 for ISP (via nbn) is via FastEthernet (HWIC-2FE Module) and I need to upgrade this to 1GBE
- Can now get a used ISR 4331 at price not much more than an additional 4 Port 1GBE Ethernet Switch Module (HWIC-4ESW)
- New router means new IOS which has benefits from throughput, features and bug fix (security) perspective
While the router has been humming away for over ten years, I would have thought that many others had completed simillar upgrade, so was a little surprised that there was not much in terms of specific information on the migration process.
Hence post.
The Process
My existing ISR 2901 includes: SEC & UC licenses and I briefly used DATA license to do some smart routing to handle fail over to alternate service. So it is running Zone Based Firewall & Ciso Unified Communications (SIP Proxy).
The IOS configuration consists of:
- Main configuration - mostly device neutral things like: hostname, domain, time and dns server details, dhcp pools, ip services to run
- Interface Additions - Lookbacks, Virtual Templates, Switch Interface vlan partitioning
- Main IP configuration - device neutral: Interface & vlan IP adddresss for IP Subnets, routing and nats
- Miscellaneous optional configuation - logging, netflow, ip performance and behaviour configurations
- Security/Firewall configuration - by far the largest part of the configuration with all the access lists, security zones, policy rules etc
- Unified Communications configuration - sip agents, dail plans etc
So process to migrate is:
- Break existing monolithic IOS config into sections, so these can be managed seperately
- Define the mapping of between ISR2901 Interfaces and vlans and new ISR 4331 Interfaces
- Establish initial admin IP connectivity (ssh, tftp)
- Starting with a clean configuation on ISR 4331 add the following into this configuration in this order: Main, Interface Additions, Security/Firewall, Main IP, Misc, Unified Comms
Doing things in this order helps avoid configuration dependency issues when you move the configuration into new router.
Differences to Manage
In moving from 2901 to 4331 the Interface naming changes, so first part of change is to create an interfaces mapping sheet. This has all the source interfaces on the 2901 and mapping of these to the interfaces / vlans on the 4331.
As virtually all of the configuration with exception of route definitions is via IP addresss, the physical interface name changes only impacts a small part of the Main IP configuration.
The hard part of the migration is due to changes in the IOS commands/features across the two routers. This requires digging into the details of change.
The "Issue #" items cover the impacts of the IOS changes.
NOTE #1: IOS XE only supports Zone-Based Firewall (i.e. define policys based on traffic classes and zones and then enrolling interfaces into zones) and not Context-Based Access Control Firewall (CBAC FW) (i.e. define access control lists and traffic types and attaching control lists to interfaces). So if you have very old CBAC FW on your IOS Router, then you will need to start from scratch with IOS XE.
NOTE #2: While IOS XE does not support CBAC FW, it still uses CBAC based mechanisms to define extended access lists and policy maps. So ZBFW has superset of features, but does not use access lists on interfaces, instead it enrolls interfaces into zones.
Issue #1 - IOS Change - ip virtual-reassembly
On my ISR 2901 15.5 "ip virtual-reassembly" definitions can be specified as being for in bound interfaces:
- "ip virtual-reassembly in" - to allow application level assembly the router need to hold and reassemble the message to do protocol compliance check.
In IOS XE 16.6.4 on ISR 4331 the specification of the direction "in" is not supported so need to change these configurations to:
- "ip virtual-reassembly" - so the "in" is redundant and now removed.
Issue #2 - IOS XE does not support "user" defined Port and Application Mapping (PAM)
The Cisco firewall uses a combination of "Context-Based Access Controls" (CBAC) and "Zone Base Firewall" (ZBFW) as it primary way of managing security configuration.
The CBAC uses port to application mapping to allow specification of security policy based on application traffic. To do this it uses IP port definitions i.e.: http == port 80, ssh == port 22, https == port 443 etc.
On IOS 15.5 these application mappings can be extented to include user defined mappings (with convention that all user mappings have a name prefix "user-").
On IOS XE this support was dropped, which is not typical with IOS which generally maintains high level of backward compatability with older versions.
As you cannot create your own definitions, the only workaround is the "hijack" some unused "system definitions". Here is the list of available applications (or services - as in "/etc/services" file..), from IOS XE 16.6.4:
#ip port-map ?
802-11-iapp IEEE 802.11 WLANs WG IAPP
ace-svr ACE Server/Propagation
aol America-Online Instant Messenger
appleqtc Apple QuickTime
bgp Border Gateway Protocol
biff Bliff mail notification
bootpc Bootstrap Protocol Client
bootps Bootstrap Protocol Server
cddbp CD Database Protocol
cifs CIFS
cisco-fna Cisco FNATIVE
cisco-net-mgmt cisco-net-mgmt
cisco-svcs cisco license/perf/GDP/X.25/ident svcs
cisco-sys Cisco SYSMAINT
cisco-tdp Cisco TDP
cisco-tna Cisco TNATIVE
citrix Citrix IMA/ADMIN/RTMP
citriximaclient Citrix IMA Client
clp Cisco Line Protocol
creativepartnr Creative Partnr
creativeserver Creative Server
cuseeme CUSeeMe Protocol
daytime Daytime (RFC 867)
dbase dBASE Unix
dbcontrol_agent Oracle dbControl Agent po
ddns-v3 Dynamic DNS Version 3
dhcp-failover DHCP Failover
discard Discard port
dns Domain Name Server
dnsix DNSIX Securit Attribute Token Map
echo Echo port
entrust-svc-handler Entrust KM/Admin Service Handler
entrust-svcs Entrust sps/aaas/aams
exec Remote Process Execution
fcip-port FCIP
finger Finger
ftp File Transfer Protocol
ftps FTP over TLS/SSL
gdoi GDOI
giop Oracle GIOP/SSL
gopher Gopher
gtpv0 GPRS Tunneling Protocol Version 0
gtpv1 GPRS Tunneling Protocol Version 1
h225ras H225 RAS over Unicast
h225rasMcast H225 RAS over Multicast
h323 H.323 Protocol (e.g, MS NetMeeting, Inte
h323callsigalt h323 Call Signal Alternate
hp-alarm-mgr HP Performance data alarm manager
hp-collector HP Performance data collector
hp-managed-node HP Performance data managed node
hsrp Hot Standby Router Protocol
http Hypertext Transfer Protocol
https Secure Hypertext Transfer Protocol
ica ica (Citrix)
icabrowser icabrowser (Citrix)
ident Authentication Service
igmpv3lite IGMP over UDP for SSM
imap Internet Message Access Protocol
imap3 Interactive Mail Access Protocol 3
imaps IMAP over TLS/SSL
ipass IPASS
ipsec-msft Microsoft IPsec NAT-T
ipx IPX
irc Internet Relay Chat Protocol
irc-serv IRC-SERV
ircs IRC over TLS/SSL
ircu IRCU
isakmp ISAKMP
iscsi iSCSI
iscsi-target iSCSI port
kazaa KAZAA
kerberos Kerberos
kermit kermit
l2tp L2TP/L2F
ldap Lightweight Directory Access Protocol
ldap-admin LDAP admin server port
ldaps LDAP over TLS/SSL
login Remote login
lotusmtap Lotus Mail Tracking Agent Protocol
lotusnote Lotus Note
mgcp Media Gateway Control Protocol
microsoft-ds Microsoft-DS
ms-cluster-net MS Cluster Net
ms-dotnetster Microsoft .NETster Port
ms-sna Microsoft SNA Server/Base
ms-sql Microsoft SQL
ms-sql-m Microsoft SQL Monitor
msexch-routing Microsoft Exchange Routing
msnmsgr MSN Instant Messenger
msrpc Microsoft Remote Procedure Call
mysql MySQL
n2h2server N2H2 Filter Service Port
ncp NCP (Novell)
net8-cman Oracle Net8 Cman/Admin
netbios-dgm NETBIOS Datagram Service
netbios-ns NETBIOS Name Service
netbios-ssn NETBIOS Session Service
netshow Microsoft NetShow
netstat Variant of systat
nfs Network File System
nntp Network News Transport Protocol
ntp Network Time Protocol
oem-agent OEM Agent (Oracle)
oracle Oracle
oracle-em-vp Oracle EM/VP
oraclenames Oracle Names
orasrv Oracle SQL*Net v1/v2
pcanywheredata pcANYWHEREdata
pcanywherestat pcANYWHEREstat
pop3 Post Office Protocol - Version 3
pop3s POP3 over TLS/SSL
pptp PPTP
pwdgen Password Generator Protocol
qmtp Quick Mail Transfer Protocol
r-winsock remote-winsock
radius RADIUS & Accounting
rdb-dbs-disp Oracle RDB
realmedia RealNetwork's Realmedia Protocol
realsecure ISS Real Secure Console Service Port
router Local Routing Process
rsvd RSVD
rsvp-encap RSVP ENCAPSULATION-1/2
rsvp_tunnel RSVP Tunnel
rtc-pm-port Oracle RTC-PM port
rtelnet Remote Telnet Service
rtsp Real Time Streaming Protocol
send SEND
shell Remote command
sip Session Initiation Protocol
sip-tls SIP-TLS
skinny Skinny Client Control Protocol
sms SMS RCINFO/XFER/CHAT
smtp Simple Mail Transfer Protocol
snmp Simple Network Management Protocol
snmptrap SNMP Trap
socks Socks
sql-net SQL-NET
sqlserv SQL Services
sqlsrv SQL Service
ssh SSH Remote Login Protocol
sshell SSLshell
ssp State Sync Protocol
streamworks StreamWorks Protocol
stun cisco STUN
sunrpc SUN Remote Procedure Call
syslog SysLog Service
syslog-conn Reliable Syslog Service
tacacs Login Host Protocol (TACACS)
tacacs-ds TACACS-Database Service
tarantella Tarantella
telnet Telnet
telnets Telnet over TLS/SSL
tftp Trivial File Transfer Protocol
time Time
timed Time server
tr-rsrb cisco RSRB
ttc Oracle TTC/SSL
uucp UUCPD/UUCP-RLOGIN
vdolive VDOLive Protocol
vqp VQP
webster Network Disctionary
who Who's service
wins Microsoft WINS
x11 X Window System
xdmcp XDM Control Protocol
ymsgr Yahoo! Instant MessengerFor my "custom" services I had combination of: service running over SSL/TLS, are http variations (WebDAV, CalDAV, CardDAV) or relate to XMPP (Jabber).
For the SSL based protocols the router cannot do inspection of contents as the packets are encrypted and so best candidate for hijacking is old unsed SSL based protocol.
For http variations the best "hijacking" candidate would be any of the "standard" systems defined ones that is based on http (I could not find such a candidate).
Here is an example of my approach for XMPP traffic:
!
! XMPP custom portmap:
! XMPP uses port 5222 / 5223 for user traffic &
! XMPP uses port 5269 for XMPP <-> XMPP Server Federation traffic
!
ip port-map user-xmpp-5222 port tcp 5222 description xmpp 5222
ip port-map user-xmpp-5223 port tcp 5223 description xmpp 5223
ip port-map user-xmpp-fed port tcp 5269 description xmpp federation
!
! To replace these need to:
! 1. Create an standard access list (ie < 100)
! 2. Choice you application to hijack
! 3. Create port-map items using hijacked applicaton
!
! Looking at details of this the firewall access configuration found the public IP address of the XMPP server so the following access list was defined:
!
access-list 10 remark XMPP port-map user-xmpp-5223 user-xmpp-fed substitute
access-list 10 permit XXX.XXX.XXX.35
!
! In this case for XMPP I decided to hijack the Lotus Notes and IPX PAM as
! there is no Lotus Notes server or IPX on my network and Lotus Notes includes two applications: lotusnote, lotusmtap with IPX for notifications
!
! So to ensure that the hijack configuration does not open any unneeded ports, I disabled the fault PAM and added new ones:
!
no ip port-map lotusnote
no ip port-map lotusmtap
no ip port-map ipx
ip port-map lotusnote port tcp 5222 list 10 description xmpp 5222
ip port-map ipx port tcp 5223 list 10 description xmpp 5223
ip port-map lotusmtap port tcp 5269 list 10 description xmpp federation 5269
!
! This creates a PAM that only applies against the hosts defined in the access list
This work around approach was applied to all the custom defined PAMs.
The corresponding ZBFW inspection and policy rules have to then refer to the "hijacked" application name and not the original "user-<APP>" one.
I hijacked:
- lotusnote, lotusmtap - for XMPP
- qmtp - for SMTP over SSL (smtps) (as smtp is used for mail transfer)
- pop3s - for CalDAV & CardDAV over SSL (as all mail collection is via imap)
- oem-agent (Oracle) - for Apple Push Notifications
- ttc (Oracle) - for Harbor Registry https. which is over SSL
As this issue is pretty fundemental I posted work around in Cisco Community.
Issue #3 - IOS XE does not support "ip inspect"
The "ip inspect" command from IOS 15.5 was used to configure the application inspection.
This is now done using "class-map type inspect <Protocol> .." and "appfw".
Here are examples for http, im & imap:
!
! Inspection policy for http
! which come originally from CCP (Cisco Configuration Proffessional)
! hence "cpp-"
!
policy-map type inspect http ccp-action-app-http
class type inspect http ccp-http-blockparam
log
allow
class type inspect http ccp-app-httpmethods
log
allow
class type inspect http ccp-http-allowparam
log
allow
!
! For ios xe this becames
! Noting that: Most http traffic will be via https
! and maintaining original ccp config
!
appfw policy-name CCP_MEDIUM
application http
strict-http action allow alarm
request-method rfc default action allow alarm
request-method extension default action allow alarm
!
! And add the follow to replace the original policy-map
!
class-map type inspect match-any HTTP-PROTOCOL
match protocol http
!
! Then use put the following into your zone policy's
!
policy-map type inspect ZONE-TO-ZONE-POLICY
...
...
class type inspect HTTP-PROTOCOL
inspect
...
...
class class-default
!
! Same pattern applies to other cases
!
! Inspection policy for im
!
policy-map type inspect im ccp-action-app-im
class type inspect aol ccp-app-aol
class type inspect msnmsgr ccp-app-msn
class type inspect ymsgr ccp-app-yahoo
class type inspect aol ccp-app-aol-otherservices
class type inspect msnmsgr ccp-app-msn-otherservices
class type inspect ymsgr ccp-app-yahoo-otherservices
!
! For ios xe this becomes
!
appfw policy-name CCP_MEDIUM
application im aol
...
...
application im msn
...
...
application im yahoo
...
...
!
!
class-map type inspect match-any IM-PROTOCOLS
match protocol ymsgr
match protocol aol
match protocol mnsmsgr
!
! As per HTTP-PROTOCOL above add IM-PROTOCOLS to your zone policies
!
!
! Inspection policy for imap
!
policy-map type inspect imap ccp-action-imap
class type inspect imap ccp-app-imap
log
!
! For ios xe this becames
!
just inspect with new IOS XE
My configuration was initially created using Cisco Configuration Professional (CCP) and there are a lot of ZBFW configration rules, for application level inspection, in this. Now most of the traffic traversing the internet is running over SSL/TLS so FW is really only doing TCP level validation. So some of the more detailed application level configuration (http payload checking for example), is unlikely to be used. In particular there should not no IMAP or POP3 exposed on network, rather these should be via IMAPS & POP3S. So take this into account when reviewng the security configuration.
Issue #4 - IOS XE does not support "parameter-map type protocol-info"
This configuration information should be moved to "appfw" for example:
!
! Old parameter0map type protocol ...
!
parameter-map type protocol-info msn-servers
server name messenger.hotmail.com
server name gateway.messenger.hotmail.com
server name webmessenger.msn.com
!
! Same configuration via appfw
!
appfw policy-name CCP_MEDIUM
application im msn
service default action allow alarm
service text-chat action allow alarm
server permit name messenger.hotmail.com
server permit name gateway.messenger.hotmail.com
server permit name webmessenger.msn.com
audit-trail on
!Issue #5 - Out of Order Packets and Packet Reassembly
With IOS there is a global configuration for management of "out of order" packets: "parameter-map type 000 global". This is no longer supported with IOS XE.
The alternative to this is to configure "ip virtual-reassembly" on each interface (this configure this for ingress & egress).
See "Issue #6 - ip flow configuration" for example.
Issue #6 - ip flow confguration
With IOS ip flow can be configured on a per interface basis via "ip flow in" and "ip flow out".
With IOS XE this is configured on a per interface basis via "ip flow monitor application-mon input" & "ip flow monitor application-mon output"
Here is example:
!
interface GigabitEthernet0/0/0
description $ETH-GBE000$FW_IN$$ETH-WAN$
ip flow monitor application-mon input
ip flow monitor application-mon output
ip address XXX.XXX.XXX.XXX 255.255.255.0
ip nat outside
ip nbar protocol-discovery
zone-member security IN
negotiation auto
ip virtual-reassembly
!Issue #7 - ip flow-top-talkers
What IOS XE Version ?
When doing your migration from 2901 to 4331 you will need to select your IOS XE version. For 4300 the options are 15.X, 16.X & 17.X releases. In general it is best to select the most "mature" version (i.e. the one with least security bugs), which has support for the hardware & features you are after. With 4300 there have been some significant differences that need to be factored into the selection: Smart Licensing (which change from the hardware embedded license to an online one) and SD-WAN which is a move away from traditional IOS configuration managed approach to an cloud automated management solution for connectivity management.
To complicate things a little the IOS XE release used a versioning convention that has the release of the Linux OS 3.x and the release of the the IOS service on top of this (15.X). With the 16.X release the 3.X prefix was removed and the IOS XE release followed the same pattern as other IOS related release, so 15 -> 16 -> 17. In addition the 16.X release took on release names: Everest, Fuji, Amsterdam, Gilbralter etc.
As I started with IOS 15.5 base configuration, my initial migration was to 15.5 IOS XE as I thought this would ensure configuration compatability. This was not the case.
To get support for some additional hardware modules, I moved to 16.6 only to find that there were critical security vulnerabilities in this release which meant moving up to 16.9 or 16.12 or even to 17.X series.
However a move to 16.12 means you must shift to Smart Licensing and the last release that does not mandate Smart Licensing is 16.9.8.
Using the "CISCO Software Checker" tool you can do a lookup on particular versions to see the list of know defects. The version within 16.X series with least number if known defects is final 16.12.9.
The current (April 2023) Cisco recommended release are: 17.6.5 & 17.3.5.
To make your choice you should:
- Review release notes for routing feature and HW support
- Review the Software Checker reports for security vulnerabilities and so an assessment against these for the your deployment, as while there might be vulnerabilities they only apply under certain conditions. Example are: "virtual reassembly" defect occures if you use large (jumbo) MTU settings, bugs which only show up when using DHCPv6, defects with certain MPLS configuration or defects which only affect particular HW).
- Decide whether you want to stay on conventional or more to Smart Licensing model.
Summary
The migration of configuration is now complete. Before pulling out the old ISR 2901 and putting in ISR 4331 I will do "Greenbone Scan" (see blog "OpenVAS / Greenbone Vulnerability Management (GVM) / Greenbone Source Edition on AWS") to check that config is at least as secure as current ISR 2901.
Not unexpectly the security configuration took the most effort and getting this fully transitioned to ISR 4331 required creation of 16 config updates to address issues which mostly related to ZBFW configuration.
Unified Communications configuration by comparison only took 2 loads to move across, including PVDM transcode configurations.
Main IP took a few rounds as initially I did this before doing full security configuration, but defining interfaces needs to go in order of: Main (including NetFlow definitions), Addition Interface (to create loopback and virtual template definitions), then Security/Firewall and only then Main IP definitions , which has references to Netflow and Scecurity/Firewall definitions.
Most the effort was spent finding IOS XE equivalents to changed / removed IOS commands. So lots of links to Cisco and other documentation below in the "References & Links" section.
Due to critical defect in 16.6 (Everest) (see cert notice below) my final configuration deployment was with 16.9 (Fuji), which required no further update to migrated configuration.
Happy IOS to IOS XE migrations.
References & Links:
- Cisco 4000 Series ISRs Software Configuration Guide - Using Cisco IOS XE Software - the initial software guide
- Cisco Configuring Application to Port Mapping - very old IOS PAM configuration information and this blog has a reasonable summary "Port to Application Mapping - Classic Firewall"
- PAM with IOS XE Issues - this was been raised as issue on Cisco Community a few times, but no clear resolution or workaround outlined: "IP Port map custom protocol in IOS XE" & "ISR 4331 cannot add ip port-map user define port"
- PAM with IOS XE Workaround - the workaround posted in Cisco Community..
- CBAC Configuration Tutorial - this tutorial provides configuration example with real world context
- "Inspection on cisco router ISR4431" - covers removal of "ip inspect" in ISR 4000 IOS XE routers
- Cisco IOS XE Zone Based Firewall Changes - there are changes to ZBFW with IOS XE which are covered in: "Cisco - Security Configuration Guide Zone-Based Firewall - Cisco IOS XE Release 3S" and the following IOS XE security configuration updates
- Cisco IOS XE 16.1 Zone-Base Firewall Updates - "Security Configuration Guide: Zone-Based Policy Firewall, Cisco IOS XE Everest 16.6.1"
- Cisco IOS XE 17 Zone Based Firewall Updates - "Security Configuration Guide - Zone-Based Policy Firewall, Cisco IOS XE 17"
- Classic IOS ZBFW - "Understand the Zone-Based Policy Firewall Design" covers classic IOS but is generally applicable to IOS XE as well
- "HTTP Services Configuration Guide, Cisco IOS Release 15M&T" - provides details on http inspection service and its configuration
- "firewall messages" due to ccp (Cisco Configuration Professional) generated config - Cisco Configuration Professional (now out of support) generated my initial configuration and there are still remants of this, here is useful post for other who may have these remnant.
- Auscert - ESB-2020.3297 - Security Bulletin for IOS XE ZBFW with dropped packets logging enabled
- Cisco Software Checker - provides an easy way to seeing what defects and vulnerabilities exists with a given software release.
- "Greenbone Vulnerability Management" - test your config before deploying using Greenbone with local install on KVM/QEMU and via AWS
Opening picture is from: "Finding the Way - Waterfowl have amazing navigational abilities that guide their migrations"
Closing Duck Picture - is uncreditted but pops up in lots of article on good public behaviour