Cisco IOS 15.5 (ISR 2901) to IOS XE 16.6 / 16.9 (ISR 4331) Migration

Status: March 2023 - Note on progress, issues identified and work arounds or IOS XE replacement outlined


Today's task is to migrate Internet connection from Cisco ISR 2901 to ISR 4331 Router.

The reason is that:

  • ISR 2901 is now over 10 years in service and EOL
  • the interface on the ISR 2901 for ISP (via nbn) is via FastEthernet (HWIC-2FE Module) and I need to upgrade this to 1GBE
  • Can now get a used ISR 4331 at price not much more than an additional 4 Port 1GBE Ethernet Switch Module (HWIC-4ESW)
  • New router means new IOS which has benefits from throughput, features and bug fix (security) perspective

While the router has been humming away for over ten years, I would have thought that many others had completed simillar upgrade, so was a little surprised that there was not much in terms of specific information on the migration process.

Hence post.


The Process

My existing ISR 2901 includes: SEC & UC licenses and I briefly used DATA license to do some smart routing to handle fail over to alternate service. So it is running Zone Based Firewall & Ciso Unified Communications (SIP Proxy).

The IOS configuration consists of:

  • Main configuration - mostly device neutral things like: hostname, domain, time and dns server details, dhcp pools, ip services to run
  • Interface Additions - Lookbacks, Virtual Templates, Switch Interface vlan partitioning
  • Main IP configuration - device neutral: Interface & vlan IP adddresss for IP Subnets, routing and nats
  • Miscellaneous optional configuation - logging, netflow, ip performance and behaviour configurations
  • Security/Firewall configuration - by far the largest part of the configuration with all the access lists, security zones, policy rules etc
  • Unified Communications configuration - sip agents, dail plans etc

So process to migrate is:

  1. Break existing monolithic IOS config into sections, so these can be managed seperately
  2. Define the mapping of between ISR2901 Interfaces and vlans and new ISR 4331 Interfaces
  3. Establish initial admin IP connectivity (ssh, tftp)
  4. Starting with a clean configuation on ISR 4331 add the following into this configuration in this order: Main, Interface Additions, Security/Firewall, Main IP,  Misc, Unified Comms

Doing things in this order helps avoid configuration dependency issues when you move the configuration into new router.

Differences to Manage

In moving from 2901 to 4331 the Interface naming changes, so first part of change is to create an interfaces mapping sheet. This has all the source interfaces on the 2901 and mapping of these to the interfaces / vlans on the 4331.

As virtually all of the configuration with exception of route definitions is via IP addresss, the physical interface name changes only impacts a small part of the Main IP configuration.

The hard part of the migration is due to changes in the IOS commands/features across the two routers. This requires digging into the details of change.

The "Issue #" items cover the impacts of the IOS changes.

NOTE #1: IOS XE only supports Zone-Based Firewall (i.e. define policys based on traffic classes and zones and then enrolling interfaces into zones) and not Context-Based Access Control Firewall (CBAC FW) (i.e. define access control lists and traffic types and attaching control lists to interfaces). So if you have very old CBAC FW on your IOS Router, then you will need to start from scratch with IOS XE.

NOTE #2: While IOS XE does not support CBAC FW, it still uses CBAC based mechanisms to define extended access lists and policy maps. So ZBFW has superset of features, but does not use access lists on interfaces, instead it enrolls interfaces into zones.

Issue #1 - IOS Change - ip virtual-reassembly

On my ISR 2901 15.5 "ip virtual-reassembly" definitions can be specified as being for in bound interfaces:

  • "ip virtual-reassembly in" - to allow application level assembly the router need to hold and reassemble the message to do protocol compliance check.

In IOS XE 16.6.4 on ISR 4331 the specification of the direction "in" is not supported so need to change these configurations to:

  • "ip virtual-reassembly" - so the "in" is redundant and now removed.

Issue #2 - IOS XE does not support "user" defined Port and Application Mapping (PAM)

The Cisco firewall uses a combination of "Context-Based Access Controls" (CBAC) and "Zone Base Firewall" (ZBFW) as it primary way of managing security configuration.

The CBAC uses port to application mapping to allow specification of security policy based on application traffic. To  do this it uses IP port definitions i.e.: http == port 80, ssh == port 22, https == port 443 etc.

On IOS 15.5 these application mappings can be extented to include user defined mappings (with convention that all user mappings have a name prefix "user-").

On IOS XE this support was dropped, which is not typical with IOS which generally maintains high level of backward compatability with older versions.

As you cannot create your own definitions, the only workaround is the "hijack" some unused "system definitions". Here is the list of available applications (or services - as in "/etc/services" file..), from IOS XE 16.6.4:

#ip port-map ?
  802-11-iapp          IEEE 802.11 WLANs WG IAPP
  ace-svr              ACE Server/Propagation
  aol                  America-Online Instant Messenger
  appleqtc             Apple QuickTime
  bgp                  Border Gateway Protocol
  biff                 Bliff mail notification
  bootpc               Bootstrap Protocol Client
  bootps               Bootstrap Protocol Server
  cddbp                CD Database Protocol
  cifs                 CIFS
  cisco-fna            Cisco FNATIVE
  cisco-net-mgmt       cisco-net-mgmt
  cisco-svcs           cisco license/perf/GDP/X.25/ident svcs
  cisco-sys            Cisco SYSMAINT
  cisco-tdp            Cisco TDP
  cisco-tna            Cisco TNATIVE
  citrix               Citrix IMA/ADMIN/RTMP
  citriximaclient      Citrix IMA Client
  clp                  Cisco Line Protocol
  creativepartnr       Creative Partnr
  creativeserver       Creative Server
  cuseeme              CUSeeMe Protocol
  daytime              Daytime (RFC 867)
  dbase                dBASE Unix
  dbcontrol_agent      Oracle dbControl Agent po
  ddns-v3              Dynamic DNS Version 3
  dhcp-failover        DHCP Failover
  discard              Discard port
  dns                  Domain Name Server
  dnsix                DNSIX Securit Attribute Token Map
  echo                 Echo port
  entrust-svc-handler  Entrust KM/Admin Service Handler
  entrust-svcs         Entrust sps/aaas/aams
  exec                 Remote Process Execution
  fcip-port            FCIP
  finger               Finger
  ftp                  File Transfer Protocol
  ftps                 FTP over TLS/SSL
  gdoi                 GDOI
  giop                 Oracle GIOP/SSL
  gopher               Gopher
  gtpv0                GPRS Tunneling Protocol Version 0
  gtpv1                GPRS Tunneling Protocol Version 1
  h225ras              H225 RAS over Unicast
  h225rasMcast         H225 RAS over Multicast
  h323                 H.323 Protocol (e.g, MS NetMeeting, Inte
  h323callsigalt       h323 Call Signal Alternate
  hp-alarm-mgr         HP Performance data alarm manager
  hp-collector         HP Performance data collector
  hp-managed-node      HP Performance data managed node
  hsrp                 Hot Standby Router Protocol
  http                 Hypertext Transfer Protocol
  https                Secure Hypertext Transfer Protocol
  ica                  ica (Citrix)
  icabrowser           icabrowser (Citrix)
  ident                Authentication Service
  igmpv3lite           IGMP over UDP for SSM
  imap                 Internet Message Access Protocol
  imap3                Interactive Mail Access Protocol 3
  imaps                IMAP over TLS/SSL
  ipass                IPASS
  ipsec-msft           Microsoft IPsec NAT-T
  ipx                  IPX
  irc                  Internet Relay Chat Protocol
  irc-serv             IRC-SERV
  ircs                 IRC over TLS/SSL
  ircu                 IRCU
  isakmp               ISAKMP
  iscsi                iSCSI
  iscsi-target         iSCSI port
  kazaa                KAZAA
  kerberos             Kerberos
  kermit               kermit
  l2tp                 L2TP/L2F
  ldap                 Lightweight Directory Access Protocol
  ldap-admin           LDAP admin server port
  ldaps                LDAP over TLS/SSL
  login                Remote login
  lotusmtap            Lotus Mail Tracking Agent Protocol
  lotusnote            Lotus Note
  mgcp                 Media Gateway Control Protocol
  microsoft-ds         Microsoft-DS
  ms-cluster-net       MS Cluster Net
  ms-dotnetster        Microsoft .NETster Port
  ms-sna               Microsoft SNA Server/Base
  ms-sql               Microsoft SQL
  ms-sql-m             Microsoft SQL Monitor
  msexch-routing       Microsoft Exchange Routing
  msnmsgr              MSN Instant Messenger
  msrpc                Microsoft Remote Procedure Call
  mysql                MySQL
  n2h2server           N2H2 Filter Service Port
  ncp                  NCP (Novell)
  net8-cman            Oracle Net8 Cman/Admin
  netbios-dgm          NETBIOS Datagram Service
  netbios-ns           NETBIOS Name Service
  netbios-ssn          NETBIOS Session Service
  netshow              Microsoft NetShow
  netstat              Variant of systat
  nfs                  Network File System
  nntp                 Network News Transport Protocol
  ntp                  Network Time Protocol
  oem-agent            OEM Agent (Oracle)
  oracle               Oracle
  oracle-em-vp         Oracle EM/VP
  oraclenames          Oracle Names
  orasrv               Oracle SQL*Net v1/v2
  pcanywheredata       pcANYWHEREdata
  pcanywherestat       pcANYWHEREstat
  pop3                 Post Office Protocol - Version 3
  pop3s                POP3 over TLS/SSL
  pptp                 PPTP
  pwdgen               Password  Generator Protocol
  qmtp                 Quick Mail Transfer Protocol
  r-winsock            remote-winsock
  radius               RADIUS & Accounting
  rdb-dbs-disp         Oracle RDB
  realmedia            RealNetwork's Realmedia Protocol
  realsecure           ISS Real Secure Console Service Port
  router               Local Routing Process
  rsvd                 RSVD
  rsvp-encap           RSVP ENCAPSULATION-1/2
  rsvp_tunnel          RSVP Tunnel
  rtc-pm-port          Oracle RTC-PM port
  rtelnet              Remote Telnet Service
  rtsp                 Real Time Streaming Protocol
  send                 SEND
  shell                Remote command
  sip                  Session Initiation Protocol
  sip-tls              SIP-TLS
  skinny               Skinny Client Control Protocol
  sms                  SMS RCINFO/XFER/CHAT
  smtp                 Simple Mail Transfer Protocol
  snmp                 Simple Network Management Protocol
  snmptrap             SNMP Trap
  socks                Socks
  sql-net              SQL-NET
  sqlserv              SQL Services
  sqlsrv               SQL Service
  ssh                  SSH Remote Login Protocol
  sshell               SSLshell
  ssp                  State Sync Protocol
  streamworks          StreamWorks Protocol
  stun                 cisco STUN
  sunrpc               SUN Remote Procedure Call
  syslog               SysLog Service
  syslog-conn          Reliable Syslog Service
  tacacs               Login Host Protocol (TACACS)
  tacacs-ds            TACACS-Database Service
  tarantella           Tarantella
  telnet               Telnet
  telnets              Telnet over TLS/SSL
  tftp                 Trivial File Transfer Protocol
  time                 Time
  timed                Time server
  tr-rsrb              cisco RSRB
  ttc                  Oracle TTC/SSL
  uucp                 UUCPD/UUCP-RLOGIN
  vdolive              VDOLive Protocol
  vqp                  VQP
  webster              Network Disctionary
  who                  Who's service
  wins                 Microsoft WINS
  x11                  X Window System
  xdmcp                XDM Control Protocol
  ymsgr                Yahoo! Instant Messenger

For my "custom" services I had combination of: service running over SSL/TLS, are http variations (WebDAV, CalDAV, CardDAV) or relate to XMPP (Jabber).

For the SSL based protocols the router cannot do inspection of contents as the packets are encrypted and so best candidate for hijacking is old unsed SSL based protocol.

For http variations the best "hijacking" candidate would be any of the "standard" systems defined ones that is based on http (I could not find such a candidate).

Here is an example of my approach for XMPP traffic:

!
! XMPP custom portmap:
!   XMPP uses port 5222 / 5223 for user traffic &
!   XMPP uses port 5269 for XMPP <-> XMPP Server Federation traffic
!
ip port-map user-xmpp-5222 port tcp 5222 description xmpp 5222
ip port-map user-xmpp-5223 port tcp 5223 description xmpp 5223
ip port-map user-xmpp-fed port tcp 5269 description xmpp federation
!
! To replace these need to:
!   1. Create an standard access list (ie < 100)
!   2. Choice you application to hijack
!   3. Create port-map items using hijacked applicaton
!
! Looking at details of this the firewall access configuration found the public IP address of the XMPP server so the following access list was defined:
!
access-list 10 remark XMPP port-map user-xmpp-5223 user-xmpp-fed substitute
access-list 10 permit XXX.XXX.XXX.35
!
! In this case for XMPP I decided to hijack the Lotus Notes and IPX PAM as
!   there is no Lotus Notes server or IPX on my network and Lotus Notes includes two applications: lotusnote, lotusmtap with IPX for notifications
!
! So to ensure that the hijack configuration does not open any unneeded ports, I disabled the fault PAM and added new ones:
!
no ip port-map lotusnote
no ip port-map lotusmtap
no ip port-map ipx
ip port-map lotusnote port tcp 5222 list 10 description xmpp 5222
ip port-map ipx port tcp 5223 list 10 description xmpp 5223
ip port-map lotusmtap port tcp 5269 list 10 description xmpp federation 5269
!
! This creates a PAM that only applies against the hosts defined in the access list

This work around approach was applied to all the custom defined PAMs.

The corresponding ZBFW inspection and policy rules have to then refer to the "hijacked" application name and not the original "user-<APP>" one.

I hijacked:

  • lotusnote, lotusmtap - for XMPP
  • qmtp - for SMTP over SSL (smtps) (as smtp is used for mail transfer)
  • pop3s - for CalDAV & CardDAV over SSL (as all mail collection is via imap)
  • oem-agent (Oracle) - for Apple Push Notifications
  • ttc (Oracle) - for Harbor Registry https. which is over SSL

As this issue is pretty fundemental I posted work around in Cisco Community.

Issue #3 - IOS XE does not support "ip inspect"

The "ip inspect" command from IOS 15.5 was used to configure the application inspection.

This is now done using "class-map type inspect <Protocol> .." and "appfw".

Here are examples for http, im & imap:

!
! Inspection policy for http
!    which come originally from CCP (Cisco Configuration Proffessional)
!    hence "cpp-"
!
policy-map type inspect http ccp-action-app-http
 class type inspect http ccp-http-blockparam
  log
  allow
 class type inspect http ccp-app-httpmethods
  log
  allow
 class type inspect http ccp-http-allowparam
  log
  allow
!
! For ios xe this becames
!   Noting that: Most http traffic will be via https
!                 and maintaining original ccp config
!
appfw policy-name CCP_MEDIUM
 application http
  strict-http action allow alarm
  request-method rfc default action allow alarm
  request-method extension default action allow alarm
!
! And add the follow to replace the original policy-map
!
class-map type inspect match-any HTTP-PROTOCOL
 match protocol http
!
! Then use put the following into your zone policy's
!
policy-map type inspect ZONE-TO-ZONE-POLICY
 ...
 ...
 class type inspect HTTP-PROTOCOL
   inspect
 ...
 ...
 class class-default
!
! Same pattern applies to other cases
!
! Inspection policy for im
!
policy-map type inspect im ccp-action-app-im
 class type inspect aol ccp-app-aol
 class type inspect msnmsgr ccp-app-msn
 class type inspect ymsgr ccp-app-yahoo
 class type inspect aol ccp-app-aol-otherservices
 class type inspect msnmsgr ccp-app-msn-otherservices
 class type inspect ymsgr ccp-app-yahoo-otherservices
!
! For ios xe this becomes
!
appfw policy-name CCP_MEDIUM
 application im aol
   ...
   ...
 application im msn
   ...
   ...
 application im yahoo
   ...
   ...
!
!
class-map type inspect match-any IM-PROTOCOLS
 match protocol ymsgr
 match protocol aol
 match protocol mnsmsgr
!
! As per HTTP-PROTOCOL above add IM-PROTOCOLS to your zone policies
!

!
! Inspection policy for imap
!
policy-map type inspect imap ccp-action-imap
 class type inspect imap ccp-app-imap
  log
!
! For ios xe this becames
!
   just inspect with new IOS XE

My configuration was initially created using Cisco Configuration Professional (CCP) and there are a lot of ZBFW configration rules, for application level inspection, in this. Now most of the traffic traversing the internet is running over SSL/TLS so FW is really only doing TCP level validation. So some of the more detailed application level configuration (http payload checking for example), is unlikely to be used. In particular there should not no IMAP or POP3 exposed on network, rather these should be via IMAPS & POP3S. So take this into account when reviewng the security configuration.

Issue #4 - IOS XE does not support "parameter-map type protocol-info"

This configuration information should be moved to "appfw" for example:

!
! Old parameter0map type protocol ...
!
parameter-map type protocol-info msn-servers
 server name messenger.hotmail.com
 server name gateway.messenger.hotmail.com
 server name webmessenger.msn.com
!
! Same configuration via appfw
!
appfw policy-name CCP_MEDIUM
  application im msn
    service default action allow alarm
    service text-chat action allow alarm
    server permit name messenger.hotmail.com
    server permit name gateway.messenger.hotmail.com
    server permit name webmessenger.msn.com
    audit-trail on
!

Issue #5 - Out of Order Packets and Packet Reassembly

With IOS there is a global configuration for management of "out of order" packets: "parameter-map type 000 global". This is no longer supported with IOS XE.

The alternative to this is to configure "ip virtual-reassembly" on each interface (this configure this for ingress & egress).

See "Issue #6 - ip flow configuration" for example.

Issue #6 - ip flow confguration

With IOS ip flow can be configured on a per interface basis via "ip flow in" and "ip flow out".

With IOS XE this is configured on a per interface basis via "ip flow monitor application-mon input" & "ip flow monitor application-mon output"

Here is example:

!
interface GigabitEthernet0/0/0
 description $ETH-GBE000$FW_IN$$ETH-WAN$
 ip flow monitor application-mon input
 ip flow monitor application-mon output
 ip address XXX.XXX.XXX.XXX 255.255.255.0
 ip nat outside
 ip nbar protocol-discovery
 zone-member security IN
 negotiation auto
 ip virtual-reassembly
!

Issue #7 - ip flow-top-talkers


What IOS XE Version ?

When doing your migration from 2901 to 4331 you will need to select your IOS XE version. For 4300 the options are 15.X, 16.X & 17.X releases. In general it is best to select the most "mature" version (i.e. the one with least security bugs), which has support for the hardware & features you are after. With 4300 there have been some significant differences that need to be factored into the selection: Smart Licensing (which change from the hardware embedded license to an online one) and SD-WAN which is a move away from traditional IOS configuration managed approach to an cloud automated management solution for connectivity management.

To complicate things a little the IOS XE release used a versioning convention that has the release of the Linux OS 3.x and the release of the the IOS service on top of this (15.X). With the 16.X release the 3.X prefix was removed and the IOS XE release followed the same pattern as other IOS related release, so 15 -> 16 -> 17. In addition the 16.X release took on release names: Everest, Fuji, Amsterdam, Gilbralter etc.

As I started with IOS 15.5 base configuration, my initial migration was to 15.5 IOS XE as I thought this would ensure configuration compatability. This was not the case.

To get support for some additional hardware modules, I moved to 16.6 only to find that there were critical security vulnerabilities in this release which meant moving up to 16.9 or 16.12 or even to 17.X series.

However a move to 16.12 means you must shift to Smart Licensing and the last release that does not mandate Smart Licensing is 16.9.8.

Using the "CISCO Software Checker" tool you can do a lookup on particular versions to see the list of know defects. The version within 16.X series with least number if known defects is final 16.12.9.

The current (April 2023) Cisco recommended release are: 17.6.5 & 17.3.5.

To make your choice you should:

  • Review release notes for routing feature and HW support
  • Review the Software Checker reports for security vulnerabilities and so an assessment against these for the your deployment, as while there might be vulnerabilities they only apply under certain conditions. Example are: "virtual reassembly" defect occures if you use large (jumbo) MTU settings, bugs which only show up when using DHCPv6, defects with certain MPLS configuration or defects which only affect particular HW).
  • Decide whether you want to stay on conventional or more to Smart Licensing model.

Summary

The migration of configuration is now complete. Before pulling out the old ISR 2901 and putting in ISR 4331 I will do "Greenbone Scan" (see blog "OpenVAS / Greenbone Vulnerability Management (GVM) / Greenbone Source Edition on AWS") to check that config is at least as secure as current ISR 2901.

Not unexpectly the security configuration took the most effort and getting this fully transitioned to ISR 4331 required creation of 16 config updates to address issues which mostly related to ZBFW configuration.

Unified Communications configuration by comparison only took 2 loads to move across, including PVDM transcode configurations.

Main IP took a few rounds as initially I did this before doing full security configuration, but defining interfaces needs to go in order of: Main (including NetFlow definitions), Addition Interface (to create loopback and virtual template definitions), then Security/Firewall and only then Main IP definitions , which has references to Netflow and Scecurity/Firewall definitions.

Most the effort was spent finding IOS XE equivalents to changed / removed IOS commands. So lots of links to Cisco and other documentation below in the "References & Links" section.

2901 to 4331 - It was a bit like this ...

Due to critical defect in 16.6 (Everest) (see cert notice below) my final configuration deployment was with 16.9 (Fuji), which required no further update to migrated configuration.


Happy IOS to IOS XE migrations.


References & Links:


Opening picture is from: "Finding the Way - Waterfowl have amazing navigational abilities that guide their migrations"

Closing Duck Picture - is uncreditted but pops up in lots of article on good public behaviour