Lets start with a couple of quotes:

• "Many companies attack the enterprise architecture excercise with lots of drawings and analysis of both existing and hoped-for systems capabilites.  ... While the architecure for a new building is captured in blue-prints, enterprise architecture is often represented in principles, policies, and technology choices. Thus the conept can be difficult for managers to get their arms around. We have found that a simple picture, which we refer to as the "core diagram" helps managers debate and eventually come to understand their company's enterprise architecture."

This is an "oldie" taken from "Enterprise Architecture as Strategy". The reason I am referencing it, is due to its statement of the value of a  "simple picture" and "core diagram".

This blog is broadly about "models" and why they are valuable (some don't even get out of bed for less than $10,000 a day ;-) ). The examples are primarily taken from work I contributed to the "LNF/GSMA Anuket Reference Model".

Status: March 2023 - Writing done, now reviewing.

Models, Work and Value

It is always good to have a clear understanding of subject at hand, so from the Oxford Dictionary.

Model (noun) - via Oxford on iPhone:

• a three-dimensional representation of a person or things or of a proposed structure, typically on a smaller scale than the original
• a thing used as an example to follow or imitate
• a simplified description, especially a mathemetical one, of a system or process, to assist calculations and predictions
• a person employed to display clothes by wearing them
• a particular design or version of a product

If your an architect of "things" (buildings, systems, towns, enterprises), then you need to be a modeller.

The reason for modelling (confirmed via ChatGPT ;-) ) is that it:

• is Essential for Understanding and Communications - I put these together, as it could be that a person has a complete understanding and the only model is the one that they maintain in their head, but if they want to communicate and share that understanding or get input into whether their model is correct then it has to be communicated and so the model moves from idea to artifact or object. Which leads to ...
• is Essential for Analysis - once formulated and communicated the model supports analysis and can be used to validate and test whether the real system will behave and perform as expected and importantly the analysis includes validation and elaboration of the model itself. The result is an improved (better) model, with broader coverage of scenarios for analysis and in it's ability to support understanding across the target audience
• can Predict - a model can provide direct predication/s. This is distinct from analysis, where the model is an input into a stakeholder/audience prediction rather than a source of prediction. A simple example from Newtoninan physics: v = u + at where v == final velocity, u == initial velocity, a == acceleration, t == time, provides a prediction (proven many times over, in its domain) of the velocity that an object will be moving at after acellerating for some time.
• is Economically Prudent - it is cheaper and faster to first create and validate via a model than via the real thing.  However be aware that only the real thing can be used to determine if the model was correct or not. This is evident with every systems investment, where you will need a model of the goal, a model of the cost and a model of the timeline (plan). Without these very few (or should I say a vanishingly small number ;-) ) will be provided funding to go from model to realisation.

These points get to the value of a model (and indirectly to the value of doing architecture).

In the realm of IT Architecture, Enterprise Architecture and Business Architecture, what are the common characteristics of models:

• Invariant Models - these are models of domain which are invariant within a domain and the same model is used across multiple organisations and instances. Well know examples include TMF Open Digital Architecture  (ODA) (Telco) or the BIAN (Banking Industry Architecture Network) Models (Banking) which are Enterprise Architecture Industry Models and the "Canonical Web Architecture" for a generalised view of Web Application Architecture. As the model is  invariant it is very useful for: analsysis of relative performance across cases, as a starting point for understanding and realisation of specific implementation and for analysis of implementation alternatives (option definition and selection). While at IBM, I was co-author in definition of BizADS (Business Architecture Description Standard), which aimed to define a model for the general description of business. This was another example of an invariant model, starting with generic demand and intent and executing a plan, transform and operate cycle:

• Specific Model - these are models which are specific to an organisation or particular implementation. The focus here is to help understand and communicate the specifics of a particular organisation or systems. This is the type of model that is shown in the case studies of "Enterprise Archtecture as Strategy" (quoted at the beginning of this blog) "core diagram". Here is an example for "bitCo". The aim was to distill: its ownership and publicly stated charter and the key business measures against this, the product BitCo sells and its go to market model, as well as it key suppliers and partners:

• Scope, Focus & Abstraction - there are a common set of modelling tools that are used irrespective of what is being modelled. This includes process and data modeling. Case in point is UML (Unified Modeling Language) based tools for systems implementation models as well as enterprise and business domain modeling (both TMF ODA Telco and BIAN Banking Models use UML). The difference is scope and focus.  The opening quote says: "While the architecure for a new building is captured in blue-prints, enterprise architecture is often represented in principles, policies, and technology choices". Many times I have seen "Solution Architecuture" (& hence modelling) as being equivalent to "building architecuture", while "Enterprise Architecture" is more akin to "Town Planning" and using "Building" architecture as metaphor for Enterprise Architecture indicates lack of understanding. I choose to disagree, there is a common set of tools and techniques across all these architecture endevours. The important aspect is to have clearly defined scope and focus and a well understood level of abstraction. With the Anuket Storage model (covered below) the scope is a particular technical sub-system, while for Multi-Cloud Interaction focus is cross-organisation / cross-cloud boundaries and hence at a much high level of abstraction. Both have common attributes.
• Formal and Informal Models - A formal model is one that compiles to some specification and typically is supported by a tool, which can enforce syntax of the specification and extract some of the semantics from the model. An informal model is made from an arbitary set of elements and there is no enforced layout/connectivity and the meaning is in the eye of the beholder. I would love to only work with formal models, so that I could also use the model as basis for tool supported analysis, round trip engineering and prediction. There is one problem though.... typically formal models are the domain of modelling experts and are not greeted with the same positiviely by broader stakeholder community. Also when initially formulating an idea and strategy imposing formal approach may inhibit rather then help. So each approach has its merits.
• "Models, Work and Value" - so now we have view of the various types of models, lets cover who they are used by and why they are valuable. The "work" in this is because while you might model a model ship as a hobby the IT/enterprise/business architecture models are typically done for work and are developed and used by multiple stakeholders. Their value is in their utility to these stakeholders. So a model's ability to aid communications and understanding is critical. The (now superseded) IEEE 1471 "Recommendation for Practice for Architectural Description of Software-Intensive Systems" provided an excellent articulation of architectural description which encompassed "stakeholder" and "concern" as fundemental part of the conceptual model. The value of the model rests in its ability to address the concerns of its stakeholders:

So lets now look at some of the Anuket Models, I was involved in defining.

Storage Model

Storage, what could be simpler ?

One of the nice things about using Clouds is that it is simple. One of the things that becomes apparent when building clouds are: the simplicity has a lot of complexity being managed behind the scenes and that the design decision made as part of the build have huge impact of how easy the platform is to manage once in operation.

While working on building a tenanted private cloud solution, I was asked to look at the storage sub-system and the alternate capabilities and way of delivering these. I worked with a number storage subject matter experts, who's job was just storage systems. The result was getting well immersed in the subject. It became apparent that it is very hard to capture the variety solutions and technology easily, there were:

• Layers - physical, operating systems, applications
• Consumption Models - block, file, object
• Characteristics and Qualities - shared, non-shared, performance, reliability, security
• Technologies - the multitude of ways that storage can be delivered and
• Perspective - are we looking this this from consumer or provider persepective.

As with most things "virtual" the entire technical stack then comes back around full circle. So what might be provided by a high level solution at the hypervisor/virtualisation layer gets consumed at a low level physical layer within the virtual machine or for performance reasons the hypervisor may allow cut through back down to the native physical layer. A simple concrete example is that a virtual disk image sits on a file system of the host machine, but is consumed as block storage by virtual machine.

The "General Cloud Storage Model" (section "3.6 Storage") aims to capture these aspects of storage in a way that covered consumer & provider perspectives , that was invariant of any technology while providing a taxonomy into which the alternate storage technologies could be mapped:

The model was refined over a number of Anuket releases (Moselle & Nile). The Anuket specification (section "3.6 Storage") provides:

• model overview,
• example stereotypical storage deployment scenarios (Software Defined Storage, Virtual NAS (or vNAS), Storage for K8S & OpenStack  and even PXE boot storage) as well as
• architectural considerations for provision of cloud storage.

I will not repeat these here, see the Anuket Reference model for details.

The model helped:

• capture the details of storage delivery
• provides a good basis for understanding the multitude of ways that this can be achieved and
• provides a basis for reviewing these and determining which is best approach in a given scenario.

The "General Cloud Storage Model" was third iteration of attempting to provide a Storage model, it has much broader coverage than prior iterations.

Iteration zero and one are various diagrams and a spreadsheet of technical analysis. A generalist architect trying to understand the world of the storage specialist.

The second iteration was included within Anuket as  "Storage for Tenant Consumption" model "Lakelse" release and only covered the tenant consumption and tiering perspective.  This provides a view of the "what" but not the "how".

The target audience for the Anuket storage model is cloud providers and architects. As storage is a foundational part of any infrastructure solution. Having been involved in building a number of cloud solutions it is apparent that the default storage choices do not result in a good outcome in many cases with the result of need to do significant rebuilds.

In general, the data held within an enterprise storage solution is the most valuable IT asset and its integrity and managability is critcal. Currently the total storage capacity worldwide and storage demand is measured in "zetta-bytes":

The impact of this is that the storage systems are getting so large, that the management of the storage and the protection of the data contained becomes more and more difficult, as you cannot simply "copy" data from one place to another. So  if you need to rebuild the storage sub-system, simply moving the stored contents becomes multi-day preparation activity.

So we need to prepare by architecting for our future storage scenarios carefully, which the "Anuket Reference Model" can help with.

As a model it is an example of a:

• Invariant Model - as it can be support analysis of many storage deployment variations
• Focus - is to support understanding and decision making
• Stakeholders & Concerns - Cloud Architects - the options and considerations for provision of storage within multi-tenanted cloud solution, Implementors / Builders of Cloud infrastructure - have clear scope and expection of capabilities that will be delivered by storage solution/s, Tenants - have clear model of alternate storage capabilities that are available for consumption and be able to clearly distinguish and select what best fits their scenario & Product Owner - have clear model of a alternate capabilities, cost and considerations, against which they can establish priorities for funding and delivery.

NOTE: #1: The original Turing machine had an "infinite tape" (see model picture at start of blog)

NOTE #2: Thank you to Mat Devanny from NetApp on the storage education.

Multi-Cloud Interactions Model

The Multi-Cloud Interactions Model:

The impetus for the Multi-Cloud Interactions Model was to address the question, "How could I expand my Cloud offering to leverage external cloud providers and services?".

The question was raised by Product Owner of an internal private Cloud platform who wanted to be able to integrate with and leverge capabilities avaliable from public Cloud Providers (initially IaaS - Infrastructure-as-a-Service capabilities provided by AWS, Azure, Google etc).

The focus of the teams efforts had been almost exclusively with ETSI NFVi style VNF/Application orchestration (MANO - Management and Orchestration) capabilities and the scope of question limited to "Should I consume public cloud using each cloud providers exposed API (AWS APIs for example), use an abstraction layer (such as Apache JClouds) or build my my own abstraction layer?".

A model that outlined overall scope of the problems was needed. The Multi-Cloud Interactions Model took a broader view than just considering application orchestration, covering the overall life cycle, including:

• Commercial Aspects - Accounts and payments
• Technical Aspects - Connectivity, Resource Managemet, Application Mangement
• End Consumption and Use - Catalog and usage transactions/conversations/communications

With the initial framework in place it was then possible to look at different scenarios as alternate instansiations of the constituent parts. This helps with understanding the composability that a set of APIs would need to support.

The model allows illustration of alternate simple XaaS scenarios:

And alternate ways to achieve objective, either through technical means, such using API brokerage to operate across multiple cloud providers:

Or via commercial model such as Cloud Brokerage:

Illustrating these alternate way of achieving ends helps in choosing the right model  for a given organisation and the implications of this.

As a model the "Multi-Cloud Interactions Model" is:

• Invariant - the model is generic and allows definition of specific cases through alternate instansiations
• Informal - the grouping of interactions is by high level classification and not detailed specification
• Focus - strategic to help determine current and future stance and capability gaps
• Stakeholders & Concerns - Cloud Strategist - organisational capabilities and gaps, market offerings and differentiation opportunities, strategic planning and risk analysis.

Within LNF / GSMA Anuket Reference team the next increment for the "Multi-Cloud Interactions Model" will be to map current technologies, standards, open source initiatives and market offerings to help with understanding what is currently available to be leverage, where there are gaps and what existing initiatives might be candidates for endorsement (as per Architecture Reference Model 1 - OpenStack for Multi-tenant VNF management & Architecture Reference Model 2 - Kubernetes for Multi-tenant CNF management).

There is also a desire to bring together the coverage of the "Multi-Cloud Interfactions Model" with the "Infrastruture Life-Cycle Model". As many of the interactions defined in the "Multi-Cloud Interactions Model" are part of the "Operate & Manage" chevrons of the Infrastructure Life-Cycle Model", as well as parts of the iteractions get re-used / re-purposed to support the "Build" & "Provision" part of the infra-structure life-cycle.

These two different views which have some areas of overlapping concern is typical when creating models, as the model inevitably reflects the focus of the author (in these cases myself ;-). So while the interactions was primarily concerned with integration and API requires to operate cloud across different organisations, the infrastructure life-cycle model was concerned with the process of building clouds and automating this and its operation.

Wind Up

Model are not diagrams and architecture is more than models.

The purpose of this blog was to provide an update on some modeling that I have been contributing to Linux Foundation Network / GSMA "Anuket Reference Model". The aim is to help Communications Service Providers (CSP) with the building and operation of Clouds, as these are the platforms that they are already using to build their current and future services on top of.

When thinking about models it is essential to think about the stakeholders and the purpose / concerns that they are addressing.

If you do this then the model will prove its worth.

The model should help in getting alignment and most importantly help reduce risk and deliver the modelled target economically.

Formal and in-formal models both have their place. While the examples here are informal, you can see my blog "an5 - Intelligent Network Design" for a formal model that applies AI to the problem of network design using a network modelling language.

Remember that the model is an intermediate step toward the end goal, so yes love the model (architects tend to do this), but drive toward the goal.

This blog is more about modelling than the actual models, for more information on the Anuket Reference Model go to the LFN  source and GSMA Cloud Infrastructure Reference Model (Currently Version 3). The team would like to hear from people who would like to contribute.

NOTE: I believe this blog post is more informative than the ChapGPT, response to the question of "What is the value of architecture model?".

References & Links:

Enterprise Architecture as Strategy - Creating a Foundation for Business Execution: Jeanne W. Ross, Peter Weill, David C. Robertson (HBA Press - 2006), available to read via ResearchGate

Linda Evangelista - a model,  credited for saying "We don't wake up for less than $10,000 a day."

LF Networking - LF (Linux Foundation) Networking which has: The primary mission (the “Mission”) of the LF Networking Fund (the “Directed Fund”) is to raise, budget and spend funds and coordinate support for open source and/or open specification projects involved with the movement or communication of data on a network (including, but not limited to, data plane, control plane, analytics, orchestration, and automation technologies for enterprise, cloud, and carrier networks, collectively the “Scope”) ..

GSMA - An industry mobile communcations body, historically "Groupe Speciale Mobile" Association - GSMA is a global organisation unifying the mobile ecosystem to discover, develop and deliver innovation foundational to positive business environments and societal change. Our vision is to unlock the full power of connectivity so that people, industry, and society thrive.

Anuket - formed via amalgamtion of CNTT, OPNFV under auspices of LFN & GSMA - Anuket delivers a common model, standardized reference infrastructure specifications, and conformance and performance frameworks for virtualized and cloud native network functions, enabling faster, more robust onboarding into production, reducing costs and accelerating communications digital transformations

"Official Document NG.126 – Cloud Infrastructure Model" - the GSMA published version of Anuket Reference Model

ETSI NFV - the ETSI NFV & NFVi works defined the initial models for Network Function Virtualisation with the ETSI NFVi Architecture providing a key industry reference model. ETSI reference architecture include definition of MANO (Management & Orchestration). Anuket is more focused on implementation.

Zettabyte Era - our alarming data traffice and storage consumption trend.

Storage Supply/Demand Crisis in a Zettabyte World - the growth in storage demand is greater than our ability to sustain supply... better empty the trash.

Model from Oxford Learners Dictonary - my definition come from the Oxford Dictionary on iPhone, it is pretty similar

IEEE 1471 - "IEEE Recommendation for Practice for Architectural Description of Software-Intensive Systems" provided an excellent definition of architecture models covering: view, viewpoint, concerns and stakeholders (accepted in 2000)

Picture: "Univeral Tuning Machine" from: https://web.mit.edu/manoli/turing/www/turing.html with its infinite length tape storage

In late 1998 and early 2000 I wrote material on "Standard Operating Environments" (SOE) for a big Telco's "Enterprise Architecture". They wanted to contain technology proliferation, avoid systems interoperability issues and build teams having common set of technical skills to ease moving people across projects. Back then there where two completing Web Application development frameworks:

• Java Application Servlets  and Java Server Pages - build on top of Sun Java and supported by tools such as IBM WebSphere, WebLogic, Apache Tomcat and JBOSS Application Servers and
• Microsoft .NET - in all its expanding guises and including: IIS, MTS, ActiveX/COM/DCOM/COM+ and ASPs (Active Server Pages) or .NET

My brain goes foggy looking back at the evolving set of Microsoft and other technologies. Many of the concepts of JSP and ASP architectures were made available via an Open Source alternative:

• LAMP (Linux, Apache, MySQL and PHP) - where PHP provided an open source variation of JSPs and ASPs

Last time I attempted using web development tools in any depth (in 2010) I found CSSs (cascading style sheets) completely non-intuiative. Now though through a process of slow osmosis, I kind of "get" how CSSs work. In 2010 I also looked at "AJAX", purchasing two very fat O'Reilly books:

1. JavaScipt - The Definitive Guide (5th Edition) (C)opyright 2006 (Javan Rhinoceros) - now up to 7th Edition (2020)... &
2. Ajax  - The Definitive Guide (1st Edition) (C)opyright 2008 (Wooly Monkey)

The learning curve looked steep and was far removed from the HTML form based templates of prior tooling.

Fast forward to 2022, once more I have a need for some Web development as part of a short Proof of Concept (PoC) project. I have a programmer allocated to assist (or two if including myself ;-) ), so what technology should we use ?

Read on to find out, how "architecture decision" drove selection of development approach and for an architectural introduction to JavaScript and its tooling.

Status: Mid Nov 2022 - First cut of prototype UI completed and intial draft of blog completed

NOTE: "JavaScript" -  trademark is owned by Oracle (was owned by Sun)

The Canonical Web Architecture

Having been in deep freeze on Web application development for over ten years I need to:

1. Determine what language and tools to use,
2. Develop a UI that uses a Thrift based server for resouce access and
3. Built and deploy the solution on Ubuntu platforms

Developing the Web application quickly (!!) is only constant from "way back when".

Lets start from the lens of "A Canonical Web Architecture":

This defines a "high level" view of Web Application architecture, with these common components:

• Browser - which has JavaScript Engine and HTML renderer
• [Optional: Web Server] - which could be external to application server to provide "reverse proxy" and SSL termination
• Application Server - which could be either a framework server, such as Apache Tomcat or JBOSS for Java based web applications or NodeJS for JavaScript based web applications
• [Optional: Database Layer] - for persistence, not required in my case, as I am using micro-services layer ("External Systems" pattern)
• [Optional: External Systems] - in my case this is exposed as Thift based micro-services (written in Java).

The architecture does not look simple... nor potentially quick to implement. If I proceeded based on historical frameworks, then I would end up installing: Apache Web Server (or NGINX), Apache Tomcat (or JBOSS) and this is before even writing a single line of application code.

To strip out complication and make this as simple as possible, my determination was to avoid unnecessary infrastructure components and layers.  Ideally the architecture would be just have an "Application Server" directly exposing the Web URI, with caveat that I can put a reverse proxy in front of this in future if needed.

That strips the Canonical Web Architecture down to single "Application Server" plus the Thrift micro-services server. Stripping out the "platform infrastructure" and using micro-service is consistent with "server-less" paradigm. Its not that there are no servers, rather that that this logic is now managed within the application framework and as part of application development and deployment.

Next decision.. what should I build the "server-less" Web application with in 2022 ?

Server-less Application for Web UI

In a classic Web Application framework the Web Server or Application Server is responsible for handling the HTTP/S protcol and passes request to the Web application through callbacks with Request/Response parameters.

In the case of JSP, ASP & PHP the call back is passed to a "template form" which extracts data from the Request and completes the form, by getting data from database or other systems, passing this back response as HTTP/XML payload.

The basic architecture is the same irrespective of Language or Framework:

---
--- Java Servlet
---
public interface Servlet {
 ...
 void service(ServletRequest req, ServletResponse res) throws ServletException, IOException
 ...
}
---
--- NodeJS
--- 1. First request handler
const RequestListener = function (req, res) { ... }
--- 2. Now pass to server creation function
const server = http.createServer(RequestListener);
---
--- or using JavaScript ES6 "arrow function" shorthand for this same thing
---
const server = http.createServer( (req, res) => { ... });
---
--- C# .NET 
---
public virtual void ProcessRequest (System.Web.HttpContext context);
--
--- where HttpContext wraps both request & repsonse object
---
public sealed class HttpContext : IServiceProvider

The goal is to strip back the infrastructure needs and keep our UI application simple by having the "application" being the "Application Server" as much as possible.  Doing a quick review of options, candidates for developing the UI could be:

• Go - has some nice features (multiple return values, delayed execution), is supported by Thrift and relatively easy to learn if you have previously used C/C++ or Java.
• Java - is a general purpose language with lots of class libraries, but these are targetted for creating WAR files for deployment into a Java application server. Though deployment via embedded Jetty is also possible and more consistent with minimalistic approach.
• JavaScript - is the programming language for the browser and so knowlege of this is required for Web UI development. Its structure shares a lot with Scheme Lisp dialect, having "eval/apply" functions that allow code to be treated as data and it has 'C' like syntax. Server client code is generated by Thrift.
• PHP - is hugely prevalent and supported as target for Thrift generated client library, but no "simple" serverless deployment option is available. A web app needs to use either NGINX or Apache Web Server.
• Python - has many advocates and it is supported by Thrift. However in using it for creating test Thrift clients, I found the syntax and indentation based structuring construct, too fiddly and prone to errors ... This is personal opinion, likely readily refuted, but as I am the "second developer" of target UI my opioion gets airplay ;-) .
• Ruby on Rails - has reputation for providing UI development innovation (as well as naming innovation ;-) , as this is "Rails" framework written in Ruby programmming language). Uses Ruby (an OO based programming language for   Model View Controller (MVC) for Web UI development.

All these options are mature and have large user bases. The criteria for selection and use include:

• Learning Curve - for language and framework / libraries (which is obviously bias'ed toward current developer skill set: Java, C/C++, Lisp).
• Adoption & Support - extent of use and availability of documentation, knowledge and expertise.
• Openess and Community - is it "open", does it have clear standards process attached to it and does it have an active community involved in its onging development.
• Utility, Tools and Libraries - are there development tools,  does it have libraries required for development of our UI and can it be readily integrated with Thrift micro-services and can I use the knowledge elsewhere (i.e. applicable for client-side and server-side programming).
• Small Footprint - low infrastructure need and ease of deployment (conceptually "serverless" not AWS Lamba "serverless).

So before venturing an opinion I looked at what the current trends are in the market (see below Appendies for details).

This showed that PHP, Java, JavaScript have strongest market support across a range of different areas. Putting the options into a simple "Decision Model" for assessment (Just Enough Architecture in action) with criteria for:

• Scoring - Exceeds or Better = 1, Meets = 0, Does not Meet or Worse = -1
• Mandatory/Screening Criteria - Thrift support and Small Footprint deployment

The result was:

JavaScript scored highest, even with low learning curve score (as target developers were JavaScript novices), based on "Small Footprint" and "Utility (Front End/Back End skills reusability) scoring.

NOTE #1: The low score for "Learning Curve" is evidenced by the two fat O'Reilly titles, that got put on the shelf some time back and my electing to tackle my last Web UI development task using PHP. PHP is much simpler, both conceptually and in practice, with easy to grasp HTML template model. In this case though PHP is not an option as it needs to be deployed via a Web Server, failing the mandatory "Small Footprint" criteria. Ditto for Java and "Ruby on Rails". The higher scores for JavaScript, Go and Java for Openess and Community is that they have formal process and corporate support to evolve, rather than pure community model or strong central controller. While the "Learning Curve" criteria is coloured as it could lends itself to personal bias, in this case did not really impact scoring.

NOTE #2: The "Decision Model" architecture work product should have screening critieria (for inclusion/exclusion) and clearly defined scoring criteria for assessment. This is preferable to "Pros/Cons" assessment which typically do not use consistent criteria across the alternates. If you have a set of  Pros/Cons turn each of these into an assessment criteria which can then be applied consistently to each option. For a complex assessment,  group criteria together into categories of concern and use weighting to apply to scores and overall groups.

NOTE #3: If we update scoring for Java based on use of embedded Jetty then it, would now pass mandatory screening criteria and come in with better scope than Go. So Java scores better than newer more fashionable options.

JavaScript Language, Engines and Frameworks

Opting to use JavaScript, here are some basic facts and about the language and tools for JavaScript development.

Some history..

JavaScript was developed at Netscape back in 1995. It is an interpreted language with roots in Scheme Lisp and 'C' style syntax. The original interpreter was embedded in the Netscape brower and subsequently reversed engineered by Microsoft. The Microsoft JScript JavaScript interpreter was realeased in 1996. Due to resulting variations in behaviour Netscape choose to submit the language into a standardization process via the European Computer Manufacturers Association (ECMA) in 1997. The result was the first version of the "ECMAScript", as JavaScript was a registered trademark of Sun, who where early partner in its development with Netscape. The ECMScript specification has contined to evolve, with most current version being "13th Edition – ECMAScript 2022". These editions are abbreviated as "ES6" for 6th Edition or more recently by year "ES2017".

JavaScript Engines

There are now three major JavaScript execution engines:

• V8 Engine - developed by Google and used in Google Chrome and Microsoft Edge browers and for the NodeJS server side JavaScript execution environment
• MonkeySpider - is the evolution of the Mozilla engine and is used in FireFox browser
• JavaScriptCore - is the Apple engine and used in the Safari browser

JavaScript Tools and Frameworks

When JavaScript was introduced in the browser, it was common to serve up a JavaScript payload having URI references to externally sourced code. These external code snippets would be pulled by the JavaScript engine from external web servers:

Serving JavaScript that contains references to external libraries raises multiple issues:

• Security - as external environment cannot be trusted (addressed by disallowing use of URI to serverr different to the originating source server)
• Dependency Failure - as the library reference could either fail to load or the version of the referenced library could change, breaking the overall JavaScript program.

These same issues occur in server-side based JavaScript environments as well. The issue is inherent to JavaScript programs, as distribution is always via source code. With Javascript source is directly executed and there is no intermediate compiled object code.  The JavaScript development tools help to manage the building of JavaScript programs using the source code based modules. The development process and example applicable tools are illustrated here (just focusing on the coding and building):

The diagram uses V8, NodeJS, Yarn, Sass, Gulp, JQuery & Git to show the roles of various tools. In taxonomy of tools, I use Java development as an analogy, where "==" means simillar to not equivalent. From above diagram, have:

• V8 - the JavaScript engine, the barebones to run a JavaScript program (== Java VM)
• NodeJS - a service side JavaScript runtime environment, provides the V8 engine, a set of libraries and other tool to make V8 useful (== Java JRE/JDK)
• Yarn - a JavaScript Package manager. Manages code libraries (as web accessible repository) used to build your JavaScript program and pulls these down into your local build environemt (== Maven for a Java project, except that Maven maintains a repository of jar files, rather than source files)
• Sass - is pre-processor for "sass" based cascading style sheets (CSSs). Sass provides a languge for definition and processing of css files. The reason this exists is that CSS were originally mostly hand crafted as part of HTML template definition, now css have become part of sophisticated HTML rendering and needed extension to lend itself to programatic manipulation. Sass if one of a number of css preprocessors (== ??, no Java equivalent)
• Gulp - a JavaScript build tool that brings together all the "bits" of your JavaScript program, so it has everything required to create a consolidated downloadable (to client side/browser) set of JavaScript programs. It also supports changing library namespaces referenced names to avoid clashes and to reduce the size of the JavaScript program (== Java ANT)
• JQuery - is just an example of a JavaScript library, it is the most widely used libraries as it is provides DOM (Document Object Model). You need it to serve, manipulate and tranverse HTML documents on client & server sides  (== Apache Xerces2 Java XML Parser, one of many DOM/XML/HTML libraries)
• Git - git is git is git and as JavaScript libraries are distributed as source code (no Jar files here thank you) the library manager like Node Package Manager and Yarn all pull code directly from git based respositorise (== git :-) is agnostic to language of source code)

Finally as many of the JavaScript tools are written in JavaScript, these also require a JavaScript runtime environment.  So it does not matter if you are creating client-side or server-side JavaScript programs, you will need JavaScript runtime environment for your JavaScript tooling. Hence why "2. Build" "chevron" in the "JavaScript Tool Chain" diagram above has "greyed-out" V8 and NodeJS logos. In this case NodeJS is there to run the tools, rather than being the used tool.

Dependency Management across Libraries

This needs to be addressed by JavaScript program build tools. This is the role of the JavaScript Package Manager. The packae manger provide a function analogous to that of Maven when building Java applications.

As part of the JavaScript program build process, the "package.json" file provides specification of which JavaScript libraries at what version are required to build the program. This is used by both NodeJS and Yarn. Taking directives from the "package.json" file, the build process brings in all the required JavaScript code into local build environment. Here is an example Yarn "package.json" from an Open Source "Commento" project:

{
  "name": "commento",
  "version": "1.0.0",
  "main": "index.js",
  "repository": "git@gitlab.com:commento/commento.git",
  "author": "Adhityaa <c.adhityaa@gmail.com> Anton Linevych anton@linevich.net",
  "license": "MIT",
  "private": true,
  "devDependencies": {
    "chartist": "0.11.0",
    "fixmyjs": "2.0.0",
    "gulp": "4.0.2",
    "gulp-clean-css": "3.9.4",
    "gulp-concat": "2.6.1",
    "gulp-eslint": "5.0.0",
    "gulp-html-minifier": "0.1.8",
    "gulp-rename": "1.3.0",
    "gulp-sass": "4.0.1",
    "gulp-sourcemaps": "2.6.4",
    "gulp-uglify": "3.0.0",
    "highlightjs": "9.10.0",
    "html-minifier": "3.5.7",
    "jquery": "3.2.1",
    "normalize-scss": "7.0.1",
    "sass": "1.5.1",
    "uglify-js": "3.4.1",
    "vue": "2.5.16"
  },
  "dependencies": {
    "eslint": "^5.10.0"
  }
}

This file provide an equivalent to the Java/Eclipse/Maven ".pom" file, which defines all the libraries that are needed to be pulled into the Java compilation process.

Next pulling all the JavaScript files together...

JavaScript Modules

Historically JavaScript had no concept of modules, you simply had files and Javascript and you could concatenate these together prior loading to execution engine or have different <script>...</script> embedded tags in your html to load different JavaScript files.

JavaScript's initial modularity mechanism was introduced through writing loading utilities which then applied the result to the "eval" function (this submits string to the JavaScript interpreter). This evolved into the "require()" statement to import and exposing definitions through "module.export" object. This practice was adopted by NodeJS and become know as "CommonJS module". As JavaScript continued to evolve through ECMA proces ES6 introduced modules into the language through the import/export declarations. Thus ES6 import/export are part of JavaScript Language, while Node "require" & "module.export" are operations on objects.

Much of the information you find via google search is illustrated using examples with "require". There are are significant differences in behaviour between the "CommonJS module" and ES6 (this is covered in section on asynchronous and Promises in "JavaScript - The Language" below). In Node how a given set of directories is managed is via the "package.json" file, which can have a entry for:

• "type": "module" – treat all .js files in this directory and below (until there is another "package.json" file as ES6 modules and enable "strict" mode (i.e. enforce ES6 language compliance, including import/export declaration support)
• "type": "commonjs"  - enable module/modules based modules through "require" and "module" classes.

Note that in Node:

• you can "import" legacy "CommonJS module" based code (this needs awareness of "package.json" placement within your source hiearchy to behave correctly,
• but you cannot use "require" based import within ES6 modules (.js files).

Node behavior is also controllable using different file extensions, with .mjs for ES6 modules and .cjs for "CommonJS module" code. Personally I do not think using file extension naming is a good idea and expect that all new JavaScript will end up being ES6 compliant.

For browser, control of module type is via script tag: <script type="module"> for ES6 compliance.

So to code and simple example...

Here is example of import for Node code. As my "resouce interface" is via Thrift based server, I needed to import the Thrift compiler generated code (thrift generated code uses "commonjs") which was imported into my ES6 UI module:

---
--- The ES6 Imports for EX.js using the thrift generated code
--- NOTE: The ../gen-nodejs is the directory for the Thrift generated code
--- The directory structure used here is important to ensure correct behaviour
--- EX-PROJECT
---   /gen-nodejs      <<== generated code (commonjs) --- packjage.json <<="=" package.json to turn on "common.js" ---- ex "module" (es6 strict) ex.js ui (es6) ex-pages.js support $ cat . ... import * as ini from 'ini'; fs 'fs'; thrift 'thrift'; ldtypes '.. gen-nodejs ex_thrfit_types.js'; ld exthrift.js'; express 'express'; pages '. x-pages.js'; the corresponding "package.json" file that is in same directory (. ) module note: this type="=" (ie es6) { "type" : "module", "dependencies": "ini": "^3.0.1", "express": "^4.18.2", "thrift": "^0.16.0" } and which up above at level .. common.js not es6 strict "commonjs", }<>

NOTE: you need to use Thrift compiler "--gen js:node,es6" flags to get this result.

To bring together all the JavaScript module into working code you can use:

• Node & Node Package Manager - for simple Server Side JavaScript and
• Node, Yarn, Gulp and Saas - for larger "AJAX" Web UI projects

So in summary:

• A JavaScript module is simply a JavaScript file that is flagged as such (explicity via package.json, script attribute or filename convention or or implicity via inclusion of require/import/export directives)
• You can import a module with "require" class invocation or ES6 "import" declaration
• You can define exposed variables, functions, classes etc within a JavaScript file using "module.export" (& "export") and ES6 "export"
• There are behaviour differences in "commonjs" & ES6 "module" mechanims
• You can use ES6 "import" to import "commonjs" modules, but you cannot use "commonjs" "require" within ES6 modules.
• You can control the module type using "package.json", file name extensions and <script type="module"> tag
• Other build tools such as: Yarn, Gulp, Saas will be applicable depending on your project needs and if your Web UI is just using Server Side JavaScript or both Server Side and Client Side (browser based) JavaScript.

JavaScript - the Language

As a language JavaScript itself should look pretty famillar to any C/C++ & Java developer. The "learning curve" scoring relates more the JavaScript execution model than its does to the core language itself. The ES6 updates provided many signficant extensions to the language with introduction of explicit classes (vs. prototype based templates) including inheritance via the "class" and "extends" keywords and with the "strict" directive abandoned full backward compatability with original JavaScript specification.

Examples of this need to flag "strict" compliance was the introducion of many new reservered words including: class, extends, const, let which could have been used as identifiers in older JavaScript programs and so would break these if executed with "strict" compliance.

The complication of JavaScript is that it was designed to run in browser and so has an asynchronous threaded execution model based on an event loop (just like many GUI frameworks).

This "asynchronous" model can cause surprises if you are not well aware of it as function that you would typically expect to act synchronously (reading a file) returns before the completion and you need to allow for this in your code.

Before ES6 this was typically managed via inclusion of callback functions for return of result and with ES6 JavaScript uses "Promises" as a generic way of handling management of asynchromous completion of requests.

Specific examples of differences in the asynchronous behavior can be seen when using the "CommonJS module" vs the ES6 "module":

• CommonJS module - require is executed synchronously while
• ES6 import - is handled asynchronously.

With ES2017 the async, await keywords where introduced to provide more control of the asynchronous execution model and allow programmer control the behaviour and avoid need for Promise.

Here is a trivial example using ES6 Promise to for a request to Thrift server:

//
// 1. Open Connection to Thrift Server
//

// Import the general and generated service functions
import * as thrift from 'thrift';
import * as ex from '../gen-nodejs/EX.js';

const host = "127.0.0.1";
const port = "9090";

let lastErr = null;
let exRes = null;
let exPromise = null;

// Check if we have a Promise
function isPromise(prom) {
  return prom != null && typeof prom.then === 'function';
}

// Handle error/exception case
ferr = function(err) { console.log("ERR>> EX: ", err); lastErr = err; };

// get our connection to Thrift based server
let connection = thrift.createConnection(host, port, {
  transport : thriftTransport,
  protocol : thriftProtocol
});
connection.on('error', ferr);

// create our client which provide handle to call our service methods
const exClient = thrift.createClient(ex, connection);

//
// 2. Now call async thrift service (exService):
//

let arg1 = {...};
let arg2 = {...};

exPromise = exClient.exService(arg1, arg2)
              .then((res) => { console.log('DBG>> exService got: ', res);
                               if (! (isPromise(res))) exRes = res;
                               return res;})
              .catch(ferr);
               
if (isPromise(exPromise)) {
  console.log('DBG>> Yep - got promise as expected!');
else {
  console.log('DBG>> What, no promise!!!');
}

//
// Explanation:
//   a. call the Thrift service function (exService) via client handle
//   b. using "then" method set up the function that will receive result
//   c. allow for error/exception case using "catch" method
//   d. NOTE: while this looks like a "try/catch" clause, it acts asychronously, so we squirel the results in the lastErr / exRes variables for later retrieval
//.  e. NOTE: the isPromise function looks to see if the returned object
//              is a promise. The only thing that defines an object
//              as a promise is that is has a "then" method,
//              so we just test to see if the object has a "then" method
//

That is it for JavaScript, as this was not supposed to be a tutorial on JavaScript, but a review of main architecture and concepts.

An experienced Java developer will find the core language pretty familiar. The ES6 additions mean that JavaScript now has full object oriented development capabilities.

The only other JavaScript concept that might be a bit foreign is in the use of "closures". This uses functions that encapsulate lexically scoped identifiers so these can be accessed outside of the initial local scope of the function definition. These should be famillar to Java developers who have used its "lamba" enhancements and to someone who has used Scheme Lisp.

And on to AJAX (or fat book #2)

As discussed above you can build a Web Application using straight JavaScript server-side programming model. But JavaScript's origins were to help create more dynamic Web pages via client-side programming using the JavaScript engine on the Browser.

This has evolved over the years from simple scripting of traditional HTML web pages to AJAX and beyond.

So what is AJAX ? The original acronym stands for "Asynchronous Javascript And Xml".

The "asynchronous" is in relation to typical http/s request where a request is made and the user waits until all the data is loaded and then the browser renders the result. With JavaScript on the browser this can execute (behaviour is controllable) on its own execution thread that is independent from the main user thread. This allow the JavaScript to make further request asynchronously and so decouples the http rendering from the request made back to server via the JavaScript program.

In original AJAX these request are made through the XMLHTTPRequest method which is used to make further HTTP requests (GET, POST, UPDATE, DELETE etc) with results returned via XML in response body.

That is the XML part, but XMLHTTPRequest does not mandate XML for payload and any data format can be returned with JSON now being the preferred format, as working with JSON in JavaScript is very easy and JSON representation is very close to JavaScript object representation.

As part of the evolution the XMLHTTPRequest has now been supplanted by "fetch" request as perferred way to get data.

NOTE: while AJAX has "asychronous" in its acronym, both the JavaScript in browser and in Node use an event loop model to achieve their asynchronous model.  The result is that by default there is only a single thread within the engine executing. This simplifies things for programmers as they do not need worry about ensuring thread safety across multiple executing thread.

The structural components

So in 2010 looking at the combination of "AJAX - The Definitive Guide" and "JavaScript - The Definitive Guide" this all seemed too daunting and not a way to fast track Web UI development. I retreated to PHP, which served my purpose. Revisiting this in 2022, is AJAX still as complicated as "fat book #2" would have you believe ?

Happily, I believe the answer is no. To make something large and complex you need to follow a simple strategy:

1. Understand the scope of ambition
2. Identify key use cases which will test the structural concepts
3. Look to distill the essential abstractions and foundation on which you can build
4. Start with the simple core and build up from this
5. Expect to have to refactor/refine if things get to complicated or you cannot readily accommodate new needs
6. Try not to underestimate the difficulting ahead (most troubled projects start with failure to anticipate the complexity ahead of them).

The problem with "fat book #2" is that it starts with a survey of ways of doing things that runs to 200 pages and then go into details on how to implement lots of different user interface elements. It fails to to capture (2) & (3), so the reader is lost before even starting.

It is important understand that AJAX did not relie on new technologies, it simply used the existing technology in a new way.  The core of this was usage model was:

• Document Object Model (DOM) - provided structural view of html document, which JavaScript uses to supports navigation and manipulation of the document. If you create your initial document programatically on server side, then it is easier to think about its manipulate on client side later ...
• Browser Events - allow you to hook JavaScript invocation to browser events (button press. mouse entry/exit, mouse click, page load etc). This allows you to control via JavaScript how the page responds to events
• Uses CSS to control page behaviour, not just its "look" / "style" - examples of this is that CSS can be used to make things visible/invisible, within viewport or out of viewport
• html <div> tags - to identify html structural elements (ie DOM hiearchy) that can be replaced/manipulated via Document Object Model (DOM) and can be assigned CSS styles. Note that div does not mark out document semantic structure (like: header, body, nav etc), just its DOM structural aspects, so div is not likely to go away soon
• fetch (was XMLHTTPRequest) to get data and your client side JavaScript program behave asychnonously to the currently rendered html page

These are the core structural elements that were used to create AJAX web applications and now simply "web applications" as the AJAX qualification has become redundant.

As always you can do this from scratch or you can leverage existing frameworks and tools to create your client-side scripted web application.

Frameworks and Web Standards

There a number of client side JavaScript and CSS frameworks and libraries available. A survey of these (by popularity/adoption) is given in the appendices.

For my pretty simple UI needs I have elected to:

1. Modularise HTML elements generation to simplify html responses from NodeJS express and allow easy application of CSS definition to elements
2. Provide limited fetch and browser event support to allow request of asynchronous request with
3. DOM element replacement to display last request result (returned via JavaScript Promises)

One aspect of Web Programming in 2022 that is now different is that it is all HTML from now on, as w3c.org XHTML efforts have not been adopted by the browser vendors (WHATWG - Apple, Google, Microsoft & Mozilla). To quote:

• "The XML syntax for HTML was formerly referred to as "XHTML", but this specification does not use that term (among other reasons, because no such term is used for the HTML syntaxes of MathML and SVG)."

So while WHATWG "living standard" may sound like "not stable", the rally around HTML provides greater certainty than the XML driven mania of XHTML, XFORM, XFRAME etc... XHTML is dead long live HTML.

NOTE: more details likely to follow as UI development evolves.

Realising Canonical Web Architecture in 2022

Adoption of JavaScript for both server-side and client-side development has proved to be viable.

• NodeJS with express has allowed rapid development of Web Application, while avoiding need for seperate HTTP and Application Servers
• Using fetch and JQuery can achieve asynchronous retreival and render the results
• Thrift compiler still generates "CommonJS module" code (even with --gen js,node,es6 flags), but using "package.json" type flag and ability to import "commonjs" module via ES6 "import" means you can work around this
• Understanding the ES6 changes NodeJS thread and asychronous Promise model and how these impact Thrift integration was area requiring most "exploration" to get working (documentation is poor)
• 80% of what you find via google has past its "use by date", as is much of "fat book #2" :-)
• But "fat book #1" 7th Edition is both thinner than 5th Edition, technically current and a better place to go for introduction to modern Web Application Development (Chapter 15: JavaScript in Web Browsers)

The first cut of our prototype Web UI is done, using JavaScript only and the learning curve barrier for AJAX development initially faced in 2010, was no longer an impediment. This is due to API improvements and starting simple with no dynamic content and then adding this once once comfortable with DOM and JavaScript.

As always the problem I found was that while there is lots of small technically detailed material, there not much in the way of high level architectural and conceptual context to make wading through the technical details productive. Hence this post.

Closing:

• expect a refresh with additional technical details on this blog as PoC UI gets further improvments in look, feel and function and
• follow up in "Just Enough Architecture - Technical Tips" blog on Thrift with JavaScript where there is a gap in the documentation required to get this doing quickly.

References and Links:

Java/Jakarta Servlets and JSPs - helped to create the greate divide between Java and .NET and now packaged as Java Enterprise Edition or Java EE

Microsoft .NET  and ASPs - Microsoft provided the Active Server Page and ASP.NET as an alternative to Java and JSPs. Now open sourced as ASP.NET

PHP - while there was battle going on with Java & .NET, PHP quietly become the most popular way to develope forms based Web applications. Can be officially found at php.net (the .net that beats .NET, in "server side languge" use share according to w3tech.com survey)

V8 JavaScript Engine - to execute JavaScript requires an engine.  On the Server side the Google Open Sourced V8 Javascript Engine is likely the most used and embedded engine. It is also used as JavaScript engine in the Chrome Browser

NodeJS - NodeJS is a JavaScript runtime framework, for executing JavaScript on server. It includes a combination of V8 JavaScript engine with a set of libraries (http & express - libraries provide alternate ways to build HTTP servers with NodeJS) and an package manager (Node Package Manager). The use of NodeJS to provide the runtime environment does not dictate what other JavaScript framework / component libraries you might wish to use. With NodeJS you can have server-side JavaScript and with browser have client side JavaScript code. The server side and client side JavaScript enviornments are independent.  There are many AJAX implementations that do not use Server-Side JavaScript.

express - a simple JavaScript http server developement framework, helps realise the ambition of having Web Application that is the Application Server.

JQuery - DOM object manipulation for JavaScript

Apache Thrift - a back to the future distributed systems development tool using IDL (remember SUN/ONC RPC / XDR etc ?) to generate client and server stubs and avoid having to worry about JSON/XML and all that other on the wire packaging nonsense.

C# IHttpHandler.ProcessRequest(HttpContext) Method - handles http request in C#, though generally .NET framework uses ASP templates

Serverless computing - a appliation development pattern that removes need to for knowledge or visibility into the applicaiton deployment infrastructure (such as Web Servers and J2EE application servers).

w3tech.com - collects web infrastructure and tooling data using web crawling

Statista - quotes survery results...

• Frameworks - "Most used web frameworks among developers worldwide, as of 2022"
• Programming Languages - "Most used programming languages among developers worldwide as of 2022"

Structure and Interpretation of Computer Programs - by Abelson & Sussman, provides a deep dive into conceptual models for programming and computing using Scheme Lisp for illustration. JavaScript has many properties that similar to Scheme and chapter 4 - Metalinguistic Abstraction provides details of the "eval/apply" loop which is consistent with JavaScript interpreter. Available as pdf.

JavaScript Reference - Documentation on Mozilla  Developer Network

ECMAScript Standard - managed by ECMA International Technical Committee TC39, which manages the language evolution process

JavaScript - The Definitive Guide - while you can get by doing google search for all your JavaScript related questions, this "fat book #1" has much to commend it and is much more current (in 7th Edition guise) and well written than much of what you will find on the web via google and complements the reference material on "Mozilla Developer Network" nicely

Building Web Applications with UML - more about Web Application Architecture in general than "building with UML" (1st Edition being assumed source for "Canonical Web Architecture" as referenced by "Developing Agent-Oriented Information Systems for the Enterprise" - Jaelson Castro, Manuel Kolp & John Mylopoulos, see below, however I first came across this via Grady Booch blog, which is no longer available on line...). 2nd Edition does not have "Canonical Web Architecture" figure.

AJAX - "Asychronous Javascript And Xml" - a web programming model that used the browsers JavaScript engine to create dynamic web content. This is significantly different from traditional web HTTP/HTML model where request resulted in "static" pages and any dynamic behaviour was through downloading of Java, Flash and ActiveX programs using the HTML <object> tag. Now all of these binary program models are obselete, thanks to JavaScript in the browser and alternate programming models like AJAX.

Thrift with JavaScript - my notes on getting these to work..

Appendices

Web Development & Programming Tools Usage

The following usage and survey data provide some useful hints in current trends and usage of Web Development Tools.

w3tech.com data is based on doing web crawling and so provides a view of what is out there, while statista is based on surveys and so give current trends and focus:

PHP's continued dominance as the most prevalent language for server-side programming is readily apparent but down from the 80% it had in 2013. ASP.NET has seen singificant decline from 19% in 2013. Ruby has lifted signficantly from < 1% to 5.7% and Java has staged a comeback from 2.8% in 2013 to 4.5%. JavaScript did not make the cut in 2013 but is now appearing.

The turning off of Flash, first by force by Apple and then its withdrawal by Adobe leaves only JavaScript on the client.

NodeJS appearing in the Web Server category is interesting as this is indicative of it support for "server-less" application server paradigm and its initial motiviation for providing non-blocking I/O model specifically tuned for web applications. Note, no other "server-less" delivery vehicle is on list (including AWS Lamba).

And now the survey results from Statista:

The survey results are hugely slanted to JavaScript based frameworks, paints a signficantly different view that the w3tech.com one, which give much more wait to Ruby, while PHP is not here at all.

On Programming Languages JavaScript is again the most popular, with Python, Java and PHP showing significant share. The appearance of "HTML/CSS" is clear indication of importance of CSS plays in creating "lovely looking" pages (like this one ;-) ).

For productive JavaScript development there are a large number of libraries available. Here is the usage based on web crawling:

Again there is discrepencies between web crawl and survery, but both have JQuery as essential, which other libraries are discretionary and likely a choice of one vs another will be required.

In telco there is an old adage that you create a service once and you assure it forever. Over the last five years Assurance has moved from "ITIL" focused to "Closed Loop Automation" to "Intent Based Network Operations".

To gauge the maturity of current and proposed Assurance capabilities, I use a coverage model, which generates a "heat map" of capabilities and on a per technology basis.

Coverage Model

The coverage model includes coverage of key ITIL processes:

• Alarm / Metric Collection - collecting data from systems that is basis for monitoring
• Incident Management - handing of incidents where focus is on faster restoration of service/s, which could be triggered from system or consumer/customer of service
• Change Management - where focus is on managing planned change in system and
• Service Request - which is focused on taking requests and manageing these

The model detail includes activities required to provide:

• Data Augmentation - to enrich the information that is availabe
• Aggregation - to aggregate multiple events resulting from alarms storm and providing summary view of this
• Root Cause Analysis - to determine underlying cause of failure / discrepancy
• Service Impact Analysis - to understand impact of failure / discrepancy
• Parent / Child Relationships - to show the relationships across alarms / incidents and discrepancies
• Notifications - to inform required parties / systems of events / incidents
• Reporting - provides aggregated reports of activitiy within monitored systems
• Problems Management - takes broader set of data to try to provide correlating events / changes that could contribute to problem
• Performance / Optimisation Management - looks at metrics and overall aggregate per formance of the network
• Closed Loop Automation - network ambition and coverage, that could be driven by events / alarms for automated break / fix automation (with notification to change management) or if looking at aggregate performance and optimisation is more likely to be served by.
• Intent Based Operation - which operates based on definition and mesurement of intent and having agents that can proposal actions drive systems to operating with defined intent thresholds.

Use of Model

The purpose of the model is not to "go all green" rather it is get an overall view of capabilities and there maturity within the different domains that might exist at the CSP. The example has domains: Applications, IT Systems / NFVi, Environmental, Optical, Ethernet, IP/MPLS & Wireless.

The purpose is to identify the targets for coverage and what capabilities would be needed to meet these.

This is consistent with current ITIL practice, which is not to for complete ITIL coverage but to target your coverage based on benefit and prioritities.

NOTE: The heat map spreadsheet is available by request.

References and Links:

ITIL - Information Technology Infrastructure Library (ITIL) was an initiative of UK Governments IT Agency to define clear processes and management of IT systems. This was expanded and adopted by Telecommunciations Industry. The "library" is now mantained by Axelos.

Closed Loop Automation - is closely related to Intent Based systems, but lacked formal specification of the intent. Initial objectives were defined in IBM's Autonomic Computing initative (2001). This outlined a sense and respond framwwork with primary goals of: Self Configuration, Self Healing, Self Optimisation and Self Protection.

Intent Based Network Operations - is currently one of TmForums strategic focus items in moving to automous network operations. To operate on intent requires clear way to specify and measure performance relative to objective and have agents that can propose actions when there is a divergence. So currently Intent based operations work best in context of overall network performance and operations

STATUS - Mar 2023 - updated an5 to have simpler and clearer goal description using "goal", "constraint" and "handler" keywords

STATUS - Jan 2022 - update example to reflect JSON support, as it trendy ;-) , but not necessarily easier to read

STATUS - 18 April 2021 - coded prototype and completed testing and measurement

Telecommunications services is a commodity business. In brief moments something new and shiny pops up that can command a premium price: the phone call, the telegram, the internet, mobile communications. Each of these growth driving cycles is then followed by need to drive out cost as commoditisation sets in and service providers try to find new and better ways automate operations, while improving customer experience. This focus on automation is key to profitability.

Within the network operations space there are three main process areas: Network Design and Build, Fulfilment and Assurance all having need for systems that support or perform some level of network design.

In general each of design & build, fulfilment & assurance focus on a set of tightly constrained areas to apply automation and then chain the various "little blocks" of automation into larger business processes.

This works to the extent that the structure of what is being managed does not change. If the structure does change, like in the case of a change in the underlying network technology or service topology then it is back to the drawing board to re-create the "little blocks" so they can be reassembled.

The "little blocks" are things like:

• Create an ethernet access service from location A to location B (a MEF example)
• Enable an alternate network path, when main path is experiencing unusually heavy load (a typical closed loop assurance response model)
• Connect VNF within NFVi platform up to internet ingress/egress point (an SDN / VNF example)

In each of the these cases the approach is to:

1. Find the right template
2. Apply the template allowing for specific considered conditions including input parameters, defined variations and resource allocations
3. If the template does not allow for condition then fall out into manual work queue

Things that "little block" approach cannot readily do are:

• Design to rebuilt an NFVi underlay network to support sharing of hypervisor attached storage across multiple Virtual Infrastructure Managers (VIM) - this is often a multi-month rebuild, requiring full redesign
• Create a new template to apply to a MEF ethernet access service,  after the  access technology has changed from FTTP (fibre to the premise) to FTTN (fibre to the node) - this requires the network engineer and the fulfilment developer to create new templates and potentially revised process to handle the new network topology.

Problems like these need an intelligent network design capability. "Intelligent" as this work is typically done by people with specialist knowledge and experience in network design.

My hypothesis is that if you start to apply Artificial Intelligence to the "bigger" network design problem then most of the "little block" problems will be solved at the same time, as these are mostly the final degenerate and terminating cases of the larger "Intelligent Network Design" problem.

This blog posting is about the architecture, implementation and testing of a prototype "intelligent network design" framework.

The purpose is to:

• Provide an example of using AI solution to network build domain and how this is different to typical telco automation
• Outline the key architectural structures of the framework
• Determine if the problem is tractable
• Gauge interest and get feedback in doing further development of the framework
• See how this could be used to support network planning and design (see diagram from Automation Life-Cycle context)

Introduction

I did an analysis on why Fulfilment systems in telecoms were so expensive to keep up to date. The result is that Fulfilment systems can be significant inhibitors to meeting "speed to market" objectives.

I found that these systems:

• dealt with abstractions that were little higher than procedural programming languages
• additional dynamic catalog driven process, allowed behaviour more like what is readily achievable with object oriented programming languages (greater abstraction and polymorphism), but this was not easily achieved,
• they did not make use of inventory data to find alternate solutions
• implementation typically centred around trying to codify network service and configuration designs provided by network engineers, which
• broke when there was any change in the structure of the underlying network (topology changes or network vendor devices changes).

The answer seemed to lie in having Fulfilment systems that were able to accomodate service topology changes, which in turn meant they needed better path finding capabilities. Path tracing is built on top of network inventory.

I documented some of this in my blog "Telco Inventory & Topology - Enabler for Automation".  Around 2016 Network Orchestration was being introduced as a way to try to decouple the network facing part of fulfilment from the customer facing part (the old Service Order Management vs Customer Order Management demarcation).

Two common technologies for Orchestration are:

• OASIS TOSCA , which is backed to ETSI and comes from IT world &
• Netconf/YANG from IETF with heavy backing by CISCO / Tail-f, from the IP network world.

Today most Orchestrators still use similar template driven process flows as traditional service order management systems, but now with inventory (or cut down Configuration Management DB - CMDB version of this) and activation more tightly coupled.

While it is relatively simple to implement a better network path tracing mechanism on top of a network graph, this only works when the edges in the graph already exist. If the edges don't exists then the graph based algorithm are not applicable.

A couple of cases where this exact situation occurs, at both a very small level and a larger level are the:

• need to apply a physical jumper link in network to realise a full circuit (such as occurs with cable patch panels, fibre distribution hubs and data centre rack building)
• need to run a new access tails to a given location (new estate building).

These problems either fall into the realm of network design or special purpose code that handles the design / assign logic for the particular case.

It seemed like there was a need to test a more general means to solve these type of problems, using an AI solution search based approach. As the search is not constrained to an existing network this will involve some "groping in the dark" to find a way get get from the known connection point to target point. Any such solution must be able to achieve this whether the "groping" is with physical or logical network constructs.

An AI based based solution is required, as the problem needs to build the constraints (i.e. network) not just resolve them. This will then allow the more typical service establishment fulfilment request to operate, but now with generated templates rather than hand crafted one. The design activity can then become one of verification.

Needs and Principles

If you are considering mostly network path design and follow on activation then the target architecture will need to use graph based inventory model and leverage the built in path search capabilities. The gap is that this presumes that the network design (and hence resulting graph representation) is already available. So ...

Decision #1 - Need to have architecture that is able to build paths from scratch not just navigate across existing paths.

Resolution - Need to implement a network design generating system, which is domain of AI planning and constraint based solvers.

Key Use Case - build a network from a collection of network components, using AI constraints based solver.

Decision #2  - Need to have a way to represent the problem space that is specific to the network design. Existing network representation models are generally not suitable for use with planning solvers (ITU X509), are focused on network configuration (Netconf/YANG) and do not have sufficiently rich network representation (Oasis TOSCA) or are targeted to a different problem such network simulation languages [14] or are just a syntactical representation choices such as JSON.

To resolve this I started to try to specify a network using JSON. It quickly became evident that trying to represent the domain problem via JSON was not intuitive. In fact it tended to obscure the problem. It seemed that the best way to generate JSON would be to write a small Java code prototype and then use it to generate the JSON output from the Java class / instance representation, rather than start with JSON and try to consume this as input into representational model.  So a chicken and egg problem.

As the domain needs to represent a network (a form of graph) the representation needed to support, the following types:

• Network - a collection of edge and nodes
• Network Element - a node within graph theory which could be a physical or logical network element
• Link - to represent the network edge and in networking this should be something that is used to link Network Elements together
• Path - a sequence network elements and links
• Interface - to create a network there needs to be a flexible way to specify interfaces that define how the elements/links that make up a path can be connected together

Resolution - Define a network modelling language that can describe the network parts as well as the build goals and template.

The defined language was - an5  (a nETOWK LANGUAGE WITH 5 CLASSES). This would then allow specification of components and interface compatibilities. The interface compatibilities are used to determine how the network can be built by plugging together the building blocks. The an5 language structure allows specification of an interface including cardinality and signatures:

• common - the common signature definition (plug = rj45)
• provides - what the interface provides (sex = female)
• needs - what the interface needs (sex = male)
• service - what service will be enabled as a result of successfully binding against an interface

The an5 language was derived from Java including similar semantics for interfaces and class inheritance, but language is mostly declarative, without need to implement methods (though this might change with need to add resource allocation behaviours).

Key Use Case - be able to write network specification using an5, compile this and then use generated package to define network build "programs"

The following is an example of an an5 specification for:

• Computer with PCIe slots
• NIC with Ethernet Ports
• Ethernet Switch
• cat6 cables

/*
  What: A small test example of set of classes
     that can be used to assemble a data centre network.
     It is example of set of a nETWORK LANGUAGE WITH 5 CLASSES
     definitions, not instances.
     
  By: John Hartley - Graphica Software/Dokmai Pty Ltd
    

*/

package an5.generic.dctypes;

@Model(
  spec = "an5",
  version = "0.2"
)

interface pcie_interface {
  common = { "type=pcie", "width=(1|4|8|16)", "gen=([1-4]\\.*)"};
}

interface pcie_slot extends pcie_interface {
  needs = {"form=(card)?"};
  provides = {"form=slot"};
  binding = "slot-%I+1";
  string name;
}

class computer extends element exposes pcie_slot {
  reflects pcie_slot[] slot;
  string name;
}

interface rj45_plug {
  common = { "plug=rj45"};
}

interface baset_sink extends rj45_plug {
  needs = { "cable=(cat([3-8].?))?",
            "gender=male", "media=copper"};
  provides = {"plug=rj45", "gender=female"};
}
 
interface ethernet_port_baset extends baset_sink {
  common = { "service=ethernet"};
  binding = "p-%I+1";
  string name,
         MAC;
}

interface ethernet_lag {
  common = { "service=(ethernet)+" }
  binding = "lag-%I";
  string name;
}

interface ethernet_vlan {
  common = { "service=(ethernet_vlan){0,4096}" }
  needs = { "service=ethernet" };
  binding = "vlan-%I";
  string name;
}

class ethernet_lag_link extends link exposes ethernet_lag {
  reflects ethernet_port_base_t[] ports;
}

class ethernet_vlan_link extends link exposes ethernet_lag, ethernet_vlan {
  reflects ethernet_lag lags[];
  reflects ethernet_port_base_t ports[];
}

interface pcie_card extends pcie_interface {
  needs = {"form=(slot)?"};
  provides = {"form=card"};
}

class pcie_nic extends element exposes pcie_card, ethernet_port_baset {
  reflects pcie_card card;
  reflects ethernet_port_baset port[];
}

class switch extends element exposes ethernet_port_baset, ethernet_vlan, ethernet_lag {
  reflects ethernet_port_baset port[];
  reflects ethernet_vlan vlan[];
  reflects ethernet_lag lag[];
  string name;
}
 
interface catx_link extends rj45_plug {
  provides = {"cable=(cat([3-8].?))?", "gender=male",
            "media=copper"};
  needs = {"plug=rj45", "gender=female"};
}

interface cat6_link extends catx_link {
  provides = {"cable=cat6"};
}
  
class cat6_cable extends link exposes catx_link {
  string type = "physical",
         colour,
         label;
   int length;
   reflects catx_link a, b;
}

/* new 0.2 spec */
goal class ethernet_lan extends network {
  @mandatory switch[] fabric;
  @mandatory computer[] hosts;
  object[] uses;
  service = { "ethernet", "(ethernet_vlan)*" };
  handler = "an5.solve.an5CreateNetwork";

  constraint class ethernet_node extends element {
    @mandatory computer host;
    @mandatory switch ether;
    object[] uses;
   }
}

/* old 0.1 spec
abstract class ethernet_lan extends network {
  @mandatory switch[] fabric;
  @mandatory computer[] hosts;
  object[] uses;
  service = { "ethernet", "(ethernet_vlan)*" };
}

abstract class ethernet_node extends element {
  @mandatory computer host;
  @mandatory switch ether;
  object[] uses;
 } */

The compiler takes (Java like) an5 specification as input and will generate compiled code for efficiency. As the an5 syntax specification is very Java like this means it it easy and possible to generate Java code directly and avoid having to create an intermediary meta-model for an5 representation (see Decision #6). This has benefits in ease of debugging models but also in overall performance of solve engine.

Decision #3 - Need to seperate out the generic CSP solver and its search algorithms from the network domain level problem reduction logic, combinatorial generators, scoring and domain knowledge based optimisation.

Resolution - Create generic AND/OR solver (see Nilsson [1] & Winston [2]) for main control structure and have this invoke domain level template. The class level structure is illustrated here:

Example Use Case - Create Network by connecting 3 computers to 1 switch/router:

• R(n) = J(R(switch/router), 3 x Computer)) - "create resulting network R(n) by joining J(network, 3 x elements) the set of elements to an initial network R(switch/router)" or
• R(n) = C(J(R(switch/router), 1 x Computer), J(R(switch/router), 1 x Computer), J(R(switch/router), 1 x Computer))) - "create a network by connecting C(J(R(n)), J(R(n)) ..) multiple networks created by joining J(R(switch/router), 1 x Computer) a set of individual networks with have a common central hub (switch/router).

This shows that there are alternate ways to reduce the initial problem into a set of smaller problems.  In the top case the result is an OR tree only, while in the lower case the result is an AND/OR tree structure. In general the solution will require AND/OR based resolutions. Research has also found that using AND/OR can result in significant reduction in the search space size in comparison to using OR tree solvers only (see Dechter [3][4]).

Decision #4 - Need to be able to provide delayed resource allocation as a result of binding to interfaces. This is essential in context of using AI search as allocation of resource at the time of initial binding could result in consumption of resources prior to determination that the search path is a successful one. So need to delay allocation until the search completes with set of valid results and only then commit resources from the finite pool. Typical example is things like IP address or VLAN Id allocation.

Resolution - Define state model for interface binding that allows for binding to be "un-committed" unless explicitly committed.

The binding state model is:

• "Open" => "Base Match" == un-committed Binding against base definition with no open unmatched items
• "Open" => "Reflecting" == un-committed partial Binding against base definition or uncommitted Binding with open unmatched items, with resulting open items and/or service outcome being re-exposed as new interface
• "Base Match" or "Reflecting" => "Committed" == commit previously un-committed Binding, doing any required resource allocation and remove any Reflecting interface from available pool

The Interface and Binding model is critical to allowing specification of virtual services such as Ethernet VLAN and aggregate links such a Ethernet LAG.

NOTE #1: That the semantics of network service such as Ethernet VLAN or Ethernet LAG is coded as an5 language specification and not through the solver runtime (see Decision #5)

Decision #5 - Network protocol interoperability and interface compatibility should only be specified through an5 language specification and not built into the an5 runtime engine

Resolution - provide Interface and associated binding mechanism that:

• allow rich specification of "plug-ability" across the components and
• allow instantiation of new services and protocols as a result of binding across these interfaces and
• support being able to dynamically add new "binding with associated service & protocols" back into the set of available interfaces for further level of "plug-ability".

The other mechanism included in Interface binding is use of "regular expressions" to allow variations across interfaces for compatibility variations (i.e. PCIe Gen 1, 2, 3 &4).

Key Use Cases - Use the interface and binding to support recursive layering of an5 interface and binding to achieve protocol layering such : as physical to data link layer and ethernet to IP layering.

Here is a sample of a set of matching interface / binding specifications and classes that use them:

// A set of an5 interface spec
// for baseT ethernet with VLAN and LAG support
//

interface rj45_plug {
  common = { "plug=rj45"};
}

interface base_t_sink extends rj45_plug {
  needs = { "cable=(cat([3-8].?))?",     <<== Regular expressions
            "gender=male", "media=copper"};
  provides = {"plug=rj45", "gender=female"};
}
 
interface ethernet_port_base_t extends base_t_sink {
  common = { "service=ethernet"};        <<== Service outcome
  binding = "p-%I+1";                    <<== Naming substitution rules
  string name,
         MAC;
}

interface ethernet_lag {
  common = { "service=(ethernet)+" }  <<== Binding cardinality
  binding = "lag-%I";
  string name;
}

interface ethernet_vlan {
  common = { "service=(ethernet_vlan){0,4096}" } <<== Binding cardinality
  needs = { "service=ethernet" };
  binding = "vlan-%I";
  string name;
}

class ethernet_lag_link extends link exposes ethernet_lag {
  reflects ethernet_port_base_t[] ports;
}

class ethernet_vlan_link extends link exposes ethernet_lag, ethernet_vlan {
  reflects ethernet_lag lags[];
  reflects ethernet_port_base_t ports[];
}

NOTE #2: in defining the interface/binding specification I reviewed Canonical JuJu which provides a means of defining component dependencies through interfaces. However JuJu did have have sufficient control mechanisms and could not accommodate the recursive exposure mechanism required to allow reflection of bound interface to participate in further protocol layers and aggregation cases.

Decision #6 - In deciding to use a specification language (Decision #3) for problem space specifications, there is need to determine how to represent this within the application layer implementation. The typical approach is to define a meta-model representation within the application layer and then create an interpreter to operate on this. As an5 provides an object oriented specification language leveraging Java, the alternative was to leverage the Java meta-model (as used by underlying Java Virtual Machine).

Resolution - Leverage Java meta-model and JVM, as this already supports using compiled code for performance and has an introspection interface to allow runtime interpretation of generated models (from compiler). Another candidate considered was to use JavaScript, however this result in use of fully interpreted model, which in context of very large combinatorial and search space calculations would have significant performance impact.

Decision #7 - More clearly distinguish the Goal specification from the network componentry parts. This was based on feedback that is was hard to see how an5 helped with specification of "target". This was related to an5 leveraging Java and not wanting to introduce to many language elements.

Resolution - Add "goal", "constraint" and "handler" keyboards (and retire Java "abstract") so that is very easy and simpler to specify goal (an5 v0.2 - March 2023). See example here:

/* new 0.2 spec */
goal class ethernet_lan extends network {
  @mandatory switch[] fabric;
  @mandatory computer[] hosts;
  object[] uses;
  service = { "ethernet", "(ethernet_vlan)*" };
  handler = "an5.solve.an5CreateNetwork";

  constraint class ethernet_node extends element {
    @mandatory computer host;
    @mandatory switch ether;
    object[] uses;
   }
}

/* old 0.1 spec
abstract class ethernet_lan extends network {
  @mandatory switch[] fabric;
  @mandatory computer[] hosts;
  object[] uses;
  service = { "ethernet", "(ethernet_vlan)*" };
}

abstract class ethernet_node extends element {
  @mandatory computer host;
  @mandatory switch ether;
  object[] uses;
 } */

Architecture

The architecture for an5, was developed iteratively and not top down:

• The first step was some paper based network definitions and instance models, to try to get idea of how to represent the problem.
• Next there was an attempt to represent these with JSON, then
• Use a specific language to represent the problem space.

As a result of this it became apparent that there was need to do some prototype development and testing to validate the overall ambition for "Intelligent Network Design", as it was to easy to miss the hidden details that needed to be dealt with and get missed in just doing architecture.

So the set of needs and decision outlined above, arose out of initial analysis and practical verification with principle of using "Just Enough Architecture" ;-) .

The overall goal for an5 are to provide:

• a general purpose network representation language and model to support planning, design, simulation and ultimately automated provisioning of IT & network platforms and services (see Cloud Automation/Life Cycle Management diagram in introduction)
• intelligent solvers that can "build networks", "connect to networks", "join networks together", "connect elements within network" and "find paths across network"
• provide network builder that allows fast "snap together" network assembly capabilities that can act as seed into intelligent network design solvers

The basic architecture packaging consists of:

• an5 - a nETWORK LANGUAGE WITH 5 CLASSES - the core (and still evolving) network specification and template language
• compiler - the an5 compiler generates Java code for the models. The generated models are packaged into Java class archives
• model - the core an5 Java class library, which contains the base classes inherited by the generated an5 classes
• solver - the Intelligent Network Design solvers
• cases - the case analysis and repository to support case detection for use in providing search optimisation by solvers
• learner - statistics and correlation analysis to identify and put probabilities against network build cases

For the initial testing and validation the focus is on: compiler,  model and solver, which form the MVP core of framework.

In creating the prototype solver the architecture has had to leverage a number of techniques and disciplines which I will outline in the Compiler and Solver package summaries.

Compiler

The compiler implementation is based on antlr. Antlr provides a mature Java based lex/yacc style compiler generator. Antlr generates a parse tree and supports tree walking call backs for symbol table build and code generation.

As per Decision #2 an5 is Java derived and so could be implemented quickly re-using (hacking ;-) ) an existing antlr grammar for Java (significantly reduced), rather than starting from scratch.

Solver

To implement any real world AI solution requires using a large number of techniques. The an5 prototype implementation - required assembling most of the basis AI building blocks as outlined in classic AI text "Artificial Intelligence by Patrick Henry Winston" [2] including:

• "Semantic Nets" for representation,
• "Generate-and-Test Method" for combinatorial generator. This initial proof has three generators,
• which use "The Problem Reduction Method" to take bigger network design problem and break it down into smaller set of sub-problems, which in turn are represented by AND/OR trees (see Decision #3)
• "Blind Methods", "Heuristically Informed Methods", "The Best Path", "Redundant Paths" search method are all used to help manage the combinatorial search space that is always lurching when there is need to search for potential solutions (formally NP-Complete problems).

When using an AI solver the generator may create sets of cases that are different from initial expectations. This was the case for the network design generation. To help understand the solver behaviour it was instrumented to collect run-time statistics as well has having skeleton "why" and "what" solution behaviour introspection.

Summary

The initial prototype only implements a portion of outlined architecture (compiler, model & solver).

While the current implementation is based on classic queue based Solver, it would be relatively easy  to use AND/OR problem breakdown to distribute solver process across a distributed set of hosts to support solving "big problems"

The key question that needs to be addressed before implementing an "Intelligent Network Design" solution is to ensure that the problem is not intractable, due to combinatorial explosion.

To check this a paper based analysis and implementation testing was done, using a simple network design as outlined in the following "The Monkey and Banana - Test Case" and "Tractability - Combinatorial Analysis and Optimisation" sections.

The Monkey and Banana - Test Case

To test the an5 representation and solver I used a network variation of "the monkey and banana" problem as observed by Wolfgang Kohler, back in 1917 [15].

In the monkey and banana problem you must be able to solve the problem of "how can  the monkey get the bananas?". The problem space has set of rooms with boxes, a monkey, a stick and a bunch of bananas that are too high to reach directly. To get the bananas the monkey needs to:

• move boxes under the banana
• stand on boxes with stick
• reach the bananas with the stick

The network equivalent as to connect 3 computers (boxes) to the network switch (bananas):

I selected this case as:

• it had the situation which is analogous to the "monkey and banana" case against which the original STRIPS[13] constraint based solver was tested,
• but with an additional layer of combinatorial generation that is needed in network domain
• dealt with simple physical and logical interface cases to allow test and development of this domain layer
• it was sufficiently non-trivial to allow testing against different search algorithms including Depth First, Breadth First, Branch and Bound, A*, British Museum (depending on search space size) and AND/OR tree based search
• it required problem reduction method to be applied

The test implementation provides the following search options (see Winston [2]) and domain optimisation choices:

public class an5SearchControl {
  public static class SearchOptions { public static final int DEPTH = 0x001,
                                      BREADTH = 0x002, BOUND = 0x004, SCORE = 0x008,
		                              COST = 0x010, REMOVE_LOCAL_EQUIVALENTS = 0x020,
                                      BIND_ALL = 0x040, BIND_UNIQUE = 0x080,
		                              THREAD = 0x100, TIMEBOX = 0x200,
                                      OPTIMAL = 0x400, KEEP_REMOVED = 0x800; }
....
}

Tractability - Combinatorial Analysis and Optimisation

The basic building blocks of AI problem solving using problem reduction and search are need for:

• Set of basic rules to allow problem reduction - to break bigger problem into set of smaller ones
• Generator - a combinatorial generator that should provide each of possible solutions for a given state of the problem solving and
• Search Engine - that controls the application of problem reduction, invokes generator and manages search for solutions.

The big enemy of AI search is combinatorial explosion, where the number of search alternates is to large to handle, making the problem intractable. To counter this the generator and search engine do not generate all solutions at once. Rather it generates a set of candidates and then tests and trims these so only a subset of the total number of search paths are expanded.

Even the simple test case outlined above would be intractable without apply some level of optimisation. Optimisations fall into a number of categories:

• Generic Search Control - applicable to any search space problem and operate by providing alternate search algorithms including: Depth First Search, Breadth First Search, Branch and Bound, Score & Cost based Search
• Domain Optimised - where the search optimisation if based on applying some pruning based on the problem domain of the search
• Case Based - where search is optimised based on historical or statistical analysis of paths which have most likelihood to provide preferred or right solution. These are ones which leverage machine learning techniques.

The other aspect of search based solutions is that with large search spaces you can not guarantee that you will find the optimal or best solution and so search is finished on finding the first solution or constrained to run for a fixed time.

In the case of above our "monkey and banana test" the set of combinations that need to be generated is equal to:

Combinations to generate = N(h) x N(s) x N(n) x N(p) x N(c)

where:

• H(h) - numbers hosts
• N(s) - number of PCI slots per host
• N(n) - number of NICs
• N(p) - number ports per NIC
• N(c) - number of cables

This is a lot of multipliers ... here what it looks like if we go from 1 host + nic -> 12 hosts + nics:

So even with the seemingly trivial "monkey + banana" network at 12 hosts the problem becomes intractable to comprehensive search.

In practice you can solve with 8 hosts but at 9 hosts memory allocation fails, due to needs of generator and at 10 hosts the size of combination space results in Java int (32 bit integer) overflow. At 7 & 8 hosts the solver spends all its time just generating the combinations, not searching them.

NOTE: you cannot have an array index of type long (64 bit integer), as it is not possible to allocate a block of memory this large.

Looking at the combinatorial behaviour shows that the largest number combination sets comes with "cable to NIC card" step, after which problem becomes trivial (direct connection of cable to switch) again:

As expected from the simple network topology the best search strategy in this case is just a "Depth First Search" as both best Scored and Cost based search end up selecting Depth First Path and so just add the overhead of calculation.

NOTE: That in general the network construction search cannot come up with an optimal scoring or cost algorithm as the estimate for "distance" to solution is never know prior to reaching a solution.

So without some other optimisation the network build is intractable.

In observing the behaviour it was apparent that the AI Solver was doing things that would not be required in real world. In particular the generator was doing a great job of building all the combination of ways that you could plug the various compatible interfaces together. So for 1 Computer it would generate every next solution combination set of:

C(h,n) = (h1-n1, h1-n2, h1-n3 ... h1-nX, h2-n1, h2-n2, h3-n3 ... h2-nX, .... hY-nX) where:

• C(h,n) - set of combination of hosts (h) and NICs (n) consisting of pairs:
• h1-n1 .. - host #1 with NIC #1, host #1 with NIC #2 etc

In additional it was also generating combinations of multiple NIC cards within single host (as each host was defined as having 4 slots).

In practice in the real world NICs and cables are treated as "commodity" devices and the result of assembling a path using any particular instance of an object of the same class of will result in the same outcome, irrespective of which instance I use.

In simple terms this means that if I have 10 Cat6 cables available to plug from a NIC port to a switch port then there is no advantage to generating all the 10 NIC port to cable to switch port combinations, as the result is same for each of the very large number of combinations. So instead you can just generate the first instance and make the cable un-available for further combinations.

To test this I created a "Local Equivalent Removal"  option within the Domain solver layer. The algorithm was "Local" as it operated solely within scope of a given interface bind request and removed cases of binds which have class template equivalency.

If you do not prune the "Local Equivalents" the result is a huge number of essentially identical search trees. Avoidance and/or caching for reuse of these cases is described in Dechter[4] as part of AND/OR search analysis.

Practically the use of "Local Equivalent Removal" was to stop the combinatorial explosion in its tracks with each new depth step now having breadth of 1. So the Domain optimisation results in a reduction from 10 to power of 12 (=power(10,12) in Excel) to 1. Thats a better return than on BitCoin !

This is a huge reduction and illustrates how application of domain knowledge takes the problem from intractable to solvable in trivial time (less than a second in practice).

There is a lot of scope for application of Domain knowledge in this context as well as Case Based optimisation.

Importantly there is good reason to believe that solving network design automation using AI techniques will yield positive results.

Conclusion & Next Steps

The an5 language provides a way to specify both network building blocks and templates. The generated class library is then easily used to "write" a network solver. The simple solver definition below was used for the "Monkey and Banana" testing. This is now mostly a declarative exercise, using the an5 device library specification shown in Decision #2 (above):

/**
 @what An example network create test file
 
 @author John Hartley - Graphica Software/Dokmai Pty Ltd
*/

import java.util.ArrayList;
import java.util.List;

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonMappingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializationFeature;
import com.fasterxml.jackson.databind.module.SimpleModule;

import an5.model.*;
import an5.solve.*;
import an5.generic.dctypes.*;


public class BuildMiniNetwork {
  public static void main(String[] args){

    /* lots of Jackson JSON Serialize/DeSerialize initialisation */
    ...
    ...
    
    /* Switch */
    String swdef = "{\"an5name\":\"switch\",\"name\":\"simple-switch\",\"reflects\":[{\"name\":\"port\",\"policy\":\"STATIC\",\"size\":24}]}";
    
    /* 12 x Computer */
    String[] cpdef = {"{\"an5name\":\"computer\",\"name\":\"hal-1\",\"reflects\":[{\"name\":\"slot\",\"policy\":\"STATIC\",\"size\":4}]}",
	  "{\"an5name\":\"computer\",\"name\":\"hal-2\",\"reflects\":[{\"name\":\"slot\",\"policy\":\"STATIC\",\"size\":4}]}",
    ...
    ...
	  "{\"an5name\":\"computer\",\"name\":\"hal-12\",\"reflects\":[{\"name\":\"slot\",\"policy\":\"STATIC\",\"size\":4}]}"};
    
    /* 12 * NIC */
    String[] nicdef = {"{\"an5name\":\"pcie_nic\",\"reflects\":[{\"name\":\"port\",\"policy\":\"STATIC\",\"size\":2}]}",
	      "{\"an5name\":\"pcie_nic\",\"reflects\":[{\"name\":\"port\",\"policy\":\"STATIC\",\"size\":2}]}",
    ...
    ...
	      "{\"an5name\":\"pcie_nic\",\"reflects\":[{\"name\":\"port\",\"policy\":\"STATIC\",\"size\":2}]}"};
    
    /* 13 * Cat6 */
    String[] cabdef = {"{\"an5name\":\"cat6_cable\",\"type\":\"physical\",\"length\":100};",
	      "{\"an5name\":\"cat6_cable\",\"type\":\"physical\",\"length\":100};",
    ...
    ...
	      "{\"an5name\":\"cat6_cable\",\"type\":\"physical\",\"length\":100};"};

    AN5CL_switch sw1 = null;
    AN5CL_computer cp[] = {null, null, null, null, null, null, null, null, null, null, null, null};
    AN5CL_pcie_nic nic[] = {null, null, null, null, null, null, null, null, null, null, null, null};
    AN5CL_cat6_cable cab[] = {null, null, null, null, null, null, null, null, null, null, null, null, null};
	    
    try {
      sw1 = new AN5CL_switch(new ObjectMapper().readTree(swdef));
      for (i = 0; i < cpdef.length; i++) {
        cp[i] = objectMapper.readValue(cpdef[i], AN5CL_computer.class);
      }
      for (i = 0; i < nicdef.length; i++) {
        nic[i] = objectMapper.readValue(nicdef[i], AN5CL_pcie_nic.class);
      }
      for (i = 0; i < cabdef.length; i++) {
        cab[i] = objectMapper.readValue(cabdef[i], AN5CL_cat6_cable.class);
      }
    } catch (JsonMappingException e1) {
      // TODO Auto-generated catch block
      e1.printStackTrace();
    } catch (JsonProcessingException e1) {
      // TODO Auto-generated catch block
      e1.printStackTrace();
    }

    an5Object[] use = {sw1,
    		           cp1, cp2, cp3, cp4, cp5, cp6,
    		           cp7, cp8, cp9, cp10, cp11, cp12,
    		           nic1, nic2, nic3, nic4, nic5, nic6,
    		           nic7, nic8, nic9, nic10, nic11, nic12,
    		           cab1, cab2, cab3, cab4, cab5, cab6,
    		           cab7, cab8, cab9 , cab10, cab11, cab12, cab13};
    List parts = new ArrayList<>();
    for (an5Object ob : use) parts.add(ob);
    
    AN5TP_ethernet_lan  netPrototype = new AN5TP_ethernet_lan();
    an5Network netResult = (an5Network)netPrototype.createInstance();
    
    an5Template netTemplate = new an5CreateNetwork(null, netPrototype, parts, netResult);
    

    an5SearchControl ctrl = new an5SearchControl();    
    
    int offFlags3 = an5SearchControl.SearchOptions.COST;
    int onFlags1 = an5SearchControl.SearchOptions.DEPTH;

    ctrl.turnOff(offFlags3);
    ctrl.turnOn(onFlags1);
    
    an5Goal makeNet = new an5Goal(netTemplate, ctrl);
    int res = makeNet.solve();
    
    System.out.println("Result was: " + ctrl.resultString(res));
    ctrl.dumpStats(System.out);
    
    for (an5GoalTree t: makeNet.found) {
      an5FoundGoal resGoal = (an5FoundGoal)t;
      if (resGoal.goal instanceof an5JoinNetwork) {
    	an5JoinNetwork jn =  (an5JoinNetwork)(resGoal.goal);
    	an5Network no = jn.joinNet;
    	no.dumpJSON(System.out);
      }
    }
    System.out.println("switch reports as" + sw1.toString());
  }

}

NOTE: All classes with name AN5TP_ are generated templates and all classes with names AN5CL_ are generated classes. The classes with prefix "an5" are from the model or solver class library.

As the code is mostly declarative it could be readily used to provide a more complete "Intelligent Network Design" automation tool by building an interpretive interface on top of this to support a web UI.

The initial an5 implementation has successfully proven the concept of being able to provide an "Intelligent Network Design" using classic AI representation, combinatorial generation and AND/OR solver with flexible control.

Of particular importance is that it shows that applying a very simple and protocol neutral optimisation you can take the problem from intractable to solvable in near real time.

In practice this means that you it is viable to apply AI to classic fulfilment problems as part of re-thinking the approach to fulfilment.

Using features such as:

• Case Based Heuristic where you could constrain the search size to find solutions to larger design problems by using reference case sets
• Allowing solver to run for bounded time you could find multiple solutions, then select "best" and "preferred" solution and convert these to templates automatically by converting from solution instance to class based templates
• These class based template in turn can be feed into "Case Based Heuristics"

In this initial proof of concept only the "Create Network" and "Connect Element" solvers were used with an OR tree control framework.

The prototype implementation is around 7,000 lines of Java and Antlr code and the example "dc-types" specification generated code is around 500 lines of Java.

Next steps could include:

• Adding additional solvers
• Complete interface specification "regular expression" binding
• Add partial interface binding and service reflection to support layered protocols implementation
• Refine an5 language syntax to include interface specification details (currently these are just sets of name/value pair strings and not validated by compiler)
• Refine an5 language to support nested templates target network specifications (ie specify network outcome + connectivity constraint templates within single template class - initial update done with an5 V0.2 March 2023)
• Add full AND/OR solving support
• Add case based heuristic support
• Add automatic template generation for application in search optimisations. This would meet goal of being able to provide behaviour consistent with existing template / parameter (+ process) fulfilment systems. With solver based solution there would not be need for explicit process definition, as process would be based on dependency needs.
• Add JSON dump/load classes (using Jackson or similar library) (done Jan 22)
• Add distributed solver and control framework
• Provide UI based network builder leveraging interface / binding model to allow snap together designs based on an5 device libraries
• Test an5 with more network models and build up set of device and network model libraries
• Add interface to device layer so an5 can plug into lower level technologies such as YANG for configuration, JuJu for automated platform building and TOSCA based orchestration engines.

So prototype has  proven the viability of the concept, but there is still lots to do. I have put the code up on github and believe that an5 is good candidate for Open Source development.  So if there are other programming and network design savvy people who would like to help flesh out the framework, please muck in ;-) .

"Intelligent Network Design" is needed to help telco improve their speed to market and make significant impact in reducing cost for network design and fulfilment systems development.

References and Links:

[1] Principles of Artificial Intelligence - Nils J. Nilson – defined AND/OR tree problem representation

[2] Artificial Intelligence (3rd Ed) - Patrick Henry Winston - creating an5 and its solver required piecing together most of the parts (representation, generate & test, search)  outlined in the first 5 chapters of this classic AI text. You can also see Patrick's lecture that cover the same topics on MIT OpenCourseWare videos

[3] Reasoning with Probabilistic and Deterministic Graphical Models - Exact Algorithms (2nd Edition) - Rina Dechter

[4] AND/OR Tree Search for Constraint Optimization - Radu Marineescu & Rina Dechter , paper on impact of introducing AND/OR into search

[5] Game-Tree Search Using Proof Numbers: The First Twenty Years - Akihiro Kishimoto,  Mark H.M. Winands, Martin Muller & Jahn-Takeshi Saito

[6] Network Design Automation - Treating Networks like Programs and Chips -George Varghese - other people are thinking about this...

[7] SAT v CSP - Toby Walsh - paper provides review of semantics and mapping between Propositional Satisfiability (SAT) and Constraint Satisfaction Problems

[8] Juju - Canonical Infrastructure Automation

[9] Compilers - Principles, Techniques and Tools - Aho, Sethi & Ullman

[10] Modern Compiler Implementation in Java - Appel

[11] lex & yacc (2nd Ed)- John R Levin, Tony Mason & Doug Brown

[12] The Java Language Specification, Third Edition

[13] STRIPS: A New Approach to the Application of Theorem Proving to Problem Solving - Richard E. Fikes & Nils J. Nilsson - original paper on STRIPS (STanford Research Institute Problem Solver) constraint based solver

[14] Network Simulation Tools Survey - Mrs. Saba Siraj, Mr. Ajay Kumar Gupta & Mrs Rinku-Badgujar

[15] The Mentality of Apes - Wolfgang Kohler - the source of the "monkey and banana" problem and assumed source of the title picture.

[16] "Telco Inventory & Topology - Enabler for Automation" - provides a PoV on role of inventory in core Telco processes

[17] OASIS Tosca - Topology and Orchestration Specification for Cloud Applications

[18] Netconf/YANG (IETF RFC 6020, 7950 and others),  and Cisco/Tail-f - YANG is a data definition language for network configuration. Cisco / Tail-f have leveraged this to provide an intent based orchestration offering

[19] Wolfgang Köhler’s the Mentality of Apes and the Animal Psychology of his
Time - Gabriel Ruiz - provide a review of Kohler's work

[20] Is CBR a Technology or a Methodology? - Ian Watson - provides a survey of "Case Based Reasoning" which uses case database to drive problem solving

[21] an5 - github respository

[22] an5 - presentation to Linux Foundation Networking / GSMA - Anuket Reference Model team

NOTE: The "monkey and banana" image can be found via google search "wolfgang kohler monkey photo", however I cannot find definitive paper/book from which this was taken.

This introduction to 5G architecture, provides a capabilities view as context to consider how 5G can enable alternate network service delivery models.  The purpose is to translate the technical capabilities  5G provides into a small set of basic build blocks, to highlight how 5G differs from 4G/LTE.

The article is the first of two, this one looks at the architecture and Part 2 looks at the business model.

As this is about the architecture, here is my “simple” illustration of 5G Architecture and why “it’s different this time”:

The 5G network architecture provides a wireless "Access Network" with (tracing left to right):

• a Radio Area Network (RAN),
• IP network underlay / overlay networks (RAN Edge, Pre-Aggregation, Aggregation) providing connectivity
• via "IP / MPLS / Transport Network" to
• Central Control (3GPP) applications including (for illustrative purposes here): Enhanced/Evolved Packet Core (EPC), IP Multimedia Sub-System (IMS) and Unified Data Management (UDM - provides 5G functions simillar to Home Location Register (HLR) & its friend Visitor Location Register (VLR) for roaming support).

As users we experience this via our phones (EUD’s End User Devices) which: connect and authenticate with the Wireless Network via the UDM, where our SIM / IMSI / Phone Numbrer subscriber information is provisioned, opening communications traffic for voice (to other subscribers and phones systems) or data services (browse the Internet) via IMS which provides a gateway to various cloud services available.  All very Monty Pythonesque at this level to the technical experts ;-) .

The communications services providers (CSPs) deploy their network assets at various "Deployment Sites". Moving from right to left the sites go from Central Data Centre (per State) to Regional Data Centres to Edge Sites (even more broadly deployed and smaller footprint sites) to Deep Edge Sites (very small footprint sites that could exist at the wireless antenna bases).

In the current wireless network (3G/4G/LTE) the access networks are mostly built from physical networking elements (NEs).

In 5G the same topology exists but rather than having sites populated with physical NEs, instead each site contains Network Function Virtualisation infrastructure (NFVi). These NFVi PoDs (built from general purpose computing and networking components) host Virtual Network Functions (VNFs) and Cloud Native Network Functions (CNFs). In the 5G architecture the NFVi is not just a "nice to have", rather the telco equipment manufactors (TEMs) for 5G Cores are building their solutions as cloud native (containerized) components that can be deployed dynamically across the distributed NFVi infrastructure. The 5G core is being built with the same technologies (Kubernetes, OpenStack, Ceph etc) that  web-scalers and digital natives build their cloud applications on. The result is a distributed Network Cloud.

Another significant change introduced with 5G are “Network Slices". This allows a single set of physical infrastructure to provide completely isolated and independent wireless networks across it. These network slices work from the Radio Access Network (RAN) all the way through to the Central Data Centre deployed VNFs / CNFs.

The "Network Slices" can support optimising the architecture's cost and operational and performance characteristics, by instantiating the CNF & VNF components differently across the Network Cloud. This is illustrated in the model by the varying size of the deployed function across the different slices.

In traditional wireless network most of the "Value" processing components, are provided by "Over The Top" (OTT) functions residing at the Point of Interconnect (in central data centre) or as cloud based services on the Internet.  In 5G architecture it is possible to provide gateways to these functions anywhere within the network topology. In the model these are illustrated as “Gateways to Value” (in 3GPP parlance this is provided by Packet Gateway and Service Capability Exposure Function - SCEF). The "Value Engines" that sit on top of the "Access Network Slices" provide a further layer of virtual function hosting for the "Value" services.

Thus the 5G architecture can provide alternate “Paths to Value”. The "Value Engines" could be hosted on the same NFVi infrastructure as the Access Network VNFs / CNFs or on its own virtualised infrastructure.  The "Value Engines" could also be operated independently from the "Access Network" functions and might have completely different organisational owners. How to deploy the "Value Engine", be this on shared or specific infrastructure or using alternate organisational providers is a business strategy decision that will have to consider: organisation capability & ambition, security, performance and other concerns.  The virtual "Gateway to Value" is a critical enabler for integration from Access Network to Value Engines.

The model illustrates the creation of alternate deployment architectures with alternate “Paths to Value” for:

• Latency Critical (lightning) – functions typically needed for  control systems and other use case where latency is critical and processing must occur as close to device a possible (autonomous vehicles or drone controls being favourite use cases),
• Low Latency (racing car) – functions  an example might be car sat-nav systems, which must provide timely map data refresh to be useful
• Enhanced Experience (hare) – where user experience is significantly improved through provision of responsive UI, as happy customer is a retained customer
• Traditional Path (snail) – this follows all the way though to Point of Interconnect (POI) for value processing and provide the cheapest way to deploy functions.

In a mature 5G deployment, the operational architecture variation are not instantiated statically rather they are provisioned and managed automatically via 5G slice aware orchestration and network management solutions.

In summary the 5G architecture is different because:

• All 5G Core network functions are provided by: VNF / CNF based software
• NFVi CNF / VNF / Value Engine Hosting environments are provided ubiquitously across all sites
• Network Slices allow sharing of physical infrastructure and cost / performance optimised deployment
• Alternate Paths to Value allow cost / performance optimised architectures via Gateways to Value (CNFs/ VNFs) for access to Value add (OTT) services within the network rather than at Points of Interconnect (POI) at Data Centre sites
• The architecture is managed dynamically through orchestration and automated network management.

Oh - and yes, 5G will provide a much faster network connection for you when you purchase your new 5G handset.

References & Links:

The real 3GPP Architecure - the 5G architecture is highly sophisicated in the capabilities it provides and the technical mechanisms it uses. See here for deeper details

5G Wiki - must have a wikipedia link ;-0

5G at the "G" - Ericsson and Telstra trial 5G at Melbourne Cricket Ground

IMS - Internet Multimedia Sub-System, is a part of overall 3GPP specified wireless communications stack, which amougst other parts includes SIP (Session Initiation Protocol) VoIP componentry

UDM - Universal Data Management provides the subscriber lookup services for SIM / IMSI / Phone Number as per Home Location Register (HLR) / Visitor Location Register (VLR) component of GSM wireless network

SCEF - Service Capabilities Exposure Function (3GPP - Requirements Document - 3GPP TS 33.187 V13.0.0 (2015-09))

Monty Python - How to Rid the World of All Known Diseases (a noble ambition indeed)

Cloud Native Computing Foundation (CNCF) - the new reference architecture for web-scale applications development

"5G for Growth & Survival – Why Telco’s need 5G" - Part 2 of 5G articles this covers the business model

Icon Analogies & Credits:

• Snail - your current 3G, 4G, LTE Architecture has Wireless Access Network terminated at POI, which provides Gateway to Value
• Hare - 5G option for better experience with shorter Path to Value than traditional wireless architecture
• Racing Car - 5G option for low latency applications with short Path to Value for critical functions
• Lightning (MS Powerpoint) - 5G option for latency critical (Ultra Reliable Low Latency Communicatons - URLLC) shortest Path to Value for our favourite 5G enabled use cases including "automonous cars", "control networks" etc

NOTE 1: This business architecture view is informed by discussion with Walter Kozlowski (Telstra Chief NFVi Architect) & Paul Bramich (Ericsson GM Telstra Digital Solutions)

NOTE 2: While author is current employee of Ericsson (Australia / New Zealand), this material (right and wrong) and opinions are the authors

NOTE 3: My reference to Monty Python brought to mind the scale of ambition of Bill & Melinda Gates Foundation, incredible in its (their) humanitarian purpose

Murphy’s law hit again recently on a Melbourne Thursday. Earlier in the day there had been a tram strike. I had a meeting to get to and tram services were only just resuming. My assumption, on leaving the office, was that I would catch a tram to get to my meeting in about 15 minutes.

I was now at one end of the CBD needing to get to a location in middle of town in around 17 minutes. Choices included: wait for tram, catch a taxi or walk fast / run to get to my meeting on time ?

With no trams in sight, I jumped into a taxi, asking for express route straight through town. The driver informed me that the direct route was closed and the detour took much longer and was traffic jammed. So out of taxi and onto "plan C", a fast hike to next major tram intersection. On way I send a message that I might be a bit delayed due to tram strike.

At the intersection I can see trams going in various directions. I ask a service officer, providing assistance to disgruntled passengers, if he knows when the next tram will be heading up Collins St. His response, “… I do not know, but it should be within next 15 to 20 minutes …”.  So wait or hike. Best to hike, as I might be a little late, but it allows me to act now and keeps me heading in the right direction, the alternative puts me at mercy of chance and luck.

On the way I thought about how this situation was analogous to many “Enterprise Architecture vs Agile” debates / discussions, the most recent being on the approach to telemetry within Network Function Virtualization infrastructure (NFVi).

So to NFVi telemetry and architecture.

Telemetry is the underpinning of many machine learning and closed loop automation use cases which aim to significantly reduce cost of operating large telco networks.  The simple view is that you feed the telemetry into a "big data" machine learning algorithm which then makes intelligent automatic remediation decisions, triggering actions such as: scaling up/down resources, moving workloads to alternate machines and taking healing actions by using alternate network paths. This focus on single use case, feeds well into agile delivery pipeline. It uses well understood technical components and the various fault use cases can be knocked off by feeding backlog and coding the actions.

A “higher level” architecture view illustrates that there are a broader set of concerns to be satisfied. Further analysis shows that the data needs for this core set of eight use case use much of the same set of underlying data (massaged and accumulated is various ways).

1. Platform Assurance – the immediate use case for “closed loop assurance”. Provides visibility of operational systems and uses metric and event data as feed to trigger remediation actions
2. Test & Verification – being able to meet non-functional requirements for performance is pretty much the life blood of NFV and performance and bench marking suites are typically fed from same metrics as assurance platforms
3. Platform Orchestration (NFVo) – needs visibility of VNF/CNF stop / start and performance data to be able to make scaling and deployment decisions
4. Capacity Management – requires rolled up summary of resource utilisation which is generated from underlying metrics collection
5. Charging / Licensing – uses consumption and utilisation metrics for charging and VNF/CNF stop / start events for license audits
6. Analytics – metric and events feed network analytics, along with other sources of enrichment data
7. Tenant Orchestration (& other consumers) – in multi-tenanted NFVi scenarios, it is desirable for tenants to get access a sub-set of underlying platform telemetry data
8. Tenant VNF / CNF Telemetry – in addition to platform level telemetry VNF/CNF functions can also generate their own application specific telemetry

Most discussions on NFVi telemetry focus on a single one or two of these use cases (typically #1 plus #3 or #4) in isolation.  What happens in practice is that you have multiple teams putting their own instrumentation into infrastructure, blocked backlogs and work done to satisfy one use case needs to be redone for the other.

The obvious solution is to do a quick strategic architecture brushstroke … a central collection point into which others can subscribe. This can be built on top of: rsyslog, Fluentd, collectd and Apache Kafka technologies providing ease of consumption and integration with real time support. The Central Collector provides data subscription point to all of the other functional uses cases, abstracted across OpenStack and Kubernetes based platforms and it meets the non-functional needs! Problem solved ;-) .

The technical reality is a little different.  OpenStack telemetry is currently (Pike, Queens, Rocky releases) in state of substantial flux and shifting from Ceilometer to collectd based collection. Riemann is used as feedback interface for many Orchestration solutions, including Cloudify. Prometheus time series database is used for collection of metrics within Kubernetes based platform while OpenStack has Gnocchi as its default time series database. Prometheus uses pull model for collection, while collectd & Fluentd are “pushers”. Both collectd and Fluentd are highly configurable both in terms of what data is collected and where it is written to. While Kafka has a great consumption model, many of the available (via GitHub) code sets use direct integration from the collection agent (collectd / fluentd) to the destination target (Prometheus,  Cloudify, Riemann, ZenOSS by way of examples) and so adding Kafka consumer requires writing additional code. Yes the details are messy!

So to the tram question: Wait for the world to align to simple strategic view, start coding a Kafka generic distribution solution or focus on particular functional use case without concern for broader needs ?

This is where Enterprise Architecture and Agile approaches work best if they work together. The value in EA view is that it provides a quick basis for review and validation of first and subsequent steps towards strategic target, while Agile view should define what is immediately actionable.

In context of NFVi telemetry the right decision is totally dependent on priority of particular use cases. Organisation are at varying levels of maturity and have different priorities. In case of an Agile delivery pipeline entering a “wait” state should be avoided.  Again in context of NFVi telemetry, there is usually more than sufficient issues or needs to ensure that delivery momentum can be maintained, even if there will need to be some adjustments on the way.

So did hiking get me on to meeting on time ?

No - I was five minutes late, but by signalling ahead that I would be late I managed to get there five minutes before  the other party. Calling ahead is an example of the virtue of transparency, a key agile tenant.

Would I have got there on time by tram ?

Who knows I might have but by the same token I could have been 30 minutes late and I did get a chance to think about NFVi telemetry while walking.  If I was waiting, I would likely haven been occupied with looking at my phone for time and getting agitated about when the next tram was arriving.

Glossary:

CNF - Cloud (Native) Network Function (a VNF built as a Container app for deployment in Kubernetes)

NFVi - Network Function Virtualisation infrastructure

NFVo - Network Function Virtualisation Orchestration

Telemetry Data - in this context covers both metric (performance) and event (syslog / snmp) data generated by NFVi platform and VNFs / CNFs running within this.

VNF - Virtualised Network Function

References:

ETSI Network Function Virtualisation (NFV) - Architectural Framework

Some Telemetry Technical Stuff:

collectd - Open Source metrics and log collection framework (for OpenStack)

Fluentd - Another Open Source metrics and log collection framework (for Kubernetes)

Gnocchi - Open Source time series database that was developed as part of Open Stack telemetry

Kafka - Open Source Streaming Platform

Kubernetes - Open Source Container Deployment and Orchestration Solution

OpenStack - Open Source Multi-Tenant Infrastructure as a Service

Prometheus - Open Source Time Series and Monitoring system

NOTES 1: Authors "first step" on Open Source telemetry was to get some bugs in collectd resolved to allow deployment on Ubuntu 18.04 & 19.04

NOTES 2: The instrumented bumblebee (picture) from Wikipedia Telemetry page (do bee's do architecture and are they agile...?)

NOTE 3: Look at what a bee can do and think about its size and energy efficiency relative to a "big data ML system", we have a long way to go ;-)

NOTES 4: Tram drivers strike for better pay..

The request for the definition of a "policy on Open Source"  gets frequently passed to the Enterprise Architecture team. So some thought about Policy in general and open source in particular.

• Policy (noun) (from OED)

plans, strategy, proposed action, blueprint, approach, scheme, strategem, programme, schedule, code, system, guidelines, intentions, notions, theory, line, position, stance, attitude

practice, custom, procedure, want, way, tack, routine, matter of course, style, pattern, convention, mode, rule

All sound and typically established with good intentions, though politicians are masters at siting policy as way of evading pointed questions on particular situtions.

The simple truth is that if you want a "policy" to be successful, especially if it is counter to current practice, then you will need to make an investment.

In the context of "Open Source" this means more than just using open source software, it means participating.  This will require investment in time, people and money. If you are going to invest, then you should have some objectives and expectation of the results. So lets lay out some simple expectations and objectives for use of Open Source within corporate IT.

Misconceptions

• Open Source is about saving money on licensing costs

Reality

• Over the life of a software solution licensing might be 10% of TCO, largest cost is on ongoing SI and enhancement (change !!)
• There is no difference in cost of configuration vs. cost of “coding” (and many Open Source developed packages are highly configurable in any case)

So why move to Open Source

• Software development is not a support function, but critical enabler for business
• Open Source projects provide a rich skills pool to drawn on as they are driven by highly skilled and motivated people and attract some of best systems development talent
• Open Source and "free" software (free as per fsf) have visible data schemas, allowing greater data visibility which is essential to achieve agility
• Open Source solutions are mostly driven by operational needs and so are rooted in clear use cases and developed incrementally around these
• Open Source solutions have functionality as good as proprietary solutions
• The community functions as shared interest / collaboration environment, a functional that that previously achieved by vendor product management teams

So

• You and your organistion will benefit by being more responsive and being better at the job than your competitors
• Adoption of Open Source should be focused on people, talent and skills, not saving money on licensing costs

Remember

• Many open source projects have been going for many years and have code and knowledge base that are equivalent to proprietry solutions
• Open Source projects are typically populated by participants who are doing their work as paid professionals and many large ICT companies have made large investments in Open Source for the same reasons outlined above

Sign posts for success

• Is ownership of software function close to business outcome owner ?
• Or conversely how many layers are there from definition of intent to delivery of digital outcome ?
• What is the ratio of evangalists and advisors on agile and open source vs the number of active participants ?
• Are you delegating definition of needs and delivery against these to supplier or are your team active particpants ?
• Are you and your supplier partner both activitely leveraging the same up stream open source projects ?
• Can the team clearly articulate the desired objective and understand how this will be achieved (a sharp focus on particular use case is illustrative of this) ?
• Are you in the Cathedral or the Bazaar :-) ?

Dangers & Risks

Yes there are dangers and risks with Open Source and some of the most commonly articulated are around security and unfettered technology proliferation.

These are valid concerns but the way to counter these is not to stop use of open source. Rather estabish the right life cycle management processes like: automatic and continuous  security scanning and comprehensive automated functional and performance testing.

Apologies for introduction of another digital buzzword towards the end of this note, but DevOps should be used to counter these risks. In bringing IT closer to business  the number one focus of DevOps should be to protect operational systems and ensure that there is on going ownership of the operational outcome by the development team.

Each of: Digital, Open Source adoption and DevOps bring accoutability closer to the owning party.

And back to Policy, in writing one try to reduce the gap from policy to result and ensure that there are clear goals and expectations around the policy.

References:

Policy - Oxford English Dictionary  (transcribed from iPhone Edition ;-) )

Open Source - has its roots in "Free Software Foundation" established by Richard Stallman

10% of TCO - based on authors analysis (as leader of Enterprise Architecture team) of large investment in enterprise systems for large telco over 5 years

The Catheral and the Bazaar - the open source musing on open source by Eric Steven Raymond

NOTE 1: In writing this piece I was wondering if this should be in the "Industry Pages" or the "Just Enough Architecture" pages, as both Open Source and Digital represent trends in "Industry" as well as a in how we approach "Architecture"

Enterprise architecture covers: business, information, applications & infrastructure.  With all this to cover how can we maintain agility. A signficant danger in enterprise architecture is getting caught up in analysis paralysis or being overwhelmed by scope.

Value Chains (as defined by Porter) provide a good way to quickly define scope of activities within an organisation. The structure is clear and easily grasped:

• Core Activies: Inputs + Value Add Activities + Sales + Service
• Supporting Activities: Facilities & Infrastructure + People + Technology + Procurement

While Porter's orginal value chain was concerned with the activities within the organisation, "Industry Level" value chains actually predate these and provide view of broader eco-system.

Within business architecture community a "Capability Model" has mostly equivalent content (in terms of activities identified) though it has a different struture than an Organistional level Value Chain.  Other variations such as value streams, chains and business component models also contain simillar content.

The important thing to remember is that each of these are either organisational or eco-systems models and the model in itself is just a starting point.

The model must have a clear audience and purpose, typically these can be:

• Rapidly communicate scope of business activities across a broad audience
• Provide a framework for comparitive analysis across organisations within the same business segment,
• Provide a framework for quantitiative analysis of business performance (investment allocation, hot spot analysis, maturity analysis, portfolio analysis),
• Provide a framework to capture performance measures, point of interaction and coupling / dependencies across processes and organisational units

There are a number approaches to developing value chains:

• Leverage existing asset - faster, likely more mature but potentially confronting to audience
• Craft one - slow, frustrating to participants, can get lost in the modelling,
• Generate it - as part of data analysis not as input into data analysis

The "Generate it" approach has distinct advantage as it drives toward using the model for analytical purposes, rather than as an end in itself.

If you are intending to do cross organisational analysis then leveraging an existing asset is essential as this is likely to provide a consistency (immutability) across the different organistions required for comparitive analysis.

"Crafting one" runs into danger of only being applicable to current context and not supporting impact of transformation on the organistional structure.

Some advise to practitioners to meet the goals of "Just Enough Architecture"...

1. Don't fall in love with your model ;-) ...

Here are examples of three industry value chains that cover the same industry (Media / Telco):

The important thing about the value chain is that is is mostly right and at this level they generally are. The value in the value chain is the analysis that sits behind it. Adopt one and do minimal updates.

2. Plan to keep the time in developing the first draft of the model short and sharp.

Remember the value chain and associated capability or compoment models are in themselves not the prime objectives. So keep time in developing a first cut (no more than a single "Sprint") to a minimum with clear aims for its target use and attribution.

3. The value of value chain is its use to support quantitiative analysis and fact based decision process. So expect that this is where most of the time will be spent

The collection of data and infomation to establish a fact base, do analysis and render this via the value chain is what takes time and brings the insight. So you need to allow for this. It is also during this time that you can get to establish relationships with component owners and get insight on information. This is essential to get traction on objectives and analysis.

4. Iterative Elaboration is key to doing "Just Enough". It is better to get first cut in place and then iterate to bring in addtional details and further attribution for next level of detailed analysis. Typically this iterative elaboration is also used to develope a more complete "capability" or "component" view of the business landscape.

This is all simple and sound advise, but to often Enterprise Architecture looses sight of goals and gets mired in obsession around details, modelling concerns and internal bluster.

Just Enough Architecture requires that architecture is grounded in providing valuable input into decision making process and helping in a timely manner.

A goal of data driven organistion is establish a process that is able to collect the analytical data in repeatable manner and re-render the analysis against this on an ongoing basis. Right now most organistional still rely on very labour intensive collection and analysis process using spreadsheets.

References:

Value Chains - originally defined by Michael Porter - "Competitive Advantage". 1985, Ch. 1, pp 11-15. The Free Press. New York.

Value Chains, Value Stream & Value Nets - Componentisation, service orientation, disaggregation & platforms drive dis-intermediation based models which view eco-systems as set of interacting and collaborating parts, which can be wired together in many ways.

Capability Model  & Planning - TOGAF defines capabilities across mutiple layers and recognises that the creation of capabilies is a process of evolution with increments and planning

Component Business Model - A modelling technique defined by PWC/IBM consultant Guy Rackham,  which views business as a set of components, which encapsulate: people, process & technology and expose service interfaces to support their interaction

NOTE 1: See Industry Pages - for an example analysis use in Media / Communications..

or John's Enterprise Architecture == JEA.

Provides structures, stories and guidance for Enterprise Architects and Digital Strategist.

Its "Just Enough Architecture" as in an Agile world it is critial that just the right amount of Enterprise Architcture is done...

• Not to much - as this results in bottle necks and potential irrelevance
• Not to little - as this is likely to result in quality and cost issues

Just Enough Architecture will cover the following areas:

• Context - where does Enterprise Architecture fit
• Value Proposition - what is the purpose and value of Enterprise Architectue
• Existenstial Angst - why is it that architects worry about their relevance and particuarly as everyone and thing is moving to agile models...
• Agile and Enterprise Architecture - how does Enterprise Architecture work in an Agile delivery world where we have Agile Everything
• Enterprise Architecture Metaphor - building cities and culture not the Tower of Babel
• Real World Examples and Learnings - industry insights and examples and lessons learnt

That is opening gambit for the Focus of this Blog, in addition there are assocated Blogs that cover:

• Industry - Industry models and examples
• Technology Tips - in addition to writing the Blog we will also write a Blog of technical tips on getting this site up and running

The objective is 1 or 2 posts a month and no more.