ONVIF Web Services Client Consumption with Java
A beginners guide to using (consuming) HTTP/HTTPS exposed APIs, including Web Services defined by WSDL, Restful HTTP APIs and others. Consumption is via Java and JavaScript for ONVIF defined Services
Status: Feb 2023 - Have tools and example "ONVIF" test device (server) going and corresponding ONVIF Java client working and simple RESTful Web Service to SOAP Web Services relay working. Code ready for use and refinement in the wild...
Status: Mar 2024 - Updated with ONVIF ws-discovery details (onvif-relay open source also expanded) and reference to ONVIF Specification pull requests.
Motivation is to "talk with" ONVIF compliant devices, that typically sit in private and non accessible networks (for very good security concern reasons).
Scope covered here is:
- "Web Services" based APIs - APIs which mostly use HTTP/HTTPS transport with SOAP/XML/JSON packaging and HTTP/HTTP based Restful Services.
- Defined by any of: "Web Services Description Language" (WSDL) or HTTP/S URI with defined Paramaters and/or Body Payload
- Client Consumption - this blog covers creating and exposing "Web Services" (to test against) and consuming existing services
- Example Server for test device emulation & ONVIF relay - expoose some ONVIF Web Service to test against and implement HTTP/HTTP JSON relay to ONVIF SOAP Web Service device
- Language - Java and JavaScript clients are only languages covered for client application
- Development Environment - Ubuntu Linux, Eclipse IDE, Maven & Command Line Tools
NOTE: To see Thrift API consumption with JavaScript have a look at blog "Apache Thrift with JavaScript".
ONVIF Development Strategy - Path To Success
ONVIF is a particlar application of SOAP based Web Services. It is unusual in continuing so use a SOAP based API. While most current APIs have pivoted to RESTful based or binary based Micro Services, ONVIF is both SOAP based and used to support an "Internet of Things" service. Where the "things" are security devices. So in ONVIF the devices that you need to "talk" with are providers of SOAP Based Web Services (as illustrated).
This is an atypical model, as you have more devices (servers) than clients (consumers). This is the inverse of the typical case, where you have a small number of central services supporting lots of clients. As the current specification and all devices out in field are SOAP based, you do not have option of alternate communications mechanism. So you are caught with need to use a "dying" protocol, that was not really defined with IoT architecture in mind.
The typical toolset SOAP Web Services is Eclipse IDE with Java Enterprise Edition (JEE) Web Development tools. Having "played" around with this more than I wished, my tips to success are:
- Don't expect Eclipse or other UI to succefully hide the details - I started using GUI tools (Eclipse and JetBrains "IntelliJ IDEA") to try work avoid details. This did not work...
- Do learn about Apache Maven - as the tooling has been under substantial flux (shift from Java Platform Enterprise Edition to Jakarta Enterprise Edition), you really need to understand how Maven works, as you cannot realistically run the required Java code generation tools directly (the class paths in the command line are rediculously long)
- Select your approach based on clear understanding of: Java Platform Enterprise Edition vs Jakarta Enterprise Edition - be aware that there is only "JSE - JDK" (Java Standard Edition - Java Development Kit) now, as all the Enterprise Edition features have been removed with JSE - JDK 11.
- Start with the bare minimum - Maven "archetypes" are supposed to make things simpler by codifying "best" practice. I found that they just obscure what is important. Better to start very bare bone. The "Best Practice" is really just a codification of "directory structure" and "namespace" choices. Most of this is handled via "plugin" and as per "Do learn about Apache Maven" once you understand this its really pretty simple to manage project configuration via the Maven "pom.xml" "<configuration>" options.
- Go to source materials - due to the signifant flux with Jakarta EE and with SOAP based Web Servcies is now a bit of a "back water", the information out there to be found via search is likely to be out of date, wrong or confusing.
Caution
- Maven is not "transparent" - unlike make or ant, maven and its "plugin" architecture is a "black box" model. With make or ant all behviour is visible as it is defined by the dependencies model and the way dependencies are satisfied is by executing explicity visible rules. With maven the dependencies define the tooling/rutime needs, while the code dependencies and the build life-cycle are encoded in compiled plugin. Hence you need to understand Maven and what it is doing and many plugins define new "targets" and Maven does not provide way to "introspect" its targets ...
Conclusion
- Creating working Java ONVIF is more complicated than it should be due to combination of errors resulting from unusable ONVIF WSDL, complexity and inflexibility (basic control of naming, single Service per WSDL) with the JAX-WS architecture and related Tools and shift from Java EE / J2EE to Jakarta EE.
The Java Programmatic Approach
On Ubuntu Linux there are two main options available to convert WSDL to Java code:
- wsimport - was part of Java EE and is now part of Jakarta EE. From Java EE to Jakarta EE the location of the wsimport class has changed over releases. So you need to ensure that the Maven plugin and the corresponding dependencies are aligned, for it to correcly pick up executable. Using the Java EE version of maven plugin will result in generation of code using the older "jaxb.xml" package (uses "javax" namespace). Using the later Jakarta EE version will create newer "jakarta.xml" packages (using "jakarta" namespace). These packages are used for the Java XML Binding (JAXB) annotations and runtime environment.
- wsdl2java - which is part of Apache Axis/CFX framework. This appears to only support older "javax" JAXB model. So if you opt to use "javax" based solution then this is an option. I have found that the wsdl2java sometimes generates easier to interpret errors. There is also a corrsponding java2js (JavaScript) compiler.
For my example I used and tested with both initially. I made slow but steady progress using wsimport (selected due to it supporting current Jakarta EE and prior Java EE models), but then changed to CXF and its WSDL2Java tool, due to a number of issues with the metro/wsimport tooling/implmentation.
With WSDL SOAP Web Services there are lots of moving parts and configuring Eclipse IDE to work with the tools is not a simple "point and click" exercise. I elected to test this with a new VM using only Maven and command line tools. This is because all of the documentation refers to Maven plugins and dependencies and the current Eclipse Enterprise Web Development uses CXF based generation. The Eclipse/CXF framework did not work auto-magically, and at this start of this project (Nov/Dec 2022) it did not support Jakarta EE.
NOTE: CXF Release 4.0.x now has Jakarta support (late Dec 2022)
The result was that I only used Eclipse IDE once I had aready got WSDL to Java generation out of way and wanted to test/debug ONVIF device server and client software.
Java Web Services and Annotations
Both wsimport and wsdl2java rely on generation of Java code with Web Service annotations to which in-turn drive generation of SOAP / XML messages. The WSDL compiler converts the WSDL defined: types, portTypes, bindings, operations and services into Java classes with annotations. These classes can then be serialised using Java to XML and XML to Java "marshalling" and "unmarshalling" methods. The generation mappings from WSDL to Java are outlined in the following diagram:
In summary:
- All types - generate Java annotated Classes
- Request / Response Types - generate Java annoted Classes
- All Message Types - Response message Java response annotated Classes, Request message Java request annotated Classes
- All Operation Types - Java Operation annotated Classes, which reference the Request / Response types
- Operations - defined as method on portType Service Object
- Service - defined via <wsdl:service name="<SERVICE_NAME>>" directive, specifies <port name="<PORT_NAME>" ... >, resulting in interface definition with name <PORT_NAME> (based on the WSDL portType name "<portType name=<PORT_NAME ..>") and a client implmentation class with name <SERIVCE_NAME> that "extends" interface class <PORT_NAME> (see Appendix A. for specific example).
The result is that all SOAP / XML is generated via the annotated Java Classes and there are no "xml" templates or XML text processing being used within framework. This should ensure services are "correct" and reduces the amount of hand written code.
The following section provides more details on the WSDL definition and wsimport compiler generation.
IoT & Compiler Client Stub Generation
WSDL (Web Service Description Language) based client "stub" generation should be the simplest thing. Just point code generator to interface definition (WSDL file) and bingo! In the case of ONVIF this is not the case.
This is combination of the nature of SOAP Web Services amd ONVIF flow through. WSDL is based on XML Schema. As with most things w3c.org related, WSDL and XML Schema are overly complicated and suffers from some really bad fundemental problems.
In context of IoT, at the very bottom of the WSDL service definition is "<wsdl:port name=<NAME> binding=<REF> >", includes the service address (such as "<soap:address location="https://i.am.here/getme"/>. The implication of this is that you need to have seperate WSDL specification of very instance of the WSDL defined service "<wsdl:service name=<NAME >". In IoT world this is problematic, as you have thousand of instances the service (each device == service instance) and unlike "big" enterprise services the IoT devices can some and go and will always have dynamically allocated IP addresses. Seperating service definition from service locations (addressing) is fundemental requirement for IoT.
Generating client / server from WSDL is known as "top down" approach for Web Service. The alternate "bottom up" approach is to write Java code with annotations and then generate WSDL from the annoated Java code.
As ONVIF devices need to confirm to defined WSDL, the "top down" approach is used. Then to allow same client to connect to different devices, the client proxy needs to change the target device programmatically (see client code example below).
WSDL Namespaces and Schema
The WSDL complications: namespaces and schema. We all know and love namespaces. These provides a way to avoid naming clashes and are availble in Java, C++ and most modern lanaguages. A typical convention is to have use see IP domain names to help manage this and ensure identifiers are "globally" unique (i.e "au.com.graphica.utils.MyCoolLib" & "java.io.File"). The problem with WSDL is that there are many namepaces required to complete even simple definitions:
- "wsdl" - for the wsdl schema itself, which is defined using "XML Schema" so needs its own explicit wsdl namespace,
- "xs" - for XML Schema name space which is the schema for defining the WSDL schema and
- "soap" - as WSDL used SOAP messaging as its origial "on wire" packaging format.
WARNING !!! - Soapbox Diversion
This exposure of the underlying "meta-layer" of definition to the developer is one of the reasons that XML and XML Schema based notation is so verbose and unfriendly to reader. A simple example of this is using of BNF (Backus-Naur Form) notation or lex/yacc to define a grammar easily. Imagine if writing a bit of Java code if I had to qualify all my Java statements with a name qualifier:
//
// exposing the meta ...
//
java:int i;
java:if (tns:i = 0; tns:i > 5; tns:i++){ ... }Gosh this is starting to look like XSLT programming ... where you end up writing code which is all via tagged / namespace XML!! I was forced to do this to create XML Binding Customisation script generator (see Appendix "E. Java Naming Schema (lower case first letter) applied to WSDL PortType/Operation names").
Add to this the fact that WSDL is built on top of many layers of "assets", exposing further additional namespaces. Really the only reason that SOAP came into existance in the first place was to get around the fact that http/https went through firewalls easily and CORBA, DCE RPC and ONC/RPC did not... and that Microsoft wanted to ignore and reinvent all the existing interprocess comminications mechanisms under the umbrealla of COM/COM+/DCOM and ultimately .NET).
The result was, we forced humans to adapt to the needs of the machine. This is the antithesis of language design and compiler wirting, where the objective has always been to try to define ways of specifying things that is easy for people and using compiler / interpreters to convert this to something that a machine can understand.
NOTE: I will move these comments later and stick to technical stuff here ;-)
WARNING !!! - End of Soapbox Diversion
With namespace and names, there is basically two things that are being done:
- defining - a name or identifier (WSDL "name=")
- referring - to a name or identifier (WSDL "elementType=")
With WDSL the "definition" is done via a specific "element", informally: "<wsdl:definition name=<DEFINED NAME> ... >", "<wsdl:portType name=<ANOTHER DEFINED NAME> ... >", "<wsdl:service name=<SERVICE DEFINED NAME> ...>"
The referencing of a defined name (identifier), can be done using a "local" (or "NCName == non-canonicalised name) or a "Qualifed Name" (QName). The QName consist of (again informally): <NAME SPACE NAME:<LOCAL NAME>. Examples from ONVIF device specification:
<wsdl:portType name="Device"> <<=== Define
<wsdl:operation name="GetServices"> <<=== Define
<wsdl:documentation>Returns information about services on the device.</wsdl:documentation>
<wsdl:input message="tds:GetServicesRequest"/> <<== Reference
<wsdl:output message="tds:GetServicesResponse"/> <<== Reference
</wsdl:operation>So to use the a WSDL or ONVIF schema you need to understand:
- How you can load and use existing namespace definition from within enclosing WSDL document and
- When something is being defined, what name space will the definition be in, so you can correctly refer to it
The set of namespaces available within an WSDL definition:
- is defined in the initial definition section ("<wsdl:definition>")
- the "targetNameSpace" which is the WSDL "this" equivalent. This is where all the name definitions within your WSDL will be put (note a given name is only required to be unique within a given "namespace" and "element type". So you can have the same name being used for WSDL "<binding name=<SAME> ...>" as for "<port name=<SAME> ....>".
This (Oracle hosted) diagram is helpful to undestand the WSDL Structure:
To define and control the namespace and schema in WDSL the following language features are used:
- "xmlns=uri" - the default (where xmlns = XML Namespace) namespace for this definition, so non-qualified definitions will be placed in this namespace. If not specified then this will be: "xmlns="http://www.w3.org/2000/10/XMLSchema"
- "xmlns:tag=uri" - defines a new namespace which can be referred to be abbreviated "tag" or full uri
- targetNameSpace = this is used with this definition or with "<import. ..>" to define the namespace that the imported schema definitions will be put into (for Java generated from WSDL this also defined the Java package namespace)
- "<wsdl:import ...>" = imports WSDL definitions from existing wsdl file, allow you to build bigger wsdl from exisiting wsdl (like the: "http://www.onvif.org/ver10/device/wsdl/devicemgmt.wsdl" wsdl, for example)
- "<ws:schema ...>" = has further "<import ...>" which can be used to import XML Schema type definitions
In WSDL 2.0 (which was essentially still born, as XML madness started to become apparent) portType (equivalent to object oriented "class" concept) becomes "interface" while bindings (equivalent to object oriented "instance" concept) remains same.
Specifications vs. Implementations
With Java there are specifications and implementation. For each of the Java Specifications that is a "Reference Implementation" (RI), but there are also other available implemetations.
For Web Services the key "specifications" are:
- Java Architecture for XML Binding (JAXB) - which defines mapping between XML and Java. This has supporting set of annotations, which historically were in "javax.*" namespace and now in "jakarta.*" namespace.
- Jakarta (formally Java) API for XML Web Services (JAX-WS) - which defines apis for implementating XML Web Services in Java. Originally defined as part of Java Enterprise Edition (JEE) and now transitioned to Jakarta EE. This provide a large number of related specifications for WSDL, SOAP, Mail etc and used JAXB for mapping to/from WSDL and Java.
And corresponding Implementations:
For JAXB it seems that the "Reference Implementation" is used universally.
For JAX-WS there are many different implementations out there. Most of these are based on historical "Java API for XML Web Services" and some have been updated to reflect current "Jakarta API for XML Web Services". Here are the main ones that you will find references to:
Metro - this was the original JAX-WS "Reference Implementation" where JAX == Java EE and was developed as part of GlassFish Application Server Project (hence close relationship between Glassfish and "Reference Implementations". The current Jakarta reference implementation is based on metro.
Axis/Axis2 - according to Wikipedia "AXIS" == Apache eXtensible Interaction System. This is an Apache implementation of JAX-WS. Axis2 is a re-disign/re-write of this. According the Axis2 web page: "Apache Axis2 is more efficient, more modular and more XML-oriented / JSON-orientated than the older version. It is carefully designed to support the easy addition of plug-in "modules" that extend their functionality for features such as security and reliability.". So more of everything .... but no Jakarta Java generation in WSDL2Java. The Axis framework includes support for languages other than Java (C/C++ ??).
CXF - Apache CXF is another Apache community project, based on IONA Technologies (the original CORBA company...) assets. This include the WSDL2Java and WSDL2JS (JavaScript) tools. With the release of CXF 4.0.0 this now has resolved the Jakarta code generation issues (Dec 22). I initially started with Metro, but ended up using CXF 4.0.x (Jakarta release) for client & server sides as it was trival to add Digest Authentication to code. To get XML Binding customisation working, I had to add soft link to have consistent file name / wsdlLocation names. This link was not needed with Metro wsimport.
GlassFish, Apache Tomcat (& TomEE), IBM WebSphere, Redhat JBOSS (& its Wildfly variation) are all Java Application Servers that support JAX-WS. These are implementations of the JAX-WS specifications. I try to avoid Java Application Servers, preferring to use embedded server a'la Jetty. Of the Application Servers, Glassfish tracks the Jakarta specification mostly closely as it it provides the "reference implementation". So GlassFish is most current relative to Jakarta. Jetty Version 11 is also Jakarta current (see below for my trival introduction to embedded Jetty), so I was able to create embedded server implementation with Jetty.
Spring - not sure where Spring fits here and there appears to be more than enough complications and variations out there already ;-)
NOTE: javax == Java Extensions
Maven Tools
I found it easier to start with Maven. Only once I had working Maven WSDL compile going via command line, did I configure Eclipse. I used Eclipse to write rquired code, not to run the code generation tools.
Maven defines a default life-cycle: validate, initialize, main/test:{generate-sources, process-sources, generate-resources, process-resources, compile, process-classes}, test, prepare, integrate, verify, install, deploy
The various Maven plugins, operate for some part of the standard life-cycle and operate on assets that are created (generated) or maintained within its standard directory tree.
The project is configured using the pom.xml (POM == Project Object Model) file.
For this project I used Maven POM to define directory structure and project directory, with softlink from /META-INF into the development tree location of the generated JAX-WS code to make WSDL naming consistent across jar and file-system (for the CXF WSDL2Java compiler). In testing across javax, jakarta, metro and cxf variations I had to break the single tree into Maven modules to make it easy to test the variations:
---
--- ONVIF project structure
---
onvif-relay
|->onvif-api <<== generated code only via compilers | |->java <<=== java code (empty) | |->patch <<=== patch directives | |->xml <<=== xslt xml binding customerisation scripts | |->sh <<=== utility script to download and patch wsdl | |->files.txt <<== wsdl src files to get and patch | |->onvif-cxf-api
| | |->src
| | | |->main
| | | | |->java <<=== empty | |->resources <<=== resource files (wsdl source) | |->META-INF <<=== place for property resouce files | ^ |->wsdl
| | | | | |->www.onvif.org <<=== onvif wsdl xsd (patched) files | |->vers10
| | | | | | |->device
| | | | | | | |->wsdl
| | | | | | | |->onvif_device.wsdl <<=== device service def | |->media
| | | | | | |->wsdl
| | | | | | |->onvif_media.wsdl <<== media service def | |->www.w3.org <<=== w3 xsd for soap | |-----------------------------------| meta-inf <<="=" link |->test
| | | |->java <<=== java test onvif device server | |->target <<=== generated files | |->generated-sources
| | |->cxf <<=== generated java from wsdl2java compiler | |->java
| | |->wget <<=== holding bay for wsdl xsd prior to patching | |->xsltproc <<=== xml binding customerisation scripts | |->onvif-metro-api
| |->src
| |-target
| |->generated-sources
| |->wsimport <<=== generated java from wsdl2java compiler | |->wget
| |->xsltproc
|
|->onvif-device
| |->src
| |->onvif-jak-device
| | |->src
| | |->target
| |
| |->onvif-jax-device
| |->src
| |->target
|
|->onvif-client
| |->src
| |->onvif-jak-client
| | |->src
| | |->target
| |
| |->onvif-jax-client
| |->src
| |->target
|
|->onvif-cxf-relay <<== cxf relay implemenation |->src
|->target
---
--- Other related projects include:
---
metro-jax-ws <<== jax-ws ri project (reference implemetation) |->jaxws-ri
| |->tools
| |->wscompile <<== wsdl compiler and related tools | wscompile <<="=" extract from jax-ws ri with eclipse usable pom --- ---< code>As detailed in appendices below the ONVIF WSDL and XSD files could not be compiled directly from http sources. Rather these had to be downloaded (via wget) and then patched.
This was done using shell script: "get-and-patch.sh", taking as arguments: the file list (with placement directives), download directory, target directory and patch directory.
To build ONVIF Java server stubs and client, the process was to:
- Run "get-and-patch.sh" script - I run this manually, as getting "maven ant" plugin to do this automatically is still on "to-do" list
- Run wsimport (or wsdl2java for cxf) - via Maven with target: "mvn -X jaxws:wsimport" (or "mvn -X generate-sources", for cxf)
- Build programs via Eclipse ...
NOTE: I split out maven modules to allow generation of cxf (wsdl2java) or metro (wsimport) versions of the onvif-api libraray (all generated code) and also seperated "jak" (Jakarta EE) and "jax" (Java EE) maven pom.xml files to control:
- wsimport plugin ("com.sun.xml.ws:jaxws-maven-plugin") options - with different version generating either "javax.*" (version 2.3.5) or "jakarta.*" (version 3.0.2) web service annotations. You need to set the "extensions" flag to true as the ONVIF WSDL uses SOAP 1.2
- javax or jakarta namespace option - the transition from Java EE to Jakarta EE means you include dependencies specific to your wsimport/cxf version choice. I started with javax namespace (having had problems with SOAP 1.2 support with Jakarta libraries), but then transitioned main branch to jakarta.
- Jetty Web Server version option - again depending on whether you are using javax or jakarta you need to select the corresponding Jetty version to provide an embedded http server: version 10 for javax and version 11 for jakarta.
In the various Maven pom.xml files you will see that I have commented out (via XML comments <!– -->) the "not used" option. By default the repository will create cxf / jakarta implementation. It is easy to change across cxf /metro by editing pom.xml and jak/jax by building alterate pom trees.
During testing I needed to create jars within the "onvif-relay" and make these available to "hacked" JAX-WS Metro RI. This required being able to load dependencies via Maven pom.xml. Here is example script to load local jars into sharable maven dependency repository:
cat src/main/sh/deploy-wsimport-3.0.2.jar.sh
#!/bin/s
groupid=onvif-relay
artifactid=onvif-naming-jaxb-tools
version=3.0.2
packaging=jar
jarpath=../metro-jax-ws/jaxws-ri/bundles/jaxws-tools/target/jaxws-tools-3.0.2.jar
mvn -X install:install-file -Dfile=${jarpath} -DgroupId=${groupid} -DartifactId=${artifactid} -Dversion=${version} -Dpackaging=${packaging} -DgeneratePom=trueThis and other helper script are in "onvif-relay" repository.
NOTE: Having Maven packaging target as "jar", "war" & "ear" and running "mvn install" should automatically build jars and install them into your Maven repository.
Eclipse Tools
I did not use any "Eclipse Java Enterprise" tools to generate java code from WSDL (as per above "Maven Tools" section). The default Eclipse tooling is based on CXF, rather than the Metro JAX-WS RI.,
I used Eclipse IDE to: build client, test ONVIF device server and run WsImport in debugger. To run WsImport in debugger you need to do Eclipse Project Import Existing Maven project and pointing this to the existing Maven "onvif-relay" repository.
NOTE: I have not included the Eclipse set up in the github repository, as this is mostly trivially established by using doing import of Maven project into Eclipse. However there is an issue in m2e (Maven 2 Eclipse) with complex projects.
Embedded Jetty
Eclipse Jetty was one of first web/application servers that let you use it as an "embedded server". That this means that is that you build your appplication uses Jetty APIs to directly instantiate the Java services without need to package these into Jars/EAR/WAR files for deployment into the Application server.
This makes development and deployment simpler as the entire application and its web server became self contained. Glass wish is also available to to be embedded, but currently its APIs provides wrappers around existing War deployment mechanisms and configuration via property files, rather than direct API calls.
This embedded development model allows you to build Java web application, much more like you would with Node JS and Javascript (see my blog: https://just.graphica.com.au/web-architecture/ for architecture view of JavaScript development).
To get your head around Embedded Jetty you need to understand its basic abstraction, which is that pretty much everything is Jetty is derived from "org.eclipse.jetty.server.Handler". What Jetty provides and a library of Handler classes which can be assembled build your server. The Handler types include:
- ConnectHandler - to handle initial http connection
- HandlerCollection - to allow a request to passed to multiple handlers
- HandlerList - to allow request to be passed sequentially through a list of handles until one flags that it handled the request
- WrapperHandlers - which allow you to wrap other http web API like HttpServlets and so deploy these in Jetty
- ProxyHandlers - to allow redirect and forwarding of http requests
- GZipHandler - to handle compressed http request/response payloads prior/after processing by regular handler
- and lots of others - see Jetty JavaDoc and Programmers Guide.
The test server used for creating the ONVIF test device is based on boiler plate embedded Jetty, which has additional handlers and handler delegation to provide "ONVIF Facade" and the individual ONVIF service implementations.
So Embedded Jetty development == Handler assembly.
Java Introspection / Reflection or JAX-WS Dispatch interface ?
The driver for use of an "onvif relay" could be:
- Devices are in private network that is not accessible to consuming client, so need to have proxy at network edge that act as go between,
- Need to easily request and extract data from the devices to include within a larger network inventory solution. This is typical case for any higher level exposure/management solution, which uses combination of OSS and device discovery data,
- Need to manage device security/visibility by having relay being responsible for device authentication/authourisation across a large network of devices and not have this being managed by consuming service,
- To allow separation of device management aspects (set time, add/remove uses, update firmware etc) of ONVIF from device consumption (get available video streams, consume video stream, Pan/Tilt/Zoom control etc).
- Due to have set of consuming client applications (and developers) that are already using Restful Web Service as preferred technical approach to expose / consume services.
Each of these requirements has to be satisfied in my target use case. Architecturally and technically there are multiple ways to create an "onvif-relay", including:
- HTTP Proxy - as ONVIF uses SOAP Web Services could just use straight HTTP proxy which just takes http/s requests proxy request and forwards to target destination
- SOAP Body - take SOAP body request and target destination and then wrap with SOAP Header and forward to ONVIF device then extract and return response SOAP Body
- RESTFul / JSON - expose ONVIF as Restful / JSON request with target destination and forward to ONVIF by repackaging as ONVIF SOAP request and then extracting results and serialising as JSON for response.
As I needed to be extract and use data via relay (2) and provided interface that was much easier to consume (5), the "onvif-relay" could not just work via HTTP / SOAP payload level. So the relay must to be able to provide;
- Semantic exposure and
- syntactic conversion from SOAP Web Service to Restful Web Service
So choice where to:
- Handwrite relay conversion for all wsdl operation (there are 177 of these just for the devicemgmt.wsdl and media.wsdl alone),
- Write compiler that generated the above 177 methods,
- Use JAX-WS dispatch approach to invoke and interpret the ONVIF requests or
- Use Java reflection/introspection to interpret the ONVIF schema as Java code and invoke and interpret from this.
I elected to use Java reflection/introspection as:
- Using Java provided way to trivally convert across SOAP/XML and JSON representation to serialising/deserialistion from Java to JSON using GSON library (or Jackson),
- Avoided need to ever directly inteface with/use SOAP/XML as this was always hidden behind the generated Java service class and all communications was managed via JAX-WS framework
- Jakarta Dispatch API is specific to JAX-WS (and poorly documented), while Java reflection/reflection is a general purpose technique and much better documented than Dispatch.
So using Java Reflection/Introspection turned problem from writing / generating 177 functions to creating a general interpretor that is readily extendable to handle other ONVIF wsdl defintions with no modification to code.
The result was a small core set of modules:
- Invokers (to handle invocation of Java services via SEI (Service Endpoint Interface) and
- Converters to handle serialisation to from JSON
Here is sample ONVIF operation via Java reflection code:
/**
@what Invoke ONVIF WSDL methods using Java reflection to allow automatiic JAX-WS <-> Restful relay
@note - this is potentially buggy as it assumes that the introspective "getFields"
returns with order consist with method arguments list.
Likely needs to be revisited to get info from annotations or
compile with -parameters flag to ensure paramter names get included in image
*/
package onvif_relay.relay.invokers;
import java.lang.reflect.Field;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import org.apache.cxf.configuration.security.AuthorizationPolicy;
import org.apache.cxf.endpoint.Client;
import org.apache.cxf.frontend.ClientProxy;
import org.apache.cxf.transport.http.HTTPConduit;
import org.apache.cxf.ws.security.wss4j.WSS4JOutInterceptor;
import org.apache.wss4j.dom.handler.WSHandlerConstants;
import org.onvif.ver10.device.wsdl.Device;
import org.onvif.ver10.device.wsdl.DeviceService;
import org.onvif.ver10.media.wsdl.Media;
import org.onvif.ver10.media.wsdl.MediaService;
import jakarta.xml.ws.Binding;
import jakarta.xml.ws.BindingProvider;
import jakarta.xml.ws.Holder;
import jakarta.xml.ws.handler.Handler;
import jakarta.xml.ws.soap.SOAPBinding;
import onvif_relay.relay.converters.JsonRequestResponse;
import onvif_relay.relay.converters.OnvifOperations;
public class InvokeOperation {
static Class<?>[] emptyArgs = {};
static Object[] emptyParams = {};
Object invokeDevice(JsonRequestResponse target, boolean doClassify, Map<String, String> ctrl) {
Object res = null;
DeviceService dserv = new DeviceService();
Device sei = dserv.getDevicePort();
Object[] useMethod = discoverMethod(sei, target);
if (useMethod != null) {
if (setupService(sei, target.target, target.user, target.password, ctrl)) {
try {
res = invokeMethod(sei, useMethod, target);
if (doClassify) {
String[] cs = classify(target, useMethod);
target.operationType = cs[0];
target.voidOperation = cs[1];
}
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
return res;
}
Object invokeMedia(JsonRequestResponse target, boolean doClassify, Map<String, String> ctrl) {
Object res = null;
MediaService dserv = new MediaService();
Media sei = dserv.getMediaPort();
Object[] useMethod = discoverMethod(sei, target);
if (useMethod != null) {
if (setupService(sei, target.target, target.user, target.password, ctrl)) {
try {
res = invokeMethod(sei, useMethod, target);
if (doClassify) {
String[] cs = classify(target, useMethod);
target.operationType = cs[0];
target.voidOperation = cs[1];
}
} catch (Exception ex) {
ex.printStackTrace();
}
}
}
return res;
}
public Object invoke(JsonRequestResponse targetRequest, boolean doClassify, Map<String, String> ctrl) {
Object res = null;
String reqType = targetRequest.request.getClass().getPackageName();
switch (reqType) {
case OnvifOperations.DeviceType: res = invokeDevice(targetRequest, doClassify, ctrl);
break;
case OnvifOperations.MediaType: res = invokeMedia(targetRequest, doClassify, ctrl);
break;
}
return res;
}
Object[] discoverMethod(Object sei, JsonRequestResponse target) {
/* Can have:
* - inputs via request parameters with return via single object (GetStreamUri) or
* - empty request returning multiple items, via object reference args (GetDeviceInformation) or
* - empty request returning single object via method return
*
* Assume: Request Object and Field return correct order...
*
*/
Object[] res = null;
Object[] strategy = new Object[]{"none", -1, -1, -1};
List<Class<?>> params = new ArrayList<>();
Class<?>[] plist = null;
Field[] opFields = target.request.getClass().getDeclaredFields();
if (opFields.length > 0) {
strategy[0] = "request";
strategy[1] = opFields.length;
for (int i = 0; i < opFields.length; i++) {
params.add(opFields[i].getType());
}
} else {
strategy[1] = 0;
if (target.response instanceof Class) {
Class respType = (Class)target.response;
opFields = respType.getDeclaredFields();
strategy[2] = opFields.length;
for (int i = 0; i < opFields.length; i++) {
params.add(opFields[i].getType());
}
switch (opFields.length) {
case 0: strategy[0] = "empty";
strategy[3] = 0;
break;
case 1: strategy[0] = "empty-with-return";
strategy[3] = 1;
break;
default: strategy[0] = "response";
strategy[3] = 0;
}
}
}
try {
plist = new Class<?>[params.size()];
if (strategy[0].equals("response")) {
for (int i = 0; i < params.size(); i++) {
// NOTE: template info is not maintained within VM runtime
// all objects wrapped via Holder<t> -> Holder<Object>
plist[i] = Holder.class;
}
} else if (((String)strategy[0]).substring(0,5).equals("empty")) {
plist = emptyArgs;
} else {
for (int i = 0; i < params.size(); i++) {
plist[i] = params.get(i);
}
}
Method op = sei.getClass().getDeclaredMethod(target.reqclass, plist);
res = new Object[]{ op, plist, strategy};
} catch (Exception ex) {
ex.printStackTrace();
}
return res;
}
String[] classify(JsonRequestResponse target, Object[] method) {
String [] cs = new String[] { "action", "false" };
Object[] strategy = (Object[])method[2];
String prefix = target.reqclass.substring(0, 3).toLowerCase();
switch (prefix) {
case "get":
case "set": cs[0] = prefix;
break;
}
if ((int)strategy[3] == 0) {
cs[1] = "true";
}
return cs;
}
Object wrapResponse(JsonRequestResponse target, Object[] useMethod, Object got) throws Exception {
Object res = null;
Class<?> cs = null;
if (target.response instanceof Class<?>) {
cs = (Class<?>)target.response;
} else if (target.response != null) {
cs = target.response.getClass();
}
Object respo = cs.getDeclaredConstructor().newInstance();
if (respo != null) {
Field[] outData = respo.getClass().getDeclaredFields();
Class<?>[] setArg = new Class<?>[1];
String nm = null, setMethod = null, getMethod = null;
Method setm = null, getm = null, addm = null;
switch (outData.length) {
case 0: break;
case 1: nm = outData[0].getName();
setMethod = "set" + nm.substring(0,1).toUpperCase() + nm.substring(1);
setArg[0] = got.getClass();
try {
setm = respo.getClass().getMethod(setMethod, setArg);
setm.invoke(respo, got);
} catch (Exception ex) {
// System.out.println(ex);
System.out.println("INFO>> InvokeOperation::WrapResponse: no set, recovering using get+add.");
getMethod = "get" + nm.substring(0,1).toUpperCase() + nm.substring(1);
getm = respo.getClass().getMethod(getMethod, null);
List<?> tlist = (List<?>)getm.invoke(respo, null);
List<?> slist = (List<?>)got;
if (slist.size() > 0) {
// setArg[0] = slist.get(0).getClass();
setArg[0] = Object.class;
addm = tlist.getClass().getMethod("add", setArg);
// tlist.addAll((Collection<?>) slist);
for (int i = 0; i < slist.size(); i++) {
addm.invoke(tlist, slist.get(i));
}
}
}
break;
default: System.out.println("ERR>> InvokeOperation::wrapResponse - multiple fields: " + respo.getClass().getCanonicalName());
}
res = respo;
}
return res;
}
Object getResponseArgs(JsonRequestResponse target, Object[] useMethod, Object[] args) throws Exception {
Object res = null;
Class<?> cs = null;
if (target.response instanceof Class<?>) {
cs = (Class<?>)target.response;
} else if (target.response != null) {
cs = target.response.getClass();
}
Object respo = cs.getDeclaredConstructor().newInstance();
if (respo != null) {
Field[] outParam = respo.getClass().getDeclaredFields();
/* Map<String, Field> respData = new HashMap<>();
for (Field f: outParam)
respData.put(f.getName().toLowerCase(), f);
Parameter[] params = method.getParameters(); */
if (outParam.length == args.length) {
Class<?>[] setArg = new Class<?>[1];
Object[] value = new Object[1];
for (int i = 0; i < args.length; i++) {
String nm = outParam[i].getName();
if (args[i] instanceof Holder<?>) {
setArg[0] = ((Holder<?>)args[i]).value.getClass();
value[0] = ((Holder<?>)args[i]).value;
} else {
setArg[0] = args[i].getClass();
value[0] = args[i];
}
String setMethod = "set" + nm.substring(0,1).toUpperCase() + nm.substring(1);
Method setm = respo.getClass().getMethod(setMethod, setArg);
setm.invoke(respo, value);
}
res = respo;
} else {
System.out.println("ERR>> InvokeOperation: '" + respo.getClass().getCanonicalName() +
"' - responce fields[" +
Integer.toString(outParam.length) + "] != args[" +
Integer.toString(args.length) + ".");
}
}
return res;
}
Object invokeMethod(Object sei, Object[] useMethod, JsonRequestResponse target) throws Exception {
Object res = null;
Class<?>[] plist = (Class[])useMethod[1];
Object[] strategy = (Object[])useMethod[2];
Method method = (Method)useMethod[0];
Object[] args = new Object[plist.length];
Method getm = null;
String getMethod = null;
if (strategy[0].equals("response")) {
Class<?>[] paramType = method.getParameterTypes();
for (int i = 0; i < paramType.length; i++) {
args[i] = paramType[i].getDeclaredConstructor().newInstance();
}
method.invoke(sei, args);
res = getResponseArgs(target, useMethod, args);
} else if (strategy[0].equals("request")) {
Field[] inputParam = target.request.getClass().getDeclaredFields();
/* Map<String, Field> reqData = new HashMap<>();
for (Field f: inputParam)
reqData.put(f.getName().toLowerCase(), f);
Parameter[] params = method.getParameters(); */
for (int i = 0; i < inputParam.length; i++) {
// String name = params[i].getName().toLowerCase();
// args[i] = reqData.get(name);
Field fld = inputParam[i];
try {
getMethod = "is" + fld.getName().substring(0,1).toUpperCase() + fld.getName().substring(1);
getm = target.request.getClass().getMethod(getMethod, null);
Object val = (Object)getm.invoke(target.request, null);
args[i] = val;
} catch (Exception ex) {
try {
getMethod = "get" + fld.getName().substring(0,1).toUpperCase() + fld.getName().substring(1);
getm = target.request.getClass().getMethod(getMethod, null);
List<?> tlist = (List<?>)getm.invoke(target.request, null);
args[i] = tlist;
} catch (Exception eex) {
args[i] = fld.get(target.request);
}
}
}
Object got = method.invoke(sei, args);
res = wrapResponse(target, useMethod, got);
} else if (((String)strategy[0]).substring(0,5).equals("empty")) {
Object got = method.invoke(sei, emptyParams);
res = wrapResponse(target, useMethod, got);
}
return res;
}
boolean setupService(Object sei, String uri, String user, String password, Map<String, String> ctrl) {
boolean res = false;
String security = ctrl.get("security");
String debug = ctrl.get("debug");
if (sei instanceof BindingProvider) {
BindingProvider bp = (BindingProvider)sei;
Binding binding = bp.getBinding();
if (debug != null && debug.equals("true") && binding instanceof SOAPBinding) {
SOAPBinding soapBinding = (SOAPBinding)binding;
Set<String> roles = soapBinding.getRoles();
System.out.println("DBG>> InvokeOperation::setupService - Roles: " + roles.toString());
}
bp.getRequestContext().put(BindingProvider.ENDPOINT_ADDRESS_PROPERTY, uri);
if (security == null || security.equals("basic")) {
bp.getRequestContext().put(BindingProvider.USERNAME_PROPERTY, user);
bp.getRequestContext().put(BindingProvider.PASSWORD_PROPERTY, password);
} else if (security != null && security.equals("digest")) {
// Note this code is cxf specific
Client client = ClientProxy.getClient(sei);
HTTPConduit httpo = (HTTPConduit)client.getConduit();
AuthorizationPolicy authPolicy = new AuthorizationPolicy();
authPolicy.setAuthorizationType("Digest");
authPolicy.setUserName(user);
authPolicy.setPassword(password);
httpo.setAuthorization(authPolicy);
} else if (security != null && security.equals("ws-security")) {
// Note this code is cxf specific
Client client = ClientProxy.getClient(sei);
Map<String, Object> outProps = new HashMap<>();
outProps.put(WSHandlerConstants.ACTION, WSHandlerConstants.USERNAME_TOKEN);
outProps.put(WSHandlerConstants.PASSWORD_TYPE, "PasswordDigest");
outProps.put(WSHandlerConstants.USER, user);
UTPasswordCallback.setAliasPassword(user, password);
outProps.put(WSHandlerConstants.PW_CALLBACK_CLASS, "onvif_relay.relay.invokers.UTPasswordCallback");
client.getOutInterceptors().add(new WSS4JOutInterceptor(outProps));
}
if (debug.equals("true")) {
List<Handler> handList = binding.getHandlerChain();
handList.add(new JakOnvifAuthHandler());
binding.setHandlerChain(handList);
}
res = true;
}
return res;
}
}Writing Servlet and embedded Jetty server on top of these was trivial.
Connecting to ONVIF Devices and Invoking Operations
The code for this example is on github: https://github.com/zebity/onvif-relay
Starting with ONVIF WSDL, this is available from "Network Interface Specification - ONVIF" page.
The ONVIF WSDL does not define any services (as these are not provided by ONVIF, rather by the device manufacturers). So you need to create a WSDL for your "device".
While the objective was create an "onvif-relay", I elected also create an ONVIF device simulator, to allow testing and verification.
So the approach was to:
- Create a software "onvif" device that could be used to test againt. This acts as a SOAP Web Service server (so can be build from WSDL generated server stubs)
- Create a "onvif" client that can talk to onvif device (SOAP Web Server) - this is "built" into JAX-WS framework which generates SEI (Service EndPoint Interface) as part of wsdl to Java generation
- Create a "relay" server that can take any "onvif" request and return the result, where the "relay" should provide any of SOAP, Restful or Thrift based interfaces to its clients and return result either as native ONVIF SOAP or JSON formatted results (to avoid all this SOAP XML complexity)
Experience with communicating with real ONVIF devices had highlighted the following issues:
- Need to ensure that the client can handle HTTP based digest authentication (ie accept a "401" result and resend request with authentication digest
- Appears to be very senstive to SOAP payload format (hence need to ensure that client adheres to the WSDL specification)
So for this project focus is initially on 1 & 2 and addressing need to support "digest authentication".
To avoid having to stand up an application server I opted to run my "onvif" test device via embedded Jetty.
Creating Java from WSDL
Initially I created a device that provides both a device management and media services, which needed to sit in new namespace ("http://www.onvif.org/ver10") above the seperate device and media name spaces used by ONVIF:
<?xml version="1.0" encoding="utf-8" ?>
<wsdl:definitions name="onvif_device" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap12/" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:tds="http://www.onvif.org/ver10/device/wsdl" xmlns:trt="http://www.onvif.org/ver10/media/wsdl" targetNamespace="http://www.onvif.org/ver10">
<wsdl:import namespace="http://www.onvif.org/ver10/device/wsdl" location="../ver10/device/wsdl/devicemgmt.wsdl"/>
<wsdl:import namespace="http://www.onvif.org/ver10/media/wsdl" location="../ver10/media/wsdl/media.wsdl"/>
<wsdl:service name="DeviceService">
<wsdl:documentation>ONVIF - Device</wsdl:documentation>
<!-- wsdl:port name="Device" binding="tns:DeviceBinding" -->
<wsdl:port name="DevicePort" binding="tds:DeviceBinding">
<soap:address location="http://127.0.0.1/onvif/device_service"/>
</wsdl:port>
</wsdl:service>
<wsdl:service name="MediaService">
<wsdl:documentation>ONVIF - Media</wsdl:documentation>
<!-- wsdl:port name="Device" binding="tns:MediaBinding" -->
<wsdl:port name="MediaPort" binding="trt:MediaBinding">
<soap:address location="http://127.0.0.1/onvif/device_service"/>
</wsdl:port>
</wsdl:service>
</wsdl:definitions>NOTE #1: Testing proved that JAX-WS does allow having multiple services from single WSDL and so result was need to break this into seperate WSDLs per ONVIF defined "portType", with each WSDL "service" having distinct URI.
NOTE #2: The <soap:address ..> tag defines the IP address of the service. This can be programmically overwritten by the client program and also by the service implementation.
NOTE #3: The wsdl import <wsdl:import ... > are from file system, as importing directly via http (as per namespace) failed (see Appendices for details)
NOTE #4: The onvif "onvif.xsd" schema file had to be edited to pickup other other schema files via file systems rather than http. The download and patch was automated with specific patch file ("patch/onvif.xsd.patch") for each of cases.
NOTE #5: When you instansiate a Web Service client it will try to go to the web to get the WSDL from target server and do a check to see if this is consistent with what it provides. By convention a web service will expose its WSDL via URL "http://IP/route?wsdl" (see below for screen shot). The support for this is generated automtically. The problem is that when you generate the server using WSDL that imports other WSDL the names get "rehashed" and are no longer consistent, so to avoid client going to server to get WSDL, it should get this as resource from its Java environment, hence reason for "resources/META-INF/wsdl/www.onvif.org/ver10/device/wsdl/onvif_device.wsdl" and copying of all included wsdl/xsd into this directory as part of build process. The maven wsimport plugin the needs to be configured to pick up correct files (see note #6).
NOTE #6: To ensure that client picks up the WSDL from Java environment (rather then via server) you need to configure the "wsdlLocation" within the maven plugin, without this the client will fail to instansiate. To streamline this and make behavior when running from packaged jar or within development environment more consistent I elected to create soft link from /META-INF to /<DEV_DIR>/onvif-relay/src/main/resources/META-INF. The same things could be achieved by doing chroot or running dev in container. So WSDL patching & maven configuration is based on convention that WSDL are alway's available from /META-INF. Hence names are consistent:
<plugin>
<groupId>com.sun.xml.ws</groupId>
<artifactId>jaxws-maven-plugin</artifactId>
<version>${project.jaxws-plugin.ver}</version>
<executions>
<execution>
<goals>
<goal>wsimport</goal>
</goals>
</execution>
</executions>
<configuration>
<wsdlLocation>/META-INF/wsdl/www.onvif.org/ver10/onvif_device.wsdl</wsdlLocation> <<== Absolute location relative to /META-INF which gets included in code
<wsdlDirectory>/META-INF/wsdl/www.onvif.org/ver10</wsdlDirectory> <<== location for running wsimport compiler (uses Unix FS link)
<wsdlFiles>
<wsdlFile>onvif_device.wsdl</wsdlFile> <<== the wsdl
</wsdlFiles>
<!-- The extension property tells JAX-WS to support soap v1.2 bindings. -->
<extension>true</extension>
</configuration>
</plugin>NOTE #7: see pom.xml for details on the dependencies for either javax (jax) or jakarta (jak) code generation option.
Create ONVIF Test Device Server
Compiling the WSDL file, will generate all the java code for SOAP XML messages and operations and two java files for the service client (DeviceService) and service implementation (Device).
To implement a server you need to create an new Java class that "implments" the interface defined by "Device.java". In this example there is Jax and Jak version of implementing class (JaxDeviceImpl.java & JakDeviceImpl). Here is Jax example, based on javax annotations:
# cat src/main/java/onvif_relay/service/JaxDeviceImpl.java
/** @what Test JAVAX onvif device service
*/
package onvif_relay.service;
import java.util.ArrayList;
import java.util.List;
import javax.xml.datatype.Duration;
import javax.xml.datatype.XMLGregorianCalendar;
import org.onvif.ver10.device.wsdl.Device;
import org.onvif.ver10.device.wsdl.DeviceServiceCapabilities;
...
...
...
import org.onvif.ver10.schema.TimeZone;
import org.onvif.ver10.schema.User;
import javax.jws.WebService; <<=== NOTE: javax.*
import import javax.xml.ws.Holder; <<=== javax.*
import @WebService( <<=== Required WebService annotation
name = "Device",
serviceName = "DeviceService",
portName = "DevicePort",
targetNamespace = "http://www.onvif.org/ver10/device/wsdl")
public class JaxDeviceImpl implements Device {
@Override
public List<Service> getServices(boolean includeCapability) {
// TODO Auto-generated method stub
return null;
}
@Override
public DeviceServiceCapabilities getServiceCapabilities() {
// TODO Auto-generated method stub
return null;
}
@Override public void getDeviceInformation(Holder<String> manufacturer, Holder<String> model, Holder<String> firmwareVersion, Holder<String> serialNumber, Holder<String> hardwareId) {
manufacturer.value = new String("john");
model.value = new String("beta");
firmwareVersion.value = new String("0.0.1");
serialNumber.value = new String("1");
hardwareId.value = new String("hw1");
}
...
...
...
}NOTE #8: Implementing class must have have @WebService annotation and has to "implement" the generated "Device" interface. All the methods (with exception of "getDeviceInformation" / "GetDeviceInformation") are automatically generated stubs (via Eclipse). For the "getDeviceInformation" I added a placeholder implementation, so the method returns a result for testing.
NOTE #9: By default all the interface object methods start with lower case (i.e. getService, getServiceCapabilities, getDeviceInformation), as JAXB & JAX-WS specification defines mapping the uses Java naming convention. This is an issue as it results in calls to upper case equivalents (ie "GetDeviceInformation) (as per WSDL) failing. Looking into this I found that the JAXB (Java Architecture for XML Binding) defines the mapping for types (as defined by XSD), it does not control the mapping of WSDL operations. For JAXB, it is possible exercise some control over the mapping at "Global, Schema, Definition and Component Scope" levels (see "Jakarta XML Binding - Section "7. Customizing XML Schema to Java Representation Binding". Here is an example of ONVIF devicemgt.wsdl patched to include "inline annotation" to specify "global" scope changes (disable Java Naming Convention & under score handling):
<wsdl:definitions xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap12/" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:tds="http://www.onvif.org/ver10/device/wsdl" targetNamespace="http://www.onvif.org/ver10/device/wsdl">
<wsdl:types>
<xs:schema targetNamespace="http://www.onvif.org/ver10/device/wsdl" xmlns:jaxb="https://jakarta.ee/xml/ns/jaxb" jaxb:version="3.0" xmlns:tt="http://www.onvif.org/ver10/schema" xmlns:tds="http://www.onvif.org/ver10/device/wsdl" elementFormDefault="qualified" version="22.12">
<xs:annotation><xs:appinfo> <<==== added annotations <jaxb:globalbinddings enablejavanamingconventions=""false"" underscorebinding=""asCharInWord"/>" < xs:appinfo>< xs:annotation> <xs:import namespace=""http://www.onvif.org/ver10/schema"" schemalocation=""../../../ver10/schema/onvif.xsd"/>" <!--="==============================-->" <xs:element name=""GetServices">" <xs:complextype> <xs:sequence> type=""xs:boolean">" <xs:annotation> <xs:documentation>indicates if the service capabilities (untyped) should be included in response.< xs:documentation> xs:element> xs:sequence> xs:complextype> ... wsdl:definitions>< code>The example uses:
- "https://jakarta.ee/xml/ns/jaxb" namespace with jaxb:version="3.0".
Alternatively use:
- "http://java.sun.com/xml/ns/jaxb" namespace with jaxb:version="2.0"
If you use the wrong version/namespace then you will get error message like this:
[WARNING] No JAXB customization was detected in the schema but the prefix "jaxb" is used for other namespace URIs. If you did intend to use JAXB customization, make sure the namespace URI is "http://java.sun.com/xml/ns/jaxb" line 1933 of file:/home/USR/Documents/dev/onvif-relay/src/main/resources/META-INF/wsdl/www.onvif.org/ver10/media/wsdl/media.wsdlNOTE 10: The XML Binding Customisation globel control flag "enableJavaNamingConventions" does not do a general enable/disable of Java Naming Convention, rather it only applies when XML Binding Customisation is being applied. This avoids having a customisation being undone, by subsequent naming convention application. The "enableJavaNamingConvention" flag controls whether the customised name is itself "mangled" to comply with the Java naming convention. This means that there is no simple way to control naming customisation, this must be done using XML schema based approach.
Now we have implementation class for the service, this can be exposed via Jetty Embedded Server. This is based on "boiler plate" code (from Jetty manual):
/**
@what Embedded Jetty Jakarta JAX-WS Device Simulator
@note: See Eclipse Jetty: Programming Guide
*/
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import org.eclipse.jetty.proxy.ConnectHandler;
import org.eclipse.jetty.security.ConstraintMapping;
import org.eclipse.jetty.security.ConstraintSecurityHandler;
import org.eclipse.jetty.security.HashLoginService;
import org.eclipse.jetty.security.UserStore;
import org.eclipse.jetty.security.authentication.DigestAuthenticator;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.servlet.ServletContextHandler;
import org.eclipse.jetty.servlet.ServletHolder;
import org.eclipse.jetty.util.security.Constraint;
import org.eclipse.jetty.util.security.Credential;
import fence.util.ConfigurationData;
import jakarta.servlet.http.HttpServlet;
import jakarta.xml.ws.Endpoint;
import jakarta.xml.ws.soap.SOAPBinding;
import onvif_relay.service.JakDeviceImpl;
import onvif_relay.service.JakMediaImpl;
import onvif_relay.servlet.OnvifFacadeServlet;
public class EmbeddedJettyJakDevice {
public static void main(String[] args) throws Exception {
System.setProperty("org.slf4j.simpleLogger.logFile", "System.out");
ConfigurationData confData = new ConfigurationData(args);
String srvPort = confData.getItem("onvif-device", "port");
String dport = confData.getItem("onvif-device", "device-port");
String mport = confData.getItem("onvif-device", "media-port");
String devrequest = confData.getItem("onvif-device", "device-service");
String medrequest = confData.getItem("onvif-device", "media-service");
String ver = confData.getItem("onvif-device", "soap-ver");
String level = confData.getItem("onvif-device", "log-level");
String dump = confData.getItem("onvif-device", "dump");
String security = confData.getItem("onvif-device", "security");
String realm = confData.getItem("onvif-device", "realm");
String auth = confData.getItem("onvif-device", "auth");
String[] cred = auth.split(":");
// ONVIF Roles: [ Administrator | Operator | User | Anonymous ]
String[] roles = {"Administrator"};
String loglevel = "warn";
if (level != null)
loglevel = level;
System.setProperty("org.slf4j.simpleLogger.defaultLogLevel", loglevel);
if (dump != null && (dump.equals("true") || dump.equals("yes"))) {
System.setProperty("com.sun.xml.ws.transport.http.HttpAdapter.dump", "true");
}
try {
System.out.println("Starting the Jetty server on port: " + srvPort + " onvif[" + dport + "," + mport + "].");
Server server = new Server(Integer.parseInt(srvPort));
JakDeviceImpl device = new JakDeviceImpl();
JakMediaImpl media = new JakMediaImpl();
// System.setProperty("jakarta.xml.ws.spi.Provider", "org.eclipse.jetty.http.spi.JettyHttpServerProvider");
String soapver = SOAPBinding.SOAP11HTTP_BINDING;
if (ver.equals("12"))
soapver = SOAPBinding.SOAP12HTTP_BINDING;
String devuri = "http://127.0.0.1:" + dport + devrequest;
Endpoint devep = Endpoint.create(soapver, device);
devep.publish(devuri);
String mediauri = "http://127.0.0.1:" + mport + medrequest;
Endpoint mediaep = Endpoint.create(soapver, media);
mediaep.publish(mediauri);
ConnectHandler proxy = new ConnectHandler();
server.setHandler(proxy);
HashLoginService loginSrv = new HashLoginService();
loginSrv.setName(realm);
UserStore creds = new UserStore();
creds.addUser(cred[0], Credential.getCredential(cred[1]), roles);
loginSrv.setUserStore(creds);
Constraint secConstraint = new Constraint();
secConstraint.setName(Constraint.__DIGEST_AUTH);
secConstraint.setRoles(roles);
secConstraint.setAuthenticate(true);
ConstraintMapping cm = new ConstraintMapping();
cm.setConstraint(secConstraint);
cm.setPathSpec("/*");
ConstraintSecurityHandler csh = new ConstraintSecurityHandler();
csh.setAuthenticator(new DigestAuthenticator());
csh.addConstraintMapping(cm);
csh.setLoginService(loginSrv);
ServletContextHandler cxtHandler = new ServletContextHandler(proxy, "/", ServletContextHandler.SESSIONS);
if (security.equals("digest")) {
cxtHandler.setSecurityHandler(csh);
}
HttpServlet srvlet = new OnvifFacadeServlet(confData);
ServletHolder holder = new ServletHolder(srvlet);
cxtHandler.addServlet(holder, "/onvif/device_service");
server.start();
server.join();
System.out.println("Stopped the simple server...");
} catch (Exception ex) {
ex.printStackTrace();
}
}
}NOTE #11: While this is running in Jetty (version 10 for javax & 11 for jakarta) there is no explicitly defined connection between Jetty http and JAX-WS implementations. While the System.setProperty() method can be used to change the implemenation providing class (SPI), this are not used in this case.
In testing I found that if you do try to use the Jetty "org.eclipse.jetty.http.spiJettyHttpServerProvider" to provide "javax.xml.ws.spi.Provider" then there is a cast exception. Testing with curl confirms that the result is being sent via the Jetty server and not directly from the Javax implementation (see below).
To test that the server is running you can see WSDL (?wsdl) or use curl to send a request to server:
$ curl --verbose http://127.0.0.1:9080/onvif/device_service -H "Content-Type: application/soap+xml" --data '<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope"><s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><GetDeviceInformation xmlns="http://www.onvif.org/ver10/device/wsdl"></GetDeviceInformation></s:Body></s:Envelope>' | xmllint --format -
* Trying 127.0.0.1:9080...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 127.0.0.1 (127.0.0.1) port 9080 (#0)
> POST /onvif/device_service HTTP/1.1
> Host: 127.0.0.1:9080
> User-Agent: curl/7.81.0
> Accept: */*
> Content-Type: application/soap+xml
> Content-Length: 283
>
} [283 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 10 Feb 2023 11:52:46 GMT
< Content-Type: application/soap+xml; charset=utf-8
< Transfer-Encoding: chunked
< Server: Jetty(11.0.12)
<
{ [692 bytes data]
100 963 0 680 100 283 57186 23799 --:--:-- --:--:-- --:--:-- 87545
* Connection #0 to host 127.0.0.1 left intact
<?xml version="1.0" encoding="UTF-8"?>
<S:Envelope xmlns:S="http://www.w3.org/2003/05/soap-envelope">
<S:Body>
<ns4:GetDeviceInformationResponse xmlns:ns4="http://www.onvif.org/ver10/device/wsdl" xmlns:ns5="http://www.onvif.org/ver10/schema" xmlns:ns6="http://www.w3.org/2005/08/addressing" xmlns:ns7="http://docs.oasis-open.org/wsn/b-2" xmlns:ns8="http://docs.oasis-open.org/wsn/t-1" xmlns:ns9="http://docs.oasis-open.org/wsrf/bf-2" xmlns:ns10="http://www.w3.org/2004/08/xop/include" xmlns:xmime="http://www.w3.org/2005/05/xmlmime">
<arg0>john</arg0>
<arg1>onvif-test</arg1>
<arg2>0.0.1</arg2>
<arg3>001</arg3>
<arg4>test#1</arg4>
</ns4:GetDeviceInformationResponse>
</S:Body>
</S:Envelope>NOTE #12: Prior to adding binding customisation, the request server has to use lower case "getDeviceInformation" as JAXB applies convention to make first letter of operations lower case. The request can also fails based on the "Content-Type". For SOAP 1.1 this is "text/xml" while for SOAP 1.2 this is "application/soap+xml"
In looking at why the server was server SOAP 1.1 rather than SOAP 1.2 (see I Appendix D.) I updated code to allow configurable selection of SOAP/HTTP binding. So you can re-run curl to to confirm if you have SOAP 1.1 or 1.2 based server.
This test example use upper case "GetDeviceInformation", as it it using XML Binding customisation.
Getting a working test server running required that:
- Apply custom names to Java methods. Initially I achieved this by hacking wsimport (not recommended) and then managed to get customisation file working (see Appendix "E. Java Naming Schema (lower case first letter) applied to WSDL PortType/Operation names & XML Binding Customisation".)
- Ensusing that service definition wrapper (onvif_device.wasl) and ONVIF wsdl (devicemgmt.wsdl) uses same namespace and where in the same directory.
Using Generated ONVIF client
To use the generated ONVIF client, is relatively straight forward. As invocation is via Java but definition is with WSDL, you need to read the WSDL and generated Java to determine interface.
In testing I found that that the client was using SOAP 1.2 and the server was using SOAP 1.1 and there were issues with ONVIF upper case WSDL operation names being converted to first letter lower case ones.
This was resolved by explicity setting SOAP 1.2 endpoint on server and adding binding customisation script to the wsimport WSDL -> Java compiler step.
My testing with client/server found that the Java client makes successful SOAP 1.2 requests with "Content-Type: application/soap+xml" but was failing with exception due to wsimport generating lower case "getDeviceInformation" rather than "GetDeviceInformation" (as specified via WSDL):
NOTE #13: RESOLVED - There appears to be asymmetry between client and server generated code, with client generating SOAP 1.2 and server providing SOAP 1.1 based service... this is true for both javax and jakarta generated code (I have tested with both cases). RESOLUTION: Change server so the EndPoint create/public explicitly request SOAP 1.2 / HTTP binding.
NOTE #14: RESOLVED - Reading the "Jakarta XML Web Services" specifications (V3.0) it has the following contradictory statements on SOAP 1.1 vs. 1.2 support, indicating that SOAP 1.1 support is required to continue, SOAP 1.2 support will be added and then says Jakarta XML Web Service will not add support for SOAP 1.2 encoding (so half in / half out for client vs server means the generated code is not compatible... strange position indeed) :
- "Goals - SOAP 1.2 - Whilst SOAP 1.1 is still widely deployed, it’s expected that services will migrate to SOAP 1.2[3][4] now that it is a W3C Recommendation. Jakarta XML Web Services will add support for SOAP 1.2 whilst requiring continued support for SOAP 1.1."
- "Non Goals - SOAP Encoding Support - Use of the SOAP encoding is essentially deprecated in the web services community, e.g., the WS-I Basic Profile[8] excludes SOAP encoding. Instead, literal usage is preferred, either in the RPC or document style. SOAP 1.1 encoding is supported in JAX-RPC 1.0 and Jakarta XML RPC but its support in Jakarta XML Web Services runs counter to the goal of delegation of data binding to Jakarta XML Binding. Therefore Jakarta XML Web Services will make support for SOAP 1.1 encoding optional and defer description of it to Jakarta XML RPC. Support for the SOAP 1.2 Encoding[4] is optional in SOAP 1.2 and Jakarta XML Web Services will not add support for SOAP 1.2 encoding.
- RESOLUTION: as per NOTE: 12 change server EndPoint create/publish to explicitly use SOAP 1.2 / HTTP binding.
NOTE #15: RESOLVED - In my first implementation, I have tried to force use of "Content-Type: text/xml" via the HTTP Headers, but adding this header was accepted (as per NOTE #12 on SOAP 1.1 / 1.2 support). RESOLUTION: As per note 12, so client code forcing "Content-Type: text/xml" was removed
NOTE #16: I have included example "public class OnvifAuthHandler implements SOAPHandler" class (just a stub at present). My testing has confirmed that is is getting invoked, for SOAP send. The aim was to provide a place to hook in ONVIF http digest authentication as per:
- "Need to ensure that the client can handle HTTP based digest authentication (ie accept a "401" result and resend request with authentication digest"
This would only be possible if you do pre-emptive digest, which requires knowledge of target device date/time setting, so alternate mechanism is needed that allow either initial probe of device or response to 401 responce.
NOTE #17: To change the authentication mechanism with Jakarta need to be done via the Jakarta Authentication framework. This requires hooking to authentication "spi" (service provider interface) but I coud not find clear example of this in context of JAX-WS so elected to swap over of CXF which includes mechanism to easily support Digest Authentication from client side.
So now have WSDL generated Java code example in place to start to test and expand onvif-relay. Need to resolve issues to progress relay.
Off to see if I can get some resolution of technical issues via StackOverflow and other github contributors ...
- RESOLVED - SOAP 1.1 vs SOAP 1.2 - Ensure have either one or the other but not half / half (see Appendix "D. ONVIF ONVIF SOAP Binding URL Goes to Dead End" for additonal information).
- RESOLVED - Automatic lower casing of Operations, can this be turned off as it is likely to cause issues / incompatiability with other ONVIF clients for the test server and ONVIF specification is clearly using upper case operation names (see Appendix "E. Java Naming Schema (lower case first letter) applied to WSDL PortType/Operation names & XML Binding Customisation" for details on trouble shooting and resolution).
The stackoverflow posting, was met by silence, so this stuff (ONVIF, SOAP based Web Services) is definitely a "back water".
I was hoping with a (warts and all) publicly available Java example, I could make for faster progress, but have mostly had to just dig deeper to resolve problems.
Now have:
- Example of EndPoint instansiation to force use of SOAP 1.2 HTTP binding (issue 1)
- Custom wsimport to generate code with ONVIF aligned method names (issue 2 workaround - no longer needed, use XML Binding Customisation Script, as below)
- XML Binding Customisation Script Generator to generate script for standard wsimport custom override (issue 2 - official way & now tested and working, see appendices)
- Script to download and patch ONVIF wsdls so they can be used to generate server and client code.
- pom.xml configurations to create and deploy jars into local maven repository, so local dependencies can be managed with pom.xml
What worked:
- Single "service" only per WSDL
- Put importing WSDL into same place within WSDL tree has the ONVIF file it is importing
- Put "main" services WSDLs in same "namespace" as the ONVIF WSDL it is wrapping
- Cannot have multiple service endpoints on same IP / port, they must have distinct ports and routes.
- Having soft link in root fs for "META-INF" -> "<DEV-LOC>/META-INF", this ensures that (cxf) tools and Java code are aligned.
What did not work:
Having multiple services defined within single wrapping WSDL
Having Wrapping WSDL in different namespace as wrapped WSDL
Adding WS-Discovery Client
In addressing need for digest authentication I moved from Metro Reference Implementation to Apache CXF Web Service framework, which also has WS-Discovery support.
Using provided test example it was easy to create a simple ONVIF camera discovery test client application, where only ONVIF requirement was to set this to use WS-Discovery V1.0 rather than default V1.1.
Running this on local network with 6 ONVIF camera, you can see result is to get the ONVIF services URL for each device:
Running test ...
[main] INFO org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean - Creating Service {http://schemas.xmlsoap.org/ws/2005/04/discovery}DiscoveryProxy from class org.apache.cxf.jaxws.support.DummyImpl
Probe, got: 6
Found: '<?xml version="1.0" encoding="UTF-8" standalone="yes"?><EndpointReference xmlns="http://www.w3.org/2005/08/addressing"><Address>http://192.168.XXX.16:80/onvif/device_service</Address><ReferenceParameters/></EndpointReference>'.
Addr: 'http://192.168.XXX.16:80/onvif/device_service'.
This should be device json details: class org.onvif.ver10.device.wsdl.GetDeviceInformationResponse
[main] INFO org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean - Creating Service {http://www.onvif.org/ver10/device/wsdl}DeviceService from WSDL: jar:file:/home/USER/.m2/repository/onvif-relay/onvif-cxf-api/1.0-SNAPSHOT/onvif-cxf-api-1.0-SNAPSHOT.jar!/META-INF/wsdl/www.onvif.org/ver10/device/wsdl/onvif_device.wsdl
read in first callo response: 'org.onvif.ver10.device.wsdl.GetDeviceInformationResponse@6ac4c3f7'.
{
"target" : "http://192.168.XXX.16:80/onvif/device_service",
"user" : "admin",
"password" : "XXXXXXXX",
"reqclass" : "GetDeviceInformation",
"respclass" : "GetDeviceInformationResponse",
"operationType" : "get",
"voidOperation" : "true",
"request" : { },
"response" : {
"Manufacturer" : "VIVOTEK",
"Model" : "FD8136",
"FirmwareVersion" : "FD8136-VVTK-0301",
"SerialNumber" : "0002D1193159",
"HardwareId" : "Mozart"
}
}
tested the response object: 'org.onvif.ver10.device.wsdl.GetDeviceInformationResponse@6ac4c3f7'.
This should be device json details: class org.onvif.ver10.device.wsdl.GetNetworkInterfacesResponse
[main] INFO org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean - Creating Service {http://www.onvif.org/ver10/device/wsdl}DeviceService from WSDL: jar:file:/home/USER/.m2/repository/onvif-relay/onvif-cxf-api/1.0-SNAPSHOT/onvif-cxf-api-1.0-SNAPSHOT.jar!/META-INF/wsdl/www.onvif.org/ver10/device/wsdl/onvif_device.wsdl
java.lang.NoSuchMethodException: org.onvif.ver10.device.wsdl.GetNetworkInterfacesResponse.setNetworkInterfaces(java.util.ArrayList)
INFO>> InvokeOperation::WrapResponse: no set, trying to recover using get+add.
read in first callo response: 'org.onvif.ver10.device.wsdl.GetNetworkInterfacesResponse@ac91282'.
{
"target" : "http://192.168.XXX.16:80/onvif/device_service",
"user" : "admin",
"password" : "XXXXXX",
"reqclass" : "GetNetworkInterfaces",
"respclass" : "GetNetworkInterfacesResponse",
"operationType" : "get",
"voidOperation" : "false",
"request" : { },
"response" : {
"NetworkInterfaces" : [ {
"Enabled" : true,
"Info" : {
"Name" : "Network interface",
"HwAddress" : "00:02:D1:21:41:52",
"MTU" : 1500
},
"Link" : {
"AdminSettings" : {
"AutoNegotiation" : true,
"Speed" : 100,
"Duplex" : "Full"
},
"OperSettings" : {
"AutoNegotiation" : true,
"Speed" : 100,
"Duplex" : "Full"
},
"InterfaceType" : 0
},
"IPv4" : {
"Enabled" : true,
"Config" : {
"Manual" : [ {
"Address" : "192.168.XXX.16",
"PrefixLength" : 24
} ],
"LinkLocal" : {
"Address" : "169.254.49.89",
"PrefixLength" : 16
},
"FromDHCP" : {
"Address" : "",
"PrefixLength" : 0
},
"DHCP" : false,
"any" : null,
"otherAttributes" : { }
}
},
"IPv6" : {
"Enabled" : false,
"Config" : {
"AcceptRouterAdvert" : true,
"DHCP" : "Auto",
"Manual" : null,
"LinkLocal" : [ {
"Address" : "",
"PrefixLength" : 0
} ],
"FromDHCP" : null,
"FromRA" : null,
"Extension" : null,
"otherAttributes" : { }
}
},
"Extension" : null,
"otherAttributes" : { },
"token" : "0"
} ]
}
}
tested the response object: 'org.onvif.ver10.device.wsdl.GetNetworkInterfacesResponse@ac91282'.
Found: '<?xml version="1.0" encoding="UTF-8" standalone="yes"?><EndpointReference xmlns="http://www.w3.org/2005/08/addressing"><Address>http://192.168.XXX.14:80/onvif/device_service</Address><ReferenceParameters/></EndpointReference>'.
Addr: 'http://192.168.XXX.14:80/onvif/device_service'.
...
...
...
So finally something that works easily with ONVIF and WebServices ;-). Only changed required as was to add support for [ws-usernameToken] security, which has authentication details being put into SOAP message rather than via HTTP headers. Adding [ws-security] required adding new maven dependency, which in turn required explicity exclusion of "ehcache" module as this bring in javax namespace libraries and not jakarta ones:
...
...
...
<dependency>
<groupId>org.apache.cxf</groupId>
<artifactId>cxf-rt-ws-security</artifactId>
<version>4.0.3</version>
<exclusions>
<exclusion>
<groupId>org.ehcache</groupId>
<artifactId>ehcache</artifactId>
</exclusion>
</exclusions>
</dependency>
...
...
...Summary
More work than expected and transition from Java EE to Jakarta EE contributed in making this more difficult that you would expect.
First basic facts.
It is not possible to build an ONVIF compliant device using straight ONVIF provided WSDL (1.1) & XSDs. The fundemental issue is that ONVIF define a single "URL" route for all requests "/onvif/device_service", which should accept device management, media and other services request.
But ONVIF breaks all its capabilities into seperate WSDL files, with its own service specific "portType" definition, which in turn force need to have seperate WSDL "service" / "port" definitions with own unque SOAP URI.
This mean that there is a core decison on how this can be implemented:
- Option #1 - Create completely seperate wrapping WSDL definitions and hence Java JAX-WS service and the put a facade in front to expose this a single "instance" routing to seperate services based on request type i.e.: "GetDeviceInformation" goes to "http://<IP>/onvif/device_service" end point and "GetServiceCapabilities" goes to "http://<IP>/onvif_media_service" end point. This means that implementor has to manage seperate service implementation project and facade creation. These can be deployed within same embedded HTTP server which handles facade URL routing.
- Option #2 - Create single WSDL as above, so can generate all code at once but still require seperate SOAP "port" defininitions that facade must route to (as per Option #1). Assuming this works can use wsimport to generate seperate implementations and instantiate these via EndPoint create/publish create. within embedded HTTP server, which handles facade URL routing.
Target was for Relay & Testing Device based on single (main) WSDL (Option #2) with Jakarta JAX-WS. Relay to expose: Restful Web Service & Thrift and talk to devices via SOAP 1.2 with http/https and Digest Authentication.
Following testing Option #1 works but does require a slightly more complicated build process, due to having to create a wrapper WSDL for each of the targeted ONVIF WSDLs. This approach also ensures that have ONVIF and wrapper namespace alignment and the customised upper case Java methods work as expected. Initial scope is for ONVIF "devicemngnt.wsdl" and "media.wsdl" WSDLs.
So have two wrapper WSDLs like (for devicemngnt.wsdl):
<?xml version="1.0" encoding="utf-8" ?>
<!-- wsdl:definitions name="onvif_media" -->
<wsdl:definitions name="onvif_device"
xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap12/"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:tds="http://www.onvif.org/ver10/device/wsdl"
xmlns:trt="http://www.onvif.org/ver10/media/wsdl"
targetNamespace="http://www.onvif.org/ver10/device/wsdl">
<!-- targetNamespace="http://www.onvif.org/ver10/media/wsdl" -->
<wsdl:import namespace="http://www.onvif.org/ver10/device/wsdl"
location="./devicemgmt.wsdl" />
<!-- wsdl:import namespace="http://www.onvif.org/ver10/media/wsdl"
location="./media.wsdl"/ -->
<wsdl:service name="DeviceService">
<wsdl:documentation>ONVIF - Device</wsdl:documentation>
<wsdl:port name="DevicePort" binding="tds:DeviceBinding">
<soap:address location="http://127.0.0.1/onvif/device_service"/>
</wsdl:port>
</wsdl:service>
<!-- wsdl:service name="MediaService">
<wsdl:documentation>ONVIF - Media</wsdl:documentation>
<wsdl:port name="MediaPort" binding="trt:MediaBinding">
<soap:address location="http://127.0.0.1/onvif/media_service"/>
</wsdl:port>
</wsdl:service -->
</wsdl:definitions>The resulting application architecture will plug together the various components in different ways to create test harness and relay:
The "ONVIF Facade" is implemented via HTTPServlet, which:
- Peeks onto SOAP payload to get the SOAP Operation request (i.e. "GetDeviceInformation")
- Does lookup to see which ONVIF service is responsible for handling the request
- Forwards requesst to particular ONVIF service by using correct port/route (i.e. "/8090/onvif/device_service" or "/8092/onvif/media_service", in this example)
This needs to look into the SOAP message to make the routing decision, so logic cannot be managed via standard URL Re-Write proxy. The facade also needs to be aware of all the published WSDL operations to determine which onvif service should be used. This was done via WSDL XSLT processor and generating method names Java code. Implementation is based on Jetty (Jakarta Version 11) ProxyServlet. This allows you to override the "rewriteTarget" method (older version of proxy had either "proxyHttpURI" or "rewriteURI" method). This method as full access to the HTTP request object, and so can peek into the SOAP message contents:
@Override
protected String rewriteTarget(HttpServletRequest request) {
String reqURL = null;
try {
InputStream is = request.getInputStream();
SOAPMessage soapReq = MessageFactory.newInstance(SOAPConstants.SOAP_1_2_PROTOCOL).createMessage(null, is);
String soapMethod = soapReq.getSOAPBody().getChildNodes().item(1).getLocalName();
System.out.println("DBG>> OnvifFacadeServlet:rewriteTarget - for: " + soapMethod);
if (DeviceOperation.contains(soapMethod) || (! MediaOperation.contains(soapMethod))) {
reqURL = "http://127.0.0.1:" + dport + devrequest;
} else {
reqURL = "http://127/0.0.1:" + mport + medrequest;
}
System.out.println("INFO>> OnvifFacadeServlet:rewriteTarget: " + reqURL);
} catch (Exception ex) {
ex.printStackTrace();
}
return reqURL;
}NOTE: Initial testing done on routing, but need to do further testing with embedded Jetty setup.
Relay RESTful HTTP to SOAP JAX-WS
Having verfied and tested both client and server parts of code, writing the relay is relatively simple, using Java Introspection/ Reflection. Here is sample code. And there is snippet above.
This has been tested with Get operations that return multiple (via Holder<T> objects, return single object (via method return interface). Need to also test with Set methods and create comprehensive auto test.
Authentication
The onvif-relay and onvif test device need to manage authentication to the devices. The onvif test device must request that a connection is authenticated (send back a 401 response as per a real device), while the relay must beable to handle a request for authentication (handle a 401 response).
This allow relay consuming client to authenticate to relay/device and then relay to handle device authentication to the device, hiding all device credentials from consuming application. ONVIF specifies that devices should provide http based Digest Authentication, so the test harness must support this.
Adding digest authentication to "test device" is via Jetty security classes. These include:
- HashLoginService - for holding "realm" and all by ConstraintSecuirtyHandler to get user id, password and realm details to check aginst
- UserStore - for holding user/password details
- Constraint - for hold roles details and authentication type ("Digest Authentication")
- ConstraintMapping - define scope of and type of protection (from Constraint)
- ConstraintSecurityHandler - the "Handler" class that is added to the http request Handling instance.
So many classes to add to simple Test Device to get Digest Authentication:
ConnectHandler proxy = new ConnectHandler();
server.setHandler(proxy);
HashLoginService loginSrv = new HashLoginService();
loginSrv.setName(realm);
UserStore creds = new UserStore();
creds.addUser(cred[0], Credential.getCredential(cred[1]), roles);
loginSrv.setUserStore(creds);
Constraint secConstraint = new Constraint();
secConstraint.setName(Constraint.__DIGEST_AUTH);
secConstraint.setRoles(roles);
secConstraint.setAuthenticate(true);
ConstraintMapping cm = new ConstraintMapping();
cm.setConstraint(secConstraint);
cm.setPathSpec("/*");
ConstraintSecurityHandler csh = new ConstraintSecurityHandler();
csh.setAuthenticator(new DigestAuthenticator());
csh.addConstraintMapping(cm);
csh.setLoginService(loginSrv);
ServletContextHandler cxtHandler = new ServletContextHandler(proxy, "/", ServletContextHandler.SESSIONS);
if (security.equals("digest")) {
cxtHandler.setSecurityHandler(csh);
}
HttpServlet srvlet = new OnvifFacadeServlet(confData);
ServletHolder holder = new ServletHolder(srvlet);
cxtHandler.addServlet(holder, "/onvif/device_service");
This generates 401 Unauthorised response and with realm, origin and nonce returned to allow client to submit digest authentication as responce to this (as per RFC 2617/7616).
Conclusion
All parts now completed for skeleton implementation.
The (default) working example is using:
- Apache cxf version 4.0.x with Jakarta APIs
- Java Introspection / Reflection to provide JSON -> JAX-WS relay
- XSLT generators to create: XML Binding Customisation scripts, code snippets required for relay and facade to allow lookup of operation names and determine if have device, media or other ONVIF request
- WS-Discovery via CXF framework tested
More time was spent in creating test and verification code (test device) and working with JAX-WS tooling than an writing the actual relay.
Work required to:
- further test/refine and replace code that relies on parameter order assumption, by either compiling with -parameter switch or reading Java code annotations
- add better JSON schema exposure (JavaScript UI ?)
- Create automatic get request generator and record result for playback in test device
- do interoperatability testing across Metro and cxf frameworks
- See if can add Digest Authentication to Metro imlementation
- Find issue with "test" device facade servlet
- (Resolved) Find issue with execution from Jar (currently code runs via Eclipse but crashes when executed directly via java)
- Fully automate build process to include XLST generators
For purpose of providing guide and example of using JAX-WS to create: ONVIF clients, testing device and relay this is done.
Please fork and create pull requests ... ;-) .
Just Enough Architecture - Technical Tips
This helped me save a lot of time talking with ONVIF devices, say thank you (suggest $ 20 - 40)
References & Links:
Jakarta EE - There is now only Java Standard Edition (JSE). The Java Platform Enterprise Edition (Java EE) & Java 2 Enterprise Edition (J2EE) with all its beans and other compilications are available as seperately packaged and distributed components. The Enterprise Edition specification is now under the stewardship of the Eclipse Foundation and re-branded as "Jakarta EE".
Jakarta JAX-WS RI (Reference Implementation) - this based is on original javax Metro RI.
OpenJDK - The Java Development Kit release under Open Source , as of JDK 11 all remnants of Java EE have been stripped of final out of JDK. So you need to be mindful of the version of libraries you are using and the change from "javax.." to "jakarta..." packaging.
RFC 2617 - HTTP Authentication: Basic and Digest Access Authentication
RFC 7616 - HTTP Digest Access Authentication (adds to with extra hash options and replaces RFC 2617 for Digest Authentication)
JEP 320 - Covers removal of Java EE and CORBA Modules from JDK. This is official nore covering removal of the Java EE (J2EE) features from JDK. These have now been transitioned to Jakarta EE.
JAXB - Java (Jakarta) Architecture for XML Binding, this includes Java code "annotations" used to control Java -> XML "marshalling" and XML -> Java "unmarshalling". The Java archives (JARs) that handle this have moved from "javax.xml.bind" to "jakarta.xml.bind". Currently that is still a lot of code which references javax.xml.bind", so you make have code with both of these.
WSDL - Web Services Description Language, the arcane, verbose and high error prone language for describing Web Service API (SOAP Envelopes, Headers and Bodies and Services). Dealing with this you can see why the development world pivoted to RESTFul Web Services as a managable alternative. And WSDL summary via Wikepedia, which is brief and fails to voice opinion that this stuff stinks ;-)
Apache CXF - a framework for creating distributed systems built around Web Services. This includes support for "traditional" SOAP Web Services and RESTful Web Serves (WS-RS). In process (Dec 2022) of being updated to Jakarta specifications
onvif.org - is industry consortium that is concerned with IP based physical security devices (Camera, Door Locks, Network Video Recorder, Video Surveillence Management)
WS-Discovery 1.0 - ONVIF uses 1.0 specification (2005), not OASIS WS-Discovery Version 1.1 (2009). The CXF WS-Discovery implementation supports both.
"onvif-relay" - blog related github repository with example code to test and develope an ONVIF relay and create onvif device test harness to allow testing of ONVIF client code
Forked Metro JAX-WS code - to test with hacked wsimport to get around issue with inconsistency between ONVIF upper case and Java first letter lower case conventions.
ONVIF Specification GitHub - the ONVIF specification is managed via GitHub based open source process. In implementing onvif-relay, I have found a couple of issues with WSDL specification: cxf compiler failure due to https redirection and empty error namespace definintion. So far only the first has been fixed via provided pull request. These need to be fixed to support clean Java code generation (seems that many are using hand crafted ONVIF code, rather than compiler generated).
Eclipse m2e Maven/Java Module Issue #173 - testing found that there is know issue with importing complex (involves use of Maven Modules and Java Modules) maven projects into Eclipse (& VSCode). This results in error: The project was not built due to "Build path contains duplicate entry: 'module-info.java' for project 'wscompile'". So I added this case to issue.
Links that helped....
- "JAX-WS SOAP Web Service Client For Java 11 With Maven" - this article has two code examples (maven pom.xml) that stripped out all the complications, providing a basis to get automatic client generation running. One is for Java EE and the other for Jakarta EE. Using this stopped the google merry-go-round trying to get a working Maven project and I updated to include Apache CXF (which I since broke and fixed again ;-) ).
- "JAX-WS Maven Plugin" - the Eclipse official page on Maven plugin to run wsimport (create annotated Java based on WSDL) and wsgen (create service definition based on annocated Java)
- Eclipse Jakarta XML Binding - page with information on Maven configurations for different Jakarta EE releases
- Stackoverflow on transition to Jakarta - there are many many questions on how to deal with removal of J2EE from JDK (complete by JDK 11), but this one has structured summary of Jakarta impact and references to materials, most of the others are incorrect and do not have any references to case.
- Java Web Service Complexity - a nice article on history and complexity of Java Web Services and steps to try to simplify this
- WSDL Structure - this simple diagram does more to make WSDL understandable than a thousand w3c words ...
- Jakarta XML Web Services Ver 3.0 - Reference for Java XML (SOAP) Web Services. Includes details of WSDL to Java mapping and annotations reference.
- Similar ONVIF Frustrations - from over 10 years ago... shows that ONVIF has not evolved substantially for long time and poor quality of documentation makes this harder then it should be... shift from Java EE to Jakarta EE adds additional complications
- "JDK 11 Illegal Access Warning" - running with javax/Jakarta WS with JDK 11 results illegal access warning. This does not stop code from running
- Jakarta XML Binding Specification - Section "7. Customizing XML Schema to Java Representation Binding" provides details on how to control the JAXB binding using annotations.
- Jakarta JAX-WS Deployment Descriptor - this is documented in Version 2.0 section "7. Deployment Descriptors" of specification but not on later version. This also has the link to "annotated schema"
Appendices
A. Create ONVIF Device WSDL using <wsdl:import ..>
If you try to directly consume the WSDL / XSD definitions and schema via http://<TARGET> then you get errors due to combination of re-direct failures, UTF-8 charset errors and security failures if you try to avoid redirects by using https://<TARGET> instead of http://<TARGET>.
So preferred alternate to modifying the ONVIF "devicemgmt.wsdl" and "media.wsdl", was to create a new device wsdl that imports the existing ONVIF WSDLs. Though getting this to work still required some changes to the ONVIF "onvif.xsd" schema file.
The "onvif_device.wsdl" WSDL, uses <wsdl:import ..> to load the ONVIF definitions. I put this into namespace above the the ONVIF device/media WSDL files:
# cat src/main/resources/META-INF/wsdl/www.onvif.org/ver10/onvif_device.wsdl
<?xml version="1.0"?>
<wsdl:definitions name="onvif_device" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap12/" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:tds="http://www.onvif.org/ver10/device/wsdl" xmlns:trt="http://www.onvif.org/ver10/media/wsdl" targetNamespace="http://www.onvif.org/ver10">
<wsdl:import namespace="http://www.onvif.org/ver10/device/wsdl" location="../ver10/device/wsdl/devicemgmt.wsdl"/>
<wsdl:import namespace="http://www.onvif.org/ver10/media/wsdl" location="../ver10/media/wsdl/media.wsdl"/>
<wsdl:service name="DeviceService">
<wsdl:documentation>ONVIF - Device</wsdl:documentation>
<!-- wsdl:port name="Device" binding="tns:DeviceBinding" -->
<wsdl:port name="DevicePort" binding="tds:DeviceBinding">
<soap:address location="http://127.0.0.1/onvif/device_service"/>
</wsdl:port>
</wsdl:service>
<wsdl:service name="MediaService">
<wsdl:documentation>ONVIF - Media</wsdl:documentation>
<!-- wsdl:port name="Device" binding="tns:MediaBinding" -->
<wsdl:port name="MediaPort" binding="trt:MediaBinding">
<soap:address location="http://127.0.0.1/onvif/device_service"/>
</wsdl:port>
</wsdl:service>
</wsdl:definitions>When this is compiled using wsimport, the results is the following client definition and server implementation stubs:
- Device.java - provides the Java interface definition, which should be "implemented" by the server implemenation class (this correlates to the ONVIF "devicemgmt.wsdl" abstract portType definition "<wsdl:portType name="Device">" (this is included via "<wsdl:import ... location=" ... devicemgmt.wsdl"/>" above)
- DeviceService.java - provides the concrete client interface or "Service Endpoint Interface" (SEI) and correlates the "<wsdl:service name="DeviceService">" definition above
- Media.java - provides the Java interface definition, which should be "implemented" by the server implemenation class (this correlates to the ONVIF "media.wsdl" abstract portType definition "<wsdl:portType name="Media">" (this is included via <wsdl:import ... location=" ... media.wsdl"/> above)
- MediaService.java - provides the concrete client interface or "Service Endpoint Interface" and correlates the "<wsdl:service name="MediaService"> definition above
Using Jakarta EE 3.0 JAX-WS classes the resulting Java code translates Service into uppercase Java Classes and operations into methods with first letter as lower case. The result is that while ONVIF WSDL has all upper case names for operations, such as "GetDeviceInformation" the generated Java has "getDeviceInformation". This results in asymmetry in WSDL/XML definition vs. Java implementation.
In testing my first server I found that it was exposing the Java operation definitions (following lower first letter convention) rather then the WSDL/XML definition...
B. Trival Update to ONVIF WSDL to Generate Java Web Services
The onvif device WSDL provides all the type, binding and port type definitions. You just need to add the "<service ..>" definition.
NOTE #1: Initially I elected to just add the services definitions into bottom of ONVIF provided WSD. The preferred alternative being to import the existing WSDL into new seperate services definition WSDL. However this resulted in a lot of errors (see Appendix C. below). Since determining cause of various issues, I returned to original approach (to import into "biggger" WSDL), which is detailed in Appendix A. above). So this approach is now superfluous.
C. Namespace and XSD includes with HTTP to HTTPS Redirect Failure
The ONVIF WSDL uses "http://<TARGET>" for all of its namespace and XSD inclusion statements. Many of the targetted URI get redirected to "https" or alternate equivalents / replacements / relocations. The problem is that some of the redirects are via html, not xml resulting in failure within the XML parser.
Here is specific example:
<?xml version="1.0" encoding="utf-8"?>
<!--<?xml-stylesheet type="text/xsl" href="onvif-schema-viewer.xsl"?>-->
<!--
Copyright (c) 2008-2022 by ONVIF: Open Network Video Interface Forum. All rights reserved.
Recipients of this document may copy, distribute, publish, or display this document so long as this copyright notice, license and disclaimer are retained with all copies of the document. No license is granted to modify this document.
THIS DOCUMENT IS PROVIDED "AS IS," AND THE CORPORATION AND ITS MEMBERS AND THEIR AFFILIATES, MAKE NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, OR TITLE; THAT THE CONTENTS OF THIS DOCUMENT ARE SUITABLE FOR ANY PURPOSE; OR THAT THE IMPLEMENTATION OF SUCH CONTENTS WILL NOT INFRINGE ANY PATENTS, COPYRIGHTS, TRADEMARKS OR OTHER RIGHTS.
IN NO EVENT WILL THE CORPORATION OR ITS MEMBERS OR THEIR AFFILIATES BE LIABLE FOR ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, PUNITIVE OR CONSEQUENTIAL DAMAGES, ARISING OUT OF OR RELATING TO ANY USE OR DISTRIBUTION OF THIS DOCUMENT, WHETHER OR NOT (1) THE CORPORATION, MEMBERS OR THEIR AFFILIATES HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES, OR (2) SUCH DAMAGES WERE REASONABLY FORESEEABLE, AND ARISING OUT OF OR RELATING TO ANY USE OR DISTRIBUTION OF THIS DOCUMENT. THE FOREGOING DISCLAIMER AND LIMITATION ON LIABILITY DO NOT APPLY TO, INVALIDATE, OR LIMIT REPRESENTATIONS AND WARRANTIES MADE BY THE MEMBERS AND THEIR RESPECTIVE AFFILIATES TO THE CORPORATION AND OTHER MEMBERS IN CERTAIN WRITTEN POLICIES OF THE CORPORATION.
-->
<xs:schema xmlns:tt="http://www.onvif.org/ver10/schema" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xmime="http://www.w3.org/2005/05/xmlmime" xmlns:wsnt="http://docs.oasis-open.org/wsn/b-2" xmlns:xop="http://www.w3.org/2004/08/xop/include" xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope" targetNamespace="http://www.onvif.org/ver10/schema" elementFormDefault="qualified" version="22.12">
<xs:include schemaLocation="common.xsd"/>
<xs:import namespace="http://www.w3.org/2005/05/xmlmime" schemaLocation="http://www.w3.org/2005/05/xmlmime"/>
<xs:import namespace="http://www.w3.org/2003/05/soap-envelope" schemaLocation="http://www.w3.org/2003/05/soap-envelope"/>
<xs:import namespace="http://docs.oasis-open.org/wsn/b-2" schemaLocation="http://docs.oasis-open.org/wsn/b-2.xsd"/>
<xs:import namespace="http://www.w3.org/2004/08/xop/include" schemaLocation="http://www.w3.org/2004/08/xop/include"/>
<!--===============================-->
<!-- Generic Types -->
<!--===============================-->
---
--- NOTE: changed -
--- http://www.w3.org/2005/05/xmlmime to https://www.w3.org/2005/05/xmlmime
--- http://www.w3.org/2003/05/soap-envelope to https://www.w3.org/2003/05/soap-envelope
--- http://www.w3.org/2004/08/xop/include to https://www.w3.org/2004/08/xop/include
---
</xs:complexType>
</xs:schema>
Without changing the includes to use https rather then http , the follow wsimport error is returned:
[DEBUG] /bin/sh -c cd '/home/USR/Documents/dev/maven/onvif_device' && '/usr/lib/jvm/java-11-openjdk-amd64/bin/java' '-cp' '/home/USR/.m2/repository/com/sun/xml/ws/jaxws-maven-plugin/3.0.2/jaxws-maven-plugin-3.0.2.jar' 'org.jvnet.jax_ws_commons.jaxws.Invoker' 'com.sun.tools.ws.wscompile.WsimportTool' '-pathfile' '/tmp/jax-ws-mvn-plugin-cp11259448073909233876.txt' '-keep' '-s' '/home/USR/Documents/dev/maven/onvif_device/target/generated-sources/wsimport' '-d' '/home/USR/Documents/dev/maven/onvif_device/target/classes' '-encoding' 'UTF-8' '-extension' '-Xnocompile' '-wsdllocation' 'devicemgmt.wsdl' 'file:/home/USR/Documents/dev/maven/onvif_device/src/main/wsdl/onvif/ver10/media/wsdl/media.wsdl' parsing WSDL...
[ERROR] Premature end of file. line 1 of http://www.w3.org/2005/05/xmlmime
[ERROR] org.xml.sax.SAXParseException; systemId: http://www.w3.org/2005/05/xmlmime; lineNumber: 1; columnNumber: 1; Premature end of file. line 11 of file:/home/USR/Documents/dev/maven/onvif_device/src/main/wsdl/onvif/ver10/media/wsdl/media.wsdl Failed to parse the WSDL.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.997 s
[INFO] Finished at: 2022-12-22T12:42:30+11:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.sun.xml.ws:jaxws-maven-plugin:3.0.2:wsimport (default-cli) on project onvif_device: Mojo failed - check output -> [Help 1]And here is result is you try to source onvif "devicemgmt.wsdl" via: "http://www.onvif.org/ver10/device/wsdl/devicemgmt.wsdl":
[DEBUG] /bin/sh -c cd '/home/USR/Documents/dev/maven/onvif_device' && '/usr/lib/jvm/java-11-openjdk-amd64/bin/java' '-cp' '/home/USR/.m2/repository/com/sun/xml/ws/jaxws-maven-plugin/3.0.2/jaxws-maven-plugin-3.0.2.jar' 'org.jvnet.jax_ws_commons.jaxws.Invoker' 'com.sun.tools.ws.wscompile.WsimportTool' '-pathfile' '/tmp/jax-ws-mvn-plugin-cp9540096234171064168.txt' '-keep' '-s' '/home/USR/Documents/dev/maven/onvif_device/target/generated-sources/wsimport' '-d' '/home/USR/Documents/dev/maven/onvif_device/target/classes' '-encoding' 'UTF-8' '-extension' '-Xnocompile' 'http://www.onvif.org/ver10/device/wsdl'
parsing WSDL...
[ERROR] DOCTYPE is disallowed when the feature "http://apache.org/xml/features/disallow-doctype-decl" set to true. line 1 of http://www.onvif.org/ver10/device/wsdl
[ERROR] DOCTYPE is disallowed when the feature "http://apache.org/xml/features/disallow-doctype-decl" set to true. Failed to read the WSDL document: http://www.onvif.org/ver10/device/wsdl, because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <wsdl:definitions>.
[ERROR] Could not find wsdl:service in the provided WSDL(s): At least one WSDL with at least one service definition needs to be provided. Failed to parse the WSDL. This error might be a bit puzzling as there is no "DOCTYPE" field in the onvif wsdl definition, the reason is that this is in the "redirect" file, which you will not see if just using the web browser to see the wsdl file:
$ curl --verbose http://www.onvif.org/ver10/device/wsdl
* Trying 190.92.159.115:80...
* Connected to www.onvif.org (190.92.159.115) port 80 (#0)
> GET /ver10/device/wsdl HTTP/1.1
> Host: www.onvif.org
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Connection: Keep-Alive
< Keep-Alive: timeout=5, max=100
< content-type: text/html
< content-length: 707
< date: Fri, 10 Feb 2023 13:28:07 GMT
< server: LiteSpeed
< location: http://www.onvif.org/ver10/device/wsdl/
< strict-transport-security: max-age=63072000; includeSubDomains
< x-frame-options: SAMEORIGIN
< x-content-type-options: nosniff
< vary: User-Agent,User-Agent
< access-control-allow-origin: https://members.onvif.org
< content-security-policy: frame-ancestors https://members.onvif.org
<
<!DOCTYPE html>
<html style="height:100%">
<head>
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no" />
<title> 301 Moved Permanently
</title></head>
<body style="color: #444; margin:0;font: normal 14px/20px Arial, Helvetica, sans-serif; height:100%; background-color: #fff;">
<div style="height:auto; min-height:100%; "> <div style="text-align: center; width:800px; margin-left: -400px; position:absolute; top: 30%; left:50%;">
<h1 style="margin:0; font-size:150px; line-height:150px; font-weight:bold;">301</h1>
<h2 style="margin-top:20px;font-size: 30px;">Moved Permanently
</h2>
<p>The document has been permanently moved.</p>
</div></div></body></html>
* Connection #0 to host www.onvif.org left intactAnd another error variation... where the returned document is html rathter than xml, then retreiving: "http://www.onvif.org/ver10/device/wsdl/" (note now have "slash" at end of URL which is there you pior example will redirect to0, all these redirects are slightly different and so return different errors: http://www.onvif.org/ver10/device/wsdl -> (via http location header) http://www.onvif.org/ver10/device/wsdl/ -> (via javascript) devicemgmt.wsdl
[DEBUG] /bin/sh -c cd '/home/USR/Documents/dev/maven/onvif_device' && '/usr/lib/jvm/java-11-openjdk-amd64/bin/java' '-cp' '/home/USR/.m2/repository/com/sun/xml/ws/jaxws-maven-plugin/3.0.2/jaxws-maven-plugin-3.0.2.jar' 'org.jvnet.jax_ws_commons.jaxws.Invoker' 'com.sun.tools.ws.wscompile.WsimportTool' '-pathfile' '/tmp/jax-ws-mvn-plugin-cp17731041581007009810.txt' '-keep' '-s' '/home/USR/Documents/dev/maven/onvif_device/target/generated-sources/wsimport' '-d' '/home/USR/Documents/dev/maven/onvif_device/target/classes' '-encoding' 'UTF-8' '-extension' '-Xnocompile' 'http://www.onvif.org/ver10/device/wsdl/'
parsing WSDL...
[ERROR] Invalid WSDL http://www.onvif.org/ver10/device/wsdl/, expected {http://schemas.xmlsoap.org/wsdl/}definitions found head at (line 1) Failed to read the WSDL document: http://www.onvif.org/ver10/device/wsdl/, because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <wsdl:definitions>.
[ERROR] Could not find wsdl:service in the provided WSDL(s): At least one WSDL with at least one service definition needs to be provided. Failed to parse the WSDL.Again do curl trace to see exactly what is happening:
$ curl --verbose http://www.onvif.org/ver10/device/wsdl/
* Trying 190.92.159.115:80...
* Connected to www.onvif.org (190.92.159.115) port 80 (#0)
> GET /ver10/device/wsdl/ HTTP/1.1
> Host: www.onvif.org
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Keep-Alive: timeout=5, max=100
< cache-control: public, max-age=30,public, must-revalidate, proxy-revalidate
< expires: Fri, 10 Feb 2023 13:34:27 GMT
< content-type: text/html
< last-modified: Thu, 02 Feb 2017 08:34:20 GMT
< etag: "70-5892ef0c-0;;;"
< accept-ranges: bytes
< content-length: 112
< date: Fri, 10 Feb 2023 13:33:57 GMT
< server: LiteSpeed
< strict-transport-security: max-age=63072000; includeSubDomains
< x-frame-options: SAMEORIGIN
< x-content-type-options: nosniff
< vary: User-Agent,User-Agent
< access-control-allow-origin: https://members.onvif.org
< content-security-policy: frame-ancestors https://members.onvif.org
< pragma: public
< x-powered-by: W3 Total Cache/0.9.4.6.4
<
<head>
<script type="text/javascript">
<!--
document.location.href="devicemgmt.wsdl";
//-->
</script>
* Connection #0 to host www.onvif.org left intactD. ONVIF SOAP Binding URL Goes to Dead End ([WARNING] SOAP port "DevicePort": uses a non-standard SOAP 1.2 binding)
The ONVIF specifcation defines that:
- "Binding definitions for an ONVIF compliant device according to this specification shall follow the requirements in [WS-I BP 2.0]. This implies that the WSDL SOAP 1.2 bindings shall be used."
The ONVIF WSDL uses the following URL for its SOAP Transport binding:
<wsdl:binding name="DeviceBinding" type="tds:Device">
<soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http"/>
<wsdl:operation name="GetServices">
<soap:operation soapAction="http://www.onvif.org/ver10/device/wsdl/GetServices"/>
<wsdl:input>
<soap:body use="literal"/>
</wsdl:input>
<wsdl:output>
<soap:body use="literal"/>
</wsdl:output>
</wsdl:operation> This is a "dead-end" reference that get redirected, to a none readable binary file:
$ wget --verbose http://schemas.xmlsoap.org/soap/http
--2023-02-11 00:43:04-- http://schemas.xmlsoap.org/soap/http
Resolving schemas.xmlsoap.org (schemas.xmlsoap.org)... 13.107.238.32, 13.107.237.32, 2620:1ec:4f:1::32, ...
Connecting to schemas.xmlsoap.org (schemas.xmlsoap.org)|13.107.238.32|:80... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: https://schemas.xmlsoap.org/soap/http [following]
--2023-02-11 00:43:05-- https://schemas.xmlsoap.org/soap/http
Connecting to schemas.xmlsoap.org (schemas.xmlsoap.org)|13.107.238.32|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://schemasxmlsoap.azurewebsites.net/soap/http/ [following]
--2023-02-11 00:43:06-- https://schemasxmlsoap.azurewebsites.net/soap/http/
Resolving schemasxmlsoap.azurewebsites.net (schemasxmlsoap.azurewebsites.net)... 13.66.212.205
Connecting to schemasxmlsoap.azurewebsites.net (schemasxmlsoap.azurewebsites.net)|13.66.212.205|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 511 [text/html]
Saving to: ‘http’
http 100%[==========================================================================>] 511 --.-KB/s in 0s
2023-02-11 00:43:07 (92.4 MB/s) - ‘http’ saved [511/511]This gets subtle visibility when running "wsimport" with the following warning message:
[WARNING] SOAP port "DevicePort": uses a non-standard SOAP 1.2 binding. line 17 of file:/home/USR/Documents/dev/onvif-relay/src/main/resources/META-INF/wsdl/www.onvif.org/ver10/onvif_device.wsdl/The error can be addressed by patching the ONVIF wsdl to use the updated namespace/schema location for SOAP1.2 HTTP Binding: http://www.w3.org/2003/05/soap/bindings/HTTP/ (as defined by "SOAP12HTTP_BINDING" constant below).
Using the WSDL as basis for generating the ONVIF device implementation, thus results in SOAP 1.1 implementation. To create a SOAP 1.2 based service requires explicitly create an "EndPoint" with a SOAP 1.2 HTTP binding:
String soapver = SOAPBinding.SOAP11HTTP_BINDING;
if (ver.equals("12"))
soapver = SOAPBinding.SOAP12HTTP_BINDING;
String uri = "http://127.0.0.1:" + port + request;
Endpoint ep = Endpoint.create(soapver, device);
ep.publish(uri);When deploying to appliation server this would require either "webservice.xml" deployment descriptor or change to service Java Code annotations to make use of SOAP 1.2 / HTTP explicit.
E. Java Naming Schema (lower case first letter) applied to WSDL PortType/Operation names & XML Binding Customisation
Testing with both WsImport (JAX & JAK) found that tools apply Java naming convention to all names (upper case Class Names & lower case first work Method names). This results in implementation that is now aligned with ONVIF WSDL.
However the tools are behaving as per specification:
Section 2.3. Operation
- "Each
wsdl:operationin awsdl:portTypeis mapped to a Java method in the corresponding Java service endpoint interface. - ◊ Conformance (Method naming): In the absence of customizations, the name of a mapped Java method MUST be the value of the
nameattribute of thewsdl:operationelement mapped according to the rules described in Section 2.8, “XML Names”."
2.8. XML Names
- "Appendix D of Jakarta XML Binding[39] defines a mapping from XML names to Java identifiers. Jakarta XML Web Services uses this mapping to convert WSDL identifiers to Java identifiers with the following modifications and additions:
- Method identifiers - When mapping
wsdl:operationnames to Java method identifiers, thegetorsetprefix is not added. Instead the first word in the word-list has its first character converted to lower case."
The official way to enforce naming based on ONVIF is to use "XML Binding Customization" (see specification section "8. Customizations"). This requires either inline XML annotations or external binding customisation file which is loaded with wsimport using -b flag. Other alternates are:
- Hack wsimport - I did this to get initial set of code that had case consistent names with ONVIF WSDL. Currently this only works with Jakarta code base. See github for fork of metro reference implemenation.
- Post process generated code - this is much harder to do than pre-processing WSDL as Java is harder to parse than XML. So I opted to create an. XLST script (extract-operation-methods.xslt) that read WSDL and generated XML Binding Customisation XML.
The XML Binding custisation XML uses XPATH to select the portType operations: "<jaxws:bindings node="wsdl:definitions/wsdl:portType[@name='Device']/wsdl:operation[@name='GetServiceCapabilities']">" or generally "<jaxws:bindings node="wsdl:definitions/wsdl:portType[@name='Device']/wsdl:operation[@*]">.
The having found the portType/Operation is is simply case to apply custom Java method name that is just the WSDL defined name: "<jaxws:methodname="@name"/>".
However JAX-WS seems to mandate that the "target" name is a fixed string, so need to go through all of WSDL do find operations and use found WSDL name as method name:
<?xml version="1.0" encoding="utf-8"?>
<jaxws:bindings xmlns:jaxws="http://java.sun.com/xml/ns/jaxws" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" wsdlLocation="../ver10/device/wsdl/devicemgmt.wsdl">
<jaxws:bindings node="wsdl:definitions/wsdl:portType[@name='Device']/wsdl:operation[@*]">
<jaxws:method name="@name"/>
</jaxws:bindings>
</jaxws:bindings> And so general generation script:
<?xml version="1.0" encoding="UTF-8"?>
<xsl:transform version="1.1" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:jaxws="https://jakarta.ee/xml/ns/jaxws">
<!-- JAX xsl:transform version="1.1" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:jaxws="http://java.sun.com/xml/ns/jaxws" -->
<xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes" />
<xsl:template match="/">
<xsl:param name="file" select="p1" />
<!-- xsl:variable name="wsdlLoc" select="document-uri()" / -->
<!-- xsl:variable name="wsdlLoc" select="concat(", $file, ")" / -->
<xsl:variable name="wsdlLoc" select=""/META-INF/wsdl/www.onvif.org/ver10/device/wsdl/devicemgmt.wsdl"" />
<jaxws:bindings xmlns:jaxws="https://jakarta.ee/xml/ns/jaxws" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:jaxb="https://jakarta.ee/xml/ns/jaxb" >
<!-- JAX jaxws:bindings xmlns:jaxws="http://java.sun.com/xml/ns/jaxws" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" -->
<xsl:attribute name="wsdlLocation">
<xsl:value-of select="$p1" />
</xsl:attribute>
<jaxb:globalBindings enableJavaNamingConventions = "false" underscoreBinding = "asCharInWord"/>
<xsl:for-each select="wsdl:definitions/wsdl:portType[@*]">
<xsl:variable name="portType" select="@name" />
<xsl:for-each select="wsdl:operation[@*]">
<xsl:variable name="operation" select="@name" />
<xsl:element name="jaxws:bindings">
<xsl:attribute name="node">
<xsl:variable name="selector" select="concat("wsdl:definitions/wsdl:portType[@name='", $portType, "']/wsdl:operation[@name='", $operation, "']")" />
<xsl:value-of select="$selector" />
<!-- xsl:value-of select="concat("wsdl:definitions/wsdl:portType[@name='Device']/wsdl:operation[@name='",$operation, "']")" / -->
</xsl:attribute>
<xsl:element name="jaxws:method">
<xsl:attribute name="name">
<xsl:value-of select="$operation" />
</xsl:attribute>
</xsl:element>
</xsl:element>
</xsl:for-each>
</xsl:for-each>
</jaxws:bindings>
</xsl:template>
</xsl:transform>The generated XML Binding Customisation script is passed to the wsimport tools via the "-b binding" option. This uses XPaths to identity the element that you wish to "customise" and the a small annotation which specifies the customisation outcome. For ONVIF case the customisation ensures that the WSDL operation names are used vertatim when mapped to the Java class methods. The WSDL operation re identified by the XPATH: "/wsdl:definitions/wsdl:portType[@name='Device']/wsdl:operation[@name='GetServiceCapabilities'" for example. Once Operation is selected this is used for the corresponding Java method (ie "GetServiceCapabilities") and no Java naming convention is applied.
As each ONVIF WSDL needed to be wrapped in "service" definition" WSDL, it was not clear how the target WSDL needed to be defined (the processing spcript requires specifing the "wsdlLocation" for the targetted file.
Should this be the:
- "wrapping WSDL" or
- the "wrapped WSDL"?
I was only able to debug this via CXF wsdl2java tool, as the metro wsimport, did not provide the required diagnostic output to allow you to find out what worked.
Testing proved that the wsdlLocation == wrapped WSDL, so the resulting customisation files looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<jaxws:bindings xmlns:jaxws="https://jakarta.ee/xml/ns/jaxws" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:jaxb="https://jakarta.ee/xml/ns/jaxb" wsdlLocation="/META-INF/wsdl/www.onvif.org/ver10/device/wsdl/devicemgmt.wsdl">
<jaxb:globalBindings enableJavaNamingConventions="false" underscoreBinding="asCharInWord"/>
<jaxws:bindings node="wsdl:definitions/wsdl:portType[@name='Device']/wsdl:operation[@name='GetServices']">
<jaxws:method name="GetServices"/>
</jaxws:bindings>
<jaxws:bindings node="wsdl:definitions/wsdl:portType[@name='Device']/wsdl:operation[@name='GetServiceCapabilities']">
<jaxws:method name="GetServiceCapabilities"/>
</jaxws:bindings>
<jaxws:bindings node="wsdl:definitions/wsdl:portType[@name='Device']/wsdl:operation[@name='GetDeviceInformation']">
<jaxws:method name="GetDeviceInformation"/>
</jaxws:bindings>
...
...
...
</jaxws:bindings>This example uses the Jakarta namespaces:
- xmlns:jaxws="https://jakarta.ee/xml/ns/jaxws"
- xmlns:jaxb="https://jakarta.ee/xml/ns/jaxb"
For javax the namespace is:
F. Diversion to Force Java Class Loading
Initially to try to work around the Java naming convention that was being enforced wsimport I thought I would try to see if I could "hook" in a method that captured the place where call was being made to convert WSDL operation names to Java Method names.
This occurs within the following code:
- com.sun.tools.ws.processor.model.Operation method: public String getJavaMethodName() which has hook to static reference to
- org.glassfish.jaxb.core.api.impl.NameConverter interface class via
- com.sun.xml.ws.spi.db.BindingHelper.
The code relies on static initialiser to establish a set of "static final variables of the "NameConverter defined as part of the NameConverter interface.
The static variable standard" which has an instantialed instance of the abstact "NameCoverter" class via org.glassfish.jaxb.core.api.impl.NameUtil.
So my initial thought was to replace the "standard" object reference with new object reference which would now apply the Java Naming convention logic.
To achieve this you would need to:
- Ensure that "hooking" class got loaded (without it being explicitly referenced by the WsImport WSDL compiler
- Make sure that the "standard" variable has already been initialised first to avoid "hooked in" replacement did not get overwritten
- Keep reference to the initial object so you could invoke it most cases and also put the reference back if you wanted to "unhook" the alternate code
- Have to use Java introspection to get around the fact that standard was marked as final and hence should not be to have have its value overridden once initialised.
This would allow change in WsImport behaviour without need to change its code...
To achieve class loading you should be able to:
- Add new Jar into class path
- Have static initialisation block, which call constructor of "NameUtil" to ensure that that static method "standard" is istantiated before hooking function. So end up with this block of code (fence.jaxws.MaintainOnvifGetSetUpperIdentifierNamesUtil.java): https://github.com/zebity/onvif-relay/blob/main/src/main/java/fence/jaxws/MaintainOnvifGetSetUpperIdentifierNamesUtil.java
This does instanstiation of new Standard() object which ensure that the static standard variable is initialized. Then the static initialisation try/catch block (as could get a number of exception (including security one...), uses Java reflection APIs for force classLoader to be called and remove "final" modifier from the static class variable. This runs but get "warning" message:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by fence.jaxws.MaintainOnvifGetSetUpperIdentifierNamesUtil (file:/home/USR/.m2/repository/onvif-relay/onvif-naming/0.0.1/onvif-naming-0.0.1.jar) to field java.lang.reflect.Field.modifiers
WARNING: Please consider reporting this to the maintainers of fence.jaxws.MaintainOnvifGetSetUpperIdentifierNamesUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
INFO>> MaintainOnvifGetSetUpperIdentifierNamesUtil Enabled!!This should work by adding new jar into class path (see src/main/xml for shell scripts to load new local libraries ito Maven repository) and the the static initializer black should get executed. I found that this this did not occurr..... A potential reason is that wsimport is not directly invoked when used via maven plugin. Rather it is making indirect load via a stand alone container environment.
So to "hack" WsImport I was forced to take fork and change code to force load. Having taken the fork it was then simpler just to make changes directly via update to com.sun.tools.ws.processor.model.Operation.
So I was not able to get wsimport hack going without actually changing code...
Here is code for changed WsImport which forces load of alternate NameConverter:
/*
* Copyright (c) 1997, 2018 Oracle and/or its affiliates. All rights reserved.
*
* This program and the accompanying materials are made available under the
* terms of the Eclipse Distribution License v. 1.0, which is available at
* http://www.eclipse.org/org/documents/edl-v10.php.
*
* SPDX-License-Identifier: BSD-3-Clause
*/
package com.sun.tools.ws;
import com.sun.tools.ws.wscompile.WsimportTool;
import fence.jaxws.MaintainOnvifGetSetUpperIdentifierNamesUtil;
/**
* WsImport tool entry point.
*
* @author Vivek Pandey
* @author Kohsuke Kawaguchi
*/
public class WsImport {
/**
* CLI entry point. Use {@link Invoker} to load proper API version
*/
/* Force use of "special ONVIF Get/Set (uppercase method names" */
static MaintainOnvifGetSetUpperIdentifierNamesUtil force = new MaintainOnvifGetSetUpperIdentifierNamesUtil();
public static void main(String[] args) throws Throwable {
System.exit(Invoker.invoke("com.sun.tools.ws.wscompile.WsimportTool", args));
}
/**
* Entry point for tool integration.
*
* <p>
* This does the same as {@link #main(String[])} except
* it doesn't invoke {@link System#exit(int)}. This method
* also doesn't play with classloaders. It's the caller's
* responsibility to set up the classloader to load all jars
* needed to run the tool.
*
* @return
* 0 if the tool runs successfully.
*/
public static int doMain(String[] args) throws Throwable {
return new WsimportTool(System.out).run(args) ? 0 : 1;
}
}Or just "hack" the getJavaMethodName method in com.sun.tools.ws.processor.model.Operation class...
public String getJavaMethodName(){
if (USE_NAME_HACK) {
String res = null;
String from = " mangler: '";
if (_javaMethod != null) {
res = _javaMethod.getName();
from = " _javaMethod: '";
} else if (customizedName != null) {
res = customizedName;
from = " customName: '";
} else {
String loc = _name.getLocalPart();
if (Character.isUpperCase(loc.charAt(0)) || loc.startsWith("Get") || loc.startsWith("Set")) {
from = " keep upper: '";
res = loc;
} else {
res = BindingHelper.mangleNameToVariableName(loc);
}
}
System.out.println("INFO>> Operation::getJavaMethodName - from " + from + res + "'");
return res;
} else {
//if JavaMethod is created return the name
if(_javaMethod != null){
return _javaMethod.getName();
}
//return the customized operation name if any without mangling
if(customizedName != null){
return customizedName;
}
return BindingHelper.mangleNameToVariableName(_name.getLocalPart());
}
} Both these options are in "hacked" metro github respository.
But recommended way is to use XML Binding Customisation Script ... as above in Apppendix "E. Java Naming Schema (lower case first letter) applied to WSDL PortType/Operation names & XML Binding Customisation"