to homepage
 Weekly emails: how to advanced search
 Glossary lookup:


> opinion > vendor perspective

Be sure to manage your metadata

by Jim Gabriel
February 18th, 2005

Cast your mind back to 1999, when IT departments the world over were panic-purchasing Y2K protection strategies ahead of the year 2000 date change. The myriad different conventions and sneaky short-cuts used by programmers for expressing and constraining dates in the previous decades forced the entire IT community to trawl through terabytes of code and data in an attempt to control the potential damage that global application failure would bring. Doom scenarios portrayed crashing markets, crashing planes, burning buildings, failing sea defences ... and all because of the 2-digit date and the corner-cutting programmer.

• print  • comment

Metadata evolution management is the real problem facing the long term lifecycle management of web services development projects.

Jim Gabriel of XML management specialist digitalML is managing director of the company's CortexML division. The CortexML suite automates the management and evolution of XML-reliant applications.

Glossary terms: metadata, XML, registry, SOA, orchestration, lookup tool

The reality of Y2K is that we managed to prevent the world ending prematurely. But it left a bad taste in the mouth for many of us, and with very good reason. We spent a lot of money clearing up a technology-induced mess, and received no tangible business benefits. The panic was well-founded, because nobody had a conceptual overview of all the places where dates were dealt with in software, and it was impossible to see how program code handled dates without first finding the code and then examining it. This was like looking for an unknown number of needles in an endless series of haystacks.

The Y2K problem is an excellent analogy for the task facing developers when metadata-driven environments need to evolve. A service-oriented architecture (SOA) is a metadata-driven architecture. Metadata is crucial to the development lifecycle of web services. Without it, the long term maintainability of the SOA is at risk, because the business logic expressed in services is not visible to the IT department at a higher level than in the code itself.

In a SOA, the business logic is expressed in message payloads that are constrained by XML schemas. These schemas define the metadata governing how messages are handled. They are externalized, standardized, and federated. This has the following advantages:

  • Enforceable contracts for processing behavior
  • Visible specifications for developers
  • Public interfaces for new partners in the SOA
  • Schema-based access to standard infrastructure such as parsers, transformation engines, and so on
  • Insulation for services from changes to schemas
  • Support for business analysts when planning changes

The disadvantages are due entirely to the limitations of metadata in general and XML schemas in particular. Nowadays we must expect schemas to change. The XML schemas describing web services message payloads are application-specific, bespoke metadata, which requires human involvement when it evolves. Unfortunately, developers modify schema-driven applications by editing the schemas. There is no other way, and there is no robust, scientific mechanism for identifying where every object has been defined and referenced. This traps developers in a manual maintenance exercise. For multiple schema families and multiple developers (or worse, multiple teams of developers), you have a very serious risk of conflicting modifications and inconsistency.

In any orchestrated set of web services used and maintained by multiple development teams, the externalized schemas and transformations describe or reference the same data objects over and over again. Schema families and their associated assets (transformations et al) present us with horrific redundancy and duplication when we try to evolve them by editing them. Modifying any object therefore presents the kind of maintenance nightmare that most of us try very hard to avoid in conventional programming environments. As with Y2K, the impact of change is difficult to predict and expensive to implement.

Metadata evolution
Managing the lifecycle of your web services development, particularly from the perspective of the evolution of metadata, is not simply a schema-versioning problem. Versioning schemas is about technical constructs and development processes, not about the management of metadata evolution. Metadata evolution management is the real problem facing the long term lifecycle management of web services development projects.

Metadata evolution management is not scalable with most current technology. To support active metadata, we need a new mix of technologies. An enterprise data dictionary platform is necessary to make all service-related metadata centrally visible to developers, wrapped in a development environment that conforms to a model-driven architecture.

Changes to metadata must be powerfully implemented in one central place (within a single-source model contained in the dictionary) and deployed out to the system via automated processes and generators, as one would expect of a model-driven development environment. The visible metadata for the community of consumers should appear as a strongly version-aware and variation-aware Enterprise Metadata Registry.

The following functionality and technology are necessary to support this concept:

  • Importers for loading existing metadata
  • Management tools for assimilating metadata into an integrated data model and dealing with redundancy and duplication
  • Design and development tools
  • Impact analysis
  • Change management
  • Fine-grained version control
  • Model-driven architecture
  • Central repository
  • Collaborative development across multiple teams
  • Release management

A technically sound, robust metadata evolution management strategy is the enabling differentiator for any modern SOA. Without such a radical new approach to managing metadata in a SOA, it is unlikely that the long term promise of flexibility and cost savings will be realized in any SOA.

More on this topic


Registry holds the keys to SOA success
How registry is implemented can spell the difference between success and failure ...

TLC to heal the IT/business rift
Let's put it bluntly the relationship between IT and business is often on the rocks ...


Copyright © 2002-2006, Procullux Media Ltd. All Rights Reserved.