The gorge between the X-volution and the real world

It is as if XML had burst a dam. The rate of new kernel technologies, new protocols, new world wide interfaces created at a rate of one per day in comparison to one per year in the former SGML times. XML will become the main WWW language within a few years. All of us will eat our hats if not.

But where is the rest? What is the situation outside, in the real world? What are the problems with publishing houses, technical documentation, training companies?

They are far behind us.

They get stuck with Windows 3.1. They talk about PDF versus SGML. They consider HTML as being very complicated. They implement intranets and look desperately for content afterwards. They evaluate tools instead of making concepts and defining goals. They seem to be purely interested in layout and they faint when seeing markup. They pay 20 people for searching and not finding the current version of their manuals but they consider $ 200.000 for a document management system as a personal attack. They are moving very slowly and often even backward. There is no awareness about document structures. There seems to be disgust at any disciplined approach towards content. There is a huge gap between technical possibilities and reality in the companies. What are the reasons?

It is one thing to develop a new technology. We X-voluzzers know very well how to do this. It is another story to spread the new ideas of information-oriented paradigms plus their consequences.

The situation in the X-World is discussed in many other presentations throughout this conference. This article tries to address the companies, the publishers and institutions which do not participate in this event, the reasons why they do not, and finally some hints what we could do to have them participate next year. Even if we will talk about reasons outside, this article takes the strict view that is up to us to act and not only to them.

There is a fictitious company producing a certain kind of machines for a broad market. The company has a long tradition and the structure of the company had one century to grow and to develop structures like an old oak tree. The documentation is spread over the whole company. There is a lot of documentation: the R&D diagrams and descriptions, drawings and lists for the construction, user manuals, installation manuals, maintenance advice, marketing brochures and many more. But the flow of the documents, the sharing of knowledge, the responsibilities are also spread all over the company.

For example the operating instructions are written by the department which as such is responsible for taking over and adapting standards; the maintenance manual is written by the department who does the training - with different tools, of course and with different demands. And if anybody wanted to get the newest information about the programming logic of an integrated circuit, the best condition would be to have a personal friend in the R&D department, because otherwise the information only gets spread when the first complaint comes from the market.

Many people in this company know about the redundancies and the problems with such chaotic structures, and there is even a very expensive SGML pilot project going on. But patient, brave and strong women and men are needed for setting out to battle. It is not technique that makes it hard. In our example the following circumstances can be observed:

1. The documentation is distributed over the whole company. So interests are fanned out in different spheres of influence up to the highest level of management.
2. Certain people responsible now for certain areas are not interested in a different solution, be it better or needed, except for the case that their territory expands.
3. Special skills and experience are needed for a strong documentation and good information flow in the industry. These skills tend to be incomplete in our fictitious example company. Most of the documentation people come from the production or R&D, being very good engineers, trainers or maintenance people. They had a lot of contacts with computing of course, but in a completely accidental way; some came over desktop publishing, some from Clipper, some had to learn Basic in order to compete with their children. So the likes and dislikes are very individual and widespread. Company regulations and interests are mostly not only disliked but seen as a real danger for these many mini-universes of limited knowledge.
4. Management does not even try to understand that it is not only an instructions-for-use leaflet but the whole knowledge infrastructure which is challenged. What are they doing? Management delegates the task to the leader of the computing department whose profile is: 56 years old, with thorough assembler language knowledge, drained and exhausted from the fight for mainframes and centralization, delegating all problems to the network administrators. The network administrators delegate the problem to external consultants who analyze the problem as one of communication and write a big report which becomes condensed to the decision to buy Lotus Notes and to install an intranet. Wow.
5. Iron rules are: New solutions cost $ 500 to $ 800 per seat, they have to work using Oracle and the server license is $ 10.000. The people responsible evaluate tools, ask for the budget, decide and install.
6. Methods, forms, tools and hardware can be changed. Structures are god-given; do not touch!

So there are lots of obstacles in our factory. Nonetheless they spent one million $ for an SGML introduction in one of the departments described above. And they are not happy:

1. In reality there is almost no choice of tools, even if an "SGML Buyer's Guide" talks us into thinking differently.
2. The prices for tools and systems are unusually high.
3. There are not many service providers (that is us!). No choice, no competition, lower quality, higher prices.
4. Some service providers did not put themselves in the customer's place but tried to talk him into so-called better solutions.
5. Design and training took place in an inconcrete and abstract form. The trainers did not learn the customer's language, so they could not communicate their content. And the students were afraid to ask, because they thought they would lose face.
6. One of the main complaints was that the providers were completely fixated on SGML solutions using their tools. But for a part of the problems other solutions turned out to be much better and much less costly.
7. Some parts of the solution were completely insufficient, some much too complicated, some simply not elegant and hence hard to maintain. A review showed, that the service provider was not experienced in SGML solutions. He had hoped to learn on the job, but he didn't due to the time pressure. So both parties lost.
8. There have been many variants of the DTDs; the first ones tended to be much too detailed. The service provider and the customer fell into the model-all-and-everything drunkenness, not asking anymore what would be needed ("who knows, we might need it in future" was the symptom of the disease). So quite expensive DTD revisions had to be made when they slept it off.
9. Some parts of the solution were problem oriented and not at all people oriented: so in certain areas the acceptance of the staff went deep into minus degrees; and not only for the reasons mentioned above.
10. And costs, costs, costs. The argument that changing a whole documentation environment is more than changing a tool only, was well understood by the customer. It is clearly seen that a clean and structured information base is a perfect starting point for the next step. But:

    They implemented different solutions for different purposes. So they saw quite well that a scanning solution for a TIFF based archive cost $ 10.000, a simple document management based on DTP tools and PDF distribution cost $50.000 and a SGML solution with DTD, prototype, pilot, training etc. cost $ 1.000.000, without a document management system . So they know that they might need SGML and all this infernal stuff, but they seriously ask whether and how it will pay off.

An again fictitious publishing house knows that an information revolution takes place. They are publishing journals and books.

They learned (from people like us) that the problem of the publisher today is that they do not own their data (but the typesetters do and neither the applications formats). Easy to understand, "so let us take possession of our data". Neutral format, long life: the SGML advantage is easy to argue.

A normal project takes place; the project diary in form of a brief outline:

1. Workshop and collection of objectives and requirements.
2. Analysis and rough design of processes and information flow.
3. DTD: Information analysis, what DTDs are available, do we need them differently, what changes are needed. Tests.
4. Legacy data: analysis, 90% quality conversion, thousands of manual corrections due to chaotic and changing source data.
5. Tools: description, requirements, evaluation, discussions, decision, adaptation to DTD, some programming.
6. Training: Internal editors cry about lost freedom. Pastoral care by the trainer. In the end: a sort of reluctant acceptance.
7. SGML doesn't print. Transformation to 3B2. Policy for making last minute corrections in typesetting and SGML data. Let us try DSSSL. Typesetting quality too low. Back to 3B2.
8. Connection with authors: test with SGML editor. No acceptance. Conversion from Word. See item 4.
9. Changes: E.g. new content element requires new DTD elements (side-heads). Change conversions, transformations, applications, documentations. Inform people.
10. After all the total cost of the solution adds up to half a million $.
11. Building up a web-site from the data including tables of contents, links and indices: a week of OmniMark programming and the website is generated in batch, error free. Proof of concept.
12. Typesetting company gets bankrupt. Data is sent to the next one. Some adaptations needed then printing can continue. Proof of concept.

The publisher is more or less satisfied. But the price has been very high in terms of time and costs for all kinds of tools and services, for acceptance and for arguing. They would like to know whether it pays off.

We, the X-voluzzers, are confronted with the following characteristics of our customers, the ones most of us get our income from:

1. The contact person is technical and
   • low in hierarchy
   • left alone by the management
   • has work enough for a 48-hour-day
   • fears to lose power
   • got stuck on tools
   • likes only tools written in C by herself
2. Missing: authors / editors
3. The contact person is managerial and
   • thinks quarterly
   • does not understand the concept
   • does not know the problem
   • is afraid of leaving the company traditions behind
   • does not want to take any risk
   • delegates everything
4. Acceptance problems
   • The management decided for a new system or a new infrastructure. But not the personnel.
   • The contact person pretends to agree but the editors, the writers, the authors, the engineers have not been asked and will not accept the new tools and the new way of thinking without being wooed.
5. Requirements not settled
   • There is no transparency at all concerning the documentation processes, so there is only a vague knowledge about their cost in money, time and quality.
   • The new technology is reduced to a question of tools and techniques
6. The electronic way of documentation is set up in parallel to the existing ways and not integrated in a common business policy.
7. Different solutions seem to be better to understand, less expensive
   • The structured approach is very good but tools and adaptation are far too expensive.
   • Others told to the customer that the future of spreading information lies in PDF and that XML is too expensive.
   • WYSIWYG.
8. The necessity for the proposed solution is fully understood, the budget however bears only 10% of the needed cost.

This paper does explicitely not focus on areas where XML is used for interfaces and protocols, but on the traditional areas of publishing and technical documentation.

There are many highlights and stunning applications of XML. And it is the serious opinion of the author that XML is the tool to choose for any information content. SThus, no heresy!

But the overall picture is that most of the potential XML users either never have heard of it, or are just beginning to brood about new approaches for their content or are in the beginning of XML experiments and mostly frustrated. Many of the reasons can be extracted from the examples above. The most important ones might be collected:

1. There is no real knowledge and no real model for "the new information" to say nothing about applications. We still got stuck in the sequential document paradigm. The authors don't write non-linear and there are almost no findings about the reception situation at the reader's end. That means we are talking about modular content and about CALS class 4, but only a few if anybody fills it with concrete life.
2. Not much comes from the universities, which is especially true for the main part of Europe: no research, no results, no tools, not even well-trained students in the areas of documentation, hypertext or XML. So the training and the research and development are paid for by the customers which has two effects: on the one hand projects tend to become very expensive and on the other hand good results cannot be made public. Yes, there are James Clark, the TEI and some more initiatives donating tools and techniques to the world. XML would not be what it is if we did not have these benefactors. But it is not enough: the lion's share of the R&D is paid for by projects of private companies.
3. The latter item is part of the reason for the tremendous cost in every perspective of even old-fashioned XML applications. Consulting, modeling, licenses: everything costs a lot, nearly nothing is for free. Sales are low in this area, development is all but easy - not even having XML replacing the baroque SGML -, a developer in a XML focused company needs skills in many areas, so prices are fair. But high.

   The spiral is spinning: the interest in good information management solutions might be high but the paying interest is low; low sales, high prices, no competitors, less quality, some almost-monopolies, high prices etc., you know the game.

   Even hardware can become an argument in this context! There are banks and insurances still using thousands of Windows 3.1 computers.

4. The acceptance for XML solutions tends to be alarmingly low in all the levels where information is produced and maintained. There are many different reasons: indolence, lack of understanding and sympathy, poor training, good or bad habits, refusal of the new, loss of layout freedom, tools too abstract, devaluation of existing skills, intellectually much higher demands due to layout vs. semantics, and there might be more.
   It can be observed much too frequently that am XML or SGML based solution has been installed but becomes obstructed or even counteracted by the ones who have to fill in the content. This may occur even when the people have been involved in planning, modeling and testing of the solution: The XML people threw a lot of very strong arguments on them. So the heads had to nod, but the hearts didn't.

Territories, delegation as an escape from indecisiveness, not enough skills - all the facts we encounter with our customers: don't worry, it'll sort itself out? Wrong!

Yes, we X-voluzzers know feasible ways. Yes, we have developed a sort of talent for a more abstract approach to the problems, because we are not buried in the daily chaos of our customers. Yes, we know the key techniques and tools to build information-based organizations. But we do not translate them well. What we are heading for is more than a method or a tool. 10 years ago Peter F. Drucker anticipated the "information-based organization":

Twenty years from now, the typical large business will have half the levels of management and one-third the managers of its counterpart today. Work will be done by specialists brought together in task forces that cut across traditional departments. Coordination and control will depend largely on employees' willingness to discipline themselves. [Peter F. Drucker, The Coming of the New Organization. In: Harvard Business Review on Knowledge Management, 1998, pp. 21-46]

He might be right - except for the period of time. Drucker might have underestimated the violent inertia of organizations to keep things unchanged in order to keep the status quo and the power.

People do not mind if things going on are called information revolution. Oddly enough in the middle of capitalism they keep calm when they are accused of participating in a revolution. Some might think that it is a revolution of information only, a thing, not touching them. But some might have a presentiment that this revolution will not take place so quietly. These hidden counter-revolutionaries are one of the reasons why the job might be so hard for us from the outside and for the ones inside the companies who feel like prophets. But this is only part of the truth.

We need to see ourselves as being only a part of a huge process which builds new structures into all companies. The X-volution is not an end in itself. Most of us need a picture of the incredibly huge organizational change which is in its beginning, which is speeding up very slowly, which costs billions of $, which costs (hopefully no blood, but:) sweat and tears. The times of crusaders have passed, so we must not use our swords, but arguments and powers of persuasion. We need to position our techniques and tools in a network of arguments and working solutions. We don't have to be believed in, but we have to be understood. This is why the word XML community is so bad! We have to understand ourselves as craftsmen and service providers. Our artifacts are viable only in a much bigger context. And this is where we have failed for 15 years now.

What can we do about it? In this paper some reasons are mentioned for the gorge between the front technologies and the Real World far behind. But how to build the bridge?

We can not change management structures, educate specialists for the industries and thin out the fat middle management layers. But we have got enough opportunities to span the gorge.

1. The topics we talk about are very abstract and intellectual. Topic maps, architectural forms, groves, meta languages: our science is complex. And some of us seem to feel good in that complexity and do not make the least effort to make themselves understandable to colleagues and, more detrimental, to customers.
   So the first advice would be to describe what we are doing for the people. Examples, calculations, figures, real texts and implementations and again examples. Do not talk about multi-directional links with role attributes and multi-layered addressing schemas for touchless indirection: show them. Present an example which proves why and what for the customer should spend hundreds of thousands of dollars for the implementation and millions for the creation and maintenance. Show him what it will look like, - more or less. Come down from the academic ladder, come down to the engineer and the boss of the computing department in a factory who have nothing else but the feeling that something goes wrong with the documents on the second floor.
   One of the open sesames is: prototype. Show results, mock them up at will but show the practical end. Don't talk about theory. Our tools and methods do have purposes. Show these and not those.
   By the way: you will notice that some of our approaches sound good, but are either not sufficient (architectural forms are my favorite example here) or are not feasible, or at least incredibly complicated. For the latter a short example: think about modular documentation with conditional texts (the author has to think about modules in some possible combinations), which I call aspects, where different readings for different readers (countries, skill levels, perspectives) are coded. Sounds reasonable, is needed in almost any technical documentation and the techniques are all but trivial, but solvable. Try to write an example which demonstrates the power of such an approach! The author needs miraculous powers to write even ten pages of such a multi-purpose document. Trying to make it happen will open our eyes for additional support needed or for alternative solutions.
   Our vis-á-vis are clever people and they get paid for being it. So they will not tell you that they did not understand your proposals and expositions and their consequences. But in most cases we are talking about a completely new infrastructure for everything which has to do with information. Nobody can grab it during a one day discussion. We have to build the model so that it can be really experienced.
   The advice is: come down to earth, be practical, show and prove what you propose, make it tangible.
2. Related to this item is the second one: Put yourselve in their position. It seems to be the main disease of the whole computing society, that they tend not to care about the user. This is bad anywhere and so it is in our profession.
   Usually we have to do with specific areas in specific industries or in publishing houses. We do not need a doctor's title in their field, but we have to begin any job with learning parts of their language. We have to understand the matters they care about, we have to understand their processes, their requirements. We have to accept their fears, to know about their staff, to consider the history of the problem and have the firmness to solve it. We need to feel where the conflicts of interest will be and how we could try to harmonize.
   My claim is: we do it insufficiently if at all. Of course we are asking our standard questions, but then a short thinking which drawer to use, and here it is, the way to solve the problem as others did before. In principle we know the main part of our solutions beforehand. We are bad role players. We are gurus. Gurus don't ask. Gurus rule.
   The second advice is to sharpen our sociological, educational, psychological and human skills. First to understand the people, then the facts, then the problem.
3. Because it is so important, it gets an item on its own: take care about acceptance from the very beginning and long after the installation.
   You might have to take many different measures for this purpose, from pastoral care to brute force, from arguments to tricks. Beside of badly performed projects missing acceptance is the main reason for failures of SGML or XML introductions.
   We need the acceptance of the project owners, of course. We need the acceptance of all the people who are connected with the new system. For many of them things change more or less dramatically when new information paradigms are to be installed. Take the utmost care about their reactions. Take great pains over monitoring the mood and the usage of the new system; react to every complaint; repair immediately when possible. Train them well. Keep in contact. And, coming to the brute force, if notorious grumblers remain: transfer them out of sight (and make sure in the beginning that such measures can be taken).
   Third: get and keep acceptance.
4. Take the blinders off: there are solutions outside. The world is sort of an ordered chaos and not every problem is solved best with an emphasis on the ordered part. There are problems which are solved best by using Word. There are good reasons for PDF in some contexts. It might be best to build up an archive of scanned TIFF images under certain conditions. A purely database based solution with a straight output to HTML can be the best.
   The fourth advice is to be open-minded. A standard is no end in itself. XML is nothing but one of many tools.
5. Never stop learning. Most of the bad solutions I have seen could be explained immediately: the skills of the designers and developers had been too narrow. In our profession good people need a background in some programming languages, database modeling, project management, workflow and business process analysis, information modeling, SGML tools and adaptation methods, string handling and conversion, tree transformation, browser technologies and development (e.g., HTML, DHTML, Javascript and CSS), typography and web site design. In their former lives they should have been psychologists, teachers, sociologists and priests. This is the superman our customer needs. Try to become it.
   Be honest, modest and reliable. There are some shamans in our profession pretending to have a lot of experience when they have learned that an angle bracket may have a special meaning. They damage themselves, the customers and us.
   This fifth advice is: you are in a very complex environment. You need many skills. It is a pleasure never to stop learning. And a fundamental condition for our business.