In section 2 we elaborate the properties of a nearly ideal tactile input/output tablet. Section 3 outlines the state of the art of tactile aids for blind computer users. Section 4 describes the structure and function of our tablet. Section 5 deals with the design of an appropriate controller. In section 6 we show how the tablet may be integrated into a personal computer system and section 7 outlines some aspects of the needed controller software. Section 8 gives some examples of appropriate applications of our tablet.
2. Properties of an nearly ideal tablet Usually the tactile pixels of an output tablet are arranged in such a way that they match the dots of a rectangular raster. A multiple of the distance between neighbouring pixels should be about 1/10 inch (2.54 mm) in order to coincide with the usual distance between Braillepoints, allowing to display Braille characters. On the other hand, the resolution of the output tablet should be less than 1/10 inch to allow the sensation of a continous line if each pixel on this line is in the on-state [1].The largest distance satisfying these conditions is 1/20 inch. This value is not ideal, because strokes in horizontal and vertical directions are felt as uninterrupted lines, while strokes with an inclination of 45 degrees are recognized as lines of isolated points.
With lines not matching these directions there is the same problem as with lowresolution screens for sighted users: these lines are broken into stairs and sometimes the user cannot decide if the steps are original and meaningful or just an effect of the low resolution. So a distance of 1/30 inch between pixels would be preferable, because steps of this height are beyond the power of tactualperception.
Displaying Braille characters, another advantage of 30 dpi in comparison with 20 dpi becomes evident. The 10 dpi resolution equals usual Braille print outs. With 20 dpi a single pixel has to serve as a character dot, while with 30 dpi four pixels form a character dot felt as one entity due to the coarseness of tactual perception.
Therefore Braille characters on a 30 dpi display are hardly distinguishable from usual print outs, while one has to get accustomed with the smaller 20 dpi dots.
The dimensions of the whole display should allow 40 characters per line, at least. If the width of a Braille character plus the space between two characters is 5/20 inch, then a 40 character line has a length of 10 inches (254 mm). In some applications it would be useful to have a tactile presentation of the usualscreen with 25 lines of 80 characters. In this case, the width of the tablet has to be 20 inches (508 mm). If characters with eight points are used and the vertical spacing is 3/10 inch, then a line plus spacing takes 6/10 inch and 25 lines take 15 inches. Having an active area of 20 by 15 inches (508x381 mm), i.e. 400x300 dots with a distance of 1/20 inch, the tablet has also the usual format of high resolution screens with a width to height ratio of 4/3.We will explain in section 8 that it should be possible to present the structure of a whole page of ink print in order to separate the drawings from the text. In most cases this could be achieved without the need for any magnification\. So, from this point of view the lower limits for the output tablet would be theusual paper format of 8 by 11 or by 12 inches.
For presenting graphic information, on the first view it seems as if the statement "the larger the tablet the better the output" might be right. However, there are two reasons for limiting the dimensions of the tablet: First, on a very large area the blind users would find isolated elements only with great effort;while on a tablet not significantly larger than the area covered by two hands,a search for objects would take about one second, on a very large tablet blinds have to search in a systematic way by partitioning the area and searching sector by sector. Second, and worse, working at the upper regions or at regions far left or right from the center of the tablet would cause a wearisome position of the body and the arms of the user. The above mentioned area of 20 inches width and 15 inches height seems to be an upper limit forced by this aspect.It should be possible to change the state of each pixel of the tablet without influencing the other pixels. For example, a cursor should be moved over the tablet without the need for resetting and setting other pixels than those coveredby the cursor. The clearing of the whole tablet and the building of a new picture covering the whole area should take so little time that the user does not have to wait for it.
The user should be able to make inputs immediately on the display area by his fingers. This facility eases and accelerates the usual nonsymbolic inputs. Moreover, blinds could make freehand drawings and could input instructions by making gestures [5]. By this method or by using tablet menus the user avoids the need for frequent changes between the tablet and a keyboard. An additional advantage is the possibility to record the users actions while he is reading the contents of the tablet. A sufficiently intelligent program would be able in some cases to detect by this means the needs of the user without waiting for commands. An example without the need for real intelligence is reading text illustrated by drawings: internally, there could be set a pointer from the text "Fig. 7" to the appropriate stored drawing. Touching the word "Fig. 7" could cause the presentation of Fig. 7 in a graphic window.The housing of the tablet should be flat and lightweighted. Small tablets should be transportable in order to use them in connection with a laptop, for example.
In contrast to this approach, a German company has developed a tablet comprising more than 7000 pins (119x59) with a resolution of about 8 dpi, i.e. with a dot distance of 0.12 inch. This tablet, used in a research projec t at the University of Stuttgart, enables perception with both hands. Additionally, it has an input facility: the user may move two ring-shaped sensors over the tablet surface [5]. (This tablet is the only available device similar to the graphic tabletdescribed in section 2. However, the resolution is too coarse and the price too high.)Technically, a tablet with the features described in section 2 could be constructed. The only true problem is the expense. An area of 20 by 15 inches with a resolution of 20 dpi comprises 120.000 display elements. It is practically impossible to construct an electronic controller with 120.000 outputs and to connect each display element to its own output. So there must be some kind of addressing by coincidence, and elements with at least two stable states have to be used. In this case, the controller for an m by n array of display elements must have just m + n outputs like the address decoder of a silicon memory device. In the above introduced example, m + n equals 700, in contrast to m * n = 120.000. For this reason, conventional electromagnetic or electrostrictive elements could not be used immediately but have to be supplemented by some mechanism in order to freeze their state. Additionally, the diameter of usual actuators is larger than 1/20 inch, forcing the mounting of the actuators in various levels. Summarizing, even if we do not take into account the price of the basic actuators, supplementing them with memory, mounting them, and electrically connecting themwould be too expensive.
Display elements using memory metals could have a sufficient small diameter tobe mounted all at the same level but they are commercially not available precision devices and could be produced only at high costs. With the electrotactile method, no better resolution than 1/10 inch could be achieved. Additionally, there is no way to have an inherent memory in these elements.
4. A tablet driven by an electrorheological fluid Reducing the costs of a tactile display is achieved most effectively by lowering the costs caused by producing and installing the mechanisms which settle and lift the tactile dots of thetablet. To reach this goal, we have developed very simple display elements driven by an electrorheological fluid (ERF) and integrated in the tablet. The termelectrorheological means that the viscosity of the fluid increases and even a semisolid state is reached if an electric field is applied to the fluid. This property enables the construction of electrically controlled valves for ERF, essentially being a channel with two electrodes inside at opposite positions on the wall [6]. Such valves have no movable mechanical elements and they may be constructed with small diameter and at low cost. In order to overcome difficultiesdirectly related to the properties of the ERF we are coopera ting with the German chemistry company Bayer AG.To illustrate the basic structure and function of a tablet with ERF- driven display elements, a description of an embodiment follows.
The tablet basically comprises a board made out of nonconductive material withan array of circular channels which are closed at the upper side by an elasticmembrane. This membrane is pre-formed to have one "pimple" over each channel and acts as the tactile element.
Inside each channel there are two metallic areas in order to form a valve. These electrodes are connected to row collection lines and to column collection lines, respectively, in such a way that in each channel one electrode is connected to a row line and the other electrode to a column line. The space under the board and inside the channels is filled with ERF. Pressure is exerted to this ERF either directly by means of a fluid pump or indirectly by means of an air pump and a membrane in order to separate the ERF from the air.The collection lines are switched to ground potential or to one of two AC voltage sources by electronic switches. The outputs of both voltage sources deliversquarewave signals shifted by 180 degrees to each other. The amplitude dependson the dimensions of the valves and on the properties of the fluid. In our prototype, the amplitude is 1000 V. The value of the voltage, the value of the pressures, and the size of the valves are tuned in such a way that each valve is completely closed if one electrode is connected to a voltage source.Switching the other electrode between the other voltage source and ground potential leaves the valve in the closed state.
5. Controller architecture and interface In the last section we showed how thedots of the tactile tablet are addressed in an x/y manner by analog switches. These switches are controlled by digital signals resulting in ground potential at the electrodes for a logical "0" and AC voltage for a "1". The switches attached to the row collection lines are controlled by a 240 stage shift register, the switches at the column collection lines by a 320 stage shift register.
In order to clear one or several screen rows and to write new information first the whole column shift register and the desired bits in the row shift register are reset to "0", all other bits set to "1". Sub-atmospheric pressure of the pump clears all dots of the selected rows. Then the dot information (x-data) isshifted in the column register. High pressure of the pump now lifts up every dot with a "0" in its column and row latches. Two special signals control the activation of the pump and determine its flow directions (sub- atmospheric/high pressure).By activating more than one row at a time, i.e. by putting more than one "0" into the row latches, it is possible to write the same information in a number of lines. Similar control operations allow the setting and resetting of a singledot, of only a part of one line, or of a rectangular clipping area of the screen without addressing the rest of the tablet. Obviously, it is also possible towrite the screen in a column by column way, preferably if an application mainly draws vertical lines.
As already mentioned, the tablet can be used as an input medium. This is achieved by the following technical mechanisms: in idle moments when no further information is available for the screen all rows and columns are scanned sequentially by shifting a single "0" through the row and column registers, respectively.This scanned signal (C) is coupled in a capacitive manner to a sensor, which is fixed to the fingertip of the user. Based on this signal the controller is able to determine the actual position of the finger - comparable to the wellknownlight pens.In order to protect the controller from high voltage, all signals between the controller and the tablet are transmitted using opto couplers.
The controller of the tablet consists of a standard microprocessor with local RAM and ROM and a lot of dedicated components. In the emulation mode (see section 6), in which the controller works in parallel to the video adapter, the ROM of the controller CPU contains specific emulation software. The graphics memoryof the tablet controller, which consists of 1 to 8 memory segments, is filled in parallel to the graphics memory of the emulated controller. The control registers of the latter are copied into the communication registers of the tablet controller and are interpreted by the emulation software. In the slave mode (seesection 6), in which the controller is explicitly addressed and managed by thehost processor, usage of the communication registers depends on the agreement between the controller software and the application program.On demand by the CPU the contents of the graphics memory are transported to the RAM by a DMA controller. There it is transformed into a form suitable for thetactile tablet and stored into the screen memory. This memory consists of 8 planes, each of which can hold the information for the whole tablet. In this way it is possible e.g. to capture 8 program outputs following each other too quickly to be analyzed by the blind user in "real" time or to present 8 "windows", which for the sighted user may be displayed (partially) overlapping. The blind user is able to select and to study one of the planes after the other. If the whole tablet or a larger part of it is to be changed the DMA controller will bring the desired section of the appropriate plane to the tablet interface control logic, whereas smaller amounts of information, like single dots or limited rectangles of dots, may be transported by the CPU itself.A dedicated module generates the control and data signals for the row and column shift registers. Furthermore it is responsible for the presentation of some kind of cursor at the current position of the tablet surface. To ease finding this position dot marks are placed at the beginning of the corresponding row andat the bottom of the column. In the input mode, as explained above, the screen/cursor control module generates the signals for scanning the tablet. The actual position of the user's fingertip is determined by means of the just selected x/y coordinates and the capacitively sensored scan signal.Another extension of the controller is a so called frame grabber, which deducts the binary information out of a video/ monitor signal and stores it into its own video memory. This information can be transported by the DMA controller to the RAM, and there the CPU may transform it into a form suitable to display on the tablet.
A pump control port delivers the signals by means of which the CPU can influence the operation of the tablet pump. Last not least the controller contains a module for speech and sound output to support the user by spoken messages or acoustic signals.
- to adapt the graphic screen informations of various graphic standards (like EGA, VGA, etc.) to a form suitable for the ERF screen, i.e. to the minor dot resolution and to the capability of a blind user to interpret graphic el ements.
On the third layer there are all the programs to emulate the interfaces and register models of standard graphic adapters and, of course, the software to use the screen as an input/ output tablet in slave mode, as mentioned above.
The outer layer comprises software that is partially running on the host CPU or even on a different PC. Its use may be restricted to the sighted user. It delivers
- support to transform standard or non standard application software int o a form suitable for the blind user, e.g. translating dialogs into Braille, removingoverlapping windows etc., - routines to analyse images in order to recognize objects on different logical levels,
- routines to transform and composite the recognized objects into images adapted to the tactile screen and the capabilities of the blind user.
Presenting each page completely on a tactile screen as a first step - obviously not in a readable way - would allow the user to identify pictures and their captions. The pictures and captions marked by the user would be separated from the text, and the pictures would be stored as pixel files with the caption or parts of it as keywords for associative access. There is already a research system supporting the document segmentation interactively by means of tactile paper print outs laid on a conventional input tablet [7]. Obviously, using a tactile tablet will ease and accelerate this task. Furthermore, instead of presenting the raw document image, each page could be segmented automatically and presentedas a set of labeled blocks on the tablet [8].Based on this segmentation, the character recognition process works faster on the pure text blocks. Reading this text translated to Braille characters the user could mark a reference to a drawing. The corresponding picture file is then accessed using the marked reference as a keyword, the picture is preprocessed to match the user's cognitive abilities and is presented on the tactile tablet.
The task of image preprocessing includes recognition of characters, filling patterns, and shaded or coloured areas. Usually, single characters should be translated and presented while strings should be replaced by a special symbol and presented in a text area outside the picture, if the corresponding symbol is touched. Some kinds of filling patterns should be replaced by patterns with bettertactile properties. In some cases, shaded areas should be replaced by their border lines or even removed. The user may zoom the picture and may set parameters in order to remove, replace, and present elements with certain properties.The user can select an appropriate partition of the tablet defining areas for graphics, the text to be read, and system messages. For example, there could befive lines of text at the bottom and the rest of the tablet could be used for graphic presentations. These five lines could be split in one line for a caption, three lines for text to be read or legends of figures, and one line for messages and user inputs.
- Reading of text/graphic-mixtures provided on paperless storages and not prepared for blinds:
If the document is generated by a word processing program as ASCII text with included pictures stored in separate files, there is no need for separating the pictures from the text but the system should provide the same functions as mentioned above. The graphic tablet eases the perception of graphics in the same way.
- Reading of text/graphic-mixtures provided on paperless storages and preparedfor blinds:
There should be no need for image preprocessing, but functions like zooming could be helpful adapting the presentation to the individual needs of the user. Even if there is no need to manipulate the pictures at all, the perception of drawings will be easier as with conventional devices, because the blind may use more than one fingertip to explore the presentation.
- Reading and writing mathematical formulas: Presenting mathematical relationson a Braille line forces the translation from the two!dimensional notation forsighted people (realize indices, limits, and arrays, for example) to an onedimensional representation. Complex formulas presented as a string of normal and control characters are nearly unreadable.
Alternatively, mathematics could be presented in a twodimensional form called "dots+" similar to the notation for sighted people [9].
There exists an approach to translate automatically textfiles created by meansof usual word processors for scientific texts into a dots+ layout for tactile print outs. With the help of a tactile tablet, blind scientists could type their mathematical texts using a conventional word processor and revise the layout on the tablet immediately. In this way, texts written by blind and by sighted people could be exchanged without effort.
- Creating drawings:
Obviously, the blind will never work easily with graphics like complex drawings of machinery or threedimensional pictures, irrespectively of the kind of support. On the other hand, blind people are able to create exact drawings on a computer, as shown in a research project at the University of Karlsruhe [10]. Additionally, it could be helpful to make a sketch in some situations in order to document an idea, for example. Drawing and sketching is well supported by a combined input/output tablet, because the objects are perceived and manipulated on the same surface, avoiding the roundabout way to move a cur!sor with an extra input device and to follow this cursor with the other hand on the output device.- Participating in scientific/engineering education: In experiments within theframework of the scientific education of blind pupils or students, the states of some instruments may become perceptable by means of acoustic signals, for example. Complex signals, presented to sighted people in a graphic way, could be presented to the blind on tactile copies in a static manner. However, because the best educational results could be achieved by involving the pupils or students in the experiment, dynamic displays presenting the current signal are preferable. For example, in order to enable the blind to vary some parameters of an electronic circuit and to get the results immediately, the recorded data provided by digital oscilloscopes could be adapted by a personal computer and displayed on a tactile tablet. Alternatively, a video camera could be positioned in front of the oscilloscope and the tactile graphic tablet could present the camera signal by means of the above mentioned frame grabber (see section 5).- Using electronic communication media: At the University of Stuttgart the tablet mentioned in section 3 with more than 7000 tactile pins is used to enable the blind to get access to a German communication service called "BTX", providing various informations as well as input facilities for ordering of goods and services or for money transfer, for example [3]. Sighted people are using a TV screen as an output device. The character set of BTX comprises graphic elements and is presented in colour. We suppose that using a tablet with high resolution would be of advantage, because graphic elements could be displayed at more detail and certain objects of different colours could be better filled with different filling patterns, for example.Another communication service supported by graphic tablets is telecopying (telefax). If the blind telecopy subscriber is using a personal computer as a receiver, then the process of reading a received page is identical to reading a scanned page of ink print text. If text illustrated with drawings should be sent, then the problem is the same as with creating drawings (see above).
- Using software not adapted to blind users: If the dialog between the user and the computer system is only text based and is organized line by line, then nographic tablet is needed. If the interface to the user is text based but is using pull down menus, then a large tablet would be more advantageous than a single line Braille output, because on the tablet the user is able to search for menu items while pointing to another area of interest with the other hand, and ifthe tablet is showing the whole monitor screen, then the menu items would be always at the same place relative to the tablet.Most of the software developed in the last few years provides graphic user interfaces (GUIs) with windows of variable positions and sizes, and icons instead of menu items or a command line. Even the characters are not coded in ASCII or in a similar code inside the video memory, because the video adapter is workingin graphic mode and the characters have to be present as patterns of pixels.
Obviously, the solution of this problem cannot be to present just the whole screen to the blind user nor to hide the whole structure of the screen from the user by using a single line Braille output.
As the sighted user, the blind should be able to get an impression of the whole screen, to manipulate the windows, and to select commands by marking items. On the other hand, he needs some additional support: replacing icons by strings,splitting the screen instead of displaying overlapping windows, and presentinginformation about the original windows, for example. In principle, there are three ways to reach these goals:
1. Adapting commonly used application software to the need of the blind. This should be easy with well described software, especially with the support by thesoftware developer. But it has to be done with each different software package.
2. Extending widely used operation systems with an interface providing all information about the contents of the screen in an abstract form (e.g. Off- Screen Model for OS2 [11]) and adapting this information. In this way, all software not bypassing the operation system can be accessed.
3. Providing an application independent system for image recognition and imageunderstanding in order to translate arbitrary GUIs into a user interface adapted to the blind. This approach would solve the GUI problem in general but givesrise to some severe difficulties.
An urgent problem arises from the increasing number of software packages with graphic user interfaces: while development and propagation of computer systems in the last two decades have created many workplaces appropriate for blind people in the areas of text processing, programming, and database management, now there is the risk to lose these achievements [12]. The blind employee has to usethe same software as his sighted colleagues in order to be able to exchange data and to be really integrated in a team. So if his colleagues get new softwarewith a graphic user interface, then he has to work with that software, too, orelse he will be disintegrated after a short time. On the other hand, if the blind's performance is considerably decreased in consequence of the problems arising from the use of graphics, then no one will let him do relevant jobs.- Reading mobility maps:
The tablet surface of up to 20 by 15 inches is easy to explore, and arbitrary portions of the map could be presented in different states of refinement. The presentation is not confused by text mixed with the graphic elements, but touching any object would cause the objects name and additional hints to be displayedin a separate text window.
On the other hand, searching and marking an object name in the text window would cause the appearence or emphasis of this object in the graphic window. So there are some advantages in comparison with static maps.
Graphic input/output tablets could enable the blind to participate in va rious kinds of information exchange increasingly based on graphics.
The main problem to be faced in future will be how to make graphics suitable for a blind person with as little intervention of the blind as possible. In order to reach this goal, beyond image preprocessing and recognition of characters as well as graphical symbols we will need some kind of knowledge based image understanding.
ERF-driven tablets seem to be the best realizable solution today and there will be improvements in consequence of developements motivated by the largescale industrial enterprise. A rough calculation lets us expect the accrued costs for one dot of our ERF-driven tablet to be only 5% of those of a piezo-driven pin. We think that our final 320x240 dot tablet will take less than twice the price of commercially available Braille lines with 80 characters, i.e. 640 dots.
At present, we are realizing a tablet prototype with a size of 16x16 dots in order to study the functionality, the reliability, and problems related to the cost efficient production, e.g. choice of appropriate materials and processes. Simultaneously, we are concerned with the construction of a simplified controller which will allow us to use the tablet as a slave input/ output device, as mentioned in section 6. In more theoretical efforts we are studying problems caused by the outer layer operating software.10. References
[1] anonymous 1990. Entwicklung eines grossflaechigen und hochaufloesenden taktilen Displays. In Zwischenbericht 1988/1989 des Modellversuchs "Informatik fuer Blinde", pp. 45-48. Karlsruhe: University of Karlsruhe.
[2] L. Kay 1984. Electronic Aids for Blind Persons: An Interdisciplinary Subject. In IEE Proceedings, Vol. 131, Pt. A, No. 7, pp. 559-576.
[3] W. Schweikhardt 1985. Bildschirmtext - ein rechnerunterstuetztes Kommunikationsmittel auch fuer Blinde. GI/OCG/OEGI-Jahrestagung. Heidelberg: Springer Verlag.
[4] H. Bruemmer 1987. Untersuchung von elektrotaktilen Schrift- und Bildtastverfahren fuer Blinde. Duesseldorf: VDI Verlag.
[5] G. Weber 1989. Reading and Pointing - Modes of Interaction for Blind Users. In Information Processing 89. Elsevier Science Publishers B.V.
[6] D.J. Peel, W.A. Bullough 1990. Miscellaneous Electro-Rheological Phenoma, Part III. In Proceedings of the 2nd International Conference on ER Fluids, pp. 141-157. Lancaster: Technomic Publishing Company.
[7] U. Mueller, M. Schenkel 1992. Improving Reading Machines for the Blind with Interactive Document Segmentation. In Computers for Handicapped Persons, Proceedings of the 3rd International Conference, pp. 363-372. Vienna: R. Oldenbourg.
[8] G. Nady, S. Seth, M. Viswanathan 1992. A Prototype Document Image AnalysisSystem for Technical Journals. In IEEE Computer 7/92, pp. 10- 22. Los Alamitos: IEEE.
[9] J.A. Gardner 1992. Personal communication. Oregon (USA): Oregon State University.
[10] C. Hessmann 1988. Parkett - Ein blindengerechtes Graphiksystem zur Erzeugung von Parkettbausteinen. Master Thesis. Karlsruhe: University of Karlsruhe.
[11] M. Emerson, D. Jameson, G. Pike, R. Schwerdtfeger, J. Thatcher 1991. GUI Access by Blind Computer Users. New York: IBM Thomas Watson Research Center.
[12] R. Heuer gen. Hallmann 1991. Des einen Freud ist des anderen Leid - Graphik-Einsatz am PC. In Horus 3/91, pp. 114-117. Marburg: DVBS.
Authors' Full Postal Address:
Joerg Fricke, Helmut Baehring