Procedures were created to allow a user to write command files that could be exectuted as single commands. These user-written procedures are executed in many cases by simply entering the command:-
:> XP PROCEDURE.PRC
Here, the `XP' command (execute procedure) is followed by the name of the file that contains the procedure: the suffix, `.PRC' is provided by default if an extension is omitted. The MDPP looks first in the current directory for the file. If it is not found it then looks in the directory pointed to by the logical `MDPPRC' which, unless the user re-defines it, is defined to point to the default procedure library in the MDPP.COM command procedure that all users execute before using the program. Note that a procedure may call another procedure if it exits: a call to another procedure terminates the current procedure.
Procedures are simpler structures than `macros', as implemented by the macro processor. They have the benefit that the user can enter information into them interactively (not possible with a macro) and can customize the output from the procedure so that they have the `look and feel' of a MDPP command. Finally, they have controll elements that allow them to start a remote compute-server, if one is available, to do the bulk of the computing for a particular task.
Possibly the most important benefit of procedures is that they are used by the MENU driver to execute processes. Thus creating a robust set of procedures to be used as tools in user's menus is an important thing to be able to do.
Procedures are text files created with your favorite text editor. A procedure file contains procedure commands, MDP commands and comments.
Comments
Comments can be inserted into a procedure file at any point. There are two comment fields that are of most use:-
`++' is the so-called `dead-line' comment. Lines proceded by this symbol are completely ignored. Put these lines anywhere into the file liberal and clarify the code.
`+-' is the so-called `in-line' comment. In-line comments are written to the screen. They can therefore be used to provide instructions to the user as to how to input information to the procedure. Use these liberally also. Note that an in-line comment is only seen and written out the the screen when the MDPP reads an input line from the procedure file (answers the question: `Why don't my cute messages get written out before it starts to do...').
Procedure Commands
There are several special procedure commands:
++$PRM=<prompt_string>
<prompt_string> is the prompt string returned to the screen. The user can enter up to 8 parameters, which are available as output registers 1 --> 8. The user should write the procedure to copy these over into registers where they can be used later: in general these parameters will be overwritten by the MDP commands which follow so if you have entered any parameters the FIRST commands in your procedure should be ones that copy these to a safe place.
++$MSK=<mask string>
The <mask string> (e.g. 00001100100111000) controls output to the screen/log. If `0' is present the line is not printed, if `1' is present that line is printed. Setting up the mask may prove to be difficult. The method most likely to work is to write the procedure using a mask consisting of all 1's (print every line), and to log the output: note that some lines printed by the regular command will not print if masking is on because they can generate variable ammounts of output (e.g. lists of input points). Take the log and write 0's in front of the lines which are not wanted (e.g. prompt strings for paramters provided explicitly or by default, or echos of these input values), and 1's in front of those wanted. The vertical line of 0's and 1's will be the mask. Now install the mask and re-run the procedure. For the most part the masking will be OK, however, `frame-shift' errors can crop up. Simply insert 0's to bring the mask into allignment. The `errors' are due to the presence of non-printing prompts which `eat' masking flags.
++$SCN
This flag will cause an error to occur if the procedure is run when MDP IS NOT in screen-management mode.
++$PRT
This flag will cause an error to occur if the procedure is run when MDP IS in screen-management mode.
++$OUT=<file_name>
If this flag is specified a new output file will be opened to receive output from the procedure. It will not work if MDP is in screen mode.
++$MCL or ++$MCL=a:b,c:d,....
This flag, ++$MCL, requests that the MACRO/LOOP processors be used to process the procedure file before input to MDPP. This allows the user
to use looping and macros from within a procedure. Additionally, if further parameters are specified (optionally), they are used to transfer data from the internal registers in MDPP to the external registers of the MACRO processor. The string `a:b' (e.g. 4:R3) means copy internal register R3 into external register 4, 5:Q2 means copy output register Q2 into external register 5. The procedure can now do arithmentic on these external register values, and substitute them into places like looping parameters, etc. There may be multiple ++$MCL=lines (you may need to set many parameters). WARNING This flag is not compatible with the ++$RMT flag.
++$RMT=<limit>/SERVER=<node>/SYNCHRONIZATION=<opt>
This is the command to activate a remote compute-server on <node>. <limit> is the current file size at which computing switches to the remote server: if the current file is smaller than this the work is done locally. <opt> is the synchronization option that controls how the local machine access the remote. For more information see the chapter `Using Remote Compute-Servers' which is below.
MDPP Commands
Normal MDPP commands follow the last Procedure command lines. There are no significant limitations on the commands that can be used, execpt that there are a set of commands that must be used at the end of the procedure: see below.
There are comment symbols that will help:-
The `in-line' comment (`+-' described above). Use it to prompt for data from the screen.
The `screen input re-direct' symbol (`+*') is used to force input from the screen. In this manner you can: (a) prompt for input using `+-', and then (b) get the input you want with `+*'. If you have suppressed all
the regular MDPP output with a mask like `++$MSK=Z' then you can essentially re-program the look and feel of the set of commands exactly as you want it.
The `screen-input confirmation' symbol (`+@') is used to solicit a confirming <cr> at a critical place, say before displaying an image, or before erasing it.
Return of control to MDPP needs to be done via special commands. These are `XP *', and `AO 7'. `XP *' causes control to be returned to the procedure driver, if it is active: if it isn't then this command is ignored. If `XP *' is ignored, then `AO 7' is executed. This returns control to MDPP Command mode; if the user was using the procedure driver, the driver is bypassed (i.e. if the user does not put an `XP *" in the procedure command list).
Remember, if you are using the procedure driver, you MUST call `XP *' at the end of a procedure to re-active the procedure driver.
If you have exited from MENU mode by executing `AO 7', then you can re-enter the MENU mode with the `XP' command: i.e. `XP' with no file name or option.
These flags should be used with the following rules in mind:
1) All parameters must appear at the begining of the procedure and before any MDP commands or blank lines.
2) Any or ALL parameters may be omitted.
3) The parameters present must appear in the following order:
++$SCN or ++$PRT (not both!),
++$PRM= (prompt),
++$MSK=,
++$MCL (do not use with ++$RMT)
++$OUT=.
The following is a dummy procedure file.
++$SCN --- A procedure command example
IE 1 ----- Forcably Switch on error trapping
[.....] -- Additional commands
ZZ ------- Trap error search
IE 2 ----- Toggle error trapping off
XP * ----- Return to MENU driver, if it is active
AO 7 ----- Return to MDPP command mode, if not using MENU mode
The MENU driver is a program which allows users to write their own customized procedures for image processing. This system allows real IDIOTS to manage to process stuff, but of course we have NEVER had an idiot use MDP (tut tut).
The menu driver reads a user-customizable menu file which contains the menu prompts, and designates the procedures to be executed if that menu item is selected. Additionally, the user can select the menu and menu item to be pointed to on return from the procedure. In this fashion the menu can lead the user through the computation, reducing the ammount of thinking!
The central element of the menu driver is the PROCEDURE. A procedure is a file containing a list of MDP commands which are to be executed as if they were a single command. The rules for writing the procedures are described elsewhere. `XP' is the command which activates the procedure.
1) Comment lines, for display. These begin with `*'
2) MENU title lines. These begin with `MENU:' followed by the menu
title.
3) MENU item lines, described in full below.
4) Indirect file
reading lines. These are preceded by a `@'
Additionally, characters following a `!' are ignored, allowing comments to be inserted in the file.
2) MENU title lines. A MENU title line begins with the string `MENU:', and is followed by the title string: e.g. "MENU:DIGITIZATON MENU". The menu title appears at the top of the window displaying that menu, and is used to identify the menu in calls to that menu.
A menu title line terminates a previous menu, and introduces the following menu. Menus call sub-menus in a tree-like fashion.
The first menu in the menu file is the top-level menu, and all sub-menus, if they are to be accessible, have to accessed via a path from this top-level menu (the exception is the UTILITIES menu discussed below). No error messages are generated if a sub-menu is inaccessible (i.e. can never be called). No checks are made for typogtraphical errors, which may make sub- menus inaccessible. If two menus have the same name, only the first is accessible: no error messages are generated.
Menus may call sub-menus in any order and from any point, however, a re-entrant menu call makes it impossible for the BACKUP command to step the user back through the sub-menus to the top-level menu. This is probably no more than a minor inconvenience since the TOP command takes the user to the top-level menu directly.
The user may write a menu with the special name "UTILITIES". This menu can be called from any sub-menu, and a prompt entry for it does not need to exist. The UTILITIES menu can call its own separate menu tree, unconnected to the main tree.
There are some special menu names:
BACKUP - This name refers to the calling menu. A sub-menu
called BACKUP will be inaccessible: don't use this name.
COMMAND - Is used to request that the user enter an MDP command. A sub-menu with this name will be inaccessible.
a) Action lines. The general format for an action line is:
<seq_num>,<prompt_string>,a,<procedure>, <return_ame>,<return_entry>
An example would be:-
4,Display Current Image,a,DISPLAY,DIGITIZE,8
This example displays the line "4. Display Current Image". If this menu option is selected MDP looks for a procedure file called MDPPRC:DISPLAY.PRC, opens it, and attempts to execute the MDP comamnds it contains. On completion of the command string, the file is closed and control returns to the menu driver. On return, the menu driver searches for a menu named DIGITIZE, and displays it in the window: entry number 8 is highlighted, since this has been selected as the most logical next step for the user (who can then select anything else he damn-well pleases: such is life in this flawed world).
If the requested procedure file is missing, no action is taken. If the return menu or entry is missing the driver returns to the current menu and entry. There are no error messages.
IMPORTANT: from the point of view of the menu driver single 2-letter commands and their modifiers are procedures. This allows the user to place single commands in the <procedure> section of the line and have the driver execute that command allone, and not have to look for a file somewhere. The command executes straight, i.e. none of the cute stuff in the procedure file to modify output, accept input or call the
MACRO processor works. The menu driver always tries to find a 2-letter command, so don't define 2-letter procedure files, they won't work.
b) Menu lines. The general format for a menu line is:
<seq_num>,<prompt_string>,m,<menu_name>, ,<menu_entry>
An example would be:-
5,Digitize an Image,m,DIGITIZE
This example displays on the screen the line "5. Digitize an Image". If this item is selected the menu driver searches for a menu named DIGITIZE, and displays it in the window the first item is highlighted since no menu entry has been selected. Generally, selection of a menu entry on a menu call is unnecessary.
4) Indirect file read lines Such a line consists of an `@' sign followed by a file name. The file should contain a series of valid menu title or menu lines. The indirect files are read into the driver at the point where the indirect read is located. Once read, reading continues in the main menu file. The indirect file lines are used to read in common menu components into the current menu, for example, the UTILITIES menu. Note: if you wnat to use the same menus under Ultrix as VMS you must remember that Ultrix file names are casesensitive, and that the `Ultrix `standard' is lower case. Therefore all the file names should appear in lower case if you want to write portable menus.
EXAMPLE
The following is an example of a simple menu illustrating many of the points made above.
*
* AUTORADIOGRAPH BRAIN SECTION PROCESSING PROGRAM
* ===============================================
*
* This is a program to allow the processing of auto * radiographs of brain sections. The user selects the * appropriate menu options to effect the computations. * Please report any bugs to Peter Canoll, Ross Smith * or Suzie Gottesman. MENU:MAIN
1,Digitize image with Calibration step,a,DZ,main,3
2,Digitize image,a,DIGITIZE,main,3
3,Display image,a,DISPLAY,main,4
4,Save image,a,SV,main,2
5,Pick area of Step-Wedge,a,PICK_AREA,MAIN,5
6,Linearize image with data entry,a,LINEAR-1,main,4
7,Linearize image using old data,a,LINEAR-2,main,4
8,Convert to BYTE internal format,a,BYTE-ME,main,4
9,Nucleus Search Menu,m,NUCLEUS_SEARCH
MENU:NUCLEUS_SEARCH
1,Read Image file,a,RD,NUCLEUS_SEARCH,2
2,Set display Max/Min range,a,MAX_MIN,NUCLEUS_SEARCH,
3 3,Display Image,a,DISPLAY,NUCLEUS_SEARCH,4
4,Trim out selected area,a,TRIM3,NUCLEUS_SEARCH,5
5,Pick area of Nucleus,a,PICK_AREA,NUCLEUS_SEARCH,5
6,Terminate processing,a,TR
@MDPPRC:UTILITIES.MENU [EOB]
Accessing The Menu Driver
.
In order to use the menu system the menu file must be identified to MDPP. This is done by defining the logical `MDP_MENU' as the file containing the menu. A typical line, included in the LOGIN.COM file would be
"$DEF MDP_MENU disk:[directory]menu-file.name".
To use the MOTIF menu-driver (rather than the SMG one which is anyway not available under Ultrix) do two things: 1) copy the file MDPXDEFAULTS.DAT to your SYS$LOGIN directory (under Ultrix, copy it to the file ~/mdpxdefaults, all lower case), and 2) define the logical MDP_USE_EVENT_DRIVER to something. This switches the driver on. Like this:
"$DEF MDP_USE_EVENT_DRIVER Y"
The menu driver is entered automatically if the mode is set to `2' in the MDPP.DFL file in the directory where MDPP is started (or in the systemwide default file). Alternatively, the user can enter the command sequence /XM/2/XP/: unmodified XP always enters the menu driver if MODE=2.
To summarize:
1) Create the menu file.
2) Define the logical to point to it (MDP_MENU).
3) Modify the default file to expect the menu-driver.
4) Make sure that the MDP_USRDFL logical points to the default file you want to use.
5) If you want to use the MOTIF menu-driver, set the logical MDP_USE_EVENT_DRIVER.
6) Yup, you noticed! 3 logicals/environmentals need to be set to get a MOTIF menu on the screen.
More about file names
Procedures are files containing MDP commands. These file names under Ultrix must always be lower case if saved on a UNIX machine. If the files are on a VMS host they will be assumed to be lower case (Multinet does this, don't worry about it...) when view via NFS from the Ultrix host. Try to write lower case file names.
Notes about the MOTIF menu-driver.
The regular terminal screen is used for data output and for terminal input if needed. Due to the way the program starts the terminal screen may remain clear until the first command is entered by from the menus. This is a `feature' not a bug....
The compute server program extension for the MDPP was develped to allow users with modestly powerful workstations or computers to tackle more complex computational tasks by using a remote compute-server. This allows the workstation to pass of heavy computational jobs to the server, but to perform a significant fraction of the work locally. This has several benefits: for the workstation user, the practical limitation imposed by the limited power of that machine is removed, and for the larger machine the overall load is reduced, vs running the entire job on that machine, since the workstation handles all of the interactive and display tasks for the user as well as much, if not most, of the computing load measured over the whole job. An obvious `target' for this type of operation would be to use MDPP on one of the older workstations, such as a VAXstation 3100, and yet to be able to access a newer Alpha workstation or mainframe for the bigger, more time-consuming tasks (such as large FFTs, etc. Note: 10242 real FFT takes 20-30mins on a MicroVAX-II and less than 3s on an Alpha/AXP 3000/500).
The goal of the programming task was to devise a mechanism where a complex image processing task on a local workstation (client) could be transferred via an electronic network to another more powerful remote compute server which would perform the task and return the results to the local machine (client). Two variations of the scheme were developed depending on whether the client and server were running VMS and connected by DECNET, or different operating systems (VMS and UNIX) connected by IP/TCP or BITNET.
Local machine (Client) Implementation:
The implementation of the distributed computation is based on a very simple message passing technique. If the computational task is to be transferred to another machine, the MDPP client writes four files to the local disk. These are: 1.) the instruction file that is to be executed by
the remote compute server (a VMS DCL command file or a UNIX bourne shell script), 2.) an `environment' data file, which allows the server to inherit the information saved during the course of the processing on the client, 3.) a list of image processing commands and ancillary data, and 4.) the current image data upon which the operation is to be performed. The instruction file is submitted to the remote compute server in a variety of ways, depending on which network protocol (BITNET, DECNET, or INTERNET) provides the most efficient link connecting the client and server.
Once the remote machine receives the instruction file, the request is either queued (in the UNIX implementation) or immediately starts execution (main frame VMS implementation). The instruction file contains the commands necessary to retrieve the environment and image files as well as containing the remote command to begin processing the image according to the clients instructions. Upon completion of the task the results are transferred back to the client machine together with a new environment file. The remote compute server then cleans up unnecessary files and stops executing.
A UNIX host (e.g. a Cray) as the Remote Server:
Computation on the Cray as the remote server, is not tightly coordinated with work on the local VMS workstation and the computation on the Cray runs independent of the workstation. The instruction file is submitted as a job file to the Cray nqs (network queueing system, the batch queues on the Cray) via the Cray Station software. Once the job begins execution, the instruction file will retrieve the image file, the environment file and the image processing command file from the VMS workstation (client) via FTP. Image data is kept in VMS internal data formats (32 bit words) and the Cray server translates this image data from VMS internal data formats (reals, integers and bytes) to and from the desired Cray internal format (64 bit words). After completion of the computation tasks, the Cray server converts the image file from Cray internal format to the VMS internal format, returns the processed image file and the appended environment file to the VMS workstation (client) via FTP, and then cleans up any extraneous scratch files before it ends execution. The local MDPP client detects the arrival of the final data file from the Cray server and signals the user that the server process has been completed.
VMS Mainframe as the Server (e.g. an Alpha):
Unlike the Cray server implementation, job submission can be tightly coordinated between a VMS client and VMS server since the two machines are connected by DECNET and use the same operating system (VMS). The server task is able to communicate with the client workstation via standard VMS task-to-task communication and therefore output information from the remote VMS server appears on the client workstation as soon as it is available. Using task-to-task communication to transfer the image increases performance because it is not necessary for the client to write an image file for the server to retrieve. This method allows the server computation to be almost completely hidden from the user. Our usage of VMS task-to-task communication works similar to an RPC call, thus the client workstation is unable to perform other tasks while the server executes.
IMPLEMENTATION
Figure 1. in the paper Smith et at., (CABIOS, 7, 501-507, 1990) illustrates how commands and procedures are processed by the MDPP. All single commands are executed locally, however, if a procedure is to be executed, the `server' command lines are interogated to see whether processing of the procedure should be done locally, or whether it should be transferred to a remote server for processing. Once the server has been selected, the files to be transferred are created, if necessary, and the remote server is started.
There are several choices as to the communication method used between client and server as illustrated in the following table. The fastest (DECchannel) uses DECnet between VMSes and direct memory-to-memory transfer of image data with the client and server processes linked. The most general method (NETLINK) method uses BITNET, or some other arbitrary link to initiate the remote server job which runs in a batch queue. Files are transferred over the INTERNET via FTP and there is no communication between client and server tasks.
Access Type And Explanation
DECCHANNEL -
Image data are transferred between client and server via
DECnet memory-to-memory transfers. The server runs in synchronous mode, with
client and server linked
DECNET -
Image data are transferred between client and server via DECnet
using an intermediate disk file on the client. The server runs in synchronous
mode, with client and server linked.
DECLINK -
Image data are transferred between client and server via
DECnet using an intermediate disk file on the client. The server job runs in a
batch queue: client and server jobs are independant.
NETLINK -
Image data are transferred between client and server via FTP
using an intermediate disk files on both the client and the server. The server
runs in synchronous mode, with client and server jobs running independantly.
The Table, describes the differnet `access' methods which the client can employ to gain access tot he server. The most efficient is the `DECCHANNEL' method, but this only works between VMS machines. The NETLINK method is most flexible, since this can be programmed on a server by server basis.
Choosing a Server
The decision as to the appropriate server to select and the size of the job which would be transferred to it for fastest response will depend on the user's local environment. In general the response time consists of three components, the overhead of remote job initiation, image data transfer time, and the time taken for the identical process to execute on the copy of MDPP on the larger machine.
This illustrates the point that the raw power of the server only influences remote execution time which may be only a small contributor to total execution time in all but the largest jobs. Therefore, remote users need to have a higher predicted computational time to justify the overhead due to job submission delays, and long transfer times for large data files over the Internet.
This heterogeneity is reflected by the fact that several `++$RMT' procedure command lines can be placed at the begining of the file. These direct the local MDPP to engauge different servers depending on the size of the job. The correct choice of the array size limit needs to be determined by experiment.
Comparison of the Elapsed and CPU times for performing a Fast Fourier Transform on a 1024x1024 array from a MV-II using a VAX 6000-410 as the remote server.
| Link Type | Elapsed Time | Overhead | CPU Time |
| Local | 18m | -- | 510s |
| DECCHANNEL | 2m | 1.5m | 30s (on remote server) |
| DECNET | 4m | 3.5m | 30s (on remote server) |
| DECLINK | 5m | 4,5m | 30s (on remote server) |
The following are simple, general guide-lines.
1) The user must have full access to the remote machine and be able to run MDPP immediately after he or she logs on. This means that the set-up for MDPP should be completed in the LOGIN.COM file (or .login file in UNIX).
2) MDPP in server-mode requires access to several .COM files in the MDPLIB: area. These must be protected so as to allow full access (W:RE).
3) In a DECnet implementation the users should have proxy access as `/DEFAULT' in both directions. (You can make it work without default access, I've done it, but if you want a quiet life don't do it. Also, if you do not have default access you have to expose the password bi-directionally which may not be cool).
4) With a UNIX server you may need to use the command file MPP:[GENERAL]SETPASS.COM to change the local password and enter the accounts and passwords of the remote server machines. This creates a `user-information file' that is read in some cases to get the information to allow a remote server task to be started. This is not all that secure since the computer has a un-encrypted file that can be pirated with all the remote info in it. Better to use the proxy login scheme wherever you can.
5) With these steps taken and an appropriately defined call in the procedure file (as in the example below) you should be in business! Just execute the procedure and watch it work!
6) OK. So it didn't work. First step. On a VMS system, set the /SYNCH switch to `DECNET'. Don't start with `DECCHAN'. The DECNET synch option tests job initiation and getting it to work is a prerequisite to the DECCHAN option to work. Take it slow.
7) The job is started by copying a file (MDP_TASK.COM) to the root account on the server machine, and then starting it (other files get copied too though). If the files are not making it, that is your first probelm. Next the job is started. If the files are getting there, but the job is not starting, that is your second problem. Finally files get copied back to the client machine. If they are not getting back look at the access rights for the reverse path. Make sure the needed files are there, then run the MDP_TASK file on the server by hand fixing the problems until it works correctly.
8) When running the DECCHANNEL server on a VMS machine remember that the server task must be installed. The system manager will need to do this for you. The server .EXE has to be linked with the `/NOTRACE' option: this is done if you request that a server task be built using CONFIG.COM. If new versions of MDPP are built they have to be re-installed every time.
9) At the begining, starting the server using DECCHAN may result in a job started that can not communicate with the client. Make sure that the nonfunctional `NETSERVER' jobs on the server are killed before trying again or you will be in hopeless shape. Also clean up the files on the server that have not been cleaned up authomatically (because the job failed to complete normally). This bit REALLY depends on `clean living' to disambiguate the process.
10) Finally: Using UNIX servers requires using the NETLINK synch option. In general each NETLINK server has to be individually configured. Look at the NETLINK files for the Cray and the PSC SuperCluster an use them as guides. With these models the process should be fairly simple.
An Example of a Simple MDPP Command Procedure
++$RMT=2000000/SERVER=CRAY/SYNCH=NETLINK ++$RMT=1000000/SERVER=MCCLB0/SYNCH=DECLINK ++$RMT=100000/SERVER=MCCLB0/SYNCH=DECCHANNEL
IE 1 -- Switch on error trapping
FF 1 -- Compute Fourier transform
MM ---- Find Max & min and histogram
ZZ ---- On error got to here and resume
IE 2 -- Switch error trapping off.
XP * -- Terminate processing if on remote server
AO 7 -- Terminate processing if running locally
TR
This example shows a simple command procedure used to perform a Fourier transform. The server selected is controlled by the first three lines of the procedure. If the array size exceeds 2,000,000 the Cray is used as server using the `NETLINK' access method. If the array size is less than 2,000,000 but greater than 100,000 the server selected is MCCLB0, a VAX 6000-410, using DECLINK access which causes a batch job to be started to compute the transform. For array sizes less than 1,000,000 and greater than 100,000 MCCLB0 is used with DECCHAN access where the user on the local machine remains linked to the server while the computation is completed. For small arrays, less that 100,000, the computation is done locally.