Linda Bissum


Quick Index

  This paper was originally published in the Proceedings for The First World Conference on System Administration, Networking andrSecurity, Washington DC, 1992.

Three Useful Tools
for
the UNIX System Administrator

Linda Bissum

ABSTRACT

It seems to be a common problem that system administrators tend to re-invent the wheels in form of many of the small tools required for the daily administration. Part of this problem is that many tools are not well distributed, and also that many of the tools which are written are done as ad-hoc shell scripts, which must be modified on an ongoing basis. This paper describes three tools, police , which removes files, cleanlog , a log maintainer and free , a file system monitor - all written as generalized utilities. They should therefore be able to adapt to changes in policy or environment without modification of the program themselves.

Introduction

This paper describes three useful system administration tools. The tools where all originally written in the Bourne Shell, back in the mid-eighties, and have proved themselves useful. However, as they where placed on an increasing number of new platforms, it became increasingly difficult to maintain a single source for all platforms. Over the last year, they therefore all have been rewritten in Perl(1). programming language. The rewritten utilities are based on the lessons learned and are also utilizing the more flexible programmability provided by Perl. The tools which are covered in this paper is police , which removes stale files, cleanlog , a log file maintainer and free , a file system space monitor.

The utility free was previously the most difficult script to maintain, and it received several improvements during the rewriting in Perl. Therefore, a discussions of this transition will also be covered, as an illustration of the differences between implementing such tools in Bourne Shell and in Perl.

Police, a file removal daemon

Temporary and stale files can be a problem on systems with limited disk space. For example, the emacs editor and the ispell spell checker leaves *.bak files which an be deleted automatically after a period of time. In a similar manner, core files can be immediatly deleted on most systems which has a binary-only license.

Police is a generalized program to remove such files. Which files should be removed is determined in a external configuration table. This table enables the system administrator to specify which files should be removed, and how old they should be before removal.

This approach has strong advantages over ad-hoc shell scripts as no modifications are necessary in the program when changes in policy require files be added or removed from the automatic deletion process.

However, as for all monitoring tools, which are actual making alterations to the system, there is a danger that files could be unintentionally be removed. The answer to this problem is twofold. First, no such tool should ever be employed on the systems, without a clear policy of which files will be removed. Second, as for police , the implementation should allow for efficient warnings before the removal. The time period of warnings before removal, as well as the time since last modification date is determined by an external configuration table, as shown in figure 1.

Finally, all file removal can be stopped by the system administrator by creating a lock file. When this is done, e-mail will be generated as usual, but no file will actually be removed. This is useful to ensure that the regular expressions in configuration performs as expected.

       core:1:3
    *.bak:1:3
    *.i:1:3
    Canceled.mail:5:7
    dead.letter:5:7

    Fig 1 - Configuration file for police
    
   

The first column in the configuration file is a regular expression specifying the name of the file. In the first line it is core . The second column shows how long a time the file should be left unnoticed (one day). When one day has passed, mail will be sent to the owner of the file, warning him that his file soon will be deleted automatically. When the number of days shown in column 3 (two days) are up, the file will be deleted unless its owner has touched it (e.i changed the date of last modification, which starts the game all over), or renamed it to keep the file permanently. The second line is for the backup files from emacs and ispell . Note that regular expressions are allowed.

The main advantages of this utility over ad hoc scripts is that the user is warned that the file is about to be deleted. If necessary, the user can change the name of the file, if it is needed in the future. Also, it is very easy for the system administrator to add files, or change any of the time periods, without any changes are required in the police utility.

The disadvantage with this utility is that it takes a long time to run. This is caused by the fact that it is necessary to run the equivalent of a find command from the root directory. The implementation is of caused based on find2perl command. The perl script generated by find2perl are then executed from an open statement (in a manner similar to popen in C).

As the utility is intended to be run at night from cron , this is a minimal problem unless the system has a large number of disks. However, in that case it can be assumed that disk space is less of a premium, obsoleting the need for a utility like police .

Cleanlog, a Log Maintainer

Various UNIX utilities are writing log files. Some of these utilities, like sendmail and syslogd are ensuring that the logs are not growing without bounds. Other utilities are not cleaning up in a similar manner. The system administrator is expected, for each log file, to invent a scheme to remove such log files on a regular basis. However, if the information in those log files is not kept for some time it can make recovery after failure more difficult.

Cleanlog is generalized tool to maintain the log files from one simple table, while giving the administrator the ability to decide for each individual file how many days they should be kept.

In principle, the system administrator specifies how many days a log file should be kept, and cleanlog will the keep the logs on a rotation basis (similar to what is done by sendmail ). Each night, when cleanlog is executed from cron , it will remove the oldest log file, and pushing all other log files one place ahead. For a seven day rotation scheme, it would be the equivalent of:

       rm logfile.7
    mv logfile.6 logfile.7
    mv logfile.5 logfile.6
            .         .
            .         .
    mv logfile logfile.1
    touch logfile
    
   

In praxis, cleanlog provides some additional functionality, as can be seen from the cleanlog configuration file in figure 2.

Configuration file for cleanlog

       /usr/lib/news:log:3:news:news:600:
    /usr/lib/news:errlog:7:news:news:600:
    /usr/lib/nn:Log:3:news:news:600:
    /usr/smail/log:log:7:root:whell:600:

    Fig 2 - cleanlog configuration example
    
   

The six fields of this table are used in the following manner: The first field is the directory where the log file is located and the second field is the basename of the log file. The third field is number of generations which should be kept of the log file. The sixth field is the file permission for the log file. The next two fields are the owner and group which the log file must belong to. Cleanlog will change the access permission on any newly created log file. The last field (unused in the example above), is intended for a shell command which must be executed after the log files have been moved. This is required in some versions of UNIX for a utility like cron , which in some cases must be killed and then restarted to ensure the log file change has taken effect. This problem is caused when a utility is opening the log files, and is keeping it open for the lifetime of the process. In that case the utility will continue to write to the same file, independent of any any file name change, or even file removal. In all cases, the older log files will be kept in a directory named OLDLOGS which will be created in the same directory as the main log file.

Free, a File System Monitor

It is a common problem that file systems are running out of free space. Unix has no way to report this prior to the event, unless the system administrator does a manual disk free check, using the df command, on a regular basis (e.i. several times a day). This is no big deal, if the system administrator is responsible for one or two systems. However, if the number of systems becomes much larger five or ten, it becomes impossible keep up.

Free is an example of solving this problem, partly through automation, partly through delegation of responsibility. The idea behind the file system monitor is based on the fact that the system administrator, in most cases, does not know which files can be deleted or archived (with the obvious exceptions of file system like root and /usr.) For most file systems, the users will have a better knowledge of what can be deleted or archived. The idea behind free is therefore based on the concept that it is appropriate to let the users look after their own file space. Sites which are adopting this view, will of cause need to have written policies stating that this is required of the users. Procedure must also be in place describing the process to request files placed on archive tapes, or to request a larger file system.

However, the problem with this policy is, that many users do not look after the file space, and many do not even know how. This is where Free is providing a useful service. free checks the amount of free space for mounted file system. If it falls below a certain level, it uses wall to broadcast a message to all users currently logged on.

Free is intended to be run from cron every hour, and will in this manner ensure that the users will get a timely warning before a file system is filling up.

If this still is happening, and if the users do not take any action, a point of view could be that they have asked for problems they are about to encounter. However, such an attitude is neither very positive, nor will it get goodwill from the user community. Therefore, the free disk monitor has a second level of checks build in. If a file systems reaches the first limit (the message level), and no action is taken, the fall back position is that when the second level (the panic level) is reached, the system administrator is notified by e-mail.

In this manner, on a reasonably well configured and administrate system, the system administrator will no longer need to worry about the free space problem on a daily basis. Most likely, the users will be able to take whatever action is necessary themselves.

There are two problems with this approach, however. First, no default will work well for all sizes of file systems, and second, not all file systems should be reported to the users. These problems have been resolved through the use of an external configuration table, as shown in figure 3.

       /dev/sd0a:3607:3150
    /dev/sd0g:95%:90%
    /usr2:10%:5%

    Fig 3 - Configuration File for Free
    
   

Free has builtin defaults, which can be set a reasonable value for most systems (e.i. 10% warning level, and a 5% panic level). However, for any file system, where such values are not reasonable, the values can be set in the configuration file. This file has three fields, where the first is either the device name or the pathname of the mount point, and the two following fields are the values for the warning and panic levels of free space. The latter can either be specified in blocks, on in a percentage of the size of the file system. File systems where no warning message should be send to the users, the warning level is set to 0.

Free Implementation Details

It is a common misconception that shell scripts are portable between UNIX systems. The syntax and semantics, at least for the Bourne Shell, are reasonable constant. However, many of the commands called from the shell script will vary in both style and location, making the portability difficult. One of the advantages of Perl over a shell script is, that operations in the shell script which are done by calling external utilities, in Perl mostly are builtin functions. However, in the case of free some of the essential operations are still done through call of external utilities, leaving the problem of portability unresolved. These are:

  • The name of the host must be determined. Depending of the version of UNIX, this is done through calling of different utilities, typically either hostname (BSD) or uuname (USG).
  • A mail user agent must be called to deliver e-mail. Less important, dependent on the mailer, it may not be possible to specify a subject line.
  • The core of the free utility is the determination of the amount of occupied and available disk space. The only truely portable solution, was to call the df program. Unfortunately, the format of the output of this program differs wildly between UNIX versions, and even between similar versions from different vendors.
       $HostName = ( "/etc/hostname",
        "/bin/hostname",
        "/usr/bin/uuname -s"
    );

    sub $HostName {
        unless ( defined $HostName ) {
            foreach $Cmd ( @HostName ) {
                chop( $HostName = `$Cmd` );
                last if ( $? == 0 );
            }
        }
        $HostName;
    }

    Fig. 4 - Perl subroutine HostName

    
   

The two first problems are solvable in a shell script by calling each possible utility, until one is found and executed. The same solution can be use in Perl, as shown for hostname in Figure 4.

I have found no truly portable solution to solve the last item. Therefore, the interface to the df program has been left dependent on the architecture.

The free utility is assuming that an external command, arch , exists, which will print the name of the current architecture to standard output. This output string is then used to make a lookup in a second configuration table, to find the name of the interface file which shall be used to execute the df command. Figure 5 is showing this table.

       sun4:bsd
    sun3:bsd
    bsdi:bsd
    hpux:hpux
    marc:bsd
    svr4:svr4
    sv3.2:usg
    sco:sco

    Fig 5 - Architecture conf. file for free
    
   

In this table, the first field contains the name of the architecture, and the second is field contains the name of the interface file.

To give the greatest possible flexibility, neither of the configuration files are required. However, if the file system table is missing, the defaults will be used for all mounted file systems, and if the architecture table is missing, the interface programs known internally to free will be executed sequentially, until one is giving a useful data, or the list is expired.

Comparison between Perl and Shell

Because Free has been implemented in both Bourne Shell and Perl, it gives a basis of comparison between the two implementation approaches. It has been my experience that many programs which can only with difficulties be implemented as a shell script are easily implemented in Perl. The code for police and cleanlog are smaller and less complex than the older shell script. Free ended up with being slightly bigger, however, much additional functionality was added to resolve problems which existed previous.

References

Larry Wall and Randal L. Schwartz. Programming Perl. O'Reilly & Associates, Inc, 1990.