crontab executes run_nightly This addition is also helpful export PATH=.:/home/regression/bin:$PATH To rebuild the HEAD executable without running any tests ./run_nightly.py --noregression --notestregression To run regression tests without rebuilding HEAD ./run_nightly.py --noupdate The structure of running nightly regressions is: run_nightly (bash script to provide python environment) run_nightly.py build_revision.py build the requested revision (default is HEAD) run_regression.py run that revision of gs on all testfiles with all paramsets test_diff.py analyze checksums: compare this run with recent and with baseline get_baseline_log.py summarize recent changes to the baseline update log These script use the environment provided by gsconf.py, which reads the file testing.cfg; as well as using the command line. run_nightly.py build_revision.py run_regression.py The data flow: All directories are relative to gsconf.root (usually /home/regression) build_revision.py input from svn gs repository output gs build tree, located at gsroot (if revsion==HEAD) - root/gs. (for specific revisions) gs installation tree (only if revision==HEAD) installtree logs/.update*,make*,config*, run_regression input comparefiles (usually /home/regression/comparefiles) - run all "appropriate" files (tested by file extension) using gs gsparamsets.py - sets of output devices for gs testing output dailydir/.db (usually /home/regression/regression/daily) - store db with (testfile-device,checksum) logs/.regression logs/.gs* is determined by the existence of the most recent checksum database, before today test_diff input baselinedb (usually /home/regression/regression/baseline/baseline.db) dailydir (usually /home/regression/regression/daily) .db .db output text report of differences. For all testfile-device entries in this revision's checksum db, if the checksum does not match the previous revision, report as a new difference if the checksum does not match the baseline, report as a difference logs/.differences get_baseline_log input baseline_log (usually /home/regression/regression/baseline/baseline.log) output text report of updates to the baseline since the specified date Details of checksums all checksum db are in "dailydir" (usually /home/regression/regression/daily) checksums are calculated using the python md5 module. these dbs are files of anydb format, with key=testfile-device, value=checksum New differences result from the comparison with a recent db. All differences result from the comparison with the baseline.db Each day (or revision) a new checksum db is created. Each day, the new checksum db is compared with the most recent day's checksum db. Each day the checksum db is also compared with the baseline checksum db (baseline.db). A checksum log can contain 15,000 or more entries (number of test files * number of devices in gsparamsets) bugs: checksum logs that are used for comparison are still named by date, and it should be revision only. each day's log is also copied to a db named by revision (but these are not yet used) There are tools to examine the checksum dbs dump_checksum.py - normal use. Be aware: you get LOTS of output dump_checksum_raw.py - for debugging dump_checksum_plus.py - specialized collate.py - gather results of several dbs and sort Details of testfiles: all testfiles in the gsconf.comparefiledir are tested. Smaller subsets (in other directories) can be used for testing. The normal situation: For each testfile, there is a set of compressed raster files (one for each device). For each raster file there is an entry in the baseline checksum db. To make the raster file and baseline checksum db up-to-date, use update_baseline.py (--help for command line options) Under the covers: gsconf.py - provide an object with information about the testing environment gstestgs.py - run gs gsparamsets.py - provide sets of output devices (one set for "normal" and one for "pdfwrite") gstestutils.py - utilities gsutil.py - utilities gssum.py - calculate checksums and insert into a checksum db rasterdb.py - create and get compressed raster files These scripts do the work of running gs and adding checksums to the daily checksum db: gscheck_all.py - run the following - creating test cases for later execution gscheck_raster.py - normal for each test file, for each device create a normal test case gscheck_pdfwrite.py - pdfwrite for each test file, for each device create a pdfwrite test case gscheck_testfiles.py - test only the files in a specified list check_* - scripts to test the other aspects of the code base (dirs, comments, docrefs, source) Repair: regen_baseline.py regenerate all checksums from the raster files (>15,000 files - takes a long time) regen_filelist.py regenerate checksums specified in a file list from the raster files (>15,000 files - takes a long time) Tests with larger scope than nightly: run_series.py - run_nightly.py with a specified range of revisions.