|
REGRESSION TESTING INSTRUCTIONS
|
Overview
This page describes how to create and compare regression test suites for nl-soar7 comprehension.
Creating a regression test suite involves building a directory comprising files representing a textual output of the grapher. For each regression sentence 4 files are created for chunked and unchunked semantic and syntactic graphs.
Comparing a regression test suite with another (i.e. usually comparing a test suite after some changes to nl-soar with a stored default suite of 'correct' values) simply involves comparing these output files.
Creating a regression set
A regression test suite is creating using the executable create-reg-set7 which is made from the C source code.
The use of the create-reg-set7 command is :
create-reg-set7 load-file regression-file tree-dir
- load-file is the name of the nl-soar load file. [ NB differences from old regression method : specify the complete name of the load file, the load-file should load all the source code required including .init.soar ]
- regression-file is the name of the regression file containing sentences to be tested. The regression file should contain one line for every sentence; each line comprising a sentence identifier and the sentence itself. It is better if the sentence is ended with a period. No comments can appear
in the file. For an example of a regression file see the file for the basic regression corpus.
- tree-dir is the name of a directory where the regression files (textual grapher output) will be dumped.
The command will start a nl-soar process and create a new source file called
regress.init.username.soar in the current directory. Unfortunately, in soar7, there is no way to automatically load this source file so this has to be done manually (i.e., type source regress.init.username.soar in the soar window. If nl-soar7 is not available in the current directory, an error will occur.
Comparing regression sets
Two directories containing regression suites are compared using the command
compare-reg-set which is made from the C source code.
The use of the compare-reg-set command is :
compare-reg-set regression-file tree-dir tree-dir2
- regression-file is the name of the regression file containing sentences to be tested.
- tree-dir and tree-dir2 are the names of the directories containing regression outputs (created by create-reg-set7 above). Typically
tree-dir1 is a new regression set created after some changes have been made to nl-soar and tree-dir2 is the default 'correct' regression set.
The output of the compare-reg-set command is an sentence by sentence analysis of the differences between regression sets as well as a summary of all sentences. Example.
This page written by Mark H. Smith, April 1997.
Updated by Julie Van Dyke, August 1997..