Carnegie Mellon
University
Information
Networking Institute
Scriptor: A Programming by Demonstration (PBD) System for Robust
Information Extraction from Web Sites
TR 2000-04
A Thesis Submitted To The Information
Networking Institute In Partial Fulfillment Of The Requirements For The Degree
Master of Science
in
Information Networking
by
Muhammad Nuri
Pittsburgh, Pennsylvania
May, 2000
1 Introduction 4
1.1 What is Programming by
Demonstration? 5
1.2 Programming by Demonstration for the
Web 6
1.3 Scope of Thesis 7
2 System Model 8
2.1 Trade-off between User Control and
Sophistication of PBD System 8
2.2 Segregating Records from a Web site
by Hints 10
2.3 Repetition of tags as an important
information in segregating records 11
2.4 Rule-based Extraction of Record
Fields 12
2.4.1 Follows <Attribute> 12
2.4.2 Precedes <Attribute> 13
2.4.3 Creating complex rules from basic
follows and precedes rules 13
2.5 Appearance-based extraction of
Record Fields 14
2.6 Sample-based Extraction of Record
Fields 15
3 Example of Use of System on Borders.com 17
3.1 Segregating records by providing
hints 17
3.2 Extracting individual fields of a
record 20
3.2.1 Extracting the title of book by
using appearance-based extraction 20
3.2.2 Extracting Price of Book by using
Rule-based Extraction 21
3.2.3 Extracting Publishing Year of Book
by Sample-based Extraction 22
3.3 Viewing the Results 23
4 Usability Study 24
4.1 Instructions to Subjects 24
4.2 Summarized Data from Experiment 25
4.2.1 Microwarehouse Computer Retailer 25
4.2.2 School of Information Science
Course List-University of Pittsburgh 26
4.2.3 E-bay Auction Site 27
4.2.4 Secapl Financial Site 28
4.2.5 Autotrader User Car Seller 29
4.3 Script Generation Time By Novice
Subjects 30
4.4 Other observations 30
4.5 Conclusions from Usability
Experiment 31
5 Conclusions and Future Work 32
6 Bibliography 33
7 Appendix B: Source Code of PBD System 34
7.1 Organization of Source Code 34
7.2 Core Function Files 34
7.3 Graphical User Interface Files 55
7.4 WebL Files 73
The question of the thesis is to determine whether its
feasible to involve a novice user in the design of information agents and the
idea is motivated by the need to extract information from a website independent
of its underlying structure. These websites are one of a kind.
Information agents gather information autonomously from
the Web. The connection between an information agent and the website from which
it collects information is “brittle”. Frequent change in the look and structure
of the website render these agents useless. These agents use scripts developed
by professional programmers. These scripts use the underlying HTML tag
structure to get information. With change in look and structure of website, the
underlying HTML tag structure changes and the script does not work anymore.
One way around this problem is to hire a professional
programmer who would reprogram the script for the information agent. However
this approach is cumbersome and has drawbacks. If the programmer tries to
modify the existing script, then we have associated errors of
copy-paste-modify. If a completely new script is written, the approach is still
infeasible that one would have to hire a programmer every time the website
changes.
Compounding on the problem above is the need to process
information from new websites, which are ever increasing on the Web. As these
websites become available, a user would need to set up new information agents
for them in helping him/her make a better decision. Thus in the face of these,
we propose to involve a user in the design of information agents. When an
information agent cannot gather from a website due to change in a website, it
would notify its novice user to “reprogram” it. In another scenario, the user
sees a new website for which it wants to set up an information agent and
“programs” a script for it.
A novice user doesn’t know how to “program” which should
be an important assumption. Here in the design of information agents, I want to
give a novice user the ability to program. This programming ability cannot be
the conventional programming language one learns in a computer science course.
Elliot Soloway, director of the Highly Interactive Computing project at the
University of Michigan, estimates that even for novices who do take a
programming class, less than 1% continue to program when the class ends. How
can then a computer be programmed without a programming language? I propose for
a Programming-by-Demonstration System.
Programming by demonstration (PBD) or programming by
example (PBE) is a powerful end-user-programming paradigm enabling users
without formal training in programming to create sophisticated programs. PBD
environments create programs for end users by observing and recording software
behaviors as they manipulate information on the graphical user interface (GUI)
level.
Maulsby concluded from its Turvy experiment that end
novice users are content to program agents and want to do so through
demonstrations with verbal hints and so on. PBD’s most important characteristic
is that everyone can do it. PBD is not much different from or more difficult from
using the computer normally. This characteristic also led me to consider PBD as
an alternative to “easier” syntactic languages. The “easiness” of syntactic
languages has been commented by Canfield etal. They argue that users need not
think like computers to accomplish their task but rather every computer system
should be simple and intuitive that they can be “programmed” with minimal
effort. The system should move to the user rather the opposite happen in
present days. The user moves to the system, thinks like a computer in order to
accomplish the task. The GUI just makes the process more apparent but does not
make it intuitive.
The Web is a great focus for PBD because of its
accessibility to a wealth of knowledge, along with the pressing need for
helping users organize, retrieve, and browser it all. Recent developments in
intelligent agents can help – but only if users are able to communicate their
requirements to and control the behavior of their agents. Typically these
agents are scripted manually which requires programming skills. With the
increasing number of web sites on the Internet, frequent change in the look and
structure of web sites, manually programming agents takes valuable time and
effort. We felt it necessary that PBD systems can be a flexible solution to the
problem. With a PBD system, a novice user could generate sophisticated scripts
in a very short time to gather specified information from the Web. An
information agent would record the interactions between the user and a
conventional direct manipulation interface and writes a program corresponding
to user’s actions. The agent can then generalize the program so it works in
other situations similar to, but not necessarily exactly the same as, the examples
on which it was taught.
Our focus of study remained on sites like Amazon, which
give listing of information items. We also attempted to make the PBD system
simple and intuitive for a novice user. The following section would explain in
detail the PBD system. We tested the system with novice subjects to determine
the system’s usability in terms of accuracy and the time taken in script
generation. The section on usability discusses the experiment we did with
novice subjects.
In this chapter I discuss the model on which the PBD
system is based. On a web page from which we wish to extract specific
information records, there are two steps involved:
1.
Segregate the records in a web page by:
§
Providing hints from the listing of information
records, or
§
Using repetition of tags in a web page as an important
information.
2.
Extract individual fields of record by:
§
Rules-based extraction,
§
Appearance-based extraction, or
§
Sample-based extraction.
Before going further into explaining each of the methods
in segregating records from a web page and then identifying individual fields
in a record, I consider it important on what considerations the model was
based.
The model has to be simplistic enough for a novice user.
Here the novice means a regular Internet surfer or a person who usually writes
his/her document using a word processor such as Word. We identified that there
is a trade-off involved in the design of the system. If finer controls are
given to the user then the system use becomes much more complicated. In
designing PBD systems for novice users, one of the crucial assumption is to
take the system should follow the principles of the novice mind. The novice
need not like to think like a computer to program but it should be the other
way around. The system should be simple and intuitive for the novice to
understand. Since we cannot give a more robust system without increasing the
user control of the system then we need to increase the sophistication of the
system. Increasing the sophistication of the system essentially means using
direct artificial methods, which take more computation time. Also such methods
are based mostly on heuristics for the Web. With the changing character of Web,
the system would be too brittle. The system model developed below is a result
of an iterative exercise keeping the trade-off between user control and
sophistication of system in mind. Further, the system was tested against
subjects to affirm its accuracy and usability and thus prove that proper
intelligence has been built into the system. The chapter on usability shows the
results in more detail.
Before information about a single record could be
extracted (such as its price of a book, its title etc.), the system needs to
know what is a record. That is done by user providing a text from the area in
the web page having list of records. The system then proposes candidate record
samples to user. The user can accept or ask the system to try for a better one.
This goes on until the user is satisfied with sample record. The following
figure illustrates the segregating of records by providing hints.
In our PBD system, the web page is modeled as a string
containing tags and text. When the user accepts the sample record, the system
finds similar records to the appearance of sample record. Thus we then have set
of records,
Record Set = S = {[R1], [R2], [R3]…}
Where Ri is the
text and tags of individual records.
In majority of the web sites I observed, the record
information was enclosing within one enclosing tag and thus I needed to gain
only those tags to identify what is in a record. However on a few sites, the
information for a single record was split among two tags. For example on Amazon
web site, the title and author was in one enclosing tag and the pricing
information was in the other enclosing tags. For such web sites, I found the
following technique helpful but remember this technique costly in terms of computation
and results are not very correct either.
On sites like book sites, auto-sites that display
listings, there is an important tag information that can be used to segregate
records from the web page. Though the information within tags changes but the
tags remain same across all records. This means that if one could remove all
the textual information from the web page and let only the tags, then there
would be a string of tags, which would be repeating it. This is a heuristic,
which would not work in all case. If there are fewer records returned, then one
cannot detect such a string of tags. The more records there are from a search
on a website, the more likelihood that the tag string would be repeating itself
very often. A simple example would illustrate the idea. If the following are
the tags left after removing all textual information from a web page,
Web Page Tags =
“<a><b><c><a><d><a><d><a><d><a><d>”
Then one can see that tag string “<a><d>” is
repeating itself four times and could most likely enclose record information.
One can note that a threshold on repetition needs to be defined too. If the
threshold were low, many tag strings would come up as possibly the tags enclosing
record information. If the threshold is very high to be greater than the number
of times the actual enclosing tag string repeats itself, then also one won’t
receive any result and there would be no tag string returned. Thus an optimum
is required which can be identified and re-identified by the user.
The individual fields are retrieved using the following
tools
This tool works like this: Anything that follows the
attribute would be picked up as the information. Consider the following simple
example,
Books = {[“Java 1.1 $19.95 20%”], [“ ABC of Java
$30.09 7%“], [“Programming by Example $49.99 13%”]…}
If price of book is desired then follows $ could be
specified. The system would return 19.95, 30.09 and 49.99
This tool works like this: Anything that precedes the
attribute would be picked up as the information. Consider the following simple
example,
Books = {[“Java 1.1 $19.95 20%”], [“ ABC of Java
$30.09 7%“], [“Programming by Example $49.99 13%”]…}
If discount on book is desired then precedes % could be
specified. The system would return 20, 7 and 13.
AND-ing or OR-ing can combine follows and precedes rules
to create more complex rules for extraction.
Appearance based extraction is based on the idea that
certain fields in a record are formatted differently (e.g. underlined, bold,
different color) from other fields in a records. This is valuable in cases
where distinguishing attributes like % and $ are not present for fields. For
example,
Books = {[“Java 1.1 $19.95 20%”], [“ ABC of Java
$30.09 7%“], [“Programming by Example $49.99 13%”]…}
We see here that there are no distinguishing attributes to
extract title of book. However, we see that title of every book is underlined.
The user specifies ABC of Java and indicates to system. The system would return
Java 1.1, ABC of Java and Programming by Example.
We can see from the above simple example that every piece
of information was extracted from the set of book records. Figure also explains
the concept. When I mention “Advanced Java”, I tell the system that anything
that looks this is a Title in a record.
Sample-based extraction works as follows: The user
specifies samples to the system and the system constructs a generic pattern for
extraction. In the above example, if the system is given samples 19.95 and
30.09, it would construct a pattern dd.dd where d stands for digit. Anything
that matches this pattern would be a price. The system would return 19.95,
30.09 and 49.99 as the price of book.
The following figure illustrates the generalization of
publishing year from a set of samples from the web site.
The following illustrates the system Scriptor which is a
practical implementation of the system model. The source code is given in the
appendix. The system was developed in Java and WebL. WebL is a web query
language developed by the Compaq Systems Lab.
You provide the system with a hint and then click on
“Click here to see” to see what it proposes is a record.
The system shows a sample to tell you if that is what
would be a typical record look like.
If you’re satisfied, you click “Yes this represent a
record” and the system advances to the following screen.
You specify to the system that anything that has
structural information like Advanced Java ( hyperlink, blue in color) is a
title.
You specify $ in the attribute box and tell the system
that anything in a record that follows a $ sign is a price of the book. If
there are multiple occurrences of $, the first one is picked for information
processing.
You specify samples to the system from two or more records
from the listing of information records. The system generalizes on these
samples.
And one can go on to view other records too as a result of
tools applied to the website.
Software was developed to experiment with the usability of
the PBD system model. An instructor acquainted the users with the software for
20 minutes. The purpose of instructor was to get them familiar with the
software.
All subjects were given the same web pages to extract
fields using rule-based, appearance-based or pattern-based extraction. Novice
subjects were selected for this experiment. The pages selected were from
diverse sites as well as diverse in appearance.
The following summarizes the data gathered from the
experiment for different web sites.
|
|
Part
|
Description
|
Price
|
|
Number of Subjects Getting
Correct Extraction
|
8
|
8
|
2
|
|
Total Subjects
|
8
|
8
|
8
|
|
Percent Correct
|
100%
|
100%
|
25%
|
|
|
CRN
|
Description
|
Prerequisites
|
|
Number of Subjects Getting
Correct Extraction
|
10
|
10
|
10
|
|
Total Subjects
|
10
|
10
|
10
|
|
Percent Correct
|
100%
|
100%
|
100%
|
|
|
Item
|
Description
|
Price
|
|
Number of Subjects Getting
Correct Extractions
|
9
|
6
|
9
|
|
Total Subjects
|
10
|
10
|
10
|
|
Percent Correct
|
90%
|
60%
|
90%
|
|
|
Ticker
|
Price
|
Change
|
|
Number of Subjects Getting
Correct Extraction
|
9
|
8
|
9
|
|
Total Subjects
|
10
|
10
|
10
|
|
Percent Correct
|
90%
|
80%
|
90%
|
|
|
Year and Car
|
Mileage
|
Color
|
Price
|
|
Number of Subjects Getting
Correct Extraction
|
7
|
10
|
10
|
10
|
|
Total Subjects
|
10
|
10
|
10
|
10
|
|
Percent Correct
|
70%
|
100%
|
100%
|
100%
|
The subjects were given web pages in the following order:
Autotrader, Secapl, E-bay, School of Information Science Course List-University
of Pittsburgh, Microwarehouse.
|
Subjects
|
Time Autotrader completed
(minutes)
|
Time Secapl
Completed
(minutes)
|
Time E-bay Completed
(minutes)
|
Time SIS completed
(minutes)
|
Time Microwarehouse
Completed
(minutes
|
|
1
|
11
|
9
|
10
|
7
|
11
|
|
2
|
6
|
7
|
4
|
3
|
7
|
|
3
|
7
|
3
|
4
|
3
|
4
|
|
4
|
5
|
4
|
10
|
4
|
4
|
|
5
|
6
|
5
|
4
|
3
|
5
|
|
6
|
10
|
6
|
5
|
5
|
7
|
|
7
|
9
|
7
|
6
|
3
|
7
|
|
8
|
10
|
3
|
9
|
3
|
7
|
|
Average
|
8
|
5.5
|
6.5
|
3.9
|
6.5
|
Subjects found rule-based extraction using follows and
precedes to be quite intuitive. They tried first these rules to extract. When
it failed they tried hints. Very few subjects used learned-pattern-based
extraction. The price field in Microwarehouse record could be obtained
correctly only through pattern-based extraction. That is why we see lesser
accuracy there.
§
Script generation time is quite impressive in
comparison with manually scripting the web sites using Perl or other language.
Moreover, these are times for novice users. A novice subject generated script
in less than 12 minutes on any site far less than the time it takes for a
programmer to code script manually.
§
Accuracy is quite impressive too. The subject obtained
good score in extracting fields with accuracy above 80% in many cases.
There are various issues that weren’t consider that I
think could improve the results even further. At present, field extraction
tools (rules-based extraction etc…) are working independently of each other. It
is quite possible to integrate them with other extraction tools using AND/OR.
The system can also be extended for expert users. Finer controls can be
introduced with modes available for novice and expert. I believe the learning
curve to PBD systems for expert would still be far lower than a comparable
syntactic language like Perl. The results can be verified by conducting a
usability study.
In the end, I conclude that it is feasible and recommended
involving a novice user in the design of information agents and thus this
approach helps in solving the problem to extract specific information from a
web site independent of its underlying structure. There are considerations that
have to be kept in mind when designing PBD systems for such one-of-a-kind web
sites namely the trade-off between user control and sophistication built into
the system. The finer controls built into the system, the lesser a PBD system
becomes appealing. Because the intuitivity of the system suffers. However, if
one try to build sophistication into the system based mainly on heuristics, the
system becomes brittle to work when systems change with time and the heuristics
become invalidated.
1.
D. Maulsby, I. Witten: Teaching
Agents to Learn: From User Study to Implementation, IEEE Computer, November
1997.
2.
H. Lieberman, B. Nardi, D. Wright: Training Agents to Recognize Text by Example, Third Conference on
Autonomous Agents, 1999
3.
D. Smith, A. Cypher, L. Tesler: Novice Programming Comes of Age, Communication of the ACM, March
2000.
4.
B. Myers, R. McDaniel, D. Wolber: Intelligence in Demonstrational Interfaces, Communication of the
ACM, March 2000.
5.
M. Bauer, D. Dengler, G. Paul, M. Meyer: Programming by Demonstration for Intelligent Agents, Communication
of the ACM, March 2000.
§
WeblEngine.java is the code that makes calls to
WebL.jar. WebL.jar is the library of files provided by Compaq Systems Lab.
§
GenericAgent.java is the code that makes calls to
WebLEngine.java.
§
EventHandler.java handles events for GUI Files and
makes subsequent calls directly to WeblEngine.java or indirectly through
GenericAgent.java
§
GuiUser.java initializes all variables and is the start
point of program.
§
All files with agt extension are WebL files loaded
dynamically by GenericAgent.java or EventHandler.java.
GenericAgent.java
import java.io.*;
import java.util.*;
import java.lang.*;
import webl.lang.*;
import webl.lang.expr.*;
import webl.util.*;
public class GenericAgent
{
static String
script = "";
static
WeblEngine webl_engine
= new
WeblEngine();
static int
number_of_patterns = 0;
static int
selected_pattern = 0;
int
search_str_set = 0;
int current_pattern
= 0;
public
GenericAgent()
{
executeScript("AgtFiles/Init.agt");
}
public void
startAgain()
{
webl_engine =
new WeblEngine();
executeScript("AgtFiles/Init.agt");
search_str_set = 0;
}
public String
executeScript(String filename)
{
String line =
null;
String
return_val = "";
try
{
BufferedReader in = new BufferedReader(
new
FileReader(filename));
while
((line = in.readLine()) != null)
{
script
+= line;
}
in.close();
System.out.println("executeScript():" + filename +
"\n"+ script + "\n");
return_val
= webl_engine.executeScript(script).print();
script =
"";
}
catch
(FileNotFoundException fnf)
{
System.err.println("GenericAgent(): " +
filename + " not found");
}
catch
(IOException ex) { System.err.println(ex); }
return
return_val;
}
public String
executeLine(String line)
{
Expr result;
System.out.println("executeLine(): " + line + "\n");
result =
webl_engine.executeScript(line);
return
result.print();
}
public String
searchWebSite(String website, String type_of_site,
String
search_string, String matching_power)
{
String search_str_line =
"import Str; import Browser; var searchInput = \""
+
search_string + "\";" ;
String
type_of_site_line =
"var
searchType = \"" + type_of_site + "\";" ;
String
website_line =
"var
website = \"http://" + website + "\";" ;
script =
search_str_line +
type_of_site_line + website_line;
if
(matching_power.length() != 0)
executeLine("repeatThreshold = "
+
matching_power + ";");
executeScript("AgtFiles/Search.agt");
return
"hello";
}
public String
setSearchString(String search_str)
{
if
(search_str_set != 0)
return
";";
search_str_set = 1;
executeLine("searchString
=\"" + search_str + "\";");
return
executeScript("AgtFiles/InitQuery.agt");
}
public void
setStarttoParent()
{
executeLine("childTag = Name(startQueryPiece);");
executeLine("startQueryPiece = Parent(startQueryPiece);");
}
public void executeQuery(int dir)
{
String query
= null;
if (dir != 0)
{
String
child_tag = executeLine("childTag;");
String
query_stmt = executeLine("queryStmt;");
query =
"Elem(P,\"" + child_tag + "\") directlyinside "
+
query_stmt;
} else {
query =
executeScript("AgtFiles/Query.agt");
}
executeLine("records = " + query + ";");
String
ShowRec = "DummyRec = " + query
+ "
contain (Pat(P,searchString)[0]);";
executeLine(ShowRec);
executeScript("AgtFiles/ShowRec.agt");
}
public int
getNumberOfPatterns()
{
String
result = executeLine(" Size(ToList(patterns));" );
number_of_patterns = Integer.parseInt(result);
return
number_of_patterns;
}
public String
getFirstPattern()
{
String index = "";
if
(number_of_patterns == 0)
return
null;
current_pattern = 0;
index =
String.valueOf(current_pattern);
executeLine("import Str; tags = Str_Search(patterns[" +
index +
"][1], \"[<](.*?)[>]\");");
executeScript("AgtFiles/Pattern.agt");
showPattern();
return
"hello";
}
public String
getLastPattern()
{
String index = "";
if
(number_of_patterns == 0 )
return
null;
current_pattern = number_of_patterns - 1;
index =
String.valueOf(current_pattern);
executeLine("import Str; tags = Str_Search(patterns[" +
index +
"][1], \"[<](.*?)[>]\");");
executeScript("AgtFiles/Pattern.agt");
showPattern();
return
"hello";
}
public String
getNextPattern()
{
String index = "";
if
(number_of_patterns == 0 )
return
null;
if
(current_pattern >= number_of_patterns)
return
null;
current_pattern ++;
index =
String.valueOf(current_pattern);
executeLine("import Str; tags = Str_Search(patterns[" +
index +
"][1], \"[<](.*?)[>]\");");
executeScript("AgtFiles/Pattern.agt");
showPattern();
return
"hello";
}
public String
getPreviousPattern()
{
String index = "";
if (number_of_patterns
== 0 )
return
null;
if
(current_pattern == 0)
return
null;
current_pattern --;
index =
String.valueOf(current_pattern);
executeLine("import Str; tags = Str_Search(patterns[" +
index +
"][1], \"[<](.*?)[>]\");");
executeScript("AgtFiles/Pattern.agt");
showPattern();
return
"hello";
}
public String
showPattern()
{
String
result = executeLine("Size(ToList(weblstmt));");
int
numberStmt = Integer.parseInt(result);
String[]
weblstmt = new String[numberStmt];
for (int i =
0; i < numberStmt; i++)
{
result =
executeLine("weblstmt[" + String.valueOf(i) + "];");
weblstmt[i] = result;
String webltext1 = "weblobj[" +
String.valueOf(i) + "] := "
+
weblstmt[i] + "; weblmarkuptxt = weblmarkuptxt + Markup(weblobj[" +
String.valueOf(i) + "][3]);";
String webltext2 = "if (Size(weblobj[" +
String.valueOf(i) +
"]) > 0 and minweblobj == -1) then minweblobj = " +
String.valueOf(i) + "; end;";
String webltext3 = "if (Size(weblobj[" +
String.valueOf(i) +
"]) < Size(weblobj[minweblobj]) ) then minweblobj = " +
String.valueOf(i) + "; end;minweblobj;";
String
webltext = webltext1 + webltext2 + webltext3;
executeLine(webltext);
}
String
browser_text1 = "import Browser;";
String browser_text2
= "Browser_ShowPage(\"<html><body><hr>\"
+weblmarkuptxt+\"<hr></body></html>\");";
String
browser_text3 = "weblmarkuptxt = \"\";";
executeLine(browser_text1 + browser_text2 +
browser_text3);
return
"hello";
}
public int
selectPattern()
{
selected_pattern = current_pattern;
executeScript("AgtFiles/Select.agt");
return
selected_pattern;
}
public int
getNumberOfResults()
{
String
result = executeLine("Size(txt);");
int
number_of_results = Integer.parseInt(result);
return
number_of_results;
}
public String
findPattern(int index, String pattern)
{
String line1
= "import Str; currList = nil;";
String line2
= "currList = Str_Search(txt[" +
String.valueOf(index) + "],\"" + pattern +
"\");" ;
String line3
=
"if
( Size(currList) != 0) then Str_Trim(currList[0][1]); else \"empty\";
end;";
String
result = executeLine(line1 + line2 + line3);
return result;
}
public String
executeStmt(int index, String stmt)
{
String line1 = "dummy =\"\";CurrentRec =
NewPiece(Markup(records["
+
String.valueOf(index) +
"]), \"text/html\");";
String line2
= "import Str;";
String line3
= stmt;
executeLine(line1);
//String
CheckNilStmt = executeLine("CheckNilStmt;");
//if
(executeLine(CheckNilStmt).equals("empty"))
/// return "empty";
String
pre_result = executeLine(line2 + line3);
StringTokenizer st = new StringTokenizer(pre_result);
String
result = "";
while
(st.hasMoreTokens())
result +=
st.nextToken() + " ";
return
result;
}
public String
executePieceQuery()
{
return
executeScript("AgtFiles/PieceSearch.agt");
}
}
EventHandler.java
import java.awt.event.*;
import java.awt.*;
import java.util.*;
import java.io.*;
public class EventHandler
extends GuiUser
implements ActionListener
{
public void
actionPerformed(ActionEvent evt)
{
if
(evt.getActionCommand() == "MSG_SEARCHGUI_CONTINUE")
SearchGui_Btn_Continue(evt);
if
(evt.getActionCommand() == "MSG_SELECTGUI_UP")
SelectGui_Btn_Up(evt);
if
(evt.getActionCommand() == "MSG_SELECTGUI_DOWN")
SelectGui_Btn_Down(evt);
if
(evt.getActionCommand() == "MSG_SELECTGUI_FIRST")
SelectGui_Btn_First(evt);
if
(evt.getActionCommand() == "MSG_SELECTGUI_LAST")
SelectGui_Btn_Last(evt);
if
(evt.getActionCommand() == "MSG_SELECTGUI_PREVIOUS")
SelectGui_Btn_Previous(evt);
if
(evt.getActionCommand() == "MSG_SELECTGUI_NEXT")
SelectGui_Btn_Next(evt);
if
(evt.getActionCommand() == "MSG_SELECTGUI_SELECT")
SelectGui_Btn_Select(evt);
if
(evt.getActionCommand() == "MSG_LABELGUI_FORMLABEL")
LabelGui_Btn_FormLabel(evt);
if
(evt.getActionCommand() == "MSG_LABELGUI_EDITLABEL")
LabelGui_Btn_EditLabel(evt);
if (evt.getActionCommand()
== "MSG_LABELGUI_REMOVELABEL")
LabelGui_Btn_RemoveLabel(evt);
if
(evt.getActionCommand() == "MSG_LABELGUI_SHOW")
LabelGui_Btn_Show(evt);
if
(evt.getActionCommand() == "MSG_LABELGUI_RESTART")
LabelGui_Btn_Restart(evt);
if
(evt.getActionCommand() == "MSG_LABELGUI_BACK")
LabelGui_Btn_Back(evt);
if
(evt.getActionCommand() == "MSG_LABELGUI_DUMP")
LabelGui_Btn_Dump(evt);
if
(evt.getActionCommand() == "MSG_RULESGUI_PRECEDES")
RulesGui_Btn_Precedes(evt);
if
(evt.getActionCommand() == "MSG_RULESGUI_FOLLOWS")
RulesGui_Btn_Follows(evt);
if
(evt.getActionCommand() == "MSG_RULESGUI_AND")
RulesGui_Btn_And(evt);
if
(evt.getActionCommand() == "MSG_RULESGUI_OR")
RulesGui_Btn_Or(evt);
if
(evt.getActionCommand() == "MSG_RULESGUI_FINISH")
RulesGui_Btn_Finish(evt);
if
(evt.getActionCommand() == "MSG_RULESGUI_FINISH2")
RulesGui_Btn_Finish2(evt);
if
(evt.getActionCommand() == "MSG_RULESGUI_CANCEL")
RulesGui_Btn_Cancel(evt);
if
(evt.getActionCommand() == "MSG_PATTERNGUI_FINISH")
PatternGui_Btn_Finish(evt);
if
(evt.getActionCommand() == "MSG_PATTERNGUI_CANCEL")
PatternGui_Btn_Cancel(evt);
if
(evt.getActionCommand() == "MSG_PATTERNGUI_SAMPLE")
PatternGui_Btn_Sample(evt);
if
(evt.getActionCommand() == "MSG_RULESGUI_SWITCH_PATTERN")
RulesGui_Btn_SwitchToPattern(evt);
if (evt.getActionCommand()
== "MSG_VIEWGUI_FIRST")
ViewGui_Btn_First(evt);
if
(evt.getActionCommand() == "MSG_VIEWGUI_LAST")
ViewGui_Btn_Last(evt);
if
(evt.getActionCommand() == "MSG_VIEWGUI_PREVIOUS")
ViewGui_Btn_Previous(evt);
if (evt.getActionCommand() ==
"MSG_VIEWGUI_NEXT")
ViewGui_Btn_Next(evt);
if
(evt.getActionCommand() == "MSG_VIEWGUI_CLOSE")
ViewGui_Btn_Close(evt);
}
void
SearchGui_Btn_Continue(ActionEvent evt)
{
String website =
search_gui.theWebsiteBox.getText();
String
type_of_site =
search_gui.theTypeOfSiteBox.getText();
String
search_for_text =
search_gui.theSearchForBox.getText();
String
matching_power =
search_gui.theMatchingPowerBox.getText();
if
(website.length() == 0)
System.err.println(
"One of the field is empty");
else
{
number_of_records = 0;
agent.executeLine("website = \"" + website +
"\";");
agent.executeScript("AgtFiles/Pages/Search.agt");
search_gui.hideGui();
select_gui.showGui();
}
}
void
SelectGui_Btn_Up(ActionEvent evt)
{
agent.setSearchString(
select_gui.theSearchForBox.getText());
agent.setStarttoParent();
agent.executeQuery(0);
}
void
SelectGui_Btn_Down(ActionEvent evt)
{
//
agent.executeQuery(1);
}
void
SelectGui_Btn_First(ActionEvent evt)
{
agent.getFirstPattern();
}
void
SelectGui_Btn_Last(ActionEvent evt)
{
agent.getLastPattern();
}
void
SelectGui_Btn_Previous(ActionEvent evt)
{
agent.getPreviousPattern();
}
void
SelectGui_Btn_Next(ActionEvent evt)
{
agent.getNextPattern();
}
void
SelectGui_Btn_Select(ActionEvent evt)
{
//
select_gui.hideGui();
agent.selectPattern();
number_of_records = agent.getNumberOfResults();
label_gui.showGui();
}
void
LabelGui_Btn_FormLabel(ActionEvent evt)
{
//
rules_gui.theLabelBox.
setEditable(true);
rules_gui.showGui();
}
void
LabelGui_Btn_EditLabel(ActionEvent evt)
{
rules_gui.theLabelBox.setText(
label_gui.theLabelList.
getSelectedItem());
rules_gui.theLabelBox.
setEditable(false);
theLabelTable.remove(label_gui.theLabelList.
getSelectedItem());
label_gui.theLabelList.remove(label_gui.theLabelList.
getSelectedItem());
rules_gui.showGui();
}
void
LabelGui_Btn_RemoveLabel(ActionEvent evt)
{
//
theLabelTable.remove(label_gui.theLabelList.
getSelectedItem());
label_gui.theLabelList.remove(
label_gui.theLabelList.
getSelectedItem());
}
void
LabelGui_Btn_Show(ActionEvent evt)
{
view_gui.theViewArea.setText("");
view_gui.showGui();
}
void
LabelGui_Btn_Restart(ActionEvent evt)
{
label_gui.hideGui();
agent.startAgain();
search_gui.showGui();
}
void
LabelGui_Btn_Back(ActionEvent evt)
{
label_gui.hideGui();
select_gui.showGui();
}
void
LabelGui_Btn_Dump(ActionEvent evt)
{
Enumeration
labels = theLabelTable.keys();
String
newline = System.getProperty("line.separator");
try {
FileWriter
ruleswriter = new FileWriter(
search_gui.theWebsiteBox.getText() + ".rules");
while
(labels.hasMoreElements() )
{
String
label = (String) labels.nextElement();
String pattern = (String)
theLabelTable.get(label);
ruleswriter.write(label + "\t:\t" + pattern + newline
+
newline + newline);
ruleswriter.flush();
System.out.println("Dumped into file...");
}
} catch
(IOException ex) {
System.err.println(ex);
}
}
void
RulesGui_Btn_Precedes(ActionEvent evt)
{
String attribute =
rules_gui.theAttributeBox.getText();
if
(attribute.length() == 0)
return;
if
(attribute.startsWith("$"))
attribute
= "[$]";
if
(attribute.length() == 1)
attribute
= "[" + attribute + "]";
if
(rules_gui.theRulesBox.
getText().endsWith("AND ") )
scratchpad_pattern = scratchpad_pattern
+
"[ ]*"
+
attribute;
if
(rules_gui.theRulesBox.
getText().endsWith("OR ") )
scratchpad_pattern = scratchpad_pattern
+
"([a-zA-Z0-9:(),. ]*[a-zA-Z0-9:(),.]+)[ ]*"
+
attribute;
if
(rules_gui.theRulesBox.
getText().length() == 0 )
scratchpad_pattern = scratchpad_pattern
+
"([^\\\\\\\\t\\\\\\\\n\\\\\\\\r\\\\\\\\f]*\\\\\\\\S+)\\\\\\\\s*"
+ attribute;
rules_gui.theRulesBox.setText(
rules_gui.theRulesBox.getText() +
"
PRECEDES " + attribute);
rules_gui.theAttributeBox.setText("");
}
void
RulesGui_Btn_Follows(ActionEvent evt)
{
String attribute =
rules_gui.theAttributeBox.getText();
if
(attribute.length() == 0)
return;
if
(attribute.startsWith("$"))
attribute
= "[$]";
if
(attribute.length() == 1)
attribute
= "[" + attribute + "]";
if (rules_gui.theRulesBox.
getText().endsWith("AND ") )
scratchpad_pattern = attribute
+
"[ ]*" + scratchpad_pattern;
if
(rules_gui.theRulesBox.
getText().endsWith("OR ") )
scratchpad_pattern = scratchpad_pattern
+
attribute
+ "[ ]*([a-zA-Z0-9:(),. ]*[a-zA-Z0-9:(),.]+)";
if
(rules_gui.theRulesBox.
getText().length() == 0 )
scratchpad_pattern = scratchpad_pattern
+ attribute
+
"\\\\\\\\s*([^\\\\\\\\t\\\\\\\\r\\\\\\\\n\\\\\\\\f]*\\\\\\\\S+)";
rules_gui.theRulesBox.setText(
rules_gui.theRulesBox.getText() +
"
FOLLOWS " + attribute);
rules_gui.theAttributeBox.setText("");
}
void
RulesGui_Btn_And(ActionEvent evt)
{
//
if
(rules_gui.theRulesBox.
getText().length() == 0)
return;
if
(rules_gui.theRulesBox.
getText().endsWith("AND ") )
return;
if (rules_gui.theRulesBox.
getText().endsWith("OR ") )
return;
rules_gui.theRulesBox.setText(
rules_gui.theRulesBox.getText() +
" AND ");
}
void
RulesGui_Btn_Or(ActionEvent evt)
{
//
if (rules_gui.theRulesBox.
getText().length() == 0)
return;
if
(rules_gui.theRulesBox.
getText().endsWith("AND ") )
return;
if
(rules_gui.theRulesBox.
getText().endsWith("OR ") )
return;
rules_gui.theRulesBox.setText(
rules_gui.theRulesBox.getText() +
" OR ");
scratchpad_pattern = scratchpad_pattern
+ " |
";
}
/*
void
RulesGui_Btn_Finish(ActionEvent evt)
{
String label
= rules_gui.theLabelBox.getText();
if
(label.length() == 0)
return;
if
(theLabelTable.containsKey(label))
return;
rules_gui.theRulesBox.setText("");
rules_gui.theLabelBox.setText("");
rules_gui.theAttributeBox.setText("");
// Add Code
Here
agent.executeLine(" labelQueryStmt =
Pat(CurrentRec,\"(?im)" +
scratchpad_pattern + "\")";
String
webl_stmt = agent.executeScript("AgtFiles/PieceSearch2.agt");
theLabelTable.put(label, webl_stmt);
scratchpad_pattern = "";
label_gui.theLabelList.add(label);
rules_gui.hideGui();
}
*/
void
RulesGui_Btn_Finish(ActionEvent evt)
{
String label
= rules_gui.theLabelBox.getText();
if
(label.length() == 0)
return;
if
(theLabelTable.containsKey(label))
return;
rules_gui.theRulesBox.setText("");
rules_gui.theLabelBox.setText("");
agent.executeLine("labelSearchString = \"\";");
rules_gui.theAttributeBox.setText("");
// Add Code
Here
agent.executeLine(" labelQueryStmt =
\"Pat(CurrentRec,\\\"(?im) " +
scratchpad_pattern + "\\\")\";");
String
webl_stmt = agent.executeScript("AgtFiles/PieceSearch2.agt");
theLabelTable.put(label, webl_stmt);
String checkstmt =
agent.executeLine("CheckNilStmt;");
theCheckTable.put(label, checkstmt);
scratchpad_pattern = "";
label_gui.theLabelList.add(label);
rules_gui.hideGui();
}
void
RulesGui_Btn_Finish2(ActionEvent evt)
{
String label
= rules_gui.theLabelBox.getText();
if
(label.length() == 0)
return;
if
(theLabelTable.containsKey(label))
return;
rules_gui.theRulesBox.setText("");
rules_gui.theLabelBox.setText("");
agent.executeLine("labelSearchString
= \"" +
rules_gui.theAttributeBox.getText() + "\";");
rules_gui.theAttributeBox.setText("");
// Add Code
Here
String
webl_stmt = agent.executePieceQuery();
theLabelTable.put(label, webl_stmt);
String
checkstmt = agent.executeLine("CheckNilStmt;");
theCheckTable.put(label, checkstmt);
scratchpad_pattern = "";
label_gui.theLabelList.add(label);
rules_gui.hideGui();
}
void
RulesGui_Btn_Cancel(ActionEvent evt)
{
rules_gui.theRulesBox.setText("");
rules_gui.theLabelBox.setText("");
rules_gui.theAttributeBox.setText("");
rules_gui.hideGui();
}
void
RulesGui_Btn_SwitchToPattern(ActionEvent evt)
{
pattern_gui.showGui();
}
void
PatternGui_Btn_Sample(ActionEvent evt)
{
pattern_gui.sampleInput();
}
void
PatternGui_Btn_Finish(ActionEvent evt)
{
String label
= rules_gui.theLabelBox.getText();
if
(label.length() == 0)
return;
if
(theLabelTable.containsKey(label))
return;
rules_gui.theRulesBox.setText("");
rules_gui.theLabelBox.setText("");
agent.executeLine("labelSearchString = \"\";");
rules_gui.theAttributeBox.setText("");
// Add Code
Here
agent.executeLine("
labelQueryStmt=\"Pat(CurrentRec,\\\"(?im) "
+
pattern_gui.getPattern() + "\\\")\";");
String webl_stmt =
agent.executeScript("AgtFiles/PieceSearch2.agt");
theLabelTable.put(label, webl_stmt);
String
checkstmt = agent.executeLine("CheckNilStmt;");
theCheckTable.put(label, checkstmt);
scratchpad_pattern = "";
label_gui.theLabelList.add(label);
rules_gui.hideGui();
pattern_gui.hideGui();
}
void
PatternGui_Btn_Cancel(ActionEvent evt)
{
pattern_gui.hideGui();
}
void
ViewGui_Btn_First(ActionEvent evt)
{
view_gui.theViewArea.setText("");
Enumeration
labels = theLabelTable.keys();
record_cursor
= 0;
String value
= "";
String line1
= "dummy =\"\";CurrentRec = NewPiece(Markup(records["
+
String.valueOf(record_cursor) +
"]), \"text/html\");";
agent.executeLine(line1);
while
(labels.hasMoreElements() )
{
String
label = (String) labels.nextElement();
String
pattern = (String) theLabelTable.get(label);
String
checkstmt = (String) theCheckTable.get(label);
if
(agent.executeLine("dummy = \"\";" +
checkstmt).equals("empty")) {
value =
"empty";
} else {
value =
agent.executeStmt(
record_cursor, pattern);
}
view_gui.theViewArea.append(
label +
": " + value + "\n");
}
}
void
ViewGui_Btn_Last(ActionEvent evt)
{
record_cursor
= number_of_records - 1;
String value
= "";
view_gui.theViewArea.setText("");
Enumeration
labels = theLabelTable.keys();
String line1
= "dummy =\"\";CurrentRec = NewPiece(Markup(records["
+
String.valueOf(record_cursor) +
"]), \"text/html\");";
agent.executeLine(line1);
while
(labels.hasMoreElements() )
{
String
label = (String) labels.nextElement();
String
pattern = (String) theLabelTable.get(label);
String
checkstmt = (String) theCheckTable.get(label);
if
(agent.executeLine("dummy = \"\";"
+checkstmt).equals("empty")) {
value =
"empty";
} else {
value =
agent.executeStmt(
record_cursor, pattern);
}
view_gui.theViewArea.append(
label +
": " + value + "\n");
}
}
void
ViewGui_Btn_Previous(ActionEvent evt)
{
String value
= "";
if
(record_cursor == 0) return;
record_cursor--;
view_gui.theViewArea.setText("");
Enumeration
labels = theLabelTable.keys();
String line1
= "dummy =\"\";CurrentRec = NewPiece(Markup(records["
+
String.valueOf(record_cursor) +
"]), \"text/html\");";
agent.executeLine(line1);
while
(labels.hasMoreElements() )
{
String
label = (String) labels.nextElement();
String
pattern = (String) theLabelTable.get(label);
String
checkstmt = (String) theCheckTable.get(label);
if
(agent.executeLine("dummy = \"\";"
+checkstmt).equals("empty")) {
value =
"empty";
} else {
value =
agent.executeStmt(
record_cursor, pattern);
}
view_gui.theViewArea.append(
label + ": " + value +
"\n");
}
}
void
ViewGui_Btn_Next(ActionEvent evt)
{
String value
= "";
if
(record_cursor ==
(number_of_records - 1) ) return;
record_cursor++;
view_gui.theViewArea.setText("");
Enumeration
labels = theLabelTable.keys();
String line1
= "dummy =\"\";CurrentRec = NewPiece(Markup(records["
+
String.valueOf(record_cursor) +
"]), \"text/html\");";
agent.executeLine(line1);
while
(labels.hasMoreElements() )
{
String
label = (String) labels.nextElement();
String
pattern = (String) theLabelTable.get(label);
String
checkstmt = (String) theCheckTable.get(label);
if
(agent.executeLine("dummy = \"\";"
+checkstmt).equals("empty")) {
value =
"empty";
} else {
value =
agent.executeStmt(
record_cursor, pattern);
}
view_gui.theViewArea.append(
label +
": " + value + "\n");
}
}
void
ViewGui_Btn_Close(ActionEvent evt)
{
view_gui.hideGui();
}
}
WeblEngine.java
import webl.lang.*;
import webl.lang.expr.*;
import webl.util.*;
import java.io.*;
import java.util.*;
public class WeblEngine {
static Machine
machine;
static Evaluator
evaluator;
static
FileWriter filewriter;
int
disable_filewrite = 0;
int
starting_line_no = 1;
String newline =
System.getProperty("line.separator");
public
WeblEngine()
{
try
{
// Load webl.properties file (contains
proxy settings etc)
Properties props = new
Properties(System.getProperties());
// We would like debugging output to the console
Log.SetLogger(new ConsoleLog(true));
// Allocate
a fresh WebL execution machine
machine =
new webl.lang.Machine("Startup.webl");
// Set the
command line arguments of the WebL machine
machine.SetARGS(new String[] {"arg1", "arg2",
"arg3"});
//
Allocate the evaluator which we will "feed" the script to
incrementally.
// Note:
An optional Logger can be passed to the Evaluator to catch syntax
// errors during script compilation,
evaluator
= new Evaluator(machine, "Scriptor");
filewriter
= new FileWriter("gen_script.txt");
}
catch
(IOException ex1) { System.out.println(ex1.toString()); }
catch
(WebLException ex2) { System.out.println(ex2.report()); }
catch
(WebLReturnException ex3)
{
// A WebL "return" statement
was executed outside a function or method
System.out.println(ex3.toString());
}
}
public Expr
executeScript(String script)
{
Expr result
= null;
try
{
result =
evaluator.Exec(script, starting_line_no);
starting_line_no++;
if
(result == null)
System.out.println("executeScript(): Syntax Error in Script");
else
System.out.println("executeScript(): " + result.print());
if
(disable_filewrite != 0) {
filewriter.write(script + newline);
filewriter.flush();
}
}
catch
(IOException ex1) { ex1.toString(); }
catch
(WebLException ex2) { ex2.report(); }
catch (WebLReturnException ex3) {
ex3.toString(); }
catch
(Exception ex) { System.err.println(ex); }
return
result;
}
public void
enableFileWriter()
{
disable_filewrite = 1;
}
public void
disableFileWriter()
{
disable_filewrite = 0;
}
}
GuiUser.java
import java.util.*;
import java.io.*;
public class GuiUser {
static SearchGui
search_gui = new SearchGui();
static SelectGui
select_gui = new SelectGui();
static RulesGui
rules_gui = new RulesGui();
static
PatternGui pattern_gui = new
PatternGui();
static LabelGui
label_gui = new LabelGui();
static ViewGui
view_gui = new ViewGui();
static
GenericAgent agent = new
GenericAgent();
static Hashtable
theLabelTable = new Hashtable();
static Hashtable
theCheckTable = new Hashtable();
static int
number_of_records;
static String
scratchpad_pattern = "";
static int
record_cursor = 0;
public static
void main(String [] args)
{
search_gui.showGui();
}
}
LabelGui.java
import java.awt.*;
import java.awt.event.*;
import java.io.*;
import java.util.*;
public class LabelGui
{
Frame
theLabelWindow = new Frame("Label");
Button
theFormLabelButton = new Button(
"Form new
label");
Button theEditLabelButton = new Button(
"Edit
selected label");
Button theRemoveLabelButton = new Button(
"Remove
Label");
Button theShowButton = new Button(
"Show!");
Button theRestartButton = new Button(
"Restart");
Button theBackButton = new Button(
"Back");
Button theDumpButton = new Button(
"Dump
Rules");
public
java.awt.List theLabelList =
new
java.awt.List(10, true);
public
LabelGui()
{
theLabelList.setMultipleMode(false);
theLabelWindow.setBackground(Color.lightGray);
theLabelWindow.setLayout(
new
BorderLayout());
theLabelWindow.add(new Label(
"Labels"), BorderLayout.NORTH);
Panel theEditPanel = new Panel(
new FlowLayout(FlowLayout.CENTER));
theEditPanel.add(theFormLabelButton);
theEditPanel.add(theEditLabelButton);
theEditPanel.add(theRemoveLabelButton);
Panel
theViewPanel = new Panel(
new
FlowLayout(FlowLayout.CENTER));
theViewPanel.add(theShowButton);
theViewPanel.add(theRestartButton);
theViewPanel.add(theBackButton);
theViewPanel.add(theDumpButton);
theLabelWindow.add(theEditPanel,
BorderLayout.NORTH);
theLabelWindow.add(theLabelList,
BorderLayout.CENTER);
theLabelWindow.add(theViewPanel,
BorderLayout.SOUTH);
WindowListener theLabelWindowListener =
new
WindowAdapter()
{
public
void windowClosing( WindowEvent evt)
{
System.exit(0);
}
};
theFormLabelButton.setActionCommand(
"MSG_LABELGUI_FORMLABEL");
theFormLabelButton.addActionListener(
new
EventHandler());
theEditLabelButton.setActionCommand(
"MSG_LABELGUI_EDITLABEL");
theEditLabelButton.addActionListener(
new
EventHandler());
theRemoveLabelButton.setActionCommand(
"MSG_LABELGUI_REMOVELABEL");
theRemoveLabelButton.addActionListener(
new
EventHandler());
theShowButton.setActionCommand(
"MSG_LABELGUI_SHOW");
theShowButton.addActionListener(
new
EventHandler());
theRestartButton.setActionCommand(
"MSG_LABELGUI_RESTART");
theRestartButton.addActionListener(
new
EventHandler());
theBackButton.setActionCommand(
"MSG_LABELGUI_BACK");
theBackButton.addActionListener(
new
EventHandler());
theDumpButton.setActionCommand(
"MSG_LABELGUI_DUMP");
theDumpButton.addActionListener(
new
EventHandler());
theLabelWindow.addWindowListener(
theLabelWindowListener);
theLabelWindow.pack();
}
public void
showGui()
{
theLabelWindow.setVisible(true);
}
public void hideGui()
{
theLabelWindow.setVisible(false);
}
}
PatternGui.java
import java.awt.*;
import java.awt.event.*;
import java.io.*;
import java.util.*;
public class PatternGui
{
Frame
thePatternWindow = new Frame("Pattern");
TextField theLearnedPatternBox
= new TextField(20);
String
theLearnedPattern = "";
Button
theFinishButton = new Button(
"Click
here when finished");
Button
theCancelButton = new Button(
"Cancel");
Panel
thePatternButtonPanel =
new Panel(new
GridLayout(2,1));
TextField
theSampleInputBox =
new
TextField(10);
Button
theSamplingButton =
new
Button("New Sample");
public
PatternGui()
{
thePatternWindow.setBackground(Color.lightGray);
thePatternWindow.setLayout(new BorderLayout());
thePatternWindow.add(theLearnedPatternBox,
BorderLayout.NORTH);
theLearnedPatternBox.setEditable(false);
thePatternButtonPanel.add(theSampleInputBox);
thePatternButtonPanel.add(theSamplingButton);
thePatternWindow.add(thePatternButtonPanel,
BorderLayout.CENTER);
Panel
theFinishPanel = new Panel();
theFinishPanel.add(theFinishButton);
theFinishPanel.add(theCancelButton);
thePatternWindow.add(theFinishPanel,
BorderLayout.SOUTH);
theFinishButton.setActionCommand(
"MSG_PATTERNGUI_FINISH");
theFinishButton.addActionListener(
new
EventHandler());
theCancelButton.setActionCommand(
"MSG_PATTERNGUI_CANCEL");
theCancelButton.addActionListener(
new
EventHandler());
theSamplingButton.setActionCommand(
"MSG_PATTERNGUI_SAMPLE");
theSamplingButton.addActionListener(
new
EventHandler());
WindowListener thePatternWindowListener =
new
WindowAdapter()
{
public
void windowClosing( WindowEvent evt)
{
System.err.println("Exiting from PatternGui");
System.exit(0);
}
};
thePatternWindow.addWindowListener(thePatternWindowListener);
thePatternWindow.pack();
}
public void
showGui()
{
init();
thePatternWindow.setVisible(true);
}
public void
hideGui()
{
thePatternWindow.setVisible(false);
}
public void
sampleInput()
{
if
(theSampleInputBox.getText().length() == 0)
return;
if
(theSampleInputBox.getText().length() != 0
&&
theLearnedPatternBox.getText().length() != 0
&& theLearnedPatternBox.getText().length() !=
theSampleInputBox.getText().length() )
return;
if
(theLearnedPatternBox.getText().length() == 0)
{
theLearnedPatternBox.setText(
theSampleInputBox.getText());
return;
}
int
sample_length = theSampleInputBox.getText().length();
char[]
learned_str =
theLearnedPatternBox.getText().toCharArray();
char[]
sample_str =
theSampleInputBox.getText().toCharArray();
String
new_learned_str = "";
for (int i =
0; i < sample_length; i++)
{
char
sample_ch = sample_str[i];
char
learned_ch = learned_str[i];
if
(sample_ch != learned_ch &&
Character.isDigit(sample_ch) &&
Character.isDigit(learned_ch) ||
learned_ch == 'd')
{ learned_ch = 'd';
new_learned_str += learned_ch;
continue; }
if
(sample_ch != learned_ch &&
Character.isLetter(sample_ch) &&
Character.isLetter(learned_ch) ||
learned_ch == 'l')
{ learned_ch = 'l';
new_learned_str += learned_ch;
continue; }
if
(sample_ch != learned_ch &&
Character.isLetter(sample_ch) &&
Character.isDigit(learned_ch) ||
learned_ch == 'a')
{ learned_ch = 'a';
new_learned_str += learned_ch;
continue; }
if
(sample_ch != learned_ch &&
Character.isDigit(sample_ch) &&
Character.isLetter(learned_ch) ||
learned_ch == 'a')
{ learned_ch = 'a';
new_learned_str += learned_ch;
continue; }
new_learned_str += learned_ch;
}
theLearnedPatternBox.setText(new_learned_str);
}
public void
init()
{
theLearnedPatternBox.setText("");
theSampleInputBox.setText("");
}
public String
getPattern()
{
if
(theLearnedPatternBox.getText().length() == 0)
return
"()";
char[] learned_str
=
theLearnedPatternBox.getText().toCharArray();
int
learned_len = theLearnedPatternBox.getText().length();
String
pattern = "";
for (int i =
0; i < learned_len; i++)
{
char ch =
learned_str[i];
if (ch
=='d')
pattern
+= "[0-9]";
if (ch
=='l')
pattern
+= "[a-zA-Z]";
if (ch
=='a')
pattern
+= "[0-9a-zA-Z]";
if (ch !=
'a' &&
ch
!= 'd' &&
ch != 'l')
pattern += "[" + ch + "]";
}
pattern =
"(" + pattern +
")";
return
pattern;
}
}
RulesGui.java
import java.awt.*;
import java.awt.event.*;
import java.io.*;
import java.util.*;
public class RulesGui
{
Frame
theRulesWindow = new Frame("Specify Rules For Item Extraction From
Record");
public TextField
theRulesBox = new TextField(30);
public TextField
theLabelBox = new TextField(20);
public TextField
theAttributeBox = new TextField(20);
public TextField
theLearnedPatternBox = new TextField(20);
Button
thePrecedesButton = new Button("Precedes Attribute");
Button
theFollowsButton = new Button("Follows Attribute");
Button
theAndButton = new Button("AND");
Button
theOrButton = new Button("OR");
Button
theSwitchButton = new Button(
"Click
here for pattern learning");
Button
theFinishButton = new Button(
"Click
here when finished");
Button
theFinish2Button = new Button(
"Click
here when finished if using hints");
Button
theCancelButton = new Button(
"Cancel");
Panel
theRulesButtonPanel =
new Panel(new
GridLayout(3,1));
Panel
thePatternButtonPanel =
new Panel(new
GridLayout(3,1));
Button
theSwitchToRulesButton =
new Button("Switch
to rules panel");
TextField
theSampleInputBox =
new
TextField(10);
Button
theSamplingButton =
new
Button("New Sample");
Button
theDoneButton =
new
Button("Done");
Panel
theLabelRulesPanel = new Panel(
new
GridLayout(2,1));
public
RulesGui()
{
theRulesWindow.setBackground(Color.lightGray);
theRulesWindow.setLayout(new BorderLayout());
theRulesBox.setEditable(false);
theRulesWindow.add(theRulesBox,
BorderLayout.NORTH);
Panel
theLabelRulesPanel = new Panel(
new
GridLayout(2,1));
Panel
theLabelAttributePanel = new Panel(
new
GridLayout(2,1));
Panel
theLabelPanel = new Panel();
theLabelPanel.add(new Label("Label name:"));
theLabelPanel.add(theLabelBox);
theLabelAttributePanel.add(theLabelPanel);
Panel
theAttributePanel = new Panel();
theAttributePanel.add(new Label("Attribute name:"));
theAttributePanel.add(theAttributeBox);
theLabelAttributePanel.add(theAttributePanel);
theLabelRulesPanel.add(theLabelAttributePanel);
Panel
thePrecedesFollowsPanel = new Panel();
thePrecedesFollowsPanel.add(thePrecedesButton);
thePrecedesFollowsPanel.add(theFollowsButton);
theRulesButtonPanel.add(thePrecedesFollowsPanel);
Panel
theAndOrPanel = new Panel();
theAndOrPanel.add(theAndButton);
theAndOrPanel.add(theOrButton);
theRulesButtonPanel.add(theAndOrPanel);
theRulesButtonPanel.add(theSwitchButton);
theLabelRulesPanel.add(theRulesButtonPanel);
theRulesWindow.add(theLabelRulesPanel,
BorderLayout.CENTER);
Panel
theFinishPanel = new Panel();
theFinishPanel.add(theFinishButton);
theFinishPanel.add(theFinish2Button);
theFinishPanel.add(theCancelButton);
theRulesWindow.add(theFinishPanel,
BorderLayout.SOUTH);
thePatternButtonPanel.add(theLearnedPatternBox);
Panel
theSamplingPanel = new Panel();
theSamplingPanel.add(theSampleInputBox);
theSamplingPanel.add(theSamplingButton);
theSamplingPanel.add(theDoneButton);
thePatternButtonPanel.add(theSamplingPanel);
thePatternButtonPanel.add(theSwitchToRulesButton);
thePrecedesButton.setActionCommand(
"MSG_RULESGUI_PRECEDES");
thePrecedesButton.addActionListener(
new
EventHandler());
theFollowsButton.setActionCommand(
"MSG_RULESGUI_FOLLOWS");
theFollowsButton.addActionListener(
new
EventHandler());
theAndButton.setActionCommand(
"MSG_RULESGUI_AND");
theAndButton.addActionListener(
new
EventHandler());
theOrButton.setActionCommand(
"MSG_RULESGUI_OR");
theOrButton.addActionListener(
new
EventHandler());
theFinishButton.setActionCommand(
"MSG_RULESGUI_FINISH");
theFinishButton.addActionListener(
new
EventHandler());
theFinish2Button.setActionCommand(
"MSG_RULESGUI_FINISH2");
theFinish2Button.addActionListener(
new
EventHandler());
theCancelButton.setActionCommand(
"MSG_RULESGUI_CANCEL");
theCancelButton.addActionListener(
new
EventHandler());
theDoneButton.setActionCommand(
"MSG_RULESGUI_DONE");
theDoneButton.addActionListener(
new
EventHandler());
theSamplingButton.setActionCommand(
"MSG_RULESGUI_SAMPLE");
theSamplingButton.addActionListener(
new
EventHandler());
theSwitchButton.setActionCommand(
"MSG_RULESGUI_SWITCH_PATTERN");
theSwitchButton.addActionListener(
new
EventHandler());
theSwitchToRulesButton.setActionCommand(
"MSG_RULESGUI_SWITCH_RULES");
theSwitchToRulesButton.addActionListener(
new EventHandler());
WindowListener theRulesWindowListener =
new
WindowAdapter()
{
public
void windowClosing( WindowEvent evt)
{
System.err.println("Exiting from RulesGui");
System.exit(0);
}
};
theRulesWindow.addWindowListener(theRulesWindowListener);
theRulesWindow.pack();
}
public void
showGui()
{
theRulesWindow.setVisible(true);
}
public void
hideGui()
{
theRulesWindow.setVisible(false);
}
public void
switchToPattern()
{
Graphics g =
theRulesButtonPanel.getGraphics();
theRulesButtonPanel = thePatternButtonPanel;
theRulesButtonPanel.validate();
theRulesButtonPanel.paintAll(g);
System.out.println("Remove
me");
}
public void
switchToRules()
{
System.out.println("Remove scomponent");
}
}
SearchGui.java
import java.awt.*;
import java.awt.event.*;
import java.util.*;
import java.io.*;
public class SearchGui
{
Frame theSearchWindow = new
Frame("Search");
public TextField
theWebsiteBox = new TextField(20);
public TextField
theTypeOfSiteBox = new TextField(20);
public TextField
theSearchForBox = new TextField(20);
public TextField
theMatchingPowerBox = new TextField(3);
Button
theContinueButton = new Button("Continue");
public
SearchGui()
{
theSearchWindow.setBackground(Color.lightGray);
theSearchWindow.setLayout(new BorderLayout());
theSearchWindow.add(
new
Panel(), BorderLayout.NORTH);
Panel
theSearchPanel = new Panel(
new
GridLayout(4, 1));
theSearchWindow.add(theSearchPanel,
BorderLayout.CENTER);
Panel
theWebsitePanel = new Panel(
new
FlowLayout(FlowLayout.CENTER));
theWebsitePanel.add(new Label(
"Enter Filename (e.g. companyname.com) :"));
theWebsitePanel.add(theWebsiteBox);
theSearchPanel.add(theWebsitePanel);
Panel
theTypeOfSitePanel = new Panel(
new
FlowLayout(FlowLayout.CENTER));
theTypeOfSitePanel.add(new Label("Type of site:"));
theTypeOfSitePanel.add(theTypeOfSiteBox);
//theSearchPanel.add(theTypeOfSitePanel);
Panel
theSearchForPanel = new Panel(
new
FlowLayout( FlowLayout.CENTER));
theSearchForPanel.add( new
Label("Search for:"));
theSearchForPanel.add( theSearchForBox);
//theSearchPanel.add( theSearchForPanel);
Panel
theMatchingPowerPanel = new Panel(
new
FlowLayout( FlowLayout.CENTER));
theMatchingPowerPanel.add(
new Label("Matching Power (3..9):"));
theMatchingPowerPanel.add( theMatchingPowerBox);
//theSearchPanel.add( theMatchingPowerPanel);
theSearchWindow.add(theContinueButton,
BorderLayout.SOUTH);
theSearchWindow.pack();
theContinueButton.setActionCommand(
"MSG_SEARCHGUI_CONTINUE");
theContinueButton.addActionListener(
new
EventHandler());
//Adds event
handler for theSearchWindow
WindowListener theSearchWindowEventHandler
= new
WindowAdapter()
{
public
void windowClosing( WindowEvent evt)
{
System.exit(0);
}
};
theSearchWindow.addWindowListener(
theSearchWindowEventHandler);
}
public void
showGui()
{
theSearchWindow.setVisible(true);
}
public void
hideGui()
{
theSearchWindow.setVisible(false);
}
}
SelectGui.java
import java.awt.*;
import java.awt.event.*;
import java.util.*;
import java.io.*;
public class SelectGui {
Frame
theSelectWindow = new Frame("Record Representation");
public TextField
theSearchForBox =
new
TextField(20);
Button
theSelectWindowUpButton =
new
Button("Click here to see");
Button theSelectWindowDownButton
=
new
Button("Previous one was better");
Button
theSelectWindowFirstButton =
new
Button("First");
Button
theSelectWindowLastButton =
new
Button("Last");
Button
theSelectWindowNextButton =
new Button("Next");
Button
theSelectWindowPrevButton =
new
Button("Previous");
Button
theSelectWindowSelectButton =
new
Button("Yes, this represents a record");
public
SelectGui()
{
theSelectWindow.setBackground(
Color.lightGray);
theSelectWindow.setLayout(
new
GridLayout(2,1));
Panel
theSearchForPanel = new Panel();
theSearchForPanel.add(new Label(
"Enter a hint: "));
theSearchForPanel.add(theSearchForBox);
theSelectWindow.add(
theSearchForPanel);
Panel
theSelectButtonPanel = new Panel();
theSelectButtonPanel.add(
theSelectWindowUpButton);
theSelectButtonPanel.add(
theSelectWindowDownButton);
theSelectButtonPanel.add(
theSelectWindowSelectButton);
theSelectWindow.add(
theSelectButtonPanel);
theSelectWindow.pack();
theSelectWindowUpButton.setActionCommand(
"MSG_SELECTGUI_UP");
theSelectWindowUpButton.addActionListener(
new
EventHandler());
theSelectWindowDownButton.setActionCommand(
"MSG_SELECTGUI_DOWN");
theSelectWindowDownButton.addActionListener(
new
EventHandler());
theSelectWindowFirstButton.setActionCommand(
"MSG_SELECTGUI_FIRST");
theSelectWindowFirstButton.addActionListener(
new
EventHandler());
theSelectWindowLastButton.setActionCommand(
"MSG_SELECTGUI_LAST");
theSelectWindowLastButton.addActionListener(
new
EventHandler());
theSelectWindowPrevButton.setActionCommand(
"MSG_SELECTGUI_PREVIOUS");
theSelectWindowPrevButton.addActionListener(
new
EventHandler());
theSelectWindowNextButton.setActionCommand(
"MSG_SELECTGUI_NEXT");
theSelectWindowNextButton.addActionListener(
new
EventHandler());
theSelectWindowSelectButton.setActionCommand(
"MSG_SELECTGUI_SELECT");
theSelectWindowSelectButton.addActionListener(
new
EventHandler());
//Adds event
handler for theSelectWindow
WindowListener theSelectWindowEventHandler
= new
WindowAdapter()
{
public
void windowClosing( WindowEvent evt)
{
System.exit(0);
}
};
theSelectWindow.addWindowListener(
theSelectWindowEventHandler);
}
public void
showGui()
{
theSelectWindow.setVisible(true);
}
public void
hideGui()
{
theSelectWindow.setVisible(false);
}
}
ViewGui.java
import java.awt.*;
import java.awt.event.*;
import java.util.*;
import java.io.*;
public class ViewGui {
Frame
theViewWindow = new Frame("View Records");
Button
theViewWindowFirstButton =
new
Button("First");
Button
theViewWindowLastButton =
new
Button("Last");
Button
theViewWindowNextButton =
new
Button("Next");
Button
theViewWindowPrevButton =
new
Button("Previous");
Button
theViewWindowCloseButton =
new Button("Close");
public TextArea
theViewArea = new TextArea(5,24);
public ViewGui()
{
theViewWindow.setBackground(Color.lightGray);
theViewWindow.setLayout(
new
BorderLayout());
theViewWindow.add(theViewArea,
BorderLayout.CENTER);
Panel
theViewButtonPanel = new Panel();
theViewButtonPanel.add(
theViewWindowFirstButton);
theViewButtonPanel.add(
theViewWindowPrevButton);
theViewButtonPanel.add(
theViewWindowNextButton);
theViewButtonPanel.add(
theViewWindowLastButton);
theViewButtonPanel.add(
theViewWindowCloseButton);
theViewWindow.add(theViewButtonPanel,
BorderLayout.SOUTH);
theViewWindow.pack();
theViewWindowFirstButton.setActionCommand(
"MSG_VIEWGUI_FIRST");
theViewWindowFirstButton.addActionListener(
new
EventHandler());
theViewWindowLastButton.setActionCommand(
"MSG_VIEWGUI_LAST");
theViewWindowLastButton.addActionListener(
new
EventHandler());
theViewWindowPrevButton.setActionCommand(
"MSG_VIEWGUI_PREVIOUS");
theViewWindowPrevButton.addActionListener(
new
EventHandler());
theViewWindowNextButton.setActionCommand(
"MSG_VIEWGUI_NEXT");
theViewWindowNextButton.addActionListener(
new
EventHandler());
theViewWindowCloseButton.setActionCommand(
"MSG_VIEWGUI_CLOSE");
theViewWindowCloseButton.addActionListener(
new
EventHandler());
//Adds event
handler for theSelectWindow
WindowListener theViewWindowEventHandler
= new
WindowAdapter()
{
public
void windowClosing( WindowEvent evt)
{
System.exit(0);
}
};
theViewWindow.addWindowListener(
theViewWindowEventHandler);
}
public void
showGui()
{
theViewWindow.setVisible(true);
}
public void
hideGui()
{
theViewWindow.setVisible(false);
}
}
Init.agt
var pathname = "";
var P = nil;
var website = "default";
var markuptxt = "";
var pieceset = nil;
var piece = nil;
var str = nil;
var pieceptr = 0;
var currList = nil;
var currptr = 0;
var attribList = nil;
var queryResults = nil;
var minlistsize = 99999;
var tags = nil;
var stack = [..];
var stackstmt = [..];
var level = 0;
var lastlevel = -1;
var index = 0;
var patternindex = 0;
var weblobj = [..];
var minweblobj = -1;
var weblmarkuptxt = "";
var txt = [];
var emptyObj = [..];
emptyObj[0]:="";
emptyObj[1]:="";
emptyObj[2]:="";
var resultList = [];
var repeatThreshold = 3;
var weblstmt = [..];
var searchString = "";
var searchStringPiece = nil;
var taggedPiece = nil;
var queryStmt = nil;
var queryPiece = nil;
var startQueryPiece = nil;
var childTag = "";
var records = nil;
var CurrentRec = nil;
var labelSearchStringPiece = nil;
var TempRecord = nil;
var labelSearchString = "";
var labelTaggedPiece = nil;
var labelQueryPiece = nil;
var labelQueryStmt = "";
var CheckNilStmt = "";
var dummy = "";
var TempPage = nil;
var DummyRec = nil;
var PatStmt = "";
InitQuery.agt
TempPage = NewPage(Markup(P), "text/html");
searchStringPiece = Pat(TempPage, searchString)[0];
taggedPiece = NewPiece("<searchtag>" +
Text(searchStringPiece)
+
"</searchtag>", "text/html");
Replace(searchStringPiece, taggedPiece);
startQueryPiece = Parent(Elem(TempPage,
"searchtag")[0]);
Pattern.agt
import Str;
stack = [..];
stackstmt = [..];
level = 0;
lastlevel = -1;
index = 0;
weblmarkuptxt = "";
weblstmt = [..];
every tag in tags do
if (
Str_StartsWith(tag[1], "[/]") == false ) then
if (level
== 0) then
stack[level] := tag[1];
stackstmt[level] :=
"Elem(P,\"" + tag[1] +
"\")";
elsif
(level > 0) then
stack[level] := tag[1];
if
(level == lastlevel-1) then
stackstmt[level] :=
" directlybefore ( Elem(P,\"" + tag[1] +
"\")";
else
stackstmt[level] :=
" contain ( Elem(P,\"" + tag[1] + "\")";
end;
end;
level =
level + 1;
elsif (
Str_StartsWith(tag[1], "[/]") == true ) then
if (level
> 0) then
lastlevel = level;
level =
level - 1;
if
(level == 0) then
lastlevel = level;
weblstmt[index] := stackstmt[level];
index = index + 1;
else
stackstmt[(level-1)] =
stackstmt[(level-1)] + stackstmt[level] + ")";
end;
DeleteField(stack, level);
DeleteField(stackstmt, level);
end;
end;
end;
PieceSearch.agt
PrintLn("PieceSearch begins");
TempRecord = NewPage("<html><body>"
+ Markup(DummyRec[0]) +
"</body></html>", "text/html");
labelSearchStringPiece =
Pat(TempRecord,
labelSearchString)[0];
labelTaggedPiece = NewPiece("<newlabel>" +
Text(labelSearchStringPiece) + "</newlabel>",
"text/html");
Replace(labelSearchStringPiece, labelTaggedPiece);
labelQueryPiece = Parent(Elem(TempRecord,
"newlabel")[0]);
labelQueryStmt = "Elem(CurrentRec, \"" +
Name(labelQueryPiece) + "\")";
while (Name(Parent(labelQueryPiece)) != "body")
do
labelQueryPiece
= Parent(labelQueryPiece);
labelQueryStmt =
labelQueryStmt + " directlyinside " +
"Elem(CurrentRec, \"" +
Name(labelQueryPiece) + "\")";
end;
dummy = "";
CheckNilStmt = "if (Size(" + labelQueryStmt +
") == 0) then dummy = \"empty\"; \n end;dummy;";
labelQueryStmt =" Str_Trim(Text((" +
labelQueryStmt +
")[0]));";
PrintLn("--------------------");
PrintLn(labelQueryStmt);
PrintLn("--------------------");
labelQueryStmt;
PieceSearch2.agt
PrintLn("PieceSearch2 begins");
dummy = "";
PatStmt = labelQueryStmt;
CheckNilStmt = "if (Size(" + PatStmt +
") == 0) then dummy = \"empty\"; \n end;dummy;";
labelQueryStmt =" Str_Trim((" +
labelQueryStmt +
")[0][1]);";
PrintLn("--------------------");
PrintLn(labelQueryStmt);
PrintLn("--------------------");
labelQueryStmt;
Query.agt
queryPiece = startQueryPiece;
queryStmt = "Elem(P, \"" + Name(queryPiece)
+ "\")";
while (Name(queryPiece) != "body") do
queryPiece =
Parent(queryPiece);
queryStmt = queryStmt
+ " directlyinside " + "Elem(P, \"" +
Name(queryPiece) + "\")";
end;
PrintLn();
PrintLn(queryStmt);
queryStmt;
Select.agt
import Str;
txt = [];
PrintLn("Select.agt starts...");
every record in records do
var
txtitem=Text(record);
txtitem = Str_Replace(txtitem, '\n', ' ');
txtitem =
Str_Replace(txtitem, '\t', ' ');
txtitem =
Str_Replace(txtitem, '\f', ' ');
txtitem =
Str_Replace(txtitem, '\r', ' ');
txt = txt +
[txtitem];
end;
PrintLn("Select.agt ends...");
Size(txt);
ShowRec.agt
import Browser;
Browser_ShowPage("<html><body>" +
Markup(DummyRec[0]) + "</body></html>");
