Regular Expressions

• 1. Introduction
• 2. Exported Names
• 3. Known bugs

The Regular Expressions Library

1. Introduction

The Regular-expressions library exports the Regular-expressions module, which contains various functions that deal with regular expressions (regexps). The module is based on Perl (version 4), and has the same semantics unless otherwise noted. The syntax for Perl-style regular expressions can be found on page 103 of Programming Perl by Larry Wall and Randal L. Schwartz. There are some differences in the way String-extensions handles regular expressions. The biggest difference is that regular expressions in Dylan are case insensitive by default. Also, when given an unparsable regexp, String-extensions will produce undefined behavior while Perl would give an error message.

A regular expression that is grammatically correct may still be illegal if it contains an infinitely quantified sub-regexp that may match the empty string. That is, if R is a regexp that can match the empty string, then any regexp containing R*, R+, and R{n,} is illegal. In this case, the Regular-expressions library will signal an <illegal-regexp> error when the regexp is parsed. Note: Perl also has this restriction, although it isn’t mentioned in Programming Perl.

In previous versions of the regular-expressions library, each basic function had a companion function that would pre-compute some information needed to use the regular expression. By using the companion function, one could avoid recomputing the same information. In the present version, the regular-expressions library caches this information, so the companion functions are no longer necessary and should be considered obsolete. However, they have been kept for backwards compatibility.

Companion functions differ in details, but they all essentially return curried versions of their corresponding basic function. For example, the following two pieces of code yield the same result:

            regexp-position("This is a string", "is");
            let is-finder = make-regexp-positioner("is");

            is-finder("This is a string");

Both pieces of code should have roughly the same performance, even if the code is inside a loop.

2. Exported Names

The following names are exported by the Regular-Expressions module of the Regular-Expressions library:

regexp-position [Function]

(big-string, regexp, #key start, end, case-sensitive)
=> variable-number-of-marks-or-#f

This function returns the index of the start of the regular expression in the big-string, or #f if the regular expression is not found. As a second value, it returns the index of the end of the regular expression in the big-string (assuming it was found; otherwise there is no second value). These values are called marks, and they come in pairs, a start-mark and an end-mark. If there are groups in the regular expression, regexp-position will return an additional pair of marks (a start and an end) for each group. If the group is matched, these marks will be integers; if the group is not matched, the marks will be #f. So

            regexp-position("This is a string", "is");

returns values(2, 4), and

            regexp-position("This is a string", "(is)(.*)ing");

returns values(2, 16, 2, 4, 4, 13), while

            regexp-position("This is a string", "(not found)(.*)ing");

returns #f. Marks are always given relative to the start of big-string, not relative to the start: keyword.

Start: and end: specify what part of big-string to look at, and they default to the beginning and end of the string, respectively. Case-sensitive defaults to false.

make-regexp-positioner [Function]

(regexp, #key byte-characters-only, need-marks, maximum-compile, case-sensitive)
=> an anonymous positioner
method (big-string, #key start, end)

Make-regexp-positioner can return several different types of positioners, and it is up to the user to specify what kind of positioner the user wants. It returns a positioner that works like regexp-position. Needs-marks, byte-characters-only, and maximum-compile are ignored, and are included only for backwards compatibility.

Note: This function is obsolete.

regexp-replace [Function]

(big-string, search-for-regexp, replace-with-string, #key count, case-sensitive, start, end)
=> new-string

This replaces all occurrences of regexp in big-string with replace-string. If count: is specified, it replaces only the first count occurrences of regexp. (This is different from Perl, which replaces only the first occurrence unless /g is specified) Replace-string can contain backreferences to the regexp. For instance,

            regexp-replace("The rain in Spain and some other text",

                           "the (.*) in (\\w*\\b)", "\\2 has its \\1")

returns "spain has its rain and some other text". If the subgroup referred to by the backreference was not matched, the reference is interpreted as the null string. For instance,

            regexp-replace("Hi there", "Hi there(, Bert)?", 

                           "What do you think\\1?")

returns "What do you think?" because ", Bert" wasn’t found.

make-regexp-replacer [Function]

(regexp, #key replace-with, case-sensitive)
=> an anonymous replacer function that is either
method (big-string, #key count, start, end)
or
method (big-string, replace-string, #key count, start, end)

The first form is returned if the replace-with: keyword isn’t supplied, otherwise the second form is returned.

Note: This function is obsolete.

translate [Generic Function]

(big-string, from-string, to-string, #key delete, start, end)
=> new-string

This is equivalent to Perl’s tr/// construct. From-string is a string specification of a character set, and to-string is another character set. Translate converts big-string character by character, according to the sets. For instance,

            translate("any string", "a-z", "A-Z")

will convert "any string" to all uppercase: "ANY STRING".

Like Perl, character ranges are not allowed to be "backwards". The following is not legal:

            translate("any string", "a-z", "z-a")

(This restriction may be removed in future releases) Unlike Perl’s tr///, translate doesn’t return the number of characters translated.

If delete: is true, any characters in the from-string that don’t have matching characters in the to-string are deleted. The following will remove all vowels from a string and convert periods to commas:

            translate("any string", ".aeiou", ",", delete: #t)

Delete: is false by default. If delete: is false and there aren’t enough characters in the to-string, the last character in the to-string is reused as many times as necessary. The following converts several punctuation characters into spaces:

            translate("any string", ",./:;[]{}()", " ");

Start: and end: indicate which part of big-string to translate. They default to the entire string.

Caveats: Translate is always case sensitive.

translate [G.F. Method]

(big-byte-string, from-byte-string, to-byte-string, #key delete, start, end)
=> new-string

The only predefined method of translate operates only on byte strings.

make-translator [Generic Function]

(from-string, to-string, #key delete)
=> an anonymous translator
method (big-string, #key start, end) => new-string

Does what you’d expect it to.

make-translator [G.F. Method]

(from-byte-string, to-byte-string, #key delete)
=> an anonymous translator
method (big-string, #key start, end) => new-byte-string

Again, the existing method on make-translator only handles byte strings.

split [Function]

(regexp, big-string, #key count, remove-empty-items, case-sensitive, start, end)
=> a variable number of strings

This is like Perl’s split function. It searches big-string from occurrences of regexp, and returns substrings that were delimited by that regexp. For instance,

            split("-", "long-dylan-identifier")

returns values("long", "dylan", "identifier"). Note that what matched the regexp is left out. Remove-empty-items, which defaults to true, magically skips over empty items, so that

            split("-", "long--with--multiple-dashes)

returns values("long", "with", "multiple", "dashes"). Count is the maximum number of strings to return. If there are n strings and count is specified, the first count - 1 strings are returned as usual, and the count’th string is the remainder, unsplit. So

            split("-", "really-long-dylan-identifier", count: 3)

returns values("really", "long", "dylan-identifier"). If remove-empty-items is true, empty items aren’t counted.

Case sensitive determines if the regexp for the delimiter should be considered case sensitive or not; it defaults to case-insensitive. Start: and end: indicate what part of the big string should be looked at for delimiters. They default to the entire string. For instance,

            split("-", "really-long-dylan-identifier", start: 8)

returns values("really-long", "dylan", "identifier").

Caveat: Unlike Perl, empty regular expressions are never legal regular expressions, so there is no way to split a string into a bunch of single character strings. Of course, in Dylan this is not a useful thing to do, so this is not really a problem.

make-splitter [Function]

(pattern :: <string>, #key case-sensitive)
=> an anonymous splitter
method (big-string, #key count, remove-empty-items, start, end) => buncha-strings

Does what you would expect.

Note: This function is obsolete.

join [Function]

(delimiter :: <string>, #rest strings) => big-string

Does the opposite of a split.

            join(":", word1, word2, word3)

is equivalent to

            concatenate(word1, ":", word2, ":", word3)

(and no more efficient) Note that there is no make-joiner.

<illegal-regexp> [Class]

When an illegal regular expression is parsed, an error of this type will be signalled.

3. Known bugs

The regular expression parser does a very poor job with syntactically invalid regular expressions. Depending on the expression, the parser may signal an error, improperly parse it, or simply crash.

A regular expression that matches a large enough substring can produce a stack overflow. This can happen much more easily under d2c than under Mindy -- as few as two dozen lines of 80 column text under d2c for Windows.

Send comments and bug reports to gwydion-bugs@cs.cmu.edu