Paste: simple toknizer

Author: doublec
Mode: factor
Date: Sun, 30 Nov 2008 22:24:20
Plain Text | 
SingleLineComment = "//" (!("\n") .)* "\n" => [[ ignore ]]
MultiLineComment  = "/*" (!("*/") .)* "*/" => [[ ignore ]]
Space             = " " | "\t" | "\r" | "\n" | SingleLineComment | MultiLineComment
Punctuation       = "(" | ")" | "[" | "]" | ";"
String            = '"' (!('"') .) '"' => [[ second >string token-string boa ]]
Word              = !(String) (!(Space | Punctuation) .)+ => [[ >string token-word boa ]]
Tok               = Spaces (Number | Word | String | Punctuation )
Toks              = Tok* Spaces

Annotation: using it

Author: doublec
Mode: factor
Date: Sun, 30 Nov 2008 22:25:32
Plain Text | 
EBNF: parse-xy
tokenizer   = <foreign tokenize-xy Tok>
Word        = . ?[ token-word? ]?        => [[ value>> ast-word boa ]]
Number      = . ?[ token-number? ]?      => [[ value>> ast-number boa ]]
St

New Annotation

Summary:
Author:
Mode:
Body: