自制腳本語言(1)--詞法分析器
目的:為了實現一個簡單的解釋器并在其基礎上進行修改優化使其成為一個初級的編譯器.
基于千葉滋老師的stone語言進行編寫,使用Java作為寫作的語言,最終完成版可以運行于JVM平臺之上.
既然是自制語言,首先簡單介紹一下語言處理器內部的流程
- 源代碼輸入,編譯器對其進行詞法分析;
- 詞法分析的結果是對語句進行了分割,產生了一系列的Token,即簡短的字符串排列;
- 接下來進行語法分析,生成抽象語法樹,到這一步接下來就有兩種走向;
- 若進行代碼生成為其他語言程序,則為編譯器;
- 若直接對抽象語法樹進行運算,則為解釋器;本文介紹的即是基于解釋執行,但到目前還是看不出來的.
詞法分析器對源代碼進行了分割,得到如下代碼中的Token對象:
Token.java
package stone;
public abstract class Token{
public static final Token EOF = new Token(-1) {}; //end of file
public static final String EOL = "\n"; //end of line
private int lineNumber;
protected Token(int line) {
lineNumber = line;
}
public int getLineNumber() {return lineNumber;}
public boolean isIdentifier() {return false;}
public boolean isNumber() {return false;}
public boolean isString() {return false;}
public int getNumber() {throw new StoneException("not number token"); }
public String getText() {return "";}
}
StoneException簡單的繼承了RuntimeException,相信有Java基礎的童鞋都可以看懂.
StoneException.java
package stone;
import stone.ast.ASTree;
public class StoneException extends RuntimeException {
public StoneException(String m) { super(m); }
public StoneException(String m, ASTree t) {
super(m + " " + t.location());
}
}
詞法分析器主體Lexer.java借助了Java語言中的regex(正則表達式類包),關于正則表達式可以參考有關內容,本文就不再贅述.
對了,正則表達式是很強大的存在,借助其可以完成很多有趣的事情,也可以稱其為瑞士軍刀了吧.
Lexer.java
package stone;
import java.io.IOException;
import java.io.LineNumberReader;
import java.io.Reader;
import java.util.ArrayList;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
public class Lexer {
public static String regexPat
= "\s((//.)|([0-9]+)|("(\\"|\\\\|\\n|[^"])")"+ "|[A-Z_a-z][A-Z_a-z0-9]|==|<=|>=|&&|\|\||\p{Punct})?";
private Pattern pattern = Pattern.compile(regexPat);
private ArrayList<Token> queue = new ArrayList<Token>();
private boolean hasMore;
private LineNumberReader reader;
public Lexer(Reader r) {
hasMore = true;
reader = new LineNumberReader(r);
}
public Token read() throws ParseException {
if (fillQueue(0))
return queue.remove(0);
else
return Token.EOF;
}
public Token peek(int i) throws ParseException {
if (fillQueue(i))
return queue.get(i);
else
return Token.EOF;
}
private boolean fillQueue(int i) throws ParseException {
while (i >= queue.size())
if (hasMore)
readLine();
else
return false;
return true;
}
protected void readLine() throws ParseException {
String line;
try {
line = reader.readLine();
} catch (IOException e) {
throw new ParseException(e);
}
if (line == null) {
hasMore = false;
return;
}
int lineNo = reader.getLineNumber();
Matcher matcher = pattern.matcher(line);
matcher.useTransparentBounds(true).useAnchoringBounds(false);
int pos = 0;
int endPos = line.length();
while (pos < endPos) {
matcher.region(pos, endPos);
if (matcher.lookingAt()) {
addToken(lineNo, matcher);
pos = matcher.end();
}
else
throw new ParseException("bad token at line " + lineNo);
}
queue.add(new IdToken(lineNo, Token.EOL));
}
protected void addToken(int lineNo, Matcher matcher) {
String m = matcher.group(1);
if (m != null) // if not a space
if (matcher.group(2) == null) { // if not a comment
Token token;
if (matcher.group(3) != null)
token = new NumToken(lineNo, Integer.parseInt(m));
else if (matcher.group(4) != null)
token = new StrToken(lineNo, toStringLiteral(m));
else
token = new IdToken(lineNo, m);
queue.add(token);
}
}
protected String toStringLiteral(String s) {
StringBuilder sb = new StringBuilder();
int len = s.length() - 1;
for (int i = 1; i < len; i++) {
char c = s.charAt(i);
if (c == '\\' && i + 1 < len) {
int c2 = s.charAt(i + 1);
if (c2 == '"' || c2 == '\\')
c = s.charAt(++i);
else if (c2 == 'n') {
++i;
c = '\n';
}
}
sb.append(c);
}
return sb.toString();
}
protected static class NumToken extends Token {
private int value;
protected NumToken(int line, int v) {
super(line);
value = v;
}
public boolean isNumber() { return true; }
public String getText() { return Integer.toString(value); }
public int getNumber() { return value; }
}
protected static class IdToken extends Token {
private String text;
protected IdToken(int line, String id) {
super(line);
text = id;
}
public boolean isIdentifier() { return true; }
public String getText() { return text; }
}
protected static class StrToken extends Token {
private String literal;
StrToken(int line, String str) {
super(line);
literal = str;
}
public boolean isString() { return true; }
public String getText() { return literal; }
}
}
異常類ParseException.java的實現,繼承Exception
package stone;
import java.io.IOException;
public class ParseException extends Exception {
public ParseException(Token t) {
this("", t);
}
public ParseException(String msg, Token t) {
super("syntax error around" + location(t) + "." + msg);
}
private static String location(Token t) {
if(t == Token.EOF)
return "the last line";
else {
return """ + t.getText() + "" at line" + t.getLineNumber();
}
}
public ParseException(IOException e) {
super(e);
}
public ParseException(String msg) {
super(msg);
}
}
到這里,詞法分析器基本完成,接下來進行測試,測試的方法是借助CodeDialog將輸入的代碼進行分割,并將分割后的Token以">+Token"的格式在console中打印出
LexerRunner.java
package stone;
import stone.*;
public class LexerRunner {
public static void main(String[] args) throws ParseException {
Lexer l = new Lexer(new CodeDialog());
for( Token t; (t = l.read()) != Token.EOF;)
System.out.println("=> " + t.getText());
}
}
CodeDialog.java作用如上所述
package stone;
import java.io.FileReader;
import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.Reader;
import javax.swing.JFileChooser;
import javax.swing.JOptionPane;
import javax.swing.JScrollPane;
import javax.swing.JTextArea;
public class CodeDialog extends Reader {
private String buffer = null;
private int pos = 0;
public int read(char[] cbuf, int off, int len) throws IOException {
if (buffer == null) {
String in = showDialog();
if (in == null)
return -1;
else {
print(in);
buffer = in + "\n";
pos = 0;
}
}
int size = 0;
int length = buffer.length();
while (pos < length && size < len)
cbuf[off + size++] = buffer.charAt(pos++);
if (pos == length)
buffer = null;
return size;
}
protected void print(String s) { System.out.println(s); }
public void close() throws IOException {}
protected String showDialog() {
JTextArea area = new JTextArea(20, 40);
JScrollPane pane = new JScrollPane(area);
int result = JOptionPane.showOptionDialog(null, pane, "Input",
JOptionPane.OK_CANCEL_OPTION,
JOptionPane.PLAIN_MESSAGE,
null, null, null);
if (result == JOptionPane.OK_OPTION)
return area.getText();
else
return null;
}
public static Reader file() throws FileNotFoundException {
JFileChooser chooser = new JFileChooser();
if (chooser.showOpenDialog(null) == JFileChooser.APPROVE_OPTION)
return new BufferedReader(new FileReader(chooser.getSelectedFile()));
else
throw new FileNotFoundException("no file specified");
}
}
