All About Class StreamTokenizer in Java



The StreamTokenizer class takes an input stream and parse it into 'tokens', allowing the tokens to be read one at a time.The parsing process is controlled by a table and numbers of flags that can be set to various status.The StreamTokenizer can recognize identifiers, numbers,quoted Strings and various other commands style.
Each byte read from the input stream is regarded as a character in the range '\u000' through '\u00FF'.The character value is used to look up five possible attributes of the character.

five possible possibilities are:

  • Whitespaces
  • alphabetic character
  • String quotes
  • comment characters
  • numerical values
In addition, an instance has four flags.They are as follows:
  1. Whether an instance are to be returned as tokens or treated as white space.
  2. Whether C-style comments are to be recognized and skipped.
  3. Whether C++-style comments are to be recognized and skipped.
  4. Whether the characters of identifiers are converted to lowercase
A typical application first constructs an instance of this class,sets up the syntax tables and repeatedly loops calling the nextToken() method in each iteration of the loop until it returns the value TT_EOF.

The structure of the class StreamTokenizer is given as :

pubic class java.io.StreamTokenizer extends java.lang.Object{
//member elements
public double nval;
//If the current token is a number,this field is used. This field contains the value of the number.
//if the current token is a number the value of the tttype file is TT_NUMBER
public String sval;
//If the current token is a word,this field is used. This field contains a String giving characters
//of the word token.When the current token is a quoted String token,this filed contains the body of 
//the String if the current token is word, the value of the tttype file is TT_WORD
public int ttype;
//after a call to the nextToken() method,this field contains the type of the token just read.For a 
//Single character token,its value is the single character converted to an integer.For a quoted Sting 
//token,its value is the quoted character,otherwise its value //is one of the following
//TT_WORD-indicates that the token is a word
//TT_NUMBER-indicates that the token is a number
//TT_EOL-indicates that the end of the line has read.the field can only have this value if the EOL is 
//significant
//method has been called with the argument true.
//TT_EOF-Indicates that the end of the input stream has been reached.
//constructors
 public StreamTokenizer(InputStream in);
//Creates a StreamTokenizer object that parse the specified input stream.The StreamTokenizer is initialized
//to the following default state:
//All byte values 'A' through 'Z','a' through 'z' and '\u00A0 through '\u0FF' are considered to be alphabetic.
//All bytes values '\u000 through '\u0020' are considered to be whitespace.'/' is a comment character.
//Single quote '\' and ouble quote "" are sting quote character.numbers are parsed .End of lines are not
//treated as whitespace,not as separate tokens.C and C++ style comments are not recognized.
//Methods
public void commentChar(int ch);
public void eolIsSignificant(boolean flag);
public void ordinaryChar(int ch);
public void ordinaryChars(int low,int hi);
public void parseNumbers();
public void pushBack();
public void quoteChar(int ch);
public void resetSyntax();
public void lowerCaseMode(boolean f1);
public void slashStarComments(boolean flag);
public void whitespaceChars(int low,int hi);
public void wordChars(int low,int hi);
public int lineno();
public int nextToken();
public String toString();
}
 
All About Class StreamTokenizer in Java All About Class StreamTokenizer in Java Reviewed by Animesh Chatterjee on October 11, 2018 Rating: 5

No comments:

Powered by Blogger.