Package EasyExtend :: Package langlets :: Package p4d :: Module langlet :: Class LangletTokenizer

Class LangletTokenizer

           object --+            
                    |            
     eeoptions.EEShow --+        
                        |        
eetokenizer.BaseTokenizer --+    
                            |    
        eetokenizer.Tokenizer --+
                                |
                               LangletTokenizer

Defines langlet specific token stream processor.

Instance Methods

[hide private]

NAME(self, pos, tok, stream)

Inherited from eetokenizer.Tokenizer: INTRON, LEFT, RIGHT, __init__, dedent_to, default, dot_start, multiple_introns, post_lex, single_intron, terminate_stream

Inherited from eetokenizer.BaseTokenizer: add_token, set_handler, tokenize_file, tokenize_fileobj, tokenize_string

Inherited from eeoptions.EEShow: maybe_grammar_check, maybe_show_cst_after, maybe_show_cst_before, maybe_show_python, maybe_show_scan, maybe_show_token

Inherited from object: __delattr__, __getattribute__, __hash__, __new__, __reduce__, __reduce_ex__, __repr__, __setattr__, __str__

Class Methods

[hide private]

Inherited from eeoptions.EEShow: show_tokenstream

Properties

[hide private]

Inherited from object: __class__

Method Details

[hide private]

NAME(self, pos, tok, stream)

Decorators:

@post