329 lines
38 KiB
Plaintext
329 lines
38 KiB
Plaintext
|
Ë
|
|||
|
|
dÀjR‰ ã óX — d Z ddlZddlZddlZddlmZmZ ddlmZ ddl m
|
|||
|
|
Z
|
|||
|
|
m Z m Z m
|
|||
|
|
Z
|
|||
|
|
mZ ddlmZmZmZmZmZmZ ddlmZ g d¢Z ej2 d « Zg d
|
|||
|
|
¢Z e d „ « Z G d „ d
|
|||
|
|
e « Z G d„ de ¬« Z G d„ de « Z! G d„ de"« Z# G d„ d« Z$ e$« Z% G d„ de&« Z' G d„ d« Z(d„ Z) G d „ d « Z* e*« Z+d „ Z, G d „ d « Z- G d!„ d"e« Z. G d#„ d$e « Z/ G d%„ d&e e/¬« Z0 G d'„ d(« Z1 G d)„ d*e0« Z2d+„ Z3 G d,„ d-e/« Z4 G d.„ d/e0e4¬« Z5y)0z·
|
|||
|
|
pygments.lexer
|
|||
|
|
~~~~~~~~~~~~~~
|
|||
|
|
|
|||
|
|
Base lexer classes.
|
|||
|
|
|
|||
|
|
:copyright: Copyright 2006-present by the Pygments team, see AUTHORS.
|
|||
|
|
:license: BSD, see LICENSE for details.
|
|||
|
|
é N)Ú
|
|||
|
|
apply_filtersÚFilter)Úget_filter_by_name)ÚErrorÚTextÚOtherÚ
|
|||
|
|
WhitespaceÚ
|
|||
|
|
_TokenType)Ú get_bool_optÚ get_int_optÚ get_list_optÚmake_analysatorÚFutureÚ guess_decode)Ú regex_opt)
|
|||
|
|
ÚLexerÚ
|
|||
|
|
RegexLexerÚExtendedRegexLexerÚDelegatingLexerÚ LexerContextÚincludeÚinheritÚbygroupsÚusingÚthisÚdefaultÚwordsÚline_rez.*?
|
|||
|
|
))s úutf-8)s ÿþ zutf-32)s þÿzutf-32be)s ÿþzutf-16)s þÿzutf-16bec ó — y)Nç © )Úxs úQ/home/antigravity/intellecton/venv/lib/python3.12/site-packages/pygments/lexer.pyú<lambda>r% " ó � ó c ó — e Zd ZdZd„ Zy)Ú LexerMetaz‚
|
|||
|
|
This metaclass automagically converts ``analyse_text`` methods into
|
|||
|
|
static methods which always return float values.
|
|||
|
|
c ó\ — d|v rt |d « |d< t j | |||« S )NÚ analyse_text)r ÚtypeÚ__new__)ÚmcsÚnameÚbasesÚds r$ r- zLexerMeta.__new__+ s3 € Ø ˜QÑ Ü /°°.Ñ0AÓ BˆAˆnÑ Ü�|‰|˜C u¨aÓ0Ð0r' N)Ú__name__Ú
|
|||
|
|
__module__Ú __qualname__Ú__doc__r- r" r' r$ r) r) % s „ ñó
|
|||
|
|
1r' r) c ó` — e Zd ZdZdZg Zg Zg Zg ZdZ dZ
|
|||
|
|
dZ dZ d„ Z
|
|||
|
|
d„ Zd„ Zd„ Zd„ Zd d „Zd
|
|||
|
|
„ Zy) r a"
|
|||
|
|
Lexer for a specific language.
|
|||
|
|
|
|||
|
|
See also :doc:`lexerdevelopment`, a high-level guide to writing
|
|||
|
|
lexers.
|
|||
|
|
|
|||
|
|
Lexer classes have attributes used for choosing the most appropriate
|
|||
|
|
lexer based on various criteria.
|
|||
|
|
|
|||
|
|
.. autoattribute:: name
|
|||
|
|
:no-value:
|
|||
|
|
.. autoattribute:: aliases
|
|||
|
|
:no-value:
|
|||
|
|
.. autoattribute:: filenames
|
|||
|
|
:no-value:
|
|||
|
|
.. autoattribute:: alias_filenames
|
|||
|
|
.. autoattribute:: mimetypes
|
|||
|
|
:no-value:
|
|||
|
|
.. autoattribute:: priority
|
|||
|
|
|
|||
|
|
Lexers included in Pygments should have two additional attributes:
|
|||
|
|
|
|||
|
|
.. autoattribute:: url
|
|||
|
|
:no-value:
|
|||
|
|
.. autoattribute:: version_added
|
|||
|
|
:no-value:
|
|||
|
|
|
|||
|
|
Lexers included in Pygments may have additional attributes:
|
|||
|
|
|
|||
|
|
.. autoattribute:: _example
|
|||
|
|
:no-value:
|
|||
|
|
|
|||
|
|
You can pass options to the constructor. The basic options recognized
|
|||
|
|
by all lexers and processed by the base `Lexer` class are:
|
|||
|
|
|
|||
|
|
``stripnl``
|
|||
|
|
Strip leading and trailing newlines from the input (default: True).
|
|||
|
|
``stripall``
|
|||
|
|
Strip all leading and trailing whitespace from the input
|
|||
|
|
(default: False).
|
|||
|
|
``ensurenl``
|
|||
|
|
Make sure that the input ends with a newline (default: True). This
|
|||
|
|
is required for some lexers that consume input linewise.
|
|||
|
|
|
|||
|
|
.. versionadded:: 1.3
|
|||
|
|
|
|||
|
|
``tabsize``
|
|||
|
|
If given and greater than 0, expand tabs in the input (default: 0).
|
|||
|
|
``encoding``
|
|||
|
|
If given, must be an encoding name. This encoding will be used to
|
|||
|
|
convert the input string to Unicode, if it is not already a Unicode
|
|||
|
|
string (default: ``'guess'``, which uses a simple UTF-8 / Locale /
|
|||
|
|
Latin1 detection. Can also be ``'chardet'`` to use the chardet
|
|||
|
|
library, if it is installed.
|
|||
|
|
``inencoding``
|
|||
|
|
Overrides the ``encoding`` if given.
|
|||
|
|
Nr c ól — || _ t |dd« | _ t |dd« | _ t |dd« | _ t |dd« | _ |j dd « | _ |j d
|
|||
|
|
« xs | j | _ g | _ t |d d « D ] }| j |« Œ y
|
|||
|
|
)a‘
|
|||
|
|
This constructor takes arbitrary options as keyword arguments.
|
|||
|
|
Every subclass must first process its own options and then call
|
|||
|
|
the `Lexer` constructor, since it processes the basic
|
|||
|
|
options like `stripnl`.
|
|||
|
|
|
|||
|
|
An example looks like this:
|
|||
|
|
|
|||
|
|
.. sourcecode:: python
|
|||
|
|
|
|||
|
|
def __init__(self, **options):
|
|||
|
|
self.compress = options.get('compress', '')
|
|||
|
|
Lexer.__init__(self, **options)
|
|||
|
|
|
|||
|
|
As these options must all be specifiable as strings (due to the
|
|||
|
|
command line usage), there are various utility functions
|
|||
|
|
available to help with that, see `Utilities`_.
|
|||
|
|
ÚstripnlTÚstripallFÚensurenlÚtabsizer ÚencodingÚguessÚ
|
|||
|
|
inencodingÚfiltersr" N) Úoptionsr r8 r9 r: r r; Úgetr< r? r
|
|||
|
|
Ú
|
|||
|
|
add_filter)Úselfr@ Úfilter_s r$ Ú__init__zLexer.__init__‘ s¡ € ð& ˆŒ Ü# G¨Y¸Ó=ˆŒ Ü$ W¨j¸%Ó@ˆŒ
|
|||
|
|
Ü$ W¨j¸$Ó?ˆŒ
|
|||
|
|
Ü" 7¨I°qÓ9ˆŒ Ø Ÿ ™ J°Ó8ˆŒ
|
|||
|
|
Ø Ÿ ™ LÓ1ÒB°T·]±]ˆŒ
|
|||
|
|
؈Œ Ü# G¨Y¸Ó;ò %ˆGØ �O‰O˜GÕ $ñ %r' c óœ — | j r'd| j j › d| j ›d�S d| j j › d�S )Nz<pygments.lexers.z with ú>)r@ Ú __class__r2 ©rC s r$ Ú__repr__zLexer.__repr__¯ sI € Ø �<Š<Ø& t§~¡~×'>Ñ'>Ð&?¸vÀdÇlÁlÐEUÐUVÐWÐ Wà& t§~¡~×'>Ñ'>Ð&?¸qÐAÐ Ar' c ór — t |t « s t |fi |¤Ž}| j j |« y)z8
|
|||
|
|
Add a new stream filter to this lexer.
|
|||
|
|
N)Ú
|
|||
|
|
isinstancer r r? Úappend)rC rD r@ s r$ rB zLexer.add_filterµ s/ € ô ˜'¤6Ô*Ü(¨Ñ<°GÑ<ˆGØ � ‰ ×јGÕ$r' c ó — y)aÁ
|
|||
|
|
A static method which is called for lexer guessing.
|
|||
|
|
|
|||
|
|
It should analyse the text and return a float in the range
|
|||
|
|
from ``0.0`` to ``1.0``. If it returns ``0.0``, the lexer
|
|||
|
|
will not be selected as the most probable one, if it returns
|
|||
|
|
``1.0``, it will be selected immediately. This is used by
|
|||
|
|
`guess_lexer`.
|
|||
|
|
|
|||
|
|
The `LexerMeta` metaclass automatically wraps this function so
|
|||
|
|
that it works like a static method (no ``self`` or ``cls``
|
|||
|
|
parameter) and the return value is automatically converted to
|
|||
|
|
`float`. If the return value is an object that is boolean `False`
|
|||
|
|
it's the same as if the return values was ``0.0``.
|
|||
|
|
Nr" )Útexts r$ r+ zLexer.analyse_text½ r& r' c óÊ — t |t « sì| j dk( rt |« \ }}ní| j dk( r„ ddl}d}t D ]6 \ }}|j |« sŒ|t |« d j |d« } n |€9|j |dd « }|j |j d« xs d d« }|}nZ|j | j « }|j d
|
|||
|
|
« r.|t d
|
|||
|
|
« d }n |j d
|
|||
|
|
« r|t d
|
|||
|
|
« d }|j d d « }|j d
|
|||
|
|
d « }| j r|j « }n | j r|j d « }| j dkD r|j# | j « }| j$ r|j' d « s|d z
|
|||
|
|
}|S # t
|
|||
|
|
$ r}t d« |‚d}~ww xY w)zVApply preprocessing such as decoding the input, removing BOM and normalizing newlines.r= Úchardetr NzkTo enable chardet encoding guessing, please install the chardet library from http://chardet.feedparser.org/Úreplacei r< r u z
|
|||
|
|
ú
|
|||
|
|
ú
|
|||
|
|
)rL Ústrr< r rQ Ú ImportErrorÚ
|
|||
|
|
_encoding_mapÚ
|
|||
|
|
startswithÚlenÚdecodeÚdetectrA rR r9 Ústripr8 r; Ú
|
|||
|
|
expandtabsr: Úendswith) rC rO Ú_rQ ÚeÚdecodedÚbomr< Úencs r$ Ú_preprocess_lexer_inputz Lexer._preprocess_lexer_inputÎ sÁ € ô ˜$¤Ô$Ø�}‰} Ò'Ü& tÓ,‘�‘aØ— ‘ )Ò+ðTÛ"ð �Ü%2ò ‘M�C˜Ø—‘ sÕ+Ø"&¤s¨3£x y /×"8Ñ"8¸À9Ó"M˜Ù ð ð
|
|||
|
|
�?Ø!Ÿ.™.¨¨e¨t¨Ó5�CØ "Ÿk™k¨#¯'©'°*Ó*=Ò*HÀØ*3ó 5�Gà ‘à—{‘{ 4§=¡=Ó1�Ø—?‘? 8Ô,Ø ¤ H£
|
|||
|
|
Ð/‘Dà�‰˜xÔ(ØœC ›M˜NÐ+�ð �|‰|˜F DÓ)ˆØ�|‰|˜D $Ó'ˆØ �=Š=Ø—:‘:“<‰DØ
|
|||
|
|
�\Š\Ø—:‘:˜dÓ#ˆDØ �<‰<˜!Ò Ø—?‘? 4§<¡<Ó0ˆDØ �=Š= § ¡ ¨tÔ!4Ø �D‰LˆDàˆ øôI #ò TÜ%ð 'Ló MàRSðTûðTús ¿G Ç G"Ç G Ç G"c óx ‡ ‡— ‰ j ‰« Šˆ ˆfd„} |« }|st |‰ j ‰ « }|S )ae
|
|||
|
|
This method is the basic interface of a lexer. It is called by
|
|||
|
|
the `highlight()` function. It must process the text and return an
|
|||
|
|
iterable of ``(tokentype, value)`` pairs from `text`.
|
|||
|
|
|
|||
|
|
Normally, you don't need to override this method. The default
|
|||
|
|
implementation processes the options recognized by all lexers
|
|||
|
|
(`stripnl`, `stripall` and so on), and then yields all tokens
|
|||
|
|
from `get_tokens_unprocessed()`, with the ``index`` dropped.
|
|||
|
|
|
|||
|
|
If `unfiltered` is set to `True`, the filtering mechanism is
|
|||
|
|
bypassed even if filters are defined.
|
|||
|
|
c 3 óN •K — ‰j ‰« D ] \ } }}||f–— Œ y w©N)Úget_tokens_unprocessed)r_ ÚtÚvrC rO s €€r$ Ústreamerz"Lexer.get_tokens.<locals>.streamer
|
|||
|
|
s0 øè ø€ Ø ×6Ñ6°tÓ<ò
|
|||
|
|
‘��1�aؘ�d“
|
|||
|
|
ñ
|
|||
|
|
ùs ƒ"%)rd r r? )rC rO Ú
|
|||
|
|
unfilteredrk Ústreams `` r$ Ú
|
|||
|
|
get_tokenszLexer.get_tokensý s= ù€ ð ×+Ñ+¨DÓ1ˆõ ñ “ˆÙÜ" 6¨4¯<©<¸Ó>ˆF؈
|
|||
|
|
r' c ó — t ‚)aS
|
|||
|
|
This method should process the text and return an iterable of
|
|||
|
|
``(index, tokentype, value)`` tuples where ``index`` is the starting
|
|||
|
|
position of the token within the input text.
|
|||
|
|
|
|||
|
|
It must be overridden by subclasses. It is recommended to
|
|||
|
|
implement it as a generator to maximize effectiveness.
|
|||
|
|
)ÚNotImplementedError)rC rO s r$ rh z Lexer.get_tokens_unprocessed s
|
|||
|
|
€ ô "Ð!r' )F)r2 r3 r4 r5 r/ ÚaliasesÚ filenamesÚalias_filenamesÚ mimetypesÚpriorityÚurlÚ
|
|||
|
|
version_addedÚ_examplerE rJ rB r+ rd rn rh r" r' r$ r r 1 sl „ ñ8ðv €Dð €Gð
|
|||
|
|
€Ið €Oð €Ið €Hð €Cð €Mð €Hò %ò<Bò %ò ò"-ó^ó0 "r' r )Ú metaclassc ó — e Zd ZdZefd„Zd„ Zy)r a
|
|||
|
|
This lexer takes two lexer as arguments. A root lexer and
|
|||
|
|
a language lexer. First everything is scanned using the language
|
|||
|
|
lexer, afterwards all ``Other`` tokens are lexed using the root
|
|||
|
|
lexer.
|
|||
|
|
|
|||
|
|
The lexers from the ``template`` lexer package use this base lexer.
|
|||
|
|
c ór — |di |¤Ž| _ |di |¤Ž| _ || _ t j | fi |¤Ž y ©Nr" )Ú
|
|||
|
|
root_lexerÚlanguage_lexerÚneedler rE )rC Ú _root_lexerÚ_language_lexerÚ_needler@ s r$ rE zDelegatingLexer.__init__+ s9 € Ù%Ñ0¨Ñ0ˆŒÙ -Ñ 8°Ñ 8ˆÔØ ˆŒ Ü
|
|||
|
|
�‰�tÑ'˜wÓ'r' c ól — d}g }g }| j j |« D ]N \ }}}|| j u r&|r |j t |« |f« g }||z
|
|||
|
|
}Œ;|j |||f« ŒP |r |j t |« |f« t || j j |« « S )NÚ )r~ rh r rM rY Ú
|
|||
|
|
do_insertionsr} )rC rO ÚbufferedÚ
|
|||
|
|
insertionsÚ
|
|||
|
|
lng_bufferÚiri rj s r$ rh z&DelegatingLexer.get_tokens_unprocessed1 s¾ € ؈؈
|
|||
|
|
؈
|
|||
|
|
Ø×*Ñ*×AÑAÀ$ÓGò -‰GˆAˆq�!Ø�D—K‘KÑ Ù Ø ×%Ñ%¤s¨8£}°jÐ&AÔBØ!#�JؘA‘
|
|||
|
|
‘à×!Ñ! 1 a¨ )Õ,ð -ñ Ø × Ñ œs 8›}¨jÐ 9Ô :Ü ˜ZØ !Ÿ_™_× CÑ CÀHÓ MóOð Or' N)r2 r3 r4 r5 r rE rh r" r' r$ r r ! s „ ñð >Có (ó Or' r c ó — e Zd ZdZy)r zI
|
|||
|
|
Indicates that a state should include rules from another state.
|
|||
|
|
N©r2 r3 r4 r5 r" r' r$ r r H s „ ñð r' r c ó — e Zd ZdZd„ Zy)Ú_inheritzC
|
|||
|
|
Indicates the a state should inherit from its superclass.
|
|||
|
|
c ó — y)Nr r" rI s r$ rJ z_inherit.__repr__S s € Ør' N)r2 r3 r4 r5 rJ r" r' r$ r� r� O s „ ñór' r� c ó — e Zd ZdZd„ Zd„ Zy)Úcombinedz:
|
|||
|
|
Indicates a state combined from multiple states.
|
|||
|
|
c ó. — t j | |« S rg )Útupler- )ÚclsÚargss r$ r- zcombined.__new__^ s € Ü�}‰}˜S $Ó'Ð'r' c ó — y rg r" )rC r” s r$ rE zcombined.__init__a s € à r' N)r2 r3 r4 r5 r- rE r" r' r$ r� r� Y s „ ñò(ó
|
|||
|
|
r' r� c ó: — e Zd ZdZd„ Zd d„Zd d„Zd d„Zd„ Zd„ Z y)
|
|||
|
|
Ú _PseudoMatchz:
|
|||
|
|
A pseudo match object constructed from a string.
|
|||
|
|
c ó — || _ || _ y rg )Ú_textÚ_start)rC ÚstartrO s r$ rE z_PseudoMatch.__init__k s € ؈Œ
|
|||
|
|
؈� r' Nc ó — | j S rg )rš ©rC Úargs r$ r› z_PseudoMatch.starto s € Ø�{‰{Ðr' c óF — | j t | j « z S rg )rš rY r™ r� s r$ Úendz_PseudoMatch.endr s € Ø�{‰{œS §¡›_Ñ,Ð,r' c ó4 — |r t d« ‚| j S )Nz
|
|||
|
|
No such group)Ú
|
|||
|
|
IndexErrorr™ r� s r$ Úgroupz_PseudoMatch.groupu s € Ù Ü ˜_Ó-Ð -Ø�z‰zÐr' c ó — | j fS rg )r™ rI s r$ Úgroupsz_PseudoMatch.groupsz s € Ø—
|
|||
|
|
‘
|
|||
|
|
ˆ}Ð r' c ó — i S rg r" rI s r$ Ú groupdictz_PseudoMatch.groupdict} s € ؈ r' rg )
|
|||
|
|
r2 r3 r4 r5 rE r› r r£ r¥ r§ r" r' r$ r— r— f s% „ ñò óó-óò
|
|||
|
|
ór' r— c ó ‡ — dˆ fd„ }|S )zL
|
|||
|
|
Callback that yields multiple actions for each group in the match.
|
|||
|
|
c
|
|||
|
|
3 ó¶ •K — t ‰« D ]® \ }}|€Œ t |« t u r1|j |dz « }|sŒ1|j |dz « ||f–— ŒK|j |dz « }|€Œb|r|j |dz « |_ || t
|
|||
|
|
|j |dz « |« |« D ] }|sŒ|–— Œ Œ° |r|j « |_ y y w)Né )Ú enumerater, r
|
|||
|
|
r£ r› Úposr— r )ÚlexerÚmatchÚctxr‰ ÚactionÚdataÚitemr” s €r$ Úcallbackzbygroups.<locals>.callback… sß øè ø€ Ü" 4› ò '‰IˆAˆv؈~ØÜ�f“ ¤Ñ+Ø —{‘{ 1 q¡5Ó)�ÚØ Ÿ+™+ a¨!¡eÓ,¨f°dÐ:Ó:à —{‘{ 1 q¡5Ó)�ØÑ#ÙØ"'§+¡+¨a°!©eÓ"4˜œÙ & uÜ'3°E·K±KÀÀAÁÓ4FÈÓ'MÈsó!Tò '˜â Ø"&›Jñ'ð 'ñ Ø—i‘i“kˆC�Gð ùs ƒ<CÁ 0CÁ1ACÂ8!Crg r" )r” r³ s ` r$ r r � s ø€ õ"ð& €Or' c ó — e Zd ZdZy)Ú_ThiszX
|
|||
|
|
Special singleton used for indicating the caller class.
|
|||
|
|
Used by ``using``.
|
|||
|
|
Nr‹ r" r' r$ rµ rµ › s „ òr' rµ c ó² ‡ ‡‡— i Šd‰v r4‰j d« }t |t t f« r|‰d< nd|f‰d< ‰ t u r dˆˆfd„ }|S dˆ ˆˆfd„ }|S )aÂ
|
|||
|
|
Callback that processes the match with a different lexer.
|
|||
|
|
|
|||
|
|
The keyword arguments are forwarded to the lexer, except `state` which
|
|||
|
|
is handled separately.
|
|||
|
|
|
|||
|
|
`state` specifies the state that the new lexer will start in, and can
|
|||
|
|
be an enumerable such as ('root', 'inline', 'string') or a simple
|
|||
|
|
string which is assumed to be on top of the root state.
|
|||
|
|
|
|||
|
|
Note: For that to work, `_other` must not be an `ExtendedRegexLexer`.
|
|||
|
|
ÚstateÚstackÚrootc 3 ó@ •K — ‰
|
|||
|
|
r9t | j « }|j ‰
|
|||
|
|
« | j di |¤Ž}n| }|j « } |j
|
|||
|
|
|j
|
|||
|
|
« fi ‰ ¤ŽD ] \ }}}||z ||f–— Œ |r|j « |_ y y wr| ) Údictr@ ÚupdaterH r› rh r£ r r¬ ) r r® r¯ r1 ÚlxÚsr‰ ri rj Ú gt_kwargsÚkwargss €€r$ r³ zusing.<locals>.callbackº s› øè ø€ ñ 䘟 ™ Ó'�Ø—‘˜Ô Ø$�U—_‘_Ñ) qÑ)‘à�Ø— ‘ “
|
|||
|
|
ˆAØ4˜2×4Ñ4°U·[±[³]ÑPÀiÑPò
|
|||
|
|
"‘��1�aؘ!‘e˜Q �kÓ!ð
|
|||
|
|
"áØ Ÿ)™)›+�•ð ùs ƒBB c 3 ó" •K — t | j « }|j ‰ « ‰ di |¤Ž}|j « } |j |j « fi ‰
|
|||
|
|
¤ŽD ] \ }}}||z ||f–— Œ |r|j
|
|||
|
|
« |_ y y wr| )r» r@ r¼ r› rh r£ r r¬ ) r r® r¯ r1 r½ r¾ r‰ ri rj Ú_otherr¿ rÀ s €€€r$ r³ zusing.<locals>.callbackÊ s‰ øè ø€ ä�U—]‘]Ó#ˆAØ
|
|||
|
|
�H‰H�VÔ Ù‘˜!‘ˆBà— ‘ “
|
|||
|
|
ˆAØ4˜2×4Ñ4°U·[±[³]ÑPÀiÑPò
|
|||
|
|
"‘��1�aؘ!‘e˜Q �kÓ!ð
|
|||
|
|
"áØ Ÿ)™)›+�•ð ùs ƒB Brg )ÚpoprL Úlistr’ r )r rÀ r¾ r³ r¿ s `` @r$ r r ¤ se ú€ ð €IØ�&ÑØ �J‰J�wÓ ˆÜ �aœ$¤˜ Ô 'Ø!"ˆI�gÒ à"(¨! ˆI�gÑ à
|
|||
|
|
”�~ö &ð6 €O÷
|
|||
|
|
&ð €Or' c ó — e Zd ZdZd„ Zy)r zÑ
|
|||
|
|
Indicates a state or state action (e.g. #pop) to apply.
|
|||
|
|
For example default('#pop') is equivalent to ('', Token, '#pop')
|
|||
|
|
Note that state tuples may be used as well.
|
|||
|
|
|
|||
|
|
.. versionadded:: 2.0
|
|||
|
|
c ó — || _ y rg )r· )rC r· s r$ rE zdefault.__init__à s € ؈�
|
|||
|
|
r' N)r2 r3 r4 r5 rE r" r' r$ r r Ø s „ ñór' r c ó — e Zd ZdZdd„Zd„ Zy)r z•
|
|||
|
|
Indicates a list of literal words that is transformed into an optimized
|
|||
|
|
regex that matches any of the words.
|
|||
|
|
|
|||
|
|
.. versionadded:: 2.0
|
|||
|
|
c ó. — || _ || _ || _ y rg )r ÚprefixÚsuffix)rC r rÉ rÊ s r$ rE zwords.__init__ë s € ؈Œ
|
|||
|
|
Ø ˆŒ Ø ˆ� r' c óZ — t | j | j | j ¬« S )N©rÉ rÊ )r r rÉ rÊ rI s r$ rA z words.getð s € ܘŸ™¨D¯K©KÀÇ Á ÔLÐLr' N)r„ r„ )r2 r3 r4 r5 rE rA r" r' r$ r r ä s „ ñó ó
|
|||
|
|
Mr' r c ó< — e Zd ZdZd„ Zd„ Zd„ Zd„ Zd
|
|||
|
|
d„Zd„ Z d „ Z
|
|||
|
|
y) ÚRegexLexerMetazw
|
|||
|
|
Metaclass for RegexLexer, creates the self._tokens attribute from
|
|||
|
|
self.tokens on the first instantiation.
|
|||
|
|
c ó‚ — t |t « r|j « }t j ||« j
|
|||
|
|
S )zBPreprocess the regular expression component of a token definition.)rL r rA ÚreÚcompiler® )r“ ÚregexÚrflagsr· s r$ Ú_process_regexz RegexLexerMeta._process_regexú s. € ä �eœVÔ $Ø—I‘I“KˆEÜ�z‰z˜% Ó(×.Ñ.Ð.r' c óR — t |« t u st |« s
|
|||
|
|
J d|›�« ‚|S )z5Preprocess the token component of a token definition.z0token type must be simple type or callable, not )r, r
|
|||
|
|
Úcallable)r“ Útokens r$ Ú_process_tokenz RegexLexerMeta._process_token s2 € ä�E‹{œjÑ(¬H°U¬Oð IØ>¸u¸iÐ Hó IÐ;àˆ r' c óä — t |t « r5|dk( ry||v r|fS |dk( r|S |dd dk( rt |dd « S J d|›�« ‚t |t « rfd| j z }| xj d z
|
|||
|
|
c_ g }|D ]3 }||k7 s
|
|||
|
|
J d
|
|||
|
|
|›�« ‚|j | j
|
|||
|
|
|||« « Œ5 |||< |fS t |t « r |D ] }||v rŒ|d v rŒ
|
|||
|
|
J d|z « ‚ |S J d |›�« ‚)
|
|||
|
|
z=Preprocess the state transition action of a token definition.ú#popéÿÿÿÿú#pushNé z#pop:zunknown new state z_tmp_%drª zcircular state ref )rÚ rÜ zunknown new state def )rL rU Úintr� Ú_tmpnameÚextendÚ_process_stater’ )r“ Ú new_stateÚ unprocessedÚ processedÚ tmp_stateÚitokensÚistates r$ Ú_process_new_statez!RegexLexerMeta._process_new_state sQ € ä �i¤Ô %à˜FÒ"ØؘkÑ)Ø!�|Ð#ؘgÒ%Ø Ð Ø˜2˜A� 'Ò)ܘI a b˜MÓ*Ð*Ð*à@Ð 2°9°-Ð @Ó@�uÜ
|
|||
|
|
˜ ¤8Ô
|
|||
|
|
,à! C§L¡LÑ0ˆIØ �LŠL˜AÑ �L؈GØ#ò
|
|||
|
|
F�Ø Ò*ÐLÐ.AÀ&ÀÐ,LÓLÐ*Ø—‘˜s× 1Ñ 1°+Ø2;¸Vó Eõ Fð
|
|||
|
|
Fð $+ˆI�iÑ Ø �<Ð Ü
|
|||
|
|
˜ ¤5Ô
|
|||
|
|
)à#ò
|
|||
|
|
2�Ø +Ò-Ø Ð"3Ò3ð2à(¨6Ñ1ó2ð 4ð
|
|||
|
|
2ð Ð à @Ð 2°9°-Ð@Ó @�5r' c ó~ — t |t « s
|
|||
|
|
J d|›�« ‚|d dk7 s
|
|||
|
|
J d|›�« ‚||v r|| S g x}||< | j }|| D �]? }t |t « r;||k7 s
|
|||
|
|
J d|›�« ‚|j | j ||t |« « « ŒOt |t « rŒ`t |t « rO| j |j ||« }|j t j d« j d|f« Œ¿t |« t u s
|
|||
|
|
J d|›�« ‚ | j! |d ||« }| j' |d
|
|||
|
|
« }
|
|||
|
|
t) |« dk( rd}n| j |d ||« }|j ||
|
|||
|
|
|f« �ŒB |S # t" $ r } t% d |d ›d
|
|||
|
|
|›d | ›d | › �« | ‚d} ~ ww xY w)z%Preprocess a single state definition.zwrong state name r ú#zinvalid state name zcircular state reference r„ Nzwrong rule def zuncompilable regex z
|
|||
|
|
in state z of z: rª é )rL rU Úflagsr rà rá r� r rè r· rM rÐ rÑ r® r, r’ rÔ Ú ExceptionÚ
|
|||
|
|
ValueErrorrØ rY ) r“ rã rä r· ÚtokensrÓ Útdefrâ ÚrexÚerrr× s r$ rá z RegexLexerMeta._process_state) sü € ä˜%¤Ô%ÐDÐ):¸5¸)Ð'DÓDÐ%Ø�Q‰x˜3ŠÐ?Ð"5°e°YÐ ?Ó?ˆØ �IÑ Ø ˜UÑ#Ð #Ø$&Ð&ˆ�˜5Ñ!Ø—‘ˆØ Ñ&ó 3ˆDܘ$¤Ô(à˜u’}ÐKÐ(AÀ%ÀÐ&KÓK�}Ø—
|
|||
|
|
‘
|
|||
|
|
˜c× 0Ñ 0°¸iÜ14°T³ó <ô =àܘ$¤Ô)ð ܘ$¤Ô(Ø × 2Ñ 2°4·:±:¸{ÈIÓ V� Ø—
|
|||
|
|
‘
|
|||
|
|
œrŸz™z¨"›~× 3Ñ 3°T¸9Ð EÔFØ䘓:¤Ñ&Ð B¨/¸$¸Ð(BÓ BÐ&ð
|
|||
|
|
rØ×(Ñ(¨¨a©°&¸%Ó@�ð ×&Ñ& t¨A¡wÓ/ˆEä�4‹y˜AŠ~Ø ‘ à × 2Ñ 2°4¸±7Ø3>À ó K� ð
|
|||
|
|
�M‰M˜3 yÐ1Ö 2ðA 3ðB ˆ
|
|||
|
|
øô ò
|
|||
|
|
rÜ Ð#6°t¸A±w°kÀÈEÈ9ÐTXÐY\ÐX_Ð_aÐbeÐafÐ!gÓhÐnqÐqûð
|
|||
|
|
rús Ä)FÆ F<Æ F7Æ7F<Nc ó” — i x}| j |< |xs | j | }t |« D ] }| j |||« Œ |S )z-Preprocess a dictionary of token definitions.)Ú _all_tokensrï rÄ rá )r“ r/ Ú tokendefsrä r· s r$ Úprocess_tokendefz RegexLexerMeta.process_tokendefT sS € à,.Ð.ˆ �C—O‘O DÑ)Ø Ò1 §¡¨DÑ!1ˆ ܘ)“_ò <ˆEØ × Ñ ˜y¨)°UÕ ;ð <àÐr' c ó² — i }i }| j D ]¥ }|j j di « }|j « D ]t \ }}|j |« }|€!|||< |j t
|
|||
|
|
« }|||< Œ:|j |d« }|€ŒO||||dz |j t
|
|||
|
|
« } || z ||< Œv Œ§ |S # t $ r Y Œ†w xY w# t $ r Y Œ•w xY w)a
|
|||
|
|
Merge tokens from superclasses in MRO order, returning a single tokendef
|
|||
|
|
dictionary.
|
|||
|
|
|
|||
|
|
Any state that is not defined by a subclass will be inherited
|
|||
|
|
automatically. States that *are* defined by subclasses will, by
|
|||
|
|
default, override that state in the superclass. If a subclass wishes to
|
|||
|
|
inherit definitions from a superclass, it can use the special value
|
|||
|
|
"inherit", which will cause the superclass' state definition to be
|
|||
|
|
included at that point in the state.
|
|||
|
|
rï Nrª )Ú__mro__Ú__dict__rA ÚitemsÚindexr rî rà )
|
|||
|
|
r“ rï Ú inheritableÚcÚtoksr· rú ÚcuritemsÚ inherit_ndxÚ new_inh_ndxs
|
|||
|
|
r$ Ú
|
|||
|
|
get_tokendefsz RegexLexerMeta.get_tokendefs\ s € ð ˆØˆ Ø—‘ò CˆAØ—:‘:—>‘> (¨BÓ/ˆDà $§
|
|||
|
|
¡
|
|||
|
|
£ ò
|
|||
|
|
C‘ ��uØ!Ÿ:™: eÓ,�ØÐ#ð
|
|||
|
|
%*�F˜5‘Mð!Ø&+§k¡k´'Ó&:˜ ð *5�K Ñ&Ø à )Ÿo™o¨e°TÓ :� Ø Ð&Ø ð 7<�˜ [°¡]Ð3ðCð #(§+¡+¬gÓ"6�Kð *5°{Ñ)B�K Ò&ñ9
|
|||
|
|
Cð CðB ˆ
|
|||
|
|
øô) &ò !Ù ð!ûô "ò Ùðús$ Á B;ÂC
|
|||
|
|
Â; CÃCÃ
|
|||
|
|
CÃCc óè — d| j vrLi | _ d| _ t | d« r
|
|||
|
|
| j rn%| j d| j
|
|||
|
|
« « | _ t j | g|¢i |¤ŽS )z:Instantiate cls after preprocessing its token definitions.Ú_tokensr Útoken_variantsr„ )
|
|||
|
|
rù rô rß Úhasattrr rö r r r, Ú__call__)r“ r” Úkwdss r$ r zRegexLexerMeta.__call__� sh € à ˜CŸL™LÑ (Ø ˆCŒOØ ˆCŒLÜ�sÐ ,Ô-°#×2DÒ2Dàà !× 2Ñ 2°2°s×7HÑ7HÓ7JÓ K�” ä�}‰}˜SÐ0 4Ò0¨4Ñ0Ð0r' rg ) r2 r3 r4 r5 rÔ rØ rè rá rö r r r" r' r$ rÎ rÎ ô s. „ ñò
|
|||
|
|
/ò ò !AòF)óVò/ób 1r' rÎ c ó4 — e Zd ZdZej
|
|||
|
|
Zi Zdd„Zy)r z±
|
|||
|
|
Base for simple stateful regular expression-based lexers.
|
|||
|
|
Simplifies the lexing process so that you need only
|
|||
|
|
provide a list of states and regular expressions.
|
|||
|
|
c # ó: K — d}| j }t |« }||d } |D �]& \ }}} |||« }
|
|||
|
|
|
|