o DDi @sUdZddlmZmZddlmZddlmZmZm Z m Z m Z ddl m Z ddlmZddlmZmZddlmZmZdd lmZmZdd lmZdd lmZejZee d <ej!Z"ee d <ej#Z$ee d<ej%Z&ee d<ej'Z(ee d<ej)Z*ee d<ej+Z,ee d<ej-Z.ee d<GdddeZ/e/j0Z1e/e d<e/j2Z3e/e d<e eddGdddZ4de5dedeefdd Z6de5d!e e5dedeefd"d#Zde5d!e e5dedeed$d$ffd%d&Z7d'e4d(ed)e edefd*d+Z8d$S),a Parso's tokenize doesn't give us tokens in the format that we'd ideally like, so this performs a small number of transformations to the token stream: - `end_pos` is precomputed as a property, instead of lazily as a method, for more efficient access. - `whitespace_before` and `whitespace_after` have been added. These include the correct indentation information. - `prefix` is removed, since we don't use it anywhere. - `ERRORTOKEN` and `ERROR_DEDENT` have been removed, because we don't intend to support error recovery. If we encounter token errors, we'll raise a ParserSyntaxError instead. If performance becomes a concern, we can rewrite this later as a fork of the original tokenize module, instead of as a wrapper. ) dataclassfield)Enum) GeneratorIteratorListOptionalSequence) add_slots)ParserSyntaxError)PythonTokenTypes TokenType)Tokentokenize_lines)PythonVersionInfo split_lines)rWhitespaceState _ERRORTOKEN _ERROR_DEDENT_INDENT_DEDENT _ENDMARKER_FSTRING_START _FSTRING_END_OPc@seZdZdZdZdS)_ParenthesisOrFStringStackEntryrN)__name__ __module__ __qualname__ PARENTHESISFSTRINGr"r"X/var/www/Datamplify/venv/lib/python3.10/site-packages/libcst/_parser/wrapped_tokenize.pyr3sr_PARENTHESIS_STACK_ENTRY_FSTRING_STACK_ENTRYF)frozenc@s`eZdZUeeed<edddZeed<edddZ e eed<ee dZ e e ed<d S) _TokenizeStatelinescCstdddddS)NrF)linecolumnabsolute_indentis_parenthesizedrr"r"r"r#Esz_TokenizeState.)default_factoryprevious_whitespace_statecCsdgS)Nr*r"r"r"r"r#r/Isindentsparenthesis_or_fstring_stackN)rrrr str__annotations__rr1rr2rlistr3rr"r"r"r#r'@s r'code version_inforeturncCsDz ddlm}||WSty!t|dd}t|||YSw)NrtokenizeT)keepends) libcst_nativer; ImportErrorrr)r7r8native_tokenizer(r"r"r#r;Os    r;r(cCs8z ddlm}||WStyt|||YSw)Nrr:)r=r;r>tokenize_lines_py)r7r(r8r?r"r"r#rYs    rNccsht|}tt||}zt|}Wn tyYdSw|D] }t|||V|}qt||dVdS)N)r'iterorig_tokenize_linesnext StopIteration_convert_token)r7r(r8stateorig_tokens_iter curr_token next_tokenr"r"r#r@es  r@rFrHrIc Cs<|j}|j}|j}|turt|d|j|d|dd|tur.td|j|d|ddd}|turOt|j dkr=dn|j d}|j d }|t|d}|durx|j} | turo|j\} } |j |j| dd| n | t urx|j |j } |tus|t us|tur| } |}nt|}t|dkr|dt|dt|d f}n |d|dt|f}|j}z+|tur| tn|tur| n|tur|d vr| tn|d vr| Wntytd |j|d|ddwt|dko|d tk}t|d|d|j d |} | |_ t||||| | |S) Nz is not a valid token.rr))r(raw_line raw_columnz,Inconsistent indentation. Expected a dedent.r*z([{z)]}z=Encountered a closing brace without a matching opening brace.)typestring start_posrr r(rrlenr2appendrpopr1rrr3rr%rrr$ IndexErrorrr)rFrHrIct_type ct_string ct_start_posrelative_indent old_indent new_indentnt_typent_line nt_columnwhitespace_beforewhitespace_after ct_end_posr( pof_stackr.r"r"r#rExs     "      rE)9__doc__ dataclassesrrenumrtypingrrrrr libcst._add_slotsr libcst._exceptionsr !libcst._parser.parso.python.tokenr r $libcst._parser.parso.python.tokenizer OrigTokenrrBlibcst._parser.parso.utilsrrlibcst._parser.types.token%libcst._parser.types.whitespace_stater ERRORTOKENrr5 ERROR_DEDENTrINDENTrDEDENTr ENDMARKERr FSTRING_STARTr FSTRING_ENDrOPrrr r$r!r%r'r4r;r@rEr"r"r"r#sn