o RDiz@s,UddlmZddlZddlZddlZddlmZddlmZm Z ddl m Z ddl m Z mZmZmZmZmZddlmZdd lmZdd lmZdd lmZmZmZdd lmZmZdd l m!Z!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*m+Z+m,Z,m-Z-ddl.m/Z/ddl.m0Z1ddl2m3Z3ddl4m5Z5m6Z6m7Z7ddl8m9Z9e e:Z;dZm?Z?ddl0m@Z@mAZAddlBmCZCddlDmEZEddlFmGZGmHZHddlImJZJgZKdeLd<dZMd ZNd!ZO "dEdFd.d/ZPe Gd0d1d1eZQGd2d3d3eZRdGd=d>ZSGd?d@d@ejTZUGdAdBdBeUZVGdCdDdDeUZWdS)H) annotationsN) b64decode)Enumunique) getLogger) TYPE_CHECKINGAnyCallableIterator NamedTupleSequence)Self)ArrowConverterContext)exponential_backoff)OK UNAUTHORIZEDurlparse) FIELD_TYPESIterUnit)'ER_FAILED_TO_CONVERT_ROW_TO_PYTHON_TYPE ER_NO_PYARROW)ErrorInterfaceErrorNotSupportedErrorProgrammingError) RetryRequestget_http_retryable_erroris_retryable_http_coderaise_failed_request_errorraise_okta_unauthorized_error)installed_pandas)pyarrow)SecretDetector) HttpConfigSessionManagerSessionManagerFactory)TimerContextManager ) DataFrame)DataTypeTable)SnowflakeConnection)SnowflakeConverterType)ResultMetadataV2SnowflakeCursor)Responsez,list[Callable[[ResultMetadataV2], DataType]]FIELD_TYPE_TO_PA_TYPEz/x-amz-server-side-encryption-customer-algorithmz)x-amz-server-side-encryption-customer-keyAES256Tdatabytescontextruse_dict_resultboolnumpynumber_to_decimalrow_unitrcheck_error_on_every_columnc CsLddlm}m}td|tjkr|d||||||S|d||||||S)Nr)PyArrowRowIteratorPyArrowTableIteratorz+Using nanoarrow as the arrow data converter)nanoarrow_arrow_iteratorr=r>loggerdebugrROW_UNIT) r4r6r7r9r:r;r<r=r>rCY/var/www/Datamplify/venv/lib/python3.10/site-packages/snowflake/connector/result_batch.py_create_nanoarrow_iterator8s.   rEc@seZdZdZdZdZdZdS)DownloadMetricsz:Defines the keywords by which to store metrics for chunks.downloadparseloadN)__name__ __module__ __qualname____doc__rGrHrIrCrCrCrDrF[s rFc@s*eZdZUdZded<ded<ded<dS)RemoteChunkInfozKSmall class that holds information about chunks that are given by back-end.strurlintuncompressedSizecompressedSizeN)rJrKrLrM__annotations__rCrCrCrDrNds  rNcursorr0_formatrOdict[str, Any]schemaSequence[ResultMetadataV2]returnlist[ResultBatch]c sgd|d}|dd}|}g}|dkr'd fd d fd d |Dn |d }tjjd|vr|d} tdt| |d} id|vrpi|dD]\} } | | <d| vrntd| d| qWn| durtdt | t t <| t <d!dd|dkrfdd | D}n fdd | D}|D]} || j 8}q|dkrtj|d|jjjd}n9|durtj||jjjjjjjd}ntd|tjddjjjjjjjd}|g|S)"NrowtypetotalrjsoncolrWrZ"tuple[str, SnowflakeConverterType]cs$|d}jj||}||fS)Ntype)upper _connection converterto_python_method)r_ type_name python_method)rUrCrDcol_to_converterzs z6create_batches_from_response..col_to_convertercsg|]}|qSrCrC.0c)rhrCrD z0create_batches_from_response.. rowsetBase64chunksz chunk size=qrmk chunkHeaders encryptionzadded chunk header: key=z, value=zqrmk=rkrNcSst|d|d|ddS)NrPrRrS)rPrRrS)rN)rkrCrCrDremote_chunk_infos z7create_batches_from_response..remote_chunk_infoc s:g|]}t|d|jjjjjdqS)rowCountjson_result_force_utf8_decodingsession_manager)JSONResultBatch_use_dict_resultrc _json_result_force_utf8_decoding_session_managercloneri) chunk_headerscolumn_convertersrUrsrXrCrDrls  c s@g|]}t|d|jjjjjjjd qS)rtrw)ArrowResultBatchryrc_numpy_arrow_number_to_decimalr{r|ri) arrow_contextr}rUrsrXrCrDrls  rowsetrz9Don't know how to construct ResultBatches from response: )r_rWrZr`)rkrWrZrN)getrrc_session_parametersr@rAlenitemsr# mask_secrets SSE_C_AESSSE_C_ALGORITHM SSE_C_KEYrowcountrx from_dataryr{r|rrrerror)rUrVr4rXrowtypes total_lenfirst_chunk_lenrest_of_chunks rowset_b64rorp header_key header_valuerk first_chunkrC)rr}rhr~rUrsrXrDcreate_batches_from_responsels             rc@seZdZdZ d;dddZed>ddZed?ddZ ed@ddZ e j dAddZ ed d!Z e j dBd$d!Z dCd&d'Z d;dDd+d,ZejdEd.d/ZdFd0d1ZejdGd3d4ZejdHd6d7Zej d;dId9d:ZdS)J ResultBatchazRepresents what the back-end calls a result chunk. These are parts of a result set of a query. They each know how to retrieve their own results and convert them into Python native formats. As you are iterating through a ResultBatch you should check whether the yielded value is an ``Exception`` in case there was some error parsing the current row we might yield one of these to allow iteration to continue instead of raising the ``Exception`` when it occurs. These objects are pickleable for easy distribution and replication. Please note that the URLs stored in these do expire. The lifetime is dictated by the Snowflake back-end, at the time of writing this this is 6 hours. They can be iterated over multiple times and in different ways. Please follow the code in ``cursor.py`` to make sure that you are using this class correctly. NrrQr}dict[str, str] | NonersRemoteChunkInfo | NonerXrYr7r8rwSessionManager | NonerZNonec Cs||_||_||_||_|durdd|Dnd|_||_||_i|_d|_|jr=t |jj }|j dd}|d|_ dSt|j|_ dS)NcSsg|]}|qSrC)_to_result_metadata_v1rjsrCrCrDrlrmz(ResultBatch.__init__../r)r_chunk_headers_remote_chunk_info_schemarXryr{_metrics_datarrPpathrsplitidrO) selfrr}rsrXr7rw parsed_url path_partsrCrCrD__init__s  zResultBatch.__init__cCs |jduS)zWhether this chunk is local.NrrrCrCrD_locals zResultBatch._local int | NonecC|jrdS|jjS)zzReturns the size of chunk in bytes in compressed form. If it's a local chunk this function returns None. N)rrrSrrCrCrDcompressed_sizezResultBatch.compressed_sizecCr)z|Returns the size of chunk in bytes in uncompressed form. If it's a local chunk this function returns None. N)rrrRrrCrCrDuncompressed_size#rzResultBatch.uncompressed_size list[str]cCsdd|jDS)NcSg|]}|jqSrC)name)rjr_rCrCrDrl/z,ResultBatch.column_names..)rrrCrCrD column_names-szResultBatch.column_namescCs|jSNr{rrCrCrDrw1zResultBatch.session_managercCs ||_dSrr)rrwrCrCrDrw5s cCs|jjSr)r{configrrCrCrD http_config9szResultBatch.http_configrr$cCs$|jr ||j_dStj|d|_dS)N)r)r{rr& get_manager)rrrCrCrDr=s 8Iterator[dict | Exception] | Iterator[tuple | Exception]cCs|S)aReturns an iterator through the data this chunk holds. In case of this chunk being a local one it iterates through the local already parsed data and if it's a remote chunk it will download, parse its data and return an iterator through it. ) create_iterrrCrCrD__iter__Ds zResultBatch.__iter__ connectionSnowflakeConnection | Noner1cKsd}|dur |jnt}ttD]$}zt}td|j|jj }||j t d}|re|j re|j j dure|j |d} td|jd| | jdi|} Wdn1s_wYn>|jdur|j|d} | jdi|} Wdn1swYntd|jd tjd d } | jdi|} | jtkrtd |j WdWn|t| jrt| j} t| | jtkrtd| ntd|d| Wdn1swYWqttfy6} z<|tdkrt| tr | jd } | | } | t |}t!d|jd|dd|d| d t"#|WYd} ~ qd} ~ ww|$|j%t&j'j(<| S)z;Downloads the data that the ``ResultBatch`` is pointing at.rNz%started downloading result batch id: )rPheaderstimeoutrPzdownloading result batch id: z with existing session rz/ with new session through local session managerF) use_poolingz)successfully downloaded result batch id: rz+Failed to fetch the large result set batch z for the z th time, backing off for zs for the reason: '')rrC))_backoff_generatorrrangeMAX_DOWNLOAD_RETRYr'r@rArrrPrDOWNLOAD_TIMEOUTrestrwuse_requests_sessionrequestr{ use_sessionr&rr status_coderrrrrr r Exception isinstanceargsnext exceptiontimesleepget_timing_millisrrFrGvalue)rrkwargs sleep_timerbackoffretrydownload_metric chunk_url request_datasessionresponselocal_session_managerrerCrCrD _downloadOs      .   8 zResultBatch._download`Iterator[dict | Exception] | Iterator[tuple | Exception] | Iterator[Table] | Iterator[DataFrame]cKt)aDownloads the data from from blob storage that this ResultChunk points at. This function is the one that does the actual work for ``self.__iter__``. It is necessary because a ``ResultBatch`` can return multiple types of iterators. A good example of this is simply iterating through ``SnowflakeCursor`` and calling ``fetch_pandas_batches`` on it. NotImplementedError)rrrCrCrDrszResultBatch.create_itercCs"tsd}t}tt||ddS)NzOptional dependency: 'pandas' is not installed, please see the following link for install instructions: https://docs.snowflake.com/en/user-guide/python-connector-pandas.html#installationmsgerrno)r!rrerrorhandler_make_exceptionr)rrrrCrCrD_check_can_use_pandassz!ResultBatch._check_can_use_pandasr*cCrrrrrCrCrD to_pandasrzResultBatch.to_pandasr,cCrrrrrCrCrDto_arrowrzResultBatch.to_arrowr cKr)zDownloads the data that the ``ResultBatch`` is pointing at and populates it into self._data. Returns the instance itself.rrrrrCrCrD populate_dataszResultBatch.populate_datar)rrQr}rrsrrXrYr7r8rwrrZr)rZr8)rZr)rZr)rZr)rwrrZr)rr$rZr)rZr)rrrZr1)rZr)rZr)rZr*rZr,rrrZr )rJrKrLrMrpropertyrrrrrwsetterrrrabcabstractmethodrrrrrrCrCrCrDrsB           V    rcseZdZdddd7fddZe d8d9ddZd:dd Zd;d"d#Zdd,d-Z d8d?d/d0Z d1d2Z d3d4Z d5d6ZZS)@rxFNrurrQr}rrsrrXrYr~,Sequence[tuple[str, SnowflakeConverterType]]r7r8rvrwrrZrc s&t||||||||_||_dSr)superrrzr~) rrr}rsrXr~r7rvrw __class__rCrDrs  zJSONResultBatch.__init__r4Sequence[Sequence[Any]]data_lenc Cs*|t|dd||||d}|||_|S)z:Initializes a ``JSONResultBatch`` from static, local data.Nr)r_parser)clsr4rrXr~r7rw new_chunkrCrCrDrs  zJSONResultBatch.from_datarr1listc Csn|jr)z t|jddd}Wnty(}zd|}t|t|dd}~ww|j}t d d|d gS) a This function loads a compressed JSON file into memory. Returns: Whatever ``json.loads`` return, but in a list. Unfortunately there's no type hint for this. For context: https://github.com/python/typing/issues/182 zutf-8strict)errorsz2failed to decode json result content due to error )rNr[]) rzrOcontentrr@rrtextr^loadsjoin)rr read_dataexcerr_msgrCrCrD_loads    zJSONResultBatch._load0list[dict | Exception] | list[tuple | Exception]c Cstd|jg}|jrq|D]^}i}z(t|j||jD]\\}}}}|dus,|dur.|n||||j<q||Wqt yn} z&d|jd|d|d| } t | |t t | tdWYd} ~ qd} ~ ww|S|D]k}dgt|j}z/d} t|j||jD]\\}}}} |dus|dur|n|||| <| d 7} q|t|Wqst y} z&d| jd|d|d| } t | |t t | tdWYd} ~ qsd} ~ ww|S) z+Parses downloaded data into its final form.zparsing for result batch id: NzFailed to convert: field z: z::z , Error: rrr)r@rArryzipr~rrappendrrrrrrrtuple) rdownloaded_data result_listrow row_result_trkvr_rridx_colrCrCrDrsj$ %    zJSONResultBatch._parserOcCd|jdS)NzJSONResultChunk()rrrCrCrD__repr__VzJSONResultBatch.__repr__rrcKs|j|d}td|jt }||}Wdn1s"wYtd|j||jtj j <t }| |}Wdn1sLwY||jtj j <|S)Nr!started loading result batch id: "finished loading result batch id: ) rr@rArr'rrrrFrIrrrH)rrrr load_metricr parse_metric parsed_datarCrCrD _fetch_dataYs   zJSONResultBatch._fetch_datar cKs|jdd|i||_|SNrrC)r*rrrCrCrDriszJSONResultBatch.populate_datarcKs(|jrt|jSt|jdd|i|Sr+)riterrr*rrCrCrDros zJSONResultBatch.create_itercCstdt|dS)Nz Trying to use arrow fetching on z which is not ArrowResultChunk)rrarrCrCrD_arrow_fetching_errorvsz%JSONResultBatch._arrow_fetching_errorcC|rr-rrCrCrDr|zJSONResultBatch.to_pandascCr.rr/rrCrCrDrr0zJSONResultBatch.to_arrow)rrQr}rrsrrXrYr~rr7r8rvr8rwrrZrr) r4rrrQrXrYr~rr7r8rwr)rr1rZr)rZrrZrO)rrrZrr)rrrZr)rJrKrLr classmethodrrrr"r*rrr-rr __classcell__rCrCrrDrxs&    8   rxcseZdZ dCdDfdd ZdEddZdFddZ dGdHd%d&Ze dCdId(d)Z dCdJd-d.Z dCdKd0d1Z dLd3d4Z dCdMd5d6Z dCdNd8d9Z dCdOd;d<Z dCdPd>d?Z dCdQdAdBZZS)RrNrrQr}rrsrr6rr7r8r9rXrYr:rwrrZrc s,t|||||| ||_||_||_dSr)rr_contextr_number_to_decimal) rrr}rsr6r7r9rXr:rwrrCrDrs  zArrowResultBatch.__init__rOcCr)NzArrowResultChunk(r r!rrCrCrDr"r#zArrowResultBatch.__repr__rr1r;rrcCst|j|j|j|j|j|S)zCreates a ``PyArrowIterator`` from a response. This is used to iterate through results in different ways depending on which mode that ``PyArrowIterator`` is in. )rEr r4ryrr5)rrr;rCrCrDrszArrowResultBatch._loadTr4 str | bytes iter_unitr<cCsBt|dkr tgSt|trt|}t||j|j|j|j ||S)zCreates a ``PyArrowIterator`` files from a str. This is used to iterate through results in different ways depending on which mode that ``PyArrowIterator`` is in. r) rr,rrOrrEr4ryrr5)rr4r7r<rCrCrD _from_datas  zArrowResultBatch._from_datarc Cs$||dd||||||d } || _| S)zr TABLE_UNIT)rrrCrCrD_get_arrow_itersz ArrowResultBatch._get_arrow_iterr,cs4tr ddtDfdd|jD}t|S)z)Returns empty Arrow table based on schemacSrrC)pa_type)rjrrCrCrDrlrz8ArrowResultBatch._create_empty_table..cs$g|]}t|j|j|qSrC)pafieldr type_coderr2rCrDrls)r!rrrDrX empty_table)rfieldsrCrGrD_create_empty_table s  z$ArrowResultBatch._create_empty_tablecCs&t|j|dd}|dur|S|S)z%Returns this batch as a pyarrow Tabler$N)rrBrJ)rrvalrCrCrDrszArrowResultBatch.to_arrowr*cKs$||j|d}|jdi|S)z(Returns this batch as a pandas DataFramer$NrC)rrr)rrrtablerCrCrDrs zArrowResultBatch.to_pandasIterator[DataFrame]cKs0g}|jdd|i|}|js||t|S)z:An iterator for this batch which yields a pandas DataFramerNrC)remptyrr,)rrr iterator_data dataframerCrCrD_get_pandas_iter's  z!ArrowResultBatch._get_pandas_iterrcKsZ|dtj}|tjkr&|dd}|dkr |jdd|i|S|j|dS|j||dS) zKThe interface used by ResultSet to create an iterator for this ResultBatch.r7 structurepandasrr$r@NrC)poprrBrArQrBr>)rrrr7rRrCrCrDr1s   zArrowResultBatch.create_iterr cKs|j|dj|_|S)Nr$)rr rrrCrCrDrDszArrowResultBatch.populate_datar)rrQr}rrsrr6rr7r8r9r8rXrYr:r8rwrrZrr1)rr1r;rrZrT)r4r6r7rr<r8rZr)r4rOrrQr6rr7r8r9r8rXrYr:r8rwr)r7rrrrZr9)rrrZr?r)rrrZr,)rrrZr*)rrrZrM)rrrZrr)rJrKrLrr"rr8r2rr>rBrJrrrQrrr3rCrCrrDrs0      !     rrU)r4r5r6rr7r8r9r8r:r8r;rr<r8) rUr0rVrOr4rWrXrYrZr[)X __future__rrr^rbase64renumrrloggingrtypingrrr r r r typing_extensionsr rrbackoff_policiesrcompatrrr constantsrr errorcoderrrrrrrnetworkrrrrr optionsr!r"rDsecret_detectorr#rwr$r%r& time_utilr'rJr@rrrSr*r+r,rr-rdr.rUr/r0vendored.requestsr1r2rTrrrrErFrNrABCrrxrrCrCrCrDs\                # vx+