o EDi3@sddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZmZddlmZddlmZddlmZddlmZdd lmZmZdd lmZdd lmZdd l m!Z!e"e#Z$Gd ddZ%e%Z&GdddZ'dS)N)ThreadPoolExecutor)tzinfo)OptionalUnionDictAnySequenceIterable GeneratorBinaryIO)Client) StreamContext) HttpClient) ExternalData) QueryContext QueryResult) QuerySummary)ClickHouseType) InsertContextc@s eZdZdS)DefaultThreadPoolExecutorN)__name__ __module__ __qualname__rr^/var/www/Datamplify/venv/lib/python3.10/site-packages/clickhouse_connect/driver/asyncclient.pyrsrc*@s eZdZdZdeddededeede ffdd Z d d Z d e e fd dZde fddZde d efddZddZ                drde e de eeee effde ee efde ee e fde ee ee ee e fffde e de ede ede ed e ed!ed"e ee efd#e ee ee effd$e ed%e ed&e ee e fd ef"d'd(Z             dsde e de eeee effde ee efde ee e fde ee ee ee e fffde e de ed!ed"e ee efd#e ee ee effd$e ed%e ed&e ee e fd efd)d*Z             dsde e de eeee effde ee efde ee e fde ee ee ee e fffde e de ed!ed"e ee efd#e ee ee effd$e ed%e ed&e ee e fd efd+d,Z             dsde e de eeee effde ee efde ee e fde ee ee ee e fffde e de ed!ed"e ee efd#e ee ee effd$e ed%e ed&e ee e fd efd-d.Z    /  dtde de eeee effde ee efd0e d1ed%e ed&e ee e fd efd2d3Z    /  dtde de eeee effde ee efd0e d1ed%e ed&e ee e fd e!j"fd4d5Z#           dude e de eeee effde ee efde ee e fde ee e fde e de ed e ed!ed%e ed&e ee e ffd6d7Z$           dude e de eeee effde ee efde ee e fde ee e fde e de ed e ed!ed%e ed&e ee e fd efd8d9Z%                drde e de eeee effde ee efde ee e fde ee e fde e de ed e ed:e ed"e e d#e ee ee effd$e ed!ed%e ed;e ed&e ee e ff ddvde de eeee effde ee efd?e ed%e ed&e ee e fd@e d edAfdBdCZ'                drde e de eeee effde ee efde ee e fde ee e fde e de ed e ed:e ed"e e d#e ee ee effd$e ed!ed%e ed;e ed&e ee e fd ef"dDdEZ(      >dvde de eeee effde ee efd?e ed%e ed&e ee e fd@e d efdFdGZ)         H      H H   dwde ee efde eeee effde ee efde ee e fde ee ee ee e fffde e de ede ede ed e ed!e ed"e ee efd#e ee ee effd:e edIedJed%e ed;e ed&e ee e fd ef(dKdLZ*     dxde de eeee effde ee efd?e ed%e ed&e ee e ff dMdNZ+     dxde de eeee effde ee efd?e ed%e ed&e ee e fd efdOdPZ,    /  dtdQe de eeee effdRee efde ee efd1ed%e ed&e ee e fd ee eee e-ffdSdTZ.d efdUdVZ/   W    H   dydXe e dReeedYee e0e fdZe e d[ee1d\ee dede ee efd!e2d&e ee e fd e-fd]d^Z3         dzdXe dZe e de edYe ee d[ee1d\ee d!e2d&e ee e fd e-fd_d`Z4   d{dXe dZe de ed&e ee e fd e-f dadbZ5   d{dXe dcedAdZe e de ed&e ee e fd e-f dddeZ6     H   d|dXe dYe ee ee fdZe e d[ee1d\ee dede ee efdRe eeed&e ee e fd e2fdfdgZ7d!e2d e-fdhdiZ8      d}dXe dYe ee djee ee9eddfe:fde ed0e e dke e d&e ee e fd e-fdldmZ;d~dndoZr?r@rArBrCrDrErFrGrHrIrJrKrLr)rr=rrArDrIrGrBrKrFr>r=r@rHr-r?rLrCrErJrr_queryzs2$z!AsyncClient.query.._queryNr9get_running_looprun_in_executorr)r-r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrNloopresultrrMrr=cs ,zAsyncClient.querycH fdd}t}| j|IdH}|S)z Variation of main query method that returns a stream of column oriented blocks. For parameters, see the create_query_context method. :return: StreamContext -- Iterable stream context that returns column oriented blocks c& jj    d SN) r=r>r?r@rArBrCrGrHrIrJrKrL)rquery_column_block_streamrrArIrGrBrKr>r=r@rHr-r?rLrCrJrr_query_column_block_stream zIAsyncClient.query_column_block_stream.._query_column_block_streamNrO)r-r=r>r?r@rArBrCrGrHrIrJrKrLrYrRrSrrXrrWs &z%AsyncClient.query_column_block_streamcrT) Variation of main query method that returns a stream of row oriented blocks. For parameters, see the create_query_context method. :return: StreamContext -- Iterable stream context that returns blocks of rows crUrV)rquery_row_block_streamrrXrr_query_row_block_streamrZzCAsyncClient.query_row_block_stream.._query_row_block_streamNrO)r-r=r>r?r@rArBrCrGrHrIrJrKrLr]rRrSrrXrr\ &z"AsyncClient.query_row_block_streamcrT)r[crUrV)rquery_rows_streamrrXrr_query_rows_streamrZz9AsyncClient.query_rows_stream.._query_rows_streamNrO)r-r=r>r?r@rArBrCrGrHrIrJrKrLr`rRrSrrXrr_r^zAsyncClient.query_rows_streamTfmt use_databasec <fdd}t} | j|IdH} | S)a Query method that simply returns the raw ClickHouse format bytes. :param query: Query statement/format string :param parameters: Optional dictionary used to format the query :param settings: Optional dictionary of ClickHouse settings (key/string values) :param fmt: ClickHouse output format :param use_database Send the database parameter to ClickHouse so the command will be executed in the client database context :param external_data External data to send with the query :param transport_settings: Optional dictionary of transport level settings (HTTP headers, etc.) :return: bytes representing raw ClickHouse return value based on format c jjdSN)r=r>r?rarbrKrL)r raw_queryrrKrar>r=r-r?rLrbrr _raw_queryz)AsyncClient.raw_query.._raw_queryNrO) r-r=r>r?rarbrKrLrhrRrSrrgrrfs zAsyncClient.raw_queryc rc)a Query method that returns the result as an io.IOBase iterator. :param query: Query statement/format string :param parameters: Optional dictionary used to format the query :param settings: Optional dictionary of ClickHouse settings (key/string values) :param fmt: ClickHouse output format :param use_database Send the database parameter to ClickHouse so the command will be executed in the client database context :param external_data External data to send with the query :param transport_settings: Optional dictionary of transport level settings (HTTP headers, etc.) :return: io.IOBase stream/iterator for the result c rdre)r raw_streamrrgrr _raw_streamsz+AsyncClient.raw_stream.._raw_streamNrO) r-r=r>r?rarbrKrLrkrRrSrrgrrjs zAsyncClient.raw_streamc  D f dd} t} | j| IdH}|S)z Query method that returns the results as a numpy array. For parameter values, see the create_query_context method. :return: Numpy array representing the result set c "jj   d SN) r=r>r?r@rArBrCrFrGrKrL)rquery_npr rArGrBrKrFr>r=r@r-r?rLrCrr _query_np3s z'AsyncClient.query_np.._query_npNrO)r-r=r>r?r@rArBrCrFrGrKrLrqrRrSrrprro! "zAsyncClient.query_npc  rl)z Query method that returns the results as a stream of numpy arrays. For parameter values, see the create_query_context method. :return: Generator that yield a numpy array per block representing the result set c rmrn)rquery_np_streamrrprr_query_np_streamOs z5AsyncClient.query_np_stream.._query_np_streamNrO)r-r=r>r?r@rArBrCrFrGrKrLrtrRrSrrprrs=rrzAsyncClient.query_np_stream use_na_valuesuse_extended_dtypescN fdd}t}| j|IdH}|S)z Query method that results the results as a pandas dataframe. For parameter values, see the create_query_context method. :return: Pandas dataframe representing the result set cn jjdiddd dddddd d d d d dd d SNr=r>r?r@rArBrCrFrurHrIrJrGrKrvrLr)rquery_dfrrArIrGrBrKrFr>r=r@rHr-r?rLrvrurCrJrr _query_dfps6z'AsyncClient.query_df.._query_dfNrO)r-r=r>r?r@rArBrCrFrurHrIrJrGrKrvrLr|rRrSrr{rrzYs , zAsyncClient.query_dfpandas use_stringsdataframe_library)z pd.DataFramez pl.DataFramec rc)a Query method using the ClickHouse Arrow format to return a DataFrame with PyArrow dtype backend. This provides better performance and memory efficiency compared to the standard query_df method, though fewer output formatting options. :param query: Query statement/format string :param parameters: Optional dictionary used to format the query :param settings: Optional dictionary of ClickHouse settings (key/string values) :param use_strings: Convert ClickHouse String type to Arrow string type (instead of binary) :param external_data: ClickHouse "external data" to send with query :param transport_settings: Optional dictionary of transport level settings (HTTP headers, etc.) :param dataframe_library: Library to use for DataFrame creation ("pandas" or "polars") :return: DataFrame (pandas or polars based on dataframe_library parameter) c jjdSN)r=r>r?r~rKrLr)rquery_df_arrowrrrKr>r=r-r?rLr~rr_query_df_arrowz3AsyncClient.query_df_arrow.._query_df_arrowNrO) r-r=r>r?r~rKrLrrrRrSrrrr}s  zAsyncClient.query_df_arrowcrw)z Query method that returns the results as a StreamContext. For parameter values, see the create_query_context method. :return: Generator that yields a Pandas dataframe per block representing the result set crxry)rquery_df_streamrr{rr_query_df_streams6z5AsyncClient.query_df_stream.._query_df_streamNrO)r-r=r>r?r@rArBrCrFrurHrIrJrGrKrvrLrrRrSrr{rrs , zAsyncClient.query_df_streamc rc)ao Query method that returns the results as a stream of DataFrames with PyArrow dtype backend. Each DataFrame represents a block from the ClickHouse response. :param query: Query statement/format string :param parameters: Optional dictionary used to format the query :param settings: Optional dictionary of ClickHouse settings (key/string values) :param use_strings: Convert ClickHouse String type to Arrow string type (instead of binary) :param external_data: ClickHouse "external data" to send with query :param transport_settings: Optional dictionary of transport level settings (HTTP headers, etc.) :param dataframe_library: Library to use for DataFrame creation ("pandas" or "polars") :return: StreamContext that yields DataFrames (pandas or polars based on dataframe_library parameter) c rr)rquery_df_arrow_streamrrrr_query_df_arrow_streamrzAAsyncClient.query_df_arrow_stream.._query_df_arrow_streamNrO) r-r=r>r?r~rKrLrrrRrSrrrrs  z!AsyncClient.query_df_arrow_streamF streaming as_pandascCs|jjdid|d|d|d|d|d|d|d|d | d | d | d | d | d|d|d|d|d|d|S)a Creates or updates a reusable QueryContext object :param query: Query statement/format string :param parameters: Optional dictionary used to format the query :param settings: Optional dictionary of ClickHouse settings (key/string values) :param query_formats: See QueryContext __init__ docstring :param column_formats: See QueryContext __init__ docstring :param encoding: See QueryContext __init__ docstring :param use_none: Use None for ClickHouse NULL instead of default values. Note that using None in Numpy arrays will force the numpy array dtype to 'object', which is often inefficient. This effect also will impact the performance of Pandas dataframes. :param column_oriented: Deprecated. Controls orientation of the QueryResult result_set property :param use_numpy: Return QueryResult columns as one-dimensional numpy arrays :param max_str_len: Limit returned ClickHouse String values to this length, which allows a Numpy structured array even with ClickHouse variable length String columns. If 0, Numpy arrays for String columns will always be object arrays :param context: An existing QueryContext to be updated with any provided parameter values :param query_tz Either a string or a pytz tzinfo object. (Strings will be converted to tzinfo objects). Values for any DateTime or DateTime64 column in the query will be converted to Python datetime.datetime objects with the selected timezone :param column_tzs A dictionary of column names to tzinfo objects (or strings that will be converted to tzinfo objects). The timezone will be applied to datetime objects returned in the query :param use_na_values: Deprecated alias for use_advanced_dtypes :param as_pandas Return the result columns as pandas.Series objects :param streaming Marker used to correctly configure streaming queries :param external_data ClickHouse "external data" to send with query :param use_extended_dtypes: Only relevant to Pandas Dataframe queries. Use Pandas "missing types", such as pandas.NA and pandas.NaT for ClickHouse NULL values, as well as extended Pandas dtypes such as IntegerArray and StringArray. Defaulted to True for query_df methods :param transport_settings: Optional dictionary of transport level settings (HTTP headers, etc.) :return: Reusable QueryContext r=r>r?r@rArBrCrDrErFrGrHrIrurrrKrvrLNr)rcreate_query_context)r-r=r>r?r@rArBrCrDrErFrGrHrIrurrrKrvrLrrrrsB5  z AsyncClient.create_query_contextc :fdd}t}|j|IdH} | S)ae Query method using the ClickHouse Arrow format to return a PyArrow table :param query: Query statement/format string :param parameters: Optional dictionary used to format the query :param settings: Optional dictionary of ClickHouse settings (key/string values) :param use_strings: Convert ClickHouse String type to Arrow string type (instead of binary) :param external_data ClickHouse "external data" to send with query :param transport_settings: Optional dictionary of transport level settings (HTTP headers, etc.) :return: PyArrow.Table cjjdSN)r=r>r?r~rKrL)r query_arrowrrKr>r=r-r?rLr~rr _query_arrowD z-AsyncClient.query_arrow.._query_arrowNrO) r-r=r>r?r~rKrLrrRrSrrrr2 zAsyncClient.query_arrowc r)a Query method that returns the results as a stream of Arrow tables :param query: Query statement/format string :param parameters: Optional dictionary used to format the query :param settings: Optional dictionary of ClickHouse settings (key/string values) :param use_strings: Convert ClickHouse String type to Arrow string type (instead of binary) :param external_data ClickHouse "external data" to send with query :param transport_settings: Optional dictionary of transport level settings (HTTP headers, etc.) :return: Generator that yields a PyArrow.Table for per block representing the result set crr)rquery_arrow_streamrrrr_query_arrow_stream_rz;AsyncClient.query_arrow_stream.._query_arrow_streamNrO) r-r=r>r?r~rKrLrrRrSrrrrMrzAsyncClient.query_arrow_streamcmddatac rc)a Client method that returns a single value instead of a result set :param cmd: ClickHouse query/command as a python format string :param parameters: Optional dictionary of key/values pairs to be formatted :param data: Optional 'data' for the command (for INSERT INTO in particular) :param settings: Optional dictionary of ClickHouse settings (key/string values) :param use_database: Send the database parameter to ClickHouse so the command will be executed in the client database context. Otherwise, no database will be specified with the command. This is useful for determining the default user database :param external_data ClickHouse "external data" to send with command/query :param transport_settings: Optional dictionary of transport level settings (HTTP headers, etc.) :return: Decoded response from ClickHouse as either a string, int, or sequence of strings, or QuerySummary if no data returned c sjjdS)N)rr>rr?rbrKrL)rcommandrrrrKr>r-r?rLrbrr_commandriz%AsyncClient.command.._commandNrO) r-rr>rr?rbrKrLrrRrSrrrrhs zAsyncClient.commandcs.fdd}t}|j|IdH}|S)z Validate the connection, does not throw an Exception (see debug logs) :return: ClickHouse server is up and reachable cs jSN)rpingrr<rr_pings zAsyncClient.ping.._pingNrO)r-rrRrSrr<rrs  zAsyncClient.ping*table column_namesdatabase column_typescolumn_type_namesc  sB f dd} t} | j| IdH} | S)a Method to insert multiple rows/data matrix of native Python objects. If context is specified arguments other than data are ignored :param table: Target table :param data: Sequence of sequences of Python data :param column_names: Ordered list of column names or '*' if column types should be retrieved from the ClickHouse table definition :param database: Target database -- will use client default database if not specified. :param column_types: ClickHouse column types. If set then column data does not need to be retrieved from the server :param column_type_names: ClickHouse column type names. If set then column data does not need to be retrieved from the server :param column_oriented: If true the data is already "pivoted" in column form :param settings: Optional dictionary of ClickHouse settings (key/string values) :param context: Optional reusable insert context to allow repeated inserts into the same table with different data batches :param transport_settings: Optional dictionary of transport level settings (HTTP headers, etc.) :return: QuerySummary with summary information, throws exception if insert fails c s jj  d S)N) rrrrrrrDr?rGrL)rinsertr rrDrrrGrrr-r?rrLrr_inserts z#AsyncClient.insert.._insertNrO)r-rrrrrrrDr?rGrLrrRrSrrrrs  zAsyncClient.insertc @ f dd} t} | j| IdH} | S)aX Insert a pandas DataFrame into ClickHouse. If context is specified arguments other than df are ignored :param table: ClickHouse table :param df: two-dimensional pandas dataframe :param database: Optional ClickHouse database :param settings: Optional dictionary of ClickHouse settings (key/string values) :param column_names: An optional list of ClickHouse column names. If not set, the DataFrame column names will be used :param column_types: ClickHouse column types. If set then column data does not need to be retrieved from the server :param column_type_names: ClickHouse column type names. If set then column data does not need to be retrieved from the server :param context: Optional reusable insert context to allow repeated inserts into the same table with different data batches :param transport_settings: Optional dictionary of transport level settings (HTTP headers, etc.) :return: QuerySummary with summary information, throws exception if insert fails c sjj d S)N) rdfrr?rrrrGrL)r insert_dfr rrrrGrrr-r?rrLrr _insert_dfs z)AsyncClient.insert_df.._insert_dfNrO) r-rrrr?rrrrGrLrrRrSrrrr zAsyncClient.insert_dfc 8fdd}t}|j|IdH}|S)a Insert a PyArrow table DataFrame into ClickHouse using raw Arrow format :param table: ClickHouse table :param arrow_table: PyArrow Table object :param database: Optional ClickHouse database :param settings: Optional dictionary of ClickHouse settings (key/string values) :param transport_settings: Optional dictionary of transport level settings (HTTP headers, etc.) :return: QuerySummary with summary information, throws exception if insert fails csjjdS)N)r arrow_tablerr?rL)r insert_arrowrrrr-r?rrLrr _insert_arrows z/AsyncClient.insert_arrow.._insert_arrowNrO) r-rrrr?rLrrRrSrrrrs zAsyncClient.insert_arrowrc r)a Insert a pandas DataFrame with PyArrow backend or a polars DataFrame into ClickHouse using Arrow format. This method is optimized for DataFrames that already use Arrow format, providing better performance than the standard insert_df method. Validation is performed and an exception will be raised if this requirement is not met. Polars DataFrames are natively Arrow-based and don't require additional validation. :param table: ClickHouse table name :param df: Pandas DataFrame with PyArrow dtype backend or Polars DataFrame :param database: Optional ClickHouse database name :param settings: Optional dictionary of ClickHouse settings (key/string values) :param transport_settings: Optional dictionary of transport level settings (HTTP headers, etc.) :return: QuerySummary with summary information, throws exception if insert fails csjjdS)N)rrrr?rL)rinsert_df_arrowrrrr-r?rrLrr_insert_df_arrowsz5AsyncClient.insert_df_arrow.._insert_df_arrowNrO) r-rrrr?rLrrRrSrrrrs  zAsyncClient.insert_df_arrowc r)a Builds a reusable insert context to hold state for a duration of an insert :param table: Target table :param database: Target database. If not set, uses the client default database :param column_names: Optional ordered list of column names. If not set, all columns ('*') will be assumed in the order specified by the table definition :param database: Target database -- will use client default database if not specified :param column_types: ClickHouse column types. Optional Sequence of ClickHouseType objects. If neither column types nor column type names are set, actual column types will be retrieved from the server. :param column_type_names: ClickHouse column type names. Specified column types by name string :param column_oriented: If true the data is already "pivoted" in column form :param settings: Optional dictionary of ClickHouse settings (key/string values) :param data: Initial dataset for insert :param transport_settings: Optional dictionary of transport level settings (HTTP headers, etc.) :return: Reusable insert context c sjj d S)N) rrrrrrDr?rrL)rcreate_insert_contextr rrDrrrrr-r?rrLrr_create_insert_context9s zAAsyncClient.create_insert_context.._create_insert_contextNrO) r-rrrrrrDr?rrLrrRrSrrrrrz!AsyncClient.create_insert_contextcs0fdd}t}|j|IdH}|S)z Subclass implementation of the data insert :context: InsertContext parameter object :return: No return, throws an exception if the insert fails csjjdS)N)rG)r data_insertrrGr-rr _data_insertJsz-AsyncClient.data_insert.._data_insertNrO)r-rGrrRrSrrrrCs zAsyncClient.data_insert insert_block compressionc rc)ar Insert data already formatted in a bytes object :param table: Table name (whether qualified with the database name or not) :param column_names: Sequence of column names :param insert_block: Binary or string data already in a recognized ClickHouse format :param settings: Optional dictionary of ClickHouse settings (key/string values) :param compression: Recognized ClickHouse `Accept-Encoding` header compression value :param transport_settings: Optional dictionary of transport level settings (HTTP headers, etc.) :param fmt: Valid clickhouse format c sjjdS)N)rrrr?rarrL)r raw_insertrrrrarr-r?rrLrr _raw_insertcs z+AsyncClient.raw_insert.._raw_insertNrO) r-rrrr?rarrLrrRrSrrrrQs zAsyncClient.raw_insertcs|Srrr<rrr __aenter__lszAsyncClient.__aenter__cs|IdHdSr)r8)r-exc_typeexc_valexc_tbrrr __aexit__oszAsyncClient.__aexit__)NNNNNNNNNNNNNNNN) NNNNNNNNNNNNN)NNNTNN) NNNNNNNNNNN)NNNNNr})NNNNNNNNFrNNNNFFNNN)NNNNN) NNrNNNFNNN) NNNNNNNNN)NNN)NNNNFNNN)NNNNNN)r2r)r2N)>rrr__doc__r)r intrrrr.r1rstrr3r5boolr7r8rrrrrrrr=r rWr\r_bytesrfioIOBaserjrorsrzrrrrrrrrrr rrrrrrrrr r rrrrrrrrs           $      "      !      "               ' )      ( (      C         )  &  '  %  r)(r9rloggingr'concurrent.futures.threadrdatetimertypingrrrrrr r r clickhouse_connect.driver.clientr clickhouse_connect.driver.commonr $clickhouse_connect.driver.httpclientr"clickhouse_connect.driver.externalrclickhouse_connect.driver.queryrr!clickhouse_connect.driver.summaryr!clickhouse_connect.datatypes.baser clickhouse_connect.driver.insertr getLoggerrr+rr)rrrrrs&  (