o KDiY@s ddlmZddlZddlmZddlmZddlmZddl Z ddl m Z m Z ddl mZmZmZmZmZdd lmZdd lmZdd lmZmZed ZGd ddZGdddZGdddeZGdddedZGdddeZ GdddeZ!GdddeZ"ee#eeefZ$dS))unicode_literalsN) OrderedDict)chain) getLogger)Field StringField) parse_tsvNO_VALUEget_subclass_names arg_to_sqlunescapeQuerySet)F)Merge Distributedclickhouse_ormc@s(eZdZdZdZdZddZddZdS) Constraintz% Defines a model constraint. NcCs ||_dS)ze Initializer. Expects an expression that ClickHouse will verify when inserting data. N)expr)selfrrS/var/www/Datamplify/venv/lib/python3.10/site-packages/infi/clickhouse_orm/models.py__init__s zConstraint.__init__cCsd|jt|jfS)z_ Returns the SQL statement for defining this constraint during table creation. zCONSTRAINT `%s` CHECK %s)namer rrrrrcreate_table_sql!szConstraint.create_table_sql)__name__ __module__ __qualname____doc__rparentrrrrrrrs  rc@sfeZdZdZdZdZddZddZeddZ ed d Z ed d Z ed dZ edddZ dS)Indexz( Defines a data-skipping index. NcCs||_||_||_dS)a Initializer. - `expr` - a column, expression, or tuple of columns and expressions to index. - `type` - the index type. Use one of the following methods to specify the type: `Index.minmax`, `Index.set`, `Index.ngrambf_v1`, `Index.tokenbf_v1` or `Index.bloom_filter`. - `granularity` - index block size (number of multiples of the `index_granularity` defined by the engine). N)rtype granularity)rrr#r$rrrr0s  zIndex.__init__cCsd|jt|j|j|jfS)zZ Returns the SQL statement for defining this index during table creation. z$INDEX `%s` %s TYPE %s GRANULARITY %d)rr rr#r$rrrrr=szIndex.create_table_sqlcCsdS)z An index that stores extremes of the specified expression (if the expression is tuple, then it stores extremes for each element of tuple). The stored info is used for skipping blocks of data like the primary key. minmaxrrrrrr%Csz Index.minmaxcCd|S)z An index that stores unique values of the specified expression (no more than max_rows rows, or unlimited if max_rows=0). Uses the values to check if the WHERE expression is not satisfiable on a block of data. zset(%d)r)max_rowsrrrsetKsz Index.setcCsd||||fS)u7 An index that stores a Bloom filter containing all ngrams from a block of data. Works only with strings. Can be used for optimization of equals, like and in expressions. - `n` — ngram size - `size_of_bloom_filter_in_bytes` — Bloom filter size in bytes (you can use large values here, for example 256 or 512, because it can be compressed well). - `number_of_hash_functions` — The number of hash functions used in the Bloom filter. - `random_seed` — The seed for Bloom filter hash functions. zngrambf_v1(%d, %d, %d, %d)r)nsize_of_bloom_filter_in_bytesnumber_of_hash_functions random_seedrrr ngrambf_v1Ts zIndex.ngrambf_v1cCsd|||fS)u An index that stores a Bloom filter containing string tokens. Tokens are sequences separated by non-alphanumeric characters. - `size_of_bloom_filter_in_bytes` — Bloom filter size in bytes (you can use large values here, for example 256 or 512, because it can be compressed well). - `number_of_hash_functions` — The number of hash functions used in the Bloom filter. - `random_seed` — The seed for Bloom filter hash functions. ztokenbf_v1(%d, %d, %d)r)r*r+r,rrr tokenbf_v1bs zIndex.tokenbf_v1皙?cCr&)z An index that stores a Bloom filter containing values of the index expression. - `false_positive` - the probability (between 0 and 1) of receiving a false positive response from the filter zbloom_filter(%f)r)false_positiverrr bloom_filteroszIndex.bloom_filter)r/)rrrr rr!rr staticmethodr%r(r-r.r1rrrrr"(s       r"cs>eZdZdZiZfddZed ddZeddZZ S) ModelBasezP A metaclass for ORM models. It adds the _fields list to model classes. c szi}i}i}|D]}t|tr!||j||j||jq|D]!\}} t| tr4| ||<q&t| tr>| ||<q&t| t rG| ||<q&t |ddd}i} d} |D]&\}} | j sb| j rgt | |<qXt| jtrtt | |<d} qX| | jtj| |<qXt|t|||tdd|D| | d}tt||t|||} t|||D]\}} t| d | t| d |q| S) NcSs |djS)Nr)creation_counter)itemrrrs z#ModelBase.__new__..)keyFTcSsg|] }|djs|qS)rreadonly.0frrr sz%ModelBase.__new__..)_fields _constraints_indexes_writable_fields _defaults_has_funcs_as_defaultsr!r) isinstancer3updater>r?r@itemsrrr"sortedalias materializedr defaultr to_pythonpytzUTCdictrsuper__new__strrsetattr)clsrbasesattrsfields constraintsindexesbaser)objdefaultshas_funcs_as_defaultsr<model __class__rrrPsR               zModelBase.__new__ AdHocModelcCslt|}|dt|}||jvr|j|Si}|D] \}}||||<q|||tf|}||j|<|S)N )listrQad_hoc_model_cachecreate_ad_hoc_fieldrPModel)rSrV model_name cache_keyrUrdb_type model_classrrrcreate_ad_hoc_models    zModelBase.create_ad_hoc_modelc Cs ddlmm}|dr|j|S|dr/|dd}|j|r+|dddSddS|drYd d |d dd D^}}|jt ||rU|dddd Sdd S|drl||dd}| |S|drdd |ddd D}t t |dksJd|||d}| |S|drt |dd}| |S|dr|d}dd ||ddd D} t||d|d} | | S|dr||dd}||S|dr||dd}||S|d} t|| s td|t|| S)NrEnumz DateTime( r)timezonez DateTime64(cSg|]}|qSrstripr;srrrr=z1ModelBase.create_ad_hoc_field.. ,) precisionrnArrayTuplecSrorrprrrrrr=rtz'No support for mixed types in tuples - FixedString Decimal(cSsg|]}t|qSr)intrq)r;r)rrrr=rNullableLowCardinalityzNo field class for %s)infi.clickhouse_orm.fieldsrrV startswith BaseEnumFieldrd DateTimeFieldsplitDateTime64Fieldr ArrayFieldlenr(FixedStringFieldindexgetattr NullableFieldLowCardinalityFieldhasattrNotImplementedError) rSrh orm_fieldsrnrw inner_fieldtypeslengthpargs field_classrrrrrdsX                      zModelBase.create_ad_hoc_field)r`) rrrr rcrP classmethodrjrd __classcell__rrr^rr3zs 7 r3cseZdZdZdZdZdZdZfddZfddZ dd Z d d Z d d Z e ddZe ddZe ddZe ddZe ejdfddZd)ddZd)ddZddZd*dd Ze d!d"Ze d+d#d$Ze d%d&Ze d'd(ZZS),rea A base class for ORM models. Each model class represent a ClickHouse table. For example: class CPUStats(Model): timestamp = DateTimeField() cpu_id = UInt16Field() cpu_percent = Float32Field() engine = Memory() NFc s`tt||j|j|D]\}}||}|r$t|||qt d|j j |fdS)a Creates a model instance, using keyword arguments as field values. Since values are immediately converted to their Pythonic type, invalid values will cause a `ValueError` to be raised. Unrecognized field names will cause an `AttributeError`. z"%s does not have a field called %sN) rOrer__dict__rErBrF get_fieldrRAttributeErrorr_r)rkwargsrvaluefieldr^rrrs zModel.__init__cs||}|r6|tkr6z||tj}||Wnty5t\}}}d ||}| |||wt t | ||dS)z When setting a field value, converts the value to its Pythonic type and validates it. This may raise a `ValueError`. z{} (field '{}')N)rr rKrLutcvalidate ValueErrorsysexc_infoformatwith_tracebackrOre __setattr__)rrrrtpvtbnew_msgr^rrr&s    zModel.__setattr__cCs(ddlm}t||sJd||_dS) Sets the `Database` that this model instance belongs to. This is done automatically when the instance is read from the database or written to it. r)Databasez+database must be database.Database instanceN)databaserrD _database)rdbrrrr set_database6s  zModel.set_databasecC|jS)z Gets the `Database` that this model instance belongs to. Returns `None` unless the instance was read from the database or written to it. )rrrrr get_database@zModel.get_databasecCs |j|S)zQ Gets a `Field` instance given its name, or `None` if not found. )r>get)rrrrrrGs zModel.get_fieldcCs |jS)z Returns the model's database table name. By default this is the class name converted to lowercase. Override this if you want to use a different table name. )rlowerrSrrr table_nameMs zModel.table_namecCr)z Return True if some of the model's fields use a function expression as a default value. This requires special handling when inserting instances. )rCrrrrr\VszModel.has_funcs_as_defaultscCsd|j|fg}g}|D]\}}|d||j|dfq|jD] }|d|q)|j D] }|d|q:|d ||d|d|j |d |S) P Returns the SQL statement for creating a table for this model. &CREATE TABLE IF NOT EXISTS `%s`.`%s` ( %s %srz %s, ) ENGINE =  ) db_namerrVrFappendget_sqlr?valuesrr@joinengine)rSrpartsrFrrcirrrr^s  zModel.create_table_sqlcCsd|j|fS)zJ Returns the SQL command for deleting this model's table. zDROP TABLE IF EXISTS `%s`.`%s`)rr)rSrrrrdrop_table_sqltszModel.drop_table_sqlc Csltt|}i}|D]}t||}t|ddp|} |t|| ||<q |di|} |dur4| || S)a Create a model instance from a tab-separated line. The line may or may not include a newline. The `field_names` list must match the fields defined in the model, but does not have to include all of them. - `line`: the TSV-formatted data. - `field_names`: names of the model fields in the data. - `timezone_in_use`: the timezone to use when parsing dates and datetimes. Some fields use their own timezones. - `database`: if given, sets the database that this instance belongs to. rnNr)iterr rrKnextr) rSline field_namestimezone_in_userrrrrfield_timezonerZrrrfrom_tsv{s   zModel.from_tsvTcs0|j|j| d}dfdd|DS)z Returns the instance's column values as a tab-separated line. A newline is not included. - `include_readonly`: if false, returns only fields that can be inserted into database. writable c3s&|]\}}|j|ddVqdS)FquoteN) to_db_string)r;rrdatarr s$zModel.to_tsv..)rrVrrF)rinclude_readonlyrVrrrto_tsvsz Model.to_tsvcCs`|j}|j| d}g}|D]\}}||tkr*||d|j||ddqd|S)a Returns the instance's column keys and values as a tab-separated line. A newline is not included. Fields that were not assigned a value are omitted. - `include_readonly`: if false, returns only fields that can be inserted into database. r=Frr)rrVrFr rrr)rrrrVrrrrrrto_tskvs   z Model.to_tskvcCs,|jr|dn|d}|d7}|dS)z^ Returns the instance as a bytestring ready to be inserted into the database. Frzutf-8)rCrrencode)rrsrrrrs zModel.to_db_stringcs@|j| d}durfdd|D}|jfdd|DS)z Returns the instance's column values as a dict. - `include_readonly`: if false, returns only fields that can be inserted into database. - `field_names`: an iterable of field names to return (optional) rNcsg|]}|vr|qSrrr:)rrrr=rz!Model.to_dict..csi|]}||qSrr)r;rrrr sz!Model.to_dict..)rVr)rrrrVr)rrrto_dicts z Model.to_dictcCs t||S)zS Returns a `QuerySet` for selecting instances of this model class. r)rSrrrr objects_ins zModel.objects_incCs|r|jS|jS)z Returns an `OrderedDict` of the model's fields (from name to `Field` instance). If `writable` is true, only writable fields are included. Callers should not modify the dictionary. )rAr>)rSrrrrrVsz Model.fieldscCr)zC Returns true if the model is marked as read only. ) _readonlyrrrr is_read_onlyrzModel.is_read_onlycCr)zF Returns true if the model represents a system table. )_systemrrrris_system_modelrzModel.is_system_model)T)TN)F)rrrr rrrrrrrrrrrr\rrrLrrrrrrrrVrrrrrr^rresB              re) metaclassc@seZdZeddZdS) BufferModelcCsBd|j||j|jjfg}|j|}||d|S)rz1CREATE TABLE IF NOT EXISTS `%s`.`%s` AS `%s`.`%s`ra)rrr main_modelrrr)rSrr engine_strrrrrs     zBufferModel.create_table_sqlN)rrrrrrrrrrsrc@s*eZdZdZdZeddZeddZdS) MergeModelz Model for Merge engine Predefines virtual _table column an controls that rows can't be inserted to this table type https://clickhouse.tech/docs/en/single/index.html#document-table_engines/merge Tr8cCst|jts Jdd|j|fg}g}|D]\}}|dkr1|d||j|dfq|d ||d|d|j |d |S) rz+engine must be an instance of engines.Merger_tablerrrrrr) rDrrrrrVrFrrrr)rSrrcolsrrrrrrs  zMergeModel.create_table_sqlN) rrrr r9rrrrrrrrrs  rcs8eZdZdZfddZeddZeddZZS)DistributedModelz: Model class for use with a `Distributed` engine. cs(t|jts Jdtt||}|S)rz1engine must be an instance of engines.Distributed)rDrrrOrr)rrresr^rrrszDistributedModel.set_databasecCsL|jjrdSdd|jD}|stdt|dkrtd|d|j_dS)a Remember: Distributed table does not store any data, just provides distributed access to it. So if we define a model with engine that has no defined table for data storage (see FooDistributed below), that table cannot be successfully created. This routine can automatically fix engine's storage table by finding the first non-distributed model among your model's superclasses. >>> class Foo(Model): ... id = UInt8Field(1) ... >>> class FooDistributed(Foo, DistributedModel): ... engine = Distributed('my_cluster') ... >>> FooDistributed.engine.table None >>> FooDistributed.fix_engine() >>> FooDistributed.engine.table However if you prefer more explicit way of doing things, you can always mention the Foo model twice without bothering with any fixes: >>> class FooDistributedVerbose(Foo, DistributedModel): ... engine = Distributed('my_cluster', Foo) >>> FooDistributedVerbose.engine.table See tests.test_engines:DistributedTestCase for more examples NcSs$g|]}t|trt|ts|qSr) issubclassrer)r;brrrr=Cs z5DistributedModel.fix_engine_table..zaWhen defining Distributed engine without the table_name ensure that your model has a parent modelrzyWhen defining Distributed engine without the table_name ensure that your model has exactly one non-distributed superclassr)rr __bases__ TypeErrorrtable)rSstorage_modelsrrrfix_engine_tables" z!DistributedModel.fix_engine_tablecCsNt|jts Jd|d|j||jjd|j|g}d|S)rz+engine must be engines.Distributed instancez5CREATE TABLE IF NOT EXISTS `{0}`.`{1}` AS `{0}`.`{2}`rr) rDrrrrrrrr)rSrrrrrrPs z!DistributedModel.create_table_sql) rrrr rrrrrrrr^rrs  2r)% __future__rr collectionsr itertoolsrloggingrrLrVrrutilsr r r r r queryrfuncsrenginesrrloggerrr"r#r3rerrrlocals__all__rrrrs,      ReR