diff --git a/impuls/__pycache__/settings.cpython-311.pyc b/impuls/__pycache__/settings.cpython-311.pyc index 900b3081..eb990cef 100644 Binary files a/impuls/__pycache__/settings.cpython-311.pyc and b/impuls/__pycache__/settings.cpython-311.pyc differ diff --git a/impuls/__pycache__/urls.cpython-311.pyc b/impuls/__pycache__/urls.cpython-311.pyc index 5ba46589..6e88f7b5 100644 Binary files a/impuls/__pycache__/urls.cpython-311.pyc and b/impuls/__pycache__/urls.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/INSTALLER b/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/INSTALLER new file mode 100644 index 00000000..a1b589e3 --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/INSTALLER @@ -0,0 +1 @@ +pip diff --git a/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/LICENSE.txt b/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/LICENSE.txt new file mode 100644 index 00000000..d6456956 --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/LICENSE.txt @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/METADATA b/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/METADATA new file mode 100644 index 00000000..72f77b9a --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/METADATA @@ -0,0 +1,387 @@ +Metadata-Version: 2.1 +Name: pymemcache +Version: 4.0.0 +Summary: A comprehensive, fast, pure Python memcached client +Home-page: https://github.com/pinterest/pymemcache +Author: Jon Parise +Author-email: jon@pinterest.com +License: Apache License 2.0 +Keywords: memcache,client,database +Classifier: Programming Language :: Python +Classifier: Programming Language :: Python :: 3 :: Only +Classifier: Programming Language :: Python :: 3.7 +Classifier: Programming Language :: Python :: 3.8 +Classifier: Programming Language :: Python :: 3.9 +Classifier: Programming Language :: Python :: 3.10 +Classifier: Programming Language :: Python :: Implementation :: PyPy +Classifier: License :: OSI Approved :: Apache Software License +Classifier: Topic :: Database +Requires-Python: >=3.7 +Description-Content-Type: text/x-rst +License-File: LICENSE.txt + +pymemcache +========== + +.. image:: https://img.shields.io/pypi/v/pymemcache.svg + :target: https://pypi.python.org/pypi/pymemcache + +.. image:: https://readthedocs.org/projects/pymemcache/badge/?version=master + :target: https://pymemcache.readthedocs.io/en/latest/ + :alt: Master Documentation Status + +A comprehensive, fast, pure-Python memcached client. + +pymemcache supports the following features: + +* Complete implementation of the memcached text protocol. +* Connections using UNIX sockets, or TCP over IPv4 or IPv6. +* Configurable timeouts for socket connect and send/recv calls. +* Access to the "noreply" flag, which can significantly increase the speed of writes. +* Flexible, modular and simple approach to serialization and deserialization. +* The (optional) ability to treat network and memcached errors as cache misses. + +Installing pymemcache +===================== + +Install from pip: + +.. code-block:: bash + + pip install pymemcache + +For development, clone from github and run the tests: + +.. code-block:: bash + + git clone https://github.com/pinterest/pymemcache.git + cd pymemcache + +Run the tests (make sure you have a local memcached server running): + +.. code-block:: bash + + tox + +Usage +===== + +See the documentation here: https://pymemcache.readthedocs.io/en/latest/ + +Django +------ + +Since version 3.2, Django has included a pymemcache-based cache backend. +See `its documentation +`__. + +On older Django versions, you can use +`django-pymemcache `_. + +Comparison with Other Libraries +=============================== + +pylibmc +------- + +The pylibmc library is a wrapper around libmemcached, implemented in C. It is +fast, implements consistent hashing, the full memcached protocol and timeouts. +It does not provide access to the "noreply" flag. It also isn't pure Python, +so using it with libraries like gevent is out of the question, and its +dependency on libmemcached poses challenges (e.g., it must be built against +the same version of libmemcached that it will use at runtime). + +python-memcached +---------------- + +The python-memcached library implements the entire memcached text protocol, has +a single timeout for all socket calls and has a flexible approach to +serialization and deserialization. It is also written entirely in Python, so +it works well with libraries like gevent. However, it is tied to using thread +locals, doesn't implement "noreply", can't treat errors as cache misses and is +slower than both pylibmc and pymemcache. It is also tied to a specific method +for handling clusters of memcached servers. + +memcache_client +--------------- + +The team at mixpanel put together a pure Python memcached client as well. It +has more fine grained support for socket timeouts, only connects to a single +host. However, it doesn't support most of the memcached API (just get, set, +delete and stats), doesn't support "noreply", has no serialization or +deserialization support and can't treat errors as cache misses. + +External Links +============== + +The memcached text protocol reference page: + https://github.com/memcached/memcached/blob/master/doc/protocol.txt + +The python-memcached library (another pure-Python library): + https://github.com/linsomniac/python-memcached + +Mixpanel's Blog post about their memcached client for Python: + https://engineering.mixpanel.com/we-went-down-so-we-wrote-a-better-pure-python-memcache-client-b409a9fe07a9 + +Mixpanel's pure Python memcached client: + https://github.com/mixpanel/memcache_client + +Bye-bye python-memcached, hello pymemcache (migration guide) + https://jugmac00.github.io/blog/bye-bye-python-memcached-hello-pymemcache/ + +Credits +======= + +* `Charles Gordon `_ +* `Dave Dash `_ +* `Dan Crosta `_ +* `Julian Berman `_ +* `Mark Shirley `_ +* `Tim Bart `_ +* `Thomas Orozco `_ +* `Marc Abramowitz `_ +* `Marc-Andre Courtois `_ +* `Julien Danjou `_ +* `INADA Naoki `_ +* `James Socol `_ +* `Joshua Harlow `_ +* `John Anderson `_ +* `Adam Chainz `_ +* `Ernest W. Durbin III `_ +* `Remco van Oosterhout `_ +* `Nicholas Charriere `_ +* `Joe Gordon `_ +* `Jon Parise `_ +* `Stephen Rosen `_ +* `Feras Alazzeh `_ +* `Moisés Guimarães de Medeiros `_ +* `Nick Pope `_ +* `Hervé Beraud `_ +* `Martin Jørgensen `_ + +We're Hiring! +============= +Are you really excited about open-source? Or great software engineering? +Pinterest is `hiring `_! + +Changelog +========= +New in version 4.0.0 +-------------------- +* Dropped Python 2 and 3.6 support + `#321 `_ + `#363 `_ +* Begin adding typing +* Add pluggable compression serde + `#407 `_ + + +New in version 3.5.2 +-------------------- +* Handle blank ``STAT`` values. + +New in version 3.5.1 +-------------------- +* ``Client.get`` returns the default when using ``ignore_exc`` and if memcached + is unavailable +* Added ``noreply`` support to ``HashClient.flush_all``. + +New in version 3.5.0 +-------------------- +* Sockets are now closed on ``MemcacheUnexpectedCloseError``. +* Added support for TCP keepalive for client sockets on Linux platforms. +* Added retrying mechanisms by wrapping clients. + +New in version 3.4.4 +-------------------- +* Idle connections will be removed from the pool after ``pool_idle_timeout``. + +New in version 3.4.3 +-------------------- +* Fix ``HashClient.{get,set}_many()`` with UNIX sockets. + +New in version 3.4.2 +-------------------- +* Remove trailing space for commands that don't take arguments, such as + ``stats``. This was a violation of the memcached protocol. + +New in version 3.4.1 +-------------------- +* CAS operations will now raise ``MemcacheIllegalInputError`` when ``None`` is + given as the ``cas`` value. + +New in version 3.4.0 +-------------------- +* Added IPv6 support for TCP socket connections. Note that IPv6 may be used in + preference to IPv4 when passing a domain name as the host if an IPv6 address + can be resolved for that domain. +* ``HashClient`` now supports UNIX sockets. + +New in version 3.3.0 +-------------------- +* ``HashClient`` can now be imported from the top-level ``pymemcache`` package + (e.g. ``pymemcache.HashClient``). +* ``HashClient.get_many()`` now longer stores ``False`` for missing keys from + unavailable clients. Instead, the result won't contain the key at all. +* Added missing ``HashClient.close()`` and ``HashClient.quit()``. + +New in version 3.2.0 +-------------------- +* ``PooledClient`` and ``HashClient`` now support custom ``Client`` classes + +New in version 3.1.1 +-------------------- +* Improve ``MockMemcacheClient`` to behave even more like ``Client`` + +New in version 3.1.0 +-------------------- +* Add TLS support for TCP sockets. +* Fix corner case when dead hashed server comes back alive. + +New in version 3.0.1 +-------------------- +* Make MockMemcacheClient more consistent with the real client. +* Pass ``encoding`` from HashClient to its pooled clients when ``use_pooling`` + is enabled. + +New in version 3.0.0 +-------------------- +* The serialization API has been reworked. Instead of consuming a serializer + and deserializer as separate arguments, client objects now expect an argument + ``serde`` to be an object which implements ``serialize`` and ``deserialize`` + as methods. (``serialize`` and ``deserialize`` are still supported but + considered deprecated.) +* Validate integer inputs for ``expire``, ``delay``, ``incr``, ``decr``, and + ``memlimit`` -- non-integer values now raise ``MemcacheIllegalInputError`` +* Validate inputs for ``cas`` -- values which are not integers or strings of + 0-9 now raise ``MemcacheIllegalInputError`` +* Add ``prepend`` and ``append`` support to ``MockMemcacheClient``. +* Add the ``touch`` method to ``HashClient``. +* Added official support for Python 3.8. + +New in version 2.2.2 +-------------------- +* Fix ``long_description`` string in Python packaging. + +New in version 2.2.1 +-------------------- +* Fix ``flags`` when setting multiple differently-typed values at once. + +New in version 2.2.0 +-------------------- +* Drop official support for Python 3.4. +* Use ``setup.cfg`` metadata instead ``setup.py`` config to generate package. +* Add ``default_noreply`` parameter to ``HashClient``. +* Add ``encoding`` parameter to ``Client`` constructors (defaults to ``ascii``). +* Add ``flags`` parameter to write operation methods. +* Handle unicode key values in ``MockMemcacheClient`` correctly. +* Improve ASCII encoding failure exception. + +New in version 2.1.1 +-------------------- +* Fix ``setup.py`` dependency on six already being installed. + +New in version 2.1.0 +-------------------- +* Public classes and exceptions can now be imported from the top-level + ``pymemcache`` package (e.g. ``pymemcache.Client``). + `#197 `_ +* Add UNIX domain socket support and document server connection options. + `#206 `_ +* Add support for the ``cache_memlimit`` command. + `#211 `_ +* Commands key are now always sent in their original order. + `#209 `_ + +New in version 2.0.0 +-------------------- +* Change set_many and set_multi api return value. `#179 `_ +* Fix support for newbytes from python-future. `#187 `_ +* Add support for Python 3.7, and drop support for Python 3.3 +* Properly batch Client.set_many() call. `#182 `_ +* Improve _check_key() and _store_cmd() performance. `#183 `_ +* Properly batch Client.delete_many() call. `#184 `_ +* Add option to explicitly set pickle version used by serde. `#190 `_ + +New in version 1.4.4 +-------------------- +* pypy3 to travis test matrix +* full benchmarks in test +* fix flake8 issues +* Have mockmemcacheclient support non-ascii strings +* Switch from using pickle format 0 to the highest available version. See `#156 `_ + + *Warning*: different versions of python have different highest pickle versions: https://docs.python.org/3/library/pickle.html + + +New in version 1.4.3 +-------------------- +* Documentation improvements +* Fixed cachedump stats command, see `#103 `_ +* Honor default_value in HashClient + +New in version 1.4.2 +-------------------- +* Drop support for python 2.6, see `#109 `_ + +New in version 1.4.1 +-------------------- +* Python 3 serializations fixes `#131 `_ +* Drop support for pypy3 +* Comment cleanup +* Add gets_many to hash_client +* Better checking for illegal chars in key + +New in version 1.4.0 +-------------------- +* Unicode keys support. It is now possible to pass the flag ``allow_unicode_keys`` when creating the clients, thanks @jogo! +* Fixed a bug where PooledClient wasn't following ``default_noreply`` arg set on init, thanks @kols! +* Improved documentation + +New in version 1.3.8 +-------------------- +* use cpickle instead of pickle when possible (python2) + +New in version 1.3.7 +-------------------- +* default parameter on get(key, default=0) +* fixed docs to autogenerate themselves with sphinx +* fix linter to work with python3 +* improve error message on illegal Input for the key +* refactor stat parsing +* fix MockMemcacheClient +* fix unicode char in middle of key bug + +New in version 1.3.6 +-------------------- +* Fix flake8 and cleanup tox building +* Fix security vulnerability by sanitizing key input + +New in version 1.3.5 +-------------------- +* Bug fix for HashClient when retries is set to zero. +* Adding the VERSION command to the clients. + +New in version 1.3.4 +-------------------- +* Bug fix for the HashClient that corrects behavior when there are no working servers. + +New in version 1.3.3 +-------------------- +* Adding caching to the Travis build. +* A bug fix for pluggable hashing in HashClient. +* Adding a default_noreply argument to the Client ctor. + +New in version 1.3.2 +-------------------- +* Making the location of Memcache Exceptions backwards compatible. + +New in version 1.3.0 +-------------------- +* Python 3 Support +* Introduced HashClient that uses consistent hasing for allocating keys across many memcached nodes. It also can detect servers going down and rebalance keys across the available nodes. +* Retry sock.recv() when it raises EINTR + +New in version 1.2.9 +-------------------- +* Introduced PooledClient a thread-safe pool of clients diff --git a/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/RECORD b/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/RECORD new file mode 100644 index 00000000..92d70f5b --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/RECORD @@ -0,0 +1,53 @@ +pymemcache-4.0.0.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4 +pymemcache-4.0.0.dist-info/LICENSE.txt,sha256=z8d0m5b2O9McPEK1xHG_dWgUBT6EfBDz6wA0F7xSPTA,11358 +pymemcache-4.0.0.dist-info/METADATA,sha256=aJhanppO-xfdUsaGxPNotz1PyKpIQArFxBDzAff40oQ,14360 +pymemcache-4.0.0.dist-info/RECORD,, +pymemcache-4.0.0.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 +pymemcache-4.0.0.dist-info/WHEEL,sha256=z9j0xAa_JmUKMpmz72K0ZGALSM_n-wQVmGbleXx2VHg,110 +pymemcache-4.0.0.dist-info/top_level.txt,sha256=A7o5woZP9MH_1OrIbwQIsJsB8UcIX4Kcj4IBeuYakx0,11 +pymemcache/__init__.py,sha256=PHZ_lmH3ue3R7QKJP-9OMuTl19MVYgDFopMFtSQCNJk,693 +pymemcache/__pycache__/__init__.cpython-311.pyc,, +pymemcache/__pycache__/exceptions.cpython-311.pyc,, +pymemcache/__pycache__/fallback.cpython-311.pyc,, +pymemcache/__pycache__/pool.cpython-311.pyc,, +pymemcache/__pycache__/serde.cpython-311.pyc,, +pymemcache/client/__init__.py,sha256=iwKIUkD67iy93gQhhiRDhD1bL-bKVNMKncg8pE7ie04,706 +pymemcache/client/__pycache__/__init__.cpython-311.pyc,, +pymemcache/client/__pycache__/base.cpython-311.pyc,, +pymemcache/client/__pycache__/hash.cpython-311.pyc,, +pymemcache/client/__pycache__/murmur3.cpython-311.pyc,, +pymemcache/client/__pycache__/rendezvous.cpython-311.pyc,, +pymemcache/client/__pycache__/retrying.cpython-311.pyc,, +pymemcache/client/base.py,sha256=95JAH7upCHoMO7b0YkOF01O5Xtb2DB2GJZsUSPSVwdw,61383 +pymemcache/client/hash.py,sha256=5v_uMGFux8xd7_aw5P7u6GQWb77gqssjJ2bruMxr3Qc,16047 +pymemcache/client/murmur3.py,sha256=HaN3Lwzl1e8gn-a8_lTgiEOdfD0cJnPR0FOaSt6iBSA,1465 +pymemcache/client/rendezvous.py,sha256=bllKETXBAemaHWXG9nv0eaaUnzzacf-1hScUU8sHTmk,1273 +pymemcache/client/retrying.py,sha256=NLMUm--hfBbElQ3o5LK9pubFMH8KlJiDrcSrSrvWiJ8,6443 +pymemcache/exceptions.py,sha256=sq1e0N8Qk79eTQWl_8rxDBHWgaSD1mzkbS94iDv6-w8,1252 +pymemcache/fallback.py,sha256=g7HlZ_BNaC9R2-lBxUYA2F0hqhFcpvWMttEsDTUOzSo,4186 +pymemcache/pool.py,sha256=rSODZOZKb7n29SyNjB0FjtHLkEmpRI0_38Nfz1Mvr4w,4384 +pymemcache/serde.py,sha256=1RgrobrPrv_sSY2i80oTMW95jcpeYsMHXspxSzb7xKI,6042 +pymemcache/test/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0 +pymemcache/test/__pycache__/__init__.cpython-311.pyc,, +pymemcache/test/__pycache__/conftest.cpython-311.pyc,, +pymemcache/test/__pycache__/test_benchmark.cpython-311.pyc,, +pymemcache/test/__pycache__/test_client.cpython-311.pyc,, +pymemcache/test/__pycache__/test_client_hash.cpython-311.pyc,, +pymemcache/test/__pycache__/test_client_retry.cpython-311.pyc,, +pymemcache/test/__pycache__/test_compression.cpython-311.pyc,, +pymemcache/test/__pycache__/test_integration.cpython-311.pyc,, +pymemcache/test/__pycache__/test_rendezvous.cpython-311.pyc,, +pymemcache/test/__pycache__/test_serde.cpython-311.pyc,, +pymemcache/test/__pycache__/test_utils.cpython-311.pyc,, +pymemcache/test/__pycache__/utils.cpython-311.pyc,, +pymemcache/test/conftest.py,sha256=9JwSnZg-2A5ceK1getNevPujqPemexd_MOyTSIssw00,3018 +pymemcache/test/test_benchmark.py,sha256=3d9xT8WyOt1pVcF_ZOnlRv98_UvscaL-8rnQL7vyDgM,2963 +pymemcache/test/test_client.py,sha256=b9C14iEq47ZG8lyQ2G9mFaLo67fHGd7jzU0qALFASzc,56913 +pymemcache/test/test_client_hash.py,sha256=GwRWNQQzu2DACQJnlt9MERxB8TTT4wBnivzGDrE06hQ,17202 +pymemcache/test/test_client_retry.py,sha256=Ahlog0AREKILTlgRyTB_SmsztPMbRI11KJVoaJcKPqc,10361 +pymemcache/test/test_compression.py,sha256=rL6pvXF4EU9eb3BFB1m2J7vch3a43-laE6ktXxYH7tk,5638 +pymemcache/test/test_integration.py,sha256=W3Rc4sjnb3VPooZrhSU9F-V-hEmyLvd6wxPL8r9uHVI,12263 +pymemcache/test/test_rendezvous.py,sha256=gClzxCNpl4qUotleOsvES5wuto1GU4_OOWXP2C6qEBw,5188 +pymemcache/test/test_serde.py,sha256=Ou5JGCRXfvDfnq4LTtAAjgGEcsHXXXCS1x9UaP3IJww,3843 +pymemcache/test/test_utils.py,sha256=uaI1scSgfV5LMTstte3iCuxTmZ3-XhlZk7QuHRuBFU4,2576 +pymemcache/test/utils.py,sha256=f5JCkYfQ_KLBxrxzgHp-cjop-LeXgwIsvBEV-VqD2DE,6870 diff --git a/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/REQUESTED b/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/REQUESTED new file mode 100644 index 00000000..e69de29b diff --git a/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/WHEEL b/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/WHEEL new file mode 100644 index 00000000..0b18a281 --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/WHEEL @@ -0,0 +1,6 @@ +Wheel-Version: 1.0 +Generator: bdist_wheel (0.37.1) +Root-Is-Purelib: true +Tag: py2-none-any +Tag: py3-none-any + diff --git a/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/top_level.txt b/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/top_level.txt new file mode 100644 index 00000000..738eceac --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache-4.0.0.dist-info/top_level.txt @@ -0,0 +1 @@ +pymemcache diff --git a/lib/python3.11/site-packages/pymemcache/__init__.py b/lib/python3.11/site-packages/pymemcache/__init__.py new file mode 100644 index 00000000..da11017a --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/__init__.py @@ -0,0 +1,14 @@ +__version__ = "4.0.0" + +from pymemcache.client.base import Client # noqa +from pymemcache.client.base import PooledClient # noqa +from pymemcache.client.hash import HashClient # noqa +from pymemcache.client.base import KeepaliveOpts # noqa + +from pymemcache.exceptions import MemcacheError # noqa +from pymemcache.exceptions import MemcacheClientError # noqa +from pymemcache.exceptions import MemcacheUnknownCommandError # noqa +from pymemcache.exceptions import MemcacheIllegalInputError # noqa +from pymemcache.exceptions import MemcacheServerError # noqa +from pymemcache.exceptions import MemcacheUnknownError # noqa +from pymemcache.exceptions import MemcacheUnexpectedCloseError # noqa diff --git a/lib/python3.11/site-packages/pymemcache/__pycache__/__init__.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/__pycache__/__init__.cpython-311.pyc new file mode 100644 index 00000000..44fd61b0 Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/__pycache__/__init__.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/__pycache__/exceptions.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/__pycache__/exceptions.cpython-311.pyc new file mode 100644 index 00000000..eb1f539f Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/__pycache__/exceptions.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/__pycache__/fallback.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/__pycache__/fallback.cpython-311.pyc new file mode 100644 index 00000000..c95d457b Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/__pycache__/fallback.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/__pycache__/pool.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/__pycache__/pool.cpython-311.pyc new file mode 100644 index 00000000..a9c9e95c Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/__pycache__/pool.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/__pycache__/serde.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/__pycache__/serde.cpython-311.pyc new file mode 100644 index 00000000..e2302933 Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/__pycache__/serde.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/client/__init__.py b/lib/python3.11/site-packages/pymemcache/client/__init__.py new file mode 100644 index 00000000..250f4e1f --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/client/__init__.py @@ -0,0 +1,14 @@ +# API Backwards compatibility + +from pymemcache.client.base import Client # noqa +from pymemcache.client.base import PooledClient # noqa +from pymemcache.client.hash import HashClient # noqa +from pymemcache.client.retrying import RetryingClient # noqa + +from pymemcache.exceptions import MemcacheError # noqa +from pymemcache.exceptions import MemcacheClientError # noqa +from pymemcache.exceptions import MemcacheUnknownCommandError # noqa +from pymemcache.exceptions import MemcacheIllegalInputError # noqa +from pymemcache.exceptions import MemcacheServerError # noqa +from pymemcache.exceptions import MemcacheUnknownError # noqa +from pymemcache.exceptions import MemcacheUnexpectedCloseError # noqa diff --git a/lib/python3.11/site-packages/pymemcache/client/__pycache__/__init__.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/client/__pycache__/__init__.cpython-311.pyc new file mode 100644 index 00000000..f6604533 Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/client/__pycache__/__init__.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/client/__pycache__/base.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/client/__pycache__/base.cpython-311.pyc new file mode 100644 index 00000000..736dd336 Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/client/__pycache__/base.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/client/__pycache__/hash.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/client/__pycache__/hash.cpython-311.pyc new file mode 100644 index 00000000..df7882ef Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/client/__pycache__/hash.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/client/__pycache__/murmur3.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/client/__pycache__/murmur3.cpython-311.pyc new file mode 100644 index 00000000..e75996c8 Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/client/__pycache__/murmur3.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/client/__pycache__/rendezvous.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/client/__pycache__/rendezvous.cpython-311.pyc new file mode 100644 index 00000000..8a45c239 Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/client/__pycache__/rendezvous.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/client/__pycache__/retrying.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/client/__pycache__/retrying.cpython-311.pyc new file mode 100644 index 00000000..55bc6e3f Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/client/__pycache__/retrying.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/client/base.py b/lib/python3.11/site-packages/pymemcache/client/base.py new file mode 100644 index 00000000..04ae0520 --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/client/base.py @@ -0,0 +1,1753 @@ +# Copyright 2012 Pinterest.com +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import errno +import platform +import socket +from functools import partial +from ssl import SSLContext +from types import ModuleType +from typing import Any, Callable, Dict, Iterable, List, Optional, Tuple, Union + +from pymemcache import pool +from pymemcache.exceptions import ( + MemcacheClientError, + MemcacheIllegalInputError, + MemcacheServerError, + MemcacheUnexpectedCloseError, + MemcacheUnknownCommandError, + MemcacheUnknownError, +) +from pymemcache.serde import LegacyWrappingSerde + +RECV_SIZE = 4096 +VALID_STORE_RESULTS = { + b"set": (b"STORED", b"NOT_STORED"), + b"add": (b"STORED", b"NOT_STORED"), + b"replace": (b"STORED", b"NOT_STORED"), + b"append": (b"STORED", b"NOT_STORED"), + b"prepend": (b"STORED", b"NOT_STORED"), + b"cas": (b"STORED", b"EXISTS", b"NOT_FOUND"), +} + +SOCKET_KEEPALIVE_SUPPORTED_SYSTEM = { + "Linux", +} + +STORE_RESULTS_VALUE = { + b"STORED": True, + b"NOT_STORED": False, + b"NOT_FOUND": None, + b"EXISTS": False, +} + +ServerSpec = Union[Tuple[str, int], str] +Key = Union[bytes, str] + + +# Some of the values returned by the "stats" command +# need mapping into native Python types +def _parse_bool_int(value: bytes) -> bool: + return int(value) != 0 + + +def _parse_bool_string_is_yes(value: bytes) -> bool: + return value == b"yes" + + +def _parse_float(value: bytes) -> float: + return float(value.replace(b":", b".")) + + +def _parse_hex(value: bytes) -> int: + return int(value, 8) + + +STAT_TYPES: Dict[bytes, Callable[[bytes], Any]] = { + # General stats + b"version": bytes, + b"rusage_user": _parse_float, + b"rusage_system": _parse_float, + b"hash_is_expanding": _parse_bool_int, + b"slab_reassign_running": _parse_bool_int, + # Settings stats + b"inter": bytes, + b"growth_factor": float, + b"stat_key_prefix": bytes, + b"umask": _parse_hex, + b"detail_enabled": _parse_bool_int, + b"cas_enabled": _parse_bool_int, + b"auth_enabled_sasl": _parse_bool_string_is_yes, + b"maxconns_fast": _parse_bool_int, + b"slab_reassign": _parse_bool_int, + b"slab_automove": _parse_bool_int, +} + +# Common helper functions. + + +def check_key_helper( + key: Key, allow_unicode_keys: bool, key_prefix: bytes = b"" +) -> bytes: + """Checks key and add key_prefix.""" + if allow_unicode_keys: + if isinstance(key, str): + key = key.encode("utf8") + elif isinstance(key, str): + try: + key = key.encode("ascii") + except (UnicodeEncodeError, UnicodeDecodeError): + raise MemcacheIllegalInputError("Non-ASCII key: %r" % key) + + key = key_prefix + key + parts = key.split() + + if len(key) > 250: + raise MemcacheIllegalInputError("Key is too long: %r" % key) + # second statement catches leading or trailing whitespace + elif len(parts) > 1 or (parts and parts[0] != key): + raise MemcacheIllegalInputError("Key contains whitespace: %r" % key) + elif b"\00" in key: + raise MemcacheIllegalInputError("Key contains null: %r" % key) + + return key + + +def normalize_server_spec(server: ServerSpec) -> ServerSpec: + if isinstance(server, tuple): + return server + if not isinstance(server, str): + raise ValueError(f"Unsupported server specification: {server!r}") + if server.startswith("unix:"): + return server[5:] + if server.startswith("/"): + return server + if ":" not in server or server.endswith("]"): + host, port = server, 11211 + else: + parts = server.rsplit(":", 1) + host, port = parts[0], int(parts[1]) + if host.startswith("["): + host = host.strip("[]") + return (host, port) + + +class KeepaliveOpts: + """ + A configuration structure to define the socket keepalive. + + This structure must be passed to a client. The client will configure + its socket keepalive by using the elements of the structure. + + Args: + idle: The time (in seconds) the connection needs to remain idle + before TCP starts sending keepalive probes. Should be a positive + integer most greater than zero. + intvl: The time (in seconds) between individual keepalive probes. + Should be a positive integer most greater than zero. + cnt: The maximum number of keepalive probes TCP should send before + dropping the connection. Should be a positive integer most greater + than zero. + """ + + __slots__ = ("idle", "intvl", "cnt") + + def __init__(self, idle: int = 1, intvl: int = 1, cnt: int = 5) -> None: + if idle < 1: + raise ValueError("The idle parameter must be greater or equal to 1.") + self.idle = idle + if intvl < 1: + raise ValueError("The intvl parameter must be greater or equal to 1.") + self.intvl = intvl + if cnt < 1: + raise ValueError("The cnt parameter must be greater or equal to 1.") + self.cnt = cnt + + +class Client: + """ + A client for a single memcached server. + + *Server Connection* + + The ``server`` parameter controls how the client connects to the memcached + server. You can either use a (host, port) tuple for a TCP connection or a + string containing the path to a UNIX domain socket. + + The ``connect_timeout`` and ``timeout`` parameters can be used to set + socket timeout values. By default, timeouts are disabled. + + When the ``no_delay`` flag is set, the ``TCP_NODELAY`` socket option will + also be set. This only applies to TCP-based connections. + + Lastly, the ``socket_module`` allows you to specify an alternate socket + implementation (such as `gevent.socket`_). + + .. _gevent.socket: http://www.gevent.org/api/gevent.socket.html + + *Keys and Values* + + Keys must have a __str__() method which should return a str with no more + than 250 ASCII characters and no whitespace or control characters. Unicode + strings must be encoded (as UTF-8, for example) unless they consist only + of ASCII characters that are neither whitespace nor control characters. + + Values must have a __str__() method to convert themselves to a byte + string. Unicode objects can be a problem since str() on a Unicode object + will attempt to encode it as ASCII (which will fail if the value contains + code points larger than U+127). You can fix this with a serializer or by + just calling encode on the string (using UTF-8, for instance). + + If you intend to use anything but str as a value, it is a good idea to use + a serializer. The pymemcache.serde library has an already implemented + serializer which pickles and unpickles data. + + *Serialization and Deserialization* + + The constructor takes an optional object, the "serializer/deserializer" + ("serde"), which is responsible for both serialization and deserialization + of objects. That object must satisfy the serializer interface by providing + two methods: `serialize` and `deserialize`. `serialize` takes two + arguments, a key and a value, and returns a tuple of two elements, the + serialized value, and an integer in the range 0-65535 (the "flags"). + `deserialize` takes three parameters, a key, value, and flags, and returns + the deserialized value. + + Here is an example using JSON for non-str values: + + .. code-block:: python + + class JSONSerde(object): + def serialize(self, key, value): + if isinstance(value, str): + return value, 1 + return json.dumps(value), 2 + + def deserialize(self, key, value, flags): + if flags == 1: + return value + + if flags == 2: + return json.loads(value) + + raise Exception("Unknown flags for value: {1}".format(flags)) + + .. note:: + + Most write operations allow the caller to provide a ``flags`` value to + support advanced interaction with the server. This will **override** the + "flags" value returned by the serializer and should therefore only be + used when you have a complete understanding of how the value should be + serialized, stored, and deserialized. + + *Error Handling* + + All of the methods in this class that talk to memcached can throw one of + the following exceptions: + + * :class:`pymemcache.exceptions.MemcacheUnknownCommandError` + * :class:`pymemcache.exceptions.MemcacheClientError` + * :class:`pymemcache.exceptions.MemcacheServerError` + * :class:`pymemcache.exceptions.MemcacheUnknownError` + * :class:`pymemcache.exceptions.MemcacheUnexpectedCloseError` + * :class:`pymemcache.exceptions.MemcacheIllegalInputError` + * :class:`socket.timeout` + * :class:`socket.error` + + Instances of this class maintain a persistent connection to memcached + which is terminated when any of these exceptions are raised. The next + call to a method on the object will result in a new connection being made + to memcached. + """ + + def __init__( + self, + server: ServerSpec, + serde=None, + serializer=None, + deserializer=None, + connect_timeout: Optional[float] = None, + timeout: Optional[float] = None, + no_delay: bool = False, + ignore_exc: bool = False, + socket_module: ModuleType = socket, + socket_keepalive: Optional[KeepaliveOpts] = None, + key_prefix: bytes = b"", + default_noreply: bool = True, + allow_unicode_keys: bool = False, + encoding: str = "ascii", + tls_context: Optional[SSLContext] = None, + ): + """ + Constructor. + + Args: + server: tuple(hostname, port) or string containing a UNIX socket path. + serde: optional serializer object, see notes in the class docs. + serializer: deprecated serialization function + deserializer: deprecated deserialization function + connect_timeout: optional float, seconds to wait for a connection to + the memcached server. Defaults to "forever" (uses the underlying + default socket timeout, which can be very long). + timeout: optional float, seconds to wait for send or recv calls on + the socket connected to memcached. Defaults to "forever" (uses the + underlying default socket timeout, which can be very long). + no_delay: optional bool, set the TCP_NODELAY flag, which may help + with performance in some cases. Defaults to False. + ignore_exc: optional bool, True to cause the "get", "gets", + "get_many" and "gets_many" calls to treat any errors as cache + misses. Defaults to False. + socket_module: socket module to use, e.g. gevent.socket. Defaults to + the standard library's socket module. + socket_keepalive: Activate the socket keepalive feature by passing + a KeepaliveOpts structure in this parameter. Disabled by default + (None). This feature is only supported on Linux platforms. + key_prefix: Prefix of key. You can use this as namespace. Defaults + to b''. + default_noreply: bool, the default value for 'noreply' as passed to + store commands (except from cas, incr, and decr, which default to + False). + allow_unicode_keys: bool, support unicode (utf8) keys + encoding: optional str, controls data encoding (defaults to 'ascii'). + + Notes: + The constructor does not make a connection to memcached. The first + call to a method on the object will do that. + """ + self.server = normalize_server_spec(server) + self.serde = serde or LegacyWrappingSerde(serializer, deserializer) + self.connect_timeout = connect_timeout + self.timeout = timeout + self.no_delay = no_delay + self.ignore_exc = ignore_exc + self.socket_module = socket_module + self.socket_keepalive = socket_keepalive + user_system = platform.system() + if self.socket_keepalive is not None: + if user_system not in SOCKET_KEEPALIVE_SUPPORTED_SYSTEM: + raise SystemError( + "Pymemcache's socket keepalive mechanism doesn't " + "support your system ({user_system}). If " + "you see this message it mean that you tried to " + "configure your socket keepalive on an unsupported " + "system. To fix the problem pass `socket_" + "keepalive=False` or use a supported system. " + "Supported systems are: {systems}".format( + user_system=user_system, + systems=", ".join(sorted(SOCKET_KEEPALIVE_SUPPORTED_SYSTEM)), + ) + ) + if not isinstance(self.socket_keepalive, KeepaliveOpts): + raise ValueError( + "Unsupported keepalive options. If you see this message " + "it means that you passed an unsupported object within " + "the param `socket_keepalive`. To fix it " + "please instantiate and pass to socket_keepalive a " + "KeepaliveOpts object. That's the only supported type " + "of structure." + ) + self.sock: Optional[socket.socket] = None + if isinstance(key_prefix, str): + key_prefix = key_prefix.encode("ascii") + if not isinstance(key_prefix, bytes): + raise TypeError("key_prefix should be bytes.") + self.key_prefix = key_prefix + self.default_noreply = default_noreply + self.allow_unicode_keys = allow_unicode_keys + self.encoding = encoding + self.tls_context = tls_context + + def check_key(self, key: Key, key_prefix: bytes) -> bytes: + """Checks key and add key_prefix.""" + return check_key_helper( + key, allow_unicode_keys=self.allow_unicode_keys, key_prefix=key_prefix + ) + + def _connect(self) -> None: + self.close() + + s = self.socket_module + + if not isinstance(self.server, tuple): + sockaddr = self.server + sock = s.socket(s.AF_UNIX, s.SOCK_STREAM) + else: + sock = None + error = None + host, port = self.server + info = s.getaddrinfo(host, port, s.AF_UNSPEC, s.SOCK_STREAM, s.IPPROTO_TCP) + for family, socktype, proto, _, sockaddr in info: + try: + sock = s.socket(family, socktype, proto) + if self.no_delay: + sock.setsockopt(s.IPPROTO_TCP, s.TCP_NODELAY, 1) + if self.tls_context: + context = self.tls_context + sock = context.wrap_socket(sock, server_hostname=host) + except Exception as e: + error = e + if sock is not None: + sock.close() + sock = None + else: + break + + if error is not None: + raise error + + try: + sock.settimeout(self.connect_timeout) + if self.socket_keepalive is not None: + sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) + sock.setsockopt( + socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, self.socket_keepalive.idle + ) + sock.setsockopt( + socket.IPPROTO_TCP, + socket.TCP_KEEPINTVL, + self.socket_keepalive.intvl, + ) + sock.setsockopt( + socket.IPPROTO_TCP, socket.TCP_KEEPCNT, self.socket_keepalive.cnt + ) + sock.connect(sockaddr) + sock.settimeout(self.timeout) + except Exception: + sock.close() + raise + + self.sock = sock + + def close(self) -> None: + """Close the connection to memcached, if it is open. The next call to a + method that requires a connection will re-open it.""" + if self.sock is not None: + try: + self.sock.close() + except Exception: + pass + finally: + self.sock = None + + disconnect_all = close + + def set( + self, + key: Key, + value: Any, + expire: int = 0, + noreply: Optional[bool] = None, + flags: Optional[int] = None, + ) -> Optional[bool]: + """ + The memcached "set" command. + + Args: + key: str, see class docs for details. + value: str, see class docs for details. + expire: optional int, number of seconds until the item is expired + from the cache, or zero for no expiry (the default). + noreply: optional bool, True to not wait for the reply (defaults to + self.default_noreply). + flags: optional int, arbitrary bit field used for server-specific + flags + + Returns: + If no exception is raised, always returns True. If an exception is + raised, the set may or may not have occurred. If noreply is True, + then a successful return does not guarantee a successful set. + """ + if noreply is None: + noreply = self.default_noreply + # Optional because _store_cmd lookup in STORE_RESULTS_VALUE can return None in some cases. + # TODO: refactor to fix + return self._store_cmd(b"set", {key: value}, expire, noreply, flags=flags)[key] + + def set_many( + self, + values: Dict[Key, Any], + expire: int = 0, + noreply: Optional[bool] = None, + flags: Optional[int] = None, + ) -> List[Key]: + """ + A convenience function for setting multiple values. + + Args: + values: dict(str, str), a dict of keys and values, see class docs + for details. + expire: optional int, number of seconds until the item is expired + from the cache, or zero for no expiry (the default). + noreply: optional bool, True to not wait for the reply (defaults to + self.default_noreply). + flags: optional int, arbitrary bit field used for server-specific + flags + + Returns: + Returns a list of keys that failed to be inserted. + If noreply is True, always returns empty list. + """ + if noreply is None: + noreply = self.default_noreply + result = self._store_cmd(b"set", values, expire, noreply, flags=flags) + return [k for k, v in result.items() if not v] + + set_multi = set_many + + def add( + self, + key: Key, + value: Any, + expire: int = 0, + noreply: Optional[bool] = None, + flags: Optional[int] = None, + ) -> bool: + """ + The memcached "add" command. + + Args: + key: str, see class docs for details. + value: str, see class docs for details. + expire: optional int, number of seconds until the item is expired + from the cache, or zero for no expiry (the default). + noreply: optional bool, True to not wait for the reply (defaults to + self.default_noreply). + flags: optional int, arbitrary bit field used for server-specific + flags + + Returns: + If ``noreply`` is True (or if it is unset and ``self.default_noreply`` + is True), the return value is always True. Otherwise the return value + is True if the value was stored, and False if it was not (because + the key already existed). + """ + if noreply is None: + noreply = self.default_noreply + response = self._store_cmd(b"add", {key: value}, expire, noreply, flags=flags)[ + key + ] + # For typing, can only be None, if cas command + assert response is not None + return response + + def replace( + self, + key: Key, + value, + expire: int = 0, + noreply: Optional[bool] = None, + flags: Optional[int] = None, + ) -> bool: + """ + The memcached "replace" command. + + Args: + key: str, see class docs for details. + value: str, see class docs for details. + expire: optional int, number of seconds until the item is expired + from the cache, or zero for no expiry (the default). + noreply: optional bool, True to not wait for the reply (defaults to + self.default_noreply). + flags: optional int, arbitrary bit field used for server-specific + flags + + Returns: + If ``noreply`` is True (or if it is unset and ``self.default_noreply`` + is True), the return value is always True. Otherwise returns True if + the value was stored and False if it wasn't (because the key didn't + already exist). + """ + if noreply is None: + noreply = self.default_noreply + response = self._store_cmd( + b"replace", {key: value}, expire, noreply, flags=flags + )[key] + # for typing + assert response is not None + return response + + def append( + self, + key: Key, + value, + expire: int = 0, + noreply: Optional[bool] = None, + flags: Optional[int] = None, + ) -> bool: + """ + The memcached "append" command. + + Args: + key: str, see class docs for details. + value: str, see class docs for details. + expire: optional int, number of seconds until the item is expired + from the cache, or zero for no expiry (the default). + noreply: optional bool, True to not wait for the reply (defaults to + self.default_noreply). + flags: optional int, arbitrary bit field used for server-specific + flags + + Returns: + True. + """ + if noreply is None: + noreply = self.default_noreply + response = self._store_cmd( + b"append", {key: value}, expire, noreply, flags=flags + )[key] + # For typing + assert response is not None + return response + + def prepend( + self, + key, + value, + expire: int = 0, + noreply: Optional[bool] = None, + flags: Optional[int] = None, + ): + """ + The memcached "prepend" command. + + Args: + key: str, see class docs for details. + value: str, see class docs for details. + expire: optional int, number of seconds until the item is expired + from the cache, or zero for no expiry (the default). + noreply: optional bool, True to not wait for the reply (defaults to + self.default_noreply). + flags: optional int, arbitrary bit field used for server-specific + flags + + Returns: + True. + """ + if noreply is None: + noreply = self.default_noreply + return self._store_cmd(b"prepend", {key: value}, expire, noreply, flags=flags)[ + key + ] + + def cas( + self, + key, + value, + cas, + expire: int = 0, + noreply=False, + flags: Optional[int] = None, + ) -> Optional[bool]: + """ + The memcached "cas" command. + + Args: + key: str, see class docs for details. + value: str, see class docs for details. + cas: int or str that only contains the characters '0'-'9'. + expire: optional int, number of seconds until the item is expired + from the cache, or zero for no expiry (the default). + noreply: optional bool, False to wait for the reply (the default). + flags: optional int, arbitrary bit field used for server-specific + flags + + Returns: + If ``noreply`` is True (or if it is unset and ``self.default_noreply`` + is True), the return value is always True. Otherwise returns None if + the key didn't exist, False if it existed but had a different cas + value and True if it existed and was changed. + """ + cas = self._check_cas(cas) + return self._store_cmd( + b"cas", {key: value}, expire, noreply, flags=flags, cas=cas + )[key] + + def get(self, key: Key, default: Optional[Any] = None) -> Any: + """ + The memcached "get" command, but only for one key, as a convenience. + + Args: + key: str, see class docs for details. + default: value that will be returned if the key was not found. + + Returns: + The value for the key, or default if the key wasn't found. + """ + return self._fetch_cmd(b"get", [key], False, key_prefix=self.key_prefix).get( + key, default + ) + + def get_many(self, keys: Iterable[Key]) -> Dict[Key, Any]: + """ + The memcached "get" command. + + Args: + keys: list(str), see class docs for details. + + Returns: + A dict in which the keys are elements of the "keys" argument list + and the values are values from the cache. The dict may contain all, + some or none of the given keys. + """ + if not keys: + return {} + + return self._fetch_cmd(b"get", keys, False, key_prefix=self.key_prefix) + + get_multi = get_many + + def gets( + self, key: Key, default: Any = None, cas_default: Any = None + ) -> Tuple[Any, Any]: + """ + The memcached "gets" command for one key, as a convenience. + + Args: + key: str, see class docs for details. + default: value that will be returned if the key was not found. + cas_default: same behaviour as default argument. + + Returns: + A tuple of (value, cas) + or (default, cas_defaults) if the key was not found. + """ + defaults = (default, cas_default) + return self._fetch_cmd(b"gets", [key], True, key_prefix=self.key_prefix).get( + key, defaults + ) + + def gets_many(self, keys: Iterable[Key]) -> Dict[Key, Tuple[Any, Any]]: + """ + The memcached "gets" command. + + Args: + keys: list(str), see class docs for details. + + Returns: + A dict in which the keys are elements of the "keys" argument list and + the values are tuples of (value, cas) from the cache. The dict may + contain all, some or none of the given keys. + """ + if not keys: + return {} + + return self._fetch_cmd(b"gets", keys, True, key_prefix=self.key_prefix) + + def delete(self, key: Key, noreply: Optional[bool] = None) -> bool: + """ + The memcached "delete" command. + + Args: + key: str, see class docs for details. + noreply: optional bool, True to not wait for the reply (defaults to + self.default_noreply). + + Returns: + If ``noreply`` is True (or if it is unset and ``self.default_noreply`` + is True), the return value is always True. Otherwise returns True if + the key was deleted, and False if it wasn't found. + """ + if noreply is None: + noreply = self.default_noreply + cmd = b"delete " + self.check_key(key, self.key_prefix) + if noreply: + cmd += b" noreply" + cmd += b"\r\n" + results = self._misc_cmd([cmd], b"delete", noreply) + if noreply: + return True + return results[0] == b"DELETED" + + def delete_many(self, keys: Iterable[Key], noreply: Optional[bool] = None) -> bool: + """ + A convenience function to delete multiple keys. + + Args: + keys: list(str), the list of keys to delete. + noreply: optional bool, True to not wait for the reply (defaults to + self.default_noreply). + + Returns: + True. If an exception is raised then all, some or none of the keys + may have been deleted. Otherwise all the keys have been sent to + memcache for deletion and if noreply is False, they have been + acknowledged by memcache. + """ + if not keys: + return True + + if noreply is None: + noreply = self.default_noreply + + cmds = [] + for key in keys: + cmds.append( + b"delete " + + self.check_key(key, self.key_prefix) + + (b" noreply" if noreply else b"") + + b"\r\n" + ) + self._misc_cmd(cmds, b"delete", noreply) + return True + + delete_multi = delete_many + + def incr( + self, key: Key, value: int, noreply: Optional[bool] = False + ) -> Optional[int]: + """ + The memcached "incr" command. + + Args: + key: str, see class docs for details. + value: int, the amount by which to increment the value. + noreply: optional bool, False to wait for the reply (the default). + + Returns: + If noreply is True, always returns None. Otherwise returns the new + value of the key, or None if the key wasn't found. + """ + key = self.check_key(key, self.key_prefix) + val = self._check_integer(value, "value") + cmd = b"incr " + key + b" " + val + if noreply: + cmd += b" noreply" + cmd += b"\r\n" + results = self._misc_cmd([cmd], b"incr", noreply) + if noreply: + return None + if results[0] == b"NOT_FOUND": + return None + return int(results[0]) + + def decr( + self, key: Key, value: int, noreply: Optional[bool] = False + ) -> Optional[int]: + """ + The memcached "decr" command. + + Args: + key: str, see class docs for details. + value: int, the amount by which to decrement the value. + noreply: optional bool, False to wait for the reply (the default). + + Returns: + If noreply is True, always returns None. Otherwise returns the new + value of the key, or None if the key wasn't found. + """ + key = self.check_key(key, self.key_prefix) + val = self._check_integer(value, "value") + cmd = b"decr " + key + b" " + val + if noreply: + cmd += b" noreply" + cmd += b"\r\n" + results = self._misc_cmd([cmd], b"decr", noreply) + if noreply: + return None + if results[0] == b"NOT_FOUND": + return None + return int(results[0]) + + def touch(self, key: Key, expire: int = 0, noreply: Optional[bool] = None) -> bool: + """ + The memcached "touch" command. + + Args: + key: str, see class docs for details. + expire: optional int, number of seconds until the item is expired + from the cache, or zero for no expiry (the default). + noreply: optional bool, True to not wait for the reply (defaults to + self.default_noreply). + + Returns: + True if the expiration time was updated, False if the key wasn't + found. + """ + if noreply is None: + noreply = self.default_noreply + key = self.check_key(key, self.key_prefix) + expire_bytes = self._check_integer(expire, "expire") + cmd = b"touch " + key + b" " + expire_bytes + if noreply: + cmd += b" noreply" + cmd += b"\r\n" + results = self._misc_cmd([cmd], b"touch", noreply) + if noreply: + return True + return results[0] == b"TOUCHED" + + def stats(self, *args): + """ + The memcached "stats" command. + + The returned keys depend on what the "stats" command returns. + A best effort is made to convert values to appropriate Python + types, defaulting to strings when a conversion cannot be made. + + Args: + *arg: extra string arguments to the "stats" command. See the + memcached protocol documentation for more information. + + Returns: + A dict of the returned stats. + """ + result = self._fetch_cmd(b"stats", args, False) + + for key, value in result.items(): + converter = STAT_TYPES.get(key, int) + try: + result[key] = converter(value) + except Exception: + pass + + return result + + def cache_memlimit(self, memlimit) -> bool: + """ + The memcached "cache_memlimit" command. + + Args: + memlimit: int, the number of megabytes to set as the new cache memory + limit. + + Returns: + If no exception is raised, always returns True. + """ + memlimit = self._check_integer(memlimit, "memlimit") + self._fetch_cmd(b"cache_memlimit", [memlimit], False) + return True + + def version(self) -> bytes: + """ + The memcached "version" command. + + Returns: + A string of the memcached version. + """ + cmd = b"version\r\n" + results = self._misc_cmd([cmd], b"version", False) + before, _, after = results[0].partition(b" ") + + if before != b"VERSION": + raise MemcacheUnknownError(f"Received unexpected response: {results[0]!r}") + return after + + def raw_command( + self, command: Union[str, bytes], end_tokens: Union[str, bytes] = "\r\n" + ) -> bytes: + """ + Sends an arbitrary command to the server and parses the response until a + specified token is encountered. + + Args: + command: str|bytes: The command to send. + end_tokens: str|bytes: The token expected at the end of the + response. If the `end_token` is not found, the client will wait + until the timeout specified in the constructor. + + Returns: + The response from the server, with the `end_token` removed. + """ + encoding = "utf8" if self.allow_unicode_keys else "ascii" + command = command.encode(encoding) if isinstance(command, str) else command + end_tokens = ( + end_tokens.encode(encoding) if isinstance(end_tokens, str) else end_tokens + ) + return self._misc_cmd([b"" + command + b"\r\n"], command, False, end_tokens)[0] + + def flush_all(self, delay: int = 0, noreply: Optional[bool] = None) -> bool: + """ + The memcached "flush_all" command. + + Args: + delay: optional int, the number of seconds to wait before flushing, + or zero to flush immediately (the default). + noreply: optional bool, True to not wait for the reply (defaults to + self.default_noreply). + + Returns: + True. + """ + if noreply is None: + noreply = self.default_noreply + delay_bytes = self._check_integer(delay, "delay") + cmd = b"flush_all " + delay_bytes + if noreply: + cmd += b" noreply" + cmd += b"\r\n" + results = self._misc_cmd([cmd], b"flush_all", noreply) + if noreply: + return True + return results[0] == b"OK" + + def quit(self) -> None: + """ + The memcached "quit" command. + + This will close the connection with memcached. Calling any other + method on this object will re-open the connection, so this object can + be re-used after quit. + """ + cmd = b"quit\r\n" + self._misc_cmd([cmd], b"quit", True) + self.close() + + def shutdown(self, graceful: bool = False) -> None: + """ + The memcached "shutdown" command. + + This will request shutdown and eventual termination of the server, + optionally preceded by a graceful stop of memcached's internal state + machine. Note that the server needs to have been started with the + shutdown protocol command enabled with the --enable-shutdown flag. + + Args: + graceful: optional bool, True to request a graceful shutdown with + SIGUSR1 (defaults to False, i.e. SIGINT shutdown). + """ + cmd = b"shutdown" + if graceful: + cmd += b" graceful" + cmd += b"\r\n" + + # The shutdown command raises a server-side error if the shutdown + # protocol command is not enabled. Otherwise, a successful shutdown + # is expected to close the remote end of the transport. + try: + self._misc_cmd([cmd], b"shutdown", False) + except MemcacheUnexpectedCloseError: + pass + + def _raise_errors(self, line: bytes, name: bytes) -> None: + if line.startswith(b"ERROR"): + raise MemcacheUnknownCommandError(name) + + if line.startswith(b"CLIENT_ERROR"): + error = line[line.find(b" ") + 1 :] + raise MemcacheClientError(error) + + if line.startswith(b"SERVER_ERROR"): + error = line[line.find(b" ") + 1 :] + raise MemcacheServerError(error) + + def _check_integer(self, value: int, name: str) -> bytes: + """Check that a value is an integer and encode it as a binary string""" + if not isinstance(value, int): + raise MemcacheIllegalInputError( + f"{name} must be integer, got bad value: {value!r}" + ) + + return str(value).encode(self.encoding) + + def _check_cas(self, cas: Union[int, str, bytes]) -> bytes: + """Check that a value is a valid input for 'cas' -- either an int or a + string containing only 0-9 + + The value will be (re)encoded so that we can accept strings or bytes. + """ + # convert non-binary values to binary + if isinstance(cas, (int, str)): + try: + cas = str(cas).encode(self.encoding) + except UnicodeEncodeError: + raise MemcacheIllegalInputError("non-ASCII cas value: %r" % cas) + elif not isinstance(cas, bytes): + raise MemcacheIllegalInputError( + "cas must be integer, string, or bytes, got bad value: %r" % cas + ) + + if not cas.isdigit(): + raise MemcacheIllegalInputError( + "cas must only contain values in 0-9, got bad value: %r" % cas + ) + + return cas + + def _extract_value( + self, + expect_cas: bool, + line: bytes, + buf: bytes, + remapped_keys: Dict[bytes, Key], + prefixed_keys: List[bytes], + ) -> Tuple[Key, Union[Any, Tuple[Any, bytes]], bytes]: + """ + This function is abstracted from _fetch_cmd to support different ways + of value extraction. In order to use this feature, _extract_value needs + to be overridden in the subclass. + """ + if expect_cas: + _, key, flags, size, cas = line.split() + else: + try: + _, key, flags, size = line.split() + except Exception as e: + raise ValueError(f"Unable to parse line {line!r}: {e}") + + value = None + try: + # For typing + assert self.sock is not None + + buf, value = _readvalue(self.sock, buf, int(size)) + except MemcacheUnexpectedCloseError: + self.close() + raise + original_key = remapped_keys[key] + value = self.serde.deserialize(original_key, value, int(flags)) + + if expect_cas: + return original_key, (value, cas), buf + else: + return original_key, value, buf + + def _fetch_cmd( + self, + name: bytes, + keys: Iterable[Key], + expect_cas: bool, + key_prefix: bytes = b"", + ) -> Dict[Key, Any]: + prefixed_keys = [self.check_key(k, key_prefix=key_prefix) for k in keys] + remapped_keys = dict(zip(prefixed_keys, keys)) + + # It is important for all keys to be listed in their original order. + cmd = name + if prefixed_keys: + cmd += b" " + b" ".join(prefixed_keys) + cmd += b"\r\n" + + try: + if self.sock is None: + self._connect() + + # For typing + assert self.sock is not None + + self.sock.sendall(cmd) + + buf = b"" + line = None + result: Dict[Key, Any] = {} + while True: + try: + buf, line = _readline(self.sock, buf) + except MemcacheUnexpectedCloseError: + self.close() + raise + self._raise_errors(line, name) + if line == b"END" or line == b"OK": + return result + elif line.startswith(b"VALUE"): + key, value, buf = self._extract_value( + expect_cas, line, buf, remapped_keys, prefixed_keys + ) + result[key] = value + elif name == b"stats" and line.startswith(b"STAT"): + key_value = line.split() + result[key_value[1]] = key_value[2] if len(key_value) > 2 else b"" + elif name == b"stats" and line.startswith(b"ITEM"): + # For 'stats cachedump' commands + key_value = line.split() + result[key_value[1]] = b" ".join(key_value[2:]) + else: + raise MemcacheUnknownError(line[:32]) + except Exception: + self.close() + if self.ignore_exc: + return {} + raise + + def _store_cmd( + self, + name: bytes, + values: Dict[Key, Any], + expire: int, + noreply: bool, + flags: Optional[int] = None, + cas: Optional[bytes] = None, + ) -> Dict[Key, Optional[bool]]: + cmds = [] + keys = [] + + extra = b"" + if cas is not None: + extra += b" " + cas + if noreply: + extra += b" noreply" + expire_bytes = self._check_integer(expire, "expire") + + for key, data in values.items(): + # must be able to reliably map responses back to the original order + keys.append(key) + + key = self.check_key(key, self.key_prefix) + data, data_flags = self.serde.serialize(key, data) + + # If 'flags' was explicitly provided, it overrides the value + # returned by the serializer. + if flags is not None: + data_flags = flags + + if not isinstance(data, bytes): + try: + data = str(data).encode(self.encoding) + except UnicodeEncodeError as e: + raise MemcacheIllegalInputError( + "Data values must be binary-safe: %s" % e + ) + + cmds.append( + name + + b" " + + key + + b" " + + str(data_flags).encode(self.encoding) + + b" " + + expire_bytes + + b" " + + str(len(data)).encode(self.encoding) + + extra + + b"\r\n" + + data + + b"\r\n" + ) + + if self.sock is None: + self._connect() + + # For typing + assert self.sock is not None + + try: + self.sock.sendall(b"".join(cmds)) + if noreply: + return {k: True for k in keys} + + results = {} + buf = b"" + line = None + for key in keys: + try: + buf, line = _readline(self.sock, buf) + except MemcacheUnexpectedCloseError: + self.close() + raise + self._raise_errors(line, name) + + if line in VALID_STORE_RESULTS[name]: + results[key] = STORE_RESULTS_VALUE[line] + else: + raise MemcacheUnknownError(line[:32]) + return results + except Exception: + self.close() + raise + + def _misc_cmd( + self, + cmds: Iterable[bytes], + cmd_name: bytes, + noreply: Optional[bool], + end_tokens=None, + ) -> List[bytes]: + + # If no end_tokens have been given, just assume standard memcached + # operations, which end in "\r\n", use regular code for that. + _reader: Callable[[socket.socket, bytes], Tuple[bytes, bytes]] + if end_tokens: + _reader = partial(_readsegment, end_tokens=end_tokens) + else: + _reader = _readline + + if self.sock is None: + self._connect() + + # For typing + assert self.sock is not None + + try: + self.sock.sendall(b"".join(cmds)) + + if noreply: + return [] + + results = [] + buf = b"" + line = None + for cmd in cmds: + try: + buf, line = _reader(self.sock, buf) + except MemcacheUnexpectedCloseError: + self.close() + raise + self._raise_errors(line, cmd_name) + results.append(line) + return results + + except Exception: + self.close() + raise + + def __setitem__(self, key: Key, value): + self.set(key, value, noreply=True) + + def __getitem__(self, key): + value = self.get(key) + if value is None: + raise KeyError + return value + + def __delitem__(self, key): + self.delete(key, noreply=True) + + +class PooledClient: + """A thread-safe pool of clients (with the same client api). + + Args: + max_pool_size: maximum pool size to use (going above this amount + triggers a runtime error), by default this is 2147483648L + when not provided (or none). + pool_idle_timeout: pooled connections are discarded if they have been + unused for this many seconds. A value of 0 indicates + that pooled connections are never discarded. + lock_generator: a callback/type that takes no arguments that will + be called to create a lock or semaphore that can + protect the pool from concurrent access (for example a + eventlet lock or semaphore could be used instead) + + Further arguments are interpreted as for :py:class:`.Client` constructor. + + Note: if `serde` is given, the same object will be used for *all* clients + in the pool. Your serde object must therefore be thread-safe. + """ + + #: :class:`Client` class used to create new clients + client_class = Client + + def __init__( + self, + server: ServerSpec, + serde=None, + serializer=None, + deserializer=None, + connect_timeout=None, + timeout=None, + no_delay=False, + ignore_exc=False, + socket_module=socket, + socket_keepalive=None, + key_prefix=b"", + max_pool_size=None, + pool_idle_timeout=0, + lock_generator=None, + default_noreply: bool = True, + allow_unicode_keys=False, + encoding="ascii", + tls_context=None, + ): + self.server = normalize_server_spec(server) + self.serde = serde or LegacyWrappingSerde(serializer, deserializer) + self.connect_timeout = connect_timeout + self.timeout = timeout + self.no_delay = no_delay + self.ignore_exc = ignore_exc + self.socket_module = socket_module + self.socket_keepalive = socket_keepalive + self.default_noreply = default_noreply + self.allow_unicode_keys = allow_unicode_keys + if isinstance(key_prefix, str): + key_prefix = key_prefix.encode("ascii") + if not isinstance(key_prefix, bytes): + raise TypeError("key_prefix should be bytes.") + self.key_prefix = key_prefix + self.client_pool = pool.ObjectPool( + self._create_client, + after_remove=lambda client: client.close(), + max_size=max_pool_size, + idle_timeout=pool_idle_timeout, + lock_generator=lock_generator, + ) + self.encoding = encoding + self.tls_context = tls_context + + def check_key(self, key: Key) -> bytes: + """Checks key and add key_prefix.""" + return check_key_helper( + key, allow_unicode_keys=self.allow_unicode_keys, key_prefix=self.key_prefix + ) + + def _create_client(self) -> Client: + return self.client_class( + self.server, + serde=self.serde, + connect_timeout=self.connect_timeout, + timeout=self.timeout, + no_delay=self.no_delay, + # We need to know when it fails *always* so that we + # can remove/destroy it from the pool... + ignore_exc=False, + socket_module=self.socket_module, + socket_keepalive=self.socket_keepalive, + key_prefix=self.key_prefix, + default_noreply=self.default_noreply, + allow_unicode_keys=self.allow_unicode_keys, + tls_context=self.tls_context, + ) + + def close(self) -> None: + self.client_pool.clear() + + disconnect_all = close + + def set( + self, + key, + value, + expire: int = 0, + noreply: Optional[bool] = None, + flags: Optional[int] = None, + ): + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + return client.set(key, value, expire=expire, noreply=noreply, flags=flags) + + def set_many( + self, + values, + expire: int = 0, + noreply: Optional[bool] = None, + flags: Optional[int] = None, + ): + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + return client.set_many(values, expire=expire, noreply=noreply, flags=flags) + + set_multi = set_many + + def replace( + self, + key, + value, + expire: int = 0, + noreply: Optional[bool] = None, + flags: Optional[int] = None, + ): + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + return client.replace( + key, value, expire=expire, noreply=noreply, flags=flags + ) + + def append( + self, + key, + value, + expire: int = 0, + noreply: Optional[bool] = None, + flags: Optional[int] = None, + ): + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + return client.append( + key, value, expire=expire, noreply=noreply, flags=flags + ) + + def prepend( + self, + key, + value, + expire: int = 0, + noreply: Optional[bool] = None, + flags: Optional[int] = None, + ): + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + return client.prepend( + key, value, expire=expire, noreply=noreply, flags=flags + ) + + def cas( + self, + key, + value, + cas, + expire: int = 0, + noreply=False, + flags: Optional[int] = None, + ): + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + return client.cas( + key, value, cas, expire=expire, noreply=noreply, flags=flags + ) + + def get(self, key: Key, default: Any = None) -> Any: + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + try: + return client.get(key, default) + except Exception: + if self.ignore_exc: + return default + else: + raise + + def get_many(self, keys: Iterable[Key]) -> Dict[Key, Any]: + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + try: + return client.get_many(keys) + except Exception: + if self.ignore_exc: + return {} + else: + raise + + get_multi = get_many + + def gets(self, key: Key) -> Tuple[Any, Any]: + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + try: + return client.gets(key) + except Exception: + if self.ignore_exc: + return (None, None) + else: + raise + + def gets_many(self, keys: Iterable[Key]) -> Dict[Key, Tuple[Any, Any]]: + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + try: + return client.gets_many(keys) + except Exception: + if self.ignore_exc: + return {} + else: + raise + + def delete(self, key: Key, noreply: Optional[bool] = None) -> bool: + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + return client.delete(key, noreply=noreply) + + def delete_many(self, keys: Iterable[Key], noreply: Optional[bool] = None) -> bool: + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + return client.delete_many(keys, noreply=noreply) + + delete_multi = delete_many + + def add( + self, + key: Key, + value, + expire: int = 0, + noreply: Optional[bool] = None, + flags: Optional[int] = None, + ): + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + return client.add(key, value, expire=expire, noreply=noreply, flags=flags) + + def incr(self, key: Key, value, noreply=False): + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + return client.incr(key, value, noreply=noreply) + + def decr(self, key: Key, value, noreply=False): + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + return client.decr(key, value, noreply=noreply) + + def touch(self, key: Key, expire: int = 0, noreply=None): + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + return client.touch(key, expire=expire, noreply=noreply) + + def stats(self, *args): + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + try: + return client.stats(*args) + except Exception: + if self.ignore_exc: + return {} + else: + raise + + def version(self) -> bytes: + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + return client.version() + + def flush_all(self, delay=0, noreply=None) -> bool: + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + return client.flush_all(delay=delay, noreply=noreply) + + def quit(self) -> None: + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + try: + client.quit() + finally: + self.client_pool.destroy(client) + + def shutdown(self, graceful: bool = False) -> None: + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + client.shutdown(graceful) + + def raw_command(self, command, end_tokens=b"\r\n"): + with self.client_pool.get_and_release(destroy_on_fail=True) as client: + return client.raw_command(command, end_tokens) + + def __setitem__(self, key: Key, value): + self.set(key, value, noreply=True) + + def __getitem__(self, key): + value = self.get(key) + if value is None: + raise KeyError + return value + + def __delitem__(self, key): + self.delete(key, noreply=True) + + +def _readline(sock: socket.socket, buf: bytes) -> Tuple[bytes, bytes]: + """Read line of text from the socket. + + Read a line of text (delimited by "\r\n") from the socket, and + return that line along with any trailing characters read from the + socket. + + Args: + sock: Socket object, should be connected. + buf: Bytes, zero or more characters, returned from an earlier + call to _readline or _readvalue (pass an empty byte string on the + first call). + + Returns: + A tuple of (buf, line) where line is the full line read from the + socket (minus the "\r\n" characters) and buf is any trailing + characters read after the "\r\n" was found (which may be an empty + byte string). + + """ + chunks: List[bytes] = [] + last_char = b"" + + while True: + # We're reading in chunks, so "\r\n" could appear in one chunk, + # or across the boundary of two chunks, so we check for both + # cases. + + # This case must appear first, since the buffer could have + # later \r\n characters in it and we want to get the first \r\n. + if last_char == b"\r" and buf[0:1] == b"\n": + # Strip the last character from the last chunk. + chunks[-1] = chunks[-1][:-1] + return buf[1:], b"".join(chunks) + else: + token_pos = buf.find(b"\r\n") + if token_pos != -1: + # Note: 2 == len(b"\r\n") + before, after = buf[:token_pos], buf[token_pos + 2 :] + chunks.append(before) + return after, b"".join(chunks) + + if buf: + chunks.append(buf) + last_char = buf[-1:] + + buf = _recv(sock, RECV_SIZE) + if not buf: + raise MemcacheUnexpectedCloseError() + + +def _readvalue(sock: socket.socket, buf: bytes, size: int): + """Read specified amount of bytes from the socket. + + Read size bytes, followed by the "\r\n" characters, from the socket, + and return those bytes and any trailing bytes read after the "\r\n". + + Args: + sock: Socket object, should be connected. + buf: String, zero or more characters, returned from an earlier + call to _readline or _readvalue (pass an empty string on the + first call). + size: Integer, number of bytes to read from the socket. + + Returns: + A tuple of (buf, value) where value is the bytes read from the + socket (there will be exactly size bytes) and buf is trailing + characters read after the "\r\n" following the bytes (but not + including the \r\n). + + """ + chunks = [] + rlen = size + 2 + while rlen - len(buf) > 0: + if buf: + rlen -= len(buf) + chunks.append(buf) + buf = _recv(sock, RECV_SIZE) + if not buf: + raise MemcacheUnexpectedCloseError() + + # Now we need to remove the \r\n from the end. There are two cases we care + # about: the \r\n is all in the last buffer, or only the \n is in the last + # buffer, and we need to remove the \r from the penultimate buffer. + + if rlen == 1: + # replace the last chunk with the same string minus the last character, + # which is always '\r' in this case. + chunks[-1] = chunks[-1][:-1] + else: + # Just remove the "\r\n" from the latest chunk + chunks.append(buf[: rlen - 2]) + + return buf[rlen:], b"".join(chunks) + + +def _readsegment( + sock: socket.socket, buf: bytes, end_tokens: bytes +) -> Tuple[bytes, bytes]: + """Read a segment from the socket. + + Read a segment from the socket, up to the first end_token sub-string/bytes, + and return that segment. + + Args: + sock: Socket object, should be connected. + buf: bytes, zero or more bytes, returned from an earlier + call to _readline, _readsegment or _readvalue (pass an empty + byte-string on the first call). + end_tokens: bytes, indicates the end of the segment, generally this is + b"\\r\\n" for memcached. + + Returns: + A tuple of (buf, line) where line is the full line read from the + socket (minus the end_tokens bytes) and buf is any trailing + characters read after the end_tokens was found (which may be an empty + bytes object). + + """ + result = bytes() + + while True: + + tokens_pos = buf.find(end_tokens) + if tokens_pos != -1: + before, after = buf[:tokens_pos], buf[tokens_pos + len(end_tokens) :] + result += before + return after, result + + buf = _recv(sock, RECV_SIZE) + if not buf: + raise MemcacheUnexpectedCloseError() + + +def _recv(sock: socket.socket, size: int) -> bytes: + """sock.recv() with retry on EINTR""" + while True: + try: + return sock.recv(size) + except OSError as e: + if e.errno != errno.EINTR: + raise diff --git a/lib/python3.11/site-packages/pymemcache/client/hash.py b/lib/python3.11/site-packages/pymemcache/client/hash.py new file mode 100644 index 00000000..e56517f8 --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/client/hash.py @@ -0,0 +1,447 @@ +import collections +import socket +import time +import logging + +from pymemcache.client.base import ( + Client, + PooledClient, + check_key_helper, + normalize_server_spec, +) +from pymemcache.client.rendezvous import RendezvousHash +from pymemcache.exceptions import MemcacheError + +logger = logging.getLogger(__name__) + + +class HashClient: + """ + A client for communicating with a cluster of memcached servers + """ + + #: :class:`Client` class used to create new clients + client_class = Client + + def __init__( + self, + servers, + hasher=RendezvousHash, + serde=None, + serializer=None, + deserializer=None, + connect_timeout=None, + timeout=None, + no_delay=False, + socket_module=socket, + socket_keepalive=None, + key_prefix=b"", + max_pool_size=None, + pool_idle_timeout=0, + lock_generator=None, + retry_attempts=2, + retry_timeout=1, + dead_timeout=60, + use_pooling=False, + ignore_exc=False, + allow_unicode_keys=False, + default_noreply=True, + encoding="ascii", + tls_context=None, + ): + """ + Constructor. + + Args: + servers: list() of tuple(hostname, port) or string containing a UNIX + socket path. + hasher: optional class three functions ``get_node``, ``add_node``, + and ``remove_node`` + defaults to Rendezvous (HRW) hash. + + use_pooling: use py:class:`.PooledClient` as the default underlying + class. ``max_pool_size`` and ``lock_generator`` can + be used with this. default: False + + retry_attempts: Amount of times a client should be tried before it + is marked dead and removed from the pool. + retry_timeout (float): Time in seconds that should pass between retry + attempts. + dead_timeout (float): Time in seconds before attempting to add a node + back in the pool. + encoding: optional str, controls data encoding (defaults to 'ascii'). + + Further arguments are interpreted as for :py:class:`.Client` + constructor. + """ + self.clients = {} + self.retry_attempts = retry_attempts + self.retry_timeout = retry_timeout + self.dead_timeout = dead_timeout + self.use_pooling = use_pooling + self.key_prefix = key_prefix + self.ignore_exc = ignore_exc + self.allow_unicode_keys = allow_unicode_keys + self._failed_clients = {} + self._dead_clients = {} + self._last_dead_check_time = time.time() + + self.hasher = hasher() + + self.default_kwargs = { + "connect_timeout": connect_timeout, + "timeout": timeout, + "no_delay": no_delay, + "socket_module": socket_module, + "socket_keepalive": socket_keepalive, + "key_prefix": key_prefix, + "serde": serde, + "serializer": serializer, + "deserializer": deserializer, + "allow_unicode_keys": allow_unicode_keys, + "default_noreply": default_noreply, + "encoding": encoding, + "tls_context": tls_context, + } + + if use_pooling is True: + self.default_kwargs.update( + { + "max_pool_size": max_pool_size, + "pool_idle_timeout": pool_idle_timeout, + "lock_generator": lock_generator, + } + ) + + for server in servers: + self.add_server(normalize_server_spec(server)) + self.encoding = encoding + self.tls_context = tls_context + + def _make_client_key(self, server): + if isinstance(server, (list, tuple)) and len(server) == 2: + return "%s:%s" % server + return server + + def add_server(self, server, port=None) -> None: + # To maintain backward compatibility, if a port is provided, assume + # that server wasn't provided as a (host, port) tuple. + if port is not None: + if not isinstance(server, str): + raise TypeError("Server must be a string when passing port.") + server = (server, port) + + _class = PooledClient if self.use_pooling else self.client_class + client = _class(server, **self.default_kwargs) + if self.use_pooling: + client.client_class = self.client_class + + key = self._make_client_key(server) + self.clients[key] = client + self.hasher.add_node(key) + + def remove_server(self, server, port=None) -> None: + # To maintain backward compatibility, if a port is provided, assume + # that server wasn't provided as a (host, port) tuple. + if port is not None: + if not isinstance(server, str): + raise TypeError("Server must be a string when passing port.") + server = (server, port) + + key = self._make_client_key(server) + dead_time = time.time() + self._failed_clients.pop(server) + self._dead_clients[server] = dead_time + self.hasher.remove_node(key) + + def _retry_dead(self) -> None: + current_time = time.time() + ldc = self._last_dead_check_time + # We have reached the retry timeout + if current_time - ldc > self.dead_timeout: + candidates = [] + for server, dead_time in self._dead_clients.items(): + if current_time - dead_time > self.dead_timeout: + candidates.append(server) + for server in candidates: + logger.debug("bringing server back into rotation %s", server) + self.add_server(server) + del self._dead_clients[server] + self._last_dead_check_time = current_time + + def _get_client(self, key): + check_key_helper(key, self.allow_unicode_keys, self.key_prefix) + if self._dead_clients: + self._retry_dead() + + server = self.hasher.get_node(key) + # We've ran out of servers to try + if server is None: + if self.ignore_exc is True: + return + raise MemcacheError("All servers seem to be down right now") + + return self.clients[server] + + def _safely_run_func(self, client, func, default_val, *args, **kwargs): + try: + if client.server in self._failed_clients: + # This server is currently failing, lets check if it is in + # retry or marked as dead + failed_metadata = self._failed_clients[client.server] + + # we haven't tried our max amount yet, if it has been enough + # time lets just retry using it + if failed_metadata["attempts"] < self.retry_attempts: + failed_time = failed_metadata["failed_time"] + if time.time() - failed_time > self.retry_timeout: + logger.debug("retrying failed server: %s", client.server) + result = func(*args, **kwargs) + # we were successful, lets remove it from the failed + # clients + self._failed_clients.pop(client.server) + return result + return default_val + else: + # We've reached our max retry attempts, we need to mark + # the sever as dead + logger.debug("marking server as dead: %s", client.server) + self.remove_server(client.server) + + result = func(*args, **kwargs) + return result + + # Connecting to the server fail, we should enter + # retry mode + except OSError: + self._mark_failed_server(client.server) + + # if we haven't enabled ignore_exc, don't move on gracefully, just + # raise the exception + if not self.ignore_exc: + raise + + return default_val + except Exception: + # any exceptions that aren't socket.error we need to handle + # gracefully as well + if not self.ignore_exc: + raise + + return default_val + + def _safely_run_set_many(self, client, values, *args, **kwargs): + failed = [] + succeeded = [] + try: + if client.server in self._failed_clients: + # This server is currently failing, lets check if it is in + # retry or marked as dead + failed_metadata = self._failed_clients[client.server] + + # we haven't tried our max amount yet, if it has been enough + # time lets just retry using it + if failed_metadata["attempts"] < self.retry_attempts: + failed_time = failed_metadata["failed_time"] + if time.time() - failed_time > self.retry_timeout: + logger.debug("retrying failed server: %s", client.server) + succeeded, failed, err = self._set_many( + client, values, *args, **kwargs + ) + if err is not None: + raise err + # we were successful, lets remove it from the failed + # clients + self._failed_clients.pop(client.server) + return failed + return values.keys() + else: + # We've reached our max retry attempts, we need to mark + # the sever as dead + logger.debug("marking server as dead: %s", client.server) + self.remove_server(client.server) + + succeeded, failed, err = self._set_many(client, values, *args, **kwargs) + if err is not None: + raise err + + return failed + + # Connecting to the server fail, we should enter + # retry mode + except OSError: + self._mark_failed_server(client.server) + + # if we haven't enabled ignore_exc, don't move on gracefully, just + # raise the exception + if not self.ignore_exc: + raise + + return list(set(values.keys()) - set(succeeded)) + except Exception: + # any exceptions that aren't socket.error we need to handle + # gracefully as well + if not self.ignore_exc: + raise + + return list(set(values.keys()) - set(succeeded)) + + def _mark_failed_server(self, server): + # This client has never failed, lets mark it for failure + if server not in self._failed_clients and self.retry_attempts > 0: + self._failed_clients[server] = { + "failed_time": time.time(), + "attempts": 0, + } + # We aren't allowing any retries, we should mark the server as + # dead immediately + elif server not in self._failed_clients and self.retry_attempts <= 0: + self._failed_clients[server] = { + "failed_time": time.time(), + "attempts": 0, + } + logger.debug("marking server as dead %s", server) + self.remove_server(server) + # This client has failed previously, we need to update the metadata + # to reflect that we have attempted it again + else: + failed_metadata = self._failed_clients[server] + failed_metadata["attempts"] += 1 + failed_metadata["failed_time"] = time.time() + self._failed_clients[server] = failed_metadata + + def _run_cmd(self, cmd, key, default_val, *args, **kwargs): + client = self._get_client(key) + + if client is None: + return default_val + + func = getattr(client, cmd) + args = list(args) + args.insert(0, key) + return self._safely_run_func(client, func, default_val, *args, **kwargs) + + def _set_many(self, client, values, *args, **kwargs): + failed = [] + succeeded = [] + + try: + failed = client.set_many(values, *args, **kwargs) + except Exception as e: + if not self.ignore_exc: + return succeeded, failed, e + + succeeded = [key for key in values if key not in failed] + return succeeded, failed, None + + def close(self): + for client in self.clients.values(): + self._safely_run_func(client, client.close, False) + + disconnect_all = close + + def set(self, key, *args, **kwargs): + return self._run_cmd("set", key, False, *args, **kwargs) + + def get(self, key, default=None, **kwargs): + return self._run_cmd("get", key, default, default=default, **kwargs) + + def incr(self, key, *args, **kwargs): + return self._run_cmd("incr", key, False, *args, **kwargs) + + def decr(self, key, *args, **kwargs): + return self._run_cmd("decr", key, False, *args, **kwargs) + + def set_many(self, values, *args, **kwargs): + client_batches = collections.defaultdict(dict) + failed = [] + + for key, value in values.items(): + client = self._get_client(key) + + if client is None: + failed.append(key) + continue + + client_batches[client.server][key] = value + + for server, values in client_batches.items(): + client = self.clients[self._make_client_key(server)] + failed += self._safely_run_set_many(client, values, *args, **kwargs) + + return failed + + set_multi = set_many + + def get_many(self, keys, gets=False, *args, **kwargs): + client_batches = collections.defaultdict(list) + end = {} + + for key in keys: + client = self._get_client(key) + + if client is None: + continue + + client_batches[client.server].append(key) + + for server, keys in client_batches.items(): + client = self.clients[self._make_client_key(server)] + new_args = list(args) + new_args.insert(0, keys) + + if gets: + get_func = client.gets_many + else: + get_func = client.get_many + + result = self._safely_run_func(client, get_func, {}, *new_args, **kwargs) + end.update(result) + + return end + + get_multi = get_many + + def gets(self, key, *args, **kwargs): + return self._run_cmd("gets", key, None, *args, **kwargs) + + def gets_many(self, keys, *args, **kwargs): + return self.get_many(keys, gets=True, *args, **kwargs) + + gets_multi = gets_many + + def add(self, key, *args, **kwargs): + return self._run_cmd("add", key, False, *args, **kwargs) + + def prepend(self, key, *args, **kwargs): + return self._run_cmd("prepend", key, False, *args, **kwargs) + + def append(self, key, *args, **kwargs): + return self._run_cmd("append", key, False, *args, **kwargs) + + def delete(self, key, *args, **kwargs): + return self._run_cmd("delete", key, False, *args, **kwargs) + + def delete_many(self, keys, *args, **kwargs) -> bool: + for key in keys: + self._run_cmd("delete", key, False, *args, **kwargs) + return True + + delete_multi = delete_many + + def cas(self, key, *args, **kwargs): + return self._run_cmd("cas", key, False, *args, **kwargs) + + def replace(self, key, *args, **kwargs): + return self._run_cmd("replace", key, False, *args, **kwargs) + + def touch(self, key, *args, **kwargs): + return self._run_cmd("touch", key, False, *args, **kwargs) + + def flush_all(self, *args, **kwargs) -> None: + for client in self.clients.values(): + self._safely_run_func(client, client.flush_all, False, *args, **kwargs) + + def quit(self) -> None: + for client in self.clients.values(): + self._safely_run_func(client, client.quit, False) diff --git a/lib/python3.11/site-packages/pymemcache/client/murmur3.py b/lib/python3.11/site-packages/pymemcache/client/murmur3.py new file mode 100644 index 00000000..b42cc090 --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/client/murmur3.py @@ -0,0 +1,55 @@ +def murmur3_32(data, seed=0): + """MurmurHash3 was written by Austin Appleby, and is placed in the + public domain. The author hereby disclaims copyright to this source + code.""" + + c1 = 0xCC9E2D51 + c2 = 0x1B873593 + + length = len(data) + h1 = seed + roundedEnd = length & 0xFFFFFFFC # round down to 4 byte block + for i in range(0, roundedEnd, 4): + # little endian load order + k1 = ( + (ord(data[i]) & 0xFF) + | ((ord(data[i + 1]) & 0xFF) << 8) + | ((ord(data[i + 2]) & 0xFF) << 16) + | (ord(data[i + 3]) << 24) + ) + k1 *= c1 + k1 = (k1 << 15) | ((k1 & 0xFFFFFFFF) >> 17) # ROTL32(k1,15) + k1 *= c2 + + h1 ^= k1 + h1 = (h1 << 13) | ((h1 & 0xFFFFFFFF) >> 19) # ROTL32(h1,13) + h1 = h1 * 5 + 0xE6546B64 + + # tail + k1 = 0 + + val = length & 0x03 + if val == 3: + k1 = (ord(data[roundedEnd + 2]) & 0xFF) << 16 + # fallthrough + if val in [2, 3]: + k1 |= (ord(data[roundedEnd + 1]) & 0xFF) << 8 + # fallthrough + if val in [1, 2, 3]: + k1 |= ord(data[roundedEnd]) & 0xFF + k1 *= c1 + k1 = (k1 << 15) | ((k1 & 0xFFFFFFFF) >> 17) # ROTL32(k1,15) + k1 *= c2 + h1 ^= k1 + + # finalization + h1 ^= length + + # fmix(h1) + h1 ^= (h1 & 0xFFFFFFFF) >> 16 + h1 *= 0x85EBCA6B + h1 ^= (h1 & 0xFFFFFFFF) >> 13 + h1 *= 0xC2B2AE35 + h1 ^= (h1 & 0xFFFFFFFF) >> 16 + + return h1 & 0xFFFFFFFF diff --git a/lib/python3.11/site-packages/pymemcache/client/rendezvous.py b/lib/python3.11/site-packages/pymemcache/client/rendezvous.py new file mode 100644 index 00000000..eb4d9ec2 --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/client/rendezvous.py @@ -0,0 +1,46 @@ +from pymemcache.client.murmur3 import murmur3_32 + + +class RendezvousHash: + """ + Implements the Highest Random Weight (HRW) hashing algorithm most + commonly referred to as rendezvous hashing. + + Originally developed as part of python-clandestined. + + Copyright (c) 2014 Ernest W. Durbin III + """ + + def __init__(self, nodes=None, seed=0, hash_function=murmur3_32): + """ + Constructor. + """ + self.nodes = [] + self.seed = seed + if nodes is not None: + self.nodes = nodes + self.hash_function = lambda x: hash_function(x, seed) + + def add_node(self, node): + if node not in self.nodes: + self.nodes.append(node) + + def remove_node(self, node): + if node in self.nodes: + self.nodes.remove(node) + else: + raise ValueError("No such node %s to remove" % (node)) + + def get_node(self, key): + high_score = -1 + winner = None + + for node in self.nodes: + score = self.hash_function(f"{node}-{key}") + + if score > high_score: + (high_score, winner) = (score, node) + elif score == high_score: + (high_score, winner) = (score, max(str(node), str(winner))) + + return winner diff --git a/lib/python3.11/site-packages/pymemcache/client/retrying.py b/lib/python3.11/site-packages/pymemcache/client/retrying.py new file mode 100644 index 00000000..22d78988 --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/client/retrying.py @@ -0,0 +1,178 @@ +""" Module containing the RetryingClient wrapper class. """ + +from time import sleep + + +def _ensure_tuple_argument(argument_name, argument_value): + """ + Helper function to ensure the given arguments are tuples of Exceptions (or + subclasses), or can at least be converted to such. + + Args: + argument_name: str, name of the argument we're checking, only used for + raising meaningful exceptions. + argument: any, the argument itself. + + Returns: + tuple[Exception]: A tuple with the elements from the argument if they are + valid. + + Exceptions: + ValueError: If the argument was not None, tuple or Iterable. + ValueError: If any of the elements of the argument is not a subclass of + Exception. + """ + + # Ensure the argument is a tuple, set or list. + if argument_value is None: + return tuple() + elif not isinstance(argument_value, (tuple, set, list)): + raise ValueError("%s must be either a tuple, a set or a list." % argument_name) + + # Convert the argument before checking contents. + argument_tuple = tuple(argument_value) + + # Check that all the elements are actually inherited from Exception. + # (Catchable) + if not all([issubclass(arg, Exception) for arg in argument_tuple]): + raise ValueError( + "%s is only allowed to contain elements that are subclasses of " + "Exception." % argument_name + ) + + return argument_tuple + + +class RetryingClient(object): + """ + Client that allows retrying calls for the other clients. + """ + + def __init__( + self, client, attempts=2, retry_delay=0, retry_for=None, do_not_retry_for=None + ): + """ + Constructor for RetryingClient. + + Args: + client: Client|PooledClient|HashClient, inner client to use for + performing actual work. + attempts: optional int, how many times to attempt an action before + failing. Must be 1 or above. Defaults to 2. + retry_delay: optional int|float, how many seconds to sleep between + each attempt. + Defaults to 0. + + retry_for: optional None|tuple|set|list, what exceptions to + allow retries for. Will allow retries for all exceptions if None. + Example: + `(MemcacheClientError, MemcacheUnexpectedCloseError)` + Accepts any class that is a subclass of Exception. + Defaults to None. + + do_not_retry_for: optional None|tuple|set|list, what + exceptions should be retried. Will not block retries for any + Exception if None. + Example: + `(IOError, MemcacheIllegalInputError)` + Accepts any class that is a subclass of Exception. + Defaults to None. + + Exceptions: + ValueError: If `attempts` is not 1 or above. + ValueError: If `retry_for` or `do_not_retry_for` is not None, tuple or + Iterable. + ValueError: If any of the elements of `retry_for` or + `do_not_retry_for` is not a subclass of Exception. + ValueError: If there is any overlap between `retry_for` and + `do_not_retry_for`. + """ + + if attempts < 1: + raise ValueError( + "`attempts` argument must be at least 1. " + "Otherwise no attempts are made." + ) + + self._client = client + self._attempts = attempts + self._retry_delay = retry_delay + self._retry_for = _ensure_tuple_argument("retry_for", retry_for) + self._do_not_retry_for = _ensure_tuple_argument( + "do_not_retry_for", do_not_retry_for + ) + + # Verify no overlap in the go/no-go exception collections. + for exc_class in self._retry_for: + if exc_class in self._do_not_retry_for: + raise ValueError( + 'Exception class "%s" was present in both `retry_for` ' + "and `do_not_retry_for`. Any exception class is only " + "allowed in a single argument." % repr(exc_class) + ) + + # Take dir from the client to speed up future checks. + self._client_dir = dir(self._client) + + def _retry(self, name, func, *args, **kwargs): + """ + Workhorse function, handles retry logic. + + Args: + name: str, Name of the function called. + func: callable, the function to retry. + *args: args, array arguments to pass to the function. + **kwargs: kwargs, keyword arguments to pass to the function. + """ + for attempt in range(self._attempts): + try: + result = func(*args, **kwargs) + return result + + except Exception as exc: + # Raise the exception to caller if either is met: + # - We've used the last attempt. + # - self._retry_for is set, and we do not match. + # - self._do_not_retry_for is set, and we do match. + # - name is not actually a member of the client class. + if ( + attempt >= self._attempts - 1 + or (self._retry_for and not isinstance(exc, self._retry_for)) + or ( + self._do_not_retry_for + and isinstance(exc, self._do_not_retry_for) + ) + or name not in self._client_dir + ): + raise exc + + # Sleep and try again. + sleep(self._retry_delay) + + # This is the real magic soup of the class, we catch anything that isn't + # strictly defined for ourselves and pass it on to whatever client we've + # been given. + def __getattr__(self, name): + + return lambda *args, **kwargs: self._retry( + name, self._client.__getattribute__(name), *args, **kwargs + ) + + # We implement these explicitly because they're "magic" functions and won't + # get passed on by __getattr__. + + def __dir__(self): + return self._client_dir + + # These magics are copied from the base client. + def __setitem__(self, key, value): + self.set(key, value, noreply=True) + + def __getitem__(self, key): + value = self.get(key) + if value is None: + raise KeyError + return value + + def __delitem__(self, key): + self.delete(key, noreply=True) diff --git a/lib/python3.11/site-packages/pymemcache/exceptions.py b/lib/python3.11/site-packages/pymemcache/exceptions.py new file mode 100644 index 00000000..138a0dbe --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/exceptions.py @@ -0,0 +1,45 @@ +class MemcacheError(Exception): + "Base exception class" + pass + + +class MemcacheClientError(MemcacheError): + """Raised when memcached fails to parse the arguments to a request, likely + due to a malformed key and/or value, a bug in this library, or a version + mismatch with memcached.""" + + pass + + +class MemcacheUnknownCommandError(MemcacheClientError): + """Raised when memcached fails to parse a request, likely due to a bug in + this library or a version mismatch with memcached.""" + + pass + + +class MemcacheIllegalInputError(MemcacheClientError): + """Raised when a key or value is not legal for Memcache (see the class docs + for Client for more details).""" + + pass + + +class MemcacheServerError(MemcacheError): + """Raised when memcached reports a failure while processing a request, + likely due to a bug or transient issue in memcached.""" + + pass + + +class MemcacheUnknownError(MemcacheError): + """Raised when this library receives a response from memcached that it + cannot parse, likely due to a bug in this library or a version mismatch + with memcached.""" + + pass + + +class MemcacheUnexpectedCloseError(MemcacheServerError): + "Raised when the connection with memcached closes unexpectedly." + pass diff --git a/lib/python3.11/site-packages/pymemcache/fallback.py b/lib/python3.11/site-packages/pymemcache/fallback.py new file mode 100644 index 00000000..e3c843f8 --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/fallback.py @@ -0,0 +1,123 @@ +# Copyright 2012 Pinterest.com +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +A client for falling back to older memcached servers when performing reads. + +It is sometimes necessary to deploy memcached on new servers, or with a +different configuration. In these cases, it is undesirable to start up an +empty memcached server and point traffic to it, since the cache will be cold, +and the backing store will have a large increase in traffic. + +This class attempts to solve that problem by providing an interface identical +to the Client interface, but which can fall back to older memcached servers +when reads to the primary server fail. The approach for upgrading memcached +servers or configuration then becomes: + + 1. Deploy a new host (or fleet) with memcached, possibly with a new + configuration. + 2. From your application servers, use FallbackClient to write and read from + the new cluster, and to read from the old cluster when there is a miss in + the new cluster. + 3. Wait until the new cache is warm enough to support the load. + 4. Switch from FallbackClient to a regular Client library for doing all + reads and writes to the new cluster. + 5. Take down the old cluster. + +Best Practices: +--------------- + - Make sure that the old client has "ignore_exc" set to True, so that it + treats failures like cache misses. That will allow you to take down the + old cluster before you switch away from FallbackClient. +""" + + +class FallbackClient: + def __init__(self, caches): + assert len(caches) > 0 + self.caches = caches + + def close(self): + "Close each of the memcached clients" + for cache in self.caches: + cache.close() + + def set(self, key, value, expire=0, noreply=True): + self.caches[0].set(key, value, expire, noreply) + + def add(self, key, value, expire=0, noreply=True): + self.caches[0].add(key, value, expire, noreply) + + def replace(self, key, value, expire=0, noreply=True): + self.caches[0].replace(key, value, expire, noreply) + + def append(self, key, value, expire=0, noreply=True): + self.caches[0].append(key, value, expire, noreply) + + def prepend(self, key, value, expire=0, noreply=True): + self.caches[0].prepend(key, value, expire, noreply) + + def cas(self, key, value, cas, expire=0, noreply=True): + self.caches[0].cas(key, value, cas, expire, noreply) + + def get(self, key): + for cache in self.caches: + result = cache.get(key) + if result is not None: + return result + return None + + def get_many(self, keys): + for cache in self.caches: + result = cache.get_many(keys) + if result: + return result + return [] + + def gets(self, key): + for cache in self.caches: + result = cache.gets(key) + if result is not None: + return result + return None + + def gets_many(self, keys): + for cache in self.caches: + result = cache.gets_many(keys) + if result: + return result + return [] + + def delete(self, key, noreply=True): + self.caches[0].delete(key, noreply) + + def incr(self, key, value, noreply=True): + self.caches[0].incr(key, value, noreply) + + def decr(self, key, value, noreply=True): + self.caches[0].decr(key, value, noreply) + + def touch(self, key, expire=0, noreply=True): + self.caches[0].touch(key, expire, noreply) + + def stats(self): + # TODO: ?? + pass + + def flush_all(self, delay=0, noreply=True): + self.caches[0].flush_all(delay, noreply) + + def quit(self): + # TODO: ?? + pass diff --git a/lib/python3.11/site-packages/pymemcache/pool.py b/lib/python3.11/site-packages/pymemcache/pool.py new file mode 100644 index 00000000..382abd9f --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/pool.py @@ -0,0 +1,134 @@ +# Copyright 2015 Yahoo.com +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import collections +import contextlib +import threading +import time +from typing import Callable, Optional, TypeVar, Deque, List, Generic, Iterator + + +T = TypeVar("T") + + +class ObjectPool(Generic[T]): + """A pool of objects that release/creates/destroys as needed.""" + + def __init__( + self, + obj_creator: Callable[[], T], + after_remove: Optional[Callable] = None, + max_size: Optional[int] = None, + idle_timeout: int = 0, + lock_generator: Optional[Callable] = None, + ): + self._used_objs: Deque[T] = collections.deque() + self._free_objs: Deque[T] = collections.deque() + self._obj_creator = obj_creator + if lock_generator is None: + self._lock = threading.Lock() + else: + self._lock = lock_generator() + self._after_remove = after_remove + max_size = max_size or 2**31 + if not isinstance(max_size, int) or max_size < 0: + raise ValueError('"max_size" must be a positive integer') + self.max_size = max_size + self.idle_timeout = idle_timeout + if idle_timeout: + self._idle_clock = time.time + else: + self._idle_clock = float + + @property + def used(self): + return tuple(self._used_objs) + + @property + def free(self): + return tuple(self._free_objs) + + @contextlib.contextmanager + def get_and_release(self, destroy_on_fail=False) -> Iterator[T]: + obj = self.get() + try: + yield obj + except Exception: + if not destroy_on_fail: + self.release(obj) + else: + self.destroy(obj) + raise + self.release(obj) + + def get(self): + with self._lock: + # Find a free object, removing any that have idled for too long. + now = self._idle_clock() + while self._free_objs: + obj = self._free_objs.popleft() + if now - obj._last_used <= self.idle_timeout: + break + + if self._after_remove is not None: + self._after_remove(obj) + else: + # No free objects, create a new one. + curr_count = len(self._used_objs) + if curr_count >= self.max_size: + raise RuntimeError( + "Too many objects," " %s >= %s" % (curr_count, self.max_size) + ) + obj = self._obj_creator() + + self._used_objs.append(obj) + obj._last_used = now + return obj + + def destroy(self, obj, silent=True) -> None: + was_dropped = False + with self._lock: + try: + self._used_objs.remove(obj) + was_dropped = True + except ValueError: + if not silent: + raise + if was_dropped and self._after_remove is not None: + self._after_remove(obj) + + def release(self, obj, silent=True) -> None: + with self._lock: + try: + self._used_objs.remove(obj) + self._free_objs.append(obj) + obj._last_used = self._idle_clock() + except ValueError: + if not silent: + raise + + def clear(self) -> None: + if self._after_remove is not None: + needs_destroy: List[T] = [] + with self._lock: + needs_destroy.extend(self._used_objs) + needs_destroy.extend(self._free_objs) + self._free_objs.clear() + self._used_objs.clear() + for obj in needs_destroy: + self._after_remove(obj) + else: + with self._lock: + self._free_objs.clear() + self._used_objs.clear() diff --git a/lib/python3.11/site-packages/pymemcache/serde.py b/lib/python3.11/site-packages/pymemcache/serde.py new file mode 100644 index 00000000..42ec922f --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/serde.py @@ -0,0 +1,193 @@ +# Copyright 2012 Pinterest.com +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +import pickle +import zlib +from functools import partial +from io import BytesIO + +FLAG_BYTES = 0 +FLAG_PICKLE = 1 << 0 +FLAG_INTEGER = 1 << 1 +FLAG_LONG = 1 << 2 +FLAG_COMPRESSED = 1 << 3 +FLAG_TEXT = 1 << 4 + +# Pickle protocol version (highest available to runtime) +# Warning with `0`: If somewhere in your value lies a slotted object, +# ie defines `__slots__`, even if you do not include it in your pickleable +# state via `__getstate__`, python will complain with something like: +# TypeError: a class that defines __slots__ without defining __getstate__ +# cannot be pickled +DEFAULT_PICKLE_VERSION = pickle.HIGHEST_PROTOCOL + + +def _python_memcache_serializer(key, value, pickle_version=None): + flags = 0 + value_type = type(value) + + # Check against exact types so that subclasses of native types will be + # restored as their native type + if value_type is bytes: + pass + + elif value_type is str: + flags |= FLAG_TEXT + value = value.encode("utf8") + + elif value_type is int: + flags |= FLAG_INTEGER + value = "%d" % value + + else: + flags |= FLAG_PICKLE + output = BytesIO() + pickler = pickle.Pickler(output, pickle_version) + pickler.dump(value) + value = output.getvalue() + + return value, flags + + +def get_python_memcache_serializer(pickle_version: int = DEFAULT_PICKLE_VERSION): + """Return a serializer using a specific pickle version""" + return partial(_python_memcache_serializer, pickle_version=pickle_version) + + +python_memcache_serializer = get_python_memcache_serializer() + + +def python_memcache_deserializer(key, value, flags): + if flags == 0: + return value + + elif flags & FLAG_TEXT: + return value.decode("utf8") + + elif flags & FLAG_INTEGER: + return int(value) + + elif flags & FLAG_LONG: + return int(value) + + elif flags & FLAG_PICKLE: + try: + buf = BytesIO(value) + unpickler = pickle.Unpickler(buf) + return unpickler.load() + except Exception: + logging.info("Pickle error", exc_info=True) + return None + + return value + + +class PickleSerde: + """ + An object which implements the serialization/deserialization protocol for + :py:class:`pymemcache.client.base.Client` and its descendants using the + :mod:`pickle` module. + + Serialization and deserialization are implemented as methods of this class. + To implement a custom serialization/deserialization method for pymemcache, + you should implement the same interface as the one provided by this object + -- :py:meth:`pymemcache.serde.PickleSerde.serialize` and + :py:meth:`pymemcache.serde.PickleSerde.deserialize`. Then, + pass your custom object to the pymemcache client object in place of + `PickleSerde`. + + For more details on the serialization protocol, see the class documentation + for :py:class:`pymemcache.client.base.Client` + """ + + def __init__(self, pickle_version: int = DEFAULT_PICKLE_VERSION) -> None: + self._serialize_func = get_python_memcache_serializer(pickle_version) + + def serialize(self, key, value): + return self._serialize_func(key, value) + + def deserialize(self, key, value, flags): + return python_memcache_deserializer(key, value, flags) + + +pickle_serde = PickleSerde() + + +class CompressedSerde: + """ + An object which implements the serialization/deserialization protocol for + :py:class:`pymemcache.client.base.Client` and its descendants with + configurable compression. + """ + + def __init__( + self, + compress=zlib.compress, + decompress=zlib.decompress, + serde=pickle_serde, + # Discovered via the `test_optimal_compression_length` test. + min_compress_len=400, + ): + self._serde = serde + self._compress = compress + self._decompress = decompress + self._min_compress_len = min_compress_len + + def serialize(self, key, value): + value, flags = self._serde.serialize(key, value) + + if len(value) > self._min_compress_len > 0: + old_value = value + value = self._compress(value) + # Don't use the compressed value if our end result is actually + # larger uncompressed. + if len(old_value) < len(value): + value = old_value + else: + flags |= FLAG_COMPRESSED + + return value, flags + + def deserialize(self, key, value, flags): + if flags & FLAG_COMPRESSED: + value = self._decompress(value) + + value = self._serde.deserialize(key, value, flags) + return value + + +compressed_serde = CompressedSerde() + + +class LegacyWrappingSerde: + """ + This class defines how to wrap legacy de/serialization functions into a + 'serde' object which implements '.serialize' and '.deserialize' methods. + It is used automatically by pymemcache.client.base.Client when the + 'serializer' or 'deserializer' arguments are given. + + The serializer_func and deserializer_func are expected to be None in the + case that they are missing. + """ + + def __init__(self, serializer_func, deserializer_func) -> None: + self.serialize = serializer_func or self._default_serialize + self.deserialize = deserializer_func or self._default_deserialize + + def _default_serialize(self, key, value): + return value, 0 + + def _default_deserialize(self, key, value, flags): + return value diff --git a/lib/python3.11/site-packages/pymemcache/test/__init__.py b/lib/python3.11/site-packages/pymemcache/test/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/lib/python3.11/site-packages/pymemcache/test/__pycache__/__init__.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/test/__pycache__/__init__.cpython-311.pyc new file mode 100644 index 00000000..57c7ed5f Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/test/__pycache__/__init__.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/test/__pycache__/conftest.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/test/__pycache__/conftest.cpython-311.pyc new file mode 100644 index 00000000..b9e82e4f Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/test/__pycache__/conftest.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_benchmark.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_benchmark.cpython-311.pyc new file mode 100644 index 00000000..6dd3b41e Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_benchmark.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_client.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_client.cpython-311.pyc new file mode 100644 index 00000000..b22605de Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_client.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_client_hash.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_client_hash.cpython-311.pyc new file mode 100644 index 00000000..9b10871e Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_client_hash.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_client_retry.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_client_retry.cpython-311.pyc new file mode 100644 index 00000000..4ccb9afa Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_client_retry.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_compression.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_compression.cpython-311.pyc new file mode 100644 index 00000000..e6a27a6b Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_compression.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_integration.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_integration.cpython-311.pyc new file mode 100644 index 00000000..0c6bf49a Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_integration.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_rendezvous.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_rendezvous.cpython-311.pyc new file mode 100644 index 00000000..8239b748 Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_rendezvous.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_serde.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_serde.cpython-311.pyc new file mode 100644 index 00000000..818f0f22 Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_serde.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_utils.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_utils.cpython-311.pyc new file mode 100644 index 00000000..2189fec4 Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/test/__pycache__/test_utils.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/test/__pycache__/utils.cpython-311.pyc b/lib/python3.11/site-packages/pymemcache/test/__pycache__/utils.cpython-311.pyc new file mode 100644 index 00000000..b6a13ab0 Binary files /dev/null and b/lib/python3.11/site-packages/pymemcache/test/__pycache__/utils.cpython-311.pyc differ diff --git a/lib/python3.11/site-packages/pymemcache/test/conftest.py b/lib/python3.11/site-packages/pymemcache/test/conftest.py new file mode 100644 index 00000000..ce532e96 --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/test/conftest.py @@ -0,0 +1,116 @@ +import os.path +import socket +import ssl + +import pytest + + +def pytest_addoption(parser): + parser.addoption( + "--server", action="store", default="localhost", help="memcached server" + ) + + parser.addoption( + "--port", action="store", default="11211", help="memcached server port" + ) + + parser.addoption( + "--tls-server", action="store", default="localhost", help="TLS memcached server" + ) + + parser.addoption( + "--tls-port", action="store", default="11212", help="TLS memcached server port" + ) + + parser.addoption( + "--size", action="store", default=1024, help="size of data in benchmarks" + ) + + parser.addoption( + "--count", + action="store", + default=10000, + help="number of iterations to run each benchmark", + ) + + parser.addoption( + "--keys", + action="store", + default=20, + help="number of keys to use for multi benchmarks", + ) + + +@pytest.fixture(scope="session") +def host(request): + return request.config.option.server + + +@pytest.fixture(scope="session") +def port(request): + return int(request.config.option.port) + + +@pytest.fixture(scope="session") +def tls_host(request): + return request.config.option.tls_server + + +@pytest.fixture(scope="session") +def tls_port(request): + return int(request.config.option.tls_port) + + +@pytest.fixture(scope="session") +def size(request): + return int(request.config.option.size) + + +@pytest.fixture(scope="session") +def count(request): + return int(request.config.option.count) + + +@pytest.fixture(scope="session") +def keys(request): + return int(request.config.option.keys) + + +@pytest.fixture(scope="session") +def pairs(size, keys): + return {"pymemcache_test:%d" % i: "X" * size for i in range(keys)} + + +@pytest.fixture(scope="session") +def tls_context(): + return ssl.create_default_context( + cafile=os.path.join(os.path.dirname(__file__), "certs/ca-root.crt") + ) + + +def pytest_generate_tests(metafunc): + if "socket_module" in metafunc.fixturenames: + socket_modules = [socket] + try: + from gevent import socket as gevent_socket # type: ignore + except ImportError: + print("Skipping gevent (not installed)") + else: + socket_modules.append(gevent_socket) + + metafunc.parametrize("socket_module", socket_modules) + + if "client_class" in metafunc.fixturenames: + from pymemcache.client.base import Client, PooledClient + from pymemcache.client.hash import HashClient + + class HashClientSingle(HashClient): + def __init__(self, server, *args, **kwargs): + super().__init__([server], *args, **kwargs) + + metafunc.parametrize("client_class", [Client, PooledClient, HashClientSingle]) + + if "key_prefix" in metafunc.fixturenames: + mark = metafunc.definition.get_closest_marker("parametrize") + if not mark or "key_prefix" not in mark.args[0]: + metafunc.parametrize("key_prefix", [b"", b"prefix"]) diff --git a/lib/python3.11/site-packages/pymemcache/test/test_benchmark.py b/lib/python3.11/site-packages/pymemcache/test/test_benchmark.py new file mode 100644 index 00000000..f1234825 --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/test/test_benchmark.py @@ -0,0 +1,118 @@ +# Copyright 2012 Pinterest.com +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import time +import pytest + +try: + import pylibmc # type: ignore + + HAS_PYLIBMC = True +except Exception: + HAS_PYLIBMC = False + +try: + import memcache # type: ignore + + HAS_MEMCACHE = True +except Exception: + HAS_MEMCACHE = False + + +try: + import pymemcache.client + + HAS_PYMEMCACHE = True +except Exception: + HAS_PYMEMCACHE = False + + +@pytest.fixture( + params=[ + "pylibmc", + "memcache", + "pymemcache", + ] +) +def client(request, host, port): + if request.param == "pylibmc": + if not HAS_PYLIBMC: + pytest.skip("requires pylibmc") + client = pylibmc.Client([f"{host}:{port}"]) + client.behaviors = {"tcp_nodelay": True} + + elif request.param == "memcache": + if not HAS_MEMCACHE: + pytest.skip("requires python-memcached") + client = memcache.Client([f"{host}:{port}"]) + + elif request.param == "pymemcache": + if not HAS_PYMEMCACHE: + pytest.skip("requires pymemcache") + client = pymemcache.client.Client((host, port)) + + else: + pytest.skip(f"unknown library {request.param}") + + client.flush_all() + return client + + +def benchmark(count, func, *args, **kwargs): + start = time.time() + + for _ in range(count): + result = func(*args, **kwargs) + + duration = time.time() - start + print(str(duration)) + + return result + + +@pytest.mark.benchmark() +def test_bench_get(request, client, pairs, count): + key = "pymemcache_test:0" + value = pairs[key] + client.set(key, value) + benchmark(count, client.get, key) + + +@pytest.mark.benchmark() +def test_bench_set(request, client, pairs, count): + key = "pymemcache_test:0" + value = pairs[key] + benchmark(count, client.set, key, value) + + +@pytest.mark.benchmark() +def test_bench_get_multi(request, client, pairs, count): + client.set_multi(pairs) + benchmark(count, client.get_multi, list(pairs)) + + +@pytest.mark.benchmark() +def test_bench_set_multi(request, client, pairs, count): + benchmark(count, client.set_multi, pairs) + + +@pytest.mark.benchmark() +def test_bench_delete(request, client, pairs, count): + benchmark(count, client.delete, next(pairs)) + + +@pytest.mark.benchmark() +def test_bench_delete_multi(request, client, pairs, count): + # deleting missing key takes the same work client-side as real keys + benchmark(count, client.delete_multi, list(pairs.keys())) diff --git a/lib/python3.11/site-packages/pymemcache/test/test_client.py b/lib/python3.11/site-packages/pymemcache/test/test_client.py new file mode 100644 index 00000000..22f8387d --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/test/test_client.py @@ -0,0 +1,1618 @@ +# Copyright 2012 Pinterest.com +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import collections +import errno +import functools +import ipaddress +import json +import os +import platform +from unittest import mock +import socket +import unittest + +import pytest + +from pymemcache.client.base import ( + PooledClient, + Client, + normalize_server_spec, + KeepaliveOpts, + check_key_helper, +) +from pymemcache.exceptions import ( + MemcacheClientError, + MemcacheServerError, + MemcacheUnexpectedCloseError, + MemcacheUnknownCommandError, + MemcacheUnknownError, + MemcacheIllegalInputError, +) + +from pymemcache import pool +from pymemcache.test.utils import MockMemcacheClient + + +def is_ipv6(address): + try: + return ipaddress.ip_address(address).version == 6 + except ValueError: + # Fail to parse as valid ip address + return False + + +@pytest.mark.parametrize( + "key,allow_unicode_keys,key_prefix,ex_exception,ex_excinfo", + [ + ("b" * 251, True, b"", MemcacheIllegalInputError, "Key is too long"), + ("foo bar", True, b"", MemcacheIllegalInputError, "Key contains whitespace"), + ("\00", True, b"", MemcacheIllegalInputError, "Key contains null"), + (None, True, b"", TypeError, None), + ], +) +@pytest.mark.unit() +def test_check_key_helper_failing_conditions( + key, allow_unicode_keys, key_prefix, ex_exception, ex_excinfo +): + + with pytest.raises(ex_exception) as excinfo: + check_key_helper(key, allow_unicode_keys, key_prefix) + + # Allow to ignore the excinfo value. Different implementation (2.7, + # pypy, 3.x) comes with various error messages for the same thing. + # We just check the kind of the exception. + if ex_excinfo: + assert ex_excinfo in str(excinfo.value) + + +@pytest.mark.unit() +def test_check_key_helper(): + assert check_key_helper(b"key", True, b"") == b"key" + assert check_key_helper("key", True) == b"key" + assert isinstance(check_key_helper(b"key", True), bytes) + assert check_key_helper("", True) == b"" + + +class MockSocket: + def __init__(self, recv_bufs, connect_failure=None, close_failure=None): + self.recv_bufs = collections.deque(recv_bufs) + self.send_bufs = [] + self.closed = False + self.timeouts = [] + self.connect_failure = connect_failure + self.close_failure = close_failure + self.connections = [] + self.socket_options = [] + + @property + def family(self): + any_ipv6 = any(is_ipv6(c[0]) for c in self.connections) + return socket.AF_INET6 if any_ipv6 else socket.AF_INET + + def sendall(self, value): + self.send_bufs.append(value) + + def close(self): + if isinstance(self.close_failure, Exception): + raise self.close_failure + self.closed = True + + def recv(self, size): + value = self.recv_bufs.popleft() + if isinstance(value, Exception): + raise value + return value + + def settimeout(self, timeout): + self.timeouts.append(timeout) + + def connect(self, server): + if isinstance(self.connect_failure, Exception): + raise self.connect_failure + self.connections.append(server) + + def setsockopt(self, level, option, value): + self.socket_options.append((level, option, value)) + + +class MockUnixSocketServer: + def __init__(self, socket_path): + if os.path.exists(socket_path): + os.remove(socket_path) + self.socket_path = socket_path + self.socket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) + + def __enter__(self): + self.socket.bind(self.socket_path) + self.socket.listen(1) + return self.socket + + def __exit__(self, *args): + self.socket.close() + os.remove(self.socket_path) + + +class MockSocketModule: + def __init__(self, connect_failure=None, close_failure=None): + self.connect_failure = connect_failure + self.close_failure = close_failure + self.sockets = [] + + def socket(self, family, type, proto=0, fileno=None): + socket = MockSocket( + [], connect_failure=self.connect_failure, close_failure=self.close_failure + ) + self.sockets.append(socket) + return socket + + def getaddrinfo(self, host, port, family=0, type=0, proto=0, flags=0): + family = family or (socket.AF_INET6 if is_ipv6(host) else socket.AF_INET) + type = type or socket.SOCK_STREAM + proto = proto or socket.IPPROTO_TCP + sockaddr = ( + ("::1", 11211, 0, 0) if family == socket.AF_INET6 else ("127.0.0.1", 11211) + ) + return [(family, type, proto, "", sockaddr)] + + def __getattr__(self, name): + return getattr(socket, name) + + +class CustomizedClient(Client): + def _extract_value(self, expect_cas, line, buf, remapped_keys, prefixed_keys): + return b"key", b"value", b"END\r\n" + + +@pytest.mark.unit() +class ClientTestMixin: + def make_client(self, mock_socket_values, **kwargs): + client = Client("localhost", **kwargs) + # mock out client._connect() rather than hard-setting client.sock to + # ensure methods are checking whether self.sock is None before + # attempting to use it + sock = MockSocket(list(mock_socket_values)) + client._connect = mock.Mock( + side_effect=functools.partial(setattr, client, "sock", sock) + ) + return client + + def make_customized_client(self, mock_socket_values, **kwargs): + client = CustomizedClient("localhost", **kwargs) + # mock out client._connect() rather than hard-setting client.sock to + # ensure methods are checking whether self.sock is None before + # attempting to use it + sock = MockSocket(list(mock_socket_values)) + client._connect = mock.Mock( + side_effect=functools.partial(setattr, client, "sock", sock) + ) + return client + + def test_set_success(self): + client = self.make_client([b"STORED\r\n"]) + result = client.set(b"key", b"value", noreply=False) + assert result is True + + # unit test for encoding passed in __init__() + client = self.make_client([b"STORED\r\n"], encoding="utf-8") + result = client.set(b"key", b"value", noreply=False) + assert result is True + + # unit test for set operation with parameter flags + client = self.make_client([b"STORED\r\n"], encoding="utf-8") + result = client.set(b"key", b"value", noreply=False, flags=0x00000030) + assert result is True + + def test_set_unicode_key(self): + client = self.make_client([b""]) + + def _set(): + client.set("\u0FFF", b"value", noreply=False) + + with pytest.raises(MemcacheIllegalInputError): + _set() + + def test_set_unicode_key_ok(self): + client = self.make_client([b"STORED\r\n"], allow_unicode_keys=True) + + result = client.set("\u0FFF", b"value", noreply=False) + assert result is True + + def test_set_unicode_key_ok_snowman(self): + client = self.make_client([b"STORED\r\n"], allow_unicode_keys=True) + + result = client.set("my☃", b"value", noreply=False) + assert result is True + + def test_set_unicode_char_in_middle_of_key(self): + client = self.make_client([b"STORED\r\n"]) + + def _set(): + client.set("helloworld_\xb1901520_%c3", b"value", noreply=False) + + with pytest.raises(MemcacheIllegalInputError): + _set() + + def test_set_unicode_char_in_middle_of_key_snowman(self): + client = self.make_client([b"STORED\r\n"]) + + def _set(): + client.set("my☃", b"value", noreply=False) + + with pytest.raises(MemcacheIllegalInputError): + _set() + + def test_set_unicode_value(self): + client = self.make_client([b""]) + + def _set(): + client.set(b"key", "\u0FFF", noreply=False) + + with pytest.raises(MemcacheIllegalInputError): + _set() + + def test_set_unicode_char_in_middle_of_key_ok(self): + client = self.make_client([b"STORED\r\n"], allow_unicode_keys=True) + + result = client.set("helloworld_\xb1901520_%c3", b"value", noreply=False) + assert result is True + + def test_set_noreply(self): + client = self.make_client([]) + result = client.set(b"key", b"value", noreply=True) + assert result is True + + # unit test for encoding passed in __init__() + client = self.make_client([], encoding="utf-8") + result = client.set(b"key", b"value", noreply=True) + assert result is True + + def test_set_many_success(self): + client = self.make_client([b"STORED\r\n"]) + result = client.set_many({b"key": b"value"}, noreply=False) + assert result == [] + + # unit test for encoding passed in __init__() + client = self.make_client([b"STORED\r\n"], encoding="utf-8") + result = client.set_many({b"key": b"value"}, noreply=False) + assert result == [] + + def test_set_multi_success(self): + # Should just map to set_many + client = self.make_client([b"STORED\r\n"]) + result = client.set_multi({b"key": b"value"}, noreply=False) + assert result == [] + + # unit test for encoding passed in __init__() + client = self.make_client([b"STORED\r\n"], encoding="utf-8") + result = client.set_multi({b"key": b"value"}, noreply=False) + assert result == [] + + def test_add_stored(self): + client = self.make_client([b"STORED\r", b"\n"]) + result = client.add(b"key", b"value", noreply=False) + assert result is True + + # unit test for encoding passed in __init__() + client = self.make_client([b"STORED\r", b"\n"], encoding="utf-8") + result = client.add(b"key", b"value", noreply=False) + assert result is True + + def test_add_not_stored(self): + client = self.make_client( + [b"STORED\r", b"\n", b"NOT_", b"STOR", b"ED", b"\r\n"] + ) + client.add(b"key", b"value", noreply=False) + result = client.add(b"key", b"value", noreply=False) + assert result is False + + # unit test for encoding passed in __init__() + client = self.make_client( + [b"STORED\r", b"\n", b"NOT_", b"STOR", b"ED", b"\r\n"], encoding="utf-8" + ) + client.add(b"key", b"value", noreply=False) + result = client.add(b"key", b"value", noreply=False) + assert result is False + + def test_get_not_found(self): + client = self.make_client([b"END\r\n"]) + result = client.get(b"key") + assert result is None + + # Unit test for customized client (override _extract_value) + client = self.make_customized_client([b"END\r\n"]) + result = client.get(b"key") + assert result is None + + def test_space_key(self): + client = self.make_client([b""]) + with pytest.raises(MemcacheIllegalInputError): + client.get(b"space key") + + with pytest.raises(MemcacheIllegalInputError): + client.set(b"space key", b"value") + + def test_get_not_found_default(self): + client = self.make_client([b"END\r\n"]) + result = client.get(b"key", default="foobar") + assert result == "foobar" + + # Unit test for customized client (override _extract_value) + client = self.make_customized_client([b"END\r\n"]) + result = client.get(b"key", default="foobar") + assert result == "foobar" + + def test_get_ignore_exc_default(self): + client = self.make_client([b"INVALID DATA\r\n"], ignore_exc=True) + result = client.get(b"key", default="foobar") + assert result == "foobar" + + # Unit test for customized client (override _extract_value) + client = self.make_client([b"INVALID DATA\r\n"], ignore_exc=True) + result = client.get(b"key", default="foobar") + assert result == "foobar" + + def test_get_found(self): + client = self.make_client( + [ + b"STORED\r\n", + b"VALUE key 0 5\r\nvalue\r\nEND\r\n", + ] + ) + client.set(b"key", b"value", noreply=False) + result = client.get(b"key") + assert result == b"value" + + # Unit test for customized client (override _extract_value) + client = self.make_customized_client( + [ + b"STORED\r\n", + b"VALUE key 0 5\r\nvalue\r\nEND\r\n", + ] + ) + client.set(b"key", b"value", noreply=False) + result = client.get(b"key") + assert result == b"value" + + def test_get_many_none_found(self): + client = self.make_client([b"END\r\n"]) + result = client.get_many([b"key1", b"key2"]) + assert result == {} + + def test_get_multi_none_found(self): + client = self.make_client([b"END\r\n"]) + result = client.get_multi([b"key1", b"key2"]) + assert result == {} + + def test_get_many_some_found(self): + client = self.make_client( + [ + b"STORED\r\n", + b"VALUE key1 0 6\r\nvalue1\r\nEND\r\n", + ] + ) + client.set(b"key1", b"value1", noreply=False) + result = client.get_many([b"key1", b"key2"]) + assert result == {b"key1": b"value1"} + + def test_get_many_all_found(self): + client = self.make_client( + [ + b"STORED\r\n", + b"STORED\r\n", + b"VALUE key1 0 6\r\nvalue1\r\n", + b"VALUE key2 0 6\r\nvalue2\r\nEND\r\n", + ] + ) + client.set(b"key1", b"value1", noreply=False) + client.set(b"key2", b"value2", noreply=False) + result = client.get_many([b"key1", b"key2"]) + assert result == {b"key1": b"value1", b"key2": b"value2"} + + def test_get_unicode_key(self): + client = self.make_client([b""]) + + def _get(): + client.get("\u0FFF") + + with pytest.raises(MemcacheIllegalInputError): + _get() + + def test_delete_not_found(self): + client = self.make_client([b"NOT_FOUND\r\n"]) + result = client.delete(b"key", noreply=False) + assert result is False + + def test_delete_found(self): + client = self.make_client([b"STORED\r", b"\n", b"DELETED\r\n"]) + client.add(b"key", b"value", noreply=False) + result = client.delete(b"key", noreply=False) + assert result is True + + def test_delete_noreply(self): + client = self.make_client([]) + result = client.delete(b"key", noreply=True) + assert result is True + + def test_delete_many_no_keys(self): + client = self.make_client([]) + result = client.delete_many([], noreply=False) + assert result is True + + def test_delete_many_none_found(self): + client = self.make_client([b"NOT_FOUND\r\n"]) + result = client.delete_many([b"key"], noreply=False) + assert result is True + + def test_delete_many_found(self): + client = self.make_client([b"STORED\r", b"\n", b"DELETED\r\n"]) + client.add(b"key", b"value", noreply=False) + result = client.delete_many([b"key"], noreply=False) + assert result is True + + def test_delete_many_some_found(self): + client = self.make_client([b"STORED\r\n", b"DELETED\r\n", b"NOT_FOUND\r\n"]) + client.add(b"key", b"value", noreply=False) + result = client.delete_many([b"key", b"key2"], noreply=False) + assert result is True + + def test_delete_multi_some_found(self): + client = self.make_client([b"STORED\r\n", b"DELETED\r\n", b"NOT_FOUND\r\n"]) + client.add(b"key", b"value", noreply=False) + result = client.delete_multi([b"key", b"key2"], noreply=False) + assert result is True + + def test_incr_not_found(self): + client = self.make_client([b"NOT_FOUND\r\n"]) + result = client.incr(b"key", 1, noreply=False) + assert result is None + + def test_incr_found(self): + client = self.make_client([b"STORED\r\n", b"1\r\n"]) + client.set(b"key", 0, noreply=False) + result = client.incr(b"key", 1, noreply=False) + assert result == 1 + + def test_incr_noreply(self): + client = self.make_client([b"STORED\r\n"]) + client.set(b"key", 0, noreply=False) + + client = self.make_client([]) + result = client.incr(b"key", 1, noreply=True) + assert result is None + + def test_decr_not_found(self): + client = self.make_client([b"NOT_FOUND\r\n"]) + result = client.decr(b"key", 1, noreply=False) + assert result is None + + def test_decr_found(self): + client = self.make_client([b"STORED\r\n", b"1\r\n"]) + client.set(b"key", 2, noreply=False) + result = client.decr(b"key", 1, noreply=False) + assert result == 1 + + +class TestClient(ClientTestMixin, unittest.TestCase): + + Client = Client + + def test_append_stored(self): + client = self.make_client([b"STORED\r\n"]) + result = client.append(b"key", b"value", noreply=False) + assert result is True + + # unit test for encoding passed in __init__() + client = self.make_client([b"STORED\r\n"], encoding="utf-8") + result = client.append(b"key", b"value", noreply=False) + assert result is True + + def test_prepend_stored(self): + client = self.make_client([b"STORED\r\n"]) + result = client.prepend(b"key", b"value", noreply=False) + assert result is True + + # unit test for encoding passed in __init__() + client = self.make_client([b"STORED\r\n"], encoding="utf-8") + result = client.prepend(b"key", b"value", noreply=False) + assert result is True + + def test_cas_malformed(self): + client = self.make_client([b"STORED\r\n"]) + with pytest.raises(MemcacheIllegalInputError): + client.cas(b"key", b"value", None, noreply=False) + + with pytest.raises(MemcacheIllegalInputError): + client.cas(b"key", b"value", "nonintegerstring", noreply=False) + + with pytest.raises(MemcacheIllegalInputError): + # even a space makes it a noninteger string + client.cas(b"key", b"value", "123 ", noreply=False) + + with pytest.raises(MemcacheIllegalInputError): + # non-ASCII digit + client.cas(b"key", b"value", "⁰", noreply=False) + + def test_cas_stored(self): + client = self.make_client([b"STORED\r\n"]) + result = client.cas(b"key", b"value", b"123", noreply=False) + assert result is True + + # unit test for encoding passed in __init__() + client = self.make_client([b"STORED\r\n"], encoding="utf-8") + result = client.cas(b"key", b"value", b"123", noreply=False) + assert result is True + + def test_cas_exists(self): + client = self.make_client([b"EXISTS\r\n"]) + result = client.cas(b"key", b"value", b"123", noreply=False) + assert result is False + + # unit test for encoding passed in __init__() + client = self.make_client([b"EXISTS\r\n"], encoding="utf-8") + result = client.cas(b"key", b"value", b"123", noreply=False) + assert result is False + + def test_cas_not_found(self): + client = self.make_client([b"NOT_FOUND\r\n"]) + result = client.cas(b"key", b"value", b"123", noreply=False) + assert result is None + + # unit test for encoding passed in __init__() + client = self.make_client([b"NOT_FOUND\r\n"], encoding="utf-8") + result = client.cas(b"key", b"value", b"123", noreply=False) + assert result is None + + def test_cr_nl_boundaries(self): + client = self.make_client( + [ + b"VALUE key1 0 6\r", + b"\nvalue1\r\n" b"VALUE key2 0 6\r\n", + b"value2\r\n" b"END\r\n", + ] + ) + result = client.get_many([b"key1", b"key2"]) + assert result == {b"key1": b"value1", b"key2": b"value2"} + + client = self.make_client( + [ + b"VALUE key1 0 6\r\n", + b"value1\r", + b"\nVALUE key2 0 6\r\n", + b"value2\r\n", + b"END\r\n", + ] + ) + result = client.get_many([b"key1", b"key2"]) + assert result == {b"key1": b"value1", b"key2": b"value2"} + + client = self.make_client( + [ + b"VALUE key1 0 6\r\n", + b"value1\r\n", + b"VALUE key2 0 6\r", + b"\nvalue2\r\n", + b"END\r\n", + ] + ) + result = client.get_many([b"key1", b"key2"]) + assert result == {b"key1": b"value1", b"key2": b"value2"} + + client = self.make_client( + [ + b"VALUE key1 0 6\r\n", + b"value1\r\n", + b"VALUE key2 0 6\r\n", + b"value2\r", + b"\nEND\r\n", + ] + ) + result = client.get_many([b"key1", b"key2"]) + assert result == {b"key1": b"value1", b"key2": b"value2"} + + client = self.make_client( + [ + b"VALUE key1 0 6\r\n", + b"value1\r\n", + b"VALUE key2 0 6\r\n", + b"value2\r\n", + b"END\r", + b"\n", + ] + ) + result = client.get_many([b"key1", b"key2"]) + assert result == {b"key1": b"value1", b"key2": b"value2"} + + client = self.make_client( + [ + b"VALUE key1 0 6\r", + b"\nvalue1\r", + b"\nVALUE key2 0 6\r", + b"\nvalue2\r", + b"\nEND\r", + b"\n", + ] + ) + result = client.get_many([b"key1", b"key2"]) + assert result == {b"key1": b"value1", b"key2": b"value2"} + + def test_delete_exception(self): + client = self.make_client([Exception("fail")]) + + def _delete(): + client.delete(b"key", noreply=False) + + with pytest.raises(Exception): + _delete() + + assert client.sock is None + + def test_flush_all(self): + client = self.make_client([b"OK\r\n"]) + result = client.flush_all(noreply=False) + assert result is True + + def test_incr_exception(self): + client = self.make_client([Exception("fail")]) + + def _incr(): + client.incr(b"key", 1) + + with pytest.raises(Exception): + _incr() + + assert client.sock is None + + def test_get_error(self): + client = self.make_client([b"ERROR\r\n"]) + + def _get(): + client.get(b"key") + + with pytest.raises(MemcacheUnknownCommandError): + _get() + + def test_get_recv_chunks(self): + client = self.make_client( + [b"VALUE key", b" 0 5\r", b"\nvalue", b"\r\n", b"END", b"\r", b"\n"] + ) + result = client.get(b"key") + assert result == b"value" + + def test_get_unknown_error(self): + client = self.make_client([b"foobarbaz\r\n"]) + + def _get(): + client.get(b"key") + + with pytest.raises(MemcacheUnknownError): + _get() + + def test_gets_not_found(self): + client = self.make_client([b"END\r\n"]) + result = client.gets(b"key") + assert result == (None, None) + + def test_gets_not_found_defaults(self): + client = self.make_client([b"END\r\n"]) + result = client.gets(b"key", default="foo", cas_default="bar") + assert result == ("foo", "bar") + + def test_gets_found(self): + client = self.make_client([b"VALUE key 0 5 10\r\nvalue\r\nEND\r\n"]) + result = client.gets(b"key") + assert result == (b"value", b"10") + + def test_gets_many_none_found(self): + client = self.make_client([b"END\r\n"]) + result = client.gets_many([b"key1", b"key2"]) + assert result == {} + + def test_gets_many_some_found(self): + client = self.make_client([b"VALUE key1 0 6 11\r\nvalue1\r\nEND\r\n"]) + result = client.gets_many([b"key1", b"key2"]) + assert result == {b"key1": (b"value1", b"11")} + + def test_touch_not_found(self): + client = self.make_client([b"NOT_FOUND\r\n"]) + result = client.touch(b"key", noreply=False) + assert result is False + + def test_touch_found(self): + client = self.make_client([b"TOUCHED\r\n"]) + result = client.touch(b"key", noreply=False) + assert result is True + + def test_quit(self): + client = self.make_client([]) + result = client.quit() + assert result is None + assert client.sock is None + + def test_shutdown(self): + client = self.make_client([MemcacheUnexpectedCloseError("shutdown")]) + result = client.shutdown() + assert result is None + + def test_shutdown_disabled(self): + def _shutdown(): + client = self.make_client([b"ERROR: shutdown not enabled\r\n"]) + client.shutdown() + + with pytest.raises(MemcacheUnknownCommandError): + _shutdown() + + def test_replace_stored(self): + client = self.make_client([b"STORED\r\n"]) + result = client.replace(b"key", b"value", noreply=False) + assert result is True + + # unit test for encoding passed in __init__() + client = self.make_client([b"STORED\r\n"], encoding="utf-8") + result = client.replace(b"key", b"value", noreply=False) + assert result is True + + def test_replace_not_stored(self): + client = self.make_client([b"NOT_STORED\r\n"]) + result = client.replace(b"key", b"value", noreply=False) + assert result is False + + # unit test for encoding passed in __init__ + client = self.make_client([b"NOT_STORED\r\n"], encoding="utf-8") + result = client.replace(b"key", b"value", noreply=False) + assert result is False + + def test_serialization(self): + class JsonSerde: + def serialize(self, key, value): + return json.dumps(value).encode("ascii"), 0 + + def deserialize(self, key, value, flags): + return json.loads(value.decode("ascii")) + + client = self.make_client([b"STORED\r\n"], serde=JsonSerde()) + client.set("key", {"c": "d"}) + assert client.sock.send_bufs == [b'set key 0 0 10 noreply\r\n{"c": "d"}\r\n'] + + def test_serialization_flags(self): + def _ser(key, value): + return value, 1 if isinstance(value, int) else 0 + + client = self.make_client([b"STORED\r\n", b"STORED\r\n"], serializer=_ser) + client.set_many( + collections.OrderedDict([(b"a", b"s"), (b"b", 0)]), noreply=False + ) + assert client.sock.send_bufs == [b"set a 0 0 1\r\ns\r\nset b 1 0 1\r\n0\r\n"] + + def test_serialization_overridden_flags(self): + def _ser(key, value): + return value, 1 if isinstance(value, int) else 0 + + client = self.make_client([b"STORED\r\n", b"STORED\r\n"], serializer=_ser) + client.set_many( + collections.OrderedDict([(b"a", b"s"), (b"b", 0)]), noreply=False, flags=5 + ) + assert client.sock.send_bufs == [b"set a 5 0 1\r\ns\r\nset b 5 0 1\r\n0\r\n"] + + def test_explicit_flags(self): + client = self.make_client([b"STORED\r\n", b"STORED\r\n"]) + client.set_many( + collections.OrderedDict([(b"a", b"s"), (b"b", 0)]), noreply=False, flags=5 + ) + assert client.sock.send_bufs == [b"set a 5 0 1\r\ns\r\nset b 5 0 1\r\n0\r\n"] + + def test_set_socket_handling(self): + client = self.make_client([b"STORED\r\n"]) + result = client.set(b"key", b"value", noreply=False) + assert result is True + assert client.sock.closed is False + assert len(client.sock.send_bufs) == 1 + + def test_set_error(self): + client = self.make_client([b"ERROR\r\n"]) + + def _set(): + client.set(b"key", b"value", noreply=False) + + with pytest.raises(MemcacheUnknownCommandError): + _set() + + def test_set_exception(self): + client = self.make_client([Exception("fail")]) + + def _set(): + client.set(b"key", b"value", noreply=False) + + with pytest.raises(Exception): + _set() + + assert client.sock is None + + def test_set_client_error(self): + client = self.make_client([b"CLIENT_ERROR some message\r\n"]) + + def _set(): + client.set("key", "value", noreply=False) + + with pytest.raises(MemcacheClientError): + _set() + + def test_set_server_error(self): + client = self.make_client([b"SERVER_ERROR some message\r\n"]) + + def _set(): + client.set(b"key", b"value", noreply=False) + + with pytest.raises(MemcacheServerError): + _set() + + def test_closing_socket_on_unexpected_closed_error(self): + client = self.make_client( + [ + b"VALUE ", + MemcacheUnexpectedCloseError("foo bar"), + ] + ) + + def _set(): + client.set(b"key", b"value", noreply=False) + + with pytest.raises(MemcacheUnexpectedCloseError): + _set() + + assert client.sock is None + + def test_set_unknown_error(self): + client = self.make_client([b"foobarbaz\r\n"]) + + def _set(): + client.set(b"key", b"value", noreply=False) + + with pytest.raises(MemcacheUnknownError): + _set() + + def test_set_key_with_space(self): + client = self.make_client([b""]) + + def _set(): + client.set(b"key has space", b"value", noreply=False) + + with pytest.raises(MemcacheIllegalInputError): + _set() + + def test_set_key_with_newline(self): + client = self.make_client([b""]) + + def _set(): + client.set(b"key\n", b"value", noreply=False) + + with pytest.raises(MemcacheIllegalInputError): + _set() + + def test_set_key_with_carriage_return(self): + client = self.make_client([b""]) + + def _set(): + client.set(b"key\r", b"value", noreply=False) + + with pytest.raises(MemcacheIllegalInputError): + _set() + + def test_set_key_with_null_character(self): + client = self.make_client([b""]) + + def _set(): + client.set(b"key\00", b"value", noreply=False) + + with pytest.raises(MemcacheIllegalInputError): + _set() + + def test_set_key_with_noninteger_expire(self): + client = self.make_client([b""]) + + class _OneLike: + """object that looks similar to the int 1""" + + def __str__(self): + return "1" + + for noreply in (True, False): + for expire in (1.5, _OneLike(), "1"): + + def _set(): + client.set(b"finekey", b"finevalue", noreply=noreply, expire=expire) + + with pytest.raises(MemcacheIllegalInputError): + _set() + + def test_set_many_socket_handling(self): + client = self.make_client([b"STORED\r\n"]) + result = client.set_many({b"key": b"value"}, noreply=False) + assert result == [] + assert client.sock.closed is False + assert len(client.sock.send_bufs) == 1 + + # unit test for encoding passed in __init__() + client = self.make_client([b"STORED\r\n"], encoding="utf-8") + result = client.set_many({b"key": b"value"}, noreply=False) + assert result == [] + assert client.sock.closed is False + assert len(client.sock.send_bufs) == 1 + + def test_set_many_exception(self): + client = self.make_client([b"STORED\r\n", Exception("fail")]) + + def _set(): + client.set_many({b"key": b"value", b"other": b"value"}, noreply=False) + + with pytest.raises(Exception): + _set() + + assert client.sock is None + + # unit test for encoding passed in __init__() + client = self.make_client([b"STORED\r\n", Exception("fail")], encoding="utf-8") + + def _set(): + client.set_many({b"key": b"value", b"other": b"value"}, noreply=False) + + with pytest.raises(Exception): + _set() + + assert client.sock is None + + def test_stats(self): + client = self.make_client([b"STAT fake_stats 1\r\n", b"END\r\n"]) + result = client.stats() + assert client.sock.send_bufs == [b"stats\r\n"] + assert result == {b"fake_stats": 1} + + def test_stats_with_args(self): + client = self.make_client([b"STAT fake_stats 1\r\n", b"END\r\n"]) + result = client.stats("some_arg") + assert client.sock.send_bufs == [b"stats some_arg\r\n"] + assert result == {b"fake_stats": 1} + + def test_stats_with_blank_value(self): + client = self.make_client([b"STAT fake_stats \r\n", b"END\r\n"]) + result = client.stats() + assert result == {b"fake_stats": b""} + + def test_stats_conversions(self): + client = self.make_client( + [ + # Most stats are converted to int + b"STAT cmd_get 2519\r\n", + b"STAT cmd_set 3099\r\n", + b"STAT evictions 939\r\n", + # Unless they can't be, they remain str + b"STAT libevent 2.0.19-stable\r\n", + # Some named stats are explicitly converted + b"STAT hash_is_expanding 0\r\n", + b"STAT rusage_user 0.609165\r\n", + b"STAT rusage_system 0.852791\r\n", + b"STAT slab_reassign_running 1\r\n", + b"STAT version 1.4.14\r\n", + b"STAT umask 777\r\n", + b"STAT auth_enabled_sasl yes\r\n", + b"END\r\n", + ] + ) + result = client.stats() + assert client.sock.send_bufs == [b"stats\r\n"] + expected = { + b"cmd_get": 2519, + b"cmd_set": 3099, + b"evictions": 939, + b"libevent": b"2.0.19-stable", + b"hash_is_expanding": False, + b"rusage_user": 0.609165, + b"rusage_system": 0.852791, + b"slab_reassign_running": True, + b"version": b"1.4.14", + b"umask": 0o777, + b"auth_enabled_sasl": True, + } + assert result == expected + + def test_stats_cachedump(self): + client = self.make_client([b"ITEM bob [7 b; 0 s]\r\n", b"END\r\n"]) + result = client.stats("cachedump", "1", "1") + assert client.sock.send_bufs == [b"stats cachedump 1 1\r\n"] + assert result == {b"bob": b"[7 b; 0 s]"} + + def test_cache_memlimit(self): + client = self.make_client([b"OK\r\n"]) + result = client.cache_memlimit(8) + assert client.sock.send_bufs == [b"cache_memlimit 8\r\n"] + assert result is True + + def test_python_dict_set_is_supported(self): + client = self.make_client([b"STORED\r\n"]) + client[b"key"] = b"value" + + def test_python_dict_get_is_supported(self): + client = self.make_client([b"VALUE key 0 5\r\nvalue\r\nEND\r\n"]) + assert client[b"key"] == b"value" + + def test_python_dict_get_not_found_is_supported(self): + client = self.make_client([b"END\r\n"]) + + def _get(): + client[b"key"] + + with pytest.raises(KeyError): + _get() + + def test_python_dict_del_is_supported(self): + client = self.make_client([b"DELETED\r\n"]) + del client[b"key"] + + def test_too_long_key(self): + client = self.make_client([b"END\r\n"]) + + with pytest.raises(MemcacheClientError): + client.get(b"x" * 251) + + def test_too_long_unicode_key(self): + client = self.make_client([b"STORED\r\n"], allow_unicode_keys=True) + + with pytest.raises(MemcacheClientError): + client.get("my☃" * 150) + + with pytest.raises(MemcacheClientError): + client.get("\u0FFF" * 150) + + def test_key_contains_space(self): + client = self.make_client([b"END\r\n"]) + with pytest.raises(MemcacheClientError): + client.get(b"abc xyz") + + def test_key_contains_nonascii(self): + client = self.make_client([b"END\r\n"]) + + with pytest.raises(MemcacheClientError): + client.get("\u3053\u3093\u306b\u3061\u306f") + + def _default_noreply_false(self, cmd, args, response): + client = self.make_client(response, default_noreply=False) + result = getattr(client, cmd)(*args) + assert result is False + + def _default_noreply_true(self, cmd, args, response): + client = self.make_client(response, default_noreply=True) + result = getattr(client, cmd)(*args) + assert result is True + + def _default_noreply_true_and_empty_list(self, cmd, args, response): + client = self.make_client(response, default_noreply=True) + result = getattr(client, cmd)(*args) + assert result == [] + + def test_default_noreply_set(self): + with pytest.raises(MemcacheUnknownError): + self._default_noreply_false("set", (b"key", b"value"), [b"UNKNOWN\r\n"]) + self._default_noreply_false("set", (b"key", b"value"), [b"NOT_STORED\r\n"]) + self._default_noreply_true("set", (b"key", b"value"), [b"NOT_STORED\r\n"]) + + def test_default_noreply_set_many(self): + with pytest.raises(MemcacheUnknownError): + client = self.make_client([b"UNKNOWN\r\n"], default_noreply=False) + result = client.set_many({b"key": b"value"}) + assert result == [b"key"] + self._default_noreply_true_and_empty_list( + "set_many", ({b"key": b"value"},), [b"NOT_STORED\r\n"] + ) + + def test_default_noreply_add(self): + self._default_noreply_false("add", (b"key", b"value"), [b"NOT_STORED\r\n"]) + self._default_noreply_true("add", (b"key", b"value"), [b"NOT_STORED\r\n"]) + + def test_default_noreply_replace(self): + self._default_noreply_false("replace", (b"key", b"value"), [b"NOT_STORED\r\n"]) + self._default_noreply_true("replace", (b"key", b"value"), [b"NOT_STORED\r\n"]) + + def test_default_noreply_append(self): + self._default_noreply_false("append", (b"key", b"value"), [b"NOT_STORED\r\n"]) + self._default_noreply_true("append", (b"key", b"value"), [b"NOT_STORED\r\n"]) + + def test_default_noreply_prepend(self): + self._default_noreply_false("prepend", (b"key", b"value"), [b"NOT_STORED\r\n"]) + self._default_noreply_true("prepend", (b"key", b"value"), [b"NOT_STORED\r\n"]) + + def test_default_noreply_touch(self): + self._default_noreply_false("touch", (b"key",), [b"NOT_FOUND\r\n"]) + self._default_noreply_true("touch", (b"key",), [b"NOT_FOUND\r\n"]) + + def test_default_noreply_flush_all(self): + self._default_noreply_false("flush_all", (), [b"__FAKE_RESPONSE__\r\n"]) + self._default_noreply_true("flush_all", (), [b"__FAKE_RESPONSE__\r\n"]) + + def test_version_success(self): + client = self.make_client([b"VERSION 1.2.3\r\n"], default_noreply=False) + result = client.version() + assert result == b"1.2.3" + + def test_version_exception(self): + client = self.make_client([b"INVALID DATA\r\n"], default_noreply=False) + with pytest.raises(MemcacheUnknownError): + client.version() + + def test_raw_command_default_end_tokens(self): + client = self.make_client([b"REPLY\r\n", b"REPLY\r\nLEFTOVER"]) + result = client.raw_command(b"misc") + assert result == b"REPLY" + result = client.raw_command(b"misc") + assert result == b"REPLY" + + def test_raw_command_custom_end_tokens(self): + client = self.make_client( + [ + b"REPLY\r\nEND\r\n", + b"REPLY\r\nEND\r\nLEFTOVER", + b"REPLYEND\r\nLEFTOVER", + b"REPLY\nLEFTOVER", + ] + ) + end_tokens = b"END\r\n" + result = client.raw_command(b"misc", end_tokens) + assert result == b"REPLY\r\n" + result = client.raw_command(b"misc", end_tokens) + assert result == b"REPLY\r\n" + result = client.raw_command(b"misc", end_tokens) + assert result == b"REPLY" + result = client.raw_command(b"misc", b"\n") + assert result == b"REPLY" + + def test_raw_command_missing_end_tokens(self): + client = self.make_client([b"REPLY", b"REPLY"]) + with pytest.raises(IndexError): + client.raw_command(b"misc") + with pytest.raises(IndexError): + client.raw_command(b"misc", b"END\r\n") + + def test_raw_command_empty_end_tokens(self): + client = self.make_client([b"REPLY"]) + + with pytest.raises(IndexError): + client.raw_command(b"misc", b"") + + def test_raw_command_types(self): + client = self.make_client( + [b"REPLY\r\n", b"REPLY\r\n", b"REPLY\r\nLEFTOVER", b"REPLY\r\nLEFTOVER"] + ) + assert client.raw_command("key") == b"REPLY" + assert client.raw_command(b"key") == b"REPLY" + assert client.raw_command("key") == b"REPLY" + assert client.raw_command(b"key") == b"REPLY" + + def test_send_end_token_types(self): + client = self.make_client( + [b"REPLY\r\n", b"REPLY\r\n", b"REPLY\r\nLEFTOVER", b"REPLY\r\nLEFTOVER"] + ) + assert client.raw_command("key", "\r\n") == b"REPLY" + assert client.raw_command(b"key", b"\r\n") == b"REPLY" + assert client.raw_command("key", "\r\n") == b"REPLY" + assert client.raw_command(b"key", b"\r\n") == b"REPLY" + + +@pytest.mark.unit() +class TestClientSocketConnect(unittest.TestCase): + def test_socket_connect_ipv4(self): + server = ("127.0.0.1", 11211) + + client = Client(server, socket_module=MockSocketModule()) + client._connect() + assert client.sock.connections == [server] + assert client.sock.family == socket.AF_INET + + timeout = 2 + connect_timeout = 3 + client = Client( + server, + connect_timeout=connect_timeout, + timeout=timeout, + socket_module=MockSocketModule(), + ) + client._connect() + assert client.sock.timeouts == [connect_timeout, timeout] + + client = Client(server, socket_module=MockSocketModule()) + client._connect() + assert client.sock.socket_options == [] + + client = Client(server, socket_module=MockSocketModule(), no_delay=True) + client._connect() + assert client.sock.socket_options == [ + (socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) + ] + + def test_socket_connect_ipv6(self): + server = ("::1", 11211) + + client = Client(server, socket_module=MockSocketModule()) + client._connect() + assert client.sock.connections == [server + (0, 0)] + assert client.sock.family == socket.AF_INET6 + + timeout = 2 + connect_timeout = 3 + client = Client( + server, + connect_timeout=connect_timeout, + timeout=timeout, + socket_module=MockSocketModule(), + ) + client._connect() + assert client.sock.timeouts == [connect_timeout, timeout] + + client = Client(server, socket_module=MockSocketModule()) + client._connect() + assert client.sock.socket_options == [] + + client = Client(server, socket_module=MockSocketModule(), no_delay=True) + client._connect() + assert client.sock.socket_options == [ + (socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) + ] + + def test_socket_connect_unix(self): + server = f"/tmp/pymemcache.{os.getpid()}" + + with MockUnixSocketServer(server): + client = Client(server) + client._connect() + assert client.sock.family == socket.AF_UNIX + + @unittest.skipIf( + "Linux" != platform.system(), "Socket keepalive only support Linux platforms." + ) + def test_linux_socket_keepalive(self): + server = ("::1", 11211) + try: + client = Client( + server, + socket_module=MockSocketModule(), + socket_keepalive=KeepaliveOpts(), + ) + client._connect() + except SystemError: + self.fail("SystemError unexpectedly raised") + with self.assertRaises(ValueError): + # A KeepaliveOpts object is expected, a ValueError will be raised + Client(server, socket_module=MockSocketModule(), socket_keepalive=True) + + @mock.patch("platform.system") + def test_osx_socket_keepalive(self, platform_mock): + platform_mock.return_value = "Darwin" + server = ("::1", 11211) + # For the moment the socket keepalive is only implemented for Linux + with self.assertRaises(SystemError): + Client( + server, + socket_module=MockSocketModule(), + socket_keepalive=KeepaliveOpts(), + ) + + @mock.patch("platform.system") + def test_windows_socket_keepalive(self, platform_mock): + platform_mock.return_value = "Windows" + server = ("::1", 11211) + # For the moment the socket keepalive is only implemented for Linux + with self.assertRaises(SystemError): + Client( + server, + socket_module=MockSocketModule(), + socket_keepalive=KeepaliveOpts(), + ) + + def test_socket_connect_closes_on_failure(self): + server = ("example.com", 11211) + + socket_module = MockSocketModule(connect_failure=OSError()) + client = Client(server, socket_module=socket_module) + with pytest.raises(OSError): + client._connect() + assert len(socket_module.sockets) == 1 + assert socket_module.sockets[0].connections == [] + assert socket_module.sockets[0].closed + + def test_socket_close(self): + server = ("example.com", 11211) + + client = Client(server, socket_module=MockSocketModule()) + client._connect() + assert client.sock is not None + + client.close() + assert client.sock is None + + def test_socket_close_exception(self): + server = ("example.com", 11211) + + socket_module = MockSocketModule(close_failure=OSError()) + client = Client(server, socket_module=socket_module) + client._connect() + assert client.sock is not None + + client.close() + assert client.sock is None + + +class TestPooledClient(ClientTestMixin, unittest.TestCase): + def make_client(self, mock_socket_values, **kwargs): + mock_client = Client("localhost", **kwargs) + mock_client.sock = MockSocket(list(mock_socket_values)) + client = PooledClient("localhost", **kwargs) + client.client_pool = pool.ObjectPool(lambda: mock_client) + return client + + def _default_noreply_false(self, cmd, args, response): + client = self.make_client(response, default_noreply=False) + result = getattr(client, cmd)(*args) + assert result is False + + def _default_noreply_true(self, cmd, args, response): + client = self.make_client(response, default_noreply=True) + result = getattr(client, cmd)(*args) + assert result is True + + def _default_noreply_true_and_empty_list(self, cmd, args, response): + client = self.make_client(response, default_noreply=True) + result = getattr(client, cmd)(*args) + assert result == [] + + def test_default_noreply_set(self): + with pytest.raises(MemcacheUnknownError): + self._default_noreply_false("set", (b"key", b"value"), [b"UNKNOWN\r\n"]) + self._default_noreply_false("set", (b"key", b"value"), [b"NOT_STORED\r\n"]) + self._default_noreply_true("set", (b"key", b"value"), [b"NOT_STORED\r\n"]) + + def test_default_noreply_set_many(self): + with pytest.raises(MemcacheUnknownError): + client = self.make_client([b"UNKNOWN\r\n"], default_noreply=False) + client.set_many({b"key": b"value"}) + self._default_noreply_true_and_empty_list( + "set_many", ({b"key": b"value"},), [b"NOT_STORED\r\n"] + ) + + def test_default_noreply_add(self): + self._default_noreply_false("add", (b"key", b"value"), [b"NOT_STORED\r\n"]) + self._default_noreply_true("add", (b"key", b"value"), [b"NOT_STORED\r\n"]) + + def test_default_noreply_replace(self): + self._default_noreply_false("replace", (b"key", b"value"), [b"NOT_STORED\r\n"]) + self._default_noreply_true("replace", (b"key", b"value"), [b"NOT_STORED\r\n"]) + + def test_default_noreply_append(self): + self._default_noreply_false("append", (b"key", b"value"), [b"NOT_STORED\r\n"]) + self._default_noreply_true("append", (b"key", b"value"), [b"NOT_STORED\r\n"]) + + def test_default_noreply_prepend(self): + self._default_noreply_false("prepend", (b"key", b"value"), [b"NOT_STORED\r\n"]) + self._default_noreply_true("prepend", (b"key", b"value"), [b"NOT_STORED\r\n"]) + + def test_default_noreply_touch(self): + self._default_noreply_false("touch", (b"key",), [b"NOT_FOUND\r\n"]) + self._default_noreply_true("touch", (b"key",), [b"NOT_FOUND\r\n"]) + + def test_default_noreply_flush_all(self): + self._default_noreply_false("flush_all", (), [b"__FAKE_RESPONSE__\r\n"]) + self._default_noreply_true("flush_all", (), [b"__FAKE_RESPONSE__\r\n"]) + + def test_custom_client(self): + class MyClient(Client): + pass + + client = PooledClient(("host", 11211)) + client.client_class = MyClient + assert isinstance(client.client_pool.get(), MyClient) + + +class TestPooledClientIdleTimeout(ClientTestMixin, unittest.TestCase): + def make_client(self, mock_socket_values, **kwargs): + mock_client = Client("localhost", **kwargs) + mock_client.sock = MockSocket(list(mock_socket_values)) + client = PooledClient("localhost", pool_idle_timeout=60, **kwargs) + client.client_pool = pool.ObjectPool(lambda: mock_client) + return client + + def test_free_idle(self): + class Counter: + count = 0 + + def increment(self, obj): + self.count += 1 + + removed = Counter() + + client = self.make_client([b"VALUE key 0 5\r\nvalue\r\nEND\r\n"] * 2) + client.client_pool._after_remove = removed.increment + client.client_pool._idle_clock = lambda: 0 + + client.set(b"key", b"value") + assert removed.count == 0 + client.get(b"key") + assert removed.count == 0 + + # Advance clock to beyond the idle timeout. + client.client_pool._idle_clock = lambda: 61 + client.get(b"key") + assert removed.count == 1 + + +class TestMockClient(ClientTestMixin, unittest.TestCase): + def make_client(self, mock_socket_values, **kwargs): + client = MockMemcacheClient("localhost", **kwargs) + client.sock = MockSocket(list(mock_socket_values)) + return client + + def test_get_found(self): + client = self.make_client( + [ + b"STORED\r\n", + b"VALUE key 0 5\r\nvalue\r\nEND\r\n", + ] + ) + result = client.set(b"key", b"value", noreply=False) + result = client.get(b"key") + assert result == b"value" + + def test_deserialization(self): + class JsonSerde: + def serialize(self, key, value): + if isinstance(value, dict): + return json.dumps(value).encode("UTF-8"), 1 + return value, 0 + + def deserialize(self, key, value, flags): + if flags == 1: + return json.loads(value.decode("UTF-8")) + return value + + client = self.make_client( + [ + b"STORED\r\n", + b"VALUE key1 0 5\r\nhello\r\nEND\r\n", + b"STORED\r\n", + b'VALUE key2 0 18\r\n{"hello": "world"}\r\nEND\r\n', + ], + serde=JsonSerde(), + ) + + result = client.set(b"key1", b"hello", noreply=False) + result = client.get(b"key1") + assert result == b"hello" + + result = client.set(b"key2", dict(hello="world"), noreply=False) + result = client.get(b"key2") + assert result == dict(hello="world") + + +class TestPrefixedClient(ClientTestMixin, unittest.TestCase): + def make_client(self, mock_socket_values, **kwargs): + client = Client("localhost", key_prefix=b"xyz:", **kwargs) + client.sock = MockSocket(list(mock_socket_values)) + return client + + def test_get_found(self): + client = self.make_client( + [ + b"STORED\r\n", + b"VALUE xyz:key 0 5\r\nvalue\r\nEND\r\n", + ] + ) + result = client.set(b"key", b"value", noreply=False) + result = client.get(b"key") + assert result == b"value" + + def test_get_many_some_found(self): + client = self.make_client( + [ + b"STORED\r\n", + b"VALUE xyz:key1 0 6\r\nvalue1\r\nEND\r\n", + ] + ) + result = client.set(b"key1", b"value1", noreply=False) + result = client.get_many([b"key1", b"key2"]) + assert result == {b"key1": b"value1"} + + def test_get_many_all_found(self): + client = self.make_client( + [ + b"STORED\r\n", + b"STORED\r\n", + b"VALUE xyz:key1 0 6\r\nvalue1\r\n", + b"VALUE xyz:key2 0 6\r\nvalue2\r\nEND\r\n", + ] + ) + result = client.set(b"key1", b"value1", noreply=False) + result = client.set(b"key2", b"value2", noreply=False) + result = client.get_many([b"key1", b"key2"]) + assert result == {b"key1": b"value1", b"key2": b"value2"} + + def test_python_dict_get_is_supported(self): + client = self.make_client([b"VALUE xyz:key 0 5\r\nvalue\r\nEND\r\n"]) + assert client[b"key"] == b"value" + + +class TestPrefixedPooledClient(TestPrefixedClient): + def make_client(self, mock_socket_values, **kwargs): + mock_client = Client("localhost", key_prefix=b"xyz:", **kwargs) + mock_client.sock = MockSocket(list(mock_socket_values)) + client = PooledClient("localhost", key_prefix=b"xyz:", **kwargs) + client.client_pool = pool.ObjectPool(lambda: mock_client) + return client + + +@pytest.mark.unit() +class TestRetryOnEINTR(unittest.TestCase): + def make_client(self, values): + client = Client("localhost") + client.sock = MockSocket(list(values)) + return client + + def test_recv(self): + client = self.make_client( + [ + b"VALUE ", + socket.error(errno.EINTR, "Interrupted system call"), + b"key1 0 6\r\nval", + socket.error(errno.EINTR, "Interrupted system call"), + b"ue1\r\nEND\r\n", + ] + ) + assert client[b"key1"] == b"value1" + + +@pytest.mark.unit() +class TestNormalizeServerSpec(unittest.TestCase): + def test_normalize_server_spec(self): + f = normalize_server_spec + assert f(("127.0.0.1", 12345)) == ("127.0.0.1", 12345) + assert f("unix:/run/memcached/socket") == "/run/memcached/socket" + assert f("/run/memcached/socket") == "/run/memcached/socket" + assert f("localhost") == ("localhost", 11211) + assert f("localhost:12345") == ("localhost", 12345) + assert f("[::1]") == ("::1", 11211) + assert f("[::1]:12345") == ("::1", 12345) + assert f("127.0.0.1") == ("127.0.0.1", 11211) + assert f("127.0.0.1:12345") == ("127.0.0.1", 12345) + + with pytest.raises(ValueError, match="Unsupported server specification"): + f(None) # type: ignore + + with pytest.raises(ValueError, match="Unsupported server specification"): + f({"host": 12345}) # type: ignore + + with pytest.raises(ValueError, match="Unsupported server specification"): + f(12345) # type: ignore + + +@pytest.mark.unit() +class TestKeepaliveopts(unittest.TestCase): + def test_keepalive_opts(self): + kao = KeepaliveOpts() + assert kao.idle == 1 and kao.intvl == 1 and kao.cnt == 5 + kao = KeepaliveOpts(idle=1, intvl=4, cnt=2) + assert kao.idle == 1 and kao.intvl == 4 and kao.cnt == 2 + kao = KeepaliveOpts(idle=8) + assert kao.idle == 8 and kao.intvl == 1 and kao.cnt == 5 + kao = KeepaliveOpts(cnt=8) + assert kao.idle == 1 and kao.intvl == 1 and kao.cnt == 8 + + with self.assertRaises(ValueError): + KeepaliveOpts(cnt=0) + with self.assertRaises(ValueError): + KeepaliveOpts(idle=-1) diff --git a/lib/python3.11/site-packages/pymemcache/test/test_client_hash.py b/lib/python3.11/site-packages/pymemcache/test/test_client_hash.py new file mode 100644 index 00000000..4cb066f0 --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/test/test_client_hash.py @@ -0,0 +1,519 @@ +from pymemcache.client.hash import HashClient +from pymemcache.client.base import Client, PooledClient +from pymemcache.exceptions import MemcacheError, MemcacheUnknownError +from pymemcache import pool + +from .test_client import ClientTestMixin, MockSocket +import unittest +import os +import pytest +from unittest import mock +import socket + + +class TestHashClient(ClientTestMixin, unittest.TestCase): + def make_client_pool(self, hostname, mock_socket_values, serializer=None, **kwargs): + mock_client = Client(hostname, serializer=serializer, **kwargs) + mock_client.sock = MockSocket(mock_socket_values) + client = PooledClient(hostname, serializer=serializer) + client.client_pool = pool.ObjectPool(lambda: mock_client) + return mock_client + + def make_client(self, *mock_socket_values, **kwargs): + current_port = 11012 + client = HashClient([], **kwargs) + ip = "127.0.0.1" + + for vals in mock_socket_values: + s = f"{ip}:{current_port}" + c = self.make_client_pool((ip, current_port), vals, **kwargs) + client.clients[s] = c + client.hasher.add_node(s) + current_port += 1 + + return client + + def make_unix_client(self, sockets, *mock_socket_values, **kwargs): + client = HashClient([], **kwargs) + + for socket_, vals in zip(sockets, mock_socket_values): + c = self.make_client_pool(socket_, vals, **kwargs) + client.clients[socket_] = c + client.hasher.add_node(socket_) + + return client + + def test_setup_client_without_pooling(self): + client_class = "pymemcache.client.hash.HashClient.client_class" + with mock.patch(client_class) as internal_client: + client = HashClient([], timeout=999, key_prefix="foo_bar_baz") + client.add_server(("127.0.0.1", "11211")) + + assert internal_client.call_args[0][0] == ("127.0.0.1", "11211") + kwargs = internal_client.call_args[1] + assert kwargs["timeout"] == 999 + assert kwargs["key_prefix"] == "foo_bar_baz" + + def test_get_many_unix(self): + pid = os.getpid() + sockets = [ + "/tmp/pymemcache.1.%d" % pid, + "/tmp/pymemcache.2.%d" % pid, + ] + client = self.make_unix_client( + sockets, + *[ + [ + b"STORED\r\n", + b"VALUE key3 0 6\r\nvalue2\r\nEND\r\n", + ], + [ + b"STORED\r\n", + b"VALUE key1 0 6\r\nvalue1\r\nEND\r\n", + ], + ], + ) + + def get_clients(key): + if key == b"key3": + return client.clients["/tmp/pymemcache.1.%d" % pid] + else: + return client.clients["/tmp/pymemcache.2.%d" % pid] + + client._get_client = get_clients + + result = client.set(b"key1", b"value1", noreply=False) + result = client.set(b"key3", b"value2", noreply=False) + result = client.get_many([b"key1", b"key3"]) + assert result == {b"key1": b"value1", b"key3": b"value2"} + + def test_get_many_all_found(self): + client = self.make_client( + *[ + [ + b"STORED\r\n", + b"VALUE key3 0 6\r\nvalue2\r\nEND\r\n", + ], + [ + b"STORED\r\n", + b"VALUE key1 0 6\r\nvalue1\r\nEND\r\n", + ], + ] + ) + + def get_clients(key): + if key == b"key3": + return client.clients["127.0.0.1:11012"] + else: + return client.clients["127.0.0.1:11013"] + + client._get_client = get_clients + + result = client.set(b"key1", b"value1", noreply=False) + result = client.set(b"key3", b"value2", noreply=False) + result = client.get_many([b"key1", b"key3"]) + assert result == {b"key1": b"value1", b"key3": b"value2"} + + def test_get_many_some_found(self): + client = self.make_client( + *[ + [ + b"END\r\n", + ], + [ + b"STORED\r\n", + b"VALUE key1 0 6\r\nvalue1\r\nEND\r\n", + ], + ] + ) + + def get_clients(key): + if key == b"key3": + return client.clients["127.0.0.1:11012"] + else: + return client.clients["127.0.0.1:11013"] + + client._get_client = get_clients + result = client.set(b"key1", b"value1", noreply=False) + result = client.get_many([b"key1", b"key3"]) + + assert result == {b"key1": b"value1"} + + def test_get_many_bad_server_data(self): + client = self.make_client( + *[ + [ + b"STORED\r\n", + b"VAXLUE key3 0 6\r\nvalue2\r\nEND\r\n", + ], + [ + b"STORED\r\n", + b"VAXLUE key1 0 6\r\nvalue1\r\nEND\r\n", + ], + ] + ) + + def get_clients(key): + if key == b"key3": + return client.clients["127.0.0.1:11012"] + else: + return client.clients["127.0.0.1:11013"] + + client._get_client = get_clients + + with pytest.raises(MemcacheUnknownError): + client.set(b"key1", b"value1", noreply=False) + client.set(b"key3", b"value2", noreply=False) + client.get_many([b"key1", b"key3"]) + + def test_get_many_bad_server_data_ignore(self): + client = self.make_client( + *[ + [ + b"STORED\r\n", + b"VAXLUE key3 0 6\r\nvalue2\r\nEND\r\n", + ], + [ + b"STORED\r\n", + b"VAXLUE key1 0 6\r\nvalue1\r\nEND\r\n", + ], + ], + ignore_exc=True, + ) + + def get_clients(key): + if key == b"key3": + return client.clients["127.0.0.1:11012"] + else: + return client.clients["127.0.0.1:11013"] + + client._get_client = get_clients + + client.set(b"key1", b"value1", noreply=False) + client.set(b"key3", b"value2", noreply=False) + result = client.get_many([b"key1", b"key3"]) + assert result == {} + + def test_gets_many(self): + client = self.make_client( + *[ + [ + b"STORED\r\n", + b"VALUE key3 0 6 1\r\nvalue2\r\nEND\r\n", + ], + [ + b"STORED\r\n", + b"VALUE key1 0 6 1\r\nvalue1\r\nEND\r\n", + ], + ] + ) + + def get_clients(key): + if key == b"key3": + return client.clients["127.0.0.1:11012"] + else: + return client.clients["127.0.0.1:11013"] + + client._get_client = get_clients + + assert client.set(b"key1", b"value1", noreply=False) is True + assert client.set(b"key3", b"value2", noreply=False) is True + result = client.gets_many([b"key1", b"key3"]) + assert result == {b"key1": (b"value1", b"1"), b"key3": (b"value2", b"1")} + + def test_touch_not_found(self): + client = self.make_client([b"NOT_FOUND\r\n"]) + result = client.touch(b"key", noreply=False) + assert result is False + + def test_touch_no_expiry_found(self): + client = self.make_client([b"TOUCHED\r\n"]) + result = client.touch(b"key", noreply=False) + assert result is True + + def test_touch_with_expiry_found(self): + client = self.make_client([b"TOUCHED\r\n"]) + result = client.touch(b"key", 1, noreply=False) + assert result is True + + def test_close(self): + client = self.make_client([]) + assert all(c.sock is not None for c in client.clients.values()) + result = client.close() + assert result is None + assert all(c.sock is None for c in client.clients.values()) + + def test_quit(self): + client = self.make_client([]) + assert all(c.sock is not None for c in client.clients.values()) + result = client.quit() + assert result is None + assert all(c.sock is None for c in client.clients.values()) + + def test_no_servers_left(self): + from pymemcache.client.hash import HashClient + + client = HashClient( + [], use_pooling=True, ignore_exc=True, timeout=1, connect_timeout=1 + ) + + hashed_client = client._get_client("foo") + assert hashed_client is None + + def test_no_servers_left_raise_exception(self): + from pymemcache.client.hash import HashClient + + client = HashClient( + [], use_pooling=True, ignore_exc=False, timeout=1, connect_timeout=1 + ) + + with pytest.raises(MemcacheError) as e: + client._get_client("foo") + + assert str(e.value) == "All servers seem to be down right now" + + def test_unavailable_servers_zero_retry_raise_exception(self): + from pymemcache.client.hash import HashClient + + client = HashClient( + [("example.com", 11211)], + use_pooling=True, + ignore_exc=False, + retry_attempts=0, + timeout=1, + connect_timeout=1, + ) + + with pytest.raises(socket.error): + client.get("foo") + + def test_no_servers_left_with_commands_return_default_value(self): + from pymemcache.client.hash import HashClient + + client = HashClient( + [], use_pooling=True, ignore_exc=True, timeout=1, connect_timeout=1 + ) + + result = client.get("foo") + assert result is None + result = client.get("foo", default="default") + assert result == "default" + result = client.set("foo", "bar") + assert result is False + + def test_no_servers_left_return_positional_default(self): + from pymemcache.client.hash import HashClient + + client = HashClient( + [], use_pooling=True, ignore_exc=True, timeout=1, connect_timeout=1 + ) + + # Ensure compatibility with clients that pass the default as a + # positional argument + result = client.get("foo", "default") + assert result == "default" + + def test_no_servers_left_with_set_many(self): + from pymemcache.client.hash import HashClient + + client = HashClient( + [], use_pooling=True, ignore_exc=True, timeout=1, connect_timeout=1 + ) + + result = client.set_many({"foo": "bar"}) + assert result == ["foo"] + + def test_no_servers_left_with_get_many(self): + from pymemcache.client.hash import HashClient + + client = HashClient( + [], use_pooling=True, ignore_exc=True, timeout=1, connect_timeout=1 + ) + + result = client.get_many(["foo", "bar"]) + assert result == {} + + def test_ignore_exec_set_many(self): + values = {"key1": "value1", "key2": "value2", "key3": "value3"} + + with pytest.raises(MemcacheUnknownError): + client = self.make_client( + *[ + [b"STORED\r\n", b"UNKNOWN\r\n", b"STORED\r\n"], + [b"STORED\r\n", b"UNKNOWN\r\n", b"STORED\r\n"], + ] + ) + client.set_many(values, noreply=False) + + client = self.make_client( + *[ + [b"STORED\r\n", b"UNKNOWN\r\n", b"STORED\r\n"], + ], + ignore_exc=True, + ) + result = client.set_many(values, noreply=False) + + assert len(result) == 0 + + def test_noreply_set_many(self): + values = {"key1": "value1", "key2": "value2", "key3": "value3"} + + client = self.make_client( + *[ + [b"STORED\r\n", b"NOT_STORED\r\n", b"STORED\r\n"], + ] + ) + result = client.set_many(values, noreply=False) + assert len(result) == 1 + + client = self.make_client( + *[ + [b"STORED\r\n", b"NOT_STORED\r\n", b"STORED\r\n"], + ] + ) + result = client.set_many(values, noreply=True) + assert result == [] + + def test_noreply_flush(self): + client = self.make_client() + client.flush_all(noreply=True) + + def test_set_many_unix(self): + values = {"key1": "value1", "key2": "value2", "key3": "value3"} + + pid = os.getpid() + sockets = ["/tmp/pymemcache.%d" % pid] + client = self.make_unix_client( + sockets, + *[ + [b"STORED\r\n", b"NOT_STORED\r\n", b"STORED\r\n"], + ], + ) + + result = client.set_many(values, noreply=False) + assert len(result) == 1 + + def test_server_encoding_pooled(self): + """ + test passed encoding from hash client to pooled clients + """ + encoding = "utf8" + from pymemcache.client.hash import HashClient + + hash_client = HashClient( + [("example.com", 11211)], use_pooling=True, encoding=encoding + ) + + for client in hash_client.clients.values(): + assert client.encoding == encoding + + def test_server_encoding_client(self): + """ + test passed encoding from hash client to clients + """ + encoding = "utf8" + from pymemcache.client.hash import HashClient + + hash_client = HashClient([("example.com", 11211)], encoding=encoding) + + for client in hash_client.clients.values(): + assert client.encoding == encoding + + @mock.patch("pymemcache.client.hash.HashClient.client_class") + def test_dead_server_comes_back(self, client_patch): + client = HashClient([], dead_timeout=0, retry_attempts=0) + client.add_server(("127.0.0.1", 11211)) + + test_client = client_patch.return_value + test_client.server = ("127.0.0.1", 11211) + + test_client.get.side_effect = socket.timeout() + with pytest.raises(socket.timeout): + client.get(b"key", noreply=False) + # Client gets removed because of socket timeout + assert ("127.0.0.1", 11211) in client._dead_clients + + test_client.get.side_effect = lambda *_, **_kw: "Some value" + # Client should be retried and brought back + assert client.get(b"key") == "Some value" + assert ("127.0.0.1", 11211) not in client._dead_clients + + @mock.patch("pymemcache.client.hash.HashClient.client_class") + def test_failed_is_retried(self, client_patch): + client = HashClient([], retry_attempts=1, retry_timeout=0) + client.add_server(("127.0.0.1", 11211)) + + assert client_patch.call_count == 1 + + test_client = client_patch.return_value + test_client.server = ("127.0.0.1", 11211) + + test_client.get.side_effect = socket.timeout() + with pytest.raises(socket.timeout): + client.get(b"key", noreply=False) + + test_client.get.side_effect = lambda *_, **_kw: "Some value" + assert client.get(b"key") == "Some value" + + assert client_patch.call_count == 1 + + def test_custom_client(self): + class MyClient(Client): + pass + + client = HashClient([]) + client.client_class = MyClient + client.add_server(("host", 11211)) + assert isinstance(client.clients["host:11211"], MyClient) + + def test_custom_client_with_pooling(self): + class MyClient(Client): + pass + + client = HashClient([], use_pooling=True) + client.client_class = MyClient + client.add_server(("host", 11211)) + assert isinstance(client.clients["host:11211"], PooledClient) + + pool = client.clients["host:11211"].client_pool + with pool.get_and_release(destroy_on_fail=True) as c: + assert isinstance(c, MyClient) + + def test_mixed_inet_and_unix_sockets(self): + expected = { + f"/tmp/pymemcache.{os.getpid()}", + ("127.0.0.1", 11211), + ("::1", 11211), + } + client = HashClient( + [ + f"/tmp/pymemcache.{os.getpid()}", + "127.0.0.1", + "127.0.0.1:11211", + "[::1]", + "[::1]:11211", + ("127.0.0.1", 11211), + ("::1", 11211), + ] + ) + assert expected == {c.server for c in client.clients.values()} + + def test_legacy_add_remove_server_signature(self): + server = ("127.0.0.1", 11211) + client = HashClient([]) + assert client.clients == {} + client.add_server(*server) # Unpack (host, port) tuple. + assert ("%s:%s" % server) in client.clients + client._mark_failed_server(server) + assert server in client._failed_clients + client.remove_server(*server) # Unpack (host, port) tuple. + assert server in client._dead_clients + assert server not in client._failed_clients + + # Ensure that server is a string if passing port argument: + with pytest.raises(TypeError): + client.add_server(server, server[-1]) + with pytest.raises(TypeError): + client.remove_server(server, server[-1]) + + # TODO: Test failover logic diff --git a/lib/python3.11/site-packages/pymemcache/test/test_client_retry.py b/lib/python3.11/site-packages/pymemcache/test/test_client_retry.py new file mode 100644 index 00000000..293270fe --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/test/test_client_retry.py @@ -0,0 +1,286 @@ +""" Test collection for the RetryingClient. """ + +import functools +import unittest +from unittest import mock + +import pytest + +from .test_client import ClientTestMixin, MockSocket +from pymemcache.client.retrying import RetryingClient +from pymemcache.client.base import Client +from pymemcache.exceptions import MemcacheUnknownError, MemcacheClientError + + +# Test pure passthroughs with no retry action. +class TestRetryingClientPassthrough(ClientTestMixin, unittest.TestCase): + def make_base_client(self, mock_socket_values, **kwargs): + base_client = Client("localhost", **kwargs) + # mock out client._connect() rather than hard-setting client.sock to + # ensure methods are checking whether self.sock is None before + # attempting to use it + sock = MockSocket(list(mock_socket_values)) + base_client._connect = mock.Mock( + side_effect=functools.partial(setattr, base_client, "sock", sock) + ) + return base_client + + def make_client(self, mock_socket_values, **kwargs): + # Create a base client to wrap. + base_client = self.make_base_client( + mock_socket_values=mock_socket_values, **kwargs + ) + + # Wrap the client in the retrying class, disable retries. + client = RetryingClient(base_client, attempts=1) + return client + + +# Retry specific tests. +@pytest.mark.unit() +class TestRetryingClient(object): + def make_base_client(self, mock_socket_values, **kwargs): + """Creates a regular mock client to wrap in the RetryClient.""" + base_client = Client("localhost", **kwargs) + # mock out client._connect() rather than hard-setting client.sock to + # ensure methods are checking whether self.sock is None before + # attempting to use it + sock = MockSocket(list(mock_socket_values)) + base_client._connect = mock.Mock( + side_effect=functools.partial(setattr, base_client, "sock", sock) + ) + return base_client + + def make_client(self, mock_socket_values, **kwargs): + """ + Creates a RetryingClient that will respond with the given values, + configured using kwargs. + """ + # Create a base client to wrap. + base_client = self.make_base_client(mock_socket_values=mock_socket_values) + + # Wrap the client in the retrying class, and pass kwargs on. + client = RetryingClient(base_client, **kwargs) + return client + + # Start testing. + def test_constructor_default(self): + base_client = self.make_base_client([]) + RetryingClient(base_client) + + with pytest.raises(TypeError): + RetryingClient() + + def test_constructor_attempts(self): + base_client = self.make_base_client([]) + rc = RetryingClient(base_client, attempts=1) + assert rc._attempts == 1 + + with pytest.raises(ValueError): + RetryingClient(base_client, attempts=0) + + def test_constructor_retry_for(self): + base_client = self.make_base_client([]) + + # Try none/default. + rc = RetryingClient(base_client, retry_for=None) + assert rc._retry_for == tuple() + + # Try with tuple. + rc = RetryingClient(base_client, retry_for=tuple([Exception])) + assert rc._retry_for == tuple([Exception]) + + # Try with list. + rc = RetryingClient(base_client, retry_for=[Exception]) + assert rc._retry_for == tuple([Exception]) + + # Try with multi element list. + rc = RetryingClient(base_client, retry_for=[Exception, IOError]) + assert rc._retry_for == (Exception, IOError) + + # With string? + with pytest.raises(ValueError): + RetryingClient(base_client, retry_for="haha!") + + # With collection of string and exceptions? + with pytest.raises(ValueError): + RetryingClient(base_client, retry_for=[Exception, str]) + + def test_constructor_do_no_retry_for(self): + base_client = self.make_base_client([]) + + # Try none/default. + rc = RetryingClient(base_client, do_not_retry_for=None) + assert rc._do_not_retry_for == tuple() + + # Try with tuple. + rc = RetryingClient(base_client, do_not_retry_for=tuple([Exception])) + assert rc._do_not_retry_for == tuple([Exception]) + + # Try with list. + rc = RetryingClient(base_client, do_not_retry_for=[Exception]) + assert rc._do_not_retry_for == tuple([Exception]) + + # Try with multi element list. + rc = RetryingClient(base_client, do_not_retry_for=[Exception, IOError]) + assert rc._do_not_retry_for == (Exception, IOError) + + # With string? + with pytest.raises(ValueError): + RetryingClient(base_client, do_not_retry_for="haha!") + + # With collection of string and exceptions? + with pytest.raises(ValueError): + RetryingClient(base_client, do_not_retry_for=[Exception, str]) + + def test_constructor_both_filters(self): + base_client = self.make_base_client([]) + + # Try none/default. + rc = RetryingClient(base_client, retry_for=None, do_not_retry_for=None) + assert rc._retry_for == tuple() + assert rc._do_not_retry_for == tuple() + + # Try a valid config. + rc = RetryingClient( + base_client, + retry_for=[Exception, IOError], + do_not_retry_for=[ValueError, MemcacheUnknownError], + ) + assert rc._retry_for == (Exception, IOError) + assert rc._do_not_retry_for == (ValueError, MemcacheUnknownError) + + # Try with overlapping filters + with pytest.raises(ValueError): + rc = RetryingClient( + base_client, + retry_for=[Exception, IOError, MemcacheUnknownError], + do_not_retry_for=[ValueError, MemcacheUnknownError], + ) + + def test_dir_passthrough(self): + base = self.make_base_client([]) + client = RetryingClient(base) + + assert dir(base) == dir(client) + + def test_retry_dict_set_is_supported(self): + client = self.make_client([b"UNKNOWN\r\n", b"STORED\r\n"]) + client[b"key"] = b"value" + + def test_retry_dict_get_is_supported(self): + client = self.make_client( + [b"UNKNOWN\r\n", b"VALUE key 0 5\r\nvalue\r\nEND\r\n"] + ) + assert client[b"key"] == b"value" + + def test_retry_dict_get_not_found_is_supported(self): + client = self.make_client([b"UNKNOWN\r\n", b"END\r\n"]) + + with pytest.raises(KeyError): + client[b"key"] + + def test_retry_dict_del_is_supported(self): + client = self.make_client([b"UNKNOWN\r\n", b"DELETED\r\n"]) + del client[b"key"] + + def test_retry_get_found(self): + client = self.make_client( + [b"UNKNOWN\r\n", b"VALUE key 0 5\r\nvalue\r\nEND\r\n"], attempts=2 + ) + result = client.get("key") + assert result == b"value" + + def test_retry_get_not_found(self): + client = self.make_client([b"UNKNOWN\r\n", b"END\r\n"], attempts=2) + result = client.get("key") + assert result is None + + def test_retry_get_exception(self): + client = self.make_client([b"UNKNOWN\r\n", b"UNKNOWN\r\n"], attempts=2) + with pytest.raises(MemcacheUnknownError): + client.get("key") + + def test_retry_set_success(self): + client = self.make_client([b"UNKNOWN\r\n", b"STORED\r\n"], attempts=2) + result = client.set("key", "value", noreply=False) + assert result is True + + def test_retry_set_fail(self): + client = self.make_client( + [b"UNKNOWN\r\n", b"UNKNOWN\r\n", b"STORED\r\n"], attempts=2 + ) + with pytest.raises(MemcacheUnknownError): + client.set("key", "value", noreply=False) + + def test_no_retry(self): + client = self.make_client( + [b"UNKNOWN\r\n", b"VALUE key 0 5\r\nvalue\r\nEND\r\n"], attempts=1 + ) + + with pytest.raises(MemcacheUnknownError): + client.get("key") + + def test_retry_for_exception_success(self): + # Test that we retry for the exception specified. + client = self.make_client( + [MemcacheClientError("Whoops."), b"VALUE key 0 5\r\nvalue\r\nEND\r\n"], + attempts=2, + retry_for=tuple([MemcacheClientError]), + ) + result = client.get("key") + assert result == b"value" + + def test_retry_for_exception_fail(self): + # Test that we do not retry for unapproved exception. + client = self.make_client( + [MemcacheUnknownError("Whoops."), b"VALUE key 0 5\r\nvalue\r\nEND\r\n"], + attempts=2, + retry_for=tuple([MemcacheClientError]), + ) + + with pytest.raises(MemcacheUnknownError): + client.get("key") + + def test_do_not_retry_for_exception_success(self): + # Test that we retry for exceptions not specified. + client = self.make_client( + [MemcacheClientError("Whoops."), b"VALUE key 0 5\r\nvalue\r\nEND\r\n"], + attempts=2, + do_not_retry_for=tuple([MemcacheUnknownError]), + ) + result = client.get("key") + assert result == b"value" + + def test_do_not_retry_for_exception_fail(self): + # Test that we do not retry for the exception specified. + client = self.make_client( + [MemcacheClientError("Whoops."), b"VALUE key 0 5\r\nvalue\r\nEND\r\n"], + attempts=2, + do_not_retry_for=tuple([MemcacheClientError]), + ) + + with pytest.raises(MemcacheClientError): + client.get("key") + + def test_both_exception_filters(self): + # Test interaction between both exception filters. + client = self.make_client( + [ + MemcacheClientError("Whoops."), + b"VALUE key 0 5\r\nvalue\r\nEND\r\n", + MemcacheUnknownError("Whoops."), + b"VALUE key 0 5\r\nvalue\r\nEND\r\n", + ], + attempts=2, + retry_for=tuple([MemcacheClientError]), + do_not_retry_for=tuple([MemcacheUnknownError]), + ) + + # Check that we succeed where allowed. + result = client.get("key") + assert result == b"value" + + # Check that no retries are attempted for the banned exception. + with pytest.raises(MemcacheUnknownError): + client.get("key") diff --git a/lib/python3.11/site-packages/pymemcache/test/test_compression.py b/lib/python3.11/site-packages/pymemcache/test/test_compression.py new file mode 100644 index 00000000..7ec6641d --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/test/test_compression.py @@ -0,0 +1,220 @@ +from pymemcache.client.base import Client +from pymemcache.serde import ( + CompressedSerde, + pickle_serde, +) + +from faker import Faker + +import pytest +import random +import string +import time +import zstd # type: ignore +import zlib + +fake = Faker(["it_IT", "en_US", "ja_JP"]) + + +def get_random_string(length): + letters = string.ascii_letters + chars = string.punctuation + digits = string.digits + total = letters + chars + digits + result_str = "".join(random.choice(total) for i in range(length)) + return result_str + + +class CustomObject: + """ + Custom class for verifying serialization + """ + + def __init__(self): + self.number = random.randint(0, 100) + self.string = fake.text() + self.object = fake.profile() + + +class CustomObjectValue: + def __init__(self, value): + self.value = value + + +def benchmark(count, func, *args, **kwargs): + start = time.time() + + for _ in range(count): + result = func(*args, **kwargs) + + duration = time.time() - start + print(str(duration)) + + return result + + +@pytest.fixture(scope="session") +def names(): + names = [] + for _ in range(15): + names.append(fake.name()) + + return names + + +@pytest.fixture(scope="session") +def paragraphs(): + paragraphs = [] + for _ in range(15): + paragraphs.append(fake.text()) + + return paragraphs + + +@pytest.fixture(scope="session") +def objects(): + objects = [] + for _ in range(15): + objects.append(CustomObject()) + + return objects + + +# Always run compression for the benchmarks +min_compress_len = 1 + +default_serde = CompressedSerde(min_compress_len=min_compress_len) + +zlib_serde = CompressedSerde( + compress=lambda value: zlib.compress(value, 9), + decompress=lambda value: zlib.decompress(value), + min_compress_len=min_compress_len, +) + +zstd_serde = CompressedSerde( + compress=lambda value: zstd.compress(value), + decompress=lambda value: zstd.decompress(value), + min_compress_len=min_compress_len, +) + +serializers = [ + None, + default_serde, + zlib_serde, + zstd_serde, +] +ids = ["none", "zlib ", "zlib9", "zstd "] + + +@pytest.mark.benchmark() +@pytest.mark.parametrize("serde", serializers, ids=ids) +def test_bench_compress_set_strings(count, host, port, serde, names): + client = Client((host, port), serde=serde, encoding="utf-8") + + def test(): + for index, name in enumerate(names): + key = f"name_{index}" + client.set(key, name) + + benchmark(count, test) + + +@pytest.mark.benchmark() +@pytest.mark.parametrize("serde", serializers, ids=ids) +def test_bench_compress_get_strings(count, host, port, serde, names): + client = Client((host, port), serde=serde, encoding="utf-8") + for index, name in enumerate(names): + key = f"name_{index}" + client.set(key, name) + + def test(): + for index, _ in enumerate(names): + key = f"name_{index}" + client.get(key) + + benchmark(count, test) + + +@pytest.mark.benchmark() +@pytest.mark.parametrize("serde", serializers, ids=ids) +def test_bench_compress_set_large_strings(count, host, port, serde, paragraphs): + client = Client((host, port), serde=serde, encoding="utf-8") + + def test(): + for index, p in enumerate(paragraphs): + key = f"paragraph_{index}" + client.set(key, p) + + benchmark(count, test) + + +@pytest.mark.benchmark() +@pytest.mark.parametrize("serde", serializers, ids=ids) +def test_bench_compress_get_large_strings(count, host, port, serde, paragraphs): + client = Client((host, port), serde=serde, encoding="utf-8") + for index, p in enumerate(paragraphs): + key = f"paragraphs_{index}" + client.set(key, p) + + def test(): + for index, _ in enumerate(paragraphs): + key = f"paragraphs_{index}" + client.get(key) + + benchmark(count, test) + + +@pytest.mark.benchmark() +@pytest.mark.parametrize("serde", serializers, ids=ids) +def test_bench_compress_set_objects(count, host, port, serde, objects): + client = Client((host, port), serde=serde, encoding="utf-8") + + def test(): + for index, o in enumerate(objects): + key = f"objects_{index}" + client.set(key, o) + + benchmark(count, test) + + +@pytest.mark.benchmark() +@pytest.mark.parametrize("serde", serializers, ids=ids) +def test_bench_compress_get_objects(count, host, port, serde, objects): + client = Client((host, port), serde=serde, encoding="utf-8") + for index, o in enumerate(objects): + key = f"objects_{index}" + client.set(key, o) + + def test(): + for index, _ in enumerate(objects): + key = f"objects_{index}" + client.get(key) + + benchmark(count, test) + + +@pytest.mark.benchmark() +def test_optimal_compression_length(): + for length in range(5, 2000): + input_data = get_random_string(length) + start = len(input_data) + + for index, serializer in enumerate(serializers[1:]): + name = ids[index + 1] + value, _ = serializer.serialize("foo", input_data) + end = len(value) + print(f"serializer={name}\t start={start}\t end={end}") + + +@pytest.mark.benchmark() +def test_optimal_compression_length_objects(): + for length in range(5, 2000): + input_data = get_random_string(length) + obj = CustomObjectValue(input_data) + start = len(pickle_serde.serialize("foo", obj)[0]) + + for index, serializer in enumerate(serializers[1:]): + name = ids[index + 1] + value, _ = serializer.serialize("foo", obj) + end = len(value) + print(f"serializer={name}\t start={start}\t end={end}") diff --git a/lib/python3.11/site-packages/pymemcache/test/test_integration.py b/lib/python3.11/site-packages/pymemcache/test/test_integration.py new file mode 100644 index 00000000..19d04f20 --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/test/test_integration.py @@ -0,0 +1,441 @@ +# Copyright 2012 Pinterest.com +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import json +from collections import defaultdict + +import pytest +from pymemcache.client.base import Client +from pymemcache.exceptions import ( + MemcacheClientError, + MemcacheIllegalInputError, + MemcacheServerError, +) +from pymemcache.serde import PickleSerde, compressed_serde, pickle_serde + + +def get_set_helper(client, key, value, key2, value2): + result = client.get(key) + assert result is None + + client.set(key, value, noreply=False) + result = client.get(key) + assert result == value + + client.set(key2, value2, noreply=True) + result = client.get(key2) + assert result == value2 + + result = client.get_many([key, key2]) + assert result == {key: value, key2: value2} + + result = client.get_many([]) + assert result == {} + + +@pytest.mark.integration() +@pytest.mark.parametrize( + "serde", + [ + pickle_serde, + compressed_serde, + ], +) +def test_get_set(client_class, host, port, serde, socket_module, key_prefix): + client = client_class( + (host, port), serde=serde, socket_module=socket_module, key_prefix=key_prefix + ) + client.flush_all() + + key = b"key" + value = b"value" + key2 = b"key2" + value2 = b"value2" + get_set_helper(client, key, value, key2, value2) + + +@pytest.mark.integration() +@pytest.mark.parametrize( + "serde", + [ + pickle_serde, + compressed_serde, + ], +) +def test_get_set_unicode_key( + client_class, host, port, serde, socket_module, key_prefix +): + client = client_class( + (host, port), + serde=serde, + socket_module=socket_module, + allow_unicode_keys=True, + key_prefix=key_prefix, + ) + client.flush_all() + + key = "こんにちは" + value = b"hello" + key2 = "my☃" + value2 = b"value2" + get_set_helper(client, key, value, key2, value2) + + +@pytest.mark.integration() +@pytest.mark.parametrize( + "serde", + [ + pickle_serde, + compressed_serde, + ], +) +def test_add_replace(client_class, host, port, serde, socket_module, key_prefix): + client = client_class( + (host, port), serde=serde, socket_module=socket_module, key_prefix=key_prefix + ) + client.flush_all() + + result = client.add(b"key", b"value", noreply=False) + assert result is True + result = client.get(b"key") + assert result == b"value" + + result = client.add(b"key", b"value2", noreply=False) + assert result is False + result = client.get(b"key") + assert result == b"value" + + result = client.replace(b"key1", b"value1", noreply=False) + assert result is False + result = client.get(b"key1") + assert result is None + + result = client.replace(b"key", b"value2", noreply=False) + assert result is True + result = client.get(b"key") + assert result == b"value2" + + +@pytest.mark.integration() +def test_append_prepend(client_class, host, port, socket_module, key_prefix): + client = client_class( + (host, port), socket_module=socket_module, key_prefix=key_prefix + ) + client.flush_all() + + result = client.append(b"key", b"value", noreply=False) + assert result is False + result = client.get(b"key") + assert result is None + + result = client.set(b"key", b"value", noreply=False) + assert result is True + result = client.append(b"key", b"after", noreply=False) + assert result is True + result = client.get(b"key") + assert result == b"valueafter" + + result = client.prepend(b"key1", b"value", noreply=False) + assert result is False + result = client.get(b"key1") + assert result is None + + result = client.prepend(b"key", b"before", noreply=False) + assert result is True + result = client.get(b"key") + assert result == b"beforevalueafter" + + +@pytest.mark.integration() +def test_cas(client_class, host, port, socket_module, key_prefix): + client = client_class( + (host, port), socket_module=socket_module, key_prefix=key_prefix + ) + client.flush_all() + result = client.cas(b"key", b"value", b"1", noreply=False) + assert result is None + + result = client.set(b"key", b"value", noreply=False) + assert result is True + + # binary, string, and raw int all match -- should all be encoded as b'1' + result = client.cas(b"key", b"value", b"1", noreply=False) + assert result is False + result = client.cas(b"key", b"value", "1", noreply=False) + assert result is False + result = client.cas(b"key", b"value", 1, noreply=False) + assert result is False + + result, cas = client.gets(b"key") + assert result == b"value" + + result = client.cas(b"key", b"value1", cas, noreply=False) + assert result is True + + result = client.cas(b"key", b"value2", cas, noreply=False) + assert result is False + + +@pytest.mark.integration() +def test_gets(client_class, host, port, socket_module, key_prefix): + client = client_class( + (host, port), socket_module=socket_module, key_prefix=key_prefix + ) + client.flush_all() + + result = client.gets(b"key") + assert result == (None, None) + + result = client.set(b"key", b"value", noreply=False) + assert result is True + result = client.gets(b"key") + assert result[0] == b"value" + + +@pytest.mark.integration() +def test_delete(client_class, host, port, socket_module, key_prefix): + client = client_class( + (host, port), socket_module=socket_module, key_prefix=key_prefix + ) + client.flush_all() + + result = client.delete(b"key", noreply=False) + assert result is False + + result = client.get(b"key") + assert result is None + result = client.set(b"key", b"value", noreply=False) + assert result is True + result = client.delete(b"key", noreply=False) + assert result is True + result = client.get(b"key") + assert result is None + + +@pytest.mark.integration() +def test_incr_decr(client_class, host, port, socket_module, key_prefix): + client = Client((host, port), socket_module=socket_module, key_prefix=key_prefix) + client.flush_all() + + result = client.incr(b"key", 1, noreply=False) + assert result is None + + result = client.set(b"key", b"0", noreply=False) + assert result is True + result = client.incr(b"key", 1, noreply=False) + assert result == 1 + + def _bad_int(): + client.incr(b"key", b"foobar") + + with pytest.raises(MemcacheClientError): + _bad_int() + + result = client.decr(b"key1", 1, noreply=False) + assert result is None + + result = client.decr(b"key", 1, noreply=False) + assert result == 0 + result = client.get(b"key") + assert result == b"0" + + +@pytest.mark.integration() +def test_touch(client_class, host, port, socket_module, key_prefix): + client = client_class( + (host, port), socket_module=socket_module, key_prefix=key_prefix + ) + client.flush_all() + + result = client.touch(b"key", noreply=False) + assert result is False + + result = client.set(b"key", b"0", 1, noreply=False) + assert result is True + + result = client.touch(b"key", noreply=False) + assert result is True + + result = client.touch(b"key", 1, noreply=False) + assert result is True + + +@pytest.mark.integration() +def test_misc(client_class, host, port, socket_module, key_prefix): + client = Client((host, port), socket_module=socket_module, key_prefix=key_prefix) + client.flush_all() + + # Ensure no exceptions are thrown + client.stats("cachedump", "1", "1") + + success = client.cache_memlimit(50) + assert success + + +@pytest.mark.integration() +def test_serialization_deserialization(host, port, socket_module): + class JsonSerde: + def serialize(self, key, value): + return json.dumps(value).encode("ascii"), 1 + + def deserialize(self, key, value, flags): + if flags == 1: + return json.loads(value.decode("ascii")) + return value + + client = Client((host, port), serde=JsonSerde(), socket_module=socket_module) + client.flush_all() + + value = {"a": "b", "c": ["d"]} + client.set(b"key", value) + result = client.get(b"key") + assert result == value + + +def serde_serialization_helper(client_class, host, port, socket_module, serde): + def check(value): + client.set(b"key", value, noreply=False) + result = client.get(b"key") + assert result == value + assert type(result) is type(value) + + client = client_class((host, port), serde=serde, socket_module=socket_module) + client.flush_all() + + check(b"byte string") + check("unicode string") + check("olé") + check("olé") + check(1) + check(123123123123123123123) + check({"a": "pickle"}) + check(["one pickle", "two pickle"]) + testdict = defaultdict(int) + testdict["one pickle"] + testdict[b"two pickle"] + check(testdict) + + +@pytest.mark.integration() +@pytest.mark.parametrize( + "serde", + [ + pickle_serde, + compressed_serde, + ], +) +def test_serde_serialization(client_class, host, port, socket_module, serde): + serde_serialization_helper(client_class, host, port, socket_module, serde) + + +@pytest.mark.integration() +def test_serde_serialization0(client_class, host, port, socket_module): + serde_serialization_helper( + client_class, host, port, socket_module, PickleSerde(pickle_version=0) + ) + + +@pytest.mark.integration() +def test_serde_serialization2(client_class, host, port, socket_module): + serde_serialization_helper( + client_class, host, port, socket_module, PickleSerde(pickle_version=2) + ) + + +@pytest.mark.integration() +def test_errors(client_class, host, port, socket_module): + client = client_class((host, port), socket_module=socket_module) + client.flush_all() + + def _key_with_ws(): + client.set(b"key with spaces", b"value", noreply=False) + + with pytest.raises(MemcacheIllegalInputError): + _key_with_ws() + + def _key_with_illegal_carriage_return(): + client.set(b"\r\nflush_all", b"value", noreply=False) + + with pytest.raises(MemcacheIllegalInputError): + _key_with_illegal_carriage_return() + + def _key_too_long(): + client.set(b"x" * 1024, b"value", noreply=False) + + with pytest.raises(MemcacheClientError): + _key_too_long() + + def _unicode_key_in_set(): + client.set("\u0FFF", b"value", noreply=False) + + with pytest.raises(MemcacheClientError): + _unicode_key_in_set() + + def _unicode_key_in_get(): + client.get("\u0FFF") + + with pytest.raises(MemcacheClientError): + _unicode_key_in_get() + + def _unicode_value_in_set(): + client.set(b"key", "\u0FFF", noreply=False) + + with pytest.raises(MemcacheClientError): + _unicode_value_in_set() + + +@pytest.mark.skip("https://github.com/pinterest/pymemcache/issues/39") +@pytest.mark.integration() +def test_tls(client_class, tls_host, tls_port, socket_module, tls_context): + client = client_class( + (tls_host, tls_port), socket_module=socket_module, tls_context=tls_context + ) + client.flush_all() + + key = b"key" + value = b"value" + key2 = b"key2" + value2 = b"value2" + get_set_helper(client, key, value, key2, value2) + + +@pytest.mark.integration() +@pytest.mark.parametrize( + "serde,should_fail", + [ + (pickle_serde, True), + (compressed_serde, False), + ], +) +def test_get_set_large( + client_class, + host, + port, + serde, + socket_module, + should_fail, +): + client = client_class((host, port), serde=serde, socket_module=socket_module) + client.flush_all() + + key = b"key" + value = b"value" * 1024 * 1024 + key2 = b"key2" + value2 = b"value2" * 1024 * 1024 + + if should_fail: + with pytest.raises(MemcacheServerError): + get_set_helper(client, key, value, key2, value2) + else: + get_set_helper(client, key, value, key2, value2) diff --git a/lib/python3.11/site-packages/pymemcache/test/test_rendezvous.py b/lib/python3.11/site-packages/pymemcache/test/test_rendezvous.py new file mode 100644 index 00000000..2582df5a --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/test/test_rendezvous.py @@ -0,0 +1,203 @@ +from pymemcache.client.rendezvous import RendezvousHash +import pytest + + +@pytest.mark.unit() +def test_init_no_options(): + rendezvous = RendezvousHash() + assert 0 == len(rendezvous.nodes) + assert 1361238019 == rendezvous.hash_function("6666") + + +@pytest.mark.unit() +def test_init(): + nodes = ["0", "1", "2"] + rendezvous = RendezvousHash(nodes=nodes) + assert 3 == len(rendezvous.nodes) + assert 1361238019 == rendezvous.hash_function("6666") + + +@pytest.mark.unit() +def test_seed(): + rendezvous = RendezvousHash(seed=10) + assert 2981722772 == rendezvous.hash_function("6666") + + +@pytest.mark.unit() +def test_add_node(): + rendezvous = RendezvousHash() + rendezvous.add_node("1") + + assert 1 == len(rendezvous.nodes) + rendezvous.add_node("1") + + assert 1 == len(rendezvous.nodes) + rendezvous.add_node("2") + + assert 2 == len(rendezvous.nodes) + rendezvous.add_node("1") + + assert 2 == len(rendezvous.nodes) + + +@pytest.mark.unit() +def test_remove_node(): + nodes = ["0", "1", "2"] + rendezvous = RendezvousHash(nodes=nodes) + rendezvous.remove_node("2") + + assert 2 == len(rendezvous.nodes) + + with pytest.raises(ValueError): + rendezvous.remove_node("2") + + assert 2 == len(rendezvous.nodes) + + rendezvous.remove_node("1") + assert 1 == len(rendezvous.nodes) + + rendezvous.remove_node("0") + assert 0 == len(rendezvous.nodes) + + +@pytest.mark.unit() +def test_get_node(): + nodes = ["0", "1", "2"] + rendezvous = RendezvousHash(nodes=nodes) + assert "0" == rendezvous.get_node("ok") + assert "1" == rendezvous.get_node("mykey") + assert "2" == rendezvous.get_node("wat") + + +@pytest.mark.unit() +def test_get_node_after_removal(): + nodes = ["0", "1", "2"] + rendezvous = RendezvousHash(nodes=nodes) + rendezvous.remove_node("1") + + assert "0" == rendezvous.get_node("ok") + assert "0" == rendezvous.get_node("mykey") + assert "2" == rendezvous.get_node("wat") + + +@pytest.mark.unit() +def test_get_node_after_addition(): + nodes = ["0", "1", "2"] + rendezvous = RendezvousHash(nodes=nodes) + assert "0" == rendezvous.get_node("ok") + assert "1" == rendezvous.get_node("mykey") + assert "2" == rendezvous.get_node("wat") + assert "2" == rendezvous.get_node("lol") + rendezvous.add_node("3") + + assert "0" == rendezvous.get_node("ok") + assert "1" == rendezvous.get_node("mykey") + assert "2" == rendezvous.get_node("wat") + assert "3" == rendezvous.get_node("lol") + + +@pytest.mark.unit() +def test_grow(): + rendezvous = RendezvousHash() + + placements = {} + + for i in range(10): + rendezvous.add_node(str(i)) + placements[str(i)] = [] + + for i in range(1000): + node = rendezvous.get_node(str(i)) + placements[node].append(i) + + new_placements = {} + + for i in range(20): + rendezvous.add_node(str(i)) + new_placements[str(i)] = [] + + for i in range(1000): + node = rendezvous.get_node(str(i)) + new_placements[node].append(i) + + keys = [k for sublist in placements.values() for k in sublist] + new_keys = [k for sublist in new_placements.values() for k in sublist] + assert sorted(keys) == sorted(new_keys) + + added = 0 + removed = 0 + + for node, assignments in new_placements.items(): + after = set(assignments) + before = set(placements.get(node, [])) + removed += len(before.difference(after)) + added += len(after.difference(before)) + + assert added == removed + assert 1062 == (added + removed) + + +@pytest.mark.unit() +def test_shrink(): + rendezvous = RendezvousHash() + + placements = {} + for i in range(10): + rendezvous.add_node(str(i)) + placements[str(i)] = [] + + for i in range(1000): + node = rendezvous.get_node(str(i)) + placements[node].append(i) + + rendezvous.remove_node("9") + new_placements = {} + for i in range(9): + new_placements[str(i)] = [] + + for i in range(1000): + node = rendezvous.get_node(str(i)) + new_placements[node].append(i) + + keys = [k for sublist in placements.values() for k in sublist] + new_keys = [k for sublist in new_placements.values() for k in sublist] + assert sorted(keys) == sorted(new_keys) + + added = 0 + removed = 0 + for node, assignments in placements.items(): + after = set(assignments) + before = set(new_placements.get(node, [])) + removed += len(before.difference(after)) + added += len(after.difference(before)) + + assert added == removed + assert 202 == (added + removed) + + +def collide(key, seed): + return 1337 + + +@pytest.mark.unit() +def test_rendezvous_collision(): + nodes = ["c", "b", "a"] + rendezvous = RendezvousHash(nodes, hash_function=collide) + + for i in range(1000): + assert "c" == rendezvous.get_node(i) + + +@pytest.mark.unit() +def test_rendezvous_names(): + nodes = [1, 2, 3, "a", "b", "lol.wat.com"] + rendezvous = RendezvousHash(nodes, hash_function=collide) + + for i in range(10): + assert "lol.wat.com" == rendezvous.get_node(i) + + nodes = [1, "a", "0"] + rendezvous = RendezvousHash(nodes, hash_function=collide) + + for i in range(10): + assert "a" == rendezvous.get_node(i) diff --git a/lib/python3.11/site-packages/pymemcache/test/test_serde.py b/lib/python3.11/site-packages/pymemcache/test/test_serde.py new file mode 100644 index 00000000..cb922715 --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/test/test_serde.py @@ -0,0 +1,147 @@ +from unittest import TestCase + +from pymemcache.serde import ( + CompressedSerde, + pickle_serde, + PickleSerde, + FLAG_BYTES, + FLAG_COMPRESSED, + FLAG_PICKLE, + FLAG_INTEGER, + FLAG_TEXT, +) +import pytest +import pickle +import sys +import zlib + + +class CustomInt(int): + """ + Custom integer type for testing. + + Entirely useless, but used to show that built in types get serialized and + deserialized back as the same type of object. + """ + + pass + + +def check(serde, value, expected_flags): + serialized, flags = serde.serialize(b"key", value) + assert flags == expected_flags + + # pymemcache stores values as byte strings, so we immediately the value + # if needed so deserialized works as it would with a real server + if not isinstance(serialized, bytes): + serialized = str(serialized).encode("ascii") + + deserialized = serde.deserialize(b"key", serialized, flags) + assert deserialized == value + + +@pytest.mark.unit() +class TestSerde: + serde = pickle_serde + + def test_bytes(self): + check(self.serde, b"value", FLAG_BYTES) + check(self.serde, b"\xc2\xa3 $ \xe2\x82\xac", FLAG_BYTES) # £ $ € + + def test_unicode(self): + check(self.serde, "value", FLAG_TEXT) + check(self.serde, "£ $ €", FLAG_TEXT) + + def test_int(self): + check(self.serde, 1, FLAG_INTEGER) + + def test_pickleable(self): + check(self.serde, {"a": "dict"}, FLAG_PICKLE) + + def test_subtype(self): + # Subclass of a native type will be restored as the same type + check(self.serde, CustomInt(123123), FLAG_PICKLE) + + +@pytest.mark.unit() +class TestSerdePickleVersion0(TestCase): + serde = PickleSerde(pickle_version=0) + + +@pytest.mark.unit() +class TestSerdePickleVersion1(TestCase): + serde = PickleSerde(pickle_version=1) + + +@pytest.mark.unit() +class TestSerdePickleVersion2(TestCase): + serde = PickleSerde(pickle_version=2) + + +@pytest.mark.unit() +class TestSerdePickleVersionHighest(TestCase): + serde = PickleSerde(pickle_version=pickle.HIGHEST_PROTOCOL) + + +@pytest.mark.parametrize("serde", [pickle_serde, CompressedSerde()]) +@pytest.mark.unit() +def test_compressed_simple(serde): + # test_bytes + check(serde, b"value", FLAG_BYTES) + check(serde, b"\xc2\xa3 $ \xe2\x82\xac", FLAG_BYTES) # £ $ € + + # test_unicode + check(serde, "value", FLAG_TEXT) + check(serde, "£ $ €", FLAG_TEXT) + + # test_int + check(serde, 1, FLAG_INTEGER) + + # test_pickleable + check(serde, {"a": "dict"}, FLAG_PICKLE) + + # test_subtype + # Subclass of a native type will be restored as the same type + check(serde, CustomInt(12312), FLAG_PICKLE) + + +@pytest.mark.parametrize( + "serde", + [ + CompressedSerde(min_compress_len=49), + # Custom compression. This could be something like lz4 + CompressedSerde( + compress=lambda value: zlib.compress(value, 9), + decompress=lambda value: zlib.decompress(value), + min_compress_len=49, + ), + ], +) +@pytest.mark.unit() +def test_compressed_complex(serde): + # test_bytes + check(serde, b"value" * 10, FLAG_BYTES | FLAG_COMPRESSED) + check(serde, b"\xc2\xa3 $ \xe2\x82\xac" * 10, FLAG_BYTES | FLAG_COMPRESSED) # £ $ € + + # test_unicode + check(serde, "value" * 10, FLAG_TEXT | FLAG_COMPRESSED) + check(serde, "£ $ €" * 10, FLAG_TEXT | FLAG_COMPRESSED) + + # test_int, doesn't make sense to compress + check(serde, sys.maxsize, FLAG_INTEGER) + + # test_pickleable + check( + serde, + { + "foo": "bar", + "baz": "qux", + "uno": "dos", + "tres": "tres", + }, + FLAG_PICKLE | FLAG_COMPRESSED, + ) + + # test_subtype + # Subclass of a native type will be restored as the same type + check(serde, CustomInt(sys.maxsize), FLAG_PICKLE | FLAG_COMPRESSED) diff --git a/lib/python3.11/site-packages/pymemcache/test/test_utils.py b/lib/python3.11/site-packages/pymemcache/test/test_utils.py new file mode 100644 index 00000000..6dc6a9a3 --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/test/test_utils.py @@ -0,0 +1,113 @@ +import pytest + +from pymemcache.test.utils import MockMemcacheClient + + +@pytest.mark.unit() +def test_get_set(): + client = MockMemcacheClient() + assert client.get(b"hello") is None + + client.set(b"hello", 12) + assert client.get(b"hello") == 12 + + +@pytest.mark.unit() +def test_get_set_unicide_key(): + client = MockMemcacheClient() + assert client.get("hello") is None + + client.set(b"hello", 12) + assert client.get("hello") == 12 + + +@pytest.mark.unit() +def test_get_set_non_ascii_value(): + client = MockMemcacheClient() + assert client.get(b"hello") is None + + # This is the value of msgpack.packb('non_ascii') + non_ascii_str = b"\xa9non_ascii" + client.set(b"hello", non_ascii_str) + assert client.get(b"hello") == non_ascii_str + + +@pytest.mark.unit() +def test_get_many_set_many(): + client = MockMemcacheClient() + client.set(b"h", 1) + + result = client.get_many([b"h", b"e", b"l", b"o"]) + assert result == {b"h": 1} + + # Convert keys into bytes + d = {k.encode("ascii"): v for k, v in dict(h=1, e=2, z=3).items()} + client.set_many(d) + assert client.get_many([b"h", b"e", b"z", b"o"]) == d + + +@pytest.mark.unit() +def test_get_many_set_many_non_ascii_values(): + client = MockMemcacheClient() + + # These are the values of calling msgpack.packb() on '1', '2', and '3' + non_ascii_1 = b"\xa11" + non_ascii_2 = b"\xa12" + non_ascii_3 = b"\xa13" + client.set(b"h", non_ascii_1) + + result = client.get_many([b"h", b"e", b"l", b"o"]) + assert result == {b"h": non_ascii_1} + + # Convert keys into bytes + d = { + k.encode("ascii"): v + for k, v in dict(h=non_ascii_1, e=non_ascii_2, z=non_ascii_3).items() + } + client.set_many(d) + assert client.get_many([b"h", b"e", b"z", b"o"]) == d + + +@pytest.mark.unit() +def test_add(): + client = MockMemcacheClient() + + client.add(b"k", 2) + assert client.get(b"k") == 2 + + client.add(b"k", 25) + assert client.get(b"k") == 2 + + +@pytest.mark.unit() +def test_delete(): + client = MockMemcacheClient() + + client.add(b"k", 2) + assert client.get(b"k") == 2 + + client.delete(b"k") + assert client.get(b"k") is None + + +@pytest.mark.unit() +def test_incr_decr(): + client = MockMemcacheClient() + + client.add(b"k", 2) + + client.incr(b"k", 4) + assert client.get(b"k") == 6 + + client.decr(b"k", 2) + assert client.get(b"k") == 4 + + +@pytest.mark.unit() +def test_prepand_append(): + client = MockMemcacheClient() + + client.set(b"k", "1") + client.append(b"k", "a") + client.prepend(b"k", "p") + assert client.get(b"k") == b"p1a" diff --git a/lib/python3.11/site-packages/pymemcache/test/utils.py b/lib/python3.11/site-packages/pymemcache/test/utils.py new file mode 100644 index 00000000..0ca7f167 --- /dev/null +++ b/lib/python3.11/site-packages/pymemcache/test/utils.py @@ -0,0 +1,223 @@ +""" +Useful testing utilities. + +This module is considered public API. + +""" + +import time + +import socket + +from pymemcache.exceptions import MemcacheClientError, MemcacheIllegalInputError +from pymemcache.serde import LegacyWrappingSerde +from pymemcache.client.base import check_key_helper + + +class MockMemcacheClient: + """ + A (partial) in-memory mock for Clients. + + """ + + def __init__( + self, + server=None, + serde=None, + serializer=None, + deserializer=None, + connect_timeout=None, + timeout=None, + no_delay=False, + ignore_exc=False, + socket_module=None, + default_noreply=True, + allow_unicode_keys=False, + encoding="ascii", + tls_context=None, + ): + + self._contents = {} + + self.serde = serde or LegacyWrappingSerde(serializer, deserializer) + self.allow_unicode_keys = allow_unicode_keys + + # Unused, but present for interface compatibility + self.server = server + self.connect_timeout = connect_timeout + self.timeout = timeout + self.no_delay = no_delay + self.ignore_exc = ignore_exc + self.socket_module = socket + self.sock = None + self.encoding = encoding + self.tls_context = tls_context + + def check_key(self, key): + """Checks key and add key_prefix.""" + return check_key_helper(key, allow_unicode_keys=self.allow_unicode_keys) + + def clear(self): + """Method used to clear/reset mock cache""" + self._contents.clear() + + def get(self, key, default=None): + key = self.check_key(key) + + if key not in self._contents: + return default + + expire, value, flags = self._contents[key] + if expire and expire < time.time(): + del self._contents[key] + return default + + return self.serde.deserialize(key, value, flags) + + def get_many(self, keys): + out = {} + for key in keys: + value = self.get(key) + if value is not None: + out[key] = value + return out + + get_multi = get_many + + def set(self, key, value, expire=0, noreply=True, flags=None): + key = self.check_key(key) + if isinstance(value, str) and not isinstance(value, bytes): + try: + value = value.encode(self.encoding) + except (UnicodeEncodeError, UnicodeDecodeError): + raise MemcacheIllegalInputError + + value, flags = self.serde.serialize(key, value) + + if expire: + expire += time.time() + + self._contents[key] = expire, value, flags + return True + + def set_many(self, values, expire=0, noreply=True, flags=None): + result = [] + for key, value in values.items(): + ret = self.set(key, value, expire, noreply, flags=flags) + if not ret: + result.append(key) + return [] if noreply else result + + set_multi = set_many + + def incr(self, key, value, noreply=False): + current = self.get(key) + present = current is not None + if present: + self.set(key, current + value, noreply=noreply) + return None if noreply or not present else current + value + + def decr(self, key, value, noreply=False): + current = self.get(key) + present = current is not None + if present: + self.set(key, current - value, noreply=noreply) + return None if noreply or not present else current - value + + def add(self, key, value, expire=0, noreply=True, flags=None): + current = self.get(key) + present = current is not None + if not present: + self.set(key, value, expire, noreply, flags=flags) + return noreply or not present + + def delete(self, key, noreply=True): + key = self.check_key(key) + current = self._contents.pop(key, None) + present = current is not None + return noreply or present + + def delete_many(self, keys, noreply=True): + for key in keys: + self.delete(key, noreply) + return True + + def prepend(self, key, value, expire=0, noreply=True, flags=None): + current = self.get(key) + if current is not None: + if isinstance(value, str) and not isinstance(value, bytes): + try: + value = value.encode(self.encoding) + except (UnicodeEncodeError, UnicodeDecodeError): + raise MemcacheIllegalInputError + self.set(key, value + current, expire, noreply, flags=flags) + return True + + def append(self, key, value, expire=0, noreply=True, flags=None): + current = self.get(key) + if current is not None: + if isinstance(value, str) and not isinstance(value, bytes): + try: + value = value.encode(self.encoding) + except (UnicodeEncodeError, UnicodeDecodeError): + raise MemcacheIllegalInputError + self.set(key, current + value, expire, noreply, flags=flags) + return True + + delete_multi = delete_many + + def stats(self, *_args): + # I make no claim that these values make any sense, but the format + # of the output is the same as for pymemcache.client.Client.stats() + return { + "version": "MockMemcacheClient", + "rusage_user": 1.0, + "rusage_system": 1.0, + "hash_is_expanding": False, + "slab_reassign_running": False, + "inter": "in-memory", + "evictions": False, + "growth_factor": 1.0, + "stat_key_prefix": "", + "umask": 0o644, + "detail_enabled": False, + "cas_enabled": False, + "auth_enabled_sasl": False, + "maxconns_fast": False, + "slab_reassign": False, + "slab_automove": False, + } + + def replace(self, key, value, expire=0, noreply=True, flags=None): + current = self.get(key) + present = current is not None + if present: + self.set(key, value, expire, noreply, flags=flags) + return noreply or present + + def cas(self, key, value, cas, expire=0, noreply=False, flags=None): + raise MemcacheClientError("CAS is not enabled for this instance") + + def touch(self, key, expire=0, noreply=True): + current = self.get(key) + present = current is not None + if present: + self.set(key, current, expire, noreply=noreply) + return True if noreply or present else False + + def cache_memlimit(self, memlimit): + return True + + def version(self): + return "MockMemcacheClient" + + def flush_all(self, delay=0, noreply=True): + self.clear() + + return noreply or self._contents == {} + + def quit(self): + pass + + def close(self): + pass diff --git a/requirements.txt b/requirements.txt index 7630336c..7ef6b277 100644 --- a/requirements.txt +++ b/requirements.txt @@ -15,6 +15,8 @@ debtcollector==2.5.0 decorator==5.1.1 Django==4.2.1 django-debug-toolbar==4.1.0 +django-ranged-response==0.2.0 +django-simple-captcha==0.5.20 dnspython==2.3.0 dogpile.cache==1.2.1 eventlet==0.33.3 diff --git a/templates/impressum.html b/templates/impressum.html index fe8ac55b..52b27034 100644 --- a/templates/impressum.html +++ b/templates/impressum.html @@ -16,6 +16,8 @@ Unser Angebot enthält Links zu externen Webseiten Dritter, auf deren Inhalte wi
Andere Disclaimer
Wir weisen darauf hin, dass die Datenübertragung im Internet (z.B. bei der Kommunikation per E-Mail) Sicherheitslücken aufweisen kann. Ein lückenloser Schutz der Daten vor dem Zugriff durch Dritte ist nicht möglich.
Der Nutzung von im Rahmen der Impressumspflicht veröffentlichten Kontaktdaten durch Dritte zur Übersendung von nicht ausdrücklich angeforderter Werbung und Informationsmaterialien wird hiermit ausdrücklich widersprochen. Die Betreiber der Seiten behalten sich ausdrücklich rechtliche Schritte im Falle der unverlangten Zusendung von Werbeinformationen, etwa durch Spam-Mails, vor.
+
Quellcode
+Der Quellcode dieser Seite ist unter https://git.denkena-consulting.com/f-denkena/impuls zu finden. Pull Requests und weitere Mithilfe ist stets willkommen. Gerne werden auch Ihre Fehlermeldungen schnellstmöglich bearbeitet.
{% endblock content %} diff --git a/website/__pycache__/models.cpython-311.pyc b/website/__pycache__/models.cpython-311.pyc index 64b43e57..062ba4fb 100644 Binary files a/website/__pycache__/models.cpython-311.pyc and b/website/__pycache__/models.cpython-311.pyc differ diff --git a/website/__pycache__/tests.cpython-311.pyc b/website/__pycache__/tests.cpython-311.pyc new file mode 100644 index 00000000..79fbc2f8 Binary files /dev/null and b/website/__pycache__/tests.cpython-311.pyc differ diff --git a/website/__pycache__/views.cpython-311.pyc b/website/__pycache__/views.cpython-311.pyc index 0bde04b4..2ac05e3d 100644 Binary files a/website/__pycache__/views.cpython-311.pyc and b/website/__pycache__/views.cpython-311.pyc differ