From d65edfb0243c9bc24b748c52d2983ef4dd627004 Mon Sep 17 00:00:00 2001 From: jesse-sony <50677487+jesse-sony@users.noreply.github.com> Date: Wed, 21 Jul 2021 08:22:18 -0400 Subject: [PATCH 001/100] Feature/local exception translator (#2650) * Create a module_internals struct Since we now have two things that are going to be module local, it felt correct to add a struct to manage them. * Add local exception translators These are added via the register_local_exception_translator function and are then applied before the global translators * Add unit tests to show the local exception translator works * Fix a bug in the unit test with the string value of KeyError * Fix a formatting issue * Rename registered_local_types_cpp() Rename it to get_registered_local_types_cpp() to disambiguate from the new member of module_internals * Add additional comments to new local exception code path * Add a register_local_exception function * Add additional unit tests for register_local_exception * Use get_local_internals like get_internals * Update documentation for new local exception feature * Add back a missing space * Clean-up some issues in the docs * Remove the code duplication when translating exceptions Separated out the exception processing into a standalone function in the details namespace. Clean-up some comments as per PR notes as well * Remove the code duplication in register_exception * Cleanup some formatting things caught by clang-format * Remove the templates from exception translators But I added a using declaration to alias the type. * Remove the extra local from local_internals variable names * Add an extra explanatory comment to local_internals * Fix a typo in the code --- docs/advanced/exceptions.rst | 72 +++++++++++-- include/pybind11/detail/class.h | 2 +- include/pybind11/detail/internals.h | 27 ++++- include/pybind11/detail/type_caster_base.h | 2 +- include/pybind11/pybind11.h | 119 ++++++++++++++++----- tests/local_bindings.h | 19 ++++ tests/pybind11_cross_module_tests.cpp | 16 ++- tests/test_exceptions.cpp | 31 +++++- tests/test_exceptions.py | 29 ++++- tests/test_local_bindings.cpp | 2 +- 10 files changed, 274 insertions(+), 45 deletions(-) diff --git a/docs/advanced/exceptions.rst b/docs/advanced/exceptions.rst index f70c202f..738fb0ba 100644 --- a/docs/advanced/exceptions.rst +++ b/docs/advanced/exceptions.rst @@ -75,9 +75,10 @@ Registering custom translators If the default exception conversion policy described above is insufficient, pybind11 also provides support for registering custom exception translators. -To register a simple exception conversion that translates a C++ exception into -a new Python exception using the C++ exception's ``what()`` method, a helper -function is available: +Similar to pybind11 classes, exception translators can be local to the module +they are defined in or global to the entire python session. To register a simple +exception conversion that translates a C++ exception into a new Python exception +using the C++ exception's ``what()`` method, a helper function is available: .. code-block:: cpp @@ -87,12 +88,20 @@ This call creates a Python exception class with the name ``PyExp`` in the given module and automatically converts any encountered exceptions of type ``CppExp`` into Python exceptions of type ``PyExp``. +A matching function is available for registering a local exception translator: + +.. code-block:: cpp + + py::register_local_exception(module, "PyExp"); + + It is possible to specify base class for the exception using the third parameter, a `handle`: .. code-block:: cpp py::register_exception(module, "PyExp", PyExc_RuntimeError); + py::register_local_exception(module, "PyExp", PyExc_RuntimeError); Then `PyExp` can be caught both as `PyExp` and `RuntimeError`. @@ -100,16 +109,18 @@ The class objects of the built-in Python exceptions are listed in the Python documentation on `Standard Exceptions `_. The default base class is `PyExc_Exception`. -When more advanced exception translation is needed, the function -``py::register_exception_translator(translator)`` can be used to register +When more advanced exception translation is needed, the functions +``py::register_exception_translator(translator)`` and +``py::register_local_exception_translator(translator)`` can be used to register functions that can translate arbitrary exception types (and which may include -additional logic to do so). The function takes a stateless callable (e.g. a +additional logic to do so). The functions takes a stateless callable (e.g. a function pointer or a lambda function without captured variables) with the call signature ``void(std::exception_ptr)``. When a C++ exception is thrown, the registered exception translators are tried in reverse order of registration (i.e. the last registered translator gets the -first shot at handling the exception). +first shot at handling the exception). All local translators will be tried +before a global translator is tried. Inside the translator, ``std::rethrow_exception`` should be used within a try block to re-throw the exception. One or more catch clauses to catch @@ -168,6 +179,53 @@ section. with ``-fvisibility=hidden``. Therefore exceptions that are used across ABI boundaries need to be explicitly exported, as exercised in ``tests/test_exceptions.h``. See also: "Problems with C++ exceptions" under `GCC Wiki `_. + +Local vs Global Exception Translators +===================================== + +When a global exception translator is registered, it will be applied across all +modules in the reverse order of registration. This can create behavior where the +order of module import influences how exceptions are translated. + +If module1 has the following translator: + +.. code-block:: cpp + + py::register_exception_translator([](std::exception_ptr p) { + try { + if (p) std::rethrow_exception(p); + } catch (const std::invalid_argument &e) { + PyErr_SetString("module1 handled this") + } + } + +and module2 has the following similar translator: + +.. code-block:: cpp + + py::register_exception_translator([](std::exception_ptr p) { + try { + if (p) std::rethrow_exception(p); + } catch (const std::invalid_argument &e) { + PyErr_SetString("module2 handled this") + } + } + +then which translator handles the invalid_argument will be determined by the +order that module1 and module2 are imported. Since exception translators are +applied in the reverse order of registration, which ever module was imported +last will "win" and that translator will be applied. + +If there are multiple pybind11 modules that share exception types (either +standard built-in or custom) loaded into a single python instance and +consistent error handling behavior is needed, then local translators should be +used. + +Changing the previous example to use ``register_local_exception_translator`` +would mean that when invalid_argument is thrown in the module2 code, the +module2 translator will always handle it, while in module1, the module1 +translator will do the same. + .. _handling_python_exceptions_cpp: Handling exceptions from Python in C++ diff --git a/include/pybind11/detail/class.h b/include/pybind11/detail/class.h index 5fee3318..15e09165 100644 --- a/include/pybind11/detail/class.h +++ b/include/pybind11/detail/class.h @@ -209,7 +209,7 @@ extern "C" inline void pybind11_meta_dealloc(PyObject *obj) { internals.direct_conversions.erase(tindex); if (tinfo->module_local) - registered_local_types_cpp().erase(tindex); + get_local_internals().registered_types_cpp.erase(tindex); else internals.registered_types_cpp.erase(tindex); internals.registered_types_py.erase(tinfo->type); diff --git a/include/pybind11/detail/internals.h b/include/pybind11/detail/internals.h index 5578c651..a621aed2 100644 --- a/include/pybind11/detail/internals.h +++ b/include/pybind11/detail/internals.h @@ -12,7 +12,11 @@ #include "../pytypes.h" PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) + +using ExceptionTranslator = void (*)(std::exception_ptr); + PYBIND11_NAMESPACE_BEGIN(detail) + // Forward declarations inline PyTypeObject *make_static_property_type(); inline PyTypeObject *make_default_metaclass(); @@ -100,7 +104,7 @@ struct internals { std::unordered_set, override_hash> inactive_override_cache; type_map> direct_conversions; std::unordered_map> patients; - std::forward_list registered_exception_translators; + std::forward_list registered_exception_translators; std::unordered_map shared_data; // Custom data to be shared across extensions std::vector loader_patient_stack; // Used by `loader_life_support` std::forward_list static_strings; // Stores the std::strings backing detail::c_str() @@ -313,12 +317,25 @@ PYBIND11_NOINLINE inline internals &get_internals() { return **internals_pp; } -/// Works like `internals.registered_types_cpp`, but for module-local registered types: -inline type_map ®istered_local_types_cpp() { - static type_map locals{}; - return locals; + +// the internals struct (above) is shared between all the modules. local_internals are only +// for a single module. Any changes made to internals may require an update to +// PYBIND11_INTERNALS_VERSION, breaking backwards compatibility. local_internals is, by design, +// restricted to a single module. Whether a module has local internals or not should not +// impact any other modules, because the only things accessing the local internals is the +// module that contains them. +struct local_internals { + type_map registered_types_cpp; + std::forward_list registered_exception_translators; +}; + +/// Works like `get_internals`, but for things which are locally registered. +inline local_internals &get_local_internals() { + static local_internals locals; + return locals; } + /// Constructs a std::string with the given arguments, stores it in `internals`, and returns its /// `c_str()`. Such strings objects have a long storage duration -- the internal strings are only /// cleared when the program exits or after interpreter shutdown (when embedding), and so are diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h index 451690dc..bdf3a398 100644 --- a/include/pybind11/detail/type_caster_base.h +++ b/include/pybind11/detail/type_caster_base.h @@ -160,7 +160,7 @@ PYBIND11_NOINLINE inline detail::type_info* get_type_info(PyTypeObject *type) { } inline detail::type_info *get_local_type_info(const std::type_index &tp) { - auto &locals = registered_local_types_cpp(); + auto &locals = get_local_internals().registered_types_cpp; auto it = locals.find(tp); if (it != locals.end()) return it->second; diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index af8e730d..0c25ca1a 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -56,6 +56,29 @@ PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) +PYBIND11_NAMESPACE_BEGIN(detail) + +// Apply all the extensions translators from a list +// Return true if one of the translators completed without raising an exception +// itself. Return of false indicates that if there are other translators +// available, they should be tried. +inline bool apply_exception_translators(std::forward_list& translators) { + auto last_exception = std::current_exception(); + + for (auto &translator : translators) { + try { + translator(last_exception); + return true; + } catch (...) { + last_exception = std::current_exception(); + } + } + return false; +} + +PYBIND11_NAMESPACE_END(detail) + + /// Wraps an arbitrary C++ function/method/lambda function/.. into a callable Python object class cpp_function : public function { public: @@ -550,6 +573,7 @@ protected: } } + /// Main dispatch logic for calls to functions bound using pybind11 static PyObject *dispatcher(PyObject *self, PyObject *args_in, PyObject *kwargs_in) { using namespace detail; @@ -830,8 +854,12 @@ protected: #endif } catch (...) { /* When an exception is caught, give each registered exception - translator a chance to translate it to a Python exception - in reverse order of registration. + translator a chance to translate it to a Python exception. First + all module-local translators will be tried in reverse order of + registration. If none of the module-locale translators handle + the exception (or there are no module-locale translators) then + the global translators will be tried, also in reverse order of + registration. A translator may choose to do one of the following: @@ -840,17 +868,15 @@ protected: - do nothing and let the exception fall through to the next translator, or - delegate translation to the next translator by throwing a new type of exception. */ - auto last_exception = std::current_exception(); - auto ®istered_exception_translators = get_internals().registered_exception_translators; - for (auto& translator : registered_exception_translators) { - try { - translator(last_exception); - } catch (...) { - last_exception = std::current_exception(); - continue; - } + auto &local_exception_translators = get_local_internals().registered_exception_translators; + if (detail::apply_exception_translators(local_exception_translators)) { return nullptr; } + auto &exception_translators = get_internals().registered_exception_translators; + if (detail::apply_exception_translators(exception_translators)) { + return nullptr; + } + PyErr_SetString(PyExc_SystemError, "Exception escaped from default exception translator!"); return nullptr; } @@ -951,6 +977,7 @@ protected: } }; + /// Wrapper for Python extension modules class module_ : public object { public: @@ -1124,7 +1151,7 @@ protected: auto tindex = std::type_index(*rec.type); tinfo->direct_conversions = &internals.direct_conversions[tindex]; if (rec.module_local) - registered_local_types_cpp()[tindex] = tinfo; + get_local_internals().registered_types_cpp[tindex] = tinfo; else internals.registered_types_cpp[tindex] = tinfo; internals.registered_types_py[(PyTypeObject *) m_ptr] = { tinfo }; @@ -1310,7 +1337,7 @@ public: generic_type::initialize(record); if (has_alias) { - auto &instances = record.module_local ? registered_local_types_cpp() : get_internals().registered_types_cpp; + auto &instances = record.module_local ? get_local_internals().registered_types_cpp : get_internals().registered_types_cpp; instances[std::type_index(typeid(type_alias))] = instances[std::type_index(typeid(type))]; } } @@ -2010,12 +2037,24 @@ template void implicitly_convertible() pybind11_fail("implicitly_convertible: Unable to find type " + type_id()); } -template -void register_exception_translator(ExceptionTranslator&& translator) { + +inline void register_exception_translator(ExceptionTranslator &&translator) { detail::get_internals().registered_exception_translators.push_front( std::forward(translator)); } + +/** + * Add a new module-local exception translator. Locally registered functions + * will be tried before any globally registered exception translators, which + * will only be invoked if the module-local handlers do not deal with + * the exception. + */ +inline void register_local_exception_translator(ExceptionTranslator &&translator) { + detail::get_local_internals().registered_exception_translators.push_front( + std::forward(translator)); +} + /** * Wrapper to generate a new Python exception type. * @@ -2049,22 +2088,20 @@ PYBIND11_NAMESPACE_BEGIN(detail) // directly in register_exception, but that makes clang <3.5 segfault - issue #1349). template exception &get_exception_object() { static exception ex; return ex; } -PYBIND11_NAMESPACE_END(detail) -/** - * Registers a Python exception in `m` of the given `name` and installs an exception translator to - * translate the C++ exception to the created Python exception using the exceptions what() method. - * This is intended for simple exception translations; for more complex translation, register the - * exception object and translator directly. - */ +// Helper function for register_exception and register_local_exception template -exception ®ister_exception(handle scope, - const char *name, - handle base = PyExc_Exception) { +exception ®ister_exception_impl(handle scope, + const char *name, + handle base, + bool isLocal) { auto &ex = detail::get_exception_object(); if (!ex) ex = exception(scope, name, base); - register_exception_translator([](std::exception_ptr p) { + auto register_func = isLocal ? ®ister_local_exception_translator + : ®ister_exception_translator; + + register_func([](std::exception_ptr p) { if (!p) return; try { std::rethrow_exception(p); @@ -2075,6 +2112,36 @@ exception ®ister_exception(handle scope, return ex; } +PYBIND11_NAMESPACE_END(detail) + +/** + * Registers a Python exception in `m` of the given `name` and installs a translator to + * translate the C++ exception to the created Python exception using the what() method. + * This is intended for simple exception translations; for more complex translation, register the + * exception object and translator directly. + */ +template +exception ®ister_exception(handle scope, + const char *name, + handle base = PyExc_Exception) { + return detail::register_exception_impl(scope, name, base, false /* isLocal */); +} + +/** + * Registers a Python exception in `m` of the given `name` and installs a translator to + * translate the C++ exception to the created Python exception using the what() method. + * This translator will only be used for exceptions that are thrown in this module and will be + * tried before global exception translators, including those registered with register_exception. + * This is intended for simple exception translations; for more complex translation, register the + * exception object and translator directly. + */ +template +exception ®ister_local_exception(handle scope, + const char *name, + handle base = PyExc_Exception) { + return detail::register_exception_impl(scope, name, base, true /* isLocal */); +} + PYBIND11_NAMESPACE_BEGIN(detail) PYBIND11_NOINLINE inline void print(const tuple &args, const dict &kwargs) { auto strings = tuple(args.size()); diff --git a/tests/local_bindings.h b/tests/local_bindings.h index 8629ed77..f11c08e6 100644 --- a/tests/local_bindings.h +++ b/tests/local_bindings.h @@ -35,6 +35,25 @@ using NonLocalVec2 = std::vector; using NonLocalMap = std::unordered_map; using NonLocalMap2 = std::unordered_map; + +// Exception that will be caught via the module local translator. +class LocalException : public std::exception { +public: + explicit LocalException(const char * m) : message{m} {} + const char * what() const noexcept override {return message.c_str();} +private: + std::string message = ""; +}; + +// Exception that will be registered with register_local_exception_translator +class LocalSimpleException : public std::exception { +public: + explicit LocalSimpleException(const char * m) : message{m} {} + const char * what() const noexcept override {return message.c_str();} +private: + std::string message = ""; +}; + PYBIND11_MAKE_OPAQUE(LocalVec); PYBIND11_MAKE_OPAQUE(LocalVec2); PYBIND11_MAKE_OPAQUE(LocalMap); diff --git a/tests/pybind11_cross_module_tests.cpp b/tests/pybind11_cross_module_tests.cpp index e79701c6..4bfd5302 100644 --- a/tests/pybind11_cross_module_tests.cpp +++ b/tests/pybind11_cross_module_tests.cpp @@ -29,11 +29,14 @@ PYBIND11_MODULE(pybind11_cross_module_tests, m) { bind_local(m, "ExternalType2", py::module_local()); // test_exceptions.py + py::register_local_exception(m, "LocalSimpleException"); m.def("raise_runtime_error", []() { PyErr_SetString(PyExc_RuntimeError, "My runtime error"); throw py::error_already_set(); }); m.def("raise_value_error", []() { PyErr_SetString(PyExc_ValueError, "My value error"); throw py::error_already_set(); }); m.def("throw_pybind_value_error", []() { throw py::value_error("pybind11 value error"); }); m.def("throw_pybind_type_error", []() { throw py::type_error("pybind11 type error"); }); m.def("throw_stop_iteration", []() { throw py::stop_iteration(); }); + m.def("throw_local_error", []() { throw LocalException("just local"); }); + m.def("throw_local_simple_error", []() { throw LocalSimpleException("external mod"); }); py::register_exception_translator([](std::exception_ptr p) { try { if (p) std::rethrow_exception(p); @@ -42,6 +45,17 @@ PYBIND11_MODULE(pybind11_cross_module_tests, m) { } }); + // translate the local exception into a key error but only in this module + py::register_local_exception_translator([](std::exception_ptr p) { + try { + if (p) { + std::rethrow_exception(p); + } + } catch (const LocalException &e) { + PyErr_SetString(PyExc_KeyError, e.what()); + } + }); + // test_local_bindings.py // Local to both: bind_local(m, "LocalType", py::module_local()) @@ -94,7 +108,7 @@ PYBIND11_MODULE(pybind11_cross_module_tests, m) { m.def("get_mixed_lg", [](int i) { return MixedLocalGlobal(i); }); // test_internal_locals_differ - m.def("local_cpp_types_addr", []() { return (uintptr_t) &py::detail::registered_local_types_cpp(); }); + m.def("local_cpp_types_addr", []() { return (uintptr_t) &py::detail::get_local_internals().registered_types_cpp; }); // test_stl_caster_vs_stl_bind py::bind_vector>(m, "VectorInt"); diff --git a/tests/test_exceptions.cpp b/tests/test_exceptions.cpp index f7bacd07..e28f0bb7 100644 --- a/tests/test_exceptions.cpp +++ b/tests/test_exceptions.cpp @@ -6,9 +6,10 @@ All rights reserved. Use of this source code is governed by a BSD-style license that can be found in the LICENSE file. */ - #include "test_exceptions.h" +#include "local_bindings.h" + #include "pybind11_tests.h" #include @@ -68,6 +69,17 @@ class MyException5_1 : public MyException5 { using MyException5::MyException5; }; + +// Exception that will be caught via the module local translator. +class MyException6 : public std::exception { +public: + explicit MyException6(const char * m) : message{m} {} + const char * what() const noexcept override {return message.c_str();} +private: + std::string message = ""; +}; + + struct PythonCallInDestructor { PythonCallInDestructor(const py::dict &d) : d(d) {} ~PythonCallInDestructor() { d["good"] = true; } @@ -138,14 +150,29 @@ TEST_SUBMODULE(exceptions, m) { // A slightly more complicated one that declares MyException5_1 as a subclass of MyException5 py::register_exception(m, "MyException5_1", ex5.ptr()); + //py::register_local_exception(m, "LocalSimpleException") + + py::register_local_exception_translator([](std::exception_ptr p) { + try { + if (p) { + std::rethrow_exception(p); + } + } catch (const MyException6 &e) { + PyErr_SetString(PyExc_RuntimeError, e.what()); + } + }); + m.def("throws1", []() { throw MyException("this error should go to a custom type"); }); m.def("throws2", []() { throw MyException2("this error should go to a standard Python exception"); }); m.def("throws3", []() { throw MyException3("this error cannot be translated"); }); m.def("throws4", []() { throw MyException4("this error is rethrown"); }); m.def("throws5", []() { throw MyException5("this is a helper-defined translated exception"); }); m.def("throws5_1", []() { throw MyException5_1("MyException5 subclass"); }); + m.def("throws6", []() { throw MyException6("MyException6 only handled in this module"); }); m.def("throws_logic_error", []() { throw std::logic_error("this error should fall through to the standard handler"); }); - m.def("throws_overflow_error", []() {throw std::overflow_error(""); }); + m.def("throws_overflow_error", []() { throw std::overflow_error(""); }); + m.def("throws_local_error", []() { throw LocalException("never caught"); }); + m.def("throws_local_simple_error", []() { throw LocalSimpleException("this mod"); }); m.def("exception_matches", []() { py::dict foo; try { diff --git a/tests/test_exceptions.py b/tests/test_exceptions.py index b9fc2693..966ae07f 100644 --- a/tests/test_exceptions.py +++ b/tests/test_exceptions.py @@ -25,7 +25,7 @@ def test_error_already_set(msg): assert msg(excinfo.value) == "foo" -def test_cross_module_exceptions(): +def test_cross_module_exceptions(msg): with pytest.raises(RuntimeError) as excinfo: cm.raise_runtime_error() assert str(excinfo.value) == "My runtime error" @@ -45,6 +45,15 @@ def test_cross_module_exceptions(): with pytest.raises(StopIteration) as excinfo: cm.throw_stop_iteration() + with pytest.raises(cm.LocalSimpleException) as excinfo: + cm.throw_local_simple_error() + assert msg(excinfo.value) == "external mod" + + with pytest.raises(KeyError) as excinfo: + cm.throw_local_error() + # KeyError is a repr of the key, so it has an extra set of quotes + assert str(excinfo.value) == "'just local'" + # TODO: FIXME @pytest.mark.xfail( @@ -221,3 +230,21 @@ def test_invalid_repr(): with pytest.raises(TypeError): m.simple_bool_passthrough(MyRepr()) + + +def test_local_translator(msg): + """Tests that a local translator works and that the local translator from + the cross module is not applied""" + with pytest.raises(RuntimeError) as excinfo: + m.throws6() + assert msg(excinfo.value) == "MyException6 only handled in this module" + + with pytest.raises(RuntimeError) as excinfo: + m.throws_local_error() + assert not isinstance(excinfo.value, KeyError) + assert msg(excinfo.value) == "never caught" + + with pytest.raises(Exception) as excinfo: + m.throws_local_simple_error() + assert not isinstance(excinfo.value, cm.LocalSimpleException) + assert msg(excinfo.value) == "this mod" diff --git a/tests/test_local_bindings.cpp b/tests/test_local_bindings.cpp index bfbab3ed..8d6e33f7 100644 --- a/tests/test_local_bindings.cpp +++ b/tests/test_local_bindings.cpp @@ -78,7 +78,7 @@ TEST_SUBMODULE(local_bindings, m) { m.def("get_mixed_lg", [](int i) { return MixedLocalGlobal(i); }); // test_internal_locals_differ - m.def("local_cpp_types_addr", []() { return (uintptr_t) &py::detail::registered_local_types_cpp(); }); + m.def("local_cpp_types_addr", []() { return (uintptr_t) &py::detail::get_local_internals().registered_types_cpp; }); // test_stl_caster_vs_stl_bind m.def("load_vector_via_caster", [](std::vector v) { From e58c6897cc7b77b92e1c49df82a3504f777b97c9 Mon Sep 17 00:00:00 2001 From: ka-bo <55587089+ka-bo@users.noreply.github.com> Date: Wed, 21 Jul 2021 18:00:57 +0200 Subject: [PATCH 002/100] Specified encoding in setup.py calls of open() (#3137) * Specified encoding in setup.py calls of open() * Fix for Python2 Co-authored-by: Karsten Bock --- setup.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/setup.py b/setup.py index ced3d695..7aa151c0 100644 --- a/setup.py +++ b/setup.py @@ -11,6 +11,7 @@ import string import subprocess import sys import tempfile +import io import setuptools.command.sdist @@ -70,7 +71,7 @@ exec(code, loc) version = loc["__version__"] # Verify that the version matches the one in C++ -with open("include/pybind11/detail/common.h") as f: +with io.open("include/pybind11/detail/common.h", encoding="utf8") as f: matches = dict(VERSION_REGEX.findall(f.read())) cpp_version = "{MAJOR}.{MINOR}.{PATCH}".format(**matches) if version != cpp_version: From ff97f101d96cc780ce348ac6b4b4464982578929 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Mon, 26 Jul 2021 11:28:36 -0700 Subject: [PATCH 003/100] Removing MSVC C4996 from pragma block at the top of pybind11.h (#3129) * Removing MSVC C4996 from pragma block at the top of pybind11.h * localtime_thread_safe, PYBIND11_COMPAT_STRDUP * Adding #include (attempt to fix MSVC 2015, 2017 errors). --- include/pybind11/chrono.h | 33 +++++++++++++++++++++++--------- include/pybind11/detail/common.h | 5 +++++ include/pybind11/pybind11.h | 19 ++++++++++++------ 3 files changed, 42 insertions(+), 15 deletions(-) diff --git a/include/pybind11/chrono.h b/include/pybind11/chrono.h index d32b5b05..c4e81f57 100644 --- a/include/pybind11/chrono.h +++ b/include/pybind11/chrono.h @@ -11,9 +11,14 @@ #pragma once #include "pybind11.h" + +#include #include #include -#include +#include + +#include + #include // Backport the PyDateTime_DELTA functions from Python3.3 if required @@ -95,6 +100,22 @@ public: PYBIND11_TYPE_CASTER(type, _("datetime.timedelta")); }; +inline std::tm *localtime_thread_safe(const std::time_t *time, std::tm *buf) { +#if defined(__STDC_WANT_LIB_EXT1__) || defined(_MSC_VER) + if (localtime_s(buf, time)) + return nullptr; + return buf; +#else + static std::mutex mtx; + std::lock_guard lock(mtx); + std::tm *tm_ptr = localtime(time); + if (tm_ptr != nullptr) { + *buf = *tm_ptr; + } + return tm_ptr; +#endif +} + // This is for casting times on the system clock into datetime.datetime instances template class type_caster> { public: @@ -162,16 +183,10 @@ public: // (https://en.cppreference.com/w/cpp/chrono/system_clock/to_time_t) std::time_t tt = system_clock::to_time_t(time_point_cast(src - us)); - // std::localtime returns a pointer to a static internal std::tm object on success, - // or null pointer otherwise - std::tm *localtime_ptr = std::localtime(&tt); + std::tm localtime; + std::tm *localtime_ptr = localtime_thread_safe(&tt, &localtime); if (!localtime_ptr) throw cast_error("Unable to represent system_clock in local time"); - - // this function uses static memory so it's best to copy it out asap just in case - // otherwise other code that is using localtime may break this (not just python code) - std::tm localtime = *localtime_ptr; - return PyDateTime_FromDateAndTime(localtime.tm_year + 1900, localtime.tm_mon + 1, localtime.tm_mday, diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 52b01933..3faf3ea3 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -125,6 +125,11 @@ # endif #endif +// https://en.cppreference.com/w/c/chrono/localtime +#if defined(__STDC_LIB_EXT1__) && !defined(__STDC_WANT_LIB_EXT1__) +# define __STDC_WANT_LIB_EXT1__ +#endif + #include #include #include diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 0c25ca1a..696e7cd3 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -19,7 +19,6 @@ # pragma warning(disable: 4100) // warning C4100: Unreferenced formal parameter # pragma warning(disable: 4127) // warning C4127: Conditional expression is constant # pragma warning(disable: 4800) // warning C4800: 'int': forcing value to bool 'true' or 'false' (performance warning) -# pragma warning(disable: 4996) // warning C4996: The POSIX name for this item is deprecated. Instead, use the ISO C and C++ conformant name # pragma warning(disable: 4522) // warning C4522: multiple assignment operators specified # pragma warning(disable: 4505) // warning C4505: 'PySlice_GetIndicesEx': unreferenced local function has been removed (PyPy only) #elif defined(__GNUG__) && !defined(__clang__) @@ -43,6 +42,8 @@ #include #include +#include + #if defined(__cpp_lib_launder) && !(defined(_MSC_VER) && (_MSC_VER < 1914)) # define PYBIND11_STD_LAUNDER std::launder # define PYBIND11_HAS_STD_LAUNDER 1 @@ -76,8 +77,13 @@ inline bool apply_exception_translators(std::forward_list& return false; } -PYBIND11_NAMESPACE_END(detail) +#if defined(_MSC_VER) +# define PYBIND11_COMPAT_STRDUP _strdup +#else +# define PYBIND11_COMPAT_STRDUP strdup +#endif +PYBIND11_NAMESPACE_END(detail) /// Wraps an arbitrary C++ function/method/lambda function/.. into a callable Python object class cpp_function : public function { @@ -276,7 +282,7 @@ protected: std::free(s); } char *operator()(const char *s) { - auto t = strdup(s); + auto t = PYBIND11_COMPAT_STRDUP(s); strings.push_back(t); return t; } @@ -520,7 +526,8 @@ protected: auto *func = (PyCFunctionObject *) m_ptr; std::free(const_cast(func->m_ml->ml_doc)); // Install docstring if it's non-empty (when at least one option is enabled) - func->m_ml->ml_doc = signatures.empty() ? nullptr : strdup(signatures.c_str()); + func->m_ml->ml_doc + = signatures.empty() ? nullptr : PYBIND11_COMPAT_STRDUP(signatures.c_str()); if (rec->is_method) { m_ptr = PYBIND11_INSTANCE_METHOD_NEW(m_ptr, rec->scope.ptr()); @@ -1525,7 +1532,7 @@ public: detail::process_attributes::init(extra..., rec_fget); if (rec_fget->doc && rec_fget->doc != doc_prev) { free(doc_prev); - rec_fget->doc = strdup(rec_fget->doc); + rec_fget->doc = PYBIND11_COMPAT_STRDUP(rec_fget->doc); } } if (rec_fset) { @@ -1533,7 +1540,7 @@ public: detail::process_attributes::init(extra..., rec_fset); if (rec_fset->doc && rec_fset->doc != doc_prev) { free(doc_prev); - rec_fset->doc = strdup(rec_fset->doc); + rec_fset->doc = PYBIND11_COMPAT_STRDUP(rec_fset->doc); } if (! rec_active) rec_active = rec_fset; } From 7904ba1a5c600ea8455b16c1010a1d46181d3148 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Mon, 26 Jul 2021 12:02:50 -0700 Subject: [PATCH 004/100] Adding pragma warning(disable: 4522) for MSVC <= 2017. (#3142) --- include/pybind11/pybind11.h | 1 - include/pybind11/pytypes.h | 7 +++++++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 696e7cd3..9c0b7238 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -19,7 +19,6 @@ # pragma warning(disable: 4100) // warning C4100: Unreferenced formal parameter # pragma warning(disable: 4127) // warning C4127: Conditional expression is constant # pragma warning(disable: 4800) // warning C4800: 'int': forcing value to bool 'true' or 'false' (performance warning) -# pragma warning(disable: 4522) // warning C4522: multiple assignment operators specified # pragma warning(disable: 4505) // warning C4505: 'PySlice_GetIndicesEx': unreferenced local function has been removed (PyPy only) #elif defined(__GNUG__) && !defined(__clang__) # pragma GCC diagnostic push diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index b483fb32..33299827 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -532,6 +532,10 @@ object object_or_cast(T &&o); // Match a PyObject*, which we want to convert directly to handle via its converting constructor inline handle object_or_cast(PyObject *ptr) { return ptr; } +#if defined(_MSC_VER) && _MSC_VER < 1920 +# pragma warning(push) +# pragma warning(disable: 4522) // warning C4522: multiple assignment operators specified +#endif template class accessor : public object_api> { using key_type = typename Policy::key_type; @@ -580,6 +584,9 @@ private: key_type key; mutable object cache; }; +#if defined(_MSC_VER) && _MSC_VER < 1920 +# pragma warning(pop) +#endif PYBIND11_NAMESPACE_BEGIN(accessor_policies) struct obj_attr { From 7f76d79508475ef2e8f17941e350b1dfa8d828b4 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 26 Jul 2021 15:05:58 -0400 Subject: [PATCH 005/100] [pre-commit.ci] pre-commit autoupdate (#3143) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit updates: - [github.com/asottile/pyupgrade: v2.21.2 → v2.23.0](https://github.com/asottile/pyupgrade/compare/v2.21.2...v2.23.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .pre-commit-config.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index aa7b4774..510b8412 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -31,7 +31,7 @@ repos: exclude: ^noxfile.py$ - repo: https://github.com/asottile/pyupgrade - rev: v2.21.2 + rev: v2.23.0 hooks: - id: pyupgrade From a0f862d428acd11dacb344ce9690e420d98cde53 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Mon, 26 Jul 2021 13:26:36 -0700 Subject: [PATCH 006/100] Removing MSVC C4800 from pragma block at the top of pybind11.h (#3141) * Adding PYBIND11_COMPAT_BOOL_CAST to appease MSVC 2015 warning C4800. * Replacing PYBIND11_COMPAT_BOOL_CAST with simpler != 0 * Extra parentheses (almost all compilers failed without these). --- include/pybind11/buffer_info.h | 2 +- include/pybind11/cast.h | 2 +- include/pybind11/detail/type_caster_base.h | 4 ++-- include/pybind11/pybind11.h | 1 - include/pybind11/pytypes.h | 6 ++++-- 5 files changed, 8 insertions(+), 7 deletions(-) diff --git a/include/pybind11/buffer_info.h b/include/pybind11/buffer_info.h index 47dc39d4..eba68d1a 100644 --- a/include/pybind11/buffer_info.h +++ b/include/pybind11/buffer_info.h @@ -83,7 +83,7 @@ struct buffer_info { view->strides ? std::vector(view->strides, view->strides + view->ndim) : detail::c_strides({view->shape, view->shape + view->ndim}, view->itemsize), - view->readonly) { + (view->readonly != 0)) { this->m_view = view; this->ownview = ownview; } diff --git a/include/pybind11/cast.h b/include/pybind11/cast.h index a748c77c..1da43235 100644 --- a/include/pybind11/cast.h +++ b/include/pybind11/cast.h @@ -322,7 +322,7 @@ public: } #endif if (res == 0 || res == 1) { - value = (bool) res; + value = (res != 0); return true; } PyErr_Clear(); diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h index bdf3a398..a6b6bea6 100644 --- a/include/pybind11/detail/type_caster_base.h +++ b/include/pybind11/detail/type_caster_base.h @@ -231,7 +231,7 @@ struct value_and_holder { return reinterpret_cast(vh[0]); } // True if this `value_and_holder` has a non-null value pointer - explicit operator bool() const { return value_ptr(); } + explicit operator bool() const { return value_ptr() != nullptr; } template H &holder() const { return reinterpret_cast(vh[1]); @@ -252,7 +252,7 @@ struct value_and_holder { bool instance_registered() const { return inst->simple_layout ? inst->simple_instance_registered - : inst->nonsimple.status[index] & instance::status_instance_registered; + : ((inst->nonsimple.status[index] & instance::status_instance_registered) != 0); } void set_instance_registered(bool v = true) const { if (inst->simple_layout) diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 9c0b7238..93b920d8 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -18,7 +18,6 @@ # pragma warning(push) # pragma warning(disable: 4100) // warning C4100: Unreferenced formal parameter # pragma warning(disable: 4127) // warning C4127: Conditional expression is constant -# pragma warning(disable: 4800) // warning C4800: 'int': forcing value to bool 'true' or 'false' (performance warning) # pragma warning(disable: 4505) // warning C4505: 'PySlice_GetIndicesEx': unreferenced local function has been removed (PyPy only) #elif defined(__GNUG__) && !defined(__clang__) # pragma GCC diagnostic push diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index 33299827..cea4e7eb 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -367,7 +367,9 @@ public: /// Check if the currently trapped error type matches the given Python exception class (or a /// subclass thereof). May also be passed a tuple to search for any exception class matches in /// the given tuple. - bool matches(handle exc) const { return PyErr_GivenExceptionMatches(m_type.ptr(), exc.ptr()); } + bool matches(handle exc) const { + return (PyErr_GivenExceptionMatches(m_type.ptr(), exc.ptr()) != 0); + } const object& type() const { return m_type; } const object& value() const { return m_value; } @@ -853,7 +855,7 @@ PYBIND11_NAMESPACE_END(detail) Name(handle h, borrowed_t) : Parent(h, borrowed_t{}) { } \ Name(handle h, stolen_t) : Parent(h, stolen_t{}) { } \ PYBIND11_DEPRECATED("Use py::isinstance(obj) instead") \ - bool check() const { return m_ptr != nullptr && (bool) CheckFun(m_ptr); } \ + bool check() const { return m_ptr != nullptr && (CheckFun(m_ptr) != 0); } \ static bool check_(handle h) { return h.ptr() != nullptr && CheckFun(h.ptr()); } \ template \ Name(const ::pybind11::detail::accessor &a) : Name(object(a)) { } From a0b975965fe5df66f20a1d29193055f8538ac94e Mon Sep 17 00:00:00 2001 From: David Hewitt <1939362+davidhewitt@users.noreply.github.com> Date: Tue, 27 Jul 2021 19:16:28 +0100 Subject: [PATCH 007/100] Allow python builtins to be used as callbacks (#1413) * Allow python builtins to be used as callbacks * Try to fix pypy segfault * Add expected fail for PyPy * Fix typo * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add more info to xfail * Add env * Try returning false * Try removing the move for pypy * Fix bugs * Try removing move * Just keep ignoring for PyPy * Add back xfail * Fix ctors * Revert change of std::move * Change to skip * Fix bug and edit comments * Remove clang-tidy bugprone fix skip bug Co-authored-by: Aaron Gokaslan Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- include/pybind11/functional.h | 34 ++++++++++++++++++++-------------- tests/test_callbacks.cpp | 6 ++++++ tests/test_callbacks.py | 11 +++++++++++ 3 files changed, 37 insertions(+), 14 deletions(-) diff --git a/include/pybind11/functional.h b/include/pybind11/functional.h index cc7099c6..b9bf3891 100644 --- a/include/pybind11/functional.h +++ b/include/pybind11/functional.h @@ -43,27 +43,33 @@ public: captured variables), in which case the roundtrip can be avoided. */ if (auto cfunc = func.cpp_function()) { - auto c = reinterpret_borrow(PyCFunction_GET_SELF(cfunc.ptr())); - auto rec = (function_record *) c; + auto cfunc_self = PyCFunction_GET_SELF(cfunc.ptr()); + if (isinstance(cfunc_self)) { + auto c = reinterpret_borrow(cfunc_self); + auto rec = (function_record *) c; - while (rec != nullptr) { - if (rec->is_stateless - && same_type(typeid(function_type), - *reinterpret_cast(rec->data[1]))) { - struct capture { - function_type f; - }; - value = ((capture *) &rec->data)->f; - return true; + while (rec != nullptr) { + if (rec->is_stateless + && same_type(typeid(function_type), + *reinterpret_cast(rec->data[1]))) { + struct capture { + function_type f; + }; + value = ((capture *) &rec->data)->f; + return true; + } + rec = rec->next; } - rec = rec->next; } + // PYPY segfaults here when passing builtin function like sum. + // Raising an fail exception here works to prevent the segfault, but only on gcc. + // See PR #1413 for full details } // ensure GIL is held during functor destruction struct func_handle { function f; - func_handle(function&& f_) : f(std::move(f_)) {} + func_handle(function &&f_) noexcept : f(std::move(f_)) {} func_handle(const func_handle& f_) { gil_scoped_acquire acq; f = f_.f; @@ -77,7 +83,7 @@ public: // to emulate 'move initialization capture' in C++11 struct func_wrapper { func_handle hfunc; - func_wrapper(func_handle&& hf): hfunc(std::move(hf)) {} + func_wrapper(func_handle &&hf) noexcept : hfunc(std::move(hf)) {} Return operator()(Args... args) const { gil_scoped_acquire acq; object retval(hfunc.f(std::forward(args)...)); diff --git a/tests/test_callbacks.cpp b/tests/test_callbacks.cpp index a208b44d..a5077103 100644 --- a/tests/test_callbacks.cpp +++ b/tests/test_callbacks.cpp @@ -156,6 +156,12 @@ TEST_SUBMODULE(callbacks, m) { .def(py::init<>()) .def("triple", [](CppBoundMethodTest &, int val) { return 3 * val; }); + // This checks that builtin functions can be passed as callbacks + // rather than throwing RuntimeError due to trying to extract as capsule + m.def("test_sum_builtin", [](const std::function &sum_builtin, const py::iterable &i) { + return sum_builtin(i); + }); + // test async Python callbacks using callback_f = std::function; m.def("test_async_callback", [](const callback_f &f, const py::list &work) { diff --git a/tests/test_callbacks.py b/tests/test_callbacks.py index 397bf63d..5bc4d177 100644 --- a/tests/test_callbacks.py +++ b/tests/test_callbacks.py @@ -3,6 +3,7 @@ import pytest from pybind11_tests import callbacks as m from threading import Thread import time +import env # NOQA: F401 def test_callbacks(): @@ -124,6 +125,16 @@ def test_movable_object(): assert m.callback_with_movable(lambda _: None) is True +@pytest.mark.skipif( + "env.PYPY", + reason="PyPy segfaults on here. See discussion on #1413.", +) +def test_python_builtins(): + """Test if python builtins like sum() can be used as callbacks""" + assert m.test_sum_builtin(sum, [1, 2, 3]) == 6 + assert m.test_sum_builtin(sum, []) == 0 + + def test_async_callbacks(): # serves as state for async callback class Item: From 7cc0ebb4753770ace452d118bb327cdd9e76dcbb Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Tue, 27 Jul 2021 14:23:52 -0700 Subject: [PATCH 008/100] fix: the CMake config in Python package had a hard coded path (#3144) --- CMakeLists.txt | 6 ++++++ tests/extra_python_package/test_files.py | 14 ++++++++++++-- tools/pybind11Config.cmake.in | 3 ++- 3 files changed, 20 insertions(+), 3 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index c988ea0b..b04311fd 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -203,6 +203,12 @@ if(PYBIND11_INSTALL) "${CMAKE_INSTALL_DATAROOTDIR}/cmake/${PROJECT_NAME}" CACHE STRING "install path for pybind11Config.cmake") + if(IS_ABSOLUTE "${CMAKE_INSTALL_INCLUDEDIR}") + set(pybind11_INCLUDEDIR "${CMAKE_INSTALL_FULL_INCLUDEDIR}") + else() + set(pybind11_INCLUDEDIR "\$\{PACKAGE_PREFIX_DIR\}/${CMAKE_INSTALL_INCLUDEDIR}") + endif() + configure_package_config_file( tools/${PROJECT_NAME}Config.cmake.in "${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}Config.cmake" INSTALL_DESTINATION ${PYBIND11_CMAKECONFIG_INSTALL_DIR}) diff --git a/tests/extra_python_package/test_files.py b/tests/extra_python_package/test_files.py index 43e93c26..337a72df 100644 --- a/tests/extra_python_package/test_files.py +++ b/tests/extra_python_package/test_files.py @@ -138,6 +138,16 @@ def test_build_sdist(monkeypatch, tmpdir): ) as f: pyproject_toml = f.read() + with contextlib.closing( + tar.extractfile( + tar.getmember( + start + "pybind11/share/cmake/pybind11/pybind11Config.cmake" + ) + ) + ) as f: + contents = f.read().decode("utf8") + assert 'set(pybind11_INCLUDE_DIR "${PACKAGE_PREFIX_DIR}/include")' in contents + files = {"pybind11/{}".format(n) for n in all_files} files |= sdist_files files |= {"pybind11{}".format(n) for n in local_sdist_files} @@ -151,11 +161,11 @@ def test_build_sdist(monkeypatch, tmpdir): .substitute(version=version, extra_cmd="") .encode() ) - assert setup_py == contents + assert setup_py == contents with open(os.path.join(MAIN_DIR, "tools", "pyproject.toml"), "rb") as f: contents = f.read() - assert pyproject_toml == contents + assert pyproject_toml == contents def test_build_global_dist(monkeypatch, tmpdir): diff --git a/tools/pybind11Config.cmake.in b/tools/pybind11Config.cmake.in index 6fa03a0f..73ec104a 100644 --- a/tools/pybind11Config.cmake.in +++ b/tools/pybind11Config.cmake.in @@ -201,7 +201,8 @@ Using ``find_package`` with version info is not recommended except for release v @PACKAGE_INIT@ # Location of pybind11/pybind11.h -set(pybind11_INCLUDE_DIR "@CMAKE_INSTALL_FULL_INCLUDEDIR@") +# This will be relative unless explicitly set as absolute +set(pybind11_INCLUDE_DIR "@pybind11_INCLUDEDIR@") set(pybind11_LIBRARY "") set(pybind11_DEFINITIONS USING_pybind11) From 9beaa925db49cf4751642a1529d34bef4a3682ce Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Tue, 27 Jul 2021 18:32:26 -0400 Subject: [PATCH 009/100] maint(clang-tidy): Improve code readability with explicit boolean casts (#3148) * maint(clang-tidy) Improve code readability * Fix minor typos * Revert optimization that removed test case * Fix comment formatting * Revert another optimization to repro an issue * Remove make_unique since it C++14 and newer only * eformat comments * Fix unsignedness of comparison * Update comment --- .clang-tidy | 13 +++++++++++++ include/pybind11/cast.h | 16 ++++++++++------ include/pybind11/detail/class.h | 7 ++++--- include/pybind11/detail/internals.h | 2 +- include/pybind11/detail/type_caster_base.h | 4 ++-- include/pybind11/embed.h | 4 ++-- include/pybind11/numpy.h | 19 +++++++++++-------- include/pybind11/pybind11.h | 15 +++++++++------ include/pybind11/pytypes.h | 12 ++++++++---- include/pybind11/stl/filesystem.h | 4 ++-- tests/test_embed/test_interpreter.cpp | 2 +- tests/test_methods_and_attributes.cpp | 1 + tests/test_numpy_dtypes.cpp | 6 ++++-- tests/test_numpy_vectorize.cpp | 2 +- tests/test_smart_ptr.cpp | 4 ++-- tests/test_stl.cpp | 6 ++---- 16 files changed, 73 insertions(+), 44 deletions(-) diff --git a/.clang-tidy b/.clang-tidy index 72304528..db5077c2 100644 --- a/.clang-tidy +++ b/.clang-tidy @@ -5,13 +5,17 @@ cppcoreguidelines-init-variables, clang-analyzer-optin.cplusplus.VirtualCall, llvm-namespace-comment, misc-misplaced-const, +misc-non-copyable-objects, misc-static-assert, misc-throw-by-value-catch-by-reference, misc-uniqueptr-reset-release, +misc-unused-parameters, modernize-avoid-bind, +modernize-make-shared, modernize-redundant-void-arg, modernize-replace-auto-ptr, modernize-replace-disallow-copy-and-assign-macro, +modernize-replace-random-shuffle, modernize-shrink-to-fit, modernize-use-auto, modernize-use-bool-literals, @@ -23,13 +27,20 @@ modernize-use-emplace, modernize-use-override, modernize-use-using, *performance*, +readability-avoid-const-params-in-decls, readability-container-size-empty, readability-else-after-return, +readability-delete-null-pointer, +readability-implicit-bool-conversion, readability-make-member-function-const, +readability-misplaced-array-index, +readability-non-const-parameter, readability-redundant-function-ptr-dereference, readability-redundant-smartptr-get, readability-redundant-string-cstr, readability-simplify-subscript-expr, +readability-static-accessed-through-instance, +readability-static-definition-in-anonymous-namespace, readability-string-compare, readability-uniqueptr-delete-release, ' @@ -39,6 +50,8 @@ CheckOptions: value: true - key: performance-unnecessary-value-param.AllowedTypes value: 'exception_ptr$;' +- key: readability-implicit-bool-conversion.AllowPointerConditions + value: true HeaderFilterRegex: 'pybind11/.*h' diff --git a/include/pybind11/cast.h b/include/pybind11/cast.h index 1da43235..898047b3 100644 --- a/include/pybind11/cast.h +++ b/include/pybind11/cast.h @@ -181,7 +181,7 @@ public: // Signed/unsigned checks happen elsewhere if (py_err || (std::is_integral::value && sizeof(py_type) != sizeof(T) && py_value != (py_type) (T) py_value)) { PyErr_Clear(); - if (py_err && convert && PyNumber_Check(src.ptr())) { + if (py_err && convert && (PyNumber_Check(src.ptr()) != 0)) { auto tmp = reinterpret_steal(std::is_floating_point::value ? PyNumber_Float(src.ptr()) : PyNumber_Long(src.ptr())); @@ -300,7 +300,7 @@ public: value = false; return true; } - if (convert || !std::strcmp("numpy.bool_", Py_TYPE(src.ptr())->tp_name)) { + if (convert || (std::strcmp("numpy.bool_", Py_TYPE(src.ptr())->tp_name) == 0)) { // (allow non-implicit conversion for numpy booleans) Py_ssize_t res = -1; @@ -501,10 +501,14 @@ public: // can fit into a single char value. if (StringCaster::UTF_N == 8 && str_len > 1 && str_len <= 4) { auto v0 = static_cast(value[0]); - size_t char0_bytes = !(v0 & 0x80) ? 1 : // low bits only: 0-127 - (v0 & 0xE0) == 0xC0 ? 2 : // 0b110xxxxx - start of 2-byte sequence - (v0 & 0xF0) == 0xE0 ? 3 : // 0b1110xxxx - start of 3-byte sequence - 4; // 0b11110xxx - start of 4-byte sequence + // low bits only: 0-127 + // 0b110xxxxx - start of 2-byte sequence + // 0b1110xxxx - start of 3-byte sequence + // 0b11110xxx - start of 4-byte sequence + size_t char0_bytes = (v0 & 0x80) == 0 ? 1 + : (v0 & 0xE0) == 0xC0 ? 2 + : (v0 & 0xF0) == 0xE0 ? 3 + : 4; if (char0_bytes == str_len) { // If we have a 128-255 value, we can decode it into a single char: diff --git a/include/pybind11/detail/class.h b/include/pybind11/detail/class.h index 15e09165..f9822c7b 100644 --- a/include/pybind11/detail/class.h +++ b/include/pybind11/detail/class.h @@ -129,8 +129,9 @@ extern "C" inline int pybind11_meta_setattro(PyObject* obj, PyObject* name, PyOb // 2. `Type.static_prop = other_static_prop` --> setattro: replace existing `static_prop` // 3. `Type.regular_attribute = value` --> setattro: regular attribute assignment const auto static_prop = (PyObject *) get_internals().static_property_type; - const auto call_descr_set = descr && value && PyObject_IsInstance(descr, static_prop) - && !PyObject_IsInstance(value, static_prop); + const auto call_descr_set = (descr != nullptr) && (value != nullptr) + && (PyObject_IsInstance(descr, static_prop) != 0) + && (PyObject_IsInstance(value, static_prop) == 0); if (call_descr_set) { // Call `static_property.__set__()` instead of replacing the `static_property`. #if !defined(PYPY_VERSION) @@ -562,7 +563,7 @@ extern "C" inline int pybind11_getbuffer(PyObject *obj, Py_buffer *view, int fla view->len = view->itemsize; for (auto s : info->shape) view->len *= s; - view->readonly = info->readonly; + view->readonly = static_cast(info->readonly); if ((flags & PyBUF_FORMAT) == PyBUF_FORMAT) view->format = const_cast(info->format.c_str()); if ((flags & PyBUF_STRIDES) == PyBUF_STRIDES) { diff --git a/include/pybind11/detail/internals.h b/include/pybind11/detail/internals.h index a621aed2..a93a83e2 100644 --- a/include/pybind11/detail/internals.h +++ b/include/pybind11/detail/internals.h @@ -297,7 +297,7 @@ PYBIND11_NOINLINE inline internals &get_internals() { PyThreadState *tstate = PyThreadState_Get(); #if PY_VERSION_HEX >= 0x03070000 internals_ptr->tstate = PyThread_tss_alloc(); - if (!internals_ptr->tstate || PyThread_tss_create(internals_ptr->tstate)) + if (!internals_ptr->tstate || (PyThread_tss_create(internals_ptr->tstate) != 0)) pybind11_fail("get_internals: could not successfully initialize the TSS key!"); PyThread_tss_set(internals_ptr->tstate, tstate); #else diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h index a6b6bea6..2a675418 100644 --- a/include/pybind11/detail/type_caster_base.h +++ b/include/pybind11/detail/type_caster_base.h @@ -238,8 +238,8 @@ struct value_and_holder { } bool holder_constructed() const { return inst->simple_layout - ? inst->simple_holder_constructed - : inst->nonsimple.status[index] & instance::status_holder_constructed; + ? inst->simple_holder_constructed + : (inst->nonsimple.status[index] & instance::status_holder_constructed) != 0u; } void set_holder_constructed(bool v = true) const { if (inst->simple_layout) diff --git a/include/pybind11/embed.h b/include/pybind11/embed.h index 204aaf98..abb1cd3c 100644 --- a/include/pybind11/embed.h +++ b/include/pybind11/embed.h @@ -76,7 +76,7 @@ struct embedded_module { using init_t = void (*)(); #endif embedded_module(const char *name, init_t init) { - if (Py_IsInitialized()) + if (Py_IsInitialized() != 0) pybind11_fail("Can't add new modules after the interpreter has been initialized"); auto result = PyImport_AppendInittab(name, init); @@ -101,7 +101,7 @@ PYBIND11_NAMESPACE_END(detail) .. _Python documentation: https://docs.python.org/3/c-api/init.html#c.Py_InitializeEx \endrst */ inline void initialize_interpreter(bool init_signal_handlers = true) { - if (Py_IsInitialized()) + if (Py_IsInitialized() != 0) pybind11_fail("The interpreter is already running"); Py_InitializeEx(init_signal_handlers ? 1 : 0); diff --git a/include/pybind11/numpy.h b/include/pybind11/numpy.h index edba8bac..7313897f 100644 --- a/include/pybind11/numpy.h +++ b/include/pybind11/numpy.h @@ -465,7 +465,9 @@ public: explicit dtype(const buffer_info &info) { dtype descr(_dtype_from_pep3118()(PYBIND11_STR_TYPE(info.format))); // If info.itemsize == 0, use the value calculated from the format string - m_ptr = descr.strip_padding(info.itemsize ? info.itemsize : descr.itemsize()).release().ptr(); + m_ptr = descr.strip_padding(info.itemsize != 0 ? info.itemsize : descr.itemsize()) + .release() + .ptr(); } explicit dtype(const std::string &format) { @@ -486,7 +488,7 @@ public: /// This is essentially the same as calling numpy.dtype(args) in Python. static dtype from_args(object args) { PyObject *ptr = nullptr; - if (!detail::npy_api::get().PyArray_DescrConverter_(args.ptr(), &ptr) || !ptr) + if ((detail::npy_api::get().PyArray_DescrConverter_(args.ptr(), &ptr) == 0) || !ptr) throw error_already_set(); return reinterpret_steal(ptr); } @@ -542,7 +544,7 @@ private: auto name = spec[0].cast(); auto format = spec[1].cast()[0].cast(); auto offset = spec[1].cast()[1].cast(); - if (!len(name) && format.kind() == 'V') + if ((len(name) == 0u) && format.kind() == 'V') continue; field_descriptors.push_back({(PYBIND11_STR_TYPE) name, format.strip_padding(format.itemsize()), offset}); } @@ -872,11 +874,12 @@ public: : array(std::move(shape), std::move(strides), ptr, base) { } explicit array_t(ShapeContainer shape, const T *ptr = nullptr, handle base = handle()) - : array_t(private_ctor{}, std::move(shape), - ExtraFlags & f_style - ? detail::f_strides(*shape, itemsize()) - : detail::c_strides(*shape, itemsize()), - ptr, base) { } + : array_t(private_ctor{}, + std::move(shape), + (ExtraFlags & f_style) != 0 ? detail::f_strides(*shape, itemsize()) + : detail::c_strides(*shape, itemsize()), + ptr, + base) {} explicit array_t(ssize_t count, const T *ptr = nullptr, handle base = handle()) : array({count}, {}, ptr, base) { } diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 93b920d8..e2cbb544 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -318,7 +318,8 @@ protected: a.descr = guarded_strdup(repr(a.value).cast().c_str()); } - rec->is_constructor = !strcmp(rec->name, "__init__") || !strcmp(rec->name, "__setstate__"); + rec->is_constructor + = (strcmp(rec->name, "__init__") == 0) || (strcmp(rec->name, "__setstate__") == 0); #if !defined(NDEBUG) && !defined(PYBIND11_DISABLE_NEW_STYLE_INIT_WARNING) if (rec->is_constructor && !rec->is_new_style_constructor) { @@ -1131,7 +1132,8 @@ protected: pybind11_fail("generic_type: cannot initialize type \"" + std::string(rec.name) + "\": an object with that name is already defined"); - if (rec.module_local ? get_local_type_info(*rec.type) : get_global_type_info(*rec.type)) + if ((rec.module_local ? get_local_type_info(*rec.type) : get_global_type_info(*rec.type)) + != nullptr) pybind11_fail("generic_type: type \"" + std::string(rec.name) + "\" is already registered!"); @@ -1209,8 +1211,9 @@ protected: void def_property_static_impl(const char *name, handle fget, handle fset, detail::function_record *rec_func) { - const auto is_static = rec_func && !(rec_func->is_method && rec_func->scope); - const auto has_doc = rec_func && rec_func->doc && pybind11::options::show_user_defined_docstrings(); + const auto is_static = (rec_func != nullptr) && !(rec_func->is_method && rec_func->scope); + const auto has_doc = (rec_func != nullptr) && (rec_func->doc != nullptr) + && pybind11::options::show_user_defined_docstrings(); auto property = handle((PyObject *) (is_static ? get_internals().static_property_type : &PyProperty_Type)); attr(name) = property(fget.ptr() ? fget : none(), @@ -2220,8 +2223,8 @@ inline function get_type_override(const void *this_ptr, const type_info *this_ty Unfortunately this doesn't work on PyPy. */ #if !defined(PYPY_VERSION) PyFrameObject *frame = PyThreadState_Get()->frame; - if (frame && (std::string) str(frame->f_code->co_name) == name && - frame->f_code->co_argcount > 0) { + if (frame != nullptr && (std::string) str(frame->f_code->co_name) == name + && frame->f_code->co_argcount > 0) { PyFrame_FastToLocals(frame); PyObject *self_caller = dict_getitem( frame->f_locals, PyTuple_GET_ITEM(frame->f_code->co_varnames, 0)); diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index cea4e7eb..161aed06 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -773,7 +773,11 @@ protected: dict_readonly(handle obj, ssize_t pos) : obj(obj), pos(pos) { increment(); } reference dereference() const { return {key, value}; } - void increment() { if (!PyDict_Next(obj.ptr(), &pos, &key, &value)) { pos = -1; } } + void increment() { + if (PyDict_Next(obj.ptr(), &pos, &key, &value) == 0) { + pos = -1; + } + } bool equal(const dict_readonly &b) const { return pos == b.pos; } private: @@ -1169,14 +1173,14 @@ public: bool_() : object(Py_False, borrowed_t{}) { } // Allow implicit conversion from and to `bool`: bool_(bool value) : object(value ? Py_True : Py_False, borrowed_t{}) { } - operator bool() const { return m_ptr && PyLong_AsLong(m_ptr) != 0; } + operator bool() const { return (m_ptr != nullptr) && PyLong_AsLong(m_ptr) != 0; } private: /// Return the truth value of an object -- always returns a new reference static PyObject *raw_bool(PyObject *op) { const auto value = PyObject_IsTrue(op); if (value == -1) return nullptr; - return handle(value ? Py_True : Py_False).inc_ref().ptr(); + return handle(value != 0 ? Py_True : Py_False).inc_ref().ptr(); } }; @@ -1607,7 +1611,7 @@ inline memoryview memoryview::from_buffer( size_t ndim = shape->size(); if (ndim != strides->size()) pybind11_fail("memoryview: shape length doesn't match strides length"); - ssize_t size = ndim ? 1 : 0; + ssize_t size = ndim != 0u ? 1 : 0; for (size_t i = 0; i < ndim; ++i) size *= (*shape)[i]; Py_buffer view; diff --git a/include/pybind11/stl/filesystem.h b/include/pybind11/stl/filesystem.h index 7a8acdb6..431b94b4 100644 --- a/include/pybind11/stl/filesystem.h +++ b/include/pybind11/stl/filesystem.h @@ -67,7 +67,7 @@ public: } PyObject* native = nullptr; if constexpr (std::is_same_v) { - if (PyUnicode_FSConverter(buf, &native)) { + if (PyUnicode_FSConverter(buf, &native) != 0) { if (auto c_str = PyBytes_AsString(native)) { // AsString returns a pointer to the internal buffer, which // must not be free'd. @@ -75,7 +75,7 @@ public: } } } else if constexpr (std::is_same_v) { - if (PyUnicode_FSDecoder(buf, &native)) { + if (PyUnicode_FSDecoder(buf, &native) != 0) { if (auto c_str = PyUnicode_AsWideCharString(native, nullptr)) { // AsWideCharString returns a new string that must be free'd. value = c_str; // Copies the string. diff --git a/tests/test_embed/test_interpreter.cpp b/tests/test_embed/test_interpreter.cpp index 6925ee97..cd50e952 100644 --- a/tests/test_embed/test_interpreter.cpp +++ b/tests/test_embed/test_interpreter.cpp @@ -103,7 +103,7 @@ bool has_pybind11_internals_builtin() { bool has_pybind11_internals_static() { auto **&ipp = py::detail::get_internals_pp(); - return ipp && *ipp; + return (ipp != nullptr) && (*ipp != nullptr); } TEST_CASE("Restart the interpreter") { diff --git a/tests/test_methods_and_attributes.cpp b/tests/test_methods_and_attributes.cpp index 9db31a0b..4cf6f08b 100644 --- a/tests/test_methods_and_attributes.cpp +++ b/tests/test_methods_and_attributes.cpp @@ -43,6 +43,7 @@ public: void add6(int other) { value += other; } // passing by value void add7(int &other) { value += other; } // passing by reference void add8(const int &other) { value += other; } // passing by const reference + // NOLINTNEXTLINE(readability-non-const-parameter) Deliberately non-const for testing void add9(int *other) { value += *other; } // passing by pointer void add10(const int *other) { value += *other; } // passing by const pointer diff --git a/tests/test_numpy_dtypes.cpp b/tests/test_numpy_dtypes.cpp index edf9cf20..340c972d 100644 --- a/tests/test_numpy_dtypes.cpp +++ b/tests/test_numpy_dtypes.cpp @@ -108,9 +108,11 @@ PYBIND11_PACKED(struct EnumStruct { std::ostream& operator<<(std::ostream& os, const StringStruct& v) { os << "a='"; - for (size_t i = 0; i < 3 && v.a[i]; i++) os << v.a[i]; + for (size_t i = 0; i < 3 && (v.a[i] != 0); i++) + os << v.a[i]; os << "',b='"; - for (size_t i = 0; i < 3 && v.b[i]; i++) os << v.b[i]; + for (size_t i = 0; i < 3 && (v.b[i] != 0); i++) + os << v.b[i]; return os << "'"; } diff --git a/tests/test_numpy_vectorize.cpp b/tests/test_numpy_vectorize.cpp index ed08a42b..b08a9f7e 100644 --- a/tests/test_numpy_vectorize.cpp +++ b/tests/test_numpy_vectorize.cpp @@ -59,7 +59,7 @@ TEST_SUBMODULE(numpy_vectorize, m) { .def(py::init()) .def_readwrite("value", &NonPODClass::value); m.def("vec_passthrough", - py::vectorize([](double *a, + py::vectorize([](const double *a, double b, // Changing this broke things // NOLINTNEXTLINE(performance-unnecessary-value-param) diff --git a/tests/test_smart_ptr.cpp b/tests/test_smart_ptr.cpp index 7fd5a9b3..57b2d894 100644 --- a/tests/test_smart_ptr.cpp +++ b/tests/test_smart_ptr.cpp @@ -101,7 +101,7 @@ private: // test_unique_nodelete // Object with a private destructor class MyObject4; -static std::unordered_set myobject4_instances; +std::unordered_set myobject4_instances; class MyObject4 { public: MyObject4(int value) : value{value} { @@ -127,7 +127,7 @@ private: // Object with std::unique_ptr where D is not matching the base class // Object with a protected destructor class MyObject4a; -static std::unordered_set myobject4a_instances; +std::unordered_set myobject4a_instances; class MyObject4a { public: MyObject4a(int i) { diff --git a/tests/test_stl.cpp b/tests/test_stl.cpp index eb8cdab7..23e2c07b 100644 --- a/tests/test_stl.cpp +++ b/tests/test_stl.cpp @@ -102,7 +102,7 @@ TEST_SUBMODULE(stl, m) { // test_set m.def("cast_set", []() { return std::set{"key1", "key2"}; }); m.def("load_set", [](const std::set &set) { - return set.count("key1") && set.count("key2") && set.count("key3"); + return (set.count("key1") != 0u) && (set.count("key2") != 0u) && (set.count("key3") != 0u); }); // test_recursive_casting @@ -196,9 +196,7 @@ TEST_SUBMODULE(stl, m) { m.def("double_or_zero", [](const opt_int& x) -> int { return x.value_or(0) * 2; }); - m.def("half_or_none", [](int x) -> opt_int { - return x ? opt_int(x / 2) : opt_int(); - }); + m.def("half_or_none", [](int x) -> opt_int { return x != 0 ? opt_int(x / 2) : opt_int(); }); m.def("test_nullopt", [](opt_int x) { return x.value_or(42); }, py::arg_v("x", std::nullopt, "None")); From 2164c2e0e78ca4bf12123f98349722907732581b Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Tue, 27 Jul 2021 15:33:31 -0700 Subject: [PATCH 010/100] Removing __INTEL_COMPILER section from pragma block at the top of pybind11.h (#3135) * Fixing `pragma warning pop` for `__INTEL_COMPILER`. * Adding push/pop to 3 tests. Removing #878 from top of pybind11.h (it was/is only needed for 1 test). * Trying again after CI failure, moving the push to the top of 2 tests. * Trying more after CI failure, adding push/pop to pybind11_tests.h, constructor_stats.h. * Moving ICC #2196 suppression to CMakeLists.txt * Fixing condition for `pragma GCC diagnostic push` in pybind11.h * Moving `pragma warning disable 2196` to common.h * Revising #ifdef to be more conservative. * Undoing insertion of notes that will hopefully soon be completely obsolete anyway. --- include/pybind11/detail/common.h | 3 +++ include/pybind11/pybind11.h | 8 ++------ tests/test_constants_and_functions.cpp | 7 +++++++ 3 files changed, 12 insertions(+), 6 deletions(-) diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 3faf3ea3..0b4e30c1 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -56,6 +56,9 @@ # elif __INTEL_COMPILER < 1900 && defined(PYBIND11_CPP14) # error pybind11 supports only C++11 with Intel C++ compiler v18. Use v19 or newer for C++14. # endif +/* The following pragma cannot be pop'ed: + https://community.intel.com/t5/Intel-C-Compiler/Inline-and-no-inline-warning/td-p/1216764 */ +# pragma warning disable 2196 // warning #2196: routine is both "inline" and "noinline" #elif defined(__clang__) && !defined(__apple_build_version__) # if __clang_major__ < 3 || (__clang_major__ == 3 && __clang_minor__ < 3) # error pybind11 requires clang 3.3 or newer diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index e2cbb544..b14aff58 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -10,16 +10,12 @@ #pragma once -#if defined(__INTEL_COMPILER) -# pragma warning push -# pragma warning disable 878 // incompatible exception specifications -# pragma warning disable 2196 // warning #2196: routine is both "inline" and "noinline" -#elif defined(_MSC_VER) +#if defined(_MSC_VER) && !defined(__INTEL_COMPILER) # pragma warning(push) # pragma warning(disable: 4100) // warning C4100: Unreferenced formal parameter # pragma warning(disable: 4127) // warning C4127: Conditional expression is constant # pragma warning(disable: 4505) // warning C4505: 'PySlice_GetIndicesEx': unreferenced local function has been removed (PyPy only) -#elif defined(__GNUG__) && !defined(__clang__) +#elif defined(__GNUG__) && !defined(__clang__) && !defined(__INTEL_COMPILER) # pragma GCC diagnostic push # pragma GCC diagnostic ignored "-Wunused-but-set-parameter" # pragma GCC diagnostic ignored "-Wattributes" diff --git a/tests/test_constants_and_functions.cpp b/tests/test_constants_and_functions.cpp index e7f01854..c0554503 100644 --- a/tests/test_constants_and_functions.cpp +++ b/tests/test_constants_and_functions.cpp @@ -133,7 +133,14 @@ TEST_SUBMODULE(constants_and_functions, m) { ; m.def("f1", f1); m.def("f2", f2); +#if defined(__INTEL_COMPILER) +# pragma warning push +# pragma warning disable 878 // incompatible exception specifications +#endif m.def("f3", f3); +#if defined(__INTEL_COMPILER) +# pragma warning pop +#endif m.def("f4", f4); // test_function_record_leaks From f4721a7b44e5513df3681ccb2c08eca50132b4bd Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Wed, 28 Jul 2021 08:58:36 -0700 Subject: [PATCH 011/100] Accommodating environments that define __STDC_WANT_LIB_EXT1__ even if __STDC_LIB_EXT1__ is not defined by the implementation. (#3151) Follow-on to PR #3129. --- include/pybind11/chrono.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/pybind11/chrono.h b/include/pybind11/chrono.h index c4e81f57..61bbcbc5 100644 --- a/include/pybind11/chrono.h +++ b/include/pybind11/chrono.h @@ -101,7 +101,7 @@ public: }; inline std::tm *localtime_thread_safe(const std::time_t *time, std::tm *buf) { -#if defined(__STDC_WANT_LIB_EXT1__) || defined(_MSC_VER) +#if (defined(__STDC_LIB_EXT1__) && defined(__STDC_WANT_LIB_EXT1__)) || defined(_MSC_VER) if (localtime_s(buf, time)) return nullptr; return buf; From b72ca7d1bd8d07470b5094017c790f6c763f9b0f Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Wed, 28 Jul 2021 17:01:21 -0700 Subject: [PATCH 012/100] Removing MSVC C4100 from pragma block at the top of pybind11.h (#3150) * Removing pragma for 4100 (to see what is still broken with the latest code). * Adding --keep-going * Revert "Adding --keep-going" This reverts commit 1c844c6ffd07a6111b644811e7e3b0a50b9d44bb. * Introducing PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100. * _MSC_VER <= 1916 * Replacing simple variadic function with variadic template (attempt to resolve MSVC 2017 failures). * Preserving existing comment (moved from pybind11.h to detail/common.h). * Adding blank lines for readability. --- include/pybind11/attr.h | 5 +++++ include/pybind11/cast.h | 1 + include/pybind11/detail/common.h | 13 +++++++++++++ include/pybind11/detail/descr.h | 1 + include/pybind11/detail/init.h | 3 +++ include/pybind11/detail/type_caster_base.h | 2 ++ include/pybind11/pybind11.h | 3 ++- 7 files changed, 27 insertions(+), 1 deletion(-) diff --git a/include/pybind11/attr.h b/include/pybind11/attr.h index 60ed9fd9..20d119f0 100644 --- a/include/pybind11/attr.h +++ b/include/pybind11/attr.h @@ -516,18 +516,22 @@ template struct process_attribute struct process_attributes { static void init(const Args&... args, function_record *r) { + PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(r); int unused[] = { 0, (process_attribute::type>::init(args, r), 0) ... }; ignore_unused(unused); } static void init(const Args&... args, type_record *r) { + PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(r); int unused[] = { 0, (process_attribute::type>::init(args, r), 0) ... }; ignore_unused(unused); } static void precall(function_call &call) { + PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(call); int unused[] = { 0, (process_attribute::type>::precall(call), 0) ... }; ignore_unused(unused); } static void postcall(function_call &call, handle fn_ret) { + PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(call, fn_ret); int unused[] = { 0, (process_attribute::type>::postcall(call, fn_ret), 0) ... }; ignore_unused(unused); } @@ -545,6 +549,7 @@ template ::value...), size_t self = constexpr_sum(std::is_same::value...)> constexpr bool expected_num_args(size_t nargs, bool has_args, bool has_kwargs) { + PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(nargs, has_args, has_kwargs); return named == 0 || (self + named + size_t(has_args) + size_t(has_kwargs)) == nargs; } diff --git a/include/pybind11/cast.h b/include/pybind11/cast.h index 898047b3..5ff0355a 100644 --- a/include/pybind11/cast.h +++ b/include/pybind11/cast.h @@ -605,6 +605,7 @@ protected: /* Implementation: Convert a C++ tuple into a Python tuple */ template static handle cast_impl(T &&src, return_value_policy policy, handle parent, index_sequence) { + PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(src, policy, parent); std::array entries{{ reinterpret_steal(make_caster::cast(std::get(std::forward(src)), policy, parent))... }}; diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 0b4e30c1..0add6272 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -913,5 +913,18 @@ inline static std::shared_ptr try_get_shared_from_this(std::enable_shared_fro #endif } +#if defined(_MSC_VER) && _MSC_VER <= 1916 + +// warning C4100: Unreferenced formal parameter +template +inline constexpr void workaround_incorrect_msvc_c4100(Args &&...) {} + +# define PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(...) \ + detail::workaround_incorrect_msvc_c4100(__VA_ARGS__) + +#else +# define PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(...) +#endif + PYBIND11_NAMESPACE_END(detail) PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) diff --git a/include/pybind11/detail/descr.h b/include/pybind11/detail/descr.h index 7cb8350e..0acfc7db 100644 --- a/include/pybind11/detail/descr.h +++ b/include/pybind11/detail/descr.h @@ -42,6 +42,7 @@ struct descr { template constexpr descr plus_impl(const descr &a, const descr &b, index_sequence, index_sequence) { + PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(b); return {a.text[Is1]..., b.text[Is2]...}; } diff --git a/include/pybind11/detail/init.h b/include/pybind11/detail/init.h index 3269e042..3ebec041 100644 --- a/include/pybind11/detail/init.h +++ b/include/pybind11/detail/init.h @@ -94,6 +94,7 @@ void construct(...) { // construct an Alias from the returned base instance. template void construct(value_and_holder &v_h, Cpp *ptr, bool need_alias) { + PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(need_alias); no_nullptr(ptr); if (Class::has_alias && need_alias && !is_alias(ptr)) { // We're going to try to construct an alias by moving the cpp type. Whether or not @@ -131,6 +132,7 @@ void construct(value_and_holder &v_h, Alias *alias_ptr, bool) { // derived type (through those holder's implicit conversion from derived class holder constructors). template void construct(value_and_holder &v_h, Holder holder, bool need_alias) { + PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(need_alias); auto *ptr = holder_helper>::get(holder); no_nullptr(ptr); // If we need an alias, check that the held pointer is actually an alias instance @@ -148,6 +150,7 @@ void construct(value_and_holder &v_h, Holder holder, bool need_alias) { // need it, we simply move-construct the cpp value into a new instance. template void construct(value_and_holder &v_h, Cpp &&result, bool need_alias) { + PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(need_alias); static_assert(std::is_move_constructible>::value, "pybind11::init() return-by-value factory function requires a movable class"); if (Class::has_alias && need_alias) diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h index 2a675418..e2d1bcb8 100644 --- a/include/pybind11/detail/type_caster_base.h +++ b/include/pybind11/detail/type_caster_base.h @@ -930,6 +930,7 @@ protected: does not have a private operator new implementation. */ template ::value>> static auto make_copy_constructor(const T *x) -> decltype(new T(*x), Constructor{}) { + PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(x); return [](const void *arg) -> void * { return new T(*reinterpret_cast(arg)); }; @@ -937,6 +938,7 @@ protected: template ::value>> static auto make_move_constructor(const T *x) -> decltype(new T(std::move(*const_cast(x))), Constructor{}) { + PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(x); return [](const void *arg) -> void * { return new T(std::move(*const_cast(reinterpret_cast(arg)))); }; diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index b14aff58..139a4111 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -12,7 +12,6 @@ #if defined(_MSC_VER) && !defined(__INTEL_COMPILER) # pragma warning(push) -# pragma warning(disable: 4100) // warning C4100: Unreferenced formal parameter # pragma warning(disable: 4127) // warning C4127: Conditional expression is constant # pragma warning(disable: 4505) // warning C4505: 'PySlice_GetIndicesEx': unreferenced local function has been removed (PyPy only) #elif defined(__GNUG__) && !defined(__clang__) && !defined(__INTEL_COMPILER) @@ -1388,12 +1387,14 @@ public: template class_ &def(const detail::initimpl::constructor &init, const Extra&... extra) { + PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(init); init.execute(*this, extra...); return *this; } template class_ &def(const detail::initimpl::alias_constructor &init, const Extra&... extra) { + PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(init); init.execute(*this, extra...); return *this; } From c14b193308d7877b158e9187079078099d1052ee Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Wed, 28 Jul 2021 18:04:14 -0700 Subject: [PATCH 013/100] chore: increase CMake upper limit (#3124) --- .github/workflows/configure.yml | 2 +- CMakeLists.txt | 6 +++--- tests/CMakeLists.txt | 4 ++-- tests/test_cmake_build/installed_embed/CMakeLists.txt | 6 ++++-- tests/test_cmake_build/installed_function/CMakeLists.txt | 3 ++- tests/test_cmake_build/installed_target/CMakeLists.txt | 3 ++- tests/test_cmake_build/subdirectory_embed/CMakeLists.txt | 6 ++++-- tests/test_cmake_build/subdirectory_function/CMakeLists.txt | 3 ++- tests/test_cmake_build/subdirectory_target/CMakeLists.txt | 3 ++- tests/test_embed/CMakeLists.txt | 1 + 10 files changed, 23 insertions(+), 14 deletions(-) diff --git a/.github/workflows/configure.yml b/.github/workflows/configure.yml index abcff1dd..924088fc 100644 --- a/.github/workflows/configure.yml +++ b/.github/workflows/configure.yml @@ -18,7 +18,7 @@ jobs: matrix: runs-on: [ubuntu-latest, macos-latest, windows-latest] arch: [x64] - cmake: [3.18] + cmake: ["3.21"] include: - runs-on: ubuntu-latest diff --git a/CMakeLists.txt b/CMakeLists.txt index b04311fd..bb01a861 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -7,13 +7,13 @@ cmake_minimum_required(VERSION 3.4) -# The `cmake_minimum_required(VERSION 3.4...3.18)` syntax does not work with +# The `cmake_minimum_required(VERSION 3.4...3.21)` syntax does not work with # some versions of VS that have a patched CMake 3.11. This forces us to emulate # the behavior using the following workaround: -if(${CMAKE_VERSION} VERSION_LESS 3.18) +if(${CMAKE_VERSION} VERSION_LESS 3.21) cmake_policy(VERSION ${CMAKE_MAJOR_VERSION}.${CMAKE_MINOR_VERSION}) else() - cmake_policy(VERSION 3.18) + cmake_policy(VERSION 3.21) endif() # Extract project version from source diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index dc70038d..d71a51e6 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -10,10 +10,10 @@ cmake_minimum_required(VERSION 3.4) # The `cmake_minimum_required(VERSION 3.4...3.18)` syntax does not work with # some versions of VS that have a patched CMake 3.11. This forces us to emulate # the behavior using the following workaround: -if(${CMAKE_VERSION} VERSION_LESS 3.18) +if(${CMAKE_VERSION} VERSION_LESS 3.21) cmake_policy(VERSION ${CMAKE_MAJOR_VERSION}.${CMAKE_MINOR_VERSION}) else() - cmake_policy(VERSION 3.18) + cmake_policy(VERSION 3.21) endif() # Only needed for CMake < 3.5 support diff --git a/tests/test_cmake_build/installed_embed/CMakeLists.txt b/tests/test_cmake_build/installed_embed/CMakeLists.txt index 64ae5c4b..f7d69399 100644 --- a/tests/test_cmake_build/installed_embed/CMakeLists.txt +++ b/tests/test_cmake_build/installed_embed/CMakeLists.txt @@ -22,5 +22,7 @@ set_target_properties(test_installed_embed PROPERTIES OUTPUT_NAME test_cmake_bui # This may be needed to resolve header conflicts, e.g. between Python release and debug headers. set_target_properties(test_installed_embed PROPERTIES NO_SYSTEM_FROM_IMPORTED ON) -add_custom_target(check_installed_embed $ - ${PROJECT_SOURCE_DIR}/../test.py) +add_custom_target( + check_installed_embed + $ ${PROJECT_SOURCE_DIR}/../test.py + DEPENDS test_installed_embed) diff --git a/tests/test_cmake_build/installed_function/CMakeLists.txt b/tests/test_cmake_build/installed_function/CMakeLists.txt index 1a502863..d7ca4db5 100644 --- a/tests/test_cmake_build/installed_function/CMakeLists.txt +++ b/tests/test_cmake_build/installed_function/CMakeLists.txt @@ -35,4 +35,5 @@ add_custom_target( PYTHONPATH=$ ${_Python_EXECUTABLE} ${PROJECT_SOURCE_DIR}/../test.py - ${PROJECT_NAME}) + ${PROJECT_NAME} + DEPENDS test_installed_function) diff --git a/tests/test_cmake_build/installed_target/CMakeLists.txt b/tests/test_cmake_build/installed_target/CMakeLists.txt index b38eb774..bc5e101f 100644 --- a/tests/test_cmake_build/installed_target/CMakeLists.txt +++ b/tests/test_cmake_build/installed_target/CMakeLists.txt @@ -42,4 +42,5 @@ add_custom_target( PYTHONPATH=$ ${_Python_EXECUTABLE} ${PROJECT_SOURCE_DIR}/../test.py - ${PROJECT_NAME}) + ${PROJECT_NAME} + DEPENDS test_installed_target) diff --git a/tests/test_cmake_build/subdirectory_embed/CMakeLists.txt b/tests/test_cmake_build/subdirectory_embed/CMakeLists.txt index dfb9cb8f..58cdd7cf 100644 --- a/tests/test_cmake_build/subdirectory_embed/CMakeLists.txt +++ b/tests/test_cmake_build/subdirectory_embed/CMakeLists.txt @@ -23,8 +23,10 @@ add_executable(test_subdirectory_embed ../embed.cpp) target_link_libraries(test_subdirectory_embed PRIVATE pybind11::embed) set_target_properties(test_subdirectory_embed PROPERTIES OUTPUT_NAME test_cmake_build) -add_custom_target(check_subdirectory_embed $ - "${PROJECT_SOURCE_DIR}/../test.py") +add_custom_target( + check_subdirectory_embed + $ "${PROJECT_SOURCE_DIR}/../test.py" + DEPENDS test_subdirectory_embed) # Test custom export group -- PYBIND11_EXPORT_NAME add_library(test_embed_lib ../embed.cpp) diff --git a/tests/test_cmake_build/subdirectory_function/CMakeLists.txt b/tests/test_cmake_build/subdirectory_function/CMakeLists.txt index 34aedcf6..01557c43 100644 --- a/tests/test_cmake_build/subdirectory_function/CMakeLists.txt +++ b/tests/test_cmake_build/subdirectory_function/CMakeLists.txt @@ -31,4 +31,5 @@ add_custom_target( PYTHONPATH=$ ${_Python_EXECUTABLE} ${PROJECT_SOURCE_DIR}/../test.py - ${PROJECT_NAME}) + ${PROJECT_NAME} + DEPENDS test_subdirectory_function) diff --git a/tests/test_cmake_build/subdirectory_target/CMakeLists.txt b/tests/test_cmake_build/subdirectory_target/CMakeLists.txt index 31d862f6..ba82fdee 100644 --- a/tests/test_cmake_build/subdirectory_target/CMakeLists.txt +++ b/tests/test_cmake_build/subdirectory_target/CMakeLists.txt @@ -37,4 +37,5 @@ add_custom_target( PYTHONPATH=$ ${_Python_EXECUTABLE} ${PROJECT_SOURCE_DIR}/../test.py - ${PROJECT_NAME}) + ${PROJECT_NAME} + DEPENDS test_subdirectory_target) diff --git a/tests/test_embed/CMakeLists.txt b/tests/test_embed/CMakeLists.txt index c960c877..3b89d6e5 100644 --- a/tests/test_embed/CMakeLists.txt +++ b/tests/test_embed/CMakeLists.txt @@ -31,6 +31,7 @@ endif() add_custom_target( cpptest COMMAND "$" + DEPENDS test_embed WORKING_DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}") pybind11_add_module(external_module THIN_LTO external_module.cpp) From b1fdbe6954be5afa0a9003c37731da60ac68baff Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 29 Jul 2021 16:10:18 -0700 Subject: [PATCH 014/100] chore: add discussions link (#3159) --- .github/ISSUE_TEMPLATE/bug-report.md | 28 -------------- .github/ISSUE_TEMPLATE/bug-report.yml | 45 +++++++++++++++++++++++ .github/ISSUE_TEMPLATE/config.yml | 3 ++ .github/ISSUE_TEMPLATE/feature-request.md | 16 -------- .github/ISSUE_TEMPLATE/question.md | 21 ----------- README.rst | 4 +- 6 files changed, 51 insertions(+), 66 deletions(-) delete mode 100644 .github/ISSUE_TEMPLATE/bug-report.md create mode 100644 .github/ISSUE_TEMPLATE/bug-report.yml delete mode 100644 .github/ISSUE_TEMPLATE/feature-request.md delete mode 100644 .github/ISSUE_TEMPLATE/question.md diff --git a/.github/ISSUE_TEMPLATE/bug-report.md b/.github/ISSUE_TEMPLATE/bug-report.md deleted file mode 100644 index ae36ea65..00000000 --- a/.github/ISSUE_TEMPLATE/bug-report.md +++ /dev/null @@ -1,28 +0,0 @@ ---- -name: Bug Report -about: File an issue about a bug -title: "[BUG] " ---- - - -Make sure you've completed the following steps before submitting your issue -- thank you! - -1. Make sure you've read the [documentation][]. Your issue may be addressed there. -2. Search the [issue tracker][] to verify that this hasn't already been reported. +1 or comment there if it has. -3. Consider asking first in the [Gitter chat room][]. -4. Include a self-contained and minimal piece of code that reproduces the problem. If that's not possible, try to make the description as clear as possible. - a. If possible, make a PR with a new, failing test to give us a starting point to work on! - -[documentation]: https://pybind11.readthedocs.io -[issue tracker]: https://github.com/pybind/pybind11/issues -[Gitter chat room]: https://gitter.im/pybind/Lobby - -*After reading, remove this checklist and the template text in parentheses below.* - -## Issue description - -(Provide a short description, state the expected behavior and what actually happens.) - -## Reproducible example code - -(The code should be minimal, have no external dependencies, isolate the function(s) that cause breakage. Submit matched and complete C++ and Python snippets that can be easily compiled and run to diagnose the issue.) diff --git a/.github/ISSUE_TEMPLATE/bug-report.yml b/.github/ISSUE_TEMPLATE/bug-report.yml new file mode 100644 index 00000000..bd6a9a8e --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug-report.yml @@ -0,0 +1,45 @@ +name: Bug Report +description: File an issue about a bug +title: "[BUG]: " +labels: [triage] +body: + - type: markdown + attributes: + value: | + Maintainers will only make a best effort to triage PRs. Please do your best to make the issue as easy to act on as possible, and only open if clearly a problem with pybind11 (ask first if unsure). + - type: checkboxes + id: steps + attributes: + label: Required prerequisites + description: Make sure you've completed the following steps before submitting your issue -- thank you! + options: + - label: Make sure you've read the [documentation](https://pybind11.readthedocs.io). Your issue may be addressed there. + required: true + - label: Search the [issue tracker](https://github.com/pybind/pybind11/issues) and [Discussions](https:/pybind/pybind11/discussions) to verify that this hasn't already been reported. +1 or comment there if it has. + required: true + - label: Consider asking first in the [Gitter chat room](https://gitter.im/pybind/Lobby) or in a [Discussion](https:/pybind/pybind11/discussions/new). + required: false + + - type: textarea + id: description + attributes: + label: Problem description + placeholder: >- + Provide a short description, state the expected behavior and what + actually happens. Include relevant information like what version of + pybind11 you are using, what system you are on, and any useful commands + / output. + validations: + required: true + + - type: textarea + id: code + attributes: + label: Reproducible example code + placeholder: >- + The code should be minimal, have no external dependencies, isolate the + function(s) that cause breakage. Submit matched and complete C++ and + Python snippets that can be easily compiled and run to diagnose the + issue. If possible, make a PR with a new, failing test to give us a + starting point to work on! + render: text diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml index 20e74313..27f9a804 100644 --- a/.github/ISSUE_TEMPLATE/config.yml +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -1,5 +1,8 @@ blank_issues_enabled: false contact_links: + - name: Ask a question + url: https://github.com/pybind/pybind11/discussions/new + about: Please ask and answer questions here, or propose new ideas. - name: Gitter room url: https://gitter.im/pybind/Lobby about: A room for discussing pybind11 with an active community diff --git a/.github/ISSUE_TEMPLATE/feature-request.md b/.github/ISSUE_TEMPLATE/feature-request.md deleted file mode 100644 index 5f6ec81e..00000000 --- a/.github/ISSUE_TEMPLATE/feature-request.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -name: Feature Request -about: File an issue about adding a feature -title: "[FEAT] " ---- - - -Make sure you've completed the following steps before submitting your issue -- thank you! - -1. Check if your feature has already been mentioned / rejected / planned in other issues. -2. If those resources didn't help, consider asking in the [Gitter chat room][] to see if this is interesting / useful to a larger audience and possible to implement reasonably, -4. If you have a useful feature that passes the previous items (or not suitable for chat), please fill in the details below. - -[Gitter chat room]: https://gitter.im/pybind/Lobby - -*After reading, remove this checklist.* diff --git a/.github/ISSUE_TEMPLATE/question.md b/.github/ISSUE_TEMPLATE/question.md deleted file mode 100644 index b199b6ee..00000000 --- a/.github/ISSUE_TEMPLATE/question.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -name: Question -about: File an issue about unexplained behavior -title: "[QUESTION] " ---- - -If you have a question, please check the following first: - -1. Check if your question has already been answered in the [FAQ][] section. -2. Make sure you've read the [documentation][]. Your issue may be addressed there. -3. If those resources didn't help and you only have a short question (not a bug report), consider asking in the [Gitter chat room][] -4. Search the [issue tracker][], including the closed issues, to see if your question has already been asked/answered. +1 or comment if it has been asked but has no answer. -5. If you have a more complex question which is not answered in the previous items (or not suitable for chat), please fill in the details below. -6. Include a self-contained and minimal piece of code that illustrates your question. If that's not possible, try to make the description as clear as possible. - -[FAQ]: http://pybind11.readthedocs.io/en/latest/faq.html -[documentation]: https://pybind11.readthedocs.io -[issue tracker]: https://github.com/pybind/pybind11/issues -[Gitter chat room]: https://gitter.im/pybind/Lobby - -*After reading, remove this checklist.* diff --git a/README.rst b/README.rst index 69a7797e..57eb06e5 100644 --- a/README.rst +++ b/README.rst @@ -3,7 +3,7 @@ **pybind11 — Seamless operability between C++11 and Python** -|Latest Documentation Status| |Stable Documentation Status| |Gitter chat| |CI| |Build status| +|Latest Documentation Status| |Stable Documentation Status| |Gitter chat| |GitHub Discussions| |CI| |Build status| |Repology| |PyPI package| |Conda-forge| |Python Versions| @@ -176,3 +176,5 @@ to the terms and conditions of this license. :target: https://repology.org/project/python:pybind11/versions .. |Python Versions| image:: https://img.shields.io/pypi/pyversions/pybind11.svg :target: https://pypi.org/project/pybind11/ +.. |GitHub Discussions| image:: https://img.shields.io/static/v1?label=Discussions&message=Ask&color=blue&logo=github + :target: https://github.com/pybind/pybind11/discussions From b42597291fd59b9bb3af4a9489ca14c910c347d6 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Fri, 30 Jul 2021 07:09:55 -0700 Subject: [PATCH 015/100] Limiting pragma for ignoring GCC 7 -Wnoexcept-type to the scope of pybind11.h. (#3161) * Moving pragma for ignoring -Wnoexcept-type to the one location where it is needed. * Trying a second location. * The previous commit worked (GitHub Actions green), but see the added comment about the dicy nature of -Wnoexcept-type ("if and only if"). * Applying reviewer suggestion. --- include/pybind11/pybind11.h | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 139a4111..16615421 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -18,9 +18,6 @@ # pragma GCC diagnostic push # pragma GCC diagnostic ignored "-Wunused-but-set-parameter" # pragma GCC diagnostic ignored "-Wattributes" -# if __GNUC__ >= 7 -# pragma GCC diagnostic ignored "-Wnoexcept-type" -# endif #endif #include "attr.h" @@ -48,6 +45,18 @@ # include #endif +/* https://stackoverflow.com/questions/46798456/handling-gccs-noexcept-type-warning + This warning is about ABI compatibility, not code health. + It is only actually needed in a couple places, but apparently GCC 7 "generates this warning if + and only if the first template instantiation ... involves noexcept" [stackoverflow], therefore + it could get triggered from seemingly random places, depending on user code. + No other GCC version generates this warning. + */ +#if defined(__GNUC__) && __GNUC__ == 7 +# pragma GCC diagnostic push +# pragma GCC diagnostic ignored "-Wnoexcept-type" +#endif + PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) PYBIND11_NAMESPACE_BEGIN(detail) @@ -2376,6 +2385,10 @@ inline function get_overload(const T *this_ptr, const char *name) { PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) +#if defined(__GNUC__) && __GNUC__ == 7 +# pragma GCC diagnostic pop // -Wnoexcept-type +#endif + #if defined(_MSC_VER) && !defined(__INTEL_COMPILER) # pragma warning(pop) #elif defined(__GNUG__) && !defined(__clang__) && !defined(__INTEL_COMPILER) From 9e8a741baac7485a7f81904bbb9c06a9ae423c40 Mon Sep 17 00:00:00 2001 From: Jerome Robert Date: Fri, 30 Jul 2021 19:48:41 +0200 Subject: [PATCH 016/100] fix: Mingw64 corrected and add a CI job to test it (#3132) * mingw64 platform string is like mingw_xxx not "mingw" See https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-python/0099-Change-the-get_platform-method-in-sysconfig-and-dist.patch * Mingw: Do not dllexport exceptions This is a fix for errors like: D:/a/pybind11/pybind11/include/pybind11/detail/common.h:735:23: error: 'dllexport' implies default visibility, but 'class pybind11::builtin_exception' has already been declared with a different visibility 735 | class PYBIND11_EXPORT builtin_exception : public std::runtime_error { | ^~~~~~~~~~~~~~~~~ * GHA: Test Mingw64 build * fix: avoid thin binaries on mingw * fix: drop lto on MinGW * Mingw64: disable PYBIND11_DEPRECATED It trigger many warnings for unknown reasons Co-authored-by: Henry Schreiner --- .github/workflows/ci.yml | 31 +++++++++++++++++++++++++++++++ include/pybind11/detail/common.h | 20 +++++++++++++++++--- include/pybind11/pytypes.h | 2 +- pybind11/setup_helpers.py | 3 +-- tests/test_exceptions.h | 2 +- tools/pybind11Common.cmake | 7 ++++++- 6 files changed, 57 insertions(+), 8 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 03c9225c..73a6fd51 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -859,3 +859,34 @@ jobs: - name: Run all checks run: cmake --build build -t check + + mingw: + runs-on: windows-latest + defaults: + run: + shell: msys2 {0} + steps: + - uses: msys2/setup-msys2@v2 + with: + install: >- + mingw-w64-x86_64-gcc + mingw-w64-x86_64-python-pip + mingw-w64-x86_64-cmake + mingw-w64-x86_64-make + mingw-w64-x86_64-python-pytest + mingw-w64-x86_64-eigen3 + mingw-w64-x86_64-boost + mingw-w64-x86_64-catch + + - uses: actions/checkout@v1 + + - name: Configure + # LTO leads to many undefined reference like + # `pybind11::detail::function_call::function_call(pybind11::detail::function_call&&) + run: cmake -G "MinGW Makefiles" -S . -B build + + - name: Build + run: cmake --build build -j 2 + + - name: Python tests + run: cmake --build build --target pytest diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 0add6272..d63e3e9c 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -89,13 +89,27 @@ # endif #endif +#if !defined(PYBIND11_EXPORT_EXCEPTION) +# ifdef __MINGW32__ +// workaround for: +// error: 'dllexport' implies default visibility, but xxx has already been declared with a different visibility +# define PYBIND11_EXPORT_EXCEPTION +# else +# define PYBIND11_EXPORT_EXCEPTION PYBIND11_EXPORT +# endif +#endif + #if defined(_MSC_VER) # define PYBIND11_NOINLINE __declspec(noinline) #else # define PYBIND11_NOINLINE __attribute__ ((noinline)) #endif -#if defined(PYBIND11_CPP14) +#if defined(__MINGW32__) +// For unknown reasons all PYBIND11_DEPRECATED member trigger a warning when declared +// whether it is used or not +# define PYBIND11_DEPRECATED(reason) +#elif defined(PYBIND11_CPP14) # define PYBIND11_DEPRECATED(reason) [[deprecated(reason)]] #else # define PYBIND11_DEPRECATED(reason) __attribute__((deprecated(reason))) @@ -740,7 +754,7 @@ PYBIND11_NAMESPACE_END(detail) # pragma warning(disable: 4275) // warning C4275: An exported class was derived from a class that wasn't exported. Can be ignored when derived from a STL class. #endif /// C++ bindings of builtin Python exceptions -class PYBIND11_EXPORT builtin_exception : public std::runtime_error { +class PYBIND11_EXPORT_EXCEPTION builtin_exception : public std::runtime_error { public: using std::runtime_error::runtime_error; /// Set the error using the Python C API @@ -751,7 +765,7 @@ public: #endif #define PYBIND11_RUNTIME_EXCEPTION(name, type) \ - class PYBIND11_EXPORT name : public builtin_exception { public: \ + class PYBIND11_EXPORT_EXCEPTION name : public builtin_exception { public: \ using builtin_exception::builtin_exception; \ name() : name("") { } \ void set_error() const override { PyErr_SetString(type, what()); } \ diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index 161aed06..4cf606e8 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -327,7 +327,7 @@ PYBIND11_NAMESPACE_END(detail) /// thrown to propagate python-side errors back through C++ which can either be caught manually or /// else falls back to the function dispatcher (which then raises the captured error back to /// python). -class PYBIND11_EXPORT error_already_set : public std::runtime_error { +class PYBIND11_EXPORT_EXCEPTION error_already_set : public std::runtime_error { public: /// Constructs a new exception from the current Python error indicator, if any. The current /// Python error indicator will be cleared. diff --git a/pybind11/setup_helpers.py b/pybind11/setup_helpers.py index 2b27f1f5..0888ab48 100644 --- a/pybind11/setup_helpers.py +++ b/pybind11/setup_helpers.py @@ -59,8 +59,7 @@ except ImportError: import distutils.errors import distutils.ccompiler - -WIN = sys.platform.startswith("win32") and sysconfig.get_platform() != "mingw" +WIN = sys.platform.startswith("win32") and "mingw" not in sysconfig.get_platform() PY2 = sys.version_info[0] < 3 MACOS = sys.platform.startswith("darwin") STD_TMPL = "/std:c++{}" if WIN else "-std=c++{}" diff --git a/tests/test_exceptions.h b/tests/test_exceptions.h index 5d02d1b3..9d428312 100644 --- a/tests/test_exceptions.h +++ b/tests/test_exceptions.h @@ -4,7 +4,7 @@ // shared exceptions for cross_module_tests -class PYBIND11_EXPORT shared_exception : public pybind11::builtin_exception { +class PYBIND11_EXPORT_EXCEPTION shared_exception : public pybind11::builtin_exception { public: using builtin_exception::builtin_exception; explicit shared_exception() : shared_exception("") {} diff --git a/tools/pybind11Common.cmake b/tools/pybind11Common.cmake index 57e42536..7afb0d0b 100644 --- a/tools/pybind11Common.cmake +++ b/tools/pybind11Common.cmake @@ -302,13 +302,18 @@ function(_pybind11_return_if_cxx_and_linker_flags_work result cxxflags linkerfla endfunction() function(_pybind11_generate_lto target prefer_thin_lto) + if(MINGW) + message(STATUS "${target} disabled (problems with undefined symbols for MinGW for now)") + return() + endif() + if(CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang") set(cxx_append "") set(linker_append "") if(CMAKE_CXX_COMPILER_ID MATCHES "Clang" AND NOT APPLE) # Clang Gold plugin does not support -Os; append -O3 to MinSizeRel builds to override it set(linker_append ";$<$:-O3>") - elseif(CMAKE_CXX_COMPILER_ID MATCHES "GNU") + elseif(CMAKE_CXX_COMPILER_ID MATCHES "GNU" AND NOT MINGW) set(cxx_append ";-fno-fat-lto-objects") endif() From e2573dc961ec11cd148c50c06abfaee0a6930694 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Fri, 30 Jul 2021 10:51:50 -0700 Subject: [PATCH 017/100] Moving pragma for MSVC warning C4505 from pybind11.h to existing list in detail/common.h (#3160) * Moving pragma for C4505 from pybind11.h to existing list in detail/common.h. * Removing 4 existing suppressions to 1. see what is still needed and 2. capture the MSVC messages. * It turns out none of the 4 pragmas are needed anymore. --- include/pybind11/detail/common.h | 3 ++- include/pybind11/pybind11.h | 1 - 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index d63e3e9c..d1e9e659 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -135,7 +135,8 @@ # define HAVE_ROUND 1 # endif # pragma warning(push) -# pragma warning(disable: 4510 4610 4512 4005) +// C4505: 'PySlice_GetIndicesEx': unreferenced local function has been removed (PyPy only) +# pragma warning(disable: 4505) # if defined(_DEBUG) && !defined(Py_DEBUG) # define PYBIND11_DEBUG_MARKER # undef _DEBUG diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 16615421..ec6bd8d3 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -13,7 +13,6 @@ #if defined(_MSC_VER) && !defined(__INTEL_COMPILER) # pragma warning(push) # pragma warning(disable: 4127) // warning C4127: Conditional expression is constant -# pragma warning(disable: 4505) // warning C4505: 'PySlice_GetIndicesEx': unreferenced local function has been removed (PyPy only) #elif defined(__GNUG__) && !defined(__clang__) && !defined(__INTEL_COMPILER) # pragma GCC diagnostic push # pragma GCC diagnostic ignored "-Wunused-but-set-parameter" From dcbda8d7ff6e0a207807208ff0364bbb74e02175 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Fri, 30 Jul 2021 11:25:29 -0700 Subject: [PATCH 018/100] Removing MSVC C4127 from pragma block at the top of pybind11.h (#3152) * Removing pragma for 4127 (to see what is still broken with the latest code). * Using new constexpr_bool() to suppress warning C4127. * One missed case, Python 2 only. * PYBIND11_SILENCE_MSVC_C4127 (more similar to the approach for C4100). --- include/pybind11/cast.h | 12 ++++++++---- include/pybind11/detail/common.h | 11 +++++++++++ include/pybind11/detail/init.h | 6 +++--- include/pybind11/pybind11.h | 11 +++-------- include/pybind11/pytypes.h | 6 +++--- 5 files changed, 28 insertions(+), 18 deletions(-) diff --git a/include/pybind11/cast.h b/include/pybind11/cast.h index 5ff0355a..211cf09b 100644 --- a/include/pybind11/cast.h +++ b/include/pybind11/cast.h @@ -384,7 +384,11 @@ template struct string_caster { const auto *buffer = reinterpret_cast(PYBIND11_BYTES_AS_STRING(utfNbytes.ptr())); size_t length = (size_t) PYBIND11_BYTES_SIZE(utfNbytes.ptr()) / sizeof(CharT); - if (UTF_N > 8) { buffer++; length--; } // Skip BOM for UTF-16/32 + // Skip BOM for UTF-16/32 + if (PYBIND11_SILENCE_MSVC_C4127(UTF_N > 8)) { + buffer++; + length--; + } value = StringType(buffer, length); // If we're loading a string_view we need to keep the encoded Python object alive: @@ -499,7 +503,7 @@ public: // out how long the first encoded character is in bytes to distinguish between these two // errors. We also allow want to allow unicode characters U+0080 through U+00FF, as those // can fit into a single char value. - if (StringCaster::UTF_N == 8 && str_len > 1 && str_len <= 4) { + if (PYBIND11_SILENCE_MSVC_C4127(StringCaster::UTF_N == 8) && str_len > 1 && str_len <= 4) { auto v0 = static_cast(value[0]); // low bits only: 0-127 // 0b110xxxxx - start of 2-byte sequence @@ -524,7 +528,7 @@ public: // UTF-16 is much easier: we can only have a surrogate pair for values above U+FFFF, thus a // surrogate pair with total length 2 instantly indicates a range error (but not a "your // string was too long" error). - else if (StringCaster::UTF_N == 16 && str_len == 2) { + else if (PYBIND11_SILENCE_MSVC_C4127(StringCaster::UTF_N == 16) && str_len == 2) { one_char = static_cast(value[0]); if (one_char >= 0xD800 && one_char < 0xE000) throw value_error("Character code point not in range(0x10000)"); @@ -778,7 +782,7 @@ struct pyobject_caster { // For Python 2, without this implicit conversion, Python code would // need to be cluttered with six.ensure_text() or similar, only to be // un-cluttered later after Python 2 support is dropped. - if (std::is_same::value && isinstance(src)) { + if (PYBIND11_SILENCE_MSVC_C4127(std::is_same::value) && isinstance(src)) { PyObject *str_from_bytes = PyUnicode_FromEncodedObject(src.ptr(), "utf-8", nullptr); if (!str_from_bytes) throw error_already_set(); value = reinterpret_steal(str_from_bytes); diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index d1e9e659..e8d83ae0 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -941,5 +941,16 @@ inline constexpr void workaround_incorrect_msvc_c4100(Args &&...) {} # define PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(...) #endif +#if defined(_MSC_VER) // All versions (as of July 2021). + +// warning C4127: Conditional expression is constant +constexpr inline bool silence_msvc_c4127(bool cond) { return cond; } + +# define PYBIND11_SILENCE_MSVC_C4127(...) detail::silence_msvc_c4127(__VA_ARGS__) + +#else +# define PYBIND11_SILENCE_MSVC_C4127(...) __VA_ARGS__ +#endif + PYBIND11_NAMESPACE_END(detail) PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) diff --git a/include/pybind11/detail/init.h b/include/pybind11/detail/init.h index 3ebec041..e795da7d 100644 --- a/include/pybind11/detail/init.h +++ b/include/pybind11/detail/init.h @@ -96,7 +96,7 @@ template void construct(value_and_holder &v_h, Cpp *ptr, bool need_alias) { PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(need_alias); no_nullptr(ptr); - if (Class::has_alias && need_alias && !is_alias(ptr)) { + if (PYBIND11_SILENCE_MSVC_C4127(Class::has_alias) && need_alias && !is_alias(ptr)) { // We're going to try to construct an alias by moving the cpp type. Whether or not // that succeeds, we still need to destroy the original cpp pointer (either the // moved away leftover, if the alias construction works, or the value itself if we @@ -136,7 +136,7 @@ void construct(value_and_holder &v_h, Holder holder, bool need_alias) { auto *ptr = holder_helper>::get(holder); no_nullptr(ptr); // If we need an alias, check that the held pointer is actually an alias instance - if (Class::has_alias && need_alias && !is_alias(ptr)) + if (PYBIND11_SILENCE_MSVC_C4127(Class::has_alias) && need_alias && !is_alias(ptr)) throw type_error("pybind11::init(): construction failed: returned holder-wrapped instance " "is not an alias instance"); @@ -153,7 +153,7 @@ void construct(value_and_holder &v_h, Cpp &&result, bool need_alias) { PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(need_alias); static_assert(std::is_move_constructible>::value, "pybind11::init() return-by-value factory function requires a movable class"); - if (Class::has_alias && need_alias) + if (PYBIND11_SILENCE_MSVC_C4127(Class::has_alias) && need_alias) construct_alias_from_cpp(is_alias_constructible{}, v_h, std::move(result)); else v_h.value_ptr() = new Cpp(std::move(result)); diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index ec6bd8d3..aeeb3368 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -10,10 +10,7 @@ #pragma once -#if defined(_MSC_VER) && !defined(__INTEL_COMPILER) -# pragma warning(push) -# pragma warning(disable: 4127) // warning C4127: Conditional expression is constant -#elif defined(__GNUG__) && !defined(__clang__) && !defined(__INTEL_COMPILER) +#if defined(__GNUG__) && !defined(__clang__) && !defined(__INTEL_COMPILER) # pragma GCC diagnostic push # pragma GCC diagnostic ignored "-Wunused-but-set-parameter" # pragma GCC diagnostic ignored "-Wattributes" @@ -166,7 +163,7 @@ protected: auto rec = unique_rec.get(); /* Store the capture object directly in the function record if there is enough space */ - if (sizeof(capture) <= sizeof(rec->data)) { + if (PYBIND11_SILENCE_MSVC_C4127(sizeof(capture) <= sizeof(rec->data))) { /* Without these pragmas, GCC warns that there might not be enough space to use the placement new operator. However, the 'if' statement above ensures that this is the case. */ @@ -2388,8 +2385,6 @@ PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) # pragma GCC diagnostic pop // -Wnoexcept-type #endif -#if defined(_MSC_VER) && !defined(__INTEL_COMPILER) -# pragma warning(pop) -#elif defined(__GNUG__) && !defined(__clang__) && !defined(__INTEL_COMPILER) +#if defined(__GNUG__) && !defined(__clang__) && !defined(__INTEL_COMPILER) # pragma GCC diagnostic pop #endif diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index 4cf606e8..c7b2501f 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -1191,9 +1191,9 @@ PYBIND11_NAMESPACE_BEGIN(detail) // unsigned type: (A)-1 != (B)-1 when A and B are unsigned types of different sizes). template Unsigned as_unsigned(PyObject *o) { - if (sizeof(Unsigned) <= sizeof(unsigned long) + if (PYBIND11_SILENCE_MSVC_C4127(sizeof(Unsigned) <= sizeof(unsigned long)) #if PY_VERSION_HEX < 0x03000000 - || PyInt_Check(o) + || PyInt_Check(o) #endif ) { unsigned long v = PyLong_AsUnsignedLong(o); @@ -1212,7 +1212,7 @@ public: template ::value, int> = 0> int_(T value) { - if (sizeof(T) <= sizeof(long)) { + if (PYBIND11_SILENCE_MSVC_C4127(sizeof(T) <= sizeof(long))) { if (std::is_signed::value) m_ptr = PyLong_FromLong((long) value); else From f4f4632e283ce5476964e9f612ecb7d7915fb035 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Tue, 3 Aug 2021 10:56:57 -0400 Subject: [PATCH 019/100] [pre-commit.ci] pre-commit autoupdate (#3167) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit updates: - [github.com/asottile/pyupgrade: v2.23.0 → v2.23.1](https://github.com/asottile/pyupgrade/compare/v2.23.0...v2.23.1) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .pre-commit-config.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 510b8412..57beb777 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -31,7 +31,7 @@ repos: exclude: ^noxfile.py$ - repo: https://github.com/asottile/pyupgrade - rev: v2.23.0 + rev: v2.23.1 hooks: - id: pyupgrade From c0756ccd932de6e8d52ef102de0e18558f157db9 Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Tue, 3 Aug 2021 13:15:48 -0400 Subject: [PATCH 020/100] fix: func_handle for rule of two (#3169) * Fix func_handle for rule of two * Apply reviewer suggestion --- include/pybind11/functional.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/include/pybind11/functional.h b/include/pybind11/functional.h index b9bf3891..bc8a8af8 100644 --- a/include/pybind11/functional.h +++ b/include/pybind11/functional.h @@ -70,9 +70,11 @@ public: struct func_handle { function f; func_handle(function &&f_) noexcept : f(std::move(f_)) {} - func_handle(const func_handle& f_) { + func_handle(const func_handle &f_) { operator=(f_); } + func_handle &operator=(const func_handle &f_) { gil_scoped_acquire acq; f = f_.f; + return *this; } ~func_handle() { gil_scoped_acquire acq; From a2b78a8c279e54f01a0502405fd2335a59c43a6d Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Tue, 3 Aug 2021 10:16:14 -0700 Subject: [PATCH 021/100] chore: changelog update (#3163) * chore: changelog update * Update docs/changelog.rst --- docs/changelog.rst | 49 ++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 47 insertions(+), 2 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 5e8a6325..2f76abe0 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -6,9 +6,54 @@ Changelog Starting with version 1.8.0, pybind11 releases use a `semantic versioning `_ policy. -Next version (WIP) ------------------- +v2.8.0 (WIP) +------------ +* Allow exception translators to be optionally registered local to a module + instead of applying globally across all pybind11 modules. Use + ``register_local_exception_translator(ExceptionTranslator&& translator)`` + instead of ``register_exception_translator(ExceptionTranslator&& + translator)`` to keep your exception remapping code local to the module. + `#2650 `_ + +v2.7.1 (Aug 3, 2021) +--------------------- + +Minor missing functionality added: + +* Allow Python builtins to be used as callbacks in CPython. + `#1413 `_ + +Bug fixes: + +* Fix regression in CMake Python package config: improper use of absolute path. + `#3144 `_ + +* Fix Mingw64 and add to the CI testing matrix. + `#3132 `_ + +* Specified UTF8-encoding in setup.py calls of open(). + `#3137 `_ + +* Add clang-tidy-readability rules to make boolean casts explicit improving + code readability. Also enabled other misc and readability clang-tidy checks. + `#3148 `_ + +* Move object in ``.pop()`` for list. + `#3116 `_ + +Backend and tidying up: + +* Removed and fixed warning suppressions. + `#3127 `_ + `#3129 `_ + `#3135 `_ + `#3141 `_ + `#3142 `_ + `#3150 `_ + `#3152 `_ + `#3160 `_ + `#3161 `_ v2.7.0 (Jul 16, 2021) From 5f34c42d70d6af0af546106503da9d045efb904e Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Tue, 3 Aug 2021 15:05:54 -0400 Subject: [PATCH 022/100] chore: bump to version 2.7.1 --- include/pybind11/detail/common.h | 2 +- pybind11/_version.py | 2 +- setup.cfg | 1 + 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index e8d83ae0..1d771433 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -11,7 +11,7 @@ #define PYBIND11_VERSION_MAJOR 2 #define PYBIND11_VERSION_MINOR 7 -#define PYBIND11_VERSION_PATCH 1.dev1 +#define PYBIND11_VERSION_PATCH 1 // Similar to Python's convention: https://docs.python.org/3/c-api/apiabiversion.html // Additional convention: 0xD = dev diff --git a/pybind11/_version.py b/pybind11/_version.py index 2bb14b96..2bb4fe36 100644 --- a/pybind11/_version.py +++ b/pybind11/_version.py @@ -8,5 +8,5 @@ def _to_int(s): return s -__version__ = "2.7.1.dev1" +__version__ = "2.7.1" version_info = tuple(_to_int(s) for s in __version__.split(".")) diff --git a/setup.cfg b/setup.cfg index 31038eb0..95963d2f 100644 --- a/setup.cfg +++ b/setup.cfg @@ -20,6 +20,7 @@ classifiers = Programming Language :: Python :: 3.7 Programming Language :: Python :: 3.8 Programming Language :: Python :: 3.9 + Programming Language :: Python :: 3.10 License :: OSI Approved :: BSD License Programming Language :: Python :: Implementation :: PyPy Programming Language :: Python :: Implementation :: CPython From 82adacb31de4d24983e240a60d608dfaed464d1a Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Tue, 3 Aug 2021 15:20:23 -0400 Subject: [PATCH 023/100] fix: include hex version in bump --- include/pybind11/detail/common.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 1d771433..b9185840 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -15,7 +15,7 @@ // Similar to Python's convention: https://docs.python.org/3/c-api/apiabiversion.html // Additional convention: 0xD = dev -#define PYBIND11_VERSION_HEX 0x020701D1 +#define PYBIND11_VERSION_HEX 0x02070100 #define PYBIND11_NAMESPACE_BEGIN(name) namespace name { #define PYBIND11_NAMESPACE_END(name) } From c30f57d2ede50c0306db54d5fb215f57c1e465a6 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Tue, 3 Aug 2021 16:11:07 -0400 Subject: [PATCH 024/100] chore: start development for 2.8.0 --- include/pybind11/detail/common.h | 6 +++--- pybind11/_version.py | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index b9185840..adb00093 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -10,12 +10,12 @@ #pragma once #define PYBIND11_VERSION_MAJOR 2 -#define PYBIND11_VERSION_MINOR 7 -#define PYBIND11_VERSION_PATCH 1 +#define PYBIND11_VERSION_MINOR 8 +#define PYBIND11_VERSION_PATCH 0.dev1 // Similar to Python's convention: https://docs.python.org/3/c-api/apiabiversion.html // Additional convention: 0xD = dev -#define PYBIND11_VERSION_HEX 0x02070100 +#define PYBIND11_VERSION_HEX 0x020800D0 #define PYBIND11_NAMESPACE_BEGIN(name) namespace name { #define PYBIND11_NAMESPACE_END(name) } diff --git a/pybind11/_version.py b/pybind11/_version.py index 2bb4fe36..610d39bf 100644 --- a/pybind11/_version.py +++ b/pybind11/_version.py @@ -8,5 +8,5 @@ def _to_int(s): return s -__version__ = "2.7.1" +__version__ = "2.8.0.dev1" version_info = tuple(_to_int(s) for s in __version__.split(".")) From 5f4d725918d7fb90d441b8c1ee1a2cf464407a6a Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Tue, 3 Aug 2021 17:03:11 -0400 Subject: [PATCH 025/100] fix: version number hex --- include/pybind11/detail/common.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index adb00093..8b8f5da3 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -15,7 +15,7 @@ // Similar to Python's convention: https://docs.python.org/3/c-api/apiabiversion.html // Additional convention: 0xD = dev -#define PYBIND11_VERSION_HEX 0x020800D0 +#define PYBIND11_VERSION_HEX 0x020800D1 #define PYBIND11_NAMESPACE_BEGIN(name) namespace name { #define PYBIND11_NAMESPACE_END(name) } From 1fafd1b44796f39e56d4c029203c0e2ba18a60dd Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Fri, 6 Aug 2021 13:03:26 -0400 Subject: [PATCH 026/100] fix: apply simpler expression with fewer workarounds --- include/pybind11/detail/type_caster_base.h | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h index e2d1bcb8..a708d87f 100644 --- a/include/pybind11/detail/type_caster_base.h +++ b/include/pybind11/detail/type_caster_base.h @@ -927,18 +927,17 @@ protected: using Constructor = void *(*)(const void *); /* Only enabled when the types are {copy,move}-constructible *and* when the type - does not have a private operator new implementation. */ + does not have a private operator new implementation. A comma operator is used in the decltype + argument to apply SFINAE to the public copy/move constructors.*/ template ::value>> - static auto make_copy_constructor(const T *x) -> decltype(new T(*x), Constructor{}) { - PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(x); + static auto make_copy_constructor(const T *) -> decltype(new T(std::declval()), Constructor{}) { return [](const void *arg) -> void * { return new T(*reinterpret_cast(arg)); }; } template ::value>> - static auto make_move_constructor(const T *x) -> decltype(new T(std::move(*const_cast(x))), Constructor{}) { - PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(x); + static auto make_move_constructor(const T *) -> decltype(new T(std::declval()), Constructor{}) { return [](const void *arg) -> void * { return new T(std::move(*const_cast(reinterpret_cast(arg)))); }; From 089328f77955685a29db9b9cf05e7c0893ee031b Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Fri, 6 Aug 2021 13:09:48 -0400 Subject: [PATCH 027/100] Revert "fix: apply simpler expression with fewer workarounds" This reverts commit 1fafd1b44796f39e56d4c029203c0e2ba18a60dd. --- include/pybind11/detail/type_caster_base.h | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h index a708d87f..e2d1bcb8 100644 --- a/include/pybind11/detail/type_caster_base.h +++ b/include/pybind11/detail/type_caster_base.h @@ -927,17 +927,18 @@ protected: using Constructor = void *(*)(const void *); /* Only enabled when the types are {copy,move}-constructible *and* when the type - does not have a private operator new implementation. A comma operator is used in the decltype - argument to apply SFINAE to the public copy/move constructors.*/ + does not have a private operator new implementation. */ template ::value>> - static auto make_copy_constructor(const T *) -> decltype(new T(std::declval()), Constructor{}) { + static auto make_copy_constructor(const T *x) -> decltype(new T(*x), Constructor{}) { + PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(x); return [](const void *arg) -> void * { return new T(*reinterpret_cast(arg)); }; } template ::value>> - static auto make_move_constructor(const T *) -> decltype(new T(std::declval()), Constructor{}) { + static auto make_move_constructor(const T *x) -> decltype(new T(std::move(*const_cast(x))), Constructor{}) { + PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(x); return [](const void *arg) -> void * { return new T(std::move(*const_cast(reinterpret_cast(arg)))); }; From 3893f37bcec9f0e9c8ab2c0069cfec8520591860 Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Fri, 6 Aug 2021 14:30:28 -0400 Subject: [PATCH 028/100] maint(clang-tidy): Bugprone enable checks (#3166) * Enable bugprone checks * Reset delta and massage config * Start to apply bugprone fixes * try to fix minor bug * Fix later * Fix perfect forwarding bugprone * Remove nolint * undo constructor delete * Fix bugprone-perfect-forwarding again * Remove TODO * Add another nolint for bugprone-exception-escape in scoped interpreter * Fix remaining bugprone errors * Properly apply bugprone-macro-parantheses * Redo formatting and remove bugprone nolint * Add coment and revert more whitespace changes * Fix typo * Fix parsing bug * Add back comma * Fix clang-tidy issue * Apply remaining clang-tidy fixes --- .clang-tidy | 4 +++ include/pybind11/cast.h | 11 ++++---- include/pybind11/detail/common.h | 39 ++++++++++++++--------------- include/pybind11/embed.h | 36 +++++++++++++------------- include/pybind11/numpy.h | 7 ++++-- include/pybind11/pybind11.h | 2 +- tests/object.h | 6 ++++- tests/pybind11_tests.h | 9 +++---- tests/test_buffers.cpp | 6 ++++- tests/test_class.cpp | 8 +++--- tests/test_multiple_inheritance.cpp | 5 +++- tests/test_numpy_dtypes.cpp | 12 +++++---- tests/test_smart_ptr.cpp | 1 + tests/test_tagbased_polymorphic.cpp | 6 ++--- tests/test_virtual_functions.cpp | 1 + 15 files changed, 85 insertions(+), 68 deletions(-) diff --git a/.clang-tidy b/.clang-tidy index db5077c2..c83b9b2f 100644 --- a/.clang-tidy +++ b/.clang-tidy @@ -1,6 +1,7 @@ FormatStyle: file Checks: ' +*bugprone*, cppcoreguidelines-init-variables, clang-analyzer-optin.cplusplus.VirtualCall, llvm-namespace-comment, @@ -43,6 +44,9 @@ readability-static-accessed-through-instance, readability-static-definition-in-anonymous-namespace, readability-string-compare, readability-uniqueptr-delete-release, +-bugprone-exception-escape, +-bugprone-reserved-identifier, +-bugprone-unused-raii, ' CheckOptions: diff --git a/include/pybind11/cast.h b/include/pybind11/cast.h index 211cf09b..988a2369 100644 --- a/include/pybind11/cast.h +++ b/include/pybind11/cast.h @@ -102,9 +102,9 @@ public: } \ return cast(*src, policy, parent); \ } \ - operator type *() { return &value; } \ - operator type &() { return value; } \ - operator type &&() && { return std::move(value); } \ + operator type *() { return &value; } /* NOLINT(bugprone-macro-parentheses) */ \ + operator type &() { return value; } /* NOLINT(bugprone-macro-parentheses) */ \ + operator type &&() && { return std::move(value); } /* NOLINT(bugprone-macro-parentheses) */ \ template \ using cast_op_type = pybind11::detail::movable_cast_op_type @@ -145,9 +145,8 @@ public: py_value = (py_type) PyFloat_AsDouble(src.ptr()); else return false; - } else if (PyFloat_Check(src.ptr())) { - return false; - } else if (!convert && !PYBIND11_LONG_CHECK(src.ptr()) && !index_check(src.ptr())) { + } else if (PyFloat_Check(src.ptr()) + || (!convert && !PYBIND11_LONG_CHECK(src.ptr()) && !index_check(src.ptr()))) { return false; } else { handle src_or_index = src; diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 8b8f5da3..27a79bfd 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -220,8 +220,8 @@ #define PYBIND11_BYTES_SIZE PyBytes_Size #define PYBIND11_LONG_CHECK(o) PyLong_Check(o) #define PYBIND11_LONG_AS_LONGLONG(o) PyLong_AsLongLong(o) -#define PYBIND11_LONG_FROM_SIGNED(o) PyLong_FromSsize_t((ssize_t) o) -#define PYBIND11_LONG_FROM_UNSIGNED(o) PyLong_FromSize_t((size_t) o) +#define PYBIND11_LONG_FROM_SIGNED(o) PyLong_FromSsize_t((ssize_t) (o)) +#define PYBIND11_LONG_FROM_UNSIGNED(o) PyLong_FromSize_t((size_t) (o)) #define PYBIND11_BYTES_NAME "bytes" #define PYBIND11_STRING_NAME "str" #define PYBIND11_SLICE_OBJECT PyObject @@ -356,24 +356,23 @@ extern "C" { }); } \endrst */ -#define PYBIND11_MODULE(name, variable) \ - static ::pybind11::module_::module_def \ - PYBIND11_CONCAT(pybind11_module_def_, name) PYBIND11_MAYBE_UNUSED; \ - PYBIND11_MAYBE_UNUSED \ - static void PYBIND11_CONCAT(pybind11_init_, name)(::pybind11::module_ &); \ - PYBIND11_PLUGIN_IMPL(name) { \ - PYBIND11_CHECK_PYTHON_VERSION \ - PYBIND11_ENSURE_INTERNALS_READY \ - auto m = ::pybind11::module_::create_extension_module( \ - PYBIND11_TOSTRING(name), nullptr, \ - &PYBIND11_CONCAT(pybind11_module_def_, name)); \ - try { \ - PYBIND11_CONCAT(pybind11_init_, name)(m); \ - return m.ptr(); \ - } PYBIND11_CATCH_INIT_EXCEPTIONS \ - } \ - void PYBIND11_CONCAT(pybind11_init_, name)(::pybind11::module_ &variable) - +#define PYBIND11_MODULE(name, variable) \ + static ::pybind11::module_::module_def PYBIND11_CONCAT(pybind11_module_def_, name) \ + PYBIND11_MAYBE_UNUSED; \ + PYBIND11_MAYBE_UNUSED \ + static void PYBIND11_CONCAT(pybind11_init_, name)(::pybind11::module_ &); \ + PYBIND11_PLUGIN_IMPL(name) { \ + PYBIND11_CHECK_PYTHON_VERSION \ + PYBIND11_ENSURE_INTERNALS_READY \ + auto m = ::pybind11::module_::create_extension_module( \ + PYBIND11_TOSTRING(name), nullptr, &PYBIND11_CONCAT(pybind11_module_def_, name)); \ + try { \ + PYBIND11_CONCAT(pybind11_init_, name)(m); \ + return m.ptr(); \ + } \ + PYBIND11_CATCH_INIT_EXCEPTIONS \ + } \ + void PYBIND11_CONCAT(pybind11_init_, name)(::pybind11::module_ & (variable)) PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) diff --git a/include/pybind11/embed.h b/include/pybind11/embed.h index abb1cd3c..6e777830 100644 --- a/include/pybind11/embed.h +++ b/include/pybind11/embed.h @@ -45,25 +45,23 @@ }); } \endrst */ -#define PYBIND11_EMBEDDED_MODULE(name, variable) \ - static ::pybind11::module_::module_def \ - PYBIND11_CONCAT(pybind11_module_def_, name); \ - static void PYBIND11_CONCAT(pybind11_init_, name)(::pybind11::module_ &); \ - static PyObject PYBIND11_CONCAT(*pybind11_init_wrapper_, name)() { \ - auto m = ::pybind11::module_::create_extension_module( \ - PYBIND11_TOSTRING(name), nullptr, \ - &PYBIND11_CONCAT(pybind11_module_def_, name)); \ - try { \ - PYBIND11_CONCAT(pybind11_init_, name)(m); \ - return m.ptr(); \ - } PYBIND11_CATCH_INIT_EXCEPTIONS \ - } \ - PYBIND11_EMBEDDED_MODULE_IMPL(name) \ - ::pybind11::detail::embedded_module PYBIND11_CONCAT(pybind11_module_, name) \ - (PYBIND11_TOSTRING(name), \ - PYBIND11_CONCAT(pybind11_init_impl_, name)); \ - void PYBIND11_CONCAT(pybind11_init_, name)(::pybind11::module_ &variable) - +#define PYBIND11_EMBEDDED_MODULE(name, variable) \ + static ::pybind11::module_::module_def PYBIND11_CONCAT(pybind11_module_def_, name); \ + static void PYBIND11_CONCAT(pybind11_init_, name)(::pybind11::module_ &); \ + static PyObject PYBIND11_CONCAT(*pybind11_init_wrapper_, name)() { \ + auto m = ::pybind11::module_::create_extension_module( \ + PYBIND11_TOSTRING(name), nullptr, &PYBIND11_CONCAT(pybind11_module_def_, name)); \ + try { \ + PYBIND11_CONCAT(pybind11_init_, name)(m); \ + return m.ptr(); \ + } \ + PYBIND11_CATCH_INIT_EXCEPTIONS \ + } \ + PYBIND11_EMBEDDED_MODULE_IMPL(name) \ + ::pybind11::detail::embedded_module PYBIND11_CONCAT(pybind11_module_, name)( \ + PYBIND11_TOSTRING(name), PYBIND11_CONCAT(pybind11_init_impl_, name)); \ + void PYBIND11_CONCAT(pybind11_init_, name)(::pybind11::module_ \ + & variable) // NOLINT(bugprone-macro-parentheses) PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) PYBIND11_NAMESPACE_BEGIN(detail) diff --git a/include/pybind11/numpy.h b/include/pybind11/numpy.h index 7313897f..aa8294e1 100644 --- a/include/pybind11/numpy.h +++ b/include/pybind11/numpy.h @@ -1551,8 +1551,11 @@ private: "pybind11::vectorize(...) requires a function with at least one vectorizable argument"); public: - template - explicit vectorize_helper(T &&f) : f(std::forward(f)) { } + template ::type>::value>> + explicit vectorize_helper(T &&f) : f(std::forward(f)) {} object operator()(typename vectorize_arg::type... args) { return run(args..., diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index aeeb3368..54213297 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -1726,7 +1726,7 @@ struct enum_base { m_base.attr(op) = cpp_function( \ [](const object &a, const object &b) { \ if (!type::handle_of(a).is(type::handle_of(b))) \ - strict_behavior; \ + strict_behavior; /* NOLINT(bugprone-macro-parentheses) */ \ return expr; \ }, \ name(op), \ diff --git a/tests/object.h b/tests/object.h index 865a9bea..6851fc62 100644 --- a/tests/object.h +++ b/tests/object.h @@ -110,7 +110,11 @@ public: /// Overwrite this reference with another reference ref& operator=(const ref& r) { - print_copy_assigned(this, "pointer", r.m_ptr); track_copy_assigned((ref_tag*) this); + if (this == &r) { + return *this; + } + print_copy_assigned(this, "pointer", r.m_ptr); + track_copy_assigned((ref_tag *) this); if (m_ptr == r.m_ptr) return *this; diff --git a/tests/pybind11_tests.h b/tests/pybind11_tests.h index d970ba8b..12d8a777 100644 --- a/tests/pybind11_tests.h +++ b/tests/pybind11_tests.h @@ -23,11 +23,10 @@ public: test_initializer(const char *submodule_name, Initializer init); }; -#define TEST_SUBMODULE(name, variable) \ - void test_submodule_##name(py::module_ &); \ - test_initializer name(#name, test_submodule_##name); \ - void test_submodule_##name(py::module_ &variable) - +#define TEST_SUBMODULE(name, variable) \ + void test_submodule_##name(py::module_ &); \ + test_initializer name(#name, test_submodule_##name); \ + void test_submodule_##name(py::module_ &(variable)) /// Dummy type which is not exported anywhere -- something to trigger a conversion error struct UnregisteredType { }; diff --git a/tests/test_buffers.cpp b/tests/test_buffers.cpp index e77c626f..c7e2c7df 100644 --- a/tests/test_buffers.cpp +++ b/tests/test_buffers.cpp @@ -40,7 +40,11 @@ TEST_SUBMODULE(buffers, m) { } Matrix &operator=(const Matrix &s) { - print_copy_assigned(this, std::to_string(m_rows) + "x" + std::to_string(m_cols) + " matrix"); + if (this == &s) { + return *this; + } + print_copy_assigned(this, + std::to_string(m_rows) + "x" + std::to_string(m_cols) + " matrix"); delete[] m_data; m_rows = s.m_rows; m_cols = s.m_cols; diff --git a/tests/test_class.cpp b/tests/test_class.cpp index 73de2a61..dff75839 100644 --- a/tests/test_class.cpp +++ b/tests/test_class.cpp @@ -492,15 +492,15 @@ using DoesntBreak5 = py::class_>; using DoesntBreak6 = py::class_, std::shared_ptr>, BreaksTramp<6>>; using DoesntBreak7 = py::class_, BreaksTramp<7>, std::shared_ptr>>; using DoesntBreak8 = py::class_, std::shared_ptr>>; -#define CHECK_BASE(N) static_assert(std::is_same>::value, \ +#define CHECK_BASE(N) static_assert(std::is_same>::value, \ "DoesntBreak" #N " has wrong type!") CHECK_BASE(1); CHECK_BASE(2); CHECK_BASE(3); CHECK_BASE(4); CHECK_BASE(5); CHECK_BASE(6); CHECK_BASE(7); CHECK_BASE(8); -#define CHECK_ALIAS(N) static_assert(DoesntBreak##N::has_alias && std::is_same>::value, \ +#define CHECK_ALIAS(N) static_assert(DoesntBreak##N::has_alias && std::is_same>::value, \ "DoesntBreak" #N " has wrong type_alias!") #define CHECK_NOALIAS(N) static_assert(!DoesntBreak##N::has_alias && std::is_void::value, \ "DoesntBreak" #N " has type alias, but shouldn't!") CHECK_ALIAS(1); CHECK_ALIAS(2); CHECK_NOALIAS(3); CHECK_ALIAS(4); CHECK_NOALIAS(5); CHECK_ALIAS(6); CHECK_ALIAS(7); CHECK_NOALIAS(8); -#define CHECK_HOLDER(N, TYPE) static_assert(std::is_same>>::value, \ +#define CHECK_HOLDER(N, TYPE) static_assert(std::is_same>>::value, \ "DoesntBreak" #N " has wrong holder_type!") CHECK_HOLDER(1, unique); CHECK_HOLDER(2, unique); CHECK_HOLDER(3, unique); CHECK_HOLDER(4, unique); CHECK_HOLDER(5, unique); CHECK_HOLDER(6, shared); CHECK_HOLDER(7, shared); CHECK_HOLDER(8, shared); @@ -510,7 +510,7 @@ CHECK_HOLDER(6, shared); CHECK_HOLDER(7, shared); CHECK_HOLDER(8, shared); // failures occurs). // We have to actually look into the type: the typedef alone isn't enough to instantiate the type: -#define CHECK_BROKEN(N) static_assert(std::is_same>::value, \ +#define CHECK_BROKEN(N) static_assert(std::is_same>::value, \ "Breaks1 has wrong type!"); //// Two holder classes: diff --git a/tests/test_multiple_inheritance.cpp b/tests/test_multiple_inheritance.cpp index d6b24d34..b5ca298d 100644 --- a/tests/test_multiple_inheritance.cpp +++ b/tests/test_multiple_inheritance.cpp @@ -108,7 +108,10 @@ TEST_SUBMODULE(multiple_inheritance, m) { // test_multiple_inheritance_python_many_bases - #define PYBIND11_BASEN(N) py::class_>(m, "BaseN" #N).def(py::init()).def("f" #N, [](BaseN &b) { return b.i + N; }) +#define PYBIND11_BASEN(N) \ + py::class_>(m, "BaseN" #N).def(py::init()).def("f" #N, [](BaseN &b) { \ + return b.i + (N); \ + }) PYBIND11_BASEN( 1); PYBIND11_BASEN( 2); PYBIND11_BASEN( 3); PYBIND11_BASEN( 4); PYBIND11_BASEN( 5); PYBIND11_BASEN( 6); PYBIND11_BASEN( 7); PYBIND11_BASEN( 8); PYBIND11_BASEN( 9); PYBIND11_BASEN(10); PYBIND11_BASEN(11); PYBIND11_BASEN(12); diff --git a/tests/test_numpy_dtypes.cpp b/tests/test_numpy_dtypes.cpp index 340c972d..bf4f4cee 100644 --- a/tests/test_numpy_dtypes.cpp +++ b/tests/test_numpy_dtypes.cpp @@ -148,11 +148,13 @@ py::array mkarray_via_buffer(size_t n) { 1, { n }, { sizeof(T) })); } -#define SET_TEST_VALS(s, i) do { \ - s.bool_ = (i) % 2 != 0; \ - s.uint_ = (uint32_t) (i); \ - s.float_ = (float) (i) * 1.5f; \ - s.ldbl_ = (long double) (i) * -2.5L; } while (0) +#define SET_TEST_VALS(s, i) \ + do { \ + (s).bool_ = (i) % 2 != 0; \ + (s).uint_ = (uint32_t) (i); \ + (s).float_ = (float) (i) *1.5f; \ + (s).ldbl_ = (long double) (i) * -2.5L; \ + } while (0) template py::array_t create_recarray(size_t n) { diff --git a/tests/test_smart_ptr.cpp b/tests/test_smart_ptr.cpp index 57b2d894..eeaa4414 100644 --- a/tests/test_smart_ptr.cpp +++ b/tests/test_smart_ptr.cpp @@ -182,6 +182,7 @@ struct SharedPtrRef { struct SharedFromThisRef { struct B : std::enable_shared_from_this { B() { print_created(this); } + // NOLINTNEXTLINE(bugprone-copy-constructor-init) B(const B &) : std::enable_shared_from_this() { print_copy_created(this); } B(B &&) noexcept : std::enable_shared_from_this() { print_move_created(this); } ~B() { print_destroyed(this); } diff --git a/tests/test_tagbased_polymorphic.cpp b/tests/test_tagbased_polymorphic.cpp index 838a168d..90f40e14 100644 --- a/tests/test_tagbased_polymorphic.cpp +++ b/tests/test_tagbased_polymorphic.cpp @@ -86,13 +86,13 @@ std::vector> create_zoo() const std::type_info* Animal::type_of_kind(Kind kind) { switch (kind) { - case Kind::Unknown: break; - + case Kind::Unknown: case Kind::Dog: break; + case Kind::Labrador: return &typeid(Labrador); case Kind::Chihuahua: return &typeid(Chihuahua); - case Kind::LastDog: break; + case Kind::LastDog: case Kind::Cat: break; case Kind::Panther: return &typeid(Panther); case Kind::LastCat: break; diff --git a/tests/test_virtual_functions.cpp b/tests/test_virtual_functions.cpp index 5280af8e..f83a7364 100644 --- a/tests/test_virtual_functions.cpp +++ b/tests/test_virtual_functions.cpp @@ -454,6 +454,7 @@ template class PyB_Tpl : public PyA_Tpl { public: using PyA_Tpl::PyA_Tpl; // Inherit constructors (via PyA_Tpl's inherited constructors) + // NOLINTNEXTLINE(bugprone-parent-virtual-call) int unlucky_number() override { PYBIND11_OVERRIDE(int, Base, unlucky_number, ); } double lucky_number() override { PYBIND11_OVERRIDE(double, Base, lucky_number, ); } }; From af7007331ab5a6f82a1fe38f041b624dea0b4084 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Fri, 6 Aug 2021 12:27:11 -0700 Subject: [PATCH 029/100] Removing GCC -Wunused-but-set-parameter from pragma block at the top of pybind11.h (#3164) * Cleanup triggered by work on pragma for GCC -Wunused-but-set-parameter. * Backing out changes to eigen.h (to be worked on later). * Adding PYBIND11_WORKAROUND_INCORRECT_GCC_UNUSED_BUT_SET_PARAMETER in type_caster_base.h (apparently needed only for older GCCs). * Apparently older compilers need a simpler overload for silence_unused_warnings(). * clang C++11 compatibility: removing constexpr * Special case for MSVC 2017: `constexpr void` return * Trying again without the silence_unused_warnings(const int *) overload. * Separate macros for ALL_GCC, OLD_GCC_UNUSED_BUT_SET_PARAMETER * Changing to __GNUC__ <= 2 (turning off) * Refined condition for PYBIND11_WORKAROUND_INCORRECT_OLD_GCC_UNUSED_BUT_SET_PARAMETER. * Quick experiment trying out suggestion by @henryiii * Introducing macro: PYBIND11_INT_ARRAY_WORKING_AROUND_MSVC_CLANG_ISSUES * Trying henryiii@ (void) expander idea. * fix: apply simpler expression with fewer workarounds * Purging new-but-already-obsoleted macro, made possible by @henryiii's commit. * Renaming `ALL_GCC` macro back to just `GCC` (because there is no `OLD` anymore, luckily). * [actions skip] Adding "All GCC versions" to comment, to be clear about it. Co-authored-by: Henry Schreiner --- include/pybind11/attr.h | 23 +++++++++++++------- include/pybind11/cast.h | 5 +++-- include/pybind11/detail/common.h | 25 ++++++++++++++-------- include/pybind11/detail/type_caster_base.h | 9 ++++---- include/pybind11/pybind11.h | 1 - 5 files changed, 38 insertions(+), 25 deletions(-) diff --git a/include/pybind11/attr.h b/include/pybind11/attr.h index 20d119f0..7188fd7e 100644 --- a/include/pybind11/attr.h +++ b/include/pybind11/attr.h @@ -517,23 +517,30 @@ template struct process_attribute struct process_attributes { static void init(const Args&... args, function_record *r) { PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(r); - int unused[] = { 0, (process_attribute::type>::init(args, r), 0) ... }; - ignore_unused(unused); + PYBIND11_WORKAROUND_INCORRECT_GCC_UNUSED_BUT_SET_PARAMETER(r); + using expander = int[]; + (void) expander{ + 0, ((void) process_attribute::type>::init(args, r), 0)...}; } static void init(const Args&... args, type_record *r) { PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(r); - int unused[] = { 0, (process_attribute::type>::init(args, r), 0) ... }; - ignore_unused(unused); + PYBIND11_WORKAROUND_INCORRECT_GCC_UNUSED_BUT_SET_PARAMETER(r); + using expander = int[]; + (void) expander{0, + (process_attribute::type>::init(args, r), 0)...}; } static void precall(function_call &call) { PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(call); - int unused[] = { 0, (process_attribute::type>::precall(call), 0) ... }; - ignore_unused(unused); + using expander = int[]; + (void) expander{0, + (process_attribute::type>::precall(call), 0)...}; } static void postcall(function_call &call, handle fn_ret) { PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(call, fn_ret); - int unused[] = { 0, (process_attribute::type>::postcall(call, fn_ret), 0) ... }; - ignore_unused(unused); + PYBIND11_WORKAROUND_INCORRECT_GCC_UNUSED_BUT_SET_PARAMETER(fn_ret); + using expander = int[]; + (void) expander{ + 0, (process_attribute::type>::postcall(call, fn_ret), 0)...}; } }; diff --git a/include/pybind11/cast.h b/include/pybind11/cast.h index 988a2369..718dc2de 100644 --- a/include/pybind11/cast.h +++ b/include/pybind11/cast.h @@ -609,6 +609,7 @@ protected: template static handle cast_impl(T &&src, return_value_policy policy, handle parent, index_sequence) { PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(src, policy, parent); + PYBIND11_WORKAROUND_INCORRECT_GCC_UNUSED_BUT_SET_PARAMETER(policy, parent); std::array entries{{ reinterpret_steal(make_caster::cast(std::get(std::forward(src)), policy, parent))... }}; @@ -1235,8 +1236,8 @@ public: // Tuples aren't (easily) resizable so a list is needed for collection, // but the actual function call strictly requires a tuple. auto args_list = list(); - int _[] = { 0, (process(args_list, std::forward(values)), 0)... }; - ignore_unused(_); + using expander = int[]; + (void) expander{0, (process(args_list, std::forward(values)), 0)...}; m_args = std::move(args_list); } diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 27a79bfd..08430b5a 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -734,9 +734,6 @@ using function_signature_t = conditional_t< template using is_lambda = satisfies_none_of, std::is_function, std::is_pointer, std::is_member_pointer>; -/// Ignore that a variable is unused in compiler warnings -inline void ignore_unused(const int *) { } - // [workaround(intel)] Internal error on fold expression /// Apply a function over each element of a parameter pack #if defined(__cpp_fold_expressions) && !defined(__INTEL_COMPILER) @@ -927,19 +924,29 @@ inline static std::shared_ptr try_get_shared_from_this(std::enable_shared_fro #endif } -#if defined(_MSC_VER) && _MSC_VER <= 1916 - -// warning C4100: Unreferenced formal parameter +// For silencing "unused" compiler warnings in special situations. template -inline constexpr void workaround_incorrect_msvc_c4100(Args &&...) {} +#if defined(_MSC_VER) && _MSC_VER >= 1910 && _MSC_VER < 1920 // MSVC 2017 +constexpr +#endif +inline void silence_unused_warnings(Args &&...) {} +// MSVC warning C4100: Unreferenced formal parameter +#if defined(_MSC_VER) && _MSC_VER <= 1916 # define PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(...) \ - detail::workaround_incorrect_msvc_c4100(__VA_ARGS__) - + detail::silence_unused_warnings(__VA_ARGS__) #else # define PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(...) #endif +// GCC -Wunused-but-set-parameter All GCC versions (as of July 2021). +#if defined(__GNUG__) && !defined(__clang__) && !defined(__INTEL_COMPILER) +# define PYBIND11_WORKAROUND_INCORRECT_GCC_UNUSED_BUT_SET_PARAMETER(...) \ + detail::silence_unused_warnings(__VA_ARGS__) +#else +# define PYBIND11_WORKAROUND_INCORRECT_GCC_UNUSED_BUT_SET_PARAMETER(...) +#endif + #if defined(_MSC_VER) // All versions (as of July 2021). // warning C4127: Conditional expression is constant diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h index e2d1bcb8..a708d87f 100644 --- a/include/pybind11/detail/type_caster_base.h +++ b/include/pybind11/detail/type_caster_base.h @@ -927,18 +927,17 @@ protected: using Constructor = void *(*)(const void *); /* Only enabled when the types are {copy,move}-constructible *and* when the type - does not have a private operator new implementation. */ + does not have a private operator new implementation. A comma operator is used in the decltype + argument to apply SFINAE to the public copy/move constructors.*/ template ::value>> - static auto make_copy_constructor(const T *x) -> decltype(new T(*x), Constructor{}) { - PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(x); + static auto make_copy_constructor(const T *) -> decltype(new T(std::declval()), Constructor{}) { return [](const void *arg) -> void * { return new T(*reinterpret_cast(arg)); }; } template ::value>> - static auto make_move_constructor(const T *x) -> decltype(new T(std::move(*const_cast(x))), Constructor{}) { - PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(x); + static auto make_move_constructor(const T *) -> decltype(new T(std::declval()), Constructor{}) { return [](const void *arg) -> void * { return new T(std::move(*const_cast(reinterpret_cast(arg)))); }; diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 54213297..ba29f964 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -12,7 +12,6 @@ #if defined(__GNUG__) && !defined(__clang__) && !defined(__INTEL_COMPILER) # pragma GCC diagnostic push -# pragma GCC diagnostic ignored "-Wunused-but-set-parameter" # pragma GCC diagnostic ignored "-Wattributes" #endif From 6ac8efe52dfd8b15caec0e376d677cadd9bec38c Mon Sep 17 00:00:00 2001 From: Eric Cousineau Date: Fri, 6 Aug 2021 15:51:53 -0400 Subject: [PATCH 030/100] test_eval: Show example of working closure (#2743) * test_eval: Show example of working closure * Extend test_eval_closure with weirder examples of closures for py::eval Co-authored-by: Yannick Jadoul Co-authored-by: Aaron Gokaslan --- tests/test_eval.cpp | 18 ++++++++++++++++++ tests/test_eval.py | 17 +++++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/tests/test_eval.cpp b/tests/test_eval.cpp index 16cdf17f..29366f67 100644 --- a/tests/test_eval.cpp +++ b/tests/test_eval.cpp @@ -98,4 +98,22 @@ TEST_SUBMODULE(eval_, m) { auto int_class = py::eval("isinstance(42, int)", global); return global; }); + + // test_eval_closure + m.def("test_eval_closure", []() { + py::dict global; + global["closure_value"] = 42; + py::dict local; + local["closure_value"] = 0; + py::exec(R"( + local_value = closure_value + + def func_global(): + return closure_value + + def func_local(): + return local_value + )", global, local); + return std::make_pair(global, local); + }); } diff --git a/tests/test_eval.py b/tests/test_eval.py index 1bb05af0..601e526f 100644 --- a/tests/test_eval.py +++ b/tests/test_eval.py @@ -33,3 +33,20 @@ def test_eval_empty_globals(): g = {} assert "__builtins__" in m.eval_empty_globals(g) assert "__builtins__" in g + + +def test_eval_closure(): + global_, local = m.test_eval_closure() + + assert global_["closure_value"] == 42 + assert local["closure_value"] == 0 + + assert "local_value" not in global_ + assert local["local_value"] == 0 + + assert "func_global" not in global_ + assert local["func_global"]() == 42 + + assert "func_local" not in global_ + with pytest.raises(NameError): + local["func_local"]() From 61ee923bb19f38ae7a590ff0e6389d4c62e8360d Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Sat, 7 Aug 2021 11:51:28 -0700 Subject: [PATCH 031/100] Consistent step name "Python tests". (#3180) --- .github/workflows/ci.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 73a6fd51..7b028098 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -664,7 +664,7 @@ jobs: -DPYTHON_EXECUTABLE=$(python3 -c "import sys; print(sys.executable)") working-directory: /build-tests - - name: Run tests + - name: Python tests run: make pytest -j 2 working-directory: /build-tests @@ -760,7 +760,7 @@ jobs: - name: Build C++11 run: cmake --build build -j 2 - - name: Run tests + - name: Python tests run: cmake --build build -t pytest win32-msvc2015: From ff590c1258abf5b472f64b5ab337c5951922f81d Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Mon, 9 Aug 2021 12:48:27 -0400 Subject: [PATCH 032/100] maint(perf): Optimize Numpy constructor to remove copies by value. (#3183) * maint(perf): Optimize Numpy Constructor with additional std::move * Add more moves --- include/pybind11/numpy.h | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/include/pybind11/numpy.h b/include/pybind11/numpy.h index aa8294e1..f0f3b60e 100644 --- a/include/pybind11/numpy.h +++ b/include/pybind11/numpy.h @@ -478,11 +478,11 @@ public: dtype(list names, list formats, list offsets, ssize_t itemsize) { dict args; - args["names"] = names; - args["formats"] = formats; - args["offsets"] = offsets; + args["names"] = std::move(names); + args["formats"] = std::move(formats); + args["offsets"] = std::move(offsets); args["itemsize"] = pybind11::int_(itemsize); - m_ptr = from_args(args).release().ptr(); + m_ptr = from_args(std::move(args)).release().ptr(); } /// This is essentially the same as calling numpy.dtype(args) in Python. @@ -560,7 +560,7 @@ private: formats.append(descr.format); offsets.append(descr.offset); } - return dtype(names, formats, offsets, itemsize); + return dtype(std::move(names), std::move(formats), std::move(offsets), itemsize); } }; @@ -1134,7 +1134,10 @@ inline PYBIND11_NOINLINE void register_structured_dtype( formats.append(field.descr); offsets.append(pybind11::int_(field.offset)); } - auto dtype_ptr = pybind11::dtype(names, formats, offsets, itemsize).release().ptr(); + auto dtype_ptr + = pybind11::dtype(std::move(names), std::move(formats), std::move(offsets), itemsize) + .release() + .ptr(); // There is an existing bug in NumPy (as of v1.11): trailing bytes are // not encoded explicitly into the format string. This will supposedly From 4c7e509fa475d0a0ede726267bc451e4f0583446 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Mon, 9 Aug 2021 10:10:38 -0700 Subject: [PATCH 033/100] PYBIND11_NOINLINE-related cleanup. (#3179) * Removing pragma for GCC -Wattributes, fixing forward declarations. * Introducing PYBIND11_NOINLINE_FWD to deal with CUDA, GCC7, GCC8. * Updating PYBIND11_NOINLINE_DCL in Doxyfile. * Trying noinline, noinline for {CUDA, GCC7, GCC8} * Trying noinline, inline for {CUDA, GCC7, GCC8} * Adding GCC -Wattributes `pragma` in 3 header files. * Introducing PYBIND11_NOINLINE_GCC_PRAGMA_ATTRIBUTES_NEEDED, used in 9 header files. * Removing ICC pragma 2196, to see if it is still needed. * Trying noinline, noinline for ICC * Trying noinline, inline for ICC * Restoring ICC pragma 2196, introducing PYBIND11_NOINLINE_FORCED, defined for testing. * Removing code accidentally left in (was for experimentation only). * Removing one-time-test define. * Removing PYBIND11_NOINLINE_FWD macro (after learning that it makes no sense). * Testing with PYBIND11_NOINLINE_DISABLED. Minor non-functional enhancements. * Removing #define PYBIND11_NOINLINE_DISABLED (test was successful). * Removing PYBIND11_NOINLINE_FORCED and enhancing comments for PYBIND11_NOINLINE. * WIP stripping back * Making -Wattributes pragma in pybind11 specific to GCC7, GCC8, CUDA. --- include/pybind11/attr.h | 2 +- include/pybind11/detail/common.h | 15 +++++++++----- include/pybind11/detail/internals.h | 6 +++--- include/pybind11/detail/type_caster_base.h | 24 +++++++++++----------- include/pybind11/detail/typeid.h | 2 +- include/pybind11/numpy.h | 4 ++-- include/pybind11/pybind11.h | 10 ++++----- include/pybind11/pytypes.h | 4 ++-- 8 files changed, 36 insertions(+), 31 deletions(-) diff --git a/include/pybind11/attr.h b/include/pybind11/attr.h index 7188fd7e..ab1fe804 100644 --- a/include/pybind11/attr.h +++ b/include/pybind11/attr.h @@ -124,7 +124,7 @@ enum op_id : int; enum op_type : int; struct undefined_t; template struct op_; -inline void keep_alive_impl(size_t Nurse, size_t Patient, function_call &call, handle ret); +void keep_alive_impl(size_t Nurse, size_t Patient, function_call &call, handle ret); /// Internal data structure which holds metadata about a keyword argument struct argument_record { diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 08430b5a..cc24a52f 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -99,10 +99,15 @@ # endif #endif -#if defined(_MSC_VER) -# define PYBIND11_NOINLINE __declspec(noinline) +// The PYBIND11_NOINLINE macro is for function DEFINITIONS. +// In contrast, FORWARD DECLARATIONS should never use this macro: +// https://stackoverflow.com/questions/9317473/forward-declaration-of-inline-functions +#if defined(PYBIND11_NOINLINE_DISABLED) // Option for maximum portability and experimentation. +# define PYBIND11_NOINLINE inline +#elif defined(_MSC_VER) +# define PYBIND11_NOINLINE __declspec(noinline) inline #else -# define PYBIND11_NOINLINE __attribute__ ((noinline)) +# define PYBIND11_NOINLINE __attribute__ ((noinline)) inline #endif #if defined(__MINGW32__) @@ -778,8 +783,8 @@ PYBIND11_RUNTIME_EXCEPTION(import_error, PyExc_ImportError) PYBIND11_RUNTIME_EXCEPTION(cast_error, PyExc_RuntimeError) /// Thrown when pybind11::cast or handle::call fail due to a type casting error PYBIND11_RUNTIME_EXCEPTION(reference_cast_error, PyExc_RuntimeError) /// Used internally -[[noreturn]] PYBIND11_NOINLINE inline void pybind11_fail(const char *reason) { throw std::runtime_error(reason); } -[[noreturn]] PYBIND11_NOINLINE inline void pybind11_fail(const std::string &reason) { throw std::runtime_error(reason); } +[[noreturn]] PYBIND11_NOINLINE void pybind11_fail(const char *reason) { throw std::runtime_error(reason); } +[[noreturn]] PYBIND11_NOINLINE void pybind11_fail(const std::string &reason) { throw std::runtime_error(reason); } template struct format_descriptor { }; diff --git a/include/pybind11/detail/internals.h b/include/pybind11/detail/internals.h index a93a83e2..b177801a 100644 --- a/include/pybind11/detail/internals.h +++ b/include/pybind11/detail/internals.h @@ -257,7 +257,7 @@ inline void translate_local_exception(std::exception_ptr p) { #endif /// Return a reference to the current `internals` data -PYBIND11_NOINLINE inline internals &get_internals() { +PYBIND11_NOINLINE internals &get_internals() { auto **&internals_pp = get_internals_pp(); if (internals_pp && *internals_pp) return **internals_pp; @@ -352,14 +352,14 @@ PYBIND11_NAMESPACE_END(detail) /// Returns a named pointer that is shared among all extension modules (using the same /// pybind11 version) running in the current interpreter. Names starting with underscores /// are reserved for internal usage. Returns `nullptr` if no matching entry was found. -inline PYBIND11_NOINLINE void *get_shared_data(const std::string &name) { +PYBIND11_NOINLINE void *get_shared_data(const std::string &name) { auto &internals = detail::get_internals(); auto it = internals.shared_data.find(name); return it != internals.shared_data.end() ? it->second : nullptr; } /// Set the shared data that can be later recovered by `get_shared_data()`. -inline PYBIND11_NOINLINE void *set_shared_data(const std::string &name, void *data) { +PYBIND11_NOINLINE void *set_shared_data(const std::string &name, void *data) { detail::get_internals().shared_data[name] = data; return data; } diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h index a708d87f..14cb27ca 100644 --- a/include/pybind11/detail/type_caster_base.h +++ b/include/pybind11/detail/type_caster_base.h @@ -81,7 +81,7 @@ public: inline std::pair all_type_info_get_cache(PyTypeObject *type); // Populates a just-created cache entry. -PYBIND11_NOINLINE inline void all_type_info_populate(PyTypeObject *t, std::vector &bases) { +PYBIND11_NOINLINE void all_type_info_populate(PyTypeObject *t, std::vector &bases) { std::vector check; for (handle parent : reinterpret_borrow(t->tp_bases)) check.push_back((PyTypeObject *) parent.ptr()); @@ -150,7 +150,7 @@ inline const std::vector &all_type_info(PyTypeObject *type) * ancestors are pybind11-registered. Throws an exception if there are multiple bases--use * `all_type_info` instead if you want to support multiple bases. */ -PYBIND11_NOINLINE inline detail::type_info* get_type_info(PyTypeObject *type) { +PYBIND11_NOINLINE detail::type_info* get_type_info(PyTypeObject *type) { auto &bases = all_type_info(type); if (bases.empty()) return nullptr; @@ -176,7 +176,7 @@ inline detail::type_info *get_global_type_info(const std::type_index &tp) { } /// Return the type info for a given C++ type; on lookup failure can either throw or return nullptr. -PYBIND11_NOINLINE inline detail::type_info *get_type_info(const std::type_index &tp, +PYBIND11_NOINLINE detail::type_info *get_type_info(const std::type_index &tp, bool throw_if_missing = false) { if (auto ltype = get_local_type_info(tp)) return ltype; @@ -191,13 +191,13 @@ PYBIND11_NOINLINE inline detail::type_info *get_type_info(const std::type_index return nullptr; } -PYBIND11_NOINLINE inline handle get_type_handle(const std::type_info &tp, bool throw_if_missing) { +PYBIND11_NOINLINE handle get_type_handle(const std::type_info &tp, bool throw_if_missing) { detail::type_info *type_info = get_type_info(tp, throw_if_missing); return handle(type_info ? ((PyObject *) type_info->type) : nullptr); } // Searches the inheritance graph for a registered Python instance, using all_type_info(). -PYBIND11_NOINLINE inline handle find_registered_python_instance(void *src, +PYBIND11_NOINLINE handle find_registered_python_instance(void *src, const detail::type_info *tinfo) { auto it_instances = get_internals().registered_instances.equal_range(src); for (auto it_i = it_instances.first; it_i != it_instances.second; ++it_i) { @@ -325,7 +325,7 @@ public: * The returned object should be short-lived: in particular, it must not outlive the called-upon * instance. */ -PYBIND11_NOINLINE inline value_and_holder instance::get_value_and_holder(const type_info *find_type /*= nullptr default in common.h*/, bool throw_if_missing /*= true in common.h*/) { +PYBIND11_NOINLINE value_and_holder instance::get_value_and_holder(const type_info *find_type /*= nullptr default in common.h*/, bool throw_if_missing /*= true in common.h*/) { // Optimize common case: if (!find_type || Py_TYPE(this) == find_type->type) return value_and_holder(this, find_type, 0, 0); @@ -349,7 +349,7 @@ PYBIND11_NOINLINE inline value_and_holder instance::get_value_and_holder(const t #endif } -PYBIND11_NOINLINE inline void instance::allocate_layout() { +PYBIND11_NOINLINE void instance::allocate_layout() { auto &tinfo = all_type_info(Py_TYPE(this)); const size_t n_types = tinfo.size(); @@ -397,19 +397,19 @@ PYBIND11_NOINLINE inline void instance::allocate_layout() { owned = true; } -PYBIND11_NOINLINE inline void instance::deallocate_layout() const { +PYBIND11_NOINLINE void instance::deallocate_layout() const { if (!simple_layout) PyMem_Free(nonsimple.values_and_holders); } -PYBIND11_NOINLINE inline bool isinstance_generic(handle obj, const std::type_info &tp) { +PYBIND11_NOINLINE bool isinstance_generic(handle obj, const std::type_info &tp) { handle type = detail::get_type_handle(tp, false); if (!type) return false; return isinstance(obj, type); } -PYBIND11_NOINLINE inline std::string error_string() { +PYBIND11_NOINLINE std::string error_string() { if (!PyErr_Occurred()) { PyErr_SetString(PyExc_RuntimeError, "Unknown internal error occurred"); return "Unknown internal error occurred"; @@ -456,7 +456,7 @@ PYBIND11_NOINLINE inline std::string error_string() { return errorString; } -PYBIND11_NOINLINE inline handle get_object_handle(const void *ptr, const detail::type_info *type ) { +PYBIND11_NOINLINE handle get_object_handle(const void *ptr, const detail::type_info *type ) { auto &instances = get_internals().registered_instances; auto range = instances.equal_range(ptr); for (auto it = range.first; it != range.second; ++it) { @@ -483,7 +483,7 @@ inline PyThreadState *get_thread_state_unchecked() { } // Forward declarations -inline void keep_alive_impl(handle nurse, handle patient); +void keep_alive_impl(handle nurse, handle patient); inline PyObject *make_new_instance(PyTypeObject *type); class type_caster_generic { diff --git a/include/pybind11/detail/typeid.h b/include/pybind11/detail/typeid.h index 148889ff..39ba8ce0 100644 --- a/include/pybind11/detail/typeid.h +++ b/include/pybind11/detail/typeid.h @@ -29,7 +29,7 @@ inline void erase_all(std::string &string, const std::string &search) { } } -PYBIND11_NOINLINE inline void clean_type_id(std::string &name) { +PYBIND11_NOINLINE void clean_type_id(std::string &name) { #if defined(__GNUG__) int status = 0; std::unique_ptr res { diff --git a/include/pybind11/numpy.h b/include/pybind11/numpy.h index f0f3b60e..2dc0f29f 100644 --- a/include/pybind11/numpy.h +++ b/include/pybind11/numpy.h @@ -104,7 +104,7 @@ struct numpy_internals { } }; -inline PYBIND11_NOINLINE void load_numpy_internals(numpy_internals* &ptr) { +PYBIND11_NOINLINE void load_numpy_internals(numpy_internals* &ptr) { ptr = &get_or_create_shared_data("_numpy_internals"); } @@ -1110,7 +1110,7 @@ struct field_descriptor { dtype descr; }; -inline PYBIND11_NOINLINE void register_structured_dtype( +PYBIND11_NOINLINE void register_structured_dtype( any_container fields, const std::type_info& tinfo, ssize_t itemsize, bool (*direct_converter)(PyObject *, void *&)) { diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index ba29f964..20df6a01 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -10,7 +10,7 @@ #pragma once -#if defined(__GNUG__) && !defined(__clang__) && !defined(__INTEL_COMPILER) +#if defined(__CUDACC__) || (defined(__GNUC__) && (__GNUC__ == 7 || __GNUC__ == 8)) # pragma GCC diagnostic push # pragma GCC diagnostic ignored "-Wattributes" #endif @@ -1875,7 +1875,7 @@ private: PYBIND11_NAMESPACE_BEGIN(detail) -inline void keep_alive_impl(handle nurse, handle patient) { +PYBIND11_NOINLINE void keep_alive_impl(handle nurse, handle patient) { if (!nurse || !patient) pybind11_fail("Could not activate keep_alive!"); @@ -1902,7 +1902,7 @@ inline void keep_alive_impl(handle nurse, handle patient) { } } -PYBIND11_NOINLINE inline void keep_alive_impl(size_t Nurse, size_t Patient, function_call &call, handle ret) { +PYBIND11_NOINLINE void keep_alive_impl(size_t Nurse, size_t Patient, function_call &call, handle ret) { auto get_arg = [&](size_t n) { if (n == 0) return ret; @@ -2152,7 +2152,7 @@ exception ®ister_local_exception(handle scope, } PYBIND11_NAMESPACE_BEGIN(detail) -PYBIND11_NOINLINE inline void print(const tuple &args, const dict &kwargs) { +PYBIND11_NOINLINE void print(const tuple &args, const dict &kwargs) { auto strings = tuple(args.size()); for (size_t i = 0; i < args.size(); ++i) { strings[i] = str(args[i]); @@ -2384,6 +2384,6 @@ PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) # pragma GCC diagnostic pop // -Wnoexcept-type #endif -#if defined(__GNUG__) && !defined(__clang__) && !defined(__INTEL_COMPILER) +#if defined(__CUDACC__) || (defined(__GNUC__) && (__GNUC__ == 7 || __GNUC__ == 8)) # pragma GCC diagnostic pop #endif diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index c7b2501f..21e14960 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -24,7 +24,7 @@ struct arg; struct arg_v; PYBIND11_NAMESPACE_BEGIN(detail) class args_proxy; -inline bool isinstance_generic(handle obj, const std::type_info &tp); +bool isinstance_generic(handle obj, const std::type_info &tp); // Accessor forward declarations template class accessor; @@ -316,7 +316,7 @@ template T reinterpret_borrow(handle h) { return {h, object::borrow template T reinterpret_steal(handle h) { return {h, object::stolen_t{}}; } PYBIND11_NAMESPACE_BEGIN(detail) -inline std::string error_string(); +std::string error_string(); PYBIND11_NAMESPACE_END(detail) #if defined(_MSC_VER) From 0be2ea065eeaf2983964ecb792d30abbcedcdc9d Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 9 Aug 2021 15:08:19 -0400 Subject: [PATCH 034/100] [pre-commit.ci] pre-commit autoupdate (#3185) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit updates: - [github.com/asottile/pyupgrade: v2.23.1 → v2.23.3](https://github.com/asottile/pyupgrade/compare/v2.23.1...v2.23.3) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .pre-commit-config.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 57beb777..ee61258d 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -31,7 +31,7 @@ repos: exclude: ^noxfile.py$ - repo: https://github.com/asottile/pyupgrade - rev: v2.23.1 + rev: v2.23.3 hooks: - id: pyupgrade From 7d3b05715bf1c8f9f7232723671bf9b780a173ac Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Thu, 12 Aug 2021 13:21:49 -0700 Subject: [PATCH 035/100] Improved workaround for Centos 8 failure (follow-on to PR #3030). (#3193) * Rollback of PR #3030 (Working around Centos 8 failure). * Adding `-DCMAKE_BUILD_TYPE=Release` * Improving existing workaround (undoing the originally intended rollback). * Fixing minor documentation bug. --- .github/workflows/ci.yml | 6 +++--- tools/pybind11Config.cmake.in | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 7b028098..e1c32b96 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -594,13 +594,13 @@ jobs: - name: VAR_BUILD_TYPE 7 if: matrix.centos == 7 - run: echo Release > VAR_BUILD_TYPE + run: echo MinSizeRel > VAR_BUILD_TYPE - # Using Debug to avoid segfault that appeared around 2021-06-04, + # Using Release to avoid segfault that appeared around 2021-06-04, # apparently when the gcc version changed from 8.3 to 8.4. - name: VAR_BUILD_TYPE 8 if: matrix.centos == 8 - run: echo Debug > VAR_BUILD_TYPE + run: echo Release > VAR_BUILD_TYPE - name: Configure shell: bash diff --git a/tools/pybind11Config.cmake.in b/tools/pybind11Config.cmake.in index 73ec104a..8f8701be 100644 --- a/tools/pybind11Config.cmake.in +++ b/tools/pybind11Config.cmake.in @@ -147,7 +147,7 @@ Add a module and setup all helpers. You can select the type of the library; the default is ``MODULE``. There are several options: ``OPT_SIZE`` - Optimize for size, even if the ``CMAKE_BUILD_TYPE`` is not ``RelSize``. + Optimize for size, even if the ``CMAKE_BUILD_TYPE`` is not ``MinSizeRel``. ``THIN_LTO`` Use thin TLO instead of regular if there's a choice (pybind11's selection is disabled if ``CMAKE_INTERPROCEDURAL_OPTIMIZATIONS`` is set). From d6841f60cc6a07105222d3650caa14ef2007f949 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 13 Aug 2021 12:27:54 -0400 Subject: [PATCH 036/100] chore(deps): bump jwlawson/actions-setup-cmake from 1.9 to 1.10 (#3196) Bumps [jwlawson/actions-setup-cmake](https://github.com/jwlawson/actions-setup-cmake) from 1.9 to 1.10. - [Release notes](https://github.com/jwlawson/actions-setup-cmake/releases) - [Commits](https://github.com/jwlawson/actions-setup-cmake/compare/v1.9...v1.10) --- updated-dependencies: - dependency-name: jwlawson/actions-setup-cmake dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/ci.yml | 12 ++++++------ .github/workflows/configure.yml | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e1c32b96..be3fa723 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -75,7 +75,7 @@ jobs: run: brew install boost - name: Update CMake - uses: jwlawson/actions-setup-cmake@v1.9 + uses: jwlawson/actions-setup-cmake@v1.10 - name: Cache wheels if: runner.os == 'macOS' @@ -193,7 +193,7 @@ jobs: debug: ${{ matrix.python-debug }} - name: Update CMake - uses: jwlawson/actions-setup-cmake@v1.9 + uses: jwlawson/actions-setup-cmake@v1.10 - name: Valgrind cache if: matrix.valgrind @@ -445,7 +445,7 @@ jobs: run: python3 -m pip install --upgrade pip - name: Update CMake - uses: jwlawson/actions-setup-cmake@v1.9 + uses: jwlawson/actions-setup-cmake@v1.10 - name: Configure shell: bash @@ -738,7 +738,7 @@ jobs: architecture: x86 - name: Update CMake - uses: jwlawson/actions-setup-cmake@v1.9 + uses: jwlawson/actions-setup-cmake@v1.10 - name: Prepare MSVC uses: ilammy/msvc-dev-cmd@v1.9.0 @@ -784,7 +784,7 @@ jobs: python-version: ${{ matrix.python }} - name: Update CMake - uses: jwlawson/actions-setup-cmake@v1.9 + uses: jwlawson/actions-setup-cmake@v1.10 - name: Prepare MSVC uses: ilammy/msvc-dev-cmd@v1.9.0 @@ -838,7 +838,7 @@ jobs: python-version: ${{ matrix.python }} - name: Update CMake - uses: jwlawson/actions-setup-cmake@v1.9 + uses: jwlawson/actions-setup-cmake@v1.10 - name: Prepare env run: python -m pip install -r tests/requirements.txt --prefer-binary diff --git a/.github/workflows/configure.yml b/.github/workflows/configure.yml index 924088fc..d37fa3c2 100644 --- a/.github/workflows/configure.yml +++ b/.github/workflows/configure.yml @@ -55,7 +55,7 @@ jobs: # An action for adding a specific version of CMake: # https://github.com/jwlawson/actions-setup-cmake - name: Setup CMake ${{ matrix.cmake }} - uses: jwlawson/actions-setup-cmake@v1.9 + uses: jwlawson/actions-setup-cmake@v1.10 with: cmake-version: ${{ matrix.cmake }} From 9df2f1ff13ee37c312b5f04fcb77f4339f92b015 Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Fri, 13 Aug 2021 12:37:05 -0400 Subject: [PATCH 037/100] maint(precommit): Apply isort (#3195) * Apply isort * Tweak isort config * Add env.py as a known_first_party * Add one missing known first party * Make config compat with older isort versions * Add another comment * Revert pyproject setting --- .pre-commit-config.yaml | 5 +++++ docs/benchmark.py | 6 +++--- docs/conf.py | 4 ++-- noxfile.py | 1 - pybind11/__init__.py | 5 ++--- pybind11/__main__.py | 2 +- pybind11/_version.pyi | 2 +- pybind11/commands.py | 1 - pybind11/setup_helpers.py | 8 ++++---- pybind11/setup_helpers.pyi | 9 ++++----- pyproject.toml | 6 ++++++ setup.py | 2 +- tests/extra_setuptools/test_setuphelper.py | 2 +- tests/test_buffers.py | 5 ++--- tests/test_builtin_casters.py | 3 +-- tests/test_call_policies.py | 3 +-- tests/test_callbacks.py | 8 +++++--- tests/test_chrono.py | 3 ++- tests/test_class.py | 3 +-- tests/test_cmake_build/test.py | 1 + tests/test_copy_move.py | 1 + tests/test_custom_type_casters.py | 1 + tests/test_eigen.py | 1 + tests/test_enum.py | 1 + tests/test_eval.py | 1 - tests/test_exceptions.py | 3 +-- tests/test_factory_constructors.py | 6 +++--- tests/test_iostream.py | 4 ++-- tests/test_kwargs_and_defaults.py | 1 - tests/test_local_bindings.py | 1 - tests/test_methods_and_attributes.py | 3 +-- tests/test_modules.py | 8 +++++--- tests/test_multiple_inheritance.py | 1 - tests/test_numpy_array.py | 1 - tests/test_numpy_dtypes.py | 1 - tests/test_numpy_vectorize.py | 1 + tests/test_opaque_types.py | 3 ++- tests/test_operator_overloading.py | 3 ++- tests/test_pickling.py | 1 - tests/test_pytypes.py | 7 ++++--- tests/test_sequences_and_iterators.py | 3 ++- tests/test_stl.py | 3 +-- tests/test_stl_binders.py | 1 - tools/libsize.py | 3 ++- tools/make_changelog.py | 2 -- 45 files changed, 73 insertions(+), 67 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index ee61258d..a5900df1 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -35,6 +35,11 @@ repos: hooks: - id: pyupgrade +- repo: https://github.com/PyCQA/isort + rev: 5.9.3 + hooks: + - id: isort + # Black, the code formatter, natively supports pre-commit - repo: https://github.com/psf/black rev: 21.7b0 diff --git a/docs/benchmark.py b/docs/benchmark.py index 33d78fb4..369470c8 100644 --- a/docs/benchmark.py +++ b/docs/benchmark.py @@ -1,8 +1,8 @@ # -*- coding: utf-8 -*- -import random -import os -import time import datetime as dt +import os +import random +import time nfns = 4 # Functions per class nargs = 4 # Arguments per function diff --git a/docs/conf.py b/docs/conf.py index 6ac054c6..458a8688 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -13,12 +13,12 @@ # All configuration values have a default; values that are commented out # serve to show the default. -import sys import os +import re import shlex import subprocess +import sys from pathlib import Path -import re DIR = Path(__file__).parent.resolve() diff --git a/noxfile.py b/noxfile.py index f0f66801..55b1d180 100644 --- a/noxfile.py +++ b/noxfile.py @@ -1,6 +1,5 @@ import nox - nox.options.sessions = ["lint", "tests", "tests_packaging"] diff --git a/pybind11/__init__.py b/pybind11/__init__.py index ad654208..64e999ba 100644 --- a/pybind11/__init__.py +++ b/pybind11/__init__.py @@ -1,8 +1,7 @@ # -*- coding: utf-8 -*- -from ._version import version_info, __version__ -from .commands import get_include, get_cmake_dir - +from ._version import __version__, version_info +from .commands import get_cmake_dir, get_include __all__ = ( "version_info", diff --git a/pybind11/__main__.py b/pybind11/__main__.py index 020988c6..3235747b 100644 --- a/pybind11/__main__.py +++ b/pybind11/__main__.py @@ -5,7 +5,7 @@ import argparse import sys import sysconfig -from .commands import get_include, get_cmake_dir +from .commands import get_cmake_dir, get_include def print_includes(): diff --git a/pybind11/_version.pyi b/pybind11/_version.pyi index 970184c7..d45e5dc9 100644 --- a/pybind11/_version.pyi +++ b/pybind11/_version.pyi @@ -1,4 +1,4 @@ -from typing import Union, Tuple +from typing import Tuple, Union def _to_int(s: str) -> Union[int, str]: ... diff --git a/pybind11/commands.py b/pybind11/commands.py index 34dbaf8a..11f81d2d 100644 --- a/pybind11/commands.py +++ b/pybind11/commands.py @@ -1,7 +1,6 @@ # -*- coding: utf-8 -*- import os - DIR = os.path.abspath(os.path.dirname(__file__)) diff --git a/pybind11/setup_helpers.py b/pybind11/setup_helpers.py index 0888ab48..4ff1a0cb 100644 --- a/pybind11/setup_helpers.py +++ b/pybind11/setup_helpers.py @@ -41,23 +41,23 @@ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. import contextlib import os +import platform import shutil import sys +import sysconfig import tempfile import threading -import platform import warnings -import sysconfig try: - from setuptools.command.build_ext import build_ext as _build_ext from setuptools import Extension as _Extension + from setuptools.command.build_ext import build_ext as _build_ext except ImportError: from distutils.command.build_ext import build_ext as _build_ext from distutils.extension import Extension as _Extension -import distutils.errors import distutils.ccompiler +import distutils.errors WIN = sys.platform.startswith("win32") and "mingw" not in sysconfig.get_platform() PY2 = sys.version_info[0] < 3 diff --git a/pybind11/setup_helpers.pyi b/pybind11/setup_helpers.pyi index 8b16c0a1..074744eb 100644 --- a/pybind11/setup_helpers.pyi +++ b/pybind11/setup_helpers.pyi @@ -1,13 +1,12 @@ # IMPORTANT: Should stay in sync with setup_helpers.py (mostly checked by CI / # pre-commit). -from typing import Any, Callable, Dict, Iterator, List, Optional, Type, TypeVar, Union -from types import TracebackType - +import contextlib +import distutils.ccompiler from distutils.command.build_ext import build_ext as _build_ext # type: ignore from distutils.extension import Extension as _Extension -import distutils.ccompiler -import contextlib +from types import TracebackType +from typing import Any, Callable, Dict, Iterator, List, Optional, Type, TypeVar, Union WIN: bool PY2: bool diff --git a/pyproject.toml b/pyproject.toml index f86470ee..7d7a1c82 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -15,6 +15,12 @@ ignore = [ "noxfile.py", ] +[tool.isort] +# Needs the compiled .so modules and env.py from tests +known_first_party = "env,pybind11_cross_module_tests,pybind11_tests," +# For black compatibility +profile = "black" + [tool.mypy] files = "pybind11" python_version = "2.7" diff --git a/setup.py b/setup.py index 7aa151c0..a2326287 100644 --- a/setup.py +++ b/setup.py @@ -4,6 +4,7 @@ # Setup script for PyPI; use CMakeFile.txt to build extension modules import contextlib +import io import os import re import shutil @@ -11,7 +12,6 @@ import string import subprocess import sys import tempfile -import io import setuptools.command.sdist diff --git a/tests/extra_setuptools/test_setuphelper.py b/tests/extra_setuptools/test_setuphelper.py index fb2d3420..c24f50af 100644 --- a/tests/extra_setuptools/test_setuphelper.py +++ b/tests/extra_setuptools/test_setuphelper.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- import os -import sys import subprocess +import sys from textwrap import dedent import pytest diff --git a/tests/test_buffers.py b/tests/test_buffers.py index 50845758..e3df7e04 100644 --- a/tests/test_buffers.py +++ b/tests/test_buffers.py @@ -1,14 +1,13 @@ # -*- coding: utf-8 -*- +import ctypes import io import struct -import ctypes import pytest import env # noqa: F401 - -from pybind11_tests import buffers as m from pybind11_tests import ConstructorStats +from pybind11_tests import buffers as m np = pytest.importorskip("numpy") diff --git a/tests/test_builtin_casters.py b/tests/test_builtin_casters.py index 1b9dbb2b..09b4b08c 100644 --- a/tests/test_builtin_casters.py +++ b/tests/test_builtin_casters.py @@ -2,9 +2,8 @@ import pytest import env # noqa: F401 - +from pybind11_tests import IncType, UserType from pybind11_tests import builtin_casters as m -from pybind11_tests import UserType, IncType def test_simple_string(): diff --git a/tests/test_call_policies.py b/tests/test_call_policies.py index af017e9d..3599cf81 100644 --- a/tests/test_call_policies.py +++ b/tests/test_call_policies.py @@ -2,9 +2,8 @@ import pytest import env # noqa: F401 - -from pybind11_tests import call_policies as m from pybind11_tests import ConstructorStats +from pybind11_tests import call_policies as m @pytest.mark.xfail("env.PYPY", reason="sometimes comes out 1 off on PyPy", strict=False) diff --git a/tests/test_callbacks.py b/tests/test_callbacks.py index 5bc4d177..9dc272a2 100644 --- a/tests/test_callbacks.py +++ b/tests/test_callbacks.py @@ -1,9 +1,11 @@ # -*- coding: utf-8 -*- -import pytest -from pybind11_tests import callbacks as m -from threading import Thread import time +from threading import Thread + +import pytest + import env # NOQA: F401 +from pybind11_tests import callbacks as m def test_callbacks(): diff --git a/tests/test_chrono.py b/tests/test_chrono.py index 12312c72..fdd73d69 100644 --- a/tests/test_chrono.py +++ b/tests/test_chrono.py @@ -1,9 +1,10 @@ # -*- coding: utf-8 -*- -from pybind11_tests import chrono as m import datetime + import pytest import env # noqa: F401 +from pybind11_tests import chrono as m def test_chrono_system_clock(): diff --git a/tests/test_class.py b/tests/test_class.py index 1ff60fe2..85d45319 100644 --- a/tests/test_class.py +++ b/tests/test_class.py @@ -2,9 +2,8 @@ import pytest import env # noqa: F401 - +from pybind11_tests import ConstructorStats, UserType from pybind11_tests import class_ as m -from pybind11_tests import UserType, ConstructorStats def test_repr(): diff --git a/tests/test_cmake_build/test.py b/tests/test_cmake_build/test.py index 87ed5135..d1a290dc 100644 --- a/tests/test_cmake_build/test.py +++ b/tests/test_cmake_build/test.py @@ -1,5 +1,6 @@ # -*- coding: utf-8 -*- import sys + import test_cmake_build assert test_cmake_build.add(1, 2) == 3 diff --git a/tests/test_copy_move.py b/tests/test_copy_move.py index 1d989522..eb1efddd 100644 --- a/tests/test_copy_move.py +++ b/tests/test_copy_move.py @@ -1,5 +1,6 @@ # -*- coding: utf-8 -*- import pytest + from pybind11_tests import copy_move_policies as m diff --git a/tests/test_custom_type_casters.py b/tests/test_custom_type_casters.py index bb74d54e..a10646ff 100644 --- a/tests/test_custom_type_casters.py +++ b/tests/test_custom_type_casters.py @@ -1,5 +1,6 @@ # -*- coding: utf-8 -*- import pytest + from pybind11_tests import custom_type_casters as m diff --git a/tests/test_eigen.py b/tests/test_eigen.py index a131dc15..e53826cb 100644 --- a/tests/test_eigen.py +++ b/tests/test_eigen.py @@ -1,5 +1,6 @@ # -*- coding: utf-8 -*- import pytest + from pybind11_tests import ConstructorStats np = pytest.importorskip("numpy") diff --git a/tests/test_enum.py b/tests/test_enum.py index e9732fa7..62f9426e 100644 --- a/tests/test_enum.py +++ b/tests/test_enum.py @@ -1,5 +1,6 @@ # -*- coding: utf-8 -*- import pytest + from pybind11_tests import enums as m diff --git a/tests/test_eval.py b/tests/test_eval.py index 601e526f..1bbd991b 100644 --- a/tests/test_eval.py +++ b/tests/test_eval.py @@ -4,7 +4,6 @@ import os import pytest import env # noqa: F401 - from pybind11_tests import eval_ as m diff --git a/tests/test_exceptions.py b/tests/test_exceptions.py index 966ae07f..d1edc39f 100644 --- a/tests/test_exceptions.py +++ b/tests/test_exceptions.py @@ -4,9 +4,8 @@ import sys import pytest import env # noqa: F401 - -from pybind11_tests import exceptions as m import pybind11_cross_module_tests as cm +from pybind11_tests import exceptions as m def test_std_exception(msg): diff --git a/tests/test_factory_constructors.py b/tests/test_factory_constructors.py index 8c6ca173..8bc02698 100644 --- a/tests/test_factory_constructors.py +++ b/tests/test_factory_constructors.py @@ -1,12 +1,12 @@ # -*- coding: utf-8 -*- -import pytest import re -import env # noqa: F401 +import pytest +import env # noqa: F401 +from pybind11_tests import ConstructorStats from pybind11_tests import factory_constructors as m from pybind11_tests.factory_constructors import tag -from pybind11_tests import ConstructorStats def test_init_factory_basic(): diff --git a/tests/test_iostream.py b/tests/test_iostream.py index e2b74d01..7f18ca65 100644 --- a/tests/test_iostream.py +++ b/tests/test_iostream.py @@ -1,9 +1,9 @@ # -*- coding: utf-8 -*- -from pybind11_tests import iostream as m import sys - from contextlib import contextmanager +from pybind11_tests import iostream as m + try: # Python 3 from io import StringIO diff --git a/tests/test_kwargs_and_defaults.py b/tests/test_kwargs_and_defaults.py index 12fe705b..ddc387ee 100644 --- a/tests/test_kwargs_and_defaults.py +++ b/tests/test_kwargs_and_defaults.py @@ -2,7 +2,6 @@ import pytest import env # noqa: F401 - from pybind11_tests import kwargs_and_defaults as m diff --git a/tests/test_local_bindings.py b/tests/test_local_bindings.py index a38564b6..52b1b633 100644 --- a/tests/test_local_bindings.py +++ b/tests/test_local_bindings.py @@ -2,7 +2,6 @@ import pytest import env # noqa: F401 - from pybind11_tests import local_bindings as m diff --git a/tests/test_methods_and_attributes.py b/tests/test_methods_and_attributes.py index 750ec02f..21909654 100644 --- a/tests/test_methods_and_attributes.py +++ b/tests/test_methods_and_attributes.py @@ -2,9 +2,8 @@ import pytest import env # noqa: F401 - -from pybind11_tests import methods_and_attributes as m from pybind11_tests import ConstructorStats +from pybind11_tests import methods_and_attributes as m def test_methods_and_attributes(): diff --git a/tests/test_modules.py b/tests/test_modules.py index 3390031a..49e1ea5e 100644 --- a/tests/test_modules.py +++ b/tests/test_modules.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- +from pybind11_tests import ConstructorStats from pybind11_tests import modules as m from pybind11_tests.modules import subsubmodule as ms -from pybind11_tests import ConstructorStats def test_nested_modules(): @@ -54,18 +54,20 @@ def test_reference_internal(): def test_importing(): - from pybind11_tests.modules import OD from collections import OrderedDict + from pybind11_tests.modules import OD + assert OD is OrderedDict assert str(OD([(1, "a"), (2, "b")])) == "OrderedDict([(1, 'a'), (2, 'b')])" def test_pydoc(): """Pydoc needs to be able to provide help() for everything inside a pybind11 module""" - import pybind11_tests import pydoc + import pybind11_tests + assert pybind11_tests.__name__ == "pybind11_tests" assert pybind11_tests.__doc__ == "pybind11 test module" assert pydoc.text.docmodule(pybind11_tests) diff --git a/tests/test_multiple_inheritance.py b/tests/test_multiple_inheritance.py index e6a7f976..a02c3130 100644 --- a/tests/test_multiple_inheritance.py +++ b/tests/test_multiple_inheritance.py @@ -2,7 +2,6 @@ import pytest import env # noqa: F401 - from pybind11_tests import ConstructorStats from pybind11_tests import multiple_inheritance as m diff --git a/tests/test_numpy_array.py b/tests/test_numpy_array.py index 548c84ba..69ba9d49 100644 --- a/tests/test_numpy_array.py +++ b/tests/test_numpy_array.py @@ -2,7 +2,6 @@ import pytest import env # noqa: F401 - from pybind11_tests import numpy_array as m np = pytest.importorskip("numpy") diff --git a/tests/test_numpy_dtypes.py b/tests/test_numpy_dtypes.py index 0a5881e4..6ea064d5 100644 --- a/tests/test_numpy_dtypes.py +++ b/tests/test_numpy_dtypes.py @@ -4,7 +4,6 @@ import re import pytest import env # noqa: F401 - from pybind11_tests import numpy_dtypes as m np = pytest.importorskip("numpy") diff --git a/tests/test_numpy_vectorize.py b/tests/test_numpy_vectorize.py index 4e6b2d19..de5c9a60 100644 --- a/tests/test_numpy_vectorize.py +++ b/tests/test_numpy_vectorize.py @@ -1,5 +1,6 @@ # -*- coding: utf-8 -*- import pytest + from pybind11_tests import numpy_vectorize as m np = pytest.importorskip("numpy") diff --git a/tests/test_opaque_types.py b/tests/test_opaque_types.py index 77379463..5495cb6b 100644 --- a/tests/test_opaque_types.py +++ b/tests/test_opaque_types.py @@ -1,7 +1,8 @@ # -*- coding: utf-8 -*- import pytest -from pybind11_tests import opaque_types as m + from pybind11_tests import ConstructorStats, UserType +from pybind11_tests import opaque_types as m def test_string_list(): diff --git a/tests/test_operator_overloading.py b/tests/test_operator_overloading.py index 5dbfb32c..b7137d15 100644 --- a/tests/test_operator_overloading.py +++ b/tests/test_operator_overloading.py @@ -1,7 +1,8 @@ # -*- coding: utf-8 -*- import pytest -from pybind11_tests import operators as m + from pybind11_tests import ConstructorStats +from pybind11_tests import operators as m def test_operator_overloading(): diff --git a/tests/test_pickling.py b/tests/test_pickling.py index 30321306..e39463d2 100644 --- a/tests/test_pickling.py +++ b/tests/test_pickling.py @@ -2,7 +2,6 @@ import pytest import env # noqa: F401 - from pybind11_tests import pickling as m try: diff --git a/tests/test_pytypes.py b/tests/test_pytypes.py index 66d6d30a..8a11b187 100644 --- a/tests/test_pytypes.py +++ b/tests/test_pytypes.py @@ -1,12 +1,13 @@ # -*- coding: utf-8 -*- from __future__ import division -import pytest + import sys -import env # noqa: F401 +import pytest -from pybind11_tests import pytypes as m +import env # noqa: F401 from pybind11_tests import debug_enabled +from pybind11_tests import pytypes as m def test_int(doc): diff --git a/tests/test_sequences_and_iterators.py b/tests/test_sequences_and_iterators.py index 70fb697f..a868c542 100644 --- a/tests/test_sequences_and_iterators.py +++ b/tests/test_sequences_and_iterators.py @@ -1,7 +1,8 @@ # -*- coding: utf-8 -*- import pytest -from pybind11_tests import sequences_and_iterators as m + from pybind11_tests import ConstructorStats +from pybind11_tests import sequences_and_iterators as m def isclose(a, b, rel_tol=1e-05, abs_tol=0.0): diff --git a/tests/test_stl.py b/tests/test_stl.py index 6f084590..3f63d6c3 100644 --- a/tests/test_stl.py +++ b/tests/test_stl.py @@ -1,9 +1,8 @@ # -*- coding: utf-8 -*- import pytest +from pybind11_tests import ConstructorStats, UserType from pybind11_tests import stl as m -from pybind11_tests import UserType -from pybind11_tests import ConstructorStats def test_vector(doc): diff --git a/tests/test_stl_binders.py b/tests/test_stl_binders.py index 84132a2b..475a9ec4 100644 --- a/tests/test_stl_binders.py +++ b/tests/test_stl_binders.py @@ -2,7 +2,6 @@ import pytest import env # noqa: F401 - from pybind11_tests import stl_binders as m diff --git a/tools/libsize.py b/tools/libsize.py index 589c317f..1551477e 100644 --- a/tools/libsize.py +++ b/tools/libsize.py @@ -1,5 +1,6 @@ # -*- coding: utf-8 -*- -from __future__ import print_function, division +from __future__ import division, print_function + import os import sys diff --git a/tools/make_changelog.py b/tools/make_changelog.py index 4e8fbf5b..629c284d 100755 --- a/tools/make_changelog.py +++ b/tools/make_changelog.py @@ -4,11 +4,9 @@ import re import ghapi.all - from rich import print from rich.syntax import Syntax - ENTRY = re.compile( r""" Suggested \s changelog \s entry: From 1bcd94c481bba513c4c63a62b1f5da13f5e7978d Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Sat, 14 Aug 2021 07:41:27 -0700 Subject: [PATCH 038/100] Removing last remnants of pragma block at the top of pybind11.h (#3186) * Removing last remnants of pragma block at the top of pybind11.h, defaulting CUDA, GCC7, GCC8 to PYBIND11_NOINLINE_DISABLED, with the option to define PYBIND11_NOINLINE_FORCED. * Unique SOSIZE prefix to make it easier to extract the sosizes from the GitHub logs. * Commenting out PYBIND11_WERROR block, for noinline testing. * Undoing accidental change. * `#define PYBIND11_NOINLINE_FORCED` * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * `#define PYBIND11_NOINLINE_DISABLED` * Going back to default (removing `#define PYBIND11_NOINLINE_DISABLED`). * `#define PYBIND11_NOINLINE_FORCED` * Undoing all changes releated to measuring sosizes. * Rollback of PR #3030 (Working around Centos 8 failure). * Disabling -Werror for GNU (experiment). * Commenting out the entire `if(PYBIND11_WERROR)` again (although that is not expected to make a difference, but who knows what I am overlooking). * Adding `-DCMAKE_BUILD_TYPE=Release` * Undoing change to tests/CMakeLists.txt (uncommenting `if(PYBIND11_WERROR)` block). * post `git rebase master -X theirs` fixups. * Adding measurements to comment for `PYBIND11_NOINLINE_FORCED`. Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- include/pybind11/detail/common.h | 11 +++++++++++ include/pybind11/pybind11.h | 9 --------- 2 files changed, 11 insertions(+), 9 deletions(-) diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index cc24a52f..8daa3390 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -99,6 +99,17 @@ # endif #endif +// For CUDA, GCC7, GCC8: +// PYBIND11_NOINLINE_FORCED is incompatible with `-Wattributes -Werror`. +// When defining PYBIND11_NOINLINE_FORCED, it is best to also use `-Wno-attributes`. +// However, the measured shared-library size saving when using noinline are only +// 1.7% for CUDA, -0.2% for GCC7, and 0.0% for GCC8 (using -DCMAKE_BUILD_TYPE=MinSizeRel, +// the default under pybind11/tests). +#if !defined(PYBIND11_NOINLINE_FORCED) && \ + (defined(__CUDACC__) || (defined(__GNUC__) && (__GNUC__ == 7 || __GNUC__ == 8))) +# define PYBIND11_NOINLINE_DISABLED +#endif + // The PYBIND11_NOINLINE macro is for function DEFINITIONS. // In contrast, FORWARD DECLARATIONS should never use this macro: // https://stackoverflow.com/questions/9317473/forward-declaration-of-inline-functions diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 20df6a01..7b7b3ca7 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -10,11 +10,6 @@ #pragma once -#if defined(__CUDACC__) || (defined(__GNUC__) && (__GNUC__ == 7 || __GNUC__ == 8)) -# pragma GCC diagnostic push -# pragma GCC diagnostic ignored "-Wattributes" -#endif - #include "attr.h" #include "gil.h" #include "options.h" @@ -2383,7 +2378,3 @@ PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) #if defined(__GNUC__) && __GNUC__ == 7 # pragma GCC diagnostic pop // -Wnoexcept-type #endif - -#if defined(__CUDACC__) || (defined(__GNUC__) && (__GNUC__ == 7 || __GNUC__ == 8)) -# pragma GCC diagnostic pop -#endif From 617cb653ec513b4e02d7104b05fb75c26d10e79e Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Sat, 14 Aug 2021 12:25:54 -0400 Subject: [PATCH 039/100] [Bugfix] Fix errant const methods (#3194) * Fix errant const methods * Remove NOLINT since clang-tidy is pretty conservative * Missed one * Fix a few more errors * Add reviewer suggested comments * Run clang-format --- include/pybind11/detail/common.h | 2 +- include/pybind11/detail/type_caster_base.h | 9 ++++++--- include/pybind11/pytypes.h | 10 +++++----- 3 files changed, 12 insertions(+), 9 deletions(-) diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 8daa3390..0d12d734 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -522,7 +522,7 @@ struct instance { void allocate_layout(); /// Destroys/deallocates all of the above - void deallocate_layout() const; + void deallocate_layout(); /// Returns the value_and_holder wrapper for the given type (or the first, if `find_type` /// omitted). Returns a default-constructed (with `.inst = nullptr`) object on failure if diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h index 14cb27ca..5a5acc2e 100644 --- a/include/pybind11/detail/type_caster_base.h +++ b/include/pybind11/detail/type_caster_base.h @@ -241,7 +241,8 @@ struct value_and_holder { ? inst->simple_holder_constructed : (inst->nonsimple.status[index] & instance::status_holder_constructed) != 0u; } - void set_holder_constructed(bool v = true) const { + // NOLINTNEXTLINE(readability-make-member-function-const) + void set_holder_constructed(bool v = true) { if (inst->simple_layout) inst->simple_holder_constructed = v; else if (v) @@ -254,7 +255,8 @@ struct value_and_holder { ? inst->simple_instance_registered : ((inst->nonsimple.status[index] & instance::status_instance_registered) != 0); } - void set_instance_registered(bool v = true) const { + // NOLINTNEXTLINE(readability-make-member-function-const) + void set_instance_registered(bool v = true) { if (inst->simple_layout) inst->simple_instance_registered = v; else if (v) @@ -397,7 +399,8 @@ PYBIND11_NOINLINE void instance::allocate_layout() { owned = true; } -PYBIND11_NOINLINE void instance::deallocate_layout() const { +// NOLINTNEXTLINE(readability-make-member-function-const) +PYBIND11_NOINLINE void instance::deallocate_layout() { if (!simple_layout) PyMem_Free(nonsimple.values_and_holders); } diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index 21e14960..dc1607ff 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -1393,7 +1393,7 @@ public: bool empty() const { return size() == 0; } detail::dict_iterator begin() const { return {*this, 0}; } detail::dict_iterator end() const { return {}; } - void clear() const { PyDict_Clear(ptr()); } + void clear() /* py-non-const */ { PyDict_Clear(ptr()); } template bool contains(T &&key) const { return PyDict_Contains(m_ptr, detail::object_or_cast(std::forward(key)).ptr()) == 1; } @@ -1435,10 +1435,10 @@ public: detail::item_accessor operator[](handle h) const { return object::operator[](h); } detail::list_iterator begin() const { return {*this, 0}; } detail::list_iterator end() const { return {*this, PyList_GET_SIZE(m_ptr)}; } - template void append(T &&val) const { + template void append(T &&val) /* py-non-const */ { PyList_Append(m_ptr, detail::object_or_cast(std::forward(val)).ptr()); } - template void insert(size_t index, T &&val) const { + template void insert(size_t index, T &&val) /* py-non-const */ { PyList_Insert(m_ptr, static_cast(index), detail::object_or_cast(std::forward(val)).ptr()); } @@ -1455,10 +1455,10 @@ public: } size_t size() const { return (size_t) PySet_Size(m_ptr); } bool empty() const { return size() == 0; } - template bool add(T &&val) const { + template bool add(T &&val) /* py-non-const */ { return PySet_Add(m_ptr, detail::object_or_cast(std::forward(val)).ptr()) == 0; } - void clear() const { PySet_Clear(m_ptr); } + void clear() /* py-non-const */ { PySet_Clear(m_ptr); } template bool contains(T &&val) const { return PySet_Contains(m_ptr, detail::object_or_cast(std::forward(val)).ptr()) == 1; } From 774b5ff90bbc2fac198f03a246e3517a53f8c637 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Tue, 17 Aug 2021 16:49:39 -0700 Subject: [PATCH 040/100] Removing obsolete eigen.h warning suppression pragmas. (#3198) * Removing all pragma from eigen.h * Removing -Werror or equivalent from tests/CMakeLists.txt * Restoring tests/CMakeLists.txt from master. * Adding 4 PYBIND11_SILENCE_MSVC_C4127. * Compatibility with -Wconversion, -Wdeprecated * Introducing PYBIND11_COMPATIBILITY_WDEPRECATED_COPY * Systematically using --verbose for compilations where possible (cmake 3.14 or newer). Also changing all `cmake -t` to `--target`, `-v` to `--verbose`, `check` to `pytest`, for consistency (to make it easier to pin-point all commands of a certain type). Also removing one `-j 2` for `pytest` in hopes of reducing flakes due to races in test_iostream and in prints from destructors. * Commenting out pragmas as an experiment to reproduce previous observation. * Removing all (newly added, but already commented-out) pragma code, adding HINT use -isystem (as cmake does). * Restoring ci.yml from master. Those changes are better handled separately. BTW: in the last CI run there was still a test_iostream flake, even without the -j 2 for running the tests (verfied by inspecting the log). --- include/pybind11/eigen.h | 41 ++++++++++------------------------------ 1 file changed, 10 insertions(+), 31 deletions(-) diff --git a/include/pybind11/eigen.h b/include/pybind11/eigen.h index 218fe270..b03e15f6 100644 --- a/include/pybind11/eigen.h +++ b/include/pybind11/eigen.h @@ -9,30 +9,14 @@ #pragma once +/* HINT: To suppress warnings originating from the Eigen headers, use -isystem. + See also: + https://stackoverflow.com/questions/2579576/i-dir-vs-isystem-dir + https://stackoverflow.com/questions/1741816/isystem-for-ms-visual-studio-c-compiler +*/ + #include "numpy.h" -#if defined(__INTEL_COMPILER) -# pragma warning(disable: 1682) // implicit conversion of a 64-bit integral type to a smaller integral type (potential portability problem) -#elif defined(__GNUG__) || defined(__clang__) -# pragma GCC diagnostic push -# pragma GCC diagnostic ignored "-Wconversion" -# pragma GCC diagnostic ignored "-Wdeprecated-declarations" -# ifdef __clang__ -// Eigen generates a bunch of implicit-copy-constructor-is-deprecated warnings with -Wdeprecated -// under Clang, so disable that warning here: -# pragma GCC diagnostic ignored "-Wdeprecated" -# endif -# if __GNUC__ >= 7 -# pragma GCC diagnostic ignored "-Wint-in-bool-context" -# endif -#endif - -#if defined(_MSC_VER) -# pragma warning(push) -# pragma warning(disable: 4127) // warning C4127: Conditional expression is constant -# pragma warning(disable: 4996) // warning C4996: std::unary_negate is deprecated in C++17 -#endif - #include #include @@ -153,7 +137,8 @@ template struct EigenProps { np_cols = a.shape(1), np_rstride = a.strides(0) / static_cast(sizeof(Scalar)), np_cstride = a.strides(1) / static_cast(sizeof(Scalar)); - if ((fixed_rows && np_rows != rows) || (fixed_cols && np_cols != cols)) + if ((PYBIND11_SILENCE_MSVC_C4127(fixed_rows) && np_rows != rows) || + (PYBIND11_SILENCE_MSVC_C4127(fixed_cols) && np_cols != cols)) return false; return {np_rows, np_cols, np_rstride, np_cstride}; @@ -165,7 +150,7 @@ template struct EigenProps { stride = a.strides(0) / static_cast(sizeof(Scalar)); if (vector) { // Eigen type is a compile-time vector - if (fixed && size != n) + if (PYBIND11_SILENCE_MSVC_C4127(fixed) && size != n) return false; // Vector size mismatch return {rows == 1 ? 1 : n, cols == 1 ? 1 : n, stride}; } @@ -179,7 +164,7 @@ template struct EigenProps { if (cols != n) return false; return {1, n, stride}; } // Otherwise it's either fully dynamic, or column dynamic; both become a column vector - if (fixed_rows && rows != n) return false; + if (PYBIND11_SILENCE_MSVC_C4127(fixed_rows) && rows != n) return false; return {n, 1, stride}; } @@ -596,9 +581,3 @@ struct type_caster::value>> { PYBIND11_NAMESPACE_END(detail) PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) - -#if defined(__GNUG__) || defined(__clang__) -# pragma GCC diagnostic pop -#elif defined(_MSC_VER) -# pragma warning(pop) -#endif From 998d45e4313067d752476338d090f388f0a5d9ed Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Thu, 19 Aug 2021 11:37:04 -0700 Subject: [PATCH 041/100] Cleanup of file-scoped and globally-scoped warning suppression pragmas across pybind11 header files. (#3201) * Removing all MSVC C4127 warning suppression pragmas. * Removing MSVC /WX (WERROR). To get a full list of all warnings. * Inserting PYBIND11_SILENCE_MSVC_C4127. Changing one runtime if to #if. * Changing PYBIND11_SILENCE_MSVC_C4127 macro to use absolute namespace (for use outside pybind11 include directory). * Restoring MSVC /WX (WERROR). * Removing globally-scoped suppression for clang -Wunsequenced. Based on an experiment under PR #3202 it is obsolete and can simply be removed. --- include/pybind11/detail/common.h | 2 +- include/pybind11/numpy.h | 13 ++----------- include/pybind11/operators.h | 14 ++------------ include/pybind11/stl.h | 9 --------- tests/test_builtin_casters.cpp | 14 +++++--------- 5 files changed, 10 insertions(+), 42 deletions(-) diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 0d12d734..cb52aa27 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -968,7 +968,7 @@ inline void silence_unused_warnings(Args &&...) {} // warning C4127: Conditional expression is constant constexpr inline bool silence_msvc_c4127(bool cond) { return cond; } -# define PYBIND11_SILENCE_MSVC_C4127(...) detail::silence_msvc_c4127(__VA_ARGS__) +# define PYBIND11_SILENCE_MSVC_C4127(...) ::pybind11::detail::silence_msvc_c4127(__VA_ARGS__) #else # define PYBIND11_SILENCE_MSVC_C4127(...) __VA_ARGS__ diff --git a/include/pybind11/numpy.h b/include/pybind11/numpy.h index 2dc0f29f..7717059f 100644 --- a/include/pybind11/numpy.h +++ b/include/pybind11/numpy.h @@ -25,11 +25,6 @@ #include #include -#if defined(_MSC_VER) -# pragma warning(push) -# pragma warning(disable: 4127) // warning C4127: Conditional expression is constant -#endif - /* This will be true on all flat address space platforms and allows us to reduce the whole npy_intp / ssize_t / Py_intptr_t business down to just ssize_t for all size and dimension types (e.g. shape, strides, indexing), instead of inflicting this @@ -747,7 +742,7 @@ public: * and the caller must take care not to access invalid dimensions or dimension indices. */ template detail::unchecked_mutable_reference mutable_unchecked() & { - if (Dims >= 0 && ndim() != Dims) + if (PYBIND11_SILENCE_MSVC_C4127(Dims >= 0) && ndim() != Dims) throw std::domain_error("array has incorrect number of dimensions: " + std::to_string(ndim()) + "; expected " + std::to_string(Dims)); return detail::unchecked_mutable_reference(mutable_data(), shape(), strides(), ndim()); @@ -761,7 +756,7 @@ public: * invalid dimensions or dimension indices. */ template detail::unchecked_reference unchecked() const & { - if (Dims >= 0 && ndim() != Dims) + if (PYBIND11_SILENCE_MSVC_C4127(Dims >= 0) && ndim() != Dims) throw std::domain_error("array has incorrect number of dimensions: " + std::to_string(ndim()) + "; expected " + std::to_string(Dims)); return detail::unchecked_reference(data(), shape(), strides(), ndim()); @@ -1708,7 +1703,3 @@ Helper vectorize(Return (Class::*f)(Args...) const) { } PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) - -#if defined(_MSC_VER) -#pragma warning(pop) -#endif diff --git a/include/pybind11/operators.h b/include/pybind11/operators.h index 086cb4cf..2a615315 100644 --- a/include/pybind11/operators.h +++ b/include/pybind11/operators.h @@ -11,13 +11,6 @@ #include "pybind11.h" -#if defined(__clang__) && !defined(__INTEL_COMPILER) -# pragma clang diagnostic ignored "-Wunsequenced" // multiple unsequenced modifications to 'self' (when using def(py::self OP Type())) -#elif defined(_MSC_VER) -# pragma warning(push) -# pragma warning(disable: 4127) // warning C4127: Conditional expression is constant -#endif - PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) PYBIND11_NAMESPACE_BEGIN(detail) @@ -58,7 +51,8 @@ template struct op_ { using op = op_impl; cl.def(op::name(), &op::execute, is_operator(), extra...); #if PY_MAJOR_VERSION < 3 - if (id == op_truediv || id == op_itruediv) + if (PYBIND11_SILENCE_MSVC_C4127(id == op_truediv) || + PYBIND11_SILENCE_MSVC_C4127(id == op_itruediv)) cl.def(id == op_itruediv ? "__idiv__" : ot == op_l ? "__div__" : "__rdiv__", &op::execute, is_operator(), extra...); #endif @@ -167,7 +161,3 @@ using detail::self; using detail::hash; PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) - -#if defined(_MSC_VER) -# pragma warning(pop) -#endif diff --git a/include/pybind11/stl.h b/include/pybind11/stl.h index 51994c65..fe391f70 100644 --- a/include/pybind11/stl.h +++ b/include/pybind11/stl.h @@ -19,11 +19,6 @@ #include #include -#if defined(_MSC_VER) -#pragma warning(push) -#pragma warning(disable: 4127) // warning C4127: Conditional expression is constant -#endif - #ifdef __has_include // std::optional (but including it in c++14 mode isn't allowed) # if defined(PYBIND11_CPP17) && __has_include() @@ -390,7 +385,3 @@ inline std::ostream &operator<<(std::ostream &os, const handle &obj) { } PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) - -#if defined(_MSC_VER) -#pragma warning(pop) -#endif diff --git a/tests/test_builtin_casters.cpp b/tests/test_builtin_casters.cpp index 51e410ca..be4431aa 100644 --- a/tests/test_builtin_casters.cpp +++ b/tests/test_builtin_casters.cpp @@ -10,11 +10,6 @@ #include "pybind11_tests.h" #include -#if defined(_MSC_VER) -# pragma warning(push) -# pragma warning(disable: 4127) // warning C4127: Conditional expression is constant -#endif - struct ConstRefCasted { int tag; }; @@ -73,7 +68,7 @@ TEST_SUBMODULE(builtin_casters, m) { std::wstring wstr; wstr.push_back(0x61); // a wstr.push_back(0x2e18); // ⸘ - if (sizeof(wchar_t) == 2) { wstr.push_back(mathbfA16_1); wstr.push_back(mathbfA16_2); } // 𝐀, utf16 + if (PYBIND11_SILENCE_MSVC_C4127(sizeof(wchar_t) == 2)) { wstr.push_back(mathbfA16_1); wstr.push_back(mathbfA16_2); } // 𝐀, utf16 else { wstr.push_back((wchar_t) mathbfA32); } // 𝐀, utf32 wstr.push_back(0x7a); // z @@ -83,11 +78,12 @@ TEST_SUBMODULE(builtin_casters, m) { m.def("good_wchar_string", [=]() { return wstr; }); // a‽𝐀z m.def("bad_utf8_string", []() { return std::string("abc\xd0" "def"); }); m.def("bad_utf16_string", [=]() { return std::u16string({ b16, char16_t(0xd800), z16 }); }); +#if PY_MAJOR_VERSION >= 3 // Under Python 2.7, invalid unicode UTF-32 characters don't appear to trigger UnicodeDecodeError - if (PY_MAJOR_VERSION >= 3) - m.def("bad_utf32_string", [=]() { return std::u32string({ a32, char32_t(0xd800), z32 }); }); - if (PY_MAJOR_VERSION >= 3 || sizeof(wchar_t) == 2) + m.def("bad_utf32_string", [=]() { return std::u32string({ a32, char32_t(0xd800), z32 }); }); + if (PYBIND11_SILENCE_MSVC_C4127(sizeof(wchar_t) == 2)) m.def("bad_wchar_string", [=]() { return std::wstring({ wchar_t(0x61), wchar_t(0xd800) }); }); +#endif m.def("u8_Z", []() -> char { return 'Z'; }); m.def("u8_eacute", []() -> char { return '\xe9'; }); m.def("u16_ibang", [=]() -> char16_t { return ib16; }); From 4f29b8a45b8d1e52d06d920f7a8140811a925fd9 Mon Sep 17 00:00:00 2001 From: Liam Keegan Date: Thu, 19 Aug 2021 20:42:55 +0200 Subject: [PATCH 042/100] ci: extend msys2 mingw CI (#3207) * extend msys2 CI - add 32-bit job - add c++11/17 c++/interface tests copied from standard ci - add numpy/scipy * account for padding of PartialStruct in numpy dtypes test with mingw32 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * msys2 ci: add c++14 tests Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .github/workflows/ci.yml | 81 +++++++++++++++++++++++++++++++------- tests/test_numpy_dtypes.py | 14 +++++-- 2 files changed, 78 insertions(+), 17 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index be3fa723..a3664ddf 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -861,32 +861,85 @@ jobs: run: cmake --build build -t check mingw: + name: "🐍 3 • windows-latest • ${{ matrix.sys }}" runs-on: windows-latest defaults: run: shell: msys2 {0} + strategy: + fail-fast: false + matrix: + include: + - { sys: mingw64, env: x86_64 } + - { sys: mingw32, env: i686 } steps: - uses: msys2/setup-msys2@v2 with: + msystem: ${{matrix.sys}} install: >- - mingw-w64-x86_64-gcc - mingw-w64-x86_64-python-pip - mingw-w64-x86_64-cmake - mingw-w64-x86_64-make - mingw-w64-x86_64-python-pytest - mingw-w64-x86_64-eigen3 - mingw-w64-x86_64-boost - mingw-w64-x86_64-catch + git + mingw-w64-${{matrix.env}}-gcc + mingw-w64-${{matrix.env}}-python-pip + mingw-w64-${{matrix.env}}-python-numpy + mingw-w64-${{matrix.env}}-python-scipy + mingw-w64-${{matrix.env}}-cmake + mingw-w64-${{matrix.env}}-make + mingw-w64-${{matrix.env}}-python-pytest + mingw-w64-${{matrix.env}}-eigen3 + mingw-w64-${{matrix.env}}-boost + mingw-w64-${{matrix.env}}-catch - - uses: actions/checkout@v1 + - uses: actions/checkout@v2 - - name: Configure + - name: Configure C++11 # LTO leads to many undefined reference like # `pybind11::detail::function_call::function_call(pybind11::detail::function_call&&) - run: cmake -G "MinGW Makefiles" -S . -B build + run: cmake -G "MinGW Makefiles" -DCMAKE_CXX_STANDARD=11 -S . -B build - - name: Build + - name: Build C++11 run: cmake --build build -j 2 - - name: Python tests - run: cmake --build build --target pytest + - name: Python tests C++11 + run: cmake --build build --target pytest -j 2 + + - name: C++11 tests + run: cmake --build build --target cpptest -j 2 + + - name: Interface test C++11 + run: cmake --build build --target test_cmake_build + + - name: Clean directory + run: git clean -fdx + + - name: Configure C++14 + run: cmake -G "MinGW Makefiles" -DCMAKE_CXX_STANDARD=14 -S . -B build2 + + - name: Build C++14 + run: cmake --build build2 -j 2 + + - name: Python tests C++14 + run: cmake --build build2 --target pytest -j 2 + + - name: C++14 tests + run: cmake --build build2 --target cpptest -j 2 + + - name: Interface test C++14 + run: cmake --build build2 --target test_cmake_build + + - name: Clean directory + run: git clean -fdx + + - name: Configure C++17 + run: cmake -G "MinGW Makefiles" -DCMAKE_CXX_STANDARD=17 -S . -B build3 + + - name: Build C++17 + run: cmake --build build3 -j 2 + + - name: Python tests C++17 + run: cmake --build build3 --target pytest -j 2 + + - name: C++17 tests + run: cmake --build build3 --target cpptest -j 2 + + - name: Interface test C++17 + run: cmake --build build3 --target test_cmake_build diff --git a/tests/test_numpy_dtypes.py b/tests/test_numpy_dtypes.py index 6ea064d5..06e57832 100644 --- a/tests/test_numpy_dtypes.py +++ b/tests/test_numpy_dtypes.py @@ -63,14 +63,20 @@ def partial_ld_offset(): def partial_dtype_fmt(): ld = np.dtype("longdouble") partial_ld_off = partial_ld_offset() - return dt_fmt().format(ld.itemsize, partial_ld_off, partial_ld_off + ld.itemsize) + partial_size = partial_ld_off + ld.itemsize + partial_end_padding = partial_size % np.dtype("uint64").alignment + return dt_fmt().format( + ld.itemsize, partial_ld_off, partial_size + partial_end_padding + ) def partial_nested_fmt(): ld = np.dtype("longdouble") partial_nested_off = 8 + 8 * (ld.alignment > 8) partial_ld_off = partial_ld_offset() - partial_nested_size = partial_nested_off * 2 + partial_ld_off + ld.itemsize + partial_size = partial_ld_off + ld.itemsize + partial_end_padding = partial_size % np.dtype("uint64").alignment + partial_nested_size = partial_nested_off * 2 + partial_size + partial_end_padding return "{{'names':['a'], 'formats':[{}], 'offsets':[{}], 'itemsize':{}}}".format( partial_dtype_fmt(), partial_nested_off, partial_nested_size ) @@ -91,10 +97,12 @@ def test_format_descriptors(): ldbl_fmt = ("4x" if ld.alignment > 4 else "") + ld.char ss_fmt = "^T{?:bool_:3xI:uint_:f:float_:" + ldbl_fmt + ":ldbl_:}" dbl = np.dtype("double") + end_padding = ld.itemsize % np.dtype("uint64").alignment partial_fmt = ( "^T{?:bool_:3xI:uint_:f:float_:" + str(4 * (dbl.alignment > 4) + dbl.itemsize + 8 * (ld.alignment > 8)) - + "xg:ldbl_:}" + + "xg:ldbl_:" + + (str(end_padding) + "x}" if end_padding > 0 else "}") ) nested_extra = str(max(8, ld.alignment)) assert m.print_format_descriptors() == [ From b3d18f382f3bc37ec4d95cfe2045db72cf0e950d Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 23 Aug 2021 15:22:12 -0400 Subject: [PATCH 043/100] [pre-commit.ci] pre-commit autoupdate (#3213) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit updates: - [github.com/asottile/pyupgrade: v2.23.3 → v2.24.0](https://github.com/asottile/pyupgrade/compare/v2.23.3...v2.24.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .pre-commit-config.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index a5900df1..96ddcf74 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -31,7 +31,7 @@ repos: exclude: ^noxfile.py$ - repo: https://github.com/asottile/pyupgrade - rev: v2.23.3 + rev: v2.24.0 hooks: - id: pyupgrade From fdac5fbf7c1c3b63b6f3067734a10db0b409e28c Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Mon, 23 Aug 2021 15:05:54 -0700 Subject: [PATCH 044/100] chore: support targeting different Python versions with nox (#3214) --- .github/CONTRIBUTING.md | 4 ++-- noxfile.py | 4 +++- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index ff1997f9..39c32b2a 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -68,8 +68,8 @@ nox -l # Run linters nox -s lint -# Run tests -nox -s tests +# Run tests on Python 3.9 +nox -s tests-3.9 # Build and preview docs nox -s docs -- serve diff --git a/noxfile.py b/noxfile.py index 55b1d180..23417982 100644 --- a/noxfile.py +++ b/noxfile.py @@ -2,6 +2,8 @@ import nox nox.options.sessions = ["lint", "tests", "tests_packaging"] +PYTHON_VERISONS = ["2.7", "3.5", "3.6", "3.7", "3.8", "3.9", "3.10"] + @nox.session(reuse_venv=True) def lint(session: nox.Session) -> None: @@ -12,7 +14,7 @@ def lint(session: nox.Session) -> None: session.run("pre-commit", "run", "-a") -@nox.session +@nox.session(python=PYTHON_VERISONS) def tests(session: nox.Session) -> None: """ Run the tests (requires a compiler). From 6cbabc4b8c86a838216cb9d17dce14167222e6f7 Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Mon, 23 Aug 2021 18:42:19 -0400 Subject: [PATCH 045/100] maint(clang-tidy): Enable cpp-coreguideline slicing checks (#3210) * maint(clang-tidy): add a clang-tidy slicing check * Add self + touch up readme * Fix typo --- .clang-tidy | 1 + README.rst | 4 ++-- include/pybind11/pybind11.h | 2 +- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/.clang-tidy b/.clang-tidy index c83b9b2f..cefffba1 100644 --- a/.clang-tidy +++ b/.clang-tidy @@ -3,6 +3,7 @@ FormatStyle: file Checks: ' *bugprone*, cppcoreguidelines-init-variables, +cppcoreguidelines-slicing, clang-analyzer-optin.cplusplus.VirtualCall, llvm-namespace-comment, misc-misplaced-const, diff --git a/README.rst b/README.rst index 57eb06e5..7ce57b03 100644 --- a/README.rst +++ b/README.rst @@ -134,9 +134,9 @@ About This project was created by `Wenzel Jakob `_. Significant features and/or improvements to the code were contributed by Jonas Adler, Lori A. Burns, -Sylvain Corlay, Eric Cousineau, Ralf Grosse-Kunstleve, Trent Houliston, Axel +Sylvain Corlay, Eric Cousineau, Aaron Gokaslan, Ralf Grosse-Kunstleve, Trent Houliston, Axel Huebl, @hulucc, Yannick Jadoul, Sergey Lyskov Johan Mabille, Tomasz Miąsko, -Dean Moldovan, Ben Pritchard, Jason Rhinelander, Boris Schäling, Pim +Dean Moldovan, Ben Pritchard, Jason Rhinelander, Boris Schäling, Pim Schellart, Henry Schreiner, Ivan Smirnov, Boris Staletic, and Patrick Stewart. We thank Google for a generous financial contribution to the continuous diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 7b7b3ca7..47b04214 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -1664,7 +1664,7 @@ inline str enum_name(handle arg) { } struct enum_base { - enum_base(handle base, handle parent) : m_base(base), m_parent(parent) { } + enum_base(const handle &base, const handle &parent) : m_base(base), m_parent(parent) { } PYBIND11_NOINLINE void init(bool is_arithmetic, bool is_convertible) { m_base.attr("__entries") = dict(); From c8ce4b8df854a630a5f02192305c183299778b84 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Mon, 23 Aug 2021 17:30:01 -0700 Subject: [PATCH 046/100] Clone of @virtuald's PR #2112 with minor enhancements. (#3215) * Add py::raise_from to enable chaining exceptions on Python 3.3+ * Use 'raise from' in initialization * Documenting the exact base version of _PyErr_FormatVFromCause, adding back `assert`s. Co-authored-by: Dustin Spicuzza --- docs/advanced/exceptions.rst | 28 ++++++++++++++++++ include/pybind11/detail/common.h | 15 ++++++++++ include/pybind11/pytypes.h | 41 +++++++++++++++++++++++++++ tests/test_embed/test_interpreter.cpp | 16 +++++++++++ tests/test_exceptions.cpp | 20 +++++++++++++ tests/test_exceptions.py | 16 +++++++++++ 6 files changed, 136 insertions(+) diff --git a/docs/advanced/exceptions.rst b/docs/advanced/exceptions.rst index 738fb0ba..2aaa0ad3 100644 --- a/docs/advanced/exceptions.rst +++ b/docs/advanced/exceptions.rst @@ -323,6 +323,34 @@ Alternately, to ignore the error, call `PyErr_Clear Any Python error must be thrown or cleared, or Python/pybind11 will be left in an invalid state. +Chaining exceptions ('raise from') +================================== + +In Python 3.3 a mechanism for indicating that exceptions were caused by other +exceptions was introduced: + +.. code-block:: py + + try: + print(1 / 0) + except Exception as exc: + raise RuntimeError("could not divide by zero") from exc + +To do a similar thing in pybind11, you can use the ``py::raise_from`` function. It +sets the current python error indicator, so to continue propagating the exception +you should ``throw py::error_already_set()`` (Python 3 only). + +.. code-block:: cpp + + try { + py::eval("print(1 / 0")); + } catch (py::error_already_set &e) { + py::raise_from(e, PyExc_RuntimeError, "could not divide by zero"); + throw py::error_already_set(); + } + +.. versionadded:: 2.8 + .. _unraisable_exceptions: Handling unraisable exceptions diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index cb52aa27..5050da80 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -315,6 +315,19 @@ extern "C" { } \ } +#if PY_VERSION_HEX >= 0x03030000 + +#define PYBIND11_CATCH_INIT_EXCEPTIONS \ + catch (pybind11::error_already_set &e) { \ + pybind11::raise_from(e, PyExc_ImportError, "initialization failed"); \ + return nullptr; \ + } catch (const std::exception &e) { \ + PyErr_SetString(PyExc_ImportError, e.what()); \ + return nullptr; \ + } \ + +#else + #define PYBIND11_CATCH_INIT_EXCEPTIONS \ catch (pybind11::error_already_set &e) { \ PyErr_SetString(PyExc_ImportError, e.what()); \ @@ -324,6 +337,8 @@ extern "C" { return nullptr; \ } \ +#endif + /** \rst ***Deprecated in favor of PYBIND11_MODULE*** diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index dc1607ff..85f6a40a 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -382,6 +382,47 @@ private: # pragma warning(pop) #endif +#if PY_VERSION_HEX >= 0x03030000 + +/// Replaces the current Python error indicator with the chosen error, performing a +/// 'raise from' to indicate that the chosen error was caused by the original error. +inline void raise_from(PyObject *type, const char *message) { + // Based on _PyErr_FormatVFromCause: + // https://github.com/python/cpython/blob/467ab194fc6189d9f7310c89937c51abeac56839/Python/errors.c#L405 + // See https://github.com/pybind/pybind11/pull/2112 for details. + PyObject *exc = nullptr, *val = nullptr, *val2 = nullptr, *tb = nullptr; + + assert(PyErr_Occurred()); + PyErr_Fetch(&exc, &val, &tb); + PyErr_NormalizeException(&exc, &val, &tb); + if (tb != nullptr) { + PyException_SetTraceback(val, tb); + Py_DECREF(tb); + } + Py_DECREF(exc); + assert(!PyErr_Occurred()); + + PyErr_SetString(type, message); + + PyErr_Fetch(&exc, &val2, &tb); + PyErr_NormalizeException(&exc, &val2, &tb); + Py_INCREF(val); + PyException_SetCause(val2, val); + PyException_SetContext(val2, val); + PyErr_Restore(exc, val2, tb); +} + +/// Sets the current Python error indicator with the chosen error, performing a 'raise from' +/// from the error contained in error_already_set to indicate that the chosen error was +/// caused by the original error. After this function is called error_already_set will +/// no longer contain an error. +inline void raise_from(error_already_set& err, PyObject *type, const char *message) { + err.restore(); + raise_from(type, message); +} + +#endif + /** \defgroup python_builtins _ Unless stated otherwise, the following C++ functions behave the same as their Python counterparts. diff --git a/tests/test_embed/test_interpreter.cpp b/tests/test_embed/test_interpreter.cpp index cd50e952..b40ff481 100644 --- a/tests/test_embed/test_interpreter.cpp +++ b/tests/test_embed/test_interpreter.cpp @@ -74,8 +74,24 @@ TEST_CASE("Import error handling") { REQUIRE_NOTHROW(py::module_::import("widget_module")); REQUIRE_THROWS_WITH(py::module_::import("throw_exception"), "ImportError: C++ Error"); +#if PY_VERSION_HEX >= 0x03030000 + REQUIRE_THROWS_WITH(py::module_::import("throw_error_already_set"), + Catch::Contains("ImportError: initialization failed")); + + auto locals = py::dict("is_keyerror"_a=false, "message"_a="not set"); + py::exec(R"( + try: + import throw_error_already_set + except ImportError as e: + is_keyerror = type(e.__cause__) == KeyError + message = str(e.__cause__) + )", py::globals(), locals); + REQUIRE(locals["is_keyerror"].cast() == true); + REQUIRE(locals["message"].cast() == "'missing'"); +#else REQUIRE_THROWS_WITH(py::module_::import("throw_error_already_set"), Catch::Contains("ImportError: KeyError")); +#endif } TEST_CASE("There can be only one interpreter") { diff --git a/tests/test_exceptions.cpp b/tests/test_exceptions.cpp index e28f0bb7..4805a024 100644 --- a/tests/test_exceptions.cpp +++ b/tests/test_exceptions.cpp @@ -262,4 +262,24 @@ TEST_SUBMODULE(exceptions, m) { m.def("simple_bool_passthrough", [](bool x) {return x;}); m.def("throw_should_be_translated_to_key_error", []() { throw shared_exception(); }); + +#if PY_VERSION_HEX >= 0x03030000 + + m.def("raise_from", []() { + PyErr_SetString(PyExc_ValueError, "inner"); + py::raise_from(PyExc_ValueError, "outer"); + throw py::error_already_set(); + }); + + m.def("raise_from_already_set", []() { + try { + PyErr_SetString(PyExc_ValueError, "inner"); + throw py::error_already_set(); + } catch (py::error_already_set& e) { + py::raise_from(e, PyExc_ValueError, "outer"); + throw py::error_already_set(); + } + }); + +#endif } diff --git a/tests/test_exceptions.py b/tests/test_exceptions.py index d1edc39f..3821eada 100644 --- a/tests/test_exceptions.py +++ b/tests/test_exceptions.py @@ -24,6 +24,22 @@ def test_error_already_set(msg): assert msg(excinfo.value) == "foo" +@pytest.mark.skipif("env.PY2") +def test_raise_from(msg): + with pytest.raises(ValueError) as excinfo: + m.raise_from() + assert msg(excinfo.value) == "outer" + assert msg(excinfo.value.__cause__) == "inner" + + +@pytest.mark.skipif("env.PY2") +def test_raise_from_already_set(msg): + with pytest.raises(ValueError) as excinfo: + m.raise_from_already_set() + assert msg(excinfo.value) == "outer" + assert msg(excinfo.value.__cause__) == "inner" + + def test_cross_module_exceptions(msg): with pytest.raises(RuntimeError) as excinfo: cm.raise_runtime_error() From 031a700dfd611efc3949d8427d8c87e91a7b6998 Mon Sep 17 00:00:00 2001 From: Jouke Witteveen Date: Thu, 26 Aug 2021 17:04:22 +0200 Subject: [PATCH 047/100] Add make_simple_namespace function and tests (#2840) Co-authored-by: Jouke Witteveen --- docs/advanced/pycpp/object.rst | 43 +++++++++++++++++++++++++++++++++- include/pybind11/cast.h | 10 ++++++++ tests/test_pytypes.cpp | 13 ++++++++++ tests/test_pytypes.py | 13 ++++++++++ 4 files changed, 78 insertions(+), 1 deletion(-) diff --git a/docs/advanced/pycpp/object.rst b/docs/advanced/pycpp/object.rst index 6c7525ce..6fa8d070 100644 --- a/docs/advanced/pycpp/object.rst +++ b/docs/advanced/pycpp/object.rst @@ -20,6 +20,47 @@ Available types include :class:`handle`, :class:`object`, :class:`bool_`, Be sure to review the :ref:`pytypes_gotchas` before using this heavily in your C++ API. +.. _instantiating_compound_types: + +Instantiating compound Python types from C++ +============================================ + +Dictionaries can be initialized in the :class:`dict` constructor: + +.. code-block:: cpp + + using namespace pybind11::literals; // to bring in the `_a` literal + py::dict d("spam"_a=py::none(), "eggs"_a=42); + +A tuple of python objects can be instantiated using :func:`py::make_tuple`: + +.. code-block:: cpp + + py::tuple tup = py::make_tuple(42, py::none(), "spam"); + +Each element is converted to a supported Python type. + +A `simple namespace`_ can be instantiated using +:func:`py::make_simple_namespace`: + +.. code-block:: cpp + + using namespace pybind11::literals; // to bring in the `_a` literal + py::object ns = py::make_simple_namespace("spam"_a=py::none(), "eggs"_a=42); + +Attributes on a namespace can be modified with the :func:`py::delattr`, +:func:`py::getattr`, and :func:`py::setattr` functions. Simple namespaces can +be useful as lightweight stand-ins for class instances. + +.. note:: + + ``make_simple_namespace`` is not available in Python 2. + +.. versionchanged:: 2.8 + ``make_simple_namespace`` added. + +.. _simple namespace: https://docs.python.org/3/library/types.html#types.SimpleNamespace + .. _casting_back_and_forth: Casting back and forth @@ -30,7 +71,7 @@ types to Python, which can be done using :func:`py::cast`: .. code-block:: cpp - MyClass *cls = ..; + MyClass *cls = ...; py::object obj = py::cast(cls); The reverse direction uses the following syntax: diff --git a/include/pybind11/cast.h b/include/pybind11/cast.h index 718dc2de..79bf506d 100644 --- a/include/pybind11/cast.h +++ b/include/pybind11/cast.h @@ -1018,6 +1018,16 @@ template = 0x03030000 +template ()>> +object make_simple_namespace(Args&&... args_) { + PyObject *ns = _PyNamespace_New(dict(std::forward(args_)...).ptr()); + if (!ns) throw error_already_set(); + return reinterpret_steal(ns); +} +#endif + /// \ingroup annotations /// Annotation for arguments struct arg { diff --git a/tests/test_pytypes.cpp b/tests/test_pytypes.cpp index d70536d3..15d007a4 100644 --- a/tests/test_pytypes.cpp +++ b/tests/test_pytypes.cpp @@ -70,6 +70,19 @@ TEST_SUBMODULE(pytypes, m) { m.def("dict_contains", [](const py::dict &dict, const char *val) { return dict.contains(val); }); + // test_tuple + m.def("get_tuple", []() { return py::make_tuple(42, py::none(), "spam"); }); + +#if PY_VERSION_HEX >= 0x03030000 + // test_simple_namespace + m.def("get_simple_namespace", []() { + auto ns = py::make_simple_namespace("attr"_a=42, "x"_a="foo", "wrong"_a=1); + py::delattr(ns, "wrong"); + py::setattr(ns, "right", py::int_(2)); + return ns; + }); +#endif + // test_str m.def("str_from_string", []() { return py::str(std::string("baz")); }); m.def("str_from_bytes", []() { return py::str(py::bytes("boo", 3)); }); diff --git a/tests/test_pytypes.py b/tests/test_pytypes.py index 8a11b187..f873658a 100644 --- a/tests/test_pytypes.py +++ b/tests/test_pytypes.py @@ -99,6 +99,19 @@ def test_dict(capture, doc): assert m.dict_keyword_constructor() == {"x": 1, "y": 2, "z": 3} +def test_tuple(): + assert m.get_tuple() == (42, None, "spam") + + +@pytest.mark.skipif("env.PY2") +def test_simple_namespace(): + ns = m.get_simple_namespace() + assert ns.attr == 42 + assert ns.x == "foo" + assert ns.right == 2 + assert not hasattr(ns, "wrong") + + def test_str(doc): assert m.str_from_string().encode().decode() == "baz" assert m.str_from_bytes().encode().decode() == "boo" From 59ad1e7d05d96ccc4e7090d5da7a328e4512cb7a Mon Sep 17 00:00:00 2001 From: Nick Cullen Date: Thu, 26 Aug 2021 17:12:35 +0200 Subject: [PATCH 048/100] reshape for numpy arrays (#984) * reshape * more tests * Update numpy.h * Update test_numpy_array.py * Update numpy.h * Update numpy.h * Update test_numpy_array.cpp * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix merge bug * Make clang-tidy happy * Add xfail for PyPy * Fix casting issue * Address reviews on additional tests * Fix ordering * Do a little more reordering * Fix typo * Try improving tests * Fix error in reshape * Add one more reshape test * streamlining new tests; removing a few stray msg Co-authored-by: ncullen93 Co-authored-by: NC Cullen Co-authored-by: Aaron Gokaslan Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ralf Grosse-Kunstleve --- include/pybind11/numpy.h | 21 +++++++++++++++++++-- tests/test_numpy_array.cpp | 7 +++++++ tests/test_numpy_array.py | 31 ++++++++++++++++++++++++++++--- 3 files changed, 54 insertions(+), 5 deletions(-) diff --git a/include/pybind11/numpy.h b/include/pybind11/numpy.h index 7717059f..0d0cbdfa 100644 --- a/include/pybind11/numpy.h +++ b/include/pybind11/numpy.h @@ -198,6 +198,8 @@ struct npy_api { // Unused. Not removed because that affects ABI of the class. int (*PyArray_SetBaseObject_)(PyObject *, PyObject *); PyObject* (*PyArray_Resize_)(PyObject*, PyArray_Dims*, int, int); + PyObject* (*PyArray_Newshape_)(PyObject*, PyArray_Dims*, int); + private: enum functions { API_PyArray_GetNDArrayCFeatureVersion = 211, @@ -212,10 +214,11 @@ private: API_PyArray_NewCopy = 85, API_PyArray_NewFromDescr = 94, API_PyArray_DescrNewFromType = 96, + API_PyArray_Newshape = 135, + API_PyArray_Squeeze = 136, API_PyArray_DescrConverter = 174, API_PyArray_EquivTypes = 182, API_PyArray_GetArrayParamsFromObject = 278, - API_PyArray_Squeeze = 136, API_PyArray_SetBaseObject = 282 }; @@ -243,11 +246,13 @@ private: DECL_NPY_API(PyArray_NewCopy); DECL_NPY_API(PyArray_NewFromDescr); DECL_NPY_API(PyArray_DescrNewFromType); + DECL_NPY_API(PyArray_Newshape); + DECL_NPY_API(PyArray_Squeeze); DECL_NPY_API(PyArray_DescrConverter); DECL_NPY_API(PyArray_EquivTypes); DECL_NPY_API(PyArray_GetArrayParamsFromObject); - DECL_NPY_API(PyArray_Squeeze); DECL_NPY_API(PyArray_SetBaseObject); + #undef DECL_NPY_API return api; } @@ -785,6 +790,18 @@ public: if (isinstance(new_array)) { *this = std::move(new_array); } } + /// Optional `order` parameter omitted, to be added as needed. + array reshape(ShapeContainer new_shape) { + detail::npy_api::PyArray_Dims d + = {reinterpret_cast(new_shape->data()), int(new_shape->size())}; + auto new_array + = reinterpret_steal(detail::npy_api::get().PyArray_Newshape_(m_ptr, &d, 0)); + if (!new_array) { + throw error_already_set(); + } + return new_array; + } + /// Ensure that the argument is a NumPy array /// In case of an error, nullptr is returned and the Python error is cleared. static array ensure(handle h, int ExtraFlags = 0) { diff --git a/tests/test_numpy_array.cpp b/tests/test_numpy_array.cpp index 5c22a3d2..4ccfd279 100644 --- a/tests/test_numpy_array.cpp +++ b/tests/test_numpy_array.cpp @@ -405,6 +405,13 @@ TEST_SUBMODULE(numpy_array, sm) { return a; }); + sm.def("reshape_initializer_list", [](py::array_t a, size_t N, size_t M, size_t O) { + return a.reshape({N, M, O}); + }); + sm.def("reshape_tuple", [](py::array_t a, const std::vector &new_shape) { + return a.reshape(new_shape); + }); + sm.def("index_using_ellipsis", [](const py::array &a) { return a[py::make_tuple(0, py::ellipsis(), 0)]; }); diff --git a/tests/test_numpy_array.py b/tests/test_numpy_array.py index 69ba9d49..e96454be 100644 --- a/tests/test_numpy_array.py +++ b/tests/test_numpy_array.py @@ -411,7 +411,7 @@ def test_array_unchecked_fixed_dims(msg): assert m.proxy_auxiliaries2_const_ref(z1) -def test_array_unchecked_dyn_dims(msg): +def test_array_unchecked_dyn_dims(): z1 = np.array([[1, 2], [3, 4]], dtype="float64") m.proxy_add2_dyn(z1, 10) assert np.all(z1 == [[11, 12], [13, 14]]) @@ -444,7 +444,7 @@ def test_initializer_list(): assert m.array_initializer_list4().shape == (1, 2, 3, 4) -def test_array_resize(msg): +def test_array_resize(): a = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9], dtype="float64") m.array_reshape2(a) assert a.size == 9 @@ -470,12 +470,37 @@ def test_array_resize(msg): @pytest.mark.xfail("env.PYPY") -def test_array_create_and_resize(msg): +def test_array_create_and_resize(): a = m.create_and_resize(2) assert a.size == 4 assert np.all(a == 42.0) +def test_reshape_initializer_list(): + a = np.arange(2 * 7 * 3) + 1 + x = m.reshape_initializer_list(a, 2, 7, 3) + assert x.shape == (2, 7, 3) + assert list(x[1][4]) == [34, 35, 36] + with pytest.raises(ValueError) as excinfo: + m.reshape_initializer_list(a, 1, 7, 3) + assert str(excinfo.value) == "cannot reshape array of size 42 into shape (1,7,3)" + + +def test_reshape_tuple(): + a = np.arange(3 * 7 * 2) + 1 + x = m.reshape_tuple(a, (3, 7, 2)) + assert x.shape == (3, 7, 2) + assert list(x[1][4]) == [23, 24] + y = m.reshape_tuple(x, (x.size,)) + assert y.shape == (42,) + with pytest.raises(ValueError) as excinfo: + m.reshape_tuple(a, (3, 7, 1)) + assert str(excinfo.value) == "cannot reshape array of size 42 into shape (3,7,1)" + with pytest.raises(ValueError) as excinfo: + m.reshape_tuple(a, ()) + assert str(excinfo.value) == "cannot reshape array of size 42 into shape ()" + + def test_index_using_ellipsis(): a = m.index_using_ellipsis(np.zeros((5, 6, 7))) assert a.shape == (6,) From db44afa33b21af09f81c70ee6449c4a4d93fbeca Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 26 Aug 2021 10:52:13 -0700 Subject: [PATCH 049/100] tests: fix pytest usage on Python 3.10 (#3221) --- tests/requirements.txt | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/requirements.txt b/tests/requirements.txt index 00cb5f11..069122b8 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -2,11 +2,11 @@ numpy==1.16.6; python_version<"3.6" and sys_platform!="win32" numpy==1.18.0; platform_python_implementation=="PyPy" and sys_platform=="darwin" and python_version>="3.6" numpy==1.19.3; (platform_python_implementation!="PyPy" or sys_platform=="linux") and python_version=="3.6" -numpy==1.20.0; (platform_python_implementation!="PyPy" or sys_platform=="linux") and python_version>="3.7" and python_version<"3.10" +numpy==1.21.2; (platform_python_implementation!="PyPy" or sys_platform=="linux") and python_version>="3.7" and python_version<"3.10" +numpy==1.21.2; platform_python_implementation!="PyPy" and sys_platform=="linux" and python_version=="3.10" pytest==4.6.9; python_version<"3.5" pytest==6.1.2; python_version=="3.5" -pytest==6.2.1; python_version>="3.6" and python_version<="3.9" -pytest @ git+https://github.com/pytest-dev/pytest@c117bc350ec1e570672fda3b2ad234fd52e72b53; python_version>="3.10" +pytest==6.2.4; python_version>="3.6" pytest-timeout scipy==1.2.3; (platform_python_implementation!="PyPy" or sys_platform=="linux") and python_version<"3.6" scipy==1.5.4; (platform_python_implementation!="PyPy" or sys_platform=="linux") and python_version>="3.6" and python_version<"3.10" From 503ff2a6fbb498138c0b7e85419de491c9860a93 Mon Sep 17 00:00:00 2001 From: Nick Cullen Date: Thu, 26 Aug 2021 23:11:01 +0200 Subject: [PATCH 050/100] view for numpy arrays (#987) * reshape * more tests * Update numpy.h * Update test_numpy_array.py * array view * test * Update test_numpy_array.cpp * Update numpy.h * Update numpy.h * Update test_numpy_array.cpp * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix merge bug * Make clang-tidy happy * Add xfail for PyPy * Fix casting issue * Fix formatting * Apply clang-tidy * Address reviews on additional tests * Fix ordering * Do a little more reordering * Fix typo * Try improving tests * Fix error in reshape * Add one more reshape test * Fix bugs and add test * Relax test * streamlining new tests; removing a few stray msg * Fix style revert * Fix clang-tidy * Misc tweaks: * Comment: matching style in file (///), responsibility sentence, consistent punctuation. * Replacing `unsigned char` with `uint8_t` for max consistency. * Removing `1` from `array_view1` because there is only one. * Partial clang-format-diff. Co-authored-by: ncullen93 Co-authored-by: NC Cullen Co-authored-by: Aaron Gokaslan Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ralf Grosse-Kunstleve --- include/pybind11/numpy.h | 18 ++++++++++++++++++ tests/test_numpy_array.cpp | 3 +++ tests/test_numpy_array.py | 15 +++++++++++++++ 3 files changed, 36 insertions(+) diff --git a/include/pybind11/numpy.h b/include/pybind11/numpy.h index 0d0cbdfa..fa128efd 100644 --- a/include/pybind11/numpy.h +++ b/include/pybind11/numpy.h @@ -199,6 +199,7 @@ struct npy_api { int (*PyArray_SetBaseObject_)(PyObject *, PyObject *); PyObject* (*PyArray_Resize_)(PyObject*, PyArray_Dims*, int, int); PyObject* (*PyArray_Newshape_)(PyObject*, PyArray_Dims*, int); + PyObject* (*PyArray_View_)(PyObject*, PyObject*, PyObject*); private: enum functions { @@ -216,6 +217,7 @@ private: API_PyArray_DescrNewFromType = 96, API_PyArray_Newshape = 135, API_PyArray_Squeeze = 136, + API_PyArray_View = 137, API_PyArray_DescrConverter = 174, API_PyArray_EquivTypes = 182, API_PyArray_GetArrayParamsFromObject = 278, @@ -248,6 +250,7 @@ private: DECL_NPY_API(PyArray_DescrNewFromType); DECL_NPY_API(PyArray_Newshape); DECL_NPY_API(PyArray_Squeeze); + DECL_NPY_API(PyArray_View); DECL_NPY_API(PyArray_DescrConverter); DECL_NPY_API(PyArray_EquivTypes); DECL_NPY_API(PyArray_GetArrayParamsFromObject); @@ -802,6 +805,21 @@ public: return new_array; } + /// Create a view of an array in a different data type. + /// This function may fundamentally reinterpret the data in the array. + /// It is the responsibility of the caller to ensure that this is safe. + /// Only supports the `dtype` argument, the `type` argument is omitted, + /// to be added as needed. + array view(const std::string &dtype) { + auto &api = detail::npy_api::get(); + auto new_view = reinterpret_steal(api.PyArray_View_( + m_ptr, dtype::from_args(pybind11::str(dtype)).release().ptr(), nullptr)); + if (!new_view) { + throw error_already_set(); + } + return new_view; + } + /// Ensure that the argument is a NumPy array /// In case of an error, nullptr is returned and the Python error is cleared. static array ensure(handle h, int ExtraFlags = 0) { diff --git a/tests/test_numpy_array.cpp b/tests/test_numpy_array.cpp index 4ccfd279..30a71acc 100644 --- a/tests/test_numpy_array.cpp +++ b/tests/test_numpy_array.cpp @@ -405,6 +405,9 @@ TEST_SUBMODULE(numpy_array, sm) { return a; }); + sm.def("array_view", + [](py::array_t a, const std::string &dtype) { return a.view(dtype); }); + sm.def("reshape_initializer_list", [](py::array_t a, size_t N, size_t M, size_t O) { return a.reshape({N, M, O}); }); diff --git a/tests/test_numpy_array.py b/tests/test_numpy_array.py index e96454be..e4138f02 100644 --- a/tests/test_numpy_array.py +++ b/tests/test_numpy_array.py @@ -476,6 +476,21 @@ def test_array_create_and_resize(): assert np.all(a == 42.0) +def test_array_view(): + a = np.ones(100 * 4).astype("uint8") + a_float_view = m.array_view(a, "float32") + assert a_float_view.shape == (100 * 1,) # 1 / 4 bytes = 8 / 32 + + a_int16_view = m.array_view(a, "int16") # 1 / 2 bytes = 16 / 32 + assert a_int16_view.shape == (100 * 2,) + + +def test_array_view_invalid(): + a = np.ones(100 * 4).astype("uint8") + with pytest.raises(TypeError): + m.array_view(a, "deadly_dtype") + + def test_reshape_initializer_list(): a = np.arange(2 * 7 * 3) + 1 x = m.reshape_initializer_list(a, 2, 7, 3) From 930bb16c797af642ed4d216cd8972bbd3276dceb Mon Sep 17 00:00:00 2001 From: Dan Date: Thu, 26 Aug 2021 17:12:54 -0400 Subject: [PATCH 051/100] Call PySys_SetArgv when initializing interpreter. (#2341) * Call PySys_SetArgv when initializing interpreter. * Document argc/argv parameters in initialize_interpreter. * Remove manual memory management from set_interpreter_argv in favor of smart pointers. * Use size_t for indexers in set_interpreter_argv. * Minimize macros for flow control in set_interpreter_argv. * Fix 'unused variable' warning on Py2 * whitespace * Define wide_char_arg_deleter outside set_interpreter_argv. * Do sys.path workaround in C++ rather than eval. * Factor out wchar conversion to a separate function. * Restore widened_argv variable declaration. * Fix undeclared widened_arg variable on some paths. * Use delete[] to match new wchar_t[]. * Fix compiler errors * Use PY_VERSION_HEX for a cleaner CVE-2008-5983 mode check. * Fix typo * Use explicit type for deleter so delete[] works cross-compiler. * Always use PySys_SetArgvEx because pybind11 doesn't support pythons that don't include it. * Remove pointless ternary operator. * Use unique_ptr.reset instead of a second initialization. * Rename add_program_dir_to_path parameter to clarify intent. * Add defined() check before evaluating HAVE_BROKEN_MBSTOWCS. * Apply clang-tidy fixes * Pre-commit * refactor: use const for set_interpreter_argv * Try to fix const issue and allocate vector properly * fix: copy strings on Python 2 * Applying clang-format-diff relative to master. The only manual change is an added empty line between pybind11 and system `#include`s. ``` git diff -U0 --no-color master | python3 $HOME/clone/llvm-project/clang/tools/clang-format/clang-format-diff.py -p1 -style=file -i ``` Co-authored-by: Boris Staletic Co-authored-by: Aaron Gokaslan Co-authored-by: Henry Schreiner Co-authored-by: Ralf W. Grosse-Kunstleve --- include/pybind11/embed.h | 101 ++++++++++++++++++++++++-- tests/test_embed/test_interpreter.cpp | 24 ++++++ tests/test_embed/test_interpreter.py | 5 ++ 3 files changed, 122 insertions(+), 8 deletions(-) diff --git a/include/pybind11/embed.h b/include/pybind11/embed.h index 6e777830..7b5d7cd2 100644 --- a/include/pybind11/embed.h +++ b/include/pybind11/embed.h @@ -12,6 +12,9 @@ #include "pybind11.h" #include "eval.h" +#include +#include + #if defined(PYPY_VERSION) # error Embedding the interpreter is not supported with PyPy #endif @@ -83,29 +86,106 @@ struct embedded_module { } }; +struct wide_char_arg_deleter { + void operator()(wchar_t *ptr) const { +#if PY_VERSION_HEX >= 0x030500f0 + // API docs: https://docs.python.org/3/c-api/sys.html#c.Py_DecodeLocale + PyMem_RawFree(ptr); +#else + delete[] ptr; +#endif + } +}; + +inline wchar_t *widen_chars(const char *safe_arg) { +#if PY_VERSION_HEX >= 0x030500f0 + wchar_t *widened_arg = Py_DecodeLocale(safe_arg, nullptr); +#else + wchar_t *widened_arg = nullptr; +# if defined(HAVE_BROKEN_MBSTOWCS) && HAVE_BROKEN_MBSTOWCS + size_t count = strlen(safe_arg); +# else + size_t count = mbstowcs(nullptr, safe_arg, 0); +# endif + if (count != static_cast(-1)) { + widened_arg = new wchar_t[count + 1]; + mbstowcs(widened_arg, safe_arg, count + 1); + } +#endif + return widened_arg; +} + +/// Python 2.x/3.x-compatible version of `PySys_SetArgv` +inline void set_interpreter_argv(int argc, const char *const *argv, bool add_program_dir_to_path) { + // Before it was special-cased in python 3.8, passing an empty or null argv + // caused a segfault, so we have to reimplement the special case ourselves. + bool special_case = (argv == nullptr || argc <= 0); + + const char *const empty_argv[]{"\0"}; + const char *const *safe_argv = special_case ? empty_argv : argv; + if (special_case) + argc = 1; + + auto argv_size = static_cast(argc); +#if PY_MAJOR_VERSION >= 3 + // SetArgv* on python 3 takes wchar_t, so we have to convert. + std::unique_ptr widened_argv(new wchar_t *[argv_size]); + std::vector> widened_argv_entries; + widened_argv_entries.reserve(argv_size); + for (size_t ii = 0; ii < argv_size; ++ii) { + widened_argv_entries.emplace_back(widen_chars(safe_argv[ii])); + if (!widened_argv_entries.back()) { + // A null here indicates a character-encoding failure or the python + // interpreter out of memory. Give up. + return; + } + widened_argv[ii] = widened_argv_entries.back().get(); + } + + auto pysys_argv = widened_argv.get(); +#else + // python 2.x + std::vector strings{safe_argv, safe_argv + argv_size}; + std::vector char_strings{argv_size}; + for (std::size_t i = 0; i < argv_size; ++i) + char_strings[i] = &strings[i][0]; + char **pysys_argv = char_strings.data(); +#endif + + PySys_SetArgvEx(argc, pysys_argv, static_cast(add_program_dir_to_path)); +} + PYBIND11_NAMESPACE_END(detail) /** \rst Initialize the Python interpreter. No other pybind11 or CPython API functions can be called before this is done; with the exception of `PYBIND11_EMBEDDED_MODULE`. The - optional parameter can be used to skip the registration of signal handlers (see the - `Python documentation`_ for details). Calling this function again after the interpreter - has already been initialized is a fatal error. + optional `init_signal_handlers` parameter can be used to skip the registration of + signal handlers (see the `Python documentation`_ for details). Calling this function + again after the interpreter has already been initialized is a fatal error. If initializing the Python interpreter fails, then the program is terminated. (This is controlled by the CPython runtime and is an exception to pybind11's normal behavior of throwing exceptions on errors.) + The remaining optional parameters, `argc`, `argv`, and `add_program_dir_to_path` are + used to populate ``sys.argv`` and ``sys.path``. + See the |PySys_SetArgvEx documentation|_ for details. + .. _Python documentation: https://docs.python.org/3/c-api/init.html#c.Py_InitializeEx + .. |PySys_SetArgvEx documentation| replace:: ``PySys_SetArgvEx`` documentation + .. _PySys_SetArgvEx documentation: https://docs.python.org/3/c-api/init.html#c.PySys_SetArgvEx \endrst */ -inline void initialize_interpreter(bool init_signal_handlers = true) { +inline void initialize_interpreter(bool init_signal_handlers = true, + int argc = 0, + const char *const *argv = nullptr, + bool add_program_dir_to_path = true) { if (Py_IsInitialized() != 0) pybind11_fail("The interpreter is already running"); Py_InitializeEx(init_signal_handlers ? 1 : 0); - // Make .py files in the working directory available by default - module_::import("sys").attr("path").cast().append("."); + detail::set_interpreter_argv(argc, argv, add_program_dir_to_path); } /** \rst @@ -167,6 +247,8 @@ inline void finalize_interpreter() { Scope guard version of `initialize_interpreter` and `finalize_interpreter`. This a move-only guard and only a single instance can exist. + See `initialize_interpreter` for a discussion of its constructor arguments. + .. code-block:: cpp #include @@ -178,8 +260,11 @@ inline void finalize_interpreter() { \endrst */ class scoped_interpreter { public: - scoped_interpreter(bool init_signal_handlers = true) { - initialize_interpreter(init_signal_handlers); + scoped_interpreter(bool init_signal_handlers = true, + int argc = 0, + const char *const *argv = nullptr, + bool add_program_dir_to_path = true) { + initialize_interpreter(init_signal_handlers, argc, argv, add_program_dir_to_path); } scoped_interpreter(const scoped_interpreter &) = delete; diff --git a/tests/test_embed/test_interpreter.cpp b/tests/test_embed/test_interpreter.cpp index b40ff481..78b64be6 100644 --- a/tests/test_embed/test_interpreter.cpp +++ b/tests/test_embed/test_interpreter.cpp @@ -23,6 +23,7 @@ public: std::string the_message() const { return message; } virtual int the_answer() const = 0; + virtual std::string argv0() const = 0; private: std::string message; @@ -32,6 +33,7 @@ class PyWidget final : public Widget { using Widget::Widget; int the_answer() const override { PYBIND11_OVERRIDE_PURE(int, Widget, the_answer); } + std::string argv0() const override { PYBIND11_OVERRIDE_PURE(std::string, Widget, argv0); } }; PYBIND11_EMBEDDED_MODULE(widget_module, m) { @@ -299,3 +301,25 @@ TEST_CASE("Reload module from file") { result = module_.attr("test")().cast(); REQUIRE(result == 2); } + +TEST_CASE("sys.argv gets initialized properly") { + py::finalize_interpreter(); + { + py::scoped_interpreter default_scope; + auto module = py::module::import("test_interpreter"); + auto py_widget = module.attr("DerivedWidget")("The question"); + const auto &cpp_widget = py_widget.cast(); + REQUIRE(cpp_widget.argv0().empty()); + } + + { + char *argv[] = {strdup("a.out")}; + py::scoped_interpreter argv_scope(true, 1, argv); + free(argv[0]); + auto module = py::module::import("test_interpreter"); + auto py_widget = module.attr("DerivedWidget")("The question"); + const auto &cpp_widget = py_widget.cast(); + REQUIRE(cpp_widget.argv0() == "a.out"); + } + py::initialize_interpreter(); +} diff --git a/tests/test_embed/test_interpreter.py b/tests/test_embed/test_interpreter.py index 6174ede4..5ab55a4b 100644 --- a/tests/test_embed/test_interpreter.py +++ b/tests/test_embed/test_interpreter.py @@ -1,4 +1,6 @@ # -*- coding: utf-8 -*- +import sys + from widget_module import Widget @@ -8,3 +10,6 @@ class DerivedWidget(Widget): def the_answer(self): return 42 + + def argv0(self): + return sys.argv[0] From cb60ed49e46fb31cc578b86b2178d8ce574617ad Mon Sep 17 00:00:00 2001 From: Ye Zhihao Date: Fri, 27 Aug 2021 05:34:24 +0800 Subject: [PATCH 052/100] Fix enum value's __int__ returning non-int when underlying type is bool or of char type (#1334) * Use equivalent_integer for enum's Scalar decision * Add test for char underlying enum * Support translating bool type in enum's Scalar * Add test for bool underlying enum * Fix comment in test * Switch from `PYBIND11_CPP20` macro to `PYBIND11_HAS_U8STRING` * Refine tests Co-authored-by: Aaron Gokaslan --- include/pybind11/pybind11.h | 21 +++++++++++-- tests/test_enum.cpp | 61 +++++++++++++++++++++++++++++++++++++ tests/test_enum.py | 28 +++++++++++++++++ 3 files changed, 108 insertions(+), 2 deletions(-) diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 47b04214..89f9cbd9 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -1814,6 +1814,19 @@ struct enum_base { handle m_parent; }; +template struct equivalent_integer {}; +template <> struct equivalent_integer { using type = int8_t; }; +template <> struct equivalent_integer { using type = uint8_t; }; +template <> struct equivalent_integer { using type = int16_t; }; +template <> struct equivalent_integer { using type = uint16_t; }; +template <> struct equivalent_integer { using type = int32_t; }; +template <> struct equivalent_integer { using type = uint32_t; }; +template <> struct equivalent_integer { using type = int64_t; }; +template <> struct equivalent_integer { using type = uint64_t; }; + +template +using equivalent_integer_t = typename equivalent_integer::value, sizeof(IntLike)>::type; + PYBIND11_NAMESPACE_END(detail) /// Binds C++ enumerations and enumeration classes to Python @@ -1824,13 +1837,17 @@ public: using Base::attr; using Base::def_property_readonly; using Base::def_property_readonly_static; - using Scalar = typename std::underlying_type::type; + using Underlying = typename std::underlying_type::type; + // Scalar is the integer representation of underlying type + using Scalar = detail::conditional_t, std::is_same + >::value, detail::equivalent_integer_t, Underlying>; template enum_(const handle &scope, const char *name, const Extra&... extra) : class_(scope, name, extra...), m_base(*this, scope) { constexpr bool is_arithmetic = detail::any_of...>::value; - constexpr bool is_convertible = std::is_convertible::value; + constexpr bool is_convertible = std::is_convertible::value; m_base.init(is_arithmetic, is_convertible); def(init([](Scalar i) { return static_cast(i); }), arg("value")); diff --git a/tests/test_enum.cpp b/tests/test_enum.cpp index 31530892..40c48d41 100644 --- a/tests/test_enum.cpp +++ b/tests/test_enum.cpp @@ -84,4 +84,65 @@ TEST_SUBMODULE(enums, m) { .value("ONE", SimpleEnum::THREE) .export_values(); }); + + // test_enum_scalar + enum UnscopedUCharEnum : unsigned char {}; + enum class ScopedShortEnum : short {}; + enum class ScopedLongEnum : long {}; + enum UnscopedUInt64Enum : std::uint64_t {}; + static_assert(py::detail::all_of< + std::is_same::Scalar, unsigned char>, + std::is_same::Scalar, short>, + std::is_same::Scalar, long>, + std::is_same::Scalar, std::uint64_t> + >::value, "Error during the deduction of enum's scalar type with normal integer underlying"); + + // test_enum_scalar_with_char_underlying + enum class ScopedCharEnum : char { Zero, Positive }; + enum class ScopedWCharEnum : wchar_t { Zero, Positive }; + enum class ScopedChar32Enum : char32_t { Zero, Positive }; + enum class ScopedChar16Enum : char16_t { Zero, Positive }; + + // test the scalar of char type enums according to chapter 'Character types' + // from https://en.cppreference.com/w/cpp/language/types + static_assert(py::detail::any_of< + std::is_same::Scalar, signed char>, // e.g. gcc on x86 + std::is_same::Scalar, unsigned char> // e.g. arm linux + >::value, "char should be cast to either signed char or unsigned char"); + static_assert( + sizeof(py::enum_::Scalar) == 2 || + sizeof(py::enum_::Scalar) == 4 + , "wchar_t should be either 16 bits (Windows) or 32 (everywhere else)"); + static_assert(py::detail::all_of< + std::is_same::Scalar, std::uint_least32_t>, + std::is_same::Scalar, std::uint_least16_t> + >::value, "char32_t, char16_t (and char8_t)'s size, signedness, and alignment is determined"); +#if defined(PYBIND11_HAS_U8STRING) + enum class ScopedChar8Enum : char8_t { Zero, Positive }; + static_assert(std::is_same::Scalar, unsigned char>::value); +#endif + + // test_char_underlying_enum + py::enum_(m, "ScopedCharEnum") + .value("Zero", ScopedCharEnum::Zero) + .value("Positive", ScopedCharEnum::Positive); + py::enum_(m, "ScopedWCharEnum") + .value("Zero", ScopedWCharEnum::Zero) + .value("Positive", ScopedWCharEnum::Positive); + py::enum_(m, "ScopedChar32Enum") + .value("Zero", ScopedChar32Enum::Zero) + .value("Positive", ScopedChar32Enum::Positive); + py::enum_(m, "ScopedChar16Enum") + .value("Zero", ScopedChar16Enum::Zero) + .value("Positive", ScopedChar16Enum::Positive); + + // test_bool_underlying_enum + enum class ScopedBoolEnum : bool { FALSE, TRUE }; + + // bool is unsigned (std::is_signed returns false) and 1-byte long, so represented with u8 + static_assert(std::is_same::Scalar, std::uint8_t>::value, ""); + + py::enum_(m, "ScopedBoolEnum") + .value("FALSE", ScopedBoolEnum::FALSE) + .value("TRUE", ScopedBoolEnum::TRUE); } diff --git a/tests/test_enum.py b/tests/test_enum.py index 62f9426e..11cab6dd 100644 --- a/tests/test_enum.py +++ b/tests/test_enum.py @@ -218,10 +218,16 @@ def test_binary_operators(): def test_enum_to_int(): m.test_enum_to_int(m.Flags.Read) m.test_enum_to_int(m.ClassWithUnscopedEnum.EMode.EFirstMode) + m.test_enum_to_int(m.ScopedCharEnum.Positive) + m.test_enum_to_int(m.ScopedBoolEnum.TRUE) m.test_enum_to_uint(m.Flags.Read) m.test_enum_to_uint(m.ClassWithUnscopedEnum.EMode.EFirstMode) + m.test_enum_to_uint(m.ScopedCharEnum.Positive) + m.test_enum_to_uint(m.ScopedBoolEnum.TRUE) m.test_enum_to_long_long(m.Flags.Read) m.test_enum_to_long_long(m.ClassWithUnscopedEnum.EMode.EFirstMode) + m.test_enum_to_long_long(m.ScopedCharEnum.Positive) + m.test_enum_to_long_long(m.ScopedBoolEnum.TRUE) def test_duplicate_enum_name(): @@ -230,6 +236,28 @@ def test_duplicate_enum_name(): assert str(excinfo.value) == 'SimpleEnum: element "ONE" already exists!' +def test_char_underlying_enum(): # Issue #1331/PR #1334: + assert type(m.ScopedCharEnum.Positive.__int__()) is int + assert int(m.ScopedChar16Enum.Zero) == 0 # int() call should successfully return + assert hash(m.ScopedChar32Enum.Positive) == 1 + assert m.ScopedCharEnum.Positive.__getstate__() == 1 # return type is long in py2.x + assert m.ScopedWCharEnum(1) == m.ScopedWCharEnum.Positive + with pytest.raises(TypeError): + # Enum should construct with a int, even with char underlying type + m.ScopedWCharEnum("0") + + +def test_bool_underlying_enum(): + assert type(m.ScopedBoolEnum.TRUE.__int__()) is int + assert int(m.ScopedBoolEnum.FALSE) == 0 + assert hash(m.ScopedBoolEnum.TRUE) == 1 + assert m.ScopedBoolEnum.TRUE.__getstate__() == 1 + assert m.ScopedBoolEnum(1) == m.ScopedBoolEnum.TRUE + # Enum could construct with a bool + # (bool is a strict subclass of int, and False will be converted to 0) + assert m.ScopedBoolEnum(False) == m.ScopedBoolEnum.FALSE + + def test_docstring_signatures(): for enum_type in [m.ScopedEnum, m.UnscopedEnum]: for attr in enum_type.__dict__.values(): From 777352fcd10bd9b57a8f626a6c7397596466ee55 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Sat, 28 Aug 2021 16:40:46 -0700 Subject: [PATCH 053/100] Adding `ssize_t_cast` to support passing `size_t` or `ssize_t` values where `ssize_t` is needed. (#3219) * Trivial change to avoid (ssize_t) cast. * Demo for safe_ssize_t idea. * Removing safe_ssize_t.cpp (proof-of-concept code) to not upset the GHA Format workflow. * Completing changes in pytypes.h * New ssize_t_cast (better replacement for safe_ssize_t). * clang-format-diff (no manual changes). * bytes_ssize_t -Wnarrowing reproducer (see PR #2692). * Backing out tuple(), list() ssize_t support, for compatibility with older compilers (to resolve link failures). * Bug fix: missing `py::` for `py::ssize_t` * Restoring tuple(), list() ssize_t support, but passing `size` by value, for compatibility with older compilers (to resolve link failures). * Full test coverage of all functions with modified signatures. --- include/pybind11/detail/common.h | 6 +++ include/pybind11/pytypes.h | 66 +++++++++++++++++++------------ include/pybind11/stl.h | 8 ++-- tests/test_pytypes.cpp | 67 ++++++++++++++++++++++++++++++++ tests/test_pytypes.py | 39 +++++++++++++++++++ 5 files changed, 158 insertions(+), 28 deletions(-) diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 5050da80..e4f00293 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -410,6 +410,12 @@ PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) using ssize_t = Py_ssize_t; using size_t = std::size_t; +template +inline ssize_t ssize_t_cast(const IntType &val) { + static_assert(sizeof(IntType) <= sizeof(ssize_t), "Implicit narrowing is not permitted."); + return static_cast(val); +} + /// Approach used to cast a previously unknown C++ instance into a Python object enum class return_value_policy : uint8_t { /** This is the default return value policy, which falls back to the policy diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index 85f6a40a..5e699e53 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -661,15 +661,17 @@ struct generic_item { struct sequence_item { using key_type = size_t; - static object get(handle obj, size_t index) { - PyObject *result = PySequence_GetItem(obj.ptr(), static_cast(index)); + template ::value, int> = 0> + static object get(handle obj, const IdxType &index) { + PyObject *result = PySequence_GetItem(obj.ptr(), ssize_t_cast(index)); if (!result) { throw error_already_set(); } return reinterpret_steal(result); } - static void set(handle obj, size_t index, handle val) { + template ::value, int> = 0> + static void set(handle obj, const IdxType &index, handle val) { // PySequence_SetItem does not steal a reference to 'val' - if (PySequence_SetItem(obj.ptr(), static_cast(index), val.ptr()) != 0) { + if (PySequence_SetItem(obj.ptr(), ssize_t_cast(index), val.ptr()) != 0) { throw error_already_set(); } } @@ -678,15 +680,17 @@ struct sequence_item { struct list_item { using key_type = size_t; - static object get(handle obj, size_t index) { - PyObject *result = PyList_GetItem(obj.ptr(), static_cast(index)); + template ::value, int> = 0> + static object get(handle obj, const IdxType &index) { + PyObject *result = PyList_GetItem(obj.ptr(), ssize_t_cast(index)); if (!result) { throw error_already_set(); } return reinterpret_borrow(result); } - static void set(handle obj, size_t index, handle val) { + template ::value, int> = 0> + static void set(handle obj, const IdxType &index, handle val) { // PyList_SetItem steals a reference to 'val' - if (PyList_SetItem(obj.ptr(), static_cast(index), val.inc_ref().ptr()) != 0) { + if (PyList_SetItem(obj.ptr(), ssize_t_cast(index), val.inc_ref().ptr()) != 0) { throw error_already_set(); } } @@ -695,15 +699,17 @@ struct list_item { struct tuple_item { using key_type = size_t; - static object get(handle obj, size_t index) { - PyObject *result = PyTuple_GetItem(obj.ptr(), static_cast(index)); + template ::value, int> = 0> + static object get(handle obj, const IdxType &index) { + PyObject *result = PyTuple_GetItem(obj.ptr(), ssize_t_cast(index)); if (!result) { throw error_already_set(); } return reinterpret_borrow(result); } - static void set(handle obj, size_t index, handle val) { + template ::value, int> = 0> + static void set(handle obj, const IdxType &index, handle val) { // PyTuple_SetItem steals a reference to 'val' - if (PyTuple_SetItem(obj.ptr(), static_cast(index), val.inc_ref().ptr()) != 0) { + if (PyTuple_SetItem(obj.ptr(), ssize_t_cast(index), val.inc_ref().ptr()) != 0) { throw error_already_set(); } } @@ -1043,8 +1049,9 @@ class str : public object { public: PYBIND11_OBJECT_CVT(str, object, PYBIND11_STR_CHECK_FUN, raw_str) - str(const char *c, size_t n) - : object(PyUnicode_FromStringAndSize(c, (ssize_t) n), stolen_t{}) { + template ::value, int> = 0> + str(const char *c, const SzType &n) + : object(PyUnicode_FromStringAndSize(c, ssize_t_cast(n)), stolen_t{}) { if (!m_ptr) pybind11_fail("Could not allocate string object!"); } @@ -1116,8 +1123,9 @@ public: if (!m_ptr) pybind11_fail("Could not allocate bytes object!"); } - bytes(const char *c, size_t n) - : object(PYBIND11_BYTES_FROM_STRING_AND_SIZE(c, (ssize_t) n), stolen_t{}) { + template ::value, int> = 0> + bytes(const char *c, const SzType &n) + : object(PYBIND11_BYTES_FROM_STRING_AND_SIZE(c, ssize_t_cast(n)), stolen_t{}) { if (!m_ptr) pybind11_fail("Could not allocate bytes object!"); } @@ -1160,7 +1168,7 @@ inline str::str(const bytes& b) { ssize_t length = 0; if (PYBIND11_BYTES_AS_STRING_AND_SIZE(b.ptr(), &buffer, &length)) pybind11_fail("Unable to extract bytes contents!"); - auto obj = reinterpret_steal(PyUnicode_FromStringAndSize(buffer, (ssize_t) length)); + auto obj = reinterpret_steal(PyUnicode_FromStringAndSize(buffer, length)); if (!obj) pybind11_fail("Could not allocate string object!"); m_ptr = obj.release().ptr(); @@ -1172,8 +1180,9 @@ class bytearray : public object { public: PYBIND11_OBJECT_CVT(bytearray, object, PyByteArray_Check, PyByteArray_FromObject) - bytearray(const char *c, size_t n) - : object(PyByteArray_FromStringAndSize(c, (ssize_t) n), stolen_t{}) { + template ::value, int> = 0> + bytearray(const char *c, const SzType &n) + : object(PyByteArray_FromStringAndSize(c, ssize_t_cast(n)), stolen_t{}) { if (!m_ptr) pybind11_fail("Could not allocate bytearray object!"); } @@ -1398,7 +1407,10 @@ public: class tuple : public object { public: PYBIND11_OBJECT_CVT(tuple, object, PyTuple_Check, PySequence_Tuple) - explicit tuple(size_t size = 0) : object(PyTuple_New((ssize_t) size), stolen_t{}) { + template ::value, int> = 0> + // Some compilers generate link errors when using `const SzType &` here: + explicit tuple(SzType size = 0) : object(PyTuple_New(ssize_t_cast(size)), stolen_t{}) { if (!m_ptr) pybind11_fail("Could not allocate tuple object!"); } size_t size() const { return (size_t) PyTuple_Size(m_ptr); } @@ -1467,7 +1479,10 @@ public: class list : public object { public: PYBIND11_OBJECT_CVT(list, object, PyList_Check, PySequence_List) - explicit list(size_t size = 0) : object(PyList_New((ssize_t) size), stolen_t{}) { + template ::value, int> = 0> + // Some compilers generate link errors when using `const SzType &` here: + explicit list(SzType size = 0) : object(PyList_New(ssize_t_cast(size)), stolen_t{}) { if (!m_ptr) pybind11_fail("Could not allocate list object!"); } size_t size() const { return (size_t) PyList_Size(m_ptr); } @@ -1479,9 +1494,12 @@ public: template void append(T &&val) /* py-non-const */ { PyList_Append(m_ptr, detail::object_or_cast(std::forward(val)).ptr()); } - template void insert(size_t index, T &&val) /* py-non-const */ { - PyList_Insert(m_ptr, static_cast(index), - detail::object_or_cast(std::forward(val)).ptr()); + template ::value, int> = 0> + void insert(const IdxType &index, ValType &&val) /* py-non-const */ { + PyList_Insert( + m_ptr, ssize_t_cast(index), detail::object_or_cast(std::forward(val)).ptr()); } }; diff --git a/include/pybind11/stl.h b/include/pybind11/stl.h index fe391f70..99b49d0e 100644 --- a/include/pybind11/stl.h +++ b/include/pybind11/stl.h @@ -168,12 +168,12 @@ public: if (!std::is_lvalue_reference::value) policy = return_value_policy_override::policy(policy); list l(src.size()); - size_t index = 0; + ssize_t index = 0; for (auto &&value : src) { auto value_ = reinterpret_steal(value_conv::cast(forward_like(value), policy, parent)); if (!value_) return handle(); - PyList_SET_ITEM(l.ptr(), (ssize_t) index++, value_.release().ptr()); // steals a reference + PyList_SET_ITEM(l.ptr(), index++, value_.release().ptr()); // steals a reference } return l.release(); } @@ -225,12 +225,12 @@ public: template static handle cast(T &&src, return_value_policy policy, handle parent) { list l(src.size()); - size_t index = 0; + ssize_t index = 0; for (auto &&value : src) { auto value_ = reinterpret_steal(value_conv::cast(forward_like(value), policy, parent)); if (!value_) return handle(); - PyList_SET_ITEM(l.ptr(), (ssize_t) index++, value_.release().ptr()); // steals a reference + PyList_SET_ITEM(l.ptr(), index++, value_.release().ptr()); // steals a reference } return l.release(); } diff --git a/tests/test_pytypes.cpp b/tests/test_pytypes.cpp index 15d007a4..96c97351 100644 --- a/tests/test_pytypes.cpp +++ b/tests/test_pytypes.cpp @@ -20,6 +20,11 @@ TEST_SUBMODULE(pytypes, m) { // test_iterable m.def("get_iterable", []{return py::iterable();}); // test_list + m.def("list_no_args", []() { return py::list{}; }); + m.def("list_ssize_t", []() { return py::list{(py::ssize_t) 0}; }); + m.def("list_size_t", []() { return py::list{(py::size_t) 0}; }); + m.def("list_insert_ssize_t", [](py::list *l) { return l->insert((py::ssize_t) 1, 83); }); + m.def("list_insert_size_t", [](py::list *l) { return l->insert((py::size_t) 3, 57); }); m.def("get_list", []() { py::list list; list.append("value"); @@ -71,6 +76,9 @@ TEST_SUBMODULE(pytypes, m) { [](const py::dict &dict, const char *val) { return dict.contains(val); }); // test_tuple + m.def("tuple_no_args", []() { return py::tuple{}; }); + m.def("tuple_ssize_t", []() { return py::tuple{(py::ssize_t) 0}; }); + m.def("tuple_size_t", []() { return py::tuple{(py::size_t) 0}; }); m.def("get_tuple", []() { return py::make_tuple(42, py::none(), "spam"); }); #if PY_VERSION_HEX >= 0x03030000 @@ -84,6 +92,8 @@ TEST_SUBMODULE(pytypes, m) { #endif // test_str + m.def("str_from_char_ssize_t", []() { return py::str{"red", (py::ssize_t) 3}; }); + m.def("str_from_char_size_t", []() { return py::str{"blue", (py::size_t) 4}; }); m.def("str_from_string", []() { return py::str(std::string("baz")); }); m.def("str_from_bytes", []() { return py::str(py::bytes("boo", 3)); }); m.def("str_from_object", [](const py::object& obj) { return py::str(obj); }); @@ -100,10 +110,14 @@ TEST_SUBMODULE(pytypes, m) { }); // test_bytes + m.def("bytes_from_char_ssize_t", []() { return py::bytes{"green", (py::ssize_t) 5}; }); + m.def("bytes_from_char_size_t", []() { return py::bytes{"purple", (py::size_t) 6}; }); m.def("bytes_from_string", []() { return py::bytes(std::string("foo")); }); m.def("bytes_from_str", []() { return py::bytes(py::str("bar", 3)); }); // test bytearray + m.def("bytearray_from_char_ssize_t", []() { return py::bytearray{"$%", (py::ssize_t) 2}; }); + m.def("bytearray_from_char_size_t", []() { return py::bytearray{"@$!", (py::size_t) 3}; }); m.def("bytearray_from_string", []() { return py::bytearray(std::string("foo")); }); m.def("bytearray_size", []() { return py::bytearray("foo").size(); }); @@ -447,4 +461,57 @@ TEST_SUBMODULE(pytypes, m) { m.def("weakref_from_object", [](const py::object &o) { return py::weakref(o); }); m.def("weakref_from_object_and_function", [](py::object o, py::function f) { return py::weakref(std::move(o), std::move(f)); }); + + // Tests below this line are for pybind11 IMPLEMENTATION DETAILS: + + m.def("sequence_item_get_ssize_t", [](const py::object &o) { + return py::detail::accessor_policies::sequence_item::get(o, (py::ssize_t) 1); + }); + m.def("sequence_item_set_ssize_t", [](const py::object &o) { + auto s = py::str{"peppa", 5}; + py::detail::accessor_policies::sequence_item::set(o, (py::ssize_t) 1, s); + }); + m.def("sequence_item_get_size_t", [](const py::object &o) { + return py::detail::accessor_policies::sequence_item::get(o, (py::size_t) 2); + }); + m.def("sequence_item_set_size_t", [](const py::object &o) { + auto s = py::str{"george", 6}; + py::detail::accessor_policies::sequence_item::set(o, (py::size_t) 2, s); + }); + m.def("list_item_get_ssize_t", [](const py::object &o) { + return py::detail::accessor_policies::list_item::get(o, (py::ssize_t) 3); + }); + m.def("list_item_set_ssize_t", [](const py::object &o) { + auto s = py::str{"rebecca", 7}; + py::detail::accessor_policies::list_item::set(o, (py::ssize_t) 3, s); + }); + m.def("list_item_get_size_t", [](const py::object &o) { + return py::detail::accessor_policies::list_item::get(o, (py::size_t) 4); + }); + m.def("list_item_set_size_t", [](const py::object &o) { + auto s = py::str{"richard", 7}; + py::detail::accessor_policies::list_item::set(o, (py::size_t) 4, s); + }); + m.def("tuple_item_get_ssize_t", [](const py::object &o) { + return py::detail::accessor_policies::tuple_item::get(o, (py::ssize_t) 5); + }); + m.def("tuple_item_set_ssize_t", []() { + auto s0 = py::str{"emely", 5}; + auto s1 = py::str{"edmond", 6}; + auto o = py::tuple{2}; + py::detail::accessor_policies::tuple_item::set(o, (py::ssize_t) 0, s0); + py::detail::accessor_policies::tuple_item::set(o, (py::ssize_t) 1, s1); + return o; + }); + m.def("tuple_item_get_size_t", [](const py::object &o) { + return py::detail::accessor_policies::tuple_item::get(o, (py::size_t) 6); + }); + m.def("tuple_item_set_size_t", []() { + auto s0 = py::str{"candy", 5}; + auto s1 = py::str{"cat", 3}; + auto o = py::tuple{2}; + py::detail::accessor_policies::tuple_item::set(o, (py::size_t) 1, s1); + py::detail::accessor_policies::tuple_item::set(o, (py::size_t) 0, s0); + return o; + }); } diff --git a/tests/test_pytypes.py b/tests/test_pytypes.py index f873658a..18847550 100644 --- a/tests/test_pytypes.py +++ b/tests/test_pytypes.py @@ -23,6 +23,15 @@ def test_iterable(doc): def test_list(capture, doc): + assert m.list_no_args() == [] + assert m.list_ssize_t() == [] + assert m.list_size_t() == [] + lins = [1, 2] + m.list_insert_ssize_t(lins) + assert lins == [1, 83, 2] + m.list_insert_size_t(lins) + assert lins == [1, 83, 2, 57] + with capture: lst = m.get_list() assert lst == ["inserted-0", "overwritten", "inserted-2"] @@ -100,6 +109,9 @@ def test_dict(capture, doc): def test_tuple(): + assert m.tuple_no_args() == () + assert m.tuple_ssize_t() == () + assert m.tuple_size_t() == () assert m.get_tuple() == (42, None, "spam") @@ -113,6 +125,8 @@ def test_simple_namespace(): def test_str(doc): + assert m.str_from_char_ssize_t().encode().decode() == "red" + assert m.str_from_char_size_t().encode().decode() == "blue" assert m.str_from_string().encode().decode() == "baz" assert m.str_from_bytes().encode().decode() == "boo" @@ -157,6 +171,8 @@ def test_str(doc): def test_bytes(doc): + assert m.bytes_from_char_ssize_t().decode() == "green" + assert m.bytes_from_char_size_t().decode() == "purple" assert m.bytes_from_string().decode() == "foo" assert m.bytes_from_str().decode() == "bar" @@ -166,6 +182,8 @@ def test_bytes(doc): def test_bytearray(doc): + assert m.bytearray_from_char_ssize_t().decode() == "$%" + assert m.bytearray_from_char_size_t().decode() == "@$!" assert m.bytearray_from_string().decode() == "foo" assert m.bytearray_size() == len("foo") @@ -603,3 +621,24 @@ def test_weakref(create_weakref, create_weakref_with_callback): del obj pytest.gc_collect() assert callback.called + + +def test_implementation_details(): + lst = [39, 43, 92, 49, 22, 29, 93, 98, 26, 57, 8] + tup = tuple(lst) + assert m.sequence_item_get_ssize_t(lst) == 43 + assert m.sequence_item_set_ssize_t(lst) is None + assert lst[1] == "peppa" + assert m.sequence_item_get_size_t(lst) == 92 + assert m.sequence_item_set_size_t(lst) is None + assert lst[2] == "george" + assert m.list_item_get_ssize_t(lst) == 49 + assert m.list_item_set_ssize_t(lst) is None + assert lst[3] == "rebecca" + assert m.list_item_get_size_t(lst) == 22 + assert m.list_item_set_size_t(lst) is None + assert lst[4] == "richard" + assert m.tuple_item_get_ssize_t(tup) == 29 + assert m.tuple_item_set_ssize_t() == ("emely", "edmond") + assert m.tuple_item_get_size_t(tup) == 93 + assert m.tuple_item_set_size_t() == ("candy", "cat") From 76d939de537dc1d95b8c387ea4633c9c3a8448d2 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 30 Aug 2021 16:00:33 -0400 Subject: [PATCH 054/100] [pre-commit.ci] pre-commit autoupdate (#3231) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit updates: - [github.com/asottile/pyupgrade: v2.24.0 → v2.25.0](https://github.com/asottile/pyupgrade/compare/v2.24.0...v2.25.0) - [github.com/psf/black: 21.7b0 → 21.8b0](https://github.com/psf/black/compare/21.7b0...21.8b0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .pre-commit-config.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 96ddcf74..06e82fe4 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -31,7 +31,7 @@ repos: exclude: ^noxfile.py$ - repo: https://github.com/asottile/pyupgrade - rev: v2.24.0 + rev: v2.25.0 hooks: - id: pyupgrade @@ -42,7 +42,7 @@ repos: # Black, the code formatter, natively supports pre-commit - repo: https://github.com/psf/black - rev: 21.7b0 + rev: 21.8b0 hooks: - id: black # By default, this ignores pyi files, though black supports them From d6474ed7d28c03ef79acd2e87ca429072727df86 Mon Sep 17 00:00:00 2001 From: Tailing Yuan Date: Tue, 31 Aug 2021 09:48:33 +0800 Subject: [PATCH 055/100] fix: memory leak in cpp_function (#3228) (#3229) * fix: memory leak in cpp_function (#3228) * add a test case to check objects are deconstructed in cpp_function * update the test case about cpp_function * fix the test case about cpp_function: remove "noexcept" * Actually calling func. CHECK(stat.alive() == 2); Manually verified that the new tests fails without the change in pybind11.h * Moving new test to test_callbacks.cpp,py, with small enhancements. * Removing new test from test_interpreter.cpp (after it was moved to test_callbacks.cpp,py). This restores test_interpreter.cpp to the current state on master. * Using py::detail::silence_unused_warnings(py_func); to make the intent clear. Co-authored-by: Ralf W. Grosse-Kunstleve --- include/pybind11/pybind11.h | 2 +- tests/test_callbacks.cpp | 43 +++++++++++++++++++++++++++++++++++-- tests/test_callbacks.py | 7 +++++- 3 files changed, 48 insertions(+), 4 deletions(-) diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 89f9cbd9..990c2130 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -175,7 +175,7 @@ protected: #endif // UB without std::launder, but without breaking ABI and/or // a significant refactoring it's "impossible" to solve. - if (!std::is_trivially_destructible::value) + if (!std::is_trivially_destructible::value) rec->free_data = [](function_record *r) { auto data = PYBIND11_STD_LAUNDER((capture *) &r->data); (void) data; diff --git a/tests/test_callbacks.cpp b/tests/test_callbacks.cpp index a5077103..58688b6e 100644 --- a/tests/test_callbacks.cpp +++ b/tests/test_callbacks.cpp @@ -81,16 +81,55 @@ TEST_SUBMODULE(callbacks, m) { }; // Export the payload constructor statistics for testing purposes: m.def("payload_cstats", &ConstructorStats::get); - /* Test cleanup of lambda closure */ - m.def("test_cleanup", []() -> std::function { + m.def("test_lambda_closure_cleanup", []() -> std::function { Payload p; + // In this situation, `Func` in the implementation of + // `cpp_function::initialize` is NOT trivially destructible. return [p]() { /* p should be cleaned up when the returned function is garbage collected */ (void) p; }; }); + class CppCallable { + public: + CppCallable() { track_default_created(this); } + ~CppCallable() { track_destroyed(this); } + CppCallable(const CppCallable &) { track_copy_created(this); } + CppCallable(CppCallable &&) noexcept { track_move_created(this); } + void operator()() {} + }; + + m.def("test_cpp_callable_cleanup", []() { + // Related issue: https://github.com/pybind/pybind11/issues/3228 + // Related PR: https://github.com/pybind/pybind11/pull/3229 + py::list alive_counts; + ConstructorStats &stat = ConstructorStats::get(); + alive_counts.append(stat.alive()); + { + CppCallable cpp_callable; + alive_counts.append(stat.alive()); + { + // In this situation, `Func` in the implementation of + // `cpp_function::initialize` IS trivially destructible, + // only `capture` is not. + py::cpp_function py_func(cpp_callable); + py::detail::silence_unused_warnings(py_func); + alive_counts.append(stat.alive()); + } + alive_counts.append(stat.alive()); + { + py::cpp_function py_func(std::move(cpp_callable)); + py::detail::silence_unused_warnings(py_func); + alive_counts.append(stat.alive()); + } + alive_counts.append(stat.alive()); + } + alive_counts.append(stat.alive()); + return alive_counts; + }); + // test_cpp_function_roundtrip /* Test if passing a function pointer from C++ -> Python -> C++ yields the original pointer */ m.def("dummy_function", &dummy_function); diff --git a/tests/test_callbacks.py b/tests/test_callbacks.py index 9dc272a2..edbb1890 100644 --- a/tests/test_callbacks.py +++ b/tests/test_callbacks.py @@ -79,13 +79,18 @@ def test_keyword_args_and_generalized_unpacking(): def test_lambda_closure_cleanup(): - m.test_cleanup() + m.test_lambda_closure_cleanup() cstats = m.payload_cstats() assert cstats.alive() == 0 assert cstats.copy_constructions == 1 assert cstats.move_constructions >= 1 +def test_cpp_callable_cleanup(): + alive_counts = m.test_cpp_callable_cleanup() + assert alive_counts == [0, 1, 2, 1, 2, 1, 0] + + def test_cpp_function_roundtrip(): """Test if passing a function pointer from C++ -> Python -> C++ yields the original pointer""" From a46f623782df6a92a737146890d9c070fc19b99b Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Tue, 31 Aug 2021 06:44:14 -0700 Subject: [PATCH 056/100] Minor tweaks. (#3230) * Minor tweaks. * Restoring tests/pybind11_tests.h version from master, removing just the comment and empty line that was added in PR #3087; those were made obsolete by the pragma cleanup that concluded with PR #3186. * [ci skip] Restoring tests/test_enum.py from master. --- tests/pybind11_tests.h | 3 --- 1 file changed, 3 deletions(-) diff --git a/tests/pybind11_tests.h b/tests/pybind11_tests.h index 12d8a777..8da0a670 100644 --- a/tests/pybind11_tests.h +++ b/tests/pybind11_tests.h @@ -1,9 +1,6 @@ #pragma once -// This must be kept first for MSVC 2015. -// Do not remove the empty line between the #includes. #include - #include #if defined(_MSC_VER) && _MSC_VER < 1910 From 49173e472f9d0f38654ccb20daf203d116b83b39 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Tue, 31 Aug 2021 08:52:04 -0700 Subject: [PATCH 057/100] Minor follow-on to PR #1334 (Fix enum value's __int__ returning non-int when underlying type is bool or of char type) (#3232) * Minor tweaks. * Restoring tests/pybind11_tests.h version from master, removing just the comment and empty line that was added in PR #3087; those were made obsolete by the pragma cleanup that concluded with PR #3186. * More-to-the-point test for Python 3. --- tests/test_enum.py | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/tests/test_enum.py b/tests/test_enum.py index 11cab6dd..85302b08 100644 --- a/tests/test_enum.py +++ b/tests/test_enum.py @@ -1,6 +1,7 @@ # -*- coding: utf-8 -*- import pytest +import env from pybind11_tests import enums as m @@ -238,20 +239,26 @@ def test_duplicate_enum_name(): def test_char_underlying_enum(): # Issue #1331/PR #1334: assert type(m.ScopedCharEnum.Positive.__int__()) is int - assert int(m.ScopedChar16Enum.Zero) == 0 # int() call should successfully return + assert int(m.ScopedChar16Enum.Zero) == 0 assert hash(m.ScopedChar32Enum.Positive) == 1 - assert m.ScopedCharEnum.Positive.__getstate__() == 1 # return type is long in py2.x + if env.PY2: + assert m.ScopedCharEnum.Positive.__getstate__() == 1 # long + else: + assert type(m.ScopedCharEnum.Positive.__getstate__()) is int assert m.ScopedWCharEnum(1) == m.ScopedWCharEnum.Positive with pytest.raises(TypeError): - # Enum should construct with a int, even with char underlying type - m.ScopedWCharEnum("0") + # Even if the underlying type is char, only an int can be used to construct the enum: + m.ScopedCharEnum("0") def test_bool_underlying_enum(): assert type(m.ScopedBoolEnum.TRUE.__int__()) is int assert int(m.ScopedBoolEnum.FALSE) == 0 assert hash(m.ScopedBoolEnum.TRUE) == 1 - assert m.ScopedBoolEnum.TRUE.__getstate__() == 1 + if env.PY2: + assert m.ScopedBoolEnum.TRUE.__getstate__() == 1 # long + else: + assert type(m.ScopedBoolEnum.TRUE.__getstate__()) is int assert m.ScopedBoolEnum(1) == m.ScopedBoolEnum.TRUE # Enum could construct with a bool # (bool is a strict subclass of int, and False will be converted to 0) From d71ba0cb73616c493d35699a8a9283aa64ef0f6b Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Thu, 2 Sep 2021 15:18:42 -0400 Subject: [PATCH 058/100] (perf): Add a missing noexcept to a pytype constructor (#3236) * Add a missing noexcept to pytypes constructor. Adds a few missing noexcept to PyType constructors for perf reasons. * Revert exception ctor --- include/pybind11/pytypes.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index 5e699e53..fa50eb1e 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -761,7 +761,7 @@ template struct arrow_proxy { T value; - arrow_proxy(T &&value) : value(std::move(value)) { } + arrow_proxy(T &&value) noexcept : value(std::move(value)) { } T *operator->() const { return &value; } }; From 39a0aac88ea2eb9d08293f71f636edc16e536a50 Mon Sep 17 00:00:00 2001 From: Thomas Ballinger Date: Wed, 8 Sep 2021 13:00:00 -0500 Subject: [PATCH 059/100] docs fix to avoid nonexistent SmartCompile (#3241) --- docs/compiling.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/compiling.rst b/docs/compiling.rst index bf7acfc8..eaf3270e 100644 --- a/docs/compiling.rst +++ b/docs/compiling.rst @@ -113,7 +113,7 @@ with the following: from pybind11.setup_helpers import ParallelCompile, naive_recompile - SmartCompile("NPY_NUM_BUILD_JOBS", needs_recompile=naive_recompile).install() + ParallelCompile("NPY_NUM_BUILD_JOBS", needs_recompile=naive_recompile).install() If you have a more complex build, you can implement a smarter function and pass From 6abf2baa62063a39ccc4a9535351de571004ddd4 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Wed, 8 Sep 2021 18:53:38 -0700 Subject: [PATCH 060/100] CodeHealth: Enabling clang-tidy google-explicit-constructor (#3250) * Adding google-explicit-constructor to .clang-tidy * clang-tidy explicit attr.h (all automatic) * clang-tidy explicit cast.h (all automatic) * clang-tidy detail/init.h (1 NOLINT) * clang-tidy detail/type_caster_base.h (2 NOLINT) * clang-tidy pybind11.h (7 NOLINT) * clang-tidy detail/common.h (3 NOLINT) * clang-tidy detail/descr.h (2 NOLINT) * clang-tidy pytypes.h (23 NOLINT, only 1 explicit) * clang-tidy eigen.h (7 NOLINT, 0 explicit) * Adding 2 explicit in functional.h * Adding 4 explicit in iostream.h * clang-tidy numpy.h (1 NOLINT, 1 explicit) * clang-tidy embed.h (0 NOLINT, 1 explicit) * clang-tidy tests/local_bindings.h (0 NOLINT, 4 explicit) * clang-tidy tests/pybind11_cross_module_tests.cpp (0 NOLINT, 1 explicit) * clang-tidy tests/pybind11_tests.h (0 NOLINT, 2 explicit) * clang-tidy tests/test_buffers.cpp (0 NOLINT, 2 explicit) * clang-tidy tests/test_builtin_casters.cpp (0 NOLINT, 4 explicit) * clang-tidy tests/test_class.cpp (0 NOLINT, 6 explicit) * clang-tidy tests/test_copy_move.cpp (0 NOLINT, 7 explicit) * clang-tidy tests/test_embed/external_module.cpp (0 NOLINT, 1 explicit) * clang-tidy tests/test_embed/test_interpreter.cpp (0 NOLINT, 1 explicit) * clang-tidy tests/object.h (0 NOLINT, 2 explicit) * clang-tidy batch of fully automatic fixes. * Workaround for MSVC 19.16.27045.0 C++17 Python 2 C++ syntax error. --- .clang-tidy | 1 + include/pybind11/attr.h | 24 ++++++++--- include/pybind11/cast.h | 14 ++++--- include/pybind11/detail/common.h | 3 ++ include/pybind11/detail/descr.h | 2 + include/pybind11/detail/init.h | 5 ++- include/pybind11/detail/type_caster_base.h | 18 ++++---- include/pybind11/eigen.h | 7 ++++ include/pybind11/embed.h | 8 ++-- include/pybind11/functional.h | 6 ++- include/pybind11/iostream.h | 14 ++++--- include/pybind11/numpy.h | 3 +- include/pybind11/pybind11.h | 9 +++- include/pybind11/pytypes.h | 25 ++++++++++- tests/local_bindings.h | 8 ++-- tests/object.h | 4 +- tests/pybind11_cross_module_tests.cpp | 2 +- tests/pybind11_tests.h | 4 +- tests/test_buffers.cpp | 4 +- tests/test_builtin_casters.cpp | 22 +++++++--- tests/test_class.cpp | 12 +++--- tests/test_copy_move.cpp | 14 +++---- tests/test_embed/external_module.cpp | 2 +- tests/test_embed/test_interpreter.cpp | 2 +- tests/test_exceptions.cpp | 4 +- tests/test_factory_constructors.cpp | 49 ++++++++++++++-------- tests/test_local_bindings.cpp | 2 +- tests/test_methods_and_attributes.cpp | 8 ++-- tests/test_modules.cpp | 2 +- tests/test_multiple_inheritance.cpp | 10 ++--- tests/test_numpy_vectorize.cpp | 4 +- tests/test_pickling.cpp | 4 +- tests/test_sequences_and_iterators.cpp | 16 +++---- tests/test_smart_ptr.cpp | 32 +++++++------- tests/test_stl.cpp | 3 +- tests/test_stl_binders.cpp | 2 +- tests/test_tagbased_polymorphic.cpp | 12 +++--- tests/test_virtual_functions.cpp | 2 +- 38 files changed, 231 insertions(+), 132 deletions(-) diff --git a/.clang-tidy b/.clang-tidy index cefffba1..a6437dd4 100644 --- a/.clang-tidy +++ b/.clang-tidy @@ -5,6 +5,7 @@ Checks: ' cppcoreguidelines-init-variables, cppcoreguidelines-slicing, clang-analyzer-optin.cplusplus.VirtualCall, +google-explicit-constructor, llvm-namespace-comment, misc-misplaced-const, misc-non-copyable-objects, diff --git a/include/pybind11/attr.h b/include/pybind11/attr.h index ab1fe804..13f68bbe 100644 --- a/include/pybind11/attr.h +++ b/include/pybind11/attr.h @@ -18,7 +18,9 @@ PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) /// @{ /// Annotation for methods -struct is_method { handle class_; is_method(const handle &c) : class_(c) { } }; +struct is_method { handle class_; + explicit is_method(const handle &c) : class_(c) {} +}; /// Annotation for operators struct is_operator { }; @@ -27,16 +29,24 @@ struct is_operator { }; struct is_final { }; /// Annotation for parent scope -struct scope { handle value; scope(const handle &s) : value(s) { } }; +struct scope { handle value; + explicit scope(const handle &s) : value(s) {} +}; /// Annotation for documentation -struct doc { const char *value; doc(const char *value) : value(value) { } }; +struct doc { const char *value; + explicit doc(const char *value) : value(value) {} +}; /// Annotation for function names -struct name { const char *value; name(const char *value) : value(value) { } }; +struct name { const char *value; + explicit name(const char *value) : value(value) {} +}; /// Annotation indicating that a function is an overload associated with a given "sibling" -struct sibling { handle value; sibling(const handle &value) : value(value.ptr()) { } }; +struct sibling { handle value; + explicit sibling(const handle &value) : value(value.ptr()) {} +}; /// Annotation indicating that a class derives from another given type template struct base { @@ -70,7 +80,9 @@ struct metaclass { }; /// Annotation that marks a class as local to the module: -struct module_local { const bool value; constexpr module_local(bool v = true) : value(v) { } }; +struct module_local { const bool value; + constexpr explicit module_local(bool v = true) : value(v) {} +}; /// Annotation to mark enums as an arithmetic type struct arithmetic { }; diff --git a/include/pybind11/cast.h b/include/pybind11/cast.h index 79bf506d..db79f57c 100644 --- a/include/pybind11/cast.h +++ b/include/pybind11/cast.h @@ -82,7 +82,7 @@ public: return caster_t::cast(&src.get(), policy, parent); } template using cast_op_type = std::reference_wrapper; - operator std::reference_wrapper() { return cast_op(subcaster); } + explicit operator std::reference_wrapper() { return cast_op(subcaster); } }; #define PYBIND11_TYPE_CASTER(type, py_name) \ @@ -279,7 +279,7 @@ public: } template using cast_op_type = void*&; - operator void *&() { return value; } + explicit operator void *&() { return value; } static constexpr auto name = _("capsule"); private: void *value = nullptr; @@ -487,8 +487,10 @@ public: return StringCaster::cast(StringType(1, src), policy, parent); } - operator CharT*() { return none ? nullptr : const_cast(static_cast(str_caster).c_str()); } - operator CharT&() { + explicit operator CharT *() { + return none ? nullptr : const_cast(static_cast(str_caster).c_str()); + } + explicit operator CharT &() { if (none) throw value_error("Cannot convert None to a character"); @@ -581,8 +583,8 @@ public: template using cast_op_type = type; - operator type() & { return implicit_cast(indices{}); } - operator type() && { return std::move(*this).implicit_cast(indices{}); } + explicit operator type() & { return implicit_cast(indices{}); } + explicit operator type() && { return std::move(*this).implicit_cast(indices{}); } protected: template diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index e4f00293..a8fd7d91 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -918,6 +918,7 @@ public: // Implicit conversion constructor from any arbitrary container type with values convertible to T template ())), T>::value>> + // NOLINTNEXTLINE(google-explicit-constructor) any_container(const Container &c) : any_container(std::begin(c), std::end(c)) { } // initializer_list's aren't deducible, so don't get matched by the above template; we need this @@ -926,9 +927,11 @@ public: any_container(const std::initializer_list &c) : any_container(c.begin(), c.end()) { } // Avoid copying if given an rvalue vector of the correct type. + // NOLINTNEXTLINE(google-explicit-constructor) any_container(std::vector &&v) : v(std::move(v)) { } // Moves the vector out of an rvalue any_container + // NOLINTNEXTLINE(google-explicit-constructor) operator std::vector &&() && { return std::move(v); } // Dereferencing obtains a reference to the underlying vector diff --git a/include/pybind11/detail/descr.h b/include/pybind11/detail/descr.h index 0acfc7db..0b498e5e 100644 --- a/include/pybind11/detail/descr.h +++ b/include/pybind11/detail/descr.h @@ -26,12 +26,14 @@ struct descr { char text[N + 1]{'\0'}; constexpr descr() = default; + // NOLINTNEXTLINE(google-explicit-constructor) constexpr descr(char const (&s)[N+1]) : descr(s, make_index_sequence()) { } template constexpr descr(char const (&s)[N+1], index_sequence) : text{s[Is]..., '\0'} { } template + // NOLINTNEXTLINE(google-explicit-constructor) constexpr descr(char c, Chars... cs) : text{c, static_cast(cs)..., '\0'} { } static constexpr std::array types() { diff --git a/include/pybind11/detail/init.h b/include/pybind11/detail/init.h index e795da7d..cace3529 100644 --- a/include/pybind11/detail/init.h +++ b/include/pybind11/detail/init.h @@ -23,7 +23,7 @@ public: } template using cast_op_type = value_and_holder &; - operator value_and_holder &() { return *value; } + explicit operator value_and_holder &() { return *value; } static constexpr auto name = _(); private: @@ -222,7 +222,8 @@ template struct factory { remove_reference_t class_factory; - factory(Func &&f) : class_factory(std::forward(f)) { } + // NOLINTNEXTLINE(google-explicit-constructor) + factory(Func &&f) : class_factory(std::forward(f)) {} // The given class either has no alias or has no separate alias factory; // this always constructs the class itself. If the class is registered with an alias diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h index 5a5acc2e..4c9ac7b8 100644 --- a/include/pybind11/detail/type_caster_base.h +++ b/include/pybind11/detail/type_caster_base.h @@ -225,7 +225,7 @@ struct value_and_holder { value_and_holder() = default; // Used for past-the-end iterator - value_and_holder(size_t index) : index{index} {} + explicit value_and_holder(size_t index) : index{index} {} template V *&value_ptr() const { return reinterpret_cast(vh[0]); @@ -274,7 +274,8 @@ private: const type_vec &tinfo; public: - values_and_holders(instance *inst) : inst{inst}, tinfo(all_type_info(Py_TYPE(inst))) {} + explicit values_and_holders(instance *inst) + : inst{inst}, tinfo(all_type_info(Py_TYPE(inst))) {} struct iterator { private: @@ -290,7 +291,8 @@ public: 0 /* index */) {} // Past-the-end iterator: - iterator(size_t end) : curr(end) {} + explicit iterator(size_t end) : curr(end) {} + public: bool operator==(const iterator &other) const { return curr.index == other.curr.index; } bool operator!=(const iterator &other) const { return curr.index != other.curr.index; } @@ -491,11 +493,11 @@ inline PyObject *make_new_instance(PyTypeObject *type); class type_caster_generic { public: - PYBIND11_NOINLINE type_caster_generic(const std::type_info &type_info) - : typeinfo(get_type_info(type_info)), cpptype(&type_info) { } + PYBIND11_NOINLINE explicit type_caster_generic(const std::type_info &type_info) + : typeinfo(get_type_info(type_info)), cpptype(&type_info) {} - type_caster_generic(const type_info *typeinfo) - : typeinfo(typeinfo), cpptype(typeinfo ? typeinfo->cpptype : nullptr) { } + explicit type_caster_generic(const type_info *typeinfo) + : typeinfo(typeinfo), cpptype(typeinfo ? typeinfo->cpptype : nullptr) {} bool load(handle src, bool convert) { return load_impl(src, convert); @@ -923,7 +925,9 @@ public: template using cast_op_type = detail::cast_op_type; + // NOLINTNEXTLINE(google-explicit-constructor) operator itype*() { return (type *) value; } + // NOLINTNEXTLINE(google-explicit-constructor) operator itype&() { if (!value) throw reference_cast_error(); return *((itype *) value); } protected: diff --git a/include/pybind11/eigen.h b/include/pybind11/eigen.h index b03e15f6..c0363827 100644 --- a/include/pybind11/eigen.h +++ b/include/pybind11/eigen.h @@ -61,6 +61,7 @@ template struct EigenConformable { EigenDStride stride{0, 0}; // Only valid if negativestrides is false! bool negativestrides = false; // If true, do not use stride! + // NOLINTNEXTLINE(google-explicit-constructor) EigenConformable(bool fits = false) : conformable{fits} {} // Matrix type: EigenConformable(EigenIndex r, EigenIndex c, @@ -88,6 +89,7 @@ template struct EigenConformable { (props::outer_stride == Eigen::Dynamic || props::outer_stride == stride.outer() || (EigenRowMajor ? rows : cols) == 1); } + // NOLINTNEXTLINE(google-explicit-constructor) operator bool() const { return conformable; } }; @@ -326,8 +328,11 @@ public: static constexpr auto name = props::descriptor; + // NOLINTNEXTLINE(google-explicit-constructor) operator Type*() { return &value; } + // NOLINTNEXTLINE(google-explicit-constructor) operator Type&() { return value; } + // NOLINTNEXTLINE(google-explicit-constructor) operator Type&&() && { return std::move(value); } template using cast_op_type = movable_cast_op_type; @@ -451,7 +456,9 @@ public: return true; } + // NOLINTNEXTLINE(google-explicit-constructor) operator Type*() { return ref.get(); } + // NOLINTNEXTLINE(google-explicit-constructor) operator Type&() { return *ref; } template using cast_op_type = pybind11::detail::cast_op_type<_T>; diff --git a/include/pybind11/embed.h b/include/pybind11/embed.h index 7b5d7cd2..9843f0f9 100644 --- a/include/pybind11/embed.h +++ b/include/pybind11/embed.h @@ -260,10 +260,10 @@ inline void finalize_interpreter() { \endrst */ class scoped_interpreter { public: - scoped_interpreter(bool init_signal_handlers = true, - int argc = 0, - const char *const *argv = nullptr, - bool add_program_dir_to_path = true) { + explicit scoped_interpreter(bool init_signal_handlers = true, + int argc = 0, + const char *const *argv = nullptr, + bool add_program_dir_to_path = true) { initialize_interpreter(init_signal_handlers, argc, argv, add_program_dir_to_path); } diff --git a/include/pybind11/functional.h b/include/pybind11/functional.h index bc8a8af8..24141ce3 100644 --- a/include/pybind11/functional.h +++ b/include/pybind11/functional.h @@ -69,6 +69,10 @@ public: // ensure GIL is held during functor destruction struct func_handle { function f; +#if !(defined(_MSC_VER) && _MSC_VER == 1916 && defined(PYBIND11_CPP17) && PY_MAJOR_VERSION < 3) + // This triggers a syntax error under very special conditions (very weird indeed). + explicit +#endif func_handle(function &&f_) noexcept : f(std::move(f_)) {} func_handle(const func_handle &f_) { operator=(f_); } func_handle &operator=(const func_handle &f_) { @@ -85,7 +89,7 @@ public: // to emulate 'move initialization capture' in C++11 struct func_wrapper { func_handle hfunc; - func_wrapper(func_handle &&hf) noexcept : hfunc(std::move(hf)) {} + explicit func_wrapper(func_handle &&hf) noexcept : hfunc(std::move(hf)) {} Return operator()(Args... args) const { gil_scoped_acquire acq; object retval(hfunc.f(std::forward(args)...)); diff --git a/include/pybind11/iostream.h b/include/pybind11/iostream.h index e4d20958..95449a07 100644 --- a/include/pybind11/iostream.h +++ b/include/pybind11/iostream.h @@ -123,7 +123,7 @@ private: } public: - pythonbuf(const object &pyostream, size_t buffer_size = 1024) + explicit pythonbuf(const object &pyostream, size_t buffer_size = 1024) : buf_size(buffer_size), d_buffer(new char[buf_size]), pywrite(pyostream.attr("write")), pyflush(pyostream.attr("flush")) { setp(d_buffer.get(), d_buffer.get() + buf_size - 1); @@ -171,8 +171,9 @@ protected: detail::pythonbuf buffer; public: - scoped_ostream_redirect(std::ostream &costream = std::cout, - const object &pyostream = module_::import("sys").attr("stdout")) + explicit scoped_ostream_redirect(std::ostream &costream = std::cout, + const object &pyostream + = module_::import("sys").attr("stdout")) : costream(costream), buffer(pyostream) { old = costream.rdbuf(&buffer); } @@ -201,8 +202,9 @@ public: \endrst */ class scoped_estream_redirect : public scoped_ostream_redirect { public: - scoped_estream_redirect(std::ostream &costream = std::cerr, - const object &pyostream = module_::import("sys").attr("stderr")) + explicit scoped_estream_redirect(std::ostream &costream = std::cerr, + const object &pyostream + = module_::import("sys").attr("stderr")) : scoped_ostream_redirect(costream, pyostream) {} }; @@ -217,7 +219,7 @@ class OstreamRedirect { std::unique_ptr redirect_stderr; public: - OstreamRedirect(bool do_stdout = true, bool do_stderr = true) + explicit OstreamRedirect(bool do_stdout = true, bool do_stderr = true) : do_stdout_(do_stdout), do_stderr_(do_stderr) {} void enter() { diff --git a/include/pybind11/numpy.h b/include/pybind11/numpy.h index fa128efd..d5584408 100644 --- a/include/pybind11/numpy.h +++ b/include/pybind11/numpy.h @@ -477,7 +477,7 @@ public: m_ptr = from_args(pybind11::str(format)).release().ptr(); } - dtype(const char *format) : dtype(std::string(format)) { } + explicit dtype(const char *format) : dtype(std::string(format)) {} dtype(list names, list formats, list offsets, ssize_t itemsize) { dict args; @@ -894,6 +894,7 @@ public: if (!is_borrowed) Py_XDECREF(h.ptr()); } + // NOLINTNEXTLINE(google-explicit-constructor) array_t(const object &o) : array(raw_array_t(o.ptr()), stolen_t{}) { if (!m_ptr) throw error_already_set(); } diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 990c2130..82d33975 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -81,10 +81,12 @@ PYBIND11_NAMESPACE_END(detail) class cpp_function : public function { public: cpp_function() = default; + // NOLINTNEXTLINE(google-explicit-constructor) cpp_function(std::nullptr_t) { } /// Construct a cpp_function from a vanilla function pointer template + // NOLINTNEXTLINE(google-explicit-constructor) cpp_function(Return (*f)(Args...), const Extra&... extra) { initialize(f, f, extra...); } @@ -92,6 +94,7 @@ public: /// Construct a cpp_function from a lambda function (possibly with internal state) template ::value>> + // NOLINTNEXTLINE(google-explicit-constructor) cpp_function(Func &&f, const Extra&... extra) { initialize(std::forward(f), (detail::function_signature_t *) nullptr, extra...); @@ -99,6 +102,7 @@ public: /// Construct a cpp_function from a class method (non-const, no ref-qualifier) template + // NOLINTNEXTLINE(google-explicit-constructor) cpp_function(Return (Class::*f)(Arg...), const Extra&... extra) { initialize([f](Class *c, Arg... args) -> Return { return (c->*f)(std::forward(args)...); }, (Return (*) (Class *, Arg...)) nullptr, extra...); @@ -108,6 +112,7 @@ public: /// A copy of the overload for non-const functions without explicit ref-qualifier /// but with an added `&`. template + // NOLINTNEXTLINE(google-explicit-constructor) cpp_function(Return (Class::*f)(Arg...)&, const Extra&... extra) { initialize([f](Class *c, Arg... args) -> Return { return (c->*f)(args...); }, (Return (*) (Class *, Arg...)) nullptr, extra...); @@ -115,6 +120,7 @@ public: /// Construct a cpp_function from a class method (const, no ref-qualifier) template + // NOLINTNEXTLINE(google-explicit-constructor) cpp_function(Return (Class::*f)(Arg...) const, const Extra&... extra) { initialize([f](const Class *c, Arg... args) -> Return { return (c->*f)(std::forward(args)...); }, (Return (*)(const Class *, Arg ...)) nullptr, extra...); @@ -124,6 +130,7 @@ public: /// A copy of the overload for const functions without explicit ref-qualifier /// but with an added `&`. template + // NOLINTNEXTLINE(google-explicit-constructor) cpp_function(Return (Class::*f)(Arg...) const&, const Extra&... extra) { initialize([f](const Class *c, Arg... args) -> Return { return (c->*f)(args...); }, (Return (*)(const Class *, Arg ...)) nullptr, extra...); @@ -2034,7 +2041,7 @@ template void implicitly_convertible() { struct set_flag { bool &flag; - set_flag(bool &flag_) : flag(flag_) { flag_ = true; } + explicit set_flag(bool &flag_) : flag(flag_) { flag_ = true; } ~set_flag() { flag = false; } }; auto implicit_caster = [](PyObject *obj, PyTypeObject *type) -> PyObject * { diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index fa50eb1e..971db85f 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -178,6 +178,7 @@ public: /// The default constructor creates a handle with a ``nullptr``-valued pointer handle() = default; /// Creates a ``handle`` from the given raw Python object pointer + // NOLINTNEXTLINE(google-explicit-constructor) handle(PyObject *ptr) : m_ptr(ptr) { } // Allow implicit conversion from PyObject* /// Return the underlying ``PyObject *`` pointer @@ -612,6 +613,7 @@ public: return obj.contains(key); } + // NOLINTNEXTLINE(google-explicit-constructor) operator object() const { return get_cache(); } PyObject *ptr() const { return get_cache().ptr(); } template T cast() const { return get_cache().template cast(); } @@ -761,6 +763,7 @@ template struct arrow_proxy { T value; + // NOLINTNEXTLINE(google-explicit-constructor) arrow_proxy(T &&value) noexcept : value(std::move(value)) { } T *operator->() const { return &value; } }; @@ -909,14 +912,17 @@ PYBIND11_NAMESPACE_END(detail) bool check() const { return m_ptr != nullptr && (CheckFun(m_ptr) != 0); } \ static bool check_(handle h) { return h.ptr() != nullptr && CheckFun(h.ptr()); } \ template \ + /* NOLINTNEXTLINE(google-explicit-constructor) */ \ Name(const ::pybind11::detail::accessor &a) : Name(object(a)) { } #define PYBIND11_OBJECT_CVT(Name, Parent, CheckFun, ConvertFun) \ PYBIND11_OBJECT_COMMON(Name, Parent, CheckFun) \ /* This is deliberately not 'explicit' to allow implicit conversion from object: */ \ + /* NOLINTNEXTLINE(google-explicit-constructor) */ \ Name(const object &o) \ : Parent(check_(o) ? o.inc_ref().ptr() : ConvertFun(o.ptr()), stolen_t{}) \ { if (!m_ptr) throw error_already_set(); } \ + /* NOLINTNEXTLINE(google-explicit-constructor) */ \ Name(object &&o) \ : Parent(check_(o) ? o.release().ptr() : ConvertFun(o.ptr()), stolen_t{}) \ { if (!m_ptr) throw error_already_set(); } @@ -933,8 +939,10 @@ PYBIND11_NAMESPACE_END(detail) #define PYBIND11_OBJECT(Name, Parent, CheckFun) \ PYBIND11_OBJECT_COMMON(Name, Parent, CheckFun) \ /* This is deliberately not 'explicit' to allow implicit conversion from object: */ \ + /* NOLINTNEXTLINE(google-explicit-constructor) */ \ Name(const object &o) : Parent(o) \ { if (m_ptr && !check_(m_ptr)) throw PYBIND11_OBJECT_CHECK_FAILED(Name, m_ptr); } \ + /* NOLINTNEXTLINE(google-explicit-constructor) */ \ Name(object &&o) : Parent(std::move(o)) \ { if (m_ptr && !check_(m_ptr)) throw PYBIND11_OBJECT_CHECK_FAILED(Name, m_ptr); } @@ -1056,11 +1064,13 @@ public: } // 'explicit' is explicitly omitted from the following constructors to allow implicit conversion to py::str from C++ string-like objects + // NOLINTNEXTLINE(google-explicit-constructor) str(const char *c = "") : object(PyUnicode_FromString(c), stolen_t{}) { if (!m_ptr) pybind11_fail("Could not allocate string object!"); } + // NOLINTNEXTLINE(google-explicit-constructor) str(const std::string &s) : str(s.data(), s.size()) { } explicit str(const bytes &b); @@ -1071,6 +1081,7 @@ public: \endrst */ explicit str(handle h) : object(raw_str(h.ptr()), stolen_t{}) { if (!m_ptr) throw error_already_set(); } + // NOLINTNEXTLINE(google-explicit-constructor) operator std::string() const { object temp = *this; if (PyUnicode_Check(m_ptr)) { @@ -1118,6 +1129,7 @@ public: PYBIND11_OBJECT(bytes, object, PYBIND11_BYTES_CHECK) // Allow implicit conversion: + // NOLINTNEXTLINE(google-explicit-constructor) bytes(const char *c = "") : object(PYBIND11_BYTES_FROM_STRING(c), stolen_t{}) { if (!m_ptr) pybind11_fail("Could not allocate bytes object!"); @@ -1130,10 +1142,12 @@ public: } // Allow implicit conversion: + // NOLINTNEXTLINE(google-explicit-constructor) bytes(const std::string &s) : bytes(s.data(), s.size()) { } explicit bytes(const pybind11::str &s); + // NOLINTNEXTLINE(google-explicit-constructor) operator std::string() const { char *buffer = nullptr; ssize_t length = 0; @@ -1222,7 +1236,9 @@ public: PYBIND11_OBJECT_CVT(bool_, object, PyBool_Check, raw_bool) bool_() : object(Py_False, borrowed_t{}) { } // Allow implicit conversion from and to `bool`: + // NOLINTNEXTLINE(google-explicit-constructor) bool_(bool value) : object(value ? Py_True : Py_False, borrowed_t{}) { } + // NOLINTNEXTLINE(google-explicit-constructor) operator bool() const { return (m_ptr != nullptr) && PyLong_AsLong(m_ptr) != 0; } private: @@ -1261,6 +1277,7 @@ public: // Allow implicit conversion from C++ integral types: template ::value, int> = 0> + // NOLINTNEXTLINE(google-explicit-constructor) int_(T value) { if (PYBIND11_SILENCE_MSVC_C4127(sizeof(T) <= sizeof(long))) { if (std::is_signed::value) @@ -1278,6 +1295,7 @@ public: template ::value, int> = 0> + // NOLINTNEXTLINE(google-explicit-constructor) operator T() const { return std::is_unsigned::value ? detail::as_unsigned(m_ptr) @@ -1291,13 +1309,17 @@ class float_ : public object { public: PYBIND11_OBJECT_CVT(float_, object, PyFloat_Check, PyNumber_Float) // Allow implicit conversion from float/double: + // NOLINTNEXTLINE(google-explicit-constructor) float_(float value) : object(PyFloat_FromDouble((double) value), stolen_t{}) { if (!m_ptr) pybind11_fail("Could not allocate float object!"); } + // NOLINTNEXTLINE(google-explicit-constructor) float_(double value = .0) : object(PyFloat_FromDouble((double) value), stolen_t{}) { if (!m_ptr) pybind11_fail("Could not allocate float object!"); } + // NOLINTNEXTLINE(google-explicit-constructor) operator float() const { return (float) PyFloat_AsDouble(m_ptr); } + // NOLINTNEXTLINE(google-explicit-constructor) operator double() const { return (double) PyFloat_AsDouble(m_ptr); } }; @@ -1372,7 +1394,7 @@ public: pybind11_fail("Could not set capsule context!"); } - capsule(void (*destructor)()) { + explicit capsule(void (*destructor)()) { m_ptr = PyCapsule_New(reinterpret_cast(destructor), nullptr, [](PyObject *o) { auto destructor = reinterpret_cast(PyCapsule_GetPointer(o, nullptr)); destructor(); @@ -1382,6 +1404,7 @@ public: pybind11_fail("Could not allocate capsule object!"); } + // NOLINTNEXTLINE(google-explicit-constructor) template operator T *() const { return get_pointer(); } diff --git a/tests/local_bindings.h b/tests/local_bindings.h index f11c08e6..4c936c19 100644 --- a/tests/local_bindings.h +++ b/tests/local_bindings.h @@ -6,7 +6,7 @@ /// Simple class used to test py::local: template class LocalBase { public: - LocalBase(int i) : i(i) { } + explicit LocalBase(int i) : i(i) { } int i = -1; }; @@ -75,11 +75,11 @@ py::class_ bind_local(Args && ...args) { namespace pets { class Pet { public: - Pet(std::string name) : name_(std::move(name)) {} + explicit Pet(std::string name) : name_(std::move(name)) {} std::string name_; const std::string &name() const { return name_; } }; } // namespace pets -struct MixGL { int i; MixGL(int i) : i{i} {} }; -struct MixGL2 { int i; MixGL2(int i) : i{i} {} }; +struct MixGL { int i; explicit MixGL(int i) : i{i} {} }; +struct MixGL2 { int i; explicit MixGL2(int i) : i{i} {} }; diff --git a/tests/object.h b/tests/object.h index 6851fc62..be21bf63 100644 --- a/tests/object.h +++ b/tests/object.h @@ -65,7 +65,7 @@ public: ref() : m_ptr(nullptr) { print_default_created(this); track_default_created((ref_tag*) this); } /// Construct a reference from a pointer - ref(T *ptr) : m_ptr(ptr) { + explicit ref(T *ptr) : m_ptr(ptr) { if (m_ptr) ((Object *) m_ptr)->incRef(); print_created(this, "from pointer", m_ptr); track_created((ref_tag*) this, "from pointer"); @@ -165,7 +165,7 @@ public: const T& operator*() const { return *m_ptr; } /// Return a pointer to the referenced object - operator T* () { return m_ptr; } + explicit operator T* () { return m_ptr; } /// Return a const pointer to the referenced object T* get_ptr() { return m_ptr; } diff --git a/tests/pybind11_cross_module_tests.cpp b/tests/pybind11_cross_module_tests.cpp index 4bfd5302..5838cb27 100644 --- a/tests/pybind11_cross_module_tests.cpp +++ b/tests/pybind11_cross_module_tests.cpp @@ -123,7 +123,7 @@ PYBIND11_MODULE(pybind11_cross_module_tests, m) { class Dog : public pets::Pet { public: - Dog(std::string name) : Pet(std::move(name)) {} + explicit Dog(std::string name) : Pet(std::move(name)) {} }; py::class_(m, "Pet", py::module_local()) .def("name", &pets::Pet::name); diff --git a/tests/pybind11_tests.h b/tests/pybind11_tests.h index 8da0a670..800ddda4 100644 --- a/tests/pybind11_tests.h +++ b/tests/pybind11_tests.h @@ -16,7 +16,7 @@ class test_initializer { using Initializer = void (*)(py::module_ &); public: - test_initializer(Initializer init); + explicit test_initializer(Initializer init); test_initializer(const char *submodule_name, Initializer init); }; @@ -32,7 +32,7 @@ struct UnregisteredType { }; class UserType { public: UserType() = default; - UserType(int i) : i(i) { } + explicit UserType(int i) : i(i) { } int value() const { return i; } void set(int set) { i = set; } diff --git a/tests/test_buffers.cpp b/tests/test_buffers.cpp index c7e2c7df..3a8e3e7b 100644 --- a/tests/test_buffers.cpp +++ b/tests/test_buffers.cpp @@ -122,7 +122,7 @@ TEST_SUBMODULE(buffers, m) { // test_inherited_protocol class SquareMatrix : public Matrix { public: - SquareMatrix(py::ssize_t n) : Matrix(n, n) { } + explicit SquareMatrix(py::ssize_t n) : Matrix(n, n) {} }; // Derived classes inherit the buffer protocol and the buffer access function py::class_(m, "SquareMatrix") @@ -173,7 +173,7 @@ TEST_SUBMODULE(buffers, m) { struct BufferReadOnly { const uint8_t value = 0; - BufferReadOnly(uint8_t value): value(value) {} + explicit BufferReadOnly(uint8_t value) : value(value) {} py::buffer_info get_buffer_info() { return py::buffer_info(&value, 1); diff --git a/tests/test_builtin_casters.cpp b/tests/test_builtin_casters.cpp index be4431aa..71c778e8 100644 --- a/tests/test_builtin_casters.cpp +++ b/tests/test_builtin_casters.cpp @@ -25,16 +25,28 @@ class type_caster { // cast operator. bool load(handle, bool) { return true; } - operator ConstRefCasted &&() { + explicit operator ConstRefCasted &&() { value = {1}; // NOLINTNEXTLINE(performance-move-const-arg) return std::move(value); } - operator ConstRefCasted&() { value = {2}; return value; } - operator ConstRefCasted*() { value = {3}; return &value; } + explicit operator ConstRefCasted &() { + value = {2}; + return value; + } + explicit operator ConstRefCasted *() { + value = {3}; + return &value; + } - operator const ConstRefCasted&() { value = {4}; return value; } - operator const ConstRefCasted*() { value = {5}; return &value; } + explicit operator const ConstRefCasted &() { + value = {4}; + return value; + } + explicit operator const ConstRefCasted *() { + value = {5}; + return &value; + } // custom cast_op to explicitly propagate types to the conversion operators. template diff --git a/tests/test_class.cpp b/tests/test_class.cpp index dff75839..246b6c1f 100644 --- a/tests/test_class.cpp +++ b/tests/test_class.cpp @@ -27,7 +27,7 @@ // test_brace_initialization struct NoBraceInitialization { - NoBraceInitialization(std::vector v) : vec{std::move(v)} {} + explicit NoBraceInitialization(std::vector v) : vec{std::move(v)} {} template NoBraceInitialization(std::initializer_list l) : vec(l) {} @@ -65,18 +65,18 @@ TEST_SUBMODULE(class_, m) { class Dog : public Pet { public: - Dog(const std::string &name) : Pet(name, "dog") {} + explicit Dog(const std::string &name) : Pet(name, "dog") {} std::string bark() const { return "Woof!"; } }; class Rabbit : public Pet { public: - Rabbit(const std::string &name) : Pet(name, "parrot") {} + explicit Rabbit(const std::string &name) : Pet(name, "parrot") {} }; class Hamster : public Pet { public: - Hamster(const std::string &name) : Pet(name, "rodent") {} + explicit Hamster(const std::string &name) : Pet(name, "rodent") {} }; class Chimera : public Pet { @@ -208,7 +208,7 @@ TEST_SUBMODULE(class_, m) { struct ConvertibleFromUserType { int i; - ConvertibleFromUserType(UserType u) : i(u.value()) { } + explicit ConvertibleFromUserType(UserType u) : i(u.value()) {} }; py::class_(m, "AcceptsUserType") @@ -263,7 +263,7 @@ TEST_SUBMODULE(class_, m) { }; struct PyAliasedHasOpNewDelSize : AliasedHasOpNewDelSize { PyAliasedHasOpNewDelSize() = default; - PyAliasedHasOpNewDelSize(int) { } + explicit PyAliasedHasOpNewDelSize(int) {} std::uint64_t j; }; struct HasOpNewDelBoth { diff --git a/tests/test_copy_move.cpp b/tests/test_copy_move.cpp index 859da9df..5fb0dd81 100644 --- a/tests/test_copy_move.cpp +++ b/tests/test_copy_move.cpp @@ -37,7 +37,7 @@ template <> lacking_move_ctor empty::instance_ = {}; class MoveOnlyInt { public: MoveOnlyInt() { print_default_created(this); } - MoveOnlyInt(int v) : value{v} { print_created(this, value); } + explicit MoveOnlyInt(int v) : value{v} { print_created(this, value); } MoveOnlyInt(MoveOnlyInt &&m) noexcept { print_move_created(this, m.value); std::swap(value, m.value); @@ -56,7 +56,7 @@ public: class MoveOrCopyInt { public: MoveOrCopyInt() { print_default_created(this); } - MoveOrCopyInt(int v) : value{v} { print_created(this, value); } + explicit MoveOrCopyInt(int v) : value{v} { print_created(this, value); } MoveOrCopyInt(MoveOrCopyInt &&m) noexcept { print_move_created(this, m.value); std::swap(value, m.value); @@ -75,7 +75,7 @@ public: class CopyOnlyInt { public: CopyOnlyInt() { print_default_created(this); } - CopyOnlyInt(int v) : value{v} { print_created(this, value); } + explicit CopyOnlyInt(int v) : value{v} { print_created(this, value); } CopyOnlyInt(const CopyOnlyInt &c) { print_copy_created(this, c.value); value = c.value; } CopyOnlyInt &operator=(const CopyOnlyInt &c) { print_copy_assigned(this, c.value); value = c.value; return *this; } ~CopyOnlyInt() { print_destroyed(this); } @@ -107,8 +107,8 @@ public: if (!src) return none().release(); return cast(*src, policy, parent); } - operator CopyOnlyInt*() { return &value; } - operator CopyOnlyInt&() { return value; } + explicit operator CopyOnlyInt *() { return &value; } + explicit operator CopyOnlyInt &() { return value; } template using cast_op_type = pybind11::detail::cast_op_type; }; PYBIND11_NAMESPACE_END(detail) @@ -219,7 +219,7 @@ TEST_SUBMODULE(copy_move_policies, m) { // #389: rvp::move should fall-through to copy on non-movable objects struct MoveIssue1 { int v; - MoveIssue1(int v) : v{v} {} + explicit MoveIssue1(int v) : v{v} {} MoveIssue1(const MoveIssue1 &c) = default; MoveIssue1(MoveIssue1 &&) = delete; }; @@ -227,7 +227,7 @@ TEST_SUBMODULE(copy_move_policies, m) { struct MoveIssue2 { int v; - MoveIssue2(int v) : v{v} {} + explicit MoveIssue2(int v) : v{v} {} MoveIssue2(MoveIssue2 &&) = default; }; py::class_(m, "MoveIssue2").def(py::init()).def_readwrite("value", &MoveIssue2::v); diff --git a/tests/test_embed/external_module.cpp b/tests/test_embed/external_module.cpp index e9a6058b..49095229 100644 --- a/tests/test_embed/external_module.cpp +++ b/tests/test_embed/external_module.cpp @@ -9,7 +9,7 @@ namespace py = pybind11; PYBIND11_MODULE(external_module, m) { class A { public: - A(int value) : v{value} {}; + explicit A(int value) : v{value} {}; int v; }; diff --git a/tests/test_embed/test_interpreter.cpp b/tests/test_embed/test_interpreter.cpp index 78b64be6..fae14f75 100644 --- a/tests/test_embed/test_interpreter.cpp +++ b/tests/test_embed/test_interpreter.cpp @@ -18,7 +18,7 @@ using namespace py::literals; class Widget { public: - Widget(std::string message) : message(std::move(message)) {} + explicit Widget(std::string message) : message(std::move(message)) {} virtual ~Widget() = default; std::string the_message() const { return message; } diff --git a/tests/test_exceptions.cpp b/tests/test_exceptions.cpp index 4805a024..25adb32e 100644 --- a/tests/test_exceptions.cpp +++ b/tests/test_exceptions.cpp @@ -81,7 +81,7 @@ private: struct PythonCallInDestructor { - PythonCallInDestructor(const py::dict &d) : d(d) {} + explicit PythonCallInDestructor(const py::dict &d) : d(d) {} ~PythonCallInDestructor() { d["good"] = true; } py::dict d; @@ -90,7 +90,7 @@ struct PythonCallInDestructor { struct PythonAlreadySetInDestructor { - PythonAlreadySetInDestructor(const py::str &s) : s(s) {} + explicit PythonAlreadySetInDestructor(const py::str &s) : s(s) {} ~PythonAlreadySetInDestructor() { py::dict foo; try { diff --git a/tests/test_factory_constructors.cpp b/tests/test_factory_constructors.cpp index 235ec4dc..660e2896 100644 --- a/tests/test_factory_constructors.cpp +++ b/tests/test_factory_constructors.cpp @@ -19,8 +19,9 @@ class TestFactory1 { friend class TestFactoryHelper; TestFactory1() : value("(empty)") { print_default_created(this); } - TestFactory1(int v) : value(std::to_string(v)) { print_created(this, value); } - TestFactory1(std::string v) : value(std::move(v)) { print_created(this, value); } + explicit TestFactory1(int v) : value(std::to_string(v)) { print_created(this, value); } + explicit TestFactory1(std::string v) : value(std::move(v)) { print_created(this, value); } + public: std::string value; TestFactory1(TestFactory1 &&) = delete; @@ -33,8 +34,9 @@ public: class TestFactory2 { friend class TestFactoryHelper; TestFactory2() : value("(empty2)") { print_default_created(this); } - TestFactory2(int v) : value(std::to_string(v)) { print_created(this, value); } - TestFactory2(std::string v) : value(std::move(v)) { print_created(this, value); } + explicit TestFactory2(int v) : value(std::to_string(v)) { print_created(this, value); } + explicit TestFactory2(std::string v) : value(std::move(v)) { print_created(this, value); } + public: TestFactory2(TestFactory2 &&m) noexcept { value = std::move(m.value); @@ -53,9 +55,10 @@ class TestFactory3 { protected: friend class TestFactoryHelper; TestFactory3() : value("(empty3)") { print_default_created(this); } - TestFactory3(int v) : value(std::to_string(v)) { print_created(this, value); } + explicit TestFactory3(int v) : value(std::to_string(v)) { print_created(this, value); } + public: - TestFactory3(std::string v) : value(std::move(v)) { print_created(this, value); } + explicit TestFactory3(std::string v) : value(std::move(v)) { print_created(this, value); } TestFactory3(TestFactory3 &&m) noexcept { value = std::move(m.value); print_move_created(this); @@ -72,13 +75,13 @@ public: class TestFactory4 : public TestFactory3 { public: TestFactory4() : TestFactory3() { print_default_created(this); } - TestFactory4(int v) : TestFactory3(v) { print_created(this, v); } + explicit TestFactory4(int v) : TestFactory3(v) { print_created(this, v); } ~TestFactory4() override { print_destroyed(this); } }; // Another class for an invalid downcast test class TestFactory5 : public TestFactory3 { public: - TestFactory5(int i) : TestFactory3(i) { print_created(this, i); } + explicit TestFactory5(int i) : TestFactory3(i) { print_created(this, i); } ~TestFactory5() override { print_destroyed(this); } }; @@ -87,7 +90,7 @@ protected: int value; bool alias = false; public: - TestFactory6(int i) : value{i} { print_created(this, i); } + explicit TestFactory6(int i) : value{i} { print_created(this, i); } TestFactory6(TestFactory6 &&f) noexcept { print_move_created(this); value = f.value; @@ -102,11 +105,20 @@ class PyTF6 : public TestFactory6 { public: // Special constructor that allows the factory to construct a PyTF6 from a TestFactory6 only // when an alias is needed: - PyTF6(TestFactory6 &&base) : TestFactory6(std::move(base)) { alias = true; print_created(this, "move", value); } - PyTF6(int i) : TestFactory6(i) { alias = true; print_created(this, i); } + explicit PyTF6(TestFactory6 &&base) : TestFactory6(std::move(base)) { + alias = true; + print_created(this, "move", value); + } + explicit PyTF6(int i) : TestFactory6(i) { + alias = true; + print_created(this, i); + } PyTF6(PyTF6 &&f) noexcept : TestFactory6(std::move(f)) { print_move_created(this); } PyTF6(const PyTF6 &f) : TestFactory6(f) { print_copy_created(this); } - PyTF6(std::string s) : TestFactory6((int) s.size()) { alias = true; print_created(this, s); } + explicit PyTF6(std::string s) : TestFactory6((int) s.size()) { + alias = true; + print_created(this, s); + } ~PyTF6() override { print_destroyed(this); } int get() override { PYBIND11_OVERRIDE(int, TestFactory6, get, /*no args*/); } }; @@ -116,7 +128,7 @@ protected: int value; bool alias = false; public: - TestFactory7(int i) : value{i} { print_created(this, i); } + explicit TestFactory7(int i) : value{i} { print_created(this, i); } TestFactory7(TestFactory7 &&f) noexcept { print_move_created(this); value = f.value; @@ -129,7 +141,10 @@ public: }; class PyTF7 : public TestFactory7 { public: - PyTF7(int i) : TestFactory7(i) { alias = true; print_created(this, i); } + explicit PyTF7(int i) : TestFactory7(i) { + alias = true; + print_created(this, i); + } PyTF7(PyTF7 &&f) noexcept : TestFactory7(std::move(f)) { print_move_created(this); } PyTF7(const PyTF7 &f) : TestFactory7(f) { print_copy_created(this); } ~PyTF7() override { print_destroyed(this); } @@ -300,7 +315,7 @@ TEST_SUBMODULE(factory_constructors, m) { // Class with a custom new operator but *without* a placement new operator (issue #948) class NoPlacementNew { public: - NoPlacementNew(int i) : i(i) { } + explicit NoPlacementNew(int i) : i(i) {} static void *operator new(std::size_t s) { auto *p = ::operator new(s); py::print("operator new called, returning", reinterpret_cast(p)); @@ -324,8 +339,8 @@ TEST_SUBMODULE(factory_constructors, m) { // Class that has verbose operator_new/operator_delete calls struct NoisyAlloc { NoisyAlloc(const NoisyAlloc &) = default; - NoisyAlloc(int i) { py::print(py::str("NoisyAlloc(int {})").format(i)); } - NoisyAlloc(double d) { py::print(py::str("NoisyAlloc(double {})").format(d)); } + explicit NoisyAlloc(int i) { py::print(py::str("NoisyAlloc(int {})").format(i)); } + explicit NoisyAlloc(double d) { py::print(py::str("NoisyAlloc(double {})").format(d)); } ~NoisyAlloc() { py::print("~NoisyAlloc()"); } static void *operator new(size_t s) { py::print("noisy new"); return ::operator new(s); } diff --git a/tests/test_local_bindings.cpp b/tests/test_local_bindings.cpp index 8d6e33f7..a5808e2f 100644 --- a/tests/test_local_bindings.cpp +++ b/tests/test_local_bindings.cpp @@ -91,7 +91,7 @@ TEST_SUBMODULE(local_bindings, m) { class Cat : public pets::Pet { public: - Cat(std::string name) : Pet(std::move(name)) {} + explicit Cat(std::string name) : Pet(std::move(name)) {} }; py::class_(m, "Pet", py::module_local()) .def("get_name", &pets::Pet::name); diff --git a/tests/test_methods_and_attributes.cpp b/tests/test_methods_and_attributes.cpp index 4cf6f08b..2d303a44 100644 --- a/tests/test_methods_and_attributes.cpp +++ b/tests/test_methods_and_attributes.cpp @@ -19,9 +19,9 @@ using overload_cast_ = pybind11::detail::overload_cast_impl; class ExampleMandA { public: ExampleMandA() { print_default_created(this); } - ExampleMandA(int value) : value(value) { print_created(this, value); } + explicit ExampleMandA(int value) : value(value) { print_created(this, value); } ExampleMandA(const ExampleMandA &e) : value(e.value) { print_copy_created(this); } - ExampleMandA(std::string&&) {} + explicit ExampleMandA(std::string &&) {} ExampleMandA(ExampleMandA &&e) noexcept : value(e.value) { print_move_created(this); } ~ExampleMandA() { print_destroyed(this); } @@ -124,14 +124,14 @@ class NoneCastTester { public: int answer = -1; NoneCastTester() = default; - NoneCastTester(int v) : answer(v) {} + explicit NoneCastTester(int v) : answer(v) {} }; struct StrIssue { int val = -1; StrIssue() = default; - StrIssue(int i) : val{i} {} + explicit StrIssue(int i) : val{i} {} }; // Issues #854, #910: incompatible function args when member function/pointer is in unregistered base class diff --git a/tests/test_modules.cpp b/tests/test_modules.cpp index 58671354..ce61c1a2 100644 --- a/tests/test_modules.cpp +++ b/tests/test_modules.cpp @@ -20,7 +20,7 @@ TEST_SUBMODULE(modules, m) { // test_reference_internal class A { public: - A(int v) : v(v) { print_created(this, v); } + explicit A(int v) : v(v) { print_created(this, v); } ~A() { print_destroyed(this); } A(const A&) { print_copy_created(this); } A& operator=(const A ©) { print_copy_assigned(this); v = copy.v; return *this; } diff --git a/tests/test_multiple_inheritance.cpp b/tests/test_multiple_inheritance.cpp index b5ca298d..6963197a 100644 --- a/tests/test_multiple_inheritance.cpp +++ b/tests/test_multiple_inheritance.cpp @@ -16,7 +16,7 @@ namespace { // Many bases for testing that multiple inheritance from many classes (i.e. requiring extra // space for holder constructed flags) works. template struct BaseN { - BaseN(int i) : i(i) { } + explicit BaseN(int i) : i(i) {} int i; }; @@ -47,12 +47,12 @@ int VanillaStaticMix2::static_value = 12; // test_multiple_inheritance_virtbase struct Base1a { - Base1a(int i) : i(i) { } + explicit Base1a(int i) : i(i) {} int foo() const { return i; } int i; }; struct Base2a { - Base2a(int i) : i(i) { } + explicit Base2a(int i) : i(i) {} int bar() const { return i; } int i; }; @@ -77,7 +77,7 @@ TEST_SUBMODULE(multiple_inheritance, m) { // test_multiple_inheritance_mix1 // test_multiple_inheritance_mix2 struct Base1 { - Base1(int i) : i(i) { } + explicit Base1(int i) : i(i) {} int foo() const { return i; } int i; }; @@ -86,7 +86,7 @@ TEST_SUBMODULE(multiple_inheritance, m) { .def("foo", &Base1::foo); struct Base2 { - Base2(int i) : i(i) { } + explicit Base2(int i) : i(i) {} int bar() const { return i; } int i; }; diff --git a/tests/test_numpy_vectorize.cpp b/tests/test_numpy_vectorize.cpp index b08a9f7e..eb5281fb 100644 --- a/tests/test_numpy_vectorize.cpp +++ b/tests/test_numpy_vectorize.cpp @@ -52,7 +52,7 @@ TEST_SUBMODULE(numpy_vectorize, m) { // Passthrough test: references and non-pod types should be automatically passed through (in the // function definition below, only `b`, `d`, and `g` are vectorized): struct NonPODClass { - NonPODClass(int v) : value{v} {} + explicit NonPODClass(int v) : value{v} {} int value; }; py::class_(m, "NonPODClass") @@ -71,7 +71,7 @@ TEST_SUBMODULE(numpy_vectorize, m) { // test_method_vectorization struct VectorizeTestClass { - VectorizeTestClass(int v) : value{v} {}; + explicit VectorizeTestClass(int v) : value{v} {}; float method(int x, float y) const { return y + (float) (x + value); } int value = 0; }; diff --git a/tests/test_pickling.cpp b/tests/test_pickling.cpp index 0d582731..b77636dd 100644 --- a/tests/test_pickling.cpp +++ b/tests/test_pickling.cpp @@ -67,7 +67,7 @@ TEST_SUBMODULE(pickling, m) { // test_roundtrip class Pickleable { public: - Pickleable(const std::string &value) : m_value(value) { } + explicit Pickleable(const std::string &value) : m_value(value) { } const std::string &value() const { return m_value; } void setExtra1(int extra1) { m_extra1 = extra1; } @@ -132,7 +132,7 @@ TEST_SUBMODULE(pickling, m) { // test_roundtrip_with_dict class PickleableWithDict { public: - PickleableWithDict(const std::string &value) : value(value) { } + explicit PickleableWithDict(const std::string &value) : value(value) { } std::string value; int extra; diff --git a/tests/test_sequences_and_iterators.cpp b/tests/test_sequences_and_iterators.cpp index d49fb1f4..b07fd197 100644 --- a/tests/test_sequences_and_iterators.cpp +++ b/tests/test_sequences_and_iterators.cpp @@ -20,7 +20,7 @@ template class NonZeroIterator { const T* ptr_; public: - NonZeroIterator(const T* ptr) : ptr_(ptr) {} + explicit NonZeroIterator(const T *ptr) : ptr_(ptr) {} const T& operator*() const { return *ptr_; } NonZeroIterator& operator++() { ++ptr_; return *this; } }; @@ -77,9 +77,9 @@ TEST_SUBMODULE(sequences_and_iterators, m) { // test_sliceable class Sliceable{ public: - Sliceable(int n): size(n) {} - int start,stop,step; - int size; + explicit Sliceable(int n) : size(n) {} + int start, stop, step; + int size; }; py::class_(m, "Sliceable") .def(py::init()) @@ -96,12 +96,12 @@ TEST_SUBMODULE(sequences_and_iterators, m) { // test_sequence class Sequence { public: - Sequence(size_t size) : m_size(size) { + explicit Sequence(size_t size) : m_size(size) { print_created(this, "of size", m_size); m_data = new float[size]; memset(m_data, 0, sizeof(float) * size); } - Sequence(const std::vector &value) : m_size(value.size()) { + explicit Sequence(const std::vector &value) : m_size(value.size()) { print_created(this, "of size", m_size, "from std::vector"); m_data = new float[m_size]; memcpy(m_data, &value[0], sizeof(float) * m_size); @@ -239,7 +239,7 @@ TEST_SUBMODULE(sequences_and_iterators, m) { class StringMap { public: StringMap() = default; - StringMap(std::unordered_map init) + explicit StringMap(std::unordered_map init) : map(std::move(init)) {} void set(const std::string &key, std::string val) { map[key] = std::move(val); } @@ -276,7 +276,7 @@ TEST_SUBMODULE(sequences_and_iterators, m) { // test_generalized_iterators class IntPairs { public: - IntPairs(std::vector> data) : data_(std::move(data)) {} + explicit IntPairs(std::vector> data) : data_(std::move(data)) {} const std::pair* begin() const { return data_.data(); } private: std::vector> data_; diff --git a/tests/test_smart_ptr.cpp b/tests/test_smart_ptr.cpp index eeaa4414..94f04330 100644 --- a/tests/test_smart_ptr.cpp +++ b/tests/test_smart_ptr.cpp @@ -24,7 +24,7 @@ template class huge_unique_ptr { std::unique_ptr ptr; uint64_t padding[10]; public: - huge_unique_ptr(T *p) : ptr(p) {} + explicit huge_unique_ptr(T *p) : ptr(p) {} T *get() { return ptr.get(); } }; @@ -33,7 +33,7 @@ template class custom_unique_ptr { std::unique_ptr impl; public: - custom_unique_ptr(T* p) : impl(p) { } + explicit custom_unique_ptr(T *p) : impl(p) {} T* get() const { return impl.get(); } T* release_ptr() { return impl.release(); } }; @@ -46,7 +46,7 @@ class shared_ptr_with_addressof_operator { std::shared_ptr impl; public: shared_ptr_with_addressof_operator( ) = default; - shared_ptr_with_addressof_operator(T* p) : impl(p) { } + explicit shared_ptr_with_addressof_operator(T *p) : impl(p) {} T* get() const { return impl.get(); } T** operator&() { throw std::logic_error("Call of overloaded operator& is not expected"); } }; @@ -59,7 +59,7 @@ class unique_ptr_with_addressof_operator { std::unique_ptr impl; public: unique_ptr_with_addressof_operator() = default; - unique_ptr_with_addressof_operator(T* p) : impl(p) { } + explicit unique_ptr_with_addressof_operator(T *p) : impl(p) {} T* get() const { return impl.get(); } T* release_ptr() { return impl.release(); } T** operator&() { throw std::logic_error("Call of overloaded operator& is not expected"); } @@ -68,7 +68,7 @@ public: // Custom object with builtin reference counting (see 'object.h' for the implementation) class MyObject1 : public Object { public: - MyObject1(int value) : value(value) { print_created(this, toString()); } + explicit MyObject1(int value) : value(value) { print_created(this, toString()); } std::string toString() const override { return "MyObject1[" + std::to_string(value) + "]"; } protected: ~MyObject1() override { print_destroyed(this); } @@ -80,7 +80,7 @@ private: class MyObject2 { public: MyObject2(const MyObject2 &) = default; - MyObject2(int value) : value(value) { print_created(this, toString()); } + explicit MyObject2(int value) : value(value) { print_created(this, toString()); } std::string toString() const { return "MyObject2[" + std::to_string(value) + "]"; } virtual ~MyObject2() { print_destroyed(this); } private: @@ -91,7 +91,7 @@ private: class MyObject3 : public std::enable_shared_from_this { public: MyObject3(const MyObject3 &) = default; - MyObject3(int value) : value(value) { print_created(this, toString()); } + explicit MyObject3(int value) : value(value) { print_created(this, toString()); } std::string toString() const { return "MyObject3[" + std::to_string(value) + "]"; } virtual ~MyObject3() { print_destroyed(this); } private: @@ -104,7 +104,7 @@ class MyObject4; std::unordered_set myobject4_instances; class MyObject4 { public: - MyObject4(int value) : value{value} { + explicit MyObject4(int value) : value{value} { print_created(this); myobject4_instances.insert(this); } @@ -130,7 +130,7 @@ class MyObject4a; std::unordered_set myobject4a_instances; class MyObject4a { public: - MyObject4a(int i) { + explicit MyObject4a(int i) { value = i; print_created(this); myobject4a_instances.insert(this); @@ -153,14 +153,14 @@ protected: // Object derived but with public destructor and no Deleter in default holder class MyObject4b : public MyObject4a { public: - MyObject4b(int i) : MyObject4a(i) { print_created(this); } + explicit MyObject4b(int i) : MyObject4a(i) { print_created(this); } ~MyObject4b() override { print_destroyed(this); } }; // test_large_holder class MyObject5 { // managed by huge_unique_ptr public: - MyObject5(int value) : value{value} { print_created(this); } + explicit MyObject5(int value) : value{value} { print_created(this); } ~MyObject5() { print_destroyed(this); } int value; }; @@ -222,7 +222,7 @@ struct TypeForHolderWithAddressOf { // test_move_only_holder_with_addressof_operator struct TypeForMoveOnlyHolderWithAddressOf { - TypeForMoveOnlyHolderWithAddressOf(int value) : value{value} { print_created(this); } + explicit TypeForMoveOnlyHolderWithAddressOf(int value) : value{value} { print_created(this); } ~TypeForMoveOnlyHolderWithAddressOf() { print_destroyed(this); } std::string toString() const { return "MoveOnlyHolderWithAddressOf[" + std::to_string(value) + "]"; @@ -242,7 +242,7 @@ struct ElementBase { }; struct ElementA : ElementBase { - ElementA(int v) : v(v) { } + explicit ElementA(int v) : v(v) {} int value() const { return v; } int v; }; @@ -291,9 +291,9 @@ TEST_SUBMODULE(smart_ptr, m) { py::implicitly_convertible(); m.def("make_object_1", []() -> Object * { return new MyObject1(1); }); - m.def("make_object_2", []() -> ref { return new MyObject1(2); }); + m.def("make_object_2", []() -> ref { return ref(new MyObject1(2)); }); m.def("make_myobject1_1", []() -> MyObject1 * { return new MyObject1(4); }); - m.def("make_myobject1_2", []() -> ref { return new MyObject1(5); }); + m.def("make_myobject1_2", []() -> ref { return ref(new MyObject1(5)); }); m.def("print_object_1", [](const Object *obj) { py::print(obj->toString()); }); m.def("print_object_2", [](ref obj) { py::print(obj->toString()); }); m.def("print_object_3", [](const ref &obj) { py::print(obj->toString()); }); @@ -328,7 +328,7 @@ TEST_SUBMODULE(smart_ptr, m) { // test_smart_ptr_refcounting m.def("test_object1_refcounting", []() { - ref o = new MyObject1(0); + auto o = ref(new MyObject1(0)); bool good = o->getRefCount() == 1; py::object o2 = py::cast(o, py::return_value_policy::reference); // always request (partial) ownership for objects with intrusive diff --git a/tests/test_stl.cpp b/tests/test_stl.cpp index 23e2c07b..7e3363c5 100644 --- a/tests/test_stl.cpp +++ b/tests/test_stl.cpp @@ -45,7 +45,8 @@ PYBIND11_MAKE_OPAQUE(std::vector>); /// Issue #528: templated constructor struct TplCtorClass { - template TplCtorClass(const T &) { } + template + explicit TplCtorClass(const T &) {} bool operator==(const TplCtorClass &) const { return true; } }; diff --git a/tests/test_stl_binders.cpp b/tests/test_stl_binders.cpp index 22847eb7..6b23e352 100644 --- a/tests/test_stl_binders.cpp +++ b/tests/test_stl_binders.cpp @@ -18,7 +18,7 @@ class El { public: El() = delete; - El(int v) : a(v) { } + explicit El(int v) : a(v) {} int a; }; diff --git a/tests/test_tagbased_polymorphic.cpp b/tests/test_tagbased_polymorphic.cpp index 90f40e14..2c7bad8b 100644 --- a/tests/test_tagbased_polymorphic.cpp +++ b/tests/test_tagbased_polymorphic.cpp @@ -37,33 +37,35 @@ struct Animal struct Dog : Animal { - Dog(const std::string& _name, Kind _kind = Kind::Dog) : Animal(_name, _kind) {} + explicit Dog(const std::string &_name, Kind _kind = Kind::Dog) : Animal(_name, _kind) {} std::string bark() const { return name_of_kind(kind) + " " + name + " goes " + sound; } std::string sound = "WOOF!"; }; struct Labrador : Dog { - Labrador(const std::string& _name, int _excitement = 9001) + explicit Labrador(const std::string &_name, int _excitement = 9001) : Dog(_name, Kind::Labrador), excitement(_excitement) {} int excitement; }; struct Chihuahua : Dog { - Chihuahua(const std::string& _name) : Dog(_name, Kind::Chihuahua) { sound = "iyiyiyiyiyi"; } + explicit Chihuahua(const std::string &_name) : Dog(_name, Kind::Chihuahua) { + sound = "iyiyiyiyiyi"; + } std::string bark() const { return Dog::bark() + " and runs in circles"; } }; struct Cat : Animal { - Cat(const std::string& _name, Kind _kind = Kind::Cat) : Animal(_name, _kind) {} + explicit Cat(const std::string &_name, Kind _kind = Kind::Cat) : Animal(_name, _kind) {} std::string purr() const { return "mrowr"; } }; struct Panther : Cat { - Panther(const std::string& _name) : Cat(_name, Kind::Panther) {} + explicit Panther(const std::string &_name) : Cat(_name, Kind::Panther) {} std::string purr() const { return "mrrrRRRRRR"; } }; diff --git a/tests/test_virtual_functions.cpp b/tests/test_virtual_functions.cpp index f83a7364..1eba534d 100644 --- a/tests/test_virtual_functions.cpp +++ b/tests/test_virtual_functions.cpp @@ -15,7 +15,7 @@ /* This is an example class that we'll want to be able to extend from Python */ class ExampleVirt { public: - ExampleVirt(int state) : state(state) { print_created(this, state); } + explicit ExampleVirt(int state) : state(state) { print_created(this, state); } ExampleVirt(const ExampleVirt &e) : state(e.state) { print_copy_created(this); } ExampleVirt(ExampleVirt &&e) noexcept : state(e.state) { print_move_created(this); From 4c6bee3514679cde3b506d5552a3a11fd09cc6d5 Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Thu, 9 Sep 2021 14:06:33 -0400 Subject: [PATCH 061/100] fix: Set __file__ constant when using eval_file (#1300) (#3233) * Set __file__ constant when using eval_file * Use const ref * Use a move instead * Revert * Improve test * Guard test with Python version * Fix tests * Dont support Python2 API * Drop Python2 eval __file__ support * Hack * Semisupport Python2 * Take2 * Remove Python2 support --- include/pybind11/eval.h | 9 +++++++++ tests/test_cmake_build/test.py | 3 +++ 2 files changed, 12 insertions(+) diff --git a/include/pybind11/eval.h b/include/pybind11/eval.h index 33fcdc09..e0f58bcf 100644 --- a/include/pybind11/eval.h +++ b/include/pybind11/eval.h @@ -136,6 +136,15 @@ object eval_file(str fname, object global = globals(), object local = object()) pybind11_fail("File \"" + fname_str + "\" could not be opened!"); } + // In Python2, this should be encoded by getfilesystemencoding. + // We don't boher setting it since Python2 is past EOL anyway. + // See PR#3233 +#if PY_VERSION_HEX >= 0x03000000 + if (!global.contains("__file__")) { + global["__file__"] = std::move(fname); + } +#endif + #if PY_VERSION_HEX < 0x03000000 && defined(PYPY_VERSION) PyObject *result = PyRun_File(f, fname_str.c_str(), start, global.ptr(), local.ptr()); diff --git a/tests/test_cmake_build/test.py b/tests/test_cmake_build/test.py index d1a290dc..972a27be 100644 --- a/tests/test_cmake_build/test.py +++ b/tests/test_cmake_build/test.py @@ -3,5 +3,8 @@ import sys import test_cmake_build +if str is not bytes: # If not Python2 + assert isinstance(__file__, str) # Test this is properly set + assert test_cmake_build.add(1, 2) == 3 print("{} imports, runs, and adds: 1 + 2 = 3".format(sys.argv[1])) From 4d5ad03e1fd09b8be26843f7cb165051ab9a9c05 Mon Sep 17 00:00:00 2001 From: Jeremy Maitin-Shepard Date: Thu, 9 Sep 2021 12:56:10 -0700 Subject: [PATCH 062/100] Avoid use of temporary `bytes` object in string_caster for UTF-8 (#3257) Fixes #3252 --- include/pybind11/cast.h | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/include/pybind11/cast.h b/include/pybind11/cast.h index db79f57c..3e621eba 100644 --- a/include/pybind11/cast.h +++ b/include/pybind11/cast.h @@ -377,6 +377,22 @@ template struct string_caster { #endif } +#if PY_VERSION_HEX >= 0x03030000 + // On Python >= 3.3, for UTF-8 we avoid the need for a temporary `bytes` + // object by using `PyUnicode_AsUTF8AndSize`. + if (PYBIND11_SILENCE_MSVC_C4127(UTF_N == 8)) { + Py_ssize_t size = -1; + const auto *buffer + = reinterpret_cast(PyUnicode_AsUTF8AndSize(load_src.ptr(), &size)); + if (!buffer) { + PyErr_Clear(); + return false; + } + value = StringType(buffer, static_cast(size)); + return true; + } +#endif + auto utfNbytes = reinterpret_steal(PyUnicode_AsEncodedString( load_src.ptr(), UTF_N == 8 ? "utf-8" : UTF_N == 16 ? "utf-16" : "utf-32", nullptr)); if (!utfNbytes) { PyErr_Clear(); return false; } From ae07d4c6c663f8cc2590d28ecd6baa09b055537f Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Fri, 10 Sep 2021 00:27:36 -0400 Subject: [PATCH 063/100] maint(Clang-Tidy): readability-const-return (#3254) * Enable clang-tidy readability-const-return * PyTest functional * Fix regression * Fix actual regression * Remove one more NOLINT * Update comment --- .clang-tidy | 3 ++- include/pybind11/cast.h | 7 +++++-- include/pybind11/detail/descr.h | 3 ++- include/pybind11/pybind11.h | 4 ++-- include/pybind11/pytypes.h | 6 +++--- tests/test_eigen.cpp | 4 ++++ 6 files changed, 18 insertions(+), 9 deletions(-) diff --git a/.clang-tidy b/.clang-tidy index a6437dd4..dbe85a8b 100644 --- a/.clang-tidy +++ b/.clang-tidy @@ -31,9 +31,10 @@ modernize-use-override, modernize-use-using, *performance*, readability-avoid-const-params-in-decls, +readability-const-return-type, readability-container-size-empty, -readability-else-after-return, readability-delete-null-pointer, +readability-else-after-return, readability-implicit-bool-conversion, readability-make-member-function-const, readability-misplaced-array-index, diff --git a/include/pybind11/cast.h b/include/pybind11/cast.h index 3e621eba..17b10157 100644 --- a/include/pybind11/cast.h +++ b/include/pybind11/cast.h @@ -1194,13 +1194,15 @@ public: } template + // NOLINTNEXTLINE(readability-const-return-type) enable_if_t::value, Return> call(Func &&f) && { - return std::move(*this).template call_impl(std::forward(f), indices{}, Guard{}); + return std::move(*this).template call_impl>(std::forward(f), indices{}, Guard{}); } template + // NOLINTNEXTLINE(readability-const-return-type) enable_if_t::value, void_type> call(Func &&f) && { - std::move(*this).template call_impl(std::forward(f), indices{}, Guard{}); + std::move(*this).template call_impl>(std::forward(f), indices{}, Guard{}); return void_type(); } @@ -1222,6 +1224,7 @@ private: } template + // NOLINTNEXTLINE(readability-const-return-type) Return call_impl(Func &&f, index_sequence, Guard &&) && { return std::forward(f)(cast_op(std::move(std::get(argcasters)))...); } diff --git a/include/pybind11/detail/descr.h b/include/pybind11/detail/descr.h index 0b498e5e..c62e541b 100644 --- a/include/pybind11/detail/descr.h +++ b/include/pybind11/detail/descr.h @@ -77,7 +77,8 @@ constexpr enable_if_t _(const T1 &d, const T2 &) { return d; } template constexpr enable_if_t _(const T1 &, const T2 &d) { return d; } -template auto constexpr _() -> decltype(int_to_str::digits) { +template +auto constexpr _() -> remove_cv_t::digits)> { return int_to_str::digits; } diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 82d33975..c4e5ea6e 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -2002,13 +2002,13 @@ template ()).first), #endif typename... Extra> -iterator make_key_iterator(Iterator first, Sentinel last, Extra &&... extra) { +iterator make_key_iterator(Iterator first, Sentinel last, Extra &&...extra) { using state = detail::iterator_state; if (!detail::get_type_info(typeid(state), false)) { class_(handle(), "iterator", pybind11::module_local()) .def("__iter__", [](state &s) -> state& { return s; }) - .def("__next__", [](state &s) -> KeyType { + .def("__next__", [](state &s) -> detail::remove_cv_t { if (!s.first_or_done) ++s.it; else diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index 971db85f..2de3f5f1 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -773,7 +773,7 @@ class sequence_fast_readonly { protected: using iterator_category = std::random_access_iterator_tag; using value_type = handle; - using reference = const handle; + using reference = handle; using pointer = arrow_proxy; sequence_fast_readonly(handle obj, ssize_t n) : ptr(PySequence_Fast_ITEMS(obj.ptr()) + n) { } @@ -816,7 +816,7 @@ class dict_readonly { protected: using iterator_category = std::forward_iterator_tag; using value_type = std::pair; - using reference = const value_type; + using reference = value_type; using pointer = arrow_proxy; dict_readonly() = default; @@ -966,7 +966,7 @@ public: using iterator_category = std::input_iterator_tag; using difference_type = ssize_t; using value_type = handle; - using reference = const handle; + using reference = handle; using pointer = const handle *; PYBIND11_OBJECT_DEFAULT(iterator, object, PyIter_Check) diff --git a/tests/test_eigen.cpp b/tests/test_eigen.cpp index 651be057..ad572b2a 100644 --- a/tests/test_eigen.cpp +++ b/tests/test_eigen.cpp @@ -178,6 +178,7 @@ TEST_SUBMODULE(eigen, m) { ReturnTester() { print_created(this); } ~ReturnTester() { print_destroyed(this); } static Eigen::MatrixXd create() { return Eigen::MatrixXd::Ones(10, 10); } + // NOLINTNEXTLINE(readability-const-return-type) static const Eigen::MatrixXd createConst() { return Eigen::MatrixXd::Ones(10, 10); } Eigen::MatrixXd &get() { return mat; } Eigen::MatrixXd *getPtr() { return &mat; } @@ -244,6 +245,9 @@ TEST_SUBMODULE(eigen, m) { // test_fixed, and various other tests m.def("fixed_r", [mat]() -> FixedMatrixR { return FixedMatrixR(mat); }); + // Our Eigen does a hack which respects constness through the numpy writeable flag. + // Therefore, the const return actually affects this type despite being an rvalue. + // NOLINTNEXTLINE(readability-const-return-type) m.def("fixed_r_const", [mat]() -> const FixedMatrixR { return FixedMatrixR(mat); }); m.def("fixed_c", [mat]() -> FixedMatrixC { return FixedMatrixC(mat); }); m.def("fixed_copy_r", [](const FixedMatrixR &m) -> FixedMatrixR { return m; }); From 121b91f99c9044e12ea1f746d7f3e586c41529c8 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Fri, 10 Sep 2021 07:16:09 -0700 Subject: [PATCH 064/100] Fixing NOLINT mishap (#3260) * Removing NOLINT pointed out by Aaron. * Removing another NOLINT. --- include/pybind11/cast.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/include/pybind11/cast.h b/include/pybind11/cast.h index 17b10157..1ec2080f 100644 --- a/include/pybind11/cast.h +++ b/include/pybind11/cast.h @@ -1200,7 +1200,6 @@ public: } template - // NOLINTNEXTLINE(readability-const-return-type) enable_if_t::value, void_type> call(Func &&f) && { std::move(*this).template call_impl>(std::forward(f), indices{}, Guard{}); return void_type(); @@ -1224,7 +1223,6 @@ private: } template - // NOLINTNEXTLINE(readability-const-return-type) Return call_impl(Func &&f, index_sequence, Guard &&) && { return std::forward(f)(cast_op(std::move(std::get(argcasters)))...); } From 0e599589fe821f86d18635c13636f3042d4c06b9 Mon Sep 17 00:00:00 2001 From: Laramie Leavitt Date: Fri, 10 Sep 2021 09:29:21 -0700 Subject: [PATCH 065/100] Fix thread safety for pybind11 loader_life_support (#3237) * Fix thread safety for pybind11 loader_life_support Fixes issue: https://github.com/pybind/pybind11/issues/2765 This converts the vector of PyObjects to either a single void* or a per-thread void* depending on the WITH_THREAD define. The new field is used by each thread to construct a stack of loader_life_support frames that can extend the life of python objects. The pointer is updated when the loader_life_support object is allocated (which happens before a call) as well as on release. Each loader_life_support maintains a set of PyObject references that need to be lifetime extended; this is done by storing them in a c++ std::unordered_set and clearing the references when the method completes. * Also update the internals version as the internal struct is no longer compatible * Add test demonstrating threading works correctly. It may be appropriate to run this under msan/tsan/etc. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update test to use lifetime-extended references rather than std::string_view, as that's a C++ 17 feature. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make loader_life_support members private * Update version to dev2 * Update test to use python threading rather than concurrent.futures * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove unnecessary env in test * Remove unnecessary pytest in test * Use native C++ thread_local in place of python per-thread data structures to retain compatability * clang-format test_thread.cpp * Add a note about debugging the py::cast() error * thread_test.py now propagates exceptions on join() calls. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove unused sys / merge * Update include order in test_thread.cpp * Remove spurious whitespace * Update comment / whitespace. * Address review comments * lint cleanup * Fix test IntStruct constructor. * Add explicit to constructor Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Aaron Gokaslan --- include/pybind11/detail/common.h | 4 +- include/pybind11/detail/internals.h | 6 +- include/pybind11/detail/type_caster_base.h | 55 ++++++++++-------- pybind11/_version.py | 2 +- tests/CMakeLists.txt | 1 + tests/test_thread.cpp | 66 ++++++++++++++++++++++ tests/test_thread.py | 44 +++++++++++++++ 7 files changed, 148 insertions(+), 30 deletions(-) create mode 100644 tests/test_thread.cpp create mode 100644 tests/test_thread.py diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index a8fd7d91..8aeb79fb 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -11,11 +11,11 @@ #define PYBIND11_VERSION_MAJOR 2 #define PYBIND11_VERSION_MINOR 8 -#define PYBIND11_VERSION_PATCH 0.dev1 +#define PYBIND11_VERSION_PATCH 0.dev2 // Similar to Python's convention: https://docs.python.org/3/c-api/apiabiversion.html // Additional convention: 0xD = dev -#define PYBIND11_VERSION_HEX 0x020800D1 +#define PYBIND11_VERSION_HEX 0x020800D2 #define PYBIND11_NAMESPACE_BEGIN(name) namespace name { #define PYBIND11_NAMESPACE_END(name) } diff --git a/include/pybind11/detail/internals.h b/include/pybind11/detail/internals.h index b177801a..39c28e77 100644 --- a/include/pybind11/detail/internals.h +++ b/include/pybind11/detail/internals.h @@ -106,7 +106,7 @@ struct internals { std::unordered_map> patients; std::forward_list registered_exception_translators; std::unordered_map shared_data; // Custom data to be shared across extensions - std::vector loader_patient_stack; // Used by `loader_life_support` + std::vector unused_loader_patient_stack_remove_at_v5; std::forward_list static_strings; // Stores the std::strings backing detail::c_str() PyTypeObject *static_property_type; PyTypeObject *default_metaclass; @@ -298,12 +298,12 @@ PYBIND11_NOINLINE internals &get_internals() { #if PY_VERSION_HEX >= 0x03070000 internals_ptr->tstate = PyThread_tss_alloc(); if (!internals_ptr->tstate || (PyThread_tss_create(internals_ptr->tstate) != 0)) - pybind11_fail("get_internals: could not successfully initialize the TSS key!"); + pybind11_fail("get_internals: could not successfully initialize the tstate TSS key!"); PyThread_tss_set(internals_ptr->tstate, tstate); #else internals_ptr->tstate = PyThread_create_key(); if (internals_ptr->tstate == -1) - pybind11_fail("get_internals: could not successfully initialize the TLS key!"); + pybind11_fail("get_internals: could not successfully initialize the tstate TLS key!"); PyThread_set_key_value(internals_ptr->tstate, tstate); #endif internals_ptr->istate = tstate->interp; diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h index 4c9ac7b8..fa8433cf 100644 --- a/include/pybind11/detail/type_caster_base.h +++ b/include/pybind11/detail/type_caster_base.h @@ -31,47 +31,54 @@ PYBIND11_NAMESPACE_BEGIN(detail) /// A life support system for temporary objects created by `type_caster::load()`. /// Adding a patient will keep it alive up until the enclosing function returns. class loader_life_support { +private: + loader_life_support* parent = nullptr; + std::unordered_set keep_alive; + + static loader_life_support** get_stack_pp() { +#if defined(WITH_THREAD) + thread_local static loader_life_support* per_thread_stack = nullptr; + return &per_thread_stack; +#else + static loader_life_support* global_stack = nullptr; + return &global_stack; +#endif + } + public: /// A new patient frame is created when a function is entered loader_life_support() { - get_internals().loader_patient_stack.push_back(nullptr); + loader_life_support** stack = get_stack_pp(); + parent = *stack; + *stack = this; } /// ... and destroyed after it returns ~loader_life_support() { - auto &stack = get_internals().loader_patient_stack; - if (stack.empty()) + loader_life_support** stack = get_stack_pp(); + if (*stack != this) pybind11_fail("loader_life_support: internal error"); - - auto ptr = stack.back(); - stack.pop_back(); - Py_CLEAR(ptr); - - // A heuristic to reduce the stack's capacity (e.g. after long recursive calls) - if (stack.capacity() > 16 && !stack.empty() && stack.capacity() / stack.size() > 2) - stack.shrink_to_fit(); + *stack = parent; + for (auto* item : keep_alive) + Py_DECREF(item); } /// This can only be used inside a pybind11-bound function, either by `argument_loader` /// at argument preparation time or by `py::cast()` at execution time. PYBIND11_NOINLINE static void add_patient(handle h) { - auto &stack = get_internals().loader_patient_stack; - if (stack.empty()) + loader_life_support* frame = *get_stack_pp(); + if (!frame) { + // NOTE: It would be nice to include the stack frames here, as this indicates + // use of pybind11::cast<> outside the normal call framework, finding such + // a location is challenging. Developers could consider printing out + // stack frame addresses here using something like __builtin_frame_address(0) throw cast_error("When called outside a bound function, py::cast() cannot " "do Python -> C++ conversions which require the creation " "of temporary values"); - - auto &list_ptr = stack.back(); - if (list_ptr == nullptr) { - list_ptr = PyList_New(1); - if (!list_ptr) - pybind11_fail("loader_life_support: error allocating list"); - PyList_SET_ITEM(list_ptr, 0, h.inc_ref().ptr()); - } else { - auto result = PyList_Append(list_ptr, h.ptr()); - if (result == -1) - pybind11_fail("loader_life_support: error adding patient"); } + + if (frame->keep_alive.insert(h.ptr()).second) + Py_INCREF(h.ptr()); } }; diff --git a/pybind11/_version.py b/pybind11/_version.py index 610d39bf..d212f1df 100644 --- a/pybind11/_version.py +++ b/pybind11/_version.py @@ -8,5 +8,5 @@ def _to_int(s): return s -__version__ = "2.8.0.dev1" +__version__ = "2.8.0.dev2" version_info = tuple(_to_int(s) for s in __version__.split(".")) diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index d71a51e6..f014771d 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -129,6 +129,7 @@ set(PYBIND11_TEST_FILES test_stl.cpp test_stl_binders.cpp test_tagbased_polymorphic.cpp + test_thread.cpp test_union.cpp test_virtual_functions.cpp) diff --git a/tests/test_thread.cpp b/tests/test_thread.cpp new file mode 100644 index 00000000..19d91768 --- /dev/null +++ b/tests/test_thread.cpp @@ -0,0 +1,66 @@ +/* + tests/test_thread.cpp -- call pybind11 bound methods in threads + + Copyright (c) 2021 Laramie Leavitt (Google LLC) + + All rights reserved. Use of this source code is governed by a + BSD-style license that can be found in the LICENSE file. +*/ + +#include +#include + +#include +#include + +#include "pybind11_tests.h" + +namespace py = pybind11; + +namespace { + +struct IntStruct { + explicit IntStruct(int v) : value(v) {}; + ~IntStruct() { value = -value; } + IntStruct(const IntStruct&) = default; + IntStruct& operator=(const IntStruct&) = default; + + int value; +}; + +} // namespace + +TEST_SUBMODULE(thread, m) { + + py::class_(m, "IntStruct").def(py::init([](const int i) { return IntStruct(i); })); + + // implicitly_convertible uses loader_life_support when an implicit + // conversion is required in order to lifetime extend the reference. + // + // This test should be run with ASAN for better effectiveness. + py::implicitly_convertible(); + + m.def("test", [](int expected, const IntStruct &in) { + { + py::gil_scoped_release release; + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + } + + if (in.value != expected) { + throw std::runtime_error("Value changed!!"); + } + }); + + m.def( + "test_no_gil", + [](int expected, const IntStruct &in) { + std::this_thread::sleep_for(std::chrono::milliseconds(5)); + if (in.value != expected) { + throw std::runtime_error("Value changed!!"); + } + }, + py::call_guard()); + + // NOTE: std::string_view also uses loader_life_support to ensure that + // the string contents remain alive, but that's a C++ 17 feature. +} diff --git a/tests/test_thread.py b/tests/test_thread.py new file mode 100644 index 00000000..f9db1bab --- /dev/null +++ b/tests/test_thread.py @@ -0,0 +1,44 @@ +# -*- coding: utf-8 -*- + +import threading + +from pybind11_tests import thread as m + + +class Thread(threading.Thread): + def __init__(self, fn): + super(Thread, self).__init__() + self.fn = fn + self.e = None + + def run(self): + try: + for i in range(10): + self.fn(i, i) + except Exception as e: + self.e = e + + def join(self): + super(Thread, self).join() + if self.e: + raise self.e + + +def test_implicit_conversion(): + a = Thread(m.test) + b = Thread(m.test) + c = Thread(m.test) + for x in [a, b, c]: + x.start() + for x in [c, b, a]: + x.join() + + +def test_implicit_conversion_no_gil(): + a = Thread(m.test_no_gil) + b = Thread(m.test_no_gil) + c = Thread(m.test_no_gil) + for x in [a, b, c]: + x.start() + for x in [c, b, a]: + x.join() From 9978ed588baa2cf804794b8ee9f765a316ab16e2 Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Fri, 10 Sep 2021 14:23:32 -0400 Subject: [PATCH 066/100] Fix capsule bug (#3261) Thanks Aaron for jumping in fixing this! --- include/pybind11/pytypes.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index 2de3f5f1..a1cf6bef 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -1414,14 +1414,19 @@ public: T* get_pointer() const { auto name = this->name(); T *result = static_cast(PyCapsule_GetPointer(m_ptr, name)); - if (!result) pybind11_fail("Unable to extract capsule contents!"); + if (!result) { + PyErr_Clear(); + pybind11_fail("Unable to extract capsule contents!"); + } return result; } /// Replaces a capsule's pointer *without* calling the destructor on the existing one. void set_pointer(const void *value) { - if (PyCapsule_SetPointer(m_ptr, const_cast(value)) != 0) + if (PyCapsule_SetPointer(m_ptr, const_cast(value)) != 0) { + PyErr_Clear(); pybind11_fail("Could not set capsule pointer"); + } } const char *name() const { return PyCapsule_GetName(m_ptr); } From 6c65ab5950a75271bddced3f4238fff2c9a427a5 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Sun, 12 Sep 2021 19:53:26 -0700 Subject: [PATCH 067/100] Follow-on to PR #3254, to address user code breakages. (#3263) * Restoring `const` removed from pytypes.h in PR #3254, adding tests reflective of user code that breaks when those `const` are removed. * clang-tidy NOLINTs (and one collateral fix). * Inserting PYBIND11_CONST_FOR_STRICT_PLATFORMS * Trying `defined(__APPLE__)` * Trying again: `auto it` for strict platforms. * Adding NOLINTNEXTLINE(bugprone-macro-parentheses), expanding comments. * Labeling all changes with `PR #3263`, for easy reference, and to make it easy to undo these changes if we decide to do so in the future. --- include/pybind11/pybind11.h | 1 + include/pybind11/pytypes.h | 11 +++++++--- tests/test_pytypes.cpp | 43 +++++++++++++++++++++++++++++++++++++ tests/test_pytypes.py | 6 ++++++ 4 files changed, 58 insertions(+), 3 deletions(-) diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index c4e5ea6e..07d2d043 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -1987,6 +1987,7 @@ iterator make_iterator(Iterator first, Sentinel last, Extra &&... extra) { throw stop_iteration(); } return *s.it; + // NOLINTNEXTLINE(readability-const-return-type) // PR #3263 }, std::forward(extra)..., Policy); } diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index a1cf6bef..d1d3dcb0 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -733,7 +733,9 @@ public: generic_iterator() = default; generic_iterator(handle seq, ssize_t index) : Policy(seq, index) { } + // NOLINTNEXTLINE(readability-const-return-type) // PR #3263 reference operator*() const { return Policy::dereference(); } + // NOLINTNEXTLINE(readability-const-return-type) // PR #3263 reference operator[](difference_type n) const { return *(*this + n); } pointer operator->() const { return **this; } @@ -773,11 +775,12 @@ class sequence_fast_readonly { protected: using iterator_category = std::random_access_iterator_tag; using value_type = handle; - using reference = handle; + using reference = const handle; // PR #3263 using pointer = arrow_proxy; sequence_fast_readonly(handle obj, ssize_t n) : ptr(PySequence_Fast_ITEMS(obj.ptr()) + n) { } + // NOLINTNEXTLINE(readability-const-return-type) // PR #3263 reference dereference() const { return *ptr; } void increment() { ++ptr; } void decrement() { --ptr; } @@ -816,12 +819,13 @@ class dict_readonly { protected: using iterator_category = std::forward_iterator_tag; using value_type = std::pair; - using reference = value_type; + using reference = const value_type; // PR #3263 using pointer = arrow_proxy; dict_readonly() = default; dict_readonly(handle obj, ssize_t pos) : obj(obj), pos(pos) { increment(); } + // NOLINTNEXTLINE(readability-const-return-type) // PR #3263 reference dereference() const { return {key, value}; } void increment() { if (PyDict_Next(obj.ptr(), &pos, &key, &value) == 0) { @@ -966,7 +970,7 @@ public: using iterator_category = std::input_iterator_tag; using difference_type = ssize_t; using value_type = handle; - using reference = handle; + using reference = const handle; // PR #3263 using pointer = const handle *; PYBIND11_OBJECT_DEFAULT(iterator, object, PyIter_Check) @@ -982,6 +986,7 @@ public: return rv; } + // NOLINTNEXTLINE(readability-const-return-type) // PR #3263 reference operator*() const { if (m_ptr && !value.ptr()) { auto& self = const_cast(*this); diff --git a/tests/test_pytypes.cpp b/tests/test_pytypes.cpp index 96c97351..2157dc09 100644 --- a/tests/test_pytypes.cpp +++ b/tests/test_pytypes.cpp @@ -462,6 +462,49 @@ TEST_SUBMODULE(pytypes, m) { m.def("weakref_from_object_and_function", [](py::object o, py::function f) { return py::weakref(std::move(o), std::move(f)); }); +// See PR #3263 for background (https://github.com/pybind/pybind11/pull/3263): +// pytypes.h could be changed to enforce the "most correct" user code below, by removing +// `const` from iterator `reference` using type aliases, but that will break existing +// user code. +#if (defined(__APPLE__) && defined(__clang__)) || defined(PYPY_VERSION) +// This is "most correct" and enforced on these platforms. +# define PYBIND11_AUTO_IT auto it +#else +// This works on many platforms and is (unfortunately) reflective of existing user code. +// NOLINTNEXTLINE(bugprone-macro-parentheses) +# define PYBIND11_AUTO_IT auto &it +#endif + + m.def("tuple_iterator", []() { + auto tup = py::make_tuple(5, 7); + int tup_sum = 0; + for (PYBIND11_AUTO_IT : tup) { + tup_sum += it.cast(); + } + return tup_sum; + }); + + m.def("dict_iterator", []() { + py::dict dct; + dct[py::int_(3)] = 5; + dct[py::int_(7)] = 11; + int kv_sum = 0; + for (PYBIND11_AUTO_IT : dct) { + kv_sum += it.first.cast() * 100 + it.second.cast(); + } + return kv_sum; + }); + + m.def("passed_iterator", [](const py::iterator &py_it) { + int elem_sum = 0; + for (PYBIND11_AUTO_IT : py_it) { + elem_sum += it.cast(); + } + return elem_sum; + }); + +#undef PYBIND11_AUTO_IT + // Tests below this line are for pybind11 IMPLEMENTATION DETAILS: m.def("sequence_item_get_ssize_t", [](const py::object &o) { diff --git a/tests/test_pytypes.py b/tests/test_pytypes.py index 18847550..070538cc 100644 --- a/tests/test_pytypes.py +++ b/tests/test_pytypes.py @@ -623,6 +623,12 @@ def test_weakref(create_weakref, create_weakref_with_callback): assert callback.called +def test_cpp_iterators(): + assert m.tuple_iterator() == 12 + assert m.dict_iterator() == 305 + 711 + assert m.passed_iterator(iter((-7, 3))) == -4 + + def test_implementation_details(): lst = [39, 43, 92, 49, 22, 29, 93, 98, 26, 57, 8] tup = tuple(lst) From 3ed31e9289acaecb333e851ecb59912c784cdd3d Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 13 Sep 2021 15:58:30 -0400 Subject: [PATCH 068/100] [pre-commit.ci] pre-commit autoupdate (#3266) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit updates: - [github.com/asottile/pyupgrade: v2.25.0 → v2.26.0](https://github.com/asottile/pyupgrade/compare/v2.25.0...v2.26.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .pre-commit-config.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 06e82fe4..fd11cf0c 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -31,7 +31,7 @@ repos: exclude: ^noxfile.py$ - repo: https://github.com/asottile/pyupgrade - rev: v2.25.0 + rev: v2.26.0 hooks: - id: pyupgrade From e0031bfcebc189b978e650cf67620788454453d3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Matthias=20K=C3=B6ppe?= Date: Wed, 15 Sep 2021 10:52:32 -0700 Subject: [PATCH 069/100] include/pybind11/numpy.h: gcc 4.8.4 does not have is_trivially_copyable (#3270) --- include/pybind11/numpy.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/pybind11/numpy.h b/include/pybind11/numpy.h index d5584408..b7747fae 100644 --- a/include/pybind11/numpy.h +++ b/include/pybind11/numpy.h @@ -322,7 +322,7 @@ template using remove_all_extents_t = typename array_info::type; template using is_pod_struct = all_of< std::is_standard_layout, // since we're accessing directly in memory we need a standard layout type -#if defined(__GLIBCXX__) && (__GLIBCXX__ < 20150422 || __GLIBCXX__ == 20150623 || __GLIBCXX__ == 20150626 || __GLIBCXX__ == 20160803) +#if defined(__GLIBCXX__) && (__GLIBCXX__ < 20150422 || __GLIBCXX__ == 20150426 || __GLIBCXX__ == 20150623 || __GLIBCXX__ == 20150626 || __GLIBCXX__ == 20160803) // libstdc++ < 5 (including versions 4.8.5, 4.9.3 and 4.9.4 which were released after 5) // don't implement is_trivially_copyable, so approximate it std::is_trivially_destructible, From 04dd3262f0420d48531a82021fa4895ba14bdcb2 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Fri, 17 Sep 2021 17:28:26 -0400 Subject: [PATCH 070/100] docs: update CHANGELOG (#3276) --- docs/advanced/pycpp/numpy.rst | 39 +++++++++--- docs/changelog.rst | 114 ++++++++++++++++++++++++++++++++++ 2 files changed, 146 insertions(+), 7 deletions(-) diff --git a/docs/advanced/pycpp/numpy.rst b/docs/advanced/pycpp/numpy.rst index 53ec8c1a..cae068e9 100644 --- a/docs/advanced/pycpp/numpy.rst +++ b/docs/advanced/pycpp/numpy.rst @@ -171,6 +171,31 @@ template parameter, and it ensures that non-conforming arguments are converted into an array satisfying the specified requirements instead of trying the next function overload. +There are several methods on arrays; the methods listed below under references +work, as well as the following functions based on the NumPy API: + +- ``.dtype()`` returns the type of the contained values. + +- ``.strides()`` returns a pointer to the strides of the array (optionally pass + an integer axis to get a number). + +- ``.flags()`` returns the flag settings. ``.writable()`` and ``.owndata()`` + are directly available. + +- ``.offset_at()`` returns the offset (optionally pass indices). + +- ``.squeeze()`` returns a view with length-1 axes removed. + +- ``.view(dtype)`` returns a view of the array with a different dtype. + +- ``.reshape({i, j, ...})`` returns a view of the array with a different shape. + ``.resize({...})`` is also available. + +- ``.index_at(i, j, ...)`` gets the count from the beginning to a given index. + + +There are also several methods for getting references (described below). + Structured types ================ @@ -345,21 +370,21 @@ The returned proxy object supports some of the same methods as ``py::array`` so that it can be used as a drop-in replacement for some existing, index-checked uses of ``py::array``: -- ``r.ndim()`` returns the number of dimensions +- ``.ndim()`` returns the number of dimensions -- ``r.data(1, 2, ...)`` and ``r.mutable_data(1, 2, ...)``` returns a pointer to +- ``.data(1, 2, ...)`` and ``r.mutable_data(1, 2, ...)``` returns a pointer to the ``const T`` or ``T`` data, respectively, at the given indices. The latter is only available to proxies obtained via ``a.mutable_unchecked()``. -- ``itemsize()`` returns the size of an item in bytes, i.e. ``sizeof(T)``. +- ``.itemsize()`` returns the size of an item in bytes, i.e. ``sizeof(T)``. -- ``ndim()`` returns the number of dimensions. +- ``.ndim()`` returns the number of dimensions. -- ``shape(n)`` returns the size of dimension ``n`` +- ``.shape(n)`` returns the size of dimension ``n`` -- ``size()`` returns the total number of elements (i.e. the product of the shapes). +- ``.size()`` returns the total number of elements (i.e. the product of the shapes). -- ``nbytes()`` returns the number of bytes used by the referenced elements +- ``.nbytes()`` returns the number of bytes used by the referenced elements (i.e. ``itemsize()`` times ``size()``). .. seealso:: diff --git a/docs/changelog.rst b/docs/changelog.rst index 2f76abe0..8b9690ca 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -9,6 +9,11 @@ Starting with version 1.8.0, pybind11 releases use a `semantic versioning v2.8.0 (WIP) ------------ +New features: + +* Added ``py::raise_from`` to enable chaining exceptions. + `#3215 `_ + * Allow exception translators to be optionally registered local to a module instead of applying globally across all pybind11 modules. Use ``register_local_exception_translator(ExceptionTranslator&& translator)`` @@ -16,6 +21,115 @@ v2.8.0 (WIP) translator)`` to keep your exception remapping code local to the module. `#2650 `_ +* Add ``make_simple_namespace`` function for instantiating Python + ``SimpleNamespace`` objects. + `#2840 `_ + +* ``pybind11::scoped_interpreter`` and ``initialize_interpreter`` have new + arguments to allow ``sys.argv`` initialization. + `#2341 `_ + +* Allow Python builtins to be used as callbacks in CPython. + `#1413 `_ + +* Added ``view`` to view arrays with a different datatype. + `#987 `_ + +* Implemented ``reshape`` on arrays. + `#984 `_ + + +Changes: + +* Set ``__file__`` constant when running ``eval_file`` in an embedded interpreter. + `#3233 `_ + +* The pybind11 proxy types ``str``, ``bytes``, ``bytearray``, ``tuple``, + ``list`` now consistently support passing ``ssize_t`` values for sizes and + indexes. Previously, only ``size_t`` was accepted in several interfaces. + `#3219 `_ + + +Fixes: + +* Bug fix: enum value's ``__int__`` returning non-int when underlying type is bool or of char type. + `#1334 `_ + +* Fixes bug in setting error state in Capsule's pointer methods. + `#3261 `_ + +* A long-standing memory leak in ``py::cpp_function::initialize`` was fixed. + `#3229 `_ + +* Fixes thread safety for some ``pybind11::type_caster`` which require lifetime extension, such as for ``std::string_view``. + `#3237 `_ + +* Restore compatibility with gcc 4.8.4 as distributed by ubuntu-trusty, linuxmint-17. + `#3270 `_ + + +Build system improvements: + +* Fix regression in CMake Python package config: improper use of absolute path. + `#3144 `_ + +* Specified UTF8-encoding in setup.py calls of open(). + `#3137 `_ + + +Backend and tidying up: + +* Optimize NumPy array construction with additional moves. + `#3183 `_ + +* Conversion to ``std::string`` and ``std::string_view`` now avoids making an + extra copy of the data on Python >= 3.3. + `#3257 `_ + +* Remove const modifier from certain C++ methods on Python collections + (``list``, ``set``, ``dict``) such as (``clear()``, ``append()``, + ``insert()``, etc...) and annotated them with ``py-non-const``. + +* Enable readability ``clang-tidy-const-return`` and remove useless consts. + `#3254 `_ + `#3194 `_ + +* The clang-tidy ``google-explicit-constructor`` option was enabled. + `#3250 `_ + +* Mark a pytype move constructor as noexcept (perf). + `#3236 `_ + +* Enable clang-tidy check to guard against inheritance slicing. + `#3210 `_ + +* Legacy warning suppression pragma were removed from eigen.h. On Unix + platforms, please use -isystem for Eigen include directories, to suppress + compiler warnings originating from Eigen headers. Note that CMake does this + by default. No adjustments are needed for Windows. + `#3198 `_ + +* Format pybind11 with isort consistent ordering of imports + `#3195 `_ + +* The warnings-suppression "pragma clamp" at the top/bottom of pybind11 was + removed, clearing the path to refactoring and IWYU cleanup. + `#3186 `_ + +* Enable most bugprone checks in clang-tidy and fix the found potential bugs + and poor coding styles. + `#3166 `_ + +* Add ``clang-tidy-readability`` rules to make boolean casts explicit improving + code readability. Also enabled other misc and readability clang-tidy checks. + `#3148 `_ + +* Move object in ``.pop()`` for list. + `#3116 `_ + + + + v2.7.1 (Aug 3, 2021) --------------------- From 14976c853b8422dc41ce7874742cab122684c97e Mon Sep 17 00:00:00 2001 From: Jeremy Maitin-Shepard Date: Mon, 20 Sep 2021 04:57:38 -0700 Subject: [PATCH 071/100] Eliminate duplicate TLS keys for loader_life_support stack (#3275) * Eliminate duplicate TLS keys for loader_life_support stack This revises the existing fix for https://github.com/pybind/pybind11/issues/2765 in https://github.com/pybind/pybind11/pull/3237 to reduce the amount of TLS storage used. The shared TLS key is stored in two different ways, depending on `PYBIND11_INTERNALS_VERSION`. If `PYBIND11_INTERNALS_VERSION == 4` (as is currently set), the TLS key is stored in the `internal::shared_data` map to avoid breaking ABI compatibility. If `PYBIND11_INTERNALS_VERSION > 4`, the TLS key is stored directly in the `internals` struct. * Fix test_pytypes.py::test_issue2361 failure on PyPy3.7 * Add github actions tests for unstable ABI --- .github/workflows/ci.yml | 18 ++ CMakeLists.txt | 7 + include/pybind11/detail/internals.h | 182 +++++++++++++++------ include/pybind11/detail/type_caster_base.h | 41 +++-- include/pybind11/gil.h | 4 +- tests/test_pytypes.py | 3 +- 6 files changed, 189 insertions(+), 66 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a3664ddf..2348212f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -144,6 +144,24 @@ jobs: if: "!(runner.os == 'Windows' && (matrix.python == 3.8 || matrix.python == 3.9 || matrix.python == '3.10-dev'))" run: cmake --build build2 --target cpptest + # Third build - C++17 mode with unstable ABI + - name: Configure (unstable ABI) + run: > + cmake -S . -B build3 + -DPYBIND11_WERROR=ON + -DDOWNLOAD_CATCH=ON + -DDOWNLOAD_EIGEN=ON + -DCMAKE_CXX_STANDARD=17 + -DPYBIND11_INTERNALS_VERSION=10000000 + "-DPYBIND11_TEST_OVERRIDE=test_call_policies.cpp;test_gil_scoped.cpp;test_thread.cpp" + ${{ matrix.args }} + + - name: Build (unstable ABI) + run: cmake --build build3 -j 2 + + - name: Python tests (unstable ABI) + run: cmake --build build3 --target pytest + - name: Interface test run: cmake --build build2 --target test_cmake_build diff --git a/CMakeLists.txt b/CMakeLists.txt index bb01a861..25cfcec2 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -89,6 +89,9 @@ endif() option(PYBIND11_INSTALL "Install pybind11 header files?" ${PYBIND11_MASTER_PROJECT}) option(PYBIND11_TEST "Build pybind11 test suite?" ${PYBIND11_MASTER_PROJECT}) option(PYBIND11_NOPYTHON "Disable search for Python" OFF) +set(PYBIND11_INTERNALS_VERSION + "" + CACHE STRING "Override the ABI version, may be used to enable the unstable ABI.") cmake_dependent_option( USE_PYTHON_INCLUDE_DIR @@ -183,6 +186,10 @@ if(NOT TARGET pybind11_headers) target_compile_features(pybind11_headers INTERFACE cxx_inheriting_constructors cxx_user_literals cxx_right_angle_brackets) + if(NOT "${PYBIND11_INTERNALS_VERSION}" STREQUAL "") + target_compile_definitions( + pybind11_headers INTERFACE "PYBIND11_INTERNALS_VERSION=${PYBIND11_INTERNALS_VERSION}") + endif() else() # It is invalid to install a target twice, too. set(PYBIND11_INSTALL OFF) diff --git a/include/pybind11/detail/internals.h b/include/pybind11/detail/internals.h index 39c28e77..7c2e4997 100644 --- a/include/pybind11/detail/internals.h +++ b/include/pybind11/detail/internals.h @@ -11,6 +11,24 @@ #include "../pytypes.h" +/// Tracks the `internals` and `type_info` ABI version independent of the main library version. +/// +/// Some portions of the code use an ABI that is conditional depending on this +/// version number. That allows ABI-breaking changes to be "pre-implemented". +/// Once the default version number is incremented, the conditional logic that +/// no longer applies can be removed. Additionally, users that need not +/// maintain ABI compatibility can increase the version number in order to take +/// advantage of any functionality/efficiency improvements that depend on the +/// newer ABI. +/// +/// WARNING: If you choose to manually increase the ABI version, note that +/// pybind11 may not be tested as thoroughly with a non-default ABI version, and +/// further ABI-incompatible changes may be made before the ABI is officially +/// changed to the new version. +#ifndef PYBIND11_INTERNALS_VERSION +# define PYBIND11_INTERNALS_VERSION 4 +#endif + PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) using ExceptionTranslator = void (*)(std::exception_ptr); @@ -25,30 +43,58 @@ inline PyObject *make_object_base_type(PyTypeObject *metaclass); // The old Python Thread Local Storage (TLS) API is deprecated in Python 3.7 in favor of the new // Thread Specific Storage (TSS) API. #if PY_VERSION_HEX >= 0x03070000 -# define PYBIND11_TLS_KEY_INIT(var) Py_tss_t *var = nullptr -# define PYBIND11_TLS_GET_VALUE(key) PyThread_tss_get((key)) -# define PYBIND11_TLS_REPLACE_VALUE(key, value) PyThread_tss_set((key), (value)) -# define PYBIND11_TLS_DELETE_VALUE(key) PyThread_tss_set((key), nullptr) -# define PYBIND11_TLS_FREE(key) PyThread_tss_free(key) -#else - // Usually an int but a long on Cygwin64 with Python 3.x -# define PYBIND11_TLS_KEY_INIT(var) decltype(PyThread_create_key()) var = 0 -# define PYBIND11_TLS_GET_VALUE(key) PyThread_get_key_value((key)) -# if PY_MAJOR_VERSION < 3 -# define PYBIND11_TLS_DELETE_VALUE(key) \ - PyThread_delete_key_value(key) -# define PYBIND11_TLS_REPLACE_VALUE(key, value) \ - do { \ - PyThread_delete_key_value((key)); \ - PyThread_set_key_value((key), (value)); \ - } while (false) +// Avoid unnecessary allocation of `Py_tss_t`, since we cannot use +// `Py_LIMITED_API` anyway. +# if PYBIND11_INTERNALS_VERSION > 4 +# define PYBIND11_TLS_KEY_REF Py_tss_t & +# ifdef __GNUC__ +// Clang on macOS warns due to `Py_tss_NEEDS_INIT` not specifying an initializer +// for every field. +# define PYBIND11_TLS_KEY_INIT(var) \ + _Pragma("GCC diagnostic push") /**/ \ + _Pragma("GCC diagnostic ignored \"-Wmissing-field-initializers\"") /**/ \ + Py_tss_t var \ + = Py_tss_NEEDS_INIT; \ + _Pragma("GCC diagnostic pop") +# else +# define PYBIND11_TLS_KEY_INIT(var) Py_tss_t var = Py_tss_NEEDS_INIT; +# endif +# define PYBIND11_TLS_KEY_CREATE(var) (PyThread_tss_create(&(var)) == 0) +# define PYBIND11_TLS_GET_VALUE(key) PyThread_tss_get(&(key)) +# define PYBIND11_TLS_REPLACE_VALUE(key, value) PyThread_tss_set(&(key), (value)) +# define PYBIND11_TLS_DELETE_VALUE(key) PyThread_tss_set(&(key), nullptr) +# define PYBIND11_TLS_FREE(key) PyThread_tss_delete(&(key)) # else -# define PYBIND11_TLS_DELETE_VALUE(key) \ - PyThread_set_key_value((key), nullptr) -# define PYBIND11_TLS_REPLACE_VALUE(key, value) \ - PyThread_set_key_value((key), (value)) +# define PYBIND11_TLS_KEY_REF Py_tss_t * +# define PYBIND11_TLS_KEY_INIT(var) Py_tss_t *var = nullptr; +# define PYBIND11_TLS_KEY_CREATE(var) \ + (((var) = PyThread_tss_alloc()) != nullptr && (PyThread_tss_create((var)) == 0)) +# define PYBIND11_TLS_GET_VALUE(key) PyThread_tss_get((key)) +# define PYBIND11_TLS_REPLACE_VALUE(key, value) PyThread_tss_set((key), (value)) +# define PYBIND11_TLS_DELETE_VALUE(key) PyThread_tss_set((key), nullptr) +# define PYBIND11_TLS_FREE(key) PyThread_tss_free(key) # endif -# define PYBIND11_TLS_FREE(key) (void)key +#else +// Usually an int but a long on Cygwin64 with Python 3.x +# define PYBIND11_TLS_KEY_REF decltype(PyThread_create_key()) +# define PYBIND11_TLS_KEY_INIT(var) PYBIND11_TLS_KEY_REF var = 0; +# define PYBIND11_TLS_KEY_CREATE(var) (((var) = PyThread_create_key()) != -1) +# define PYBIND11_TLS_GET_VALUE(key) PyThread_get_key_value((key)) +# if PY_MAJOR_VERSION < 3 || defined(PYPY_VERSION) +// On CPython < 3.4 and on PyPy, `PyThread_set_key_value` strangely does not set +// the value if it has already been set. Instead, it must first be deleted and +// then set again. +# define PYBIND11_TLS_DELETE_VALUE(key) PyThread_delete_key_value(key) +# define PYBIND11_TLS_REPLACE_VALUE(key, value) \ + do { \ + PyThread_delete_key_value((key)); \ + PyThread_set_key_value((key), (value)); \ + } while (false) +# else +# define PYBIND11_TLS_DELETE_VALUE(key) PyThread_set_key_value((key), nullptr) +# define PYBIND11_TLS_REPLACE_VALUE(key, value) PyThread_set_key_value((key), (value)) +# endif +# define PYBIND11_TLS_FREE(key) (void) key #endif // Python loads modules by default with dlopen with the RTLD_LOCAL flag; under libc++ and possibly @@ -106,22 +152,31 @@ struct internals { std::unordered_map> patients; std::forward_list registered_exception_translators; std::unordered_map shared_data; // Custom data to be shared across extensions +#if PYBIND11_INTERNALS_VERSION == 4 std::vector unused_loader_patient_stack_remove_at_v5; +#endif std::forward_list static_strings; // Stores the std::strings backing detail::c_str() PyTypeObject *static_property_type; PyTypeObject *default_metaclass; PyObject *instance_base; #if defined(WITH_THREAD) - PYBIND11_TLS_KEY_INIT(tstate); + PYBIND11_TLS_KEY_INIT(tstate) +# if PYBIND11_INTERNALS_VERSION > 4 + PYBIND11_TLS_KEY_INIT(loader_life_support_tls_key) +# endif // PYBIND11_INTERNALS_VERSION > 4 PyInterpreterState *istate = nullptr; ~internals() { +# if PYBIND11_INTERNALS_VERSION > 4 + PYBIND11_TLS_FREE(loader_life_support_tls_key); +# endif // PYBIND11_INTERNALS_VERSION > 4 + // This destructor is called *after* Py_Finalize() in finalize_interpreter(). - // That *SHOULD BE* fine. The following details what happens when PyThread_tss_free is called. - // PYBIND11_TLS_FREE is PyThread_tss_free on python 3.7+. On older python, it does nothing. - // PyThread_tss_free calls PyThread_tss_delete and PyMem_RawFree. - // PyThread_tss_delete just calls TlsFree (on Windows) or pthread_key_delete (on *NIX). Neither - // of those have anything to do with CPython internals. - // PyMem_RawFree *requires* that the `tstate` be allocated with the CPython allocator. + // That *SHOULD BE* fine. The following details what happens when PyThread_tss_free is + // called. PYBIND11_TLS_FREE is PyThread_tss_free on python 3.7+. On older python, it does + // nothing. PyThread_tss_free calls PyThread_tss_delete and PyMem_RawFree. + // PyThread_tss_delete just calls TlsFree (on Windows) or pthread_key_delete (on *NIX). + // Neither of those have anything to do with CPython internals. PyMem_RawFree *requires* + // that the `tstate` be allocated with the CPython allocator. PYBIND11_TLS_FREE(tstate); } #endif @@ -153,9 +208,6 @@ struct type_info { bool module_local : 1; }; -/// Tracks the `internals` and `type_info` ABI version independent of the main library version -#define PYBIND11_INTERNALS_VERSION 4 - /// On MSVC, debug and release builds are not ABI-compatible! #if defined(_MSC_VER) && defined(_DEBUG) # define PYBIND11_BUILD_TYPE "_debug" @@ -291,21 +343,21 @@ PYBIND11_NOINLINE internals &get_internals() { internals_ptr = new internals(); #if defined(WITH_THREAD) - #if PY_VERSION_HEX < 0x03090000 - PyEval_InitThreads(); - #endif +# if PY_VERSION_HEX < 0x03090000 + PyEval_InitThreads(); +# endif PyThreadState *tstate = PyThreadState_Get(); - #if PY_VERSION_HEX >= 0x03070000 - internals_ptr->tstate = PyThread_tss_alloc(); - if (!internals_ptr->tstate || (PyThread_tss_create(internals_ptr->tstate) != 0)) - pybind11_fail("get_internals: could not successfully initialize the tstate TSS key!"); - PyThread_tss_set(internals_ptr->tstate, tstate); - #else - internals_ptr->tstate = PyThread_create_key(); - if (internals_ptr->tstate == -1) - pybind11_fail("get_internals: could not successfully initialize the tstate TLS key!"); - PyThread_set_key_value(internals_ptr->tstate, tstate); - #endif + if (!PYBIND11_TLS_KEY_CREATE(internals_ptr->tstate)) { + pybind11_fail("get_internals: could not successfully initialize the tstate TSS key!"); + } + PYBIND11_TLS_REPLACE_VALUE(internals_ptr->tstate, tstate); + +# if PYBIND11_INTERNALS_VERSION > 4 + if (!PYBIND11_TLS_KEY_CREATE(internals_ptr->loader_life_support_tls_key)) { + pybind11_fail("get_internals: could not successfully initialize the " + "loader_life_support TSS key!"); + } +# endif internals_ptr->istate = tstate->interp; #endif builtins[id] = capsule(internals_pp); @@ -317,7 +369,6 @@ PYBIND11_NOINLINE internals &get_internals() { return **internals_pp; } - // the internals struct (above) is shared between all the modules. local_internals are only // for a single module. Any changes made to internals may require an update to // PYBIND11_INTERNALS_VERSION, breaking backwards compatibility. local_internals is, by design, @@ -325,8 +376,41 @@ PYBIND11_NOINLINE internals &get_internals() { // impact any other modules, because the only things accessing the local internals is the // module that contains them. struct local_internals { - type_map registered_types_cpp; - std::forward_list registered_exception_translators; + type_map registered_types_cpp; + std::forward_list registered_exception_translators; +#if defined(WITH_THREAD) && PYBIND11_INTERNALS_VERSION == 4 + + // For ABI compatibility, we can't store the loader_life_support TLS key in + // the `internals` struct directly. Instead, we store it in `shared_data` and + // cache a copy in `local_internals`. If we allocated a separate TLS key for + // each instance of `local_internals`, we could end up allocating hundreds of + // TLS keys if hundreds of different pybind11 modules are loaded (which is a + // plausible number). + PYBIND11_TLS_KEY_INIT(loader_life_support_tls_key) + + // Holds the shared TLS key for the loader_life_support stack. + struct shared_loader_life_support_data { + PYBIND11_TLS_KEY_INIT(loader_life_support_tls_key) + shared_loader_life_support_data() { + if (!PYBIND11_TLS_KEY_CREATE(loader_life_support_tls_key)) { + pybind11_fail("local_internals: could not successfully initialize the " + "loader_life_support TLS key!"); + } + } + // We can't help but leak the TLS key, because Python never unloads extension modules. + }; + + local_internals() { + auto &internals = get_internals(); + // Get or create the `loader_life_support_stack_key`. + auto &ptr = internals.shared_data["_life_support"]; + if (!ptr) { + ptr = new shared_loader_life_support_data; + } + loader_life_support_tls_key + = static_cast(ptr)->loader_life_support_tls_key; + } +#endif // defined(WITH_THREAD) && PYBIND11_INTERNALS_VERSION == 4 }; /// Works like `get_internals`, but for things which are locally registered. diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h index fa8433cf..f804d9d1 100644 --- a/include/pybind11/detail/type_caster_base.h +++ b/include/pybind11/detail/type_caster_base.h @@ -35,30 +35,43 @@ private: loader_life_support* parent = nullptr; std::unordered_set keep_alive; - static loader_life_support** get_stack_pp() { #if defined(WITH_THREAD) - thread_local static loader_life_support* per_thread_stack = nullptr; - return &per_thread_stack; -#else - static loader_life_support* global_stack = nullptr; - return &global_stack; -#endif + // Store stack pointer in thread-local storage. + static PYBIND11_TLS_KEY_REF get_stack_tls_key() { +# if PYBIND11_INTERNALS_VERSION == 4 + return get_local_internals().loader_life_support_tls_key; +# else + return get_internals().loader_life_support_tls_key; +# endif } + static loader_life_support *get_stack_top() { + return static_cast(PYBIND11_TLS_GET_VALUE(get_stack_tls_key())); + } + static void set_stack_top(loader_life_support *value) { + PYBIND11_TLS_REPLACE_VALUE(get_stack_tls_key(), value); + } +#else + // Use single global variable for stack. + static loader_life_support **get_stack_pp() { + static loader_life_support *global_stack = nullptr; + return global_stack; + } + static loader_life_support *get_stack_top() { return *get_stack_pp(); } + static void set_stack_top(loader_life_support *value) { *get_stack_pp() = value; } +#endif public: /// A new patient frame is created when a function is entered loader_life_support() { - loader_life_support** stack = get_stack_pp(); - parent = *stack; - *stack = this; + parent = get_stack_top(); + set_stack_top(this); } /// ... and destroyed after it returns ~loader_life_support() { - loader_life_support** stack = get_stack_pp(); - if (*stack != this) + if (get_stack_top() != this) pybind11_fail("loader_life_support: internal error"); - *stack = parent; + set_stack_top(parent); for (auto* item : keep_alive) Py_DECREF(item); } @@ -66,7 +79,7 @@ public: /// This can only be used inside a pybind11-bound function, either by `argument_loader` /// at argument preparation time or by `py::cast()` at execution time. PYBIND11_NOINLINE static void add_patient(handle h) { - loader_life_support* frame = *get_stack_pp(); + loader_life_support *frame = get_stack_top(); if (!frame) { // NOTE: It would be nice to include the stack frames here, as this indicates // use of pybind11::cast<> outside the normal call framework, finding such diff --git a/include/pybind11/gil.h b/include/pybind11/gil.h index 32f1a896..b73aaa3f 100644 --- a/include/pybind11/gil.h +++ b/include/pybind11/gil.h @@ -50,7 +50,7 @@ PYBIND11_NAMESPACE_END(detail) class gil_scoped_acquire { public: PYBIND11_NOINLINE gil_scoped_acquire() { - auto const &internals = detail::get_internals(); + auto &internals = detail::get_internals(); tstate = (PyThreadState *) PYBIND11_TLS_GET_VALUE(internals.tstate); if (!tstate) { @@ -132,7 +132,7 @@ public: // `get_internals()` must be called here unconditionally in order to initialize // `internals.tstate` for subsequent `gil_scoped_acquire` calls. Otherwise, an // initialization race could occur as multiple threads try `gil_scoped_acquire`. - const auto &internals = detail::get_internals(); + auto &internals = detail::get_internals(); tstate = PyEval_SaveThread(); if (disassoc) { auto key = internals.tstate; diff --git a/tests/test_pytypes.py b/tests/test_pytypes.py index 070538cc..cc27e60a 100644 --- a/tests/test_pytypes.py +++ b/tests/test_pytypes.py @@ -467,7 +467,8 @@ def test_issue2361(): assert m.issue2361_str_implicit_copy_none() == "None" with pytest.raises(TypeError) as excinfo: assert m.issue2361_dict_implicit_copy_none() - assert "'NoneType' object is not iterable" in str(excinfo.value) + assert "NoneType" in str(excinfo.value) + assert "iterable" in str(excinfo.value) @pytest.mark.parametrize( From d0f3c51f01271b71e6228a1d002bad347dc4aca1 Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Mon, 20 Sep 2021 10:42:14 -0400 Subject: [PATCH 072/100] Enable defining custom __new__ (#3265) * Enable defining custom __new__ * See if xfail needed * Qualify auto self * Unconditionally defining PYBIND11_DISABLE_NEW_STYLE_INIT_WARNING. Returning pointer from "__init__" instead of reference. * Use new style __init__ * Simplify __new__ creation * Reviewer suggestions * Match indentation Co-authored-by: Ralf W. Grosse-Kunstleve --- include/pybind11/pybind11.h | 3 ++- tests/test_class.cpp | 15 +++++++++++++++ tests/test_class.py | 8 ++++++++ 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 07d2d043..b8f5a6ba 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -416,7 +416,8 @@ protected: detail::function_record *chain = nullptr, *chain_start = rec; if (rec->sibling) { if (PyCFunction_Check(rec->sibling.ptr())) { - auto rec_capsule = reinterpret_borrow(PyCFunction_GET_SELF(rec->sibling.ptr())); + auto *self = PyCFunction_GET_SELF(rec->sibling.ptr()); + capsule rec_capsule = isinstance(self) ? reinterpret_borrow(self) : capsule(self); chain = (detail::function_record *) rec_capsule; /* Never append a method to an overload chain of a parent class; instead, hide the parent's overloads in this case */ diff --git a/tests/test_class.cpp b/tests/test_class.cpp index 246b6c1f..0b998330 100644 --- a/tests/test_class.cpp +++ b/tests/test_class.cpp @@ -47,10 +47,25 @@ TEST_SUBMODULE(class_, m) { } ~NoConstructor() { print_destroyed(this); } }; + struct NoConstructorNew { + NoConstructorNew() = default; + NoConstructorNew(const NoConstructorNew &) = default; + NoConstructorNew(NoConstructorNew &&) = default; + static NoConstructorNew *new_instance() { + auto *ptr = new NoConstructorNew(); + print_created(ptr, "via new_instance"); + return ptr; + } + ~NoConstructorNew() { print_destroyed(this); } + }; py::class_(m, "NoConstructor") .def_static("new_instance", &NoConstructor::new_instance, "Return an instance"); + py::class_(m, "NoConstructorNew") + .def(py::init([](NoConstructorNew *self) { return self; })) // Need a NOOP __init__ + .def_static("__new__", [](const py::object *) { return NoConstructorNew::new_instance(); } ); + // test_inheritance class Pet { public: diff --git a/tests/test_class.py b/tests/test_class.py index 85d45319..caafe206 100644 --- a/tests/test_class.py +++ b/tests/test_class.py @@ -25,6 +25,14 @@ def test_instance(msg): assert cstats.alive() == 0 +def test_instance_new(msg): + instance = m.NoConstructorNew() # .__new__(m.NoConstructor.__class__) + cstats = ConstructorStats.get(m.NoConstructorNew) + assert cstats.alive() == 1 + del instance + assert cstats.alive() == 0 + + def test_type(): assert m.check_type(1) == m.DerivedClass1 with pytest.raises(RuntimeError) as execinfo: From 6e6975e2171cd5fa5efc109a708dc81addae1880 Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Mon, 20 Sep 2021 16:03:21 -0400 Subject: [PATCH 073/100] Fix test case with __new__ (#3285) --- tests/test_class.cpp | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/test_class.cpp b/tests/test_class.cpp index 0b998330..52a41a3b 100644 --- a/tests/test_class.cpp +++ b/tests/test_class.cpp @@ -63,8 +63,9 @@ TEST_SUBMODULE(class_, m) { .def_static("new_instance", &NoConstructor::new_instance, "Return an instance"); py::class_(m, "NoConstructorNew") - .def(py::init([](NoConstructorNew *self) { return self; })) // Need a NOOP __init__ - .def_static("__new__", [](const py::object *) { return NoConstructorNew::new_instance(); } ); + .def(py::init([](const NoConstructorNew &self) { return self; })) // Need a NOOP __init__ + .def_static("__new__", + [](const py::object &) { return NoConstructorNew::new_instance(); }); // test_inheritance class Pet { From 077a16e9d11bb946babbe8001a1dd42704eae3ec Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 20 Sep 2021 16:18:09 -0400 Subject: [PATCH 074/100] [pre-commit.ci] pre-commit autoupdate (#3286) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit updates: - [github.com/psf/black: 21.8b0 → 21.9b0](https://github.com/psf/black/compare/21.8b0...21.9b0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .pre-commit-config.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index fd11cf0c..b9e37e5f 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -42,7 +42,7 @@ repos: # Black, the code formatter, natively supports pre-commit - repo: https://github.com/psf/black - rev: 21.8b0 + rev: 21.9b0 hooks: - id: black # By default, this ignores pyi files, though black supports them From ee0c5ee405e7a532410797687da28a20b89cd62b Mon Sep 17 00:00:00 2001 From: Bruce Merry <1963944+bmerry@users.noreply.github.com> Date: Tue, 21 Sep 2021 19:37:19 +0200 Subject: [PATCH 075/100] Add make_value_iterator (#3271) * Add make_value_iterator This is the counterpart to make_key_iterator, and will allow implementing a `value` method in `bind_map` (although doing so is left for a subsequent PR). I made a few design changes to reduce copy-and-paste boilerplate. Previously detail::iterator_state had a boolean template parameter to indicate whether it was being used for make_iterator or make_key_iterator. I replaced the boolean with a class that determines how to dereference the iterator. This allows for a generic implementation of `__next__`. I also added the ValueType and Extra... parameters to the iterator_state template args, because I think it was a bug that they were missing: if make_iterator is called twice with different values of these, only the first set has effect (because the state class is only registered once). There is still a potential issue in that the *values* of the extra arguments are latched on the first call, but since most policies are empty classes this should be even less common. * Add some remove_cv_t to appease clang-tidy * Make iterator_access and friends take reference For some reason I'd accidentally made it take a const value, which caused some issues with third-party packages. * Another attempt to remove remove_cv_t from iterators Some of the return types were const (non-reference) types because of the pecularities of decltype: `decltype((*it).first)` is the *declared* type of the member of the pair, rather than the type of the expression. So if the reference type of the iterator is `pair &`, then the decltype is `const int`. Wrapping an extra set of parentheses to form `decltype(((*it).first))` would instead give `const int &`. This means that the existing make_key_iterator actually returns by value from `__next__`, rather than by reference. Since for mapping types, keys are always const, this probably hasn't been noticed, but it will affect make_value_iterator if the Python code tries to mutate the returned objects. I've changed things to use double parentheses so that make_iterator, make_key_iterator and make_value_iterator should now all return the reference type of the iterator. I'll still need to add a test for that; for now I'm just checking whether I can keep Clang-Tidy happy. * Add back some NOLINTNEXTLINE to appease Clang-Tidy This is favoured over using remove_cv_t because in some cases a const value return type is deliberate (particularly for Eigen). * Add a unit test for iterator referencing Ensure that make_iterator, make_key_iterator and make_value_iterator return references to the container elements, rather than copies. The test for make_key_iterator fails to compile on master, which gives me confidence that this branch has fixed it. * Make the iterator_access etc operator() const I'm actually a little surprised it compiled at all given that the operator() is called on a temporary, but I don't claim to fully understand all the different value types in C++11. * Attempt to work around compiler bugs https://godbolt.org/ shows an example where ICC gets the wrong result for a decltype used as the default for a template argument, and CI also showed problems with PGI. This is a shot in the dark to see if it fixes things. * Make a test constructor explicit (Clang-Tidy) * Fix unit test on GCC 4.8.5 It seems to require the arguments to the std::pair constructor to be implicitly convertible to the types in the pair, rather than just requiring is_constructible. * Remove DOXYGEN_SHOULD_SKIP_THIS guards Now that a complex decltype expression has been replaced by a simpler nested type, I'm hoping Doxygen will be able to build it without issues. * Add comment to explain iterator_state template params --- docs/reference.rst | 3 + include/pybind11/pybind11.h | 118 ++++++++++++++++++------- tests/test_sequences_and_iterators.cpp | 58 ++++++++++++ tests/test_sequences_and_iterators.py | 29 ++++++ 4 files changed, 176 insertions(+), 32 deletions(-) diff --git a/docs/reference.rst b/docs/reference.rst index a678d41c..e64a0351 100644 --- a/docs/reference.rst +++ b/docs/reference.rst @@ -63,6 +63,9 @@ Convenience functions converting to Python types .. doxygenfunction:: make_key_iterator(Iterator, Sentinel, Extra &&...) .. doxygenfunction:: make_key_iterator(Type &, Extra&&...) +.. doxygenfunction:: make_value_iterator(Iterator, Sentinel, Extra &&...) +.. doxygenfunction:: make_value_iterator(Type &, Extra&&...) + .. _extras: Passing extra arguments to ``def`` or ``class_`` diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index b8f5a6ba..ac95b3a3 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -1955,25 +1955,52 @@ inline std::pair all_t return res; } -template +/* There are a large number of apparently unused template arguments because + * each combination requires a separate py::class_ registration. + */ +template struct iterator_state { Iterator it; Sentinel end; bool first_or_done; }; -PYBIND11_NAMESPACE_END(detail) +// Note: these helpers take the iterator by non-const reference because some +// iterators in the wild can't be dereferenced when const. +template +struct iterator_access { + using result_type = decltype((*std::declval())); + // NOLINTNEXTLINE(readability-const-return-type) // PR #3263 + result_type operator()(Iterator &it) const { + return *it; + } +}; -/// Makes a python iterator from a first and past-the-end C++ InputIterator. -template +struct iterator_key_access { + using result_type = decltype(((*std::declval()).first)); + result_type operator()(Iterator &it) const { + return (*it).first; + } +}; + +template +struct iterator_value_access { + using result_type = decltype(((*std::declval()).second)); + result_type operator()(Iterator &it) const { + return (*it).second; + } +}; + +template ()), -#endif + typename ValueType, typename... Extra> -iterator make_iterator(Iterator first, Sentinel last, Extra &&... extra) { - using state = detail::iterator_state; +iterator make_iterator_impl(Iterator first, Sentinel last, Extra &&... extra) { + using state = detail::iterator_state; + // TODO: state captures only the types of Extra, not the values if (!detail::get_type_info(typeid(state), false)) { class_(handle(), "iterator", pybind11::module_local()) @@ -1987,7 +2014,7 @@ iterator make_iterator(Iterator first, Sentinel last, Extra &&... extra) { s.first_or_done = true; throw stop_iteration(); } - return *s.it; + return Access()(s.it); // NOLINTNEXTLINE(readability-const-return-type) // PR #3263 }, std::forward(extra)..., Policy); } @@ -1995,35 +2022,55 @@ iterator make_iterator(Iterator first, Sentinel last, Extra &&... extra) { return cast(state{first, last, true}); } -/// Makes an python iterator over the keys (`.first`) of a iterator over pairs from a +PYBIND11_NAMESPACE_END(detail) + +/// Makes a python iterator from a first and past-the-end C++ InputIterator. +template ::result_type, + typename... Extra> +iterator make_iterator(Iterator first, Sentinel last, Extra &&... extra) { + return detail::make_iterator_impl< + detail::iterator_access, + Policy, + Iterator, + Sentinel, + ValueType, + Extra...>(first, last, std::forward(extra)...); +} + +/// Makes a python iterator over the keys (`.first`) of a iterator over pairs from a /// first and past-the-end InputIterator. template ()).first), -#endif + typename KeyType = typename detail::iterator_key_access::result_type, typename... Extra> iterator make_key_iterator(Iterator first, Sentinel last, Extra &&...extra) { - using state = detail::iterator_state; + return detail::make_iterator_impl< + detail::iterator_key_access, + Policy, + Iterator, + Sentinel, + KeyType, + Extra...>(first, last, std::forward(extra)...); +} - if (!detail::get_type_info(typeid(state), false)) { - class_(handle(), "iterator", pybind11::module_local()) - .def("__iter__", [](state &s) -> state& { return s; }) - .def("__next__", [](state &s) -> detail::remove_cv_t { - if (!s.first_or_done) - ++s.it; - else - s.first_or_done = false; - if (s.it == s.end) { - s.first_or_done = true; - throw stop_iteration(); - } - return (*s.it).first; - }, std::forward(extra)..., Policy); - } - - return cast(state{first, last, true}); +/// Makes a python iterator over the values (`.second`) of a iterator over pairs from a +/// first and past-the-end InputIterator. +template ::result_type, + typename... Extra> +iterator make_value_iterator(Iterator first, Sentinel last, Extra &&...extra) { + return detail::make_iterator_impl< + detail::iterator_value_access, + Policy, Iterator, + Sentinel, + ValueType, + Extra...>(first, last, std::forward(extra)...); } /// Makes an iterator over values of an stl container or other container supporting @@ -2040,6 +2087,13 @@ template (std::begin(value), std::end(value), extra...); } +/// Makes an iterator over the values (`.second`) of a stl map-like container supporting +/// `std::begin()`/`std::end()` +template iterator make_value_iterator(Type &value, Extra&&... extra) { + return make_value_iterator(std::begin(value), std::end(value), extra...); +} + template void implicitly_convertible() { struct set_flag { bool &flag; diff --git a/tests/test_sequences_and_iterators.cpp b/tests/test_sequences_and_iterators.cpp index b07fd197..66d64726 100644 --- a/tests/test_sequences_and_iterators.cpp +++ b/tests/test_sequences_and_iterators.cpp @@ -15,6 +15,7 @@ #include #include +#include template class NonZeroIterator { @@ -32,6 +33,29 @@ bool operator==(const NonZeroIterator>& it, const NonZeroSentine return !(*it).first || !(*it).second; } +class NonCopyableInt { +public: + explicit NonCopyableInt(int value) : value_(value) {} + NonCopyableInt(const NonCopyableInt &) = delete; + NonCopyableInt(NonCopyableInt &&other) noexcept : value_(other.value_) { + other.value_ = -1; // detect when an unwanted move occurs + } + NonCopyableInt &operator=(const NonCopyableInt &) = delete; + NonCopyableInt &operator=(NonCopyableInt &&other) noexcept { + value_ = other.value_; + other.value_ = -1; // detect when an unwanted move occurs + return *this; + } + int get() const { return value_; } + void set(int value) { value_ = value; } + ~NonCopyableInt() = default; +private: + int value_; +}; +using NonCopyableIntPair = std::pair; +PYBIND11_MAKE_OPAQUE(std::vector); +PYBIND11_MAKE_OPAQUE(std::vector); + template py::list test_random_access_iterator(PythonType x) { if (x.size() < 5) @@ -271,6 +295,10 @@ TEST_SUBMODULE(sequences_and_iterators, m) { .def( "items", [](const StringMap &map) { return py::make_iterator(map.begin(), map.end()); }, + py::keep_alive<0, 1>()) + .def( + "values", + [](const StringMap &map) { return py::make_value_iterator(map.begin(), map.end()); }, py::keep_alive<0, 1>()); // test_generalized_iterators @@ -289,8 +317,38 @@ TEST_SUBMODULE(sequences_and_iterators, m) { .def("nonzero_keys", [](const IntPairs& s) { return py::make_key_iterator(NonZeroIterator>(s.begin()), NonZeroSentinel()); }, py::keep_alive<0, 1>()) + .def("nonzero_values", [](const IntPairs& s) { + return py::make_value_iterator(NonZeroIterator>(s.begin()), NonZeroSentinel()); + }, py::keep_alive<0, 1>()) ; + // test_iterater_referencing + py::class_(m, "NonCopyableInt") + .def(py::init()) + .def("set", &NonCopyableInt::set) + .def("__int__", &NonCopyableInt::get) + ; + py::class_>(m, "VectorNonCopyableInt") + .def(py::init<>()) + .def("append", [](std::vector &vec, int value) { + vec.emplace_back(value); + }) + .def("__iter__", [](std::vector &vec) { + return py::make_iterator(vec.begin(), vec.end()); + }) + ; + py::class_>(m, "VectorNonCopyableIntPair") + .def(py::init<>()) + .def("append", [](std::vector &vec, const std::pair &value) { + vec.emplace_back(NonCopyableInt(value.first), NonCopyableInt(value.second)); + }) + .def("keys", [](std::vector &vec) { + return py::make_key_iterator(vec.begin(), vec.end()); + }) + .def("values", [](std::vector &vec) { + return py::make_value_iterator(vec.begin(), vec.end()); + }) + ; #if 0 // Obsolete: special data structure for exposing custom iterator types to python diff --git a/tests/test_sequences_and_iterators.py b/tests/test_sequences_and_iterators.py index a868c542..2c73eff2 100644 --- a/tests/test_sequences_and_iterators.py +++ b/tests/test_sequences_and_iterators.py @@ -25,6 +25,10 @@ def test_generalized_iterators(): assert list(m.IntPairs([(1, 2), (2, 0), (0, 3), (4, 5)]).nonzero_keys()) == [1] assert list(m.IntPairs([(0, 3), (1, 2), (3, 4)]).nonzero_keys()) == [] + assert list(m.IntPairs([(1, 2), (3, 4), (0, 5)]).nonzero_values()) == [2, 4] + assert list(m.IntPairs([(1, 2), (2, 0), (0, 3), (4, 5)]).nonzero_values()) == [2] + assert list(m.IntPairs([(0, 3), (1, 2), (3, 4)]).nonzero_values()) == [] + # __next__ must continue to raise StopIteration it = m.IntPairs([(0, 0)]).nonzero() for _ in range(3): @@ -37,6 +41,30 @@ def test_generalized_iterators(): next(it) +def test_iterator_referencing(): + """Test that iterators reference rather than copy their referents.""" + vec = m.VectorNonCopyableInt() + vec.append(3) + vec.append(5) + assert [int(x) for x in vec] == [3, 5] + # Increment everything to make sure the referents can be mutated + for x in vec: + x.set(int(x) + 1) + assert [int(x) for x in vec] == [4, 6] + + vec = m.VectorNonCopyableIntPair() + vec.append([3, 4]) + vec.append([5, 7]) + assert [int(x) for x in vec.keys()] == [3, 5] + assert [int(x) for x in vec.values()] == [4, 7] + for x in vec.keys(): + x.set(int(x) + 1) + for x in vec.values(): + x.set(int(x) + 10) + assert [int(x) for x in vec.keys()] == [4, 6] + assert [int(x) for x in vec.values()] == [14, 17] + + def test_sliceable(): sliceable = m.Sliceable(100) assert sliceable[::] == (0, 100, 1) @@ -140,6 +168,7 @@ def test_map_iterator(): assert sm[k] == expected[k] for k, v in sm.items(): assert v == expected[k] + assert list(sm.values()) == [expected[k] for k in sm] it = iter(m.StringMap({})) for _ in range(3): # __next__ must continue to raise StopIteration From 0fb981b21966339b811bc770b11f71eddd918f13 Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Wed, 22 Sep 2021 15:38:50 -0400 Subject: [PATCH 076/100] Add blacken-docs and pycln pre-commit hooks (#3292) * Apply blacken-docs and fix language-hints * Add blacken-docs pre-commit hook * Add pycln pre-commit hook * Enable a few builtin hooks * Black no longer ignores pyi files --- .pre-commit-config.yaml | 30 ++++++++++++++++++++--------- docs/advanced/cast/custom.rst | 2 ++ docs/advanced/cast/eigen.rst | 4 ++-- docs/advanced/cast/strings.rst | 32 +++++++++++++++---------------- docs/advanced/classes.rst | 8 +++++--- docs/advanced/embedding.rst | 1 + docs/advanced/functions.rst | 7 ++++--- docs/advanced/pycpp/numpy.rst | 6 +++--- docs/advanced/pycpp/object.rst | 1 + docs/advanced/pycpp/utilities.rst | 2 +- docs/advanced/smart_ptrs.rst | 1 + docs/benchmark.py | 1 - docs/changelog.rst | 1 + docs/classes.rst | 22 ++++++++++----------- docs/compiling.rst | 11 ++--------- docs/conf.py | 1 - docs/faq.rst | 2 +- 17 files changed, 72 insertions(+), 60 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index b9e37e5f..4245641b 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -19,8 +19,10 @@ repos: hooks: - id: check-added-large-files - id: check-case-conflict + - id: check-docstring-first - id: check-merge-conflict - id: check-symlinks + - id: check-toml - id: check-yaml - id: debug-statements - id: end-of-file-fixer @@ -42,12 +44,16 @@ repos: # Black, the code formatter, natively supports pre-commit - repo: https://github.com/psf/black - rev: 21.9b0 + rev: 21.9b0 # Keep in sync with blacken-docs hooks: - id: black - # By default, this ignores pyi files, though black supports them - types: [text] - files: \.pyi?$ + +- repo: https://github.com/asottile/blacken-docs + rev: v1.11.0 + hooks: + - id: blacken-docs + additional_dependencies: + - black==21.9b0 # keep in sync with black hook # Changes tabs to spaces - repo: https://github.com/Lucas-C/pre-commit-hooks @@ -55,6 +61,12 @@ repos: hooks: - id: remove-tabs +# Autoremoves unused imports +- repo: https://github.com/hadialqattan/pycln + rev: v1.0.3 + hooks: + - id: pycln + # Flake8 also supports pre-commit natively (same author) - repo: https://github.com/PyCQA/flake8 rev: 3.9.2 @@ -86,7 +98,7 @@ repos: # Checks the manifest for missing files (native support) - repo: https://github.com/mgedmin/check-manifest - rev: "0.46" + rev: "0.47" hooks: - id: check-manifest # This is a slow hook, so only run this if --hook-stage manual is passed @@ -100,10 +112,10 @@ repos: exclude: ".supp$" args: ["-L", "nd,ot,thist"] -- repo: https://github.com/shellcheck-py/shellcheck-py - rev: v0.7.2.1 - hooks: - - id: shellcheck +- repo: https://github.com/shellcheck-py/shellcheck-py + rev: v0.7.2.1 + hooks: + - id: shellcheck # The original pybind11 checks for a few C++ style items - repo: local diff --git a/docs/advanced/cast/custom.rst b/docs/advanced/cast/custom.rst index a779444c..19b93534 100644 --- a/docs/advanced/cast/custom.rst +++ b/docs/advanced/cast/custom.rst @@ -26,7 +26,9 @@ The following Python snippet demonstrates the intended usage from the Python sid def __int__(self): return 123 + from example import print + print(A()) To register the necessary conversion routines, it is necessary to add an diff --git a/docs/advanced/cast/eigen.rst b/docs/advanced/cast/eigen.rst index e01472d5..80f10134 100644 --- a/docs/advanced/cast/eigen.rst +++ b/docs/advanced/cast/eigen.rst @@ -112,7 +112,7 @@ example: .. code-block:: python a = MyClass() - m = a.get_matrix() # flags.writeable = True, flags.owndata = False + m = a.get_matrix() # flags.writeable = True, flags.owndata = False v = a.view_matrix() # flags.writeable = False, flags.owndata = False c = a.copy_matrix() # flags.writeable = True, flags.owndata = True # m[5,6] and v[5,6] refer to the same element, c[5,6] does not. @@ -203,7 +203,7 @@ adding the ``order='F'`` option when creating an array: .. code-block:: python - myarray = np.array(source, order='F') + myarray = np.array(source, order="F") Such an object will be passable to a bound function accepting an ``Eigen::Ref`` (or similar column-major Eigen type). diff --git a/docs/advanced/cast/strings.rst b/docs/advanced/cast/strings.rst index e25701ec..cfd7e7b7 100644 --- a/docs/advanced/cast/strings.rst +++ b/docs/advanced/cast/strings.rst @@ -36,13 +36,13 @@ everywhere `_. } ); -.. code-block:: python +.. code-block:: pycon - >>> utf8_test('🎂') + >>> utf8_test("🎂") utf-8 is icing on the cake. 🎂 - >>> utf8_charptr('🍕') + >>> utf8_charptr("🍕") My favorite food is 🍕 @@ -80,7 +80,7 @@ raise a ``UnicodeDecodeError``. } ); -.. code-block:: python +.. code-block:: pycon >>> isinstance(example.std_string_return(), str) True @@ -114,7 +114,7 @@ conversion has the same overhead as implicit conversion. } ); -.. code-block:: python +.. code-block:: pycon >>> str_output() 'Send your résumé to Alice in HR' @@ -143,7 +143,7 @@ returned to Python as ``bytes``, then one can return the data as a } ); -.. code-block:: python +.. code-block:: pycon >>> example.return_bytes() b'\xba\xd0\xba\xd0' @@ -160,7 +160,7 @@ encoding, but cannot convert ``std::string`` back to ``bytes`` implicitly. } ); -.. code-block:: python +.. code-block:: pycon >>> isinstance(example.asymmetry(b"have some bytes"), str) True @@ -229,16 +229,16 @@ character. m.def("pass_char", [](char c) { return c; }); m.def("pass_wchar", [](wchar_t w) { return w; }); -.. code-block:: python +.. code-block:: pycon - >>> example.pass_char('A') + >>> example.pass_char("A") 'A' While C++ will cast integers to character types (``char c = 0x65;``), pybind11 does not convert Python integers to characters implicitly. The Python function ``chr()`` can be used to convert integers to characters. -.. code-block:: python +.. code-block:: pycon >>> example.pass_char(0x65) TypeError @@ -259,17 +259,17 @@ a combining acute accent). The combining character will be lost if the two-character sequence is passed as an argument, even though it renders as a single grapheme. -.. code-block:: python +.. code-block:: pycon - >>> example.pass_wchar('é') + >>> example.pass_wchar("é") 'é' - >>> combining_e_acute = 'e' + '\u0301' + >>> combining_e_acute = "e" + "\u0301" >>> combining_e_acute 'é' - >>> combining_e_acute == 'é' + >>> combining_e_acute == "é" False >>> example.pass_wchar(combining_e_acute) @@ -278,9 +278,9 @@ single grapheme. Normalizing combining characters before passing the character literal to C++ may resolve *some* of these issues: -.. code-block:: python +.. code-block:: pycon - >>> example.pass_wchar(unicodedata.normalize('NFC', combining_e_acute)) + >>> example.pass_wchar(unicodedata.normalize("NFC", combining_e_acute)) 'é' In some languages (Thai for example), there are `graphemes that cannot be diff --git a/docs/advanced/classes.rst b/docs/advanced/classes.rst index 9d1b1f0d..7f8fcdf4 100644 --- a/docs/advanced/classes.rst +++ b/docs/advanced/classes.rst @@ -136,7 +136,7 @@ a virtual method call. u'woof! woof! woof! ' >>> class Cat(Animal): ... def go(self, n_times): - ... return "meow! " * n_times + ... return "meow! " * n_times ... >>> c = Cat() >>> call_go(c) @@ -159,8 +159,9 @@ Here is an example: class Dachshund(Dog): def __init__(self, name): - Dog.__init__(self) # Without this, a TypeError is raised. + Dog.__init__(self) # Without this, a TypeError is raised. self.name = name + def bark(self): return "yap!" @@ -1153,6 +1154,7 @@ error: >>> class PyFinalChild(IsFinal): ... pass + ... TypeError: type 'IsFinal' is not an acceptable base type .. note:: This attribute is currently ignored on PyPy @@ -1247,7 +1249,7 @@ Accessing the type object You can get the type object from a C++ class that has already been registered using: -.. code-block:: python +.. code-block:: cpp py::type T_py = py::type::of(); diff --git a/docs/advanced/embedding.rst b/docs/advanced/embedding.rst index dfdaad2d..a435b8a7 100644 --- a/docs/advanced/embedding.rst +++ b/docs/advanced/embedding.rst @@ -122,6 +122,7 @@ embedding the interpreter. This makes it easy to import local Python files: """calc.py located in the working directory""" + def add(i, j): return i + j diff --git a/docs/advanced/functions.rst b/docs/advanced/functions.rst index a537cb65..1178d072 100644 --- a/docs/advanced/functions.rst +++ b/docs/advanced/functions.rst @@ -272,7 +272,7 @@ And used in Python as usual: .. code-block:: pycon - >>> print_dict({'foo': 123, 'bar': 'hello'}) + >>> print_dict({"foo": 123, "bar": "hello"}) key=foo, value=123 key=bar, value=hello @@ -377,10 +377,11 @@ argument in a function definition: def f(a, *, b): # a can be positional or via keyword; b must be via keyword pass + f(a=1, b=2) # good f(b=2, a=1) # good - f(1, b=2) # good - f(1, 2) # TypeError: f() takes 1 positional argument but 2 were given + f(1, b=2) # good + f(1, 2) # TypeError: f() takes 1 positional argument but 2 were given Pybind11 provides a ``py::kw_only`` object that allows you to implement the same behaviour by specifying the object between positional and keyword-only diff --git a/docs/advanced/pycpp/numpy.rst b/docs/advanced/pycpp/numpy.rst index cae068e9..30daeeff 100644 --- a/docs/advanced/pycpp/numpy.rst +++ b/docs/advanced/pycpp/numpy.rst @@ -258,8 +258,8 @@ by the compiler. The result is returned as a NumPy array of type .. code-block:: pycon - >>> x = np.array([[1, 3],[5, 7]]) - >>> y = np.array([[2, 4],[6, 8]]) + >>> x = np.array([[1, 3], [5, 7]]) + >>> y = np.array([[2, 4], [6, 8]]) >>> z = 3 >>> result = vectorized_func(x, y, z) @@ -403,7 +403,7 @@ In Python 2, the syntactic sugar ``...`` is not available, but the singleton .. code-block:: python - a = # a NumPy array + a = ... # a NumPy array b = a[0, ..., 0] The function ``py::ellipsis()`` function can be used to perform the same diff --git a/docs/advanced/pycpp/object.rst b/docs/advanced/pycpp/object.rst index 6fa8d070..8bffb83e 100644 --- a/docs/advanced/pycpp/object.rst +++ b/docs/advanced/pycpp/object.rst @@ -173,6 +173,7 @@ Keyword arguments are also supported. In Python, there is the usual call syntax: def f(number, say, to): ... # function code + f(1234, say="hello", to=some_instance) # keyword call in Python In C++, the same call can be made using: diff --git a/docs/advanced/pycpp/utilities.rst b/docs/advanced/pycpp/utilities.rst index 30429e84..bf90a62f 100644 --- a/docs/advanced/pycpp/utilities.rst +++ b/docs/advanced/pycpp/utilities.rst @@ -66,7 +66,7 @@ extra type, `py::scoped_estream_redirect `, is identica except for defaulting to ``std::cerr`` and ``sys.stderr``; this can be useful with `py::call_guard`, which allows multiple items, but uses the default constructor: -.. code-block:: py +.. code-block:: cpp // Alternative: Call single function using call guard m.def("noisy_func", &call_noisy_function, diff --git a/docs/advanced/smart_ptrs.rst b/docs/advanced/smart_ptrs.rst index da57748c..5a222010 100644 --- a/docs/advanced/smart_ptrs.rst +++ b/docs/advanced/smart_ptrs.rst @@ -77,6 +77,7 @@ segmentation fault). .. code-block:: python from example import Parent + print(Parent().get_child()) The problem is that ``Parent::get_child()`` returns a pointer to an instance of diff --git a/docs/benchmark.py b/docs/benchmark.py index 369470c8..f1907936 100644 --- a/docs/benchmark.py +++ b/docs/benchmark.py @@ -2,7 +2,6 @@ import datetime as dt import os import random -import time nfns = 4 # Functions per class nargs = 4 # Arguments per function diff --git a/docs/changelog.rst b/docs/changelog.rst index 8b9690ca..6bdf6a6e 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -1142,6 +1142,7 @@ v2.2.0 (August 31, 2017) from cpp_module import CppBase1, CppBase2 + class PyDerived(CppBase1, CppBase2): def __init__(self): CppBase1.__init__(self) # C++ bases must be initialized explicitly diff --git a/docs/classes.rst b/docs/classes.rst index f3610ef3..a4463e41 100644 --- a/docs/classes.rst +++ b/docs/classes.rst @@ -44,12 +44,12 @@ interactive Python session demonstrating this example is shown below: % python >>> import example - >>> p = example.Pet('Molly') + >>> p = example.Pet("Molly") >>> print(p) >>> p.getName() u'Molly' - >>> p.setName('Charly') + >>> p.setName("Charly") >>> p.getName() u'Charly' @@ -122,10 +122,10 @@ This makes it possible to write .. code-block:: pycon - >>> p = example.Pet('Molly') + >>> p = example.Pet("Molly") >>> p.name u'Molly' - >>> p.name = 'Charly' + >>> p.name = "Charly" >>> p.name u'Charly' @@ -174,10 +174,10 @@ Native Python classes can pick up new attributes dynamically: .. code-block:: pycon >>> class Pet: - ... name = 'Molly' + ... name = "Molly" ... >>> p = Pet() - >>> p.name = 'Charly' # overwrite existing + >>> p.name = "Charly" # overwrite existing >>> p.age = 2 # dynamically add a new attribute By default, classes exported from C++ do not support this and the only writable @@ -195,7 +195,7 @@ Trying to set any other attribute results in an error: .. code-block:: pycon >>> p = example.Pet() - >>> p.name = 'Charly' # OK, attribute defined in C++ + >>> p.name = "Charly" # OK, attribute defined in C++ >>> p.age = 2 # fail AttributeError: 'Pet' object has no attribute 'age' @@ -213,7 +213,7 @@ Now everything works as expected: .. code-block:: pycon >>> p = example.Pet() - >>> p.name = 'Charly' # OK, overwrite value in C++ + >>> p.name = "Charly" # OK, overwrite value in C++ >>> p.age = 2 # OK, dynamically add a new attribute >>> p.__dict__ # just like a native Python class {'age': 2} @@ -280,7 +280,7 @@ expose fields and methods of both types: .. code-block:: pycon - >>> p = example.Dog('Molly') + >>> p = example.Dog("Molly") >>> p.name u'Molly' >>> p.bark() @@ -486,7 +486,7 @@ typed enums. .. code-block:: pycon - >>> p = Pet('Lucy', Pet.Cat) + >>> p = Pet("Lucy", Pet.Cat) >>> p.type Kind.Cat >>> int(p.type) @@ -508,7 +508,7 @@ The ``name`` property returns the name of the enum value as a unicode string. .. code-block:: pycon - >>> p = Pet( "Lucy", Pet.Cat ) + >>> p = Pet("Lucy", Pet.Cat) >>> pet_type = p.type >>> pet_type Pet.Cat diff --git a/docs/compiling.rst b/docs/compiling.rst index eaf3270e..07385104 100644 --- a/docs/compiling.rst +++ b/docs/compiling.rst @@ -42,10 +42,7 @@ An example of a ``setup.py`` using pybind11's helpers: ), ] - setup( - ..., - ext_modules=ext_modules - ) + setup(..., ext_modules=ext_modules) If you want to do an automatic search for the highest supported C++ standard, that is supported via a ``build_ext`` command override; it will only affect @@ -64,11 +61,7 @@ that is supported via a ``build_ext`` command override; it will only affect ), ] - setup( - ..., - cmdclass={"build_ext": build_ext}, - ext_modules=ext_modules - ) + setup(..., cmdclass={"build_ext": build_ext}, ext_modules=ext_modules) If you have single-file extension modules that are directly stored in the Python source tree (``foo.cpp`` in the same directory as where a ``foo.py`` diff --git a/docs/conf.py b/docs/conf.py index 458a8688..092e274e 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -15,7 +15,6 @@ import os import re -import shlex import subprocess import sys from pathlib import Path diff --git a/docs/faq.rst b/docs/faq.rst index d6a048b0..e2f477b1 100644 --- a/docs/faq.rst +++ b/docs/faq.rst @@ -54,7 +54,7 @@ provided by the caller -- in fact, it does nothing at all. .. code-block:: python def increment(i): - i += 1 # nope.. + i += 1 # nope.. pybind11 is also affected by such language-level conventions, which means that binding ``increment`` or ``increment_ptr`` will also create Python functions From b06a6f4f6294dd3550b1a2b6053421d8df145d3c Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Wed, 22 Sep 2021 17:41:56 -0400 Subject: [PATCH 077/100] feat: Slice allowing None with py::object or std::optional (#1101) * Adding nullptr slices Using example from #1095 Some fixes from @wjakob's review Stop clang-tidy from complaining New proposal for py::slice constructor Eric's suggested changes: simplify testing; shift def's * chore: drop MSVC pragma (hopefully unneeded) * Apply suggestions from code review --- include/pybind11/detail/common.h | 21 ++++++++++++++++++- include/pybind11/pytypes.h | 25 ++++++++++++++++++++--- include/pybind11/stl.h | 28 ++++++++------------------ tests/test_sequences_and_iterators.cpp | 17 ++++++++++++++++ tests/test_sequences_and_iterators.py | 11 ++++++++++ 5 files changed, 78 insertions(+), 24 deletions(-) diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 8aeb79fb..3bd84da5 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -161,7 +161,26 @@ // https://en.cppreference.com/w/c/chrono/localtime #if defined(__STDC_LIB_EXT1__) && !defined(__STDC_WANT_LIB_EXT1__) -# define __STDC_WANT_LIB_EXT1__ +# define __STDC_WANT_LIB_EXT1__ +#endif + +#ifdef __has_include +// std::optional (but including it in c++14 mode isn't allowed) +# if defined(PYBIND11_CPP17) && __has_include() +# define PYBIND11_HAS_OPTIONAL 1 +# endif +// std::experimental::optional (but not allowed in c++11 mode) +# if defined(PYBIND11_CPP14) && (__has_include() && \ + !__has_include()) +# define PYBIND11_HAS_EXP_OPTIONAL 1 +# endif +// std::variant +# if defined(PYBIND11_CPP17) && __has_include() +# define PYBIND11_HAS_VARIANT 1 +# endif +#elif defined(_MSC_VER) && defined(PYBIND11_CPP17) +# define PYBIND11_HAS_OPTIONAL 1 +# define PYBIND11_HAS_VARIANT 1 #endif #include diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index d1d3dcb0..383663b5 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -14,6 +14,10 @@ #include #include +#if defined(PYBIND11_HAS_OPTIONAL) +# include +#endif + PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) /* A few forward declarations */ @@ -1345,11 +1349,20 @@ private: class slice : public object { public: PYBIND11_OBJECT_DEFAULT(slice, object, PySlice_Check) - slice(ssize_t start_, ssize_t stop_, ssize_t step_) { - int_ start(start_), stop(stop_), step(step_); + slice(handle start, handle stop, handle step) { m_ptr = PySlice_New(start.ptr(), stop.ptr(), step.ptr()); - if (!m_ptr) pybind11_fail("Could not allocate slice object!"); + if (!m_ptr) + pybind11_fail("Could not allocate slice object!"); } + +#ifdef PYBIND11_HAS_OPTIONAL + slice(std::optional start, std::optional stop, std::optional step) + : slice(index_to_object(start), index_to_object(stop), index_to_object(step)) {} +#else + slice(ssize_t start_, ssize_t stop_, ssize_t step_) + : slice(int_(start_), int_(stop_), int_(step_)) {} +#endif + bool compute(size_t length, size_t *start, size_t *stop, size_t *step, size_t *slicelength) const { return PySlice_GetIndicesEx((PYBIND11_SLICE_OBJECT *) m_ptr, @@ -1364,6 +1377,12 @@ public: stop, step, slicelength) == 0; } + +private: + template + static object index_to_object(T index) { + return index ? object(int_(*index)) : object(none()); + } }; class capsule : public object { diff --git a/include/pybind11/stl.h b/include/pybind11/stl.h index 99b49d0e..2c017b4f 100644 --- a/include/pybind11/stl.h +++ b/include/pybind11/stl.h @@ -9,6 +9,7 @@ #pragma once +#include "detail/common.h" #include "pybind11.h" #include #include @@ -19,28 +20,15 @@ #include #include -#ifdef __has_include -// std::optional (but including it in c++14 mode isn't allowed) -# if defined(PYBIND11_CPP17) && __has_include() -# include -# define PYBIND11_HAS_OPTIONAL 1 -# endif -// std::experimental::optional (but not allowed in c++11 mode) -# if defined(PYBIND11_CPP14) && (__has_include() && \ - !__has_include()) -# include -# define PYBIND11_HAS_EXP_OPTIONAL 1 -# endif -// std::variant -# if defined(PYBIND11_CPP17) && __has_include() -# include -# define PYBIND11_HAS_VARIANT 1 -# endif -#elif defined(_MSC_VER) && defined(PYBIND11_CPP17) +// See `detail/common.h` for implementation of these guards. +#if defined(PYBIND11_HAS_OPTIONAL) # include +#elif defined(PYBIND11_HAS_EXP_OPTIONAL) +# include +#endif + +#if defined(PYBIND11_HAS_VARIANT) # include -# define PYBIND11_HAS_OPTIONAL 1 -# define PYBIND11_HAS_VARIANT 1 #endif PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) diff --git a/tests/test_sequences_and_iterators.cpp b/tests/test_sequences_and_iterators.cpp index 66d64726..72d96cb4 100644 --- a/tests/test_sequences_and_iterators.cpp +++ b/tests/test_sequences_and_iterators.cpp @@ -17,6 +17,11 @@ #include #include +#ifdef PYBIND11_HAS_OPTIONAL +#include +#endif // PYBIND11_HAS_OPTIONAL + + template class NonZeroIterator { const T* ptr_; @@ -117,6 +122,18 @@ TEST_SUBMODULE(sequences_and_iterators, m) { return std::make_tuple(istart, istop, istep); }); + m.def("make_forward_slice_size_t", []() { return py::slice(0, -1, 1); }); + m.def("make_reversed_slice_object", []() { return py::slice(py::none(), py::none(), py::int_(-1)); }); +#ifdef PYBIND11_HAS_OPTIONAL + m.attr("has_optional") = true; + m.def("make_reversed_slice_size_t_optional_verbose", []() { return py::slice(std::nullopt, std::nullopt, -1); }); + // Warning: The following spelling may still compile if optional<> is not present and give wrong answers. + // Please use with caution. + m.def("make_reversed_slice_size_t_optional", []() { return py::slice({}, {}, -1); }); +#else + m.attr("has_optional") = false; +#endif + // test_sequence class Sequence { public: diff --git a/tests/test_sequences_and_iterators.py b/tests/test_sequences_and_iterators.py index 2c73eff2..79689391 100644 --- a/tests/test_sequences_and_iterators.py +++ b/tests/test_sequences_and_iterators.py @@ -16,6 +16,17 @@ def allclose(a_list, b_list, rel_tol=1e-05, abs_tol=0.0): ) +def test_slice_constructors(): + assert m.make_forward_slice_size_t() == slice(0, -1, 1) + assert m.make_reversed_slice_object() == slice(None, None, -1) + + +@pytest.mark.skipif(not m.has_optional, reason="no ") +def test_slice_constructors_explicit_optional(): + assert m.make_reversed_slice_size_t_optional() == slice(None, None, -1) + assert m.make_reversed_slice_size_t_optional_verbose() == slice(None, None, -1) + + def test_generalized_iterators(): assert list(m.IntPairs([(1, 2), (3, 4), (0, 5)]).nonzero()) == [(1, 2), (3, 4)] assert list(m.IntPairs([(1, 2), (2, 0), (0, 3), (4, 5)]).nonzero()) == [(1, 2)] From 1dc9a23caea407db3b7e148b1e9cb962a235b5ed Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 22 Sep 2021 22:38:04 -0400 Subject: [PATCH 078/100] chore(deps): bump jwlawson/actions-setup-cmake from 1.10 to 1.11 (#3294) Bumps [jwlawson/actions-setup-cmake](https://github.com/jwlawson/actions-setup-cmake) from 1.10 to 1.11. - [Release notes](https://github.com/jwlawson/actions-setup-cmake/releases) - [Commits](https://github.com/jwlawson/actions-setup-cmake/compare/v1.10...v1.11) --- updated-dependencies: - dependency-name: jwlawson/actions-setup-cmake dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/ci.yml | 12 ++++++------ .github/workflows/configure.yml | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2348212f..0b4968cc 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -75,7 +75,7 @@ jobs: run: brew install boost - name: Update CMake - uses: jwlawson/actions-setup-cmake@v1.10 + uses: jwlawson/actions-setup-cmake@v1.11 - name: Cache wheels if: runner.os == 'macOS' @@ -211,7 +211,7 @@ jobs: debug: ${{ matrix.python-debug }} - name: Update CMake - uses: jwlawson/actions-setup-cmake@v1.10 + uses: jwlawson/actions-setup-cmake@v1.11 - name: Valgrind cache if: matrix.valgrind @@ -463,7 +463,7 @@ jobs: run: python3 -m pip install --upgrade pip - name: Update CMake - uses: jwlawson/actions-setup-cmake@v1.10 + uses: jwlawson/actions-setup-cmake@v1.11 - name: Configure shell: bash @@ -756,7 +756,7 @@ jobs: architecture: x86 - name: Update CMake - uses: jwlawson/actions-setup-cmake@v1.10 + uses: jwlawson/actions-setup-cmake@v1.11 - name: Prepare MSVC uses: ilammy/msvc-dev-cmd@v1.9.0 @@ -802,7 +802,7 @@ jobs: python-version: ${{ matrix.python }} - name: Update CMake - uses: jwlawson/actions-setup-cmake@v1.10 + uses: jwlawson/actions-setup-cmake@v1.11 - name: Prepare MSVC uses: ilammy/msvc-dev-cmd@v1.9.0 @@ -856,7 +856,7 @@ jobs: python-version: ${{ matrix.python }} - name: Update CMake - uses: jwlawson/actions-setup-cmake@v1.10 + uses: jwlawson/actions-setup-cmake@v1.11 - name: Prepare env run: python -m pip install -r tests/requirements.txt --prefer-binary diff --git a/.github/workflows/configure.yml b/.github/workflows/configure.yml index d37fa3c2..d60025e5 100644 --- a/.github/workflows/configure.yml +++ b/.github/workflows/configure.yml @@ -55,7 +55,7 @@ jobs: # An action for adding a specific version of CMake: # https://github.com/jwlawson/actions-setup-cmake - name: Setup CMake ${{ matrix.cmake }} - uses: jwlawson/actions-setup-cmake@v1.10 + uses: jwlawson/actions-setup-cmake@v1.11 with: cmake-version: ${{ matrix.cmake }} From 2fa3fcfda5bb1aa1e8efc4d9cf90951e8055375e Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Wed, 22 Sep 2021 22:50:29 -0400 Subject: [PATCH 079/100] Revert "Add make_value_iterator (#3271)" This reverts commit ee0c5ee405e7a532410797687da28a20b89cd62b. --- docs/reference.rst | 3 - include/pybind11/pybind11.h | 118 +++++++------------------ tests/test_sequences_and_iterators.cpp | 58 ------------ tests/test_sequences_and_iterators.py | 29 ------ 4 files changed, 32 insertions(+), 176 deletions(-) diff --git a/docs/reference.rst b/docs/reference.rst index e64a0351..a678d41c 100644 --- a/docs/reference.rst +++ b/docs/reference.rst @@ -63,9 +63,6 @@ Convenience functions converting to Python types .. doxygenfunction:: make_key_iterator(Iterator, Sentinel, Extra &&...) .. doxygenfunction:: make_key_iterator(Type &, Extra&&...) -.. doxygenfunction:: make_value_iterator(Iterator, Sentinel, Extra &&...) -.. doxygenfunction:: make_value_iterator(Type &, Extra&&...) - .. _extras: Passing extra arguments to ``def`` or ``class_`` diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index ac95b3a3..b8f5a6ba 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -1955,52 +1955,25 @@ inline std::pair all_t return res; } -/* There are a large number of apparently unused template arguments because - * each combination requires a separate py::class_ registration. - */ -template +template struct iterator_state { Iterator it; Sentinel end; bool first_or_done; }; -// Note: these helpers take the iterator by non-const reference because some -// iterators in the wild can't be dereferenced when const. -template -struct iterator_access { - using result_type = decltype((*std::declval())); - // NOLINTNEXTLINE(readability-const-return-type) // PR #3263 - result_type operator()(Iterator &it) const { - return *it; - } -}; +PYBIND11_NAMESPACE_END(detail) -template -struct iterator_key_access { - using result_type = decltype(((*std::declval()).first)); - result_type operator()(Iterator &it) const { - return (*it).first; - } -}; - -template -struct iterator_value_access { - using result_type = decltype(((*std::declval()).second)); - result_type operator()(Iterator &it) const { - return (*it).second; - } -}; - -template ()), +#endif typename... Extra> -iterator make_iterator_impl(Iterator first, Sentinel last, Extra &&... extra) { - using state = detail::iterator_state; - // TODO: state captures only the types of Extra, not the values +iterator make_iterator(Iterator first, Sentinel last, Extra &&... extra) { + using state = detail::iterator_state; if (!detail::get_type_info(typeid(state), false)) { class_(handle(), "iterator", pybind11::module_local()) @@ -2014,7 +1987,7 @@ iterator make_iterator_impl(Iterator first, Sentinel last, Extra &&... extra) { s.first_or_done = true; throw stop_iteration(); } - return Access()(s.it); + return *s.it; // NOLINTNEXTLINE(readability-const-return-type) // PR #3263 }, std::forward(extra)..., Policy); } @@ -2022,55 +1995,35 @@ iterator make_iterator_impl(Iterator first, Sentinel last, Extra &&... extra) { return cast(state{first, last, true}); } -PYBIND11_NAMESPACE_END(detail) - -/// Makes a python iterator from a first and past-the-end C++ InputIterator. -template ::result_type, - typename... Extra> -iterator make_iterator(Iterator first, Sentinel last, Extra &&... extra) { - return detail::make_iterator_impl< - detail::iterator_access, - Policy, - Iterator, - Sentinel, - ValueType, - Extra...>(first, last, std::forward(extra)...); -} - -/// Makes a python iterator over the keys (`.first`) of a iterator over pairs from a +/// Makes an python iterator over the keys (`.first`) of a iterator over pairs from a /// first and past-the-end InputIterator. template ::result_type, +#ifndef DOXYGEN_SHOULD_SKIP_THIS // Issue in breathe 4.26.1 + typename KeyType = decltype((*std::declval()).first), +#endif typename... Extra> iterator make_key_iterator(Iterator first, Sentinel last, Extra &&...extra) { - return detail::make_iterator_impl< - detail::iterator_key_access, - Policy, - Iterator, - Sentinel, - KeyType, - Extra...>(first, last, std::forward(extra)...); -} + using state = detail::iterator_state; -/// Makes a python iterator over the values (`.second`) of a iterator over pairs from a -/// first and past-the-end InputIterator. -template ::result_type, - typename... Extra> -iterator make_value_iterator(Iterator first, Sentinel last, Extra &&...extra) { - return detail::make_iterator_impl< - detail::iterator_value_access, - Policy, Iterator, - Sentinel, - ValueType, - Extra...>(first, last, std::forward(extra)...); + if (!detail::get_type_info(typeid(state), false)) { + class_(handle(), "iterator", pybind11::module_local()) + .def("__iter__", [](state &s) -> state& { return s; }) + .def("__next__", [](state &s) -> detail::remove_cv_t { + if (!s.first_or_done) + ++s.it; + else + s.first_or_done = false; + if (s.it == s.end) { + s.first_or_done = true; + throw stop_iteration(); + } + return (*s.it).first; + }, std::forward(extra)..., Policy); + } + + return cast(state{first, last, true}); } /// Makes an iterator over values of an stl container or other container supporting @@ -2087,13 +2040,6 @@ template (std::begin(value), std::end(value), extra...); } -/// Makes an iterator over the values (`.second`) of a stl map-like container supporting -/// `std::begin()`/`std::end()` -template iterator make_value_iterator(Type &value, Extra&&... extra) { - return make_value_iterator(std::begin(value), std::end(value), extra...); -} - template void implicitly_convertible() { struct set_flag { bool &flag; diff --git a/tests/test_sequences_and_iterators.cpp b/tests/test_sequences_and_iterators.cpp index 72d96cb4..f4a78ae9 100644 --- a/tests/test_sequences_and_iterators.cpp +++ b/tests/test_sequences_and_iterators.cpp @@ -15,7 +15,6 @@ #include #include -#include #ifdef PYBIND11_HAS_OPTIONAL #include @@ -38,29 +37,6 @@ bool operator==(const NonZeroIterator>& it, const NonZeroSentine return !(*it).first || !(*it).second; } -class NonCopyableInt { -public: - explicit NonCopyableInt(int value) : value_(value) {} - NonCopyableInt(const NonCopyableInt &) = delete; - NonCopyableInt(NonCopyableInt &&other) noexcept : value_(other.value_) { - other.value_ = -1; // detect when an unwanted move occurs - } - NonCopyableInt &operator=(const NonCopyableInt &) = delete; - NonCopyableInt &operator=(NonCopyableInt &&other) noexcept { - value_ = other.value_; - other.value_ = -1; // detect when an unwanted move occurs - return *this; - } - int get() const { return value_; } - void set(int value) { value_ = value; } - ~NonCopyableInt() = default; -private: - int value_; -}; -using NonCopyableIntPair = std::pair; -PYBIND11_MAKE_OPAQUE(std::vector); -PYBIND11_MAKE_OPAQUE(std::vector); - template py::list test_random_access_iterator(PythonType x) { if (x.size() < 5) @@ -312,10 +288,6 @@ TEST_SUBMODULE(sequences_and_iterators, m) { .def( "items", [](const StringMap &map) { return py::make_iterator(map.begin(), map.end()); }, - py::keep_alive<0, 1>()) - .def( - "values", - [](const StringMap &map) { return py::make_value_iterator(map.begin(), map.end()); }, py::keep_alive<0, 1>()); // test_generalized_iterators @@ -334,38 +306,8 @@ TEST_SUBMODULE(sequences_and_iterators, m) { .def("nonzero_keys", [](const IntPairs& s) { return py::make_key_iterator(NonZeroIterator>(s.begin()), NonZeroSentinel()); }, py::keep_alive<0, 1>()) - .def("nonzero_values", [](const IntPairs& s) { - return py::make_value_iterator(NonZeroIterator>(s.begin()), NonZeroSentinel()); - }, py::keep_alive<0, 1>()) ; - // test_iterater_referencing - py::class_(m, "NonCopyableInt") - .def(py::init()) - .def("set", &NonCopyableInt::set) - .def("__int__", &NonCopyableInt::get) - ; - py::class_>(m, "VectorNonCopyableInt") - .def(py::init<>()) - .def("append", [](std::vector &vec, int value) { - vec.emplace_back(value); - }) - .def("__iter__", [](std::vector &vec) { - return py::make_iterator(vec.begin(), vec.end()); - }) - ; - py::class_>(m, "VectorNonCopyableIntPair") - .def(py::init<>()) - .def("append", [](std::vector &vec, const std::pair &value) { - vec.emplace_back(NonCopyableInt(value.first), NonCopyableInt(value.second)); - }) - .def("keys", [](std::vector &vec) { - return py::make_key_iterator(vec.begin(), vec.end()); - }) - .def("values", [](std::vector &vec) { - return py::make_value_iterator(vec.begin(), vec.end()); - }) - ; #if 0 // Obsolete: special data structure for exposing custom iterator types to python diff --git a/tests/test_sequences_and_iterators.py b/tests/test_sequences_and_iterators.py index 79689391..44069fdd 100644 --- a/tests/test_sequences_and_iterators.py +++ b/tests/test_sequences_and_iterators.py @@ -36,10 +36,6 @@ def test_generalized_iterators(): assert list(m.IntPairs([(1, 2), (2, 0), (0, 3), (4, 5)]).nonzero_keys()) == [1] assert list(m.IntPairs([(0, 3), (1, 2), (3, 4)]).nonzero_keys()) == [] - assert list(m.IntPairs([(1, 2), (3, 4), (0, 5)]).nonzero_values()) == [2, 4] - assert list(m.IntPairs([(1, 2), (2, 0), (0, 3), (4, 5)]).nonzero_values()) == [2] - assert list(m.IntPairs([(0, 3), (1, 2), (3, 4)]).nonzero_values()) == [] - # __next__ must continue to raise StopIteration it = m.IntPairs([(0, 0)]).nonzero() for _ in range(3): @@ -52,30 +48,6 @@ def test_generalized_iterators(): next(it) -def test_iterator_referencing(): - """Test that iterators reference rather than copy their referents.""" - vec = m.VectorNonCopyableInt() - vec.append(3) - vec.append(5) - assert [int(x) for x in vec] == [3, 5] - # Increment everything to make sure the referents can be mutated - for x in vec: - x.set(int(x) + 1) - assert [int(x) for x in vec] == [4, 6] - - vec = m.VectorNonCopyableIntPair() - vec.append([3, 4]) - vec.append([5, 7]) - assert [int(x) for x in vec.keys()] == [3, 5] - assert [int(x) for x in vec.values()] == [4, 7] - for x in vec.keys(): - x.set(int(x) + 1) - for x in vec.values(): - x.set(int(x) + 10) - assert [int(x) for x in vec.keys()] == [4, 6] - assert [int(x) for x in vec.values()] == [14, 17] - - def test_sliceable(): sliceable = m.Sliceable(100) assert sliceable[::] == (0, 100, 1) @@ -179,7 +151,6 @@ def test_map_iterator(): assert sm[k] == expected[k] for k, v in sm.items(): assert v == expected[k] - assert list(sm.values()) == [expected[k] for k in sm] it = iter(m.StringMap({})) for _ in range(3): # __next__ must continue to raise StopIteration From 5f46e47da8ef4c43252a934e9f461296140df1d2 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 23 Sep 2021 08:01:06 -0400 Subject: [PATCH 080/100] tests: check simple iteration of pairs (#3296) --- tests/test_sequences_and_iterators.cpp | 13 +++++++++++++ tests/test_sequences_and_iterators.py | 9 +++++++++ 2 files changed, 22 insertions(+) diff --git a/tests/test_sequences_and_iterators.cpp b/tests/test_sequences_and_iterators.cpp index f4a78ae9..16f8f5b2 100644 --- a/tests/test_sequences_and_iterators.cpp +++ b/tests/test_sequences_and_iterators.cpp @@ -295,6 +295,8 @@ TEST_SUBMODULE(sequences_and_iterators, m) { public: explicit IntPairs(std::vector> data) : data_(std::move(data)) {} const std::pair* begin() const { return data_.data(); } + // .end() only required for py::make_iterator(self) overload + const std::pair* end() const { return data_.data() + data_.size(); } private: std::vector> data_; }; @@ -306,6 +308,17 @@ TEST_SUBMODULE(sequences_and_iterators, m) { .def("nonzero_keys", [](const IntPairs& s) { return py::make_key_iterator(NonZeroIterator>(s.begin()), NonZeroSentinel()); }, py::keep_alive<0, 1>()) + .def("simple_iterator", [](IntPairs& self) { + return py::make_iterator(self); + }, py::keep_alive<0, 1>()) + .def("simple_keys", [](IntPairs& self) { + return py::make_key_iterator(self); + }, py::keep_alive<0, 1>()) + + // test iterator with keep_alive (doesn't work so not used at runtime, but tests compile) + .def("make_iterator_keep_alive", [](IntPairs& self) { + return py::make_iterator(self, py::keep_alive<0, 1>()); + }, py::keep_alive<0, 1>()) ; diff --git a/tests/test_sequences_and_iterators.py b/tests/test_sequences_and_iterators.py index 44069fdd..902f4914 100644 --- a/tests/test_sequences_and_iterators.py +++ b/tests/test_sequences_and_iterators.py @@ -48,6 +48,15 @@ def test_generalized_iterators(): next(it) +def test_generalized_iterators_simple(): + assert list(m.IntPairs([(1, 2), (3, 4), (0, 5)]).simple_iterator()) == [ + (1, 2), + (3, 4), + (0, 5), + ] + assert list(m.IntPairs([(1, 2), (3, 4), (0, 5)]).simple_keys()) == [1, 3, 0] + + def test_sliceable(): sliceable = m.Sliceable(100) assert sliceable[::] == (0, 100, 1) From 2a78abffd85904849a1c9f9c6705f2d5a0d27018 Mon Sep 17 00:00:00 2001 From: Jeremy Maitin-Shepard Date: Thu, 23 Sep 2021 10:36:25 -0700 Subject: [PATCH 081/100] Ensure PYBIND11_TLS_REPLACE_VALUE evaluates its arguments only once (#3290) --- include/pybind11/detail/internals.h | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/include/pybind11/detail/internals.h b/include/pybind11/detail/internals.h index 7c2e4997..98d21eb9 100644 --- a/include/pybind11/detail/internals.h +++ b/include/pybind11/detail/internals.h @@ -84,12 +84,13 @@ inline PyObject *make_object_base_type(PyTypeObject *metaclass); // On CPython < 3.4 and on PyPy, `PyThread_set_key_value` strangely does not set // the value if it has already been set. Instead, it must first be deleted and // then set again. +inline void tls_replace_value(PYBIND11_TLS_KEY_REF key, void *value) { + PyThread_delete_key_value(key); + PyThread_set_key_value(key, value); +} # define PYBIND11_TLS_DELETE_VALUE(key) PyThread_delete_key_value(key) # define PYBIND11_TLS_REPLACE_VALUE(key, value) \ - do { \ - PyThread_delete_key_value((key)); \ - PyThread_set_key_value((key), (value)); \ - } while (false) + ::pybind11::detail::tls_replace_value((key), (value)) # else # define PYBIND11_TLS_DELETE_VALUE(key) PyThread_set_key_value((key), nullptr) # define PYBIND11_TLS_REPLACE_VALUE(key, value) PyThread_set_key_value((key), (value)) From 21282e645a638600be9bf44dd535dbae5416005e Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 23 Sep 2021 15:06:07 -0400 Subject: [PATCH 082/100] feat: reapply fixed version of #3271 (#3293) * Add make_value_iterator (#3271) * Add make_value_iterator This is the counterpart to make_key_iterator, and will allow implementing a `value` method in `bind_map` (although doing so is left for a subsequent PR). I made a few design changes to reduce copy-and-paste boilerplate. Previously detail::iterator_state had a boolean template parameter to indicate whether it was being used for make_iterator or make_key_iterator. I replaced the boolean with a class that determines how to dereference the iterator. This allows for a generic implementation of `__next__`. I also added the ValueType and Extra... parameters to the iterator_state template args, because I think it was a bug that they were missing: if make_iterator is called twice with different values of these, only the first set has effect (because the state class is only registered once). There is still a potential issue in that the *values* of the extra arguments are latched on the first call, but since most policies are empty classes this should be even less common. * Add some remove_cv_t to appease clang-tidy * Make iterator_access and friends take reference For some reason I'd accidentally made it take a const value, which caused some issues with third-party packages. * Another attempt to remove remove_cv_t from iterators Some of the return types were const (non-reference) types because of the pecularities of decltype: `decltype((*it).first)` is the *declared* type of the member of the pair, rather than the type of the expression. So if the reference type of the iterator is `pair &`, then the decltype is `const int`. Wrapping an extra set of parentheses to form `decltype(((*it).first))` would instead give `const int &`. This means that the existing make_key_iterator actually returns by value from `__next__`, rather than by reference. Since for mapping types, keys are always const, this probably hasn't been noticed, but it will affect make_value_iterator if the Python code tries to mutate the returned objects. I've changed things to use double parentheses so that make_iterator, make_key_iterator and make_value_iterator should now all return the reference type of the iterator. I'll still need to add a test for that; for now I'm just checking whether I can keep Clang-Tidy happy. * Add back some NOLINTNEXTLINE to appease Clang-Tidy This is favoured over using remove_cv_t because in some cases a const value return type is deliberate (particularly for Eigen). * Add a unit test for iterator referencing Ensure that make_iterator, make_key_iterator and make_value_iterator return references to the container elements, rather than copies. The test for make_key_iterator fails to compile on master, which gives me confidence that this branch has fixed it. * Make the iterator_access etc operator() const I'm actually a little surprised it compiled at all given that the operator() is called on a temporary, but I don't claim to fully understand all the different value types in C++11. * Attempt to work around compiler bugs https://godbolt.org/ shows an example where ICC gets the wrong result for a decltype used as the default for a template argument, and CI also showed problems with PGI. This is a shot in the dark to see if it fixes things. * Make a test constructor explicit (Clang-Tidy) * Fix unit test on GCC 4.8.5 It seems to require the arguments to the std::pair constructor to be implicitly convertible to the types in the pair, rather than just requiring is_constructible. * Remove DOXYGEN_SHOULD_SKIP_THIS guards Now that a complex decltype expression has been replaced by a simpler nested type, I'm hoping Doxygen will be able to build it without issues. * Add comment to explain iterator_state template params * fix: regression in #3271 Co-authored-by: Bruce Merry <1963944+bmerry@users.noreply.github.com> --- docs/reference.rst | 3 + include/pybind11/pybind11.h | 120 ++++++++++++++++++------- tests/test_sequences_and_iterators.cpp | 77 +++++++++++++++- tests/test_sequences_and_iterators.py | 30 +++++++ 4 files changed, 195 insertions(+), 35 deletions(-) diff --git a/docs/reference.rst b/docs/reference.rst index a678d41c..e64a0351 100644 --- a/docs/reference.rst +++ b/docs/reference.rst @@ -63,6 +63,9 @@ Convenience functions converting to Python types .. doxygenfunction:: make_key_iterator(Iterator, Sentinel, Extra &&...) .. doxygenfunction:: make_key_iterator(Type &, Extra&&...) +.. doxygenfunction:: make_value_iterator(Iterator, Sentinel, Extra &&...) +.. doxygenfunction:: make_value_iterator(Type &, Extra&&...) + .. _extras: Passing extra arguments to ``def`` or ``class_`` diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index b8f5a6ba..16535f19 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -1955,25 +1955,54 @@ inline std::pair all_t return res; } -template +/* There are a large number of apparently unused template arguments because + * each combination requires a separate py::class_ registration. + */ +template struct iterator_state { Iterator it; Sentinel end; bool first_or_done; }; -PYBIND11_NAMESPACE_END(detail) +// Note: these helpers take the iterator by non-const reference because some +// iterators in the wild can't be dereferenced when const. C++ needs the extra parens in decltype +// to enforce an lvalue. The & after Iterator is required for MSVC < 16.9. SFINAE cannot be +// reused for result_type due to bugs in ICC, NVCC, and PGI compilers. See PR #3293. +template ()))> +struct iterator_access { + using result_type = decltype((*std::declval())); + // NOLINTNEXTLINE(readability-const-return-type) // PR #3263 + result_type operator()(Iterator &it) const { + return *it; + } +}; -/// Makes a python iterator from a first and past-the-end C++ InputIterator. -template ()).first)) > +struct iterator_key_access { + using result_type = decltype(((*std::declval()).first)); + result_type operator()(Iterator &it) const { + return (*it).first; + } +}; + +template ()).second))> +struct iterator_value_access { + using result_type = decltype(((*std::declval()).second)); + result_type operator()(Iterator &it) const { + return (*it).second; + } +}; + +template ()), -#endif + typename ValueType, typename... Extra> -iterator make_iterator(Iterator first, Sentinel last, Extra &&... extra) { - using state = detail::iterator_state; +iterator make_iterator_impl(Iterator first, Sentinel last, Extra &&... extra) { + using state = detail::iterator_state; + // TODO: state captures only the types of Extra, not the values if (!detail::get_type_info(typeid(state), false)) { class_(handle(), "iterator", pybind11::module_local()) @@ -1987,7 +2016,7 @@ iterator make_iterator(Iterator first, Sentinel last, Extra &&... extra) { s.first_or_done = true; throw stop_iteration(); } - return *s.it; + return Access()(s.it); // NOLINTNEXTLINE(readability-const-return-type) // PR #3263 }, std::forward(extra)..., Policy); } @@ -1995,35 +2024,55 @@ iterator make_iterator(Iterator first, Sentinel last, Extra &&... extra) { return cast(state{first, last, true}); } -/// Makes an python iterator over the keys (`.first`) of a iterator over pairs from a +PYBIND11_NAMESPACE_END(detail) + +/// Makes a python iterator from a first and past-the-end C++ InputIterator. +template ::result_type, + typename... Extra> +iterator make_iterator(Iterator first, Sentinel last, Extra &&... extra) { + return detail::make_iterator_impl< + detail::iterator_access, + Policy, + Iterator, + Sentinel, + ValueType, + Extra...>(first, last, std::forward(extra)...); +} + +/// Makes a python iterator over the keys (`.first`) of a iterator over pairs from a /// first and past-the-end InputIterator. template ()).first), -#endif + typename KeyType = typename detail::iterator_key_access::result_type, typename... Extra> iterator make_key_iterator(Iterator first, Sentinel last, Extra &&...extra) { - using state = detail::iterator_state; + return detail::make_iterator_impl< + detail::iterator_key_access, + Policy, + Iterator, + Sentinel, + KeyType, + Extra...>(first, last, std::forward(extra)...); +} - if (!detail::get_type_info(typeid(state), false)) { - class_(handle(), "iterator", pybind11::module_local()) - .def("__iter__", [](state &s) -> state& { return s; }) - .def("__next__", [](state &s) -> detail::remove_cv_t { - if (!s.first_or_done) - ++s.it; - else - s.first_or_done = false; - if (s.it == s.end) { - s.first_or_done = true; - throw stop_iteration(); - } - return (*s.it).first; - }, std::forward(extra)..., Policy); - } - - return cast(state{first, last, true}); +/// Makes a python iterator over the values (`.second`) of a iterator over pairs from a +/// first and past-the-end InputIterator. +template ::result_type, + typename... Extra> +iterator make_value_iterator(Iterator first, Sentinel last, Extra &&...extra) { + return detail::make_iterator_impl< + detail::iterator_value_access, + Policy, Iterator, + Sentinel, + ValueType, + Extra...>(first, last, std::forward(extra)...); } /// Makes an iterator over values of an stl container or other container supporting @@ -2040,6 +2089,13 @@ template (std::begin(value), std::end(value), extra...); } +/// Makes an iterator over the values (`.second`) of a stl map-like container supporting +/// `std::begin()`/`std::end()` +template iterator make_value_iterator(Type &value, Extra&&... extra) { + return make_value_iterator(std::begin(value), std::end(value), extra...); +} + template void implicitly_convertible() { struct set_flag { bool &flag; diff --git a/tests/test_sequences_and_iterators.cpp b/tests/test_sequences_and_iterators.cpp index 16f8f5b2..9de69338 100644 --- a/tests/test_sequences_and_iterators.cpp +++ b/tests/test_sequences_and_iterators.cpp @@ -15,6 +15,7 @@ #include #include +#include #ifdef PYBIND11_HAS_OPTIONAL #include @@ -37,6 +38,29 @@ bool operator==(const NonZeroIterator>& it, const NonZeroSentine return !(*it).first || !(*it).second; } +class NonCopyableInt { +public: + explicit NonCopyableInt(int value) : value_(value) {} + NonCopyableInt(const NonCopyableInt &) = delete; + NonCopyableInt(NonCopyableInt &&other) noexcept : value_(other.value_) { + other.value_ = -1; // detect when an unwanted move occurs + } + NonCopyableInt &operator=(const NonCopyableInt &) = delete; + NonCopyableInt &operator=(NonCopyableInt &&other) noexcept { + value_ = other.value_; + other.value_ = -1; // detect when an unwanted move occurs + return *this; + } + int get() const { return value_; } + void set(int value) { value_ = value; } + ~NonCopyableInt() = default; +private: + int value_; +}; +using NonCopyableIntPair = std::pair; +PYBIND11_MAKE_OPAQUE(std::vector); +PYBIND11_MAKE_OPAQUE(std::vector); + template py::list test_random_access_iterator(PythonType x) { if (x.size() < 5) @@ -288,6 +312,10 @@ TEST_SUBMODULE(sequences_and_iterators, m) { .def( "items", [](const StringMap &map) { return py::make_iterator(map.begin(), map.end()); }, + py::keep_alive<0, 1>()) + .def( + "values", + [](const StringMap &map) { return py::make_value_iterator(map.begin(), map.end()); }, py::keep_alive<0, 1>()); // test_generalized_iterators @@ -308,19 +336,62 @@ TEST_SUBMODULE(sequences_and_iterators, m) { .def("nonzero_keys", [](const IntPairs& s) { return py::make_key_iterator(NonZeroIterator>(s.begin()), NonZeroSentinel()); }, py::keep_alive<0, 1>()) + .def("nonzero_values", [](const IntPairs& s) { + return py::make_value_iterator(NonZeroIterator>(s.begin()), NonZeroSentinel()); + }, py::keep_alive<0, 1>()) + + // test single-argument make_iterator .def("simple_iterator", [](IntPairs& self) { return py::make_iterator(self); }, py::keep_alive<0, 1>()) .def("simple_keys", [](IntPairs& self) { return py::make_key_iterator(self); }, py::keep_alive<0, 1>()) + .def("simple_values", [](IntPairs& self) { + return py::make_value_iterator(self); + }, py::keep_alive<0, 1>()) - // test iterator with keep_alive (doesn't work so not used at runtime, but tests compile) - .def("make_iterator_keep_alive", [](IntPairs& self) { - return py::make_iterator(self, py::keep_alive<0, 1>()); + // Test iterator with an Extra (doesn't do anything useful, so not used + // at runtime, but tests need to be able to compile with the correct + // overload. See PR #3293. + .def("_make_iterator_extras", [](IntPairs& self) { + return py::make_iterator(self, py::call_guard()); + }, py::keep_alive<0, 1>()) + .def("_make_key_extras", [](IntPairs& self) { + return py::make_key_iterator(self, py::call_guard()); + }, py::keep_alive<0, 1>()) + .def("_make_value_extras", [](IntPairs& self) { + return py::make_value_iterator(self, py::call_guard()); }, py::keep_alive<0, 1>()) ; + // test_iterater_referencing + py::class_(m, "NonCopyableInt") + .def(py::init()) + .def("set", &NonCopyableInt::set) + .def("__int__", &NonCopyableInt::get) + ; + py::class_>(m, "VectorNonCopyableInt") + .def(py::init<>()) + .def("append", [](std::vector &vec, int value) { + vec.emplace_back(value); + }) + .def("__iter__", [](std::vector &vec) { + return py::make_iterator(vec.begin(), vec.end()); + }) + ; + py::class_>(m, "VectorNonCopyableIntPair") + .def(py::init<>()) + .def("append", [](std::vector &vec, const std::pair &value) { + vec.emplace_back(NonCopyableInt(value.first), NonCopyableInt(value.second)); + }) + .def("keys", [](std::vector &vec) { + return py::make_key_iterator(vec.begin(), vec.end()); + }) + .def("values", [](std::vector &vec) { + return py::make_value_iterator(vec.begin(), vec.end()); + }) + ; #if 0 // Obsolete: special data structure for exposing custom iterator types to python diff --git a/tests/test_sequences_and_iterators.py b/tests/test_sequences_and_iterators.py index 902f4914..38e2ab5b 100644 --- a/tests/test_sequences_and_iterators.py +++ b/tests/test_sequences_and_iterators.py @@ -36,6 +36,10 @@ def test_generalized_iterators(): assert list(m.IntPairs([(1, 2), (2, 0), (0, 3), (4, 5)]).nonzero_keys()) == [1] assert list(m.IntPairs([(0, 3), (1, 2), (3, 4)]).nonzero_keys()) == [] + assert list(m.IntPairs([(1, 2), (3, 4), (0, 5)]).nonzero_values()) == [2, 4] + assert list(m.IntPairs([(1, 2), (2, 0), (0, 3), (4, 5)]).nonzero_values()) == [2] + assert list(m.IntPairs([(0, 3), (1, 2), (3, 4)]).nonzero_values()) == [] + # __next__ must continue to raise StopIteration it = m.IntPairs([(0, 0)]).nonzero() for _ in range(3): @@ -55,6 +59,31 @@ def test_generalized_iterators_simple(): (0, 5), ] assert list(m.IntPairs([(1, 2), (3, 4), (0, 5)]).simple_keys()) == [1, 3, 0] + assert list(m.IntPairs([(1, 2), (3, 4), (0, 5)]).simple_values()) == [2, 4, 5] + + +def test_iterator_referencing(): + """Test that iterators reference rather than copy their referents.""" + vec = m.VectorNonCopyableInt() + vec.append(3) + vec.append(5) + assert [int(x) for x in vec] == [3, 5] + # Increment everything to make sure the referents can be mutated + for x in vec: + x.set(int(x) + 1) + assert [int(x) for x in vec] == [4, 6] + + vec = m.VectorNonCopyableIntPair() + vec.append([3, 4]) + vec.append([5, 7]) + assert [int(x) for x in vec.keys()] == [3, 5] + assert [int(x) for x in vec.values()] == [4, 7] + for x in vec.keys(): + x.set(int(x) + 1) + for x in vec.values(): + x.set(int(x) + 10) + assert [int(x) for x in vec.keys()] == [4, 6] + assert [int(x) for x in vec.values()] == [14, 17] def test_sliceable(): @@ -160,6 +189,7 @@ def test_map_iterator(): assert sm[k] == expected[k] for k, v in sm.items(): assert v == expected[k] + assert list(sm.values()) == [expected[k] for k in sm] it = iter(m.StringMap({})) for _ in range(3): # __next__ must continue to raise StopIteration From 6ad3f874a797ed554b2fea82dd2b798826a83fa8 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 23 Sep 2021 15:42:16 -0400 Subject: [PATCH 083/100] fix(build): avoid a possible warning about shadowed variables and changing behaviors (#3220) --- tools/pybind11Tools.cmake | 42 +++++++++++++++++---------------------- 1 file changed, 18 insertions(+), 24 deletions(-) diff --git a/tools/pybind11Tools.cmake b/tools/pybind11Tools.cmake index 32313539..cc5ca21c 100644 --- a/tools/pybind11Tools.cmake +++ b/tools/pybind11Tools.cmake @@ -45,31 +45,25 @@ list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_LIST_DIR}") find_package(PythonLibsNew ${PYBIND11_PYTHON_VERSION} MODULE REQUIRED ${_pybind11_quiet}) list(REMOVE_AT CMAKE_MODULE_PATH -1) +# Makes a normal variable a cached variable +macro(_PYBIND11_PROMOTE_TO_CACHE NAME) + set(_tmp_ptc "${${NAME}}") + # CMake 3.21 complains if a cached variable is shadowed by a normal one + unset(${NAME}) + set(${NAME} + "${_tmp_ptc}" + CACHE INTERNAL "") +endmacro() + # Cache variables so pybind11_add_module can be used in parent projects -set(PYTHON_INCLUDE_DIRS - ${PYTHON_INCLUDE_DIRS} - CACHE INTERNAL "") -set(PYTHON_LIBRARIES - ${PYTHON_LIBRARIES} - CACHE INTERNAL "") -set(PYTHON_MODULE_PREFIX - ${PYTHON_MODULE_PREFIX} - CACHE INTERNAL "") -set(PYTHON_MODULE_EXTENSION - ${PYTHON_MODULE_EXTENSION} - CACHE INTERNAL "") -set(PYTHON_VERSION_MAJOR - ${PYTHON_VERSION_MAJOR} - CACHE INTERNAL "") -set(PYTHON_VERSION_MINOR - ${PYTHON_VERSION_MINOR} - CACHE INTERNAL "") -set(PYTHON_VERSION - ${PYTHON_VERSION} - CACHE INTERNAL "") -set(PYTHON_IS_DEBUG - "${PYTHON_IS_DEBUG}" - CACHE INTERNAL "") +_pybind11_promote_to_cache(PYTHON_INCLUDE_DIRS) +_pybind11_promote_to_cache(PYTHON_LIBRARIES) +_pybind11_promote_to_cache(PYTHON_MODULE_PREFIX) +_pybind11_promote_to_cache(PYTHON_MODULE_EXTENSION) +_pybind11_promote_to_cache(PYTHON_VERSION_MAJOR) +_pybind11_promote_to_cache(PYTHON_VERSION_MINOR) +_pybind11_promote_to_cache(PYTHON_VERSION) +_pybind11_promote_to_cache(PYTHON_IS_DEBUG) if(PYBIND11_MASTER_PROJECT) if(PYTHON_MODULE_EXTENSION MATCHES "pypy") From 409be8336f4c597bcca9d9eaa09df0abc065afff Mon Sep 17 00:00:00 2001 From: Wenzel Jakob Date: Fri, 24 Sep 2021 13:03:57 +0200 Subject: [PATCH 084/100] CMake: react to python version changes The new FindPython-based variant of the CMake scripts caches information about the chosen Python version that can become stale. For example, suppose I configure a simple pybind11-based project as follows ``` cmake -S . -B build -GNinja -DPython_ROOT= ``` which will generate `my_extension.cpython-38-x86_64-linux-gnu.so`. A subsequent change to the python version like ``` cmake -S . -B build -GNinja -DPython_ROOT= ``` does not update all necessary build system information. In particular, the compiled file is still called `my_extension.cpython-38-x86_64-linux-gnu.so`. This commit fixes the problem by detecting changes in `Python_EXECUTABLE` and re-running Python as needed. Note that the previous way of detecting Python does not seem to be affected, it always specifies the right suffix. --- tools/pybind11NewTools.cmake | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/tools/pybind11NewTools.cmake b/tools/pybind11NewTools.cmake index a20803bc..b93f9145 100644 --- a/tools/pybind11NewTools.cmake +++ b/tools/pybind11NewTools.cmake @@ -82,6 +82,15 @@ if(NOT DEFINED ${_Python}_EXECUTABLE) endif() +if(NOT ${_Python}_EXECUTABLE STREQUAL PYTHON_EXECUTABLE_LAST) + # Detect changes to the Python version/binary in subsequent CMake runs, and refresh config if needed + unset(PYTHON_IS_DEBUG CACHE) + unset(PYTHON_MODULE_EXTENSION CACHE) + set(PYTHON_EXECUTABLE_LAST + "${${_Python}_EXECUTABLE}" + CACHE INTERNAL "Python executable during the last CMake run") +endif() + if(NOT DEFINED PYTHON_IS_DEBUG) # Debug check - see https://stackoverflow.com/questions/646518/python-how-to-detect-debug-Interpreter execute_process( From 62c4909cce37829303329da77e427d9d3a907d3a Mon Sep 17 00:00:00 2001 From: Jeremy Maitin-Shepard Date: Fri, 24 Sep 2021 12:08:22 -0700 Subject: [PATCH 085/100] Add `custom_type_setup` attribute (#3287) * Fix `pybind11::object::operator=` to be safe if `*this` is accessible from Python * Add `custom_type_setup` attribute This allows for custom modifications to the PyHeapTypeObject prior to calling `PyType_Ready`. This may be used, for example, to define `tp_traverse` and `tp_clear` functions. --- docs/advanced/classes.rst | 34 ++++++++++++++++++++++ include/pybind11/attr.h | 29 ++++++++++++++++++ include/pybind11/detail/class.h | 6 ++-- include/pybind11/pytypes.h | 5 +++- tests/CMakeLists.txt | 1 + tests/test_custom_type_setup.cpp | 41 ++++++++++++++++++++++++++ tests/test_custom_type_setup.py | 50 ++++++++++++++++++++++++++++++++ 7 files changed, 163 insertions(+), 3 deletions(-) create mode 100644 tests/test_custom_type_setup.cpp create mode 100644 tests/test_custom_type_setup.py diff --git a/docs/advanced/classes.rst b/docs/advanced/classes.rst index 7f8fcdf4..5f01a2f1 100644 --- a/docs/advanced/classes.rst +++ b/docs/advanced/classes.rst @@ -1261,3 +1261,37 @@ object, just like ``type(ob)`` in Python. Other types, like ``py::type::of()``, do not work, see :ref:`type-conversions`. .. versionadded:: 2.6 + +Custom type setup +================= + +For advanced use cases, such as enabling garbage collection support, you may +wish to directly manipulate the `PyHeapTypeObject` corresponding to a +``py::class_`` definition. + +You can do that using ``py::custom_type_setup``: + +.. code-block:: cpp + + struct OwnsPythonObjects { + py::object value = py::none(); + }; + py::class_ cls( + m, "OwnsPythonObjects", py::custom_type_setup([](PyHeapTypeObject *heap_type) { + auto *type = &heap_type->ht_type; + type->tp_flags |= Py_TPFLAGS_HAVE_GC; + type->tp_traverse = [](PyObject *self_base, visitproc visit, void *arg) { + auto &self = py::cast(py::handle(self_base)); + Py_VISIT(self.value.ptr()); + return 0; + }; + type->tp_clear = [](PyObject *self_base) { + auto &self = py::cast(py::handle(self_base)); + self.value = py::none(); + return 0; + }; + })); + cls.def(py::init<>()); + cls.def_readwrite("value", &OwnsPythonObjects::value); + +.. versionadded:: 2.8 diff --git a/include/pybind11/attr.h b/include/pybind11/attr.h index 13f68bbe..0dedbc08 100644 --- a/include/pybind11/attr.h +++ b/include/pybind11/attr.h @@ -12,6 +12,8 @@ #include "cast.h" +#include + PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) /// \addtogroup annotations @@ -79,6 +81,23 @@ struct metaclass { explicit metaclass(handle value) : value(value) { } }; +/// Specifies a custom callback with signature `void (PyHeapTypeObject*)` that +/// may be used to customize the Python type. +/// +/// The callback is invoked immediately before `PyType_Ready`. +/// +/// Note: This is an advanced interface, and uses of it may require changes to +/// work with later versions of pybind11. You may wish to consult the +/// implementation of `make_new_python_type` in `detail/classes.h` to understand +/// the context in which the callback will be run. +struct custom_type_setup { + using callback = std::function; + + explicit custom_type_setup(callback value) : value(std::move(value)) {} + + callback value; +}; + /// Annotation that marks a class as local to the module: struct module_local { const bool value; constexpr explicit module_local(bool v = true) : value(v) {} @@ -272,6 +291,9 @@ struct type_record { /// Custom metaclass (optional) handle metaclass; + /// Custom type setup. + custom_type_setup::callback custom_type_setup_callback; + /// Multiple inheritance marker bool multiple_inheritance : 1; @@ -476,6 +498,13 @@ struct process_attribute : process_attribute_default static void init(const dynamic_attr &, type_record *r) { r->dynamic_attr = true; } }; +template <> +struct process_attribute { + static void init(const custom_type_setup &value, type_record *r) { + r->custom_type_setup_callback = value.value; + } +}; + template <> struct process_attribute : process_attribute_default { static void init(const is_final &, type_record *r) { r->is_final = true; } diff --git a/include/pybind11/detail/class.h b/include/pybind11/detail/class.h index f9822c7b..b9376b4c 100644 --- a/include/pybind11/detail/class.h +++ b/include/pybind11/detail/class.h @@ -683,11 +683,13 @@ inline PyObject* make_new_python_type(const type_record &rec) { if (rec.buffer_protocol) enable_buffer_protocol(heap_type); + if (rec.custom_type_setup_callback) + rec.custom_type_setup_callback(heap_type); + if (PyType_Ready(type) < 0) pybind11_fail(std::string(rec.name) + ": PyType_Ready failed (" + error_string() + ")!"); - assert(rec.dynamic_attr ? PyType_HasFeature(type, Py_TPFLAGS_HAVE_GC) - : !PyType_HasFeature(type, Py_TPFLAGS_HAVE_GC)); + assert(!rec.dynamic_attr || PyType_HasFeature(type, Py_TPFLAGS_HAVE_GC)); /* Register type with the parent scope */ if (rec.scope) diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index 383663b5..f54d5fad 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -259,8 +259,11 @@ public: object& operator=(const object &other) { other.inc_ref(); - dec_ref(); + // Use temporary variable to ensure `*this` remains valid while + // `Py_XDECREF` executes, in case `*this` is accessible from Python. + handle temp(m_ptr); m_ptr = other.m_ptr; + temp.dec_ref(); return *this; } diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index f014771d..70303879 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -104,6 +104,7 @@ set(PYBIND11_TEST_FILES test_constants_and_functions.cpp test_copy_move.cpp test_custom_type_casters.cpp + test_custom_type_setup.cpp test_docstring_options.cpp test_eigen.cpp test_enum.cpp diff --git a/tests/test_custom_type_setup.cpp b/tests/test_custom_type_setup.cpp new file mode 100644 index 00000000..42fae05d --- /dev/null +++ b/tests/test_custom_type_setup.cpp @@ -0,0 +1,41 @@ +/* + tests/test_custom_type_setup.cpp -- Tests `pybind11::custom_type_setup` + + Copyright (c) Google LLC + + All rights reserved. Use of this source code is governed by a + BSD-style license that can be found in the LICENSE file. +*/ + +#include + +#include "pybind11_tests.h" + +namespace py = pybind11; + +namespace { + +struct OwnsPythonObjects { + py::object value = py::none(); +}; +} // namespace + +TEST_SUBMODULE(custom_type_setup, m) { + py::class_ cls( + m, "OwnsPythonObjects", py::custom_type_setup([](PyHeapTypeObject *heap_type) { + auto *type = &heap_type->ht_type; + type->tp_flags |= Py_TPFLAGS_HAVE_GC; + type->tp_traverse = [](PyObject *self_base, visitproc visit, void *arg) { + auto &self = py::cast(py::handle(self_base)); + Py_VISIT(self.value.ptr()); + return 0; + }; + type->tp_clear = [](PyObject *self_base) { + auto &self = py::cast(py::handle(self_base)); + self.value = py::none(); + return 0; + }; + })); + cls.def(py::init<>()); + cls.def_readwrite("value", &OwnsPythonObjects::value); +} diff --git a/tests/test_custom_type_setup.py b/tests/test_custom_type_setup.py new file mode 100644 index 00000000..ef96f081 --- /dev/null +++ b/tests/test_custom_type_setup.py @@ -0,0 +1,50 @@ +# -*- coding: utf-8 -*- + +import gc +import weakref + +import pytest + +import env # noqa: F401 +from pybind11_tests import custom_type_setup as m + + +@pytest.fixture +def gc_tester(): + """Tests that an object is garbage collected. + + Assumes that any unreferenced objects are fully collected after calling + `gc.collect()`. That is true on CPython, but does not appear to reliably + hold on PyPy. + """ + + weak_refs = [] + + def add_ref(obj): + # PyPy does not support `gc.is_tracked`. + if hasattr(gc, "is_tracked"): + assert gc.is_tracked(obj) + weak_refs.append(weakref.ref(obj)) + + yield add_ref + + gc.collect() + for ref in weak_refs: + assert ref() is None + + +# PyPy does not seem to reliably garbage collect. +@pytest.mark.skipif("env.PYPY") +def test_self_cycle(gc_tester): + obj = m.OwnsPythonObjects() + obj.value = obj + gc_tester(obj) + + +# PyPy does not seem to reliably garbage collect. +@pytest.mark.skipif("env.PYPY") +def test_indirect_cycle(gc_tester): + obj = m.OwnsPythonObjects() + obj_list = [obj] + obj.value = obj_list + gc_tester(obj) From d58699c9ab9d20010b15aa38382ad517a8280179 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Fri, 24 Sep 2021 17:47:03 -0400 Subject: [PATCH 086/100] fix(cmake): reduce chance for variable collision (#3302) --- tools/pybind11NewTools.cmake | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/pybind11NewTools.cmake b/tools/pybind11NewTools.cmake index b93f9145..0a7b8470 100644 --- a/tools/pybind11NewTools.cmake +++ b/tools/pybind11NewTools.cmake @@ -82,11 +82,11 @@ if(NOT DEFINED ${_Python}_EXECUTABLE) endif() -if(NOT ${_Python}_EXECUTABLE STREQUAL PYTHON_EXECUTABLE_LAST) +if(NOT ${_Python}_EXECUTABLE STREQUAL PYBIND11_PYTHON_EXECUTABLE_LAST) # Detect changes to the Python version/binary in subsequent CMake runs, and refresh config if needed unset(PYTHON_IS_DEBUG CACHE) unset(PYTHON_MODULE_EXTENSION CACHE) - set(PYTHON_EXECUTABLE_LAST + set(PYBIND11_PYTHON_EXECUTABLE_LAST "${${_Python}_EXECUTABLE}" CACHE INTERNAL "Python executable during the last CMake run") endif() From 6bce3bd72e8d0838cdb1586abd6d3cfb9c598a8f Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Fri, 24 Sep 2021 23:48:38 -0400 Subject: [PATCH 087/100] docs: update CHANGELOG (#3304) --- docs/changelog.rst | 40 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 6bdf6a6e..04e2714f 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -38,21 +38,42 @@ New features: * Implemented ``reshape`` on arrays. `#984 `_ +* Enable defining custom ``__new__`` methods on classes by fixing bug + preventing overriding methods if they have non-pybind11 siblings. + `#3265 `_ + +* Add ``make_value_iterator()``, and fix ``make_key_iterator()`` to return + references instead of copies. + `#3293 `_ + +* ``pybind11::custom_type_setup`` was added, for customizing the + ``PyHeapTypeObject`` corresponding to a class, which may be useful for + enabling garbage collection support, among other things. + `#3287 `_ + + Changes: * Set ``__file__`` constant when running ``eval_file`` in an embedded interpreter. `#3233 `_ +* Python objects and (C++17) ``std::optional`` now accepted in ``py::slice`` + constructor. + `#1101 `_ + * The pybind11 proxy types ``str``, ``bytes``, ``bytearray``, ``tuple``, ``list`` now consistently support passing ``ssize_t`` values for sizes and indexes. Previously, only ``size_t`` was accepted in several interfaces. `#3219 `_ +* Avoid evaluating ``PYBIND11_TLS_REPLACE_VALUE`` arguments more than once. + `#3290 `_ Fixes: -* Bug fix: enum value's ``__int__`` returning non-int when underlying type is bool or of char type. +* Bug fix: enum value's ``__int__`` returning non-int when underlying type is + bool or of char type. `#1334 `_ * Fixes bug in setting error state in Capsule's pointer methods. @@ -61,7 +82,8 @@ Fixes: * A long-standing memory leak in ``py::cpp_function::initialize`` was fixed. `#3229 `_ -* Fixes thread safety for some ``pybind11::type_caster`` which require lifetime extension, such as for ``std::string_view``. +* Fixes thread safety for some ``pybind11::type_caster`` which require lifetime + extension, such as for ``std::string_view``. `#3237 `_ * Restore compatibility with gcc 4.8.4 as distributed by ubuntu-trusty, linuxmint-17. @@ -73,12 +95,26 @@ Build system improvements: * Fix regression in CMake Python package config: improper use of absolute path. `#3144 `_ +* Cached Python version information could become stale when CMake was re-run + with a different Python version. The build system now detects this and + updates this information. + `#3299 `_ + * Specified UTF8-encoding in setup.py calls of open(). `#3137 `_ +* Fix a harmless warning from CMake 3.21 with the classic Python discovery. + `#3220 `_ + Backend and tidying up: +* Reduced thread-local storage required for keeping alive temporary data for + type conversion to one key per ABI version, rather than one key per extension + module. This makes the total thread-local storage required by pybind11 2 + keys per ABI version. + `#3275 `_ + * Optimize NumPy array construction with additional moves. `#3183 `_ From 6be64304b7e5cd4792fcd25d031127b99ef474bc Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Tue, 28 Sep 2021 10:09:08 -0400 Subject: [PATCH 088/100] [pre-commit.ci] pre-commit autoupdate (#3312) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit updates: - [github.com/asottile/pyupgrade: v2.26.0 → v2.28.0](https://github.com/asottile/pyupgrade/compare/v2.26.0...v2.28.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .pre-commit-config.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 4245641b..05bfce85 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -33,7 +33,7 @@ repos: exclude: ^noxfile.py$ - repo: https://github.com/asottile/pyupgrade - rev: v2.26.0 + rev: v2.28.0 hooks: - id: pyupgrade From c9bbf8d2eed098f3c10ae461bab2c0f9541d38f7 Mon Sep 17 00:00:00 2001 From: Philipp Bucher Date: Tue, 28 Sep 2021 16:09:38 +0200 Subject: [PATCH 089/100] docs: fix minor typo (#3311) --- docs/compiling.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/compiling.rst b/docs/compiling.rst index 07385104..75608bd5 100644 --- a/docs/compiling.rst +++ b/docs/compiling.rst @@ -340,7 +340,7 @@ standard explicitly with set(CMAKE_CXX_STANDARD 14 CACHE STRING "C++ version selection") # or 11, 14, 17, 20 set(CMAKE_CXX_STANDARD_REQUIRED ON) # optional, ensure standard is supported - set(CMAKE_CXX_EXTENSIONS OFF) # optional, keep compiler extensionsn off + set(CMAKE_CXX_EXTENSIONS OFF) # optional, keep compiler extensions off The variables can also be set when calling CMake from the command line using the ``-D=`` flag. You can also manually set ``CXX_STANDARD`` From a1830d5e6e27cb4c40955399f3826736a59f5cd5 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Wed, 29 Sep 2021 06:18:58 -0400 Subject: [PATCH 090/100] docs: mention title conventions in PR template (#3313) --- .github/pull_request_template.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 97a6ff7d..54b7f510 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,3 +1,7 @@ + ## Description From b4e1ab8caa483350fbcfc7397b3b5cb33c49050f Mon Sep 17 00:00:00 2001 From: xaedes Date: Thu, 30 Sep 2021 20:45:06 +0200 Subject: [PATCH 091/100] Docs: Demonstrate non-enum internal types in example (#3314) * Docs: Demonstrate non-enum internal types in example Previously example only demonstrated internal enumeration type. To show that it works for other internal types the same way the example was updated with an additional struct Pet::Attributes type. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- docs/classes.rst | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/docs/classes.rst b/docs/classes.rst index a4463e41..13fa8b53 100644 --- a/docs/classes.rst +++ b/docs/classes.rst @@ -446,8 +446,7 @@ you can use ``py::detail::overload_cast_impl`` with an additional set of parenth Enumerations and internal types =============================== -Let's now suppose that the example class contains an internal enumeration type, -e.g.: +Let's now suppose that the example class contains internal types like enumerations, e.g.: .. code-block:: cpp @@ -457,10 +456,15 @@ e.g.: Cat }; + struct Attributes { + float age = 0; + }; + Pet(const std::string &name, Kind type) : name(name), type(type) { } std::string name; Kind type; + Attributes attr; }; The binding code for this example looks as follows: @@ -471,15 +475,21 @@ The binding code for this example looks as follows: pet.def(py::init()) .def_readwrite("name", &Pet::name) - .def_readwrite("type", &Pet::type); + .def_readwrite("type", &Pet::type) + .def_readwrite("attr", &Pet::attr); py::enum_(pet, "Kind") .value("Dog", Pet::Kind::Dog) .value("Cat", Pet::Kind::Cat) .export_values(); -To ensure that the ``Kind`` type is created within the scope of ``Pet``, the -``pet`` :class:`class_` instance must be supplied to the :class:`enum_`. + py::class_ attributes(pet, "Attributes") + .def(py::init<>()) + .def_readwrite("age", &Pet::Attributes::age); + + +To ensure that the nested types ``Kind`` and ``Attributes`` are created within the scope of ``Pet``, the +``pet`` :class:`class_` instance must be supplied to the :class:`enum_` and :class:`class_` constructor. The :func:`enum_::export_values` function exports the enum entries into the parent scope, which should be skipped for newer C++11-style strongly typed enums. From b3573ac9615b705d6e5b9bf598a3378edf079f06 Mon Sep 17 00:00:00 2001 From: Bruce Merry <1963944+bmerry@users.noreply.github.com> Date: Fri, 1 Oct 2021 15:24:36 +0200 Subject: [PATCH 092/100] feat: add `.keys` and `.values` to bind_map (#3310) * Add `.keys` and `.values` to bind_map Both of these implement views (rather than just iterators), and `.items` is also upgraded to a view. In practical terms, this allows a view to be iterated multiple times and have its size taken, neither of which works with an iterator. The views implement `__len__`, `__iter__`, and the keys view implements `__contains__`. Testing membership also works in item and value views because Python falls back to iteration. This won't be optimal for item values since it's linear rather than O(log n) or O(1), but I didn't fancy trying to get all the corner cases to match Python behaviour (tuple of wrong types, wrong length tuple, not a tuple etc). Missing relative to Python dictionary views is `__reversed__` (only added to Python in 3.8). Implementing that could break code that binds custom map classes which don't provide `rbegin`/`rend` (at least without doing clever things with SFINAE), so I've not tried. The size increase on my system is 131072 bytes, which is rather large (5%) but also suspiciously round (2^17) and makes me suspect some quantisation effect. * bind_map: support any object in __contains__ Add extra overload of `__contains__` (for both the map itself and KeysView) which takes an arbitrary object and returns false. * Take py::object by const reference in __contains__ To keep clang-tidy happy. * Removing stray `py::` (detected via interactive testing in Google environment). Co-authored-by: Ralf W. Grosse-Kunstleve --- include/pybind11/stl_bind.h | 78 +++++++++++++++++++++++++++++++++++-- tests/test_stl_binders.py | 30 +++++++++++++- 2 files changed, 104 insertions(+), 4 deletions(-) diff --git a/include/pybind11/stl_bind.h b/include/pybind11/stl_bind.h index 82317b37..050be83c 100644 --- a/include/pybind11/stl_bind.h +++ b/include/pybind11/stl_bind.h @@ -595,6 +595,23 @@ template auto map_if_insertion_operator(Class_ & ); } +template +struct keys_view +{ + Map ↦ +}; + +template +struct values_view +{ + Map ↦ +}; + +template +struct items_view +{ + Map ↦ +}; PYBIND11_NAMESPACE_END(detail) @@ -602,6 +619,9 @@ template , typename... class_ bind_map(handle scope, const std::string &name, Args&&... args) { using KeyType = typename Map::key_type; using MappedType = typename Map::mapped_type; + using KeysView = detail::keys_view; + using ValuesView = detail::values_view; + using ItemsView = detail::items_view; using Class_ = class_; // If either type is a non-module-local bound type then make the map binding non-local as well; @@ -615,6 +635,12 @@ class_ bind_map(handle scope, const std::string &name, Args&&. } Class_ cl(scope, name.c_str(), pybind11::module_local(local), std::forward(args)...); + class_ keys_view( + scope, ("KeysView[" + name + "]").c_str(), pybind11::module_local(local)); + class_ values_view( + scope, ("ValuesView[" + name + "]").c_str(), pybind11::module_local(local)); + class_ items_view( + scope, ("ItemsView[" + name + "]").c_str(), pybind11::module_local(local)); cl.def(init<>()); @@ -628,12 +654,22 @@ class_ bind_map(handle scope, const std::string &name, Args&&. cl.def("__iter__", [](Map &m) { return make_key_iterator(m.begin(), m.end()); }, - keep_alive<0, 1>() /* Essential: keep list alive while iterator exists */ + keep_alive<0, 1>() /* Essential: keep map alive while iterator exists */ + ); + + cl.def("keys", + [](Map &m) { return KeysView{m}; }, + keep_alive<0, 1>() /* Essential: keep map alive while view exists */ + ); + + cl.def("values", + [](Map &m) { return ValuesView{m}; }, + keep_alive<0, 1>() /* Essential: keep map alive while view exists */ ); cl.def("items", - [](Map &m) { return make_iterator(m.begin(), m.end()); }, - keep_alive<0, 1>() /* Essential: keep list alive while iterator exists */ + [](Map &m) { return ItemsView{m}; }, + keep_alive<0, 1>() /* Essential: keep map alive while view exists */ ); cl.def("__getitem__", @@ -654,6 +690,8 @@ class_ bind_map(handle scope, const std::string &name, Args&&. return true; } ); + // Fallback for when the object is not of the key type + cl.def("__contains__", [](Map &, const object &) -> bool { return false; }); // Assignment provided only if the type is copyable detail::map_assignment(cl); @@ -669,6 +707,40 @@ class_ bind_map(handle scope, const std::string &name, Args&&. cl.def("__len__", &Map::size); + keys_view.def("__len__", [](KeysView &view) { return view.map.size(); }); + keys_view.def("__iter__", + [](KeysView &view) { + return make_key_iterator(view.map.begin(), view.map.end()); + }, + keep_alive<0, 1>() /* Essential: keep view alive while iterator exists */ + ); + keys_view.def("__contains__", + [](KeysView &view, const KeyType &k) -> bool { + auto it = view.map.find(k); + if (it == view.map.end()) + return false; + return true; + } + ); + // Fallback for when the object is not of the key type + keys_view.def("__contains__", [](KeysView &, const object &) -> bool { return false; }); + + values_view.def("__len__", [](ValuesView &view) { return view.map.size(); }); + values_view.def("__iter__", + [](ValuesView &view) { + return make_value_iterator(view.map.begin(), view.map.end()); + }, + keep_alive<0, 1>() /* Essential: keep view alive while iterator exists */ + ); + + items_view.def("__len__", [](ItemsView &view) { return view.map.size(); }); + items_view.def("__iter__", + [](ItemsView &view) { + return make_iterator(view.map.begin(), view.map.end()); + }, + keep_alive<0, 1>() /* Essential: keep view alive while iterator exists */ + ); + return cl; } diff --git a/tests/test_stl_binders.py b/tests/test_stl_binders.py index 475a9ec4..a68dcd31 100644 --- a/tests/test_stl_binders.py +++ b/tests/test_stl_binders.py @@ -160,15 +160,43 @@ def test_map_string_double(): mm["b"] = 2.5 assert list(mm) == ["a", "b"] - assert list(mm.items()) == [("a", 1), ("b", 2.5)] assert str(mm) == "MapStringDouble{a: 1, b: 2.5}" + assert "b" in mm + assert "c" not in mm + assert 123 not in mm + + # Check that keys, values, items are views, not merely iterable + keys = mm.keys() + values = mm.values() + items = mm.items() + assert list(keys) == ["a", "b"] + assert len(keys) == 2 + assert "a" in keys + assert "c" not in keys + assert 123 not in keys + assert list(items) == [("a", 1), ("b", 2.5)] + assert len(items) == 2 + assert ("b", 2.5) in items + assert "hello" not in items + assert ("b", 2.5, None) not in items + assert list(values) == [1, 2.5] + assert len(values) == 2 + assert 1 in values + assert 2 not in values + # Check that views update when the map is updated + mm["c"] = -1 + assert list(keys) == ["a", "b", "c"] + assert list(values) == [1, 2.5, -1] + assert list(items) == [("a", 1), ("b", 2.5), ("c", -1)] um = m.UnorderedMapStringDouble() um["ua"] = 1.1 um["ub"] = 2.6 assert sorted(list(um)) == ["ua", "ub"] + assert list(um.keys()) == list(um) assert sorted(list(um.items())) == [("ua", 1.1), ("ub", 2.6)] + assert list(zip(um.keys(), um.values())) == list(um.items()) assert "UnorderedMapStringDouble" in str(um) From ad9665560597a473774ea19dc87a40aef02090b3 Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Sun, 3 Oct 2021 20:15:37 -0400 Subject: [PATCH 093/100] fix: replace free() with std::free() (#3321) * Disambiguate free() to use std::free() * Add cstdlib include --- include/pybind11/pybind11.h | 5 +++-- tests/test_embed/test_interpreter.cpp | 3 ++- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 16535f19..370e52cf 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -16,6 +16,7 @@ #include "detail/class.h" #include "detail/init.h" +#include #include #include #include @@ -1536,7 +1537,7 @@ public: char *doc_prev = rec_fget->doc; /* 'extra' field may include a property-specific documentation string */ detail::process_attributes::init(extra..., rec_fget); if (rec_fget->doc && rec_fget->doc != doc_prev) { - free(doc_prev); + std::free(doc_prev); rec_fget->doc = PYBIND11_COMPAT_STRDUP(rec_fget->doc); } } @@ -1544,7 +1545,7 @@ public: char *doc_prev = rec_fset->doc; detail::process_attributes::init(extra..., rec_fset); if (rec_fset->doc && rec_fset->doc != doc_prev) { - free(doc_prev); + std::free(doc_prev); rec_fset->doc = PYBIND11_COMPAT_STRDUP(rec_fset->doc); } if (! rec_active) rec_active = rec_fset; diff --git a/tests/test_embed/test_interpreter.cpp b/tests/test_embed/test_interpreter.cpp index fae14f75..20bcade0 100644 --- a/tests/test_embed/test_interpreter.cpp +++ b/tests/test_embed/test_interpreter.cpp @@ -8,6 +8,7 @@ #include +#include #include #include #include @@ -315,7 +316,7 @@ TEST_CASE("sys.argv gets initialized properly") { { char *argv[] = {strdup("a.out")}; py::scoped_interpreter argv_scope(true, 1, argv); - free(argv[0]); + std::free(argv[0]); auto module = py::module::import("test_interpreter"); auto py_widget = module.attr("DerivedWidget")("The question"); const auto &cpp_widget = py_widget.cast(); From d7a7edc12bc4c7aec4afd16aee1ed032bb974164 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Mon, 4 Oct 2021 12:31:53 -0400 Subject: [PATCH 094/100] tests: support Eigen configuration --- tests/CMakeLists.txt | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 70303879..6ebcba9b 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -171,6 +171,14 @@ set(PYBIND11_CROSS_MODULE_TESTS test_exceptions.py test_local_bindings.py test_s set(PYBIND11_CROSS_MODULE_GIL_TESTS test_gil_scoped.py) +set(PYBIND11_EIGEN_REPO + "https://gitlab.com/libeigen/eigen.git" + CACHE STRING "Eigen repository to use for tests") +# This hash is for 3.3.8, using a hash for security reasons +set(PYBIND11_EIGEN_VERSION + "dc252fbf00079ccab57948a164b1421703fe4361" + CACHE STRING "Eigen version to use for tests") + # Check if Eigen is available; if not, remove from PYBIND11_TEST_FILES (but # keep it in PYBIND11_PYTEST_FILES, so that we get the "eigen is not installed" # skip message). @@ -184,13 +192,11 @@ if(PYBIND11_TEST_FILES_EIGEN_I GREATER -1) message(FATAL_ERROR "CMake 3.11+ required when using DOWNLOAD_EIGEN") endif() - set(EIGEN3_VERSION_STRING "3.3.8") - include(FetchContent) FetchContent_Declare( eigen - GIT_REPOSITORY https://gitlab.com/libeigen/eigen.git - GIT_TAG ${EIGEN3_VERSION_STRING}) + GIT_REPOSITORY "${PYBIND11_EIGEN_REPO}" + GIT_TAG "${PYBIND11_EIGEN_VERSION}") FetchContent_GetProperties(eigen) if(NOT eigen_POPULATED) From 9f146a56227eab9b3bf5e847fc5610853372f587 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Sun, 3 Oct 2021 14:57:41 -0700 Subject: [PATCH 095/100] All `-DDOWNLOAD_EIGEN=OFF` (to work around gitlab eigen outage). --- .github/workflows/ci.yml | 8 ++++++++ .github/workflows/format.yml | 1 + 2 files changed, 9 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0b4968cc..279a1e7a 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -102,6 +102,7 @@ jobs: -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON + -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup -DCMAKE_CXX_STANDARD=11 ${{ matrix.args }} @@ -129,6 +130,7 @@ jobs: -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON + -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup -DCMAKE_CXX_STANDARD=17 ${{ matrix.args }} ${{ matrix.args2 }} @@ -151,6 +153,7 @@ jobs: -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON + -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup -DCMAKE_CXX_STANDARD=17 -DPYBIND11_INTERNALS_VERSION=10000000 "-DPYBIND11_TEST_OVERRIDE=test_call_policies.cpp;test_gil_scoped.cpp;test_thread.cpp" @@ -249,6 +252,7 @@ jobs: -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON + -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup -DCMAKE_CXX_STANDARD=17 - name: Build @@ -628,6 +632,7 @@ jobs: -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON + -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup -DCMAKE_CXX_STANDARD=11 -DPYTHON_EXECUTABLE=$(python3 -c "import sys; print(sys.executable)") @@ -774,6 +779,7 @@ jobs: -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON + -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup ${{ matrix.args }} - name: Build C++11 run: cmake --build build -j 2 @@ -820,6 +826,7 @@ jobs: -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON + -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup - name: Build C++14 run: cmake --build build -j 2 @@ -869,6 +876,7 @@ jobs: -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON + -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup -DCMAKE_CXX_STANDARD=${{ matrix.std }} ${{ matrix.args }} diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml index ab7b4050..7869912b 100644 --- a/.github/workflows/format.yml +++ b/.github/workflows/format.yml @@ -41,6 +41,7 @@ jobs: cmake -S . -B build -DCMAKE_CXX_CLANG_TIDY="$(which clang-tidy)" -DDOWNLOAD_EIGEN=ON + -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup -DDOWNLOAD_CATCH=ON -DCMAKE_CXX_STANDARD=17 From 591db0b930f6739fe276bce9a7d5b6647bd00851 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Fri, 1 Oct 2021 09:30:55 -0400 Subject: [PATCH 096/100] docs: update CHANGELOG for 2.8 --- docs/changelog.rst | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 04e2714f..4a58f2d5 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -6,8 +6,8 @@ Changelog Starting with version 1.8.0, pybind11 releases use a `semantic versioning `_ policy. -v2.8.0 (WIP) ------------- +v2.8.0 (Oct 4, 2021) +-------------------- New features: @@ -46,13 +46,17 @@ New features: references instead of copies. `#3293 `_ +* Improve the classes generated by ``bind_map``: `#3310 `_ + * Change ``.items`` from an iterator to a dictionary view. + * Add ``.keys`` and ``.values`` (both dictionary views). + * Allow ``__contains__`` to take any object. + * ``pybind11::custom_type_setup`` was added, for customizing the ``PyHeapTypeObject`` corresponding to a class, which may be useful for enabling garbage collection support, among other things. `#3287 `_ - Changes: * Set ``__file__`` constant when running ``eval_file`` in an embedded interpreter. @@ -106,6 +110,9 @@ Build system improvements: * Fix a harmless warning from CMake 3.21 with the classic Python discovery. `#3220 `_ +* Eigen repo and version can now be specified as cmake options. + `#3324 `_ + Backend and tidying up: From 20aae3e61a91dbae5d71c3b11677369d94a82884 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Mon, 4 Oct 2021 14:47:45 -0400 Subject: [PATCH 097/100] ci: disable Eigen due to Cert issue on CentOS --- .github/workflows/ci.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 279a1e7a..416d2a4e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -624,6 +624,7 @@ jobs: if: matrix.centos == 8 run: echo Release > VAR_BUILD_TYPE + # Temporally disabling EIGEN due to SSL issue in CentOS 7 - name: Configure shell: bash run: > @@ -631,8 +632,7 @@ jobs: -DCMAKE_BUILD_TYPE=$(cat VAR_BUILD_TYPE) -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON - -DDOWNLOAD_EIGEN=ON - -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup + -DDOWNLOAD_EIGEN=OFF -DCMAKE_CXX_STANDARD=11 -DPYTHON_EXECUTABLE=$(python3 -c "import sys; print(sys.executable)") From c9a319c607a5fcddf14417483a451bd85083da72 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Mon, 4 Oct 2021 15:33:58 -0400 Subject: [PATCH 098/100] chore: version 2.8.0 final --- include/pybind11/detail/common.h | 4 ++-- pybind11/_version.py | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 3bd84da5..31716e5b 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -11,11 +11,11 @@ #define PYBIND11_VERSION_MAJOR 2 #define PYBIND11_VERSION_MINOR 8 -#define PYBIND11_VERSION_PATCH 0.dev2 +#define PYBIND11_VERSION_PATCH 0 // Similar to Python's convention: https://docs.python.org/3/c-api/apiabiversion.html // Additional convention: 0xD = dev -#define PYBIND11_VERSION_HEX 0x020800D2 +#define PYBIND11_VERSION_HEX 0x02080000 #define PYBIND11_NAMESPACE_BEGIN(name) namespace name { #define PYBIND11_NAMESPACE_END(name) } diff --git a/pybind11/_version.py b/pybind11/_version.py index d212f1df..704191b9 100644 --- a/pybind11/_version.py +++ b/pybind11/_version.py @@ -8,5 +8,5 @@ def _to_int(s): return s -__version__ = "2.8.0.dev2" +__version__ = "2.8.0" version_info = tuple(_to_int(s) for s in __version__.split(".")) From 3747dc2c4f807fba952380f7bd9f367ab9c53a0a Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Mon, 4 Oct 2021 16:38:15 -0400 Subject: [PATCH 099/100] Revert "All `-DDOWNLOAD_EIGEN=OFF` (to work around gitlab eigen outage)." (#3326) This reverts commit 9f146a56227eab9b3bf5e847fc5610853372f587. --- .github/workflows/ci.yml | 9 +-------- .github/workflows/format.yml | 1 - 2 files changed, 1 insertion(+), 9 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 416d2a4e..6843a924 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -102,7 +102,6 @@ jobs: -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON - -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup -DCMAKE_CXX_STANDARD=11 ${{ matrix.args }} @@ -130,7 +129,6 @@ jobs: -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON - -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup -DCMAKE_CXX_STANDARD=17 ${{ matrix.args }} ${{ matrix.args2 }} @@ -153,7 +151,6 @@ jobs: -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON - -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup -DCMAKE_CXX_STANDARD=17 -DPYBIND11_INTERNALS_VERSION=10000000 "-DPYBIND11_TEST_OVERRIDE=test_call_policies.cpp;test_gil_scoped.cpp;test_thread.cpp" @@ -252,7 +249,6 @@ jobs: -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON - -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup -DCMAKE_CXX_STANDARD=17 - name: Build @@ -632,7 +628,7 @@ jobs: -DCMAKE_BUILD_TYPE=$(cat VAR_BUILD_TYPE) -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON - -DDOWNLOAD_EIGEN=OFF + -DDOWNLOAD_EIGEN=ON -DCMAKE_CXX_STANDARD=11 -DPYTHON_EXECUTABLE=$(python3 -c "import sys; print(sys.executable)") @@ -779,7 +775,6 @@ jobs: -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON - -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup ${{ matrix.args }} - name: Build C++11 run: cmake --build build -j 2 @@ -826,7 +821,6 @@ jobs: -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON - -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup - name: Build C++14 run: cmake --build build -j 2 @@ -876,7 +870,6 @@ jobs: -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON - -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup -DCMAKE_CXX_STANDARD=${{ matrix.std }} ${{ matrix.args }} diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml index 7869912b..ab7b4050 100644 --- a/.github/workflows/format.yml +++ b/.github/workflows/format.yml @@ -41,7 +41,6 @@ jobs: cmake -S . -B build -DCMAKE_CXX_CLANG_TIDY="$(which clang-tidy)" -DDOWNLOAD_EIGEN=ON - -DPYBIND11_EIGEN_REPO=https://gitlab.com/libeigen/eigen-backup -DDOWNLOAD_CATCH=ON -DCMAKE_CXX_STANDARD=17 From 97976c16fb7652f7faf02d76756666ef87adbe7d Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Mon, 4 Oct 2021 16:38:31 -0400 Subject: [PATCH 100/100] [pre-commit.ci] pre-commit autoupdate (#3325) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit updates: - [github.com/asottile/pyupgrade: v2.28.0 → v2.29.0](https://github.com/asottile/pyupgrade/compare/v2.28.0...v2.29.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .pre-commit-config.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 05bfce85..2df31460 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -33,7 +33,7 @@ repos: exclude: ^noxfile.py$ - repo: https://github.com/asottile/pyupgrade - rev: v2.28.0 + rev: v2.29.0 hooks: - id: pyupgrade