A breaking upgrade in protobuf from version 4.21.0 broke multiple Google cloud client libraries and leads to typeerror: descriptors cannot not be created directly.
The solutions to this problem are –
- Downgrade the protobuf package to 3.20.x or lower.
- Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
This new version offers significantly better parsing performance than previous releases, especially for large payloads. These changes are mentioned in Protocol Buffer Guide.
Affected Packages
Here is the list of affected packages from this change which leads to typeerror: descriptors cannot not be created directly –
- Onnx – Open Neural Network Exchange
- TensorFlow 1.15.5
- google-cloud-datastore
- Ray
- Keras
- googleapis-common-protos
- google-cloud-datastore
Most of the google-cloud libraries are prone to this error.
Solution 1: Downgrade protobuf package
It’s not always evident if there is protobuf installed in your project. This is because different libraries have dependencies and they install them.
If you are using google-cloud libraries then probably protobuf is installed by them.
One of the solution is to downgrade protobuf package to version 3.20.x or lower. You can use pip for that –
pip install protobuf==3.20.1
Using requirements.txt file
You can also downgrade the package using requirements.txt file. Append the protobuf attribute at the end of this file. When the parser read requirements.txt file, it will override the previously mentioned protobuf versions and choose the last one.
... ... protobuf==3.20.1
Solution 2: Setting PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION Flag
Another solution to resolve “typeerror: descriptors cannot not be created directly” is to set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION flag to be equal to python. It’s an environment variable which can be set using this command –
set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python
This flag is also used to share messages between Python and C++. In that case we set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp.
The problem with this approach is that it is much slower. This is because it uses pure-Python parsing. While the new version of protobuf is upgraded to improve significant performance in parsing and we couldn’t use that.
Conclusion
Due to the breaking changes in protobuf version 4.21.0 most of the libraries which were dependent on it broke on update. This version is better in parsing performance but is not backward compatible. So, the libraries which are not upgraded with this change are still dependent on previous versions of protobuf and throwing typeerror: descriptors cannot not be created directly.
