Bumps the version of Python used for type-checking to 3.11. That is to
say, pyright now supports type-checking idioms that were introduced
after 3.7 like `FooType | None`.
Regenerated `Pipfile.lock` after updating `Pipfile` using `pipenv lock --dev --keep-outdated`.
Adds support for type checking python code using pyright.
Also adds a dependency on `mypy-protobuf` in order to generate `.pyi`
type stubs for generated protobuf code, without which the type checker
gets very confused.
pyright was chosen because it seems like the leading python type checker
these days, and because it has broad python3 version support.
This PR also integrates pyright with `butler.py lint`, such that changed
files can optionally be type-checked if the `--type-check` command-line
argument is set. This allows for local experimentation before we decide
on the next steps.
### Motivation
Last attempted was
[reverted](https://github.com/google/clusterfuzz/pull/4064) due to an
issue in utask main scheduler, related to protos and gcp batch
This PR attempts to fix the following issue:
```
AttributeError: Unknown field for InstancePolicy: boot_disk
at .__setattr__ ( /mnt/scratch0/clusterfuzz/src/third_party/proto/message.py:932 )
at ._get_allocation_policy ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:182 )
at ._create_job ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:225 )
at .create_uworker_main_batch_jobs ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:139 )
at .schedule_utask_mains ( /mnt/scratch0/clusterfuzz/src/python/bot/startup/run_bot.py:105 )
at .task_loop ( /mnt/scratch0/clusterfuzz/src/python/bot/startup/run_bot.py:136 )
```
### Testing strategy
This issue was caused because batch was set to 0.2.0, and the boot_disk
proto field was only [introduced in
0.6.0](https://cloud.google.com/python/docs/reference/batch/latest/changelog).
Returning the dependency to 0.11.0 solved the issue. This was tested
manually through
[use_batch.py](bcae6b19f3/src/clusterfuzz/_internal/tests/core/google_cloud_utils/use_batch.py (L4)),
pointing to our internal gcp service.
Evidence that a batch job is successfully created:
This reverts commit 7e57e8821d35ec2b12c957239ae75c9bf33964f7.
New error:
```
AttributeError: Unknown field for InstancePolicy: boot_disk
at .__setattr__ ( /mnt/scratch0/clusterfuzz/src/third_party/proto/message.py:932 )
at ._get_allocation_policy ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:182 )
at ._create_job ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:225 )
at .create_uworker_main_batch_jobs ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:139 )
at .schedule_utask_mains ( /mnt/scratch0/clusterfuzz/src/python/bot/startup/run_bot.py:105 )
at .task_loop ( /mnt/scratch0/clusterfuzz/src/python/bot/startup/run_bot.py:136 )
```
They are broken in prod causing this error:
https://pantheon.corp.google.com/errors/detail/CPv0x-n-iqv-jQE?e=-13802955&mods=logs_tg_prod&project=google.com:clusterfuzz
```
Unable to update heartbeat.
Traceback (most recent call last):
File "/mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/datastore/data_handler.py", line 1130, in update_heartbeat
heartbeat = ndb.Key(data_types.Heartbeat, bot_name).get()
File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/_options.py", line 102, in wrapper
return wrapped(*pass_args, **kwargs)
File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/utils.py", line 150, in positional_wrapper
return wrapped(*args, **kwds)
File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/key.py", line 859, in get
return self.get_async(_options=_options).result()
File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/tasklets.py", line 210, in result
self.check_success()
File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/tasklets.py", line 157, in check_success
raise self._exception
File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/tasklets.py", line 319, in _advance_tasklet
yielded = self.generator.throw(type(error), error, traceback)
File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/key.py", line 944, in get
entity_pb = yield _datastore_api.lookup(self._key, _options)
File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/tasklets.py", line 323, in _advance_tasklet
yielded = self.generator.send(send_value)
File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/_datastore_api.py", line 165, in lookup
entity_pb = yield batch.add(key)
File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/ndb/_datastore_api.py", line 217, in add
todo_key = key.to_protobuf()._pb.SerializeToString()
File "/mnt/scratch0/clusterfuzz/src/third_party/google/cloud/datastore/key.py", line 324, in to_protobuf
key.path.append(element)
TypeError: Parameter to MergeFrom() must be instance of same class: expected google.datastore.v1.Key.PathElement got PathElement.
```
Vitor says this isn't a deterministic bug.
Some of our current dependencies are incompatible with Python 3.11. This
PR attempts to bump the least possible amount of dependencies (more
specifically, grpcio, grpcio-tools, psutil and markupsafe), so we can
isolate deployment issues between dependency bumps and the python
upgrade itself.
I think I must've installed a newer psutil to get this working last
time.
https://github.com/giampaolo/psutil/issues/2165
I've confirmed this works on a linux machine without a compiler
CC @mihaimaruseac
1. Change Docker images to use Python3
2. Change local scripts to support Python3
3. Fix installation of Chromium dependencies
4. Remove profiling code, the profiling package only supports up to
Python3.9
5. Recompile all protos and change some uworker serialization code that
has to work differently due to some changes in dependencies
6. Add many no-member pylint exemptions since pylint is now
misunderstanding proto members.
7. Change platforms for binary downloads during deploy
8. Upgrade packages that no longer work or have to be upgraded by
because of depdency/dependent upgrades
[Used everywhere]
a. grpcio
b. grpcio-tools
c. protobuf
[development]
d. parameterized
e. WebTest
[bots]
f. google-cloud-batch
g. google-cloud-core
h. google-cloud-datastore
i. google-cloud-logging
j. google-cloud-monitoring
k. google-cloud-ndb
l. google-cloud-storage
m. google-api-python-client
n. google-auth
o. google-api-core
[appengine]
p. cryptography
q. firebase-admin
r. PyJWT
9. Remove no longer used pip packages:
a. twisted
b. lxml
* Pin all vendored dependencies.
Use a Pipfile to generate the requirements.txt to vendor.
For App Engine only dependencies, we use "dev-packages" as a hack to
make sure that version resolution always works and doesn't introduce
conflicts.
* fixes
Move most imports into an `clusterfuzz._internal` dir, and
change internal imports to be of the form: `from clusterfuzz._internal.base.* import ` etc.
This avoids a lot of global import namespace pollution, and makes our
libClusterFuzz setup much less hacky.
Actual changes:
- Rename most of "src/python" to "src/clusterfuzz/_internal".
- Keep "src/python/bot/startup" to avoid issues with existing startup
scripts.
- Move "protos" to "src/clusterfuzz/_internal" as well.
- Convert all imports to be relative from clusterfuzz._internal.
- Use isort for import ordering, remove our own implementation for
checking of ordering during lint.
- Temporarily add some lint exclusion rules, as there were quite a few
that were failing.
- Relax line length rules for test files.
- Add a license check exclusion for process_state.proto, since it has a
difference license from what we expect.
Tests before deploying:
- [x] stage on GCE bot
- [x] deploy to App Engine staging
- [x] test pip package
Fixes#2192.
- Remove tensorflow from Pipfile as it repeatedly causes issues when a
new Python version needs to be supported. It's not required for the
majority of ClusterFuzz devs and most things run fine without it.
- Upgrade grpcio.
Fix unittests failing on HEAD because of a bug in pyfakefs.
Since all of these tests used fakefs, I tried upgrading it, which seemed to work.
Also upgraded the pipenv.lock file.
Fixes#2246
* Remove some dependencies for libClusterFuzz.
- Move selenium to Pipfile.
- Move python-dateutil to appengine requirements.
- Remove hyperlink pin.
- Remove pytz.
- Remove configparser. This was a Python 2 backport of a Python 3
module.
- Remove peach, antlr dependencies for Pip package.
- Also some import fixes for the pip package.
* remove debug print
First release that contains:
a04a928cf3
We upgrade conservatively for now to avoid any other issues that may
arise while we migrate from Python 2 -> 3.
Besides the typical py2->3 problem in src/python/bot/fuzzers/ml/rnn/utils.py, another issue here is that tensorflow 1.8.0 is way too old and wasn't even built for Python3.7, and official tensorflow GCS bucket doesn't have it.
1.13.2 seems to be the first version built for Python3.7, and it's also not that new to break code compatibility. Should be acceptable in the meantime. After migration we'll update TF and its dependencies altogether (#1540).
