GitHub/clusterfuzz
1
0
Fork 0
mirror of https://github.com/google/clusterfuzz.git synced 2025-03-16 05:02:07 +08:00
Code Issues Packages Projects Releases Wiki Activity
clusterfuzz/Pipfile

31 lines
564 B
TOML
Raw Normal View History

Preliminary environment setup for Python 3. (#1415) Add Pipfile/Pipfile.lock for Python 3 dependencies. This is currently just based on src/local/requirements.txt and docker/ci/requirements.txt. Add python 3 env setup to install_deps_linux.bash. To use, set `PY3=1`. A separate checkout should be used from the one used for Python 2 development. Some minor bug fixes to get butler lint/format and unit tests running. The vast majority of them are still failing though. Vendor webapp2 and Jinja2 in src/third_party.
2020-02-05 16:12:14 +11:00
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"
[packages]
Attempting to bump proto/grpcio in Python 3.7 - 3rd attempt (#4066) ### Motivation Last attempted was [reverted](https://github.com/google/clusterfuzz/pull/4064) due to an issue in utask main scheduler, related to protos and gcp batch This PR attempts to fix the following issue: ``` AttributeError: Unknown field for InstancePolicy: boot_disk at .__setattr__ ( /mnt/scratch0/clusterfuzz/src/third_party/proto/message.py:932 ) at ._get_allocation_policy ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:182 ) at ._create_job ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:225 ) at .create_uworker_main_batch_jobs ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:139 ) at .schedule_utask_mains ( /mnt/scratch0/clusterfuzz/src/python/bot/startup/run_bot.py:105 ) at .task_loop ( /mnt/scratch0/clusterfuzz/src/python/bot/startup/run_bot.py:136 ) ``` ### Testing strategy This issue was caused because batch was set to 0.2.0, and the boot_disk proto field was only [introduced in 0.6.0](https://cloud.google.com/python/docs/reference/batch/latest/changelog). Returning the dependency to 0.11.0 solved the issue. This was tested manually through [use_batch.py](https://github.com/google/clusterfuzz/blob/bcae6b19f399a74ca105404284a2cad8e174627e/src/clusterfuzz/_internal/tests/core/google_cloud_utils/use_batch.py#L4), pointing to our internal gcp service. Evidence that a batch job is successfully created: ![image](https://github.com/google/clusterfuzz/assets/5733577/85b8c476-e1c0-4e89-b5ce-3c2d0d1cc578) ![image](https://github.com/google/clusterfuzz/assets/5733577/39469891-0a5c-4d66-ae81-c12240a08c6f)
2024-06-28 13:30:00 -03:00
cryptography = "==37.0.4"
Preliminary environment setup for Python 3. (#1415) Add Pipfile/Pipfile.lock for Python 3 dependencies. This is currently just based on src/local/requirements.txt and docker/ci/requirements.txt. Add python 3 env setup to install_deps_linux.bash. To use, set `PY3=1`. A separate checkout should be used from the one used for Python 2 development. Some minor bug fixes to get butler lint/format and unit tests running. The vast majority of them are still failing though. Vendor webapp2 and Jinja2 in src/third_party.
2020-02-05 16:12:14 +11:00
crcmod = "==1.7"
Reinstall future. (#3704) Some fuzzers depend on it.
2024-02-01 13:37:07 -05:00
future = "==0.17.1"
Attempting to bump proto/grpcio in Python 3.7 - 3rd attempt (#4066) ### Motivation Last attempted was [reverted](https://github.com/google/clusterfuzz/pull/4064) due to an issue in utask main scheduler, related to protos and gcp batch This PR attempts to fix the following issue: ``` AttributeError: Unknown field for InstancePolicy: boot_disk at .__setattr__ ( /mnt/scratch0/clusterfuzz/src/third_party/proto/message.py:932 ) at ._get_allocation_policy ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:182 ) at ._create_job ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:225 ) at .create_uworker_main_batch_jobs ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:139 ) at .schedule_utask_mains ( /mnt/scratch0/clusterfuzz/src/python/bot/startup/run_bot.py:105 ) at .task_loop ( /mnt/scratch0/clusterfuzz/src/python/bot/startup/run_bot.py:136 ) ``` ### Testing strategy This issue was caused because batch was set to 0.2.0, and the boot_disk proto field was only [introduced in 0.6.0](https://cloud.google.com/python/docs/reference/batch/latest/changelog). Returning the dependency to 0.11.0 solved the issue. This was tested manually through [use_batch.py](https://github.com/google/clusterfuzz/blob/bcae6b19f399a74ca105404284a2cad8e174627e/src/clusterfuzz/_internal/tests/core/google_cloud_utils/use_batch.py#L4), pointing to our internal gcp service. Evidence that a batch job is successfully created: ![image](https://github.com/google/clusterfuzz/assets/5733577/85b8c476-e1c0-4e89-b5ce-3c2d0d1cc578) ![image](https://github.com/google/clusterfuzz/assets/5733577/39469891-0a5c-4d66-ae81-c12240a08c6f)
2024-06-28 13:30:00 -03:00
protobuf = "==4.23.4"
Bumping deps to an intermediary step compatible with Python3.11 (#4005) Some of our current dependencies are incompatible with Python 3.11. This PR attempts to bump the least possible amount of dependencies (more specifically, grpcio, grpcio-tools, psutil and markupsafe), so we can isolate deployment issues between dependency bumps and the python upgrade itself.
2024-06-07 13:40:29 -03:00
psutil = "==5.9.4"
Preliminary environment setup for Python 3. (#1415) Add Pipfile/Pipfile.lock for Python 3 dependencies. This is currently just based on src/local/requirements.txt and docker/ci/requirements.txt. Add python 3 env setup to install_deps_linux.bash. To use, set `PY3=1`. A separate checkout should be used from the one used for Python 2 development. Some minor bug fixes to get butler lint/format and unit tests running. The vast majority of them are still failing though. Vendor webapp2 and Jinja2 in src/third_party.
2020-02-05 16:12:14 +11:00
Remove some Python 3 dependencies. (#1612) Some of these seem to be unused.
2020-03-19 09:11:38 +11:00
[dev-packages]
Remove some dependencies for libClusterFuzz. (#2095) * Remove some dependencies for libClusterFuzz. - Move selenium to Pipfile. - Move python-dateutil to appengine requirements. - Remove hyperlink pin. - Remove pytz. - Remove configparser. This was a Python 2 backport of a Python 3 module. - Remove peach, antlr dependencies for Pip package. - Also some import fixes for the pip package. * remove debug print
2020-10-20 08:41:38 +11:00
Fabric = "==1.14.1"
Attempting to bump proto/grpcio in Python 3.7 - 3rd attempt (#4066) ### Motivation Last attempted was [reverted](https://github.com/google/clusterfuzz/pull/4064) due to an issue in utask main scheduler, related to protos and gcp batch This PR attempts to fix the following issue: ``` AttributeError: Unknown field for InstancePolicy: boot_disk at .__setattr__ ( /mnt/scratch0/clusterfuzz/src/third_party/proto/message.py:932 ) at ._get_allocation_policy ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:182 ) at ._create_job ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:225 ) at .create_uworker_main_batch_jobs ( /mnt/scratch0/clusterfuzz/src/clusterfuzz/_internal/google_cloud_utils/batch.py:139 ) at .schedule_utask_mains ( /mnt/scratch0/clusterfuzz/src/python/bot/startup/run_bot.py:105 ) at .task_loop ( /mnt/scratch0/clusterfuzz/src/python/bot/startup/run_bot.py:136 ) ``` ### Testing strategy This issue was caused because batch was set to 0.2.0, and the boot_disk proto field was only [introduced in 0.6.0](https://cloud.google.com/python/docs/reference/batch/latest/changelog). Returning the dependency to 0.11.0 solved the issue. This was tested manually through [use_batch.py](https://github.com/google/clusterfuzz/blob/bcae6b19f399a74ca105404284a2cad8e174627e/src/clusterfuzz/_internal/tests/core/google_cloud_utils/use_batch.py#L4), pointing to our internal gcp service. Evidence that a batch job is successfully created: ![image](https://github.com/google/clusterfuzz/assets/5733577/85b8c476-e1c0-4e89-b5ce-3c2d0d1cc578) ![image](https://github.com/google/clusterfuzz/assets/5733577/39469891-0a5c-4d66-ae81-c12240a08c6f)
2024-06-28 13:30:00 -03:00
grpcio-tools = "==1.62.2"
Remove some dependencies for libClusterFuzz. (#2095) * Remove some dependencies for libClusterFuzz. - Move selenium to Pipfile. - Move python-dateutil to appengine requirements. - Remove hyperlink pin. - Remove pytz. - Remove configparser. This was a Python 2 backport of a Python 3 module. - Remove peach, antlr dependencies for Pip package. - Also some import fixes for the pip package. * remove debug print
2020-10-20 08:41:38 +11:00
gunicorn = "*"
Restructure ClusterFuzz to be a proper package. (#2383) Move most imports into an `clusterfuzz._internal` dir, and change internal imports to be of the form: `from clusterfuzz._internal.base.* import ` etc. This avoids a lot of global import namespace pollution, and makes our libClusterFuzz setup much less hacky. Actual changes: - Rename most of "src/python" to "src/clusterfuzz/_internal". - Keep "src/python/bot/startup" to avoid issues with existing startup scripts. - Move "protos" to "src/clusterfuzz/_internal" as well. - Convert all imports to be relative from clusterfuzz._internal. - Use isort for import ordering, remove our own implementation for checking of ordering during lint. - Temporarily add some lint exclusion rules, as there were quite a few that were failing. - Relax line length rules for test files. - Add a license check exclusion for process_state.proto, since it has a difference license from what we expect. Tests before deploying: - [x] stage on GCE bot - [x] deploy to App Engine staging - [x] test pip package Fixes #2192.
2021-07-26 14:42:30 +10:00
isort = "*"
Add pyright support. (#3452) Adds support for type checking python code using pyright. Also adds a dependency on `mypy-protobuf` in order to generate `.pyi` type stubs for generated protobuf code, without which the type checker gets very confused. pyright was chosen because it seems like the leading python type checker these days, and because it has broad python3 version support. This PR also integrates pyright with `butler.py lint`, such that changed files can optionally be type-checked if the `--type-check` command-line argument is set. This allows for local experimentation before we decide on the next steps.
2024-09-03 02:07:24 +02:00
mypy-protobuf = "==3.4.0"
nodeenv = "==1.9.1"
[python upgrade] Upgrade test-only packages (#3376) This will make it possible to upgrade to Python3.11 Relands changes reverted in #3264.
2023-10-02 15:12:15 -04:00
parameterized = "==0.9.0"
Remove some Python 3 dependencies. (#1612) Some of these seem to be unused.
2020-03-19 09:11:38 +11:00
paramiko = "==2.6.0"
Update pipenv version (#2982) Update pipenv version to fix [issue 2981](https://github.com/google/clusterfuzz/issues/2981) --------- Co-authored-by: Holly Gong <gongh@google.com>
2023-03-30 15:33:17 +11:00
pipenv = "==2022.8.5"
Fix some flaky tests. (#2347) * Reenable incorrectly marked flaky tests. * Try ulimit fix. * Another try. * try pyfakefs upgrade. * Fix another test failure. * Try upgrading grpcio
2021-05-21 12:48:07 -07:00
pyfakefs = "==4.4.0"
Remove some dependencies for libClusterFuzz. (#2095) * Remove some dependencies for libClusterFuzz. - Move selenium to Pipfile. - Move python-dateutil to appengine requirements. - Remove hyperlink pin. - Remove pytz. - Remove configparser. This was a Python 2 backport of a Python 3 module. - Remove peach, antlr dependencies for Pip package. - Also some import fixes for the pip package. * remove debug print
2020-10-20 08:41:38 +11:00
pylint = "~=2.4"
Use protos directly in utasks (#3536) This change removes the uworker_io wrappers and uses protos directly within the utasks. This change also removes the Json user defined proto.
2023-12-11 07:33:13 +01:00
pylint-protobuf = "==0.21.0"
Update pyright to 1.1.393 and type-checking python version to 3.11 (#4655) Bumps the version of Python used for type-checking to 3.11. That is to say, pyright now supports type-checking idioms that were introduced after 3.7 like `FooType | None`. Regenerated `Pipfile.lock` after updating `Pipfile` using `pipenv lock --dev --keep-outdated`.
2025-02-03 18:31:25 +01:00
pyright = "==1.1.393"
Remove some dependencies for libClusterFuzz. (#2095) * Remove some dependencies for libClusterFuzz. - Move selenium to Pipfile. - Move python-dateutil to appengine requirements. - Remove hyperlink pin. - Remove pytz. - Remove configparser. This was a Python 2 backport of a Python 3 module. - Remove peach, antlr dependencies for Pip package. - Also some import fixes for the pip package. * remove debug print
2020-10-20 08:41:38 +11:00
selenium = "==3.141.0"
Fix some flaky tests. (#2347) * Reenable incorrectly marked flaky tests. * Try ulimit fix. * Another try. * try pyfakefs upgrade. * Fix another test failure. * Try upgrading grpcio
2021-05-21 12:48:07 -07:00
twine = "*"
[python upgrade] Upgrade test-only packages (#3376) This will make it possible to upgrade to Python3.11 Relands changes reverted in #3264.
2023-10-02 15:12:15 -04:00
WebTest = "==3.0.0"
Add pyright support. (#3452) Adds support for type checking python code using pyright. Also adds a dependency on `mypy-protobuf` in order to generate `.pyi` type stubs for generated protobuf code, without which the type checker gets very confused. pyright was chosen because it seems like the leading python type checker these days, and because it has broad python3 version support. This PR also integrates pyright with `butler.py lint`, such that changed files can optionally be type-checked if the `--type-check` command-line argument is set. This allows for local experimentation before we decide on the next steps.
2024-09-03 02:07:24 +02:00
yapf = "==0.22.0"
Reference in New Issue Copy Permalink