Convert test runner to use pypiserver package as standalone process (#5284)

* Check point progress on moving tests to pypiserver.

* Allow other indexes to mimic the pypi json API for package hashes.

* Fix these tests that run on lower python versions only.

* Try adding the pypiserver to the github actions to only run once.

* Expand the scope of fixtures for pypiserver.

* try to accomedate microsoft runner.

* Windows networking troubles.

* Remove running as a background job.

* Try to condtionally invoke different start pypi-server commands

* Try to condtionally invoke different start pypi-server commands

* Try to condtionally invoke different start pypi-server commands

* Try to condtionally invoke different start pypi-server commands

* Try to condtionally invoke different start pypi-server commands

* Try to condtionally invoke different start pypi-server commands

* Try to condtionally invoke different start pypi-server commands

* Try to introduce pypi as the root index because setuptools-scm is not in our pypi artifacts.

* see if the windows tests run faster (and the other tests) by supplying waitress.

* Only use waitress on windows because the others are fast on the default.  Fix requests pollution.

* Supply a suitable Pipfile instead for these two failing tests.

* More requests resolver cross test contamination cleanup.

* remove problematic tests because even on my version of python 3.8.12 this does not work due to AttributeError: 'HTMLParser' object has no attribute 'unescape'

* fix mirror install test.

* Fix Pipfile.

* Fix Pipfile for real

* Fix tests

* Cleanup test naming and more test enhancements.

* Save this optimization for a subsequent PR.

* Cleanup the TemporaryDirectory between tests.

* resolve merge conflict.

* Cleanup path initalization -- it hsould always be a TemporaryDirectory for tests that gets cleanedup.

* Cleanup path initalization -- it hsould always be a TemporaryDirectory for tests that gets cleanedup.

* tableflip on those requests tests that read the setup metadata in reqlib from other tests.

* Update developer documentation for running tests.

* add news fragment.

* Try gunicorn perfoormance for linux/mac

* Only use gunicorn on linux based on the results of last run.

.github/workflows/ci.yaml

@@ -85,7 +85,7 @@ jobs:
           echo ::set-output name=path::$(python -c "import sys; print(sys.executable)")
       - name: Install latest pip, setuptools, wheel
         run: |
-          python -m pip install --upgrade pip setuptools wheel --upgrade
+          python -m pip install --upgrade pip setuptools wheel
       - name: Install dependencies
         env:
           PIPENV_DEFAULT_PYTHON_VERSION: ${{ matrix.python-version }}
@@ -97,6 +97,18 @@ jobs:
           git submodule update --init --recursive
           python -m pip install -e . --upgrade
           pipenv install --deploy --dev --python=${{ steps.python-path.outputs.path }}
+      - name: Run pypiserver Windows
+        run: |
+         cmd /c start pipenv run pypi-server run -v --host=0.0.0.0 --port=8080 --hash-algo=sha256 --disable-fallback ./tests/pypi/ ./tests/fixtures
+        if: ${{matrix.os  == 'Windows' }}
+      - name: Run pypiserver Mac
+        run: |
+          pipenv run pypi-server run -v --host=0.0.0.0 --port=8080 --hash-algo=sha256 --disable-fallback ./tests/pypi/ ./tests/fixtures &
+        if: ${{matrix.os  == 'MacOS' }}
+      - name: Run pypiserver Ubuntu
+        run: |
+          pipenv run pypi-server run -v --host=0.0.0.0 --port=8080 --hash-algo=sha256 --disable-fallback ./tests/pypi/ ./tests/fixtures &
+        if: ${{matrix.os  == 'Ubuntu' }}
       - name: Run tests
         env:
           PIPENV_DEFAULT_PYTHON_VERSION: ${{ matrix.python-version }}