Advice for setting up the environment of a python project

If you haven't yet, read the Python Packaging User Guide which discusses many topics in this area.

So for each new project is it best to create a new virtual environment?

Yes absolutely. Tools like uv and Hatch can do this automatically. You can of course do it by hand with python -m venv as well.

I would like to add a requirements.txt...

In your pyproject.toml, there should be a dependencies line that lists the libraries you depend on; also see the Dependencies and requirements section of the Python Packaging User Guide. A best practice here is to declare a range of versions you accept, typically from some minor version up to the next major version.

dependencies = [
  "pandas ~=2.0",      # avoid recently-released Pandas 3.0 for now
  "Django >=4.2, 

If you're only planning on using these projects on one machine, then you can get by with just conda environments instead of also making Python virtual environments. However, virtual environments are convenient for portability to different machines.

I'm a big fan of using uv, and apparently it's possible to use uv within a conda environment; see this article for instance. With uv, all you need to produce a requirements.txt for the given project is the command

uv pip freeze > requirements.txt