Using pip-compile --no-index doesn't exclude the --extra-index-url from being added to requirements.txt

[jackton1]

Prior to 4.x.x using pip-compile and --no-index option correctly excluded the --extra-index-url from the generated requirements.txt

This doesn't seem to work with the latest release.

Environment Versions

1. OS Type: Mac OS
2. Python version: Python 3.7.3
3. pip version: pip 19.3.1
4. pip-tools version: pip-compile, version 4.3.0

Steps to replicate

$ pip-compile --extra-index-url=https://mycustomusername:[email protected] --no-index

Expected result

In the requirements.txt

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --no-index --output-file requirements.txt setup.py
#

Actual result

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --extra-index-url=https://mycustomusername:[email protected] --no-index
#

[atugushev]

Hello @jackton1

Thanks for the issue! The fact is the --no-index option has nothing to do with To update, run: header. This option controls whether pip-compile should print out --index-url or --extra-index-url to the output (below the header/commentaries).

Default behavior without --no-index option:

$ echo "" | pip-compile - -qo- --extra-index-url https://foo
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --extra-index-url=https://foo --output-file=- -
#
--extra-index-url https://foo

See --no-index skips --extra-index-url in the output:

$ echo "" | pip-compile - -qo- --extra-index-url https://foo --no-index
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile --extra-index-url=https://foo --no-index --output-file=- -
#

Notice the To update, run: header, there's always --extra-index-url regardless of --no-index.

Prior to 4.x.x using pip-compile and --no-index option correctly excluded the --extra-index-url from the generated requirements.txt

That's because before 4.X --index-url and --extra-index-url were never printed to the header section.

Most likely, you need the --no-header option, which is the recommended way to hide sensitive data from the output.

Should we remove --index-url/--extra-index-url from the header if there is --no-index? I'd say no but would like to hear opinions.

[jackton1]

Personally from the help information it mentions the index url which one could assume relates
to -I, --index-url.

 --index / --no-index            Add index URL to generated file

If this is the case using a similar option --no-extra-index could mean exclude the extra index url explicitly else this could also be another option. Since I'll like to keep the header because it includes warning that the requirements.txt was autogenerated and shouldn't be manually modified.

This could be a new option --no-extra-index or assuming the presence of --no-index also means exclude the extra-index-url from the requirements.txt header.

I prefer the former since it's clear and explicit. @atugushev

[atugushev]

@jackton1

I prefer the former since it's clear and explicit

I must note that --index-url and --extra-index-url were never printed to the output before 3.7.0 regardless whether or not the --no-index was set on. Thus, this former behavior was kind of a bug.

See also #811 (comment) wich could be related.

[jackton1]

Yeah it was quite obvious that this was a bug prior to 4.x.x

I also agree that --emit-index-url would be a viable option.

I'll be keen on finding a middle ground since we recently upgraded to 4.x.x and lost the previous bug/behaviour.

[atugushev]

Let's fix it. See #811 (comment).

[atugushev]

Here is the fix #1146. Please test and review it, any feedback is welcomed!

[jackton1]

Will do. Thanks.

[AndydeCleyre]

@jackton1

Is this resolved with the current redaction behavior?

Thanks for any update!

[jackton1]

Yes this works as expected using --no-emit-index-url but the index URL is still included with the credentials getting redacted and replaced by ***

[AndydeCleyre]

Thanks! Do you think it's sufficient to close the issue?

[jackton1]

Yes, I’ll close this. Thanks