Error: The 'Referer' header must be modified using the appropriate property or method.

[AgentSam]

Checklist

• I looked at https://github.com/pbatard/rufus/wiki/FAQ to see if my question has already been answered.
• I performed a search in the issue tracker for similar issues using keywords relevant to my problem, such as the error message I got from the log.
• I clicked the 'Log' button or pressed Ctrl-L in Rufus, and copy/pasted the log into the line that says <FULL LOG> below.
• The log I am copying is the FULL log, starting with the line Rufus version: x.y.z - I have NOT removed any part of it.

Additionally (if applicable):

• I ran a bad blocks check, by clicking Show advanced format options then Check device for bad blocks, and confirmed that my USB is not defective.
• I also tried one or more of the following:
  • Using a different USB drive.
  • Plugging the USB into a different port.
  • Running Rufus on a different computer.
• If using an image, I clicked on the (✓) button to compute the MD5, SHA1 and SHA256 checksums, which are therefore present in the log I copied. I confirmed, by performing an internet search, that these values match the ones from the official image.

Issue description

I was trying to download a Windows 10 ISO-image with Rufus on Windows 7 SP1, 64-bit, Ultimate-Edition.

I got as far as selecting the Version, Release, Edition and Language options (as shown below).

At this point, when clicking on "Continue", Rufus came up with an ERROR message, stating:

Error: The 'Referer' header must be modified using the appropriate property or method.
Parameter name: name

This completely prevented me from accessing the ISO-downloads, but gave me no clue as to which component of my Windows 7 SP1 installation might have failed to set the "referer" header.

I can only surmise that this could be related to the older version of Internet Explorer on my Windows 7 setup -- but then again, I have also read elsewhere that the current version of the downloader script does not rely on IE.

Also, this error does not come up on my Windows 10 setup at all. In other words, on my Windows 10 setup, RUFUS downloads ISO-images just fine. I have been using it as a workaround, but obviously it'd be "nice to know" how to fix this and what's causing it. I guess there must be some component of Windows 7 that I should try to update.

Also, technically this problem is propably related to the FIDO PowerShell script to download Windows ISOs, but since I'm using it via RUFUS decided I'd post the issue here at the RUFUS issue tracker.

Log

Rufus x86 v3.20.1929
Windows version: Windows 7 SP1 64-bit
Syslinux versions: 4.07/2013-07-25, 6.04/pre1
Grub versions: 0.4.6a, 2.06
System locale ID: 0x0409 (en-US)
Will use default UI locale 0x0409
SetLGP: Successfully set NoDriveTypeAutorun policy to 0x0000009E
Localization set to 'en-US'
0 devices found
Downloading https://github.com/pbatard/Fido/releases/download/v1.31/Fido.ps1.lzma
File length: 17.6 KB
Successfully downloaded 'Fido.ps1.lzma'
Download signature is valid ✓
Script signature is valid ✓
Please Wait...
Querying https://www.microsoft.com/en-US/software-download/
Querying https://www.microsoft.com/en-US/api/controls/contentinclude/html?pageId=a8f8f489-4c7f-463a-9ca6-5cff94d8d041&host=www.microsoft.com&segments=software-download,Windows10ISO&query=&action=getskuinformationbyproductedition&sessionId=59e44920-e918-4e0d-b801-283cb702079b&productEditionId=2084&sdVersion=2
Querying https://www.microsoft.com/en-US/api/controls/contentinclude/html?pageId=6e2a1789-ef16-4f27-a296-74ef7ef5d96b&host=www.microsoft.com&segments=software-download,Windows10ISO&query=&action=GetProductDownloadLinksBySku&sessionId=59e44920-e918-4e0d-b801-283cb702079b&skuId=13378&language=English&sdVersion=2
Error: The 'Referer' header must be modified using the appropriate property or method.
Parameter name: name

[pbatard]

This is most likely a consequence of pbatard/Fido#41 and its fix where we added -Headers @{'Referer' = 'https://www.microsoft.com/software-download/windows11'} when calling Invoke-WebRequest.

I guess that older versions of PowerShell, such as the one used by Windows 7, do not support using -Headers to set the referer property.

I've had a quick look to see if there was a specific "old PowerShell" way of doing this, but couldn't find any.

Now of course, your problem becomes is that I no longer support Windows 7 in any way shape or form, so, unless somebody else wants to spend their time investigating and proposing a fix, I am currently planning to drop the ISO download functionality of Rufus on Windows 7 platforms that use a version of PowerShell that is incompatible with setting the referrer as a result of this issue, since you won't be able to download much of anything...

I have also opened pbatard/Fido#44, to give people who are interested in fixing this a chance, since I all I am planning to do at this stage, is disable Fido downloads on any platform that does not meet the PowerShell requirements for Invoke-WebRequest -Headers @{ 'Referer' = '...' }

[teknixstuff]

I'll have a look at this tomorrow.

[AgentSam]

Sidenote: I'm also curious as to whether the "referer" header is absolutely necessary?

Do the Microsoft download servers block (or somehow cripple) the ISO-downloads if the referer-header is not set, or if one simply goes directly to attempt the download with the direct ISO-link?

And on a related note, if one chooses the option to "Download via browser" in the settings dialog, does the referer-header somehow get passed to the browser in this case?

[pbatard]

Do the Microsoft download servers block (or somehow cripple) the ISO-downloads if the referer-header is not set

Yup. Per pbatard/Fido#41, if you don't set the referer in the header, then the POST query that provides the download links will be rejected by the Microsoft servers. That's the whole reason we added the referer in the first place, and why the script no longer works for people who use a PowerShell version that doesn't support -Headers @{'Referer' = 'https://www.microsoft.com/software-download/windows11'}

Note that we need to issue two requests before we can get the ISO download links and that the download links themselves don't need anything special to work. But to get to these links, you must get the Microsoft servers to answer the requests, and you need a referer set.

if one chooses the option to "Download via browser" in the settings dialog, does the referer-header somehow get passed to the browser in this case?

Again, there's no need for a referer or anything special once you get the download links.
But the download links have a time limit of 24 hours, which means we need a way to generate them, which means we need to query the Microsoft servers for them, which means that we need to make these servers think that our query comes from a Microsoft page, else they will flat out reject the query.

[AgentSam]

Roger that. Thanks for the clarifications, Pete.

[pbatard]

For your reference, from the log above:

1. Querying https://www.microsoft.com/en-US/software-download/
   This is just a run of the mill check to confirm that the user can access the Microsoft download page. It's not exactly needed, but it allows us to validate the locale, or fallback to en-US.

2. Querying https://www.microsoft.com/en-US/api/controls/contentinclude/html?pageId=a8f8f489-4c7f-463a-9ca6-5cff94d8d041&host=www.microsoft.com&segments=software-download,Windows10ISO&query=&action=getskuinformationbyproductedition&sessionId=59e44920-e918-4e0d-b801-283cb702079b&productEditionId=2084&sdVersion=2
   This is a GET query to the Microsoft servers to get the languages available for a specific version of Windows. This query does not require a referer to be set.

3. Querying https://www.microsoft.com/en-US/api/controls/contentinclude/html?pageId=6e2a1789-ef16-4f27-a296-74ef7ef5d96b&host=www.microsoft.com&segments=software-download,Windows10ISO&query=&action=GetProductDownloadLinksBySku&sessionId=59e44920-e918-4e0d-b801-283cb702079b&skuId=13378&language=English&sdVersion=2
   This is a POST query to obtain the download links (ISO URLs that are valid for 24 hours) for the various architectures available for a specific version of Windows in the language selected by the user (from the list of languages that was retrieved from the previous query). Microsoft recently changed their servers so that this query will fail unless the referer field is set to a Microsoft URL. Thus, you won't be able to get ISO URLs on Windows 7 unless you can set a referer when issuing the query.

[AgentSam]

OK. Roger that. Thank you for the additional clarifications, again.

[pbatard]

See pbatard/Fido#44 (comment)

[pbatard]

Closed in pbatard/Fido@9beb231

[AgentSam]

Thank you so much -- for managing to find a backward-compatibility fix for this!

In other words, I can confirm that Windows 10/11 ISO downloads now work flawlessly with Rufus 3.20 (via the latest fido.ps1 script), even on Windows 7 SP1.

(And forgive me for being so "entrenched" that I still need Windows 7 on some of my systems.)

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue if you think you have a related problem or query.