Need more control over TLS

[JTJag]

Quite often it happens that tls interceptors do not bypass the protection.
I checked manually and found that on some sites not only cipher suites are checked, but also the presence of GREASE and the contents of some extensions. I understand that to control this, need to rewrite the entire HTTPS agent and create your own TLS implementation. This solution will definitely be slower than native libraries. I have already achieved some results in this direction. Are you considering adding such functionality to got-scrapping?

[barjin]

Hello and thank you for your interest in this project!

Indeed, with our in-house web scraping projects, we also noticed that online services are using JA3 (or similar TLS) fingerprints more and more. For a moment, we entertained the idea of creating our own, JS-only implementation of the native HTTP functionality - but abandoned this promptly.

Right now, we are going through other options - such as Node native bindings for Rust and Go, which both allow you to modify the entire HTTP / TLS stack a bit better, while still keeping the developer experience ok-ish. This is still a long-term goal, though.

You're mentioning you have achieved some results in this direction - would you mind sharing? While we are not planning to implement the HTTP agent all again in JS, seeing your approach might help change our opinion on that :)

Cheers!