Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(web): fix bug and add mock users' behaviours by chromedp #701

Merged
merged 3 commits into from
Oct 4, 2024
Merged

feat(web): fix bug and add mock users' behaviours by chromedp #701

merged 3 commits into from
Oct 4, 2024

Conversation

chuang8511
Copy link
Contributor

Because

  • there is a handshake error
  • the page we target to scrape is dynamic

This commit

  • before full investigation about why there is a handshake issue, we use curl to fetch the static content
  • we add mock users' behaviours with scrolling down as a default

@linear Linear
Copy link

linear bot commented Oct 4, 2024

INS-6490 Investigate handshake failure with web scraper on url-to-catalog

donch1989
donch1989 reviewed Oct 4, 2024
View reviewed changes
pkg/component/operator/web/v0/scrape_webpage.go Outdated
return doc, nil
}

// TODO: Investigate the root cause of the handshake error and remove this temporary solution.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chuang8511
This solution is a bit weird.
Let me help investigate it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I will come back to it when I finish the dinner. Probably 9 p.m. London time.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I'll directly push changes to your branch if I have progess

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This issue should be this

By default, cipher suites without ECDHE support are no longer offered by either clients or servers during pre-TLS 1.3 handshakes. This change can be reverted with the tlsrsakex=1 GODEBUG setting.

We can add this flag back.

@donch1989 donch1989 force-pushed the chunhao/ins-6490-web-operator branch from 994d545 to de50f47 Compare October 4, 2024 17:14
@donch1989 donch1989 merged commit ca312fe into main Oct 4, 2024
11 checks passed
@donch1989 donch1989 deleted the chunhao/ins-6490-web-operator branch October 4, 2024 17:42
@droplet-bot droplet-bot mentioned this pull request Oct 4, 2024
Merged
donch1989 added a commit that referenced this pull request Oct 4, 2024
@chuang8511 @donch1989
feat(web): fix bug and add mock users' behaviours by chromedp (#701)
a578b4f 
Because

- there is a handshake error
- the page we target to scrape is dynamic

This commit

- before full investigation about why there is a handshake issue, we use
curl to fetch the static content
- we add mock users' behaviours with scrolling down as a default

---------

Co-authored-by: Chang, Hui-Tang <[email protected]>
jvallesm pushed a commit that referenced this pull request Oct 8, 2024
@droplet-bot
chore(main): release 0.42.0-beta (#700)
2c29c7f 
🤖 I have created a release *beep* *boop*
---


##
[0.42.0-beta](v0.41.0-beta...v0.42.0-beta)
(2024-10-08)


### Features

* **document:** repair pdf with libreoffice
([#683](#683))
([b6a738f](b6a738f))
* **integration:** add identity to OAuth connections
([#699](#699))
([e2e9237](e2e9237))
* merge worker into main process and optimize graceful shutdown
mechanism
([#713](#713))
([69b64a0](69b64a0))
* **universalai:** add instill credit function for universal ai
component
([#680](#680))
([9ce10b5](9ce10b5))
* use explicit parameter for target namespace when cloning pipeline
([#711](#711))
([ab2e3a6](ab2e3a6))
* **web:** fix bug and add mock users' behaviours by chromedp
([#701](#701))
([ca312fe](ca312fe))


### Bug Fixes

* **component:** use kebab-case for icon name
([#703](#703))
([cd587e2](cd587e2))
* **document:** catch the error if the bbox is out of boundary
([#718](#718))
([4fe03e6](4fe03e6))
* fix the component render error was not being returned
([#707](#707))
([931f067](931f067))
* fix the wrong target namespace when cloning pipeline
([#706](#706))
([75a757b](75a757b))
* **integration:** update GitHub scopes to read/write repository issues
([#709](#709))
([c1cf4ce](c1cf4ce))
* **text:** fix the bug about the list position
([#712](#712))
([49aed28](49aed28))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@donch1989 donch1989 donch1989 left review comments

@GeorgeWilliamStrong GeorgeWilliamStrong Awaiting requested review from GeorgeWilliamStrong GeorgeWilliamStrong is a code owner

@pinglin pinglin Awaiting requested review from pinglin

@xiaofei-du xiaofei-du Awaiting requested review from xiaofei-du

@jvallesm jvallesm Awaiting requested review from jvallesm jvallesm is a code owner

Assignees
No one assigned
Labels
instill component instill vdp
Projects
No open projects
🔮 Instill Core
Status: No status
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@chuang8511 @donch1989 @droplet-bot