Skip to content

Latest commit

 

History

History
584 lines (562 loc) · 29.3 KB

TODO.md

File metadata and controls

584 lines (562 loc) · 29.3 KB

TODO

Done
  • Test types
  • Add binary
  • Subscribe on workers ready
  • Parallel (need pre-build? worker-farm?)
    • Custom server runner
    • Patch mocha runner with cluster
  • Allow customize hooks to non-storybook env
  • Defined default config + deep merge
  • Build correct d.ts files. For now, you should remove require types from lib/creevey.js after build.
  • Pass config to server initialization (use process.cwd())
  • Single fork for single browser thread
  • Mocha workers for server
  • Add test parser
  • Selenium keep alive
  • Allow start/stop tests
  • reconnect to selenium (sending keep-alive)
  • subscribe websocket api (status commands)
  • websocket api types
  • Generate static page in report dir
  • Save/Load results in/from report dir (js/json)
  • Save images report in multiple runs
  • Not graceful exit master process on error in workers
  • Add worker timeout and restart it
  • Ignore elements from screenshot
  • Custom mocha reporter for worker
  • Allow to use Teamcity reporter
  • Add worker ready event
  • Support load test files from glob patterns
  • Better handle websocket messages
  • Allow define sets (test files, address, browsers)
    • Filter tests by path in parser
    • Merge common config with browser config
  • Web interface
    • webpack build
    • usage react-ui
    • output tests tree
    • allow start/stop
    • comm with API by test id
    • approve images
    • Offline mode, load report data
    • Output test error message
    • output reported images
    • better images view like github does
      • SlideView
      • BlendView
    • switch images by hotkeys
    • output test status (pending)
    • update/re-calc suite status
    • use classnames (emotion)
    • ApprovedRetry
    • Fix incorrect output new images
    • output skipped tests
    • Output test error message
    • Fit large images inside TestResultView
    • Allow view fullscale images
    • Invert expect/actual color
    • Better output test error message
  • Test grep regexp don't work with parenthesis
  • Don't respect skip flag from report json
  • Browser resolution option
  • Fix TeamCity reporter <unknown test name>
  • Output images for Teamcity reporter
  • Setup viewport size
  • Color output in console
  • Server state sync
  • Config npmignore files
  • Status runner
  • Add Storybook for web ui components
  • Restart selenium driver on timeout
  • Export decorator from creevey
  • Improve ts support for creevey (like webpack does)
    • Don't work with ts-node + esnext modules
  • Define simple version for browsers
  • Simplify types re-export for lib usage
  • Calc correct viewport size
  • Fix skip/unskip tests between run without delete report dir
  • Allow clean approved images
  • Generate runtime tests based on stories
  • Reload IE page on start (don't handle storybook hot-reload)
  • Allow leave reason comment for skipped tests
  • Update args readme (config, parser, ...)
  • Add cli arguments
    • config
    • parser
    • ui
    • reporter
    • update
    • init
    • port
  • Storybook integration
  • Allow Composite images
  • Slide story don't work correctly, must be fixed
  • Better error message about open storybook page
  • Exit if worker got UnhandledPromiseRejectionWarning
  • Serialize skip regexp
  • Husky, lint-staged
  • Rework UI
    • Improve UI performance on initial load
    • Put tests tree into side page
    • Output test result view into main page block
    • Output error message multiline
    • Show icons for skipped tests
    • Allow check/uncheck tests without results
    • Add bottom padding into test tree
  • Update incorrect work with new structure directory
  • Reset button nowrap style
  • Skip by browser regexp don't work (webdriver serialization)
  • Convert storycase to export name
  • Allow assert multiple images in one test (chai toMatchImages())
  • Remove Loader, use require.context
  • Remove tests parser ability, support only storybook
  • Add babel-plugin-typescript-to-proptypes Seems this plugin doesn't do so much
  • Allow skip tests inside story
  • Not properly work with CRA (need to install ts-node or @babel/register) Add notes in readme
  • Lint sort imports
  • Allow define mocha hooks
  • Optimize stories load process (don't import other stuff like react, components and other browser libs/styles/images/fonts)
  • Don't output tests without status if filtering by status
  • Patch @babel/register hook to allow use '.ts' along side with '.tsx' extensions
  • HotReload tests files without restart
  • has workaround Can't use By and Key helpers from selenium-webdriver in tests, because webpack try to build bundle with selenium-webdriver
  • Chai used as deps, but in stories should imported explicitly. Add chai to peerdeps
  • Add description for types properties, like config/decorator/etc
  • Init pirates before any compiler (fix error with ts-node (allowJs: true) and pirates order)
  • Add delay option for stories. To allow wait some time before real test started (right after switch story)
  • Add composite screenshot helper (this.takeScreenshot should be composite)
  • Don't apply scrollbar hiding styles in composite images
  • Add authors
  • Handle error on mocha hooks
  • Bugs
    • Reconnect on WebDriverError: Session timed out or not found
    • On Teamcity cli exits with -1 code without any output
    • Don't handle correctly storybook render story errors
    • Readlink don't work on windows. Need to change storybook framework detection
    • Restart workers output errors NoSuchSessionError: Tried to run command without establishing a connection and TypeError: _cluster.default.disconnect is not a function
    • In chrome 80 creevey sometime failed with error MoveTargetOutOfBoundsError: move target out of bounds
    • For firefox composite images captured without scrollbars, but image width has scrollbar width
    • Unexpected loaded state. Did you call load twice?
    • [BABEL] Note: The code generator has deoptimised the styling of /home/kich/Projects/creevey/report/storybook/tmp-8207-HTp79b5JhpxQ-.js as it exceeds the max of 500KB.
    • webpack-hot-middleware's client requires EventSource to work. You should include a polyfill if you want to support this browser: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events#Tools
  • Cutoff subcomponents parameter
  • Apply AST transformation on storybook config directory (cut decorators)
  • EPIPE Error on SIGINT :(
  • Remove unnecessary deps and code, for example pirates, require.context, interpret, (?)other
  • Optimize stories loading
    • Add debug output on fail transformation
    • Use proxy to handle side-effects
    • AST transformation to exclude all source code except stories meta and tests (support only CSF)
  • Add examples
    • Angular
    • Create React App
  • Docs
    • Update framework examples
    • Add delay option
    • Record screencast with Creevey UI
    • Update Readme.md (also describe scenarios or how to capture screenshots)
    • Add topics in top of readme
    • Add instruction for various frameworks
      • Angular
      • Create React App
  • Storybook Integration
  • Allow define custom extensions to ignore it while story loading process
  • Github Actions
    • Add linting job
    • Allow run ui tests in cli by yarn test:ui

(v0.6.x)

  • Install core-js and regenerator runtime as deps, to fix storybook deps issue, where storybook require own core-js, but module path for bundle resolve incorrect
  • Fix match story and absolute path to file with that story (fix hot-reloading issue in CRA)
    • Add note about storybook version

(v0.7)

  • Add same font as Storybook uses
  • Update Readme
    • Best practices: "use git-lfs in your repo"
    • Default captureElement: #root
    • By default creevey use docker, but you can disable it by specifying gridUrl or override for some browsers
      • About CI, can't simply run docker-in-docker env for now (especially in circle ci because of isolated remote docker. As possible solution use machine executor oblador/loki#183 (comment))
    • before/after hooks
    • creevey addon
    • new params config, don't need decorator any more
    • update using types (export const Slide: Story<React.ComponentProps<typeof ImagesView>> & CreeveyStory = () => ImagesView('slide');)
    • Why need to use findElement({ css: 'selector' }). Improve it in creevey-selenium add null-loader
    • Write about differences with other tools
    • How's using Creevey?
    • Add how to start docker and IE11 especially
    • Add instruction for various frameworks
      • Vue
  • Support Storybook 6.x
  • Add @babel/code-frame to output loader error There is babel issue babel/babel#8617
  • Support declarative 6.0 decorators format, like this https://github.com/storybookjs/storybook/tree/master/addons/knobs/src/preset
  • Store creevey storybook bundle in cache dir using find-cache-dir pkg
  • Move addon ImageViews to shared and use it in client UI
  • Cleanup nodejs storybook bundle (Don't load any of addons)
  • Build addon for ie11
  • Add human readable error message if test failed with window.__CREEVEY_SELECT_STORY__ is not a function
  • Update examples
  • Add creevey in awesome list
  • Creevey as Addon PoC
  • Update Eslint to v7
  • Simplify hot-reloading logic, for v6.x fixed removing tests issue
  • Bugs
    • Mocha worker Possible EventEmitter memory leak detected. 11 error listeners added
    • IPC_CHANNEL_CLOSED error infinity loop, could reproduce with invalid gridUrl
    • Error mocha instance already disposed in [email protected]
    • Tests not removing in hot-reloading process
    • Don't end all worker processes, especially if worker has errors
    • Highlight success/failed screenshot previews
    • Creevey don't work with docs addon (cleanup bundle)
    • export const parameters = {}; in preview.js lead to error Singleton client API not yet initialized, cannot call addParameters
    • Storybook addons override creevey parameters in stories (wait fix from storybook, send PR)
    • Add regenerator runtime to report main.js
    • Don't reset scroll on swap images
    • Don't fail build with mdx stories (just ignore it for now)
    • Re-disable animations on storybook reload
    • Don't stop rebuilding if rebuild failed due syntax error
    • Cut off all exports in preview.js except creevey params
    • Client UI don't show statuses on first run
    • Cut off loaders parameters for stories storybookjs/storybook#12699
  • Features
    • Add args type for CSFStory (Can't support 5.x and 6.x in same time)
    • Support run tests inside docker
    • Allow define saucelabs/browserstack-local init/dispose functions
    • Allow define custom localhost resolver in config (write function for storybookUrl)
    • Output unnecessary images when creevey run from cli
  • Storybook integration
    • Render tests UI as a part of storybook UI
  • Write stories on new components
    • SideBar
    • ResultPageHeader

(v0.7.x)

  • Bugs
    • Add timeout to resolver
    • Incorrect merge skip params (global + local)
    • Fix unnecessary images report for windows
    • Store cache inside creevey package dir (fix core-js versions)
    • Correctly resize images in views using correct proportions (smaller image should shrink if larger shrink too, max-width/max-height doesn't work)
    • Scale images properly (Use naturalWidth image prop for scale in views)
    • Images switch freeze
    • creevey-loader private members
    • Fix teamcity preview images (TeamCity bug)
    • Fix height in addon
    • Fix firefox 61 in skbkontur selenium grid
    • creevey-loader top-level property access
    • wait-on doesn't work properly
    • Don't pull docker local images
    • Remove support ts config file version
    • MDX docsOnly: true
    • Listen to exception before waitForReady
    • 0.7.30 did not compile for us (Cannot find module '@storybook/builder-webpack4'). https://github.com/iTwin/iTwinUI-react
    • Take a look on preview.tsx in react-ui, it isn't transpile
    • Creevey update should update only failed images
    • Teamcity reporter doesn't output failed test with maxRetries > 0
    • E2E tests outputs skip option with duplicated values
    • hot-reloading doesn't work with re-exported stories
    • Angular stop working with babel-loader
    • Creevey update doesn't work on source repo
    • NPM Ctrl+C doesn't exit creevey
    • Creevey addon ui don't work with storybook 5.3 (Drop Storybook 5.x in Creevey 0.8)
    • Make work extract without defining creevey as an addon
    • Run creevey with Vite
  • Add docs addon and mdx stories for old storybooks in e2e tests
  • Add extract stories to e2e tests
  • Change viewport height to 786 in config
  • Save TeamCity config in repo
  • Gitlab browse report fix
  • Add Storybook 6.2 e2e tests
  • Start server early and wait for build
  • Setup TeamCity CI
  • Update CircleCI and GitLab according by github actions
  • Write config description instead of config example
  • Hide some advanced docs in other pages
  • update demo video
  • Rewrite description to more clear one
  • Add ABBYY logo
  • Rework github actions workflows
  • Add CI example in docs
  • Add Storybook integration tests
    • Init projects with various frameworks
    • Init storybook using storybook cli
    • Add creevey config
    • Test webpack building (include bundle size)
    • Test stories tests in output
  • Features
    • Show multiple tests for browser in storybook UI
    • Allow run multiple tests from storybook UI
    • Add waitForReady parameter
    • Allow to ignore elements in capturing screenshot
    • Add link go to runner in addon UI
    • Show side-by-side diff vertically or horizontally depends on aspect
    • Add resolveStorybookUrl to config
    • Support teamcity screenshots diff UI https://www.jetbrains.com/help/teamcity/including-third-party-reports-in-the-build-results.html
    • Save webpack stats.json for debug
    • Send list of available browsers from api
    • Handle main.js for 6.x+, remove addons from it
    • Add extract command for CLI
    • Pass grep option for update command
    • Extract stories.json as a part of build storybook
    • Update data.js right after tests finish even in --ui
    • Add fallback option, load tests from browser (hmr and tests are disabled in this case)
      • Send PR to Storybook to allow use HMR for stories
    • Allow run creevey against static-storybook folder (Depends on fallback tests loading)
    • Implement first iteration of mdx support (support only stories without docs)
      • Move webpack/update to separate folder/file
    • Mdx e2e tests
      • Add tests loader and e2e
  • Fix todos in browser.ts and no-shadow rule
  • Rename webpack to bundler. Move bundler.ts
  • Add debug output for webdriver build and resolve and story switch
  • Improve and approve storybook.examples e2e tests
  • Fix github actions for forked storybook repo
  • Be able to run storybook examples e2e in CI
  • Improve Docker
  • Docs
    • Example link
    • Add vue3 example
    • Add github actions for creevey-examples
    • Ignore elements
    • Describe use cases
      • MDX, animations, CI + docker, custom images (options), sauceLabs, standalone, waitForReady, extract, etc
      • use /** @type {import("webpack").Configuration } */ thing for creevey.config
    • Rewrite config docs to more detail (simple setup, use cases, config description)
    • How to setup creevey report in TeamCity
    • Add new options (selenoidPath, webdriverCommand, etc)
    • Lazy-load components (use https://storybook.js.org/docs/react/writing-stories/loaders)
    • Add best practices for stories
      • Git LFS
      • Avoid write side-effects
      • Don't generate CSF dynamically
      • Do side-effect in separate files (examples)
  • Demo Page
    • Expose official storybook page with creevey
    • Run creevey API somewhere in VPS
    • Support changing args (pass new args values to creevey server -> server sends them to browser in docker)
    • (Optional) On approve save args values for story
    • (Question) How handle multiple users? Github auth? Autoscale docker hosting
      • Limit session time 5-10 min then stop/remove docker container
      • Limit 1 session per IP, reuse same container
      • Start new instance by request from browser
      • Write nodejs proxy app, that starts creevey on random port
      • Use Google cloud k8s

First priority (v0.8)

  • Support es modules
  • Expose new API for captureElement and ComponentStoryFn/Obj with creevey params
  • Wrap creevey capture inside storybook instrumenter
    • Assert images on capture
  • Check storyStoreV7 feature
  • Migrate to Vite
  • Can't use play function in mdx
  • Update don't work with docs? check
  • Remove report on each start
  • Bugs
    • Stories with restricted characters
    • Output error that can't find storybook dir or you forgot to add creevey as an addon
    • Fix taking composite screenshots with hidden scrollbar
      • Don't use scrollBarWidth or hasScrollBar helpers
      • Take document.documentElement.clientWidth/Height instead of window rect
      • For each screenshot after scroll, take elementRect coordinates
      • Iterate by screen images and calculate resulting x/y coordinates for composite image
      • If image width/height greater than viewport width/height then scroll bar is captured
    • Stop stdout from workers after shutdown event (Issue with yarn: yarnpkg/yarn#4667)
  • Test with yarn2
  • Think about how to test with ESM (try to use import() from esm directory)
  • Try https://docs.gitlab.com/runner/executors/docker.html#the-privileged-mode
  • Move docker config options to separate prop docker
  • Move from yarn to npm@7
  • Drop support nodejs 10
  • Download webdriver binary automatically (see bigtest as example)
  • Rename screenDir config option
  • Drop storiesOf and Storybook v5.x support
    • Could we drop more entry points from webpack config? (generated entry for example) (Nope)
  • Fix png logos
  • Support GitLab CI (used services and standalone selenoid)
  • Add edge cases for e2e tests (Add on demand)
  • Figure out if I need use my own react and setup this https://reactjs.org/blog/2020/09/22/introducing-the-new-jsx-transform.html or I need to use react from storybook (Storybook uses optional react deps, so it's better to leave as-is)
  • Support esm/cjs builds
  • Research webpack 5 (Doesn't need after using babel for building)
  • Bring all examples from storybook repo and test with creevey (It's too expensive, e2e tests are pretty enough)
    • Save here images/configs/bundle/stories.json
    • Grab storybook examples from same version as installed here
  • Annotate types for storybook 6.2 (It seems like was needed for webpack config builder)
  • Features
    • CSFv3 + Interactors => PoC run tests in browser
      • Require support static storybook
      • Add config option to switch using play method
      • Check the story events story render and play finish
      • Can we catch play method errors?
      • Add capture command for Creevey websocket server
    • Create/Update stories directly from Storybook (by changing args)
      • Sync Args values with remote browsers
    • update should remove unnecessary images
    • Use native composite screenshots for browsers which support it
    • Get other storybook options, like babel configs from main.js
    • Implement chromatic capture element resolving logic
      • If only one child inside #root node => capture #root > *
      • If more children => capture #root
      • Else capture viewport
    • Capture storybook docs pages (where should be defined creevey parameters?)
      • Listen event Events.DOCS_RENDERED
      • The root element is #docs-root
    • Support JUnit mocha reporter
    • Change skip option API (described somewhere in telegram, like object with keys or other format)
    • Move creevey config inside addon
      • Describe storybook config dir in args
      • How to deal with fallback option?
      • Load addons from storybook api
    • Try to use odiff tool (https://github.com/dmtrKovalenko/odiff)
    • Run extract command as part of storybook building process (Think about storybook built-in extract command)
      • Execute extract with custom bundlers
    • Support Storybook Composition https://storybook.js.org/docs/react/workflows/storybook-composition
    • Support stories.svelte https://storybook.js.org/blog/storybook-for-svelte/
    • Webpack 5 support (Doesn't need after using babel for building)
    • Allow defined params for knobs and args to capture story with different states (Doesn't need with CSFv3)

Second priority (v0.8.x)

  • Add instruction for various frameworks
    • Web components
    • Create React App Typescript
    • Gatsby
    • Next.js
    • https://nx.dev/
  • Bugs
    • SKB Kontur Selenium Grid resolve url timeout
    • IE don't work in github actions maybe out of sync?
    • IE fail because out of sync. Add explicit wait for each browser action
    • We need to define flag when story threw an error. Check flag in selenium and capture screenshot for debug
    • MDX imported stories missed source parameters in creevey (allow to import such stories)
    • ERR! Runtime error! Check your browser console. ERR! ResizeObserver loop limit exceeded (in addon)
    • Creevey nodejs console output This browser doesn't support requestAnimationFrame.
    • webpack config dll references (disable dll plugin)
    • Scrollbar is not visible on dark theme in ResultPage
    • Don't have hot reload on preview config storybook
  • Check latest storybook docs on useful cases
  • Add google analytics (send reports, versions, addons, framework, configs, package.json?)
  • Test standalone selenoid + webdriver work
  • Add more tests on different esnext features (test babel-parser + plugins)
  • Add custom docker images with node+selenoid+browser (We won't need them, if we make creevey-as-a-service image)
  • Features
    • Simplify work with monorepos https://github.com/adiun/vite-monorepo
    • Support other browser automation tools
      • Playwright
      • Puppeteer
    • Improve waitForReady for interaction tests
      • await this.waitForReady(() => this.browser.sendKeys().perform())
      • await this.takeScreenshot()
    • Stop gif animations (investigate)
    • Add status approved, apply after approve and reset after run
    • Output browser logs for debug
    • Support switch between globals #108
    • Merge stories from nodejs bundle and browser, output warning to user if some stories are missing in nodejs
    • Add HTML diff view
    • Add option to apply custom styles to #root or something else
    • Improve creevey-addon webpack config to allow use import { By } from 'selenium' and maybe other stuff (add creevey-selenium or improve creevey-loader)
    • Wait for resources loaded (fonts, images, etc) How?
    • Allow set viewport sizes for story (use width x height as postfix for browser name in UI)
    • Add fuzzy search and highlight
    • Improve creevey-loader
      • Support re-export stories
      • Don't warn user on imported tests
      • Check storiesOf/addDecorators/addParameters import from @storybook
      • Output warnings when somewhere is spread/rest is used
      • Remove unused side-effects from nested scopes
      • Support exclude/include stories parameter
      • Correctly cutoff re-exported stories/parameters

Third priority (v0.9)

  • Rename in skip option to browsers
  • Integrate effection https://github.com/thefrontside/effection
  • Experiment with html2canvas
  • Update Readme
    • How to deal with animations (CREEVEY_ENV)
  • Add bootstrap script, that build and install current version into examples or use monorepo
  • Transform to monorepo
    • creevey
    • chai-images
    • creevey-selenium
    • creevey-docker
    • creevey-storybook
    • examples
  • Features
    • Add test editor inside the addon (user be able to write/change tests for story)
    • Allow save approved screenshots in separate storage, like S3
    • Add API to allow to use third party "stories" resolvers to support not only storybook
    • Allow to extend this.browser API
    • Try AWS Lambda (Think about Creevey-as-a-Service. Deploy Creevey server. And it could be used in gitlab as service)
    • Allow to select elements for capture from storybook UI
    • Allow to ignore elements or rects in storybook UI
    • Support docker-in-docker (start storybook and creevey inside docker)
      • Start storybook inside docker
      • Allow define custom storybook image
    • Improve CLI add grep/kind/story option
    • Setup CREEVEY_ENV (in project use if (CREEVEY_ENV) {} and addon define function that check if it inside creevey or not)
    • Allow pass components into findElement
      • The idea is, add some transformation to creevey-addon, and replace it component to [data-comp-name~="MyComponent"]
      • Then need to investigate how to inject such data-attributes to html node (especially for non-react frameworks)
    • Add init cli option
    • Easy way to ignore stories/kinds from UI
    • Allow to restart tests on story changes
    • Add only option as opposite for skip
    • Creevey-as-a-Service
  • Improve Docker
    • Add vnc
    • Add video recording
  • Bugs
    • Fix flex-shrink in side-by-side view not work with disabled cache

Not in first time

  • Correctly reload and reset tests statuses according source code file dependencies
  • Improve css filter for blend view, try to reach pixelmatch output
  • Profile tests loading process (maybe we don't need workers at all)
  • Always save images even if test with matchImages failed
  • Add liftoff https://github.com/js-cli/js-liftoff
  • Tests on images view components with various scenarios (same/diff sizes, less/bigger viewport, elements with width/height not integer size)
  • Bugs
    • Firefox double click if clicks in different tests
  • Improve CLI
    • Output cli help
  • Rework UI
    • Show removed tests results, mark these as removed
    • Allow hide skipped tests in UI
    • Allow switch between 1:1 and fit image views
  • Add unit tests
  • Rewrite to use worker_threads instead of cluster to allow use shared memory
  • Support mocha options for workers
  • Programmic API
  • Add logger lib

Maybe Never

  • vscode mocha explorer
    • codelens run not work (need full path)
    • tests run with default timeout even if it changed in config
    • cwd and require conflict each other
    • mocha opts and mocha bin is not prefect support
  • Use own runner instead of mocha
  • Allow use creevey without storybook
  • Support third-party test runners

The new beginning

  • Use odiff

  • Kontur creevey config

  • Add similar to playwright tracing

  • Show instructions for git-lfs

  • Replace mocha to manual runner (or another variant)

  • Remove chai and provide expect specially for images (maybe another API)

  • Try pkgroll or use storybook addon-kit template

  • Add shortcuts for approve/view switch/image switch