Sequence trouble

[bendelonlee]

Thanks for creating factory bot cypress!

The Problem

Noticing some unexpected behavior for sequences in a test scenario.

create_list works as expected:

items = factory.create_list(:item, 2)
items[0].name 
# => test name 1
items[1].name 
# => test name 2

This does not:

item1 = factory.create(:item)
item2 = factory.create(:item)
item1.name
# => test name 1
item2.name
# => test name 1

It can lead to a lot of boilerplate test setup needing to be added when dealing with items with associations that have uniqueness validators.

My config

My cypress-on-rails version is 1.11.0 (latest).
My factory-bot version is 6.2.

My pretty standard factory_bot.rb

Array.wrap(command_options).map do |factory_options|
  factory_method = factory_options.shift
  begin
    logger.debug "running #{factory_method}, #{factory_options}"
    CypressOnRails::SmartFactoryWrapper.public_send(factory_method, *factory_options)
  rescue => e
    logger.error "#{e.class}: #{e.message}"
    logger.error e.backtrace.join("\n")
    logger.error e.record.inspect.to_s if e.is_a?(ActiveRecord::RecordInvalid)
    raise e
  end
end

My cypress_helper.rb

begin
  require "database_cleaner-active_record"
rescue LoadError => e
  puts e.message
  begin
    require "database_cleaner"
  rescue LoadError => e
    puts e.message
  end
end

begin
  require "factory_bot_rails"
rescue LoadError => e
  puts e.message
end

require "cypress_on_rails/smart_factory_wrapper"

factory = CypressOnRails::SimpleRailsFactory
factory = FactoryBot if defined?(FactoryBot)

CypressOnRails::SmartFactoryWrapper.configure(
  always_reload: !Rails.configuration.cache_classes,
  factory: factory,
  files: [
    Rails.root.join("test", "factories.rb"),
    Rails.root.join("test", "factories", "**", "*.rb"),
  ]
)

My clean.rb

if defined?(DatabaseCleaner)
  DatabaseCleaner.strategy = :truncation
  DatabaseCleaner.clean

  load Rails.root.join("db", "bootstrap.rb")
else
  logger.warn "add database_cleaner or update cypress/app_commands/clean.rb"
end

Rails.logger.info "APPCLEANED" # used by log_fail.rb

Hack-y solution

I can get desired behavior by adding this to the top of the scenario file:

factory.instance.always_reload = false
factory.instance.factory::Internal.sequences.clear

and this to the bottom:

factory.instance.always_reload = true
factory.instance.factory::Internal.sequences.clear

What hasn't worked

Leaving out factory.instance.factory::Internal.sequences.clear makes the sequence not reset between test runs.
I tried adding

CypressOnRails::SmartFactoryWrapper.instance.factory::Internal.sequences.clear

to clean.rb but it didn't have any effect.

Related issue:

#66 was closed, with a fix.

[grantspeelman]

Yes, I have the same experience when using sequences. Currently we prefer to something like https://github.com/faker-ruby/faker for our factories instead of sequences.

Maybe what you could try is in your cypress_helper.rb change

  always_reload: !Rails.configuration.cache_classes,

to

  always_reload: false

then in your clean.rb add

FactoryBot.reload

Let me know if this gives you the desired results. If it does, maybe I'll see about tweaking the code to make this the default.

[bendelonlee]

Thanks for the quick response!

if defined?(DatabaseCleaner)
# tried putting it here as well
  DatabaseCleaner.strategy = :truncation
  DatabaseCleaner.clean
# and here

  load Rails.root.join("db", "bootstrap.rb")
else
  logger.warn "add database_cleaner or update cypress/app_commands/clean.rb"
end

FactoryBot.reload

Rails.logger.info "APPCLEANED" # used by log_fail.rb

When I put FactoryBot.reload in clean.rb, unfortunately I get an error after the second test run:

Factory not registered: "item"

[grantspeelman]

can you please give the factory-bot-sequences #95 branch a try.

in your Gemfile

gem 'cypress-on-rails', '~> 1.0', github: 'shakacode/cypress-on-rails', branch: 'factory-bot-sequences'

then in your clean.rb instead of FactoryBot.reload use CypressOnRails::SmartFactoryWrapper.reload

[bendelonlee]

I tried it and it works exactly as expected!

[grantspeelman]

That's great news. I'll get #95 merged 👍🏽 .
Thank you for your help, i might even start using sequences now again 😄

[bendelonlee]

Sorry to report, the fix is flaky in a weird way.

When it works it works repeatedly, ie hitting

will consistently lead to success.

But then I'll edit either the test, the setup, or the code under test and will get an error like

Validation failed: Item name Item 1 already exists

indicating factorybot is resetting the sequences between calls rather than between tests again. Hitting rerun will consistently get this error.

Editing whitespace (or unrelated code like console logs) in test, setup, code eventually tends to make the error go away.

Wasn't able to isolate exactly what kinds of changes make the problem appear or disappear.

[grantspeelman]

@bendelonlee can you check the log file to see how the factories are getting reloaded.
You should see "Loading Factories" each time the factories are reloaded.

I'll investigate some more in the meantime.

[grantspeelman]

can you please give the reload-tweaks #98 branch a try.

in your Gemfile

gem 'cypress-on-rails', '~> 1.0', github: 'shakacode/cypress-on-rails', branch: 'reload-tweaks'

[grantspeelman]

I have merged the branch and released a new version. Hopefully this fixes the issue for you 👍🏽

[bendelonlee]

👍

I haven't had time to thoroughly test whether the flakiness is gone. Takes time because I wasn't able to reproduce reliably. Thanks for the fix! I'll let you know either way

[bendelonlee]

Unfortunately I had to remove sequence usage from tests because it wasn't resetting the sequence between tests on CI (though worked locally) and haven't had time to do much further investigation.