WIP: On-device tests #146

t184256 · 2021-12-04T13:53:18Z

I've decided to quick-and-dirty automate a few things I often check manually on PRs and releases inside a disposable installation.

@Gerschtli thoughts? Other high-profile scenarios I'm missing?

Gerschtli

I love it! Thanks a lot, the first step for test automation :) Had some comments, but beware, I did not run/test any of my suggestions, so there may be some typos.

I do not see any big thing missing, but I think at some point we should have a test for every option, but that doesn't need to be a on-device test.

nix-on-droid/nix-on-droid.sh

Gerschtli · 2021-12-04T18:42:48Z

tests/on-device/.run.sh

+    cp -r ~/.config.bak ~/.config
+fi
+
+nix-shell -p bats --run 'bats .'


Scripts should depend on the working directory from where they are called.

With dirname "$(readlink -f "$0")", you get the directory where your script is, so

# somewhere at the top SCRIPT_DIR="$(dirname "$(readlink -f "$0")")" # then here nix-shell -p bats --run "bats ${SCRIPT_DIR}"

Incorporated.

Gerschtli · 2021-12-04T18:44:43Z

tests/on-device/.run.sh

+if [[ ! -d ~/.config.bak ]]; then
+    mv ~/.config ~/.config.bak
+    cp -r ~/.config.bak ~/.config
+fi


Wouldn't be backupping the ~/.config/nixpkgs directory be enough? That would reduce possible unwanted side effects.

tests/on-device/.run.sh

tests/on-device/config-flake.bats

Gerschtli · 2021-12-04T18:57:50Z

tests/on-device/hello.bats

+  out=$(nix-shell -p hello --run hello)
+  [[ "$out" == 'Hello, world!' ]]


A bit more idiomatic :)

Suggested change

out=$(nix-shell -p hello --run hello)

[[ "$out" == 'Hello, world!' ]]

run nix-shell -p hello --run hello

[[ "$output" == 'Hello, world!' ]]

doesn't check the retcode that way

Ah okay, then maybe be more explicit with what you want to test with [[ $status -eq 0 ]].

run also collects stderr into "$output", nah

If you want to have stdout and stderr separate, then you need --separate-stderr

Additionally, you can use --separate-stderr to split stdout and stderr into $output/$stderr and ${lines[@]}/${stderr_lines[@]}.

https://bats-core.readthedocs.io/en/stable/writing-tests.html#run-test-other-commands

Gerschtli · 2021-12-04T18:59:08Z

tests/on-device/lib.sh

@@ -0,0 +1,18 @@
+switch_to_default_config() {
+  rm -rf ~/.config/nix


This shouldn't be needed, if you always you the cli args for nix

I tried to follow what we recommend in the README, not what's easier to script. But I'm having second thoughts, yeah

Gerschtli · 2021-12-04T18:59:48Z

tests/on-device/lib.sh

+  CHANNEL_DIR=$HOME/.cache/nix-on-droid-self-test/channel
+  cp $CHANNEL_DIR/modules/environment/login/nix-on-droid.nix.default \
+    ~/.config/nixpkgs/nix-on-droid.nix


Suggested change

CHANNEL_DIR=$HOME/.cache/nix-on-droid-self-test/channel

cp $CHANNEL_DIR/modules/environment/login/nix-on-droid.nix.default \

~/.config/nixpkgs/nix-on-droid.nix

CHANNEL_DIR=$HOME/.cache/nix-on-droid-self-test/channel

cp "$(nix-instantiate --eval --expr "<nix-on-droid/modules/environment/login/nix-on-droid.nix.default>")" \

~/.config/nixpkgs/nix-on-droid.nix

Gerschtli · 2021-12-04T19:03:42Z

tests/on-device/config-h-m.nix

+    { pkgs, lib, ... }:
+    {
+      home.stateVersion = "21.05";
+      nixpkgs.overlays = lib.mkIf (lib.hasAttrByPath [ "nixpkgs" "overlays" ] config) config.nixpkgs.overlays;


Why not enable home-manager.useGlobalPkgs instead of this monstrosity? I am still not sure if that complexity is really necessary for setting just the overlay :D

because that's what we recommend to users

Shouldn't be needed for 21.11 anymore

tests/on-device/gnumake.bats

t184256 · 2021-12-10T08:47:02Z

Not finished yet, will need more time with this.

Gerschtli · 2021-12-10T19:15:25Z

tests/on-device/.run.sh

+    cp -r ~/.config.bak ~/.config
+fi
+
+pushd "$SCRIPT_DIR"  # for lib.sh import


You can replace the pushd stuff with this nice feature to load shared code: https://bats-core.readthedocs.io/en/stable/writing-tests.html#load-share-common-code

tests/on-device/gnumake.bats

t184256 · 2021-12-10T22:42:13Z

added config-flake-h-m test, it currently fails with attribute nixpkgs missing

Gerschtli · 2021-12-10T22:52:01Z

Sorry, my phone is really slow today. So can you test for me if the tests would fail with the following changes?

diff --git i/tests/on-device/config-flake-h-m.cfg.nix w/tests/on-device/config-flake-h-m.cfg.nix
index 2df66f5..9c0310b 100644
--- i/tests/on-device/config-flake-h-m.cfg.nix
+++ w/tests/on-device/config-flake-h-m.cfg.nix
@@ -8,12 +8,5 @@
   # no nixpkgs.overlays defined
   environment.packages = with pkgs; [ zsh ];
 
-  home-manager.config =
-    { pkgs, ... }:
-    {
-      home.stateVersion = "21.05";
-
-      nixpkgs.overlays = config.nixpkgs.overlays;
-      home.packages = with pkgs; [ dash ];
-    };
+  nixpkgs.overlays = [];
 }

Because I am not sure, if the problem is in nix-on-droid's or home-manager's nixpkgs definition..

t184256 · 2021-12-10T23:13:15Z

(after also commenting out the now-missing experimentalFeatures) In a flake setup, the options nixpkgs.* should be used...

Gerschtli · 2021-12-10T23:32:45Z

That should not print this message for nixpkgs.overlays = [];.. But this is at least a different error than with home-manager. I checked their source code and they always define the option definitions, so I really do not have a clue. I am trying to reproduce it on my phone..

Gerschtli · 2021-12-10T23:36:13Z

I totally messed up the condition of the assertion.. :D I will push it to my PR in a second.

Gerschtli · 2021-12-10T23:59:15Z

tests/on-device/.run.sh

+    cp -r ~/.config.bak ~/.config
+fi
+
+bats "${SCRIPT_DIR}" --verbose-run -x -T


These options throw an error on my phone..

Gerschtli · 2021-12-10T23:59:45Z

tests/on-device/config-default.bats

@@ -0,0 +1,7 @@
+# Copyright (c) 2019-2021, see AUTHORS. Licensed under MIT License, see LICENSE.
+
+load lib.sh


I think you need to write load lib and rename lib.sh to lib.bash

Gerschtli · 2021-12-11T00:18:29Z

The tests just seem to hang when I run them.. Only the lines

1..6
ok 1 default config can be switched into

t184256 · 2021-12-11T00:20:26Z

Yeah, flake ones take a lot of time to run... I'm actually working on an environment to debug them on PC, hope this makes sense in the end.

Gerschtli · 2021-12-11T12:57:45Z

So my testing branch (based of latest release-21.11 (#147) + cherry-picked commits from this PR + 7a1f8eb) runs perfectly fine :)

So if you incorporate 7a1f8eb into this PR, from my point of view, #147 and this PR can be merged :)

t184256 · 2021-12-11T13:06:27Z

Awesome! Now I just must somehow contain the testing scope creep...

Gerschtli · 2021-12-11T13:41:17Z

Oh boy, that script looks complicated :D

Gerschtli · 2021-12-12T00:21:58Z

While using your tests for my flake PR just as a quick heads up before I lose the thought: If you are changing nix-on-droid script itself you need to do a manual switch and run the test and do manual rollbacks in case of failures. Maybe (not necessarily in this PR) we find a good solution for improving UX for that case :)

t184256 changed the title ~~On-device tests~~ WIP: On-device tests Dec 4, 2021

Gerschtli reviewed Dec 4, 2021

View reviewed changes

t184256 force-pushed the on-device-test branch from 32a2ca9 to 6afab1b Compare December 10, 2021 00:17

Gerschtli reviewed Dec 10, 2021

View reviewed changes

tests/on-device/gnumake.bats Show resolved Hide resolved

t184256 force-pushed the on-device-test branch 2 times, most recently from 7324bd7 to f607198 Compare December 10, 2021 22:41

Gerschtli reviewed Dec 10, 2021

View reviewed changes

t184256 added 2 commits December 12, 2021 02:17

modules/environment/login/login-inner: correct greeting

72c0114

tests/on-device: add

8510c23

t184256 force-pushed the on-device-test branch from f607198 to 8510c23 Compare December 12, 2021 10:55

t184256 merged commit 343ec2c into master Dec 12, 2021

t184256 deleted the on-device-test branch December 15, 2021 21:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: On-device tests #146

WIP: On-device tests #146

t184256 commented Dec 4, 2021

Gerschtli left a comment

Gerschtli Dec 4, 2021

t184256 Dec 10, 2021

Gerschtli Dec 4, 2021

Gerschtli Dec 4, 2021

t184256 Dec 9, 2021

Gerschtli Dec 9, 2021

t184256 Dec 10, 2021

Gerschtli Dec 10, 2021

Gerschtli Dec 4, 2021

t184256 Dec 9, 2021

Gerschtli Dec 4, 2021

Gerschtli Dec 4, 2021

t184256 Dec 9, 2021

Gerschtli Dec 9, 2021

t184256 commented Dec 10, 2021

Gerschtli Dec 10, 2021

t184256 commented Dec 10, 2021

Gerschtli commented Dec 10, 2021 •

edited

Loading

t184256 commented Dec 10, 2021

Gerschtli commented Dec 10, 2021

Gerschtli commented Dec 10, 2021

Gerschtli Dec 10, 2021

Gerschtli Dec 10, 2021

Gerschtli commented Dec 11, 2021

t184256 commented Dec 11, 2021

Gerschtli commented Dec 11, 2021

t184256 commented Dec 11, 2021

Gerschtli commented Dec 11, 2021 •

edited

Loading

Gerschtli commented Dec 12, 2021

		out=$(nix-shell -p hello --run hello)
		[[ "$out" == 'Hello, world!' ]]

		@@ -0,0 +1,18 @@
		switch_to_default_config() {
		rm -rf ~/.config/nix

		@@ -0,0 +1,7 @@
		# Copyright (c) 2019-2021, see AUTHORS. Licensed under MIT License, see LICENSE.

		load lib.sh

WIP: On-device tests #146

WIP: On-device tests #146

Conversation

t184256 commented Dec 4, 2021

Gerschtli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

t184256 commented Dec 10, 2021

Choose a reason for hiding this comment

t184256 commented Dec 10, 2021

Gerschtli commented Dec 10, 2021 • edited Loading

t184256 commented Dec 10, 2021

Gerschtli commented Dec 10, 2021

Gerschtli commented Dec 10, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Gerschtli commented Dec 11, 2021

t184256 commented Dec 11, 2021

Gerschtli commented Dec 11, 2021

t184256 commented Dec 11, 2021

Gerschtli commented Dec 11, 2021 • edited Loading

Gerschtli commented Dec 12, 2021

Gerschtli commented Dec 10, 2021 •

edited

Loading

Gerschtli commented Dec 11, 2021 •

edited

Loading