Support custom path for installed extensions #1940

motorina0 · 2023-09-13T08:03:26Z

Summary

Long discussion here: #1579

By default extensions are installed in a hard-coded directory: lnbits/extensions.
This PR allow users to specify a custom directory where the extensions will be installed by specifying an environment variable.

The reason for having a custom extension path is that some users do not want to perform write operations in their lnbits directory (maybe they are using Nix which installs programs to immutable locations).

Required changes for extensions:

this PR is backwards compatible. If the custom extensions path is not used then there is no need for extensions to be updated
if the custom extensions path is used then extensions must be updated (see comment below)

Config Steps:

update the .env file by configuring LNBITS_EXTENSIONS_PATH. This location must allow WRITE operations.

LNBITS_EXTENSIONS_PATH="/Users/moto/Documents/GitHub/motorina0/lnbits/data/code"

installed extensions MUST be updated in order to support this custom path feature

Testing

Backwards compatible

test that the PR is backwards compatible:
- do not use the LNBITS_EXTENSIONS_PATH
- install/uninstal/upgrade extensions

Extension Install

set LNBITS_EXTENSIONS_PATH to a valid path
make sure the extension to be installed has been updated (see comment below)
install extension
check that the extension works as expected
upgrade extension and check that the new features are present
check that the lnbits/extensions directory is empty/does not exist
check that the LNBITS_EXTENSIONS_PATH has an extensions folder that contains the installed extension
install older version of the extension and check that the changes are rolled back

Test data:

manifest file: https://raw.githubusercontent.com/motorina0/watchonly/main/manifest.json
there are two releases:
- v0.1-custom-path-ready - contains the updated paths for the extension
- v0.2-ui-api-updates - contains code changes for the extension (see screenshot 1)

Screenshots

1: UI and API updates

motorina0 · 2023-09-13T10:19:15Z

Required Extension Changes

Extension changes in order to be compatible:

the below example is for tpos (can be adapted for any extension)
in the config.json file set the min LNbits version to the version that will have this feature:

{
 "min_lnbits_version": "xx.yy",
 ...
}

in the tpos/__init__.py file:

Change 1:

Remove the "app" field from tpos_static_files

From:

tpos_static_files = [
    {
        "path": "/tpos/static",
        "app": StaticFiles(directory="lnbits/extensions/tpos/static"),
        "name": "tpos_static",
    }
]

To:

tpos_static_files = [
    {
        "path": "/tpos/static",
        "name": "tpos_static",
    }
]

Change 2:

Make the template_renderer path relative by removing the lnbits/extensions prefix:

From:

def tpos_renderer():
    return template_renderer(["lnbits/extensions/tpos/templates"])

To:

def tpos_renderer():
    return template_renderer(["tpos/templates"])

codecov · 2023-09-13T11:49:39Z

Codecov Report

Merging #1940 (3dd91b8) into dev (1646b08) will increase coverage by 0.24%.
Report is 2 commits behind head on dev.
The diff coverage is 27.08%.

@@            Coverage Diff             @@
##              dev    #1940      +/-   ##
==========================================
+ Coverage   59.56%   59.80%   +0.24%     
==========================================
  Files          50       50              
  Lines        7500     7531      +31     
==========================================
+ Hits         4467     4504      +37     
+ Misses       3033     3027       -6

Files Changed	Coverage Δ
lnbits/helpers.py	`59.49% <0.00%> (-0.77%)`	⬇️
lnbits/server.py	`38.63% <0.00%> (ø)`
lnbits/extension_manager.py	`53.11% <14.28%> (-1.54%)`	⬇️
lnbits/app.py	`66.55% <35.71%> (-1.06%)`	⬇️
lnbits/settings.py	`89.47% <100.00%> (+0.18%)`	⬆️

... and 3 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

motorina0 · 2023-09-18T08:10:11Z

Comparability table for "custom extension path support":

LNbits/Extension	Ext No	Ext Yes
LNbits No	OK	NOK
LNbits Yes	OK	OK

callebtc · 2023-09-19T09:55:08Z

lnbits/app.py

+def register_custom_extensions_path():
+    if settings.has_default_extension_path:
+        return
+    default_ext_path = "lnbits/extensions"


should use os.path.join() instead of a string concatenation.

callebtc · 2023-09-19T09:55:20Z

lnbits/app.py

+        )
+        logger.warning(
+            f"You can move the existing '{default_ext_path}' directory to: "
+            + f" '{settings.lnbits_extensions_path}/extensions'"


should use os.path.join() instead of a string concatenation.

its just a log message here

callebtc · 2023-09-19T09:55:57Z

lnbits/app.py

+    sys.path.append(str(Path(settings.lnbits_extensions_path, "extensions")))
+    sys.path.append(str(Path(settings.lnbits_extensions_path, "upgrades")))


again os.path.join would help

what is wrong with Path?

i prefer Path

callebtc · 2023-09-19T09:57:34Z

lnbits/extension_manager.py

+        )
+
+        tmp_dir = Path(settings.lnbits_data_folder, "unzip-temp", self.hash)
+        shutil.rmtree(tmp_dir, True)


I think this one can error (for example insufficient permissions), consider try: except:

in that case it is OK for the exception to be propagated (should not be hidden)

callebtc · 2023-09-19T09:57:50Z

lnbits/extension_manager.py

+        shutil.rmtree(self.ext_upgrade_dir, True)
+        shutil.copytree(
+            Path(tmp_dir, generated_dir_name),
+            Path(self.ext_upgrade_dir),
        )
+        shutil.rmtree(tmp_dir, True)


Same here, all this can error.

let it fail ❄️ , let it fail ❄️ , let it fail ❄️ ☃️ 🏔️

callebtc · 2023-09-19T09:58:10Z

lnbits/extension_manager.py

-        with open(
-            Path(self.ext_upgrade_dir, self.id, "config.json"), "r+"
-        ) as json_file:
+        with open(Path(self.ext_upgrade_dir, "config.json"), "r+") as json_file:


Should you check for existence of this file before opening?

if the file does not exist then open will throw

this is fine extract_archive should fail in this case

callebtc · 2023-09-19T09:58:49Z

lnbits/settings.py

+    @property
+    def has_default_extension_path(self) -> bool:
+        return self.lnbits_extensions_path == "./lnbits"


needs os.path.join

dni

LGTM! awesome

prusnak · 2023-09-20T08:59:19Z

lnbits/helpers.py

@@ -30,6 +30,10 @@ def url_for(endpoint: str, external: Optional[bool] = False, **params: Any) -> s
 def template_renderer(additional_folders: Optional[List] = None) -> Jinja2Templates:
    folders = ["lnbits/templates", "lnbits/core/templates"]
    if additional_folders:
+        additional_folders += [
+            Path(settings.lnbits_extensions_path, "extensions", f)


Maybe we want to perform this hack to allow compatibility with old extensions?

But I agree it is just better to perform a clean cut.

Suggested change

Path(settings.lnbits_extensions_path, "extensions", f)

Path(settings.lnbits_extensions_path, "extensions", f[18:] if f.startswith("lnbits/extensions/" else f)

callebtc

LGTM

* [FIX] correct amount in fiat tracking (#1934) * better debug log * fix: convert msat to sat for fiat amount * [CHORE] update to version 0.11.0 (#1933) * [FIX] codeql for dev branch (#1935) * [FEAT] create data folder if it doesnt exist (#1930) * [FEAT] improve update_admin_settings (#1903) * [FEAT] improve update_admin_settings while working on the push notification pr i found it very hard just to update 2 settings inside the db, so i improved upon update_admin_settings. now you just need to provide a dict with key/values you want to update inside db. also debugging the endpoints for update_settings i found despite the type of `EditableSettings` fastapi did in fact pass a dict. * t * use `EditableSettings` as param in update_settings * fix settings model validation we previously overrode the pydantic validation with our own method * make `LnbitsSettings` a `BaseModel` and only add `BaseSettings` later this allows us to instantiate `EditableSettings` without the environment values being loaded in * add test * forbid extra fields in update api * fixup * add test * test datadir * move UpdateSettings * fix compat * fixup webpush --------- Co-authored-by: jacksn <[email protected]> * [TEST] use clean db in postgres tests (#1928) * [CORE] Document CoreLightningRestWallet (#1920) * document CoreLightningRestWallet * pushed to comment * fixed base64 comment --------- Co-authored-by: Ben <[email protected]> * [TEST] use test data as mockdata (#1929) * add db group to cli * delete mock_data.zip * generate migration data through tests * [BUG] self payments for the same wallet (#1914) * [BUG] self payments for fakewallet make it possible to pay to yourself on fakewallet f * bugfix selfpayments for fakewallet * delete by wallet and fix previous_payment check (#1919) --------- Co-authored-by: callebtc <[email protected]> * [REFACTOR] core/__init__ to not have circular import issues (#1876) * F541 fix remove unused workflow and combine linters into one add lnbits/static to ruff ignore remote setupnode ignore upgrades for mypy ignore F401 for __init__ files unused noqa ignore upgrades for black F821: undefine name disabled and logged webhook_listener for opennode and lnpay because they are obvisouly not working E402: module level import not at top of file fixup revert breaking changes wait for PR #1876 #1876 E721 fixes, only popped up for python3.9 not 3.10 [REFACTOR] core/__init__ to not have circular import issues WIP add db for backwards compat fix pyright make mypy happy again pyright did not catch those, i think mypy got confused with relative imports. maybe we should use absolute ones everywhere E402: module level import not at top of file dont forget to add core_app rebase on ruff pr f remo format * fix clnrest * ignore E402 in conftest * refactoring issues --------- Co-authored-by: jacksn <[email protected]> * make wallets name prettier! (#1918) * make wallets name prettier! * Update lnbits/core/templates/admin/index.html Co-authored-by: dni ⚡ <[email protected]> * Update lnbits/core/templates/admin/index.html Co-authored-by: dni ⚡ <[email protected]> --------- Co-authored-by: Arc <[email protected]> Co-authored-by: dni ⚡ <[email protected]> * Fix payments chart (#1851) * feat: payment history endpoint * test payment history * use new endpoint in frontend * refactor tests * [CHORE] bugfix formatting prettier (#1936) * do not show deleted wallets (#1942) * do not show deleted wallets * rename parameter for clarity * extension loader: do not panic! (#1945) * Revert "do not show deleted wallets (#1942)" (#1947) This reverts commit 1ee2d53. * remove LNURL-p description_hash validation (#1951) * [CHORE] update dependencies, unsafe pip packages, fastapi (#1609) * d * middleware needs to be initialised before startup runs on python3.11 update secp256k1 update psycopg * update fastapi to v0.103 * fix webpush * bump dependencies * ruff issue * lock versions * bump versions --------- Co-authored-by: jacksn <[email protected]> * [FEAT] Focus cursor to textarea (#1959) * Focus cursor to textarea Fix need for additional click to textarea before pasting * [FEAT] autofocus textarea on paste request closes #1959 * format arba * bundle --------- Co-authored-by: dni ⚡ <[email protected]> * add editorconfig config, fix issues (only whitespace changes) (#1958) * adding bolt11 lib and removing bolt11.py from the codebase (#1817) * add latest bolt11 lib decode exception handling for create_payment fix json response for decode bugfix hexing description hash improvement on bolt11 lib update to bolt11 2.0.1 fix clnrest * bolt 2.0.4 * refactor core/crud.py * catch bolt11 erxception clnrest * Updated guide for extension development for current extension management process (#1962) * Startup optimization: nonblocking expiry check (#1943) * nonblocking expiry check * autoruff * smaller interval --------- Co-authored-by: dni ⚡ <[email protected]> * Support custom path for installed extensions (#1940) * feat: the more you fuck around the more you learn * feat: backwards compatible static file loading * refactor: update paths for extension files * refactor: var renaming * doc: update `LNBITS_EXTENSIONS_PATH` documentation * fix: default folder install * feat: install ext without external path * doc: `PYTHONPATH` no longer required * fix: add warnings * fix: missing path * refactor: re-order statements * fix: hardcoded path separator --------- Co-authored-by: dni ⚡ <[email protected]> * [FEAT] Node Managment (#1895) * [FEAT] Node Managment feat: node dashboard channels and transactions fix: update channel variables better types refactor ui add onchain balances and backend_name mock values for fake wallet remove app tab start implementing peers and channel management peer and channel management implement channel closing add channel states, better errors seperate payments and invoices on transactions tab display total channel balance feat: optional public page feat: show node address fix: port conversion feat: details dialog on transactions fix: peer info without alias fix: rename channel balances small improvements to channels tab feat: pagination on transactions tab test caching transactions refactor: move WALLET into wallets module fix: backwards compatibility refactor: move get_node_class to nodes modules post merge bundle fundle feat: disconnect peer feat: initial lnd support only use filtered channels for total balance adjust closing logic add basic node tests add setting for disabling transactions tab revert unnecessary changes add tests for invoices and payments improve payment and invoice implementations the previously used invoice fixture has a session scope, but a new invoice is required tests and bug fixes for channels api use query instead of body in channel delete delete requests should generally not use a body take node id through path instead of body for delete endpoint add peer management tests more tests for errors improve error handling rename id and pubkey to peer_id for consistency remove dead code fix http status codes make cache keys safer cache node public info comments for node settings rename node prop in frontend adjust tests to new status codes cln: use amount_msat instead of value for onchain balance turn transactions tab off by default enable transactions in tests only allow super user to create or delete fix prop name in admin navbar --------- Co-authored-by: jacksn <[email protected]> * [FEAT] cleanup GET /wallet endpoint, add wallet api routes (#1932) * [FEAT] cleanup GET /wallet endpoint, add wallet api route this removes the functionalitiy to create accounts and wallets via the GET /wallet endpoint in generic.py it add endpoints inside the api.py for it and the frontend is modified to use the api endpoints this also simplifies for the `feat/login` for the route. * remove stale generic tests and add api tests * bug wrong endpoint create account * vlad nitpick * added checkif deleted is 404 * reload after renaming wallet * another iteration with vlad * create new wallet if it none exist * fix delete refresh * formatting * Don't create user if allowed users is set (#1822) * dont allow creating of users when allowed users are set * add info to .env.example --------- Co-authored-by: dni ⚡ <[email protected]> * fix typos (#1969) * fix: node ranks (#1968) * [BUG] bundle is broken (#1972) * [BUG] Fix bad transation call (#1973) * fix: `:label="$('xxx')` to :label="$t('xxx') * chore: commit bundles * chore: remove extra file * fix: catch sse exceptions (#1971) * Add Markdown capability globally (#1965) * add markdown capability globally * add markdown to site description * add showdown to package.json remove it from vendor.json, its bundled * formatting * io * Update lnbits/core/templates/core/index.html --------- Co-authored-by: dni ⚡ <[email protected]> * feat: remove commit version (#1980) * nix: fix nixosModule and build (#1979) * lnbits/settings: get LNBITS_COMMIT from envvar if available There is no other way of setting the git commit at runtime, for build systems like Nix that allow you to hermetically define envvars derived from the source that will be available reproducibly at runtime * nix: update potree2nix and nixosModule This is a general refactor that gets everything building and passing the vmTest again * Update nix/modules/lnbits-service.nix --------- Co-authored-by: Pavol Rusnak <[email protected]> * catch exceptions in migrations for extensions (#1987) * fix: admin_ui js error (#1982) * fix: admin_ui js error undefined formData.lnbits_allowed_currencies JS error in console * vlad suggestion * update: to 0.11.1 * revert: update --------- Co-authored-by: jackstar12 <[email protected]> Co-authored-by: dni ⚡ <[email protected]> Co-authored-by: jacksn <[email protected]> Co-authored-by: Ben <[email protected]> Co-authored-by: Tiago Vasconcelos <[email protected]> Co-authored-by: Arc <[email protected]> Co-authored-by: arbadacarba <[email protected]> Co-authored-by: Pavol Rusnak <[email protected]> Co-authored-by: blackcoffeexbt <[email protected]> Co-authored-by: Vlad Stan <[email protected]> Co-authored-by: Matthew Croughan <[email protected]>

* feat: the more you fuck around the more you learn * feat: backwards compatible static file loading * refactor: update paths for extension files * refactor: var renaming * doc: update `LNBITS_EXTENSIONS_PATH` documentation * fix: default folder install * feat: install ext without external path * doc: `PYTHONPATH` no longer required * fix: add warnings * fix: missing path * refactor: re-order statements * fix: hardcoded path separator --------- Co-authored-by: dni ⚡ <[email protected]>

This reverts commit 6835e9e.

motorina0 added 4 commits September 5, 2023 17:08

chore: experiment

d4e9bfb

chore: experiment

7f0d0ec

feat: the more you fuck around the more you learn

beef12d

chore: code format

b6701b4

motorina0 added 9 commits September 13, 2023 13:32

chore: some code clean-up

91ccd41

feat: backwards compatible static file loading

b5cb1fe

chore: code clean-up

2d277bb

refactor: update paths for extension files

0b38a9e

fix: static files path

01fd045

chore: code clean-up

bf3ac3d

fix: keep old paths too

33ac830

chore: revert test change

69ac96f

refactor: var renaming

e457494

motorina0 added 11 commits September 13, 2023 15:25

doc: update LNBITS_EXTENSIONS_PATH documentation

1f90a52

fix: default folder install

2b7b0d4

refactor: rename property

c3cedad

feat: install ext without external path

b125434

doc: PYTHONPATH no longer required

c8bc411

chore: code clean-up

84a6a60

fix: add warnings

4b571f2

chore: fix typo

592a1fb

fix: missing path

9389968

refactor: re-order statements

d1e699f

doc: typo

2023c32

motorina0 marked this pull request as ready for review September 14, 2023 10:41

motorina0 requested review from prusnak, dni and callebtc September 14, 2023 10:41

This was referenced Sep 17, 2023

allow custom path lnbits/lnaddress#3

Merged

allow custom path lnbits/watchonly#5

Merged

allow custom path lnbits/nostrclient#20

Merged

allow custom path lnbits/nostrrelay#17

Merged

allow custom path lnbits/nostrmarket#83

Merged

callebtc reviewed Sep 19, 2023

View reviewed changes

talvasconcelos approved these changes Sep 19, 2023

View reviewed changes

callebtc requested changes Sep 19, 2023

View reviewed changes

fix: hardcoded path separator

ca01a37

dni approved these changes Sep 20, 2023

View reviewed changes

dni mentioned this pull request Sep 20, 2023

[FEAT] move out extensions from /lnbits #1917

Closed

dni added this to the Release Version 0.11.0 milestone Sep 20, 2023

dni added the enhancement Make something better label Sep 20, 2023

prusnak reviewed Sep 20, 2023

View reviewed changes

blackcoffeexbt assigned blackcoffeexbt and unassigned blackcoffeexbt Sep 23, 2023

blackcoffeexbt self-requested a review September 23, 2023 08:50

callebtc approved these changes Sep 23, 2023

View reviewed changes

motorina0 mentioned this pull request Sep 25, 2023

Updated dev user guide for extension development #1962

Merged

Merge branch 'dev' into import_external_module

3dd91b8

dni merged commit c536df0 into dev Sep 25, 2023

dni deleted the import_external_module branch September 25, 2023 10:44

dni mentioned this pull request Oct 5, 2023

update main branch to 0.11.0 (#1994) #1995

Closed

motorina0 mentioned this pull request Oct 17, 2023

fix: explicitly specify the DB name when migrating #2039

Merged

dni added a commit that referenced this pull request Oct 17, 2023

Revert "Support custom path for installed extensions (#1940)"

ab4fb41

This reverts commit 6835e9e.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support custom path for installed extensions #1940

Support custom path for installed extensions #1940

motorina0 commented Sep 13, 2023 •

edited

Loading

motorina0 commented Sep 13, 2023 •

edited

Loading

codecov bot commented Sep 13, 2023 •

edited

Loading

motorina0 commented Sep 18, 2023

callebtc Sep 19, 2023

callebtc Sep 19, 2023

motorina0 Sep 19, 2023 •

edited

Loading

callebtc Sep 19, 2023

motorina0 Sep 19, 2023

dni Sep 20, 2023

callebtc Sep 19, 2023

motorina0 Sep 19, 2023

callebtc Sep 19, 2023

motorina0 Sep 19, 2023 •

edited

Loading

callebtc Sep 19, 2023

motorina0 Sep 19, 2023

callebtc Sep 19, 2023

dni left a comment

prusnak Sep 20, 2023 •

edited

Loading

callebtc left a comment

		sys.path.append(str(Path(settings.lnbits_extensions_path, "extensions")))
		sys.path.append(str(Path(settings.lnbits_extensions_path, "upgrades")))

	Path(settings.lnbits_extensions_path, "extensions", f)
	Path(settings.lnbits_extensions_path, "extensions", f[18:] if f.startswith("lnbits/extensions/" else f)

Support custom path for installed extensions #1940

Support custom path for installed extensions #1940

Conversation

motorina0 commented Sep 13, 2023 • edited Loading

Summary

Testing

Screenshots

motorina0 commented Sep 13, 2023 • edited Loading

Required Extension Changes

Change 1:

Change 2:

codecov bot commented Sep 13, 2023 • edited Loading

Codecov Report

motorina0 commented Sep 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

motorina0 Sep 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

motorina0 Sep 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dni left a comment

Choose a reason for hiding this comment

prusnak Sep 20, 2023 • edited Loading

Choose a reason for hiding this comment

callebtc left a comment

Choose a reason for hiding this comment

motorina0 commented Sep 13, 2023 •

edited

Loading

motorina0 commented Sep 13, 2023 •

edited

Loading

codecov bot commented Sep 13, 2023 •

edited

Loading

motorina0 Sep 19, 2023 •

edited

Loading

motorina0 Sep 19, 2023 •

edited

Loading

prusnak Sep 20, 2023 •

edited

Loading