Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retire Request, upgrade to Node 16 #344

Merged
merged 108 commits into from
Nov 18, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
108 commits
Select commit Hold shift + click to select a range
5d27d1f
node-fetch: update using new response object
nleush Oct 13, 2021
8b2c74c
plugins: simple export
nleush Oct 13, 2021
dcf9623
validators
nleush Oct 13, 2021
6ed00f3
system plugins
nleush Oct 13, 2021
2454b14
plugins: exports and imports
nleush Oct 14, 2021
29bc742
core: update with ES6 export import
nleush Oct 14, 2021
0ee79cf
tests: ES6 updates
nleush Oct 14, 2021
e5a824d
remove all old exports
nleush Oct 14, 2021
ff33a7c
utils: use fetch for `getUriStatus`
nleush Oct 14, 2021
e3f57f4
Update config.local.js.SAMPLE
nleush Oct 14, 2021
3fa0fd8
bugfix import export
nleush Oct 15, 2021
e4e031b
bugfix cache.js import export
nleush Oct 15, 2021
0c253dd
core: add `iframelyRun` param; add `templates`, `plugin` param for va…
nleush Oct 15, 2021
ae577e2
fix using request import
nleush Oct 15, 2021
1a448b2
fix import `logging.js`
nleush Oct 15, 2021
a73348b
core: add `templates`, `plugin` param for validators
nleush Oct 15, 2021
709ac76
remove unused
nleush Oct 15, 2021
ac5b0cb
fix typo
nleush Oct 15, 2021
2f317b8
bugfix abort
nleush Oct 15, 2021
0d858b2
core: fix `plugin` param for validators
nleush Oct 15, 2021
bd0a75d
validators: fix imports
nleush Oct 15, 2021
c7dc3e6
bugfixes for testrun
nleush Oct 15, 2021
f26ca31
fix plugins utils import
nleush Oct 15, 2021
fcb4a79
bugfixes: fetch
nleush Oct 15, 2021
64447f0
fix cheerio import
nleush Oct 18, 2021
5e63e67
fetch: fix compression, use native
nleush Oct 18, 2021
24d5f84
node-fetch: use `redirect` and `follow` options
nleush Oct 18, 2021
c51068c
use fetch-h2
nleush Oct 19, 2021
a517bd8
cheerio: fix import htmlparser2
nleush Oct 19, 2021
35a0cc0
fix import imagesize
nleush Oct 19, 2021
12a2b9d
htmlparser: fix pause resume
nleush Oct 19, 2021
39d7105
upgrade cheerio
nleush Oct 19, 2021
bce7fbd
upgrade `htmlparser2`
nleush Oct 19, 2021
1b5c4f1
fix `cheerio` import
nleush Oct 19, 2021
2fee845
fix imports
nleush Oct 19, 2021
c5caf9b
fix audit
nleush Oct 19, 2021
b1e4e4e
reivew the use of cheerio and query-strings
iparamonau Oct 20, 2021
7b08142
fix sax import
nleush Oct 20, 2021
17cb1d5
use `helix-fetch`
nleush Oct 20, 2021
26e53dc
fetch: disable cache
nleush Oct 20, 2021
0f9deed
Get rid of some of the underscores
iparamonau Oct 21, 2021
76150ee
request: implement using `helix-fetch` with most features
nleush Oct 21, 2021
c651a37
bugfix typo `this.result` -> `this._result`
nleush Oct 21, 2021
c95dc95
api request: user default user agent
nleush Oct 22, 2021
fd3ac1a
fetch: remove obsolete EventEmitter; use `keep-alive`; remove obsolet…
nleush Oct 22, 2021
089bca6
bugfix fetch wrapper
nleush Oct 22, 2021
4218737
remove `DISABLE_HTTP2` from config sample
nleush Oct 22, 2021
ca72a7e
plugins: remove obsolete `request` options
nleush Oct 25, 2021
578ea71
fetch: add timeout
nleush Oct 25, 2021
7bb0b13
utils: `rejectUnauthorized: true` for `getHead` request
nleush Oct 25, 2021
f2c95b0
core: return used http2 protocol
nleush Oct 26, 2021
430407a
tests: fix imports
nleush Oct 26, 2021
473911a
Merge branch 'develop' into node-fetch
nleush Oct 26, 2021
f377db8
tests: update imports
nleush Oct 26, 2021
004b164
config: update sample ssl usage with async file load
nleush Oct 26, 2021
597bc04
mamcached: fix import
nleush Oct 26, 2021
0dd47b2
server: fix import https
nleush Oct 26, 2021
6db1fb5
remove global process.env.NODE_TLS_REJECT_UNAUTHORIZED
nleush Oct 26, 2021
0016d60
fix redis import
nleush Oct 26, 2021
f1e08f7
fix imports
nleush Oct 26, 2021
5871e8d
slideshare: fix merge
nleush Oct 26, 2021
137c501
slideshare: revert wrong merge
nleush Oct 26, 2021
2b68924
Indent sample config
iparamonau Oct 26, 2021
b325e33
twitter: remove now obsolete 1.1 oauth APi
iparamonau Oct 27, 2021
85ad209
twitter - remove one more obsolete reference
iparamonau Oct 27, 2021
5d93d5d
twitter - clean up obsolete async
iparamonau Oct 27, 2021
475cfb6
request: `json: false` prevents json parsing
nleush Oct 27, 2021
8ab16b5
utils: allow use custom proxy
nleush Oct 27, 2021
b41a5cc
fix using cheerio `$script.text()`
nleush Oct 27, 2021
957c755
fetch: fix default follow redirect if option is not specified
nleush Oct 27, 2021
921915a
fetch: fix default follow redirect if option is not specified
nleush Oct 27, 2021
9500e14
tests: fix ld-json
iparamonau Oct 27, 2021
5322bef
tests: fix scribd
iparamonau Oct 28, 2021
9c57803
try fix `ABORT_ERR`
nleush Oct 28, 2021
a7316f1
Merge branch 'node-fetch' of github.com:itteco/iframely into node-fetch
nleush Oct 28, 2021
edfd472
fetch: allow disable http2
nleush Oct 28, 2021
574bc46
use `cheerio` 0.22
nleush Oct 28, 2021
063c21e
fix cheerio import
nleush Oct 28, 2021
afa68a3
remove oembed-iframe
iparamonau Oct 28, 2021
a6175a7
Revert "remove oembed-iframe"
iparamonau Oct 28, 2021
a35e754
test skipping plugins for mandatory params
nleush Oct 28, 2021
849c8c1
Merge branch 'node-fetch' of github.com:itteco/iframely into node-fetch
nleush Oct 28, 2021
4d309e5
review oembed-iframe
iparamonau Oct 28, 2021
eaf547f
Merge branch 'node-fetch' of https://github.com/itteco/iframely into …
iparamonau Oct 28, 2021
488bec7
temp fix AbortError for uri status
nleush Oct 29, 2021
4cf4f74
fix `__dirname` in `utils.js`
nleush Nov 1, 2021
a495cee
fix `package.json` `main`
nleush Nov 1, 2021
7264134
fix import `GracefulServer`
nleush Nov 5, 2021
a7da5b1
fix `tester.js` for `pm2`
nleush Nov 5, 2021
f79e650
disable decode html entities in htmlparser2
nleush Nov 8, 2021
759278d
upgrade iconv-lite
nleush Nov 8, 2021
1440478
fetch: retry request on http1 when http2 error
nleush Nov 9, 2021
00903fe
return `response.abort()` logic
nleush Nov 9, 2021
cd15330
fetch: fix `FetchError ABORT_ERR` when shared session aborted for spe…
nleush Nov 10, 2021
1ddef37
Update .gitignore
nleush Nov 10, 2021
6336780
htmlparser: do not abort on http error code
nleush Nov 10, 2021
5c24bf3
Merge branch 'develop' into node-fetch
nleush Nov 11, 2021
d81041c
fix config loading
nleush Nov 12, 2021
0b0b8a0
config comment
nleush Nov 12, 2021
0aff940
config: always use `iframely` config; use spread to merge objects
nleush Nov 12, 2021
d9e7500
fetch: remove hash part of url
nleush Nov 15, 2021
ee3751f
fetch: HostAbortController to prevent abort opened connections on host
nleush Nov 16, 2021
987c8c8
fetch: use timeout for response data
nleush Nov 16, 2021
4f0b1d6
bugfix
nleush Nov 16, 2021
f14074e
core: make wider post plugins iteration context
nleush Nov 15, 2021
381bc0e
domains: remove request `timeout: 1000`
nleush Nov 17, 2021
63acdf0
always abort on head request
nleush Nov 17, 2021
b0da2b6
Merge branch 'node-fetch-prevent-host-abort' into node-fetch
nleush Nov 17, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
8 changes: 2 additions & 6 deletions ansible-docker-iframely/templates/config.local.js.j2
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
(function() {
var config = {
export default {

// Specify a path for custom plugins. Custom plugins will override core plugins.
// CUSTOM_PLUGINS_PATH: __dirname + '/yourcustom-plugin-folder',
Expand Down Expand Up @@ -259,7 +258,4 @@
}
}
*/
};

module.exports = config;
})();
};
31 changes: 17 additions & 14 deletions app.js
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
var sysUtils = require('./utils');
import { cacheMiddleware, NotFound } from './utils.js';
import CONFIG from './config.loader.js';
global.CONFIG = CONFIG;

console.log("");
console.log("Starting Iframely...");
Expand All @@ -8,13 +10,12 @@ if (!CONFIG.baseAppUrl) {
console.warn('Warning: CONFIG.baseAppUrl not set, default value used');
}

var path = require('path');
var express = require('express');
var jsonxml = require('jsontoxml');
import express from 'express';
import * as jsonxml from 'jsontoxml';

var NotFound = sysUtils.NotFound;
const app = express();

var app = express();
export default app;

app.set('view engine', 'ejs');

Expand All @@ -40,12 +41,17 @@ app.use(function(req, res, next) {
next();
});

app.use(sysUtils.cacheMiddleware);
app.use(cacheMiddleware);

import apiViews from './modules/api/views.js';
import debugViews from './modules/debug/views.js';
apiViews(app);
debugViews(app);

require('./modules/api/views')(app);
require('./modules/debug/views')(app);
require('./modules/tests-ui/views')(app);
if (CONFIG.tests) {
const testViews = await import('./modules/tests-ui/views.js');
testViews.default(app);
}

app.use(logErrors);
app.use(errorHandler);
Expand Down Expand Up @@ -168,7 +174,4 @@ app.get('/', function(req, res) {
res.end();
});

process.title = "iframely";
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";

module.exports = app;
process.title = "iframely";
6 changes: 3 additions & 3 deletions cluster.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
var GracefulCluster = require('graceful-cluster').GracefulCluster;
var sysUtils = require('./utils');
import { GracefulCluster } from 'graceful-cluster';
import * as sysUtils from './utils.js';

process.title = 'iframely-cluster';

Expand All @@ -10,6 +10,6 @@ GracefulCluster.start({
restartOnTimeout: CONFIG.CLUSTER_WORKER_RESTART_ON_PERIOD,
restartOnMemory: CONFIG.CLUSTER_WORKER_RESTART_ON_MEMORY_USED,
serverFunction: function() {
require('./server');
import('./server');
}
});
37 changes: 19 additions & 18 deletions config.js
Original file line number Diff line number Diff line change
@@ -1,19 +1,18 @@
(function() {
import * as _ from 'underscore';
import * as path from 'path';
import * as fs from 'fs';

// Monkey patch before you require http for the first time.
var majorVersion = process.version.match(/v(\d+)\./);
majorVersion = parseInt(majorVersion);
if (majorVersion < 10) {
process.binding('http_parser').HTTPParser = require('http-parser-js').HTTPParser;
}

var _ = require('underscore');
var path = require('path');
var fs = require('fs');
import { fileURLToPath } from 'url';
import { dirname } from 'path';

var version = require('./package.json').version;
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);

import { readFile } from 'fs/promises';
const json = JSON.parse(await readFile(new URL('./package.json', import.meta.url)));
var version = json.version;

var config = {
const config = {

baseAppUrl: "",
port: 8061,
Expand Down Expand Up @@ -383,7 +382,8 @@
// Providers config loader.
var local_config_path = path.resolve(__dirname, "config.providers.js");
if (fs.existsSync(local_config_path)) {
var local = require(local_config_path);
var local = await import(local_config_path);
local = local && local.default;
_.extend(config, local);
}

Expand All @@ -397,11 +397,13 @@

// Try config by NODE_ENV.
if (fs.existsSync(env_config_path)) {
var local = require(env_config_path);
var local = await import(env_config_path);
local = local && local.default;

} else if (fs.existsSync(local_config_path)) {
// Else - try local config.
var local = require(local_config_path);
var local = await import(local_config_path);
local = local && local.default;
}

_.extend(config, local);
Expand Down Expand Up @@ -429,5 +431,4 @@
config.HTTP2_RETRY_CODES[item] = 1;
});

module.exports = config;
})();
export default config;
5 changes: 5 additions & 0 deletions config.loader.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
import iframelyConfig from './config.js';
// Load global config from exec dir, because `iframely` can be used as library.
var globalConfig = await import(process.cwd() + '/config.js');
globalConfig = globalConfig && globalConfig.default;
export default {...iframelyConfig, ...globalConfig};
Loading