diff --git a/.github/workflows/build-assets.yml b/.github/workflows/build-assets.yml new file mode 100644 index 0000000000..a3c2a9d348 --- /dev/null +++ b/.github/workflows/build-assets.yml @@ -0,0 +1,32 @@ +name: build assets + +on: + push: + branches: + - master + paths: + - 'lib/**' + +jobs: + build: + runs-on: ubuntu-latest + + strategy: + matrix: + node-version: + - 14.x + + steps: + - name: Checkout + uses: actions/checkout@v2 + - name: Use Node.js ${{ matrix.node-version }} + uses: actions/setup-node@v1 + with: + node-version: ${{ matrix.node-version }} + - name: build file + run: npm ci && npm run build:all + - name: Deploy + uses: peaceiris/actions-gh-pages@v3 + with: + github_token: ${{ secrets.GITHUB_TOKEN }} + publish_dir: ./assets \ No newline at end of file diff --git a/.gitignore b/.gitignore index fc0f64a663..eed3c9f7ff 100644 --- a/.gitignore +++ b/.gitignore @@ -19,3 +19,5 @@ app-minimal/ .now .vercel + +assets/build/ \ No newline at end of file diff --git a/assets/404.html b/assets/404.html new file mode 100644 index 0000000000..f8c6427942 --- /dev/null +++ b/assets/404.html @@ -0,0 +1,4 @@ + \ No newline at end of file diff --git a/assets/CNAME b/assets/CNAME new file mode 100644 index 0000000000..5ab3a9991b --- /dev/null +++ b/assets/CNAME @@ -0,0 +1 @@ +rsshub.js.org \ No newline at end of file diff --git a/assets/index.html b/assets/index.html new file mode 100644 index 0000000000..f8c6427942 --- /dev/null +++ b/assets/index.html @@ -0,0 +1,4 @@ + \ No newline at end of file diff --git a/docs/.vuepress/nav/zh.js b/docs/.vuepress/nav/zh.js index 26c1c336c3..1f46309ef8 100644 --- a/docs/.vuepress/nav/zh.js +++ b/docs/.vuepress/nav/zh.js @@ -14,6 +14,10 @@ module.exports = [ { text: '详细规范', items: [ + { + text: '路由规范', + link: '/joinus/script-standard', + }, { text: '日期处理', link: '/joinus/pub-date', diff --git a/docs/joinus/script-standard.md b/docs/joinus/script-standard.md new file mode 100644 index 0000000000..fa6e261e16 --- /dev/null +++ b/docs/joinus/script-standard.md @@ -0,0 +1,150 @@ +# 路由规范 + +::: warning 警告 + +这个规范仍在制定过程中，可能会随着时间推移而发生改变，请记得多回来看看！ + +::: + +在编写新的路由时，RSSHub会读取文件夹中的： + +- `router.js`注册路由 +- `maintainer.js`获取路由路径，维护者 +- `radar.js`获取路由所对应的网站，以及匹配规则：https://github.com/DIYgod/RSSHub-Radar/ +- `templates` 渲染模版 + +**以上文件为所有插件必备** + +``` +├───lib/v2 +│ ├───furstar +│ ├─── templates +│ ├─── description.art +│ ├─── router.js +│ ├─── maintainer.js +│ ├─── radar.js +│ ├─── someOtherJs.js +│ └───test +│ └───someOtherScript +... +``` + +**所有符合条件的，在`/v2`路径下的路由，将会被自动载入，无需更新`router.js`** + +## 路由示例 + +参考`furstar`: `./lib/v2/furstar` + +可以复制该文件夹作为新路由模版 + +## 注册路由 + +`router.js` 应当导出一个方法，我们在初始化路由的时候，会提供一个`@koa/router`对象 + +### 命名规范 + +我们会默认将所有的路由文件夹名字附加在真正的路由前面。路由维护者可以认定自己获取的就是根，我们会在附加对应的命名空间，在这空间底下，开发者有所有的控制权 + +### 例子 + +```js +module.exports = function (router) { + router.get('/characters/:lang?', require('./index')); + router.get('/artists/:lang?', require('./artists')); + router.get('/archive/:lang?', require('./archive')); +}; +``` + +## 维护者列表 + +`maintainer.js` 应当导出一个对象，在我们获取路径相关信息时，将会在从这里调取开发者信息等 + +- key: `@koa/router` 对应的路径匹配 +- value: 数组，包含所有开发者的Github Username + +Github ID可能是更好的选择，但是后续处理不便，目前暂定Username + +### 例子 + +```js +module.exports = { + '/characters/:lang?': ['NeverBehave'], + '/artists/:lang?': ['NeverBehave'], + '/archive/:lang?': ['NeverBehave'], +}; +``` + +`npm run build:maintainer` 将会在`assets/build`下生成一份贡献者清单 + +## Radar Rules + +书写方式：https://docs.rsshub.app/joinus/quick-start.html#ti-jiao-xin-de-rsshub-radar-gui-ze + +**我们目前要求所有路由，必须包含这个文件，并且包含对应的域名 -- 我们不要求完全的路由匹配，最低要求是在对应的网站，可以显示支持即可。这个文件后续会用于帮助bug反馈。** + +### 例子 + +```js +module.exports = { + 'furstar.jp': { + _name: 'Furstar', + '.': [ + { + title: '最新售卖角色列表', + docs: 'https://docs.rsshub.app/shopping.html#furstar-zui-xin-shou-mai-jiao-se-lie-biao', + source: ['/:lang', '/'], + target: '/characters/:lang', + }, + { + title: '已经出售的角色列表', + docs: 'https://docs.rsshub.app/shopping.html#furstar-yi-jing-chu-shou-de-jiao-se-lie-biao', + source: ['/:lang/archive.php', '/archive.php'], + target: '/archive/:lang', + }, + { + title: '画师列表', + docs: 'https://docs.rsshub.app/shopping.html#furstar-hua-shi-lie-biao', + source: ['/'], + target: '/artists', + }, + ], + }, +}; +``` + + +`npm run build:radar` 将会在`/assets/build/`下生成一份完整的`radar-rules.js` + + +## Template + +我们目前要求所有路由，在渲染`description`等带HTML的内容时，**必须**使用art引擎进行排版 + +art说明文档：https://aui.github.io/art-template/docs/ + +同时，所有模版应该放在插件`templates`文件夹中 -- 后续我们会以此提供自定义模版切换/渲染等需求 + +### 例子 + +```art +
+ + {{ if link !== null }} + {{name}} + {{ else }} + {{name}} + {{ /if }} +
+``` + +```js +const { art } = require('@/utils/render'); +const renderAuthor = (author) => art(path.join(__dirname, 'templates/author.art'), author); +``` + +## ctx.state.json + +插件目前可以提供一个自定义的对象，用于调试 -- 访问对应路由+`.debug.json`即可获取到对应内容 + +我们对这个部分格式内容没有任何限制，完全可选，目前会继续观察这个选项的发展方向 + diff --git a/docs/parameter.md b/docs/parameter.md index b5e302b554..7e449bb946 100644 --- a/docs/parameter.md +++ b/docs/parameter.md @@ -94,6 +94,16 @@ RSSHub 同时支持 RSS 2.0 和 Atom 输出格式，在路由末尾添加 `.rss` - Atom - - 和 filter 或其他 URL query 一起使用 `https://rsshub.app/bilibili/user/coin/2267573.atom?filter=微小微|赤九玖|暴走大事件` +### debug + +在路由末尾添加 `.debug.json`且实例运行在`debugInfo=true`的情况下，RSShub将会返回插件设置在`ctx.state.json`的内容 + +这功能皆在方便开发者调试问题，方便用户自行开发需要的功能。插件作者可以酌情考虑使用，没有格式要求。 + +举例： + +- `/furstar/characters/cn.debug.json` + ## 输出简讯 可以使用 `brief` 参数输出特定字数 ( ≥ `100` 字 ) 的纯文本内容 diff --git a/lib/api_router.js b/lib/api_router.js index eb72e2e000..4afc4ae9e6 100644 --- a/lib/api_router.js +++ b/lib/api_router.js @@ -1,15 +1,13 @@ const Router = require('@koa/router'); const router = new Router(); -const routes = require('./router'); +const maintainer = require('./maintainer'); router.get('/routes/:name?', (ctx) => { - const allRoutes = Array.from(routes.stack); - allRoutes.shift(); const result = {}; let counter = 0; - allRoutes.forEach((i) => { - const path = i.path; + Object.keys(maintainer).forEach((i) => { + const path = i; const top = path.split('/')[1]; if (!ctx.params.name || top === ctx.params.name) { diff --git a/lib/app.js b/lib/app.js index b51ca2314a..39979dffce 100644 --- a/lib/app.js +++ b/lib/app.js @@ -16,8 +16,10 @@ const favicon = require('koa-favicon'); const debug = require('./middleware/debug'); const accessControl = require('./middleware/access-control'); const antiHotlink = require('./middleware/anti-hotlink'); +const loadOnDemand = require('./middleware/load-on-demand'); const router = require('./router'); +const core_router = require('./core_router'); const protected_router = require('./protected_router'); const mount = require('koa-mount'); @@ -70,16 +72,21 @@ app.use(antiHotlink); // 3 filter content app.use(parameter); +// No Cache routes +app.use(mount('/', core_router.routes())).use(core_router.allowedMethods()); +// API router +app.use(mount('/api', api_router.routes())).use(api_router.allowedMethods()); + // 2 cache app.use(cache(app)); +// 1 load on demand +app.use(loadOnDemand(app)); + // router app.use(mount('/', router.routes())).use(router.allowedMethods()); // routes the require authentication app.use(mount('/protected', protected_router.routes())).use(protected_router.allowedMethods()); -// API router -app.use(mount('/api', api_router.routes())).use(api_router.allowedMethods()); - module.exports = app; diff --git a/lib/config.js b/lib/config.js index 6474d0fd52..5c2a89d14f 100644 --- a/lib/config.js +++ b/lib/config.js @@ -33,6 +33,7 @@ const calculateValue = () => { }, cache: { type: typeof envs.CACHE_TYPE === 'undefined' ? 'memory' : envs.CACHE_TYPE, // 缓存类型，支持 'memory' 和 'redis'，设为空可以禁止缓存 + requestTimeout: parseInt(envs.CACHE_REQUEST_TIMEOUT) || 60, routeExpire: parseInt(envs.CACHE_EXPIRE) || 5 * 60, // 路由缓存时间，单位为秒 contentExpire: parseInt(envs.CACHE_CONTENT_EXPIRE) || 1 * 60 * 60, // 不变内容缓存时间，单位为秒 }, diff --git a/lib/core_router.js b/lib/core_router.js new file mode 100644 index 0000000000..823485a702 --- /dev/null +++ b/lib/core_router.js @@ -0,0 +1,17 @@ +const Router = require('@koa/router'); +const config = require('@/config').value; +const router = new Router(); + +// Load Core Route +router.get('/', require('./routes/index')); + +router.get('/robots.txt', (ctx) => { + if (config.disallowRobot) { + ctx.set('Content-Type', 'text/plain'); + ctx.body = 'User-agent: *\nDisallow: /'; + } else { + ctx.throw(404, 'Not Found'); + } +}); + +module.exports = router; diff --git a/lib/errors/RequestInProgress.js b/lib/errors/RequestInProgress.js new file mode 100644 index 0000000000..222be7e91d --- /dev/null +++ b/lib/errors/RequestInProgress.js @@ -0,0 +1,3 @@ +class RequestInProgressError extends Error {} + +module.exports = RequestInProgressError; diff --git a/lib/errors/index.js b/lib/errors/index.js new file mode 100644 index 0000000000..63fe1f2820 --- /dev/null +++ b/lib/errors/index.js @@ -0,0 +1,3 @@ +module.exports = { + RequestInProgressError: require('./RequestInProgress'), +}; diff --git a/lib/maintainer.js b/lib/maintainer.js new file mode 100644 index 0000000000..8c343764cd --- /dev/null +++ b/lib/maintainer.js @@ -0,0 +1,27 @@ +const dirname = __dirname + '/v2'; + +// 遍历整个 routes 文件夹，收集模块 maintainer.js +const maintainerPath = require('require-all')({ + dirname, + filter: /maintainer\.js$/, +}); + +const maintainers = {}; + +// 将收集到的自定义模块进行合并 +for (const dir in maintainerPath) { + const routes = maintainerPath[dir]['maintainer.js']; // Do not merge other file + for (const key in routes) { + maintainers['/' + dir + key] = routes[key]; + } +} + +// 兼容旧版路由 +const router = require('./router'); +router.stack.forEach((e) => { + if (!maintainers[e.path]) { + maintainers[e.path] = []; + } +}); + +module.exports = maintainers; diff --git a/lib/middleware/cache.js b/lib/middleware/cache.js deleted file mode 100644 index 0a302f7479..0000000000 --- a/lib/middleware/cache.js +++ /dev/null @@ -1,182 +0,0 @@ -const Lru = require('lru-cache'); -const md5 = require('@/utils/md5'); -const config = require('@/config').value; -const logger = require('@/utils/logger'); - -let Redis; - -module.exports = function (app) { - let available = false; - const globalCache = { - get: null, - set: null, - }; - - if (config.cache.type === 'redis') { - Redis = Redis || require('ioredis'); - const redisClient = new Redis(config.redis.url); - - redisClient.on('error', (error) => { - available = false; - logger.error('Redis error: ', error); - }); - redisClient.on('end', () => { - available = false; - }); - redisClient.on('connect', () => { - available = true; - logger.info('Redis connected.'); - }); - - app.context.cache = { - get: async (key, refresh = true) => { - if (key && available) { - let value = await redisClient.get(key); - if (value && refresh) { - redisClient.expire(key, config.cache.contentExpire); - value = value + ''; - } - return value; - } - }, - set(key, value, maxAge = config.cache.contentExpire) { - if (!available) { - return; - } - if (!value || value === 'undefined') { - value = ''; - } - if (typeof value === 'object') { - value = JSON.stringify(value); - } - if (key) { - redisClient.set(key, value, 'NX', 'EX', maxAge); - } - }, - client: redisClient, - globalCache, - }; - globalCache.get = async (key) => { - if (key && available) { - const value = await redisClient.get(key); - return value; - } - }; - globalCache.set = app.context.cache.set; - } else if (config.cache.type === 'memory') { - const pageCache = new Lru({ - maxAge: config.cache.routeExpire * 1000, - max: Infinity, - }); - - const routeCache = new Lru({ - maxAge: config.cache.routeExpire * 1000, - max: Infinity, - updateAgeOnGet: true, - }); - - app.context.cache = { - get: (key, refresh = true) => { - if (key && available) { - let value = (refresh ? routeCache : pageCache).get(key); - if (value) { - value = value + ''; - } - return value; - } - }, - set: (key, value, maxAge = config.cache.contentExpire, refresh = true) => { - if (!value || value === 'undefined') { - value = ''; - } - if (typeof value === 'object') { - value = JSON.stringify(value); - } - if (key && available) { - return (refresh ? routeCache : pageCache).set(key, value, maxAge * 1000); - } - }, - client: [pageCache, routeCache], - globalCache, - }; - globalCache.get = (key) => { - if (key && available) { - return pageCache.get(key); - } - }; - globalCache.set = (key, value, maxAge) => { - if (!value || value === 'undefined') { - value = ''; - } - if (typeof value === 'object') { - value = JSON.stringify(value); - } - if (key) { - return pageCache.set(key, value, maxAge * 1000); - } - }; - available = true; - } else { - app.context.cache = { - get: () => null, - set: () => null, - }; - } - - app.context.cache.tryGet = async function (key, getValueFunc, maxAge = config.cache.contentExpire) { - let v = await this.get(key); - if (!v) { - v = await getValueFunc(); - this.set(key, v, maxAge); - } else { - let parsed; - try { - parsed = JSON.parse(v); - } catch (e) { - parsed = null; - } - if (parsed) { - v = parsed; - } - } - - return v; - }; - - return async function cache(ctx, next) { - const key = 'koa-redis-cache:' + md5(ctx.request.path); - - if (!available) { - return next(); - } - - try { - const value = await globalCache.get(key); - - if (value) { - ctx.response.status = 200; - if (config.cache.type === 'redis') { - ctx.response.set({ - 'X-Koa-Redis-Cache': 'true', - }); - } else if (config.cache.type === 'memory') { - ctx.response.set({ - 'X-Koa-Memory-Cache': 'true', - }); - } - ctx.state.data = JSON.parse(value); - return; - } - } catch (e) { - // - } - - await next(); - - if (ctx.response.get('Cache-Control') !== 'no-cache' && ctx.state && ctx.state.data) { - ctx.state.data.lastBuildDate = new Date().toUTCString(); - const body = JSON.stringify(ctx.state.data); - await globalCache.set(key, body, config.cache.routeExpire); - } - }; -}; diff --git a/lib/middleware/cache/index.js b/lib/middleware/cache/index.js new file mode 100644 index 0000000000..e12f5e5196 --- /dev/null +++ b/lib/middleware/cache/index.js @@ -0,0 +1,129 @@ +const md5 = require('@/utils/md5'); +const config = require('@/config').value; +const logger = require('@/utils/logger'); +const { RequestInProgressError } = require('@/errors'); + +const globalCache = { + get: () => null, + set: () => null, +}; + +let cacheModule = { + get: () => null, + set: () => null, + status: { available: false }, + clients: {}, +}; + +if (config.cache.type === 'redis') { + cacheModule = require('./redis'); + const { redisClient } = cacheModule.clients; + globalCache.get = async (key) => { + if (key && cacheModule.status.available) { + const value = await redisClient.get(key); + return value; + } + }; + globalCache.set = cacheModule.set; +} else if (config.cache.type === 'memory') { + cacheModule = require('./memory'); + const { pageCache } = cacheModule.clients; + globalCache.get = (key) => { + if (key && cacheModule.status.available) { + return pageCache.get(key); + } + }; + globalCache.set = (key, value, maxAge) => { + if (!value || value === 'undefined') { + value = ''; + } + if (typeof value === 'object') { + value = JSON.stringify(value); + } + if (key) { + return pageCache.set(key, value, maxAge * 1000); + } + }; +} else { + logger.error('Cache not available, concurrent requests are not limited. This could lead to bad behavior.'); +} + +// only give cache string, as the `!` condition tricky +// md5 is used to shrink key size +// plz, write these tips in comments! +module.exports = function (app) { + const { get, set, status } = cacheModule; + app.context.cache = { + ...cacheModule, + tryGet: async (key, getValueFunc, maxAge = config.cache.contentExpire) => { + let v = await get(key); + if (!v) { + v = await getValueFunc(); + set(key, v, maxAge); + } else { + let parsed; + try { + parsed = JSON.parse(v); + } catch (e) { + parsed = null; + } + if (parsed) { + v = parsed; + } + } + + return v; + }, + globalCache, + }; + + return async (ctx, next) => { + const key = 'koa-redis-cache:' + md5(ctx.request.path); + const controlKey = 'path-requested:' + md5(ctx.request.path); + + if (!status.available) { + return next(); + } + + const isRequesting = await globalCache.get(controlKey); + + if (isRequesting === '1') { + throw new RequestInProgressError('This path is currently fetching, please come back later!'); + } + + try { + const value = await globalCache.get(key); + + if (value) { + ctx.response.status = 200; + if (config.cache.type === 'redis') { + ctx.response.set({ + 'X-Koa-Redis-Cache': 'true', + }); + } else if (config.cache.type === 'memory') { + ctx.response.set({ + 'X-Koa-Memory-Cache': 'true', + }); + } + ctx.state.data = JSON.parse(value); + return; + } + } catch (e) { + // + } + + // Doesn't hit the cache? We need to let others know! + await globalCache.set(controlKey, '1', config.cache.requestTimeout); + await next(); + + if (ctx.response.get('Cache-Control') !== 'no-cache' && ctx.state && ctx.state.data) { + ctx.state.data.lastBuildDate = new Date().toUTCString(); + const body = JSON.stringify(ctx.state.data); + await globalCache.set(key, body, config.cache.routeExpire); + } + + // We need to let it go, even no cache set. + // Wait to set cache so the next request could be handled correctly + await globalCache.set(controlKey, '0', config.cache.requestTimeout); + }; +}; diff --git a/lib/middleware/cache/memory.js b/lib/middleware/cache/memory.js new file mode 100644 index 0000000000..6eff393aa3 --- /dev/null +++ b/lib/middleware/cache/memory.js @@ -0,0 +1,42 @@ +const Lru = require('lru-cache'); +const config = require('@/config').value; + +const status = { available: false }; + +const pageCache = new Lru({ + maxAge: config.cache.routeExpire * 1000, + max: Infinity, +}); + +const routeCache = new Lru({ + maxAge: config.cache.routeExpire * 1000, + max: Infinity, + updateAgeOnGet: true, +}); + +status.available = true; + +module.exports = { + get: (key, refresh = true) => { + if (key && status.available) { + let value = (refresh ? routeCache : pageCache).get(key); + if (value) { + value = value + ''; + } + return value; + } + }, + set: (key, value, maxAge = config.cache.contentExpire, refresh = true) => { + if (!value || value === 'undefined') { + value = ''; + } + if (typeof value === 'object') { + value = JSON.stringify(value); + } + if (key && status.available) { + return (refresh ? routeCache : pageCache).set(key, value, maxAge * 1000); + } + }, + clients: { pageCache, routeCache }, + status, +}; diff --git a/lib/middleware/cache/redis.js b/lib/middleware/cache/redis.js new file mode 100644 index 0000000000..d461bc87c4 --- /dev/null +++ b/lib/middleware/cache/redis.js @@ -0,0 +1,48 @@ +const config = require('@/config').value; +const Redis = require('ioredis'); +const logger = require('@/utils/logger'); + +const redisClient = new Redis(config.redis.url); + +const status = { available: false }; + +redisClient.on('error', (error) => { + status.available = false; + logger.error('Redis error: ', error); +}); +redisClient.on('end', () => { + status.available = false; +}); +redisClient.on('connect', () => { + status.available = true; + logger.info('Redis connected.'); +}); + +module.exports = { + get: async (key, refresh = true) => { + if (key && status.available) { + let value = await redisClient.get(key); + if (value && refresh) { + redisClient.expire(key, config.cache.contentExpire); + value = value + ''; + } + return value; + } + }, + set: (key, value, maxAge = config.cache.contentExpire) => { + if (!status.available) { + return; + } + if (!value || value === 'undefined') { + value = ''; + } + if (typeof value === 'object') { + value = JSON.stringify(value); + } + if (key) { + return redisClient.set(key, value, 'EX', maxAge); // setMode: https://redis.io/commands/set + } + }, + clients: { redisClient }, + status, +}; diff --git a/lib/middleware/load-on-demand.js b/lib/middleware/load-on-demand.js new file mode 100644 index 0000000000..0c15bb4230 --- /dev/null +++ b/lib/middleware/load-on-demand.js @@ -0,0 +1,36 @@ +const mount = require('koa-mount'); +const Router = require('@koa/router'); +const routes = require('../v2router'); +const loadedRoutes = new Set(); + +module.exports = function (app) { + return async function (ctx, next) { + const p = ctx.request.path.split('/').filter(Boolean); + let modName = null; + let mounted = false; + + if (p.length > 0) { + modName = p[0]; + if (!loadedRoutes.has(modName)) { + const mod = routes[modName]; + // Mount module + if (mod) { + mounted = true; + loadedRoutes.add(modName); + const router = new Router(); + mod(router); + app.use(mount(`/${modName}`, router.routes())).use(router.allowedMethods()); + } + } else { + mounted = true; + } + } + + await next(); + + // We should only add it when koa router matched + if (mounted && ctx._matchedRoute) { + ctx._matchedRoute = `/${modName}${ctx._matchedRoute}`; + } + }; +}; diff --git a/lib/middleware/onerror.js b/lib/middleware/onerror.js index 474f491f6f..ea71b528e2 100644 --- a/lib/middleware/onerror.js +++ b/lib/middleware/onerror.js @@ -3,6 +3,8 @@ const config = require('@/config').value; const art = require('art-template'); const path = require('path'); +const { RequestInProgressError } = require('@/errors'); + let Sentry; if (config.sentry.dsn) { @@ -33,77 +35,10 @@ module.exports = async (ctx, next) => { let message = err; if (err.name && (err.name === 'HTTPError' || err.name === 'RequestError')) { message = `${err.message}: target website might be blocking our access, you can host your own RSSHub instance for a better usability.`; - - const templateZh = ` - -- 完整路由地址，包含所有必选与可选参数 - \`${ctx.path}\` -- 预期是什么 - -- 实际发生了什么 - -- 部署相关信息 - - - -| Env | Value | -| ------------------ | ------------- | -| OS | | -| Node version |${process.version} | -| if Docker, version | | - -- 额外信息（日志、报错等） -\`\`\` - ${err.message} -\`\`\` `; - - const templateEn = ` - -- The involved route, with all required and optional parameters - \`${ctx.path}\` -- What is expected - -- What is actually happening - -- Self-deployment information - - - -| Env | Value | -| ------------------ | ------------- | -| OS | | -| Node version |${process.version} | -| if Docker, version | | - -- Additional info (logs errors etc) -\`\`\` - ${err.message} -\`\`\` `; - - message += `

如果您认为 RSSHub 导致了该错误，请在 GitHub 报告。`; - - message += `

If you believe this is an error caused by RSSHub, please report me on GitHub.`; } else if (err instanceof Error) { message = process.env.NODE_ENV === 'production' ? err.message : err.stack; } + logger.error(`Error in ${ctx.request.path}: ${message}`); if (config.isPackage) { @@ -116,7 +51,11 @@ Please ensure you have deployed the [master branch](https://github.com/DIYgod/RS ctx.set({ 'Content-Type': 'text/html; charset=UTF-8', }); - if (ctx.status === 403) { + + if (err instanceof RequestInProgressError) { + ctx.status = 503; + message = err.message; + } else if (ctx.status === 403) { message = err.message; } else { ctx.status = 404; @@ -127,6 +66,8 @@ Please ensure you have deployed the [master branch](https://github.com/DIYgod/RS ctx.body = art(path.resolve(__dirname, '../views/error.art'), { requestPath, message, + errorPath: ctx.path, + nodeVersion: process.version, }); } diff --git a/lib/middleware/template.js b/lib/middleware/template.js index e0d0a299f0..754164b710 100644 --- a/lib/middleware/template.js +++ b/lib/middleware/template.js @@ -1,14 +1,9 @@ const art = require('art-template'); const path = require('path'); const config = require('@/config').value; -const typeRegex = /\.(atom|rss)$/; -const unsupportedRegex = /\.json$/; +const typeRegex = /\.(atom|rss|debug\.json)$/; module.exports = async (ctx, next) => { - if (ctx.request.path.match(unsupportedRegex)) { - throw Error('JSON output had been removed, see: https://github.com/DIYgod/RSSHub/issues/1114'); - } - if (ctx.headers['user-agent'] && ctx.headers['user-agent'].includes('Reeder')) { ctx.request.path = ctx.request.path.replace(/.com$/, ''); } @@ -18,6 +13,17 @@ module.exports = async (ctx, next) => { await next(); + if (ctx.state.type[1] === 'debug.json' && config.debugInfo) { + ctx.set({ + 'Content-Type': 'application/json; charset=UTF-8', + }); + if (ctx.state.json) { + ctx.body = JSON.stringify(ctx.state.json, null, 4); + } else { + ctx.body = JSON.stringify({ message: "plugin does not set json" }); + } + } + if (!ctx.body) { let template; diff --git a/assets/radar-rules.js b/lib/radar-rules.js similarity index 99% rename from assets/radar-rules.js rename to lib/radar-rules.js index 820356b393..70c6a948d7 100644 --- a/assets/radar-rules.js +++ b/lib/radar-rules.js @@ -1,4 +1,4 @@ -({ +module.exports = { 'bilibili.com': { _name: 'bilibili', www: [ @@ -3116,4 +3116,4 @@ }, ], }, -}); +}; diff --git a/lib/radar.js b/lib/radar.js new file mode 100644 index 0000000000..8784cb9c34 --- /dev/null +++ b/lib/radar.js @@ -0,0 +1,22 @@ +const dirname = __dirname + '/v2'; +const toSource = require('tosource'); + +const radarRules = require('require-all')({ + dirname, + filter: /radar\.js$/, +}); + +let rules = {}; + +for (const dir in radarRules) { + const rule = radarRules[dir]['radar.js']; // Do not merge other file + rules = { ...rules, ...rule }; +} + +const oldRules = require('./radar-rules.js'); // Match old rules +rules = { ...rules, ...oldRules }; + +module.exports = { + rules, + toSource: () => `(${toSource(rules)})`, +}; diff --git a/lib/router.js b/lib/router.js index ea9edad20a..79278c7098 100644 --- a/lib/router.js +++ b/lib/router.js @@ -1,24 +1,6 @@ const Router = require('@koa/router'); -const config = require('@/config').value; const router = new Router(); -// 遍历整个 routes 文件夹，导入模块路由 router.js 和 router-custom.js 文件 -// 格式参考用例：routes/epicgames/router.js -const RouterPath = require('require-all')({ - dirname: __dirname + '/routes', - filter: /^.*router([-_]custom[s]?)?\.js$/, -}); - -// 将收集到的自定义模块路由进行合并 -for (const project in RouterPath) { - for (const routerName in RouterPath[project]) { - const proRouter = RouterPath[project][routerName](); - proRouter.stack.forEach((nestedLayer) => { - router.stack.push(nestedLayer); - }); - } -} - const RouterHandlerMap = new Map(); // 懒加载 Route Handler，Route 首次被请求时才会 require 相关文件 @@ -32,20 +14,7 @@ const lazyloadRouteHandler = (routeHandlerPath) => (ctx) => { return handler(ctx); }; -// index -router.get('/', lazyloadRouteHandler('./routes/index')); - -router.get('/robots.txt', (ctx) => { - if (config.disallowRobot) { - ctx.set('Content-Type', 'text/plain'); - ctx.body = 'User-agent: *\nDisallow: /'; - } else { - ctx.throw(404, 'Not Found'); - } -}); - -// test -router.get('/test/:id', lazyloadRouteHandler('./routes/test')); +// Deprecated: DO NOT ADD ROUTE HERE // RSSHub router.get('/rsshub/rss', lazyloadRouteHandler('./routes/rsshub/routes')); // 弃用 @@ -3234,11 +3203,6 @@ router.get('/liequtv/room/:id', lazyloadRouteHandler('./routes/liequtv/room')); // Behance router.get('/behance/:user/:type?', lazyloadRouteHandler('./routes/behance/index')); -// furstar.jp -router.get('/furstar/characters/:lang?', lazyloadRouteHandler('./routes/furstar/index')); -router.get('/furstar/artists/:lang?', lazyloadRouteHandler('./routes/furstar/artists')); -router.get('/furstar/archive/:lang?', lazyloadRouteHandler('./routes/furstar/archive')); - // 北京物资学院 router.get('/bwu/news', lazyloadRouteHandler('./routes/universities/bwu/news')); @@ -4149,7 +4113,7 @@ router.get('/ehentai/search/:params?/:page?/:bittorrent?', lazyloadRouteHandler( router.get('/ehentai/tag/:tag/:page?/:bittorrent?', lazyloadRouteHandler('./routes/ehentai/tag')); // 字型故事 -router.get('/fontstory', require('./routes/fontstory/tw')); +router.get('/fontstory', lazyloadRouteHandler('./routes/fontstory/tw')); // HKEPC router.get('/hkepc/:category?', lazyloadRouteHandler('./routes/hkepc/index')); @@ -4248,4 +4212,6 @@ router.get('/hket/:category?', lazyloadRouteHandler('./routes/hket/index')); // s-hentai router.get('/s-hentai/:id?', lazyloadRouteHandler('./routes/s-hentai')); +// Deprecated: DO NOT ADD ROUTE HERE + module.exports = router; diff --git a/lib/routes/epicgames/router.js b/lib/routes/epicgames/router.js deleted file mode 100644 index 2bf7cd69f7..0000000000 --- a/lib/routes/epicgames/router.js +++ /dev/null @@ -1,9 +0,0 @@ -// 文件名必须为 router.js。 -// Epic Games - -module.exports = () => { - const Router = require('@koa/router'); - const router = new Router(); - router.get('/epicgames/:collection', require('./index')); - return router; -}; diff --git a/lib/utils/render.js b/lib/utils/render.js new file mode 100644 index 0000000000..0eb5a32e4c --- /dev/null +++ b/lib/utils/render.js @@ -0,0 +1,7 @@ +const art = require('art-template'); + +// We may add more control over it later + +module.exports = { + art, +}; diff --git a/lib/routes/epicgames/index.js b/lib/v2/epicgames/index.js similarity index 100% rename from lib/routes/epicgames/index.js rename to lib/v2/epicgames/index.js diff --git a/lib/v2/epicgames/maintainer.js b/lib/v2/epicgames/maintainer.js new file mode 100644 index 0000000000..c908094eb1 --- /dev/null +++ b/lib/v2/epicgames/maintainer.js @@ -0,0 +1,3 @@ +module.exports = { + '/epicgames/:collection': ['DIYgod', 'NeverBehave', 'Zyx-A', 'junfengP'], +}; diff --git a/lib/v2/epicgames/router.js b/lib/v2/epicgames/router.js new file mode 100644 index 0000000000..db773af1fb --- /dev/null +++ b/lib/v2/epicgames/router.js @@ -0,0 +1,3 @@ +module.exports = function (router) { + router.get('/epicgames/:collection', require('./index')); +}; diff --git a/lib/routes/epicgames/supportedList.js b/lib/v2/epicgames/supportedList.js similarity index 100% rename from lib/routes/epicgames/supportedList.js rename to lib/v2/epicgames/supportedList.js diff --git a/lib/routes/furstar/archive.js b/lib/v2/furstar/archive.js similarity index 100% rename from lib/routes/furstar/archive.js rename to lib/v2/furstar/archive.js diff --git a/lib/routes/furstar/artists.js b/lib/v2/furstar/artists.js similarity index 100% rename from lib/routes/furstar/artists.js rename to lib/v2/furstar/artists.js diff --git a/lib/routes/furstar/index.js b/lib/v2/furstar/index.js similarity index 95% rename from lib/routes/furstar/index.js rename to lib/v2/furstar/index.js index 89c99f4c31..8223368571 100644 --- a/lib/routes/furstar/index.js +++ b/lib/v2/furstar/index.js @@ -13,6 +13,10 @@ module.exports = async (ctx) => { const details = await Promise.all(info.map((e) => utils.detailPage(e.detailPage, ctx.cache))); + ctx.state.json = { + info + }; + ctx.state.data = { title: 'Furstar 最新角色', link: 'https://furstar.jp', diff --git a/lib/v2/furstar/maintainer.js b/lib/v2/furstar/maintainer.js new file mode 100644 index 0000000000..373f96fd9c --- /dev/null +++ b/lib/v2/furstar/maintainer.js @@ -0,0 +1,5 @@ +module.exports = { + '/characters/:lang?': ['NeverBehave'], + '/artists/:lang?': ['NeverBehave'], + '/archive/:lang?': ['NeverBehave'], +}; diff --git a/lib/v2/furstar/radar.js b/lib/v2/furstar/radar.js new file mode 100644 index 0000000000..201317292f --- /dev/null +++ b/lib/v2/furstar/radar.js @@ -0,0 +1,25 @@ +module.exports = { + 'furstar.jp': { + _name: 'Furstar', + '.': [ + { + title: '最新售卖角色列表', + docs: 'https://docs.rsshub.app/shopping.html#furstar-zui-xin-shou-mai-jiao-se-lie-biao', + source: ['/:lang', '/'], + target: '/characters/:lang', + }, + { + title: '已经出售的角色列表', + docs: 'https://docs.rsshub.app/shopping.html#furstar-yi-jing-chu-shou-de-jiao-se-lie-biao', + source: ['/:lang/archive.php', '/archive.php'], + target: '/archive/:lang', + }, + { + title: '画师列表', + docs: 'https://docs.rsshub.app/shopping.html#furstar-hua-shi-lie-biao', + source: ['/'], + target: '/artists', + }, + ], + }, +}; diff --git a/lib/v2/furstar/router.js b/lib/v2/furstar/router.js new file mode 100644 index 0000000000..10824ba02b --- /dev/null +++ b/lib/v2/furstar/router.js @@ -0,0 +1,5 @@ +module.exports = function (router) { + router.get('/characters/:lang?', require('./index')); + router.get('/artists/:lang?', require('./artists')); + router.get('/archive/:lang?', require('./archive')); +}; diff --git a/lib/v2/furstar/templates/author.art b/lib/v2/furstar/templates/author.art new file mode 100644 index 0000000000..3d0443e32b --- /dev/null +++ b/lib/v2/furstar/templates/author.art @@ -0,0 +1,8 @@ +
+ + {{ if link !== null }} + {{name}} + {{ else }} + {{name}} + {{ /if }} +
\ No newline at end of file diff --git a/lib/v2/furstar/templates/description.art b/lib/v2/furstar/templates/description.art new file mode 100644 index 0000000000..8e8ad92fb2 --- /dev/null +++ b/lib/v2/furstar/templates/description.art @@ -0,0 +1,5 @@ +

{{ desc }}

+{{ each pics }} + +{{/each}} +{{ author }} \ No newline at end of file diff --git a/lib/routes/furstar/utils.js b/lib/v2/furstar/utils.js similarity index 86% rename from lib/routes/furstar/utils.js rename to lib/v2/furstar/utils.js index 2cad3582eb..04e48f6f02 100644 --- a/lib/routes/furstar/utils.js +++ b/lib/v2/furstar/utils.js @@ -1,18 +1,19 @@ const cheerio = require('cheerio'); const got = require('@/utils/got'); +const { art } = require('@/utils/render'); +const path = require('path'); + const base = 'https://furstar.jp'; + const langBase = (lang) => (lang ? `${base}/${lang}` : base); // en, cn, (none, for JP) -const renderAuthor = (author) => `
- - ${author.name} -
`; - -const renderDesc = (desc, pics, author) => { - const rpics = pics.map((e) => ``).join('\n'); - const rauthor = renderAuthor(author); - return `

${desc}

${rpics}${rauthor}`; -}; +const renderAuthor = (author) => art(path.join(__dirname, 'templates/author.art'), author); +const renderDesc = (desc, pics, author) => + art(path.join(__dirname, 'templates/description.art'), { + desc, + pics, + author: renderAuthor(author), + }); const authorDetail = (el) => { const $ = cheerio.load(el); diff --git a/lib/routes/test/index.js b/lib/v2/test/index.js similarity index 100% rename from lib/routes/test/index.js rename to lib/v2/test/index.js diff --git a/lib/v2/test/maintainer.js b/lib/v2/test/maintainer.js new file mode 100644 index 0000000000..49378171dd --- /dev/null +++ b/lib/v2/test/maintainer.js @@ -0,0 +1,3 @@ +module.exports = { + '/:id': ['DIYgod', 'NeverBehave'], +}; diff --git a/lib/v2/test/router.js b/lib/v2/test/router.js new file mode 100644 index 0000000000..641c23df3a --- /dev/null +++ b/lib/v2/test/router.js @@ -0,0 +1,3 @@ +module.exports = function (router) { + router.get('/:id', require('./index')); +}; diff --git a/lib/v2router.js b/lib/v2router.js new file mode 100644 index 0000000000..cf2b5cec90 --- /dev/null +++ b/lib/v2router.js @@ -0,0 +1,17 @@ +const dirname = __dirname + '/v2'; + +// 遍历整个 routes 文件夹，收集模块路由 router.js +const RouterPath = require('require-all')({ + dirname, + filter: /router\.js$/, +}); + +const routes = {}; + +// 将收集到的自定义模块路由进行合并 +for (const dir in RouterPath) { + const project = RouterPath[dir]['router.js']; // Do not merge other file + routes[dir] = project; +} + +module.exports = routes; diff --git a/lib/views/error.art b/lib/views/error.art index 831d7f35be..5f22bb4f89 100644 --- a/lib/views/error.art +++ b/lib/views/error.art @@ -39,6 +39,15 @@ 
Route requested: {{@ requestPath }}
Error message: {{@ message }}
+ 
+Helpful Information to provide when opening issue: 
+Path: {{@ errorPath }}
+Node version: {{@ nodeVersion}}
+
+ 
+如果您认为 RSSHub 导致了该错误，请在Github按照模版，复制本页面信息进行汇报
+If you believe this is an error caused by RSSHub, please report on github
+

在线文档与支持，请访问docs.rsshub.app。

diff --git a/package-lock.json b/package-lock.json index 7dab85150e..3b3728f9ca 100644 --- a/package-lock.json +++ b/package-lock.json @@ -76,10 +76,12 @@ "@vuepress/plugin-back-to-top": "1.8.2", "@vuepress/plugin-google-analytics": "1.8.2", "@vuepress/plugin-pwa": "1.8.2", + "ci-info": "^2.0.0", "cross-env": "7.0.3", "eslint": "7.29.0", "eslint-config-prettier": "8.3.0", "eslint-plugin-prettier": "3.4.0", + "fs-extra": "^8.1.0", "jest": "26.6.3", "mockdate": "3.0.5", "nock": "13.0.11", @@ -94,6 +96,7 @@ "staged-git-files": "1.2.0", "string-width": "4.2.2", "supertest": "6.1.3", + "tosource": "2.0.0-alpha.3", "vuepress": "1.8.2", "yorkie": "2.0.0" } @@ -22930,6 +22933,15 @@ "x-ray-scraper": "^3.0.5" } }, + "node_modules/tosource": { + "version": "2.0.0-alpha.3", + "resolved": "https://registry.npmjs.org/tosource/-/tosource-2.0.0-alpha.3.tgz", + "integrity": "sha512-KAB2lrSS48y91MzFPFuDg4hLbvDiyTjOVgaK7Erw+5AmZXNq4sFRVn8r6yxSLuNs15PaokrDRpS61ERY9uZOug==", + "dev": true, + "engines": { + "node": ">=10" + } + }, "node_modules/touch": { "version": "3.1.0", "resolved": "https://registry.npmjs.org/touch/-/touch-3.1.0.tgz", @@ -45089,6 +45101,12 @@ "x-ray-scraper": "^3.0.5" } }, + "tosource": { + "version": "2.0.0-alpha.3", + "resolved": "https://registry.npmjs.org/tosource/-/tosource-2.0.0-alpha.3.tgz", + "integrity": "sha512-KAB2lrSS48y91MzFPFuDg4hLbvDiyTjOVgaK7Erw+5AmZXNq4sFRVn8r6yxSLuNs15PaokrDRpS61ERY9uZOug==", + "dev": true + }, "touch": { "version": "3.1.0", "resolved": "https://registry.npmjs.org/touch/-/touch-3.1.0.tgz", diff --git a/package.json b/package.json index 08ddc0d534..4f999715e4 100644 --- a/package.json +++ b/package.json @@ -12,12 +12,16 @@ "profiling": "NODE_ENV=production node --prof lib/index.js", "docs:dev": "vuepress dev docs", "docs:build": "vuepress build docs", + "build:all": "npm run build:radar && npm run build:maintainer", + "build:radar": "node scripts/workflow/build-radar.js", + "build:maintainer": "node scripts/workflow/build-maintainer.js", "lint": "eslint .", "format": "eslint \"**/*.js\" --fix && node docs/.format/format.js && prettier \"**/*.{js,json}\" --write", "format:staged": "eslint \"**/*.js\" --fix && node docs/.format/format.js --staged && pretty-quick --staged --verbose --pattern \"**/*.{js,json}\"", "format:check": "eslint \"**/*.js\" && prettier-check \"**/*.{js,json}\"", "test": "npm run format:check && cross-env NODE_ENV=test jest --coverage --runInBand --forceExit", - "jest": "cross-env NODE_ENV=test jest --runInBand --forceExit" + "jest": "cross-env NODE_ENV=test jest --runInBand --forceExit", + "jest:watch": "cross-env NODE_ENV=test jest --watch" }, "repository": { "type": "git", @@ -44,10 +48,12 @@ "@vuepress/plugin-back-to-top": "1.8.2", "@vuepress/plugin-google-analytics": "1.8.2", "@vuepress/plugin-pwa": "1.8.2", + "ci-info": "^2.0.0", "cross-env": "7.0.3", "eslint": "7.29.0", "eslint-config-prettier": "8.3.0", "eslint-plugin-prettier": "3.4.0", + "fs-extra": "^8.1.0", "jest": "26.6.3", "mockdate": "3.0.5", "nock": "13.0.11", @@ -62,9 +68,9 @@ "staged-git-files": "1.2.0", "string-width": "4.2.2", "supertest": "6.1.3", + "tosource": "2.0.0-alpha.3", "vuepress": "1.8.2", - "yorkie": "2.0.0", - "fs-extra": "^8.1.0" + "yorkie": "2.0.0" }, "dependencies": { "@koa/router": "10.0.0", diff --git a/scripts/workflow/build-maintainer.js b/scripts/workflow/build-maintainer.js new file mode 100644 index 0000000000..f57f488b8a --- /dev/null +++ b/scripts/workflow/build-maintainer.js @@ -0,0 +1,16 @@ +/** */ +const fs = require('fs'); +const path = require('path'); +const target = path.join(__dirname, '../../assets/build/maintainer.json'); +const maintainer = require(path.join(__dirname, '../../lib/maintainer.js')); + +const count = Object.keys(maintainer).length; +const uniqueMaintainer = new Set(); +Object.values(maintainer) + .flat() + .forEach((e) => uniqueMaintainer.add(e)); + +// eslint-disable-next-line no-console +console.log(`We have ${count} routes and maintained by ${uniqueMaintainer.size} contributors!`); + +fs.writeFileSync(target, JSON.stringify(maintainer, null, 4)); diff --git a/scripts/workflow/build-radar.js b/scripts/workflow/build-radar.js new file mode 100644 index 0000000000..822b33afe5 --- /dev/null +++ b/scripts/workflow/build-radar.js @@ -0,0 +1,6 @@ +const fs = require('fs'); +const path = require('path'); +const target = path.join(__dirname, '../../assets/build/radar-rules.js'); +const radar = require(path.join(__dirname, '../../lib/radar.js')); + +fs.writeFileSync(target, radar.toSource()); diff --git a/test/middleware/cache.js b/test/middleware/cache.js index eb66470c69..bae584f079 100644 --- a/test/middleware/cache.js +++ b/test/middleware/cache.js @@ -134,8 +134,8 @@ describe('cache', () => { it('redis with quit', async () => { process.env.CACHE_TYPE = 'redis'; server = require('../../lib/index'); - const client = require('../../lib/app').context.cache.client; - await client.quit(); + const { redisClient } = require('../../lib/app').context.cache.clients; + await redisClient.quit(); const request = supertest(server); const response1 = await request.get('/test/cache'); diff --git a/test/middleware/template.js b/test/middleware/template.js index 1ba6b4296c..3f454250af 100644 --- a/test/middleware/template.js +++ b/test/middleware/template.js @@ -60,8 +60,8 @@ describe('template', () => { it(`.json`, async () => { const response = await request.get('/test/1.json'); - expect(response.status).toBe(404); - expect(response.text).toMatch(/Error: JSON output had been removed/); + const responseXML = await request.get('/test/1.rss'); + expect(response.text.slice(0, 50)).toEqual(responseXML.text.slice(0, 50)); }); it(`long title`, async () => {