Update Kunyu Version 1.7.0

CHANGELOG

@@ -4,6 +4,10 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [v1.7.0] - 2021-3-18
+### Added
+- Part of the code has been refactored to optimize the program structure
+- Added "PupilSearch" command
 
 ## [v1.6.5] - 2021-2-25
 ### Added

README.md

@@ -55,7 +55,7 @@ kunyu init --apikey <your zoomeye key> --seebug <your seebug key>
 ```
 ![](./images/setinfo.png)
 
-You need to log in with ZoomEye credentials before using this tool for information collection.
+The first time you use it, you need to use the ZoomEye login credentials to use this tool to collect information.Currently, ZoomEye registered users are given 1w query quota every month, which is enough for daily work.
 
 ZoomEye access address: https://www.zoomeye.org/
 
@@ -95,14 +95,14 @@ Global commands:
         Seebug <query>                            Search Seebug vulnerability information
         set <option>                              Set Global arguments values
         view/views <ID>                           Look over banner row data information
-        SearchKeyWord                             Query sensitive information by keyword
+        PupilSearch <URL>/<ID>                    Example Query sensitive interfaces
         Pocsuite3                                 Invoke the pocsuite component
         ExportPath                                Returns the path of the output file
         CreateMap                                 Generate an IP distribution heat map
         AliveScan                                 The viability of the last retrieval
         clear                                     Clear the console screen
         help                                      Print Help info
-        exit                                      Exit KunYu & 
+        exit                                      Exit KunYu &
 ```
 
 **OPTIONS**
@@ -114,6 +114,12 @@ ZoomEye:
 	stype <v4/v6>					stype <v4/v6> Set to get data type IPV4 or IPV6
 	btype <host/web> 				Set the API interface for batch query
 	timeout <num>					Set the timeout period of Kunyu HTTP request
+	thread  						Set PupilSearch Thread Number(default is 10)
+	deep   							Set PupilSearch Search Deep(default is 2)
+	all  							PupilSearch Add All Url To Check List
+	fuzz   							PupilSearch Add Api To Check List
+    proxy  							PupilSearch HTTP Proxy
+
 ```
 
 ## Use case introduction
@@ -196,11 +202,21 @@ Command format: **views ID**
 
 ![](./images/views.png)
 
-**Collection of Sensitive Information** 
+**PupilSearch Sensitive Information Collection**
+
+After Kunyu v1.7.0, the KeyWord command was removed and replaced with PupilSearch, which is the function of extracting sensitive data. Of course, it also supports the extraction of historical banner information through spatial mapping. For example, such as accesskey, the banner in historical data leaks sensitive data. Information, even if the service is changed now, but the AK/SK has not expired, it can still be used directly, understand everything, and support the extraction of sensitive information **(ID number, IP, JWT, API interface, appid, appkey, GithubAccessKey, default username \password, email, etc.)**.
+
+**Command format:**
+
+PupilSearch https://www.domain.com/
+
+PupilSearch ID (extract sensitive information from the banner returned by spatial mapping)
+
+![](./images/pupilsearch_1.png)
 
-After Kunyu v1.6.0, the acquisition of sensitive information in the banner has been added. Normally use the relevant grammar and set the number of pages. Kunyu will automatically collect the sensitive data in the banner information of the last query result, and then use the SearchKeyWord command to view the result . **Currently, testing will continue to focus on this feature point**. 
+![](./images/pupilsearch_2.png)
 
-![](./images/keyword.png)
+![](./images/pupilsearch_3.png)
 
 **System command execution** 
 

doc/README_CN.md

@@ -56,7 +56,7 @@ kunyu init --apikey <your zoomeye key> --seebug <your seebug key>
 ```
 ![](../images/setinfo.png)
 
-初次使用需要通过ZoomEye登录凭证，才使用该工具进行信息收集。
+初次使用需要通过ZoomEye登录凭证，才使用该工具进行信息收集，目前ZoomEye注册用户每月赠送1w条查询额度，足够日常工作使用。
 
 **ZoomEye访问地址：https://www.zoomeye.org/**
 
@@ -92,14 +92,16 @@ Global commands:
         SearchDomain <Domain>                     Domain name associated/subdomain search
         EncodeHash <encryption> <query>           Encryption method interface 
         HostCrash <IP> <Domain>                   Host Header Scan hidden assets
+        show <config>/<rule>                      Show can set options or Kunyu config
         Seebug <query>                            Search Seebug vulnerability information
         set <option>                              Set Global arguments values
-        view/views <ID>                           Look over http/ssl row data information
-        SearchKeyWord                             Query sensitive information by keyword
+        view/views <ID>                           Look over banner row data information
+        PupilSearch <URL>/<ID>                    Example Query sensitive interfaces
         Pocsuite3                                 Invoke the pocsuite component
         ExportPath                                Returns the path of the output file
+        CreateMap                                 Generate an IP distribution heat map
+        AliveScan                                 The viability of the last retrieval
         clear                                     Clear the console screen
-        show                                      Show can set options
         help                                      Print Help info
         exit                                      Exit KunYu &
 ```
@@ -108,11 +110,16 @@ Global commands:
 
 ```
 ZoomEye:
-	page <Number>    查询返回页数(默认查询一页，每页20条数据)
-	dtype <0/1>      查询关联域名/子域名(设置0为查询关联域名，反之为子域名)
-	stype <v4/v6>	 设置获取数据类型IPV4或IPV6，默认为 ipv4,ipv6 全选
-	btype <host/web> 设置批量查询的API接口(默认为HOST)
-	timeout <num>	 设置Kunyu HTTP请求的超时时间
+		page <Number>    			查询返回页数(默认查询一页，每页20条数据)
+		dtype <0/1>      			查询关联域名/子域名(设置0为查询关联域名，反之为子域名)
+		stype <v4/v6>	 			设置获取数据类型IPV4或IPV6，默认为 ipv4,ipv6 全选
+		btype <host/web> 			设置批量查询的API接口(默认为HOST)
+		timeout <num>	 			设置Kunyu HTTP请求的超时时间
+		thread  		 			设置PupilSearch线程数量(默认为10)
+		deep   			 			设置PupilSearch递归深度(默认为2)
+		all  			 			PupilSearch Add All Url To Check List
+		fuzz   			 			PupilSearch Add Api To Check List
+    	proxy  			 			PupilSearch HTTP Proxy
 ```
 
 ## 使用案例
@@ -195,11 +202,21 @@ SearchIcon /root/favicon.ico
 
 ![](../images/views.png)
 
-**敏感信息收集**
+**PupilSearch敏感信息收集**
 
-在Kunyu v1.6.0版本后，增加了对banner中敏感信息的获取，平时使用正常使用相关语法，设置页数，Kunyu会自动收集上一次查询结果banner信息中的敏感数据，然后通过SearchKeyWord命令查看结果。**目前将持续测试关注该功能点**。
+在Kunyu v1.7.0版本后，移除了KeyWord命令替换为PupilSearch，就是提取敏感数据的功能，当然也支持通过空间测绘提取历史banner信息，举个例子像accesskey这种，历史数据中banner泄露了敏感信息，哪怕现在换了服务，但是AK/SK没有过期，依旧可以直接利用，懂得都懂，支持提取敏感信息**（身份证号、IP、JWT、API接口、appid、appkey、GithubAccessKey，default username\password、邮箱等）**。
 
-![](../images/keyword.png)
+命令格式：
+
+**PupilSearch https://www.domain.com/**
+
+**PupilSearch ID	(通过空间测绘返回的banner提取敏感信息)**
+
+![](../images/pupilsearch_1.png)
+
+![](../images/pupilsearch_2.png)
+
+![](../images/pupilsearch_3.png)
 
 **系统命令执行**
 

kunyu/config/__init__.py

@@ -1,9 +1,9 @@
 #!/usr/bin/env python
 # encoding: utf-8
-'''
+"""
 @author: 风起
 @contact: [email protected]
 @File: __init__.py
 @Time: 2021/6/15 17:50
-'''
+"""
 

kunyu/config/__version__.py

@@ -15,7 +15,7 @@
 __python_version__ = sys.version.split()[0]
 __platform__ = platform.platform()
 __url__ = "https://github.com/knownsec/Kunyu"
-__version__ = '1.6.5'
+__version__ = '1.7.0'
 __author__ = '风起'
 __Team__ = 'KnownSec 404 Team'
 __author_email__ = '[email protected]'

kunyu/config/setting.py

@@ -52,6 +52,8 @@
     r'([a-zA-Z0-9][-_.a-zA-Z0-9]{0,61}[a-zA-Z0-9])).'\
     r'([a-zA-Z]{2,13}|[a-zA-Z0-9-]{2,30}.[a-zA-Z]{2,3})$'
 
+NUMBER_CHECK_REGEX = "^\+?[1-9][0-9]*$"
+
 UA = [
         "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_8; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",
         "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50",

kunyu/core/PupilMain.py

@@ -0,0 +1,85 @@
+"""
+@author: 风起
+@contact: [email protected]
+@File: export.py
+@Time: 2022/3/18 15:53
+"""
+
+import re
+import time
+
+from rich.console import Console
+
+from kunyu.lib import TrackUrl
+from kunyu.utils.log import logger
+import kunyu.lib.SerachData as SerachData
+from kunyu.lib.GlobalVariable import globalVariables as gbv
+
+console = Console(color_system="auto", record=True)
+
+class ParaInit:
+    def __init__(self, url, thread, trackdeep, mode, fuzz):
+        # There should be some / not less, for example,
+        # the target domain name is aaa.com/sso/, and it cannot be lazy as aaa.com/sso
+        self.url = url
+        gbv.url=url
+        gbv.thread = thread
+        gbv.apiresult = []
+        gbv.track = []  # add dynamically
+        # 0 means to search only for the current url without backtracking other urls,
+        # 1 means backtracking the url of the result obtained from the current url,
+        # 2 means to continue
+        self.trackdeep = trackdeep
+        gbv.urlresult = []
+        # Whether to traverse all URLs found. Choose carefully, it's a bit long
+        gbv.searchany = mode
+        # Whether to add the interface to the backtracking queue
+        gbv.fuzz = fuzz
+        gbv.trackhistory = []  # Record already processed urls to prevent repeated queries
+        gbv.Sensitiveinformation = {"Jwt": [], "Ip": [], "Email": [], "ChinaIdCard": [], "AccessKey": [],
+                                    "SecretKey": [], "AppId": [], "UserName": [], "PassWord": [],
+                                    "SSHKey": [], "RSAKey": [], "GithubAccessKey": []}
+        # Backtracking depth
+        gbv.trackdeep = 0  # current depth
+        gbv.deep = trackdeep  # allow depth
+        gbv.track.append([])
+        gbv.track[0].append(url)
+
+    def url_handler(self):
+        # Fill in the url, the url cannot be www.aaa.com, it needs to be www.aaa.com/,
+        # because the data of the first and last / will be deleted later for js concatenation.
+        # If it is not added, the host will be removed.
+        pattern = re.compile('/')
+        if len(re.findall(pattern, self.url[:])) == 2:
+            self.url = self.url + "/"
+
+    def main(self):
+        self.url_handler()
+        return self.url, self.trackdeep
+
+
+class Pupil:
+    def __init__(self) -> None:
+        console.log("PupilSearch KeyWord Start:", style="green")
+
+    def main(self, proxy):
+        start_time = time.time()
+        gbv.proxy = proxy
+        TrackUrl.Track().main()
+        logger.info("PupilSearch Total time:{}".format(time.time() - start_time))
+
+    def response_main(self, raw_data):
+        start_time = time.time()
+        SerachData.DataHandler("http://www.kunyu.com/", raw_data).main()
+        if gbv.apiresult:
+            logger.warning("Retrieving results API:")
+            console.print(list(set(gbv.apiresult)))
+            print("")
+        if gbv.urlresult:
+            logger.warning("Retrieving results URL:")
+            console.print((list(set(gbv.urlresult))))
+            print("")
+        if gbv.Sensitiveinformation:
+            logger.warning("Retrieving results KeyWord:")
+            console.print(gbv.Sensitiveinformation)
+        logger.info("PupilSearch Total time:{}".format(time.time() - start_time))

kunyu/core/__init__.py

@@ -7,7 +7,6 @@
 @Time: 2021/6/21 16:26
 '''
 
-
 import os
 import sys
 import json