+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/SONiC Foundation Technical Charter.pdf b/SONiC Foundation Technical Charter.pdf
new file mode 100644
index 0000000000..019c42d0ef
Binary files /dev/null and b/SONiC Foundation Technical Charter.pdf differ
diff --git a/Supported-Devices-and-Platforms.html b/Supported-Devices-and-Platforms.html
index 53d8a72eef..fe11a034c5 100644
--- a/Supported-Devices-and-Platforms.html
+++ b/Supported-Devices-and-Platforms.html
@@ -551,7 +551,7 @@
Innovium
+
Marvell
Teralynx 7
128x100G
@@ -749,7 +749,7 @@
Innovium
+
Marvell
Teralynx 7
32x400G
diff --git a/assets/img/ONEEF_Logo_White_nogradient.png b/assets/img/ONEEF_Logo_White_nogradient.png
new file mode 100644
index 0000000000..7e0014ad06
--- /dev/null
+++ b/assets/img/ONEEF_Logo_White_nogradient.png
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/assets/img/all_partners2_1920x1320.jpg b/assets/img/all_partners2_1920x1320.jpg
index 096450dc4b..da9bd237b3 100644
Binary files a/assets/img/all_partners2_1920x1320.jpg and b/assets/img/all_partners2_1920x1320.jpg differ
diff --git a/assets/img/lf_logo.svg b/assets/img/lf_logo.svg
new file mode 100644
index 0000000000..f043eef33e
--- /dev/null
+++ b/assets/img/lf_logo.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/assets/img/slider/2.jpg b/assets/img/slider/2.jpg
index c94dce7e06..c7e45be8c9 100644
Binary files a/assets/img/slider/2.jpg and b/assets/img/slider/2.jpg differ
diff --git a/assets/img/slider/3.jpg b/assets/img/slider/3.jpg
new file mode 100644
index 0000000000..307e6f53ab
Binary files /dev/null and b/assets/img/slider/3.jpg differ
diff --git a/assets/img/sonic_fund_1.jpg b/assets/img/sonic_fund_1.jpg
new file mode 100644
index 0000000000..977bf90073
Binary files /dev/null and b/assets/img/sonic_fund_1.jpg differ
diff --git a/doc/Dump-Utility.md b/doc/Dump-Utility.md
index 940017af13..5ed9398dc3 100644
--- a/doc/Dump-Utility.md
+++ b/doc/Dump-Utility.md
@@ -18,10 +18,13 @@
* [2.3 Helper Methods Available](#23-helper-methods-available)
* [2.4 Match Infrastructure](#24-Match-Infrastructure)
* [2.5 MatchRequest Examples](#25-matchrequest-examples)
- * [3. Unit Tests](#3-unit-tests)
- * [4. TechSupport](#4-techsupport)
+ * [3. Modules](#3-modules)
+ * [3.1 ACL Table](#31-acl-table)
+ * [3.2 ACL Rule](#32-acl-rule)
+ * [4. Unit Tests](#4-unit-tests)
+ * [5. TechSupport](#5-techsupport)
-### Revision
+### Revision
| Rev | Date | Author | Change Description |
|:---:|:-----------:|:-------------------------|:----------------------|
@@ -571,7 +574,7 @@ To add a new module, these guidelines have to be followed.
}
}
Note: DB_NAME could be either of ["CONFIG_DB", "APPL_DB", "ASIC_DB", "STATE_DB", ....] or "CONFIG_FILE".
-COPP is a example which will have a entry "CONFIG_FILE". This entry is required for COPP because, the default COPP entries are read from copp_cfg.json by coppmgr and are not present in the CONFIG_DB.
+COPP is a example which will have a entry "CONFIG_FILE". This entry is required for COPP because, the default COPP entries are read from copp_cfg.json by coppmgr and are not present in the CONFIG_DB.
Any extra entries which are added by the user are present in the CONFIG_DB. Hence it is required to have an extra "CONFIG_FILE" option.
```
@@ -639,7 +642,7 @@ class Port(Executor):
```
1) display_template(dbs=['CONFIG_DB', 'APPL_DB', 'ASIC_DB', 'STATE_DB']): Returns a dictionary of format JSON Template 1
2) MatchEngine / MatchRequest: Provided to abstract the heavy lifting in fetching the required data from redis-db/config-files. More info in the next section.
-3) verbose_print(str_): prints to the stdout based on verbosity provided by the user.
+3) verbose_print(str_): prints to the stdout based on verbosity provided by the user.
4) handle_error(err_str, excep=False): Prints the error output to stdout, if any experienced by the module, Set excep = True, to raise an exception
5) handle_multiple_keys_matched_error(err_str, key_to_go_with="", excep=False): When a filtering criteria specified by the module matches multiple keys, wherein it is expected to match ony one, this method can be used.
```
@@ -657,7 +660,7 @@ To Abstract this functionality out, a MatchEngine class is created. A MatchReque
{
"Table": "", # Mandatory, A Valid Table Name
"key_pattern": "", # Optional, Defaults to "*". Eg: "*" will match all the keys.
- # tablekey_pattern is directly applied for filtering entries
+ # tablekey_pattern is directly applied for filtering entries
"field": "", # Optional, Defaults to None
"value": "", # Optional, Value to match, Defaults to None
"return_fields": [
@@ -668,8 +671,8 @@ To Abstract this functionality out, a MatchEngine class is created. A MatchReque
# Only one of the db/file fields should have a non-empty string.
"just_keys": "true|false" # Mandatory, if true, Only Returns the keys matched. Does not return field-value pairs. Defaults to True
"ns" : DEFAULT_NAMESPACE # namespace argument, if nothing is provided, default namespace is used
- "match_entire_list" : False # Some of the fields in redis can consist of multiple values eg: trap_ids = "bgp,bgpv6,ospf".
- When this arg is set to true, entire list is matched incluing the ",".
+ "match_entire_list" : False # Some of the fields in redis can consist of multiple values eg: trap_ids = "bgp,bgpv6,ospf".
+ When this arg is set to true, entire list is matched incluing the ",".
When False, the values are split based on "," and individual items are matched with
}
```
@@ -684,7 +687,7 @@ To Abstract this functionality out, a MatchEngine class is created. A MatchReque
###### JSON Template 3: Return Dictionary by the MatchEngine:
```
-{
+{
"error": "", # Error String, if any. Empty Otherwise
"keys": [], # Match found for the request
"return_values": {} # Return Values for the corresponding return_fields passed
@@ -705,13 +708,13 @@ Possible Error strings returned by the MatchEngine.
9) "Field is provided, but no value is provided to compare with"
10) "When Just_keys is set to False, return_fields should be empty"
11) "Return Fields should be of list type"
-
+
Other Errors:
12) "No Entries found for Table|key_pattern provided.
13) "Connection Error"
14) "No Keys found after applying the filtering criteria"
- Note: Run the "dump state arg" command with -v option to print these errors any other exceptions thrown to the stdout.
+ Note: Run the "dump state arg" command with -v option to print these errors any other exceptions thrown to the stdout.
```
@@ -728,10 +731,10 @@ req.just_keys = True
req.return_fields = []
Return Dict:
-{
- "error": "",
- "keys": ["ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x1500000000052f"],
- "return_values": {}
+{
+ "error": "",
+ "keys": ["ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x1500000000052f"],
+ "return_values": {}
}
2) Fetch the entry for ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x150000000002cf from ASIC_DB (Keys + Field-Value Pairs)
@@ -744,13 +747,13 @@ req.just_keys = False
req.return_fields = []
Return Dict:
-{
- "error": "",
+{
+ "error": "",
"keys": [{"ASIC_STATE:SAI_OBJECT_TYPE_QUEUE:oid:0x1500000000052f": {
"NULL": "NULL",
"SAI_QUEUE_ATTR_TYPE": "SAI_QUEUE_TYPE_UNICAST",
- "SAI_QUEUE_ATTR_INDEX": "4"}],
- "return_values": {}
+ "SAI_QUEUE_ATTR_INDEX": "4"}],
+ "return_values": {}
}
@@ -765,10 +768,10 @@ req.db = "ASIC_DB"
req.return_fields = ["SAI_HOSTIF_TRAP_ATTR_TRAP_GROUP"]
Return Dict:
-{
- "error": "",
- "keys": ["ASIC_STATE:SAI_OBJECT_TYPE_HOSTIF_TRAP:oid:0x22000000000592"],
- "return_values": {"ASIC_STATE:SAI_OBJECT_TYPE_HOSTIF_TRAP:oid:0x22000000000592" : {"SAI_HOSTIF_TRAP_ATTR_TRAP_GROUP" : "oid:0x11000000000591"}}
+{
+ "error": "",
+ "keys": ["ASIC_STATE:SAI_OBJECT_TYPE_HOSTIF_TRAP:oid:0x22000000000592"],
+ "return_values": {"ASIC_STATE:SAI_OBJECT_TYPE_HOSTIF_TRAP:oid:0x22000000000592" : {"SAI_HOSTIF_TRAP_ATTR_TRAP_GROUP" : "oid:0x11000000000591"}}
}
4) Fetch the entry for COPP_TRAP table which has trap_id sample_packet from the copp_cfg.json file
@@ -782,14 +785,213 @@ req.file = "/etc/sonic/copp_cfg.json"
req.return_fields = ["trap_group"]
Return Dict:
-{
- "error": "",
- "keys": ["COPP_TRAP|sflow"],
- "return_values": {"COPP_TRAP|sflow" : {"trap_group" : "queue2_group1"}}
+{
+ "error": "",
+ "keys": ["COPP_TRAP|sflow"],
+ "return_values": {"COPP_TRAP|sflow" : {"trap_group" : "queue2_group1"}}
}
```
-## 3 **Unit Tests**:
+## 3 **Modules**:
+
+### 3.1 ACL Table
+
+ACL Tables configuration resides in CONFIG DB in table *ACL_TABLE*. Tables in *ACL_TABLE* may reference another table *ACL_TABLE_TYPE* that holds the table type configuration or may use one of default table types (L3, L3V6, MIRROR, etc.). This information is dumped from CONFIG DB. Orchagent processes *ACL_TABLE* and *ACL_TABLE_TYPE* configuration and manages the following ASIC resources:
+ - *SAI_OBJECT_TYPE_ACL_TABLE*
+ - *SAI_OBJECT_TYPE_ACL_TABLE_GROUP*
+ - *SAI_OBJECT_TYPE_ACL_TABLE_GROUP_MEMBER*.
+
+ There is no direct mapping of CONFIG DB ACL tables to ASIC DB objects. However, for dump utility purposes, we can correlate *ACL_TABLE* and *SAI_OBJECT_TYPE_ACL_TABLE* if there is at least one ACL rule in the table. We may use the *ACL_COUNTER_RULE_MAP* in COUNTERS DB in order to find the *SAI_OBJECT_TYPE_ACL_COUNTER* and extract the table OID from it. It is done using the following schema:
+
+```mermaid
+flowchart LR
+ ACL_TABLE --> B{Has at least 1 rule}
+ B --> |Yes| ACL_COUNTER_RULE_MAP
+ B --> |No| C[No entries in ASIC DB dump]
+ ACL_COUNTER_RULE_MAP --> COUNTER_OID
+ COUNTER_OID --> TABLE_OID
+ TABLE_OID --> TABLE_GROUP_MEMBER
+ TABLE_GROUP_MEMBER --> TABLE_GROUP
+```
+
+```
+admin@sonic:~$ dump state acl_table DATA_L3 -t
++------------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| acl_table_name | DB_NAME | DUMP |
++==================+===========+===========================================================================================================================================================================================================+
+| DATA_L3 | CONFIG_DB | +-------------------+-----------------------------+ |
+| | | | Keys | field-value pairs | |
+| | | +===================+=============================+ |
+| | | | ACL_TABLE|DATA_L3 | +-------------+-----------+ | |
+| | | | | | field | value | | |
+| | | | | |-------------+-----------| | |
+| | | | | | policy_desc | DATA_L3 | | |
+| | | | | | ports | Ethernet0 | | |
+| | | | | | | Ethernet4 | | |
+| | | | | | stage | ingress | | |
+| | | | | | type | L3 | | |
+| | | | | +-------------+-----------+ | |
+| | | +-------------------+-----------------------------+ |
++------------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+| DATA_L3 | ASIC_DB | +-----------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------+ |
+| | | | Keys | field-value pairs | |
+| | | +=======================================================================+===============================================================================================================================+ |
+| | | | ASIC_STATE:SAI_OBJECT_TYPE_ACL_TABLE:oid:0x7000000000600 | +---------------------------------------------+-----------------------------------------------------------------------------+ | |
+| | | | | | field | value | | |
+| | | | | |---------------------------------------------+-----------------------------------------------------------------------------| | |
+| | | | | | SAI_ACL_TABLE_ATTR_ACL_BIND_POINT_TYPE_LIST | 2:SAI_ACL_BIND_POINT_TYPE_PORT,SAI_ACL_BIND_POINT_TYPE_LAG | | |
+| | | | | | SAI_ACL_TABLE_ATTR_ACL_STAGE | SAI_ACL_STAGE_INGRESS | | |
+| | | | | | SAI_ACL_TABLE_ATTR_FIELD_ACL_IP_TYPE | true | | |
+| | | | | | SAI_ACL_TABLE_ATTR_FIELD_ACL_RANGE_TYPE | 2:SAI_ACL_RANGE_TYPE_L4_SRC_PORT_RANGE,SAI_ACL_RANGE_TYPE_L4_DST_PORT_RANGE | | |
+| | | | | | SAI_ACL_TABLE_ATTR_FIELD_DST_IP | true | | |
+| | | | | | SAI_ACL_TABLE_ATTR_FIELD_ETHER_TYPE | true | | |
+| | | | | | SAI_ACL_TABLE_ATTR_FIELD_ICMP_CODE | true | | |
+| | | | | | SAI_ACL_TABLE_ATTR_FIELD_ICMP_TYPE | true | | |
+| | | | | | SAI_ACL_TABLE_ATTR_FIELD_IP_PROTOCOL | true | | |
+| | | | | | SAI_ACL_TABLE_ATTR_FIELD_L4_DST_PORT | true | | |
+| | | | | | SAI_ACL_TABLE_ATTR_FIELD_L4_SRC_PORT | true | | |
+| | | | | | SAI_ACL_TABLE_ATTR_FIELD_OUTER_VLAN_ID | true | | |
+| | | | | | SAI_ACL_TABLE_ATTR_FIELD_SRC_IP | true | | |
+| | | | | | SAI_ACL_TABLE_ATTR_FIELD_TCP_FLAGS | true | | |
+| | | | | +---------------------------------------------+-----------------------------------------------------------------------------+ | |
+| | | +-----------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------+ |
+| | | | ASIC_STATE:SAI_OBJECT_TYPE_ACL_TABLE_GROUP_MEMBER:oid:0xc000000000602 | +----------------------------------------------------+---------------------+ | |
+| | | | | | field | value | | |
+| | | | | |----------------------------------------------------+---------------------| | |
+| | | | | | SAI_ACL_TABLE_GROUP_MEMBER_ATTR_ACL_TABLE_GROUP_ID | oid:0xb0000000005f7 | | |
+| | | | | | SAI_ACL_TABLE_GROUP_MEMBER_ATTR_ACL_TABLE_ID | oid:0x7000000000600 | | |
+| | | | | | SAI_ACL_TABLE_GROUP_MEMBER_ATTR_PRIORITY | 100 | | |
+| | | | | +----------------------------------------------------+---------------------+ | |
+| | | +-----------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------+ |
+| | | | ASIC_STATE:SAI_OBJECT_TYPE_ACL_TABLE_GROUP_MEMBER:oid:0xc000000000601 | +----------------------------------------------------+---------------------+ | |
+| | | | | | field | value | | |
+| | | | | |----------------------------------------------------+---------------------| | |
+| | | | | | SAI_ACL_TABLE_GROUP_MEMBER_ATTR_ACL_TABLE_GROUP_ID | oid:0xb0000000005f5 | | |
+| | | | | | SAI_ACL_TABLE_GROUP_MEMBER_ATTR_ACL_TABLE_ID | oid:0x7000000000600 | | |
+| | | | | | SAI_ACL_TABLE_GROUP_MEMBER_ATTR_PRIORITY | 100 | | |
+| | | | | +----------------------------------------------------+---------------------+ | |
+| | | +-----------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------+ |
+| | | | ASIC_STATE:SAI_OBJECT_TYPE_ACL_TABLE_GROUP:oid:0xb0000000005f7 | +---------------------------------------------------+-----------------------------------+ | |
+| | | | | | field | value | | |
+| | | | | |---------------------------------------------------+-----------------------------------| | |
+| | | | | | SAI_ACL_TABLE_GROUP_ATTR_ACL_BIND_POINT_TYPE_LIST | 1:SAI_ACL_BIND_POINT_TYPE_PORT | | |
+| | | | | | SAI_ACL_TABLE_GROUP_ATTR_ACL_STAGE | SAI_ACL_STAGE_INGRESS | | |
+| | | | | | SAI_ACL_TABLE_GROUP_ATTR_TYPE | SAI_ACL_TABLE_GROUP_TYPE_PARALLEL | | |
+| | | | | +---------------------------------------------------+-----------------------------------+ | |
+| | | +-----------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------+ |
+| | | | ASIC_STATE:SAI_OBJECT_TYPE_ACL_TABLE_GROUP:oid:0xb0000000005f5 | +---------------------------------------------------+-----------------------------------+ | |
+| | | | | | field | value | | |
+| | | | | |---------------------------------------------------+-----------------------------------| | |
+| | | | | | SAI_ACL_TABLE_GROUP_ATTR_ACL_BIND_POINT_TYPE_LIST | 1:SAI_ACL_BIND_POINT_TYPE_PORT | | |
+| | | | | | SAI_ACL_TABLE_GROUP_ATTR_ACL_STAGE | SAI_ACL_STAGE_INGRESS | | |
+| | | | | | SAI_ACL_TABLE_GROUP_ATTR_TYPE | SAI_ACL_TABLE_GROUP_TYPE_PARALLEL | | |
+| | | | | +---------------------------------------------------+-----------------------------------+ | |
+| | | +-----------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------+ |
+| | | +---------------------+---------------------+ |
+| | | | vid | rid | |
+| | | +=====================+=====================+ |
+| | | | oid:0x7000000000600 | oid:0x100000007 | |
+| | | +---------------------+---------------------+ |
+| | | | oid:0xc000000000602 | oid:0x100010000000c | |
+| | | +---------------------+---------------------+ |
+| | | | oid:0xc000000000601 | oid:0x10000000c | |
+| | | +---------------------+---------------------+ |
+| | | | oid:0xb0000000005f7 | oid:0x10000000b | |
+| | | +---------------------+---------------------+ |
+| | | | oid:0xb0000000005f5 | oid:0xb | |
+| | | +---------------------+---------------------+ |
++------------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+```
+
+### 3.2 ACL Rule
+
+ACL Rules configuration resides in CONFIG DB in table *ACL_RULE*. This information is dumped from CONFIG DB. Orchagent processes *ACL_RULE* configuration and manages the following ASIC resources:
+ - *SAI_OBJECT_TYPE_ACL_ENTRY*
+ - *SAI_OBJECT_TYPE_ACL_COUNTER*
+ - *SAI_OBJECT_TYPE_ACL_RANGE*
+
+The mapping from CONFIG DB to ASIC DB data is done using the following schema:
+
+```mermaid
+flowchart LR
+ ACL_RULE --> ACL_COUNTER_RULE_MAP
+ ACL_COUNTER_RULE_MAP --> COUNTER_OID
+ COUNTER_OID --> RULE_OID
+ RULE_OID --> RANGE_OID
+```
+
+*NOTE*: Mirror rules might be not present in ASIC DB depending on the mirror session state.
+
+Example:
+```
+admin@sonic:~$ dump state acl_rule 'DATA_L3|R1' -t
++-----------------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
+| acl_rule_name | DB_NAME | DUMP |
++=================+===========+==========================================================================================================================================================+
+| DATA_L3|R1 | CONFIG_DB | +---------------------+---------------------------------+ |
+| | | | Keys | field-value pairs | |
+| | | +=====================+=================================+ |
+| | | | ACL_RULE|DATA_L3|R1 | +-------------------+---------+ | |
+| | | | | | field | value | | |
+| | | | | |-------------------+---------| | |
+| | | | | | L4_DST_PORT_RANGE | 90-95 | | |
+| | | | | | L4_SRC_PORT_RANGE | 80-100 | | |
+| | | | | | PACKET_ACTION | FORWARD | | |
+| | | | | | PRIORITY | 9995 | | |
+| | | | | +-------------------+---------+ | |
+| | | +---------------------+---------------------------------+ |
++-----------------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
+| DATA_L3|R1 | ASIC_DB | +------------------------------------------------------------+-----------------------------------------------------------------------------------------+ |
+| | | | Keys | field-value pairs | |
+| | | +============================================================+=========================================================================================+ |
+| | | | ASIC_STATE:SAI_OBJECT_TYPE_ACL_COUNTER:oid:0x9000000000606 | +------------------------------------------+---------------------+ | |
+| | | | | | field | value | | |
+| | | | | |------------------------------------------+---------------------| | |
+| | | | | | SAI_ACL_COUNTER_ATTR_ENABLE_BYTE_COUNT | true | | |
+| | | | | | SAI_ACL_COUNTER_ATTR_ENABLE_PACKET_COUNT | true | | |
+| | | | | | SAI_ACL_COUNTER_ATTR_TABLE_ID | oid:0x7000000000600 | | |
+| | | | | +------------------------------------------+---------------------+ | |
+| | | +------------------------------------------------------------+-----------------------------------------------------------------------------------------+ |
+| | | | ASIC_STATE:SAI_OBJECT_TYPE_ACL_ENTRY:oid:0x8000000000609 | +-----------------------------------------+-------------------------------------------+ | |
+| | | | | | field | value | | |
+| | | | | |-----------------------------------------+-------------------------------------------| | |
+| | | | | | SAI_ACL_ENTRY_ATTR_ACTION_COUNTER | oid:0x9000000000606 | | |
+| | | | | | SAI_ACL_ENTRY_ATTR_ACTION_PACKET_ACTION | SAI_PACKET_ACTION_FORWARD | | |
+| | | | | | SAI_ACL_ENTRY_ATTR_ADMIN_STATE | true | | |
+| | | | | | SAI_ACL_ENTRY_ATTR_FIELD_ACL_RANGE_TYPE | 2:oid:0xa000000000607,oid:0xa000000000608 | | |
+| | | | | | SAI_ACL_ENTRY_ATTR_PRIORITY | 9995 | | |
+| | | | | | SAI_ACL_ENTRY_ATTR_TABLE_ID | oid:0x7000000000600 | | |
+| | | | | +-----------------------------------------+-------------------------------------------+ | |
+| | | +------------------------------------------------------------+-----------------------------------------------------------------------------------------+ |
+| | | | ASIC_STATE:SAI_OBJECT_TYPE_ACL_RANGE:oid:0xa000000000607 | +--------------------------+--------------------------------------+ | |
+| | | | | | field | value | | |
+| | | | | |--------------------------+--------------------------------------| | |
+| | | | | | SAI_ACL_RANGE_ATTR_LIMIT | 80,100 | | |
+| | | | | | SAI_ACL_RANGE_ATTR_TYPE | SAI_ACL_RANGE_TYPE_L4_SRC_PORT_RANGE | | |
+| | | | | +--------------------------+--------------------------------------+ | |
+| | | +------------------------------------------------------------+-----------------------------------------------------------------------------------------+ |
+| | | | ASIC_STATE:SAI_OBJECT_TYPE_ACL_RANGE:oid:0xa000000000608 | +--------------------------+--------------------------------------+ | |
+| | | | | | field | value | | |
+| | | | | |--------------------------+--------------------------------------| | |
+| | | | | | SAI_ACL_RANGE_ATTR_LIMIT | 90,95 | | |
+| | | | | | SAI_ACL_RANGE_ATTR_TYPE | SAI_ACL_RANGE_TYPE_L4_DST_PORT_RANGE | | |
+| | | | | +--------------------------+--------------------------------------+ | |
+| | | +------------------------------------------------------------+-----------------------------------------------------------------------------------------+ |
+| | | +---------------------+--------------------+ |
+| | | | vid | rid | |
+| | | +=====================+====================+ |
+| | | | oid:0x9000000000606 | oid:0x600000070009 | |
+| | | +---------------------+--------------------+ |
+| | | | oid:0x8000000000609 | oid:0x100010008 | |
+| | | +---------------------+--------------------+ |
+| | | | oid:0xa000000000607 | oid:0xa | |
+| | | +---------------------+--------------------+ |
+| | | | oid:0xa000000000608 | oid:0x20000000a | |
+| | | +---------------------+--------------------+ |
++-----------------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
+```
+
+## 4 **Unit Tests**:
| S.No | Test case synopsis |
|------|-----------------------------------------------------------------------------------------------------------------------------------------|
@@ -798,10 +1000,10 @@ Return Dict:
| 3 | Verify all the options in the CLI is working as expected |
| 4 | Verify the namespace arg is working as expected |
| 5 | Verify dump cli options are working as expected |
-| 6 | Unit tests should be added for every new module added |
+| 6 | Unit tests should be added for every new module added |
-## 4 **TechSupport**
+## 5 **TechSupport**
Output for every which extends from Executor class will be added to the techsupport dump.
Every Json file will have the corresponding output: `dump state all -k`. Output will be printed in JSON format for TechSupport Dumps.
Only the related keys information will be present in the unified_dump_folder as entire DB dumps are already present in the dump/folder.
@@ -815,9 +1017,9 @@ $BASE
├── port
├──
├── .......
- ├──
+ ├──
├──── etc/
- ├──── log/
+ ├──── log/
├──── ......
├──── ......
```
diff --git a/doc/auto_techsupport_and_coredump_mgmt.md b/doc/auto_techsupport_and_coredump_mgmt.md
index 45640d2423..2341abbd95 100644
--- a/doc/auto_techsupport_and_coredump_mgmt.md
+++ b/doc/auto_techsupport_and_coredump_mgmt.md
@@ -7,25 +7,29 @@
* [1. Overview](#1-overview)
* [2. High Level Requirements](#2-high-level-requirements)
* [3. Core Dump Generation in SONiC](#3-core-dump-generation-in-sonic)
- * [4. Schema Additions](#4-schema-additions)
- * [6. CLI Enhancements](#5-cli-enhancements)
+ * [4. Memory usage based techsupport invocation](#4-Memory-usage-based-techsupport-invocation)
+ * [5. Schema Additions](#5-schema-additions)
+ * [6. CLI Enhancements](#6-cli-enhancements)
* [7. Design](#6-design)
- * [6.1 Modifications to coredump-compress script](#61-Modifications-to-coredump-compress-script)
- * [6.2 coredump_gen_handler script](#62-coredump_gen_handler-script)
- * [6.3 Modifications to generate_dump script](#64-Modifications-to-generate-dump-script)
- * [6.4 techsupport_cleanup script](#65-techsupport_cleanup-script)
- * [6.5 Warmboot consideration](#65-Warmboot-consideration)
- * [6.6 MultiAsic consideration](#66-MultiAsic-consideration)
- * [6.7 Design choices for max-techsupport-limit & max-techsupport-limit arguments](#67-Design-choices-for-max-core-limit-&-max-techsupport-limit-arguments)
- * [8. Test Plan](#7-Test-Plan)
- * [9. SONiC-to-SONiC Upgrade Considerations](#8-SONiC-to-SONiC-Upgrade-Considerations)
+ * [7.1 Modifications to coredump-compress script](#71-Modifications-to-coredump-compress-script)
+ * [7.2 coredump_gen_handler script](#72-coredump_gen_handler-script)
+ * [7.3 Modifications to generate_dump script](#73-Modifications-to-generate-dump-script)
+ * [7.4 techsupport_cleanup script](#74-techsupport_cleanup-script)
+ * [7.5 Warmboot consideration](#75-Warmboot-consideration)
+ * [7.6 MultiAsic consideration](#76-MultiAsic-consideration)
+ * [7.7 Design choices for max-techsupport-limit & max-techsupport-limit arguments](#77-Design-choices-for-max-core-limit-&-max-techsupport-limit-arguments)
+ * [7.8 Techsupport Locking](#78-Techsupport-Locking)
+ * [8. Test Plan](#8-Test-Plan)
+ * [9. SONiC-to-SONiC Upgrade Considerations](#9-SONiC-to-SONiC-Upgrade-Considerations)
+ * [10. App Extension Consideration](#9-App-Extension-Considerations)
+ * [11. Open questions](#10-Open-questions)
### Revision
| Rev | Date | Author | Change Description |
|:---:|:-----------:|:-------------------------|:----------------------|
| 1.0 | 06/22/2021 | Vivek Reddy Karri | Auto Invocation of Techsupport, triggered by a core dump |
-| 1.1 | TBD | Vivek Reddy Karri | Add the capability to Register/Deregister app extension to AUTO_TECHSUPPORT_FEATURE table |
+| 1.1 | 04/08/2022 | Vivek Reddy Karri | Add the capability to Register/Deregister app extension to AUTO_TECHSUPPORT_FEATURE table |
| 2.0 | TBD | Vivek Reddy Karri | Extending Support for Kernel Dumps |
| 3.0 | 02/2022 | Stepan Blyshchak | Extending Support for memory usage threshold crossed |
@@ -380,13 +384,13 @@ sonic_dump_r-lionfish-16_20210901_222408 teamd python3.1630535045.34.
```
-## 6. Design
+## 7. Design
-### 6.1 Modifications to coredump-compress script
+### 7.1 Modifications to coredump-compress script
The coredump-compress script is updated to invoke the `coredump_gen_handler` script once it is done writing the core file to /var/core. Any stdout/stderr seen during the execution of `coredump_gen_handler` script is redirected to `/tmp/coredump_gen_handler.log`. This script is enhanced to determine which container the dump belongs to and passes it to the coredump_gen_handler script.
-### 6.2 coredump_gen_handler script
+### 7.2 coredump_gen_handler script
A script under the name `coredump_gen_handler.py` is added to `/usr/local/bin/` directory which will be invoked after a coredump is generated. The script first checks if this feature is enabled by the user. The script then verifies if a core dump file is created within the last 20 sec and if yes, it moves forward.
@@ -405,11 +409,11 @@ DATE sonic NOTICE coredump_gen_handler[pid]: coredump_cleanup is disabled. No c
DATE sonic ERR coredump_gen_handler[pid]: "show techsupport --since '2 days ago'" was run, but no techsupport dump is found
```
-### 6.3 Modifications to generate_dump script
+### 7.3 Modifications to generate_dump script
The generate_dump script is updated to invoke the `techsupport_cleanup` script to handle the cleanup of techsupport files. Any stdout/stderr seen during the execution of `techsupport_cleanup` script is redirected to `/tmp/coredump_gen_handler.log`
-### 6.4 techsupport_cleanup script
+### 7.4 techsupport_cleanup script
A script under the name `techsupport_cleanup.py` is added to `/usr/local/bin/` directory which will be invoked after a techsupport dump is created. The script first checks if the feature is enabled by the user. It then checks if the limit configured by the user has crossed and deletes the old techsupport files, if any.
@@ -419,17 +423,17 @@ DATE sonic NOTICE techsupport_cleanup[pid]: techsupport_cleanup is disabled. No
DATE sonic INFO coredump_gen_handler[pid]: max-techsupport-size argument is not set. No Cleanup is performed, current size occupied: 456 MB
```
-### 6.5 Warmboot consideration
+### 7.5 Warmboot consideration
No changes to this flow
-### 6.6 MultiAsic consideration
+### 7.6 MultiAsic consideration
Configuration specified for the default feature name in the AUTO_TECHSUPPORT_FEATURE table is applied across all the masic instances.
i.e. rate_limit_interval defined in the AUTO_TECHSUPPORT_FEATURE|swss key is applied for swss1, swss2, etc
-### 6.7 Design choices for max-techsupport-limit & max-techsupport-limit arguments
+### 7.7 Design choices for max-techsupport-limit & max-techsupport-limit arguments
Firstly, Size-based cleanup design was inspired from MaxUse= Argument in the systemd-coredump.conf https://www.freedesktop.org/software/systemd/man/coredump.conf.html
@@ -452,7 +456,15 @@ A default value of 5% would amount to a minimum of 500 MB which is a already a d
Although if the admin feels otherwise, these values are configurable.
-## 7. Test Plan
+### 7.8 Techsupport Locking
+
+Recently, an enhancement was made to techsupport script to only run one instance at a time by using a locking mechanism. When other script instance of techsupport tries to run, it'll exit with a relevent code. This would apply nevertheless of how a techsupport was invoked i.e. manual or through auto-techsupport.
+
+With this change, rate-limit-interval of zero would not make any difference. The locking mechanism would implicitly impose a minimum rate-limit-interval of techsupport execution time. And since, the techsupport execution time can't be found out and varies based on underlying machine and system state, the range of values configurable for the rate-limit-interval is left unchanged
+
+A relevant message will be logged to syslog when the invocation fails because of LOCKFAIL exit code.
+
+## 8. Test Plan
Enhance the existing techsupport sonic-mgmt test with the following cases.
@@ -462,96 +474,25 @@ Enhance the existing techsupport sonic-mgmt test with the following cases.
| 2 | Check if the techsupport cleanup is working as expected |
| 3 | Check if the global rate-& & per-process rate-limit-interval is working as expected |
| 4 | Check if the core-dump cleanup is working as expected |
-| 5 | Check if the core-dump generated when reaching memory threshold |
-## 8. SONiC-to-SONiC Upgrade Considerations
+| 5 | Check if the core-dump generated when reaching memory threshold |
-The default config required for auto_techsupport is present in the init_cfg.json. Therefore, when a clean installation of SONiC is performed, the configuration is found in the config DB and the feature is active.
+## 9. SONiC-to-SONiC Upgrade Considerations
-However, in the case of SONiC-SONiC upgrade, the previous config_db.json is migrated and init_cfg.json is not involved. In that case, it is the responsibility of the admin to provide the config, if the admin wants to leverage this feature.
+The configuration in the init_cfg.json is loaded to the running config i.e. CONFIG_DB even in the case of SONiC-SONiC upgrade from a older image which doesn't support this feature.
-Load this Example config provided below to enable the feature. Each of the fields are explained in Section 4 and can be modified accordingly
+### 10 App Extension Considerations
+
+Detailed Info related to Appliation Extension can be found here: https://github.com/Azure/SONiC/blob/master/doc/sonic-application-extension/sonic-application-extention-hld.md
+
+A new AUTO_TECHSUPPORT_FEATURE register/deregister option will be introduced. The existing FeatureRegistry class will be enahcned to also add/delete configuration related to AUTO_TECHSUPPORT_FEATURE table.
+
+This will be run when the application installs/uninstalls. Since, the auto-techsupport feature uses compile time flag to determine whether to enable/disable itself, it is not possible to determine that at runtime when the application is installed.
+
+Thus the decision to whether or not to enable the new feature will be based on the current values of AUTO_TECHSUPPORT & AUTO_TECHSUPPORT_FEATURE tables. The default value for new feature will be disabled if the global state is shown disabled in init_cfg.json. If not, the feature will be enabled. The rate-limit-interval & memory threshold is set to 600 & 10% by default.
+
+When the app get un-installed, all the config will be cleared unless keep-config option is used.
-```
-{
- "AUTO_TECHSUPPORT": {
- "GLOBAL": {
- "state": "enabled",
- "rate_limit_interval": "180",
- "max_techsupport_limit": "10.0",
- "max_core_limit": "5.0",
- "available_mem_threashold": "10.0",
- "since": "2 days ago"
- }
- },
- "AUTO_TECHSUPPORT_FEATURE": {
- "bgp": {
- "state": "enabled",
- "rate_limit_interval": "600"
- },
- "database": {
- "state": "enabled",
- "rate_limit_interval": "600"
- },
- "lldp": {
- "state": "enabled",
- "rate_limit_interval": "600"
- },
- "pmon": {
- "state": "enabled",
- "rate_limit_interval": "600"
- },
- "radv": {
- "state": "enabled",
- "rate_limit_interval": "600"
- },
- "snmp": {
- "state": "enabled",
- "rate_limit_interval": "600"
- },
- "swss": {
- "state": "enabled",
- "rate_limit_interval": "600"
- },
- "syncd": {
- "state": "enabled",
- "rate_limit_interval": "600"
- },
- "teamd": {
- "state": "enabled",
- "rate_limit_interval": "600"
- },
- "dhcp_relay": {
- "state": "enabled",
- "rate_limit_interval": "600"
- },
- "mgmt-framework": {
- "state": "enabled",
- "rate_limit_interval": "600"
- },
- "mux": {
- "state": "enabled",
- "rate_limit_interval": "600"
- },
- "nat": {
- "state": "enabled",
- "rate_limit_interval": "600"
- },
- "sflow": {
- "state": "enabled",
- "rate_limit_interval": "600"
- },
- "macsec": {
- "state": "enabled",
- "rate_limit_interval": "600"
- },
- "telemetry": {
- "state": "enabled",
- "rate_limit_interval": "600"
- }
- }
-}
-```
-# Open question
+## 11. Open questions
1. Is 10 % free memory/90 % used memory threshold a reasonable default?
diff --git a/doc/fast-reboot/BGP.svg b/doc/fast-reboot/BGP.svg
new file mode 100644
index 0000000000..7d640569cb
--- /dev/null
+++ b/doc/fast-reboot/BGP.svg
@@ -0,0 +1,4 @@
+
+
+
+
\ No newline at end of file
diff --git a/doc/fast-reboot/Fast-reboot_Flow_Improvements_HLD.md b/doc/fast-reboot/Fast-reboot_Flow_Improvements_HLD.md
new file mode 100644
index 0000000000..f7b044b82e
--- /dev/null
+++ b/doc/fast-reboot/Fast-reboot_Flow_Improvements_HLD.md
@@ -0,0 +1,229 @@
+# Fast-reboot Flow Improvements HLD
+
+# Table of Contents
+- [Fast-reboot Flow Improvements HLD](#fast-reboot-flow-improvements-hld)
+- [Table of Contents](#table-of-contents)
+- [List of Figures](#list-of-figures)
+- [1 Overview](#1-overview)
+- [2 Functional Requirements](#2-functional-requirements)
+- [3 Use Cases](#3-use-cases)
+ - [3.1 In-Service restart and upgrade](#31-in-service-restart-and-upgrade)
+ - [3.2 In-Service restart and upgrade from a vendor NOS to SONiC NOS](#32-in-service-restart-and-upgrade-from-a-vendor-nos-to-sonic-nos)
+- [4 Reconciliation at syncd](#4-reconciliation-at-syncd)
+ - [4.1 Orchagent Point Of View](#41-orchagent-point-of-view)
+ - [4.2 Syncd Point Of View - INIT/APPLY view framework](#42-syncd-point-of-view---initapply-view-framework)
+ - [4.3 neighsyncd Point Of View](#43-neighsyncd-point-of-view)
+ - [4.4 fpmsyncd Point Of View](#44-fpmsyncd-point-of-view)
+ - [4.5 Reboot finalizer](#45-reboot-finalizer)
+- [5 SONiC Application Extension Infrastructre Integration](#5-sonic-application-extension-infrastructre-integration)
+
+# List of Figures
+* [Syncd and Orchagent](#42-syncd-point-of-view---initapply-view-framework)
+* [Neighbors](#43-neighsyncd-point-of-view)
+* [BGP](#44-fpmsyncd-point-of-view)
+* [Finalizer](#45-reboot-finalizer)
+
+# 1 Overview
+
+The goal of SONiC fast-reboot is to be able to restart and upgrade SONiC software with a data plane disruption less than 30 seconds and control plane less than 90 seconds.
+With current implementation there is no indication of the fast-reboot status, meaning we don't have a way to determine if the flow has finished or not.
+Some feature flows in SONiC are delayed with a timer to keep the CPU dedicated to the fast-reboot init flow for best perforamnce, like enablement of flex counters.
+In order to have such indicator, re-use of the fastfast-reboot infrastructure can be used.
+
+Each network application will experience similar processing flow.
+Application and corresponding orchagent sub modules need to work together to restore the preboot state and push it to the ASIC.
+Take neighbor as an example, upon restart operation every neighbor we had prior to the reboot should be created again after resetting the ASIC.
+We should also synchronize the actual neighbor state after recovering it, the MAC of the neighbor could have changed, went down for some reason etc.
+In this case, restore_neighbors.py script will align the network state with the switch state by sending ARP/NDP to all known neighbors prior the reboot.
+neighsyncd will update the internal cache with all neighbors and push all to APP DB, orchagent will then add/remove/update any neighbor and get syncd to program the HW with the new data.
+
+In addition to the recover mechanism, the warmboot-finalizer can be enhanced to finalize fast-reboot as well and introduce a new flag indicating the process is done.
+This new flag can be used later on for any functionality, we want to start only after init flow finished in case of fast-reboot.
+This is to prevent interference in the fast-reboot reconciliation process and impair the performance, for example enablement of flex counters.
+
+References:
+https://github.com/Azure/SONiC/blob/master/doc/fast-reboot/fastreboot.pdf
+https://github.com/Azure/sonic-swss/blob/master/neighsyncd/restore_neighbors.py
+https://github.com/Azure/sonic-buildimage/blob/master/dockers/docker-orchagent/enable_counters.py
+
+# 2 Functional Requirements
+
+The new Fast-reboot design should meet the following requirments:
+
+- Reboot the switch into a new SONiC software version using kexec - less than 5 seconds.
+- Upgrade the switch FW by the new SONiC image if needed.
+- Recover all application's state with the new image to the previous state prior the reboot.
+- Recover ASIC state after reset to the previous state prior the reboot.
+- Recover the Kernel internal DB state after reset to the previous state prior the reboot.
+- Sync the Kernel and ASIC with changes on the network which happen during fast-reboot.
+- Control plane downtime will not exceed 90 seconds.
+- Data plane downtime will not exceed 30 seconds.
+
+# 3 Use Cases
+
+## 3.1 In-Service restart and upgrade
+
+### SWSS docker
+
+When swss docker starts with the new kernel, all the port/LAG, vlan, interface, arp and route data should be restored from CONFIG DB, APP DB, Linux Kernel and other reliable sources. There could be ARP, FDB changes during the restart window, proper sync processing should be performed.
+
+### Syncd docker
+
+The restart of syncd docker should leave data plane intact until it starts again with the new kernel. After restart, syncd configures the HW with the state prior the reboot by all network applications.
+
+## 3.2 In-Service restart and upgrade from a vendor NOS to SONiC NOS
+
+Fast-reboot will finish successfully from a different NOS than SONiC with two possible scenarios:
+ - Dump files of default gateway, neighbors and fdb tables are provided to the new image in a format that meet the SONiC scheme, as SONiC does prior the reboot.
+ - On this scenario all should work exacly the same as the switch rebooted from SONiC to SONiC.
+
+ - Dump files of default gateway, neighbors and fdb tables are not provided to the new image as SONiC does prior the reboot.
+ - On this scenario fast-reboot will finish successfully, but with low performance since all neighbors and fdb entries will be created by the slow path.
+
+Here are test results for both scenarios tested on a SONiC switch based on 202111 branch hash 339e68e1d, Nvidia SN2700 platform.
+
+ - With dump files:
+
+```
+2022-05-18 14:48:32 : --------------------------------------------------
+2022-05-18 14:48:32 : Summary:
+2022-05-18 14:48:32 : --------------------------------------------------
+2022-05-18 14:48:32 : Longest downtime period was 0:00:28.066813
+2022-05-18 14:48:32 : Reboot time was 0:01:45.150628
+2022-05-18 14:48:32 : Expected downtime is less then 0:00:30
+2022-05-18 14:48:32 : --------------------------------------------------
+2022-05-18 14:48:32 : Additional info:
+2022-05-18 14:48:32 : --------------------------------------------------
+2022-05-18 14:48:32 : INFO:10.213.84.185:PortChannel interface state changed 1 times
+2022-05-18 14:48:32 : INFO:10.213.84.184:PortChannel interface state changed 1 times
+2022-05-18 14:48:32 : INFO:10.213.84.187:PortChannel interface state changed 1 times
+2022-05-18 14:48:32 : INFO:10.213.84.186:PortChannel interface state changed 1 times
+2022-05-18 14:48:32 : INFO:10.213.84.181:PortChannel interface state changed 1 times
+2022-05-18 14:48:32 : INFO:10.213.84.183:PortChannel interface state changed 1 times
+2022-05-18 14:48:32 : INFO:10.213.84.182:PortChannel interface state changed 1 times
+2022-05-18 14:48:32 : INFO:10.213.84.188:PortChannel interface state changed 1 times
+2022-05-18 14:48:32 : --------------------------------------------------
+2022-05-18 14:48:32 : ==================================================
+```
+
+ - With no dump files:
+
+```
+2022-05-18 15:56:33 : --------------------------------------------------
+2022-05-18 15:56:33 : Summary:
+2022-05-18 15:56:33 : --------------------------------------------------
+2022-05-18 15:56:33 : Longest downtime period was 0:00:25.112824
+2022-05-18 15:56:33 : Reboot time was 0:01:36.778029
+2022-05-18 15:56:33 : Expected downtime is less then 0:00:30
+2022-05-18 15:56:33 : --------------------------------------------------
+2022-05-18 15:56:33 : Additional info:
+2022-05-18 15:56:33 : --------------------------------------------------
+2022-05-18 15:56:33 : INFO:10.213.84.185:PortChannel interface state changed 1 times
+2022-05-18 15:56:33 : INFO:10.213.84.184:PortChannel interface state changed 1 times
+2022-05-18 15:56:33 : INFO:10.213.84.187:PortChannel interface state changed 1 times
+2022-05-18 15:56:33 : INFO:10.213.84.186:PortChannel interface state changed 1 times
+2022-05-18 15:56:33 : INFO:10.213.84.181:PortChannel interface state changed 1 times
+2022-05-18 15:56:33 : INFO:10.213.84.183:PortChannel interface state changed 1 times
+2022-05-18 15:56:33 : INFO:10.213.84.182:PortChannel interface state changed 1 times
+2022-05-18 15:56:33 : INFO:10.213.84.188:PortChannel interface state changed 1 times
+2022-05-18 15:56:33 : --------------------------------------------------
+2022-05-18 15:56:33 : --------------------------------------------------
+2022-05-18 15:56:33 : Fails:
+2022-05-18 15:56:33 : --------------------------------------------------
+2022-05-18 15:56:33 : FAILED:dut:Total downtime period must be less then 0:00:30 seconds. It was 48.5044789314
+2022-05-18 15:56:33 : ==================================================
+```
+
+A proper automatic functional test case covering the scenario of vendor NOS to SONiC NOS reboot should be implemented as part of this feature.
+The test will simulate such scenario by dropping the dump files and perform fast-reboot without it.
+
+# 4 Reconciliation at syncd
+
+## 4.1 Orchagent Point Of View
+
+If dump files provided by the previous image prior the reboot, all tables should be pushed to APP DB for reconciling orchagent.
+If no dumps are provided, orchagent will reconcile with no information from prior the reboot.
+On this case all ARP and FDB entries will be created by the slow path, meaning ARP packets will exchange between the switch and the neighbors and will be created according to the neighbors response, from kernel to the ASIC.
+Same for FDB entries which will be created by the kernel as well, depends on the traffic flowing through the switch.
+When orchagent starts with the new SONiC image, the same infrastructure we use to reconcile fastfast-boot will start.
+After INIT_VIEW and create_switch functions sent to syncd (reset of the ASIC took place here), 'warmRestoreAndSyncUp' will be executed.
+This function will populate m_toSync with all tasks for syncd, by APP DB and CONFIG DB prior the reboot.
+To verify orchagent reached the same state as before the reboot, 'warmRestoreValidation' will verify no pending tasks left in the queue, meaning all proccessed succesfully and in the pipeline for syncd to configure the HW.
+At the end APPLY_VIEW will be sent to syncd to finalize the process, from this point orchagent enter the main loop and operates normally.
+
+### NOTICE
+
+'warmRestoreValidation' might fail the operation just like in fastfast-reboot case, if the way orchagent process an event from the DB is handled differently with the new software version the task will fail to execute and fast-reboot will fail along with it.
+This is solvable by the db migrator.
+
+## 4.2 Syncd Point Of View - INIT/APPLY view framework
+
+Syncd starts with the fast-reboot flag, trigger the ASIC reset when create_switch is requested from orchagent.
+In addition, on this case temp view flag will set to false since it is not required, no comparison logic needed since current view is empty.
+Basically INIT and APPLY view requests from orchagent are ignored by syncd, but bound the process from start to end.
+During reconsilations process of orchagent, syncd will recieve all tasks to restore the previous state.
+All other network applications will do the same as we do today for warm-reboot.
+
+
+
+## 4.3 neighsyncd Point Of View
+
+Neighbors configuration is a crucial part of the L3 switch software. It is best when the neighbor configuration on the hardware is in sync with the actual switch neighbors on the network. It can't be assumed that neighbors won't change during warm restart window, while the software is restarting, the SONiC switch software has to be ready for scenarios in which during the restart window:
+ - Existing neighbors went down, e.g: VMs crashed on the server connected to ToR switch which undergoes fast-reboot.
+ - New neighbors appeared on the network, e.g: VMs created on the server connected to ToR switch which undergoes fast-reboot.
+ - MAC changes, e.g: VMs re-created or re-configured on the server connected to ToR switch which undergoes fast-reboot.
+
+
+
+## 4.4 fpmsyncd Point Of View
+
+ - When BGP on a router restarts, all the BGP peers detect that the session went down and then came up. This "down/up" transition results in a "routing flap" and causes BGP route re-computation, generation of BGP routing updates, and unnecessary churn to the forwarding tables.
+ - BGP capability, termed "Graceful Restart Capability", is defined that would allow a BGP speaker to express its ability to preserve forwarding state during BGP restart.
+ - An UPDATE message with no reachable Network Layer Reachability Information (NLRI) and empty withdrawn NLRI is specified as the End-of-RIB marker that can be used by a BGP speaker to indicate to its peer the completion of the initial routing update after the session is established.
+
+
+
+## 4.5 Reboot finalizer
+
+Today we have a tool used for warm-reboot to collect all reconsiliation flags from the different network applications.
+This tool can be enhanced to consider fast-reboot as well and introduce a new flag indicating the end of the process.
+This flag can be used to trigger any flow or application we want to delay until init flow has finished.
+
+
+
+# 5 SONiC Application Extension Infrastructre Integration
+
+A SONiC package can specify an order of shutdown on fast-reboot for a service. A "bgp" may specify "radv" in this field in order to avoid radv to announce departure and cause hosts to lose default gateway, while "teamd" service has to stop before "syncd", but after "swss" to be able to send the last LACP PDU through CPU port right before CPU port becomes unavailable.
+
+The fast-reboot service shutdown script has to be auto-generated from a template /usr/share/sonic/templates/fast-shutdown.sh.j2. The template is derived from the fast-reboot script from sonic-utlities.
+
+A services shutdown is an ordered executions of systemctl stop {{ service }} commands with an exception for "swss" service after which a syncd pre-shutdown is requested and database backup is prepared for next boot. A service specific actions that are executed on fast-shutdown are hidden inside the service stop script action.
+
+NOTE: the assumption here is that syncd pre-shutdown is bound to swss service stop when swss service is doing system level shutdown.
+
+The *-shutdown.sh are imported and executed in corresponding *-reboot scripts.
+
+###### fast-shutdown.sh.j2 snippet
+```
+...
+{% for service in shutdown_orider %}
+systemctl stop {{ service }}
+{% endfor %}
+...
+```
+
+reboot-finalizer.sh (warm-finalizer.sh) script must also be templatized and updated based on process reconciles flag.
+
+###### manifest path
+
+| Path | Value | Mandatory | Description |
+| ----------------------------- | --------------- | --------- | -------------------------------------------------------------------------------------------------------------------- |
+| /service/fast-shutdown/ | object | no | Fast reboot related properties. Used to generate the fast-reboot script. |
+| /service/fast-shutdown/after | lits of strings | no | Same as for warm-shutdown. |
+| /service/fast-shutdown/before | lits of strings | no | Same as for warm-shutdown. |
+| /processes | object | no | Processes infromation |
+| /processes/[name]/reconciles | boolean | no | Wether process performs warm-boot reconciliation, the warmboot-finalizer service has to wait for. Defaults to False. |
+
+
+This chapter it taken from SONiC Application Extension Infrastructure HLD:
+https://github.com/Azure/SONiC/blob/master/doc/sonic-application-extension/sonic-application-extention-hld.md#warmboot-and-fastboot-design-impact
diff --git a/doc/fast-reboot/Finalizer.svg b/doc/fast-reboot/Finalizer.svg
new file mode 100644
index 0000000000..1decb1a38a
--- /dev/null
+++ b/doc/fast-reboot/Finalizer.svg
@@ -0,0 +1,4 @@
+
+
+
+
\ No newline at end of file
diff --git a/doc/fast-reboot/Neighbors.svg b/doc/fast-reboot/Neighbors.svg
new file mode 100644
index 0000000000..193119e839
--- /dev/null
+++ b/doc/fast-reboot/Neighbors.svg
@@ -0,0 +1,4 @@
+
+
+
+
\ No newline at end of file
diff --git a/doc/fast-reboot/Orchagent_Syncd.svg b/doc/fast-reboot/Orchagent_Syncd.svg
new file mode 100644
index 0000000000..563c956a4e
--- /dev/null
+++ b/doc/fast-reboot/Orchagent_Syncd.svg
@@ -0,0 +1,4 @@
+
+
+
+
\ No newline at end of file
diff --git a/doc/fips/SONiC-OpenSSL-FIPS-140-3.md b/doc/fips/SONiC-OpenSSL-FIPS-140-3.md
new file mode 100644
index 0000000000..aa750592d5
--- /dev/null
+++ b/doc/fips/SONiC-OpenSSL-FIPS-140-3.md
@@ -0,0 +1,168 @@
+# SONIC OpenSSL FIPS 140-3 HLD
+
+## Revision
+
+| Rev | Date | Author | Change Description |
+| :---: | :--------: | :--------: | ------------------ |
+| 0.1 | 2022-02-22 | Xuhui Miao | Initial version |
+
+## Table of Contents
+- [Abbreviation](#abbreviation)
+- [Requirement](#requirement)
+- [The cryptographic modules in SONiC](#the-cryptographic-modules-in-SONiC)
+- [OpenSSL FIPS 140-3](#OpenSSL-FIPS-140-3)
+ * [OpenSSL Engine](#OpenSSL-Engine)
+ * [SymCrypt OpenSSL Engine](#symCrypt-openSSL-engine)
+ * [OpenSSL configuration for SymCrypt Engine](#OpenSSL-configuration-for-SymCrypt-Engine)
+ * [OpenSSL configuration enhancement](#OpenSSL-configuration-enhancement)
+ * [SymCrypt OpenSSL Engine debian package](#SymCrypt-OpenSSL-Engine-debian-package)
+- [Kerberos Cryptographic Module](#Kerberos-Cryptographic-Module)
+- [Golang Cryptographic Module](#Golang-Cryptographic-Module)
+- [Application Impact](#Application-Impact)
+- [SONiC FIPS Configuration](#SONiC-FIPS-Configuration)
+ * [Enable FIPS on system level](#Enable-FIPS-on-system-level)
+ * [Enable FIPS on application level](#Enable-FIPS-on-application-level)
+ * [SONiC Build Options](#SONiC-Build-Options)
+- [Q&A](#Q&A)
+
+
+## Abbreviation
+
+| Abbreviation | Description |
+| ------------ | -------------------------------------------- |
+| CAVP | Cryptographic Algorithm Validation Program |
+| CST | Cryptographic and Security Test |
+| CMVP | Cryptographic Module Validation Program |
+| FIPS | Federal Information Processing Standard |
+
+## Requirement
+SONiC only uses cryptographic modules validated by FIPS 140-3, Make SONiC compliant with FIPS 140-3.
+
+## The cryptographic modules in SONiC
+
+| Module | Use Scenarios | Description |
+| -------------------- | --------------- | -------------------------------------------- |
+| OpenSSL | Python, OpenSSH | Cyptography and SSL/TLS ToolKit |
+| Kerberos | OpenSSH | Kerboros contains builtin crypto module |
+| Golang | sonic-restapi | Golang contains builtin crypto module |
+| Libgcrypto | GPG | A general purpose cryptographic library originally based on code from GnuPG |
+| Kernel Crypto | -- | Linux crypto kernel module |
+
+## Scopes:
+In Scopes:
+1. OpenSSL
+2. Kerberos
+3. Golang
+
+Out of Scopes:
+1. Linux Kernel
+2. Libgcrypt
+
+
+## OpenSSL FIPS 140-3
+
+
+
+### OpenSSL Engine
+OpenSSL supports engine cryptographic modules in the form of engine objects, and provides a reference-counted mechanism to allow them to be dynamically loaded in and out of the running application. An engine object can implement one or all cryptographic algorithms.
+
+### SymCrypt OpenSSL Engine
+The [SymCrypt engine for OpenSSL (SCOSSL)](https://github.com/microsoft/SymCrypt-OpenSSL) allows the use of OpenSSL with [SymCrypt](https://github.com/microsoft/SymCrypt) as the provider for core cryptographic operations. It leverages the OpenSSL engine interface to override the cryptographic implementations in OpenSSL's libcrypto. The primary motivation for this is to support FIPS certification, as OpenSSL 1.1.1 does not have a FIPS-certified cryptographic module. Microsoft will submit the FIPS 140-3 reports for SymCrypt to CMVP.
+
+The SymCrypt Engine is one of the implementation to support FIPS, The [wolfSSL engine](https://github.com/wolfSSL/wolfEngine) is another option.
+
+### OpenSSL configuration for SymCrypt Engine
+
+/usr/lib/ssl/openssl-fips.cnf
+```
+openssl_conf = openssl_init
+[ openssl_init ]
+engines = engine_section
+
+[ engine_section ]
+symcrypt = symcrypt_section
+
+[ symcrypt_section ]
+engine_id = symcrypt
+dynamic_path = /usr/lib/x86_64-linux-gnu/libsymcryptengine.so
+default_algorithms = ALL
+```
+
+### OpenSSL configuration enhancement
+When fips=1 is set in /proc/cmdline, the OpenSSL default config file is changed to "/usr/lib/ssl/openssl-fips.cnf", otherwise, the config file "/usr/lib/ssl/openssl.cnf" is used.
+
+### SymCrypt OpenSSL Engine debian package
+Provide SymCrypt OpenSSL debian package.
+Package name: symcrypt-openssl
+Current version: 0.1
+
+Package file name example: symcrypt-openssl_0.1_amd64.deb
+Files in the packages:
+```
+/usr/lib/ssl/openssl.cnf
+/usr/lib/ssl/openssl-fips.cnf
+/usr/lib/x86_64-linux-gnu/libsymcrypt.so
+/usr/lib/x86_64-linux-gnu/libsymcryptengine.so
+```
+
+## Kerberos Cryptographic Module
+Kerberos will use the builtin cryptographic module by default, but it allows to change the build option to use OpenSSl, see [MIT Kerberos features](https://web.mit.edu/kerberos/krb5-1.13/doc/mitK5features.html). SONiC will change the build option to use OpenSSL instead of the builtin one. It is not configurable to use the Kerberos builtin cryptographic module when OpenSSL used.
+
+## Golang Cryptographic Module
+Golang has its own cryptographic module (see [crypto](https://github.com/golang/go/tree/master/src/crypto)) without FIPS supports. There are some branches with branch name starting with "dev.boringcrypto" (see [golang branches](https://github.com/golang/go/branches/all?query=dev.boringcrypto)), changing the Golang cryptographic APIs' referenece to use [BoringSSL](https://github.com/google/boringssl). Although BoringSSL is an open source project, but it used by Google only, not intened for general use.
+
+To support FIPS for Golang, RedHat offers an alternative solution (see [here](https://developers.redhat.com/blog/2019/06/24/go-and-fips-140-2-on-red-hat-enterprise-linux)), it builds on top of the Golang's dev.bringcrypt branches, has ability to call into OpenSSL, not BoringSSL. SONiC can reuse the RedHat sulotion, one difference is that RedHat supports FIPS for OpenSSL directly, SONiC uses OpenSSL Engine.
+
+How OpenSSL Engine works in Golang?
+
+
+When FIPS enabled, both of the BoringSSL Enable Option and the SymCrypt Enabled option will be set.
+
+
+## Application Impact
+Some of functions of a application might be broken when using the cryptographic algorithms that are not FIPS compliant. It is relied on the tests of the applications to detect all the impact functions.
+For OpenSSH, Centos provides a [patch](https://git.centos.org/rpms/openssh/raw/c8/f/SOURCES/openssh-7.7p1-fips.patch) which is compiant with FIPS 140-2. We can apply the patch and verify if it can pass all the OpenSSH test cases when FIPS enabled.
+
+## SONiC FIPS Configuration
+
+### Enable FIPS on system level
+Add the Linux System parameter fips=1, in grub config, one of implemetation as below:
+
+cat /etc/grub.d/99-fips.cfg
+```
+GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT fips=1"
+```
+
+To validate the FIPS enabled, grep 'fips=1' /proc/cmdline.
+
+### Enable FIPS on application level
+```
+export ENABLE_FIPS=1
+```
+
+Alternative option for the golang applications only:
+```
+export GOLANG_FIPS=1
+```
+
+Alternative option for the OpenSSL applications only:
+
+see https://www.openssl.org/docs/manmaster/man7/openssl-env.html
+```
+export OPENSSL_CONFIG=/usr/lib/ssl/openssl-fips.cnf
+```
+
+### SONiC Build Options
+Support to enable/disable the FIPS feature, the feature is enabled by default in rules/config as below.
+```
+ENABLE_FIPS_FEATURE ?= y
+```
+Support to enable/disable FIPS config, the flage is disabled by default. IF the option is set, then the fips is enabled by default in the image, not necesary to do the config in system level or application level.
+```
+ENABLE_FIPS ?= n
+```
+If the ENABLE_FIPS_FEATURE is not set, then the option ENABLE_FIPS is useless.
+
+## Q&A
+### Does SymCrypt use Linux Kernel crypto module?
+SymCrypt on Linux does not rely on Kernel crypt for FIPS certification today.
\ No newline at end of file
diff --git a/doc/fips/images/fips-overview.png b/doc/fips/images/fips-overview.png
new file mode 100644
index 0000000000..bbeee405d1
Binary files /dev/null and b/doc/fips/images/fips-overview.png differ
diff --git a/doc/fips/images/golang-api.png b/doc/fips/images/golang-api.png
new file mode 100644
index 0000000000..c599fd9f42
Binary files /dev/null and b/doc/fips/images/golang-api.png differ
diff --git a/doc/fwutil/fwutil.md b/doc/fwutil/fwutil.md
index 82a871b569..0becc2568b 100755
--- a/doc/fwutil/fwutil.md
+++ b/doc/fwutil/fwutil.md
@@ -576,7 +576,7 @@ New component api is introduced to support the component firmware auto-update as
"""
raise NotImplementedError
```
-The return_code of auto_update_firmware() which indicates the firmware auto-update status, will be logged in "fw_au_status" under "/var/platform/" directory by fwutil and the status file will be used for the `fwutil show update status` command.
+The return_code of auto_update_firmware() which indicates the firmware auto-update status, will be logged in "fw_au_status" under "/var/firmwareupdate/" directory by fwutil and the status file will be used for the `fwutil show update status` command.
In case that a firmware update needs any additional step to complete the firmware update but the installation time is longer than the boot time requirement, auto-update platform api is expected to install the firmware and perform the complete action during the reboot via `platform_fw_au_reboot_handle` or `platform_reboot` plugin.
For example, some cpld update needs a power cycle to complete the firmware update and some cpld update needs a register triggered power-cycle to give some refresh time for the new firmware to be effective on the system.
@@ -602,7 +602,7 @@ In this example, BIOS firmware got updated, CPLD firmware got installed but powe
*powercycle can be triggered by cold reboot script in this case.
```bash
-admin@sonic:~/fwutil$ cat /var/platform/fw_au_status
+admin@sonic:~/fwutil$ cat /var/firmwareupdate/fw_au_status
{
"MSN2700/BIOS":
{
@@ -634,7 +634,7 @@ And the components available for the update are BIOS, and SSD.
In this example, BIOS firmware got updated, CPLD firmware update was skipped since the update completion can not be done for warm, and SSD firmware update is scheduled during warm boot.
```bash
-admin@sonic:~/fwutil$ cat /var/platform/fw_au_status
+admin@sonic:~/fwutil$ cat /var/firmwareupdate/fw_au_status
{
"MSN2700/BIOS":
{
@@ -668,9 +668,9 @@ Automatic FW installation requires default platform_components.json to be create
_sonic-buildimage/device///platform_components.json_
Recommended image path is "/lib/firmware/".
-Here is the /var/platform directory structure while fwutil handles the `fwutil update all fw` and `fwutil show update status` command.
+Here is the /var/firmwareupdate directory structure while fwutil handles the `fwutil update all fw` and `fwutil show update status` command.
```
-/var/platform/
+/var/firmwareupdate/
|--- fw_au_status
|--- _fw_au_task*
```
diff --git a/doc/ip-interface/loopback-action/images/feature_overview.svg b/doc/ip-interface/loopback-action/images/feature_overview.svg
new file mode 100755
index 0000000000..3518d45921
--- /dev/null
+++ b/doc/ip-interface/loopback-action/images/feature_overview.svg
@@ -0,0 +1,3 @@
+
+
+
\ No newline at end of file
diff --git a/doc/ip-interface/loopback-action/images/interface_vlan.svg b/doc/ip-interface/loopback-action/images/interface_vlan.svg
new file mode 100755
index 0000000000..70143e6b1f
--- /dev/null
+++ b/doc/ip-interface/loopback-action/images/interface_vlan.svg
@@ -0,0 +1,3 @@
+
+
+
\ No newline at end of file
diff --git a/doc/ip-interface/loopback-action/ip-interface-loopback-action-design.md b/doc/ip-interface/loopback-action/ip-interface-loopback-action-design.md
new file mode 100644
index 0000000000..76ea8deed5
--- /dev/null
+++ b/doc/ip-interface/loopback-action/ip-interface-loopback-action-design.md
@@ -0,0 +1,361 @@
+# SONiC IP Interface Loopback Action
+## High level design document
+## Table of contents
+- [Revision](#revision)
+- [Scope](#scope)
+- [1 Introduction](#1-introduction)
+ - [1.1 Feature overview](#11-feature-overview)
+ - [1.2 Requirements](#12-requirements)
+ - [1.2.1 Functionality](#121-functionality)
+ - [1.2.2 Command interface](#122-command-interface)
+ - [1.2.3 Error handling](#123-error-handling)
+ - [1.2.4 Event logging](#124-event-logging)
+- [2 Design](#2-design)
+ - [2.1 Overview](#21-overview)
+ - [2.2 DB schema](#22-db-schema)
+ - [2.2.1 Config DB](#221-config-db)
+ - [2.2.2 App DB](#222-app-db)
+ - [2.2.3 Configuration migration](#223-configuration-migration)
+ - [2.3 Flows](#23-flows)
+ - [2.3.1 Loopback action set interface vlan](#231-loopback-action-set-interface-vlan)
+ - [2.3.1.1 Handle set in runtime](#2311-handle-set-in-runtime)
+ - [2.3.1.2 Handle set in init](#2312-handle-set-in-init)
+ - [2.3.2 Loopback action set interface Ethernet](#232-loopback-action-set-interface-ethernet)
+ - [2.3.3 Loopback action set interface port-channel](#233-loopback-action-set-interface-port-channel)
+ - [2.3.4 Loopback action set subinterface](#234-loopback-action-set-subinterface)
+ - [2.3.5 IP interface creation](#235-ip-interface-creation)
+ - [2.4 SAI](#24-sai)
+ - [2.5 Statistics](#25-statistics)
+ - [2.6 CLI](#26-cli)
+ - [2.6.1 Config command](#261-config-command)
+ - [2.6.2 Show command](#262-show-command)
+ - [2.7 YANG model](#27-yang-model)
+ - [2.8 Warm/Fast boot](#28-warmfast-boot)
+- [3 Test plan](#3-test-plan)
+ - [3.1 Unit tests via VS](#31-unit-tests-via-vs)
+ - [3.2 System tests](#32-system-tests)
+
+## Revision
+| Rev | Date | Author | Description |
+|:---:|:----------:|:--------------:|:----------------|
+| 0.1 | 18/05/2022 | Lior Avramov | Initial version |
+
+## Scope
+This document describes the high level design of IP interface loopback action feature in SONiC
+
+## List of figures
+[Figure 1: feature overview](#figure-1-feature-overview)
+[Figure 2: interface vlan](#figure-2-interface-vlan)
+
+# 1 Introduction
+## 1.1 Feature overview
+IP interface loopback action is a feature that allows user to change the way router handles routed packets for which egress port equals to ingress port.
+1. When loopback action is configured to drop, those packets will be dropped. Drppoed packets due to loopback action are counted in rif statistics, see [2.5 Statistics](#25-statistics)
+2. When loopback action is configured to forward, those packets will be forwarded as the pipeline defined.
+
+
+###### Figure 1: feature overview
+
+## 1.2 Requirements
+### 1.2.1 Functionality
+**This feature will support the following functionality:**
+1. Loopback action can be set on an IP interface.
+2. An IP Interface can be one of the following:
+ 1. Interface vlan
+ 2. Interface Ethernet
+ 3. Interface port-channel
+ 4. Subinterface
+3. Loopback action can be set to "drop" or "forward".
+4. CLI will reject loopback action setting on a non IP interface. Interface is considered as an IP interface if it has entry in:
+ 1. VLAN_INTERFACE table for interface vlan (e.g. VLAN_INTERFACE|Vlan100)
+ 2. INTERFACE table for interface Ethernet (e.g. INTERFACE|Ethernet232)
+ 3. PORTCHANNEL_INTERFACE table for interface port-channel (e.g. PORTCHANNEL_INTERFACE|PortChannel1)
+ 4. VLAN_SUB_INTERFACE table for subinterface (e.g. VLAN_SUB_INTERFACE|Ethernet0.10)
+5. The configured loopback action can be viewed by show command in CLI.
+6. When SONiC configuration of loopback action is missing the system will behave as it is today, based on SAI behavioural model, forward the loopbacked packets.
+7. No changes required in Kernel.
+
+### 1.2.2 Command interface
+**This feature will support the following commands:**
+1. config: set IP interface loopback action.
+2. show: display the configured loopback action.
+
+### 1.2.3 Error handling
+**This feature will provide error handling for the next situations:**
+1. In CLI: invalid action, invalid interface, non ip interface
+2. In SWSS: invalid action, invalid DB operation, invalid interface
+
+### 1.2.4 Event logging
+**This feature will provide event logging for:** loopback action set on IP interface.
+
+###### Table 1: Event logging
+| Event | Severity |
+|:------------------------------------------|:---------|
+| IP interface loopback action set: success | NOTICE |
+| IP interface loopback action set: error | ERROR |
+
+# 2 Design
+## 2.1 Overview
+Intfmgrd subscribes for changes in L3 interface tables VLAN_INTERFACE, INTERFACE and PORTCHANNEL_INTERFACE in config DB. Whenever there is a change in one of these tables, intfmgrd is notified. Intfmgrd then updates INTF_TABLE table in app DB. Upon change in INTF_TABLE, intforch is notified.
+Intforch performs validity checks, call the relevant SAI API and updates internal cache. The flow in intfmgr is based on the current interface configuration flow.
+
+## 2.2 DB schema
+### 2.2.1 Config DB
+To support loopback action set on L3 interface, a new field-value pair is defined in CONFIG_DB in the following tables.
+Valid values are "drop" or "forward".
+
+1. VLAN_INTERFACE
+```
+VLAN_INTERFACE|interface-name
+ "loopback_action": action (OPTIONAL)
+```
+Example:
+```
+"VLAN_INTERFACE": {
+ "Vlan100": {
+ "loopback_action": "drop",
+ "mac_addr": "00:01:02:03:04:10",
+ "ipv6_use_link_local_only": "enable"
+ },
+}
+```
+
+2. INTERFACE
+```
+INTERFACE|interface-name
+ "loopback_action": action
+```
+Example:
+
+```
+"INTERFACE": {
+ "Ethernet248": {
+ "loopback_action": "forward",
+ "mac_addr": "00:01:02:03:04:11",
+ "ipv6_use_link_local_only": "enable"
+ },
+}
+```
+3. PORTCHANNEL_INTERFACE
+```
+PORTCHANNEL_INTERFACE|interface-name
+ "loopback_action": action
+```
+Example:
+```
+"PORTCHANNEL_INTERFACE": {
+ "PortChannel1": {
+ "loopback_action": "drop",
+ "mac_addr": "00:01:02:03:04:12",
+ "ipv6_use_link_local_only": "enable"
+ },
+}
+```
+
+4. VLAN_SUB_INTERFACE
+```
+VLAN_SUB_INTERFACE|interface-name
+ "loopback_action": action
+```
+Example:
+```
+"VLAN_SUB_INTERFACE": {
+ "Ethernet0.10": {
+ "loopback_action": "drop",
+ "mac_addr": "00:01:02:03:04:12",
+ "ipv6_use_link_local_only": "enable"
+ },
+}
+```
+
+### 2.2.2 App DB
+To support loopback action set on L3 interface, a new field-value pair is defined in APP DB in table INTF_TABLE.
+```
+INTF_TABLE|interface-name
+ "loopback_action": action
+```
+Example for interface vlan:
+```
+ "INTF_TABLE": {
+ "Vlan100": {
+ "loopback_action": "drop",
+ "mac_addr": "00:01:02:03:04:10",
+ "ipv6_use_link_local_only": "enable"
+
+```
+Example for interface Ethernet:
+```
+ "INTF_TABLE": {
+ "Ethernet248": {
+ "loopback_action": "drop",
+ "mac_addr": "00:01:02:03:04:11",
+ "ipv6_use_link_local_only": "enable"
+ }
+ }
+```
+Example for interface port-channel:
+```
+ "INTF_TABLE": {
+ "PortChannel1": {
+ "loopback_action": "drop",
+ "mac_addr": "00:01:02:03:04:12",
+ "ipv6_use_link_local_only": "enable"
+ }
+ }
+```
+Example for subinterface:
+```
+ "INTF_TABLE": {
+ "Ethernet0.10": {
+ "loopback_action": "drop",
+ "mac_addr": "00:01:02:03:04:12",
+ "ipv6_use_link_local_only": "enable"
+ }
+ }
+```
+
+### 2.2.3 Configuration migration
+When upgrading from SW image which does not support the new schema to a one which supports, the new image will not contain loopback action configuration. All existing IP interfaces will behave as today, based on SAI behavioural model, forward the loopbacked packets.
+
+## 2.3 Flows
+
+### 2.3.1 Loopback action set interface vlan
+#### 2.3.1.1 Handle set in runtime
+
+###### Figure 2: interface vlan
+
+#### 2.3.1.2 Handle set in init
+In init, loopback action configuration is being handled as part of the IP interface creation.
+During init, all field-value pairs of each key in INTF_TABLE table (e.g. INTF_TABLE|Vlan100) are being received in intfOrch together at once.
+Those field-value pairs are the interface attributes, e.g. mac address, loopback action etc.
+
+### 2.3.2 Loopback action set interface Ethernet
+Loopback action set on interface Ethernet flow is the same as interface vlan flow besides config DB and app DB keys being used. Used keys are defined in [2.2.1 Config DB](#221-config-db).
+
+### 2.3.3 Loopback action set interface port-channel
+Loopback action set on interface port-channel flow is the same as interface vlan flow besides config DB and app DB keys being used. Used keys are defined in [2.2.1 Config DB](#221-config-db).
+
+### 2.3.4 Loopback action set subinterface
+Loopback action set on subinterface flow is the same as interface vlan flow besides config DB and app DB keys being used. Used keys are defined in [2.2.1 Config DB](#221-config-db).
+
+### 2.3.5 IP interface creation
+If loopback action was not configured by the user, default loopback action is derived from SAI behavioral model which is forward.
+
+## 2.4 SAI
+The following will be used for setting IP interface loopback action.
+
+SAI API:
+```
+sai_router_intfs_api->set_router_interface_attribute()
+sai_router_intfs_api->create_router_interface()
+```
+
+SAI attribute:
+###### Table 2: SAI attribute
+| Loopback action | SAI attribute | Attribute value |
+|------------------|---------------------------------------------------|----------------------------|
+| drop | SAI_ROUTER_INTERFACE_ATTR_LOOPBACK_PACKET_ACTION | SAI_PACKET_ACTION_DROP |
+| forward | SAI_ROUTER_INTERFACE_ATTR_LOOPBACK_PACKET_ACTION | SAI_PACKET_ACTION_FORWARD |
+
+
+```
+typedef enum _sai_router_interface_attr_t
+{
+ /**
+ * @brief Packet action when a packet ingress and gets routed on the same RIF
+ *
+ * @type sai_packet_action_t
+ * @flags CREATE_AND_SET
+ * @default SAI_PACKET_ACTION_FORWARD
+ */
+ SAI_ROUTER_INTERFACE_ATTR_LOOPBACK_PACKET_ACTION,
+
+} sai_router_interface_attr_t;
+```
+
+## 2.5 Statistics
+Packets that are dropped due to loopback action will be counted in TX_ERR in IP interface statistics.
+```
+root@r-lionfish-14:/home/admin# show interfaces counters rif
+ IFACE RX_OK RX_BPS RX_PPS RX_ERR TX_OK TX_BPS TX_PPS TX_ERR
+------------ ------- -------- -------- -------- ------- -------- -------- --------
+ Ethernet236 4 0.00 B/s 0.00/s 1 0 0.00 B/s 0.00/s 0
+PortChannel1 0 0.00 B/s 0.00/s 0 0 0.00 B/s 0.00/s 0
+ Vlan2 0 0.00 B/s 0.00/s 0 0 0.00 B/s 0.00/s 0
+Ethernet0.10 0 0.00 B/s 0.00/s 0 0 0.00 B/s 0.00/s 0
+```
+
+## 2.6 CLI
+### 2.6.1 Config command
+A new SONiC CLI command is introduced to configure IP interface loopback action.
+```
+# Set loopback action per INTERFACE/PORTCHANNEL_INTERFACE/VLAN_INTERFACE/VLAN_SUB_INTERFACE
+config interface loopback-action drop|forward
+```
+
+Examples:
+```
+config interface loopback-action Ethernet248 drop
+config interface loopback-action Vlan100 forward
+```
+
+### 2.6.2 Show command
+A new SONiC CLI command is introduced to show the configured loopback action.
+IP interfaces that user did not configure loopback action on them, will not be shown in the table.
+```
+show ip interfaces loopback-action
+```
+
+Example:
+```
+root@sonic:~# show ip interfaces loopback-action
+Interface Action
+------------ ----------
+Ethernet232 drop
+Vlan100 forward
+```
+
+## 2.7 YANG model
+The following YANG models will be extended in order to provide support for loopback action:
+1. sonic-interface.yang -> container INTERFACE
+2. sonic-portchannel.yang -> container PORTCHANNEL_INTERFACE
+3. sonic-vlan.yang -> container VLAN_INTERFACE
+4. sonic-vlan-sub-interface.yang -> container VLAN_SUB_INTERFACE
+
+New leaf loopback_action will be added:
+```
+leaf loopback_action {
+ description "Packet action when a packet ingress and gets routed on the same IP interface";
+ type string {
+ pattern "drop|forward";
+ }
+}
+```
+
+## 2.8 Warm/Fast boot
+No special handling is required
+
+# 3 Test plan
+## 3.1 Unit tests via VS
+Add the following test case in file sonic-swss/tests/test_interface.py.
+1. Create IP interface by writing to config DB
+2. Set loopback action on IP interface
+3. Check table INTF_TABLE in App DB
+4. Check table SAI_OBJECT_TYPE_ROUTER_INTERFACE in ASIC DB
+5. Create port-channel IP interface and repeat steps 2 to 4.
+6. Create interface VLAN and repeat steps 2 to 4.
+7. Create subinterface and repeat steps 2 to 4.
+
+## 3.2 System tests
+Add the following test in sonic-mgmt.
+1. Create IP interface Ethernet.
+2. Loopback action not configured, send traffic, verify loopbacked packets are forwarded (SAI default).
+3. Set loopback action to drop, send traffic, verify loopbacked packets are dropped and TX_ERR in rif counter increased.
+4. Save config and reboot, send traffic, verify loopbacked packets are dropped and TX_ERR in rif counter increased.
+5. Set loopback action to forward, send traffic, verify loopbacked packets are forwarded.
+6. Save config and reboot, send traffic, verify loopbacked packets are forwarded.
+7. Create port-channel IP interface and repeat steps 2 to 6.
+8. Create interface VLAN and repeat steps 2 to 6.
+9. Create subinterface and repeat steps 2 to 6.
+10. Set loopback action on non IP interface.
+11. Set loopback action on non existing interface.
diff --git a/doc/port_auto_neg/flow_rmt_adv.png b/doc/port_auto_neg/flow_rmt_adv.png
new file mode 100644
index 0000000000..184a9fc770
Binary files /dev/null and b/doc/port_auto_neg/flow_rmt_adv.png differ
diff --git a/doc/port_auto_neg/port-auto-negotiation-design.md b/doc/port_auto_neg/port-auto-negotiation-design.md
index 41a9ef83ae..dfebd151c0 100644
--- a/doc/port_auto_neg/port-auto-negotiation-design.md
+++ b/doc/port_auto_neg/port-auto-negotiation-design.md
@@ -26,7 +26,7 @@
- [SWSS Enhancements](#swss-enhancements)
- [portsyncd and portmgrd Consideration](#portsyncd-and-portmgrd-consideration)
- [Port Breakout Consideration](#port-breakout-consideration)
- - [xcvrd Consideration](#xcvrd-consideration)
+ - [PMON xcvrd Consideration](#pmon-xcvrd-consideration)
- [Warmboot and Fastboot Design Impact](#warmboot-and-fastboot-design-impact)
- [Restrictions/Limitations](#restrictions/limitations)
- [Testing Requirements/Design](#testing-requirements/design)
@@ -36,10 +36,11 @@
### Revision
- | Rev | Date | Author | Change Description |
- |:---:|:-----------:|:------------------:|-----------------------------------|
- | 0.1 | | Junchao Chen | Initial version |
- | 0.2 | | Junchao Chen | Fix review comment |
+ | Rev | Date | Author | Change Description |
+ |:---:|:-----------:|:-------------------:|--------------------------------------------|
+ | 0.1 | | Junchao Chen | Initial version |
+ | 0.2 | | Junchao Chen | Fix review comment |
+ | 0.3 | | Dante (Kuo-Jung) Su | Add support for SFPs and operational states|
### Scope
This document is the design document for port auto negotiation feature on SONiC. This includes the requirements, CLI change, DB schema change, DB migrator change, yang model change and swss change.
@@ -51,7 +52,12 @@ N/A
The IEEE 802.3 standard defines a set of Ethernet protocols that are comprised of speed rate and interface type. It allows for configuring multiple values at the same time for port provisioning and advertising to the remote side. However, on SONiC, user can configure the speed of port, and user can configure auto negotiation mode via config DB. Port attributes such as interface type, advertised speeds, advertised interface types are not supported.
-The feature in this document is to address the above issues.
+The feature in this document is to address the above issues, and support the following medium typs
+
+- Twisted-pair cables (Non-Gearbox design)
+- SFP/QSFP/QSFPDD CR/KR transceivers (Non-Gearbox design)
+
+The complexities behind vendor-specific SAI implementation is outside the scope of this document.
### Requirements
@@ -86,6 +92,11 @@ Currently, SAI already defines a few port attributes to support port auto negoti
3. If autoneg is enabled and adv_speeds is not configured or empty, SAI must advertise it with all supported speeds.
4. If autoneg is enabled and adv_interface_types is not configured or empty, SAI must advertise it with all supported interface types.
5. If autoneg is disabled and interface_type is not configured, SAI must use SAI_PORT_INTERFACE_TYPE_NONE.
+6. If autoneg is enabled, the administrative port speed updates should not disable the autoneg. The configured speed should be cached in swss#orchagent and gets replayed when autoneg is transitioned from enabled to disabled.
+7. If autoneg is enabled, while the administrative interface type updates via CONFIG_DB should be blocked, the dynamic interface type updates from pmon#xcvrd via APPL_DB should be delivered to the SAI to update the autoneg advertisement. Please refer to [PMON xcvrd Consideration](#pmon-xcvrd-consideration) for details.
+8. If autoneg is enabled on a SFP/QSFP/QSFPDD port, SAI should also activate the link-training to dynamically tune the TX FIR.
+9. If autoneg is enabled on a SFP/QSFP/QSFPDD port, when it's transitioned to disabled, the port speed, FEC, interface type and TX FIR should be restored with the corresponding values in the APPL_DB. If the individual configuration is not present in the APPL_DB, the SAI driver defaults should be restored. Precisely speaking, default interface type is SAI_PORT_INTERFACE_TYPE_NONE, default fec mode is SAI_PORT_FEC_MODE_NONE and TX FIR is vendor-specific, as it's NOT a set of constants applicable to all switch ASIC.
+10. If autoneg is enabled on a SFP/QSFP/QSFPDD port, the "fec" field of APPL_DB should override the result of autoneg.
The related port attributes are listed below:
```cpp
@@ -234,26 +245,29 @@ This command always replace the advertised interface types instead of append. Fo
##### Show interfaces auto negotiation status
-As command `show interfaces status` already has 11 columns, a new CLI command will be added to display the port auto negotiation status. All data of this command are fetched from **APPL_DB**.
+As command `show interfaces status` already has 11 columns, a new CLI command will be added to display the port auto negotiation status. All data of this command are fetched from **APPL_DB** and **STATE_DB**.
```
Format:
- show interfaces autoneg-status
+ show interfaces autoneg status
Arguments:
interface_name: optional. Name of the interface to be shown. e.g: Ethernet0. If interface_name is not given, this command shows auto negotiation status for all interfaces.
Example:
- show interfaces autoneg-status
- show interfaces autoneg-status Ethernet0
+ show interfaces autoneg status
+ show interfaces autoneg status Ethernet0
Return:
error message if interface_name is invalid otherwise:
- Interface Auto-Neg Mode Speed Adv Speeds Type Adv Types Oper Admin
------------ --------------- ------- ------------ ------ ----------- ------ -------
- Ethernet0 disabled 25G N/A N/A N/A down up
- Ethernet32 enabled 40G 40G,100G KR4 KR4,CR4 up up
+ Interface Auto-Neg Mode Speed Adv Speeds Rmt Adv Speeds Type Adv Types Oper Admin
+----------- --------------- ------- ------------ ---------------- ------ ----------- ------ -------
+ Ethernet0 enabled 400G N/A 400G CR4 CR4 up up
+ Ethernet8 enabled 400G N/A N/A KR4 KR4 down up
+ Ethernet16 enabled 400G N/A 400G CR4 CR4 up up
+ Ethernet24 N/A 400G N/A N/A N/A N/A down up
+ Ethernet32 disabled 400G N/A N/A N/A N/A down up
```
##### CLI validation
@@ -379,7 +393,7 @@ The change in APP_DB is similar to CONFIG_DB. 3 new fields **adv_speeds**, **int
Valid value of the new fields are the same as **PORT** table in CONFIG_DB.
Here is the table to map the fields and SAI attributes:
-| **Parameter** | **sai_port_attr_t**
+| **Parameter** | **sai_port_attr_t** |
|---------------------|------------------------------------------------|
| adv_interface_types | SAI_PORT_ATTR_ADVERTISED_INTERFACE_TYPE |
| adv_speeds | SAI_PORT_ATTR_ADVERTISED_SPEED |
@@ -390,15 +404,19 @@ Here is the table to map the fields and SAI attributes:
#### State DB Enhancements
To support validate interface speed on CLI side, a new field **supported_speeds** will be added to **PORT_TABLE**.
+To support checking remote advertised speeds, a new field **rmt_adv_speeds** will be added to **PORT_TABLE**.
; Defines information for port state
key = PORT_TABLE:port_name ; state of the port
; field = value
...
supported_speeds = STRING ; supported speed list
- speed = STRING ; operational speed
+ speed = STRING ; operational speed
+ rmt_adv_speeds = STRING ; advertised speed list of the remote
-An example value of supported_speeds could be "10000,25000,40000,100000".
+- supported_speeds: A list of supported speeds separated by commas, for example, "10000,25000,40000,100000".
+- speed: Operational speed, for example, "100000".
+- rmt_adv_speeds: A list of supported speeds advertised by the connected remote, for example, "10000,25000,40000,100000".
Before this feature, port speed in APP DB indicates both the configured speed and the operational speed. It is OK without this feature because port operational speed must be configured speed or port operational status is down. However, this is not true with this feature. Consider following flow:
@@ -413,6 +431,13 @@ To overcome this issue, following changes are required:
1. Put port operational speed to STATE DB PORT_TABLE
2. intfutil, portstat, voqutil shall be change to get port operational speed from STATE DB first. For backward compatible, intfutil, portstat, voqutil shall still get port operational speed from APP DB if port speed is not available in STATE DB or port operational state is down.
+Here is the table to map the fields and SAI attributes:
+| **Parameter** | **sai_port_attr_t** |
+|---------------------|------------------------------------------------|
+| rmt_adv_speeds | SAI_PORT_ATTR_REMOTE_ADVERTISED_SPEED |
+| speed | SAI_PORT_ATTR_SPEED |
+| supported_speeds | SAI_PORT_ATTR_SUPPORTED_SPEED |
+
#### YANG Model Enhancements
The port yang model needs to update according to DB schema change. The yang model changes of new fields are described below:
@@ -420,7 +445,7 @@ The port yang model needs to update according to DB schema change. The yang mode
```
leaf autoneg {
type string {
- pattern "0|1";
+ pattern "off"|"on";
}
}
@@ -470,6 +495,8 @@ Will be migrated to:
#### SWSS Enhancements
+##### Setting Auto Negotiation
+
The current SONiC speed setting flow in PortsOrch can be described in following pseudo code:
```
@@ -511,6 +538,14 @@ else if autoneg == false:
setInterfaceType(port, interface_type)
```
+##### Getting Remote Advertisement
+
+A new periodic timer task will be introduced into PortsOrch, it periodically loops through physical ports and update the per-port remote advertisement if autoneg is enabled and the link is down.
+
+
+
+##### Backward Compatibility Considerations
+
SONiC usually does not call SAI interface when there is no related configuration in APPL_DB. In order to keep backward compatible, this feature also apply this rule.
swss will do validation for auto negotiation related fields, although it still CANNOT guarantee that all parameters passed to SAI will be accepted by SAI. swss validation for these field are described below:
@@ -520,6 +555,7 @@ swss will do validation for auto negotiation related fields, although it still C
3. interface_type value from APPL_DB must be able to transfer to a valid interface type value. For invalid value, swss must catch the exception, log the error value and skip the rest configuration of this port.
4. adv_interface_types value from APPL_DB must be able to transfer to a list of valid interface type values. For invalid value, swss must catch the exception, log the error value and skip the rest configuration of this port.
+
#### portsyncd and portmgrd Consideration
No changes for portsyncd and portmgrd.
@@ -617,18 +653,15 @@ I choose this solution because:
Dynamic port breakout feature also introduces a hwsku.json file to describe the port capability. It defines the default dynamic breakout mode for now. As we won't automatically set auto negotiation attributes for a port after port breakout, it is not necessary to change the hwsku.json in this feature.
-#### xcvrd Consideration
-
-It is recommended to use CLI/CONFIG_DB to setting the port auto negotiation attributes. But there is still other way.
-
-There is a media_setting.json which is used for setting the default value for some port attributes. This media_setting.json is handled by xcvrd for setting pre-emphasis value for ports. xcvrd process the file this way:
+#### PMON xcvrd Consideration
-1. When xcvrd start, it reads the media_setting.json and set pre-emphasis values to APPL_DB for each port
-2. When a new cable is inserted, xcvrd uses the value in media_setting.json to set pre-emphasis value to APPL_DB for this port.
+While it is possible to use CLI/CONFIG_DB for setting the port auto negotiation attributes, this feature is also available via [media_settings.json](https://github.com/Azure/SONiC/blob/master/doc/media-settings/Media-based-Port-settings.md)
-xcvrd just reads the configuration from media_settings.json and set value to APPL_DB. xcvrd does not care what the configuration is. So user could use media_settings.json to specify the default value for autoneg, adv_speeds and other port attributes. **Nothing needs to be changed in xcvrd**.
+- **Interface Type**
+ Unfortunately, it's not straightforward for users to identify which interface type is most appropriate to the attached transceivers, and the link will not get up unless the connected devices are both advertising the same interface type. This requires domain knowledge, correct EERPOM information and the individual hardware datasheet review process. Hence it's recommended to use [media_settings.json](https://github.com/Azure/SONiC/blob/master/doc/media-settings/Media-based-Port-settings.md) to automate the while process. A interface type update request from [media_settings.json](https://github.com/Azure/SONiC/blob/master/doc/media-settings/Media-based-Port-settings.md) is triggered by the transceiver detection of pmon#xcvrd, which leverages **APPL_DB** instead of **CONFIG_DB**, hence the requests will not be blocked by **portsyncd**. If the interface type update request arrives when autoneg is enabled, it should alter the advertisement and restart the autoneg. That said, if the user has interface type configured in both CONFIG_DB and media_settings.json, it wouldn't be possible to predict which would take precedence as there is no logic giving one priority over other. Hence please be sure to use either CLI/CONFIG_DB or media_settings.json at a time, never have both of them activated.
-However, it is worthy mentioning that: if user have port attributes configured both in CONFIG_DB and media_settings.json, the value in media_settings.json will override the value in CONFIG_DB after rebooting, restarting pmon or re-insert cables. Base on that, if user choose to use media_settings.json, they probably should not use CLI or CONFIG_DB to avoid configuration loss after rebooting, restarting pmon or re-insert cables.
+- **Pre-Emphasis**
+ Typically, prior to some process, such as transmission over cable, or recording to phonograph record or tape, the input frequency range most susceptible to noise is boosted. This is referred to as "pre-emphasis" - before the process the signal will undergo. While this is rarely necessay to the native RJ45 ports, it's important to SFP/QSFP/QSFPDD ports. For optical arrangements (e.g. SR/LR/DR transceivers), the loss is miniscule. The loss budget allocated for the host PCB board is small, which implies the pre-emphasis calibrated is supposed to work regardless of the fiber cable attached. On the other hand, on passive media, there is a significant amount of frequency dependent loss and the channel loss can range up to the CR/KR loss spec. It is therefore important and useful to activate Link-Training to "train" the TX FIR in both directions to help equalize that loss. IEEE 802.3 Clause 72, Clause 73 and Clause 93 define the auto-negotiation and link-training support for CR/KR transceivers, while the support of opticals (e.g. SR/LR/DR) are never in the scope. On the other hand, when autoneg is enabled, the link-training will also be activated, and the link-training handshake will only get started after negotiation if autoneg is enabled. Furtherly, link-training is to dynamically tune hardware signals at runtime, as a result, the static pre-emphasis parameters in media_setting.json is not necessary and will not be taken into autoneg process.
### Warmboot and Fastboot Design Impact
diff --git a/doc/port_link_training/flow-lt-config.png b/doc/port_link_training/flow-lt-config.png
new file mode 100644
index 0000000000..82f8d33726
Binary files /dev/null and b/doc/port_link_training/flow-lt-config.png differ
diff --git a/doc/port_link_training/flow-lt-status.png b/doc/port_link_training/flow-lt-status.png
new file mode 100644
index 0000000000..133e3662ec
Binary files /dev/null and b/doc/port_link_training/flow-lt-status.png differ
diff --git a/doc/port_link_training/port-link-training-design.md b/doc/port_link_training/port-link-training-design.md
new file mode 100644
index 0000000000..18db296b67
--- /dev/null
+++ b/doc/port_link_training/port-link-training-design.md
@@ -0,0 +1,343 @@
+# SONiC Port Link Training Design #
+
+## Table of Content
+
+- [Revision](#revision)
+- [Scope](#scope)
+- [Abbreviations](#abbreviations)
+- [Overview](#overview)
+- [Requirements](#requirements)
+- [Architecture Design](#architecture-design)
+- [High-Level Design](#high-level-design)
+- [SAI API Requirement](#sai-api-requirement)
+- [Configuration and management ](#configuration-and-management)
+ - [CLI Enhancements](#cli-enhancements)
+ - [Config link training mode](#config-link-training-mode)
+ - [Show link training status](#show-link-training-status)
+ - [Config DB Enhancements](#config-db-enhancements)
+ - [Application DB Enhancements](#application-db-enhancements)
+ - [State DB Enhancements](#state-db-enhancements)
+ - [YANG Model Enhancements](#yang-model-enhancements)
+ - [DB Migration Considerations](#db-migration-considerations)
+ - [SWSS Enhancements](#swss-enhancements)
+- [Warmboot Design Impact](#warmboot-design-impact)
+- [Limitations](#limitations)
+- [Testing Requirements](#testing-requirements)
+ - [Unit Test cases](#unit-test-cases)
+ - [System Test cases](#system-test-cases)
+- [Action items](#action-items)
+
+### Revision
+
+ | Rev | Date | Author | Change Description |
+ |:---:|:-----------:|:------------------:|-----------------------------------|
+ | 0.1 | | Dante (Kuo-Jung) Su| Initial version |
+ | 0.2 | | Dante (Kuo-Jung) Su| Addressing code review comments |
+
+### Scope
+This document is the design document for port link training feature on SONiC. This includes the requirements, DB schema change, DB migrator change, yang model change and swss change.
+
+The link-training hardware controls in this document is SAI specific, while the Gearbox design is outside the scope of this document.
+
+### Abbreviations
+
+ | Term | Meaning |
+ |:-------:|:----------------------------------------|
+ | ASIC | Application-Specific Integrated Circuit |
+ | BER | Bit Error Rate |
+ | FIR | Finite Impulse Response |
+ | SFP | Small Form-factor Pluggable transceiver |
+
+### Overview
+
+Link training is a process by which the transmitter and receiver on a high-speed serial link communicate with each other in order to tune their equalization settings. In theory, link training enables automatic tuning of the FIR filter for each channel in an ASIC to achieve the desired bit error rate (BER)
+
+In current SONiC implementation, user can leverage the platform-specific [media_settings.json](https://github.com/Azure/SONiC/blob/master/doc/media-settings/Media-based-Port-settings.md) to statically update the TX FIR per attached transceiver to improve BER. However, the ODM vendors rarely provide the pre-calibrated pre-emphasis for the CR/KR transceivers, which could result in the link reliability issues.
+
+The IEEE 802.3 standard defines a set of link training protocols for various mediums, and the feature in this document is to focus on IEEE clause 72 and 93 to dynamically improve the link quality over the SFP coppers and backplanes.
+
+This feature could be activated with or without port auto negotiation.
+
+### Requirements
+
+The main goal of this document is to discuss the design of following requirement:
+
+- Allow user to configure link training
+- Allow user to get the operational link training status
+
+### Architecture Design
+
+This feature does not change the existing SONiC architecture, while it has to change the configuration flow for port link training which will be covered in orchagent.
+
+### High-Level Design
+
+- SAI API requirements is covered in section [SAI API Requirement](#sai-api-requirement).
+- A few new CLI commands will be added to sonic-utilities sub module. These CLI commands support user to configure link training mode as well as show port link training status. See detail description in section [CLI Enhancements](#cli-enhancements).
+- A few new fields will be added to existing table in APPL_DB and CONFIG_DB to support link training attributes. See detail description in section [Config DB Enhancements](#config-db-enhancements), [Application DB Enhancements](#application-db-enhancements) and [State DB Enhancements](#state-db-enhancements)
+- YANG Model needs update according to the DB schema change. See detail description in section [YANG Model Enhancements](#yang-model-enhancements)
+- Port configuration setting flow will be changed in orchagent of sonic-swss. See detail description in section [SWSS Enhancements](#swss-enhancements).
+
+### SAI API Requirement
+
+First of all, we'll leverage the existing SAI port attributes listed below
+
+```cpp
+ /**
+ * @brief Enable/Disable Port Link Training
+ *
+ * @type bool
+ * @flags CREATE_AND_SET
+ * @default false
+ */
+ SAI_PORT_ATTR_LINK_TRAINING_ENABLE,
+
+ /**
+ * @brief Link training failure status and error codes
+ *
+ * @type sai_port_link_training_failure_status_t
+ * @flags READ_ONLY
+ */
+ SAI_PORT_ATTR_LINK_TRAINING_FAILURE_STATUS,
+
+ /**
+ * @brief Status whether the receiver trained or not trained to receive data
+ *
+ * @type sai_port_link_training_rx_status_t
+ * @flags READ_ONLY
+ */
+ SAI_PORT_ATTR_LINK_TRAINING_RX_STATUS,
+```
+
+On the other hand, a new port attribute will be introduced for the link-training ability query, and please note the link-training ability could be unavailable on the certain ports on a switch silicon (e.g. The uplink/management ports), hence we need to do the query at per-port basis.
+
+```cpp
+ /**
+ * @brief Query link-training support
+ *
+ * @type bool
+ * @flags READ_ONLY
+ */
+ SAI_PORT_ATTR_SUPPORTED_LINK_TRAINING_MODE,
+```
+
+Vendor-specific SAI implementation is not in the scope of this document, but there are some common requirements for SAI:
+
+1. swss must check the link-training abilities before making the corresponding requests to syncd.
+2. SAI implementation must return error code if any of the above attributes is not supported, swss and syncd must not crash.
+3. SAI implementation must keep port link training disabled by default for backward compatibility. As long as swss and SAI keep backward compatible, user need not change anything after this feature is implemented and available in SONiC.
+
+### Configuration and management
+
+#### CLI Enhancements
+
+A few new CLI commands are designed to support port link training.
+
+##### Config link training mode
+
+
+```
+Format:
+ config interface link-training
+
+Arguments:
+ interface_name: [mandatory] name of the interface to be configured. e.g: Ethernet0
+ mode: [mandatory] link training mode, can be either "on" or "off"
+
+Example:
+ config interface link-training Ethernet0 on
+ config interface link-training Ethernet0 off
+
+Return:
+ - error message if interface_name or mode is invalid
+ - empty upon success
+```
+
+##### Show link training status
+
+A new CLI command will be added to display the port link training status. All data of this command are fetched from **STATE_DB**.
+
+```
+Format:
+ show interfaces link-training status
+
+Arguments:
+ interface_name: [optional] Name of the interface to be shown. e.g: Ethernet0.
+ If interface_name is not given, this command shows link training status for all interfaces.
+
+Example:
+ show interfaces link-training status
+ show interfaces link-training status Ethernet0
+
+Return:
+ error message if interface_name is invalid otherwise:
+
+admin@sonic:~$ show interfaces link-training status
+ Interface LT Oper LT Admin Oper Admin
+----------- ----------- ---------- ------ -------
+ Ethernet0 trained on up up
+ Ethernet8 off - down up
+ Ethernet16 trained on up up
+ Ethernet24 off - down up
+ Ethernet32 not trained on down up
+ Ethernet40 off - down up
+ Ethernet48 off - down up
+```
+
+
+#### Config DB Enhancements
+
+A new field **link_training** will be added to **PORT** table:
+
+ ; Defines information for port configuration
+ key = PORT|port_name ; configuration of the port
+ ; field = value
+ ...
+ link_training = STRING ; link training configuration
+
+Valid value of the new fields are described below:
+
+- link_training:
+String value, the administratively specified port link training configuration.
+When "link_training" is not specified, the port link training should be disabled
+
+ | Mode | Description |
+ |:----:|:---------------------------------------------------------------:|
+ | on | Enable link-training |
+ | off | Disable link-training |
+
+#### Application DB Enhancements
+
+The following fields will be introduced into **PORT_TABLE** table:
+
+ ; Defines information for port configuration
+ key = PORT_TABLE:port_name ; configuration of the port
+ ; field = value
+ ...
+ link_training = STRING ; operational link training admin config
+
+- link_training:
+String value, the port link training admin config.
+
+#### State DB Enhancements
+
+The following fields will be introduced into **PORT_TABLE** table:
+
+ ; Defines information for port link-training states
+ key = PORT_TABLE:port_name ; port link-training states
+ ; field = value
+ link_training_status = STRING ; port link-training operational status
+
+- link_training_status:
+String value, the operational port link training status. The list of possible values is as follow
+
+ | Status | Description |
+ |:-----------:|:------------------------------------------------------------------:|
+ | off | Disabled |
+ | on | Enabled, while the further operational status is not yet available |
+ | trained | Enabled, while the pre-emphasis is tuned successfully |
+ | not_trained | Enabled, while the pre-emphasis is not yet tuned, no further failure is available |
+ | frame_lock | Enabled, while the training frame delineation is detected |
+ | snr_low | Enabled, while the SNR low threshold is detected |
+ | timeout | Enabled, while the training process is timed out |
+
+#### SAI attributes
+
+Here is the table to map the fields and SAI attributes:
+
+| **Parameter** | **sai_port_attr_t** |
+|:--------------------------|:-------------------------------------------|
+| link_training | SAI_PORT_ATTR_LINK_TRAINING_ENABLE |
+| link_training_status | SAI_PORT_ATTR_LINK_TRAINING_RX_STATUS, SAI_PORT_ATTR_LINK_TRAINING_FAILURE_STATUS |
+
+#### YANG Model Enhancements
+
+The port yang model needs to update according to DB schema change. The yang model changes of new fields are described below:
+
+```
+leaf link_training {
+ type string {
+ pattern "on|off";
+ }
+}
+```
+
+These new yang leaf will be added to sonic-port.yang.
+
+#### DB Migration Considerations
+
+By having port link training disabled if "link_training" is not specified in the CONFIG_DB, this feature is fully backward compatible.
+
+#### SWSS Enhancements
+
+##### Port Link-Training Ability Flag
+
+During system startup, PortsOrch should query for the per-port link-training abilities from syncd, and have these per-port flags maintained in the **m_port_cap_lt** field of **Port** object.
+
+i.e.
+```cpp
+class Port
+{
+ ......
+ bool m_port_cap_lt = false; /* Port Capability - LinkTraining */
+};
+```
+
+##### Port Link-Training Configuration Flow
+
+in PortsOrch, the port Link-Training configuration flow is as follows, the link-training may or may not be disabled when auto-negotiation is activated, it depends on the switch ASIC limitations in the individual SAI implementation, and the pre-emphasis configuration request should be saved and replayed upon link-training configuration updates:
+
+
+
+##### Port Link-Training Status Poll
+
+A timer thread will be introduced into PortsOrch for polling link-training status, the per-port status poll will be activated upon the following events
+
+- Link-Training is transitoned to be enabled on a port
+- Link is down on a port with Link-Training enabled
+
+On the other hand, the per-port status poll will be de-activated upon the following events
+
+- Link-Training is transitoned to be disabled on a port
+- Link is up on a port with Link-Training enabled
+
+This timer is a single-threaded task that polls the status of all ports one by one, the program flow is as follows
+
+
+
+### Warmboot Design Impact
+
+SAI and lower layer must not flap port during warmboot no matter what link training parameter is given.
+
+### Limitations
+
+N/A
+
+### Testing Requirements
+
+#### Unit Test cases
+
+For **sonic-swss**, we will leverage the existing [test_port.py](https://github.com/Azure/sonic-swss/blob/master/tests/test_port.py) for this. A few new test cases will be added:
+
+1. Test attribute **link_training** on both direct and warm-reboot scenario. Verify SAI_PORT_ATTR_LINK_TRAINING_ENABLE is in ASIC_DB and has correct value.
+
+For **sonic-utilities**, we will leverage the existing [unit test framework](https://github.com/Azure/sonic-utilities/tree/master/tests) for this. A few new test cases will be added:
+
+1. Test command `config interface link-training `. Verify the command return error if given invalid interface_name or mode.
+
+#### System Test cases
+
+The following tests will be introduced into **sonic-mgmt** for this feature.
+
+- test_link_training
+ Test all candidate ports by having interface/media type updated to copper followed by link-training enablement. Verification criteria:
+ - All ports are up when link-training is activated
+ - All ports are up with link-training status equals to 'trained'
+- test_link_training_negative
+ Test all candidate ports by having interface/media type updated to fiber followed by link-training enablement. Verification criteria:
+ - All ports are down when link-training is activated
+ - All ports are down without link-training status equals to 'trained'
+
+### Action items
+
+TBD
+
diff --git a/doc/profiling/img/bootchart.svg b/doc/profiling/img/bootchart.svg
new file mode 100755
index 0000000000..d49029549b
--- /dev/null
+++ b/doc/profiling/img/bootchart.svg
@@ -0,0 +1,56808 @@
+
+
+
diff --git a/doc/profiling/sonic_bootchart.md b/doc/profiling/sonic_bootchart.md
new file mode 100755
index 0000000000..aee21d21e0
--- /dev/null
+++ b/doc/profiling/sonic_bootchart.md
@@ -0,0 +1,231 @@
+
+# SONiC Boot Chart #
+
+
+## Table of Content
+
+- [Revision](#revision)
+- [Scope](#scope)
+- [Definitions/Abbreviations](#definitionsabbreviations)
+- [Overview](#overview)
+- [Requirements](#requirements)
+- [Architecture Design](#architecture-design)
+- [High-Level Design](#high-level-design)
+- [SAI API](#sai-api)
+- [Configuration and management](#configuration-and-management)
+ - [Manifest (if the feature is an Application Extension)](#manifest-if-the-feature-is-an-application-extension)
+ - [CLI/YANG model Enhancements](#cliyang-model-enhancements)
+ - [Config DB Enhancements](#config-db-enhancements)
+- [Warmboot and Fastboot Design Impact](#warmboot-and-fastboot-design-impact)
+- [Restrictions/Limitations](#restrictionslimitations)
+- [Testing Requirements/Design](#testing-requirementsdesign)
+ - [Unit Test cases](#unit-test-cases)
+ - [System Test cases](#system-test-cases)
+- [Open/Action items](#openaction-items)
+
+### Revision
+
+### Scope
+
+SONiC OS is highly modular and composible system, where every new feature is
+often implemented as a new script, utility, daemon or docker container. A lot of
+these components start at SONiC boot togather with respect to their dependencies
+requirements. Majority of start scripts use short-living small utilities written
+in python, bash or other scripting languages, invoke Jinja2 template generation.
+This leads to potential boot time performance degradation that we need a tool to
+detect and analyze.
+
+### Definitions/Abbreviations
+
+N/A
+
+### Overview
+
+This document describes an integration of one of systemd tools called
+*systemd-bootchart*. This tool is a sampling based system profiler that is used
+to analyze boot up performance but not limited to and can be used to collect
+samples after the system is booted. The output produced by systemd-bootchart is
+an SVG image that looks like this:
+
+
+
+
+
+### Requirements
+
+- Integrate systemd-bootchart with SONiC OS
+- systemd-bootchart is by default installed in the system
+- systemd-bootchart is by default disabled and user need to enable it via CLI
+- SONiC CLI tool to interact with systemd-bootchart
+- Support commands to enable/disable systemd-bootchart
+- Configure the amount of samples to collect and frequency
+- Displaying systemd-bootchart configuration and generated SVG plots
+
+### Architecture Design
+
+N/A
+
+### High-Level Design
+
+SONiC build system includes a new build-time flag to INCLUDE_BOOTCHART (y/n).
+When this flag is set a *systemd-bootchart* debian package is installed in SONiC
+host from upstream debian repositories (Installed size 128 KB).
+
+Example INCLUDE_BOOTCHART flag usage:
+
+Include bootchart:
+```
+make INCLUDE_BOOTCHART=y target/sonic-mellanox.bin
+```
+
+Do not include bootchart:
+```
+make INCLUDE_BOOTCHART=n target/sonic-mellanox.bin
+```
+
+SONiC provides a default configuration for bootchart located at
+/etc/systemd/bootchart.conf:
+
+```ini
+[Bootchart]
+Samples=4500
+Frequency=25
+```
+
+This configuration means that samples are collected for 3 minutes of uptime with
+a frequency of 25 samples per second.
+
+SONiC utilities is extendeded with a new script under *scripts/* to include a
+new utility *sonic-bootchart*.
+
+Also a flag added to the build incdicating default status:
+
+```
+ENABLE_BOOTCHART=y
+```
+
+### SAI API
+
+N/A
+
+### Configuration and management
+
+*sonic-bootchart* is a standalone utility, like *sonic-kdump-config*,
+**sonic-installer*, etc.
+
+Command line interface:
+
+```
+admin@sonic:~$ sudo sonic-bootchart
+Usage: sonic-bootchart [OPTIONS] COMMAND [ARGS]...
+
+ Main CLI group
+
+Options:
+ --help Show this message and exit.
+
+Commands:
+ config Configure bootchart NOTE: This command requires elevated (root)...
+ disable Disable bootchart NOTE: This command requires elevated (root)...
+ enable Enable bootchart NOTE: This command requires elevated (root)...
+ show Display bootchart configuration
+```
+
+Enable/Disable bootchart:
+
+```
+admin@sonic:~$ sudo sonic-bootchart enable
+Running command: systemctl enable systemd-bootchart
+admin@sonic:~$ sudo sonic-bootchart disable
+Running command: systemctl disable systemd-bootchart
+```
+
+In case image is built without bootchart included (INCLUDE_BOOTCHART=n)
+sonic-bootchart will print an error:
+
+```
+admin@sonic:~$ sudo sonic-bootchart enable
+systemd-bootchart is not installed
+```
+
+Once bootchart is enabled a systemd-bootchart.service is enabled in systemd
+which starts a daemon early at boot that collects samples. This setting is
+permanent, meaning, after it is enabled, bootchart will always run at boot until
+disabled, *config save*, *config reload* do not affect this setting.
+
+
+Update configuration example:
+
+```
+admin@sonic:~$ sudo sonic-bootchart config --time-span 50 --frequency 10
+```
+
+This command will update /etc/systemd/bootchart.conf. The next time bootchart
+will run (at system start after reboot) it will take the new values.
+*systemd-bootchart* saves the plots to a temporary directory /run/log that is
+*flushed after reboot.
+
+Displaying configuration, operational status and output SVG plots:
+
+```
+admin@sonic:~$ sudo sonic-bootchart show
+Status Operational Status Frequency Time Span (sec) Output
+-------- -------------------- ----------- ---------------- ------------------------------------
+enabled in-active 10 50 /run/log/bootchart-20220504-1325.svg
+```
+
+Fields description:
+
+| Field | Comment |
+| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
+| Status | Output of "systemctl is-enabled systemd-bootchart". Shows whether this service is configured to start at boot (enabled/disabled) |
+| Operational Status | Output of "systemctl is-action systemd-bootchart". Shows whether this service is currently collecting samples (active/in-active) |
+| Frequency | How frequent to collect samples per second |
+| Time Span | The time how long after boot samples will be collected. Based on it the samples count is calculated as Frequency x Time Span |
+| Output | If systemd-bootchart finished collecting samples (in-active) this column will display resulting plots, otherwise this column is empty |
+
+
+Usage example:
+
+- First make sure you have SONiC image built with INCLUDE_BOOTCHART=y
+- Enable bootchart: "sudo sonic-bootchart enable"
+- Perform any kind of reboot: "sudo reboot" or "sudo warm-reboot" or "sudo
+fast-reboot"
+ - Wait till system reboots and the plot is generated by quering
+"sudo sonic-bootchart show"
+
+#### Manifest (if the feature is an Application Extension)
+
+N/A
+
+#### CLI/YANG model Enhancements
+
+N/A
+
+#### Config DB Enhancements
+
+N/A
+
+### Warmboot and Fastboot Design Impact
+N/A
+
+### Restrictions/Limitations
+
+### Testing Requirements/Design
+N/A
+
+#### Unit Test cases
+
+Cover CLI with UT for:
+ - enabling bootchart
+ - disabling bootchart
+ - config/show commands
+
+#### System Test cases
+
+N/A
+
+### Open/Action items
+
+- SONiC installer bootchart config migration. Being able to run this tool for reboot with SONiC-2-SONiC upgrade
+- Requested a possibility to run the tool in runtime after boot
diff --git a/doc/qos/reclaim-reserved-buffer.md b/doc/qos/reclaim-reserved-buffer.md
index dd6e8d8791..f2b5df0dbb 100644
--- a/doc/qos/reclaim-reserved-buffer.md
+++ b/doc/qos/reclaim-reserved-buffer.md
@@ -166,6 +166,7 @@ The following snippet of code is to generate `PORT_INACTIVE` which contains all
They need to be moved from the middle of `buffer_template.j2` to the place just before the vendor specific template is imported, so that the vendor specific template has access to `PORT_INACTIVE`.
```json
+{% raw %} # ignore this line please
{%- set PORT_ALL = [] %}
{%- if PORT is not defined %}
@@ -192,16 +193,19 @@ They need to be moved from the middle of `buffer_template.j2` to the place just
{# Import default values from device HWSKU folder #}
{%- import 'buffers_defaults_%s.j2' % filename_postfix as defs with context %}
+{% endraw %} # ignore this line please
```
The vairable `port_names_inactive` also need to be generated by the following snipped of code.
```json
+{% raw %} # ignore this line please
{%- set port_names_list_inactive = [] %}
{%- for port in PORT_INACTIVE %}
{%- if port_names_list_inactive.append(port) %}{%- endif %}
{%- endfor %}
{%- set port_names_inactive = port_names_list_inactive | join(',') %}
+{% endraw %} # ignore this line please
```
##### 7.1.1.2 Macro to generate buffer pool and profiles #####
@@ -211,6 +215,7 @@ Zero buffer profiles should be defined for ingress/egress and lossless/lossy tra
This macro is defined in vendor specific buffer template files.
```json
+{% raw %} # ignore this line please
{%- macro generate_buffer_pool_and_profiles() %}
"BUFFER_POOL": {
{%- if dynamic_mode is not defined and PORT_INACTIVE is defined and PORT_INACTIVE|length > 0 %}
@@ -302,6 +307,7 @@ This macro is defined in vendor specific buffer template files.
}
},
{%- endmacro %}
+{% endraw %} # ignore this line please
```
##### 7.1.1.3 Macro to apply zero buffer profiles to inactive ports #####
@@ -321,6 +327,7 @@ These macros are defined in vendor specific buffer template files.
generate_profile_lists_with_inactive_ports:
```json
+{% raw %} # ignore this line please
{%- macro generate_profile_lists_with_inactive_ports(port_names_active, port_names_inactive) %}
"BUFFER_PORT_INGRESS_PROFILE_LIST": {
{% for port in port_names_active.split(',') %}
@@ -365,11 +372,13 @@ generate_profile_lists_with_inactive_ports:
{% endif %}
}
{%- endmacro %}
+{% endraw %} # ignore this line please
```
generate_pg_profiles_with_inactive_ports:
```json
+{% raw %} # ignore this line please
{%- macro generate_pg_profiles(port_names_active, port_names_inactive) %}
"BUFFER_PG": {
{% for port in port_names_active.split(',') %}
@@ -403,11 +412,13 @@ generate_pg_profiles_with_inactive_ports:
{% endif %}
}
{%- endmacro %}
+{% endraw %} # ignore this line please
```
generate_queue_buffers_with_inactive_ports:
```json
+{% raw %} # ignore this line please
{%- macro generate_queue_buffers(port_names_active, port_names_inactive) %}
"BUFFER_QUEUE": {
{% for port in port_names_active.split(',') %}
@@ -466,6 +477,7 @@ generate_queue_buffers_with_inactive_ports:
{% endif %}
}
{%- endmacro %}
+{% endraw %} # ignore this line please
```
Assume port `Ethernet0` is admin down, an example is:
diff --git a/doc/qos/tunnel_dscp_remapping.md b/doc/qos/tunnel_dscp_remapping.md
index c93e787c67..d20fc108cd 100644
--- a/doc/qos/tunnel_dscp_remapping.md
+++ b/doc/qos/tunnel_dscp_remapping.md
@@ -105,7 +105,7 @@ Before remapping to queue 2 and 6, both queues are required to be cleared. Hence
"30": "1",
"31": "1",
"32": "1",
- "33": "1",
+ "33": "2", // Original map "33" : "1"
"34": "1",
"35": "1",
"36": "1",
@@ -180,7 +180,7 @@ Before remapping to queue 2 and 6, both queues are required to be cleared. Hence
"AZURE_TUNNEL": {
"0": "8",
"1": "0",
- "2": "0",
+ "2": "33",
"3": "2",
"4": "6",
"5": "46",
diff --git a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md
index aef8958462..227c0d4176 100644
--- a/doc/sfp-cmis/Interface-Link-bring-up-sequence.md
+++ b/doc/sfp-cmis/Interface-Link-bring-up-sequence.md
@@ -23,8 +23,8 @@ Deterministic Approach for Interface Link bring-up sequence
* [Table 2: References](#table-2-references)
# Revision
-| Rev | Date | Author | Change Description |
-|:---:|:-----------:|:----------------------------------:|-----------------------------------|
+| Rev | Date | Author | Change Description |
+|:---:|:-----------:|:----------------------------------:|-------------------------------------|
| 0.1 | 08/16/2021 | Shyam Kumar | Initial version
| 0.2 | 12/13/2021 | Shyam Kumar, Jaganathan Anbalagan | Added uses-cases, workflows
| 0.3 | 01/19/2022 | Shyam Kumar, Jaganathan Anbalagan | Addressed review-comments
@@ -32,6 +32,8 @@ Deterministic Approach for Interface Link bring-up sequence
| 0.5 | 01/28/2022 | Shyam Kumar, Jaganathan Anbalagan | Addressed further review-comments
| 0.6 | 02/02/2022 | Shyam Kumar | Added feature-enablement workflow
| 0.7 | 02/02/2022 | Jaganathan Anbalagan | Added Breakout Handling
+| 0.8 | 02/16/2022 | Shyam Kumar | Updated feature-enablement workflow
+| 0.9 | 04/05/2022 | Shyam Kumar | Addressed review comments |
# About this Manual
@@ -149,11 +151,18 @@ Please refer to the flow/sequence diagrams which covers the following required
# Feature enablement
This feature (optics Interface Link bring-up sequence) would be enabled on per platform basis.
There could be cases where vendor(s)/platform(s) may take time to shift from existing codebase to the model (work-flows) described in this document.
- In order to avoid any breakage and ensure gradual migration of different platforms/vendors to this model, there would be new field (flag) in xcvrd to enable/disable this feature.
- When xcvrd spawns on LC/board, it would invoke platform plugin to check with the platform (hwsku) whether this feature is yet supported on underlying platform (board/LC) or not
+ In order to avoid any breakage and ensure gradual migration of different platforms/vendors to this model, will add this new workflow to enable/disable this feature:
+
+ In order to enable this feature, the platform would set ‘skip_xcvrd_cmis_mgr’ to ‘false’ in their respective pmon_daemon_control.json as part of platform bootstrap. When xcvrd would spawn on that hwsku (LC/board), it would parse ‘skip_xcvrd_cmis_mgr’ and if found 'false', it would launch CMIS task manager. This implies enabling this feature.
+
+Else, if ‘skip_xcvrd_cmis_mgr’ is set/found 'true' by xcvrd, it would skip launching CMIS task manager and this feature would remain disabled.
+If a platform/vendor does not specify/set ‘skip_xcvrd_cmis_mgr’, xcvrd would exercise the default workflow (i.e. when xcvrd detects QSFP-DD, it would luanch CMIS task manager and initialize the module per CMIS specification).
+
+Note: This feature flag (skip_xcvrd_cmis_mgr) was added as a flexibility in case vendor/platform intend to disable this feature and not use CMIS task manager. However, techinically, as mentioned in this document, that should not be the case.
Workflow :
- 
+ 
+
# Transceiver Initialization
diff --git a/doc/sonic-build-system/web_file_server_population_script.md b/doc/sonic-build-system/web_file_server_population_script.md
new file mode 100644
index 0000000000..31c3806868
--- /dev/null
+++ b/doc/sonic-build-system/web_file_server_population_script.md
@@ -0,0 +1,129 @@
+# Web file server population script
+
+
+## _Revision History_
+
+Date | Description
+----------- | -------------
+3/9/22 | First draft
+
+
+## _Table of Contents_
+
+1. [Overview](#Overview)
+ 1. [In scope](#subparagraph1)
+2. [Download web packages from external file storage](#paragraph1)
+ 1. [Work Items](#subparagraph1)
+3. [Upload web packages to trusted file storage](#paragraph2)
+ 1. [Work Items](#subparagraph1)
+4. [User Input](#paragraph3)
+5. [Script outputs](#paragraph4)
+6. [Build flow integration](#paragraph5)
+7. [Source tree location](#paragraph6)
+8. [Q&A](#paragraph7)
+
+
+
+## Overview
+The SONiC reproducible build framework provides a way to lock the versions of external packages. It will lock the versions for all the docker images including the slave docker images, host images, and docker slave containers which are used to build the SONiC targets, such as deb files, python wheels, etc.
+SONiC reproducible build allows to identify specific version of a web package retrieved by wget/curl and download it from an external file storage.
+The web site may be not stable, and the web packages may be replaced to another one unexpectedly, so we do not want to depend on the web packages at a remote web site.
+Before a web package is used in the SONiC repository, the package should be uploaded to a trusted file storage. Currently SONiC offers to do that manually or automatically via Jenkins pipeline.
+The file Server population script is a complementary utility for “SONiC reproducible build” and suppose to ease the process of downloading the web packages from an external file storage and uploading them to trusted file storage.
+The script will be triggered manually or part of the build flow (part of “make freeze”) whenever the developer intend to populate the web server which packages associated to specific tag.
+
+### In scope:
+- Download web packages from external file storage
+- Rename and upload web packages to trusted file storage
+- Script output
+
+## Download web packages from external file storage
+The script will identify all relevant web packages, collect their hash value and url and then download them from the remote site to a local cache.
+The script search over the source tree path recursively and look for ‘version-web’ files.
+Those file are being parsed in order to retrieve the path to the package and its hash value. The script is expecting for such format:
+[package url]==[hash value]
+
+```sh
+For example:
+https://storage.googleapis.com/golang/go1.14.2.linux-amd64.tar.gz==ebef065e4d35572af5b03e2be957a9c6c5063b38
+```
+
+### Work Items:
+- Identify relevant versions-web files
+- Parse versions-web files and identify relevant web packages
+- Download web packages from their external file storage
+
+
+## Upload web packages to trusted file storage
+
+Although there is no version for a package downloaded by wget/curl, the hash value of the package file can be used as the version value.
+The script would rename the Web package and then upload it from cache to a trusted file server. The file name format in the file storage is as below:
+[original file name]-[hash value]
+```sh
+For an example:
+go1.14.2.linux-amd64.tar.gz-ebef065e4d35572af5b03e2be957a9c6c5063b38
+```
+For the same web package, it supports to have multiple versions of the file in the file storage
+for example:
+```sh
+For an example:
+go1.14.2.linux-amd64.tar.gz-ebef065e4d35572af5b03e2be957a9c6c5063b38
+go1.14.2.linux-amd64.tar.gz-b73d6366c0bce441d20138c150e2ddcdf406e654
+```
+Same file name is used with different hash
+
+### Work Items:
+- Rename web package
+- Upload web package to trusted file storage
+- Generate output version file (optional)
+
+## User Input:
+The user would provide the script the following parameters:
+
+| Name | Description | Tag | Type | Default value |
+|------------- |------------------------------------------------------------------------------------------------------------ |-------------- |--------- |------------------------------------ |
+| source | Search path for source tree where to look for compilation artifacts (version files) | -s, --source | String | CWD (current working directory) |
+| cache | Path for temporary storage where to download Web Packages | -c, --cache | String | CWD/tmp |
+| output | Output file name to hold the list of packages. The file will be created in the format of version files | -o, --output | String | none |
+| print | Print level verbosity | -p, --print | integer | 1 (info level) |
+| user | User for trusted server authentication | -u, --user | String | none |
+| key | key server authentication | -k, --key | String | none |
+| Destination | URL for destination web file server | -d, --dest | String | none |
+
+
+## Script outputs:
+
+The script would print out to STDOUT its current download progress (progress bar) and error messages in case of errors.
+The script can produce a list of packages that were uploaded in a format of version file.
+The script would return 0 in case of success and an errors code (see table below) in case of failure.
+### Error codes:
+
+Code | Description | Comments
+------------ | -------------------------------------| -------------
+ 0 | Success |
+-1 | Cannot write to cache |
+-2 | Cannot access trusted server |
+-3 | Cannot authenticate trusted server |
+
+
+## Build flow integration:
+Script will be integrated into SONiC build environment as part of “make freeze” where invocation of the script is optional. User may choose to call script by defining following environment variables:
+
+```sh
+export SONIC_REPRODUCABLE_BUILD_UPLAOD_ENABLE=1
+export SONIC_REPRODUCABLE_BUILD_UPLAOD_USER=
+export SONIC_REPRODUCABLE_BUILD_UPLAOD_KEY=
+export SONIC_REPRODUCABLE_BUILD_UPLAOD_DEST=< URL for destination web file server>
+```
+
+Another approach is to call the script manually and provide as an input the pat where the script would search for version files (version files which were generated as part of "make freeze" invocation or by previous invocation of the script).
+
+## Source tree location:
+
+ The script will be located under
+sonic-buildimage/scripts/populate_file_web_server.py
+
+## Q&A
+Who should run this script?
+When should user run this script?
+
diff --git a/doc/sonic-flags/control-sonic-behaviors-with-sonic-flags.md b/doc/sonic-flags/control-sonic-behaviors-with-sonic-flags.md
new file mode 100644
index 0000000000..51015a5fe8
--- /dev/null
+++ b/doc/sonic-flags/control-sonic-behaviors-with-sonic-flags.md
@@ -0,0 +1,150 @@
+# Control SONiC behaviors with SYSTEM_DEFAULTS table
+
+## 1 Table of Content ###
+
+- [Revision](#11-revision)
+- [Scope](#2-scope)
+- [Definitions/Abbreviations](#3-definitionsabbreviations)
+- [Overview](#4-overview)
+- [Design](#5-design)
+- [Change required](#6-change-required)
+- [Test requirement](#7-test-requirement)
+
+
+### 1.1 Revision ###
+| Rev | Date | Author | Change Description |
+|:---:|:-----------:|:------------------:|-----------------------------------|
+| 0.1 | | Bing Wang | Initial version |
+
+
+## 2 Scope ##
+
+This document covers high level design of `SYSTEM_DEFAULTS` table in SONiC.
+
+## 3 Definitions/Abbreviations ##
+
+
+| Term | Meaning |
+|:--------:|:---------------------------------------------:|
+| | |
+
+
+## 4 Overview
+
+A number of flags are required to turn on/off certain feature or control the behaviors of various features in SONiC. Currently, these flags are put into `DEVICE_METADATA` table.
+
+```
+ "DEVICE_METADATA": {
+ "localhost": {
+ "default_bgp_status": "down",
+ "default_pfcwd_status": "enable",
+ "synchronous_mode": "enable",
+ "dhcp_server": "enable"
+ }
+ }
+```
+As a result, the `DEVICE_METADATA` table is inflating rapidly as we are having more and more flags, although these flags seem not to be categorized into `DEVICE_METADATA`
+
+To have a better management of the flags, a new table `SYSTEM_DEFAULTS` is introduced in this design.
+
+## 5 Design ##
+
+### 5.1 DB Schema
+
+A new table `SYSTEM_DEFAULTS` is added into config_db.
+```
+ key = SYSTEM_DEFAULTS|feature_name; feature name must bt unique
+
+ ;field = value
+ status = 1*255VCHAR ; The value is a string, which can be 'enable'/'disable', 'down'/'up' or any string.
+ custom_field = 1*255VCHAR ; The name of custom_field can be any custom string.
+```
+Below is a sample of `SYSTEM_DEFAULTS` table
+
+```
+"SYSTEM_DEFAULTS": {
+ "tunnel_qos_remap": {
+ "status": "enabled"
+ }
+ "default_bgp_status": {
+ "status": "down"
+ }
+ "synchronous_mode": {
+ "status": "enable"
+ }
+ "dhcp_server": {
+ "status": "enable"
+ }
+ }
+```
+
+### 5.2 How to update flags in `SYSTEM_DEFAULTS` table
+
+#### 5.2.1 Set default value with `init_cfg.json`
+
+The default value of flags in `SYSTEM_DEFAULTS` table can be set in `init_cfg.json` and loaded into db at system startup. These flags are usually set at image being build, and are unlikely to change at runtime.
+If the values in `config_db.json` is changed by user, it will not be rewritten back by `init_cfg.json` as `config_db.json` is loaded after `init_cfg.json` in [docker_image_ctl.j2](https://github.com/Azure/sonic-buildimage/blob/master/files/build_templates/docker_image_ctl.j2)
+
+```
+if [ -r /etc/sonic/config_db$DEV.json ]; then
+ if [ -r /etc/sonic/init_cfg.json ]; then
+ $SONIC_CFGGEN -j /etc/sonic/init_cfg.json -j /etc/sonic/config_db$DEV.json --write-to-db
+ else
+ $SONIC_CFGGEN -j /etc/sonic/config_db$DEV.json --write-to-db
+ fi
+fi
+```
+For example, the value of `default_bgp_status` is down in `init_cfg.json` if `shutdown_bgp_on_start` is set to `y` when image is being built. If we modify the value of `default_bgp_status` in `config_db.json` to `up`, it will keep `up`.
+#### 5.2.2 Parse from `minigraph.xml` when loading minigraph
+
+For the flags that can be changed by reconfiguration, we can update entries in `minigraph.xml`, and parse the new values in to config_db with minigraph parser at reloading minigraph.
+For example, to turn on/off the `tunnel_qos_remap` feature, a new section will be defined in `minigraph.xml`
+
+```
+
+
+
+ TunnelQosRemapEnabled
+ True
+
+
+
+```
+The new section will be parsed by `minigraph.py`, and the parsed value will be merged with the values defined in `init_cfg.json`, and finally written into `config_db`. If there are duplicated entries in `init_cfg.json` and `minigraph.xml`, the values in `minigraph.xml` will overwritten the values defined in `init_cfg.json`.
+#### 5.2.3 Update value directly in db memory
+For some behavior change, we may don't have to interrupt dataplane. To support controlling SONiC behavior on-the-fly, we can update the value of flags in memory with tools like `sonic-cfggen`, `configlet` or `config apply-patch`.
+
+### 5.3 How to consume flags in `SYSTEM_DEFAULTS` table
+
+#### 5.3.1 Consume at service startup or reload
+All of the flags in `SYSTEM_DEFAULTS` table can be consumed at service startup or reload as we do now. We can use the flags to render templates or control the running path of code.
+
+#### 5.3.2 Consume on-the-fly without interrupting traffic
+The `SYSTEM_DEFAULTS` table can be subscribed by components that are interested on the flags. Hence, the in-memory change of flags will be consumed by running service, and take effect without reloading if possible.
+
+## 6 Change required ##
+### 6.1 Template update
+1. Templates that generate default values in `DEVICE_METADATA|localhost` table are required to be updated. The generated flags will be put into `SYSTEM_DEFAULTS` table now.
+2. Templates that depend on `DEVICE_METADATA|localhost` table are required to be updated.
+
+### 6.2 Yang model update
+A new Yang model is to be added to restrict the valid flags in `SYSTEM_DEFAULTS` table. The existing entries for flags in current [sonic-device_metadata.yang](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-device_metadata.yang) are to be removed.
+
+### 6.3 Code change
+1. Update `db_migrator.py` to migrate flags from `DEVICE_METADATA|localhost` table to `SYSTEM_DEFAULTS` table. Current flags include
+
+|Flag|Source|
+|--|--|
+|default_bgp_status| From init_cfg.json, the value is determined by option shutdown_bgp_on_start when image being built|
+|default_pfcwd_status|From init_cfg.json, the value is determined by option enable_pfcwd_on_start when image being built|
+|synchronous_mode|From init_cfg.json, the value is determined by option include_p4rt when image being built|
+|buffer_model|From init_cfg.json, the value is determined by option default_buffer_model when image being built|
+|dhcp_server| Parse from minigraph.xml|
+
+
+2. Add code to subscribe `SYSTEM_DEFAULTS` table to get the update notification for components that is interested in the `SYSTEM_DEFAULTS` change. Currently, all orchs and daemons don't support changing flag controlled behaviors without restarting service, so no code change is required for existing components.
+
+## 7 Test requirement
+TBA
+
+
diff --git a/doc/subport/sonic-sub-port-intf-hld.md b/doc/subport/sonic-sub-port-intf-hld.md
index 1e0ad3f4bd..cf5db32ec9 100644
--- a/doc/subport/sonic-sub-port-intf-hld.md
+++ b/doc/subport/sonic-sub-port-intf-hld.md
@@ -172,7 +172,7 @@ Example configuration:
"Ethernet0.100|192.0.0.1/21": {},
"Ethernet0.100|fc0a::/112": {}
"Eth64.10": {
- “vlan” : 100,
+ "vlan" : 100,
"admin_status" : "up"
},
"Eth64.10|192.168.0.1/21": {},
diff --git a/doc/syslog/images/ssip_add_flow.svg b/doc/syslog/images/ssip_add_flow.svg
new file mode 100755
index 0000000000..bac8e8a70a
--- /dev/null
+++ b/doc/syslog/images/ssip_add_flow.svg
@@ -0,0 +1,232 @@
+
+
+
+
diff --git a/doc/syslog/images/ssip_remove_flow.svg b/doc/syslog/images/ssip_remove_flow.svg
new file mode 100755
index 0000000000..c2d3b42be6
--- /dev/null
+++ b/doc/syslog/images/ssip_remove_flow.svg
@@ -0,0 +1,231 @@
+
+
+
+
diff --git a/doc/syslog/syslog-design.md b/doc/syslog/syslog-design.md
new file mode 100644
index 0000000000..702c3a8bd7
--- /dev/null
+++ b/doc/syslog/syslog-design.md
@@ -0,0 +1,558 @@
+# SONiC Syslog Source IP
+
+## High Level Design document
+
+## Table of contents
+
+- [Revision](#revision)
+- [About this manual](#about-this-manual)
+- [Scope](#scope)
+- [Abbreviations](#abbreviations)
+- [1 Introduction](#1-introduction)
+ - [1.1 Feature overview](#11-feature-overview)
+ - [1.2 Requirements](#12-requirements)
+ - [1.2.1 Functionality](#121-functionality)
+ - [1.2.2 Command interface](#122-command-interface)
+ - [1.2.3 Error handling](#123-error-handling)
+ - [1.2.4 Event logging](#124-event-logging)
+- [2 Design](#2-design)
+ - [2.1 Overview](#21-overview)
+ - [2.2 Syslog Forwarding Output Module](#22-syslog-forwarding-output-module)
+ - [2.2.1 Overview](#221-overview)
+ - [2.2.2 IpFreeBind](#222-ipfreebind)
+ - [2.3 Configuration agent](#23-configuration-agent)
+ - [2.3.1 Overview](#231-overview)
+ - [2.3.2 SSIP parameters](#232-ssip-parameters)
+ - [2.3.3 SSIP configuration](#233-ssip-configuration)
+ - [2.3.3.1 VRF/Source: `unset/unset`](#2331-vrfsource-unsetunset)
+ - [2.3.3.2 VRF/Source: `unset/set`](#2332-vrfsource-unsetset)
+ - [2.3.3.3 VRF/Source: `set/unset`](#2333-vrfsource-setunset)
+ - [2.3.3.4 VRF/Source: `set/set`](#2334-vrfsource-setset)
+ - [2.4 DB schema](#24-db-schema)
+ - [2.4.1 Config DB](#241-config-db)
+ - [2.4.1.1 SSIP table](#2411-ssip-table)
+ - [2.4.2 Data sample](#242-data-sample)
+ - [2.4.3 Configuration sample](#243-configuration-sample)
+ - [2.4.4 Configuration migration](#244-configuration-migration)
+ - [2.5 Flows](#25-flows)
+ - [2.5.1 SSIP add](#251-ssip-add)
+ - [2.5.2 SSIP remove](#252-ssip-remove)
+ - [2.6 CLI](#26-cli)
+ - [2.6.1 Command structure](#261-command-structure)
+ - [2.6.2 Usage examples](#262-usage-examples)
+ - [2.6.2.1 Config command group](#2621-config-command-group)
+ - [2.6.2.2 Show command group](#2622-show-command-group)
+ - [2.7 YANG model](#27-yang-model)
+ - [2.8 Warm/Fast boot](#28-warmfast-boot)
+- [3 Test plan](#3-test-plan)
+ - [3.1 Unit tests via VS](#31-unit-tests-via-vs)
+ - [3.2 Data plane tests via PTF](#32-data-plane-tests-via-ptf)
+
+## Revision
+
+| Rev | Date | Author | Description |
+|:---:|:----------:|:--------------:|:----------------|
+| 0.1 | 18/04/2022 | Nazarii Hnydyn | Initial version |
+
+## About this manual
+
+This document provides general information about Syslog Source IP implementation in SONiC
+
+## Scope
+
+This document describes the high level design of Syslog Source IP feature in SONiC
+
+**In scope:**
+1. Syslog Source IP configuration for UDP protocol
+
+**Out of scope:**
+1. Syslog Source IP configuration for TCP protocol
+
+## Abbreviations
+
+| Term | Meaning |
+|:------|:------------------------------------------|
+| SONiC | Software for Open Networking in the Cloud |
+| SSIP | Syslog Source IP |
+| IP | Internet Protocol |
+| UDP | User Datagram Protocol |
+| TCP | Transmission Control Protocol |
+| DB | Database |
+| VRF | Virtual Routing and Forwarding |
+| OMFWD | Syslog Forwarding Output Module |
+| CLI | Сommand-line Interface |
+| YANG | Yet Another Next Generation |
+| VS | Virtual Switch |
+| PTF | Packet Test Framework |
+
+## List of figures
+
+[Figure 1: SSIP add flow](#figure-1-ssip-add-flow)
+[Figure 2: SSIP remove flow](#figure-2-ssip-remove-flow)
+
+## List of tables
+
+[Table 1: Event logging](#table-1-event-logging)
+[Table 2: Syslog Forwarding Output Module](#table-2-syslog-forwarding-output-module)
+[Table 3: SSIP parameters](#table-3-ssip-parameters)
+
+# 1 Introduction
+
+## 1.1 Feature overview
+
+SSIP is a feature which allows user to change UDP packet source IP address.
+Any configured address can be used for source IP mangling.
+This might be useful for security reasons.
+
+SSIP also extends the existing syslog implementation with VRF device and server UDP port configuration support.
+The feature doesn't change the existing DB schema which makes it fully backward compatible.
+
+## 1.2 Requirements
+
+### 1.2.1 Functionality
+
+**This feature will support the following functionality:**
+1. Syslog Source IP address configuration
+2. Syslog server UDP port configuration
+3. Syslog VRF device support
+
+### 1.2.2 Command interface
+
+**This feature will support the following commands:**
+1. config: add/delete syslog server configuration
+2. show: display syslog server configuration
+
+### 1.2.3 Error handling
+
+**This feature will provide error handling for the next situations:**
+1. Invalid object reference
+2. Invalid options/parameters
+
+### 1.2.4 Event logging
+
+**This feature will provide event logging for the next situations:**
+1. Syslog server add/delete
+
+###### Table 1: Event logging
+
+| Event | Severity |
+|:----------------------------------|:---------|
+| Syslog server add/delete: success | NOTICE |
+| Syslog server add/delete: error | ERROR |
+
+# 2 Design
+
+## 2.1 Overview
+
+SSIP will reuse syslog `omfwd` functionality which offers the next features:
+1. Source IP address configuration
+2. Server UDP port configuration
+3. VRF device configuration
+
+## 2.2 Syslog Forwarding Output Module
+
+### 2.2.1 Overview
+
+###### Table 2: Syslog Forwarding Output Module
+
+| Parameter | Default | Description |
+|:-----------|:---------|:------------------------------------------------------------------------------------------|
+| target | none | Name or IP-Address of the system that shall receive messages. Any resolvable name is fine |
+| address | none | Bind socket to a given local IP address. This option is only supported for UDP, not TCP |
+| port | 514 | Name or numerical value of port to use when connecting to target |
+| protocol | udp | Type of protocol to use for forwarding (e.g., tcp/udp) |
+| device | none | Bind socket to given device (e.g., eth0/vrf0) |
+| ipfreebind | 2 | Manages the IP_FREEBIND option on the UDP socket |
+
+**MAN page:** [omfwd](
+ https://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html#omfwd-syslog-forwarding-output-module
+ "Syslog Forwarding Output Module"
+)
+
+### 2.2.2 IpFreeBind
+
+**IP_FREEBIND (since Linux 2.4)**
+```
+If enabled, this boolean option allows binding to an IP address that is nonlocal or does not (yet) exist.
+This permits listening on a socket, without requiring the underlying network interface
+or the specified dynamic IP address to be up at the time that the application is trying to bind to it.
+This option is the per-socket equivalent of the ip_nonlocal_bind.
+```
+
+**MAN page:** [ip_freebind](
+ https://linux.die.net/man/7/ip
+ "IP_FREEBIND"
+)
+
+## 2.3 Configuration agent
+
+### 2.3.1 Overview
+
+Configuration management of `rsyslogd` is done by `rsyslog-config` service.
+The service is triggered by CLI once data is validated and pushed to Config DB.
+
+The `rsyslog-config` service performs the next actions:
+1. Renders `rsyslog.conf.j2` template with `sonic-cfggen` to generate a new `rsyslogd` config file
+2. Restarts `rsyslog` service which triggers `rsyslogd` to load a new config file
+
+### 2.3.2 SSIP parameters
+
+**SSIP will have the next parameter mapping:**
+
+###### Table 3: SSIP parameters
+
+| SONiC | Rsyslogd | Config DB Schema |
+|:-------|:---------|:---------------------------|
+| key | target | SYSLOG_SERVER\|key |
+| source | address | SYSLOG_SERVER\|key\|source |
+| port | port | SYSLOG_SERVER\|key\|port |
+| vrf | device | SYSLOG_SERVER\|key\|vrf |
+
+### 2.3.3 SSIP configuration
+
+SSIP offers `vrf` and `source` parameters for flexible configuration management.
+Each parameter combination requires a dedicated handling approach.
+
+**Note:**
+1. The destination might be not reachable over the specified `vrf`/`source`: no way to check - user's responsibility
+2. Additional validation is required when MGMT/DATA VRF is removed while reference still exists in syslog configuration
+
+#### 2.3.3.1 VRF/Source: `unset/unset`
+
+Linux kernel decides which source IP to use within the default VRF.
+
+**Example:**
+```
+*.* action(type="omfwd" target="2.2.2.2" protocol="udp")
+```
+
+#### 2.3.3.2 VRF/Source: `unset/set`
+
+Check if source IP is configured on any default VRF member:
+yes - set source IP, no - generate error
+
+**Example:**
+```
+*.* action(type="omfwd" target="2.2.2.2" protocol="udp" address="1.1.1.1")
+```
+
+#### 2.3.3.3 VRF/Source: `set/unset`
+
+Check VRF type:
+1. Default
+2. MGMT
+3. DATA
+
+Default VRF:
+1. Skip VRF configuration
+
+MGMT VRF:
+1. Check if MGMT VRF is enabled:
+yes - set VRF, no - generate error
+
+DATA VRF:
+1. Check if VRF is a member of SONiC VRF table:
+yes - set VRF, no - generate error
+
+**Example:**
+```
+Default VRF:
+*.* action(type="omfwd" target="2.2.2.2" protocol="udp")
+MGMT VRF:
+*.* action(type="omfwd" target="2.2.2.2" protocol="udp" device="mgmt")
+DATA VRF:
+*.* action(type="omfwd" target="2.2.2.2" protocol="udp" device="Vrf-Data")
+```
+
+#### 2.3.3.4 VRF/Source: `set/set`
+
+Check VRF type:
+1. Default
+2. MGMT
+3. DATA
+
+Default VRF:
+1. Check if source IP is configured on any DEFAULT VRF member:
+yes - set source IP, no - generate error
+2. Skip VRF configuration
+
+MGMT VRF:
+1. Check if MGMT VRF is enabled:
+yes - set VRF, no - generate error
+2. Check if source IP is configured on any MGMT VRF member:
+yes - set source IP, no - generate error
+
+DATA VRF:
+1. Check if VRF is a member of SONiC VRF table:
+yes - set VRF, no - generate error
+2. Check if source IP is configured on any DATA VRF member:
+yes - set source IP, no - generate error
+
+**Example:**
+```
+Default VRF:
+*.* action(type="omfwd" target="2.2.2.2" protocol="udp" address="1.1.1.1")
+MGMT VRF:
+*.* action(type="omfwd" target="2.2.2.2" protocol="udp" address="1.1.1.1" device="mgmt")
+DATA VRF:
+*.* action(type="omfwd" target="2.2.2.2" protocol="udp" address="1.1.1.1" device="Vrf-Data")
+```
+
+## 2.4 DB schema
+
+### 2.4.1 Config DB
+
+#### 2.4.1.1 SSIP table
+```abnf
+; defines schema for syslog table configuration attributes
+key = SYSLOG_SERVER|server_ip_address ; server IP address. Must be unique
+
+; field = value
+source = ip-addr ; source IP address
+port = 1*5DIGIT ; server UDP port (0..65535)
+vrf = vrf-device ; VRF device
+
+; value annotations
+h16 = 1*4HEXDIG
+ls32 = h16 ":" h16
+dec-octet = DIGIT ; 0-9
+ / %x31-39 DIGIT ; 10-99
+ / "1" 2DIGIT ; 100-199
+ / "2" %x30-34 DIGIT ; 200-249
+ / "25" %x30-35 ; 250-255
+ipv4-addr = dec-octet "." dec-octet "." dec-octet "." dec-octet
+ipv6-addr = 6( h16 ":" ) ls32
+ / "::" 5( h16 ":" ) ls32
+ / [ h16 ] "::" 4( h16 ":" ) ls32
+ / [ *1( h16 ":" ) h16 ] "::" 3( h16 ":" ) ls32
+ / [ *2( h16 ":" ) h16 ] "::" 2( h16 ":" ) ls32
+ / [ *3( h16 ":" ) h16 ] "::" h16 ":" ls32
+ / [ *4( h16 ":" ) h16 ] "::" ls32
+ / [ *5( h16 ":" ) h16 ] "::" h16
+ / [ *6( h16 ":" ) h16 ] "::"
+ip-addr = ipv4-addr / ipv6-addr
+vrf-default = "default"
+vrf-mgmt = "mgmt"
+vrf-data = "Vrf-" 1*255VCHAR
+vrf-device = vrf-default / vrf-mgmt / vrf-data
+```
+
+### 2.4.2 Data sample
+
+**Config DB:**
+```bash
+redis-cli -n 4 HGETALL 'SYSLOG_SERVER|2.2.2.2'
+1) "source"
+2) "1.1.1.1"
+3) "port"
+4) "514"
+5) "vrf"
+6) "default"
+
+redis-cli -n 4 HGETALL 'SYSLOG_SERVER|3.3.3.3'
+1) "source"
+2) "1.1.1.1"
+3) "port"
+4) "514"
+5) "vrf"
+6) "mgmt"
+
+redis-cli -n 4 HGETALL 'SYSLOG_SERVER|2222::2222'
+1) "source"
+2) "1111::1111"
+3) "port"
+4) "514"
+5) "vrf"
+6) "Vrf-Data"
+```
+
+### 2.4.3 Configuration sample
+
+**Syslog remote logging:**
+```json
+{
+ "SYSLOG_SERVER": {
+ "2.2.2.2": {
+ "source": "1.1.1.1",
+ "port": "514",
+ "vrf": "default"
+ },
+ "3.3.3.3": {
+ "source": "1.1.1.1",
+ "port": "514",
+ "vrf": "mgmt"
+ },
+ "2222::2222": {
+ "source": "1111::1111",
+ "port": "514",
+ "vrf": "Vrf-Data"
+ }
+ }
+}
+```
+
+### 2.4.4 Configuration migration
+
+No special handling is required
+
+## 2.5 Flows
+
+### 2.5.1 SSIP add
+
+
+
+###### Figure 1: SSIP add flow
+
+### 2.5.2 SSIP remove
+
+
+
+###### Figure 2: SSIP remove flow
+
+## 2.6 CLI
+
+### 2.6.1 Command structure
+
+**User interface**:
+```
+config
+|--- syslog
+ |--- add OPTIONS
+ |--- del
+
+show
+|--- syslog
+```
+
+**Options:**
+
+_config syslog add_
+1. `-s|--source` - source ip address
+2. `-p|--port` - server udp port
+3. `-r|--vrf` - vrf device
+
+### 2.6.2 Usage examples
+
+#### 2.6.2.1 Config command group
+
+**The following command adds/deletes syslog server:**
+```bash
+config syslog add '2.2.2.2' \
+--source '1.1.1.1' \
+--port '514' \
+--vrf 'default'
+config syslog del '2.2.2.2'
+```
+
+#### 2.6.2.2 Show command group
+
+**The following command shows syslog server configuration:**
+```bash
+root@sonic:/home/admin# show syslog
+SERVER IP SOURCE IP PORT VRF
+----------- ----------- ------ --------
+2.2.2.2 1.1.1.1 514 default
+3.3.3.3 1.1.1.1 514 mgmt
+2222::2222 1111::1111 514 Vrf-Data
+```
+
+## 2.7 YANG model
+
+An existing YANG model `sonic-syslog.yang` will be extended in order to provide support for SSIP.
+
+**Skeleton code:**
+```yang
+module sonic-syslog {
+
+ yang-version 1.1;
+
+ namespace "http://github.com/Azure/sonic-system-syslog";
+ prefix syslog;
+
+ import ietf-inet-types {
+ prefix inet;
+ }
+
+ import sonic-vrf {
+ prefix vrf;
+ }
+
+ description "Syslog YANG Module for SONiC OS: remote syslog logging";
+
+ revision 2022-04-18 {
+ description "Initial revision.";
+ }
+
+ typedef vrf-device {
+ description "Represents syslog VRF device";
+ type enumeration {
+ enum default;
+ enum mgmt;
+ }
+ }
+
+ container sonic-syslog {
+
+ container SYSLOG_SERVER {
+
+ description "SYSLOG_SERVER part of config_db.json";
+
+ list SYSLOG_SERVER_LIST {
+
+ key "server_address";
+
+ leaf server_address {
+ description "Syslog server IP address";
+ type inet:ip-address;
+ }
+
+ leaf source {
+ description "Syslog source IP address";
+ type inet:ip-address;
+ }
+
+ leaf port {
+ description "Syslog server UDP port";
+ type inet:port-number;
+ }
+
+ leaf vrf {
+ description "Syslog VRF device";
+ type union {
+ type leafref {
+ path "/vrf:sonic-vrf/vrf:VRF/vrf:VRF_LIST/vrf:name";
+ }
+ type vrf-device;
+ }
+ }
+
+ }
+ /* end of list SYSLOG_SERVER_LIST */
+ }
+ /* end of container SYSLOG_SERVER */
+ }
+ /* end of container sonic-syslog */
+}
+/* end of module sonic-syslog */
+```
+
+## 2.8 Warm/Fast boot
+
+No special handling is required
+
+# 3 Test plan
+
+## 3.1 Unit tests via VS
+
+SSIP basic configuration test:
+1. Verify rsyslog.conf after syslog server creation/removal
+
+SSIP extended configuration test:
+1. Create syslog server with IPv4/IPv6 source IP
+2. Verify rsyslog.conf
+3. Create syslog server with default/mgmt/data VRF device
+4. Verify rsyslog.conf
+
+## 3.2 Data plane tests via PTF
+
+TBD
diff --git a/doc/system_health_monitoring/system-ready-HLD.md b/doc/system_health_monitoring/system-ready-HLD.md
new file mode 100644
index 0000000000..c55698c1ec
--- /dev/null
+++ b/doc/system_health_monitoring/system-ready-HLD.md
@@ -0,0 +1,439 @@
+
+
+# System Ready HLD
+
+#### Rev 0.3
+
+# Table of Contents
+
+- [List of Tables](#list-of-tables)
+- [Revision](#revision)
+- [Definition/Abbreviation](#definitionabbreviation)
+- [About This Manual](#about-this-manual)
+- [1 Introduction and Scope](#1-introduction-and-scope)
+ - [1.1 Limitation of Existing tools](#11-existingtools-limitation)
+ - [1.2 Benefits of this feature](#12-benefits-of-this-feature)
+- [2 Feature Requirements](#2-feature-requirements)
+ - [2.1 Functional Requirements](#21-functional-requirements)
+ - [2.2 Configuration and Management Requirements](#22-configuration-and-management-requirements)
+ - [2.3 Scalability Requirements](#23-scalability-requirements)
+ - [2.4 Warm Boot Requirements](#24-warm-boot-requirements)
+- [3 Feature Description](#3-feature-description)
+- [4 Feature Design](#4-feature-design)
+ - [4.1 Overview](#41-design-overview)
+ - [4.2 Sysmonitor](#42-db-changes)
+ - [4.2.1 Subtasks in Sysmonitor](#421-subtasks-in-sysmonitor)
+ - [4.3 Service Identification](#43-service-identification)
+ - [4.4 System ready Framework logic](#44-systemready-fremework-logic)
+ - [4.5 Provision for apps to mark closest UP status](#45-provision-for-apps-to-mark-UP)
+ - [4.5.1 CONFIG_DB Changes](#451-config-db-changes)
+ - [4.5.2 STATE_DB Changes](#452-state-db-changes)
+ - [4.5.3 Feature yang Changes](#453-feature-yang-changes)
+ - [4.6 Syslogs](#46-syslogs)
+- [5 CLI](#5-cli)
+ - [5.1 Output Format](#51-cli-output-format)
+ - [5.2 show system-health sysready-status](#53-system-health sysready-status)
+ - [5.3 show system-health sysready-status brief](#54-system-health sysready-status-brief)
+ - [5.4 show system-health sysready-status detail](#55-system-health sysready-status-detail)
+- [6 Serviceability and Debug](#6-serviceability-and-debug)
+- [7 Warm reboot Support](#7-warm-reboot-support)
+- [8 Unit Test Cases ](#8-unit-test-cases)
+- [9 References ](#9-references)
+
+# List of Tables
+
+[Table 1: Abbreviations](#table-1-abbreviations)
+
+# Revision
+| Rev | Date | Author | Change Description |
+|:--:|:--------:|:-----------------:|:------------------------------------------------------------:|
+| 0.1 | | Senthil Kumar Guruswamy | Initial version |
+| 0.2 | | Senthil Kumar Guruswamy | Update as per review comments |
+| 0.3 | | Senthil Kumar Guruswamy | Integrate systemready to system-health |
+
+
+# Definition/Abbreviation
+
+### Table 1: Abbreviations
+
+| **Term** | **Meaning** |
+| -------- | ----------------------------------------- |
+| FEATURE | Docker/Service |
+| App | Docker/Service |
+
+
+# About this Manual
+
+This document provides general information about the System Ready feature implementation in SONiC.
+
+
+# 1 Introduction and Scope
+
+This document describes the Functionality and High level design of the System Ready feature.
+
+At present, there is no mechanism to know that the system is up and ready with all the essential sonic services and also, all the docker apps are ready along with port ready status to start the network traffic.
+With the asynchronous architecture of SONiC, we will not be able to verify if the config has been applied all the way down to the hardware.
+However, if we could get the closest up status of each docker app considering their config receive ready and port ready, the system readiness could be arrived.
+
+A new python based System monitor tool is introduced to monitor all the essential system host services including docker wrapper services on an event based model and declare the system is ready.
+This framework gives provision for docker apps to notify its closest up status.
+CLIs are provided to fetch the current system status and also service running status and its app ready status along with failure reason if any.
+This feature will be part of system-health framework.
+
+## 1.1 Limitation of Existing tools:
+ - Monit tool is a poll based approach which monitors the configured services for every 1 minute.
+ - Container_checker in monit checks only for running status of expected containers.
+ - Monits custom script execution can only run a logic to take some action but it is yet again a poll based approach.
+
+
+## 1.2 Benefits of this feature:
+ - Event based model where the feedback is immediate
+ - Know the overall system status through syslog and as well through CLIs
+ - It brings in the concept of application readiness to allow each application/service/docker to declare themselves as ready based on different application specific criteria.
+ - Combatibility with application extension framework.
+ SONiC package installation process will register new feature in CONFIG DB.
+ Third party dockers(signature verified) gets integrated into sonic os and runs similar to the existing dockers accessing db etc.
+ Now, once the feature is enabled, it becomes part of either sonic.target or multi-user.target and when it starts, it automatically comes under the system monitor framework watchlist.
+ Hence any third party docker needs to follow the framework logic by including "check_up_status" field in their manifest json file while registering itself in CONFIG_DB and also make use of the provision given to docker apps to mark its closest up status in STATE_DB.
+
+
+# 2 Feature Requirements
+
+## 2.1 Functional Requirements
+
+Following requirements are addressed by the design presented in this document:
+
+1. Identify the list of sonic services to be monitored.
+2. system-health to include the sysmon framework to check system status of all the service units and receive service state change notifications to declare the system ready status.
+3. Provision for apps to notify its closest up status in STATE DB. This should internally cover Port ready status. Also support application extension framework.
+4. Appropriate system ready syslogs to be raised.
+5. New CLI to be introduced to know the current system status all services.
+ - "show system-health sysready-status" covers the overall system status.
+6. During the techsupport data collection, the new CLI to be included for debugging.
+
+
+## 2.2 Configuration and Management Requirements
+
+This feature will support CLI and no configuration command is provided for any congiruations.
+
+
+## 2.3 Scalability Requirements
+
+NA
+
+## 2.4 Warm Boot Requirements
+
+warmboot-finalizer sonic service to be monitored as part of all services.
+
+
+# 3 Feature Description
+
+This feature provides framework to determine the current system status to declare the system is (almost) ready for network traffic.
+
+System ready is arrived at considering the following factors.
+1. All sonic docker services and its UP status(including Portready status)
+2. All sonic host services
+
+
+# 4 Feature Design
+## 4.1 Design Overview
+
+- System-health's newly introduced sysmonitor subtask tracks the sonic host service list, all the docker wrapper services for their running status and also, their app ready status including portready and declare the system is ready.
+- When sysmonitor daemon boots up, it polls for the service list status once and maintains the operational data in STATE_DB and publishes the system ready status in form of syslog and as well as in STATE_DB.
+- Subsequently, when any service state changes, sysmonitor gets the event notification for that service to be checked for its status and update the STATE_DB promptly.
+- Hence the system status is always up-to-date to be notifed to user in the form of syslog, STATE_DB update and as well as could be fetched by appropriate CLIs.
+
+
+## 4.2 Sysmonitor
+
+Sysmonitor is the subtask of system-health service which does the job of checking the service status and updating the system readiness.
+
+### 4.2.1 Subtasks in Sysmonitor
+
+1. subscribe to system dbus
+ - With the dbus subscription, any systemd events gets notified to this task and it puts the event in the multiprocessing queue.
+
+2. subscribe to the new FEATURE table in STATE_DB of Redis database
+ - With the STATE_DB feature table subscription, any input to the FEATURE table gets notified to this task and it puts the event in the queue.
+
+3. Main task
+ - Runs through the polling of all service status check once and listen for events in queue populated by dbus task and statedb task to take appropriate action of checking the specific event unit status
+ and updating system status in the STATE_DB.
+
+
+
+## 4.3 Service Identification
+
+- It covers the enabled services from FEATURE table of CONFIG_DB.
+- Also, since the idea is to cover only the sonic services but not the general system services, sysmonitor tracks services under "multi-user.target" and "sonic.target"
+- This covers all the sonic docker services and most of the sonic host services.
+
+
+## 4.4 System ready Framework logic
+
+The system ready framework design should just not only display the current status of the services in the system
+but align the services within framework to flag the status as "Down" if the service is not running when it was intended to be running.
+
+- For services:
+ - Loaded, enabled/enabled-runtime/static, active & running, active & exited state services are considered 'OK'.
+ - For active and running services, up_status marked by docker app should be True to be considered 'OK'.
+ - Failed state services are considered 'Down'.
+ - Activating state services are considered as 'Starting'.
+ - Deactivating state services are considered as 'Stopping'.
+ - Inactive state services category:
+ - oneshot services are considered as 'OK'.
+ - Special services with condition pathexists check failure are considered as 'OK'.
+ - Other services in inactive state are considered to be 'Down'.
+ - Any service type other than oneshot if by default goes to inactive state, RemainAfterExit=yes entry needs to be added to their service unit file to be inline with the framework.
+
+
+## 4.5 Provision for apps to mark closest UP status
+
+The feature provides framework for services to mark its closest UP status.
+In simple, each app is responsible in marking its closest up status in STATE_DB. Sysmonitor tool just reads from it.
+
+### 4.5.1 CONFIG_DB Changes (init_cfg.json)
+
+Docker apps marking their UP status in STATE_DB will input an entry in FEATURE table of CONFIG_DB with check_up_status flag set to true through /etc/sonic/init_cfg.json file change.
+Sysmonitor checks for the check_up_status flag in CONFIG_DB before reading the app ready status from STATE_DB.
+If the flag does not exist or if set to False, then sysmonitor will not read the app ready status but just checks the running status of the service.
+
+For application extension package support,
+a new manifest variable is introduced to control whether "check_up_status" should be up true or false which will also be an indication whether docker implements marking the up_status flag in STATE_DB.
+
+```
+- Schema in /etc/sonic/init_cfg.json
+ This json file will be fed to FEATURE table of CONFIG_DB during bootup.
+ This json file will be factory default and no config command will be provided for "check_up_status" entry to be updated in CONFIG_DB later.
+ {
+ "FEATURE": {
+ "": {
+ ...
+ "state": "enabled",
+ "check_up_status": "true"
+ }
+ }
+ }
+```
+
+### 4.5.2 STATE_DB Changes
+- Docker apps which rely on config, can mark 'up_status' to true in STATE_DB when they are ready to receive configs from CONFIG_DB and/or some extra dependencies are met.
+- Respective apps should mark its up_status considering Port ready status. Hence there is no separate logic check needed by system monitoring tool
+- Any docker app which has multiple independent daemons can maintain a separate intermediate key-value in the redis-db for each of the daemons and the startup script that invokes each of these daemons can determine the status from the redis entries by each daemon and finally update the STATE_DB up_status.
+- Along with up_status, docker apps should update the fail_reason field with appropriate reason in case of failure or empty string in case of success.
+- Also, update_time field to be fed in as well in the format of epoch time.
+
+
+For instances,
+- swss docker app can wait for port init done and wait for Vrfmgr, Intfmgr and Vxlanmgr to be ready before marking its up status.
+- Other apps like udld,stp etc once after waiting for PortInitDone status and interface db creation, they could mark their UP status.
+- Database app which is the first/base app to be up, may set the UP_STATUS to True once all the required number of redis-server instances are in running state.
+ However, as the supervisor monitors all the redis db instances as part of critical process list, there is no need of a separate app ready status in database docker.
+
+
+STATE_DB:
+```
+- sonic-db-cli STATE_DB HSET "FEATURE|" up_status true
+- sonic-db-cli STATE_DB HSET "FEATURE|" fail_reason "" / ""
+- sonic-db-cli STATE_DB HSET "FEATURE|" update_time ""
+
+- Schema in STATE_DB
+ sonic-db-dump -n STATE_DB output
+ "FEATURE|": {
+ "type": "hash",
+ "value": {
+ "up_status": "true",
+ "fail_reason": "",
+ "update_time": ""
+
+ }
+ },
+
+- Example:
+ "FEATURE|bgp": {
+ "type": "hash",
+ "value": {
+ "fail_reason": "",
+ "update_time": "1634119649.7268105",
+ "up_status": "true"
+ }
+ },
+
+```
+
+
+In addition to this, sysmonitor posts the system status to SYSTEM_READY table in STATE_DB as below.
+
+```
+ "SYSTEM_READY|SYSTEM_STATE": {
+ "type": "hash",
+ "value": {
+ "status": "up"
+ }
+ }
+```
+
+### 4.5.3 Feature yang Changes
+
+Following field is added to the sonic-feature.yang file.
+
+```
+container sonic-feature {
+
+ container FEATURE {
+
+ ...
+
+ leaf check_up_status {
+ description "This configuration controls the system ready tool to check
+ the app ready/up status";
+ type boolean;
+ default false;
+ }
+ }
+}
+```
+
+## 4.6 Syslogs:
+
+- Syslog to be generated for any Sonic systemd services that changes event(active/inactive)
+- A dedicated task in sysmonitor runs to listen for any service state change events.
+ Upon receiving any state change events, the task check for its status and identify the particular event service that caused the change and raise a syslog for that service.
+- syslog is generated for "System is ready" and "System is not ready - One or more services are not up" scenario, only when there is a change between the two states.
+
+```
+Example 1:
+ Jul 02 11:53:51.020133 2021 sonic INFO system#monitor: System is not ready
+ Jul 02 11:53:51.020133 2021 sonic INFO system#monitor: System is not ready - one or more services are not up
+
+Example 2:
+ Jul 02 17:01:00.454978 2021 sonic INFO system#monitor: pmon.service service state changed to [inactive/dead]
+ Jul 02 17:01:28.725101 2021 sonic INFO system#monitor: pmon.service service state changed to [active/running]
+```
+
+## 5 CLI:
+
+Only Click command is supported and not KLISH commands as KLISH will not work if Mgmt-framework service is down.
+- show system-health sysready-status
+
+
+### 5.1 Output Format:
+ 1. Short message indicating system is ready or not
+ 2. Header - Service-Name, Service-Status, App-Ready-Status, Down-Reason
+ 3. List of servies and it status values
+ 4. Output Strings for Service-Status and App-Ready-Status:
+ "OK" - when a service is up
+ "Down" - to emphasise a service is not running when it was intended to be running.
+ "Starting" - Initializing
+ "Stopping" - Deactivating
+ 5. Down-Reason will be extracted from "Result" property of systemctl command and then displayed appropriately.
+ Different reasons can be:
+ start-limit-hit
+ exit-code
+ Inactive
+ custom strings from apps
+
+```
+ <"System is ready"/"System is not ready - one or more services are not up">
+ Service-Name Service-Status App-Ready-Status Down-Reason
+ OK OK -
+ Down Down start-limit-hit
+ OK Down Inactive
+```
+
+
+### 5.2 show system-health sysready-status
+
+```
+ root@sonic:/# show system-health sysready-status
+ System is ready
+
+ Service-Name Service-Status App-Ready-Status Down-Reason
+ as7712-pddf-platform-monitor OK OK -
+ bgp OK OK -
+ caclmgrd OK OK -
+ config-setup OK OK -
+ containerd OK OK -
+
+ root@sonic:/# show system-health sysready-status
+ System is not ready - one or more services are not up
+
+ Service-Name Service-Status App-Ready-Status Down-Reason
+ aaastatsd Down Down start-limit-hit
+ as7712-pddf-platform-monitor OK OK -
+ bgp OK Starting -
+ caclmgrd OK OK -
+ config-setup OK OK -
+ ntp-config Starting Starting -
+ pmon OK Down Transceiver daemon is not up
+```
+
+### 5.3 show system-health sysready-status brief
+- The output of this command will just display the brief status of the entire sonic services.
+- Output format of the CLI :
+
+```
+ <"System is ready"/"System is not ready - one or more services are not up">
+
+ Example 1:
+ root@sonic:/# show system-health sysready-status brief
+ System is not ready - one or more services are not up
+
+ root@sonic:/#
+```
+
+### 5.4 show system-health sysready-status detail
+- The output of this command will display AppStatus-UpdateTime column in addition to the existing to give more info on the time the app readiness was marked.
+- Output format of the CLI :
+
+```
+ <"System is ready"/"System is not ready - one or more services are not up">
+ Service-Name Service-Status App-Ready-Status Down-Reason AppStatus-UpdateTime
+ -------------------------------- ---------------- ------------------ ------------- ----------------------
+ as7326-56x-pddf-platform-monitor OK OK - -
+
+
+ Example 1:
+ root@sonic:/# show system-health sysready-status detail
+ System is not ready - one or more services are not up
+
+ Service-Name Service-Status App-Ready-Status Down-Reason AppStatus-UpdateTime
+ -------------------------------- ---------------- ------------------ ------------- ----------------------
+ as7326-56x-pddf-platform-monitor OK OK - -
+ auditd OK OK - -
+ bgp OK OK - -
+ caclmgrd OK OK - -
+ config-chassisdb OK OK - -
+
+
+ root@sonic:/#
+```
+
+## 6 Serviceability and Debug
+
+- The system logging mechanisms explained in section 4.6 shall be used.
+- The show commands can be used as debug aids.
+- Techsupport:
+ In generate dump tool, show system-health sysready-status detail CLI is included to be saved to the dump in the name of system-health-sysready.status
+
+
+
+## 7 Warm Reboot Support
+
+Sysmonitor monitors the running status of warmboot-finalizer.service
+This can be enhanced to hook to the actual completion of warboot-finalizer service later.
+
+
+## 8 Unit Test Cases
+
+1. Check show system-health sysready-status
+2. Check show system-health sysready-status brief
+3. Check show system-health sysready-status detail
+4. Make any of the docker apps down and check failed apps details are shown
+5. Make any of the host apps down and check failed apps details are shown
+6. Check top command for sysmonitor CPU and memory usage
+7. Check syslogs for any service state change.
+8. Check syslog for overall system status change.
+
+## 9 References
+NA
diff --git a/doc/vxlan/Overlay ECMP with BFD.md b/doc/vxlan/Overlay ECMP with BFD.md
index 4485cd6237..28ab66ba75 100644
--- a/doc/vxlan/Overlay ECMP with BFD.md
+++ b/doc/vxlan/Overlay ECMP with BFD.md
@@ -1,6 +1,6 @@
# Overlay ECMP with BFD monitoring
## High Level Design Document
-### Rev 1.1
+### Rev 1.5
# Table of Contents
@@ -37,6 +37,8 @@
| 1.2 | 10/18/2021 | Prince Sunny/Shi Su | Test Plan added |
| 1.3 | 11/01/2021 | Prince Sunny | IPv6 test cases added |
| 1.4 | 12/03/2021 | Prince Sunny | Added scaling section, extra test cases |
+| 1.5 | 04/11/2022 | Prince Sunny | Test plan for Health monitoring |
+| 1.6 | 04/23/2022 | Storm Liang | Update 2.6 BGP secion & add Test plan for BGP |
# About this Manual
This document provides general information about the Vxlan Overlay ECMP feature implementation in SONiC with BFD support. This is an extension to the existing VNET Vxlan support as defined in the [Vxlan HLD](https://github.com/Azure/SONiC/blob/master/doc/vxlan/Vxlan_hld.md)
@@ -143,6 +145,7 @@ VNET_ROUTE_TUNNEL_TABLE:{{vnet_name}}:{{prefix}}
Proposed:
```
+{% raw %} # ignore this line please
VNET_ROUTE_TUNNEL_TABLE:{{vnet_name}}:{{prefix}}
"endpoint": {{ip_address1},{ip_address2},...}
"endpoint_monitor": {{ip_address1},{ip_address2},...} (OPTIONAL)
@@ -150,6 +153,7 @@ VNET_ROUTE_TUNNEL_TABLE:{{vnet_name}}:{{prefix}}
"vni": {{vni1},{vni2},...} (OPTIONAL)
"weight": {{w1},{w2},...} (OPTIONAL)
“profile”: {{profile_name}} (OPTIONAL)
+{% endraw %} # ignore this line please
```
```
@@ -224,18 +228,101 @@ When an endpoint is deemed unhealthy, router shall perform the following actions
## 2.6 BGP
Advertise VNET routes
-The overlay routes programmed on the device must be advertised to BGP peers. This can be achieved by the “network” command.
+
+VnetOrch shall create an entry in STATE_DB for the active overlay routes eligible to be advertised by BGP to peers. Based on the health of overlay nexthop, the entry shall be added or removed.
+
+```
+STATE_DB|ADVERTISE_NETWORK_TABLE|{{ip_prefix}}
+ "profile": {{profile_name}}
+```
+
+The above entry shall be subscribed for by bgpcfgd and advertised by the “network” command.
For example:
```
router bgp 1
address-family ipv4 unicast
- network 10.0.0.0/8
+ network 10.0.0.0/8 route-map FROM_SDN_SLB_ROUTES_RM
exit-address-family
- ```
+```
+Notes: Currently, only one profile_name is supported
+
+This configuration example says that network 10.0.0.0/8 will be announced to all neighbors. FRR bgpd doesn’t care about IGP routes when announcing its routes.
+The profile would be transformed to route-map and associated with IP prefix.
-This configuration example says that network 10.0.0.0/8 will be announced to all neighbors. FRR bgpd doesn’t care about IGP routes when announcing its routes.
+Application shall create a profile in APP_DB, which would be associated to the IP prefix when advertised by "network" command.
+```
+APPL_DB:BGP_PROFILE_TABLE:{{profile_name}}
+ "community_id": {{community_string}}
+```
+
+The above entry shall be subscribed for by bgpcfgd and created/updated by "route-map" command.
+
+For example:
+```
+route-map FROM_SDN_SLB_ROUTES_RM permit 100
+ set community 1234:1235
+```
+
+Below will go through several use cases by manipulating tables.
+
+### Use case A: To advertise route 10.0.0.0/8 with community id "1234:1235"
+Step 1: add/update one route-map entry in the state db.
+```
+APPL_DB:BGP_PROFILE_TABLE:FROM_SDN_SLB_ROUTES
+ "community_id": "1234:1235"
+```
+
+Example command to add this entry
+```
+sonic-db-cli APPL_DB HSET "BGP_PROFILE_TABLE:FROM_SDN_SLB_ROUTES" "community_id" "1234:1235"
+```
+
+Step 2: add route entry in the state db
+```
+STATE_DB|ADVERTISE_NETWORK_TABLE|10.0.0.0/8
+ "profile": "FROM_SDN_SLB_ROUTES"
+```
+Example command to add this entry
+```
+sonic-db-cli STATE_DB HSET "ADVERTISE_NETWORK_TABLE|10.0.0.0/8" "profile" "FROM_SDN_SLB_ROUTES"
+```
+
+### Use case B: To advertise route 10.0.0.0/8 without community id (without profile)
+Step 1: add route entry in the state db
+```
+STATE_DB|ADVERTISE_NETWORK_TABLE|10.0.0.0/8
+ "": ""
+```
+Example command to add this entry
+```
+sonic-db-cli STATE_DB HSET "ADVERTISE_NETWORK_TABLE|10.0.0.0/8" "" ""
+```
+
+### Use case C: 10.0.0.0/8 with community id "1234:1235", re advertise route 10.0.0.0/8 with new community id "1234:1236"
+Step 1: add/update one route-map entry in the state db.
+```
+APPL_DB:BGP_PROFILE_TABLE:FROM_SDN_SLB_ROUTES
+ "community_id": "1234:1236"
+```
+
+Example command to add this entry
+```
+sonic-db-cli APPL_DB HSET "BGP_PROFILE_TABLE:FROM_SDN_SLB_ROUTES" "community_id" "1234:1236"
+```
+
+### Use case D: To remove route 10.0.0.0/8 with community id "1234:1235"
+Step 1: Delete the route entry in the state db.
+
+~~STATE_DB|ADVERTISE_NETWORK_TABLE|10.0.0.0/8~~
+
+Example command to delete this entry
+```
+sonic-db-cli STATE_DB DEL "ADVERTISE_NETWORK_TABLE|10.0.0.0/8"
+```
+
+Notes: the BGP_PROFILE_TABLE table need to be removed explicitly, there is no ref-count in the bgpcfgd layer.
## 2.7 CLI
@@ -269,6 +356,7 @@ Create VNET and Vxlan tunnel as an below:
"scope": "default"
}
}
+}
```
Similarly for IPv6 tunnels
@@ -287,6 +375,7 @@ Similarly for IPv6 tunnels
"scope": "default"
}
}
+}
```
Note: It can be safely assumed that only one type of tunnel exists - i.e, either IPv4 or IPv6 for this use-case
@@ -321,7 +410,7 @@ With IPv6 tunnels, prefixes can be either IPv4 or IPv6
### Test Cases
-#### Overlay ECMP
+### 2.8.1 Overlay ECMP
It is assumed that the endpoint IPs may not have exact match underlay route but may have an LPM underlay route or a default route. Test must consider both IPv4 and IPv6 traffic for routes configured as example shown above
@@ -351,10 +440,110 @@ It is assumed that the endpoint IPs may not have exact match underlay route but
|Create/Delete overlay nexthop groups upto 512 | CRM | Verify crm resourse for nexthop_group |
|Create/Delete overlay nexthop group members upto 128 | CRM | Verify crm resourse for nexthop_group_member |
-#### BFD and health monitoring
+### 2.8.2 BFD and health monitoring
+
+Health monitoring requires 'endpoint_monitor' and 'advertise_prefix' attributes to be provided.
+
+Reference tables
+
+**Config_DB**
+```
+{
+ "VNET|Vnet_3000": {
+ "vxlan_tunnel": "tunnel_v4",
+ "vni": "3000",
+ "scope": "default",
+ "advertise_prefix": "true",
+ }
+
+ "VNET|Vnet_3001": {
+ "vxlan_tunnel": "tunnel_v6",
+ "vni": "3001",
+ "scope": "default",
+ "advertise_prefix": "true"
+ }
+}
+```
-TBD
+**APP_DB**
+```
+[
+ "VNET_ROUTE_TUNNEL_TABLE:Vnet_3000:100.100.2.1/32": {
+ "endpoint": "1.1.1.2",
+ "endpoint_monitor": "1.1.2.2"
+ "profile": "FROM_SDN_SLB_ROUTES"
+ }
+
+ "BFD_SESSION:default:default:1.1.2.2": {
+ "multihop": "true",
+ "local_addr": "10.1.0.32"
+ }
+
+ "VNET_ROUTE_TUNNEL_TABLE:Vnet_3001:2000::1/128": {
+ "endpoint": "fc02:1000::1",
+ "endpoint_monitor": "fc02:1000::2"
+ "profile": "FROM_SDN_SLB_ROUTES"
+ }
+
+ "BFD_SESSION:default:default:fc02:1000::2": {
+ "multihop": "true",
+ "local_addr": "fc00:1::32"
+ }
+
+ "BGP_PROFILE_TABLE:FROM_SDN_SLB_ROUTES": {
+ "community_id": "1234:1236"
+ }
+]
+```
-#### BGP advertising
+**STATE_DB**
+```
+{
+ "ADVERTISE_NETWORK_TABLE|100.100.2.1/32": {
+ "profile": "FROM_SDN_SLB_ROUTES"
+ }
+
+ "BFD_SESSION_TABLE|default|default|1.1.2.2": {
+ "state":"Up"
+ }
+
+ "ADVERTISE_NETWORK_TABLE|2000::1/128": {
+ "profile": "FROM_SDN_SLB_ROUTES"
+ }
+
+ "BFD_SESSION_TABLE|default|default|fc02:1000::2": {
+ "state":"Down"
+ }
+}
+```
+The below cases are executed first for IPv4 and repeat the same for IPv6.
-TBD
+| Step | Goal | Expected results |
+|-|-|-|
+| Create a tunnel route to a single endpoint a and monitor a'. Set BFD state a' to UP. Send packets to the route prefix dst| Tunnel route create and BFD functions | Packets are received only at endpoint a. BFD session for endpoint a' is up. Verify advertise table is present |
+| Set the tunnel route to another endpoint b and monitor b'. Set BFD state for b' to UP. Send packets to the route prefix dst | Tunnel route set and BFD functions | Packets are received only at endpoint b. BFD session for b' is created and the state is up. BFD session for endpoint a' is removed |
+| Remove the created tunnel route. Send packets to the route prefix dst. | Tunne route remove function and BFD | Packets are not received at any ports. BFD session for endpoint b' is removed. Verify advertise table is removed |
+| Create tunnel route 1 to two endpoints A = {a1, a2}. Set BFD state for a1' and a2' to UP. Send multiple packets (varying tuple) to the route 1's prefix dst. | ECMP route create with BFD | Packets are received at both a1 and a2. All BFD session states are UP |
+| Create tunnel route 2 to endpoint group A. Send multiple packets (varying tuple) to route 2’s prefix dst | ECMP route create with BFD | Packets are received at both a1 and a2 |
+| Set tunnel route 2 to endpoint group B = {b1, b2}. Set BFD state for b1' and b2' to UP. Send multiple packets (varying tuple) to route 2’s prefix dst | ECMP route set with BFD | Packets are received at both b1 and b2. All BFD session states are UP |
+| Set tunnel route 2 to shared endpoints a1 and b1 with monitor a1' and b1'. Send packets to route 2’s prefix dst | NHG modify with BFD | Packets are recieved at a1 or b1. Nexthop group and the corresponding BFD sessions for endpoint group B are removed. ALL BFD sessions states are UP. |
+| Remove tunnel route 2. Send packets to route 2’s prefix dst | ECMP route remove with BFD | Packets are not recieved at any ports with dst IP of a1 or b1. Unused BFD sessions are removed. Verify advertise table is removed |
+| Set BFD state for a1' to UP and a2' to Down. Send multiple packets (varying tuple) to the route 1's prefix dst. | Health state change | Packets are received only at endpoint a1. Verify advertise table is present |
+| Set BFD state for a1' to Down. Send packets to the route 1's prefix dst. | Health state change | Packets are not received at any ports. Verify advertise table is removed |
+| Set BFD state for a2' to UP. Send packets to the route 1's prefix dst. | Health state change | Packets are received only at endpoint a2. Verify advertise table is present |
+| Set BFD state for a1' to UP. Send multiple packets (varying tuple) to the route 1's prefix dst. | Health state change | Packets are received at both a1 and a2. Verify advertise table is present |
+| Set BFD state for a1' to Down and a2' to Down. Send multiple packets (varying tuple) to the route 1's prefix dst. | Health state change | Packets are not received at any ports. Verify advertise table is removed |
+| Remove tunnel route 1. Send multiple packets (varying tuple) to the route 1's prefix dst. | Route remove with BFD down | Packets are not received at any ports. BFD sessions are removed. Verify advertise table is removed |
+|Create/Delete overlay routes to 4k with unique endpoints upto 4k | BFD Scaling | Verify all 4k BFD sessions are succesfully created and sessions alive |
+
+### 2.8.3 BGP advertising
+The below cases are executed first for IPv4 and repeat the same for IPv6.
+| Step | Goal | Expected results |
+|-|-|-|
+| Create a tunnel route and advertise the tunnel route to all neighbor without community id | BGP | ALL BGP neighbors can recieve the advertised BGP routes |
+| Create a tunnel route and advertise the tunnel route to all neighbor with community id | BGP | ALL BGP neighbors can recieve the advertised BGP routes with community id |
+| Update a tunnel route and advertise the tunnel route to all neighbor with new community id | BGP | ALL BGP neighbors can recieve the advertised BGP routes with new community id |
+| Create a tunnel route and advertise the tunnel route to all neighbor with BGP profile, but create the profile later| BGP | ALL BGP neighbors can recieve the advertised BGP routes without community id first, after the profile table created, the community id would be added and all BGP neighbors can recieve this update and associate the community id with the route |
+| Delete a tunnel route | BGP | ALL BGP neighbors can remove the previously advertised BGP routes |
+| Create 4k tunnel routes and advertise all tunnel routes to all neighbor with community id | BGP scale | ALL BGP neighbors can recieve 4k advertised BGP routes with community id and record the time |
+| Updat BGP_PROFILE_TABLE with new community id for 4k tunnel routes and advertise all tunnel routes to all neighbor with new community id | BGP scale | ALL BGP neighbors can recieve 4k advertised BGP routes with new community id and record the time |
diff --git a/generate_sonic_image_links.sh b/generate_sonic_image_links.sh
index aae526e660..106d32bdfa 100644
--- a/generate_sonic_image_links.sh
+++ b/generate_sonic_image_links.sh
@@ -7,6 +7,9 @@ git pull origin sonic_latest_images_links
#set -euo pipefail
+
+DEFID_BRCM_CMT="$(curl -s 'https://sonic-build.azurewebsites.net/ui/sonic/pipelines/138/builds?branchName=master' | jq -r '.value[0].id')"
+
DEFID_BRCM="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/definitions?name=Azure.sonic-buildimage.official.broadcom' | jq -r '.value[0].id')"
DEFID_MLNX="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/definitions?name=Azure.sonic-buildimage.official.mellanox' | jq -r '.value[0].id')"
DEFID_VS="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/definitions?name=Azure.sonic-buildimage.official.vs' | jq -r '.value[0].id')"
@@ -21,7 +24,7 @@ DEFID_NPH="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/definition
echo '{' > sonic_image_links.json
first=1
-for BRANCH in master 202106 202012 201911 201811
+for BRANCH in master 202111 202106 202012 201911 201811
do
if [ -z "${first}" ]; then
echo ',' >> sonic_image_links.json
@@ -67,82 +70,118 @@ do
ARTF_MRV="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds/'"${BUILD_MRV}"'/artifacts?artifactName=sonic-buildimage.marvell-armhf&api-version=5.1' | jq -r '.resource.downloadUrl')"
ARTF_NPH="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds/'"${BUILD_NPH}"'/artifacts?artifactName=sonic-buildimage.nephos&api-version=5.1' | jq -r '.resource.downloadUrl')"
+ COMMIT_BRCM_1="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_BRCM}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[0].sourceVersion')"
+ COMMIT_BRCM_2="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_BRCM}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[1].sourceVersion')"
+ COMMIT_MLNX_1="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_MLNX}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[0].sourceVersion')"
+ COMMIT_MLNX_2="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_MLNX}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[1].sourceVersion')"
+ COMMIT_VS_1="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_VS}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[0].sourceVersion')"
+ COMMIT_VS_2="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_VS}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[1].sourceVersion')"
+ COMMIT_INNO_1="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_INNO}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[0].sourceVersion')"
+ COMMIT_INNO_2="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_INNO}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[1].sourceVersion')"
+ COMMIT_BFT_1="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_BFT}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[0].sourceVersion')"
+ COMMIT_BFT_2="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_BFT}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[1].sourceVersion')"
+ COMMIT_CHE_1="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_CHE}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[0].sourceVersion')"
+ COMMIT_CHE_2="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_CHE}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[1].sourceVersion')"
+ COMMIT_CTC_1="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_CTC}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[0].sourceVersion')"
+ COMMIT_CTC_2="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_CTC}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[1].sourceVersion')"
+ COMMIT_CTC64_1="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_CTC64}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[0].sourceVersion')"
+ COMMIT_CTC64_2="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_CTC64}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[1].sourceVersion')"
+ COMMIT_GRC_1="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_GRC}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[0].sourceVersion')"
+ COMMIT_GRC_2="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_GRC}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[1].sourceVersion')"
+ COMMIT_MRV_1="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_MRV}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[0].sourceVersion')"
+ COMMIT_MRV_2="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_MRV}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[1].sourceVersion')"
+ COMMIT_NPH_1="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_NPH}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[0].sourceVersion')"
+ COMMIT_NPH_2="$(curl -s 'https://dev.azure.com/mssonic/build/_apis/build/builds?definitions='"${DEFID_NPH}"'&branchName=refs/heads/'"${BRANCH}"'&$top=2&resultFilter=succeeded&api-version=6.0' | jq -r '.value[1].sourceVersion')"
+
echo "\"${BRANCH}\": {" >> sonic_image_links.json
echo "\"sonic-broadcom.bin\": {" >> sonic_image_links.json
echo " \"url\": \"$(echo "${ARTF_BRCM}" | sed 's/format=zip/format=file\&subpath=\/target\/sonic-broadcom.bin/')\"," >> sonic_image_links.json
echo " \"build-url\": \"https://dev.azure.com/mssonic/build/_build/results?buildId=${BUILD_BRCM}&view=results\"," >> sonic_image_links.json
+ echo " \"diff\": \"https://github.com/Azure/sonic-buildimage/compare/"${COMMIT_BRCM_2}"..."${COMMIT_BRCM_1}"\"," >> sonic_image_links.json
echo " \"build\": \"${BUILD_BRCM}\"," >> sonic_image_links.json
echo " \"date\": \"${BUILD_BRCM_TS}\"" >> sonic_image_links.json
echo " }," >> sonic_image_links.json
echo "\"sonic-aboot-broadcom.swi\": {" >> sonic_image_links.json
echo " \"url\": \"$(echo "${ARTF_BRCM}" | sed 's/format=zip/format=file\&subpath=\/target\/sonic-aboot-broadcom.swi/')\"," >> sonic_image_links.json
echo " \"build-url\": \"https://dev.azure.com/mssonic/build/_build/results?buildId=${BUILD_BRCM}&view=results\"," >> sonic_image_links.json
+ echo " \"diff\": \"https://github.com/Azure/sonic-buildimage/compare/"${COMMIT_BRCM_2}"..."${COMMIT_BRCM_1}"\"," >> sonic_image_links.json
echo " \"build\": \"${BUILD_BRCM}\"," >> sonic_image_links.json
echo " \"date\": \"${BUILD_BRCM_TS}\"" >> sonic_image_links.json
echo " }," >> sonic_image_links.json
echo "\"sonic-mellanox.bin\": {" >> sonic_image_links.json
echo " \"url\": \"$(echo "${ARTF_MLNX}" | sed 's/format=zip/format=file\&subpath=\/target\/sonic-mellanox.bin/')\"," >> sonic_image_links.json
echo " \"build-url\": \"https://dev.azure.com/mssonic/build/_build/results?buildId=${BUILD_MLNX}&view=results\"," >> sonic_image_links.json
+ echo " \"diff\": \"https://github.com/Azure/sonic-buildimage/compare/"${COMMIT_MLNX_2}"..."${COMMIT_MLNX_1}"\"," >> sonic_image_links.json
echo " \"build\": \"${BUILD_MLNX}\"," >> sonic_image_links.json
echo " \"date\": \"${BUILD_MLNX_TS}\"" >> sonic_image_links.json
echo " }," >> sonic_image_links.json
echo "\"sonic-vs.img.gz\": {" >> sonic_image_links.json
echo " \"url\": \"$(echo "${ARTF_VS}" | sed 's/format=zip/format=file\&subpath=\/target\/sonic-vs.img.gz/')\"," >> sonic_image_links.json
echo " \"build-url\": \"https://dev.azure.com/mssonic/build/_build/results?buildId=${BUILD_VS}&view=results\"," >> sonic_image_links.json
+ echo " \"diff\": \"https://github.com/Azure/sonic-buildimage/compare/"${COMMIT_VS_2}"..."${COMMIT_VS_1}"\"," >> sonic_image_links.json
echo " \"build\": \"${BUILD_VS}\"," >> sonic_image_links.json
echo " \"date\": \"${BUILD_VS_TS}\"" >> sonic_image_links.json
echo " }," >> sonic_image_links.json
echo "\"sonic-innovium.bin\": {" >> sonic_image_links.json
echo " \"url\": \"$(echo "${ARTF_INNO}" | sed 's/format=zip/format=file\&subpath=\/target\/sonic-innovium.bin/')\"," >> sonic_image_links.json
echo " \"build-url\": \"https://dev.azure.com/mssonic/build/_build/results?buildId=${BUILD_INNO}&view=results\"," >> sonic_image_links.json
+ echo " \"diff\": \"https://github.com/Azure/sonic-buildimage/compare/"${COMMIT_INNO_2}"..."${COMMIT_INNO_1}"\"," >> sonic_image_links.json
echo " \"build\": \"${BUILD_INNO}\"," >> sonic_image_links.json
echo " \"date\": \"${BUILD_INNO_TS}\"" >> sonic_image_links.json
echo " }," >> sonic_image_links.json
echo "\"sonic-innovium-dbg.bin\": {" >> sonic_image_links.json
echo " \"url\": \"$(echo "${ARTF_INNO}" | sed 's/format=zip/format=file\&subpath=\/target\/sonic-innovium-dbg.bin/')\"," >> sonic_image_links.json
echo " \"build-url\": \"https://dev.azure.com/mssonic/build/_build/results?buildId=${BUILD_INNO}&view=results\"," >> sonic_image_links.json
+ echo " \"diff\": \"https://github.com/Azure/sonic-buildimage/compare/"${COMMIT_INNO_2}"..."${COMMIT_INNO_1}"\"," >> sonic_image_links.json
echo " \"build\": \"${BUILD_INNO}\"," >> sonic_image_links.json
echo " \"date\": \"${BUILD_INNO_TS}\"" >> sonic_image_links.json
echo " }," >> sonic_image_links.json
echo "\"sonic-barefoot.bin\": {" >> sonic_image_links.json
echo " \"url\": \"$(echo "${ARTF_BFT}" | sed 's/format=zip/format=file\&subpath=\/target\/sonic-barefoot.bin/')\"," >> sonic_image_links.json
echo " \"build-url\": \"https://dev.azure.com/mssonic/build/_build/results?buildId=${BUILD_BFT}&view=results\"," >> sonic_image_links.json
+ echo " \"diff\": \"https://github.com/Azure/sonic-buildimage/compare/"${COMMIT_BFT_2}"..."${COMMIT_BFT_1}"\"," >> sonic_image_links.json
echo " \"build\": \"${BUILD_BFT}\"," >> sonic_image_links.json
echo " \"date\": \"${BUILD_BFT_TS}\"" >> sonic_image_links.json
echo " }," >> sonic_image_links.json
echo "\"sonic-centec.bin\": {" >> sonic_image_links.json
echo " \"url\": \"$(echo "${ARTF_CTC}" | sed 's/format=zip/format=file\&subpath=\/target\/sonic-centec.bin/')\"," >> sonic_image_links.json
echo " \"build-url\": \"https://dev.azure.com/mssonic/build/_build/results?buildId=${BUILD_CTC}&view=results\"," >> sonic_image_links.json
+ echo " \"diff\": \"https://github.com/Azure/sonic-buildimage/compare/"${COMMIT_CTC_2}"..."${COMMIT_CTC_1}"\"," >> sonic_image_links.json
echo " \"build\": \"${BUILD_CTC}\"," >> sonic_image_links.json
echo " \"date\": \"${BUILD_CTC_TS}\"" >> sonic_image_links.json
echo " }," >> sonic_image_links.json
echo "\"sonic-centec-arm64.bin\": {" >> sonic_image_links.json
echo " \"url\": \"$(echo "${ARTF_CTC64}" | sed 's/format=zip/format=file\&subpath=\/target\/sonic-centec-arm64.bin/')\"," >> sonic_image_links.json
echo " \"build-url\": \"https://dev.azure.com/mssonic/build/_build/results?buildId=${BUILD_CTC64}&view=results\"," >> sonic_image_links.json
+ echo " \"diff\": \"https://github.com/Azure/sonic-buildimage/compare/"${COMMIT_CTC64_2}"..."${COMMIT_CTC64_1}"\"," >> sonic_image_links.json
echo " \"build\": \"${BUILD_CTC64}\"," >> sonic_image_links.json
echo " \"date\": \"${BUILD_CTC64_TS}\"" >> sonic_image_links.json
echo " }," >> sonic_image_links.json
echo "\"sonic-generic.bin\": {" >> sonic_image_links.json
echo " \"url\": \"$(echo "${ARTF_GRC}" | sed 's/format=zip/format=file\&subpath=\/target\/sonic-generic.bin/')\"," >> sonic_image_links.json
echo " \"build-url\": \"https://dev.azure.com/mssonic/build/_build/results?buildId=${BUILD_GRC}&view=results\"," >> sonic_image_links.json
+ echo " \"diff\": \"https://github.com/Azure/sonic-buildimage/compare/"${COMMIT_GRC_2}"..."${COMMIT_GRC_1}"\"," >> sonic_image_links.json
echo " \"build\": \"${BUILD_GRC}\"," >> sonic_image_links.json
echo " \"date\": \"${BUILD_GRC_TS}\"" >> sonic_image_links.json
echo " }," >> sonic_image_links.json
echo "\"sonic-generic-dbg.bin\": {" >> sonic_image_links.json
echo " \"url\": \"$(echo "${ARTF_GRC}" | sed 's/format=zip/format=file\&subpath=\/target\/sonic-generic-dbg.bin/')\"," >> sonic_image_links.json
echo " \"build-url\": \"https://dev.azure.com/mssonic/build/_build/results?buildId=${BUILD_GRC}&view=results\"," >> sonic_image_links.json
+ echo " \"diff\": \"https://github.com/Azure/sonic-buildimage/compare/"${COMMIT_GRC_2}"..."${COMMIT_GRC_1}"\"," >> sonic_image_links.json
echo " \"build\": \"${BUILD_GRC}\"," >> sonic_image_links.json
echo " \"date\": \"${BUILD_GRC_TS}\"" >> sonic_image_links.json
echo " }," >> sonic_image_links.json
echo "\"sonic-marvell-armhf.bin\": {" >> sonic_image_links.json
echo " \"url\": \"$(echo "${ARTF_MRV}" | sed 's/format=zip/format=file\&subpath=\/target\/sonic-marvell-armhf.bin/')\"," >> sonic_image_links.json
echo " \"build-url\": \"https://dev.azure.com/mssonic/build/_build/results?buildId=${BUILD_MRV}&view=results\"," >> sonic_image_links.json
+ echo " \"diff\": \"https://github.com/Azure/sonic-buildimage/compare/"${COMMIT_MRV_2}"..."${COMMIT_MRV_1}"\"," >> sonic_image_links.json
echo " \"build\": \"${BUILD_MRV}\"," >> sonic_image_links.json
echo " \"date\": \"${BUILD_MRV_TS}\"" >> sonic_image_links.json
echo " }," >> sonic_image_links.json
echo "\"sonic-nephos.bin\": {" >> sonic_image_links.json
echo " \"url\": \"$(echo "${ARTF_NPH}" | sed 's/format=zip/format=file\&subpath=\/target\/sonic-nephos.bin/')\"," >> sonic_image_links.json
echo " \"build-url\": \"https://dev.azure.com/mssonic/build/_build/results?buildId=${BUILD_NPH}&view=results\"," >> sonic_image_links.json
+ echo " \"diff\": \"https://github.com/Azure/sonic-buildimage/compare/"${COMMIT_NPH_2}"..."${COMMIT_NPH_1}"\"," >> sonic_image_links.json
echo " \"build\": \"${BUILD_NPH}\"," >> sonic_image_links.json
echo " \"date\": \"${BUILD_NPH_TS}\"" >> sonic_image_links.json
echo " }" >> sonic_image_links.json
@@ -153,3 +192,4 @@ echo "\n}" >> sonic_image_links.json
git add sonic_image_links.json
git commit -m "latest links for sonic images in dedicated branch sonic_latest_images_links"
git push -f --set-upstream origin sonic_latest_images_links
+
diff --git a/images/pfc_wd/image.png b/images/pfc_wd/image.png
new file mode 100644
index 0000000000..966e3eb0e1
Binary files /dev/null and b/images/pfc_wd/image.png differ
diff --git a/index.html b/index.html
index e4efca9291..75f120a000 100644
--- a/index.html
+++ b/index.html
@@ -60,7 +60,7 @@
+
- 
+ 
-
-
+