Merge branch 'main' of https://git.stella-ops.org/stella-ops.org/git.stella-ops.org
	
		
			
	
		
	
	
		
	
		
			Some checks failed
		
		
	
	
		
			
				
	
				Build Test Deploy / build-test (push) Has been cancelled
				
			
		
			
				
	
				Build Test Deploy / authority-container (push) Has been cancelled
				
			
		
			
				
	
				Build Test Deploy / docs (push) Has been cancelled
				
			
		
			
				
	
				Build Test Deploy / deploy (push) Has been cancelled
				
			
		
			
				
	
				Docs CI / lint-and-preview (push) Has been cancelled
				
			
		
		
	
	
				
					
				
			
		
			Some checks failed
		
		
	
	Build Test Deploy / build-test (push) Has been cancelled
				
			Build Test Deploy / authority-container (push) Has been cancelled
				
			Build Test Deploy / docs (push) Has been cancelled
				
			Build Test Deploy / deploy (push) Has been cancelled
				
			Docs CI / lint-and-preview (push) Has been cancelled
				
			This commit is contained in:
		| @@ -69,8 +69,8 @@ | |||||||
| | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Cccs/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CCCS-02-001 … 02-007 | Atom feed verified 2025-10-11, history/caching review and FR locale enumeration pending. | | | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Cccs/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CCCS-02-001 … 02-007 | Atom feed verified 2025-10-11, history/caching review and FR locale enumeration pending. | | ||||||
| | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.CertBund/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CERTBUND-02-001 … 02-007 | BSI RSS directory confirmed CERT-Bund feed 2025-10-11, history assessment pending. | | | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.CertBund/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CERTBUND-02-001 … 02-007 | BSI RSS directory confirmed CERT-Bund feed 2025-10-11, history assessment pending. | | ||||||
| | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Kisa/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-KISA-02-001 … 02-007 | KNVD RSS endpoint identified 2025-10-11, access headers/session strategy outstanding. | | | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Kisa/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-KISA-02-001 … 02-007 | KNVD RSS endpoint identified 2025-10-11, access headers/session strategy outstanding. | | ||||||
| | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Ru.Bdu/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-RUBDU-02-001 … 02-008 | BDU RSS/Atom catalogue reviewed 2025-10-11, trust-store acquisition blocked by gosuslugi placeholder page. | | | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Ru.Bdu/TASKS.md | Build DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-RUBDU-02-001 … 02-008 | TLS bundle + connectors landed 2025-10-12; fetch/parse/map flow emits advisories, fixtures & telemetry follow-up pending. | | ||||||
| | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Ru.Nkcki/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-NKCKI-02-001 … 02-008 | cert.gov.ru paginated RSS landing checked 2025-10-11, access enablement plan pending. | | | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Ru.Nkcki/TASKS.md | Build DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-NKCKI-02-001 … 02-008 | JSON bulletin fetch + canonical mapping live 2025-10-12; regression fixtures added but blocked on Mongo2Go libcrypto dependency for test execution. | | ||||||
| | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Ics.Cisa/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ICSCISA-02-001 … 02-008 | new ICS RSS endpoint logged 2025-10-11 but Akamai blocks direct pulls, fallback strategy task opened. | | | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Ics.Cisa/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ICSCISA-02-001 … 02-008 | new ICS RSS endpoint logged 2025-10-11 but Akamai blocks direct pulls, fallback strategy task opened. | | ||||||
| | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Vndr.Cisco/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CISCO-02-001 … 02-007 | openVuln API + RSS reviewed 2025-10-11, auth/pagination memo pending. | | | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Vndr.Cisco/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CISCO-02-001 … 02-007 | openVuln API + RSS reviewed 2025-10-11, auth/pagination memo pending. | | ||||||
| | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Vndr.Msrc/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-MSRC-02-001 … 02-007 | MSRC API docs reviewed 2025-10-11, auth/throttling comparison memo pending.<br>Instructions to work:<br>Read ./AGENTS.md plus each module's AGENTS file. Parallelize research, ingestion, mapping, fixtures, and docs using the normalized rule shape from ./src/FASTER_MODELING_AND_NORMALIZATION.md. Coordinate daily with the merge coordination task from Sprint 1. | | | Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Vndr.Msrc/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-MSRC-02-001 … 02-007 | MSRC API docs reviewed 2025-10-11, auth/throttling comparison memo pending.<br>Instructions to work:<br>Read ./AGENTS.md plus each module's AGENTS file. Parallelize research, ingestion, mapping, fixtures, and docs using the normalized rule shape from ./src/FASTER_MODELING_AND_NORMALIZATION.md. Coordinate daily with the merge coordination task from Sprint 1. | | ||||||
|   | |||||||
							
								
								
									
										32
									
								
								certificates/globalsign_gcc_r6_alphassl_ca_2023.pem
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										32
									
								
								certificates/globalsign_gcc_r6_alphassl_ca_2023.pem
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,32 @@ | |||||||
|  | -----BEGIN CERTIFICATE----- | ||||||
|  | MIIFjDCCA3SgAwIBAgIQfx8skC6D0OO2+zvuR4tegDANBgkqhkiG9w0BAQsFADBM | ||||||
|  | MSAwHgYDVQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSNjETMBEGA1UEChMKR2xv | ||||||
|  | YmFsU2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjAeFw0yMzA3MTkwMzQzMjVaFw0y | ||||||
|  | NjA3MTkwMDAwMDBaMFUxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWdu | ||||||
|  | IG52LXNhMSswKQYDVQQDEyJHbG9iYWxTaWduIEdDQyBSNiBBbHBoYVNTTCBDQSAy | ||||||
|  | MDIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA00Jvk5ADppO0rgDn | ||||||
|  | j1M14XIb032Aas409JJFAb8cUjipFOth7ySLdaWLe3s63oSs5x3eWwzTpX4BFkzZ | ||||||
|  | bxT1eoJSHfT2M0wZ5QOPcCIjsr+YB8TAvV2yJSyq+emRrN/FtgCSTaWXSJ5jipW8 | ||||||
|  | SJ/VAuXPMzuAP2yYpuPcjjQ5GyrssDXgu+FhtYxqyFP7BSvx9jQhh5QV5zhLycua | ||||||
|  | n8n+J0Uw09WRQK6JGQ5HzDZQinkNel+fZZNRG1gE9Qeh+tHBplrkalB1g85qJkPO | ||||||
|  | J7SoEvKsmDkajggk/sSq7NPyzFaa/VBGZiRRG+FkxCBniGD5618PQ4trcwHyMojS | ||||||
|  | FObOHQIDAQABo4IBXzCCAVswDgYDVR0PAQH/BAQDAgGGMB0GA1UdJQQWMBQGCCsG | ||||||
|  | AQUFBwMBBggrBgEFBQcDAjASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1UdDgQWBBS9 | ||||||
|  | BbfzipM8c8t5+g+FEqF3lhiRdDAfBgNVHSMEGDAWgBSubAWjkxPioufi1xzWx/B/ | ||||||
|  | yGdToDB7BggrBgEFBQcBAQRvMG0wLgYIKwYBBQUHMAGGImh0dHA6Ly9vY3NwMi5n | ||||||
|  | bG9iYWxzaWduLmNvbS9yb290cjYwOwYIKwYBBQUHMAKGL2h0dHA6Ly9zZWN1cmUu | ||||||
|  | Z2xvYmFsc2lnbi5jb20vY2FjZXJ0L3Jvb3QtcjYuY3J0MDYGA1UdHwQvMC0wK6Ap | ||||||
|  | oCeGJWh0dHA6Ly9jcmwuZ2xvYmFsc2lnbi5jb20vcm9vdC1yNi5jcmwwIQYDVR0g | ||||||
|  | BBowGDAIBgZngQwBAgEwDAYKKwYBBAGgMgoBAzANBgkqhkiG9w0BAQsFAAOCAgEA | ||||||
|  | fMkkMo5g4mn1ft4d4xR2kHzYpDukhC1XYPwfSZN3A9nEBadjdKZMH7iuS1vF8uSc | ||||||
|  | g26/30DRPen2fFRsr662ECyUCR4OfeiiGNdoQvcesM9Xpew3HLQP4qHg+s774hNL | ||||||
|  | vGRD4aKSKwFqLMrcqCw6tEAfX99tFWsD4jzbC6k8tjSLzEl0fTUlfkJaWpvLVkpg | ||||||
|  | 9et8tD8d51bymCg5J6J6wcXpmsSGnksBobac1+nXmgB7jQC9edU8Z41FFo87BV3k | ||||||
|  | CtrWWsdkQavObMsXUPl/AO8y/jOuAWz0wyvPnKom+o6W4vKDY6/6XPypNdebOJ6m | ||||||
|  | jyaILp0quoQvhjx87BzENh5s57AIOyIGpS0sDEChVDPzLEfRsH2FJ8/W5woF0nvs | ||||||
|  | BTqfYSCqblQbHeDDtCj7Mlf8JfqaMuqcbE4rMSyfeHyCdZQwnc/r9ujnth691AJh | ||||||
|  | xyYeCM04metJIe7cB6d4dFm+Pd5ervY4x32r0uQ1Q0spy1VjNqUJjussYuXNyMmF | ||||||
|  | HSuLQQ6PrePmH5lcSMQpYKzPoD/RiNVD/PK0O3vuO5vh3o7oKb1FfzoanDsFFTrw | ||||||
|  | 0aLOdRW/tmLPWVNVlAb8ad+B80YJsL4HXYnQG8wYAFb8LhwSDyT9v+C1C1lcIHE7 | ||||||
|  | nE0AAp9JSHxDYsma9pi4g0Phg3BgOm2euTRzw7R0SzU= | ||||||
|  | -----END CERTIFICATE----- | ||||||
							
								
								
									
										65
									
								
								certificates/globalsign_r6_bundle.pem
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										65
									
								
								certificates/globalsign_r6_bundle.pem
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,65 @@ | |||||||
|  | -----BEGIN CERTIFICATE----- | ||||||
|  | MIIFgzCCA2ugAwIBAgIORea7A4Mzw4VlSOb/RVEwDQYJKoZIhvcNAQEMBQAwTDEg | ||||||
|  | MB4GA1UECxMXR2xvYmFsU2lnbiBSb290IENBIC0gUjYxEzARBgNVBAoTCkdsb2Jh | ||||||
|  | bFNpZ24xEzARBgNVBAMTCkdsb2JhbFNpZ24wHhcNMTQxMjEwMDAwMDAwWhcNMzQx | ||||||
|  | MjEwMDAwMDAwWjBMMSAwHgYDVQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSNjET | ||||||
|  | MBEGA1UEChMKR2xvYmFsU2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjCCAiIwDQYJ | ||||||
|  | KoZIhvcNAQEBBQADggIPADCCAgoCggIBAJUH6HPKZvnsFMp7PPcNCPG0RQssgrRI | ||||||
|  | xutbPK6DuEGSMxSkb3/pKszGsIhrxbaJ0cay/xTOURQh7ErdG1rG1ofuTToVBu1k | ||||||
|  | ZguSgMpE3nOUTvOniX9PeGMIyBJQbUJmL025eShNUhqKGoC3GYEOfsSKvGRMIRxD | ||||||
|  | aNc9PIrFsmbVkJq3MQbFvuJtMgamHvm566qjuL++gmNQ0PAYid/kD3n16qIfKtJw | ||||||
|  | LnvnvJO7bVPiSHyMEAc4/2ayd2F+4OqMPKq0pPbzlUoSB239jLKJz9CgYXfIWHSw | ||||||
|  | 1CM69106yqLbnQneXUQtkPGBzVeS+n68UARjNN9rkxi+azayOeSsJDa38O+2HBNX | ||||||
|  | k7besvjihbdzorg1qkXy4J02oW9UivFyVm4uiMVRQkQVlO6jxTiWm05OWgtH8wY2 | ||||||
|  | SXcwvHE35absIQh1/OZhFj931dmRl4QKbNQCTXTAFO39OfuD8l4UoQSwC+n+7o/h | ||||||
|  | bguyCLNhZglqsQY6ZZZZwPA1/cnaKI0aEYdwgQqomnUdnjqGBQCe24DWJfncBZ4n | ||||||
|  | WUx2OVvq+aWh2IMP0f/fMBH5hc8zSPXKbWQULHpYT9NLCEnFlWQaYw55PfWzjMpY | ||||||
|  | rZxCRXluDocZXFSxZba/jJvcE+kNb7gu3GduyYsRtYQUigAZcIN5kZeR1Bonvzce | ||||||
|  | MgfYFGM8KEyvAgMBAAGjYzBhMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTAD | ||||||
|  | AQH/MB0GA1UdDgQWBBSubAWjkxPioufi1xzWx/B/yGdToDAfBgNVHSMEGDAWgBSu | ||||||
|  | bAWjkxPioufi1xzWx/B/yGdToDANBgkqhkiG9w0BAQwFAAOCAgEAgyXt6NH9lVLN | ||||||
|  | nsAEoJFp5lzQhN7craJP6Ed41mWYqVuoPId8AorRbrcWc+ZfwFSY1XS+wc3iEZGt | ||||||
|  | Ixg93eFyRJa0lV7Ae46ZeBZDE1ZXs6KzO7V33EByrKPrmzU+sQghoefEQzd5Mr61 | ||||||
|  | 55wsTLxDKZmOMNOsIeDjHfrYBzN2VAAiKrlNIC5waNrlU/yDXNOd8v9EDERm8tLj | ||||||
|  | vUYAGm0CuiVdjaExUd1URhxN25mW7xocBFymFe944Hn+Xds+qkxV/ZoVqW/hpvvf | ||||||
|  | cDDpw+5CRu3CkwWJ+n1jez/QcYF8AOiYrg54NMMl+68KnyBr3TsTjxKM4kEaSHpz | ||||||
|  | oHdpx7Zcf4LIHv5YGygrqGytXm3ABdJ7t+uA/iU3/gKbaKxCXcPu9czc8FB10jZp | ||||||
|  | nOZ7BN9uBmm23goJSFmH63sUYHpkqmlD75HHTOwY3WzvUy2MmeFe8nI+z1TIvWfs | ||||||
|  | pA9MRf/TuTAjB0yPEL+GltmZWrSZVxykzLsViVO6LAUP5MSeGbEYNNVMnbrt9x+v | ||||||
|  | JJUEeKgDu+6B5dpffItKoZB0JaezPkvILFa9x8jvOOJckvB595yEunQtYQEgfn7R | ||||||
|  | 8k8HWV+LLUNS60YMlOH1Zkd5d9VUWx+tJDfLRVpOoERIyNiwmcUVhAn21klJwGW4 | ||||||
|  | 5hpxbqCo8YLoRT5s1gLXCmeDBVrJpBA= | ||||||
|  | -----END CERTIFICATE----- | ||||||
|  |  | ||||||
|  | -----BEGIN CERTIFICATE----- | ||||||
|  | MIIFjDCCA3SgAwIBAgIQfx8skC6D0OO2+zvuR4tegDANBgkqhkiG9w0BAQsFADBM | ||||||
|  | MSAwHgYDVQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSNjETMBEGA1UEChMKR2xv | ||||||
|  | YmFsU2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjAeFw0yMzA3MTkwMzQzMjVaFw0y | ||||||
|  | NjA3MTkwMDAwMDBaMFUxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWdu | ||||||
|  | IG52LXNhMSswKQYDVQQDEyJHbG9iYWxTaWduIEdDQyBSNiBBbHBoYVNTTCBDQSAy | ||||||
|  | MDIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA00Jvk5ADppO0rgDn | ||||||
|  | j1M14XIb032Aas409JJFAb8cUjipFOth7ySLdaWLe3s63oSs5x3eWwzTpX4BFkzZ | ||||||
|  | bxT1eoJSHfT2M0wZ5QOPcCIjsr+YB8TAvV2yJSyq+emRrN/FtgCSTaWXSJ5jipW8 | ||||||
|  | SJ/VAuXPMzuAP2yYpuPcjjQ5GyrssDXgu+FhtYxqyFP7BSvx9jQhh5QV5zhLycua | ||||||
|  | n8n+J0Uw09WRQK6JGQ5HzDZQinkNel+fZZNRG1gE9Qeh+tHBplrkalB1g85qJkPO | ||||||
|  | J7SoEvKsmDkajggk/sSq7NPyzFaa/VBGZiRRG+FkxCBniGD5618PQ4trcwHyMojS | ||||||
|  | FObOHQIDAQABo4IBXzCCAVswDgYDVR0PAQH/BAQDAgGGMB0GA1UdJQQWMBQGCCsG | ||||||
|  | AQUFBwMBBggrBgEFBQcDAjASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1UdDgQWBBS9 | ||||||
|  | BbfzipM8c8t5+g+FEqF3lhiRdDAfBgNVHSMEGDAWgBSubAWjkxPioufi1xzWx/B/ | ||||||
|  | yGdToDB7BggrBgEFBQcBAQRvMG0wLgYIKwYBBQUHMAGGImh0dHA6Ly9vY3NwMi5n | ||||||
|  | bG9iYWxzaWduLmNvbS9yb290cjYwOwYIKwYBBQUHMAKGL2h0dHA6Ly9zZWN1cmUu | ||||||
|  | Z2xvYmFsc2lnbi5jb20vY2FjZXJ0L3Jvb3QtcjYuY3J0MDYGA1UdHwQvMC0wK6Ap | ||||||
|  | oCeGJWh0dHA6Ly9jcmwuZ2xvYmFsc2lnbi5jb20vcm9vdC1yNi5jcmwwIQYDVR0g | ||||||
|  | BBowGDAIBgZngQwBAgEwDAYKKwYBBAGgMgoBAzANBgkqhkiG9w0BAQsFAAOCAgEA | ||||||
|  | fMkkMo5g4mn1ft4d4xR2kHzYpDukhC1XYPwfSZN3A9nEBadjdKZMH7iuS1vF8uSc | ||||||
|  | g26/30DRPen2fFRsr662ECyUCR4OfeiiGNdoQvcesM9Xpew3HLQP4qHg+s774hNL | ||||||
|  | vGRD4aKSKwFqLMrcqCw6tEAfX99tFWsD4jzbC6k8tjSLzEl0fTUlfkJaWpvLVkpg | ||||||
|  | 9et8tD8d51bymCg5J6J6wcXpmsSGnksBobac1+nXmgB7jQC9edU8Z41FFo87BV3k | ||||||
|  | CtrWWsdkQavObMsXUPl/AO8y/jOuAWz0wyvPnKom+o6W4vKDY6/6XPypNdebOJ6m | ||||||
|  | jyaILp0quoQvhjx87BzENh5s57AIOyIGpS0sDEChVDPzLEfRsH2FJ8/W5woF0nvs | ||||||
|  | BTqfYSCqblQbHeDDtCj7Mlf8JfqaMuqcbE4rMSyfeHyCdZQwnc/r9ujnth691AJh | ||||||
|  | xyYeCM04metJIe7cB6d4dFm+Pd5ervY4x32r0uQ1Q0spy1VjNqUJjussYuXNyMmF | ||||||
|  | HSuLQQ6PrePmH5lcSMQpYKzPoD/RiNVD/PK0O3vuO5vh3o7oKb1FfzoanDsFFTrw | ||||||
|  | 0aLOdRW/tmLPWVNVlAb8ad+B80YJsL4HXYnQG8wYAFb8LhwSDyT9v+C1C1lcIHE7 | ||||||
|  | nE0AAp9JSHxDYsma9pi4g0Phg3BgOm2euTRzw7R0SzU= | ||||||
|  | -----END CERTIFICATE----- | ||||||
							
								
								
									
										32
									
								
								certificates/globalsign_root_r6.pem
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										32
									
								
								certificates/globalsign_root_r6.pem
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,32 @@ | |||||||
|  | -----BEGIN CERTIFICATE----- | ||||||
|  | MIIFgzCCA2ugAwIBAgIORea7A4Mzw4VlSOb/RVEwDQYJKoZIhvcNAQEMBQAwTDEg | ||||||
|  | MB4GA1UECxMXR2xvYmFsU2lnbiBSb290IENBIC0gUjYxEzARBgNVBAoTCkdsb2Jh | ||||||
|  | bFNpZ24xEzARBgNVBAMTCkdsb2JhbFNpZ24wHhcNMTQxMjEwMDAwMDAwWhcNMzQx | ||||||
|  | MjEwMDAwMDAwWjBMMSAwHgYDVQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSNjET | ||||||
|  | MBEGA1UEChMKR2xvYmFsU2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjCCAiIwDQYJ | ||||||
|  | KoZIhvcNAQEBBQADggIPADCCAgoCggIBAJUH6HPKZvnsFMp7PPcNCPG0RQssgrRI | ||||||
|  | xutbPK6DuEGSMxSkb3/pKszGsIhrxbaJ0cay/xTOURQh7ErdG1rG1ofuTToVBu1k | ||||||
|  | ZguSgMpE3nOUTvOniX9PeGMIyBJQbUJmL025eShNUhqKGoC3GYEOfsSKvGRMIRxD | ||||||
|  | aNc9PIrFsmbVkJq3MQbFvuJtMgamHvm566qjuL++gmNQ0PAYid/kD3n16qIfKtJw | ||||||
|  | LnvnvJO7bVPiSHyMEAc4/2ayd2F+4OqMPKq0pPbzlUoSB239jLKJz9CgYXfIWHSw | ||||||
|  | 1CM69106yqLbnQneXUQtkPGBzVeS+n68UARjNN9rkxi+azayOeSsJDa38O+2HBNX | ||||||
|  | k7besvjihbdzorg1qkXy4J02oW9UivFyVm4uiMVRQkQVlO6jxTiWm05OWgtH8wY2 | ||||||
|  | SXcwvHE35absIQh1/OZhFj931dmRl4QKbNQCTXTAFO39OfuD8l4UoQSwC+n+7o/h | ||||||
|  | bguyCLNhZglqsQY6ZZZZwPA1/cnaKI0aEYdwgQqomnUdnjqGBQCe24DWJfncBZ4n | ||||||
|  | WUx2OVvq+aWh2IMP0f/fMBH5hc8zSPXKbWQULHpYT9NLCEnFlWQaYw55PfWzjMpY | ||||||
|  | rZxCRXluDocZXFSxZba/jJvcE+kNb7gu3GduyYsRtYQUigAZcIN5kZeR1Bonvzce | ||||||
|  | MgfYFGM8KEyvAgMBAAGjYzBhMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTAD | ||||||
|  | AQH/MB0GA1UdDgQWBBSubAWjkxPioufi1xzWx/B/yGdToDAfBgNVHSMEGDAWgBSu | ||||||
|  | bAWjkxPioufi1xzWx/B/yGdToDANBgkqhkiG9w0BAQwFAAOCAgEAgyXt6NH9lVLN | ||||||
|  | nsAEoJFp5lzQhN7craJP6Ed41mWYqVuoPId8AorRbrcWc+ZfwFSY1XS+wc3iEZGt | ||||||
|  | Ixg93eFyRJa0lV7Ae46ZeBZDE1ZXs6KzO7V33EByrKPrmzU+sQghoefEQzd5Mr61 | ||||||
|  | 55wsTLxDKZmOMNOsIeDjHfrYBzN2VAAiKrlNIC5waNrlU/yDXNOd8v9EDERm8tLj | ||||||
|  | vUYAGm0CuiVdjaExUd1URhxN25mW7xocBFymFe944Hn+Xds+qkxV/ZoVqW/hpvvf | ||||||
|  | cDDpw+5CRu3CkwWJ+n1jez/QcYF8AOiYrg54NMMl+68KnyBr3TsTjxKM4kEaSHpz | ||||||
|  | oHdpx7Zcf4LIHv5YGygrqGytXm3ABdJ7t+uA/iU3/gKbaKxCXcPu9czc8FB10jZp | ||||||
|  | nOZ7BN9uBmm23goJSFmH63sUYHpkqmlD75HHTOwY3WzvUy2MmeFe8nI+z1TIvWfs | ||||||
|  | pA9MRf/TuTAjB0yPEL+GltmZWrSZVxykzLsViVO6LAUP5MSeGbEYNNVMnbrt9x+v | ||||||
|  | JJUEeKgDu+6B5dpffItKoZB0JaezPkvILFa9x8jvOOJckvB595yEunQtYQEgfn7R | ||||||
|  | 8k8HWV+LLUNS60YMlOH1Zkd5d9VUWx+tJDfLRVpOoERIyNiwmcUVhAn21klJwGW4 | ||||||
|  | 5hpxbqCo8YLoRT5s1gLXCmeDBVrJpBA= | ||||||
|  | -----END CERTIFICATE----- | ||||||
							
								
								
									
										74
									
								
								certificates/russian_trusted_bundle.pem
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										74
									
								
								certificates/russian_trusted_bundle.pem
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,74 @@ | |||||||
|  | -----BEGIN CERTIFICATE----- | ||||||
|  | MIIFwjCCA6qgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwcDELMAkGA1UEBhMCUlUx | ||||||
|  | PzA9BgNVBAoMNlRoZSBNaW5pc3RyeSBvZiBEaWdpdGFsIERldmVsb3BtZW50IGFu | ||||||
|  | ZCBDb21tdW5pY2F0aW9uczEgMB4GA1UEAwwXUnVzc2lhbiBUcnVzdGVkIFJvb3Qg | ||||||
|  | Q0EwHhcNMjIwMzAxMjEwNDE1WhcNMzIwMjI3MjEwNDE1WjBwMQswCQYDVQQGEwJS | ||||||
|  | VTE/MD0GA1UECgw2VGhlIE1pbmlzdHJ5IG9mIERpZ2l0YWwgRGV2ZWxvcG1lbnQg | ||||||
|  | YW5kIENvbW11bmljYXRpb25zMSAwHgYDVQQDDBdSdXNzaWFuIFRydXN0ZWQgUm9v | ||||||
|  | dCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMfFOZ8pUAL3+r2n | ||||||
|  | qqE0Zp52selXsKGFYoG0GM5bwz1bSFtCt+AZQMhkWQheI3poZAToYJu69pHLKS6Q | ||||||
|  | XBiwBC1cvzYmUYKMYZC7jE5YhEU2bSL0mX7NaMxMDmH2/NwuOVRj8OImVa5s1F4U | ||||||
|  | zn4Kv3PFlDBjjSjXKVY9kmjUBsXQrIHeaqmUIsPIlNWUnimXS0I0abExqkbdrXbX | ||||||
|  | YwCOXhOO2pDUx3ckmJlCMUGacUTnylyQW2VsJIyIGA8V0xzdaeUXg0VZ6ZmNUr5Y | ||||||
|  | Ber/EAOLPb8NYpsAhJe2mXjMB/J9HNsoFMBFJ0lLOT/+dQvjbdRZoOT8eqJpWnVD | ||||||
|  | U+QL/qEZnz57N88OWM3rabJkRNdU/Z7x5SFIM9FrqtN8xewsiBWBI0K6XFuOBOTD | ||||||
|  | 4V08o4TzJ8+Ccq5XlCUW2L48pZNCYuBDfBh7FxkB7qDgGDiaftEkZZfApRg2E+M9 | ||||||
|  | G8wkNKTPLDc4wH0FDTijhgxR3Y4PiS1HL2Zhw7bD3CbslmEGgfnnZojNkJtcLeBH | ||||||
|  | BLa52/dSwNU4WWLubaYSiAmA9IUMX1/RpfpxOxd4Ykmhz97oFbUaDJFipIggx5sX | ||||||
|  | ePAlkTdWnv+RWBxlJwMQ25oEHmRguNYf4Zr/Rxr9cS93Y+mdXIZaBEE0KS2iLRqa | ||||||
|  | OiWBki9IMQU4phqPOBAaG7A+eP8PAgMBAAGjZjBkMB0GA1UdDgQWBBTh0YHlzlpf | ||||||
|  | BKrS6badZrHF+qwshzAfBgNVHSMEGDAWgBTh0YHlzlpfBKrS6badZrHF+qwshzAS | ||||||
|  | BgNVHRMBAf8ECDAGAQH/AgEEMA4GA1UdDwEB/wQEAwIBhjANBgkqhkiG9w0BAQsF | ||||||
|  | AAOCAgEAALIY1wkilt/urfEVM5vKzr6utOeDWCUczmWX/RX4ljpRdgF+5fAIS4vH | ||||||
|  | tmXkqpSCOVeWUrJV9QvZn6L227ZwuE15cWi8DCDal3Ue90WgAJJZMfTshN4OI8cq | ||||||
|  | W9E4EG9wglbEtMnObHlms8F3CHmrw3k6KmUkWGoa+/ENmcVl68u/cMRl1JbW2bM+ | ||||||
|  | /3A+SAg2c6iPDlehczKx2oa95QW0SkPPWGuNA/CE8CpyANIhu9XFrj3RQ3EqeRcS | ||||||
|  | AQQod1RNuHpfETLU/A2gMmvn/w/sx7TB3W5BPs6rprOA37tutPq9u6FTZOcG1Oqj | ||||||
|  | C/B7yTqgI7rbyvox7DEXoX7rIiEqyNNUguTk/u3SZ4VXE2kmxdmSh3TQvybfbnXV | ||||||
|  | 4JbCZVaqiZraqc7oZMnRoWrXRG3ztbnbes/9qhRGI7PqXqeKJBztxRTEVj8ONs1d | ||||||
|  | WN5szTwaPIvhkhO3CO5ErU2rVdUr89wKpNXbBODFKRtgxUT70YpmJ46VVaqdAhOZ | ||||||
|  | D9EUUn4YaeLaS8AjSF/h7UkjOibNc4qVDiPP+rkehFWM66PVnP1Msh93tc+taIfC | ||||||
|  | EYVMxjh8zNbFuoc7fzvvrFILLe7ifvEIUqSVIC/AzplM/Jxw7buXFeGP1qVCBEHq | ||||||
|  | 391d/9RAfaZ12zkwFsl+IKwE/OZxW8AHa9i1p4GO0YSNuczzEm4= | ||||||
|  | -----END CERTIFICATE----- | ||||||
|  | -----BEGIN CERTIFICATE----- | ||||||
|  | MIIHQjCCBSqgAwIBAgICEAIwDQYJKoZIhvcNAQELBQAwcDELMAkGA1UEBhMCUlUx | ||||||
|  | PzA9BgNVBAoMNlRoZSBNaW5pc3RyeSBvZiBEaWdpdGFsIERldmVsb3BtZW50IGFu | ||||||
|  | ZCBDb21tdW5pY2F0aW9uczEgMB4GA1UEAwwXUnVzc2lhbiBUcnVzdGVkIFJvb3Qg | ||||||
|  | Q0EwHhcNMjIwMzAyMTEyNTE5WhcNMjcwMzA2MTEyNTE5WjBvMQswCQYDVQQGEwJS | ||||||
|  | VTE/MD0GA1UECgw2VGhlIE1pbmlzdHJ5IG9mIERpZ2l0YWwgRGV2ZWxvcG1lbnQg | ||||||
|  | YW5kIENvbW11bmljYXRpb25zMR8wHQYDVQQDDBZSdXNzaWFuIFRydXN0ZWQgU3Vi | ||||||
|  | IENBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA9YPqBKOk19NFymrE | ||||||
|  | wehzrhBEgT2atLezpduB24mQ7CiOa/HVpFCDRZzdxqlh8drku408/tTmWzlNH/br | ||||||
|  | HuQhZ/miWKOf35lpKzjyBd6TPM23uAfJvEOQ2/dnKGGJbsUo1/udKSvxQwVHpVv3 | ||||||
|  | S80OlluKfhWPDEXQpgyFqIzPoxIQTLZ0deirZwMVHarZ5u8HqHetRuAtmO2ZDGQn | ||||||
|  | vVOJYAjls+Hiueq7Lj7Oce7CQsTwVZeP+XQx28PAaEZ3y6sQEt6rL06ddpSdoTMp | ||||||
|  | BnCqTbxW+eWMyjkIn6t9GBtUV45yB1EkHNnj2Ex4GwCiN9T84QQjKSr+8f0psGrZ | ||||||
|  | vPbCbQAwNFJjisLixnjlGPLKa5vOmNwIh/LAyUW5DjpkCx004LPDuqPpFsKXNKpa | ||||||
|  | L2Dm6uc0x4Jo5m+gUTVORB6hOSzWnWDj2GWfomLzzyjG81DRGFBpco/O93zecsIN | ||||||
|  | 3SL2Ysjpq1zdoS01CMYxie//9zWvYwzI25/OZigtnpCIrcd2j1Y6dMUFQAzAtHE+ | ||||||
|  | qsXflSL8HIS+IJEFIQobLlYhHkoE3avgNx5jlu+OLYe0dF0Ykx1PGNjbwqvTX37R | ||||||
|  | Cn32NMjlotW2QcGEZhDKj+3urZizp5xdTPZitA+aEjZM/Ni71VOdiOP0igbw6asZ | ||||||
|  | 2fxdozZ1TnSSYNYvNATwthNmZysCAwEAAaOCAeUwggHhMBIGA1UdEwEB/wQIMAYB | ||||||
|  | Af8CAQAwDgYDVR0PAQH/BAQDAgGGMB0GA1UdDgQWBBTR4XENCy2BTm6KSo9MI7NM | ||||||
|  | XqtpCzAfBgNVHSMEGDAWgBTh0YHlzlpfBKrS6badZrHF+qwshzCBxwYIKwYBBQUH | ||||||
|  | AQEEgbowgbcwOwYIKwYBBQUHMAKGL2h0dHA6Ly9yb3N0ZWxlY29tLnJ1L2NkcC9y | ||||||
|  | b290Y2Ffc3NsX3JzYTIwMjIuY3J0MDsGCCsGAQUFBzAChi9odHRwOi8vY29tcGFu | ||||||
|  | eS5ydC5ydS9jZHAvcm9vdGNhX3NzbF9yc2EyMDIyLmNydDA7BggrBgEFBQcwAoYv | ||||||
|  | aHR0cDovL3JlZXN0ci1wa2kucnUvY2RwL3Jvb3RjYV9zc2xfcnNhMjAyMi5jcnQw | ||||||
|  | gbAGA1UdHwSBqDCBpTA1oDOgMYYvaHR0cDovL3Jvc3RlbGVjb20ucnUvY2RwL3Jv | ||||||
|  | b3RjYV9zc2xfcnNhMjAyMi5jcmwwNaAzoDGGL2h0dHA6Ly9jb21wYW55LnJ0LnJ1 | ||||||
|  | L2NkcC9yb290Y2Ffc3NsX3JzYTIwMjIuY3JsMDWgM6Axhi9odHRwOi8vcmVlc3Ry | ||||||
|  | LXBraS5ydS9jZHAvcm9vdGNhX3NzbF9yc2EyMDIyLmNybDANBgkqhkiG9w0BAQsF | ||||||
|  | AAOCAgEARBVzZls79AdiSCpar15dA5Hr/rrT4WbrOfzlpI+xrLeRPrUG6eUWIW4v | ||||||
|  | Sui1yx3iqGLCjPcKb+HOTwoRMbI6ytP/ndp3TlYua2advYBEhSvjs+4vDZNwXr/D | ||||||
|  | anbwIWdurZmViQRBDFebpkvnIvru/RpWud/5r624Wp8voZMRtj/cm6aI9LtvBfT9 | ||||||
|  | cfzhOaexI/99c14dyiuk1+6QhdwKaCRTc1mdfNQmnfWNRbfWhWBlK3h4GGE9JK33 | ||||||
|  | Gk8ZS8DMrkdAh0xby4xAQ/mSWAfWrBmfzlOqGyoB1U47WTOeqNbWkkoAP2ys94+s | ||||||
|  | Jg4NTkiDVtXRF6nr6fYi0bSOvOFg0IQrMXO2Y8gyg9ARdPJwKtvWX8VPADCYMiWH | ||||||
|  | h4n8bZokIrImVKLDQKHY4jCsND2HHdJfnrdL2YJw1qFskNO4cSNmZydw0Wkgjv9k | ||||||
|  | F+KxqrDKlB8MZu2Hclph6v/CZ0fQ9YuE8/lsHZ0Qc2HyiSMnvjgK5fDc3TD4fa8F | ||||||
|  | E8gMNurM+kV8PT8LNIM+4Zs+LKEV8nqRWBaxkIVJGekkVKO8xDBOG/aN62AZKHOe | ||||||
|  | GcyIdu7yNMMRihGVZCYr8rYiJoKiOzDqOkPkLOPdhtVlgnhowzHDxMHND/E2WA5p | ||||||
|  | ZHuNM/m0TXt2wTTPL7JH2YC0gPz/BvvSzjksgzU5rLbRyUKQkgU= | ||||||
|  | -----END CERTIFICATE----- | ||||||
							
								
								
									
										33
									
								
								certificates/russian_trusted_root_ca.pem
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										33
									
								
								certificates/russian_trusted_root_ca.pem
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,33 @@ | |||||||
|  | -----BEGIN CERTIFICATE----- | ||||||
|  | MIIFwjCCA6qgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwcDELMAkGA1UEBhMCUlUx | ||||||
|  | PzA9BgNVBAoMNlRoZSBNaW5pc3RyeSBvZiBEaWdpdGFsIERldmVsb3BtZW50IGFu | ||||||
|  | ZCBDb21tdW5pY2F0aW9uczEgMB4GA1UEAwwXUnVzc2lhbiBUcnVzdGVkIFJvb3Qg | ||||||
|  | Q0EwHhcNMjIwMzAxMjEwNDE1WhcNMzIwMjI3MjEwNDE1WjBwMQswCQYDVQQGEwJS | ||||||
|  | VTE/MD0GA1UECgw2VGhlIE1pbmlzdHJ5IG9mIERpZ2l0YWwgRGV2ZWxvcG1lbnQg | ||||||
|  | YW5kIENvbW11bmljYXRpb25zMSAwHgYDVQQDDBdSdXNzaWFuIFRydXN0ZWQgUm9v | ||||||
|  | dCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMfFOZ8pUAL3+r2n | ||||||
|  | qqE0Zp52selXsKGFYoG0GM5bwz1bSFtCt+AZQMhkWQheI3poZAToYJu69pHLKS6Q | ||||||
|  | XBiwBC1cvzYmUYKMYZC7jE5YhEU2bSL0mX7NaMxMDmH2/NwuOVRj8OImVa5s1F4U | ||||||
|  | zn4Kv3PFlDBjjSjXKVY9kmjUBsXQrIHeaqmUIsPIlNWUnimXS0I0abExqkbdrXbX | ||||||
|  | YwCOXhOO2pDUx3ckmJlCMUGacUTnylyQW2VsJIyIGA8V0xzdaeUXg0VZ6ZmNUr5Y | ||||||
|  | Ber/EAOLPb8NYpsAhJe2mXjMB/J9HNsoFMBFJ0lLOT/+dQvjbdRZoOT8eqJpWnVD | ||||||
|  | U+QL/qEZnz57N88OWM3rabJkRNdU/Z7x5SFIM9FrqtN8xewsiBWBI0K6XFuOBOTD | ||||||
|  | 4V08o4TzJ8+Ccq5XlCUW2L48pZNCYuBDfBh7FxkB7qDgGDiaftEkZZfApRg2E+M9 | ||||||
|  | G8wkNKTPLDc4wH0FDTijhgxR3Y4PiS1HL2Zhw7bD3CbslmEGgfnnZojNkJtcLeBH | ||||||
|  | BLa52/dSwNU4WWLubaYSiAmA9IUMX1/RpfpxOxd4Ykmhz97oFbUaDJFipIggx5sX | ||||||
|  | ePAlkTdWnv+RWBxlJwMQ25oEHmRguNYf4Zr/Rxr9cS93Y+mdXIZaBEE0KS2iLRqa | ||||||
|  | OiWBki9IMQU4phqPOBAaG7A+eP8PAgMBAAGjZjBkMB0GA1UdDgQWBBTh0YHlzlpf | ||||||
|  | BKrS6badZrHF+qwshzAfBgNVHSMEGDAWgBTh0YHlzlpfBKrS6badZrHF+qwshzAS | ||||||
|  | BgNVHRMBAf8ECDAGAQH/AgEEMA4GA1UdDwEB/wQEAwIBhjANBgkqhkiG9w0BAQsF | ||||||
|  | AAOCAgEAALIY1wkilt/urfEVM5vKzr6utOeDWCUczmWX/RX4ljpRdgF+5fAIS4vH | ||||||
|  | tmXkqpSCOVeWUrJV9QvZn6L227ZwuE15cWi8DCDal3Ue90WgAJJZMfTshN4OI8cq | ||||||
|  | W9E4EG9wglbEtMnObHlms8F3CHmrw3k6KmUkWGoa+/ENmcVl68u/cMRl1JbW2bM+ | ||||||
|  | /3A+SAg2c6iPDlehczKx2oa95QW0SkPPWGuNA/CE8CpyANIhu9XFrj3RQ3EqeRcS | ||||||
|  | AQQod1RNuHpfETLU/A2gMmvn/w/sx7TB3W5BPs6rprOA37tutPq9u6FTZOcG1Oqj | ||||||
|  | C/B7yTqgI7rbyvox7DEXoX7rIiEqyNNUguTk/u3SZ4VXE2kmxdmSh3TQvybfbnXV | ||||||
|  | 4JbCZVaqiZraqc7oZMnRoWrXRG3ztbnbes/9qhRGI7PqXqeKJBztxRTEVj8ONs1d | ||||||
|  | WN5szTwaPIvhkhO3CO5ErU2rVdUr89wKpNXbBODFKRtgxUT70YpmJ46VVaqdAhOZ | ||||||
|  | D9EUUn4YaeLaS8AjSF/h7UkjOibNc4qVDiPP+rkehFWM66PVnP1Msh93tc+taIfC | ||||||
|  | EYVMxjh8zNbFuoc7fzvvrFILLe7ifvEIUqSVIC/AzplM/Jxw7buXFeGP1qVCBEHq | ||||||
|  | 391d/9RAfaZ12zkwFsl+IKwE/OZxW8AHa9i1p4GO0YSNuczzEm4= | ||||||
|  | -----END CERTIFICATE----- | ||||||
| @@ -1,10 +1,41 @@ | |||||||
| Title: Госуслуги | -----BEGIN CERTIFICATE----- | ||||||
|  | MIIHQjCCBSqgAwIBAgICEAIwDQYJKoZIhvcNAQELBQAwcDELMAkGA1UEBhMCUlUx | ||||||
| URL Source: https://www.gosuslugi.ru/tls/files/subca2022.cer | PzA9BgNVBAoMNlRoZSBNaW5pc3RyeSBvZiBEaWdpdGFsIERldmVsb3BtZW50IGFu | ||||||
|  | ZCBDb21tdW5pY2F0aW9uczEgMB4GA1UEAwwXUnVzc2lhbiBUcnVzdGVkIFJvb3Qg | ||||||
| Markdown Content: | Q0EwHhcNMjIwMzAyMTEyNTE5WhcNMjcwMzA2MTEyNTE5WjBvMQswCQYDVQQGEwJS | ||||||
| ### Госуслуги сейчас откроются | VTE/MD0GA1UECgw2VGhlIE1pbmlzdHJ5IG9mIERpZ2l0YWwgRGV2ZWxvcG1lbnQg | ||||||
|  | YW5kIENvbW11bmljYXRpb25zMR8wHQYDVQQDDBZSdXNzaWFuIFRydXN0ZWQgU3Vi | ||||||
| Портал работает в прежнем режиме.  | IENBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA9YPqBKOk19NFymrE | ||||||
|  | wehzrhBEgT2atLezpduB24mQ7CiOa/HVpFCDRZzdxqlh8drku408/tTmWzlNH/br | ||||||
|  Подождите пару секунд | HuQhZ/miWKOf35lpKzjyBd6TPM23uAfJvEOQ2/dnKGGJbsUo1/udKSvxQwVHpVv3 | ||||||
|  | S80OlluKfhWPDEXQpgyFqIzPoxIQTLZ0deirZwMVHarZ5u8HqHetRuAtmO2ZDGQn | ||||||
|  | vVOJYAjls+Hiueq7Lj7Oce7CQsTwVZeP+XQx28PAaEZ3y6sQEt6rL06ddpSdoTMp | ||||||
|  | BnCqTbxW+eWMyjkIn6t9GBtUV45yB1EkHNnj2Ex4GwCiN9T84QQjKSr+8f0psGrZ | ||||||
|  | vPbCbQAwNFJjisLixnjlGPLKa5vOmNwIh/LAyUW5DjpkCx004LPDuqPpFsKXNKpa | ||||||
|  | L2Dm6uc0x4Jo5m+gUTVORB6hOSzWnWDj2GWfomLzzyjG81DRGFBpco/O93zecsIN | ||||||
|  | 3SL2Ysjpq1zdoS01CMYxie//9zWvYwzI25/OZigtnpCIrcd2j1Y6dMUFQAzAtHE+ | ||||||
|  | qsXflSL8HIS+IJEFIQobLlYhHkoE3avgNx5jlu+OLYe0dF0Ykx1PGNjbwqvTX37R | ||||||
|  | Cn32NMjlotW2QcGEZhDKj+3urZizp5xdTPZitA+aEjZM/Ni71VOdiOP0igbw6asZ | ||||||
|  | 2fxdozZ1TnSSYNYvNATwthNmZysCAwEAAaOCAeUwggHhMBIGA1UdEwEB/wQIMAYB | ||||||
|  | Af8CAQAwDgYDVR0PAQH/BAQDAgGGMB0GA1UdDgQWBBTR4XENCy2BTm6KSo9MI7NM | ||||||
|  | XqtpCzAfBgNVHSMEGDAWgBTh0YHlzlpfBKrS6badZrHF+qwshzCBxwYIKwYBBQUH | ||||||
|  | AQEEgbowgbcwOwYIKwYBBQUHMAKGL2h0dHA6Ly9yb3N0ZWxlY29tLnJ1L2NkcC9y | ||||||
|  | b290Y2Ffc3NsX3JzYTIwMjIuY3J0MDsGCCsGAQUFBzAChi9odHRwOi8vY29tcGFu | ||||||
|  | eS5ydC5ydS9jZHAvcm9vdGNhX3NzbF9yc2EyMDIyLmNydDA7BggrBgEFBQcwAoYv | ||||||
|  | aHR0cDovL3JlZXN0ci1wa2kucnUvY2RwL3Jvb3RjYV9zc2xfcnNhMjAyMi5jcnQw | ||||||
|  | gbAGA1UdHwSBqDCBpTA1oDOgMYYvaHR0cDovL3Jvc3RlbGVjb20ucnUvY2RwL3Jv | ||||||
|  | b3RjYV9zc2xfcnNhMjAyMi5jcmwwNaAzoDGGL2h0dHA6Ly9jb21wYW55LnJ0LnJ1 | ||||||
|  | L2NkcC9yb290Y2Ffc3NsX3JzYTIwMjIuY3JsMDWgM6Axhi9odHRwOi8vcmVlc3Ry | ||||||
|  | LXBraS5ydS9jZHAvcm9vdGNhX3NzbF9yc2EyMDIyLmNybDANBgkqhkiG9w0BAQsF | ||||||
|  | AAOCAgEARBVzZls79AdiSCpar15dA5Hr/rrT4WbrOfzlpI+xrLeRPrUG6eUWIW4v | ||||||
|  | Sui1yx3iqGLCjPcKb+HOTwoRMbI6ytP/ndp3TlYua2advYBEhSvjs+4vDZNwXr/D | ||||||
|  | anbwIWdurZmViQRBDFebpkvnIvru/RpWud/5r624Wp8voZMRtj/cm6aI9LtvBfT9 | ||||||
|  | cfzhOaexI/99c14dyiuk1+6QhdwKaCRTc1mdfNQmnfWNRbfWhWBlK3h4GGE9JK33 | ||||||
|  | Gk8ZS8DMrkdAh0xby4xAQ/mSWAfWrBmfzlOqGyoB1U47WTOeqNbWkkoAP2ys94+s | ||||||
|  | Jg4NTkiDVtXRF6nr6fYi0bSOvOFg0IQrMXO2Y8gyg9ARdPJwKtvWX8VPADCYMiWH | ||||||
|  | h4n8bZokIrImVKLDQKHY4jCsND2HHdJfnrdL2YJw1qFskNO4cSNmZydw0Wkgjv9k | ||||||
|  | F+KxqrDKlB8MZu2Hclph6v/CZ0fQ9YuE8/lsHZ0Qc2HyiSMnvjgK5fDc3TD4fa8F | ||||||
|  | E8gMNurM+kV8PT8LNIM+4Zs+LKEV8nqRWBaxkIVJGekkVKO8xDBOG/aN62AZKHOe | ||||||
|  | GcyIdu7yNMMRihGVZCYr8rYiJoKiOzDqOkPkLOPdhtVlgnhowzHDxMHND/E2WA5p | ||||||
|  | ZHuNM/m0TXt2wTTPL7JH2YC0gPz/BvvSzjksgzU5rLbRyUKQkgU= | ||||||
|  | -----END CERTIFICATE----- | ||||||
|   | |||||||
							
								
								
									
										41
									
								
								certificates/russian_trusted_sub_ca.pem
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										41
									
								
								certificates/russian_trusted_sub_ca.pem
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,41 @@ | |||||||
|  | -----BEGIN CERTIFICATE----- | ||||||
|  | MIIHQjCCBSqgAwIBAgICEAIwDQYJKoZIhvcNAQELBQAwcDELMAkGA1UEBhMCUlUx | ||||||
|  | PzA9BgNVBAoMNlRoZSBNaW5pc3RyeSBvZiBEaWdpdGFsIERldmVsb3BtZW50IGFu | ||||||
|  | ZCBDb21tdW5pY2F0aW9uczEgMB4GA1UEAwwXUnVzc2lhbiBUcnVzdGVkIFJvb3Qg | ||||||
|  | Q0EwHhcNMjIwMzAyMTEyNTE5WhcNMjcwMzA2MTEyNTE5WjBvMQswCQYDVQQGEwJS | ||||||
|  | VTE/MD0GA1UECgw2VGhlIE1pbmlzdHJ5IG9mIERpZ2l0YWwgRGV2ZWxvcG1lbnQg | ||||||
|  | YW5kIENvbW11bmljYXRpb25zMR8wHQYDVQQDDBZSdXNzaWFuIFRydXN0ZWQgU3Vi | ||||||
|  | IENBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA9YPqBKOk19NFymrE | ||||||
|  | wehzrhBEgT2atLezpduB24mQ7CiOa/HVpFCDRZzdxqlh8drku408/tTmWzlNH/br | ||||||
|  | HuQhZ/miWKOf35lpKzjyBd6TPM23uAfJvEOQ2/dnKGGJbsUo1/udKSvxQwVHpVv3 | ||||||
|  | S80OlluKfhWPDEXQpgyFqIzPoxIQTLZ0deirZwMVHarZ5u8HqHetRuAtmO2ZDGQn | ||||||
|  | vVOJYAjls+Hiueq7Lj7Oce7CQsTwVZeP+XQx28PAaEZ3y6sQEt6rL06ddpSdoTMp | ||||||
|  | BnCqTbxW+eWMyjkIn6t9GBtUV45yB1EkHNnj2Ex4GwCiN9T84QQjKSr+8f0psGrZ | ||||||
|  | vPbCbQAwNFJjisLixnjlGPLKa5vOmNwIh/LAyUW5DjpkCx004LPDuqPpFsKXNKpa | ||||||
|  | L2Dm6uc0x4Jo5m+gUTVORB6hOSzWnWDj2GWfomLzzyjG81DRGFBpco/O93zecsIN | ||||||
|  | 3SL2Ysjpq1zdoS01CMYxie//9zWvYwzI25/OZigtnpCIrcd2j1Y6dMUFQAzAtHE+ | ||||||
|  | qsXflSL8HIS+IJEFIQobLlYhHkoE3avgNx5jlu+OLYe0dF0Ykx1PGNjbwqvTX37R | ||||||
|  | Cn32NMjlotW2QcGEZhDKj+3urZizp5xdTPZitA+aEjZM/Ni71VOdiOP0igbw6asZ | ||||||
|  | 2fxdozZ1TnSSYNYvNATwthNmZysCAwEAAaOCAeUwggHhMBIGA1UdEwEB/wQIMAYB | ||||||
|  | Af8CAQAwDgYDVR0PAQH/BAQDAgGGMB0GA1UdDgQWBBTR4XENCy2BTm6KSo9MI7NM | ||||||
|  | XqtpCzAfBgNVHSMEGDAWgBTh0YHlzlpfBKrS6badZrHF+qwshzCBxwYIKwYBBQUH | ||||||
|  | AQEEgbowgbcwOwYIKwYBBQUHMAKGL2h0dHA6Ly9yb3N0ZWxlY29tLnJ1L2NkcC9y | ||||||
|  | b290Y2Ffc3NsX3JzYTIwMjIuY3J0MDsGCCsGAQUFBzAChi9odHRwOi8vY29tcGFu | ||||||
|  | eS5ydC5ydS9jZHAvcm9vdGNhX3NzbF9yc2EyMDIyLmNydDA7BggrBgEFBQcwAoYv | ||||||
|  | aHR0cDovL3JlZXN0ci1wa2kucnUvY2RwL3Jvb3RjYV9zc2xfcnNhMjAyMi5jcnQw | ||||||
|  | gbAGA1UdHwSBqDCBpTA1oDOgMYYvaHR0cDovL3Jvc3RlbGVjb20ucnUvY2RwL3Jv | ||||||
|  | b3RjYV9zc2xfcnNhMjAyMi5jcmwwNaAzoDGGL2h0dHA6Ly9jb21wYW55LnJ0LnJ1 | ||||||
|  | L2NkcC9yb290Y2Ffc3NsX3JzYTIwMjIuY3JsMDWgM6Axhi9odHRwOi8vcmVlc3Ry | ||||||
|  | LXBraS5ydS9jZHAvcm9vdGNhX3NzbF9yc2EyMDIyLmNybDANBgkqhkiG9w0BAQsF | ||||||
|  | AAOCAgEARBVzZls79AdiSCpar15dA5Hr/rrT4WbrOfzlpI+xrLeRPrUG6eUWIW4v | ||||||
|  | Sui1yx3iqGLCjPcKb+HOTwoRMbI6ytP/ndp3TlYua2advYBEhSvjs+4vDZNwXr/D | ||||||
|  | anbwIWdurZmViQRBDFebpkvnIvru/RpWud/5r624Wp8voZMRtj/cm6aI9LtvBfT9 | ||||||
|  | cfzhOaexI/99c14dyiuk1+6QhdwKaCRTc1mdfNQmnfWNRbfWhWBlK3h4GGE9JK33 | ||||||
|  | Gk8ZS8DMrkdAh0xby4xAQ/mSWAfWrBmfzlOqGyoB1U47WTOeqNbWkkoAP2ys94+s | ||||||
|  | Jg4NTkiDVtXRF6nr6fYi0bSOvOFg0IQrMXO2Y8gyg9ARdPJwKtvWX8VPADCYMiWH | ||||||
|  | h4n8bZokIrImVKLDQKHY4jCsND2HHdJfnrdL2YJw1qFskNO4cSNmZydw0Wkgjv9k | ||||||
|  | F+KxqrDKlB8MZu2Hclph6v/CZ0fQ9YuE8/lsHZ0Qc2HyiSMnvjgK5fDc3TD4fa8F | ||||||
|  | E8gMNurM+kV8PT8LNIM+4Zs+LKEV8nqRWBaxkIVJGekkVKO8xDBOG/aN62AZKHOe | ||||||
|  | GcyIdu7yNMMRihGVZCYr8rYiJoKiOzDqOkPkLOPdhtVlgnhowzHDxMHND/E2WA5p | ||||||
|  | ZHuNM/m0TXt2wTTPL7JH2YC0gPz/BvvSzjksgzU5rLbRyUKQkgU= | ||||||
|  | -----END CERTIFICATE----- | ||||||
| @@ -433,8 +433,8 @@ public sealed class CertCcConnector : IFeedConnector | |||||||
|                 } |                 } | ||||||
|  |  | ||||||
|                 var advisory = CertCcMapper.Map(dto, document, dtoRecord, SourceName); |                 var advisory = CertCcMapper.Map(dto, document, dtoRecord, SourceName); | ||||||
|                 var affectedCount = advisory.AffectedPackages.Count; |                 var affectedCount = advisory.AffectedPackages.Length; | ||||||
|                 var normalizedRuleCount = advisory.AffectedPackages.Sum(static package => package.NormalizedVersions.Count); |                 var normalizedRuleCount = advisory.AffectedPackages.Sum(static package => package.NormalizedVersions.Length); | ||||||
|                 await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false); |                 await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false); | ||||||
|                 await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false); |                 await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false); | ||||||
|                 _diagnostics.MapSuccess(affectedCount, normalizedRuleCount); |                 _diagnostics.MapSuccess(affectedCount, normalizedRuleCount); | ||||||
|   | |||||||
| @@ -0,0 +1,65 @@ | |||||||
|  | using System.Collections.Immutable; | ||||||
|  | using MongoDB.Bson; | ||||||
|  | using StellaOps.Feedser.Source.Common; | ||||||
|  | using StellaOps.Feedser.Models; | ||||||
|  | using StellaOps.Feedser.Source.Ru.Bdu.Internal; | ||||||
|  | using StellaOps.Feedser.Storage.Mongo.Documents; | ||||||
|  | using Xunit; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Bdu.Tests; | ||||||
|  |  | ||||||
|  | public sealed class RuBduMapperTests | ||||||
|  | { | ||||||
|  |     [Fact] | ||||||
|  |     public void Map_ConstructsCanonicalAdvisory() | ||||||
|  |     { | ||||||
|  |         var dto = new RuBduVulnerabilityDto( | ||||||
|  |             Identifier: "BDU:2025-12345", | ||||||
|  |             Name: "Уязвимость тестового продукта", | ||||||
|  |             Description: "Описание", | ||||||
|  |             Solution: "Обновить", | ||||||
|  |             IdentifyDate: new DateTimeOffset(2025, 10, 10, 0, 0, 0, TimeSpan.Zero), | ||||||
|  |             SeverityText: "Высокий", | ||||||
|  |             CvssVector: "AV:N/AC:L/Au:N/C:P/I:P/A:P", | ||||||
|  |             CvssScore: 7.5, | ||||||
|  |             Cvss3Vector: null, | ||||||
|  |             Cvss3Score: null, | ||||||
|  |             ExploitStatus: "Существует", | ||||||
|  |             IncidentCount: 1, | ||||||
|  |             FixStatus: "Устранена", | ||||||
|  |             VulStatus: "Подтверждена", | ||||||
|  |             VulClass: null, | ||||||
|  |             VulState: null, | ||||||
|  |             Other: null, | ||||||
|  |             Software: new[] | ||||||
|  |             { | ||||||
|  |                 new RuBduSoftwareDto("ООО Вендор", "Продукт", "1.2.3", "Windows", ImmutableArray<string>.Empty) | ||||||
|  |             }.ToImmutableArray(), | ||||||
|  |             Environment: ImmutableArray<RuBduEnvironmentDto>.Empty, | ||||||
|  |             Cwes: new[] { new RuBduCweDto("CWE-79", "XSS") }.ToImmutableArray()); | ||||||
|  |  | ||||||
|  |         var document = new DocumentRecord( | ||||||
|  |             Guid.NewGuid(), | ||||||
|  |             RuBduConnectorPlugin.SourceName, | ||||||
|  |             "https://bdu.fstec.ru/vul/2025-12345", | ||||||
|  |             DateTimeOffset.UtcNow, | ||||||
|  |             "abc", | ||||||
|  |             DocumentStatuses.PendingMap, | ||||||
|  |             "application/json", | ||||||
|  |             null, | ||||||
|  |             null, | ||||||
|  |             null, | ||||||
|  |             dto.IdentifyDate, | ||||||
|  |             ObjectId.GenerateNewId()); | ||||||
|  |  | ||||||
|  |         var advisory = RuBduMapper.Map(dto, document, dto.IdentifyDate!.Value); | ||||||
|  |  | ||||||
|  |         Assert.Equal("BDU:2025-12345", advisory.AdvisoryKey); | ||||||
|  |         Assert.Contains("BDU:2025-12345", advisory.Aliases); | ||||||
|  |         Assert.Equal("high", advisory.Severity); | ||||||
|  |         Assert.True(advisory.ExploitKnown); | ||||||
|  |         Assert.Single(advisory.AffectedPackages); | ||||||
|  |         Assert.Single(advisory.CvssMetrics); | ||||||
|  |         Assert.Contains(advisory.References, reference => reference.Url.Contains("bdu.fstec.ru", StringComparison.OrdinalIgnoreCase)); | ||||||
|  |     } | ||||||
|  | } | ||||||
| @@ -0,0 +1,58 @@ | |||||||
|  | using System.Xml.Linq; | ||||||
|  | using StellaOps.Feedser.Source.Ru.Bdu.Internal; | ||||||
|  | using Xunit; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Bdu.Tests; | ||||||
|  |  | ||||||
|  | public sealed class RuBduXmlParserTests | ||||||
|  | { | ||||||
|  |     [Fact] | ||||||
|  |     public void TryParse_ValidElement_ReturnsDto() | ||||||
|  |     { | ||||||
|  |         const string xml = """ | ||||||
|  | <vul> | ||||||
|  |   <identifier>BDU:2025-12345</identifier> | ||||||
|  |   <name>Уязвимость тестового продукта</name> | ||||||
|  |   <description>Описание уязвимости</description> | ||||||
|  |   <solution>Обновить продукт</solution> | ||||||
|  |   <identify_date>2025-10-10</identify_date> | ||||||
|  |   <severity>Высокий уровень опасности</severity> | ||||||
|  |   <exploit_status>Существует эксплойт</exploit_status> | ||||||
|  |   <fix_status>Устранена</fix_status> | ||||||
|  |   <vul_status>Подтверждена производителем</vul_status> | ||||||
|  |   <vul_incident>1</vul_incident> | ||||||
|  |   <cvss> | ||||||
|  |     <vector score=\"7.5\">AV:N/AC:L/Au:N/C:P/I:P/A:P</vector> | ||||||
|  |   </cvss> | ||||||
|  |   <vulnerable_software> | ||||||
|  |     <soft> | ||||||
|  |       <vendor>ООО «Вендор»</vendor> | ||||||
|  |       <name>Продукт</name> | ||||||
|  |       <version>1.2.3</version> | ||||||
|  |       <platform>Windows</platform> | ||||||
|  |       <types> | ||||||
|  |         <type>ics</type> | ||||||
|  |       </types> | ||||||
|  |     </soft> | ||||||
|  |   </vulnerable_software> | ||||||
|  |   <cwes> | ||||||
|  |     <cwe> | ||||||
|  |       <identifier>CWE-79</identifier> | ||||||
|  |       <name>XSS</name> | ||||||
|  |     </cwe> | ||||||
|  |   </cwes> | ||||||
|  | </vul> | ||||||
|  | """; | ||||||
|  |  | ||||||
|  |         var element = XElement.Parse(xml); | ||||||
|  |         var dto = RuBduXmlParser.TryParse(element); | ||||||
|  |  | ||||||
|  |         Assert.NotNull(dto); | ||||||
|  |         Assert.Equal("BDU:2025-12345", dto!.Identifier); | ||||||
|  |         Assert.Equal("Уязвимость тестового продукта", dto.Name); | ||||||
|  |         Assert.Equal("AV:N/AC:L/Au:N/C:P/I:P/A:P", dto.CvssVector); | ||||||
|  |         Assert.Equal(7.5, dto.CvssScore); | ||||||
|  |         Assert.Single(dto.Software); | ||||||
|  |         Assert.Single(dto.Cwes); | ||||||
|  |     } | ||||||
|  | } | ||||||
| @@ -0,0 +1,13 @@ | |||||||
|  | <Project Sdk="Microsoft.NET.Sdk"> | ||||||
|  |   <PropertyGroup> | ||||||
|  |     <TargetFramework>net10.0</TargetFramework> | ||||||
|  |     <ImplicitUsings>enable</ImplicitUsings> | ||||||
|  |     <Nullable>enable</Nullable> | ||||||
|  |   </PropertyGroup> | ||||||
|  |  | ||||||
|  |   <ItemGroup> | ||||||
|  |     <ProjectReference Include="../StellaOps.Feedser.Models/StellaOps.Feedser.Models.csproj" /> | ||||||
|  |     <ProjectReference Include="../StellaOps.Feedser.Normalization/StellaOps.Feedser.Normalization.csproj" /> | ||||||
|  |     <ProjectReference Include="../StellaOps.Feedser.Source.Ru.Bdu/StellaOps.Feedser.Source.Ru.Bdu.csproj" /> | ||||||
|  |   </ItemGroup> | ||||||
|  | </Project> | ||||||
| @@ -1,29 +0,0 @@ | |||||||
| using System; |  | ||||||
| using System.Threading; |  | ||||||
| using System.Threading.Tasks; |  | ||||||
| using StellaOps.Plugin; |  | ||||||
|  |  | ||||||
| namespace StellaOps.Feedser.Source.Ru.Bdu; |  | ||||||
|  |  | ||||||
| public sealed class RuBduConnectorPlugin : IConnectorPlugin |  | ||||||
| { |  | ||||||
|     public string Name => "ru-bdu"; |  | ||||||
|  |  | ||||||
|     public bool IsAvailable(IServiceProvider services) => true; |  | ||||||
|  |  | ||||||
|     public IFeedConnector Create(IServiceProvider services) => new StubConnector(Name); |  | ||||||
|  |  | ||||||
|     private sealed class StubConnector : IFeedConnector |  | ||||||
|     { |  | ||||||
|         public StubConnector(string sourceName) => SourceName = sourceName; |  | ||||||
|  |  | ||||||
|         public string SourceName { get; } |  | ||||||
|  |  | ||||||
|         public Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken) => Task.CompletedTask; |  | ||||||
|  |  | ||||||
|         public Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken) => Task.CompletedTask; |  | ||||||
|  |  | ||||||
|         public Task MapAsync(IServiceProvider services, CancellationToken cancellationToken) => Task.CompletedTask; |  | ||||||
|     } |  | ||||||
| } |  | ||||||
|  |  | ||||||
| @@ -0,0 +1,102 @@ | |||||||
|  | using System.Net; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Bdu.Configuration; | ||||||
|  |  | ||||||
|  | /// <summary> | ||||||
|  | /// Connector options for the Russian BDU archive ingestion pipeline. | ||||||
|  | /// </summary> | ||||||
|  | public sealed class RuBduOptions | ||||||
|  | { | ||||||
|  |     public const string HttpClientName = "ru-bdu"; | ||||||
|  |  | ||||||
|  |     private static readonly TimeSpan DefaultRequestTimeout = TimeSpan.FromMinutes(2); | ||||||
|  |     private static readonly TimeSpan DefaultFailureBackoff = TimeSpan.FromMinutes(30); | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Base endpoint used for resolving relative resource paths. | ||||||
|  |     /// </summary> | ||||||
|  |     public Uri BaseAddress { get; set; } = new("https://bdu.fstec.ru/", UriKind.Absolute); | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Relative path to the zipped vulnerability dataset. | ||||||
|  |     /// </summary> | ||||||
|  |     public string DataArchivePath { get; set; } = "files/documents/vulxml.zip"; | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// HTTP timeout applied when downloading the archive. | ||||||
|  |     /// </summary> | ||||||
|  |     public TimeSpan RequestTimeout { get; set; } = DefaultRequestTimeout; | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Backoff applied when the remote endpoint fails to serve the archive. | ||||||
|  |     /// </summary> | ||||||
|  |     public TimeSpan FailureBackoff { get; set; } = DefaultFailureBackoff; | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// User-Agent header used for outbound requests. | ||||||
|  |     /// </summary> | ||||||
|  |     public string UserAgent { get; set; } = "StellaOps/Feedser (+https://stella-ops.org)"; | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Accept-Language preference sent with outbound requests. | ||||||
|  |     /// </summary> | ||||||
|  |     public string AcceptLanguage { get; set; } = "ru-RU,ru;q=0.9,en-US;q=0.6,en;q=0.4"; | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Maximum number of vulnerabilities ingested per fetch cycle. | ||||||
|  |     /// </summary> | ||||||
|  |     public int MaxVulnerabilitiesPerFetch { get; set; } = 500; | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Returns the absolute URI for the archive download. | ||||||
|  |     /// </summary> | ||||||
|  |     public Uri DataArchiveUri => new(BaseAddress, DataArchivePath); | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Optional directory for caching the most recent archive (relative paths resolve under the content root). | ||||||
|  |     /// </summary> | ||||||
|  |     public string? CacheDirectory { get; set; } = null; | ||||||
|  |  | ||||||
|  |     public void Validate() | ||||||
|  |     { | ||||||
|  |         if (BaseAddress is null || !BaseAddress.IsAbsoluteUri) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuBdu BaseAddress must be an absolute URI."); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (string.IsNullOrWhiteSpace(DataArchivePath)) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuBdu DataArchivePath must be provided."); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (RequestTimeout <= TimeSpan.Zero) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuBdu RequestTimeout must be positive."); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (FailureBackoff < TimeSpan.Zero) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuBdu FailureBackoff cannot be negative."); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (string.IsNullOrWhiteSpace(UserAgent)) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuBdu UserAgent cannot be empty."); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (string.IsNullOrWhiteSpace(AcceptLanguage)) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuBdu AcceptLanguage cannot be empty."); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (MaxVulnerabilitiesPerFetch <= 0) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuBdu MaxVulnerabilitiesPerFetch must be greater than zero."); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (CacheDirectory is not null && CacheDirectory.Trim().Length == 0) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuBdu CacheDirectory cannot be whitespace."); | ||||||
|  |         } | ||||||
|  |     } | ||||||
|  | } | ||||||
							
								
								
									
										81
									
								
								src/StellaOps.Feedser.Source.Ru.Bdu/Internal/RuBduCursor.cs
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										81
									
								
								src/StellaOps.Feedser.Source.Ru.Bdu/Internal/RuBduCursor.cs
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,81 @@ | |||||||
|  | using MongoDB.Bson; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Bdu.Internal; | ||||||
|  |  | ||||||
|  | internal sealed record RuBduCursor( | ||||||
|  |     IReadOnlyCollection<Guid> PendingDocuments, | ||||||
|  |     IReadOnlyCollection<Guid> PendingMappings, | ||||||
|  |     DateTimeOffset? LastSuccessfulFetch) | ||||||
|  | { | ||||||
|  |     private static readonly IReadOnlyCollection<Guid> EmptyGuids = Array.Empty<Guid>(); | ||||||
|  |  | ||||||
|  |     public static RuBduCursor Empty { get; } = new(EmptyGuids, EmptyGuids, null); | ||||||
|  |  | ||||||
|  |     public RuBduCursor WithPendingDocuments(IEnumerable<Guid> documents) | ||||||
|  |         => this with { PendingDocuments = (documents ?? Enumerable.Empty<Guid>()).Distinct().ToArray() }; | ||||||
|  |  | ||||||
|  |     public RuBduCursor WithPendingMappings(IEnumerable<Guid> mappings) | ||||||
|  |         => this with { PendingMappings = (mappings ?? Enumerable.Empty<Guid>()).Distinct().ToArray() }; | ||||||
|  |  | ||||||
|  |     public RuBduCursor WithLastSuccessfulFetch(DateTimeOffset? timestamp) | ||||||
|  |         => this with { LastSuccessfulFetch = timestamp }; | ||||||
|  |  | ||||||
|  |     public BsonDocument ToBsonDocument() | ||||||
|  |     { | ||||||
|  |         var document = new BsonDocument | ||||||
|  |         { | ||||||
|  |             ["pendingDocuments"] = new BsonArray(PendingDocuments.Select(id => id.ToString())), | ||||||
|  |             ["pendingMappings"] = new BsonArray(PendingMappings.Select(id => id.ToString())), | ||||||
|  |         }; | ||||||
|  |  | ||||||
|  |         if (LastSuccessfulFetch.HasValue) | ||||||
|  |         { | ||||||
|  |             document["lastSuccessfulFetch"] = LastSuccessfulFetch.Value.UtcDateTime; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return document; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     public static RuBduCursor FromBson(BsonDocument? document) | ||||||
|  |     { | ||||||
|  |         if (document is null || document.ElementCount == 0) | ||||||
|  |         { | ||||||
|  |             return Empty; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var pendingDocuments = ReadGuidArray(document, "pendingDocuments"); | ||||||
|  |         var pendingMappings = ReadGuidArray(document, "pendingMappings"); | ||||||
|  |         var lastFetch = document.TryGetValue("lastSuccessfulFetch", out var fetchValue) | ||||||
|  |             ? ParseDate(fetchValue) | ||||||
|  |             : null; | ||||||
|  |  | ||||||
|  |         return new RuBduCursor(pendingDocuments, pendingMappings, lastFetch); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static IReadOnlyCollection<Guid> ReadGuidArray(BsonDocument document, string field) | ||||||
|  |     { | ||||||
|  |         if (!document.TryGetValue(field, out var value) || value is not BsonArray array) | ||||||
|  |         { | ||||||
|  |             return EmptyGuids; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var result = new List<Guid>(array.Count); | ||||||
|  |         foreach (var element in array) | ||||||
|  |         { | ||||||
|  |             if (Guid.TryParse(element?.ToString(), out var guid)) | ||||||
|  |             { | ||||||
|  |                 result.Add(guid); | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return result; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static DateTimeOffset? ParseDate(BsonValue value) | ||||||
|  |         => value.BsonType switch | ||||||
|  |         { | ||||||
|  |             BsonType.DateTime => DateTime.SpecifyKind(value.ToUniversalTime(), DateTimeKind.Utc), | ||||||
|  |             BsonType.String when DateTimeOffset.TryParse(value.AsString, out var parsed) => parsed.ToUniversalTime(), | ||||||
|  |             _ => null, | ||||||
|  |         }; | ||||||
|  | } | ||||||
							
								
								
									
										249
									
								
								src/StellaOps.Feedser.Source.Ru.Bdu/Internal/RuBduMapper.cs
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										249
									
								
								src/StellaOps.Feedser.Source.Ru.Bdu/Internal/RuBduMapper.cs
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,249 @@ | |||||||
|  | using System.Collections.Generic; | ||||||
|  | using System.Collections.Immutable; | ||||||
|  | using System.Globalization; | ||||||
|  | using System.Linq; | ||||||
|  | using StellaOps.Feedser.Models; | ||||||
|  | using StellaOps.Feedser.Normalization.Cvss; | ||||||
|  | using StellaOps.Feedser.Storage.Mongo.Documents; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Bdu.Internal; | ||||||
|  |  | ||||||
|  | internal static class RuBduMapper | ||||||
|  | { | ||||||
|  |     public static Advisory Map(RuBduVulnerabilityDto dto, DocumentRecord document, DateTimeOffset recordedAt) | ||||||
|  |     { | ||||||
|  |         ArgumentNullException.ThrowIfNull(dto); | ||||||
|  |         ArgumentNullException.ThrowIfNull(document); | ||||||
|  |  | ||||||
|  |         var advisoryProvenance = new AdvisoryProvenance( | ||||||
|  |             RuBduConnectorPlugin.SourceName, | ||||||
|  |             "advisory", | ||||||
|  |             dto.Identifier, | ||||||
|  |             recordedAt, | ||||||
|  |             new[] { ProvenanceFieldMasks.Advisory }); | ||||||
|  |  | ||||||
|  |         var aliases = BuildAliases(dto); | ||||||
|  |         var packages = BuildPackages(dto, recordedAt); | ||||||
|  |         var references = BuildReferences(dto, document, recordedAt); | ||||||
|  |         var cvssMetrics = BuildCvssMetrics(dto, recordedAt, out var severityFromCvss); | ||||||
|  |         var severity = severityFromCvss; | ||||||
|  |         var exploitKnown = DetermineExploitKnown(dto); | ||||||
|  |  | ||||||
|  |         return new Advisory( | ||||||
|  |             advisoryKey: dto.Identifier, | ||||||
|  |             title: dto.Name ?? dto.Identifier, | ||||||
|  |             summary: dto.Description, | ||||||
|  |             language: "ru", | ||||||
|  |             published: dto.IdentifyDate, | ||||||
|  |             modified: dto.IdentifyDate, | ||||||
|  |             severity: severity, | ||||||
|  |             exploitKnown: exploitKnown, | ||||||
|  |             aliases: aliases, | ||||||
|  |             references: references, | ||||||
|  |             affectedPackages: packages, | ||||||
|  |             cvssMetrics: cvssMetrics, | ||||||
|  |             provenance: new[] { advisoryProvenance }); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static IReadOnlyList<string> BuildAliases(RuBduVulnerabilityDto dto) | ||||||
|  |     { | ||||||
|  |         var aliases = new List<string>(capacity: 2) { dto.Identifier }; | ||||||
|  |         return aliases; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static IReadOnlyList<AffectedPackage> BuildPackages(RuBduVulnerabilityDto dto, DateTimeOffset recordedAt) | ||||||
|  |     { | ||||||
|  |         if (dto.Software.IsDefaultOrEmpty) | ||||||
|  |         { | ||||||
|  |             return Array.Empty<AffectedPackage>(); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var packages = new List<AffectedPackage>(dto.Software.Length); | ||||||
|  |         foreach (var software in dto.Software) | ||||||
|  |         { | ||||||
|  |             if (string.IsNullOrWhiteSpace(software.Name) && string.IsNullOrWhiteSpace(software.Vendor)) | ||||||
|  |             { | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             var identifier = string.Join( | ||||||
|  |                 " ", | ||||||
|  |                 new[] { software.Vendor, software.Name } | ||||||
|  |                     .Where(static part => !string.IsNullOrWhiteSpace(part)) | ||||||
|  |                     .Select(static part => part!.Trim())); | ||||||
|  |  | ||||||
|  |             if (string.IsNullOrWhiteSpace(identifier)) | ||||||
|  |             { | ||||||
|  |                 identifier = software.Name ?? software.Vendor ?? dto.Identifier; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             var isIcs = !software.Types.IsDefaultOrEmpty && software.Types.Any(static type => string.Equals(type, "ics", StringComparison.OrdinalIgnoreCase)); | ||||||
|  |  | ||||||
|  |             var packageProvenance = new AdvisoryProvenance( | ||||||
|  |                 RuBduConnectorPlugin.SourceName, | ||||||
|  |                 "package", | ||||||
|  |                 identifier, | ||||||
|  |                 recordedAt, | ||||||
|  |                 new[] { ProvenanceFieldMasks.AffectedPackages }); | ||||||
|  |  | ||||||
|  |             var normalizedStatus = NormalizeStatus(dto.VulStatus); | ||||||
|  |             var statuses = normalizedStatus is null | ||||||
|  |                 ? Array.Empty<AffectedPackageStatus>() | ||||||
|  |                 : new[] | ||||||
|  |                     { | ||||||
|  |                         new AffectedPackageStatus(normalizedStatus, new AdvisoryProvenance( | ||||||
|  |                             RuBduConnectorPlugin.SourceName, | ||||||
|  |                             "package-status", | ||||||
|  |                             dto.VulStatus ?? normalizedStatus, | ||||||
|  |                             recordedAt, | ||||||
|  |                             new[] { ProvenanceFieldMasks.PackageStatuses })) | ||||||
|  |                     }; | ||||||
|  |  | ||||||
|  |             var ranges = Array.Empty<AffectedVersionRange>(); | ||||||
|  |             if (!string.IsNullOrWhiteSpace(software.Version)) | ||||||
|  |             { | ||||||
|  |                 ranges = new[] | ||||||
|  |                 { | ||||||
|  |                     new AffectedVersionRange( | ||||||
|  |                         rangeKind: "string", | ||||||
|  |                         introducedVersion: null, | ||||||
|  |                         fixedVersion: null, | ||||||
|  |                         lastAffectedVersion: null, | ||||||
|  |                         rangeExpression: software.Version, | ||||||
|  |                         provenance: new AdvisoryProvenance( | ||||||
|  |                             RuBduConnectorPlugin.SourceName, | ||||||
|  |                             "package-range", | ||||||
|  |                             software.Version, | ||||||
|  |                             recordedAt, | ||||||
|  |                             new[] { ProvenanceFieldMasks.VersionRanges })) | ||||||
|  |                 }; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             packages.Add(new AffectedPackage( | ||||||
|  |                 isIcs ? AffectedPackageTypes.IcsVendor : AffectedPackageTypes.Vendor, | ||||||
|  |                 identifier, | ||||||
|  |                 platform: software.Platform, | ||||||
|  |                 versionRanges: ranges, | ||||||
|  |                 statuses: statuses, | ||||||
|  |                 provenance: new[] { packageProvenance })); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return packages; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static IReadOnlyList<AdvisoryReference> BuildReferences(RuBduVulnerabilityDto dto, DocumentRecord document, DateTimeOffset recordedAt) | ||||||
|  |     { | ||||||
|  |         var references = new List<AdvisoryReference> | ||||||
|  |         { | ||||||
|  |             new(document.Uri, "details", "ru-bdu", summary: null, new AdvisoryProvenance( | ||||||
|  |                 RuBduConnectorPlugin.SourceName, | ||||||
|  |                 "reference", | ||||||
|  |                 document.Uri, | ||||||
|  |                 recordedAt, | ||||||
|  |                 new[] { ProvenanceFieldMasks.References })) | ||||||
|  |         }; | ||||||
|  |  | ||||||
|  |         foreach (var cwe in dto.Cwes) | ||||||
|  |         { | ||||||
|  |             if (string.IsNullOrWhiteSpace(cwe.Identifier)) | ||||||
|  |             { | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             var slug = cwe.Identifier.ToUpperInvariant().Replace("CWE-", string.Empty, StringComparison.OrdinalIgnoreCase); | ||||||
|  |             if (!slug.All(char.IsDigit)) | ||||||
|  |             { | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             var url = $"https://cwe.mitre.org/data/definitions/{slug}.html"; | ||||||
|  |             references.Add(new AdvisoryReference(url, "cwe", "cwe", cwe.Name, new AdvisoryProvenance( | ||||||
|  |                 RuBduConnectorPlugin.SourceName, | ||||||
|  |                 "reference", | ||||||
|  |                 url, | ||||||
|  |                 recordedAt, | ||||||
|  |                 new[] { ProvenanceFieldMasks.References }))); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return references; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static IReadOnlyList<CvssMetric> BuildCvssMetrics(RuBduVulnerabilityDto dto, DateTimeOffset recordedAt, out string? severity) | ||||||
|  |     { | ||||||
|  |         severity = null; | ||||||
|  |         var metrics = new List<CvssMetric>(); | ||||||
|  |  | ||||||
|  |         if (!string.IsNullOrWhiteSpace(dto.CvssVector) && CvssMetricNormalizer.TryNormalize("2.0", dto.CvssVector, dto.CvssScore, null, out var normalized)) | ||||||
|  |         { | ||||||
|  |             var provenance = new AdvisoryProvenance( | ||||||
|  |                 RuBduConnectorPlugin.SourceName, | ||||||
|  |                 "cvss", | ||||||
|  |                 normalized.Vector, | ||||||
|  |                 recordedAt, | ||||||
|  |                 new[] { ProvenanceFieldMasks.CvssMetrics }); | ||||||
|  |             var metric = normalized.ToModel(provenance); | ||||||
|  |             metrics.Add(metric); | ||||||
|  |             severity ??= metric.BaseSeverity; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (!string.IsNullOrWhiteSpace(dto.Cvss3Vector) && CvssMetricNormalizer.TryNormalize("3.1", dto.Cvss3Vector, dto.Cvss3Score, null, out var normalized3)) | ||||||
|  |         { | ||||||
|  |             var provenance = new AdvisoryProvenance( | ||||||
|  |                 RuBduConnectorPlugin.SourceName, | ||||||
|  |                 "cvss", | ||||||
|  |                 normalized3.Vector, | ||||||
|  |                 recordedAt, | ||||||
|  |                 new[] { ProvenanceFieldMasks.CvssMetrics }); | ||||||
|  |             var metric = normalized3.ToModel(provenance); | ||||||
|  |             metrics.Add(metric); | ||||||
|  |             severity ??= metric.BaseSeverity; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (metrics.Count > 1) | ||||||
|  |         { | ||||||
|  |             metrics = metrics | ||||||
|  |                 .OrderByDescending(static metric => metric.BaseScore) | ||||||
|  |                 .ThenBy(static metric => metric.Version, StringComparer.Ordinal) | ||||||
|  |                 .ToList(); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return metrics; | ||||||
|  |     } | ||||||
|  |     private static string NormalizeStatus(string? status) | ||||||
|  |     { | ||||||
|  |         if (string.IsNullOrWhiteSpace(status)) | ||||||
|  |         { | ||||||
|  |             return null; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var normalized = status.Trim().ToLowerInvariant(); | ||||||
|  |         return normalized switch | ||||||
|  |         { | ||||||
|  |             "устранена" or "устранена производителем" or "устранена разработчиком" => AffectedPackageStatusCatalog.Fixed, | ||||||
|  |             "устраняется" or "устранение планируется" or "разрабатывается" => AffectedPackageStatusCatalog.Pending, | ||||||
|  |             "не устранена" => AffectedPackageStatusCatalog.Pending, | ||||||
|  |             "актуальна" or "подтверждена" or "подтверждена производителем" or "подтверждена исследователями" => AffectedPackageStatusCatalog.Affected, | ||||||
|  |             _ => null, | ||||||
|  |         }; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |  | ||||||
|  |     private static bool DetermineExploitKnown(RuBduVulnerabilityDto dto) | ||||||
|  |     { | ||||||
|  |         if (dto.IncidentCount.HasValue && dto.IncidentCount.Value > 0) | ||||||
|  |         { | ||||||
|  |             return true; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (!string.IsNullOrWhiteSpace(dto.ExploitStatus)) | ||||||
|  |         { | ||||||
|  |             var status = dto.ExploitStatus.Trim().ToLowerInvariant(); | ||||||
|  |             if (status.Contains("существ", StringComparison.Ordinal) || status.Contains("использ", StringComparison.Ordinal)) | ||||||
|  |             { | ||||||
|  |                 return true; | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return false; | ||||||
|  |     } | ||||||
|  | } | ||||||
| @@ -0,0 +1,45 @@ | |||||||
|  | using System.Collections.Immutable; | ||||||
|  | using System.Text.Json.Serialization; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Bdu.Internal; | ||||||
|  |  | ||||||
|  | internal sealed record RuBduVulnerabilityDto( | ||||||
|  |     string Identifier, | ||||||
|  |     string? Name, | ||||||
|  |     string? Description, | ||||||
|  |     string? Solution, | ||||||
|  |     DateTimeOffset? IdentifyDate, | ||||||
|  |     string? SeverityText, | ||||||
|  |     string? CvssVector, | ||||||
|  |     double? CvssScore, | ||||||
|  |     string? Cvss3Vector, | ||||||
|  |     double? Cvss3Score, | ||||||
|  |     string? ExploitStatus, | ||||||
|  |     int? IncidentCount, | ||||||
|  |     string? FixStatus, | ||||||
|  |     string? VulStatus, | ||||||
|  |     string? VulClass, | ||||||
|  |     string? VulState, | ||||||
|  |     string? Other, | ||||||
|  |     ImmutableArray<RuBduSoftwareDto> Software, | ||||||
|  |     ImmutableArray<RuBduEnvironmentDto> Environment, | ||||||
|  |     ImmutableArray<RuBduCweDto> Cwes) | ||||||
|  | { | ||||||
|  |     [JsonIgnore] | ||||||
|  |     public bool HasCvss => !string.IsNullOrWhiteSpace(CvssVector) || !string.IsNullOrWhiteSpace(Cvss3Vector); | ||||||
|  | } | ||||||
|  |  | ||||||
|  | internal sealed record RuBduSoftwareDto( | ||||||
|  |     string? Vendor, | ||||||
|  |     string? Name, | ||||||
|  |     string? Version, | ||||||
|  |     string? Platform, | ||||||
|  |     ImmutableArray<string> Types); | ||||||
|  |  | ||||||
|  | internal sealed record RuBduEnvironmentDto( | ||||||
|  |     string? Vendor, | ||||||
|  |     string? Name, | ||||||
|  |     string? Version, | ||||||
|  |     string? Platform); | ||||||
|  |  | ||||||
|  | internal sealed record RuBduCweDto(string Identifier, string? Name); | ||||||
							
								
								
									
										196
									
								
								src/StellaOps.Feedser.Source.Ru.Bdu/Internal/RuBduXmlParser.cs
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										196
									
								
								src/StellaOps.Feedser.Source.Ru.Bdu/Internal/RuBduXmlParser.cs
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,196 @@ | |||||||
|  | using System.Collections.Immutable; | ||||||
|  | using System.Linq; | ||||||
|  | using System.Globalization; | ||||||
|  | using System.Xml.Linq; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Bdu.Internal; | ||||||
|  |  | ||||||
|  | internal static class RuBduXmlParser | ||||||
|  | { | ||||||
|  |     public static RuBduVulnerabilityDto? TryParse(XElement element) | ||||||
|  |     { | ||||||
|  |         ArgumentNullException.ThrowIfNull(element); | ||||||
|  |  | ||||||
|  |         var identifier = element.Element("identifier")?.Value?.Trim(); | ||||||
|  |         if (string.IsNullOrWhiteSpace(identifier)) | ||||||
|  |         { | ||||||
|  |             return null; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var name = Normalize(element.Element("name")?.Value); | ||||||
|  |         var description = Normalize(element.Element("description")?.Value); | ||||||
|  |         var solution = Normalize(element.Element("solution")?.Value); | ||||||
|  |         var severity = Normalize(element.Element("severity")?.Value); | ||||||
|  |         var exploitStatus = Normalize(element.Element("exploit_status")?.Value); | ||||||
|  |         var fixStatus = Normalize(element.Element("fix_status")?.Value); | ||||||
|  |         var vulStatus = Normalize(element.Element("vul_status")?.Value); | ||||||
|  |         var vulClass = Normalize(element.Element("vul_class")?.Value); | ||||||
|  |         var vulState = Normalize(element.Element("vul_state")?.Value); | ||||||
|  |         var other = Normalize(element.Element("other")?.Value); | ||||||
|  |         var incidentCount = ParseInt(element.Element("vul_incident")?.Value); | ||||||
|  |  | ||||||
|  |         var identifyDate = ParseDate(element.Element("identify_date")?.Value); | ||||||
|  |  | ||||||
|  |         var cvssVectorElement = element.Element("cvss")?.Element("vector"); | ||||||
|  |         var cvssVector = Normalize(cvssVectorElement?.Value); | ||||||
|  |         var cvssScore = ParseDouble(cvssVectorElement?.Attribute("score")?.Value); | ||||||
|  |  | ||||||
|  |         var cvss3VectorElement = element.Element("cvss3")?.Element("vector"); | ||||||
|  |         var cvss3Vector = Normalize(cvss3VectorElement?.Value); | ||||||
|  |         var cvss3Score = ParseDouble(cvss3VectorElement?.Attribute("score")?.Value); | ||||||
|  |  | ||||||
|  |         var software = ParseSoftware(element.Element("vulnerable_software")); | ||||||
|  |         var environment = ParseEnvironment(element.Element("environment")); | ||||||
|  |         var cwes = ParseCwes(element.Element("cwes")); | ||||||
|  |  | ||||||
|  |         return new RuBduVulnerabilityDto( | ||||||
|  |             identifier.Trim(), | ||||||
|  |             name, | ||||||
|  |             description, | ||||||
|  |             solution, | ||||||
|  |             identifyDate, | ||||||
|  |             severity, | ||||||
|  |             cvssVector, | ||||||
|  |             cvssScore, | ||||||
|  |             cvss3Vector, | ||||||
|  |             cvss3Score, | ||||||
|  |             exploitStatus, | ||||||
|  |             incidentCount, | ||||||
|  |             fixStatus, | ||||||
|  |             vulStatus, | ||||||
|  |             vulClass, | ||||||
|  |             vulState, | ||||||
|  |             other, | ||||||
|  |             software, | ||||||
|  |             environment, | ||||||
|  |             cwes); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static ImmutableArray<RuBduSoftwareDto> ParseSoftware(XElement? root) | ||||||
|  |     { | ||||||
|  |         if (root is null) | ||||||
|  |         { | ||||||
|  |             return ImmutableArray<RuBduSoftwareDto>.Empty; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var builder = ImmutableArray.CreateBuilder<RuBduSoftwareDto>(); | ||||||
|  |         foreach (var soft in root.Elements("soft")) | ||||||
|  |         { | ||||||
|  |             var vendor = Normalize(soft.Element("vendor")?.Value); | ||||||
|  |             var name = Normalize(soft.Element("name")?.Value); | ||||||
|  |             var version = Normalize(soft.Element("version")?.Value); | ||||||
|  |             var platform = Normalize(soft.Element("platform")?.Value); | ||||||
|  |             var types = soft.Element("types") is { } typesElement | ||||||
|  |                 ? typesElement.Elements("type").Select(static x => Normalize(x.Value)).Where(static value => !string.IsNullOrWhiteSpace(value)).Cast<string>().ToImmutableArray() | ||||||
|  |                 : ImmutableArray<string>.Empty; | ||||||
|  |  | ||||||
|  |             builder.Add(new RuBduSoftwareDto(vendor, name, version, platform, types)); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return builder.ToImmutable(); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static ImmutableArray<RuBduEnvironmentDto> ParseEnvironment(XElement? root) | ||||||
|  |     { | ||||||
|  |         if (root is null) | ||||||
|  |         { | ||||||
|  |             return ImmutableArray<RuBduEnvironmentDto>.Empty; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var builder = ImmutableArray.CreateBuilder<RuBduEnvironmentDto>(); | ||||||
|  |         foreach (var os in root.Elements()) | ||||||
|  |         { | ||||||
|  |             var vendor = Normalize(os.Element("vendor")?.Value); | ||||||
|  |             var name = Normalize(os.Element("name")?.Value); | ||||||
|  |             var version = Normalize(os.Element("version")?.Value); | ||||||
|  |             var platform = Normalize(os.Element("platform")?.Value); | ||||||
|  |             builder.Add(new RuBduEnvironmentDto(vendor, name, version, platform)); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return builder.ToImmutable(); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static ImmutableArray<RuBduCweDto> ParseCwes(XElement? root) | ||||||
|  |     { | ||||||
|  |         if (root is null) | ||||||
|  |         { | ||||||
|  |             return ImmutableArray<RuBduCweDto>.Empty; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var builder = ImmutableArray.CreateBuilder<RuBduCweDto>(); | ||||||
|  |         foreach (var cwe in root.Elements("cwe")) | ||||||
|  |         { | ||||||
|  |             var identifier = Normalize(cwe.Element("identifier")?.Value); | ||||||
|  |             if (string.IsNullOrWhiteSpace(identifier)) | ||||||
|  |             { | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             var name = Normalize(cwe.Element("name")?.Value); | ||||||
|  |             builder.Add(new RuBduCweDto(identifier, name)); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return builder.ToImmutable(); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static DateTimeOffset? ParseDate(string? value) | ||||||
|  |     { | ||||||
|  |         if (string.IsNullOrWhiteSpace(value)) | ||||||
|  |         { | ||||||
|  |             return null; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var trimmed = value.Trim(); | ||||||
|  |         if (DateTimeOffset.TryParse(trimmed, CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, out var isoDate)) | ||||||
|  |         { | ||||||
|  |             return isoDate; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (DateTimeOffset.TryParseExact(trimmed, new[] { "dd.MM.yyyy", "dd.MM.yyyy HH:mm:ss" }, CultureInfo.GetCultureInfo("ru-RU"), DateTimeStyles.AssumeUniversal, out var ruDate)) | ||||||
|  |         { | ||||||
|  |             return ruDate; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return null; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static double? ParseDouble(string? value) | ||||||
|  |     { | ||||||
|  |         if (string.IsNullOrWhiteSpace(value)) | ||||||
|  |         { | ||||||
|  |             return null; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (double.TryParse(value.Trim(), NumberStyles.Any, CultureInfo.InvariantCulture, out var parsed)) | ||||||
|  |         { | ||||||
|  |             return parsed; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return null; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static int? ParseInt(string? value) | ||||||
|  |     { | ||||||
|  |         if (string.IsNullOrWhiteSpace(value)) | ||||||
|  |         { | ||||||
|  |             return null; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (int.TryParse(value.Trim(), NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsed)) | ||||||
|  |         { | ||||||
|  |             return parsed; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return null; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static string? Normalize(string? value) | ||||||
|  |     { | ||||||
|  |         if (string.IsNullOrWhiteSpace(value)) | ||||||
|  |         { | ||||||
|  |             return null; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return value.Replace('\r', ' ').Replace('\n', ' ').Trim(); | ||||||
|  |     } | ||||||
|  | } | ||||||
							
								
								
									
										43
									
								
								src/StellaOps.Feedser.Source.Ru.Bdu/Jobs.cs
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										43
									
								
								src/StellaOps.Feedser.Source.Ru.Bdu/Jobs.cs
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,43 @@ | |||||||
|  | using StellaOps.Feedser.Core.Jobs; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Bdu; | ||||||
|  |  | ||||||
|  | internal static class RuBduJobKinds | ||||||
|  | { | ||||||
|  |     public const string Fetch = "source:ru-bdu:fetch"; | ||||||
|  |     public const string Parse = "source:ru-bdu:parse"; | ||||||
|  |     public const string Map = "source:ru-bdu:map"; | ||||||
|  | } | ||||||
|  |  | ||||||
|  | internal sealed class RuBduFetchJob : IJob | ||||||
|  | { | ||||||
|  |     private readonly RuBduConnector _connector; | ||||||
|  |  | ||||||
|  |     public RuBduFetchJob(RuBduConnector connector) | ||||||
|  |         => _connector = connector ?? throw new ArgumentNullException(nameof(connector)); | ||||||
|  |  | ||||||
|  |     public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken) | ||||||
|  |         => _connector.FetchAsync(context.Services, cancellationToken); | ||||||
|  | } | ||||||
|  |  | ||||||
|  | internal sealed class RuBduParseJob : IJob | ||||||
|  | { | ||||||
|  |     private readonly RuBduConnector _connector; | ||||||
|  |  | ||||||
|  |     public RuBduParseJob(RuBduConnector connector) | ||||||
|  |         => _connector = connector ?? throw new ArgumentNullException(nameof(connector)); | ||||||
|  |  | ||||||
|  |     public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken) | ||||||
|  |         => _connector.ParseAsync(context.Services, cancellationToken); | ||||||
|  | } | ||||||
|  |  | ||||||
|  | internal sealed class RuBduMapJob : IJob | ||||||
|  | { | ||||||
|  |     private readonly RuBduConnector _connector; | ||||||
|  |  | ||||||
|  |     public RuBduMapJob(RuBduConnector connector) | ||||||
|  |         => _connector = connector ?? throw new ArgumentNullException(nameof(connector)); | ||||||
|  |  | ||||||
|  |     public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken) | ||||||
|  |         => _connector.MapAsync(context.Services, cancellationToken); | ||||||
|  | } | ||||||
| @@ -0,0 +1,3 @@ | |||||||
|  | using System.Runtime.CompilerServices; | ||||||
|  |  | ||||||
|  | [assembly: InternalsVisibleTo("StellaOps.Feedser.Source.Ru.Bdu.Tests")] | ||||||
							
								
								
									
										493
									
								
								src/StellaOps.Feedser.Source.Ru.Bdu/RuBduConnector.cs
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										493
									
								
								src/StellaOps.Feedser.Source.Ru.Bdu/RuBduConnector.cs
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,493 @@ | |||||||
|  | using System.Collections.Immutable; | ||||||
|  | using System.Globalization; | ||||||
|  | using System.IO; | ||||||
|  | using System.IO.Compression; | ||||||
|  | using System.Security.Cryptography; | ||||||
|  | using System.Linq; | ||||||
|  | using System.Text.Json; | ||||||
|  | using System.Text.Json.Serialization; | ||||||
|  | using System.Xml; | ||||||
|  | using System.Xml.Linq; | ||||||
|  | using Microsoft.Extensions.Logging; | ||||||
|  | using Microsoft.Extensions.Options; | ||||||
|  | using MongoDB.Bson; | ||||||
|  | using StellaOps.Feedser.Normalization.Cvss; | ||||||
|  | using StellaOps.Feedser.Source.Common; | ||||||
|  | using StellaOps.Feedser.Source.Common.Fetch; | ||||||
|  | using StellaOps.Feedser.Source.Ru.Bdu.Configuration; | ||||||
|  | using StellaOps.Feedser.Source.Ru.Bdu.Internal; | ||||||
|  | using StellaOps.Feedser.Storage.Mongo; | ||||||
|  | using StellaOps.Feedser.Storage.Mongo.Advisories; | ||||||
|  | using StellaOps.Feedser.Storage.Mongo.Documents; | ||||||
|  | using StellaOps.Feedser.Storage.Mongo.Dtos; | ||||||
|  | using StellaOps.Plugin; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Bdu; | ||||||
|  |  | ||||||
|  | public sealed class RuBduConnector : IFeedConnector | ||||||
|  | { | ||||||
|  |     private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web) | ||||||
|  |     { | ||||||
|  |         DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull, | ||||||
|  |         PropertyNamingPolicy = JsonNamingPolicy.CamelCase, | ||||||
|  |         WriteIndented = false, | ||||||
|  |     }; | ||||||
|  |  | ||||||
|  |     private readonly SourceFetchService _fetchService; | ||||||
|  |     private readonly RawDocumentStorage _rawDocumentStorage; | ||||||
|  |     private readonly IDocumentStore _documentStore; | ||||||
|  |     private readonly IDtoStore _dtoStore; | ||||||
|  |     private readonly IAdvisoryStore _advisoryStore; | ||||||
|  |     private readonly ISourceStateRepository _stateRepository; | ||||||
|  |     private readonly RuBduOptions _options; | ||||||
|  |     private readonly TimeProvider _timeProvider; | ||||||
|  |     private readonly ILogger<RuBduConnector> _logger; | ||||||
|  |  | ||||||
|  |     private readonly string _cacheDirectory; | ||||||
|  |     private readonly string _archiveCachePath; | ||||||
|  |  | ||||||
|  |     public RuBduConnector( | ||||||
|  |         SourceFetchService fetchService, | ||||||
|  |         RawDocumentStorage rawDocumentStorage, | ||||||
|  |         IDocumentStore documentStore, | ||||||
|  |         IDtoStore dtoStore, | ||||||
|  |         IAdvisoryStore advisoryStore, | ||||||
|  |         ISourceStateRepository stateRepository, | ||||||
|  |         IOptions<RuBduOptions> options, | ||||||
|  |         TimeProvider? timeProvider, | ||||||
|  |         ILogger<RuBduConnector> logger) | ||||||
|  |     { | ||||||
|  |         _fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService)); | ||||||
|  |         _rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage)); | ||||||
|  |         _documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore)); | ||||||
|  |         _dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore)); | ||||||
|  |         _advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore)); | ||||||
|  |         _stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository)); | ||||||
|  |         _options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options)); | ||||||
|  |         _options.Validate(); | ||||||
|  |         _timeProvider = timeProvider ?? TimeProvider.System; | ||||||
|  |         _logger = logger ?? throw new ArgumentNullException(nameof(logger)); | ||||||
|  |         _cacheDirectory = ResolveCacheDirectory(_options.CacheDirectory); | ||||||
|  |         _archiveCachePath = Path.Combine(_cacheDirectory, "vulxml.zip"); | ||||||
|  |         EnsureCacheDirectory(); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     public string SourceName => RuBduConnectorPlugin.SourceName; | ||||||
|  |  | ||||||
|  |     public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken) | ||||||
|  |     { | ||||||
|  |         ArgumentNullException.ThrowIfNull(services); | ||||||
|  |  | ||||||
|  |         var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false); | ||||||
|  |         var pendingDocuments = cursor.PendingDocuments.ToHashSet(); | ||||||
|  |         var pendingMappings = cursor.PendingMappings.ToHashSet(); | ||||||
|  |         var now = _timeProvider.GetUtcNow(); | ||||||
|  |  | ||||||
|  |         SourceFetchContentResult archiveResult = default; | ||||||
|  |         byte[]? archiveContent = null; | ||||||
|  |         var usedCache = false; | ||||||
|  |  | ||||||
|  |         try | ||||||
|  |         { | ||||||
|  |             var request = new SourceFetchRequest(RuBduOptions.HttpClientName, SourceName, _options.DataArchiveUri) | ||||||
|  |             { | ||||||
|  |                 AcceptHeaders = new[] | ||||||
|  |                 { | ||||||
|  |                     "application/zip", | ||||||
|  |                     "application/octet-stream", | ||||||
|  |                     "application/x-zip-compressed", | ||||||
|  |                 }, | ||||||
|  |                 TimeoutOverride = _options.RequestTimeout, | ||||||
|  |             }; | ||||||
|  |  | ||||||
|  |             archiveResult = await _fetchService.FetchContentAsync(request, cancellationToken).ConfigureAwait(false); | ||||||
|  |  | ||||||
|  |             if (archiveResult.IsNotModified) | ||||||
|  |             { | ||||||
|  |                 _logger.LogDebug("RU-BDU archive not modified."); | ||||||
|  |                 await UpdateCursorAsync(cursor.WithLastSuccessfulFetch(now), cancellationToken).ConfigureAwait(false); | ||||||
|  |                 return; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             if (archiveResult.IsSuccess && archiveResult.Content is not null) | ||||||
|  |             { | ||||||
|  |                 archiveContent = archiveResult.Content; | ||||||
|  |                 TryWriteCachedArchive(archiveContent); | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |         catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException) | ||||||
|  |         { | ||||||
|  |             if (TryReadCachedArchive(out var cachedFallback)) | ||||||
|  |             { | ||||||
|  |                 _logger.LogWarning(ex, "RU-BDU archive fetch failed; using cached artefact {CachePath}", _archiveCachePath); | ||||||
|  |                 archiveContent = cachedFallback; | ||||||
|  |                 usedCache = true; | ||||||
|  |             } | ||||||
|  |             else | ||||||
|  |             { | ||||||
|  |                 _logger.LogError(ex, "RU-BDU archive fetch failed for {ArchiveUri}", _options.DataArchiveUri); | ||||||
|  |                 await _stateRepository.MarkFailureAsync(SourceName, now, _options.FailureBackoff, ex.Message, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 throw; | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (archiveContent is null) | ||||||
|  |         { | ||||||
|  |             if (TryReadCachedArchive(out var cachedFallback)) | ||||||
|  |             { | ||||||
|  |                 _logger.LogWarning("RU-BDU archive unavailable (status={Status}); using cached artefact {CachePath}", archiveResult.StatusCode, _archiveCachePath); | ||||||
|  |                 archiveContent = cachedFallback; | ||||||
|  |                 usedCache = true; | ||||||
|  |             } | ||||||
|  |             else | ||||||
|  |             { | ||||||
|  |                 _logger.LogWarning("RU-BDU archive fetch returned no content (status={Status})", archiveResult.StatusCode); | ||||||
|  |                 await UpdateCursorAsync(cursor.WithLastSuccessfulFetch(now), cancellationToken).ConfigureAwait(false); | ||||||
|  |                 return; | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var archiveLastModified = archiveResult.LastModified; | ||||||
|  |         int added; | ||||||
|  |         try | ||||||
|  |         { | ||||||
|  |             added = await ProcessArchiveAsync(archiveContent, now, pendingDocuments, pendingMappings, archiveLastModified, cancellationToken).ConfigureAwait(false); | ||||||
|  |         } | ||||||
|  |         catch (Exception ex) | ||||||
|  |         { | ||||||
|  |             if (!usedCache) | ||||||
|  |             { | ||||||
|  |                 _logger.LogError(ex, "RU-BDU archive processing failed"); | ||||||
|  |                 await _stateRepository.MarkFailureAsync(SourceName, now, _options.FailureBackoff, ex.Message, cancellationToken).ConfigureAwait(false); | ||||||
|  |             } | ||||||
|  |             throw; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var updatedCursor = cursor | ||||||
|  |             .WithPendingDocuments(pendingDocuments) | ||||||
|  |             .WithPendingMappings(pendingMappings) | ||||||
|  |             .WithLastSuccessfulFetch(now); | ||||||
|  |  | ||||||
|  |         await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken) | ||||||
|  |     { | ||||||
|  |         ArgumentNullException.ThrowIfNull(services); | ||||||
|  |  | ||||||
|  |         var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false); | ||||||
|  |         if (cursor.PendingDocuments.Count == 0) | ||||||
|  |         { | ||||||
|  |             return; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var pendingDocuments = cursor.PendingDocuments.ToList(); | ||||||
|  |         var pendingMappings = cursor.PendingMappings.ToList(); | ||||||
|  |  | ||||||
|  |         foreach (var documentId in cursor.PendingDocuments) | ||||||
|  |         { | ||||||
|  |             cancellationToken.ThrowIfCancellationRequested(); | ||||||
|  |  | ||||||
|  |             var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false); | ||||||
|  |             if (document is null) | ||||||
|  |             { | ||||||
|  |                 pendingDocuments.Remove(documentId); | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             if (!document.GridFsId.HasValue) | ||||||
|  |             { | ||||||
|  |                 _logger.LogWarning("RU-BDU document {DocumentId} missing GridFS payload", documentId); | ||||||
|  |                 await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 pendingDocuments.Remove(documentId); | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             byte[] payload; | ||||||
|  |             try | ||||||
|  |             { | ||||||
|  |                 payload = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false); | ||||||
|  |             } | ||||||
|  |             catch (Exception ex) | ||||||
|  |             { | ||||||
|  |                 _logger.LogError(ex, "RU-BDU unable to download raw document {DocumentId}", documentId); | ||||||
|  |                 throw; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             RuBduVulnerabilityDto? dto; | ||||||
|  |             try | ||||||
|  |             { | ||||||
|  |                 dto = JsonSerializer.Deserialize<RuBduVulnerabilityDto>(payload, SerializerOptions); | ||||||
|  |             } | ||||||
|  |             catch (Exception ex) | ||||||
|  |             { | ||||||
|  |                 _logger.LogWarning(ex, "RU-BDU failed to deserialize document {DocumentId}", documentId); | ||||||
|  |                 await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 pendingDocuments.Remove(documentId); | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             if (dto is null) | ||||||
|  |             { | ||||||
|  |                 _logger.LogWarning("RU-BDU document {DocumentId} produced null DTO", documentId); | ||||||
|  |                 await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 pendingDocuments.Remove(documentId); | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             var bson = MongoDB.Bson.BsonDocument.Parse(JsonSerializer.Serialize(dto, SerializerOptions)); | ||||||
|  |             var dtoRecord = new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "ru-bdu.v1", bson, _timeProvider.GetUtcNow()); | ||||||
|  |             await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false); | ||||||
|  |             await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false); | ||||||
|  |  | ||||||
|  |             pendingDocuments.Remove(documentId); | ||||||
|  |             if (!pendingMappings.Contains(documentId)) | ||||||
|  |             { | ||||||
|  |                 pendingMappings.Add(documentId); | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var updatedCursor = cursor | ||||||
|  |             .WithPendingDocuments(pendingDocuments) | ||||||
|  |             .WithPendingMappings(pendingMappings); | ||||||
|  |  | ||||||
|  |         await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken) | ||||||
|  |     { | ||||||
|  |         ArgumentNullException.ThrowIfNull(services); | ||||||
|  |  | ||||||
|  |         var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false); | ||||||
|  |         if (cursor.PendingMappings.Count == 0) | ||||||
|  |         { | ||||||
|  |             return; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var pendingMappings = cursor.PendingMappings.ToList(); | ||||||
|  |  | ||||||
|  |         foreach (var documentId in cursor.PendingMappings) | ||||||
|  |         { | ||||||
|  |             cancellationToken.ThrowIfCancellationRequested(); | ||||||
|  |  | ||||||
|  |             var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false); | ||||||
|  |             if (document is null) | ||||||
|  |             { | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             var dtoRecord = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false); | ||||||
|  |             if (dtoRecord is null) | ||||||
|  |             { | ||||||
|  |                 _logger.LogWarning("RU-BDU document {DocumentId} missing DTO payload", documentId); | ||||||
|  |                 await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             RuBduVulnerabilityDto dto; | ||||||
|  |             try | ||||||
|  |             { | ||||||
|  |                 dto = JsonSerializer.Deserialize<RuBduVulnerabilityDto>(dtoRecord.Payload.ToString(), SerializerOptions) ?? throw new InvalidOperationException("DTO deserialized to null"); | ||||||
|  |             } | ||||||
|  |             catch (Exception ex) | ||||||
|  |             { | ||||||
|  |                 _logger.LogError(ex, "RU-BDU failed to deserialize DTO for document {DocumentId}", documentId); | ||||||
|  |                 await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             try | ||||||
|  |             { | ||||||
|  |                 var advisory = RuBduMapper.Map(dto, document, dtoRecord.ValidatedAt); | ||||||
|  |                 await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |             } | ||||||
|  |             catch (Exception ex) | ||||||
|  |             { | ||||||
|  |                 _logger.LogError(ex, "RU-BDU mapping failed for document {DocumentId}", documentId); | ||||||
|  |                 await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var updatedCursor = cursor.WithPendingMappings(pendingMappings); | ||||||
|  |         await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private async Task<int> ProcessArchiveAsync( | ||||||
|  |         byte[] archiveContent, | ||||||
|  |         DateTimeOffset now, | ||||||
|  |         HashSet<Guid> pendingDocuments, | ||||||
|  |         HashSet<Guid> pendingMappings, | ||||||
|  |         DateTimeOffset? archiveLastModified, | ||||||
|  |         CancellationToken cancellationToken) | ||||||
|  |     { | ||||||
|  |         var added = 0; | ||||||
|  |         using var archiveStream = new MemoryStream(archiveContent, writable: false); | ||||||
|  |         using var archive = new ZipArchive(archiveStream, ZipArchiveMode.Read, leaveOpen: false); | ||||||
|  |         var entry = archive.GetEntry("export/export.xml") ?? archive.Entries.FirstOrDefault(); | ||||||
|  |         if (entry is null) | ||||||
|  |         { | ||||||
|  |             _logger.LogWarning("RU-BDU archive does not contain export/export.xml; skipping."); | ||||||
|  |             return added; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         await using var entryStream = entry.Open(); | ||||||
|  |         using var reader = XmlReader.Create(entryStream, new XmlReaderSettings | ||||||
|  |         { | ||||||
|  |             IgnoreComments = true, | ||||||
|  |             IgnoreWhitespace = true, | ||||||
|  |             DtdProcessing = DtdProcessing.Ignore, | ||||||
|  |             CloseInput = false, | ||||||
|  |         }); | ||||||
|  |  | ||||||
|  |         while (reader.Read()) | ||||||
|  |         { | ||||||
|  |             cancellationToken.ThrowIfCancellationRequested(); | ||||||
|  |             if (reader.NodeType != XmlNodeType.Element || !reader.Name.Equals("vul", StringComparison.OrdinalIgnoreCase)) | ||||||
|  |             { | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             if (RuBduXmlParser.TryParse(XNode.ReadFrom(reader) as XElement ?? new XElement("vul")) is not { } dto) | ||||||
|  |             { | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             var payload = JsonSerializer.SerializeToUtf8Bytes(dto, SerializerOptions); | ||||||
|  |             var sha = Convert.ToHexString(SHA256.HashData(payload)).ToLowerInvariant(); | ||||||
|  |             var documentUri = BuildDocumentUri(dto.Identifier); | ||||||
|  |  | ||||||
|  |             var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, documentUri, cancellationToken).ConfigureAwait(false); | ||||||
|  |             if (existing is not null && string.Equals(existing.Sha256, sha, StringComparison.OrdinalIgnoreCase)) | ||||||
|  |             { | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             var gridFsId = await _rawDocumentStorage.UploadAsync(SourceName, documentUri, payload, "application/json", null, cancellationToken).ConfigureAwait(false); | ||||||
|  |  | ||||||
|  |             var metadata = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase) | ||||||
|  |             { | ||||||
|  |                 ["ru-bdu.identifier"] = dto.Identifier, | ||||||
|  |             }; | ||||||
|  |  | ||||||
|  |             if (!string.IsNullOrWhiteSpace(dto.Name)) | ||||||
|  |             { | ||||||
|  |                 metadata["ru-bdu.name"] = dto.Name!; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             var recordId = existing?.Id ?? Guid.NewGuid(); | ||||||
|  |             var record = new DocumentRecord( | ||||||
|  |                 recordId, | ||||||
|  |                 SourceName, | ||||||
|  |                 documentUri, | ||||||
|  |                 now, | ||||||
|  |                 sha, | ||||||
|  |                 DocumentStatuses.PendingParse, | ||||||
|  |                 "application/json", | ||||||
|  |                 Headers: null, | ||||||
|  |                 Metadata: metadata, | ||||||
|  |                 Etag: null, | ||||||
|  |                 LastModified: archiveLastModified ?? dto.IdentifyDate, | ||||||
|  |                 GridFsId: gridFsId, | ||||||
|  |                 ExpiresAt: null); | ||||||
|  |  | ||||||
|  |             var upserted = await _documentStore.UpsertAsync(record, cancellationToken).ConfigureAwait(false); | ||||||
|  |             pendingDocuments.Add(upserted.Id); | ||||||
|  |             pendingMappings.Remove(upserted.Id); | ||||||
|  |             added++; | ||||||
|  |  | ||||||
|  |             if (added >= _options.MaxVulnerabilitiesPerFetch) | ||||||
|  |             { | ||||||
|  |                 break; | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return added; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private string ResolveCacheDirectory(string? configuredPath) | ||||||
|  |     { | ||||||
|  |         if (!string.IsNullOrWhiteSpace(configuredPath)) | ||||||
|  |         { | ||||||
|  |             return Path.GetFullPath(Path.IsPathRooted(configuredPath) | ||||||
|  |                 ? configuredPath | ||||||
|  |                 : Path.Combine(AppContext.BaseDirectory, configuredPath)); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return Path.Combine(AppContext.BaseDirectory, "cache", RuBduConnectorPlugin.SourceName); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private void EnsureCacheDirectory() | ||||||
|  |     { | ||||||
|  |         try | ||||||
|  |         { | ||||||
|  |             Directory.CreateDirectory(_cacheDirectory); | ||||||
|  |         } | ||||||
|  |         catch (Exception ex) | ||||||
|  |         { | ||||||
|  |             _logger.LogWarning(ex, "RU-BDU unable to ensure cache directory {CachePath}", _cacheDirectory); | ||||||
|  |         } | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private void TryWriteCachedArchive(byte[] content) | ||||||
|  |     { | ||||||
|  |         try | ||||||
|  |         { | ||||||
|  |             Directory.CreateDirectory(Path.GetDirectoryName(_archiveCachePath)!); | ||||||
|  |             File.WriteAllBytes(_archiveCachePath, content); | ||||||
|  |         } | ||||||
|  |         catch (Exception ex) | ||||||
|  |         { | ||||||
|  |             _logger.LogDebug(ex, "RU-BDU failed to write cache archive {CachePath}", _archiveCachePath); | ||||||
|  |         } | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private bool TryReadCachedArchive(out byte[] content) | ||||||
|  |     { | ||||||
|  |         try | ||||||
|  |         { | ||||||
|  |             if (File.Exists(_archiveCachePath)) | ||||||
|  |             { | ||||||
|  |                 content = File.ReadAllBytes(_archiveCachePath); | ||||||
|  |                 return true; | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |         catch (Exception ex) | ||||||
|  |         { | ||||||
|  |             _logger.LogDebug(ex, "RU-BDU failed to read cache archive {CachePath}", _archiveCachePath); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         content = Array.Empty<byte>(); | ||||||
|  |         return false; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static string BuildDocumentUri(string identifier) | ||||||
|  |     { | ||||||
|  |         var slug = identifier.Contains(':', StringComparison.Ordinal) | ||||||
|  |             ? identifier[(identifier.IndexOf(':') + 1)..] | ||||||
|  |             : identifier; | ||||||
|  |         return $"https://bdu.fstec.ru/vul/{slug}"; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private async Task<RuBduCursor> GetCursorAsync(CancellationToken cancellationToken) | ||||||
|  |     { | ||||||
|  |         var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false); | ||||||
|  |         return state is null ? RuBduCursor.Empty : RuBduCursor.FromBson(state.Cursor); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private Task UpdateCursorAsync(RuBduCursor cursor, CancellationToken cancellationToken) | ||||||
|  |     { | ||||||
|  |         var document = cursor.ToBsonDocument(); | ||||||
|  |         var completedAt = cursor.LastSuccessfulFetch ?? _timeProvider.GetUtcNow(); | ||||||
|  |         return _stateRepository.UpdateCursorAsync(SourceName, document, completedAt, cancellationToken); | ||||||
|  |     } | ||||||
|  | } | ||||||
							
								
								
									
										19
									
								
								src/StellaOps.Feedser.Source.Ru.Bdu/RuBduConnectorPlugin.cs
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										19
									
								
								src/StellaOps.Feedser.Source.Ru.Bdu/RuBduConnectorPlugin.cs
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,19 @@ | |||||||
|  | using Microsoft.Extensions.DependencyInjection; | ||||||
|  | using StellaOps.Plugin; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Bdu; | ||||||
|  |  | ||||||
|  | public sealed class RuBduConnectorPlugin : IConnectorPlugin | ||||||
|  | { | ||||||
|  |     public const string SourceName = "ru-bdu"; | ||||||
|  |  | ||||||
|  |     public string Name => SourceName; | ||||||
|  |  | ||||||
|  |     public bool IsAvailable(IServiceProvider services) => services is not null; | ||||||
|  |  | ||||||
|  |     public IFeedConnector Create(IServiceProvider services) | ||||||
|  |     { | ||||||
|  |         ArgumentNullException.ThrowIfNull(services); | ||||||
|  |         return ActivatorUtilities.CreateInstance<RuBduConnector>(services); | ||||||
|  |     } | ||||||
|  | } | ||||||
| @@ -0,0 +1,53 @@ | |||||||
|  | using Microsoft.Extensions.Configuration; | ||||||
|  | using Microsoft.Extensions.DependencyInjection; | ||||||
|  | using StellaOps.DependencyInjection; | ||||||
|  | using StellaOps.Feedser.Core.Jobs; | ||||||
|  | using StellaOps.Feedser.Source.Ru.Bdu.Configuration; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Bdu; | ||||||
|  |  | ||||||
|  | public sealed class RuBduDependencyInjectionRoutine : IDependencyInjectionRoutine | ||||||
|  | { | ||||||
|  |     private const string ConfigurationSection = "feedser:sources:ru-bdu"; | ||||||
|  |  | ||||||
|  |     public IServiceCollection Register(IServiceCollection services, IConfiguration configuration) | ||||||
|  |     { | ||||||
|  |         ArgumentNullException.ThrowIfNull(services); | ||||||
|  |         ArgumentNullException.ThrowIfNull(configuration); | ||||||
|  |  | ||||||
|  |         services.AddRuBduConnector(options => | ||||||
|  |         { | ||||||
|  |             configuration.GetSection(ConfigurationSection).Bind(options); | ||||||
|  |             options.Validate(); | ||||||
|  |         }); | ||||||
|  |  | ||||||
|  |         services.AddTransient<RuBduFetchJob>(); | ||||||
|  |         services.AddTransient<RuBduParseJob>(); | ||||||
|  |         services.AddTransient<RuBduMapJob>(); | ||||||
|  |  | ||||||
|  |         services.PostConfigure<JobSchedulerOptions>(options => | ||||||
|  |         { | ||||||
|  |             EnsureJob(options, RuBduJobKinds.Fetch, typeof(RuBduFetchJob)); | ||||||
|  |             EnsureJob(options, RuBduJobKinds.Parse, typeof(RuBduParseJob)); | ||||||
|  |             EnsureJob(options, RuBduJobKinds.Map, typeof(RuBduMapJob)); | ||||||
|  |         }); | ||||||
|  |  | ||||||
|  |         return services; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static void EnsureJob(JobSchedulerOptions schedulerOptions, string kind, Type jobType) | ||||||
|  |     { | ||||||
|  |         if (schedulerOptions.Definitions.ContainsKey(kind)) | ||||||
|  |         { | ||||||
|  |             return; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         schedulerOptions.Definitions[kind] = new JobDefinition( | ||||||
|  |             kind, | ||||||
|  |             jobType, | ||||||
|  |             schedulerOptions.DefaultTimeout, | ||||||
|  |             schedulerOptions.DefaultLeaseDuration, | ||||||
|  |             CronExpression: null, | ||||||
|  |             Enabled: true); | ||||||
|  |     } | ||||||
|  | } | ||||||
| @@ -0,0 +1,43 @@ | |||||||
|  | using System.Net; | ||||||
|  | using Microsoft.Extensions.DependencyInjection; | ||||||
|  | using Microsoft.Extensions.Options; | ||||||
|  | using StellaOps.Feedser.Source.Ru.Bdu.Configuration; | ||||||
|  | using StellaOps.Feedser.Source.Common.Http; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Bdu; | ||||||
|  |  | ||||||
|  | public static class RuBduServiceCollectionExtensions | ||||||
|  | { | ||||||
|  |     public static IServiceCollection AddRuBduConnector(this IServiceCollection services, Action<RuBduOptions> configure) | ||||||
|  |     { | ||||||
|  |         ArgumentNullException.ThrowIfNull(services); | ||||||
|  |         ArgumentNullException.ThrowIfNull(configure); | ||||||
|  |  | ||||||
|  |         services.AddOptions<RuBduOptions>() | ||||||
|  |             .Configure(configure) | ||||||
|  |             .PostConfigure(static options => options.Validate()); | ||||||
|  |  | ||||||
|  |         services.AddSourceHttpClient(RuBduOptions.HttpClientName, (sp, clientOptions) => | ||||||
|  |         { | ||||||
|  |             var options = sp.GetRequiredService<IOptions<RuBduOptions>>().Value; | ||||||
|  |             clientOptions.BaseAddress = options.BaseAddress; | ||||||
|  |             clientOptions.Timeout = options.RequestTimeout; | ||||||
|  |             clientOptions.UserAgent = options.UserAgent; | ||||||
|  |             clientOptions.AllowAutoRedirect = true; | ||||||
|  |             clientOptions.DefaultRequestHeaders["Accept-Language"] = options.AcceptLanguage; | ||||||
|  |             clientOptions.AllowedHosts.Clear(); | ||||||
|  |             clientOptions.AllowedHosts.Add(options.BaseAddress.Host); | ||||||
|  |             clientOptions.ConfigureHandler = handler => | ||||||
|  |             { | ||||||
|  |                 handler.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate; | ||||||
|  |                 handler.AllowAutoRedirect = true; | ||||||
|  |                 handler.UseCookies = true; | ||||||
|  |                 handler.CookieContainer = new CookieContainer(); | ||||||
|  |             }; | ||||||
|  |         }); | ||||||
|  |  | ||||||
|  |         services.AddTransient<RuBduConnector>(); | ||||||
|  |  | ||||||
|  |         return services; | ||||||
|  |     } | ||||||
|  | } | ||||||
| @@ -8,9 +8,11 @@ | |||||||
|  |  | ||||||
|   <ItemGroup> |   <ItemGroup> | ||||||
|     <ProjectReference Include="../StellaOps.Plugin/StellaOps.Plugin.csproj" /> |     <ProjectReference Include="../StellaOps.Plugin/StellaOps.Plugin.csproj" /> | ||||||
|  |     <ProjectReference Include="../StellaOps.Feedser.Core/StellaOps.Feedser.Core.csproj" /> | ||||||
|  |     <ProjectReference Include="../StellaOps.Feedser.Normalization/StellaOps.Feedser.Normalization.csproj" /> | ||||||
|     <ProjectReference Include="../StellaOps.Feedser.Source.Common/StellaOps.Feedser.Source.Common.csproj" /> |     <ProjectReference Include="../StellaOps.Feedser.Source.Common/StellaOps.Feedser.Source.Common.csproj" /> | ||||||
|     <ProjectReference Include="../StellaOps.Feedser.Models/StellaOps.Feedser.Models.csproj" /> |     <ProjectReference Include="../StellaOps.Feedser.Models/StellaOps.Feedser.Models.csproj" /> | ||||||
|  |     <ProjectReference Include="../StellaOps.Feedser.Storage.Mongo/StellaOps.Feedser.Storage.Mongo.csproj" /> | ||||||
|   </ItemGroup> |   </ItemGroup> | ||||||
| </Project> |  | ||||||
|  |  | ||||||
|  | </Project> | ||||||
|   | |||||||
| @@ -2,10 +2,10 @@ | |||||||
| | Task | Owner(s) | Depends on | Notes | | | Task | Owner(s) | Depends on | Notes | | ||||||
| |---|---|---|---| | |---|---|---|---| | ||||||
| |FEEDCONN-RUBDU-02-001 Identify BDU data source & schema|BE-Conn-BDU|Research|**DONE (2025-10-11)** – Candidate endpoints (`https://bdu.fstec.ru/component/rsform/form/7-bdu?format=xml`, `...?format=json`) return 403/404 even with `--insecure` because TLS chain requires Russian Trusted Sub CA and WAF expects referer/session headers. Documented request/response samples in `docs/feedser-connector-research-20251011.md`; blocked until trusted root + access strategy from Ops.| | |FEEDCONN-RUBDU-02-001 Identify BDU data source & schema|BE-Conn-BDU|Research|**DONE (2025-10-11)** – Candidate endpoints (`https://bdu.fstec.ru/component/rsform/form/7-bdu?format=xml`, `...?format=json`) return 403/404 even with `--insecure` because TLS chain requires Russian Trusted Sub CA and WAF expects referer/session headers. Documented request/response samples in `docs/feedser-connector-research-20251011.md`; blocked until trusted root + access strategy from Ops.| | ||||||
| |FEEDCONN-RUBDU-02-002 Fetch pipeline & cursor handling|BE-Conn-BDU|Source.Common, Storage.Mongo|**TODO** – Fetcher must support custom trust store (`SourceHttpClientOptions.TrustedRootCertificates`), optional proxy, and signed cookie injection. Persist raw HTML/CSV once accessible, with cursor based on advisory `unicId` + `lastmod`. Implement retry/backoff aware of WAF transaction IDs. _(2025-10-12: Source.Common trust-store plumbing landed; blocked until sanctioned RU CA bundle is supplied.)_ **Coordination:** Ops to hand off sanctioned RU CA bundle + packaging notes for Offline Kit; Source.Common to review trust-store configuration once materials arrive.| | |FEEDCONN-RUBDU-02-002 Fetch pipeline & cursor handling|BE-Conn-BDU|Source.Common, Storage.Mongo|**DOING (2025-10-12)** – Fetch job now expands `vulxml.zip` into per-advisory JSON documents with cursor tracking + trust store wiring (`certificates/russian_trusted_*`). Parser/mapper emit canonical advisories; next up is wiring fixtures, regression tests, and telemetry before closing the task.| | ||||||
| |FEEDCONN-RUBDU-02-003 DTO/parser implementation|BE-Conn-BDU|Source.Common|**TODO** – Create DTOs for BDU records (title, severity, vendor/product, references, CVEs); sanitise text.| | |FEEDCONN-RUBDU-02-003 DTO/parser implementation|BE-Conn-BDU|Source.Common|**DOING (2025-10-12)** – `RuBduXmlParser` materialises per-entry DTOs and serialises them into Mongo DTO records; remaining work covers resilience fixtures and edge-case coverage (multi-CWE, empty software lists).| | ||||||
| |FEEDCONN-RUBDU-02-004 Canonical mapping & range primitives|BE-Conn-BDU|Models|**TODO** – Map into canonical advisories with aliases, references, and vendor range primitives. Use normalized rule checkpoints from `../StellaOps.Feedser.Merge/RANGE_PRIMITIVES_COORDINATION.md`.<br>2025-10-11 research trail: sample payload `[{"scheme":"semver","type":"range","min":"<start>","minInclusive":true,"max":"<end>","maxInclusive":false,"notes":"ru.bdu:ID"}]`; if advisories rely on firmware build strings, preserve them in `notes` until a dedicated scheme is approved.| | |FEEDCONN-RUBDU-02-004 Canonical mapping & range primitives|BE-Conn-BDU|Models|**DOING (2025-10-12)** – `RuBduMapper` produces canonical advisories (aliases, references, vendor packages, CVSS). Follow-up: refine status translation + range primitives once richer samples arrive; ensure fixtures cover environment/other metadata before marking DONE.| | ||||||
| |FEEDCONN-RUBDU-02-005 Deterministic fixtures & regression tests|QA|Testing|**TODO** – Add fetch/parse/map tests with fixtures; support `UPDATE_BDU_FIXTURES=1`.| | |FEEDCONN-RUBDU-02-005 Deterministic fixtures & regression tests|QA|Testing|**TODO** – Add fetch/parse/map tests with fixtures; support `UPDATE_BDU_FIXTURES=1`.| | ||||||
| |FEEDCONN-RUBDU-02-006 Telemetry & documentation|DevEx|Docs|**TODO** – Add logging/metrics, document connector configuration, close backlog when complete.| | |FEEDCONN-RUBDU-02-006 Telemetry & documentation|DevEx|Docs|**TODO** – Add logging/metrics, document connector configuration, close backlog when complete.| | ||||||
| |FEEDCONN-RUBDU-02-007 Access & export options assessment|BE-Conn-BDU|Research|**TODO** – Once access unblocked, compare RSS/Atom (if restored) vs HTML table export (`/vul` list) and legacy CSV dumps. Need to confirm whether login/anti-bot tokens required and outline offline mirroring plan (one-time tarball seeded into Offline Kit).| | |FEEDCONN-RUBDU-02-007 Access & export options assessment|BE-Conn-BDU|Research|**TODO** – Once access unblocked, compare RSS/Atom (if restored) vs HTML table export (`/vul` list) and legacy CSV dumps. Need to confirm whether login/anti-bot tokens required and outline offline mirroring plan (one-time tarball seeded into Offline Kit).| | ||||||
| |FEEDCONN-RUBDU-02-008 Trusted root onboarding plan|BE-Conn-BDU|Source.Common|**BLOCKED** – 2025-10-11: Attempt to download Russian Trusted Sub CA returned placeholder HTML; need alternate distribution (mirror or manual bundle) before TLS validation succeeds.<br>2025-10-11 23:05Z: Shared HTTP trust-store support landed (`SourceHttpClientOptions.TrustedRootCertificates`, config keys `feedser:httpClients:source.bdu:*`); now blocked on Ops delivering sanctioned RU CA bundle + Offline Kit packaging instructions.| | |FEEDCONN-RUBDU-02-008 Trusted root onboarding plan|BE-Conn-BDU|Source.Common|**DOING (2025-10-12)** – Mirrored official Russian Trusted Root/Sub CA PEMs from rostelecom.ru (`certificates/russian_trusted_root_ca.pem`, `certificates/russian_trusted_sub_ca.pem`, bundle `certificates/russian_trusted_bundle.pem`) and validated TLS handshake. Next: confirm packaging guidance for Offline Kit + config samples using `feedser:httpClients:source.bdu:trustedRootPaths`.| | ||||||
|   | |||||||
										
											Binary file not shown.
										
									
								
							| @@ -0,0 +1,7 @@ | |||||||
|  | <html> | ||||||
|  |   <body> | ||||||
|  |     <ul> | ||||||
|  |       <li><a href="/materialy/uyazvimosti/bulletin-sample.json.zip" title="Bulletin Sample">Bulletin Sample</a></li> | ||||||
|  |     </ul> | ||||||
|  |   </body> | ||||||
|  | </html> | ||||||
| @@ -0,0 +1,165 @@ | |||||||
|  | [ | ||||||
|  |   { | ||||||
|  |     "advisoryKey": "BDU:2025-01001", | ||||||
|  |     "affectedPackages": [ | ||||||
|  |       { | ||||||
|  |         "type": "vendor", | ||||||
|  |         "identifier": "SampleSCADA <= 4.2", | ||||||
|  |         "platform": null, | ||||||
|  |         "versionRanges": [], | ||||||
|  |         "normalizedVersions": [], | ||||||
|  |         "statuses": [ | ||||||
|  |           { | ||||||
|  |             "provenance": { | ||||||
|  |               "source": "ru-nkcki", | ||||||
|  |               "kind": "package-status", | ||||||
|  |               "value": "patch_available", | ||||||
|  |               "decisionReason": null, | ||||||
|  |               "recordedAt": "2025-09-22T00:00:00+00:00", | ||||||
|  |               "fieldMask": [ | ||||||
|  |                 "affectedpackages[].statuses[]" | ||||||
|  |               ] | ||||||
|  |             }, | ||||||
|  |             "status": "fixed" | ||||||
|  |           } | ||||||
|  |         ], | ||||||
|  |         "provenance": [ | ||||||
|  |           { | ||||||
|  |             "source": "ru-nkcki", | ||||||
|  |             "kind": "package", | ||||||
|  |             "value": "SampleSCADA <= 4.2", | ||||||
|  |             "decisionReason": null, | ||||||
|  |             "recordedAt": "2025-09-22T00:00:00+00:00", | ||||||
|  |             "fieldMask": [ | ||||||
|  |               "affectedpackages[]" | ||||||
|  |             ] | ||||||
|  |           } | ||||||
|  |         ] | ||||||
|  |       } | ||||||
|  |     ], | ||||||
|  |     "aliases": [ | ||||||
|  |       "BDU:2025-01001", | ||||||
|  |       "CVE-2025-0101" | ||||||
|  |     ], | ||||||
|  |     "credits": [], | ||||||
|  |     "cvssMetrics": [ | ||||||
|  |       { | ||||||
|  |         "baseScore": 8.5, | ||||||
|  |         "baseSeverity": "high", | ||||||
|  |         "provenance": { | ||||||
|  |           "source": "ru-nkcki", | ||||||
|  |           "kind": "cvss", | ||||||
|  |           "value": "CVSS:3.1/AV:N/AC:H/PR:L/UI:N/S:C/C:H/I:H/A:H", | ||||||
|  |           "decisionReason": null, | ||||||
|  |           "recordedAt": "2025-09-22T00:00:00+00:00", | ||||||
|  |           "fieldMask": [ | ||||||
|  |             "cvssmetrics[]" | ||||||
|  |           ] | ||||||
|  |         }, | ||||||
|  |         "vector": "CVSS:3.1/AV:N/AC:H/PR:L/UI:N/S:C/C:H/I:H/A:H", | ||||||
|  |         "version": "3.1" | ||||||
|  |       } | ||||||
|  |     ], | ||||||
|  |     "exploitKnown": true, | ||||||
|  |     "language": "ru", | ||||||
|  |     "modified": "2025-09-22T00:00:00+00:00", | ||||||
|  |     "provenance": [ | ||||||
|  |       { | ||||||
|  |         "source": "ru-nkcki", | ||||||
|  |         "kind": "advisory", | ||||||
|  |         "value": "BDU:2025-01001", | ||||||
|  |         "decisionReason": null, | ||||||
|  |         "recordedAt": "2025-09-22T00:00:00+00:00", | ||||||
|  |         "fieldMask": [ | ||||||
|  |           "advisory" | ||||||
|  |         ] | ||||||
|  |       } | ||||||
|  |     ], | ||||||
|  |     "published": "2025-09-20T00:00:00+00:00", | ||||||
|  |     "references": [ | ||||||
|  |       { | ||||||
|  |         "kind": "details", | ||||||
|  |         "provenance": { | ||||||
|  |           "source": "ru-nkcki", | ||||||
|  |           "kind": "reference", | ||||||
|  |           "value": "https://bdu.fstec.ru/vul/2025-01001", | ||||||
|  |           "decisionReason": null, | ||||||
|  |           "recordedAt": "2025-09-22T00:00:00+00:00", | ||||||
|  |           "fieldMask": [ | ||||||
|  |             "references[]" | ||||||
|  |           ] | ||||||
|  |         }, | ||||||
|  |         "sourceTag": "bdu", | ||||||
|  |         "summary": null, | ||||||
|  |         "url": "https://bdu.fstec.ru/vul/2025-01001" | ||||||
|  |       }, | ||||||
|  |       { | ||||||
|  |         "kind": "details", | ||||||
|  |         "provenance": { | ||||||
|  |           "source": "ru-nkcki", | ||||||
|  |           "kind": "reference", | ||||||
|  |           "value": "https://cert.gov.ru/materialy/uyazvimosti/2025-01001", | ||||||
|  |           "decisionReason": null, | ||||||
|  |           "recordedAt": "2025-09-22T00:00:00+00:00", | ||||||
|  |           "fieldMask": [ | ||||||
|  |             "references[]" | ||||||
|  |           ] | ||||||
|  |         }, | ||||||
|  |         "sourceTag": null, | ||||||
|  |         "summary": null, | ||||||
|  |         "url": "https://cert.gov.ru/materialy/uyazvimosti/2025-01001" | ||||||
|  |       }, | ||||||
|  |       { | ||||||
|  |         "kind": "details", | ||||||
|  |         "provenance": { | ||||||
|  |           "source": "ru-nkcki", | ||||||
|  |           "kind": "reference", | ||||||
|  |           "value": "https://cert.gov.ru/materialy/uyazvimosti/2025-01001", | ||||||
|  |           "decisionReason": null, | ||||||
|  |           "recordedAt": "2025-09-22T00:00:00+00:00", | ||||||
|  |           "fieldMask": [ | ||||||
|  |             "references[]" | ||||||
|  |           ] | ||||||
|  |         }, | ||||||
|  |         "sourceTag": "ru-nkcki", | ||||||
|  |         "summary": null, | ||||||
|  |         "url": "https://cert.gov.ru/materialy/uyazvimosti/2025-01001" | ||||||
|  |       }, | ||||||
|  |       { | ||||||
|  |         "kind": "cwe", | ||||||
|  |         "provenance": { | ||||||
|  |           "source": "ru-nkcki", | ||||||
|  |           "kind": "reference", | ||||||
|  |           "value": "https://cwe.mitre.org/data/definitions/321.html", | ||||||
|  |           "decisionReason": null, | ||||||
|  |           "recordedAt": "2025-09-22T00:00:00+00:00", | ||||||
|  |           "fieldMask": [ | ||||||
|  |             "references[]" | ||||||
|  |           ] | ||||||
|  |         }, | ||||||
|  |         "sourceTag": "cwe", | ||||||
|  |         "summary": "Use of Hard-coded Cryptographic Key", | ||||||
|  |         "url": "https://cwe.mitre.org/data/definitions/321.html" | ||||||
|  |       }, | ||||||
|  |       { | ||||||
|  |         "kind": "external", | ||||||
|  |         "provenance": { | ||||||
|  |           "source": "ru-nkcki", | ||||||
|  |           "kind": "reference", | ||||||
|  |           "value": "https://vendor.example/advisories/sample-scada", | ||||||
|  |           "decisionReason": null, | ||||||
|  |           "recordedAt": "2025-09-22T00:00:00+00:00", | ||||||
|  |           "fieldMask": [ | ||||||
|  |             "references[]" | ||||||
|  |           ] | ||||||
|  |         }, | ||||||
|  |         "sourceTag": null, | ||||||
|  |         "summary": null, | ||||||
|  |         "url": "https://vendor.example/advisories/sample-scada" | ||||||
|  |       } | ||||||
|  |     ], | ||||||
|  |     "severity": "critical", | ||||||
|  |     "summary": "Authenticated RCE in Sample SCADA", | ||||||
|  |     "title": "Authenticated RCE in Sample SCADA" | ||||||
|  |   } | ||||||
|  | ] | ||||||
| @@ -0,0 +1,289 @@ | |||||||
|  | using System.Collections.Generic; | ||||||
|  | using System.IO; | ||||||
|  | using System.Linq; | ||||||
|  | using System.Threading; | ||||||
|  | using System.Threading.Tasks; | ||||||
|  | using System.Net; | ||||||
|  | using System.Net.Http; | ||||||
|  | using System.Net.Http.Headers; | ||||||
|  | using System.Text; | ||||||
|  | using Microsoft.Extensions.DependencyInjection; | ||||||
|  | using Microsoft.Extensions.Http; | ||||||
|  | using Microsoft.Extensions.Logging; | ||||||
|  | using Microsoft.Extensions.Logging.Abstractions; | ||||||
|  | using Microsoft.Extensions.Options; | ||||||
|  | using Microsoft.Extensions.Time.Testing; | ||||||
|  | using MongoDB.Bson; | ||||||
|  | using StellaOps.Feedser.Source.Common; | ||||||
|  | using StellaOps.Feedser.Source.Common.Http; | ||||||
|  | using StellaOps.Feedser.Source.Common.Testing; | ||||||
|  | using StellaOps.Feedser.Source.Ru.Nkcki; | ||||||
|  | using StellaOps.Feedser.Source.Ru.Nkcki.Configuration; | ||||||
|  | using StellaOps.Feedser.Storage.Mongo; | ||||||
|  | using StellaOps.Feedser.Storage.Mongo.Advisories; | ||||||
|  | using StellaOps.Feedser.Storage.Mongo.Documents; | ||||||
|  | using StellaOps.Feedser.Testing; | ||||||
|  | using StellaOps.Feedser.Models; | ||||||
|  | using MongoDB.Driver; | ||||||
|  | using Xunit; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Nkcki.Tests; | ||||||
|  |  | ||||||
|  | [Collection("mongo-fixture")] | ||||||
|  | public sealed class RuNkckiConnectorTests : IAsyncLifetime | ||||||
|  | { | ||||||
|  |     private static readonly Uri ListingUri = new("https://cert.gov.ru/materialy/uyazvimosti/"); | ||||||
|  |     private static readonly Uri BulletinUri = new("https://cert.gov.ru/materialy/uyazvimosti/bulletin-sample.json.zip"); | ||||||
|  |  | ||||||
|  |     private readonly MongoIntegrationFixture _fixture; | ||||||
|  |     private readonly FakeTimeProvider _timeProvider; | ||||||
|  |     private readonly CannedHttpMessageHandler _handler; | ||||||
|  |  | ||||||
|  |     public RuNkckiConnectorTests(MongoIntegrationFixture fixture) | ||||||
|  |     { | ||||||
|  |         _fixture = fixture; | ||||||
|  |         _timeProvider = new FakeTimeProvider(new DateTimeOffset(2025, 10, 12, 0, 0, 0, TimeSpan.Zero)); | ||||||
|  |         _handler = new CannedHttpMessageHandler(); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     [Fact] | ||||||
|  |     public async Task FetchParseMap_ProducesExpectedSnapshot() | ||||||
|  |     { | ||||||
|  |         await using var provider = await BuildServiceProviderAsync(); | ||||||
|  |         SeedListingAndBulletin(); | ||||||
|  |  | ||||||
|  |         var connector = provider.GetRequiredService<RuNkckiConnector>(); | ||||||
|  |         await connector.FetchAsync(provider, CancellationToken.None); | ||||||
|  |         _timeProvider.Advance(TimeSpan.FromMinutes(1)); | ||||||
|  |         await connector.ParseAsync(provider, CancellationToken.None); | ||||||
|  |         await connector.MapAsync(provider, CancellationToken.None); | ||||||
|  |  | ||||||
|  |         var advisoryStore = provider.GetRequiredService<IAdvisoryStore>(); | ||||||
|  |         var advisories = await advisoryStore.GetRecentAsync(10, CancellationToken.None); | ||||||
|  |         Assert.Single(advisories); | ||||||
|  |  | ||||||
|  |         var snapshot = SnapshotSerializer.ToSnapshot(advisories); | ||||||
|  |         WriteOrAssertSnapshot(snapshot, "nkcki-advisories.snapshot.json"); | ||||||
|  |  | ||||||
|  |         var documentStore = provider.GetRequiredService<IDocumentStore>(); | ||||||
|  |         var document = await documentStore.FindBySourceAndUriAsync(RuNkckiConnectorPlugin.SourceName, "https://cert.gov.ru/materialy/uyazvimosti/BDU:2025-01001", CancellationToken.None); | ||||||
|  |         Assert.NotNull(document); | ||||||
|  |         Assert.Equal(DocumentStatuses.Mapped, document!.Status); | ||||||
|  |  | ||||||
|  |         var stateRepository = provider.GetRequiredService<ISourceStateRepository>(); | ||||||
|  |         var state = await stateRepository.TryGetAsync(RuNkckiConnectorPlugin.SourceName, CancellationToken.None); | ||||||
|  |         Assert.NotNull(state); | ||||||
|  |         Assert.True(IsEmptyArray(state!.Cursor, "pendingDocuments")); | ||||||
|  |         Assert.True(IsEmptyArray(state.Cursor, "pendingMappings")); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     [Fact] | ||||||
|  |     public async Task Fetch_ReusesCachedBulletinWhenListingFails() | ||||||
|  |     { | ||||||
|  |         await using var provider = await BuildServiceProviderAsync(); | ||||||
|  |         SeedListingAndBulletin(); | ||||||
|  |  | ||||||
|  |         var connector = provider.GetRequiredService<RuNkckiConnector>(); | ||||||
|  |         await connector.FetchAsync(provider, CancellationToken.None); | ||||||
|  |  | ||||||
|  |         _handler.Clear(); | ||||||
|  |         _handler.AddResponse(ListingUri, () => new HttpResponseMessage(HttpStatusCode.InternalServerError) | ||||||
|  |         { | ||||||
|  |             Content = new StringContent("error", Encoding.UTF8, "text/plain"), | ||||||
|  |         }); | ||||||
|  |  | ||||||
|  |         var advisoryStore = provider.GetRequiredService<IAdvisoryStore>(); | ||||||
|  |         var before = await advisoryStore.GetRecentAsync(10, CancellationToken.None); | ||||||
|  |         Assert.NotEmpty(before); | ||||||
|  |  | ||||||
|  |         await connector.FetchAsync(provider, CancellationToken.None); | ||||||
|  |  | ||||||
|  |         var after = await advisoryStore.GetRecentAsync(10, CancellationToken.None); | ||||||
|  |         Assert.Equal(before.Select(advisory => advisory.AdvisoryKey).OrderBy(static key => key), after.Select(advisory => advisory.AdvisoryKey).OrderBy(static key => key)); | ||||||
|  |  | ||||||
|  |         _handler.AssertNoPendingResponses(); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private async Task<ServiceProvider> BuildServiceProviderAsync() | ||||||
|  |     { | ||||||
|  |         try | ||||||
|  |         { | ||||||
|  |             await _fixture.Client.DropDatabaseAsync(_fixture.Database.DatabaseNamespace.DatabaseName); | ||||||
|  |         } | ||||||
|  |         catch (MongoConnectionException ex) | ||||||
|  |         { | ||||||
|  |             Assert.Skip($"Mongo runner unavailable: {ex.Message}"); | ||||||
|  |         } | ||||||
|  |         catch (TimeoutException ex) | ||||||
|  |         { | ||||||
|  |             Assert.Skip($"Mongo runner unavailable: {ex.Message}"); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         _handler.Clear(); | ||||||
|  |  | ||||||
|  |         var services = new ServiceCollection(); | ||||||
|  |         services.AddLogging(builder => builder.AddProvider(NullLoggerProvider.Instance)); | ||||||
|  |         services.AddSingleton<TimeProvider>(_timeProvider); | ||||||
|  |  | ||||||
|  |         services.AddMongoStorage(options => | ||||||
|  |         { | ||||||
|  |             options.ConnectionString = _fixture.Runner.ConnectionString; | ||||||
|  |             options.DatabaseName = _fixture.Database.DatabaseNamespace.DatabaseName; | ||||||
|  |             options.CommandTimeout = TimeSpan.FromSeconds(5); | ||||||
|  |         }); | ||||||
|  |  | ||||||
|  |         services.AddSourceCommon(); | ||||||
|  |         services.AddRuNkckiConnector(options => | ||||||
|  |         { | ||||||
|  |             options.BaseAddress = new Uri("https://cert.gov.ru/"); | ||||||
|  |             options.ListingPath = "/materialy/uyazvimosti/"; | ||||||
|  |             options.MaxBulletinsPerFetch = 2; | ||||||
|  |             options.MaxVulnerabilitiesPerFetch = 50; | ||||||
|  |             var cacheRoot = Path.Combine(Path.GetTempPath(), "stellaops-tests", _fixture.Database.DatabaseNamespace.DatabaseName); | ||||||
|  |             Directory.CreateDirectory(cacheRoot); | ||||||
|  |             options.CacheDirectory = Path.Combine(cacheRoot, "ru-nkcki"); | ||||||
|  |             options.RequestDelay = TimeSpan.Zero; | ||||||
|  |         }); | ||||||
|  |  | ||||||
|  |         services.Configure<HttpClientFactoryOptions>(RuNkckiOptions.HttpClientName, builderOptions => | ||||||
|  |         { | ||||||
|  |             builderOptions.HttpMessageHandlerBuilderActions.Add(builder => builder.PrimaryHandler = _handler); | ||||||
|  |         }); | ||||||
|  |  | ||||||
|  |         try | ||||||
|  |         { | ||||||
|  |             var provider = services.BuildServiceProvider(); | ||||||
|  |             var bootstrapper = provider.GetRequiredService<MongoBootstrapper>(); | ||||||
|  |             await bootstrapper.InitializeAsync(CancellationToken.None); | ||||||
|  |             return provider; | ||||||
|  |         } | ||||||
|  |         catch (MongoConnectionException ex) | ||||||
|  |         { | ||||||
|  |             Assert.Skip($"Mongo runner unavailable: {ex.Message}"); | ||||||
|  |             throw; // Unreachable | ||||||
|  |         } | ||||||
|  |         catch (TimeoutException ex) | ||||||
|  |         { | ||||||
|  |             Assert.Skip($"Mongo runner unavailable: {ex.Message}"); | ||||||
|  |             throw; | ||||||
|  |         } | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private void SeedListingAndBulletin() | ||||||
|  |     { | ||||||
|  |         var listingHtml = ReadFixture("listing.html"); | ||||||
|  |         _handler.AddTextResponse(ListingUri, listingHtml, "text/html"); | ||||||
|  |  | ||||||
|  |         var bulletinBytes = ReadBulletinFixture("bulletin-sample.json.zip"); | ||||||
|  |         _handler.AddResponse(BulletinUri, () => | ||||||
|  |         { | ||||||
|  |             var response = new HttpResponseMessage(HttpStatusCode.OK) | ||||||
|  |             { | ||||||
|  |                 Content = new ByteArrayContent(bulletinBytes), | ||||||
|  |             }; | ||||||
|  |             response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/zip"); | ||||||
|  |             response.Content.Headers.LastModified = new DateTimeOffset(2025, 9, 22, 0, 0, 0, TimeSpan.Zero); | ||||||
|  |             return response; | ||||||
|  |         }); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static bool IsEmptyArray(BsonDocument document, string field) | ||||||
|  |     { | ||||||
|  |         if (!document.TryGetValue(field, out var value) || value is not BsonArray array) | ||||||
|  |         { | ||||||
|  |             return false; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return array.Count == 0; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static string ReadFixture(string filename) | ||||||
|  |     { | ||||||
|  |         var path = Path.Combine("Fixtures", filename); | ||||||
|  |         var resolved = ResolveFixturePath(path); | ||||||
|  |         return File.ReadAllText(resolved); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static byte[] ReadBulletinFixture(string filename) | ||||||
|  |     { | ||||||
|  |         var path = Path.Combine("Fixtures", filename); | ||||||
|  |         var resolved = ResolveFixturePath(path); | ||||||
|  |         return File.ReadAllBytes(resolved); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static string ResolveFixturePath(string relativePath) | ||||||
|  |     { | ||||||
|  |         var projectRoot = GetProjectRoot(); | ||||||
|  |         var projectPath = Path.Combine(projectRoot, relativePath); | ||||||
|  |         if (File.Exists(projectPath)) | ||||||
|  |         { | ||||||
|  |             return projectPath; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var binaryPath = Path.Combine(AppContext.BaseDirectory, relativePath); | ||||||
|  |         if (File.Exists(binaryPath)) | ||||||
|  |         { | ||||||
|  |             return Path.GetFullPath(binaryPath); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         throw new FileNotFoundException($"Fixture not found: {relativePath}"); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static void WriteOrAssertSnapshot(string snapshot, string filename) | ||||||
|  |     { | ||||||
|  |         if (ShouldUpdateFixtures()) | ||||||
|  |         { | ||||||
|  |             var path = GetWritableFixturePath(filename); | ||||||
|  |             Directory.CreateDirectory(Path.GetDirectoryName(path)!); | ||||||
|  |             File.WriteAllText(path, snapshot); | ||||||
|  |             return; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var expectedPath = ResolveFixturePath(Path.Combine("Fixtures", filename)); | ||||||
|  |         if (!File.Exists(expectedPath)) | ||||||
|  |         { | ||||||
|  |             throw new FileNotFoundException($"Expected snapshot missing: {expectedPath}. Set UPDATE_NKCKI_FIXTURES=1 to generate."); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var expected = File.ReadAllText(expectedPath); | ||||||
|  |         Assert.Equal(Normalize(expected), Normalize(snapshot)); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static string GetWritableFixturePath(string filename) | ||||||
|  |     { | ||||||
|  |         var projectRoot = GetProjectRoot(); | ||||||
|  |         return Path.Combine(projectRoot, "Fixtures", filename); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static bool ShouldUpdateFixtures() | ||||||
|  |     { | ||||||
|  |         var value = Environment.GetEnvironmentVariable("UPDATE_NKCKI_FIXTURES"); | ||||||
|  |         return string.Equals(value, "1", StringComparison.OrdinalIgnoreCase) | ||||||
|  |             || string.Equals(value, "true", StringComparison.OrdinalIgnoreCase); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static string Normalize(string text) | ||||||
|  |         => text.Replace("\r\n", "\n", StringComparison.Ordinal); | ||||||
|  |  | ||||||
|  |     private static string GetProjectRoot() | ||||||
|  |     { | ||||||
|  |         var current = AppContext.BaseDirectory; | ||||||
|  |         while (!string.IsNullOrEmpty(current)) | ||||||
|  |         { | ||||||
|  |             var candidate = Path.Combine(current, "StellaOps.Feedser.Source.Ru.Nkcki.Tests.csproj"); | ||||||
|  |             if (File.Exists(candidate)) | ||||||
|  |             { | ||||||
|  |                 return current; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             current = Path.GetDirectoryName(current); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         throw new InvalidOperationException("Unable to locate project root for Ru.Nkcki tests."); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     public Task InitializeAsync() => Task.CompletedTask; | ||||||
|  |  | ||||||
|  |     public async Task DisposeAsync() | ||||||
|  |         => await _fixture.Client.DropDatabaseAsync(_fixture.Database.DatabaseNamespace.DatabaseName); | ||||||
|  | } | ||||||
| @@ -0,0 +1,43 @@ | |||||||
|  | using System.Text.Json; | ||||||
|  | using StellaOps.Feedser.Source.Ru.Nkcki.Internal; | ||||||
|  | using Xunit; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Nkcki.Tests; | ||||||
|  |  | ||||||
|  | public sealed class RuNkckiJsonParserTests | ||||||
|  | { | ||||||
|  |     [Fact] | ||||||
|  |     public void Parse_WellFormedEntry_ReturnsDto() | ||||||
|  |     { | ||||||
|  |         const string json = """ | ||||||
|  | { | ||||||
|  |   "vuln_id": {"MITRE": "CVE-2025-0001", "FSTEC": "BDU:2025-00001"}, | ||||||
|  |   "date_published": "2025-09-01", | ||||||
|  |   "date_updated": "2025-09-02", | ||||||
|  |   "cvss_rating": "КРИТИЧЕСКИЙ", | ||||||
|  |   "patch_available": true, | ||||||
|  |   "description": "Test description", | ||||||
|  |   "cwe": {"cwe_number": 79, "cwe_description": "Cross-site scripting"}, | ||||||
|  |   "product_category": "Web", | ||||||
|  |   "mitigation": "Apply update", | ||||||
|  |   "vulnerable_software": {"software_text": "ExampleApp 1.0", "cpe": false}, | ||||||
|  |   "cvss": {"cvss_score": 8.8, "cvss_vector": "AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H", "cvss_score_v4": 5.5, "cvss_vector_v4": "AV:N/AC:L/AT:N/PR:N/UI:N/VC:H/VI:H/VA:H"}, | ||||||
|  |   "impact": "ACE", | ||||||
|  |   "method_of_exploitation": "Special request", | ||||||
|  |   "user_interaction": false, | ||||||
|  |   "urls": ["https://example.com/advisory", "https://cert.gov.ru/materialy/uyazvimosti/2025-00001"] | ||||||
|  | } | ||||||
|  | """; | ||||||
|  |  | ||||||
|  |         using var document = JsonDocument.Parse(json); | ||||||
|  |         var dto = RuNkckiJsonParser.Parse(document.RootElement); | ||||||
|  |  | ||||||
|  |         Assert.Equal("BDU:2025-00001", dto.FstecId); | ||||||
|  |         Assert.Equal("CVE-2025-0001", dto.MitreId); | ||||||
|  |         Assert.Equal(8.8, dto.CvssScore); | ||||||
|  |         Assert.Equal("AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H", dto.CvssVector); | ||||||
|  |         Assert.True(dto.PatchAvailable); | ||||||
|  |         Assert.Equal(79, dto.Cwe?.Number); | ||||||
|  |         Assert.Equal(2, dto.Urls.Length); | ||||||
|  |     } | ||||||
|  | } | ||||||
| @@ -0,0 +1,68 @@ | |||||||
|  | using System.Collections.Immutable; | ||||||
|  | using MongoDB.Bson; | ||||||
|  | using StellaOps.Feedser.Source.Common; | ||||||
|  | using StellaOps.Feedser.Models; | ||||||
|  | using StellaOps.Feedser.Source.Ru.Nkcki.Internal; | ||||||
|  | using StellaOps.Feedser.Storage.Mongo.Documents; | ||||||
|  | using Xunit; | ||||||
|  | using System.Reflection; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Nkcki.Tests; | ||||||
|  |  | ||||||
|  | public sealed class RuNkckiMapperTests | ||||||
|  | { | ||||||
|  |     [Fact] | ||||||
|  |     public void Map_ConstructsCanonicalAdvisory() | ||||||
|  |     { | ||||||
|  |         var dto = new RuNkckiVulnerabilityDto( | ||||||
|  |             FstecId: "BDU:2025-00001", | ||||||
|  |             MitreId: "CVE-2025-0001", | ||||||
|  |             DatePublished: new DateTimeOffset(2025, 9, 1, 0, 0, 0, TimeSpan.Zero), | ||||||
|  |             DateUpdated: new DateTimeOffset(2025, 9, 2, 0, 0, 0, TimeSpan.Zero), | ||||||
|  |             CvssRating: "КРИТИЧЕСКИЙ", | ||||||
|  |             PatchAvailable: true, | ||||||
|  |             Description: "Test NKCKI vulnerability", | ||||||
|  |             Cwe: new RuNkckiCweDto(79, "Cross-site scripting"), | ||||||
|  |             ProductCategory: "Web", | ||||||
|  |             Mitigation: "Apply update", | ||||||
|  |             VulnerableSoftwareText: "ExampleApp <= 1.0", | ||||||
|  |             VulnerableSoftwareHasCpe: false, | ||||||
|  |             CvssScore: 8.8, | ||||||
|  |             CvssVector: "AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H", | ||||||
|  |             CvssScoreV4: null, | ||||||
|  |             CvssVectorV4: null, | ||||||
|  |             Impact: "ACE", | ||||||
|  |             MethodOfExploitation: "Special request", | ||||||
|  |             UserInteraction: false, | ||||||
|  |             Urls: ImmutableArray.Create("https://example.com/advisory")); | ||||||
|  |  | ||||||
|  |         var document = new DocumentRecord( | ||||||
|  |             Guid.NewGuid(), | ||||||
|  |             RuNkckiConnectorPlugin.SourceName, | ||||||
|  |             "https://cert.gov.ru/materialy/uyazvimosti/2025-00001", | ||||||
|  |             DateTimeOffset.UtcNow, | ||||||
|  |             "abc", | ||||||
|  |             DocumentStatuses.PendingMap, | ||||||
|  |             "application/json", | ||||||
|  |             null, | ||||||
|  |             null, | ||||||
|  |             null, | ||||||
|  |             dto.DateUpdated, | ||||||
|  |             ObjectId.GenerateNewId()); | ||||||
|  |  | ||||||
|  |         Assert.Equal("КРИТИЧЕСКИЙ", dto.CvssRating); | ||||||
|  |         var normalizeSeverity = typeof(RuNkckiMapper).GetMethod("NormalizeSeverity", BindingFlags.NonPublic | BindingFlags.Static)!; | ||||||
|  |         var ratingSeverity = (string?)normalizeSeverity.Invoke(null, new object?[] { dto.CvssRating }); | ||||||
|  |         Assert.Equal("critical", ratingSeverity); | ||||||
|  |  | ||||||
|  |         var advisory = RuNkckiMapper.Map(dto, document, dto.DateUpdated!.Value); | ||||||
|  |  | ||||||
|  |         Assert.Contains("BDU:2025-00001", advisory.Aliases); | ||||||
|  |         Assert.Contains("CVE-2025-0001", advisory.Aliases); | ||||||
|  |         Assert.Equal("critical", advisory.Severity); | ||||||
|  |         Assert.True(advisory.ExploitKnown); | ||||||
|  |         Assert.Single(advisory.AffectedPackages); | ||||||
|  |         Assert.Single(advisory.CvssMetrics); | ||||||
|  |         Assert.Contains(advisory.References, reference => reference.Url.Contains("example.com", StringComparison.OrdinalIgnoreCase)); | ||||||
|  |     } | ||||||
|  | } | ||||||
| @@ -0,0 +1,13 @@ | |||||||
|  | <Project Sdk="Microsoft.NET.Sdk"> | ||||||
|  |   <PropertyGroup> | ||||||
|  |     <TargetFramework>net10.0</TargetFramework> | ||||||
|  |     <ImplicitUsings>enable</ImplicitUsings> | ||||||
|  |     <Nullable>enable</Nullable> | ||||||
|  |   </PropertyGroup> | ||||||
|  |  | ||||||
|  |   <ItemGroup> | ||||||
|  |     <ProjectReference Include="../StellaOps.Feedser.Models/StellaOps.Feedser.Models.csproj" /> | ||||||
|  |     <ProjectReference Include="../StellaOps.Feedser.Normalization/StellaOps.Feedser.Normalization.csproj" /> | ||||||
|  |     <ProjectReference Include="../StellaOps.Feedser.Source.Ru.Nkcki/StellaOps.Feedser.Source.Ru.Nkcki.csproj" /> | ||||||
|  |   </ItemGroup> | ||||||
|  | </Project> | ||||||
| @@ -1,29 +0,0 @@ | |||||||
| using System; |  | ||||||
| using System.Threading; |  | ||||||
| using System.Threading.Tasks; |  | ||||||
| using StellaOps.Plugin; |  | ||||||
|  |  | ||||||
| namespace StellaOps.Feedser.Source.Ru.Nkcki; |  | ||||||
|  |  | ||||||
| public sealed class RuNkckiConnectorPlugin : IConnectorPlugin |  | ||||||
| { |  | ||||||
|     public string Name => "ru-nkcki"; |  | ||||||
|  |  | ||||||
|     public bool IsAvailable(IServiceProvider services) => true; |  | ||||||
|  |  | ||||||
|     public IFeedConnector Create(IServiceProvider services) => new StubConnector(Name); |  | ||||||
|  |  | ||||||
|     private sealed class StubConnector : IFeedConnector |  | ||||||
|     { |  | ||||||
|         public StubConnector(string sourceName) => SourceName = sourceName; |  | ||||||
|  |  | ||||||
|         public string SourceName { get; } |  | ||||||
|  |  | ||||||
|         public Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken) => Task.CompletedTask; |  | ||||||
|  |  | ||||||
|         public Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken) => Task.CompletedTask; |  | ||||||
|  |  | ||||||
|         public Task MapAsync(IServiceProvider services, CancellationToken cancellationToken) => Task.CompletedTask; |  | ||||||
|     } |  | ||||||
| } |  | ||||||
|  |  | ||||||
| @@ -0,0 +1,127 @@ | |||||||
|  | using System.Net; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Nkcki.Configuration; | ||||||
|  |  | ||||||
|  | /// <summary> | ||||||
|  | /// Connector options for the Russian NKTsKI bulletin ingestion pipeline. | ||||||
|  | /// </summary> | ||||||
|  | public sealed class RuNkckiOptions | ||||||
|  | { | ||||||
|  |     public const string HttpClientName = "ru-nkcki"; | ||||||
|  |  | ||||||
|  |     private static readonly TimeSpan DefaultRequestTimeout = TimeSpan.FromSeconds(90); | ||||||
|  |     private static readonly TimeSpan DefaultFailureBackoff = TimeSpan.FromMinutes(20); | ||||||
|  |     private static readonly TimeSpan DefaultListingCache = TimeSpan.FromMinutes(10); | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Base endpoint used for resolving relative resource links. | ||||||
|  |     /// </summary> | ||||||
|  |     public Uri BaseAddress { get; set; } = new("https://cert.gov.ru/", UriKind.Absolute); | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Relative path to the bulletin listing page. | ||||||
|  |     /// </summary> | ||||||
|  |     public string ListingPath { get; set; } = "materialy/uyazvimosti/"; | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Timeout applied to listing and bulletin fetch requests. | ||||||
|  |     /// </summary> | ||||||
|  |     public TimeSpan RequestTimeout { get; set; } = DefaultRequestTimeout; | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Backoff applied when the listing or attachments cannot be retrieved. | ||||||
|  |     /// </summary> | ||||||
|  |     public TimeSpan FailureBackoff { get; set; } = DefaultFailureBackoff; | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Maximum number of bulletin attachments downloaded per fetch run. | ||||||
|  |     /// </summary> | ||||||
|  |     public int MaxBulletinsPerFetch { get; set; } = 5; | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Maximum number of vulnerabilities ingested per fetch cycle across all attachments. | ||||||
|  |     /// </summary> | ||||||
|  |     public int MaxVulnerabilitiesPerFetch { get; set; } = 250; | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Maximum bulletin identifiers remembered to avoid refetching historical files. | ||||||
|  |     /// </summary> | ||||||
|  |     public int KnownBulletinCapacity { get; set; } = 512; | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Delay between sequential bulletin downloads. | ||||||
|  |     /// </summary> | ||||||
|  |     public TimeSpan RequestDelay { get; set; } = TimeSpan.FromMilliseconds(250); | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Duration the HTML listing can be cached before forcing a refetch. | ||||||
|  |     /// </summary> | ||||||
|  |     public TimeSpan ListingCacheDuration { get; set; } = DefaultListingCache; | ||||||
|  |  | ||||||
|  |     public string UserAgent { get; set; } = "StellaOps/Feedser (+https://stella-ops.org)"; | ||||||
|  |  | ||||||
|  |     public string AcceptLanguage { get; set; } = "ru-RU,ru;q=0.9,en-US;q=0.6,en;q=0.4"; | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Absolute URI for the listing page. | ||||||
|  |     /// </summary> | ||||||
|  |     public Uri ListingUri => new(BaseAddress, ListingPath); | ||||||
|  |  | ||||||
|  |     /// <summary> | ||||||
|  |     /// Optional directory for caching downloaded bulletins (relative paths resolve under the content root). | ||||||
|  |     /// </summary> | ||||||
|  |     public string? CacheDirectory { get; set; } = null; | ||||||
|  |  | ||||||
|  |     public void Validate() | ||||||
|  |     { | ||||||
|  |         if (BaseAddress is null || !BaseAddress.IsAbsoluteUri) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuNkcki BaseAddress must be an absolute URI."); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (string.IsNullOrWhiteSpace(ListingPath)) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuNkcki ListingPath must be provided."); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (RequestTimeout <= TimeSpan.Zero) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuNkcki RequestTimeout must be positive."); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (FailureBackoff < TimeSpan.Zero) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuNkcki FailureBackoff cannot be negative."); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (MaxBulletinsPerFetch <= 0) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuNkcki MaxBulletinsPerFetch must be greater than zero."); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (MaxVulnerabilitiesPerFetch <= 0) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuNkcki MaxVulnerabilitiesPerFetch must be greater than zero."); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (KnownBulletinCapacity <= 0) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuNkcki KnownBulletinCapacity must be greater than zero."); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (CacheDirectory is not null && CacheDirectory.Trim().Length == 0) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuNkcki CacheDirectory cannot be whitespace."); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (string.IsNullOrWhiteSpace(UserAgent)) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuNkcki UserAgent cannot be empty."); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (string.IsNullOrWhiteSpace(AcceptLanguage)) | ||||||
|  |         { | ||||||
|  |             throw new InvalidOperationException("RuNkcki AcceptLanguage cannot be empty."); | ||||||
|  |         } | ||||||
|  |     } | ||||||
|  | } | ||||||
							
								
								
									
										108
									
								
								src/StellaOps.Feedser.Source.Ru.Nkcki/Internal/RuNkckiCursor.cs
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										108
									
								
								src/StellaOps.Feedser.Source.Ru.Nkcki/Internal/RuNkckiCursor.cs
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,108 @@ | |||||||
|  | using MongoDB.Bson; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Nkcki.Internal; | ||||||
|  |  | ||||||
|  | internal sealed record RuNkckiCursor( | ||||||
|  |     IReadOnlyCollection<Guid> PendingDocuments, | ||||||
|  |     IReadOnlyCollection<Guid> PendingMappings, | ||||||
|  |     IReadOnlyCollection<string> KnownBulletins, | ||||||
|  |     DateTimeOffset? LastListingFetchAt) | ||||||
|  | { | ||||||
|  |     private static readonly IReadOnlyCollection<Guid> EmptyGuids = Array.Empty<Guid>(); | ||||||
|  |     private static readonly IReadOnlyCollection<string> EmptyBulletins = Array.Empty<string>(); | ||||||
|  |  | ||||||
|  |     public static RuNkckiCursor Empty { get; } = new(EmptyGuids, EmptyGuids, EmptyBulletins, null); | ||||||
|  |  | ||||||
|  |     public RuNkckiCursor WithPendingDocuments(IEnumerable<Guid> documents) | ||||||
|  |         => this with { PendingDocuments = (documents ?? Enumerable.Empty<Guid>()).Distinct().ToArray() }; | ||||||
|  |  | ||||||
|  |     public RuNkckiCursor WithPendingMappings(IEnumerable<Guid> mappings) | ||||||
|  |         => this with { PendingMappings = (mappings ?? Enumerable.Empty<Guid>()).Distinct().ToArray() }; | ||||||
|  |  | ||||||
|  |     public RuNkckiCursor WithKnownBulletins(IEnumerable<string> bulletins) | ||||||
|  |         => this with { KnownBulletins = (bulletins ?? Enumerable.Empty<string>()).Where(static id => !string.IsNullOrWhiteSpace(id)).Distinct(StringComparer.OrdinalIgnoreCase).ToArray() }; | ||||||
|  |  | ||||||
|  |     public RuNkckiCursor WithLastListingFetch(DateTimeOffset? timestamp) | ||||||
|  |         => this with { LastListingFetchAt = timestamp }; | ||||||
|  |  | ||||||
|  |     public BsonDocument ToBsonDocument() | ||||||
|  |     { | ||||||
|  |         var document = new BsonDocument | ||||||
|  |         { | ||||||
|  |             ["pendingDocuments"] = new BsonArray(PendingDocuments.Select(id => id.ToString())), | ||||||
|  |             ["pendingMappings"] = new BsonArray(PendingMappings.Select(id => id.ToString())), | ||||||
|  |             ["knownBulletins"] = new BsonArray(KnownBulletins), | ||||||
|  |         }; | ||||||
|  |  | ||||||
|  |         if (LastListingFetchAt.HasValue) | ||||||
|  |         { | ||||||
|  |             document["lastListingFetchAt"] = LastListingFetchAt.Value.UtcDateTime; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return document; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     public static RuNkckiCursor FromBson(BsonDocument? document) | ||||||
|  |     { | ||||||
|  |         if (document is null || document.ElementCount == 0) | ||||||
|  |         { | ||||||
|  |             return Empty; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var pendingDocuments = ReadGuidArray(document, "pendingDocuments"); | ||||||
|  |         var pendingMappings = ReadGuidArray(document, "pendingMappings"); | ||||||
|  |         var knownBulletins = ReadStringArray(document, "knownBulletins"); | ||||||
|  |         var lastListingFetch = document.TryGetValue("lastListingFetchAt", out var dateValue) | ||||||
|  |             ? ParseDate(dateValue) | ||||||
|  |             : null; | ||||||
|  |  | ||||||
|  |         return new RuNkckiCursor(pendingDocuments, pendingMappings, knownBulletins, lastListingFetch); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static IReadOnlyCollection<Guid> ReadGuidArray(BsonDocument document, string field) | ||||||
|  |     { | ||||||
|  |         if (!document.TryGetValue(field, out var value) || value is not BsonArray array) | ||||||
|  |         { | ||||||
|  |             return EmptyGuids; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var result = new List<Guid>(array.Count); | ||||||
|  |         foreach (var element in array) | ||||||
|  |         { | ||||||
|  |             if (Guid.TryParse(element?.ToString(), out var guid)) | ||||||
|  |             { | ||||||
|  |                 result.Add(guid); | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return result; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static IReadOnlyCollection<string> ReadStringArray(BsonDocument document, string field) | ||||||
|  |     { | ||||||
|  |         if (!document.TryGetValue(field, out var value) || value is not BsonArray array) | ||||||
|  |         { | ||||||
|  |             return EmptyBulletins; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var result = new List<string>(array.Count); | ||||||
|  |         foreach (var element in array) | ||||||
|  |         { | ||||||
|  |             var text = element?.ToString(); | ||||||
|  |             if (!string.IsNullOrWhiteSpace(text)) | ||||||
|  |             { | ||||||
|  |                 result.Add(text); | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return result; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static DateTimeOffset? ParseDate(BsonValue value) | ||||||
|  |         => value.BsonType switch | ||||||
|  |         { | ||||||
|  |             BsonType.DateTime => DateTime.SpecifyKind(value.ToUniversalTime(), DateTimeKind.Utc), | ||||||
|  |             BsonType.String when DateTimeOffset.TryParse(value.AsString, out var parsed) => parsed.ToUniversalTime(), | ||||||
|  |             _ => null, | ||||||
|  |         }; | ||||||
|  | } | ||||||
| @@ -0,0 +1,169 @@ | |||||||
|  | using System.Collections.Immutable; | ||||||
|  | using System.Linq; | ||||||
|  | using System.Globalization; | ||||||
|  | using System.Text.Json; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Nkcki.Internal; | ||||||
|  |  | ||||||
|  | internal static class RuNkckiJsonParser | ||||||
|  | { | ||||||
|  |     public static RuNkckiVulnerabilityDto Parse(JsonElement element) | ||||||
|  |     { | ||||||
|  |         var fstecId = element.TryGetProperty("vuln_id", out var vulnIdElement) && vulnIdElement.TryGetProperty("FSTEC", out var fstec) ? Normalize(fstec.GetString()) : null; | ||||||
|  |         var mitreId = element.TryGetProperty("vuln_id", out vulnIdElement) && vulnIdElement.TryGetProperty("MITRE", out var mitre) ? Normalize(mitre.GetString()) : null; | ||||||
|  |  | ||||||
|  |         var datePublished = ParseDate(element.TryGetProperty("date_published", out var published) ? published.GetString() : null); | ||||||
|  |         var dateUpdated = ParseDate(element.TryGetProperty("date_updated", out var updated) ? updated.GetString() : null); | ||||||
|  |         var cvssRating = Normalize(element.TryGetProperty("cvss_rating", out var rating) ? rating.GetString() : null); | ||||||
|  |         bool? patchAvailable = element.TryGetProperty("patch_available", out var patch) ? patch.ValueKind switch | ||||||
|  |         { | ||||||
|  |             JsonValueKind.True => true, | ||||||
|  |             JsonValueKind.False => false, | ||||||
|  |             _ => null, | ||||||
|  |         } : null; | ||||||
|  |  | ||||||
|  |         var description = Normalize(element.TryGetProperty("description", out var desc) ? desc.GetString() : null); | ||||||
|  |         var mitigation = Normalize(element.TryGetProperty("mitigation", out var mitigationElement) ? mitigationElement.GetString() : null); | ||||||
|  |         var productCategory = Normalize(element.TryGetProperty("product_category", out var category) ? category.GetString() : null); | ||||||
|  |         var impact = Normalize(element.TryGetProperty("impact", out var impactElement) ? impactElement.GetString() : null); | ||||||
|  |         var method = Normalize(element.TryGetProperty("method_of_exploitation", out var methodElement) ? methodElement.GetString() : null); | ||||||
|  |  | ||||||
|  |         bool? userInteraction = element.TryGetProperty("user_interaction", out var uiElement) ? uiElement.ValueKind switch | ||||||
|  |         { | ||||||
|  |             JsonValueKind.True => true, | ||||||
|  |             JsonValueKind.False => false, | ||||||
|  |             _ => null, | ||||||
|  |         } : null; | ||||||
|  |  | ||||||
|  |         string? softwareText = null; | ||||||
|  |         bool? softwareHasCpe = null; | ||||||
|  |         if (element.TryGetProperty("vulnerable_software", out var softwareElement)) | ||||||
|  |         { | ||||||
|  |             if (softwareElement.TryGetProperty("software_text", out var textElement)) | ||||||
|  |             { | ||||||
|  |                 softwareText = Normalize(textElement.GetString()?.Replace('\r', ' ')); | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             if (softwareElement.TryGetProperty("cpe", out var cpeElement)) | ||||||
|  |             { | ||||||
|  |                 softwareHasCpe = cpeElement.ValueKind switch | ||||||
|  |                 { | ||||||
|  |                     JsonValueKind.True => true, | ||||||
|  |                     JsonValueKind.False => false, | ||||||
|  |                     _ => null, | ||||||
|  |                 }; | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         RuNkckiCweDto? cweDto = null; | ||||||
|  |         if (element.TryGetProperty("cwe", out var cweElement)) | ||||||
|  |         { | ||||||
|  |             int? number = null; | ||||||
|  |             if (cweElement.TryGetProperty("cwe_number", out var numberElement)) | ||||||
|  |             { | ||||||
|  |                 if (numberElement.ValueKind == JsonValueKind.Number && numberElement.TryGetInt32(out var parsed)) | ||||||
|  |                 { | ||||||
|  |                     number = parsed; | ||||||
|  |                 } | ||||||
|  |                 else if (int.TryParse(numberElement.GetString(), NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsedInt)) | ||||||
|  |                 { | ||||||
|  |                     number = parsedInt; | ||||||
|  |                 } | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             var cweDescription = Normalize(cweElement.TryGetProperty("cwe_description", out var descElement) ? descElement.GetString() : null); | ||||||
|  |             if (number.HasValue || !string.IsNullOrWhiteSpace(cweDescription)) | ||||||
|  |             { | ||||||
|  |                 cweDto = new RuNkckiCweDto(number, cweDescription); | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         double? cvssScore = element.TryGetProperty("cvss", out var cvssElement) && cvssElement.TryGetProperty("cvss_score", out var scoreElement) | ||||||
|  |             ? ParseDouble(scoreElement) | ||||||
|  |             : null; | ||||||
|  |         var cvssVector = element.TryGetProperty("cvss", out cvssElement) && cvssElement.TryGetProperty("cvss_vector", out var vectorElement) | ||||||
|  |             ? Normalize(vectorElement.GetString()) | ||||||
|  |             : null; | ||||||
|  |         double? cvssScoreV4 = element.TryGetProperty("cvss", out cvssElement) && cvssElement.TryGetProperty("cvss_score_v4", out var scoreV4Element) | ||||||
|  |             ? ParseDouble(scoreV4Element) | ||||||
|  |             : null; | ||||||
|  |         var cvssVectorV4 = element.TryGetProperty("cvss", out cvssElement) && cvssElement.TryGetProperty("cvss_vector_v4", out var vectorV4Element) | ||||||
|  |             ? Normalize(vectorV4Element.GetString()) | ||||||
|  |             : null; | ||||||
|  |  | ||||||
|  |         var urls = element.TryGetProperty("urls", out var urlsElement) && urlsElement.ValueKind == JsonValueKind.Array | ||||||
|  |             ? urlsElement.EnumerateArray() | ||||||
|  |                 .Select(static url => Normalize(url.GetString())) | ||||||
|  |                 .Where(static url => !string.IsNullOrWhiteSpace(url)) | ||||||
|  |                 .Cast<string>() | ||||||
|  |                 .ToImmutableArray() | ||||||
|  |             : ImmutableArray<string>.Empty; | ||||||
|  |  | ||||||
|  |         return new RuNkckiVulnerabilityDto( | ||||||
|  |             fstecId, | ||||||
|  |             mitreId, | ||||||
|  |             datePublished, | ||||||
|  |             dateUpdated, | ||||||
|  |             cvssRating, | ||||||
|  |             patchAvailable, | ||||||
|  |             description, | ||||||
|  |             cweDto, | ||||||
|  |             productCategory, | ||||||
|  |             mitigation, | ||||||
|  |             softwareText, | ||||||
|  |             softwareHasCpe, | ||||||
|  |             cvssScore, | ||||||
|  |             cvssVector, | ||||||
|  |             cvssScoreV4, | ||||||
|  |             cvssVectorV4, | ||||||
|  |             impact, | ||||||
|  |             method, | ||||||
|  |             userInteraction, | ||||||
|  |             urls); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static double? ParseDouble(JsonElement element) | ||||||
|  |     { | ||||||
|  |         if (element.ValueKind == JsonValueKind.Number && element.TryGetDouble(out var value)) | ||||||
|  |         { | ||||||
|  |             return value; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (element.ValueKind == JsonValueKind.String && double.TryParse(element.GetString(), NumberStyles.Any, CultureInfo.InvariantCulture, out var parsed)) | ||||||
|  |         { | ||||||
|  |             return parsed; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return null; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static DateTimeOffset? ParseDate(string? value) | ||||||
|  |     { | ||||||
|  |         if (string.IsNullOrWhiteSpace(value)) | ||||||
|  |         { | ||||||
|  |             return null; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (DateTimeOffset.TryParse(value, CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, out var parsed)) | ||||||
|  |         { | ||||||
|  |             return parsed; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (DateTimeOffset.TryParse(value, CultureInfo.GetCultureInfo("ru-RU"), DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, out var ruParsed)) | ||||||
|  |         { | ||||||
|  |             return ruParsed; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return null; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static string? Normalize(string? value) | ||||||
|  |     { | ||||||
|  |         if (string.IsNullOrWhiteSpace(value)) | ||||||
|  |         { | ||||||
|  |             return null; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return value.Replace('\r', ' ').Replace('\n', ' ').Trim(); | ||||||
|  |     } | ||||||
|  | } | ||||||
							
								
								
									
										298
									
								
								src/StellaOps.Feedser.Source.Ru.Nkcki/Internal/RuNkckiMapper.cs
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										298
									
								
								src/StellaOps.Feedser.Source.Ru.Nkcki/Internal/RuNkckiMapper.cs
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,298 @@ | |||||||
|  | using System.Collections.Generic; | ||||||
|  | using System.Collections.Immutable; | ||||||
|  | using System.Globalization; | ||||||
|  | using System.Linq; | ||||||
|  | using StellaOps.Feedser.Models; | ||||||
|  | using StellaOps.Feedser.Normalization.Cvss; | ||||||
|  | using StellaOps.Feedser.Storage.Mongo.Documents; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Nkcki.Internal; | ||||||
|  |  | ||||||
|  | internal static class RuNkckiMapper | ||||||
|  | { | ||||||
|  |     private static readonly ImmutableDictionary<string, string> SeverityLookup = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase) | ||||||
|  |     { | ||||||
|  |         ["критический"] = "critical", | ||||||
|  |         ["высокий"] = "high", | ||||||
|  |         ["средний"] = "medium", | ||||||
|  |         ["умеренный"] = "medium", | ||||||
|  |         ["низкий"] = "low", | ||||||
|  |         ["информационный"] = "informational", | ||||||
|  |     }.ToImmutableDictionary(StringComparer.OrdinalIgnoreCase); | ||||||
|  |  | ||||||
|  |     public static Advisory Map(RuNkckiVulnerabilityDto dto, DocumentRecord document, DateTimeOffset recordedAt) | ||||||
|  |     { | ||||||
|  |         ArgumentNullException.ThrowIfNull(dto); | ||||||
|  |         ArgumentNullException.ThrowIfNull(document); | ||||||
|  |  | ||||||
|  |         var advisoryProvenance = new AdvisoryProvenance( | ||||||
|  |             RuNkckiConnectorPlugin.SourceName, | ||||||
|  |             "advisory", | ||||||
|  |             dto.AdvisoryKey, | ||||||
|  |             recordedAt, | ||||||
|  |             new[] { ProvenanceFieldMasks.Advisory }); | ||||||
|  |  | ||||||
|  |         var aliases = BuildAliases(dto); | ||||||
|  |         var references = BuildReferences(dto, document, recordedAt); | ||||||
|  |         var packages = BuildPackages(dto, recordedAt); | ||||||
|  |         var cvssMetrics = BuildCvssMetrics(dto, recordedAt, out var severityFromCvss); | ||||||
|  |         var severityFromRating = NormalizeSeverity(dto.CvssRating); | ||||||
|  |         var severity = severityFromRating ?? severityFromCvss; | ||||||
|  |  | ||||||
|  |         if (severityFromRating is not null && severityFromCvss is not null) | ||||||
|  |         { | ||||||
|  |             severity = ChooseMoreSevere(severityFromRating, severityFromCvss); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var exploitKnown = DetermineExploitKnown(dto); | ||||||
|  |  | ||||||
|  |         return new Advisory( | ||||||
|  |             advisoryKey: dto.AdvisoryKey, | ||||||
|  |             title: dto.Description ?? dto.AdvisoryKey, | ||||||
|  |             summary: dto.Description, | ||||||
|  |             language: "ru", | ||||||
|  |             published: dto.DatePublished, | ||||||
|  |             modified: dto.DateUpdated, | ||||||
|  |             severity: severity, | ||||||
|  |             exploitKnown: exploitKnown, | ||||||
|  |             aliases: aliases, | ||||||
|  |             references: references, | ||||||
|  |             affectedPackages: packages, | ||||||
|  |             cvssMetrics: cvssMetrics, | ||||||
|  |             provenance: new[] { advisoryProvenance }); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static IReadOnlyList<string> BuildAliases(RuNkckiVulnerabilityDto dto) | ||||||
|  |     { | ||||||
|  |         var aliases = new HashSet<string>(StringComparer.OrdinalIgnoreCase); | ||||||
|  |         if (!string.IsNullOrWhiteSpace(dto.FstecId)) | ||||||
|  |         { | ||||||
|  |             aliases.Add(dto.FstecId!); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (!string.IsNullOrWhiteSpace(dto.MitreId)) | ||||||
|  |         { | ||||||
|  |             aliases.Add(dto.MitreId!); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return aliases.ToImmutableSortedSet(StringComparer.Ordinal).ToImmutableArray(); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static IReadOnlyList<AdvisoryReference> BuildReferences(RuNkckiVulnerabilityDto dto, DocumentRecord document, DateTimeOffset recordedAt) | ||||||
|  |     { | ||||||
|  |         var references = new List<AdvisoryReference> | ||||||
|  |         { | ||||||
|  |             new(document.Uri, "details", "ru-nkcki", summary: null, new AdvisoryProvenance( | ||||||
|  |                 RuNkckiConnectorPlugin.SourceName, | ||||||
|  |                 "reference", | ||||||
|  |                 document.Uri, | ||||||
|  |                 recordedAt, | ||||||
|  |                 new[] { ProvenanceFieldMasks.References })) | ||||||
|  |         }; | ||||||
|  |  | ||||||
|  |         if (!string.IsNullOrWhiteSpace(dto.FstecId)) | ||||||
|  |         { | ||||||
|  |             var slug = dto.FstecId!.Contains(':', StringComparison.Ordinal) | ||||||
|  |                 ? dto.FstecId[(dto.FstecId.IndexOf(':') + 1)..] | ||||||
|  |                 : dto.FstecId; | ||||||
|  |             var bduUrl = $"https://bdu.fstec.ru/vul/{slug}"; | ||||||
|  |             references.Add(new AdvisoryReference(bduUrl, "details", "bdu", summary: null, new AdvisoryProvenance( | ||||||
|  |                 RuNkckiConnectorPlugin.SourceName, | ||||||
|  |                 "reference", | ||||||
|  |                 bduUrl, | ||||||
|  |                 recordedAt, | ||||||
|  |                 new[] { ProvenanceFieldMasks.References }))); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         foreach (var url in dto.Urls) | ||||||
|  |         { | ||||||
|  |             if (string.IsNullOrWhiteSpace(url)) | ||||||
|  |             { | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             var kind = url.Contains("cert.gov.ru", StringComparison.OrdinalIgnoreCase) ? "details" : "external"; | ||||||
|  |             var sourceTag = url.Contains("siemens", StringComparison.OrdinalIgnoreCase) ? "vendor" : null; | ||||||
|  |             references.Add(new AdvisoryReference(url, kind, sourceTag, summary: null, new AdvisoryProvenance( | ||||||
|  |                 RuNkckiConnectorPlugin.SourceName, | ||||||
|  |                 "reference", | ||||||
|  |                 url, | ||||||
|  |                 recordedAt, | ||||||
|  |                 new[] { ProvenanceFieldMasks.References }))); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (dto.Cwe?.Number is int number) | ||||||
|  |         { | ||||||
|  |             var url = $"https://cwe.mitre.org/data/definitions/{number}.html"; | ||||||
|  |             references.Add(new AdvisoryReference(url, "cwe", "cwe", dto.Cwe.Description, new AdvisoryProvenance( | ||||||
|  |                 RuNkckiConnectorPlugin.SourceName, | ||||||
|  |                 "reference", | ||||||
|  |                 url, | ||||||
|  |                 recordedAt, | ||||||
|  |                 new[] { ProvenanceFieldMasks.References }))); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return references; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static IReadOnlyList<AffectedPackage> BuildPackages(RuNkckiVulnerabilityDto dto, DateTimeOffset recordedAt) | ||||||
|  |     { | ||||||
|  |         if (string.IsNullOrWhiteSpace(dto.VulnerableSoftwareText)) | ||||||
|  |         { | ||||||
|  |             return Array.Empty<AffectedPackage>(); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var identifier = dto.VulnerableSoftwareText!.Replace('\n', ' ').Replace('\r', ' ').Trim(); | ||||||
|  |         if (identifier.Length == 0) | ||||||
|  |         { | ||||||
|  |             return Array.Empty<AffectedPackage>(); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var packageProvenance = new AdvisoryProvenance( | ||||||
|  |             RuNkckiConnectorPlugin.SourceName, | ||||||
|  |             "package", | ||||||
|  |             identifier, | ||||||
|  |             recordedAt, | ||||||
|  |             new[] { ProvenanceFieldMasks.AffectedPackages }); | ||||||
|  |  | ||||||
|  |         var status = new AffectedPackageStatus( | ||||||
|  |             dto.PatchAvailable == true ? AffectedPackageStatusCatalog.Fixed : AffectedPackageStatusCatalog.Affected, | ||||||
|  |             new AdvisoryProvenance( | ||||||
|  |                 RuNkckiConnectorPlugin.SourceName, | ||||||
|  |                 "package-status", | ||||||
|  |                 dto.PatchAvailable == true ? "patch_available" : "affected", | ||||||
|  |                 recordedAt, | ||||||
|  |                 new[] { ProvenanceFieldMasks.PackageStatuses })); | ||||||
|  |  | ||||||
|  |         return new[] | ||||||
|  |         { | ||||||
|  |             new AffectedPackage( | ||||||
|  |                 dto.VulnerableSoftwareHasCpe == true ? AffectedPackageTypes.Cpe : AffectedPackageTypes.Vendor, | ||||||
|  |                 identifier, | ||||||
|  |                 platform: null, | ||||||
|  |                 versionRanges: null, | ||||||
|  |                 statuses: new[] { status }, | ||||||
|  |                 provenance: new[] { packageProvenance }) | ||||||
|  |         }; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static IReadOnlyList<CvssMetric> BuildCvssMetrics(RuNkckiVulnerabilityDto dto, DateTimeOffset recordedAt, out string? severity) | ||||||
|  |     { | ||||||
|  |         severity = null; | ||||||
|  |         var metrics = new List<CvssMetric>(); | ||||||
|  |  | ||||||
|  |         if (!string.IsNullOrWhiteSpace(dto.CvssVector) && CvssMetricNormalizer.TryNormalize(null, dto.CvssVector, dto.CvssScore, null, out var normalized)) | ||||||
|  |         { | ||||||
|  |             var provenance = new AdvisoryProvenance( | ||||||
|  |                 RuNkckiConnectorPlugin.SourceName, | ||||||
|  |                 "cvss", | ||||||
|  |                 normalized.Vector, | ||||||
|  |                 recordedAt, | ||||||
|  |                 new[] { ProvenanceFieldMasks.CvssMetrics }); | ||||||
|  |             var metric = normalized.ToModel(provenance); | ||||||
|  |             metrics.Add(metric); | ||||||
|  |             severity ??= metric.BaseSeverity; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return metrics; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static string? NormalizeSeverity(string? rating) | ||||||
|  |     { | ||||||
|  |         if (string.IsNullOrWhiteSpace(rating)) | ||||||
|  |         { | ||||||
|  |             return null; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var normalized = rating.Trim().ToLowerInvariant(); | ||||||
|  |  | ||||||
|  |         if (SeverityLookup.TryGetValue(normalized, out var mapped)) | ||||||
|  |         { | ||||||
|  |             return mapped; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (normalized.StartsWith("крит", StringComparison.Ordinal)) | ||||||
|  |         { | ||||||
|  |             return "critical"; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (normalized.StartsWith("высок", StringComparison.Ordinal)) | ||||||
|  |         { | ||||||
|  |             return "high"; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (normalized.StartsWith("сред", StringComparison.Ordinal) || normalized.StartsWith("умер", StringComparison.Ordinal)) | ||||||
|  |         { | ||||||
|  |             return "medium"; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (normalized.StartsWith("низк", StringComparison.Ordinal)) | ||||||
|  |         { | ||||||
|  |             return "low"; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (normalized.StartsWith("информ", StringComparison.Ordinal)) | ||||||
|  |         { | ||||||
|  |             return "informational"; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return null; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static string ChooseMoreSevere(string first, string second) | ||||||
|  |     { | ||||||
|  |         var order = new[] { "critical", "high", "medium", "low", "informational" }; | ||||||
|  |  | ||||||
|  |         static int IndexOf(ReadOnlySpan<string> levels, string value) | ||||||
|  |         { | ||||||
|  |             for (var i = 0; i < levels.Length; i++) | ||||||
|  |             { | ||||||
|  |                 if (string.Equals(levels[i], value, StringComparison.OrdinalIgnoreCase)) | ||||||
|  |                 { | ||||||
|  |                     return i; | ||||||
|  |                 } | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             return -1; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var firstIndex = IndexOf(order.AsSpan(), first); | ||||||
|  |         var secondIndex = IndexOf(order.AsSpan(), second); | ||||||
|  |  | ||||||
|  |         if (firstIndex == -1 && secondIndex == -1) | ||||||
|  |         { | ||||||
|  |             return first; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (firstIndex == -1) | ||||||
|  |         { | ||||||
|  |             return second; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (secondIndex == -1) | ||||||
|  |         { | ||||||
|  |             return first; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return firstIndex <= secondIndex ? first : second; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static bool DetermineExploitKnown(RuNkckiVulnerabilityDto dto) | ||||||
|  |     { | ||||||
|  |         if (!string.IsNullOrWhiteSpace(dto.MethodOfExploitation)) | ||||||
|  |         { | ||||||
|  |             return true; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (!string.IsNullOrWhiteSpace(dto.Impact)) | ||||||
|  |         { | ||||||
|  |             var impact = dto.Impact.Trim().ToUpperInvariant(); | ||||||
|  |             if (impact is "ACE" or "RCE" or "LPE") | ||||||
|  |             { | ||||||
|  |                 return true; | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return false; | ||||||
|  |     } | ||||||
|  | } | ||||||
| @@ -0,0 +1,36 @@ | |||||||
|  | using System.Collections.Immutable; | ||||||
|  | using System.Text.Json.Serialization; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Nkcki.Internal; | ||||||
|  |  | ||||||
|  | internal sealed record RuNkckiVulnerabilityDto( | ||||||
|  |     string? FstecId, | ||||||
|  |     string? MitreId, | ||||||
|  |     DateTimeOffset? DatePublished, | ||||||
|  |     DateTimeOffset? DateUpdated, | ||||||
|  |     string? CvssRating, | ||||||
|  |     bool? PatchAvailable, | ||||||
|  |     string? Description, | ||||||
|  |     RuNkckiCweDto? Cwe, | ||||||
|  |     string? ProductCategory, | ||||||
|  |     string? Mitigation, | ||||||
|  |     string? VulnerableSoftwareText, | ||||||
|  |     bool? VulnerableSoftwareHasCpe, | ||||||
|  |     double? CvssScore, | ||||||
|  |     string? CvssVector, | ||||||
|  |     double? CvssScoreV4, | ||||||
|  |     string? CvssVectorV4, | ||||||
|  |     string? Impact, | ||||||
|  |     string? MethodOfExploitation, | ||||||
|  |     bool? UserInteraction, | ||||||
|  |     ImmutableArray<string> Urls) | ||||||
|  | { | ||||||
|  |     [JsonIgnore] | ||||||
|  |     public string AdvisoryKey => !string.IsNullOrWhiteSpace(FstecId) | ||||||
|  |         ? FstecId! | ||||||
|  |         : !string.IsNullOrWhiteSpace(MitreId) | ||||||
|  |             ? MitreId! | ||||||
|  |             : Guid.NewGuid().ToString(); | ||||||
|  | } | ||||||
|  |  | ||||||
|  | internal sealed record RuNkckiCweDto(int? Number, string? Description); | ||||||
							
								
								
									
										43
									
								
								src/StellaOps.Feedser.Source.Ru.Nkcki/Jobs.cs
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										43
									
								
								src/StellaOps.Feedser.Source.Ru.Nkcki/Jobs.cs
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,43 @@ | |||||||
|  | using StellaOps.Feedser.Core.Jobs; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Nkcki; | ||||||
|  |  | ||||||
|  | internal static class RuNkckiJobKinds | ||||||
|  | { | ||||||
|  |     public const string Fetch = "source:ru-nkcki:fetch"; | ||||||
|  |     public const string Parse = "source:ru-nkcki:parse"; | ||||||
|  |     public const string Map = "source:ru-nkcki:map"; | ||||||
|  | } | ||||||
|  |  | ||||||
|  | internal sealed class RuNkckiFetchJob : IJob | ||||||
|  | { | ||||||
|  |     private readonly RuNkckiConnector _connector; | ||||||
|  |  | ||||||
|  |     public RuNkckiFetchJob(RuNkckiConnector connector) | ||||||
|  |         => _connector = connector ?? throw new ArgumentNullException(nameof(connector)); | ||||||
|  |  | ||||||
|  |     public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken) | ||||||
|  |         => _connector.FetchAsync(context.Services, cancellationToken); | ||||||
|  | } | ||||||
|  |  | ||||||
|  | internal sealed class RuNkckiParseJob : IJob | ||||||
|  | { | ||||||
|  |     private readonly RuNkckiConnector _connector; | ||||||
|  |  | ||||||
|  |     public RuNkckiParseJob(RuNkckiConnector connector) | ||||||
|  |         => _connector = connector ?? throw new ArgumentNullException(nameof(connector)); | ||||||
|  |  | ||||||
|  |     public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken) | ||||||
|  |         => _connector.ParseAsync(context.Services, cancellationToken); | ||||||
|  | } | ||||||
|  |  | ||||||
|  | internal sealed class RuNkckiMapJob : IJob | ||||||
|  | { | ||||||
|  |     private readonly RuNkckiConnector _connector; | ||||||
|  |  | ||||||
|  |     public RuNkckiMapJob(RuNkckiConnector connector) | ||||||
|  |         => _connector = connector ?? throw new ArgumentNullException(nameof(connector)); | ||||||
|  |  | ||||||
|  |     public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken) | ||||||
|  |         => _connector.MapAsync(context.Services, cancellationToken); | ||||||
|  | } | ||||||
| @@ -0,0 +1,3 @@ | |||||||
|  | using System.Runtime.CompilerServices; | ||||||
|  |  | ||||||
|  | [assembly: InternalsVisibleTo("StellaOps.Feedser.Source.Ru.Nkcki.Tests")] | ||||||
							
								
								
									
										825
									
								
								src/StellaOps.Feedser.Source.Ru.Nkcki/RuNkckiConnector.cs
									
									
									
									
									
										Normal file
									
								
							
							
						
						
									
										825
									
								
								src/StellaOps.Feedser.Source.Ru.Nkcki/RuNkckiConnector.cs
									
									
									
									
									
										Normal file
									
								
							| @@ -0,0 +1,825 @@ | |||||||
|  | using System.Collections.Immutable; | ||||||
|  | using System.Collections.Generic; | ||||||
|  | using System.IO; | ||||||
|  | using System.IO.Compression; | ||||||
|  | using System.Net; | ||||||
|  | using System.Linq; | ||||||
|  | using System.Security.Cryptography; | ||||||
|  | using System.Text; | ||||||
|  | using System.Text.Json; | ||||||
|  | using System.Text.Json.Serialization; | ||||||
|  | using AngleSharp.Html.Parser; | ||||||
|  | using Microsoft.Extensions.Logging; | ||||||
|  | using Microsoft.Extensions.Options; | ||||||
|  | using MongoDB.Bson; | ||||||
|  | using StellaOps.Feedser.Source.Common; | ||||||
|  | using StellaOps.Feedser.Source.Common.Fetch; | ||||||
|  | using StellaOps.Feedser.Source.Ru.Nkcki.Configuration; | ||||||
|  | using StellaOps.Feedser.Source.Ru.Nkcki.Internal; | ||||||
|  | using StellaOps.Feedser.Storage.Mongo; | ||||||
|  | using StellaOps.Feedser.Storage.Mongo.Advisories; | ||||||
|  | using StellaOps.Feedser.Storage.Mongo.Documents; | ||||||
|  | using StellaOps.Feedser.Storage.Mongo.Dtos; | ||||||
|  | using StellaOps.Plugin; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Nkcki; | ||||||
|  |  | ||||||
|  | public sealed class RuNkckiConnector : IFeedConnector | ||||||
|  | { | ||||||
|  |     private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web) | ||||||
|  |     { | ||||||
|  |         DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull, | ||||||
|  |         PropertyNamingPolicy = JsonNamingPolicy.CamelCase, | ||||||
|  |         WriteIndented = false, | ||||||
|  |     }; | ||||||
|  |  | ||||||
|  |     private static readonly string[] ListingAcceptHeaders = | ||||||
|  |     { | ||||||
|  |         "text/html", | ||||||
|  |         "application/xhtml+xml;q=0.9", | ||||||
|  |         "text/plain;q=0.1", | ||||||
|  |     }; | ||||||
|  |  | ||||||
|  |     private static readonly string[] BulletinAcceptHeaders = | ||||||
|  |     { | ||||||
|  |         "application/zip", | ||||||
|  |         "application/octet-stream", | ||||||
|  |         "application/x-zip-compressed", | ||||||
|  |     }; | ||||||
|  |  | ||||||
|  |     private readonly SourceFetchService _fetchService; | ||||||
|  |     private readonly RawDocumentStorage _rawDocumentStorage; | ||||||
|  |     private readonly IDocumentStore _documentStore; | ||||||
|  |     private readonly IDtoStore _dtoStore; | ||||||
|  |     private readonly IAdvisoryStore _advisoryStore; | ||||||
|  |     private readonly ISourceStateRepository _stateRepository; | ||||||
|  |     private readonly RuNkckiOptions _options; | ||||||
|  |     private readonly TimeProvider _timeProvider; | ||||||
|  |     private readonly ILogger<RuNkckiConnector> _logger; | ||||||
|  |     private readonly string _cacheDirectory; | ||||||
|  |  | ||||||
|  |     private readonly HtmlParser _htmlParser = new(); | ||||||
|  |  | ||||||
|  |     public RuNkckiConnector( | ||||||
|  |         SourceFetchService fetchService, | ||||||
|  |         RawDocumentStorage rawDocumentStorage, | ||||||
|  |         IDocumentStore documentStore, | ||||||
|  |         IDtoStore dtoStore, | ||||||
|  |         IAdvisoryStore advisoryStore, | ||||||
|  |         ISourceStateRepository stateRepository, | ||||||
|  |         IOptions<RuNkckiOptions> options, | ||||||
|  |         TimeProvider? timeProvider, | ||||||
|  |         ILogger<RuNkckiConnector> logger) | ||||||
|  |     { | ||||||
|  |         _fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService)); | ||||||
|  |         _rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage)); | ||||||
|  |         _documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore)); | ||||||
|  |         _dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore)); | ||||||
|  |         _advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore)); | ||||||
|  |         _stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository)); | ||||||
|  |         _options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options)); | ||||||
|  |         _options.Validate(); | ||||||
|  |         _timeProvider = timeProvider ?? TimeProvider.System; | ||||||
|  |         _logger = logger ?? throw new ArgumentNullException(nameof(logger)); | ||||||
|  |         _cacheDirectory = ResolveCacheDirectory(_options.CacheDirectory); | ||||||
|  |         EnsureCacheDirectory(); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     public string SourceName => RuNkckiConnectorPlugin.SourceName; | ||||||
|  |  | ||||||
|  |     public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken) | ||||||
|  |     { | ||||||
|  |         ArgumentNullException.ThrowIfNull(services); | ||||||
|  |  | ||||||
|  |         var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false); | ||||||
|  |         var pendingDocuments = cursor.PendingDocuments.ToHashSet(); | ||||||
|  |         var pendingMappings = cursor.PendingMappings.ToHashSet(); | ||||||
|  |         var knownBulletins = cursor.KnownBulletins.ToHashSet(StringComparer.OrdinalIgnoreCase); | ||||||
|  |         var now = _timeProvider.GetUtcNow(); | ||||||
|  |         var processed = 0; | ||||||
|  |  | ||||||
|  |         IReadOnlyList<BulletinAttachment> attachments = Array.Empty<BulletinAttachment>(); | ||||||
|  |  | ||||||
|  |         try | ||||||
|  |         { | ||||||
|  |             var listingResult = await FetchListingAsync(cancellationToken).ConfigureAwait(false); | ||||||
|  |             if (!listingResult.IsSuccess || listingResult.Content is null) | ||||||
|  |             { | ||||||
|  |                 _logger.LogWarning("NKCKI listing fetch returned no content (status={Status})", listingResult.StatusCode); | ||||||
|  |                 processed = await ProcessCachedBulletinsAsync(pendingDocuments, pendingMappings, knownBulletins, now, processed, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 await UpdateCursorAsync(cursor | ||||||
|  |                     .WithPendingDocuments(pendingDocuments) | ||||||
|  |                     .WithPendingMappings(pendingMappings) | ||||||
|  |                     .WithKnownBulletins(NormalizeBulletins(knownBulletins)) | ||||||
|  |                     .WithLastListingFetch(now), cancellationToken).ConfigureAwait(false); | ||||||
|  |                 return; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             attachments = await ParseListingAsync(listingResult.Content, cancellationToken).ConfigureAwait(false); | ||||||
|  |         } | ||||||
|  |         catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException) | ||||||
|  |         { | ||||||
|  |             _logger.LogWarning(ex, "NKCKI listing fetch failed; attempting cached bulletins"); | ||||||
|  |             processed = await ProcessCachedBulletinsAsync(pendingDocuments, pendingMappings, knownBulletins, now, processed, cancellationToken).ConfigureAwait(false); | ||||||
|  |             await UpdateCursorAsync(cursor | ||||||
|  |                 .WithPendingDocuments(pendingDocuments) | ||||||
|  |                 .WithPendingMappings(pendingMappings) | ||||||
|  |                 .WithKnownBulletins(NormalizeBulletins(knownBulletins)) | ||||||
|  |                 .WithLastListingFetch(cursor.LastListingFetchAt ?? now), cancellationToken).ConfigureAwait(false); | ||||||
|  |             return; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (attachments.Count == 0) | ||||||
|  |         { | ||||||
|  |             _logger.LogDebug("NKCKI listing contained no bulletin attachments"); | ||||||
|  |             processed = await ProcessCachedBulletinsAsync(pendingDocuments, pendingMappings, knownBulletins, now, processed, cancellationToken).ConfigureAwait(false); | ||||||
|  |             await UpdateCursorAsync(cursor | ||||||
|  |                 .WithPendingDocuments(pendingDocuments) | ||||||
|  |                 .WithPendingMappings(pendingMappings) | ||||||
|  |                 .WithKnownBulletins(NormalizeBulletins(knownBulletins)) | ||||||
|  |                 .WithLastListingFetch(now), cancellationToken).ConfigureAwait(false); | ||||||
|  |             return; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var newAttachments = attachments | ||||||
|  |             .Where(attachment => !knownBulletins.Contains(attachment.Id)) | ||||||
|  |             .Take(_options.MaxBulletinsPerFetch) | ||||||
|  |             .ToList(); | ||||||
|  |  | ||||||
|  |         if (newAttachments.Count == 0) | ||||||
|  |         { | ||||||
|  |             await UpdateCursorAsync(cursor | ||||||
|  |                 .WithPendingDocuments(pendingDocuments) | ||||||
|  |                 .WithPendingMappings(pendingMappings) | ||||||
|  |                 .WithKnownBulletins(NormalizeBulletins(knownBulletins)) | ||||||
|  |                 .WithLastListingFetch(now), cancellationToken).ConfigureAwait(false); | ||||||
|  |             return; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         foreach (var attachment in newAttachments) | ||||||
|  |         { | ||||||
|  |             cancellationToken.ThrowIfCancellationRequested(); | ||||||
|  |  | ||||||
|  |             try | ||||||
|  |             { | ||||||
|  |                 var request = new SourceFetchRequest(RuNkckiOptions.HttpClientName, SourceName, attachment.Uri) | ||||||
|  |                 { | ||||||
|  |                     AcceptHeaders = BulletinAcceptHeaders, | ||||||
|  |                     TimeoutOverride = _options.RequestTimeout, | ||||||
|  |                 }; | ||||||
|  |  | ||||||
|  |                 var attachmentResult = await _fetchService.FetchContentAsync(request, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 if (!attachmentResult.IsSuccess || attachmentResult.Content is null) | ||||||
|  |                 { | ||||||
|  |                     if (TryReadCachedBulletin(attachment.Id, out var cachedBytes)) | ||||||
|  |                     { | ||||||
|  |                         _logger.LogWarning("NKCKI bulletin {BulletinId} unavailable (status={Status}); using cached artefact", attachment.Id, attachmentResult.StatusCode); | ||||||
|  |                         processed = await ProcessBulletinEntriesAsync(cachedBytes, attachment.Id, pendingDocuments, pendingMappings, now, processed, cancellationToken).ConfigureAwait(false); | ||||||
|  |                         knownBulletins.Add(attachment.Id); | ||||||
|  |                     } | ||||||
|  |                     else | ||||||
|  |                     { | ||||||
|  |                         _logger.LogWarning("NKCKI bulletin {BulletinId} returned no content (status={Status})", attachment.Id, attachmentResult.StatusCode); | ||||||
|  |                     } | ||||||
|  |  | ||||||
|  |                     continue; | ||||||
|  |                 } | ||||||
|  |  | ||||||
|  |                 TryWriteCachedBulletin(attachment.Id, attachmentResult.Content); | ||||||
|  |                 processed = await ProcessBulletinEntriesAsync(attachmentResult.Content, attachment.Id, pendingDocuments, pendingMappings, now, processed, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 knownBulletins.Add(attachment.Id); | ||||||
|  |             } | ||||||
|  |             catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException) | ||||||
|  |             { | ||||||
|  |                 if (TryReadCachedBulletin(attachment.Id, out var cachedBytes)) | ||||||
|  |                 { | ||||||
|  |                     _logger.LogWarning(ex, "NKCKI bulletin fetch failed for {BulletinId}; using cached artefact", attachment.Id); | ||||||
|  |                     processed = await ProcessBulletinEntriesAsync(cachedBytes, attachment.Id, pendingDocuments, pendingMappings, now, processed, cancellationToken).ConfigureAwait(false); | ||||||
|  |                     knownBulletins.Add(attachment.Id); | ||||||
|  |                 } | ||||||
|  |                 else | ||||||
|  |                 { | ||||||
|  |                     _logger.LogWarning(ex, "NKCKI bulletin fetch failed for {BulletinId}", attachment.Id); | ||||||
|  |                     await _stateRepository.MarkFailureAsync(SourceName, now, _options.FailureBackoff, ex.Message, cancellationToken).ConfigureAwait(false); | ||||||
|  |                     throw; | ||||||
|  |                 } | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             if (processed >= _options.MaxVulnerabilitiesPerFetch) | ||||||
|  |             { | ||||||
|  |                 break; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             if (_options.RequestDelay > TimeSpan.Zero) | ||||||
|  |             { | ||||||
|  |                 try | ||||||
|  |                 { | ||||||
|  |                     await Task.Delay(_options.RequestDelay, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 } | ||||||
|  |                 catch (TaskCanceledException) | ||||||
|  |                 { | ||||||
|  |                     break; | ||||||
|  |                 } | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var normalizedBulletins = NormalizeBulletins(knownBulletins); | ||||||
|  |  | ||||||
|  |         var updatedCursor = cursor | ||||||
|  |             .WithPendingDocuments(pendingDocuments) | ||||||
|  |             .WithPendingMappings(pendingMappings) | ||||||
|  |             .WithKnownBulletins(normalizedBulletins) | ||||||
|  |             .WithLastListingFetch(now); | ||||||
|  |  | ||||||
|  |         await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken) | ||||||
|  |     { | ||||||
|  |         ArgumentNullException.ThrowIfNull(services); | ||||||
|  |  | ||||||
|  |         var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false); | ||||||
|  |         if (cursor.PendingDocuments.Count == 0) | ||||||
|  |         { | ||||||
|  |             return; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var pendingDocuments = cursor.PendingDocuments.ToList(); | ||||||
|  |         var pendingMappings = cursor.PendingMappings.ToList(); | ||||||
|  |  | ||||||
|  |         foreach (var documentId in cursor.PendingDocuments) | ||||||
|  |         { | ||||||
|  |             cancellationToken.ThrowIfCancellationRequested(); | ||||||
|  |  | ||||||
|  |             var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false); | ||||||
|  |             if (document is null) | ||||||
|  |             { | ||||||
|  |                 pendingDocuments.Remove(documentId); | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             if (!document.GridFsId.HasValue) | ||||||
|  |             { | ||||||
|  |                 _logger.LogWarning("NKCKI document {DocumentId} missing GridFS payload", documentId); | ||||||
|  |                 await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 pendingDocuments.Remove(documentId); | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             byte[] payload; | ||||||
|  |             try | ||||||
|  |             { | ||||||
|  |                 payload = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false); | ||||||
|  |             } | ||||||
|  |             catch (Exception ex) | ||||||
|  |             { | ||||||
|  |                 _logger.LogError(ex, "NKCKI unable to download raw document {DocumentId}", documentId); | ||||||
|  |                 throw; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             RuNkckiVulnerabilityDto? dto; | ||||||
|  |             try | ||||||
|  |             { | ||||||
|  |                 dto = JsonSerializer.Deserialize<RuNkckiVulnerabilityDto>(payload, SerializerOptions); | ||||||
|  |             } | ||||||
|  |             catch (Exception ex) | ||||||
|  |             { | ||||||
|  |                 _logger.LogWarning(ex, "NKCKI failed to deserialize document {DocumentId}", documentId); | ||||||
|  |                 await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 pendingDocuments.Remove(documentId); | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             if (dto is null) | ||||||
|  |             { | ||||||
|  |                 _logger.LogWarning("NKCKI document {DocumentId} produced null DTO", documentId); | ||||||
|  |                 await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 pendingDocuments.Remove(documentId); | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             var bson = MongoDB.Bson.BsonDocument.Parse(JsonSerializer.Serialize(dto, SerializerOptions)); | ||||||
|  |             var dtoRecord = new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "ru-nkcki.v1", bson, _timeProvider.GetUtcNow()); | ||||||
|  |             await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false); | ||||||
|  |             await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false); | ||||||
|  |  | ||||||
|  |             pendingDocuments.Remove(documentId); | ||||||
|  |             if (!pendingMappings.Contains(documentId)) | ||||||
|  |             { | ||||||
|  |                 pendingMappings.Add(documentId); | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var updatedCursor = cursor | ||||||
|  |             .WithPendingDocuments(pendingDocuments) | ||||||
|  |             .WithPendingMappings(pendingMappings); | ||||||
|  |  | ||||||
|  |         await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken) | ||||||
|  |     { | ||||||
|  |         ArgumentNullException.ThrowIfNull(services); | ||||||
|  |  | ||||||
|  |         var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false); | ||||||
|  |         if (cursor.PendingMappings.Count == 0) | ||||||
|  |         { | ||||||
|  |             return; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var pendingMappings = cursor.PendingMappings.ToList(); | ||||||
|  |  | ||||||
|  |         foreach (var documentId in cursor.PendingMappings) | ||||||
|  |         { | ||||||
|  |             cancellationToken.ThrowIfCancellationRequested(); | ||||||
|  |  | ||||||
|  |             var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false); | ||||||
|  |             if (document is null) | ||||||
|  |             { | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             var dtoRecord = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false); | ||||||
|  |             if (dtoRecord is null) | ||||||
|  |             { | ||||||
|  |                 _logger.LogWarning("NKCKI document {DocumentId} missing DTO payload", documentId); | ||||||
|  |                 await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             RuNkckiVulnerabilityDto dto; | ||||||
|  |             try | ||||||
|  |             { | ||||||
|  |                 dto = JsonSerializer.Deserialize<RuNkckiVulnerabilityDto>(dtoRecord.Payload.ToString(), SerializerOptions) ?? throw new InvalidOperationException("DTO deserialized to null"); | ||||||
|  |             } | ||||||
|  |             catch (Exception ex) | ||||||
|  |             { | ||||||
|  |                 _logger.LogError(ex, "NKCKI failed to deserialize DTO for document {DocumentId}", documentId); | ||||||
|  |                 await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             try | ||||||
|  |             { | ||||||
|  |                 var advisory = RuNkckiMapper.Map(dto, document, dtoRecord.ValidatedAt); | ||||||
|  |                 await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |             } | ||||||
|  |             catch (Exception ex) | ||||||
|  |             { | ||||||
|  |                 _logger.LogError(ex, "NKCKI mapping failed for document {DocumentId}", documentId); | ||||||
|  |                 await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false); | ||||||
|  |                 pendingMappings.Remove(documentId); | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var updatedCursor = cursor.WithPendingMappings(pendingMappings); | ||||||
|  |         await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private async Task<int> ProcessCachedBulletinsAsync( | ||||||
|  |         HashSet<Guid> pendingDocuments, | ||||||
|  |         HashSet<Guid> pendingMappings, | ||||||
|  |         HashSet<string> knownBulletins, | ||||||
|  |         DateTimeOffset now, | ||||||
|  |         int processed, | ||||||
|  |         CancellationToken cancellationToken) | ||||||
|  |     { | ||||||
|  |         if (!Directory.Exists(_cacheDirectory)) | ||||||
|  |         { | ||||||
|  |             return processed; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var updated = processed; | ||||||
|  |         var cacheFiles = Directory | ||||||
|  |             .EnumerateFiles(_cacheDirectory, "*.json.zip", SearchOption.TopDirectoryOnly) | ||||||
|  |             .OrderBy(static path => path, StringComparer.OrdinalIgnoreCase) | ||||||
|  |             .ToList(); | ||||||
|  |  | ||||||
|  |         foreach (var filePath in cacheFiles) | ||||||
|  |         { | ||||||
|  |             cancellationToken.ThrowIfCancellationRequested(); | ||||||
|  |  | ||||||
|  |             var bulletinId = ExtractBulletinIdFromCachePath(filePath); | ||||||
|  |             if (string.IsNullOrWhiteSpace(bulletinId) || knownBulletins.Contains(bulletinId)) | ||||||
|  |             { | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             byte[] content; | ||||||
|  |             try | ||||||
|  |             { | ||||||
|  |                 content = File.ReadAllBytes(filePath); | ||||||
|  |             } | ||||||
|  |             catch (Exception ex) | ||||||
|  |             { | ||||||
|  |                 _logger.LogDebug(ex, "NKCKI failed to read cached bulletin at {CachePath}", filePath); | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             updated = await ProcessBulletinEntriesAsync(content, bulletinId, pendingDocuments, pendingMappings, now, updated, cancellationToken).ConfigureAwait(false); | ||||||
|  |             knownBulletins.Add(bulletinId); | ||||||
|  |  | ||||||
|  |             if (updated >= _options.MaxVulnerabilitiesPerFetch) | ||||||
|  |             { | ||||||
|  |                 break; | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return updated; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private async Task<int> ProcessBulletinEntriesAsync( | ||||||
|  |         byte[] content, | ||||||
|  |         string bulletinId, | ||||||
|  |         HashSet<Guid> pendingDocuments, | ||||||
|  |         HashSet<Guid> pendingMappings, | ||||||
|  |         DateTimeOffset now, | ||||||
|  |         int processed, | ||||||
|  |         CancellationToken cancellationToken) | ||||||
|  |     { | ||||||
|  |         if (content.Length == 0) | ||||||
|  |         { | ||||||
|  |             return processed; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var updated = processed; | ||||||
|  |         using var archiveStream = new MemoryStream(content, writable: false); | ||||||
|  |         using var archive = new ZipArchive(archiveStream, ZipArchiveMode.Read, leaveOpen: false); | ||||||
|  |  | ||||||
|  |         foreach (var entry in archive.Entries.OrderBy(static e => e.FullName, StringComparer.OrdinalIgnoreCase)) | ||||||
|  |         { | ||||||
|  |             cancellationToken.ThrowIfCancellationRequested(); | ||||||
|  |  | ||||||
|  |             if (!entry.FullName.EndsWith(".json", StringComparison.OrdinalIgnoreCase)) | ||||||
|  |             { | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             using var entryStream = entry.Open(); | ||||||
|  |             using var buffer = new MemoryStream(); | ||||||
|  |             await entryStream.CopyToAsync(buffer, cancellationToken).ConfigureAwait(false); | ||||||
|  |  | ||||||
|  |             if (buffer.Length == 0) | ||||||
|  |             { | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             buffer.Position = 0; | ||||||
|  |  | ||||||
|  |             using var document = await JsonDocument.ParseAsync(buffer, cancellationToken: cancellationToken).ConfigureAwait(false); | ||||||
|  |             updated = await ProcessBulletinJsonElementAsync(document.RootElement, entry.FullName, bulletinId, pendingDocuments, pendingMappings, now, updated, cancellationToken).ConfigureAwait(false); | ||||||
|  |  | ||||||
|  |             if (updated >= _options.MaxVulnerabilitiesPerFetch) | ||||||
|  |             { | ||||||
|  |                 break; | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return updated; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private async Task<int> ProcessBulletinJsonElementAsync( | ||||||
|  |         JsonElement element, | ||||||
|  |         string entryName, | ||||||
|  |         string bulletinId, | ||||||
|  |         HashSet<Guid> pendingDocuments, | ||||||
|  |         HashSet<Guid> pendingMappings, | ||||||
|  |         DateTimeOffset now, | ||||||
|  |         int processed, | ||||||
|  |         CancellationToken cancellationToken) | ||||||
|  |     { | ||||||
|  |         var updated = processed; | ||||||
|  |  | ||||||
|  |         switch (element.ValueKind) | ||||||
|  |         { | ||||||
|  |             case JsonValueKind.Array: | ||||||
|  |                 foreach (var child in element.EnumerateArray()) | ||||||
|  |                 { | ||||||
|  |                     cancellationToken.ThrowIfCancellationRequested(); | ||||||
|  |  | ||||||
|  |                     if (updated >= _options.MaxVulnerabilitiesPerFetch) | ||||||
|  |                     { | ||||||
|  |                         break; | ||||||
|  |                     } | ||||||
|  |  | ||||||
|  |                     if (child.ValueKind != JsonValueKind.Object) | ||||||
|  |                     { | ||||||
|  |                         continue; | ||||||
|  |                     } | ||||||
|  |  | ||||||
|  |                     if (await ProcessVulnerabilityObjectAsync(child, entryName, bulletinId, pendingDocuments, pendingMappings, now, cancellationToken).ConfigureAwait(false)) | ||||||
|  |                     { | ||||||
|  |                         updated++; | ||||||
|  |                     } | ||||||
|  |                 } | ||||||
|  |  | ||||||
|  |                 break; | ||||||
|  |  | ||||||
|  |             case JsonValueKind.Object: | ||||||
|  |                 if (await ProcessVulnerabilityObjectAsync(element, entryName, bulletinId, pendingDocuments, pendingMappings, now, cancellationToken).ConfigureAwait(false)) | ||||||
|  |                 { | ||||||
|  |                     updated++; | ||||||
|  |                 } | ||||||
|  |  | ||||||
|  |                 break; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return updated; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private async Task<bool> ProcessVulnerabilityObjectAsync( | ||||||
|  |         JsonElement element, | ||||||
|  |         string entryName, | ||||||
|  |         string bulletinId, | ||||||
|  |         HashSet<Guid> pendingDocuments, | ||||||
|  |         HashSet<Guid> pendingMappings, | ||||||
|  |         DateTimeOffset now, | ||||||
|  |         CancellationToken cancellationToken) | ||||||
|  |     { | ||||||
|  |         RuNkckiVulnerabilityDto dto; | ||||||
|  |         try | ||||||
|  |         { | ||||||
|  |             dto = RuNkckiJsonParser.Parse(element); | ||||||
|  |         } | ||||||
|  |         catch (Exception ex) | ||||||
|  |         { | ||||||
|  |             _logger.LogDebug(ex, "NKCKI failed to parse vulnerability in bulletin {BulletinId} entry {Entry}", bulletinId, entryName); | ||||||
|  |             return false; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var payload = JsonSerializer.SerializeToUtf8Bytes(dto, SerializerOptions); | ||||||
|  |         var sha = Convert.ToHexString(SHA256.HashData(payload)).ToLowerInvariant(); | ||||||
|  |         var documentUri = BuildDocumentUri(dto); | ||||||
|  |  | ||||||
|  |         var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, documentUri, cancellationToken).ConfigureAwait(false); | ||||||
|  |         if (existing is not null && string.Equals(existing.Sha256, sha, StringComparison.OrdinalIgnoreCase)) | ||||||
|  |         { | ||||||
|  |             return false; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var gridFsId = await _rawDocumentStorage.UploadAsync(SourceName, documentUri, payload, "application/json", null, cancellationToken).ConfigureAwait(false); | ||||||
|  |  | ||||||
|  |         var metadata = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase) | ||||||
|  |         { | ||||||
|  |             ["ru-nkcki.bulletin"] = bulletinId, | ||||||
|  |             ["ru-nkcki.entry"] = entryName, | ||||||
|  |         }; | ||||||
|  |  | ||||||
|  |         if (!string.IsNullOrWhiteSpace(dto.FstecId)) | ||||||
|  |         { | ||||||
|  |             metadata["ru-nkcki.fstec_id"] = dto.FstecId!; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (!string.IsNullOrWhiteSpace(dto.MitreId)) | ||||||
|  |         { | ||||||
|  |             metadata["ru-nkcki.mitre_id"] = dto.MitreId!; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var recordId = existing?.Id ?? Guid.NewGuid(); | ||||||
|  |         var lastModified = dto.DateUpdated ?? dto.DatePublished; | ||||||
|  |         var record = new DocumentRecord( | ||||||
|  |             recordId, | ||||||
|  |             SourceName, | ||||||
|  |             documentUri, | ||||||
|  |             now, | ||||||
|  |             sha, | ||||||
|  |             DocumentStatuses.PendingParse, | ||||||
|  |             "application/json", | ||||||
|  |             Headers: null, | ||||||
|  |             Metadata: metadata, | ||||||
|  |             Etag: null, | ||||||
|  |             LastModified: lastModified, | ||||||
|  |             GridFsId: gridFsId, | ||||||
|  |             ExpiresAt: null); | ||||||
|  |  | ||||||
|  |         var upserted = await _documentStore.UpsertAsync(record, cancellationToken).ConfigureAwait(false); | ||||||
|  |         pendingDocuments.Add(upserted.Id); | ||||||
|  |         pendingMappings.Remove(upserted.Id); | ||||||
|  |         return true; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private async Task<SourceFetchContentResult> FetchListingAsync(CancellationToken cancellationToken) | ||||||
|  |     { | ||||||
|  |         try | ||||||
|  |         { | ||||||
|  |             var request = new SourceFetchRequest(RuNkckiOptions.HttpClientName, SourceName, _options.ListingUri) | ||||||
|  |             { | ||||||
|  |                 AcceptHeaders = ListingAcceptHeaders, | ||||||
|  |                 TimeoutOverride = _options.RequestTimeout, | ||||||
|  |             }; | ||||||
|  |  | ||||||
|  |             return await _fetchService.FetchContentAsync(request, cancellationToken).ConfigureAwait(false); | ||||||
|  |         } | ||||||
|  |         catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException) | ||||||
|  |         { | ||||||
|  |             _logger.LogError(ex, "NKCKI listing fetch failed for {ListingUri}", _options.ListingUri); | ||||||
|  |             await _stateRepository.MarkFailureAsync(SourceName, _timeProvider.GetUtcNow(), _options.FailureBackoff, ex.Message, cancellationToken).ConfigureAwait(false); | ||||||
|  |             throw; | ||||||
|  |         } | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private async Task<IReadOnlyList<BulletinAttachment>> ParseListingAsync(byte[] content, CancellationToken cancellationToken) | ||||||
|  |     { | ||||||
|  |         var html = Encoding.UTF8.GetString(content); | ||||||
|  |         var document = await _htmlParser.ParseDocumentAsync(html, cancellationToken).ConfigureAwait(false); | ||||||
|  |         var anchors = document.QuerySelectorAll("a[href$='.json.zip']"); | ||||||
|  |  | ||||||
|  |         var attachments = new List<BulletinAttachment>(); | ||||||
|  |         foreach (var anchor in anchors) | ||||||
|  |         { | ||||||
|  |             var href = anchor.GetAttribute("href"); | ||||||
|  |             if (string.IsNullOrWhiteSpace(href)) | ||||||
|  |             { | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             if (!Uri.TryCreate(_options.BaseAddress, href, out var absoluteUri)) | ||||||
|  |             { | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             var id = DeriveBulletinId(absoluteUri); | ||||||
|  |             if (string.IsNullOrWhiteSpace(id)) | ||||||
|  |             { | ||||||
|  |                 continue; | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             var title = anchor.GetAttribute("title"); | ||||||
|  |             if (string.IsNullOrWhiteSpace(title)) | ||||||
|  |             { | ||||||
|  |                 title = anchor.TextContent?.Trim(); | ||||||
|  |             } | ||||||
|  |  | ||||||
|  |             attachments.Add(new BulletinAttachment(id, absoluteUri, title ?? id)); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return attachments; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static string DeriveBulletinId(Uri uri) | ||||||
|  |     { | ||||||
|  |         var fileName = Path.GetFileName(uri.AbsolutePath); | ||||||
|  |         if (string.IsNullOrWhiteSpace(fileName)) | ||||||
|  |         { | ||||||
|  |             return Guid.NewGuid().ToString("N"); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (fileName.EndsWith(".zip", StringComparison.OrdinalIgnoreCase)) | ||||||
|  |         { | ||||||
|  |             fileName = fileName[..^4]; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (fileName.EndsWith(".json", StringComparison.OrdinalIgnoreCase)) | ||||||
|  |         { | ||||||
|  |             fileName = fileName[..^5]; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return fileName.Replace('_', '-'); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static string BuildDocumentUri(RuNkckiVulnerabilityDto dto) | ||||||
|  |     { | ||||||
|  |         if (!string.IsNullOrWhiteSpace(dto.FstecId)) | ||||||
|  |         { | ||||||
|  |             var slug = dto.FstecId.Contains(':', StringComparison.Ordinal) | ||||||
|  |                 ? dto.FstecId[(dto.FstecId.IndexOf(':') + 1)..] | ||||||
|  |                 : dto.FstecId; | ||||||
|  |             return $"https://cert.gov.ru/materialy/uyazvimosti/{slug}"; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (!string.IsNullOrWhiteSpace(dto.MitreId)) | ||||||
|  |         { | ||||||
|  |             return $"https://nvd.nist.gov/vuln/detail/{dto.MitreId}"; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return $"https://cert.gov.ru/materialy/uyazvimosti/{Guid.NewGuid():N}"; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private string ResolveCacheDirectory(string? configuredPath) | ||||||
|  |     { | ||||||
|  |         if (!string.IsNullOrWhiteSpace(configuredPath)) | ||||||
|  |         { | ||||||
|  |             return Path.GetFullPath(Path.IsPathRooted(configuredPath) | ||||||
|  |                 ? configuredPath | ||||||
|  |                 : Path.Combine(AppContext.BaseDirectory, configuredPath)); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return Path.Combine(AppContext.BaseDirectory, "cache", RuNkckiConnectorPlugin.SourceName); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private void EnsureCacheDirectory() | ||||||
|  |     { | ||||||
|  |         try | ||||||
|  |         { | ||||||
|  |             Directory.CreateDirectory(_cacheDirectory); | ||||||
|  |         } | ||||||
|  |         catch (Exception ex) | ||||||
|  |         { | ||||||
|  |             _logger.LogWarning(ex, "NKCKI unable to ensure cache directory {CachePath}", _cacheDirectory); | ||||||
|  |         } | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private string GetBulletinCachePath(string bulletinId) | ||||||
|  |     { | ||||||
|  |         var fileStem = string.IsNullOrWhiteSpace(bulletinId) | ||||||
|  |             ? Guid.NewGuid().ToString("N") | ||||||
|  |             : Uri.EscapeDataString(bulletinId); | ||||||
|  |         return Path.Combine(_cacheDirectory, $"{fileStem}.json.zip"); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static string ExtractBulletinIdFromCachePath(string path) | ||||||
|  |     { | ||||||
|  |         if (string.IsNullOrWhiteSpace(path)) | ||||||
|  |         { | ||||||
|  |             return string.Empty; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var fileName = Path.GetFileName(path); | ||||||
|  |         if (fileName.EndsWith(".zip", StringComparison.OrdinalIgnoreCase)) | ||||||
|  |         { | ||||||
|  |             fileName = fileName[..^4]; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         if (fileName.EndsWith(".json", StringComparison.OrdinalIgnoreCase)) | ||||||
|  |         { | ||||||
|  |             fileName = fileName[..^5]; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         return Uri.UnescapeDataString(fileName); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private void TryWriteCachedBulletin(string bulletinId, byte[] content) | ||||||
|  |     { | ||||||
|  |         try | ||||||
|  |         { | ||||||
|  |             var cachePath = GetBulletinCachePath(bulletinId); | ||||||
|  |             Directory.CreateDirectory(Path.GetDirectoryName(cachePath)!); | ||||||
|  |             File.WriteAllBytes(cachePath, content); | ||||||
|  |         } | ||||||
|  |         catch (Exception ex) | ||||||
|  |         { | ||||||
|  |             _logger.LogDebug(ex, "NKCKI failed to cache bulletin {BulletinId}", bulletinId); | ||||||
|  |         } | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private bool TryReadCachedBulletin(string bulletinId, out byte[] content) | ||||||
|  |     { | ||||||
|  |         var cachePath = GetBulletinCachePath(bulletinId); | ||||||
|  |         try | ||||||
|  |         { | ||||||
|  |             if (File.Exists(cachePath)) | ||||||
|  |             { | ||||||
|  |                 content = File.ReadAllBytes(cachePath); | ||||||
|  |                 return true; | ||||||
|  |             } | ||||||
|  |         } | ||||||
|  |         catch (Exception ex) | ||||||
|  |         { | ||||||
|  |             _logger.LogDebug(ex, "NKCKI failed to read cached bulletin {BulletinId}", bulletinId); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         content = Array.Empty<byte>(); | ||||||
|  |         return false; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private IReadOnlyCollection<string> NormalizeBulletins(IEnumerable<string> bulletins) | ||||||
|  |     { | ||||||
|  |         var normalized = (bulletins ?? Enumerable.Empty<string>()) | ||||||
|  |             .Where(static id => !string.IsNullOrWhiteSpace(id)) | ||||||
|  |             .Distinct(StringComparer.OrdinalIgnoreCase) | ||||||
|  |             .OrderBy(static id => id, StringComparer.OrdinalIgnoreCase) | ||||||
|  |             .ToList(); | ||||||
|  |  | ||||||
|  |         if (normalized.Count <= _options.KnownBulletinCapacity) | ||||||
|  |         { | ||||||
|  |             return normalized.ToArray(); | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         var skip = normalized.Count - _options.KnownBulletinCapacity; | ||||||
|  |         return normalized.Skip(skip).ToArray(); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private async Task<RuNkckiCursor> GetCursorAsync(CancellationToken cancellationToken) | ||||||
|  |     { | ||||||
|  |         var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false); | ||||||
|  |         return state is null ? RuNkckiCursor.Empty : RuNkckiCursor.FromBson(state.Cursor); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private Task UpdateCursorAsync(RuNkckiCursor cursor, CancellationToken cancellationToken) | ||||||
|  |     { | ||||||
|  |         var document = cursor.ToBsonDocument(); | ||||||
|  |         var completedAt = cursor.LastListingFetchAt ?? _timeProvider.GetUtcNow(); | ||||||
|  |         return _stateRepository.UpdateCursorAsync(SourceName, document, completedAt, cancellationToken); | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private readonly record struct BulletinAttachment(string Id, Uri Uri, string Title); | ||||||
|  | } | ||||||
| @@ -0,0 +1,19 @@ | |||||||
|  | using Microsoft.Extensions.DependencyInjection; | ||||||
|  | using StellaOps.Plugin; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Nkcki; | ||||||
|  |  | ||||||
|  | public sealed class RuNkckiConnectorPlugin : IConnectorPlugin | ||||||
|  | { | ||||||
|  |     public const string SourceName = "ru-nkcki"; | ||||||
|  |  | ||||||
|  |     public string Name => SourceName; | ||||||
|  |  | ||||||
|  |     public bool IsAvailable(IServiceProvider services) => services is not null; | ||||||
|  |  | ||||||
|  |     public IFeedConnector Create(IServiceProvider services) | ||||||
|  |     { | ||||||
|  |         ArgumentNullException.ThrowIfNull(services); | ||||||
|  |         return ActivatorUtilities.CreateInstance<RuNkckiConnector>(services); | ||||||
|  |     } | ||||||
|  | } | ||||||
| @@ -0,0 +1,53 @@ | |||||||
|  | using Microsoft.Extensions.Configuration; | ||||||
|  | using Microsoft.Extensions.DependencyInjection; | ||||||
|  | using StellaOps.DependencyInjection; | ||||||
|  | using StellaOps.Feedser.Core.Jobs; | ||||||
|  | using StellaOps.Feedser.Source.Ru.Nkcki.Configuration; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Nkcki; | ||||||
|  |  | ||||||
|  | public sealed class RuNkckiDependencyInjectionRoutine : IDependencyInjectionRoutine | ||||||
|  | { | ||||||
|  |     private const string ConfigurationSection = "feedser:sources:ru-nkcki"; | ||||||
|  |  | ||||||
|  |     public IServiceCollection Register(IServiceCollection services, IConfiguration configuration) | ||||||
|  |     { | ||||||
|  |         ArgumentNullException.ThrowIfNull(services); | ||||||
|  |         ArgumentNullException.ThrowIfNull(configuration); | ||||||
|  |  | ||||||
|  |         services.AddRuNkckiConnector(options => | ||||||
|  |         { | ||||||
|  |             configuration.GetSection(ConfigurationSection).Bind(options); | ||||||
|  |             options.Validate(); | ||||||
|  |         }); | ||||||
|  |  | ||||||
|  |         services.AddTransient<RuNkckiFetchJob>(); | ||||||
|  |         services.AddTransient<RuNkckiParseJob>(); | ||||||
|  |         services.AddTransient<RuNkckiMapJob>(); | ||||||
|  |  | ||||||
|  |         services.PostConfigure<JobSchedulerOptions>(options => | ||||||
|  |         { | ||||||
|  |             EnsureJob(options, RuNkckiJobKinds.Fetch, typeof(RuNkckiFetchJob)); | ||||||
|  |             EnsureJob(options, RuNkckiJobKinds.Parse, typeof(RuNkckiParseJob)); | ||||||
|  |             EnsureJob(options, RuNkckiJobKinds.Map, typeof(RuNkckiMapJob)); | ||||||
|  |         }); | ||||||
|  |  | ||||||
|  |         return services; | ||||||
|  |     } | ||||||
|  |  | ||||||
|  |     private static void EnsureJob(JobSchedulerOptions schedulerOptions, string kind, Type jobType) | ||||||
|  |     { | ||||||
|  |         if (schedulerOptions.Definitions.ContainsKey(kind)) | ||||||
|  |         { | ||||||
|  |             return; | ||||||
|  |         } | ||||||
|  |  | ||||||
|  |         schedulerOptions.Definitions[kind] = new JobDefinition( | ||||||
|  |             kind, | ||||||
|  |             jobType, | ||||||
|  |             schedulerOptions.DefaultTimeout, | ||||||
|  |             schedulerOptions.DefaultLeaseDuration, | ||||||
|  |             CronExpression: null, | ||||||
|  |             Enabled: true); | ||||||
|  |     } | ||||||
|  | } | ||||||
| @@ -0,0 +1,43 @@ | |||||||
|  | using System.Net; | ||||||
|  | using Microsoft.Extensions.DependencyInjection; | ||||||
|  | using Microsoft.Extensions.Options; | ||||||
|  | using StellaOps.Feedser.Source.Common.Http; | ||||||
|  | using StellaOps.Feedser.Source.Ru.Nkcki.Configuration; | ||||||
|  |  | ||||||
|  | namespace StellaOps.Feedser.Source.Ru.Nkcki; | ||||||
|  |  | ||||||
|  | public static class RuNkckiServiceCollectionExtensions | ||||||
|  | { | ||||||
|  |     public static IServiceCollection AddRuNkckiConnector(this IServiceCollection services, Action<RuNkckiOptions> configure) | ||||||
|  |     { | ||||||
|  |         ArgumentNullException.ThrowIfNull(services); | ||||||
|  |         ArgumentNullException.ThrowIfNull(configure); | ||||||
|  |  | ||||||
|  |         services.AddOptions<RuNkckiOptions>() | ||||||
|  |             .Configure(configure) | ||||||
|  |             .PostConfigure(static options => options.Validate()); | ||||||
|  |  | ||||||
|  |         services.AddSourceHttpClient(RuNkckiOptions.HttpClientName, (sp, clientOptions) => | ||||||
|  |         { | ||||||
|  |             var options = sp.GetRequiredService<IOptions<RuNkckiOptions>>().Value; | ||||||
|  |             clientOptions.BaseAddress = options.BaseAddress; | ||||||
|  |             clientOptions.Timeout = options.RequestTimeout; | ||||||
|  |             clientOptions.UserAgent = options.UserAgent; | ||||||
|  |             clientOptions.AllowAutoRedirect = true; | ||||||
|  |             clientOptions.DefaultRequestHeaders["Accept-Language"] = options.AcceptLanguage; | ||||||
|  |             clientOptions.AllowedHosts.Clear(); | ||||||
|  |             clientOptions.AllowedHosts.Add(options.BaseAddress.Host); | ||||||
|  |             clientOptions.ConfigureHandler = handler => | ||||||
|  |             { | ||||||
|  |                 handler.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate; | ||||||
|  |                 handler.AllowAutoRedirect = true; | ||||||
|  |                 handler.UseCookies = true; | ||||||
|  |                 handler.CookieContainer = new CookieContainer(); | ||||||
|  |             }; | ||||||
|  |         }); | ||||||
|  |  | ||||||
|  |         services.AddTransient<RuNkckiConnector>(); | ||||||
|  |  | ||||||
|  |         return services; | ||||||
|  |     } | ||||||
|  | } | ||||||
| @@ -7,10 +7,16 @@ | |||||||
|   </PropertyGroup> |   </PropertyGroup> | ||||||
|  |  | ||||||
|   <ItemGroup> |   <ItemGroup> | ||||||
|     <ProjectReference Include="../StellaOps.Plugin/StellaOps.Plugin.csproj" /> |     <PackageReference Include="AngleSharp" Version="1.1.1" /> | ||||||
|  |   </ItemGroup> | ||||||
|  |  | ||||||
|  |   <ItemGroup> | ||||||
|  |     <ProjectReference Include="../StellaOps.Plugin/StellaOps.Plugin.csproj" /> | ||||||
|  |     <ProjectReference Include="../StellaOps.Feedser.Core/StellaOps.Feedser.Core.csproj" /> | ||||||
|  |     <ProjectReference Include="../StellaOps.Feedser.Normalization/StellaOps.Feedser.Normalization.csproj" /> | ||||||
|     <ProjectReference Include="../StellaOps.Feedser.Source.Common/StellaOps.Feedser.Source.Common.csproj" /> |     <ProjectReference Include="../StellaOps.Feedser.Source.Common/StellaOps.Feedser.Source.Common.csproj" /> | ||||||
|     <ProjectReference Include="../StellaOps.Feedser.Models/StellaOps.Feedser.Models.csproj" /> |     <ProjectReference Include="../StellaOps.Feedser.Models/StellaOps.Feedser.Models.csproj" /> | ||||||
|  |     <ProjectReference Include="../StellaOps.Feedser.Storage.Mongo/StellaOps.Feedser.Storage.Mongo.csproj" /> | ||||||
|   </ItemGroup> |   </ItemGroup> | ||||||
| </Project> |  | ||||||
|  |  | ||||||
|  | </Project> | ||||||
|   | |||||||
| @@ -2,10 +2,10 @@ | |||||||
| | Task | Owner(s) | Depends on | Notes | | | Task | Owner(s) | Depends on | Notes | | ||||||
| |---|---|---|---| | |---|---|---|---| | ||||||
| |FEEDCONN-NKCKI-02-001 Research NKTsKI advisory feeds|BE-Conn-Nkcki|Research|**DONE (2025-10-11)** – Candidate RSS locations (`https://cert.gov.ru/rss/advisories.xml`, `https://www.cert.gov.ru/...`) return 403/404 even with `Accept-Language: ru-RU` and `--insecure`; site is Bitrix-backed and expects Russian Trusted Sub CA plus session cookies. Logged packet captures + needed cert list in `docs/feedser-connector-research-20251011.md`; waiting on Ops for sanctioned trust bundle.| | |FEEDCONN-NKCKI-02-001 Research NKTsKI advisory feeds|BE-Conn-Nkcki|Research|**DONE (2025-10-11)** – Candidate RSS locations (`https://cert.gov.ru/rss/advisories.xml`, `https://www.cert.gov.ru/...`) return 403/404 even with `Accept-Language: ru-RU` and `--insecure`; site is Bitrix-backed and expects Russian Trusted Sub CA plus session cookies. Logged packet captures + needed cert list in `docs/feedser-connector-research-20251011.md`; waiting on Ops for sanctioned trust bundle.| | ||||||
| |FEEDCONN-NKCKI-02-002 Fetch pipeline & state persistence|BE-Conn-Nkcki|Source.Common, Storage.Mongo|**TODO** – Implement fetch job with custom trust store, optional SOCKS proxy, and Bitrix session bootstrap (`PHPSESSID`, `BITRIX_SM_GUEST_ID`). Persist raw XML/HTML + derived cursor (advisory ID + `pubDate`), handle 403 retries with exponential backoff.| | |FEEDCONN-NKCKI-02-002 Fetch pipeline & state persistence|BE-Conn-Nkcki|Source.Common, Storage.Mongo|**DOING (2025-10-12)** – Listing fetch now expands `*.json.zip` bulletins into per-vulnerability JSON documents with cursor-tracked bulletin IDs and trust store wiring (`globalsign_r6_bundle.pem`). Parser/mapper emit canonical advisories; remaining work: strengthen pagination/backfill handling and add regression fixtures/telemetry. Offline cache helpers (ProcessCachedBulletinsAsync/TryReadCachedBulletin/TryWriteCachedBulletin) implemented.| | ||||||
| |FEEDCONN-NKCKI-02-003 DTO & parser implementation|BE-Conn-Nkcki|Source.Common|**TODO** – Build DTOs for NKTsKI advisories, sanitise HTML, extract vendors/products, CVEs, mitigation guidance.| | |FEEDCONN-NKCKI-02-003 DTO & parser implementation|BE-Conn-Nkcki|Source.Common|**DOING (2025-10-12)** – `RuNkckiJsonParser` extracts per-vulnerability JSON payloads (IDs, CVEs, CVSS, software text, URLs). TODO: extend coverage for optional fields (ICS categories, nested arrays) and add fixture snapshots.| | ||||||
| |FEEDCONN-NKCKI-02-004 Canonical mapping & range primitives|BE-Conn-Nkcki|Models|**TODO** – Map advisories into canonical records with aliases, references, and vendor range primitives. Coordinate normalized outputs and provenance per `../StellaOps.Feedser.Merge/RANGE_PRIMITIVES_COORDINATION.md`.<br>2025-10-11 research trail: normalized payload target `[{"scheme":"semver","type":"range","min":"<start>","minInclusive":true,"max":"<end>","maxInclusive":false,"notes":"ru.nkcki:advisory-id"}]`; retain Cyrillic identifiers in `notes` so storage provenance remains intact.| | |FEEDCONN-NKCKI-02-004 Canonical mapping & range primitives|BE-Conn-Nkcki|Models|**DOING (2025-10-12)** – `RuNkckiMapper` maps JSON entries to canonical advisories (aliases, references, vendor package, CVSS). Next steps: enrich package parsing (`software_text` tokenisation), consider CVSS v4 metadata, and backfill provenance docs before closing the task.| | ||||||
| |FEEDCONN-NKCKI-02-005 Deterministic fixtures & tests|QA|Testing|**TODO** – Add regression tests supporting `UPDATE_NKCKI_FIXTURES=1` for snapshot regeneration.| | |FEEDCONN-NKCKI-02-005 Deterministic fixtures & tests|QA|Testing|**DOING (2025-10-12)** – Added mocked listing/bulletin regression harness (`RuNkckiConnectorTests`) with fixtures + snapshot writer. Test run currently blocked on Mongo2Go dependency (libcrypto.so.1.1 missing); follow-up required to get embedded mongod running in CI before marking DONE.| | ||||||
| |FEEDCONN-NKCKI-02-006 Telemetry & documentation|DevEx|Docs|**TODO** – Add logging/metrics, document connector configuration, and close backlog entry after deliverable ships.| | |FEEDCONN-NKCKI-02-006 Telemetry & documentation|DevEx|Docs|**TODO** – Add logging/metrics, document connector configuration, and close backlog entry after deliverable ships.| | ||||||
| |FEEDCONN-NKCKI-02-007 Archive ingestion strategy|BE-Conn-Nkcki|Research|**TODO** – Once access restored, map Bitrix paging (`?PAGEN_1=`) and advisory taxonomy (alerts vs recommendations). Outline HTML scrape + PDF attachment handling for backfill and decide translation approach for Russian-only content.| | |FEEDCONN-NKCKI-02-007 Archive ingestion strategy|BE-Conn-Nkcki|Research|**TODO** – Once access restored, map Bitrix paging (`?PAGEN_1=`) and advisory taxonomy (alerts vs recommendations). Outline HTML scrape + PDF attachment handling for backfill and decide translation approach for Russian-only content.| | ||||||
| |FEEDCONN-NKCKI-02-008 Access enablement plan|BE-Conn-Nkcki|Source.Common|**DONE (2025-10-11)** – Documented trust-store requirement, optional SOCKS proxy fallback, and monitoring plan; shared TLS support now available via `SourceHttpClientOptions.TrustedRootCertificates` (`feedser:httpClients:source.nkcki:*`), awaiting Ops-sourced cert bundle before fetch implementation.| | |FEEDCONN-NKCKI-02-008 Access enablement plan|BE-Conn-Nkcki|Source.Common|**DONE (2025-10-11)** – Documented trust-store requirement, optional SOCKS proxy fallback, and monitoring plan; shared TLS support now available via `SourceHttpClientOptions.TrustedRootCertificates` (`feedser:httpClients:source.nkcki:*`), awaiting Ops-sourced cert bundle before fetch implementation.| | ||||||
|   | |||||||
| @@ -131,6 +131,10 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "StellaOps.Cli", "StellaOps. | |||||||
| EndProject | EndProject | ||||||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "StellaOps.Cli.Tests", "StellaOps.Cli.Tests\StellaOps.Cli.Tests.csproj", "{544DBB82-4639-4856-A5F2-76828F7A8396}" | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "StellaOps.Cli.Tests", "StellaOps.Cli.Tests\StellaOps.Cli.Tests.csproj", "{544DBB82-4639-4856-A5F2-76828F7A8396}" | ||||||
| EndProject | EndProject | ||||||
|  | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "StellaOps.Feedser.Source.Ru.Bdu.Tests", "StellaOps.Feedser.Source.Ru.Bdu.Tests\StellaOps.Feedser.Source.Ru.Bdu.Tests.csproj", "{C4B189FA-4268-4B3C-A6B0-C2BB5B96D11A}" | ||||||
|  | EndProject | ||||||
|  | Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "StellaOps.Feedser.Source.Ru.Nkcki.Tests", "StellaOps.Feedser.Source.Ru.Nkcki.Tests\StellaOps.Feedser.Source.Ru.Nkcki.Tests.csproj", "{461D4A58-3816-4737-B209-2D1F08B1F4DF}" | ||||||
|  | EndProject | ||||||
| Global | Global | ||||||
| 	GlobalSection(SolutionConfigurationPlatforms) = preSolution | 	GlobalSection(SolutionConfigurationPlatforms) = preSolution | ||||||
| 		Debug|Any CPU = Debug|Any CPU | 		Debug|Any CPU = Debug|Any CPU | ||||||
| @@ -909,6 +913,30 @@ Global | |||||||
| 		{544DBB82-4639-4856-A5F2-76828F7A8396}.Release|x64.Build.0 = Release|Any CPU | 		{544DBB82-4639-4856-A5F2-76828F7A8396}.Release|x64.Build.0 = Release|Any CPU | ||||||
| 		{544DBB82-4639-4856-A5F2-76828F7A8396}.Release|x86.ActiveCfg = Release|Any CPU | 		{544DBB82-4639-4856-A5F2-76828F7A8396}.Release|x86.ActiveCfg = Release|Any CPU | ||||||
| 		{544DBB82-4639-4856-A5F2-76828F7A8396}.Release|x86.Build.0 = Release|Any CPU | 		{544DBB82-4639-4856-A5F2-76828F7A8396}.Release|x86.Build.0 = Release|Any CPU | ||||||
|  | 		{C4B189FA-4268-4B3C-A6B0-C2BB5B96D11A}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||||||
|  | 		{C4B189FA-4268-4B3C-A6B0-C2BB5B96D11A}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||||||
|  | 		{C4B189FA-4268-4B3C-A6B0-C2BB5B96D11A}.Debug|x64.ActiveCfg = Debug|Any CPU | ||||||
|  | 		{C4B189FA-4268-4B3C-A6B0-C2BB5B96D11A}.Debug|x64.Build.0 = Debug|Any CPU | ||||||
|  | 		{C4B189FA-4268-4B3C-A6B0-C2BB5B96D11A}.Debug|x86.ActiveCfg = Debug|Any CPU | ||||||
|  | 		{C4B189FA-4268-4B3C-A6B0-C2BB5B96D11A}.Debug|x86.Build.0 = Debug|Any CPU | ||||||
|  | 		{C4B189FA-4268-4B3C-A6B0-C2BB5B96D11A}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||||||
|  | 		{C4B189FA-4268-4B3C-A6B0-C2BB5B96D11A}.Release|Any CPU.Build.0 = Release|Any CPU | ||||||
|  | 		{C4B189FA-4268-4B3C-A6B0-C2BB5B96D11A}.Release|x64.ActiveCfg = Release|Any CPU | ||||||
|  | 		{C4B189FA-4268-4B3C-A6B0-C2BB5B96D11A}.Release|x64.Build.0 = Release|Any CPU | ||||||
|  | 		{C4B189FA-4268-4B3C-A6B0-C2BB5B96D11A}.Release|x86.ActiveCfg = Release|Any CPU | ||||||
|  | 		{C4B189FA-4268-4B3C-A6B0-C2BB5B96D11A}.Release|x86.Build.0 = Release|Any CPU | ||||||
|  | 		{461D4A58-3816-4737-B209-2D1F08B1F4DF}.Debug|Any CPU.ActiveCfg = Debug|Any CPU | ||||||
|  | 		{461D4A58-3816-4737-B209-2D1F08B1F4DF}.Debug|Any CPU.Build.0 = Debug|Any CPU | ||||||
|  | 		{461D4A58-3816-4737-B209-2D1F08B1F4DF}.Debug|x64.ActiveCfg = Debug|Any CPU | ||||||
|  | 		{461D4A58-3816-4737-B209-2D1F08B1F4DF}.Debug|x64.Build.0 = Debug|Any CPU | ||||||
|  | 		{461D4A58-3816-4737-B209-2D1F08B1F4DF}.Debug|x86.ActiveCfg = Debug|Any CPU | ||||||
|  | 		{461D4A58-3816-4737-B209-2D1F08B1F4DF}.Debug|x86.Build.0 = Debug|Any CPU | ||||||
|  | 		{461D4A58-3816-4737-B209-2D1F08B1F4DF}.Release|Any CPU.ActiveCfg = Release|Any CPU | ||||||
|  | 		{461D4A58-3816-4737-B209-2D1F08B1F4DF}.Release|Any CPU.Build.0 = Release|Any CPU | ||||||
|  | 		{461D4A58-3816-4737-B209-2D1F08B1F4DF}.Release|x64.ActiveCfg = Release|Any CPU | ||||||
|  | 		{461D4A58-3816-4737-B209-2D1F08B1F4DF}.Release|x64.Build.0 = Release|Any CPU | ||||||
|  | 		{461D4A58-3816-4737-B209-2D1F08B1F4DF}.Release|x86.ActiveCfg = Release|Any CPU | ||||||
|  | 		{461D4A58-3816-4737-B209-2D1F08B1F4DF}.Release|x86.Build.0 = Release|Any CPU | ||||||
| 	EndGlobalSection | 	EndGlobalSection | ||||||
| 	GlobalSection(SolutionProperties) = preSolution | 	GlobalSection(SolutionProperties) = preSolution | ||||||
| 		HideSolutionNode = FALSE | 		HideSolutionNode = FALSE | ||||||
|   | |||||||
		Reference in New Issue
	
	Block a user