Merge branch 'main' of https://git.stella-ops.org/stella-ops.org/git.stella-ops.org
Some checks failed
Build Test Deploy / build-test (push) Has been cancelled
Build Test Deploy / authority-container (push) Has been cancelled
Build Test Deploy / docs (push) Has been cancelled
Build Test Deploy / deploy (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
Some checks failed
Build Test Deploy / build-test (push) Has been cancelled
Build Test Deploy / authority-container (push) Has been cancelled
Build Test Deploy / docs (push) Has been cancelled
Build Test Deploy / deploy (push) Has been cancelled
Docs CI / lint-and-preview (push) Has been cancelled
This commit is contained in:
@@ -69,8 +69,8 @@
|
|||||||
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Cccs/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CCCS-02-001 … 02-007 | Atom feed verified 2025-10-11, history/caching review and FR locale enumeration pending. |
|
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Cccs/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CCCS-02-001 … 02-007 | Atom feed verified 2025-10-11, history/caching review and FR locale enumeration pending. |
|
||||||
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.CertBund/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CERTBUND-02-001 … 02-007 | BSI RSS directory confirmed CERT-Bund feed 2025-10-11, history assessment pending. |
|
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.CertBund/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CERTBUND-02-001 … 02-007 | BSI RSS directory confirmed CERT-Bund feed 2025-10-11, history assessment pending. |
|
||||||
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Kisa/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-KISA-02-001 … 02-007 | KNVD RSS endpoint identified 2025-10-11, access headers/session strategy outstanding. |
|
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Kisa/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-KISA-02-001 … 02-007 | KNVD RSS endpoint identified 2025-10-11, access headers/session strategy outstanding. |
|
||||||
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Ru.Bdu/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-RUBDU-02-001 … 02-008 | BDU RSS/Atom catalogue reviewed 2025-10-11, trust-store acquisition blocked by gosuslugi placeholder page. |
|
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Ru.Bdu/TASKS.md | Build DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-RUBDU-02-001 … 02-008 | TLS bundle + connectors landed 2025-10-12; fetch/parse/map flow emits advisories, fixtures & telemetry follow-up pending. |
|
||||||
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Ru.Nkcki/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-NKCKI-02-001 … 02-008 | cert.gov.ru paginated RSS landing checked 2025-10-11, access enablement plan pending. |
|
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Ru.Nkcki/TASKS.md | Build DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-NKCKI-02-001 … 02-008 | JSON bulletin fetch + canonical mapping live 2025-10-12; regression fixtures added but blocked on Mongo2Go libcrypto dependency for test execution. |
|
||||||
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Ics.Cisa/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ICSCISA-02-001 … 02-008 | new ICS RSS endpoint logged 2025-10-11 but Akamai blocks direct pulls, fallback strategy task opened. |
|
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Ics.Cisa/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-ICSCISA-02-001 … 02-008 | new ICS RSS endpoint logged 2025-10-11 but Akamai blocks direct pulls, fallback strategy task opened. |
|
||||||
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Vndr.Cisco/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CISCO-02-001 … 02-007 | openVuln API + RSS reviewed 2025-10-11, auth/pagination memo pending. |
|
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Vndr.Cisco/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-CISCO-02-001 … 02-007 | openVuln API + RSS reviewed 2025-10-11, auth/pagination memo pending. |
|
||||||
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Vndr.Msrc/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-MSRC-02-001 … 02-007 | MSRC API docs reviewed 2025-10-11, auth/throttling comparison memo pending.<br>Instructions to work:<br>Read ./AGENTS.md plus each module's AGENTS file. Parallelize research, ingestion, mapping, fixtures, and docs using the normalized rule shape from ./src/FASTER_MODELING_AND_NORMALIZATION.md. Coordinate daily with the merge coordination task from Sprint 1. |
|
| Sprint 2 | Connector & Data Implementation Wave | src/StellaOps.Feedser.Source.Vndr.Msrc/TASKS.md | Research DOING | Team Connector Expansion – Regional & Vendor Feeds | FEEDCONN-MSRC-02-001 … 02-007 | MSRC API docs reviewed 2025-10-11, auth/throttling comparison memo pending.<br>Instructions to work:<br>Read ./AGENTS.md plus each module's AGENTS file. Parallelize research, ingestion, mapping, fixtures, and docs using the normalized rule shape from ./src/FASTER_MODELING_AND_NORMALIZATION.md. Coordinate daily with the merge coordination task from Sprint 1. |
|
||||||
|
|||||||
32
certificates/globalsign_gcc_r6_alphassl_ca_2023.pem
Normal file
32
certificates/globalsign_gcc_r6_alphassl_ca_2023.pem
Normal file
@@ -0,0 +1,32 @@
|
|||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIFjDCCA3SgAwIBAgIQfx8skC6D0OO2+zvuR4tegDANBgkqhkiG9w0BAQsFADBM
|
||||||
|
MSAwHgYDVQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSNjETMBEGA1UEChMKR2xv
|
||||||
|
YmFsU2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjAeFw0yMzA3MTkwMzQzMjVaFw0y
|
||||||
|
NjA3MTkwMDAwMDBaMFUxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWdu
|
||||||
|
IG52LXNhMSswKQYDVQQDEyJHbG9iYWxTaWduIEdDQyBSNiBBbHBoYVNTTCBDQSAy
|
||||||
|
MDIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA00Jvk5ADppO0rgDn
|
||||||
|
j1M14XIb032Aas409JJFAb8cUjipFOth7ySLdaWLe3s63oSs5x3eWwzTpX4BFkzZ
|
||||||
|
bxT1eoJSHfT2M0wZ5QOPcCIjsr+YB8TAvV2yJSyq+emRrN/FtgCSTaWXSJ5jipW8
|
||||||
|
SJ/VAuXPMzuAP2yYpuPcjjQ5GyrssDXgu+FhtYxqyFP7BSvx9jQhh5QV5zhLycua
|
||||||
|
n8n+J0Uw09WRQK6JGQ5HzDZQinkNel+fZZNRG1gE9Qeh+tHBplrkalB1g85qJkPO
|
||||||
|
J7SoEvKsmDkajggk/sSq7NPyzFaa/VBGZiRRG+FkxCBniGD5618PQ4trcwHyMojS
|
||||||
|
FObOHQIDAQABo4IBXzCCAVswDgYDVR0PAQH/BAQDAgGGMB0GA1UdJQQWMBQGCCsG
|
||||||
|
AQUFBwMBBggrBgEFBQcDAjASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1UdDgQWBBS9
|
||||||
|
BbfzipM8c8t5+g+FEqF3lhiRdDAfBgNVHSMEGDAWgBSubAWjkxPioufi1xzWx/B/
|
||||||
|
yGdToDB7BggrBgEFBQcBAQRvMG0wLgYIKwYBBQUHMAGGImh0dHA6Ly9vY3NwMi5n
|
||||||
|
bG9iYWxzaWduLmNvbS9yb290cjYwOwYIKwYBBQUHMAKGL2h0dHA6Ly9zZWN1cmUu
|
||||||
|
Z2xvYmFsc2lnbi5jb20vY2FjZXJ0L3Jvb3QtcjYuY3J0MDYGA1UdHwQvMC0wK6Ap
|
||||||
|
oCeGJWh0dHA6Ly9jcmwuZ2xvYmFsc2lnbi5jb20vcm9vdC1yNi5jcmwwIQYDVR0g
|
||||||
|
BBowGDAIBgZngQwBAgEwDAYKKwYBBAGgMgoBAzANBgkqhkiG9w0BAQsFAAOCAgEA
|
||||||
|
fMkkMo5g4mn1ft4d4xR2kHzYpDukhC1XYPwfSZN3A9nEBadjdKZMH7iuS1vF8uSc
|
||||||
|
g26/30DRPen2fFRsr662ECyUCR4OfeiiGNdoQvcesM9Xpew3HLQP4qHg+s774hNL
|
||||||
|
vGRD4aKSKwFqLMrcqCw6tEAfX99tFWsD4jzbC6k8tjSLzEl0fTUlfkJaWpvLVkpg
|
||||||
|
9et8tD8d51bymCg5J6J6wcXpmsSGnksBobac1+nXmgB7jQC9edU8Z41FFo87BV3k
|
||||||
|
CtrWWsdkQavObMsXUPl/AO8y/jOuAWz0wyvPnKom+o6W4vKDY6/6XPypNdebOJ6m
|
||||||
|
jyaILp0quoQvhjx87BzENh5s57AIOyIGpS0sDEChVDPzLEfRsH2FJ8/W5woF0nvs
|
||||||
|
BTqfYSCqblQbHeDDtCj7Mlf8JfqaMuqcbE4rMSyfeHyCdZQwnc/r9ujnth691AJh
|
||||||
|
xyYeCM04metJIe7cB6d4dFm+Pd5ervY4x32r0uQ1Q0spy1VjNqUJjussYuXNyMmF
|
||||||
|
HSuLQQ6PrePmH5lcSMQpYKzPoD/RiNVD/PK0O3vuO5vh3o7oKb1FfzoanDsFFTrw
|
||||||
|
0aLOdRW/tmLPWVNVlAb8ad+B80YJsL4HXYnQG8wYAFb8LhwSDyT9v+C1C1lcIHE7
|
||||||
|
nE0AAp9JSHxDYsma9pi4g0Phg3BgOm2euTRzw7R0SzU=
|
||||||
|
-----END CERTIFICATE-----
|
||||||
65
certificates/globalsign_r6_bundle.pem
Normal file
65
certificates/globalsign_r6_bundle.pem
Normal file
@@ -0,0 +1,65 @@
|
|||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIFgzCCA2ugAwIBAgIORea7A4Mzw4VlSOb/RVEwDQYJKoZIhvcNAQEMBQAwTDEg
|
||||||
|
MB4GA1UECxMXR2xvYmFsU2lnbiBSb290IENBIC0gUjYxEzARBgNVBAoTCkdsb2Jh
|
||||||
|
bFNpZ24xEzARBgNVBAMTCkdsb2JhbFNpZ24wHhcNMTQxMjEwMDAwMDAwWhcNMzQx
|
||||||
|
MjEwMDAwMDAwWjBMMSAwHgYDVQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSNjET
|
||||||
|
MBEGA1UEChMKR2xvYmFsU2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjCCAiIwDQYJ
|
||||||
|
KoZIhvcNAQEBBQADggIPADCCAgoCggIBAJUH6HPKZvnsFMp7PPcNCPG0RQssgrRI
|
||||||
|
xutbPK6DuEGSMxSkb3/pKszGsIhrxbaJ0cay/xTOURQh7ErdG1rG1ofuTToVBu1k
|
||||||
|
ZguSgMpE3nOUTvOniX9PeGMIyBJQbUJmL025eShNUhqKGoC3GYEOfsSKvGRMIRxD
|
||||||
|
aNc9PIrFsmbVkJq3MQbFvuJtMgamHvm566qjuL++gmNQ0PAYid/kD3n16qIfKtJw
|
||||||
|
LnvnvJO7bVPiSHyMEAc4/2ayd2F+4OqMPKq0pPbzlUoSB239jLKJz9CgYXfIWHSw
|
||||||
|
1CM69106yqLbnQneXUQtkPGBzVeS+n68UARjNN9rkxi+azayOeSsJDa38O+2HBNX
|
||||||
|
k7besvjihbdzorg1qkXy4J02oW9UivFyVm4uiMVRQkQVlO6jxTiWm05OWgtH8wY2
|
||||||
|
SXcwvHE35absIQh1/OZhFj931dmRl4QKbNQCTXTAFO39OfuD8l4UoQSwC+n+7o/h
|
||||||
|
bguyCLNhZglqsQY6ZZZZwPA1/cnaKI0aEYdwgQqomnUdnjqGBQCe24DWJfncBZ4n
|
||||||
|
WUx2OVvq+aWh2IMP0f/fMBH5hc8zSPXKbWQULHpYT9NLCEnFlWQaYw55PfWzjMpY
|
||||||
|
rZxCRXluDocZXFSxZba/jJvcE+kNb7gu3GduyYsRtYQUigAZcIN5kZeR1Bonvzce
|
||||||
|
MgfYFGM8KEyvAgMBAAGjYzBhMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTAD
|
||||||
|
AQH/MB0GA1UdDgQWBBSubAWjkxPioufi1xzWx/B/yGdToDAfBgNVHSMEGDAWgBSu
|
||||||
|
bAWjkxPioufi1xzWx/B/yGdToDANBgkqhkiG9w0BAQwFAAOCAgEAgyXt6NH9lVLN
|
||||||
|
nsAEoJFp5lzQhN7craJP6Ed41mWYqVuoPId8AorRbrcWc+ZfwFSY1XS+wc3iEZGt
|
||||||
|
Ixg93eFyRJa0lV7Ae46ZeBZDE1ZXs6KzO7V33EByrKPrmzU+sQghoefEQzd5Mr61
|
||||||
|
55wsTLxDKZmOMNOsIeDjHfrYBzN2VAAiKrlNIC5waNrlU/yDXNOd8v9EDERm8tLj
|
||||||
|
vUYAGm0CuiVdjaExUd1URhxN25mW7xocBFymFe944Hn+Xds+qkxV/ZoVqW/hpvvf
|
||||||
|
cDDpw+5CRu3CkwWJ+n1jez/QcYF8AOiYrg54NMMl+68KnyBr3TsTjxKM4kEaSHpz
|
||||||
|
oHdpx7Zcf4LIHv5YGygrqGytXm3ABdJ7t+uA/iU3/gKbaKxCXcPu9czc8FB10jZp
|
||||||
|
nOZ7BN9uBmm23goJSFmH63sUYHpkqmlD75HHTOwY3WzvUy2MmeFe8nI+z1TIvWfs
|
||||||
|
pA9MRf/TuTAjB0yPEL+GltmZWrSZVxykzLsViVO6LAUP5MSeGbEYNNVMnbrt9x+v
|
||||||
|
JJUEeKgDu+6B5dpffItKoZB0JaezPkvILFa9x8jvOOJckvB595yEunQtYQEgfn7R
|
||||||
|
8k8HWV+LLUNS60YMlOH1Zkd5d9VUWx+tJDfLRVpOoERIyNiwmcUVhAn21klJwGW4
|
||||||
|
5hpxbqCo8YLoRT5s1gLXCmeDBVrJpBA=
|
||||||
|
-----END CERTIFICATE-----
|
||||||
|
|
||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIFjDCCA3SgAwIBAgIQfx8skC6D0OO2+zvuR4tegDANBgkqhkiG9w0BAQsFADBM
|
||||||
|
MSAwHgYDVQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSNjETMBEGA1UEChMKR2xv
|
||||||
|
YmFsU2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjAeFw0yMzA3MTkwMzQzMjVaFw0y
|
||||||
|
NjA3MTkwMDAwMDBaMFUxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9iYWxTaWdu
|
||||||
|
IG52LXNhMSswKQYDVQQDEyJHbG9iYWxTaWduIEdDQyBSNiBBbHBoYVNTTCBDQSAy
|
||||||
|
MDIzMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA00Jvk5ADppO0rgDn
|
||||||
|
j1M14XIb032Aas409JJFAb8cUjipFOth7ySLdaWLe3s63oSs5x3eWwzTpX4BFkzZ
|
||||||
|
bxT1eoJSHfT2M0wZ5QOPcCIjsr+YB8TAvV2yJSyq+emRrN/FtgCSTaWXSJ5jipW8
|
||||||
|
SJ/VAuXPMzuAP2yYpuPcjjQ5GyrssDXgu+FhtYxqyFP7BSvx9jQhh5QV5zhLycua
|
||||||
|
n8n+J0Uw09WRQK6JGQ5HzDZQinkNel+fZZNRG1gE9Qeh+tHBplrkalB1g85qJkPO
|
||||||
|
J7SoEvKsmDkajggk/sSq7NPyzFaa/VBGZiRRG+FkxCBniGD5618PQ4trcwHyMojS
|
||||||
|
FObOHQIDAQABo4IBXzCCAVswDgYDVR0PAQH/BAQDAgGGMB0GA1UdJQQWMBQGCCsG
|
||||||
|
AQUFBwMBBggrBgEFBQcDAjASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1UdDgQWBBS9
|
||||||
|
BbfzipM8c8t5+g+FEqF3lhiRdDAfBgNVHSMEGDAWgBSubAWjkxPioufi1xzWx/B/
|
||||||
|
yGdToDB7BggrBgEFBQcBAQRvMG0wLgYIKwYBBQUHMAGGImh0dHA6Ly9vY3NwMi5n
|
||||||
|
bG9iYWxzaWduLmNvbS9yb290cjYwOwYIKwYBBQUHMAKGL2h0dHA6Ly9zZWN1cmUu
|
||||||
|
Z2xvYmFsc2lnbi5jb20vY2FjZXJ0L3Jvb3QtcjYuY3J0MDYGA1UdHwQvMC0wK6Ap
|
||||||
|
oCeGJWh0dHA6Ly9jcmwuZ2xvYmFsc2lnbi5jb20vcm9vdC1yNi5jcmwwIQYDVR0g
|
||||||
|
BBowGDAIBgZngQwBAgEwDAYKKwYBBAGgMgoBAzANBgkqhkiG9w0BAQsFAAOCAgEA
|
||||||
|
fMkkMo5g4mn1ft4d4xR2kHzYpDukhC1XYPwfSZN3A9nEBadjdKZMH7iuS1vF8uSc
|
||||||
|
g26/30DRPen2fFRsr662ECyUCR4OfeiiGNdoQvcesM9Xpew3HLQP4qHg+s774hNL
|
||||||
|
vGRD4aKSKwFqLMrcqCw6tEAfX99tFWsD4jzbC6k8tjSLzEl0fTUlfkJaWpvLVkpg
|
||||||
|
9et8tD8d51bymCg5J6J6wcXpmsSGnksBobac1+nXmgB7jQC9edU8Z41FFo87BV3k
|
||||||
|
CtrWWsdkQavObMsXUPl/AO8y/jOuAWz0wyvPnKom+o6W4vKDY6/6XPypNdebOJ6m
|
||||||
|
jyaILp0quoQvhjx87BzENh5s57AIOyIGpS0sDEChVDPzLEfRsH2FJ8/W5woF0nvs
|
||||||
|
BTqfYSCqblQbHeDDtCj7Mlf8JfqaMuqcbE4rMSyfeHyCdZQwnc/r9ujnth691AJh
|
||||||
|
xyYeCM04metJIe7cB6d4dFm+Pd5ervY4x32r0uQ1Q0spy1VjNqUJjussYuXNyMmF
|
||||||
|
HSuLQQ6PrePmH5lcSMQpYKzPoD/RiNVD/PK0O3vuO5vh3o7oKb1FfzoanDsFFTrw
|
||||||
|
0aLOdRW/tmLPWVNVlAb8ad+B80YJsL4HXYnQG8wYAFb8LhwSDyT9v+C1C1lcIHE7
|
||||||
|
nE0AAp9JSHxDYsma9pi4g0Phg3BgOm2euTRzw7R0SzU=
|
||||||
|
-----END CERTIFICATE-----
|
||||||
32
certificates/globalsign_root_r6.pem
Normal file
32
certificates/globalsign_root_r6.pem
Normal file
@@ -0,0 +1,32 @@
|
|||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIFgzCCA2ugAwIBAgIORea7A4Mzw4VlSOb/RVEwDQYJKoZIhvcNAQEMBQAwTDEg
|
||||||
|
MB4GA1UECxMXR2xvYmFsU2lnbiBSb290IENBIC0gUjYxEzARBgNVBAoTCkdsb2Jh
|
||||||
|
bFNpZ24xEzARBgNVBAMTCkdsb2JhbFNpZ24wHhcNMTQxMjEwMDAwMDAwWhcNMzQx
|
||||||
|
MjEwMDAwMDAwWjBMMSAwHgYDVQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSNjET
|
||||||
|
MBEGA1UEChMKR2xvYmFsU2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjCCAiIwDQYJ
|
||||||
|
KoZIhvcNAQEBBQADggIPADCCAgoCggIBAJUH6HPKZvnsFMp7PPcNCPG0RQssgrRI
|
||||||
|
xutbPK6DuEGSMxSkb3/pKszGsIhrxbaJ0cay/xTOURQh7ErdG1rG1ofuTToVBu1k
|
||||||
|
ZguSgMpE3nOUTvOniX9PeGMIyBJQbUJmL025eShNUhqKGoC3GYEOfsSKvGRMIRxD
|
||||||
|
aNc9PIrFsmbVkJq3MQbFvuJtMgamHvm566qjuL++gmNQ0PAYid/kD3n16qIfKtJw
|
||||||
|
LnvnvJO7bVPiSHyMEAc4/2ayd2F+4OqMPKq0pPbzlUoSB239jLKJz9CgYXfIWHSw
|
||||||
|
1CM69106yqLbnQneXUQtkPGBzVeS+n68UARjNN9rkxi+azayOeSsJDa38O+2HBNX
|
||||||
|
k7besvjihbdzorg1qkXy4J02oW9UivFyVm4uiMVRQkQVlO6jxTiWm05OWgtH8wY2
|
||||||
|
SXcwvHE35absIQh1/OZhFj931dmRl4QKbNQCTXTAFO39OfuD8l4UoQSwC+n+7o/h
|
||||||
|
bguyCLNhZglqsQY6ZZZZwPA1/cnaKI0aEYdwgQqomnUdnjqGBQCe24DWJfncBZ4n
|
||||||
|
WUx2OVvq+aWh2IMP0f/fMBH5hc8zSPXKbWQULHpYT9NLCEnFlWQaYw55PfWzjMpY
|
||||||
|
rZxCRXluDocZXFSxZba/jJvcE+kNb7gu3GduyYsRtYQUigAZcIN5kZeR1Bonvzce
|
||||||
|
MgfYFGM8KEyvAgMBAAGjYzBhMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTAD
|
||||||
|
AQH/MB0GA1UdDgQWBBSubAWjkxPioufi1xzWx/B/yGdToDAfBgNVHSMEGDAWgBSu
|
||||||
|
bAWjkxPioufi1xzWx/B/yGdToDANBgkqhkiG9w0BAQwFAAOCAgEAgyXt6NH9lVLN
|
||||||
|
nsAEoJFp5lzQhN7craJP6Ed41mWYqVuoPId8AorRbrcWc+ZfwFSY1XS+wc3iEZGt
|
||||||
|
Ixg93eFyRJa0lV7Ae46ZeBZDE1ZXs6KzO7V33EByrKPrmzU+sQghoefEQzd5Mr61
|
||||||
|
55wsTLxDKZmOMNOsIeDjHfrYBzN2VAAiKrlNIC5waNrlU/yDXNOd8v9EDERm8tLj
|
||||||
|
vUYAGm0CuiVdjaExUd1URhxN25mW7xocBFymFe944Hn+Xds+qkxV/ZoVqW/hpvvf
|
||||||
|
cDDpw+5CRu3CkwWJ+n1jez/QcYF8AOiYrg54NMMl+68KnyBr3TsTjxKM4kEaSHpz
|
||||||
|
oHdpx7Zcf4LIHv5YGygrqGytXm3ABdJ7t+uA/iU3/gKbaKxCXcPu9czc8FB10jZp
|
||||||
|
nOZ7BN9uBmm23goJSFmH63sUYHpkqmlD75HHTOwY3WzvUy2MmeFe8nI+z1TIvWfs
|
||||||
|
pA9MRf/TuTAjB0yPEL+GltmZWrSZVxykzLsViVO6LAUP5MSeGbEYNNVMnbrt9x+v
|
||||||
|
JJUEeKgDu+6B5dpffItKoZB0JaezPkvILFa9x8jvOOJckvB595yEunQtYQEgfn7R
|
||||||
|
8k8HWV+LLUNS60YMlOH1Zkd5d9VUWx+tJDfLRVpOoERIyNiwmcUVhAn21klJwGW4
|
||||||
|
5hpxbqCo8YLoRT5s1gLXCmeDBVrJpBA=
|
||||||
|
-----END CERTIFICATE-----
|
||||||
74
certificates/russian_trusted_bundle.pem
Normal file
74
certificates/russian_trusted_bundle.pem
Normal file
@@ -0,0 +1,74 @@
|
|||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIFwjCCA6qgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwcDELMAkGA1UEBhMCUlUx
|
||||||
|
PzA9BgNVBAoMNlRoZSBNaW5pc3RyeSBvZiBEaWdpdGFsIERldmVsb3BtZW50IGFu
|
||||||
|
ZCBDb21tdW5pY2F0aW9uczEgMB4GA1UEAwwXUnVzc2lhbiBUcnVzdGVkIFJvb3Qg
|
||||||
|
Q0EwHhcNMjIwMzAxMjEwNDE1WhcNMzIwMjI3MjEwNDE1WjBwMQswCQYDVQQGEwJS
|
||||||
|
VTE/MD0GA1UECgw2VGhlIE1pbmlzdHJ5IG9mIERpZ2l0YWwgRGV2ZWxvcG1lbnQg
|
||||||
|
YW5kIENvbW11bmljYXRpb25zMSAwHgYDVQQDDBdSdXNzaWFuIFRydXN0ZWQgUm9v
|
||||||
|
dCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMfFOZ8pUAL3+r2n
|
||||||
|
qqE0Zp52selXsKGFYoG0GM5bwz1bSFtCt+AZQMhkWQheI3poZAToYJu69pHLKS6Q
|
||||||
|
XBiwBC1cvzYmUYKMYZC7jE5YhEU2bSL0mX7NaMxMDmH2/NwuOVRj8OImVa5s1F4U
|
||||||
|
zn4Kv3PFlDBjjSjXKVY9kmjUBsXQrIHeaqmUIsPIlNWUnimXS0I0abExqkbdrXbX
|
||||||
|
YwCOXhOO2pDUx3ckmJlCMUGacUTnylyQW2VsJIyIGA8V0xzdaeUXg0VZ6ZmNUr5Y
|
||||||
|
Ber/EAOLPb8NYpsAhJe2mXjMB/J9HNsoFMBFJ0lLOT/+dQvjbdRZoOT8eqJpWnVD
|
||||||
|
U+QL/qEZnz57N88OWM3rabJkRNdU/Z7x5SFIM9FrqtN8xewsiBWBI0K6XFuOBOTD
|
||||||
|
4V08o4TzJ8+Ccq5XlCUW2L48pZNCYuBDfBh7FxkB7qDgGDiaftEkZZfApRg2E+M9
|
||||||
|
G8wkNKTPLDc4wH0FDTijhgxR3Y4PiS1HL2Zhw7bD3CbslmEGgfnnZojNkJtcLeBH
|
||||||
|
BLa52/dSwNU4WWLubaYSiAmA9IUMX1/RpfpxOxd4Ykmhz97oFbUaDJFipIggx5sX
|
||||||
|
ePAlkTdWnv+RWBxlJwMQ25oEHmRguNYf4Zr/Rxr9cS93Y+mdXIZaBEE0KS2iLRqa
|
||||||
|
OiWBki9IMQU4phqPOBAaG7A+eP8PAgMBAAGjZjBkMB0GA1UdDgQWBBTh0YHlzlpf
|
||||||
|
BKrS6badZrHF+qwshzAfBgNVHSMEGDAWgBTh0YHlzlpfBKrS6badZrHF+qwshzAS
|
||||||
|
BgNVHRMBAf8ECDAGAQH/AgEEMA4GA1UdDwEB/wQEAwIBhjANBgkqhkiG9w0BAQsF
|
||||||
|
AAOCAgEAALIY1wkilt/urfEVM5vKzr6utOeDWCUczmWX/RX4ljpRdgF+5fAIS4vH
|
||||||
|
tmXkqpSCOVeWUrJV9QvZn6L227ZwuE15cWi8DCDal3Ue90WgAJJZMfTshN4OI8cq
|
||||||
|
W9E4EG9wglbEtMnObHlms8F3CHmrw3k6KmUkWGoa+/ENmcVl68u/cMRl1JbW2bM+
|
||||||
|
/3A+SAg2c6iPDlehczKx2oa95QW0SkPPWGuNA/CE8CpyANIhu9XFrj3RQ3EqeRcS
|
||||||
|
AQQod1RNuHpfETLU/A2gMmvn/w/sx7TB3W5BPs6rprOA37tutPq9u6FTZOcG1Oqj
|
||||||
|
C/B7yTqgI7rbyvox7DEXoX7rIiEqyNNUguTk/u3SZ4VXE2kmxdmSh3TQvybfbnXV
|
||||||
|
4JbCZVaqiZraqc7oZMnRoWrXRG3ztbnbes/9qhRGI7PqXqeKJBztxRTEVj8ONs1d
|
||||||
|
WN5szTwaPIvhkhO3CO5ErU2rVdUr89wKpNXbBODFKRtgxUT70YpmJ46VVaqdAhOZ
|
||||||
|
D9EUUn4YaeLaS8AjSF/h7UkjOibNc4qVDiPP+rkehFWM66PVnP1Msh93tc+taIfC
|
||||||
|
EYVMxjh8zNbFuoc7fzvvrFILLe7ifvEIUqSVIC/AzplM/Jxw7buXFeGP1qVCBEHq
|
||||||
|
391d/9RAfaZ12zkwFsl+IKwE/OZxW8AHa9i1p4GO0YSNuczzEm4=
|
||||||
|
-----END CERTIFICATE-----
|
||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIHQjCCBSqgAwIBAgICEAIwDQYJKoZIhvcNAQELBQAwcDELMAkGA1UEBhMCUlUx
|
||||||
|
PzA9BgNVBAoMNlRoZSBNaW5pc3RyeSBvZiBEaWdpdGFsIERldmVsb3BtZW50IGFu
|
||||||
|
ZCBDb21tdW5pY2F0aW9uczEgMB4GA1UEAwwXUnVzc2lhbiBUcnVzdGVkIFJvb3Qg
|
||||||
|
Q0EwHhcNMjIwMzAyMTEyNTE5WhcNMjcwMzA2MTEyNTE5WjBvMQswCQYDVQQGEwJS
|
||||||
|
VTE/MD0GA1UECgw2VGhlIE1pbmlzdHJ5IG9mIERpZ2l0YWwgRGV2ZWxvcG1lbnQg
|
||||||
|
YW5kIENvbW11bmljYXRpb25zMR8wHQYDVQQDDBZSdXNzaWFuIFRydXN0ZWQgU3Vi
|
||||||
|
IENBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA9YPqBKOk19NFymrE
|
||||||
|
wehzrhBEgT2atLezpduB24mQ7CiOa/HVpFCDRZzdxqlh8drku408/tTmWzlNH/br
|
||||||
|
HuQhZ/miWKOf35lpKzjyBd6TPM23uAfJvEOQ2/dnKGGJbsUo1/udKSvxQwVHpVv3
|
||||||
|
S80OlluKfhWPDEXQpgyFqIzPoxIQTLZ0deirZwMVHarZ5u8HqHetRuAtmO2ZDGQn
|
||||||
|
vVOJYAjls+Hiueq7Lj7Oce7CQsTwVZeP+XQx28PAaEZ3y6sQEt6rL06ddpSdoTMp
|
||||||
|
BnCqTbxW+eWMyjkIn6t9GBtUV45yB1EkHNnj2Ex4GwCiN9T84QQjKSr+8f0psGrZ
|
||||||
|
vPbCbQAwNFJjisLixnjlGPLKa5vOmNwIh/LAyUW5DjpkCx004LPDuqPpFsKXNKpa
|
||||||
|
L2Dm6uc0x4Jo5m+gUTVORB6hOSzWnWDj2GWfomLzzyjG81DRGFBpco/O93zecsIN
|
||||||
|
3SL2Ysjpq1zdoS01CMYxie//9zWvYwzI25/OZigtnpCIrcd2j1Y6dMUFQAzAtHE+
|
||||||
|
qsXflSL8HIS+IJEFIQobLlYhHkoE3avgNx5jlu+OLYe0dF0Ykx1PGNjbwqvTX37R
|
||||||
|
Cn32NMjlotW2QcGEZhDKj+3urZizp5xdTPZitA+aEjZM/Ni71VOdiOP0igbw6asZ
|
||||||
|
2fxdozZ1TnSSYNYvNATwthNmZysCAwEAAaOCAeUwggHhMBIGA1UdEwEB/wQIMAYB
|
||||||
|
Af8CAQAwDgYDVR0PAQH/BAQDAgGGMB0GA1UdDgQWBBTR4XENCy2BTm6KSo9MI7NM
|
||||||
|
XqtpCzAfBgNVHSMEGDAWgBTh0YHlzlpfBKrS6badZrHF+qwshzCBxwYIKwYBBQUH
|
||||||
|
AQEEgbowgbcwOwYIKwYBBQUHMAKGL2h0dHA6Ly9yb3N0ZWxlY29tLnJ1L2NkcC9y
|
||||||
|
b290Y2Ffc3NsX3JzYTIwMjIuY3J0MDsGCCsGAQUFBzAChi9odHRwOi8vY29tcGFu
|
||||||
|
eS5ydC5ydS9jZHAvcm9vdGNhX3NzbF9yc2EyMDIyLmNydDA7BggrBgEFBQcwAoYv
|
||||||
|
aHR0cDovL3JlZXN0ci1wa2kucnUvY2RwL3Jvb3RjYV9zc2xfcnNhMjAyMi5jcnQw
|
||||||
|
gbAGA1UdHwSBqDCBpTA1oDOgMYYvaHR0cDovL3Jvc3RlbGVjb20ucnUvY2RwL3Jv
|
||||||
|
b3RjYV9zc2xfcnNhMjAyMi5jcmwwNaAzoDGGL2h0dHA6Ly9jb21wYW55LnJ0LnJ1
|
||||||
|
L2NkcC9yb290Y2Ffc3NsX3JzYTIwMjIuY3JsMDWgM6Axhi9odHRwOi8vcmVlc3Ry
|
||||||
|
LXBraS5ydS9jZHAvcm9vdGNhX3NzbF9yc2EyMDIyLmNybDANBgkqhkiG9w0BAQsF
|
||||||
|
AAOCAgEARBVzZls79AdiSCpar15dA5Hr/rrT4WbrOfzlpI+xrLeRPrUG6eUWIW4v
|
||||||
|
Sui1yx3iqGLCjPcKb+HOTwoRMbI6ytP/ndp3TlYua2advYBEhSvjs+4vDZNwXr/D
|
||||||
|
anbwIWdurZmViQRBDFebpkvnIvru/RpWud/5r624Wp8voZMRtj/cm6aI9LtvBfT9
|
||||||
|
cfzhOaexI/99c14dyiuk1+6QhdwKaCRTc1mdfNQmnfWNRbfWhWBlK3h4GGE9JK33
|
||||||
|
Gk8ZS8DMrkdAh0xby4xAQ/mSWAfWrBmfzlOqGyoB1U47WTOeqNbWkkoAP2ys94+s
|
||||||
|
Jg4NTkiDVtXRF6nr6fYi0bSOvOFg0IQrMXO2Y8gyg9ARdPJwKtvWX8VPADCYMiWH
|
||||||
|
h4n8bZokIrImVKLDQKHY4jCsND2HHdJfnrdL2YJw1qFskNO4cSNmZydw0Wkgjv9k
|
||||||
|
F+KxqrDKlB8MZu2Hclph6v/CZ0fQ9YuE8/lsHZ0Qc2HyiSMnvjgK5fDc3TD4fa8F
|
||||||
|
E8gMNurM+kV8PT8LNIM+4Zs+LKEV8nqRWBaxkIVJGekkVKO8xDBOG/aN62AZKHOe
|
||||||
|
GcyIdu7yNMMRihGVZCYr8rYiJoKiOzDqOkPkLOPdhtVlgnhowzHDxMHND/E2WA5p
|
||||||
|
ZHuNM/m0TXt2wTTPL7JH2YC0gPz/BvvSzjksgzU5rLbRyUKQkgU=
|
||||||
|
-----END CERTIFICATE-----
|
||||||
33
certificates/russian_trusted_root_ca.pem
Normal file
33
certificates/russian_trusted_root_ca.pem
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIFwjCCA6qgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwcDELMAkGA1UEBhMCUlUx
|
||||||
|
PzA9BgNVBAoMNlRoZSBNaW5pc3RyeSBvZiBEaWdpdGFsIERldmVsb3BtZW50IGFu
|
||||||
|
ZCBDb21tdW5pY2F0aW9uczEgMB4GA1UEAwwXUnVzc2lhbiBUcnVzdGVkIFJvb3Qg
|
||||||
|
Q0EwHhcNMjIwMzAxMjEwNDE1WhcNMzIwMjI3MjEwNDE1WjBwMQswCQYDVQQGEwJS
|
||||||
|
VTE/MD0GA1UECgw2VGhlIE1pbmlzdHJ5IG9mIERpZ2l0YWwgRGV2ZWxvcG1lbnQg
|
||||||
|
YW5kIENvbW11bmljYXRpb25zMSAwHgYDVQQDDBdSdXNzaWFuIFRydXN0ZWQgUm9v
|
||||||
|
dCBDQTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAMfFOZ8pUAL3+r2n
|
||||||
|
qqE0Zp52selXsKGFYoG0GM5bwz1bSFtCt+AZQMhkWQheI3poZAToYJu69pHLKS6Q
|
||||||
|
XBiwBC1cvzYmUYKMYZC7jE5YhEU2bSL0mX7NaMxMDmH2/NwuOVRj8OImVa5s1F4U
|
||||||
|
zn4Kv3PFlDBjjSjXKVY9kmjUBsXQrIHeaqmUIsPIlNWUnimXS0I0abExqkbdrXbX
|
||||||
|
YwCOXhOO2pDUx3ckmJlCMUGacUTnylyQW2VsJIyIGA8V0xzdaeUXg0VZ6ZmNUr5Y
|
||||||
|
Ber/EAOLPb8NYpsAhJe2mXjMB/J9HNsoFMBFJ0lLOT/+dQvjbdRZoOT8eqJpWnVD
|
||||||
|
U+QL/qEZnz57N88OWM3rabJkRNdU/Z7x5SFIM9FrqtN8xewsiBWBI0K6XFuOBOTD
|
||||||
|
4V08o4TzJ8+Ccq5XlCUW2L48pZNCYuBDfBh7FxkB7qDgGDiaftEkZZfApRg2E+M9
|
||||||
|
G8wkNKTPLDc4wH0FDTijhgxR3Y4PiS1HL2Zhw7bD3CbslmEGgfnnZojNkJtcLeBH
|
||||||
|
BLa52/dSwNU4WWLubaYSiAmA9IUMX1/RpfpxOxd4Ykmhz97oFbUaDJFipIggx5sX
|
||||||
|
ePAlkTdWnv+RWBxlJwMQ25oEHmRguNYf4Zr/Rxr9cS93Y+mdXIZaBEE0KS2iLRqa
|
||||||
|
OiWBki9IMQU4phqPOBAaG7A+eP8PAgMBAAGjZjBkMB0GA1UdDgQWBBTh0YHlzlpf
|
||||||
|
BKrS6badZrHF+qwshzAfBgNVHSMEGDAWgBTh0YHlzlpfBKrS6badZrHF+qwshzAS
|
||||||
|
BgNVHRMBAf8ECDAGAQH/AgEEMA4GA1UdDwEB/wQEAwIBhjANBgkqhkiG9w0BAQsF
|
||||||
|
AAOCAgEAALIY1wkilt/urfEVM5vKzr6utOeDWCUczmWX/RX4ljpRdgF+5fAIS4vH
|
||||||
|
tmXkqpSCOVeWUrJV9QvZn6L227ZwuE15cWi8DCDal3Ue90WgAJJZMfTshN4OI8cq
|
||||||
|
W9E4EG9wglbEtMnObHlms8F3CHmrw3k6KmUkWGoa+/ENmcVl68u/cMRl1JbW2bM+
|
||||||
|
/3A+SAg2c6iPDlehczKx2oa95QW0SkPPWGuNA/CE8CpyANIhu9XFrj3RQ3EqeRcS
|
||||||
|
AQQod1RNuHpfETLU/A2gMmvn/w/sx7TB3W5BPs6rprOA37tutPq9u6FTZOcG1Oqj
|
||||||
|
C/B7yTqgI7rbyvox7DEXoX7rIiEqyNNUguTk/u3SZ4VXE2kmxdmSh3TQvybfbnXV
|
||||||
|
4JbCZVaqiZraqc7oZMnRoWrXRG3ztbnbes/9qhRGI7PqXqeKJBztxRTEVj8ONs1d
|
||||||
|
WN5szTwaPIvhkhO3CO5ErU2rVdUr89wKpNXbBODFKRtgxUT70YpmJ46VVaqdAhOZ
|
||||||
|
D9EUUn4YaeLaS8AjSF/h7UkjOibNc4qVDiPP+rkehFWM66PVnP1Msh93tc+taIfC
|
||||||
|
EYVMxjh8zNbFuoc7fzvvrFILLe7ifvEIUqSVIC/AzplM/Jxw7buXFeGP1qVCBEHq
|
||||||
|
391d/9RAfaZ12zkwFsl+IKwE/OZxW8AHa9i1p4GO0YSNuczzEm4=
|
||||||
|
-----END CERTIFICATE-----
|
||||||
@@ -1,10 +1,41 @@
|
|||||||
Title: Госуслуги
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIHQjCCBSqgAwIBAgICEAIwDQYJKoZIhvcNAQELBQAwcDELMAkGA1UEBhMCUlUx
|
||||||
URL Source: https://www.gosuslugi.ru/tls/files/subca2022.cer
|
PzA9BgNVBAoMNlRoZSBNaW5pc3RyeSBvZiBEaWdpdGFsIERldmVsb3BtZW50IGFu
|
||||||
|
ZCBDb21tdW5pY2F0aW9uczEgMB4GA1UEAwwXUnVzc2lhbiBUcnVzdGVkIFJvb3Qg
|
||||||
Markdown Content:
|
Q0EwHhcNMjIwMzAyMTEyNTE5WhcNMjcwMzA2MTEyNTE5WjBvMQswCQYDVQQGEwJS
|
||||||
### Госуслуги сейчас откроются
|
VTE/MD0GA1UECgw2VGhlIE1pbmlzdHJ5IG9mIERpZ2l0YWwgRGV2ZWxvcG1lbnQg
|
||||||
|
YW5kIENvbW11bmljYXRpb25zMR8wHQYDVQQDDBZSdXNzaWFuIFRydXN0ZWQgU3Vi
|
||||||
Портал работает в прежнем режиме.
|
IENBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA9YPqBKOk19NFymrE
|
||||||
|
wehzrhBEgT2atLezpduB24mQ7CiOa/HVpFCDRZzdxqlh8drku408/tTmWzlNH/br
|
||||||
Подождите пару секунд
|
HuQhZ/miWKOf35lpKzjyBd6TPM23uAfJvEOQ2/dnKGGJbsUo1/udKSvxQwVHpVv3
|
||||||
|
S80OlluKfhWPDEXQpgyFqIzPoxIQTLZ0deirZwMVHarZ5u8HqHetRuAtmO2ZDGQn
|
||||||
|
vVOJYAjls+Hiueq7Lj7Oce7CQsTwVZeP+XQx28PAaEZ3y6sQEt6rL06ddpSdoTMp
|
||||||
|
BnCqTbxW+eWMyjkIn6t9GBtUV45yB1EkHNnj2Ex4GwCiN9T84QQjKSr+8f0psGrZ
|
||||||
|
vPbCbQAwNFJjisLixnjlGPLKa5vOmNwIh/LAyUW5DjpkCx004LPDuqPpFsKXNKpa
|
||||||
|
L2Dm6uc0x4Jo5m+gUTVORB6hOSzWnWDj2GWfomLzzyjG81DRGFBpco/O93zecsIN
|
||||||
|
3SL2Ysjpq1zdoS01CMYxie//9zWvYwzI25/OZigtnpCIrcd2j1Y6dMUFQAzAtHE+
|
||||||
|
qsXflSL8HIS+IJEFIQobLlYhHkoE3avgNx5jlu+OLYe0dF0Ykx1PGNjbwqvTX37R
|
||||||
|
Cn32NMjlotW2QcGEZhDKj+3urZizp5xdTPZitA+aEjZM/Ni71VOdiOP0igbw6asZ
|
||||||
|
2fxdozZ1TnSSYNYvNATwthNmZysCAwEAAaOCAeUwggHhMBIGA1UdEwEB/wQIMAYB
|
||||||
|
Af8CAQAwDgYDVR0PAQH/BAQDAgGGMB0GA1UdDgQWBBTR4XENCy2BTm6KSo9MI7NM
|
||||||
|
XqtpCzAfBgNVHSMEGDAWgBTh0YHlzlpfBKrS6badZrHF+qwshzCBxwYIKwYBBQUH
|
||||||
|
AQEEgbowgbcwOwYIKwYBBQUHMAKGL2h0dHA6Ly9yb3N0ZWxlY29tLnJ1L2NkcC9y
|
||||||
|
b290Y2Ffc3NsX3JzYTIwMjIuY3J0MDsGCCsGAQUFBzAChi9odHRwOi8vY29tcGFu
|
||||||
|
eS5ydC5ydS9jZHAvcm9vdGNhX3NzbF9yc2EyMDIyLmNydDA7BggrBgEFBQcwAoYv
|
||||||
|
aHR0cDovL3JlZXN0ci1wa2kucnUvY2RwL3Jvb3RjYV9zc2xfcnNhMjAyMi5jcnQw
|
||||||
|
gbAGA1UdHwSBqDCBpTA1oDOgMYYvaHR0cDovL3Jvc3RlbGVjb20ucnUvY2RwL3Jv
|
||||||
|
b3RjYV9zc2xfcnNhMjAyMi5jcmwwNaAzoDGGL2h0dHA6Ly9jb21wYW55LnJ0LnJ1
|
||||||
|
L2NkcC9yb290Y2Ffc3NsX3JzYTIwMjIuY3JsMDWgM6Axhi9odHRwOi8vcmVlc3Ry
|
||||||
|
LXBraS5ydS9jZHAvcm9vdGNhX3NzbF9yc2EyMDIyLmNybDANBgkqhkiG9w0BAQsF
|
||||||
|
AAOCAgEARBVzZls79AdiSCpar15dA5Hr/rrT4WbrOfzlpI+xrLeRPrUG6eUWIW4v
|
||||||
|
Sui1yx3iqGLCjPcKb+HOTwoRMbI6ytP/ndp3TlYua2advYBEhSvjs+4vDZNwXr/D
|
||||||
|
anbwIWdurZmViQRBDFebpkvnIvru/RpWud/5r624Wp8voZMRtj/cm6aI9LtvBfT9
|
||||||
|
cfzhOaexI/99c14dyiuk1+6QhdwKaCRTc1mdfNQmnfWNRbfWhWBlK3h4GGE9JK33
|
||||||
|
Gk8ZS8DMrkdAh0xby4xAQ/mSWAfWrBmfzlOqGyoB1U47WTOeqNbWkkoAP2ys94+s
|
||||||
|
Jg4NTkiDVtXRF6nr6fYi0bSOvOFg0IQrMXO2Y8gyg9ARdPJwKtvWX8VPADCYMiWH
|
||||||
|
h4n8bZokIrImVKLDQKHY4jCsND2HHdJfnrdL2YJw1qFskNO4cSNmZydw0Wkgjv9k
|
||||||
|
F+KxqrDKlB8MZu2Hclph6v/CZ0fQ9YuE8/lsHZ0Qc2HyiSMnvjgK5fDc3TD4fa8F
|
||||||
|
E8gMNurM+kV8PT8LNIM+4Zs+LKEV8nqRWBaxkIVJGekkVKO8xDBOG/aN62AZKHOe
|
||||||
|
GcyIdu7yNMMRihGVZCYr8rYiJoKiOzDqOkPkLOPdhtVlgnhowzHDxMHND/E2WA5p
|
||||||
|
ZHuNM/m0TXt2wTTPL7JH2YC0gPz/BvvSzjksgzU5rLbRyUKQkgU=
|
||||||
|
-----END CERTIFICATE-----
|
||||||
|
|||||||
41
certificates/russian_trusted_sub_ca.pem
Normal file
41
certificates/russian_trusted_sub_ca.pem
Normal file
@@ -0,0 +1,41 @@
|
|||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIHQjCCBSqgAwIBAgICEAIwDQYJKoZIhvcNAQELBQAwcDELMAkGA1UEBhMCUlUx
|
||||||
|
PzA9BgNVBAoMNlRoZSBNaW5pc3RyeSBvZiBEaWdpdGFsIERldmVsb3BtZW50IGFu
|
||||||
|
ZCBDb21tdW5pY2F0aW9uczEgMB4GA1UEAwwXUnVzc2lhbiBUcnVzdGVkIFJvb3Qg
|
||||||
|
Q0EwHhcNMjIwMzAyMTEyNTE5WhcNMjcwMzA2MTEyNTE5WjBvMQswCQYDVQQGEwJS
|
||||||
|
VTE/MD0GA1UECgw2VGhlIE1pbmlzdHJ5IG9mIERpZ2l0YWwgRGV2ZWxvcG1lbnQg
|
||||||
|
YW5kIENvbW11bmljYXRpb25zMR8wHQYDVQQDDBZSdXNzaWFuIFRydXN0ZWQgU3Vi
|
||||||
|
IENBMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA9YPqBKOk19NFymrE
|
||||||
|
wehzrhBEgT2atLezpduB24mQ7CiOa/HVpFCDRZzdxqlh8drku408/tTmWzlNH/br
|
||||||
|
HuQhZ/miWKOf35lpKzjyBd6TPM23uAfJvEOQ2/dnKGGJbsUo1/udKSvxQwVHpVv3
|
||||||
|
S80OlluKfhWPDEXQpgyFqIzPoxIQTLZ0deirZwMVHarZ5u8HqHetRuAtmO2ZDGQn
|
||||||
|
vVOJYAjls+Hiueq7Lj7Oce7CQsTwVZeP+XQx28PAaEZ3y6sQEt6rL06ddpSdoTMp
|
||||||
|
BnCqTbxW+eWMyjkIn6t9GBtUV45yB1EkHNnj2Ex4GwCiN9T84QQjKSr+8f0psGrZ
|
||||||
|
vPbCbQAwNFJjisLixnjlGPLKa5vOmNwIh/LAyUW5DjpkCx004LPDuqPpFsKXNKpa
|
||||||
|
L2Dm6uc0x4Jo5m+gUTVORB6hOSzWnWDj2GWfomLzzyjG81DRGFBpco/O93zecsIN
|
||||||
|
3SL2Ysjpq1zdoS01CMYxie//9zWvYwzI25/OZigtnpCIrcd2j1Y6dMUFQAzAtHE+
|
||||||
|
qsXflSL8HIS+IJEFIQobLlYhHkoE3avgNx5jlu+OLYe0dF0Ykx1PGNjbwqvTX37R
|
||||||
|
Cn32NMjlotW2QcGEZhDKj+3urZizp5xdTPZitA+aEjZM/Ni71VOdiOP0igbw6asZ
|
||||||
|
2fxdozZ1TnSSYNYvNATwthNmZysCAwEAAaOCAeUwggHhMBIGA1UdEwEB/wQIMAYB
|
||||||
|
Af8CAQAwDgYDVR0PAQH/BAQDAgGGMB0GA1UdDgQWBBTR4XENCy2BTm6KSo9MI7NM
|
||||||
|
XqtpCzAfBgNVHSMEGDAWgBTh0YHlzlpfBKrS6badZrHF+qwshzCBxwYIKwYBBQUH
|
||||||
|
AQEEgbowgbcwOwYIKwYBBQUHMAKGL2h0dHA6Ly9yb3N0ZWxlY29tLnJ1L2NkcC9y
|
||||||
|
b290Y2Ffc3NsX3JzYTIwMjIuY3J0MDsGCCsGAQUFBzAChi9odHRwOi8vY29tcGFu
|
||||||
|
eS5ydC5ydS9jZHAvcm9vdGNhX3NzbF9yc2EyMDIyLmNydDA7BggrBgEFBQcwAoYv
|
||||||
|
aHR0cDovL3JlZXN0ci1wa2kucnUvY2RwL3Jvb3RjYV9zc2xfcnNhMjAyMi5jcnQw
|
||||||
|
gbAGA1UdHwSBqDCBpTA1oDOgMYYvaHR0cDovL3Jvc3RlbGVjb20ucnUvY2RwL3Jv
|
||||||
|
b3RjYV9zc2xfcnNhMjAyMi5jcmwwNaAzoDGGL2h0dHA6Ly9jb21wYW55LnJ0LnJ1
|
||||||
|
L2NkcC9yb290Y2Ffc3NsX3JzYTIwMjIuY3JsMDWgM6Axhi9odHRwOi8vcmVlc3Ry
|
||||||
|
LXBraS5ydS9jZHAvcm9vdGNhX3NzbF9yc2EyMDIyLmNybDANBgkqhkiG9w0BAQsF
|
||||||
|
AAOCAgEARBVzZls79AdiSCpar15dA5Hr/rrT4WbrOfzlpI+xrLeRPrUG6eUWIW4v
|
||||||
|
Sui1yx3iqGLCjPcKb+HOTwoRMbI6ytP/ndp3TlYua2advYBEhSvjs+4vDZNwXr/D
|
||||||
|
anbwIWdurZmViQRBDFebpkvnIvru/RpWud/5r624Wp8voZMRtj/cm6aI9LtvBfT9
|
||||||
|
cfzhOaexI/99c14dyiuk1+6QhdwKaCRTc1mdfNQmnfWNRbfWhWBlK3h4GGE9JK33
|
||||||
|
Gk8ZS8DMrkdAh0xby4xAQ/mSWAfWrBmfzlOqGyoB1U47WTOeqNbWkkoAP2ys94+s
|
||||||
|
Jg4NTkiDVtXRF6nr6fYi0bSOvOFg0IQrMXO2Y8gyg9ARdPJwKtvWX8VPADCYMiWH
|
||||||
|
h4n8bZokIrImVKLDQKHY4jCsND2HHdJfnrdL2YJw1qFskNO4cSNmZydw0Wkgjv9k
|
||||||
|
F+KxqrDKlB8MZu2Hclph6v/CZ0fQ9YuE8/lsHZ0Qc2HyiSMnvjgK5fDc3TD4fa8F
|
||||||
|
E8gMNurM+kV8PT8LNIM+4Zs+LKEV8nqRWBaxkIVJGekkVKO8xDBOG/aN62AZKHOe
|
||||||
|
GcyIdu7yNMMRihGVZCYr8rYiJoKiOzDqOkPkLOPdhtVlgnhowzHDxMHND/E2WA5p
|
||||||
|
ZHuNM/m0TXt2wTTPL7JH2YC0gPz/BvvSzjksgzU5rLbRyUKQkgU=
|
||||||
|
-----END CERTIFICATE-----
|
||||||
@@ -433,8 +433,8 @@ public sealed class CertCcConnector : IFeedConnector
|
|||||||
}
|
}
|
||||||
|
|
||||||
var advisory = CertCcMapper.Map(dto, document, dtoRecord, SourceName);
|
var advisory = CertCcMapper.Map(dto, document, dtoRecord, SourceName);
|
||||||
var affectedCount = advisory.AffectedPackages.Count;
|
var affectedCount = advisory.AffectedPackages.Length;
|
||||||
var normalizedRuleCount = advisory.AffectedPackages.Sum(static package => package.NormalizedVersions.Count);
|
var normalizedRuleCount = advisory.AffectedPackages.Sum(static package => package.NormalizedVersions.Length);
|
||||||
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
||||||
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
||||||
_diagnostics.MapSuccess(affectedCount, normalizedRuleCount);
|
_diagnostics.MapSuccess(affectedCount, normalizedRuleCount);
|
||||||
|
|||||||
@@ -0,0 +1,65 @@
|
|||||||
|
using System.Collections.Immutable;
|
||||||
|
using MongoDB.Bson;
|
||||||
|
using StellaOps.Feedser.Source.Common;
|
||||||
|
using StellaOps.Feedser.Models;
|
||||||
|
using StellaOps.Feedser.Source.Ru.Bdu.Internal;
|
||||||
|
using StellaOps.Feedser.Storage.Mongo.Documents;
|
||||||
|
using Xunit;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Bdu.Tests;
|
||||||
|
|
||||||
|
public sealed class RuBduMapperTests
|
||||||
|
{
|
||||||
|
[Fact]
|
||||||
|
public void Map_ConstructsCanonicalAdvisory()
|
||||||
|
{
|
||||||
|
var dto = new RuBduVulnerabilityDto(
|
||||||
|
Identifier: "BDU:2025-12345",
|
||||||
|
Name: "Уязвимость тестового продукта",
|
||||||
|
Description: "Описание",
|
||||||
|
Solution: "Обновить",
|
||||||
|
IdentifyDate: new DateTimeOffset(2025, 10, 10, 0, 0, 0, TimeSpan.Zero),
|
||||||
|
SeverityText: "Высокий",
|
||||||
|
CvssVector: "AV:N/AC:L/Au:N/C:P/I:P/A:P",
|
||||||
|
CvssScore: 7.5,
|
||||||
|
Cvss3Vector: null,
|
||||||
|
Cvss3Score: null,
|
||||||
|
ExploitStatus: "Существует",
|
||||||
|
IncidentCount: 1,
|
||||||
|
FixStatus: "Устранена",
|
||||||
|
VulStatus: "Подтверждена",
|
||||||
|
VulClass: null,
|
||||||
|
VulState: null,
|
||||||
|
Other: null,
|
||||||
|
Software: new[]
|
||||||
|
{
|
||||||
|
new RuBduSoftwareDto("ООО Вендор", "Продукт", "1.2.3", "Windows", ImmutableArray<string>.Empty)
|
||||||
|
}.ToImmutableArray(),
|
||||||
|
Environment: ImmutableArray<RuBduEnvironmentDto>.Empty,
|
||||||
|
Cwes: new[] { new RuBduCweDto("CWE-79", "XSS") }.ToImmutableArray());
|
||||||
|
|
||||||
|
var document = new DocumentRecord(
|
||||||
|
Guid.NewGuid(),
|
||||||
|
RuBduConnectorPlugin.SourceName,
|
||||||
|
"https://bdu.fstec.ru/vul/2025-12345",
|
||||||
|
DateTimeOffset.UtcNow,
|
||||||
|
"abc",
|
||||||
|
DocumentStatuses.PendingMap,
|
||||||
|
"application/json",
|
||||||
|
null,
|
||||||
|
null,
|
||||||
|
null,
|
||||||
|
dto.IdentifyDate,
|
||||||
|
ObjectId.GenerateNewId());
|
||||||
|
|
||||||
|
var advisory = RuBduMapper.Map(dto, document, dto.IdentifyDate!.Value);
|
||||||
|
|
||||||
|
Assert.Equal("BDU:2025-12345", advisory.AdvisoryKey);
|
||||||
|
Assert.Contains("BDU:2025-12345", advisory.Aliases);
|
||||||
|
Assert.Equal("high", advisory.Severity);
|
||||||
|
Assert.True(advisory.ExploitKnown);
|
||||||
|
Assert.Single(advisory.AffectedPackages);
|
||||||
|
Assert.Single(advisory.CvssMetrics);
|
||||||
|
Assert.Contains(advisory.References, reference => reference.Url.Contains("bdu.fstec.ru", StringComparison.OrdinalIgnoreCase));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
using System.Xml.Linq;
|
||||||
|
using StellaOps.Feedser.Source.Ru.Bdu.Internal;
|
||||||
|
using Xunit;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Bdu.Tests;
|
||||||
|
|
||||||
|
public sealed class RuBduXmlParserTests
|
||||||
|
{
|
||||||
|
[Fact]
|
||||||
|
public void TryParse_ValidElement_ReturnsDto()
|
||||||
|
{
|
||||||
|
const string xml = """
|
||||||
|
<vul>
|
||||||
|
<identifier>BDU:2025-12345</identifier>
|
||||||
|
<name>Уязвимость тестового продукта</name>
|
||||||
|
<description>Описание уязвимости</description>
|
||||||
|
<solution>Обновить продукт</solution>
|
||||||
|
<identify_date>2025-10-10</identify_date>
|
||||||
|
<severity>Высокий уровень опасности</severity>
|
||||||
|
<exploit_status>Существует эксплойт</exploit_status>
|
||||||
|
<fix_status>Устранена</fix_status>
|
||||||
|
<vul_status>Подтверждена производителем</vul_status>
|
||||||
|
<vul_incident>1</vul_incident>
|
||||||
|
<cvss>
|
||||||
|
<vector score=\"7.5\">AV:N/AC:L/Au:N/C:P/I:P/A:P</vector>
|
||||||
|
</cvss>
|
||||||
|
<vulnerable_software>
|
||||||
|
<soft>
|
||||||
|
<vendor>ООО «Вендор»</vendor>
|
||||||
|
<name>Продукт</name>
|
||||||
|
<version>1.2.3</version>
|
||||||
|
<platform>Windows</platform>
|
||||||
|
<types>
|
||||||
|
<type>ics</type>
|
||||||
|
</types>
|
||||||
|
</soft>
|
||||||
|
</vulnerable_software>
|
||||||
|
<cwes>
|
||||||
|
<cwe>
|
||||||
|
<identifier>CWE-79</identifier>
|
||||||
|
<name>XSS</name>
|
||||||
|
</cwe>
|
||||||
|
</cwes>
|
||||||
|
</vul>
|
||||||
|
""";
|
||||||
|
|
||||||
|
var element = XElement.Parse(xml);
|
||||||
|
var dto = RuBduXmlParser.TryParse(element);
|
||||||
|
|
||||||
|
Assert.NotNull(dto);
|
||||||
|
Assert.Equal("BDU:2025-12345", dto!.Identifier);
|
||||||
|
Assert.Equal("Уязвимость тестового продукта", dto.Name);
|
||||||
|
Assert.Equal("AV:N/AC:L/Au:N/C:P/I:P/A:P", dto.CvssVector);
|
||||||
|
Assert.Equal(7.5, dto.CvssScore);
|
||||||
|
Assert.Single(dto.Software);
|
||||||
|
Assert.Single(dto.Cwes);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
<Project Sdk="Microsoft.NET.Sdk">
|
||||||
|
<PropertyGroup>
|
||||||
|
<TargetFramework>net10.0</TargetFramework>
|
||||||
|
<ImplicitUsings>enable</ImplicitUsings>
|
||||||
|
<Nullable>enable</Nullable>
|
||||||
|
</PropertyGroup>
|
||||||
|
|
||||||
|
<ItemGroup>
|
||||||
|
<ProjectReference Include="../StellaOps.Feedser.Models/StellaOps.Feedser.Models.csproj" />
|
||||||
|
<ProjectReference Include="../StellaOps.Feedser.Normalization/StellaOps.Feedser.Normalization.csproj" />
|
||||||
|
<ProjectReference Include="../StellaOps.Feedser.Source.Ru.Bdu/StellaOps.Feedser.Source.Ru.Bdu.csproj" />
|
||||||
|
</ItemGroup>
|
||||||
|
</Project>
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
using System;
|
|
||||||
using System.Threading;
|
|
||||||
using System.Threading.Tasks;
|
|
||||||
using StellaOps.Plugin;
|
|
||||||
|
|
||||||
namespace StellaOps.Feedser.Source.Ru.Bdu;
|
|
||||||
|
|
||||||
public sealed class RuBduConnectorPlugin : IConnectorPlugin
|
|
||||||
{
|
|
||||||
public string Name => "ru-bdu";
|
|
||||||
|
|
||||||
public bool IsAvailable(IServiceProvider services) => true;
|
|
||||||
|
|
||||||
public IFeedConnector Create(IServiceProvider services) => new StubConnector(Name);
|
|
||||||
|
|
||||||
private sealed class StubConnector : IFeedConnector
|
|
||||||
{
|
|
||||||
public StubConnector(string sourceName) => SourceName = sourceName;
|
|
||||||
|
|
||||||
public string SourceName { get; }
|
|
||||||
|
|
||||||
public Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken) => Task.CompletedTask;
|
|
||||||
|
|
||||||
public Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken) => Task.CompletedTask;
|
|
||||||
|
|
||||||
public Task MapAsync(IServiceProvider services, CancellationToken cancellationToken) => Task.CompletedTask;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
using System.Net;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Bdu.Configuration;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Connector options for the Russian BDU archive ingestion pipeline.
|
||||||
|
/// </summary>
|
||||||
|
public sealed class RuBduOptions
|
||||||
|
{
|
||||||
|
public const string HttpClientName = "ru-bdu";
|
||||||
|
|
||||||
|
private static readonly TimeSpan DefaultRequestTimeout = TimeSpan.FromMinutes(2);
|
||||||
|
private static readonly TimeSpan DefaultFailureBackoff = TimeSpan.FromMinutes(30);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Base endpoint used for resolving relative resource paths.
|
||||||
|
/// </summary>
|
||||||
|
public Uri BaseAddress { get; set; } = new("https://bdu.fstec.ru/", UriKind.Absolute);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Relative path to the zipped vulnerability dataset.
|
||||||
|
/// </summary>
|
||||||
|
public string DataArchivePath { get; set; } = "files/documents/vulxml.zip";
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// HTTP timeout applied when downloading the archive.
|
||||||
|
/// </summary>
|
||||||
|
public TimeSpan RequestTimeout { get; set; } = DefaultRequestTimeout;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Backoff applied when the remote endpoint fails to serve the archive.
|
||||||
|
/// </summary>
|
||||||
|
public TimeSpan FailureBackoff { get; set; } = DefaultFailureBackoff;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// User-Agent header used for outbound requests.
|
||||||
|
/// </summary>
|
||||||
|
public string UserAgent { get; set; } = "StellaOps/Feedser (+https://stella-ops.org)";
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Accept-Language preference sent with outbound requests.
|
||||||
|
/// </summary>
|
||||||
|
public string AcceptLanguage { get; set; } = "ru-RU,ru;q=0.9,en-US;q=0.6,en;q=0.4";
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Maximum number of vulnerabilities ingested per fetch cycle.
|
||||||
|
/// </summary>
|
||||||
|
public int MaxVulnerabilitiesPerFetch { get; set; } = 500;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Returns the absolute URI for the archive download.
|
||||||
|
/// </summary>
|
||||||
|
public Uri DataArchiveUri => new(BaseAddress, DataArchivePath);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Optional directory for caching the most recent archive (relative paths resolve under the content root).
|
||||||
|
/// </summary>
|
||||||
|
public string? CacheDirectory { get; set; } = null;
|
||||||
|
|
||||||
|
public void Validate()
|
||||||
|
{
|
||||||
|
if (BaseAddress is null || !BaseAddress.IsAbsoluteUri)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuBdu BaseAddress must be an absolute URI.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (string.IsNullOrWhiteSpace(DataArchivePath))
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuBdu DataArchivePath must be provided.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (RequestTimeout <= TimeSpan.Zero)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuBdu RequestTimeout must be positive.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (FailureBackoff < TimeSpan.Zero)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuBdu FailureBackoff cannot be negative.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (string.IsNullOrWhiteSpace(UserAgent))
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuBdu UserAgent cannot be empty.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (string.IsNullOrWhiteSpace(AcceptLanguage))
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuBdu AcceptLanguage cannot be empty.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (MaxVulnerabilitiesPerFetch <= 0)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuBdu MaxVulnerabilitiesPerFetch must be greater than zero.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (CacheDirectory is not null && CacheDirectory.Trim().Length == 0)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuBdu CacheDirectory cannot be whitespace.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
81
src/StellaOps.Feedser.Source.Ru.Bdu/Internal/RuBduCursor.cs
Normal file
81
src/StellaOps.Feedser.Source.Ru.Bdu/Internal/RuBduCursor.cs
Normal file
@@ -0,0 +1,81 @@
|
|||||||
|
using MongoDB.Bson;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Bdu.Internal;
|
||||||
|
|
||||||
|
internal sealed record RuBduCursor(
|
||||||
|
IReadOnlyCollection<Guid> PendingDocuments,
|
||||||
|
IReadOnlyCollection<Guid> PendingMappings,
|
||||||
|
DateTimeOffset? LastSuccessfulFetch)
|
||||||
|
{
|
||||||
|
private static readonly IReadOnlyCollection<Guid> EmptyGuids = Array.Empty<Guid>();
|
||||||
|
|
||||||
|
public static RuBduCursor Empty { get; } = new(EmptyGuids, EmptyGuids, null);
|
||||||
|
|
||||||
|
public RuBduCursor WithPendingDocuments(IEnumerable<Guid> documents)
|
||||||
|
=> this with { PendingDocuments = (documents ?? Enumerable.Empty<Guid>()).Distinct().ToArray() };
|
||||||
|
|
||||||
|
public RuBduCursor WithPendingMappings(IEnumerable<Guid> mappings)
|
||||||
|
=> this with { PendingMappings = (mappings ?? Enumerable.Empty<Guid>()).Distinct().ToArray() };
|
||||||
|
|
||||||
|
public RuBduCursor WithLastSuccessfulFetch(DateTimeOffset? timestamp)
|
||||||
|
=> this with { LastSuccessfulFetch = timestamp };
|
||||||
|
|
||||||
|
public BsonDocument ToBsonDocument()
|
||||||
|
{
|
||||||
|
var document = new BsonDocument
|
||||||
|
{
|
||||||
|
["pendingDocuments"] = new BsonArray(PendingDocuments.Select(id => id.ToString())),
|
||||||
|
["pendingMappings"] = new BsonArray(PendingMappings.Select(id => id.ToString())),
|
||||||
|
};
|
||||||
|
|
||||||
|
if (LastSuccessfulFetch.HasValue)
|
||||||
|
{
|
||||||
|
document["lastSuccessfulFetch"] = LastSuccessfulFetch.Value.UtcDateTime;
|
||||||
|
}
|
||||||
|
|
||||||
|
return document;
|
||||||
|
}
|
||||||
|
|
||||||
|
public static RuBduCursor FromBson(BsonDocument? document)
|
||||||
|
{
|
||||||
|
if (document is null || document.ElementCount == 0)
|
||||||
|
{
|
||||||
|
return Empty;
|
||||||
|
}
|
||||||
|
|
||||||
|
var pendingDocuments = ReadGuidArray(document, "pendingDocuments");
|
||||||
|
var pendingMappings = ReadGuidArray(document, "pendingMappings");
|
||||||
|
var lastFetch = document.TryGetValue("lastSuccessfulFetch", out var fetchValue)
|
||||||
|
? ParseDate(fetchValue)
|
||||||
|
: null;
|
||||||
|
|
||||||
|
return new RuBduCursor(pendingDocuments, pendingMappings, lastFetch);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IReadOnlyCollection<Guid> ReadGuidArray(BsonDocument document, string field)
|
||||||
|
{
|
||||||
|
if (!document.TryGetValue(field, out var value) || value is not BsonArray array)
|
||||||
|
{
|
||||||
|
return EmptyGuids;
|
||||||
|
}
|
||||||
|
|
||||||
|
var result = new List<Guid>(array.Count);
|
||||||
|
foreach (var element in array)
|
||||||
|
{
|
||||||
|
if (Guid.TryParse(element?.ToString(), out var guid))
|
||||||
|
{
|
||||||
|
result.Add(guid);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static DateTimeOffset? ParseDate(BsonValue value)
|
||||||
|
=> value.BsonType switch
|
||||||
|
{
|
||||||
|
BsonType.DateTime => DateTime.SpecifyKind(value.ToUniversalTime(), DateTimeKind.Utc),
|
||||||
|
BsonType.String when DateTimeOffset.TryParse(value.AsString, out var parsed) => parsed.ToUniversalTime(),
|
||||||
|
_ => null,
|
||||||
|
};
|
||||||
|
}
|
||||||
249
src/StellaOps.Feedser.Source.Ru.Bdu/Internal/RuBduMapper.cs
Normal file
249
src/StellaOps.Feedser.Source.Ru.Bdu/Internal/RuBduMapper.cs
Normal file
@@ -0,0 +1,249 @@
|
|||||||
|
using System.Collections.Generic;
|
||||||
|
using System.Collections.Immutable;
|
||||||
|
using System.Globalization;
|
||||||
|
using System.Linq;
|
||||||
|
using StellaOps.Feedser.Models;
|
||||||
|
using StellaOps.Feedser.Normalization.Cvss;
|
||||||
|
using StellaOps.Feedser.Storage.Mongo.Documents;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Bdu.Internal;
|
||||||
|
|
||||||
|
internal static class RuBduMapper
|
||||||
|
{
|
||||||
|
public static Advisory Map(RuBduVulnerabilityDto dto, DocumentRecord document, DateTimeOffset recordedAt)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(dto);
|
||||||
|
ArgumentNullException.ThrowIfNull(document);
|
||||||
|
|
||||||
|
var advisoryProvenance = new AdvisoryProvenance(
|
||||||
|
RuBduConnectorPlugin.SourceName,
|
||||||
|
"advisory",
|
||||||
|
dto.Identifier,
|
||||||
|
recordedAt,
|
||||||
|
new[] { ProvenanceFieldMasks.Advisory });
|
||||||
|
|
||||||
|
var aliases = BuildAliases(dto);
|
||||||
|
var packages = BuildPackages(dto, recordedAt);
|
||||||
|
var references = BuildReferences(dto, document, recordedAt);
|
||||||
|
var cvssMetrics = BuildCvssMetrics(dto, recordedAt, out var severityFromCvss);
|
||||||
|
var severity = severityFromCvss;
|
||||||
|
var exploitKnown = DetermineExploitKnown(dto);
|
||||||
|
|
||||||
|
return new Advisory(
|
||||||
|
advisoryKey: dto.Identifier,
|
||||||
|
title: dto.Name ?? dto.Identifier,
|
||||||
|
summary: dto.Description,
|
||||||
|
language: "ru",
|
||||||
|
published: dto.IdentifyDate,
|
||||||
|
modified: dto.IdentifyDate,
|
||||||
|
severity: severity,
|
||||||
|
exploitKnown: exploitKnown,
|
||||||
|
aliases: aliases,
|
||||||
|
references: references,
|
||||||
|
affectedPackages: packages,
|
||||||
|
cvssMetrics: cvssMetrics,
|
||||||
|
provenance: new[] { advisoryProvenance });
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IReadOnlyList<string> BuildAliases(RuBduVulnerabilityDto dto)
|
||||||
|
{
|
||||||
|
var aliases = new List<string>(capacity: 2) { dto.Identifier };
|
||||||
|
return aliases;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IReadOnlyList<AffectedPackage> BuildPackages(RuBduVulnerabilityDto dto, DateTimeOffset recordedAt)
|
||||||
|
{
|
||||||
|
if (dto.Software.IsDefaultOrEmpty)
|
||||||
|
{
|
||||||
|
return Array.Empty<AffectedPackage>();
|
||||||
|
}
|
||||||
|
|
||||||
|
var packages = new List<AffectedPackage>(dto.Software.Length);
|
||||||
|
foreach (var software in dto.Software)
|
||||||
|
{
|
||||||
|
if (string.IsNullOrWhiteSpace(software.Name) && string.IsNullOrWhiteSpace(software.Vendor))
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
var identifier = string.Join(
|
||||||
|
" ",
|
||||||
|
new[] { software.Vendor, software.Name }
|
||||||
|
.Where(static part => !string.IsNullOrWhiteSpace(part))
|
||||||
|
.Select(static part => part!.Trim()));
|
||||||
|
|
||||||
|
if (string.IsNullOrWhiteSpace(identifier))
|
||||||
|
{
|
||||||
|
identifier = software.Name ?? software.Vendor ?? dto.Identifier;
|
||||||
|
}
|
||||||
|
|
||||||
|
var isIcs = !software.Types.IsDefaultOrEmpty && software.Types.Any(static type => string.Equals(type, "ics", StringComparison.OrdinalIgnoreCase));
|
||||||
|
|
||||||
|
var packageProvenance = new AdvisoryProvenance(
|
||||||
|
RuBduConnectorPlugin.SourceName,
|
||||||
|
"package",
|
||||||
|
identifier,
|
||||||
|
recordedAt,
|
||||||
|
new[] { ProvenanceFieldMasks.AffectedPackages });
|
||||||
|
|
||||||
|
var normalizedStatus = NormalizeStatus(dto.VulStatus);
|
||||||
|
var statuses = normalizedStatus is null
|
||||||
|
? Array.Empty<AffectedPackageStatus>()
|
||||||
|
: new[]
|
||||||
|
{
|
||||||
|
new AffectedPackageStatus(normalizedStatus, new AdvisoryProvenance(
|
||||||
|
RuBduConnectorPlugin.SourceName,
|
||||||
|
"package-status",
|
||||||
|
dto.VulStatus ?? normalizedStatus,
|
||||||
|
recordedAt,
|
||||||
|
new[] { ProvenanceFieldMasks.PackageStatuses }))
|
||||||
|
};
|
||||||
|
|
||||||
|
var ranges = Array.Empty<AffectedVersionRange>();
|
||||||
|
if (!string.IsNullOrWhiteSpace(software.Version))
|
||||||
|
{
|
||||||
|
ranges = new[]
|
||||||
|
{
|
||||||
|
new AffectedVersionRange(
|
||||||
|
rangeKind: "string",
|
||||||
|
introducedVersion: null,
|
||||||
|
fixedVersion: null,
|
||||||
|
lastAffectedVersion: null,
|
||||||
|
rangeExpression: software.Version,
|
||||||
|
provenance: new AdvisoryProvenance(
|
||||||
|
RuBduConnectorPlugin.SourceName,
|
||||||
|
"package-range",
|
||||||
|
software.Version,
|
||||||
|
recordedAt,
|
||||||
|
new[] { ProvenanceFieldMasks.VersionRanges }))
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
packages.Add(new AffectedPackage(
|
||||||
|
isIcs ? AffectedPackageTypes.IcsVendor : AffectedPackageTypes.Vendor,
|
||||||
|
identifier,
|
||||||
|
platform: software.Platform,
|
||||||
|
versionRanges: ranges,
|
||||||
|
statuses: statuses,
|
||||||
|
provenance: new[] { packageProvenance }));
|
||||||
|
}
|
||||||
|
|
||||||
|
return packages;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IReadOnlyList<AdvisoryReference> BuildReferences(RuBduVulnerabilityDto dto, DocumentRecord document, DateTimeOffset recordedAt)
|
||||||
|
{
|
||||||
|
var references = new List<AdvisoryReference>
|
||||||
|
{
|
||||||
|
new(document.Uri, "details", "ru-bdu", summary: null, new AdvisoryProvenance(
|
||||||
|
RuBduConnectorPlugin.SourceName,
|
||||||
|
"reference",
|
||||||
|
document.Uri,
|
||||||
|
recordedAt,
|
||||||
|
new[] { ProvenanceFieldMasks.References }))
|
||||||
|
};
|
||||||
|
|
||||||
|
foreach (var cwe in dto.Cwes)
|
||||||
|
{
|
||||||
|
if (string.IsNullOrWhiteSpace(cwe.Identifier))
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
var slug = cwe.Identifier.ToUpperInvariant().Replace("CWE-", string.Empty, StringComparison.OrdinalIgnoreCase);
|
||||||
|
if (!slug.All(char.IsDigit))
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
var url = $"https://cwe.mitre.org/data/definitions/{slug}.html";
|
||||||
|
references.Add(new AdvisoryReference(url, "cwe", "cwe", cwe.Name, new AdvisoryProvenance(
|
||||||
|
RuBduConnectorPlugin.SourceName,
|
||||||
|
"reference",
|
||||||
|
url,
|
||||||
|
recordedAt,
|
||||||
|
new[] { ProvenanceFieldMasks.References })));
|
||||||
|
}
|
||||||
|
|
||||||
|
return references;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IReadOnlyList<CvssMetric> BuildCvssMetrics(RuBduVulnerabilityDto dto, DateTimeOffset recordedAt, out string? severity)
|
||||||
|
{
|
||||||
|
severity = null;
|
||||||
|
var metrics = new List<CvssMetric>();
|
||||||
|
|
||||||
|
if (!string.IsNullOrWhiteSpace(dto.CvssVector) && CvssMetricNormalizer.TryNormalize("2.0", dto.CvssVector, dto.CvssScore, null, out var normalized))
|
||||||
|
{
|
||||||
|
var provenance = new AdvisoryProvenance(
|
||||||
|
RuBduConnectorPlugin.SourceName,
|
||||||
|
"cvss",
|
||||||
|
normalized.Vector,
|
||||||
|
recordedAt,
|
||||||
|
new[] { ProvenanceFieldMasks.CvssMetrics });
|
||||||
|
var metric = normalized.ToModel(provenance);
|
||||||
|
metrics.Add(metric);
|
||||||
|
severity ??= metric.BaseSeverity;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!string.IsNullOrWhiteSpace(dto.Cvss3Vector) && CvssMetricNormalizer.TryNormalize("3.1", dto.Cvss3Vector, dto.Cvss3Score, null, out var normalized3))
|
||||||
|
{
|
||||||
|
var provenance = new AdvisoryProvenance(
|
||||||
|
RuBduConnectorPlugin.SourceName,
|
||||||
|
"cvss",
|
||||||
|
normalized3.Vector,
|
||||||
|
recordedAt,
|
||||||
|
new[] { ProvenanceFieldMasks.CvssMetrics });
|
||||||
|
var metric = normalized3.ToModel(provenance);
|
||||||
|
metrics.Add(metric);
|
||||||
|
severity ??= metric.BaseSeverity;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (metrics.Count > 1)
|
||||||
|
{
|
||||||
|
metrics = metrics
|
||||||
|
.OrderByDescending(static metric => metric.BaseScore)
|
||||||
|
.ThenBy(static metric => metric.Version, StringComparer.Ordinal)
|
||||||
|
.ToList();
|
||||||
|
}
|
||||||
|
|
||||||
|
return metrics;
|
||||||
|
}
|
||||||
|
private static string NormalizeStatus(string? status)
|
||||||
|
{
|
||||||
|
if (string.IsNullOrWhiteSpace(status))
|
||||||
|
{
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
var normalized = status.Trim().ToLowerInvariant();
|
||||||
|
return normalized switch
|
||||||
|
{
|
||||||
|
"устранена" or "устранена производителем" or "устранена разработчиком" => AffectedPackageStatusCatalog.Fixed,
|
||||||
|
"устраняется" or "устранение планируется" or "разрабатывается" => AffectedPackageStatusCatalog.Pending,
|
||||||
|
"не устранена" => AffectedPackageStatusCatalog.Pending,
|
||||||
|
"актуальна" or "подтверждена" or "подтверждена производителем" or "подтверждена исследователями" => AffectedPackageStatusCatalog.Affected,
|
||||||
|
_ => null,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
private static bool DetermineExploitKnown(RuBduVulnerabilityDto dto)
|
||||||
|
{
|
||||||
|
if (dto.IncidentCount.HasValue && dto.IncidentCount.Value > 0)
|
||||||
|
{
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!string.IsNullOrWhiteSpace(dto.ExploitStatus))
|
||||||
|
{
|
||||||
|
var status = dto.ExploitStatus.Trim().ToLowerInvariant();
|
||||||
|
if (status.Contains("существ", StringComparison.Ordinal) || status.Contains("использ", StringComparison.Ordinal))
|
||||||
|
{
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
using System.Collections.Immutable;
|
||||||
|
using System.Text.Json.Serialization;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Bdu.Internal;
|
||||||
|
|
||||||
|
internal sealed record RuBduVulnerabilityDto(
|
||||||
|
string Identifier,
|
||||||
|
string? Name,
|
||||||
|
string? Description,
|
||||||
|
string? Solution,
|
||||||
|
DateTimeOffset? IdentifyDate,
|
||||||
|
string? SeverityText,
|
||||||
|
string? CvssVector,
|
||||||
|
double? CvssScore,
|
||||||
|
string? Cvss3Vector,
|
||||||
|
double? Cvss3Score,
|
||||||
|
string? ExploitStatus,
|
||||||
|
int? IncidentCount,
|
||||||
|
string? FixStatus,
|
||||||
|
string? VulStatus,
|
||||||
|
string? VulClass,
|
||||||
|
string? VulState,
|
||||||
|
string? Other,
|
||||||
|
ImmutableArray<RuBduSoftwareDto> Software,
|
||||||
|
ImmutableArray<RuBduEnvironmentDto> Environment,
|
||||||
|
ImmutableArray<RuBduCweDto> Cwes)
|
||||||
|
{
|
||||||
|
[JsonIgnore]
|
||||||
|
public bool HasCvss => !string.IsNullOrWhiteSpace(CvssVector) || !string.IsNullOrWhiteSpace(Cvss3Vector);
|
||||||
|
}
|
||||||
|
|
||||||
|
internal sealed record RuBduSoftwareDto(
|
||||||
|
string? Vendor,
|
||||||
|
string? Name,
|
||||||
|
string? Version,
|
||||||
|
string? Platform,
|
||||||
|
ImmutableArray<string> Types);
|
||||||
|
|
||||||
|
internal sealed record RuBduEnvironmentDto(
|
||||||
|
string? Vendor,
|
||||||
|
string? Name,
|
||||||
|
string? Version,
|
||||||
|
string? Platform);
|
||||||
|
|
||||||
|
internal sealed record RuBduCweDto(string Identifier, string? Name);
|
||||||
196
src/StellaOps.Feedser.Source.Ru.Bdu/Internal/RuBduXmlParser.cs
Normal file
196
src/StellaOps.Feedser.Source.Ru.Bdu/Internal/RuBduXmlParser.cs
Normal file
@@ -0,0 +1,196 @@
|
|||||||
|
using System.Collections.Immutable;
|
||||||
|
using System.Linq;
|
||||||
|
using System.Globalization;
|
||||||
|
using System.Xml.Linq;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Bdu.Internal;
|
||||||
|
|
||||||
|
internal static class RuBduXmlParser
|
||||||
|
{
|
||||||
|
public static RuBduVulnerabilityDto? TryParse(XElement element)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(element);
|
||||||
|
|
||||||
|
var identifier = element.Element("identifier")?.Value?.Trim();
|
||||||
|
if (string.IsNullOrWhiteSpace(identifier))
|
||||||
|
{
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
var name = Normalize(element.Element("name")?.Value);
|
||||||
|
var description = Normalize(element.Element("description")?.Value);
|
||||||
|
var solution = Normalize(element.Element("solution")?.Value);
|
||||||
|
var severity = Normalize(element.Element("severity")?.Value);
|
||||||
|
var exploitStatus = Normalize(element.Element("exploit_status")?.Value);
|
||||||
|
var fixStatus = Normalize(element.Element("fix_status")?.Value);
|
||||||
|
var vulStatus = Normalize(element.Element("vul_status")?.Value);
|
||||||
|
var vulClass = Normalize(element.Element("vul_class")?.Value);
|
||||||
|
var vulState = Normalize(element.Element("vul_state")?.Value);
|
||||||
|
var other = Normalize(element.Element("other")?.Value);
|
||||||
|
var incidentCount = ParseInt(element.Element("vul_incident")?.Value);
|
||||||
|
|
||||||
|
var identifyDate = ParseDate(element.Element("identify_date")?.Value);
|
||||||
|
|
||||||
|
var cvssVectorElement = element.Element("cvss")?.Element("vector");
|
||||||
|
var cvssVector = Normalize(cvssVectorElement?.Value);
|
||||||
|
var cvssScore = ParseDouble(cvssVectorElement?.Attribute("score")?.Value);
|
||||||
|
|
||||||
|
var cvss3VectorElement = element.Element("cvss3")?.Element("vector");
|
||||||
|
var cvss3Vector = Normalize(cvss3VectorElement?.Value);
|
||||||
|
var cvss3Score = ParseDouble(cvss3VectorElement?.Attribute("score")?.Value);
|
||||||
|
|
||||||
|
var software = ParseSoftware(element.Element("vulnerable_software"));
|
||||||
|
var environment = ParseEnvironment(element.Element("environment"));
|
||||||
|
var cwes = ParseCwes(element.Element("cwes"));
|
||||||
|
|
||||||
|
return new RuBduVulnerabilityDto(
|
||||||
|
identifier.Trim(),
|
||||||
|
name,
|
||||||
|
description,
|
||||||
|
solution,
|
||||||
|
identifyDate,
|
||||||
|
severity,
|
||||||
|
cvssVector,
|
||||||
|
cvssScore,
|
||||||
|
cvss3Vector,
|
||||||
|
cvss3Score,
|
||||||
|
exploitStatus,
|
||||||
|
incidentCount,
|
||||||
|
fixStatus,
|
||||||
|
vulStatus,
|
||||||
|
vulClass,
|
||||||
|
vulState,
|
||||||
|
other,
|
||||||
|
software,
|
||||||
|
environment,
|
||||||
|
cwes);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static ImmutableArray<RuBduSoftwareDto> ParseSoftware(XElement? root)
|
||||||
|
{
|
||||||
|
if (root is null)
|
||||||
|
{
|
||||||
|
return ImmutableArray<RuBduSoftwareDto>.Empty;
|
||||||
|
}
|
||||||
|
|
||||||
|
var builder = ImmutableArray.CreateBuilder<RuBduSoftwareDto>();
|
||||||
|
foreach (var soft in root.Elements("soft"))
|
||||||
|
{
|
||||||
|
var vendor = Normalize(soft.Element("vendor")?.Value);
|
||||||
|
var name = Normalize(soft.Element("name")?.Value);
|
||||||
|
var version = Normalize(soft.Element("version")?.Value);
|
||||||
|
var platform = Normalize(soft.Element("platform")?.Value);
|
||||||
|
var types = soft.Element("types") is { } typesElement
|
||||||
|
? typesElement.Elements("type").Select(static x => Normalize(x.Value)).Where(static value => !string.IsNullOrWhiteSpace(value)).Cast<string>().ToImmutableArray()
|
||||||
|
: ImmutableArray<string>.Empty;
|
||||||
|
|
||||||
|
builder.Add(new RuBduSoftwareDto(vendor, name, version, platform, types));
|
||||||
|
}
|
||||||
|
|
||||||
|
return builder.ToImmutable();
|
||||||
|
}
|
||||||
|
|
||||||
|
private static ImmutableArray<RuBduEnvironmentDto> ParseEnvironment(XElement? root)
|
||||||
|
{
|
||||||
|
if (root is null)
|
||||||
|
{
|
||||||
|
return ImmutableArray<RuBduEnvironmentDto>.Empty;
|
||||||
|
}
|
||||||
|
|
||||||
|
var builder = ImmutableArray.CreateBuilder<RuBduEnvironmentDto>();
|
||||||
|
foreach (var os in root.Elements())
|
||||||
|
{
|
||||||
|
var vendor = Normalize(os.Element("vendor")?.Value);
|
||||||
|
var name = Normalize(os.Element("name")?.Value);
|
||||||
|
var version = Normalize(os.Element("version")?.Value);
|
||||||
|
var platform = Normalize(os.Element("platform")?.Value);
|
||||||
|
builder.Add(new RuBduEnvironmentDto(vendor, name, version, platform));
|
||||||
|
}
|
||||||
|
|
||||||
|
return builder.ToImmutable();
|
||||||
|
}
|
||||||
|
|
||||||
|
private static ImmutableArray<RuBduCweDto> ParseCwes(XElement? root)
|
||||||
|
{
|
||||||
|
if (root is null)
|
||||||
|
{
|
||||||
|
return ImmutableArray<RuBduCweDto>.Empty;
|
||||||
|
}
|
||||||
|
|
||||||
|
var builder = ImmutableArray.CreateBuilder<RuBduCweDto>();
|
||||||
|
foreach (var cwe in root.Elements("cwe"))
|
||||||
|
{
|
||||||
|
var identifier = Normalize(cwe.Element("identifier")?.Value);
|
||||||
|
if (string.IsNullOrWhiteSpace(identifier))
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
var name = Normalize(cwe.Element("name")?.Value);
|
||||||
|
builder.Add(new RuBduCweDto(identifier, name));
|
||||||
|
}
|
||||||
|
|
||||||
|
return builder.ToImmutable();
|
||||||
|
}
|
||||||
|
|
||||||
|
private static DateTimeOffset? ParseDate(string? value)
|
||||||
|
{
|
||||||
|
if (string.IsNullOrWhiteSpace(value))
|
||||||
|
{
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
var trimmed = value.Trim();
|
||||||
|
if (DateTimeOffset.TryParse(trimmed, CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, out var isoDate))
|
||||||
|
{
|
||||||
|
return isoDate;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (DateTimeOffset.TryParseExact(trimmed, new[] { "dd.MM.yyyy", "dd.MM.yyyy HH:mm:ss" }, CultureInfo.GetCultureInfo("ru-RU"), DateTimeStyles.AssumeUniversal, out var ruDate))
|
||||||
|
{
|
||||||
|
return ruDate;
|
||||||
|
}
|
||||||
|
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static double? ParseDouble(string? value)
|
||||||
|
{
|
||||||
|
if (string.IsNullOrWhiteSpace(value))
|
||||||
|
{
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (double.TryParse(value.Trim(), NumberStyles.Any, CultureInfo.InvariantCulture, out var parsed))
|
||||||
|
{
|
||||||
|
return parsed;
|
||||||
|
}
|
||||||
|
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static int? ParseInt(string? value)
|
||||||
|
{
|
||||||
|
if (string.IsNullOrWhiteSpace(value))
|
||||||
|
{
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (int.TryParse(value.Trim(), NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsed))
|
||||||
|
{
|
||||||
|
return parsed;
|
||||||
|
}
|
||||||
|
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string? Normalize(string? value)
|
||||||
|
{
|
||||||
|
if (string.IsNullOrWhiteSpace(value))
|
||||||
|
{
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
return value.Replace('\r', ' ').Replace('\n', ' ').Trim();
|
||||||
|
}
|
||||||
|
}
|
||||||
43
src/StellaOps.Feedser.Source.Ru.Bdu/Jobs.cs
Normal file
43
src/StellaOps.Feedser.Source.Ru.Bdu/Jobs.cs
Normal file
@@ -0,0 +1,43 @@
|
|||||||
|
using StellaOps.Feedser.Core.Jobs;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Bdu;
|
||||||
|
|
||||||
|
internal static class RuBduJobKinds
|
||||||
|
{
|
||||||
|
public const string Fetch = "source:ru-bdu:fetch";
|
||||||
|
public const string Parse = "source:ru-bdu:parse";
|
||||||
|
public const string Map = "source:ru-bdu:map";
|
||||||
|
}
|
||||||
|
|
||||||
|
internal sealed class RuBduFetchJob : IJob
|
||||||
|
{
|
||||||
|
private readonly RuBduConnector _connector;
|
||||||
|
|
||||||
|
public RuBduFetchJob(RuBduConnector connector)
|
||||||
|
=> _connector = connector ?? throw new ArgumentNullException(nameof(connector));
|
||||||
|
|
||||||
|
public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken)
|
||||||
|
=> _connector.FetchAsync(context.Services, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
internal sealed class RuBduParseJob : IJob
|
||||||
|
{
|
||||||
|
private readonly RuBduConnector _connector;
|
||||||
|
|
||||||
|
public RuBduParseJob(RuBduConnector connector)
|
||||||
|
=> _connector = connector ?? throw new ArgumentNullException(nameof(connector));
|
||||||
|
|
||||||
|
public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken)
|
||||||
|
=> _connector.ParseAsync(context.Services, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
internal sealed class RuBduMapJob : IJob
|
||||||
|
{
|
||||||
|
private readonly RuBduConnector _connector;
|
||||||
|
|
||||||
|
public RuBduMapJob(RuBduConnector connector)
|
||||||
|
=> _connector = connector ?? throw new ArgumentNullException(nameof(connector));
|
||||||
|
|
||||||
|
public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken)
|
||||||
|
=> _connector.MapAsync(context.Services, cancellationToken);
|
||||||
|
}
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
using System.Runtime.CompilerServices;
|
||||||
|
|
||||||
|
[assembly: InternalsVisibleTo("StellaOps.Feedser.Source.Ru.Bdu.Tests")]
|
||||||
493
src/StellaOps.Feedser.Source.Ru.Bdu/RuBduConnector.cs
Normal file
493
src/StellaOps.Feedser.Source.Ru.Bdu/RuBduConnector.cs
Normal file
@@ -0,0 +1,493 @@
|
|||||||
|
using System.Collections.Immutable;
|
||||||
|
using System.Globalization;
|
||||||
|
using System.IO;
|
||||||
|
using System.IO.Compression;
|
||||||
|
using System.Security.Cryptography;
|
||||||
|
using System.Linq;
|
||||||
|
using System.Text.Json;
|
||||||
|
using System.Text.Json.Serialization;
|
||||||
|
using System.Xml;
|
||||||
|
using System.Xml.Linq;
|
||||||
|
using Microsoft.Extensions.Logging;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using MongoDB.Bson;
|
||||||
|
using StellaOps.Feedser.Normalization.Cvss;
|
||||||
|
using StellaOps.Feedser.Source.Common;
|
||||||
|
using StellaOps.Feedser.Source.Common.Fetch;
|
||||||
|
using StellaOps.Feedser.Source.Ru.Bdu.Configuration;
|
||||||
|
using StellaOps.Feedser.Source.Ru.Bdu.Internal;
|
||||||
|
using StellaOps.Feedser.Storage.Mongo;
|
||||||
|
using StellaOps.Feedser.Storage.Mongo.Advisories;
|
||||||
|
using StellaOps.Feedser.Storage.Mongo.Documents;
|
||||||
|
using StellaOps.Feedser.Storage.Mongo.Dtos;
|
||||||
|
using StellaOps.Plugin;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Bdu;
|
||||||
|
|
||||||
|
public sealed class RuBduConnector : IFeedConnector
|
||||||
|
{
|
||||||
|
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web)
|
||||||
|
{
|
||||||
|
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
|
||||||
|
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
||||||
|
WriteIndented = false,
|
||||||
|
};
|
||||||
|
|
||||||
|
private readonly SourceFetchService _fetchService;
|
||||||
|
private readonly RawDocumentStorage _rawDocumentStorage;
|
||||||
|
private readonly IDocumentStore _documentStore;
|
||||||
|
private readonly IDtoStore _dtoStore;
|
||||||
|
private readonly IAdvisoryStore _advisoryStore;
|
||||||
|
private readonly ISourceStateRepository _stateRepository;
|
||||||
|
private readonly RuBduOptions _options;
|
||||||
|
private readonly TimeProvider _timeProvider;
|
||||||
|
private readonly ILogger<RuBduConnector> _logger;
|
||||||
|
|
||||||
|
private readonly string _cacheDirectory;
|
||||||
|
private readonly string _archiveCachePath;
|
||||||
|
|
||||||
|
public RuBduConnector(
|
||||||
|
SourceFetchService fetchService,
|
||||||
|
RawDocumentStorage rawDocumentStorage,
|
||||||
|
IDocumentStore documentStore,
|
||||||
|
IDtoStore dtoStore,
|
||||||
|
IAdvisoryStore advisoryStore,
|
||||||
|
ISourceStateRepository stateRepository,
|
||||||
|
IOptions<RuBduOptions> options,
|
||||||
|
TimeProvider? timeProvider,
|
||||||
|
ILogger<RuBduConnector> logger)
|
||||||
|
{
|
||||||
|
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
||||||
|
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
||||||
|
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
||||||
|
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
||||||
|
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
||||||
|
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||||
|
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
||||||
|
_options.Validate();
|
||||||
|
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||||
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
|
_cacheDirectory = ResolveCacheDirectory(_options.CacheDirectory);
|
||||||
|
_archiveCachePath = Path.Combine(_cacheDirectory, "vulxml.zip");
|
||||||
|
EnsureCacheDirectory();
|
||||||
|
}
|
||||||
|
|
||||||
|
public string SourceName => RuBduConnectorPlugin.SourceName;
|
||||||
|
|
||||||
|
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
||||||
|
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
||||||
|
var now = _timeProvider.GetUtcNow();
|
||||||
|
|
||||||
|
SourceFetchContentResult archiveResult = default;
|
||||||
|
byte[]? archiveContent = null;
|
||||||
|
var usedCache = false;
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var request = new SourceFetchRequest(RuBduOptions.HttpClientName, SourceName, _options.DataArchiveUri)
|
||||||
|
{
|
||||||
|
AcceptHeaders = new[]
|
||||||
|
{
|
||||||
|
"application/zip",
|
||||||
|
"application/octet-stream",
|
||||||
|
"application/x-zip-compressed",
|
||||||
|
},
|
||||||
|
TimeoutOverride = _options.RequestTimeout,
|
||||||
|
};
|
||||||
|
|
||||||
|
archiveResult = await _fetchService.FetchContentAsync(request, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
|
if (archiveResult.IsNotModified)
|
||||||
|
{
|
||||||
|
_logger.LogDebug("RU-BDU archive not modified.");
|
||||||
|
await UpdateCursorAsync(cursor.WithLastSuccessfulFetch(now), cancellationToken).ConfigureAwait(false);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (archiveResult.IsSuccess && archiveResult.Content is not null)
|
||||||
|
{
|
||||||
|
archiveContent = archiveResult.Content;
|
||||||
|
TryWriteCachedArchive(archiveContent);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException)
|
||||||
|
{
|
||||||
|
if (TryReadCachedArchive(out var cachedFallback))
|
||||||
|
{
|
||||||
|
_logger.LogWarning(ex, "RU-BDU archive fetch failed; using cached artefact {CachePath}", _archiveCachePath);
|
||||||
|
archiveContent = cachedFallback;
|
||||||
|
usedCache = true;
|
||||||
|
}
|
||||||
|
else
|
||||||
|
{
|
||||||
|
_logger.LogError(ex, "RU-BDU archive fetch failed for {ArchiveUri}", _options.DataArchiveUri);
|
||||||
|
await _stateRepository.MarkFailureAsync(SourceName, now, _options.FailureBackoff, ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
|
throw;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (archiveContent is null)
|
||||||
|
{
|
||||||
|
if (TryReadCachedArchive(out var cachedFallback))
|
||||||
|
{
|
||||||
|
_logger.LogWarning("RU-BDU archive unavailable (status={Status}); using cached artefact {CachePath}", archiveResult.StatusCode, _archiveCachePath);
|
||||||
|
archiveContent = cachedFallback;
|
||||||
|
usedCache = true;
|
||||||
|
}
|
||||||
|
else
|
||||||
|
{
|
||||||
|
_logger.LogWarning("RU-BDU archive fetch returned no content (status={Status})", archiveResult.StatusCode);
|
||||||
|
await UpdateCursorAsync(cursor.WithLastSuccessfulFetch(now), cancellationToken).ConfigureAwait(false);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var archiveLastModified = archiveResult.LastModified;
|
||||||
|
int added;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
added = await ProcessArchiveAsync(archiveContent, now, pendingDocuments, pendingMappings, archiveLastModified, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
if (!usedCache)
|
||||||
|
{
|
||||||
|
_logger.LogError(ex, "RU-BDU archive processing failed");
|
||||||
|
await _stateRepository.MarkFailureAsync(SourceName, now, _options.FailureBackoff, ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
throw;
|
||||||
|
}
|
||||||
|
|
||||||
|
var updatedCursor = cursor
|
||||||
|
.WithPendingDocuments(pendingDocuments)
|
||||||
|
.WithPendingMappings(pendingMappings)
|
||||||
|
.WithLastSuccessfulFetch(now);
|
||||||
|
|
||||||
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
if (cursor.PendingDocuments.Count == 0)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
var pendingDocuments = cursor.PendingDocuments.ToList();
|
||||||
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
|
foreach (var documentId in cursor.PendingDocuments)
|
||||||
|
{
|
||||||
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
|
if (document is null)
|
||||||
|
{
|
||||||
|
pendingDocuments.Remove(documentId);
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!document.GridFsId.HasValue)
|
||||||
|
{
|
||||||
|
_logger.LogWarning("RU-BDU document {DocumentId} missing GridFS payload", documentId);
|
||||||
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
|
pendingDocuments.Remove(documentId);
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
byte[] payload;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
payload = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogError(ex, "RU-BDU unable to download raw document {DocumentId}", documentId);
|
||||||
|
throw;
|
||||||
|
}
|
||||||
|
|
||||||
|
RuBduVulnerabilityDto? dto;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
dto = JsonSerializer.Deserialize<RuBduVulnerabilityDto>(payload, SerializerOptions);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogWarning(ex, "RU-BDU failed to deserialize document {DocumentId}", documentId);
|
||||||
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
|
pendingDocuments.Remove(documentId);
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (dto is null)
|
||||||
|
{
|
||||||
|
_logger.LogWarning("RU-BDU document {DocumentId} produced null DTO", documentId);
|
||||||
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
|
pendingDocuments.Remove(documentId);
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
var bson = MongoDB.Bson.BsonDocument.Parse(JsonSerializer.Serialize(dto, SerializerOptions));
|
||||||
|
var dtoRecord = new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "ru-bdu.v1", bson, _timeProvider.GetUtcNow());
|
||||||
|
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
||||||
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
|
pendingDocuments.Remove(documentId);
|
||||||
|
if (!pendingMappings.Contains(documentId))
|
||||||
|
{
|
||||||
|
pendingMappings.Add(documentId);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var updatedCursor = cursor
|
||||||
|
.WithPendingDocuments(pendingDocuments)
|
||||||
|
.WithPendingMappings(pendingMappings);
|
||||||
|
|
||||||
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
if (cursor.PendingMappings.Count == 0)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
|
foreach (var documentId in cursor.PendingMappings)
|
||||||
|
{
|
||||||
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
|
if (document is null)
|
||||||
|
{
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
var dtoRecord = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
|
if (dtoRecord is null)
|
||||||
|
{
|
||||||
|
_logger.LogWarning("RU-BDU document {DocumentId} missing DTO payload", documentId);
|
||||||
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
RuBduVulnerabilityDto dto;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
dto = JsonSerializer.Deserialize<RuBduVulnerabilityDto>(dtoRecord.Payload.ToString(), SerializerOptions) ?? throw new InvalidOperationException("DTO deserialized to null");
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogError(ex, "RU-BDU failed to deserialize DTO for document {DocumentId}", documentId);
|
||||||
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var advisory = RuBduMapper.Map(dto, document, dtoRecord.ValidatedAt);
|
||||||
|
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
||||||
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogError(ex, "RU-BDU mapping failed for document {DocumentId}", documentId);
|
||||||
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
||||||
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<int> ProcessArchiveAsync(
|
||||||
|
byte[] archiveContent,
|
||||||
|
DateTimeOffset now,
|
||||||
|
HashSet<Guid> pendingDocuments,
|
||||||
|
HashSet<Guid> pendingMappings,
|
||||||
|
DateTimeOffset? archiveLastModified,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
var added = 0;
|
||||||
|
using var archiveStream = new MemoryStream(archiveContent, writable: false);
|
||||||
|
using var archive = new ZipArchive(archiveStream, ZipArchiveMode.Read, leaveOpen: false);
|
||||||
|
var entry = archive.GetEntry("export/export.xml") ?? archive.Entries.FirstOrDefault();
|
||||||
|
if (entry is null)
|
||||||
|
{
|
||||||
|
_logger.LogWarning("RU-BDU archive does not contain export/export.xml; skipping.");
|
||||||
|
return added;
|
||||||
|
}
|
||||||
|
|
||||||
|
await using var entryStream = entry.Open();
|
||||||
|
using var reader = XmlReader.Create(entryStream, new XmlReaderSettings
|
||||||
|
{
|
||||||
|
IgnoreComments = true,
|
||||||
|
IgnoreWhitespace = true,
|
||||||
|
DtdProcessing = DtdProcessing.Ignore,
|
||||||
|
CloseInput = false,
|
||||||
|
});
|
||||||
|
|
||||||
|
while (reader.Read())
|
||||||
|
{
|
||||||
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
if (reader.NodeType != XmlNodeType.Element || !reader.Name.Equals("vul", StringComparison.OrdinalIgnoreCase))
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (RuBduXmlParser.TryParse(XNode.ReadFrom(reader) as XElement ?? new XElement("vul")) is not { } dto)
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
var payload = JsonSerializer.SerializeToUtf8Bytes(dto, SerializerOptions);
|
||||||
|
var sha = Convert.ToHexString(SHA256.HashData(payload)).ToLowerInvariant();
|
||||||
|
var documentUri = BuildDocumentUri(dto.Identifier);
|
||||||
|
|
||||||
|
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, documentUri, cancellationToken).ConfigureAwait(false);
|
||||||
|
if (existing is not null && string.Equals(existing.Sha256, sha, StringComparison.OrdinalIgnoreCase))
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
var gridFsId = await _rawDocumentStorage.UploadAsync(SourceName, documentUri, payload, "application/json", null, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
|
var metadata = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase)
|
||||||
|
{
|
||||||
|
["ru-bdu.identifier"] = dto.Identifier,
|
||||||
|
};
|
||||||
|
|
||||||
|
if (!string.IsNullOrWhiteSpace(dto.Name))
|
||||||
|
{
|
||||||
|
metadata["ru-bdu.name"] = dto.Name!;
|
||||||
|
}
|
||||||
|
|
||||||
|
var recordId = existing?.Id ?? Guid.NewGuid();
|
||||||
|
var record = new DocumentRecord(
|
||||||
|
recordId,
|
||||||
|
SourceName,
|
||||||
|
documentUri,
|
||||||
|
now,
|
||||||
|
sha,
|
||||||
|
DocumentStatuses.PendingParse,
|
||||||
|
"application/json",
|
||||||
|
Headers: null,
|
||||||
|
Metadata: metadata,
|
||||||
|
Etag: null,
|
||||||
|
LastModified: archiveLastModified ?? dto.IdentifyDate,
|
||||||
|
GridFsId: gridFsId,
|
||||||
|
ExpiresAt: null);
|
||||||
|
|
||||||
|
var upserted = await _documentStore.UpsertAsync(record, cancellationToken).ConfigureAwait(false);
|
||||||
|
pendingDocuments.Add(upserted.Id);
|
||||||
|
pendingMappings.Remove(upserted.Id);
|
||||||
|
added++;
|
||||||
|
|
||||||
|
if (added >= _options.MaxVulnerabilitiesPerFetch)
|
||||||
|
{
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return added;
|
||||||
|
}
|
||||||
|
|
||||||
|
private string ResolveCacheDirectory(string? configuredPath)
|
||||||
|
{
|
||||||
|
if (!string.IsNullOrWhiteSpace(configuredPath))
|
||||||
|
{
|
||||||
|
return Path.GetFullPath(Path.IsPathRooted(configuredPath)
|
||||||
|
? configuredPath
|
||||||
|
: Path.Combine(AppContext.BaseDirectory, configuredPath));
|
||||||
|
}
|
||||||
|
|
||||||
|
return Path.Combine(AppContext.BaseDirectory, "cache", RuBduConnectorPlugin.SourceName);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void EnsureCacheDirectory()
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
Directory.CreateDirectory(_cacheDirectory);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogWarning(ex, "RU-BDU unable to ensure cache directory {CachePath}", _cacheDirectory);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void TryWriteCachedArchive(byte[] content)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
Directory.CreateDirectory(Path.GetDirectoryName(_archiveCachePath)!);
|
||||||
|
File.WriteAllBytes(_archiveCachePath, content);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogDebug(ex, "RU-BDU failed to write cache archive {CachePath}", _archiveCachePath);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private bool TryReadCachedArchive(out byte[] content)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
if (File.Exists(_archiveCachePath))
|
||||||
|
{
|
||||||
|
content = File.ReadAllBytes(_archiveCachePath);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogDebug(ex, "RU-BDU failed to read cache archive {CachePath}", _archiveCachePath);
|
||||||
|
}
|
||||||
|
|
||||||
|
content = Array.Empty<byte>();
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string BuildDocumentUri(string identifier)
|
||||||
|
{
|
||||||
|
var slug = identifier.Contains(':', StringComparison.Ordinal)
|
||||||
|
? identifier[(identifier.IndexOf(':') + 1)..]
|
||||||
|
: identifier;
|
||||||
|
return $"https://bdu.fstec.ru/vul/{slug}";
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<RuBduCursor> GetCursorAsync(CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
||||||
|
return state is null ? RuBduCursor.Empty : RuBduCursor.FromBson(state.Cursor);
|
||||||
|
}
|
||||||
|
|
||||||
|
private Task UpdateCursorAsync(RuBduCursor cursor, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
var document = cursor.ToBsonDocument();
|
||||||
|
var completedAt = cursor.LastSuccessfulFetch ?? _timeProvider.GetUtcNow();
|
||||||
|
return _stateRepository.UpdateCursorAsync(SourceName, document, completedAt, cancellationToken);
|
||||||
|
}
|
||||||
|
}
|
||||||
19
src/StellaOps.Feedser.Source.Ru.Bdu/RuBduConnectorPlugin.cs
Normal file
19
src/StellaOps.Feedser.Source.Ru.Bdu/RuBduConnectorPlugin.cs
Normal file
@@ -0,0 +1,19 @@
|
|||||||
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
|
using StellaOps.Plugin;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Bdu;
|
||||||
|
|
||||||
|
public sealed class RuBduConnectorPlugin : IConnectorPlugin
|
||||||
|
{
|
||||||
|
public const string SourceName = "ru-bdu";
|
||||||
|
|
||||||
|
public string Name => SourceName;
|
||||||
|
|
||||||
|
public bool IsAvailable(IServiceProvider services) => services is not null;
|
||||||
|
|
||||||
|
public IFeedConnector Create(IServiceProvider services)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
return ActivatorUtilities.CreateInstance<RuBduConnector>(services);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
using Microsoft.Extensions.Configuration;
|
||||||
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
|
using StellaOps.DependencyInjection;
|
||||||
|
using StellaOps.Feedser.Core.Jobs;
|
||||||
|
using StellaOps.Feedser.Source.Ru.Bdu.Configuration;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Bdu;
|
||||||
|
|
||||||
|
public sealed class RuBduDependencyInjectionRoutine : IDependencyInjectionRoutine
|
||||||
|
{
|
||||||
|
private const string ConfigurationSection = "feedser:sources:ru-bdu";
|
||||||
|
|
||||||
|
public IServiceCollection Register(IServiceCollection services, IConfiguration configuration)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
ArgumentNullException.ThrowIfNull(configuration);
|
||||||
|
|
||||||
|
services.AddRuBduConnector(options =>
|
||||||
|
{
|
||||||
|
configuration.GetSection(ConfigurationSection).Bind(options);
|
||||||
|
options.Validate();
|
||||||
|
});
|
||||||
|
|
||||||
|
services.AddTransient<RuBduFetchJob>();
|
||||||
|
services.AddTransient<RuBduParseJob>();
|
||||||
|
services.AddTransient<RuBduMapJob>();
|
||||||
|
|
||||||
|
services.PostConfigure<JobSchedulerOptions>(options =>
|
||||||
|
{
|
||||||
|
EnsureJob(options, RuBduJobKinds.Fetch, typeof(RuBduFetchJob));
|
||||||
|
EnsureJob(options, RuBduJobKinds.Parse, typeof(RuBduParseJob));
|
||||||
|
EnsureJob(options, RuBduJobKinds.Map, typeof(RuBduMapJob));
|
||||||
|
});
|
||||||
|
|
||||||
|
return services;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void EnsureJob(JobSchedulerOptions schedulerOptions, string kind, Type jobType)
|
||||||
|
{
|
||||||
|
if (schedulerOptions.Definitions.ContainsKey(kind))
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
schedulerOptions.Definitions[kind] = new JobDefinition(
|
||||||
|
kind,
|
||||||
|
jobType,
|
||||||
|
schedulerOptions.DefaultTimeout,
|
||||||
|
schedulerOptions.DefaultLeaseDuration,
|
||||||
|
CronExpression: null,
|
||||||
|
Enabled: true);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
using System.Net;
|
||||||
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using StellaOps.Feedser.Source.Ru.Bdu.Configuration;
|
||||||
|
using StellaOps.Feedser.Source.Common.Http;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Bdu;
|
||||||
|
|
||||||
|
public static class RuBduServiceCollectionExtensions
|
||||||
|
{
|
||||||
|
public static IServiceCollection AddRuBduConnector(this IServiceCollection services, Action<RuBduOptions> configure)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
ArgumentNullException.ThrowIfNull(configure);
|
||||||
|
|
||||||
|
services.AddOptions<RuBduOptions>()
|
||||||
|
.Configure(configure)
|
||||||
|
.PostConfigure(static options => options.Validate());
|
||||||
|
|
||||||
|
services.AddSourceHttpClient(RuBduOptions.HttpClientName, (sp, clientOptions) =>
|
||||||
|
{
|
||||||
|
var options = sp.GetRequiredService<IOptions<RuBduOptions>>().Value;
|
||||||
|
clientOptions.BaseAddress = options.BaseAddress;
|
||||||
|
clientOptions.Timeout = options.RequestTimeout;
|
||||||
|
clientOptions.UserAgent = options.UserAgent;
|
||||||
|
clientOptions.AllowAutoRedirect = true;
|
||||||
|
clientOptions.DefaultRequestHeaders["Accept-Language"] = options.AcceptLanguage;
|
||||||
|
clientOptions.AllowedHosts.Clear();
|
||||||
|
clientOptions.AllowedHosts.Add(options.BaseAddress.Host);
|
||||||
|
clientOptions.ConfigureHandler = handler =>
|
||||||
|
{
|
||||||
|
handler.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
|
||||||
|
handler.AllowAutoRedirect = true;
|
||||||
|
handler.UseCookies = true;
|
||||||
|
handler.CookieContainer = new CookieContainer();
|
||||||
|
};
|
||||||
|
});
|
||||||
|
|
||||||
|
services.AddTransient<RuBduConnector>();
|
||||||
|
|
||||||
|
return services;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,16 +1,18 @@
|
|||||||
<Project Sdk="Microsoft.NET.Sdk">
|
<Project Sdk="Microsoft.NET.Sdk">
|
||||||
|
|
||||||
<PropertyGroup>
|
<PropertyGroup>
|
||||||
<TargetFramework>net10.0</TargetFramework>
|
<TargetFramework>net10.0</TargetFramework>
|
||||||
<ImplicitUsings>enable</ImplicitUsings>
|
<ImplicitUsings>enable</ImplicitUsings>
|
||||||
<Nullable>enable</Nullable>
|
<Nullable>enable</Nullable>
|
||||||
</PropertyGroup>
|
</PropertyGroup>
|
||||||
|
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<ProjectReference Include="../StellaOps.Plugin/StellaOps.Plugin.csproj" />
|
<ProjectReference Include="../StellaOps.Plugin/StellaOps.Plugin.csproj" />
|
||||||
|
<ProjectReference Include="../StellaOps.Feedser.Core/StellaOps.Feedser.Core.csproj" />
|
||||||
<ProjectReference Include="../StellaOps.Feedser.Source.Common/StellaOps.Feedser.Source.Common.csproj" />
|
<ProjectReference Include="../StellaOps.Feedser.Normalization/StellaOps.Feedser.Normalization.csproj" />
|
||||||
<ProjectReference Include="../StellaOps.Feedser.Models/StellaOps.Feedser.Models.csproj" />
|
<ProjectReference Include="../StellaOps.Feedser.Source.Common/StellaOps.Feedser.Source.Common.csproj" />
|
||||||
</ItemGroup>
|
<ProjectReference Include="../StellaOps.Feedser.Models/StellaOps.Feedser.Models.csproj" />
|
||||||
</Project>
|
<ProjectReference Include="../StellaOps.Feedser.Storage.Mongo/StellaOps.Feedser.Storage.Mongo.csproj" />
|
||||||
|
</ItemGroup>
|
||||||
|
|
||||||
|
</Project>
|
||||||
|
|||||||
@@ -2,10 +2,10 @@
|
|||||||
| Task | Owner(s) | Depends on | Notes |
|
| Task | Owner(s) | Depends on | Notes |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
|FEEDCONN-RUBDU-02-001 Identify BDU data source & schema|BE-Conn-BDU|Research|**DONE (2025-10-11)** – Candidate endpoints (`https://bdu.fstec.ru/component/rsform/form/7-bdu?format=xml`, `...?format=json`) return 403/404 even with `--insecure` because TLS chain requires Russian Trusted Sub CA and WAF expects referer/session headers. Documented request/response samples in `docs/feedser-connector-research-20251011.md`; blocked until trusted root + access strategy from Ops.|
|
|FEEDCONN-RUBDU-02-001 Identify BDU data source & schema|BE-Conn-BDU|Research|**DONE (2025-10-11)** – Candidate endpoints (`https://bdu.fstec.ru/component/rsform/form/7-bdu?format=xml`, `...?format=json`) return 403/404 even with `--insecure` because TLS chain requires Russian Trusted Sub CA and WAF expects referer/session headers. Documented request/response samples in `docs/feedser-connector-research-20251011.md`; blocked until trusted root + access strategy from Ops.|
|
||||||
|FEEDCONN-RUBDU-02-002 Fetch pipeline & cursor handling|BE-Conn-BDU|Source.Common, Storage.Mongo|**TODO** – Fetcher must support custom trust store (`SourceHttpClientOptions.TrustedRootCertificates`), optional proxy, and signed cookie injection. Persist raw HTML/CSV once accessible, with cursor based on advisory `unicId` + `lastmod`. Implement retry/backoff aware of WAF transaction IDs. _(2025-10-12: Source.Common trust-store plumbing landed; blocked until sanctioned RU CA bundle is supplied.)_ **Coordination:** Ops to hand off sanctioned RU CA bundle + packaging notes for Offline Kit; Source.Common to review trust-store configuration once materials arrive.|
|
|FEEDCONN-RUBDU-02-002 Fetch pipeline & cursor handling|BE-Conn-BDU|Source.Common, Storage.Mongo|**DOING (2025-10-12)** – Fetch job now expands `vulxml.zip` into per-advisory JSON documents with cursor tracking + trust store wiring (`certificates/russian_trusted_*`). Parser/mapper emit canonical advisories; next up is wiring fixtures, regression tests, and telemetry before closing the task.|
|
||||||
|FEEDCONN-RUBDU-02-003 DTO/parser implementation|BE-Conn-BDU|Source.Common|**TODO** – Create DTOs for BDU records (title, severity, vendor/product, references, CVEs); sanitise text.|
|
|FEEDCONN-RUBDU-02-003 DTO/parser implementation|BE-Conn-BDU|Source.Common|**DOING (2025-10-12)** – `RuBduXmlParser` materialises per-entry DTOs and serialises them into Mongo DTO records; remaining work covers resilience fixtures and edge-case coverage (multi-CWE, empty software lists).|
|
||||||
|FEEDCONN-RUBDU-02-004 Canonical mapping & range primitives|BE-Conn-BDU|Models|**TODO** – Map into canonical advisories with aliases, references, and vendor range primitives. Use normalized rule checkpoints from `../StellaOps.Feedser.Merge/RANGE_PRIMITIVES_COORDINATION.md`.<br>2025-10-11 research trail: sample payload `[{"scheme":"semver","type":"range","min":"<start>","minInclusive":true,"max":"<end>","maxInclusive":false,"notes":"ru.bdu:ID"}]`; if advisories rely on firmware build strings, preserve them in `notes` until a dedicated scheme is approved.|
|
|FEEDCONN-RUBDU-02-004 Canonical mapping & range primitives|BE-Conn-BDU|Models|**DOING (2025-10-12)** – `RuBduMapper` produces canonical advisories (aliases, references, vendor packages, CVSS). Follow-up: refine status translation + range primitives once richer samples arrive; ensure fixtures cover environment/other metadata before marking DONE.|
|
||||||
|FEEDCONN-RUBDU-02-005 Deterministic fixtures & regression tests|QA|Testing|**TODO** – Add fetch/parse/map tests with fixtures; support `UPDATE_BDU_FIXTURES=1`.|
|
|FEEDCONN-RUBDU-02-005 Deterministic fixtures & regression tests|QA|Testing|**TODO** – Add fetch/parse/map tests with fixtures; support `UPDATE_BDU_FIXTURES=1`.|
|
||||||
|FEEDCONN-RUBDU-02-006 Telemetry & documentation|DevEx|Docs|**TODO** – Add logging/metrics, document connector configuration, close backlog when complete.|
|
|FEEDCONN-RUBDU-02-006 Telemetry & documentation|DevEx|Docs|**TODO** – Add logging/metrics, document connector configuration, close backlog when complete.|
|
||||||
|FEEDCONN-RUBDU-02-007 Access & export options assessment|BE-Conn-BDU|Research|**TODO** – Once access unblocked, compare RSS/Atom (if restored) vs HTML table export (`/vul` list) and legacy CSV dumps. Need to confirm whether login/anti-bot tokens required and outline offline mirroring plan (one-time tarball seeded into Offline Kit).|
|
|FEEDCONN-RUBDU-02-007 Access & export options assessment|BE-Conn-BDU|Research|**TODO** – Once access unblocked, compare RSS/Atom (if restored) vs HTML table export (`/vul` list) and legacy CSV dumps. Need to confirm whether login/anti-bot tokens required and outline offline mirroring plan (one-time tarball seeded into Offline Kit).|
|
||||||
|FEEDCONN-RUBDU-02-008 Trusted root onboarding plan|BE-Conn-BDU|Source.Common|**BLOCKED** – 2025-10-11: Attempt to download Russian Trusted Sub CA returned placeholder HTML; need alternate distribution (mirror or manual bundle) before TLS validation succeeds.<br>2025-10-11 23:05Z: Shared HTTP trust-store support landed (`SourceHttpClientOptions.TrustedRootCertificates`, config keys `feedser:httpClients:source.bdu:*`); now blocked on Ops delivering sanctioned RU CA bundle + Offline Kit packaging instructions.|
|
|FEEDCONN-RUBDU-02-008 Trusted root onboarding plan|BE-Conn-BDU|Source.Common|**DOING (2025-10-12)** – Mirrored official Russian Trusted Root/Sub CA PEMs from rostelecom.ru (`certificates/russian_trusted_root_ca.pem`, `certificates/russian_trusted_sub_ca.pem`, bundle `certificates/russian_trusted_bundle.pem`) and validated TLS handshake. Next: confirm packaging guidance for Offline Kit + config samples using `feedser:httpClients:source.bdu:trustedRootPaths`.|
|
||||||
|
|||||||
Binary file not shown.
@@ -0,0 +1,7 @@
|
|||||||
|
<html>
|
||||||
|
<body>
|
||||||
|
<ul>
|
||||||
|
<li><a href="/materialy/uyazvimosti/bulletin-sample.json.zip" title="Bulletin Sample">Bulletin Sample</a></li>
|
||||||
|
</ul>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -0,0 +1,165 @@
|
|||||||
|
[
|
||||||
|
{
|
||||||
|
"advisoryKey": "BDU:2025-01001",
|
||||||
|
"affectedPackages": [
|
||||||
|
{
|
||||||
|
"type": "vendor",
|
||||||
|
"identifier": "SampleSCADA <= 4.2",
|
||||||
|
"platform": null,
|
||||||
|
"versionRanges": [],
|
||||||
|
"normalizedVersions": [],
|
||||||
|
"statuses": [
|
||||||
|
{
|
||||||
|
"provenance": {
|
||||||
|
"source": "ru-nkcki",
|
||||||
|
"kind": "package-status",
|
||||||
|
"value": "patch_available",
|
||||||
|
"decisionReason": null,
|
||||||
|
"recordedAt": "2025-09-22T00:00:00+00:00",
|
||||||
|
"fieldMask": [
|
||||||
|
"affectedpackages[].statuses[]"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"status": "fixed"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"provenance": [
|
||||||
|
{
|
||||||
|
"source": "ru-nkcki",
|
||||||
|
"kind": "package",
|
||||||
|
"value": "SampleSCADA <= 4.2",
|
||||||
|
"decisionReason": null,
|
||||||
|
"recordedAt": "2025-09-22T00:00:00+00:00",
|
||||||
|
"fieldMask": [
|
||||||
|
"affectedpackages[]"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"aliases": [
|
||||||
|
"BDU:2025-01001",
|
||||||
|
"CVE-2025-0101"
|
||||||
|
],
|
||||||
|
"credits": [],
|
||||||
|
"cvssMetrics": [
|
||||||
|
{
|
||||||
|
"baseScore": 8.5,
|
||||||
|
"baseSeverity": "high",
|
||||||
|
"provenance": {
|
||||||
|
"source": "ru-nkcki",
|
||||||
|
"kind": "cvss",
|
||||||
|
"value": "CVSS:3.1/AV:N/AC:H/PR:L/UI:N/S:C/C:H/I:H/A:H",
|
||||||
|
"decisionReason": null,
|
||||||
|
"recordedAt": "2025-09-22T00:00:00+00:00",
|
||||||
|
"fieldMask": [
|
||||||
|
"cvssmetrics[]"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"vector": "CVSS:3.1/AV:N/AC:H/PR:L/UI:N/S:C/C:H/I:H/A:H",
|
||||||
|
"version": "3.1"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"exploitKnown": true,
|
||||||
|
"language": "ru",
|
||||||
|
"modified": "2025-09-22T00:00:00+00:00",
|
||||||
|
"provenance": [
|
||||||
|
{
|
||||||
|
"source": "ru-nkcki",
|
||||||
|
"kind": "advisory",
|
||||||
|
"value": "BDU:2025-01001",
|
||||||
|
"decisionReason": null,
|
||||||
|
"recordedAt": "2025-09-22T00:00:00+00:00",
|
||||||
|
"fieldMask": [
|
||||||
|
"advisory"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"published": "2025-09-20T00:00:00+00:00",
|
||||||
|
"references": [
|
||||||
|
{
|
||||||
|
"kind": "details",
|
||||||
|
"provenance": {
|
||||||
|
"source": "ru-nkcki",
|
||||||
|
"kind": "reference",
|
||||||
|
"value": "https://bdu.fstec.ru/vul/2025-01001",
|
||||||
|
"decisionReason": null,
|
||||||
|
"recordedAt": "2025-09-22T00:00:00+00:00",
|
||||||
|
"fieldMask": [
|
||||||
|
"references[]"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"sourceTag": "bdu",
|
||||||
|
"summary": null,
|
||||||
|
"url": "https://bdu.fstec.ru/vul/2025-01001"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"kind": "details",
|
||||||
|
"provenance": {
|
||||||
|
"source": "ru-nkcki",
|
||||||
|
"kind": "reference",
|
||||||
|
"value": "https://cert.gov.ru/materialy/uyazvimosti/2025-01001",
|
||||||
|
"decisionReason": null,
|
||||||
|
"recordedAt": "2025-09-22T00:00:00+00:00",
|
||||||
|
"fieldMask": [
|
||||||
|
"references[]"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"sourceTag": null,
|
||||||
|
"summary": null,
|
||||||
|
"url": "https://cert.gov.ru/materialy/uyazvimosti/2025-01001"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"kind": "details",
|
||||||
|
"provenance": {
|
||||||
|
"source": "ru-nkcki",
|
||||||
|
"kind": "reference",
|
||||||
|
"value": "https://cert.gov.ru/materialy/uyazvimosti/2025-01001",
|
||||||
|
"decisionReason": null,
|
||||||
|
"recordedAt": "2025-09-22T00:00:00+00:00",
|
||||||
|
"fieldMask": [
|
||||||
|
"references[]"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"sourceTag": "ru-nkcki",
|
||||||
|
"summary": null,
|
||||||
|
"url": "https://cert.gov.ru/materialy/uyazvimosti/2025-01001"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"kind": "cwe",
|
||||||
|
"provenance": {
|
||||||
|
"source": "ru-nkcki",
|
||||||
|
"kind": "reference",
|
||||||
|
"value": "https://cwe.mitre.org/data/definitions/321.html",
|
||||||
|
"decisionReason": null,
|
||||||
|
"recordedAt": "2025-09-22T00:00:00+00:00",
|
||||||
|
"fieldMask": [
|
||||||
|
"references[]"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"sourceTag": "cwe",
|
||||||
|
"summary": "Use of Hard-coded Cryptographic Key",
|
||||||
|
"url": "https://cwe.mitre.org/data/definitions/321.html"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"kind": "external",
|
||||||
|
"provenance": {
|
||||||
|
"source": "ru-nkcki",
|
||||||
|
"kind": "reference",
|
||||||
|
"value": "https://vendor.example/advisories/sample-scada",
|
||||||
|
"decisionReason": null,
|
||||||
|
"recordedAt": "2025-09-22T00:00:00+00:00",
|
||||||
|
"fieldMask": [
|
||||||
|
"references[]"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"sourceTag": null,
|
||||||
|
"summary": null,
|
||||||
|
"url": "https://vendor.example/advisories/sample-scada"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"severity": "critical",
|
||||||
|
"summary": "Authenticated RCE in Sample SCADA",
|
||||||
|
"title": "Authenticated RCE in Sample SCADA"
|
||||||
|
}
|
||||||
|
]
|
||||||
@@ -0,0 +1,289 @@
|
|||||||
|
using System.Collections.Generic;
|
||||||
|
using System.IO;
|
||||||
|
using System.Linq;
|
||||||
|
using System.Threading;
|
||||||
|
using System.Threading.Tasks;
|
||||||
|
using System.Net;
|
||||||
|
using System.Net.Http;
|
||||||
|
using System.Net.Http.Headers;
|
||||||
|
using System.Text;
|
||||||
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
|
using Microsoft.Extensions.Http;
|
||||||
|
using Microsoft.Extensions.Logging;
|
||||||
|
using Microsoft.Extensions.Logging.Abstractions;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using Microsoft.Extensions.Time.Testing;
|
||||||
|
using MongoDB.Bson;
|
||||||
|
using StellaOps.Feedser.Source.Common;
|
||||||
|
using StellaOps.Feedser.Source.Common.Http;
|
||||||
|
using StellaOps.Feedser.Source.Common.Testing;
|
||||||
|
using StellaOps.Feedser.Source.Ru.Nkcki;
|
||||||
|
using StellaOps.Feedser.Source.Ru.Nkcki.Configuration;
|
||||||
|
using StellaOps.Feedser.Storage.Mongo;
|
||||||
|
using StellaOps.Feedser.Storage.Mongo.Advisories;
|
||||||
|
using StellaOps.Feedser.Storage.Mongo.Documents;
|
||||||
|
using StellaOps.Feedser.Testing;
|
||||||
|
using StellaOps.Feedser.Models;
|
||||||
|
using MongoDB.Driver;
|
||||||
|
using Xunit;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Nkcki.Tests;
|
||||||
|
|
||||||
|
[Collection("mongo-fixture")]
|
||||||
|
public sealed class RuNkckiConnectorTests : IAsyncLifetime
|
||||||
|
{
|
||||||
|
private static readonly Uri ListingUri = new("https://cert.gov.ru/materialy/uyazvimosti/");
|
||||||
|
private static readonly Uri BulletinUri = new("https://cert.gov.ru/materialy/uyazvimosti/bulletin-sample.json.zip");
|
||||||
|
|
||||||
|
private readonly MongoIntegrationFixture _fixture;
|
||||||
|
private readonly FakeTimeProvider _timeProvider;
|
||||||
|
private readonly CannedHttpMessageHandler _handler;
|
||||||
|
|
||||||
|
public RuNkckiConnectorTests(MongoIntegrationFixture fixture)
|
||||||
|
{
|
||||||
|
_fixture = fixture;
|
||||||
|
_timeProvider = new FakeTimeProvider(new DateTimeOffset(2025, 10, 12, 0, 0, 0, TimeSpan.Zero));
|
||||||
|
_handler = new CannedHttpMessageHandler();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task FetchParseMap_ProducesExpectedSnapshot()
|
||||||
|
{
|
||||||
|
await using var provider = await BuildServiceProviderAsync();
|
||||||
|
SeedListingAndBulletin();
|
||||||
|
|
||||||
|
var connector = provider.GetRequiredService<RuNkckiConnector>();
|
||||||
|
await connector.FetchAsync(provider, CancellationToken.None);
|
||||||
|
_timeProvider.Advance(TimeSpan.FromMinutes(1));
|
||||||
|
await connector.ParseAsync(provider, CancellationToken.None);
|
||||||
|
await connector.MapAsync(provider, CancellationToken.None);
|
||||||
|
|
||||||
|
var advisoryStore = provider.GetRequiredService<IAdvisoryStore>();
|
||||||
|
var advisories = await advisoryStore.GetRecentAsync(10, CancellationToken.None);
|
||||||
|
Assert.Single(advisories);
|
||||||
|
|
||||||
|
var snapshot = SnapshotSerializer.ToSnapshot(advisories);
|
||||||
|
WriteOrAssertSnapshot(snapshot, "nkcki-advisories.snapshot.json");
|
||||||
|
|
||||||
|
var documentStore = provider.GetRequiredService<IDocumentStore>();
|
||||||
|
var document = await documentStore.FindBySourceAndUriAsync(RuNkckiConnectorPlugin.SourceName, "https://cert.gov.ru/materialy/uyazvimosti/BDU:2025-01001", CancellationToken.None);
|
||||||
|
Assert.NotNull(document);
|
||||||
|
Assert.Equal(DocumentStatuses.Mapped, document!.Status);
|
||||||
|
|
||||||
|
var stateRepository = provider.GetRequiredService<ISourceStateRepository>();
|
||||||
|
var state = await stateRepository.TryGetAsync(RuNkckiConnectorPlugin.SourceName, CancellationToken.None);
|
||||||
|
Assert.NotNull(state);
|
||||||
|
Assert.True(IsEmptyArray(state!.Cursor, "pendingDocuments"));
|
||||||
|
Assert.True(IsEmptyArray(state.Cursor, "pendingMappings"));
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task Fetch_ReusesCachedBulletinWhenListingFails()
|
||||||
|
{
|
||||||
|
await using var provider = await BuildServiceProviderAsync();
|
||||||
|
SeedListingAndBulletin();
|
||||||
|
|
||||||
|
var connector = provider.GetRequiredService<RuNkckiConnector>();
|
||||||
|
await connector.FetchAsync(provider, CancellationToken.None);
|
||||||
|
|
||||||
|
_handler.Clear();
|
||||||
|
_handler.AddResponse(ListingUri, () => new HttpResponseMessage(HttpStatusCode.InternalServerError)
|
||||||
|
{
|
||||||
|
Content = new StringContent("error", Encoding.UTF8, "text/plain"),
|
||||||
|
});
|
||||||
|
|
||||||
|
var advisoryStore = provider.GetRequiredService<IAdvisoryStore>();
|
||||||
|
var before = await advisoryStore.GetRecentAsync(10, CancellationToken.None);
|
||||||
|
Assert.NotEmpty(before);
|
||||||
|
|
||||||
|
await connector.FetchAsync(provider, CancellationToken.None);
|
||||||
|
|
||||||
|
var after = await advisoryStore.GetRecentAsync(10, CancellationToken.None);
|
||||||
|
Assert.Equal(before.Select(advisory => advisory.AdvisoryKey).OrderBy(static key => key), after.Select(advisory => advisory.AdvisoryKey).OrderBy(static key => key));
|
||||||
|
|
||||||
|
_handler.AssertNoPendingResponses();
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<ServiceProvider> BuildServiceProviderAsync()
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await _fixture.Client.DropDatabaseAsync(_fixture.Database.DatabaseNamespace.DatabaseName);
|
||||||
|
}
|
||||||
|
catch (MongoConnectionException ex)
|
||||||
|
{
|
||||||
|
Assert.Skip($"Mongo runner unavailable: {ex.Message}");
|
||||||
|
}
|
||||||
|
catch (TimeoutException ex)
|
||||||
|
{
|
||||||
|
Assert.Skip($"Mongo runner unavailable: {ex.Message}");
|
||||||
|
}
|
||||||
|
|
||||||
|
_handler.Clear();
|
||||||
|
|
||||||
|
var services = new ServiceCollection();
|
||||||
|
services.AddLogging(builder => builder.AddProvider(NullLoggerProvider.Instance));
|
||||||
|
services.AddSingleton<TimeProvider>(_timeProvider);
|
||||||
|
|
||||||
|
services.AddMongoStorage(options =>
|
||||||
|
{
|
||||||
|
options.ConnectionString = _fixture.Runner.ConnectionString;
|
||||||
|
options.DatabaseName = _fixture.Database.DatabaseNamespace.DatabaseName;
|
||||||
|
options.CommandTimeout = TimeSpan.FromSeconds(5);
|
||||||
|
});
|
||||||
|
|
||||||
|
services.AddSourceCommon();
|
||||||
|
services.AddRuNkckiConnector(options =>
|
||||||
|
{
|
||||||
|
options.BaseAddress = new Uri("https://cert.gov.ru/");
|
||||||
|
options.ListingPath = "/materialy/uyazvimosti/";
|
||||||
|
options.MaxBulletinsPerFetch = 2;
|
||||||
|
options.MaxVulnerabilitiesPerFetch = 50;
|
||||||
|
var cacheRoot = Path.Combine(Path.GetTempPath(), "stellaops-tests", _fixture.Database.DatabaseNamespace.DatabaseName);
|
||||||
|
Directory.CreateDirectory(cacheRoot);
|
||||||
|
options.CacheDirectory = Path.Combine(cacheRoot, "ru-nkcki");
|
||||||
|
options.RequestDelay = TimeSpan.Zero;
|
||||||
|
});
|
||||||
|
|
||||||
|
services.Configure<HttpClientFactoryOptions>(RuNkckiOptions.HttpClientName, builderOptions =>
|
||||||
|
{
|
||||||
|
builderOptions.HttpMessageHandlerBuilderActions.Add(builder => builder.PrimaryHandler = _handler);
|
||||||
|
});
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var provider = services.BuildServiceProvider();
|
||||||
|
var bootstrapper = provider.GetRequiredService<MongoBootstrapper>();
|
||||||
|
await bootstrapper.InitializeAsync(CancellationToken.None);
|
||||||
|
return provider;
|
||||||
|
}
|
||||||
|
catch (MongoConnectionException ex)
|
||||||
|
{
|
||||||
|
Assert.Skip($"Mongo runner unavailable: {ex.Message}");
|
||||||
|
throw; // Unreachable
|
||||||
|
}
|
||||||
|
catch (TimeoutException ex)
|
||||||
|
{
|
||||||
|
Assert.Skip($"Mongo runner unavailable: {ex.Message}");
|
||||||
|
throw;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void SeedListingAndBulletin()
|
||||||
|
{
|
||||||
|
var listingHtml = ReadFixture("listing.html");
|
||||||
|
_handler.AddTextResponse(ListingUri, listingHtml, "text/html");
|
||||||
|
|
||||||
|
var bulletinBytes = ReadBulletinFixture("bulletin-sample.json.zip");
|
||||||
|
_handler.AddResponse(BulletinUri, () =>
|
||||||
|
{
|
||||||
|
var response = new HttpResponseMessage(HttpStatusCode.OK)
|
||||||
|
{
|
||||||
|
Content = new ByteArrayContent(bulletinBytes),
|
||||||
|
};
|
||||||
|
response.Content.Headers.ContentType = new MediaTypeHeaderValue("application/zip");
|
||||||
|
response.Content.Headers.LastModified = new DateTimeOffset(2025, 9, 22, 0, 0, 0, TimeSpan.Zero);
|
||||||
|
return response;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
private static bool IsEmptyArray(BsonDocument document, string field)
|
||||||
|
{
|
||||||
|
if (!document.TryGetValue(field, out var value) || value is not BsonArray array)
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
return array.Count == 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string ReadFixture(string filename)
|
||||||
|
{
|
||||||
|
var path = Path.Combine("Fixtures", filename);
|
||||||
|
var resolved = ResolveFixturePath(path);
|
||||||
|
return File.ReadAllText(resolved);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static byte[] ReadBulletinFixture(string filename)
|
||||||
|
{
|
||||||
|
var path = Path.Combine("Fixtures", filename);
|
||||||
|
var resolved = ResolveFixturePath(path);
|
||||||
|
return File.ReadAllBytes(resolved);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string ResolveFixturePath(string relativePath)
|
||||||
|
{
|
||||||
|
var projectRoot = GetProjectRoot();
|
||||||
|
var projectPath = Path.Combine(projectRoot, relativePath);
|
||||||
|
if (File.Exists(projectPath))
|
||||||
|
{
|
||||||
|
return projectPath;
|
||||||
|
}
|
||||||
|
|
||||||
|
var binaryPath = Path.Combine(AppContext.BaseDirectory, relativePath);
|
||||||
|
if (File.Exists(binaryPath))
|
||||||
|
{
|
||||||
|
return Path.GetFullPath(binaryPath);
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new FileNotFoundException($"Fixture not found: {relativePath}");
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void WriteOrAssertSnapshot(string snapshot, string filename)
|
||||||
|
{
|
||||||
|
if (ShouldUpdateFixtures())
|
||||||
|
{
|
||||||
|
var path = GetWritableFixturePath(filename);
|
||||||
|
Directory.CreateDirectory(Path.GetDirectoryName(path)!);
|
||||||
|
File.WriteAllText(path, snapshot);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
var expectedPath = ResolveFixturePath(Path.Combine("Fixtures", filename));
|
||||||
|
if (!File.Exists(expectedPath))
|
||||||
|
{
|
||||||
|
throw new FileNotFoundException($"Expected snapshot missing: {expectedPath}. Set UPDATE_NKCKI_FIXTURES=1 to generate.");
|
||||||
|
}
|
||||||
|
|
||||||
|
var expected = File.ReadAllText(expectedPath);
|
||||||
|
Assert.Equal(Normalize(expected), Normalize(snapshot));
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string GetWritableFixturePath(string filename)
|
||||||
|
{
|
||||||
|
var projectRoot = GetProjectRoot();
|
||||||
|
return Path.Combine(projectRoot, "Fixtures", filename);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static bool ShouldUpdateFixtures()
|
||||||
|
{
|
||||||
|
var value = Environment.GetEnvironmentVariable("UPDATE_NKCKI_FIXTURES");
|
||||||
|
return string.Equals(value, "1", StringComparison.OrdinalIgnoreCase)
|
||||||
|
|| string.Equals(value, "true", StringComparison.OrdinalIgnoreCase);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string Normalize(string text)
|
||||||
|
=> text.Replace("\r\n", "\n", StringComparison.Ordinal);
|
||||||
|
|
||||||
|
private static string GetProjectRoot()
|
||||||
|
{
|
||||||
|
var current = AppContext.BaseDirectory;
|
||||||
|
while (!string.IsNullOrEmpty(current))
|
||||||
|
{
|
||||||
|
var candidate = Path.Combine(current, "StellaOps.Feedser.Source.Ru.Nkcki.Tests.csproj");
|
||||||
|
if (File.Exists(candidate))
|
||||||
|
{
|
||||||
|
return current;
|
||||||
|
}
|
||||||
|
|
||||||
|
current = Path.GetDirectoryName(current);
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new InvalidOperationException("Unable to locate project root for Ru.Nkcki tests.");
|
||||||
|
}
|
||||||
|
|
||||||
|
public Task InitializeAsync() => Task.CompletedTask;
|
||||||
|
|
||||||
|
public async Task DisposeAsync()
|
||||||
|
=> await _fixture.Client.DropDatabaseAsync(_fixture.Database.DatabaseNamespace.DatabaseName);
|
||||||
|
}
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
using System.Text.Json;
|
||||||
|
using StellaOps.Feedser.Source.Ru.Nkcki.Internal;
|
||||||
|
using Xunit;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Nkcki.Tests;
|
||||||
|
|
||||||
|
public sealed class RuNkckiJsonParserTests
|
||||||
|
{
|
||||||
|
[Fact]
|
||||||
|
public void Parse_WellFormedEntry_ReturnsDto()
|
||||||
|
{
|
||||||
|
const string json = """
|
||||||
|
{
|
||||||
|
"vuln_id": {"MITRE": "CVE-2025-0001", "FSTEC": "BDU:2025-00001"},
|
||||||
|
"date_published": "2025-09-01",
|
||||||
|
"date_updated": "2025-09-02",
|
||||||
|
"cvss_rating": "КРИТИЧЕСКИЙ",
|
||||||
|
"patch_available": true,
|
||||||
|
"description": "Test description",
|
||||||
|
"cwe": {"cwe_number": 79, "cwe_description": "Cross-site scripting"},
|
||||||
|
"product_category": "Web",
|
||||||
|
"mitigation": "Apply update",
|
||||||
|
"vulnerable_software": {"software_text": "ExampleApp 1.0", "cpe": false},
|
||||||
|
"cvss": {"cvss_score": 8.8, "cvss_vector": "AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H", "cvss_score_v4": 5.5, "cvss_vector_v4": "AV:N/AC:L/AT:N/PR:N/UI:N/VC:H/VI:H/VA:H"},
|
||||||
|
"impact": "ACE",
|
||||||
|
"method_of_exploitation": "Special request",
|
||||||
|
"user_interaction": false,
|
||||||
|
"urls": ["https://example.com/advisory", "https://cert.gov.ru/materialy/uyazvimosti/2025-00001"]
|
||||||
|
}
|
||||||
|
""";
|
||||||
|
|
||||||
|
using var document = JsonDocument.Parse(json);
|
||||||
|
var dto = RuNkckiJsonParser.Parse(document.RootElement);
|
||||||
|
|
||||||
|
Assert.Equal("BDU:2025-00001", dto.FstecId);
|
||||||
|
Assert.Equal("CVE-2025-0001", dto.MitreId);
|
||||||
|
Assert.Equal(8.8, dto.CvssScore);
|
||||||
|
Assert.Equal("AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H", dto.CvssVector);
|
||||||
|
Assert.True(dto.PatchAvailable);
|
||||||
|
Assert.Equal(79, dto.Cwe?.Number);
|
||||||
|
Assert.Equal(2, dto.Urls.Length);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
using System.Collections.Immutable;
|
||||||
|
using MongoDB.Bson;
|
||||||
|
using StellaOps.Feedser.Source.Common;
|
||||||
|
using StellaOps.Feedser.Models;
|
||||||
|
using StellaOps.Feedser.Source.Ru.Nkcki.Internal;
|
||||||
|
using StellaOps.Feedser.Storage.Mongo.Documents;
|
||||||
|
using Xunit;
|
||||||
|
using System.Reflection;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Nkcki.Tests;
|
||||||
|
|
||||||
|
public sealed class RuNkckiMapperTests
|
||||||
|
{
|
||||||
|
[Fact]
|
||||||
|
public void Map_ConstructsCanonicalAdvisory()
|
||||||
|
{
|
||||||
|
var dto = new RuNkckiVulnerabilityDto(
|
||||||
|
FstecId: "BDU:2025-00001",
|
||||||
|
MitreId: "CVE-2025-0001",
|
||||||
|
DatePublished: new DateTimeOffset(2025, 9, 1, 0, 0, 0, TimeSpan.Zero),
|
||||||
|
DateUpdated: new DateTimeOffset(2025, 9, 2, 0, 0, 0, TimeSpan.Zero),
|
||||||
|
CvssRating: "КРИТИЧЕСКИЙ",
|
||||||
|
PatchAvailable: true,
|
||||||
|
Description: "Test NKCKI vulnerability",
|
||||||
|
Cwe: new RuNkckiCweDto(79, "Cross-site scripting"),
|
||||||
|
ProductCategory: "Web",
|
||||||
|
Mitigation: "Apply update",
|
||||||
|
VulnerableSoftwareText: "ExampleApp <= 1.0",
|
||||||
|
VulnerableSoftwareHasCpe: false,
|
||||||
|
CvssScore: 8.8,
|
||||||
|
CvssVector: "AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:H",
|
||||||
|
CvssScoreV4: null,
|
||||||
|
CvssVectorV4: null,
|
||||||
|
Impact: "ACE",
|
||||||
|
MethodOfExploitation: "Special request",
|
||||||
|
UserInteraction: false,
|
||||||
|
Urls: ImmutableArray.Create("https://example.com/advisory"));
|
||||||
|
|
||||||
|
var document = new DocumentRecord(
|
||||||
|
Guid.NewGuid(),
|
||||||
|
RuNkckiConnectorPlugin.SourceName,
|
||||||
|
"https://cert.gov.ru/materialy/uyazvimosti/2025-00001",
|
||||||
|
DateTimeOffset.UtcNow,
|
||||||
|
"abc",
|
||||||
|
DocumentStatuses.PendingMap,
|
||||||
|
"application/json",
|
||||||
|
null,
|
||||||
|
null,
|
||||||
|
null,
|
||||||
|
dto.DateUpdated,
|
||||||
|
ObjectId.GenerateNewId());
|
||||||
|
|
||||||
|
Assert.Equal("КРИТИЧЕСКИЙ", dto.CvssRating);
|
||||||
|
var normalizeSeverity = typeof(RuNkckiMapper).GetMethod("NormalizeSeverity", BindingFlags.NonPublic | BindingFlags.Static)!;
|
||||||
|
var ratingSeverity = (string?)normalizeSeverity.Invoke(null, new object?[] { dto.CvssRating });
|
||||||
|
Assert.Equal("critical", ratingSeverity);
|
||||||
|
|
||||||
|
var advisory = RuNkckiMapper.Map(dto, document, dto.DateUpdated!.Value);
|
||||||
|
|
||||||
|
Assert.Contains("BDU:2025-00001", advisory.Aliases);
|
||||||
|
Assert.Contains("CVE-2025-0001", advisory.Aliases);
|
||||||
|
Assert.Equal("critical", advisory.Severity);
|
||||||
|
Assert.True(advisory.ExploitKnown);
|
||||||
|
Assert.Single(advisory.AffectedPackages);
|
||||||
|
Assert.Single(advisory.CvssMetrics);
|
||||||
|
Assert.Contains(advisory.References, reference => reference.Url.Contains("example.com", StringComparison.OrdinalIgnoreCase));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
<Project Sdk="Microsoft.NET.Sdk">
|
||||||
|
<PropertyGroup>
|
||||||
|
<TargetFramework>net10.0</TargetFramework>
|
||||||
|
<ImplicitUsings>enable</ImplicitUsings>
|
||||||
|
<Nullable>enable</Nullable>
|
||||||
|
</PropertyGroup>
|
||||||
|
|
||||||
|
<ItemGroup>
|
||||||
|
<ProjectReference Include="../StellaOps.Feedser.Models/StellaOps.Feedser.Models.csproj" />
|
||||||
|
<ProjectReference Include="../StellaOps.Feedser.Normalization/StellaOps.Feedser.Normalization.csproj" />
|
||||||
|
<ProjectReference Include="../StellaOps.Feedser.Source.Ru.Nkcki/StellaOps.Feedser.Source.Ru.Nkcki.csproj" />
|
||||||
|
</ItemGroup>
|
||||||
|
</Project>
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
using System;
|
|
||||||
using System.Threading;
|
|
||||||
using System.Threading.Tasks;
|
|
||||||
using StellaOps.Plugin;
|
|
||||||
|
|
||||||
namespace StellaOps.Feedser.Source.Ru.Nkcki;
|
|
||||||
|
|
||||||
public sealed class RuNkckiConnectorPlugin : IConnectorPlugin
|
|
||||||
{
|
|
||||||
public string Name => "ru-nkcki";
|
|
||||||
|
|
||||||
public bool IsAvailable(IServiceProvider services) => true;
|
|
||||||
|
|
||||||
public IFeedConnector Create(IServiceProvider services) => new StubConnector(Name);
|
|
||||||
|
|
||||||
private sealed class StubConnector : IFeedConnector
|
|
||||||
{
|
|
||||||
public StubConnector(string sourceName) => SourceName = sourceName;
|
|
||||||
|
|
||||||
public string SourceName { get; }
|
|
||||||
|
|
||||||
public Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken) => Task.CompletedTask;
|
|
||||||
|
|
||||||
public Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken) => Task.CompletedTask;
|
|
||||||
|
|
||||||
public Task MapAsync(IServiceProvider services, CancellationToken cancellationToken) => Task.CompletedTask;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@@ -0,0 +1,127 @@
|
|||||||
|
using System.Net;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Nkcki.Configuration;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Connector options for the Russian NKTsKI bulletin ingestion pipeline.
|
||||||
|
/// </summary>
|
||||||
|
public sealed class RuNkckiOptions
|
||||||
|
{
|
||||||
|
public const string HttpClientName = "ru-nkcki";
|
||||||
|
|
||||||
|
private static readonly TimeSpan DefaultRequestTimeout = TimeSpan.FromSeconds(90);
|
||||||
|
private static readonly TimeSpan DefaultFailureBackoff = TimeSpan.FromMinutes(20);
|
||||||
|
private static readonly TimeSpan DefaultListingCache = TimeSpan.FromMinutes(10);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Base endpoint used for resolving relative resource links.
|
||||||
|
/// </summary>
|
||||||
|
public Uri BaseAddress { get; set; } = new("https://cert.gov.ru/", UriKind.Absolute);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Relative path to the bulletin listing page.
|
||||||
|
/// </summary>
|
||||||
|
public string ListingPath { get; set; } = "materialy/uyazvimosti/";
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Timeout applied to listing and bulletin fetch requests.
|
||||||
|
/// </summary>
|
||||||
|
public TimeSpan RequestTimeout { get; set; } = DefaultRequestTimeout;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Backoff applied when the listing or attachments cannot be retrieved.
|
||||||
|
/// </summary>
|
||||||
|
public TimeSpan FailureBackoff { get; set; } = DefaultFailureBackoff;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Maximum number of bulletin attachments downloaded per fetch run.
|
||||||
|
/// </summary>
|
||||||
|
public int MaxBulletinsPerFetch { get; set; } = 5;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Maximum number of vulnerabilities ingested per fetch cycle across all attachments.
|
||||||
|
/// </summary>
|
||||||
|
public int MaxVulnerabilitiesPerFetch { get; set; } = 250;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Maximum bulletin identifiers remembered to avoid refetching historical files.
|
||||||
|
/// </summary>
|
||||||
|
public int KnownBulletinCapacity { get; set; } = 512;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Delay between sequential bulletin downloads.
|
||||||
|
/// </summary>
|
||||||
|
public TimeSpan RequestDelay { get; set; } = TimeSpan.FromMilliseconds(250);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Duration the HTML listing can be cached before forcing a refetch.
|
||||||
|
/// </summary>
|
||||||
|
public TimeSpan ListingCacheDuration { get; set; } = DefaultListingCache;
|
||||||
|
|
||||||
|
public string UserAgent { get; set; } = "StellaOps/Feedser (+https://stella-ops.org)";
|
||||||
|
|
||||||
|
public string AcceptLanguage { get; set; } = "ru-RU,ru;q=0.9,en-US;q=0.6,en;q=0.4";
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Absolute URI for the listing page.
|
||||||
|
/// </summary>
|
||||||
|
public Uri ListingUri => new(BaseAddress, ListingPath);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Optional directory for caching downloaded bulletins (relative paths resolve under the content root).
|
||||||
|
/// </summary>
|
||||||
|
public string? CacheDirectory { get; set; } = null;
|
||||||
|
|
||||||
|
public void Validate()
|
||||||
|
{
|
||||||
|
if (BaseAddress is null || !BaseAddress.IsAbsoluteUri)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuNkcki BaseAddress must be an absolute URI.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (string.IsNullOrWhiteSpace(ListingPath))
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuNkcki ListingPath must be provided.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (RequestTimeout <= TimeSpan.Zero)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuNkcki RequestTimeout must be positive.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (FailureBackoff < TimeSpan.Zero)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuNkcki FailureBackoff cannot be negative.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (MaxBulletinsPerFetch <= 0)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuNkcki MaxBulletinsPerFetch must be greater than zero.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (MaxVulnerabilitiesPerFetch <= 0)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuNkcki MaxVulnerabilitiesPerFetch must be greater than zero.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (KnownBulletinCapacity <= 0)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuNkcki KnownBulletinCapacity must be greater than zero.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (CacheDirectory is not null && CacheDirectory.Trim().Length == 0)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuNkcki CacheDirectory cannot be whitespace.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (string.IsNullOrWhiteSpace(UserAgent))
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuNkcki UserAgent cannot be empty.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (string.IsNullOrWhiteSpace(AcceptLanguage))
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("RuNkcki AcceptLanguage cannot be empty.");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
108
src/StellaOps.Feedser.Source.Ru.Nkcki/Internal/RuNkckiCursor.cs
Normal file
108
src/StellaOps.Feedser.Source.Ru.Nkcki/Internal/RuNkckiCursor.cs
Normal file
@@ -0,0 +1,108 @@
|
|||||||
|
using MongoDB.Bson;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Nkcki.Internal;
|
||||||
|
|
||||||
|
internal sealed record RuNkckiCursor(
|
||||||
|
IReadOnlyCollection<Guid> PendingDocuments,
|
||||||
|
IReadOnlyCollection<Guid> PendingMappings,
|
||||||
|
IReadOnlyCollection<string> KnownBulletins,
|
||||||
|
DateTimeOffset? LastListingFetchAt)
|
||||||
|
{
|
||||||
|
private static readonly IReadOnlyCollection<Guid> EmptyGuids = Array.Empty<Guid>();
|
||||||
|
private static readonly IReadOnlyCollection<string> EmptyBulletins = Array.Empty<string>();
|
||||||
|
|
||||||
|
public static RuNkckiCursor Empty { get; } = new(EmptyGuids, EmptyGuids, EmptyBulletins, null);
|
||||||
|
|
||||||
|
public RuNkckiCursor WithPendingDocuments(IEnumerable<Guid> documents)
|
||||||
|
=> this with { PendingDocuments = (documents ?? Enumerable.Empty<Guid>()).Distinct().ToArray() };
|
||||||
|
|
||||||
|
public RuNkckiCursor WithPendingMappings(IEnumerable<Guid> mappings)
|
||||||
|
=> this with { PendingMappings = (mappings ?? Enumerable.Empty<Guid>()).Distinct().ToArray() };
|
||||||
|
|
||||||
|
public RuNkckiCursor WithKnownBulletins(IEnumerable<string> bulletins)
|
||||||
|
=> this with { KnownBulletins = (bulletins ?? Enumerable.Empty<string>()).Where(static id => !string.IsNullOrWhiteSpace(id)).Distinct(StringComparer.OrdinalIgnoreCase).ToArray() };
|
||||||
|
|
||||||
|
public RuNkckiCursor WithLastListingFetch(DateTimeOffset? timestamp)
|
||||||
|
=> this with { LastListingFetchAt = timestamp };
|
||||||
|
|
||||||
|
public BsonDocument ToBsonDocument()
|
||||||
|
{
|
||||||
|
var document = new BsonDocument
|
||||||
|
{
|
||||||
|
["pendingDocuments"] = new BsonArray(PendingDocuments.Select(id => id.ToString())),
|
||||||
|
["pendingMappings"] = new BsonArray(PendingMappings.Select(id => id.ToString())),
|
||||||
|
["knownBulletins"] = new BsonArray(KnownBulletins),
|
||||||
|
};
|
||||||
|
|
||||||
|
if (LastListingFetchAt.HasValue)
|
||||||
|
{
|
||||||
|
document["lastListingFetchAt"] = LastListingFetchAt.Value.UtcDateTime;
|
||||||
|
}
|
||||||
|
|
||||||
|
return document;
|
||||||
|
}
|
||||||
|
|
||||||
|
public static RuNkckiCursor FromBson(BsonDocument? document)
|
||||||
|
{
|
||||||
|
if (document is null || document.ElementCount == 0)
|
||||||
|
{
|
||||||
|
return Empty;
|
||||||
|
}
|
||||||
|
|
||||||
|
var pendingDocuments = ReadGuidArray(document, "pendingDocuments");
|
||||||
|
var pendingMappings = ReadGuidArray(document, "pendingMappings");
|
||||||
|
var knownBulletins = ReadStringArray(document, "knownBulletins");
|
||||||
|
var lastListingFetch = document.TryGetValue("lastListingFetchAt", out var dateValue)
|
||||||
|
? ParseDate(dateValue)
|
||||||
|
: null;
|
||||||
|
|
||||||
|
return new RuNkckiCursor(pendingDocuments, pendingMappings, knownBulletins, lastListingFetch);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IReadOnlyCollection<Guid> ReadGuidArray(BsonDocument document, string field)
|
||||||
|
{
|
||||||
|
if (!document.TryGetValue(field, out var value) || value is not BsonArray array)
|
||||||
|
{
|
||||||
|
return EmptyGuids;
|
||||||
|
}
|
||||||
|
|
||||||
|
var result = new List<Guid>(array.Count);
|
||||||
|
foreach (var element in array)
|
||||||
|
{
|
||||||
|
if (Guid.TryParse(element?.ToString(), out var guid))
|
||||||
|
{
|
||||||
|
result.Add(guid);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IReadOnlyCollection<string> ReadStringArray(BsonDocument document, string field)
|
||||||
|
{
|
||||||
|
if (!document.TryGetValue(field, out var value) || value is not BsonArray array)
|
||||||
|
{
|
||||||
|
return EmptyBulletins;
|
||||||
|
}
|
||||||
|
|
||||||
|
var result = new List<string>(array.Count);
|
||||||
|
foreach (var element in array)
|
||||||
|
{
|
||||||
|
var text = element?.ToString();
|
||||||
|
if (!string.IsNullOrWhiteSpace(text))
|
||||||
|
{
|
||||||
|
result.Add(text);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static DateTimeOffset? ParseDate(BsonValue value)
|
||||||
|
=> value.BsonType switch
|
||||||
|
{
|
||||||
|
BsonType.DateTime => DateTime.SpecifyKind(value.ToUniversalTime(), DateTimeKind.Utc),
|
||||||
|
BsonType.String when DateTimeOffset.TryParse(value.AsString, out var parsed) => parsed.ToUniversalTime(),
|
||||||
|
_ => null,
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -0,0 +1,169 @@
|
|||||||
|
using System.Collections.Immutable;
|
||||||
|
using System.Linq;
|
||||||
|
using System.Globalization;
|
||||||
|
using System.Text.Json;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Nkcki.Internal;
|
||||||
|
|
||||||
|
internal static class RuNkckiJsonParser
|
||||||
|
{
|
||||||
|
public static RuNkckiVulnerabilityDto Parse(JsonElement element)
|
||||||
|
{
|
||||||
|
var fstecId = element.TryGetProperty("vuln_id", out var vulnIdElement) && vulnIdElement.TryGetProperty("FSTEC", out var fstec) ? Normalize(fstec.GetString()) : null;
|
||||||
|
var mitreId = element.TryGetProperty("vuln_id", out vulnIdElement) && vulnIdElement.TryGetProperty("MITRE", out var mitre) ? Normalize(mitre.GetString()) : null;
|
||||||
|
|
||||||
|
var datePublished = ParseDate(element.TryGetProperty("date_published", out var published) ? published.GetString() : null);
|
||||||
|
var dateUpdated = ParseDate(element.TryGetProperty("date_updated", out var updated) ? updated.GetString() : null);
|
||||||
|
var cvssRating = Normalize(element.TryGetProperty("cvss_rating", out var rating) ? rating.GetString() : null);
|
||||||
|
bool? patchAvailable = element.TryGetProperty("patch_available", out var patch) ? patch.ValueKind switch
|
||||||
|
{
|
||||||
|
JsonValueKind.True => true,
|
||||||
|
JsonValueKind.False => false,
|
||||||
|
_ => null,
|
||||||
|
} : null;
|
||||||
|
|
||||||
|
var description = Normalize(element.TryGetProperty("description", out var desc) ? desc.GetString() : null);
|
||||||
|
var mitigation = Normalize(element.TryGetProperty("mitigation", out var mitigationElement) ? mitigationElement.GetString() : null);
|
||||||
|
var productCategory = Normalize(element.TryGetProperty("product_category", out var category) ? category.GetString() : null);
|
||||||
|
var impact = Normalize(element.TryGetProperty("impact", out var impactElement) ? impactElement.GetString() : null);
|
||||||
|
var method = Normalize(element.TryGetProperty("method_of_exploitation", out var methodElement) ? methodElement.GetString() : null);
|
||||||
|
|
||||||
|
bool? userInteraction = element.TryGetProperty("user_interaction", out var uiElement) ? uiElement.ValueKind switch
|
||||||
|
{
|
||||||
|
JsonValueKind.True => true,
|
||||||
|
JsonValueKind.False => false,
|
||||||
|
_ => null,
|
||||||
|
} : null;
|
||||||
|
|
||||||
|
string? softwareText = null;
|
||||||
|
bool? softwareHasCpe = null;
|
||||||
|
if (element.TryGetProperty("vulnerable_software", out var softwareElement))
|
||||||
|
{
|
||||||
|
if (softwareElement.TryGetProperty("software_text", out var textElement))
|
||||||
|
{
|
||||||
|
softwareText = Normalize(textElement.GetString()?.Replace('\r', ' '));
|
||||||
|
}
|
||||||
|
|
||||||
|
if (softwareElement.TryGetProperty("cpe", out var cpeElement))
|
||||||
|
{
|
||||||
|
softwareHasCpe = cpeElement.ValueKind switch
|
||||||
|
{
|
||||||
|
JsonValueKind.True => true,
|
||||||
|
JsonValueKind.False => false,
|
||||||
|
_ => null,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
RuNkckiCweDto? cweDto = null;
|
||||||
|
if (element.TryGetProperty("cwe", out var cweElement))
|
||||||
|
{
|
||||||
|
int? number = null;
|
||||||
|
if (cweElement.TryGetProperty("cwe_number", out var numberElement))
|
||||||
|
{
|
||||||
|
if (numberElement.ValueKind == JsonValueKind.Number && numberElement.TryGetInt32(out var parsed))
|
||||||
|
{
|
||||||
|
number = parsed;
|
||||||
|
}
|
||||||
|
else if (int.TryParse(numberElement.GetString(), NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsedInt))
|
||||||
|
{
|
||||||
|
number = parsedInt;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var cweDescription = Normalize(cweElement.TryGetProperty("cwe_description", out var descElement) ? descElement.GetString() : null);
|
||||||
|
if (number.HasValue || !string.IsNullOrWhiteSpace(cweDescription))
|
||||||
|
{
|
||||||
|
cweDto = new RuNkckiCweDto(number, cweDescription);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
double? cvssScore = element.TryGetProperty("cvss", out var cvssElement) && cvssElement.TryGetProperty("cvss_score", out var scoreElement)
|
||||||
|
? ParseDouble(scoreElement)
|
||||||
|
: null;
|
||||||
|
var cvssVector = element.TryGetProperty("cvss", out cvssElement) && cvssElement.TryGetProperty("cvss_vector", out var vectorElement)
|
||||||
|
? Normalize(vectorElement.GetString())
|
||||||
|
: null;
|
||||||
|
double? cvssScoreV4 = element.TryGetProperty("cvss", out cvssElement) && cvssElement.TryGetProperty("cvss_score_v4", out var scoreV4Element)
|
||||||
|
? ParseDouble(scoreV4Element)
|
||||||
|
: null;
|
||||||
|
var cvssVectorV4 = element.TryGetProperty("cvss", out cvssElement) && cvssElement.TryGetProperty("cvss_vector_v4", out var vectorV4Element)
|
||||||
|
? Normalize(vectorV4Element.GetString())
|
||||||
|
: null;
|
||||||
|
|
||||||
|
var urls = element.TryGetProperty("urls", out var urlsElement) && urlsElement.ValueKind == JsonValueKind.Array
|
||||||
|
? urlsElement.EnumerateArray()
|
||||||
|
.Select(static url => Normalize(url.GetString()))
|
||||||
|
.Where(static url => !string.IsNullOrWhiteSpace(url))
|
||||||
|
.Cast<string>()
|
||||||
|
.ToImmutableArray()
|
||||||
|
: ImmutableArray<string>.Empty;
|
||||||
|
|
||||||
|
return new RuNkckiVulnerabilityDto(
|
||||||
|
fstecId,
|
||||||
|
mitreId,
|
||||||
|
datePublished,
|
||||||
|
dateUpdated,
|
||||||
|
cvssRating,
|
||||||
|
patchAvailable,
|
||||||
|
description,
|
||||||
|
cweDto,
|
||||||
|
productCategory,
|
||||||
|
mitigation,
|
||||||
|
softwareText,
|
||||||
|
softwareHasCpe,
|
||||||
|
cvssScore,
|
||||||
|
cvssVector,
|
||||||
|
cvssScoreV4,
|
||||||
|
cvssVectorV4,
|
||||||
|
impact,
|
||||||
|
method,
|
||||||
|
userInteraction,
|
||||||
|
urls);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static double? ParseDouble(JsonElement element)
|
||||||
|
{
|
||||||
|
if (element.ValueKind == JsonValueKind.Number && element.TryGetDouble(out var value))
|
||||||
|
{
|
||||||
|
return value;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (element.ValueKind == JsonValueKind.String && double.TryParse(element.GetString(), NumberStyles.Any, CultureInfo.InvariantCulture, out var parsed))
|
||||||
|
{
|
||||||
|
return parsed;
|
||||||
|
}
|
||||||
|
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static DateTimeOffset? ParseDate(string? value)
|
||||||
|
{
|
||||||
|
if (string.IsNullOrWhiteSpace(value))
|
||||||
|
{
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (DateTimeOffset.TryParse(value, CultureInfo.InvariantCulture, DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, out var parsed))
|
||||||
|
{
|
||||||
|
return parsed;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (DateTimeOffset.TryParse(value, CultureInfo.GetCultureInfo("ru-RU"), DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, out var ruParsed))
|
||||||
|
{
|
||||||
|
return ruParsed;
|
||||||
|
}
|
||||||
|
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string? Normalize(string? value)
|
||||||
|
{
|
||||||
|
if (string.IsNullOrWhiteSpace(value))
|
||||||
|
{
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
return value.Replace('\r', ' ').Replace('\n', ' ').Trim();
|
||||||
|
}
|
||||||
|
}
|
||||||
298
src/StellaOps.Feedser.Source.Ru.Nkcki/Internal/RuNkckiMapper.cs
Normal file
298
src/StellaOps.Feedser.Source.Ru.Nkcki/Internal/RuNkckiMapper.cs
Normal file
@@ -0,0 +1,298 @@
|
|||||||
|
using System.Collections.Generic;
|
||||||
|
using System.Collections.Immutable;
|
||||||
|
using System.Globalization;
|
||||||
|
using System.Linq;
|
||||||
|
using StellaOps.Feedser.Models;
|
||||||
|
using StellaOps.Feedser.Normalization.Cvss;
|
||||||
|
using StellaOps.Feedser.Storage.Mongo.Documents;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Nkcki.Internal;
|
||||||
|
|
||||||
|
internal static class RuNkckiMapper
|
||||||
|
{
|
||||||
|
private static readonly ImmutableDictionary<string, string> SeverityLookup = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase)
|
||||||
|
{
|
||||||
|
["критический"] = "critical",
|
||||||
|
["высокий"] = "high",
|
||||||
|
["средний"] = "medium",
|
||||||
|
["умеренный"] = "medium",
|
||||||
|
["низкий"] = "low",
|
||||||
|
["информационный"] = "informational",
|
||||||
|
}.ToImmutableDictionary(StringComparer.OrdinalIgnoreCase);
|
||||||
|
|
||||||
|
public static Advisory Map(RuNkckiVulnerabilityDto dto, DocumentRecord document, DateTimeOffset recordedAt)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(dto);
|
||||||
|
ArgumentNullException.ThrowIfNull(document);
|
||||||
|
|
||||||
|
var advisoryProvenance = new AdvisoryProvenance(
|
||||||
|
RuNkckiConnectorPlugin.SourceName,
|
||||||
|
"advisory",
|
||||||
|
dto.AdvisoryKey,
|
||||||
|
recordedAt,
|
||||||
|
new[] { ProvenanceFieldMasks.Advisory });
|
||||||
|
|
||||||
|
var aliases = BuildAliases(dto);
|
||||||
|
var references = BuildReferences(dto, document, recordedAt);
|
||||||
|
var packages = BuildPackages(dto, recordedAt);
|
||||||
|
var cvssMetrics = BuildCvssMetrics(dto, recordedAt, out var severityFromCvss);
|
||||||
|
var severityFromRating = NormalizeSeverity(dto.CvssRating);
|
||||||
|
var severity = severityFromRating ?? severityFromCvss;
|
||||||
|
|
||||||
|
if (severityFromRating is not null && severityFromCvss is not null)
|
||||||
|
{
|
||||||
|
severity = ChooseMoreSevere(severityFromRating, severityFromCvss);
|
||||||
|
}
|
||||||
|
|
||||||
|
var exploitKnown = DetermineExploitKnown(dto);
|
||||||
|
|
||||||
|
return new Advisory(
|
||||||
|
advisoryKey: dto.AdvisoryKey,
|
||||||
|
title: dto.Description ?? dto.AdvisoryKey,
|
||||||
|
summary: dto.Description,
|
||||||
|
language: "ru",
|
||||||
|
published: dto.DatePublished,
|
||||||
|
modified: dto.DateUpdated,
|
||||||
|
severity: severity,
|
||||||
|
exploitKnown: exploitKnown,
|
||||||
|
aliases: aliases,
|
||||||
|
references: references,
|
||||||
|
affectedPackages: packages,
|
||||||
|
cvssMetrics: cvssMetrics,
|
||||||
|
provenance: new[] { advisoryProvenance });
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IReadOnlyList<string> BuildAliases(RuNkckiVulnerabilityDto dto)
|
||||||
|
{
|
||||||
|
var aliases = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||||
|
if (!string.IsNullOrWhiteSpace(dto.FstecId))
|
||||||
|
{
|
||||||
|
aliases.Add(dto.FstecId!);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!string.IsNullOrWhiteSpace(dto.MitreId))
|
||||||
|
{
|
||||||
|
aliases.Add(dto.MitreId!);
|
||||||
|
}
|
||||||
|
|
||||||
|
return aliases.ToImmutableSortedSet(StringComparer.Ordinal).ToImmutableArray();
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IReadOnlyList<AdvisoryReference> BuildReferences(RuNkckiVulnerabilityDto dto, DocumentRecord document, DateTimeOffset recordedAt)
|
||||||
|
{
|
||||||
|
var references = new List<AdvisoryReference>
|
||||||
|
{
|
||||||
|
new(document.Uri, "details", "ru-nkcki", summary: null, new AdvisoryProvenance(
|
||||||
|
RuNkckiConnectorPlugin.SourceName,
|
||||||
|
"reference",
|
||||||
|
document.Uri,
|
||||||
|
recordedAt,
|
||||||
|
new[] { ProvenanceFieldMasks.References }))
|
||||||
|
};
|
||||||
|
|
||||||
|
if (!string.IsNullOrWhiteSpace(dto.FstecId))
|
||||||
|
{
|
||||||
|
var slug = dto.FstecId!.Contains(':', StringComparison.Ordinal)
|
||||||
|
? dto.FstecId[(dto.FstecId.IndexOf(':') + 1)..]
|
||||||
|
: dto.FstecId;
|
||||||
|
var bduUrl = $"https://bdu.fstec.ru/vul/{slug}";
|
||||||
|
references.Add(new AdvisoryReference(bduUrl, "details", "bdu", summary: null, new AdvisoryProvenance(
|
||||||
|
RuNkckiConnectorPlugin.SourceName,
|
||||||
|
"reference",
|
||||||
|
bduUrl,
|
||||||
|
recordedAt,
|
||||||
|
new[] { ProvenanceFieldMasks.References })));
|
||||||
|
}
|
||||||
|
|
||||||
|
foreach (var url in dto.Urls)
|
||||||
|
{
|
||||||
|
if (string.IsNullOrWhiteSpace(url))
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
var kind = url.Contains("cert.gov.ru", StringComparison.OrdinalIgnoreCase) ? "details" : "external";
|
||||||
|
var sourceTag = url.Contains("siemens", StringComparison.OrdinalIgnoreCase) ? "vendor" : null;
|
||||||
|
references.Add(new AdvisoryReference(url, kind, sourceTag, summary: null, new AdvisoryProvenance(
|
||||||
|
RuNkckiConnectorPlugin.SourceName,
|
||||||
|
"reference",
|
||||||
|
url,
|
||||||
|
recordedAt,
|
||||||
|
new[] { ProvenanceFieldMasks.References })));
|
||||||
|
}
|
||||||
|
|
||||||
|
if (dto.Cwe?.Number is int number)
|
||||||
|
{
|
||||||
|
var url = $"https://cwe.mitre.org/data/definitions/{number}.html";
|
||||||
|
references.Add(new AdvisoryReference(url, "cwe", "cwe", dto.Cwe.Description, new AdvisoryProvenance(
|
||||||
|
RuNkckiConnectorPlugin.SourceName,
|
||||||
|
"reference",
|
||||||
|
url,
|
||||||
|
recordedAt,
|
||||||
|
new[] { ProvenanceFieldMasks.References })));
|
||||||
|
}
|
||||||
|
|
||||||
|
return references;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IReadOnlyList<AffectedPackage> BuildPackages(RuNkckiVulnerabilityDto dto, DateTimeOffset recordedAt)
|
||||||
|
{
|
||||||
|
if (string.IsNullOrWhiteSpace(dto.VulnerableSoftwareText))
|
||||||
|
{
|
||||||
|
return Array.Empty<AffectedPackage>();
|
||||||
|
}
|
||||||
|
|
||||||
|
var identifier = dto.VulnerableSoftwareText!.Replace('\n', ' ').Replace('\r', ' ').Trim();
|
||||||
|
if (identifier.Length == 0)
|
||||||
|
{
|
||||||
|
return Array.Empty<AffectedPackage>();
|
||||||
|
}
|
||||||
|
|
||||||
|
var packageProvenance = new AdvisoryProvenance(
|
||||||
|
RuNkckiConnectorPlugin.SourceName,
|
||||||
|
"package",
|
||||||
|
identifier,
|
||||||
|
recordedAt,
|
||||||
|
new[] { ProvenanceFieldMasks.AffectedPackages });
|
||||||
|
|
||||||
|
var status = new AffectedPackageStatus(
|
||||||
|
dto.PatchAvailable == true ? AffectedPackageStatusCatalog.Fixed : AffectedPackageStatusCatalog.Affected,
|
||||||
|
new AdvisoryProvenance(
|
||||||
|
RuNkckiConnectorPlugin.SourceName,
|
||||||
|
"package-status",
|
||||||
|
dto.PatchAvailable == true ? "patch_available" : "affected",
|
||||||
|
recordedAt,
|
||||||
|
new[] { ProvenanceFieldMasks.PackageStatuses }));
|
||||||
|
|
||||||
|
return new[]
|
||||||
|
{
|
||||||
|
new AffectedPackage(
|
||||||
|
dto.VulnerableSoftwareHasCpe == true ? AffectedPackageTypes.Cpe : AffectedPackageTypes.Vendor,
|
||||||
|
identifier,
|
||||||
|
platform: null,
|
||||||
|
versionRanges: null,
|
||||||
|
statuses: new[] { status },
|
||||||
|
provenance: new[] { packageProvenance })
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
private static IReadOnlyList<CvssMetric> BuildCvssMetrics(RuNkckiVulnerabilityDto dto, DateTimeOffset recordedAt, out string? severity)
|
||||||
|
{
|
||||||
|
severity = null;
|
||||||
|
var metrics = new List<CvssMetric>();
|
||||||
|
|
||||||
|
if (!string.IsNullOrWhiteSpace(dto.CvssVector) && CvssMetricNormalizer.TryNormalize(null, dto.CvssVector, dto.CvssScore, null, out var normalized))
|
||||||
|
{
|
||||||
|
var provenance = new AdvisoryProvenance(
|
||||||
|
RuNkckiConnectorPlugin.SourceName,
|
||||||
|
"cvss",
|
||||||
|
normalized.Vector,
|
||||||
|
recordedAt,
|
||||||
|
new[] { ProvenanceFieldMasks.CvssMetrics });
|
||||||
|
var metric = normalized.ToModel(provenance);
|
||||||
|
metrics.Add(metric);
|
||||||
|
severity ??= metric.BaseSeverity;
|
||||||
|
}
|
||||||
|
|
||||||
|
return metrics;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string? NormalizeSeverity(string? rating)
|
||||||
|
{
|
||||||
|
if (string.IsNullOrWhiteSpace(rating))
|
||||||
|
{
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
var normalized = rating.Trim().ToLowerInvariant();
|
||||||
|
|
||||||
|
if (SeverityLookup.TryGetValue(normalized, out var mapped))
|
||||||
|
{
|
||||||
|
return mapped;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (normalized.StartsWith("крит", StringComparison.Ordinal))
|
||||||
|
{
|
||||||
|
return "critical";
|
||||||
|
}
|
||||||
|
|
||||||
|
if (normalized.StartsWith("высок", StringComparison.Ordinal))
|
||||||
|
{
|
||||||
|
return "high";
|
||||||
|
}
|
||||||
|
|
||||||
|
if (normalized.StartsWith("сред", StringComparison.Ordinal) || normalized.StartsWith("умер", StringComparison.Ordinal))
|
||||||
|
{
|
||||||
|
return "medium";
|
||||||
|
}
|
||||||
|
|
||||||
|
if (normalized.StartsWith("низк", StringComparison.Ordinal))
|
||||||
|
{
|
||||||
|
return "low";
|
||||||
|
}
|
||||||
|
|
||||||
|
if (normalized.StartsWith("информ", StringComparison.Ordinal))
|
||||||
|
{
|
||||||
|
return "informational";
|
||||||
|
}
|
||||||
|
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string ChooseMoreSevere(string first, string second)
|
||||||
|
{
|
||||||
|
var order = new[] { "critical", "high", "medium", "low", "informational" };
|
||||||
|
|
||||||
|
static int IndexOf(ReadOnlySpan<string> levels, string value)
|
||||||
|
{
|
||||||
|
for (var i = 0; i < levels.Length; i++)
|
||||||
|
{
|
||||||
|
if (string.Equals(levels[i], value, StringComparison.OrdinalIgnoreCase))
|
||||||
|
{
|
||||||
|
return i;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
|
||||||
|
var firstIndex = IndexOf(order.AsSpan(), first);
|
||||||
|
var secondIndex = IndexOf(order.AsSpan(), second);
|
||||||
|
|
||||||
|
if (firstIndex == -1 && secondIndex == -1)
|
||||||
|
{
|
||||||
|
return first;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (firstIndex == -1)
|
||||||
|
{
|
||||||
|
return second;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (secondIndex == -1)
|
||||||
|
{
|
||||||
|
return first;
|
||||||
|
}
|
||||||
|
|
||||||
|
return firstIndex <= secondIndex ? first : second;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static bool DetermineExploitKnown(RuNkckiVulnerabilityDto dto)
|
||||||
|
{
|
||||||
|
if (!string.IsNullOrWhiteSpace(dto.MethodOfExploitation))
|
||||||
|
{
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!string.IsNullOrWhiteSpace(dto.Impact))
|
||||||
|
{
|
||||||
|
var impact = dto.Impact.Trim().ToUpperInvariant();
|
||||||
|
if (impact is "ACE" or "RCE" or "LPE")
|
||||||
|
{
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
using System.Collections.Immutable;
|
||||||
|
using System.Text.Json.Serialization;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Nkcki.Internal;
|
||||||
|
|
||||||
|
internal sealed record RuNkckiVulnerabilityDto(
|
||||||
|
string? FstecId,
|
||||||
|
string? MitreId,
|
||||||
|
DateTimeOffset? DatePublished,
|
||||||
|
DateTimeOffset? DateUpdated,
|
||||||
|
string? CvssRating,
|
||||||
|
bool? PatchAvailable,
|
||||||
|
string? Description,
|
||||||
|
RuNkckiCweDto? Cwe,
|
||||||
|
string? ProductCategory,
|
||||||
|
string? Mitigation,
|
||||||
|
string? VulnerableSoftwareText,
|
||||||
|
bool? VulnerableSoftwareHasCpe,
|
||||||
|
double? CvssScore,
|
||||||
|
string? CvssVector,
|
||||||
|
double? CvssScoreV4,
|
||||||
|
string? CvssVectorV4,
|
||||||
|
string? Impact,
|
||||||
|
string? MethodOfExploitation,
|
||||||
|
bool? UserInteraction,
|
||||||
|
ImmutableArray<string> Urls)
|
||||||
|
{
|
||||||
|
[JsonIgnore]
|
||||||
|
public string AdvisoryKey => !string.IsNullOrWhiteSpace(FstecId)
|
||||||
|
? FstecId!
|
||||||
|
: !string.IsNullOrWhiteSpace(MitreId)
|
||||||
|
? MitreId!
|
||||||
|
: Guid.NewGuid().ToString();
|
||||||
|
}
|
||||||
|
|
||||||
|
internal sealed record RuNkckiCweDto(int? Number, string? Description);
|
||||||
43
src/StellaOps.Feedser.Source.Ru.Nkcki/Jobs.cs
Normal file
43
src/StellaOps.Feedser.Source.Ru.Nkcki/Jobs.cs
Normal file
@@ -0,0 +1,43 @@
|
|||||||
|
using StellaOps.Feedser.Core.Jobs;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Nkcki;
|
||||||
|
|
||||||
|
internal static class RuNkckiJobKinds
|
||||||
|
{
|
||||||
|
public const string Fetch = "source:ru-nkcki:fetch";
|
||||||
|
public const string Parse = "source:ru-nkcki:parse";
|
||||||
|
public const string Map = "source:ru-nkcki:map";
|
||||||
|
}
|
||||||
|
|
||||||
|
internal sealed class RuNkckiFetchJob : IJob
|
||||||
|
{
|
||||||
|
private readonly RuNkckiConnector _connector;
|
||||||
|
|
||||||
|
public RuNkckiFetchJob(RuNkckiConnector connector)
|
||||||
|
=> _connector = connector ?? throw new ArgumentNullException(nameof(connector));
|
||||||
|
|
||||||
|
public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken)
|
||||||
|
=> _connector.FetchAsync(context.Services, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
internal sealed class RuNkckiParseJob : IJob
|
||||||
|
{
|
||||||
|
private readonly RuNkckiConnector _connector;
|
||||||
|
|
||||||
|
public RuNkckiParseJob(RuNkckiConnector connector)
|
||||||
|
=> _connector = connector ?? throw new ArgumentNullException(nameof(connector));
|
||||||
|
|
||||||
|
public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken)
|
||||||
|
=> _connector.ParseAsync(context.Services, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
internal sealed class RuNkckiMapJob : IJob
|
||||||
|
{
|
||||||
|
private readonly RuNkckiConnector _connector;
|
||||||
|
|
||||||
|
public RuNkckiMapJob(RuNkckiConnector connector)
|
||||||
|
=> _connector = connector ?? throw new ArgumentNullException(nameof(connector));
|
||||||
|
|
||||||
|
public Task ExecuteAsync(JobExecutionContext context, CancellationToken cancellationToken)
|
||||||
|
=> _connector.MapAsync(context.Services, cancellationToken);
|
||||||
|
}
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
using System.Runtime.CompilerServices;
|
||||||
|
|
||||||
|
[assembly: InternalsVisibleTo("StellaOps.Feedser.Source.Ru.Nkcki.Tests")]
|
||||||
825
src/StellaOps.Feedser.Source.Ru.Nkcki/RuNkckiConnector.cs
Normal file
825
src/StellaOps.Feedser.Source.Ru.Nkcki/RuNkckiConnector.cs
Normal file
@@ -0,0 +1,825 @@
|
|||||||
|
using System.Collections.Immutable;
|
||||||
|
using System.Collections.Generic;
|
||||||
|
using System.IO;
|
||||||
|
using System.IO.Compression;
|
||||||
|
using System.Net;
|
||||||
|
using System.Linq;
|
||||||
|
using System.Security.Cryptography;
|
||||||
|
using System.Text;
|
||||||
|
using System.Text.Json;
|
||||||
|
using System.Text.Json.Serialization;
|
||||||
|
using AngleSharp.Html.Parser;
|
||||||
|
using Microsoft.Extensions.Logging;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using MongoDB.Bson;
|
||||||
|
using StellaOps.Feedser.Source.Common;
|
||||||
|
using StellaOps.Feedser.Source.Common.Fetch;
|
||||||
|
using StellaOps.Feedser.Source.Ru.Nkcki.Configuration;
|
||||||
|
using StellaOps.Feedser.Source.Ru.Nkcki.Internal;
|
||||||
|
using StellaOps.Feedser.Storage.Mongo;
|
||||||
|
using StellaOps.Feedser.Storage.Mongo.Advisories;
|
||||||
|
using StellaOps.Feedser.Storage.Mongo.Documents;
|
||||||
|
using StellaOps.Feedser.Storage.Mongo.Dtos;
|
||||||
|
using StellaOps.Plugin;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Nkcki;
|
||||||
|
|
||||||
|
public sealed class RuNkckiConnector : IFeedConnector
|
||||||
|
{
|
||||||
|
private static readonly JsonSerializerOptions SerializerOptions = new(JsonSerializerDefaults.Web)
|
||||||
|
{
|
||||||
|
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
|
||||||
|
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
|
||||||
|
WriteIndented = false,
|
||||||
|
};
|
||||||
|
|
||||||
|
private static readonly string[] ListingAcceptHeaders =
|
||||||
|
{
|
||||||
|
"text/html",
|
||||||
|
"application/xhtml+xml;q=0.9",
|
||||||
|
"text/plain;q=0.1",
|
||||||
|
};
|
||||||
|
|
||||||
|
private static readonly string[] BulletinAcceptHeaders =
|
||||||
|
{
|
||||||
|
"application/zip",
|
||||||
|
"application/octet-stream",
|
||||||
|
"application/x-zip-compressed",
|
||||||
|
};
|
||||||
|
|
||||||
|
private readonly SourceFetchService _fetchService;
|
||||||
|
private readonly RawDocumentStorage _rawDocumentStorage;
|
||||||
|
private readonly IDocumentStore _documentStore;
|
||||||
|
private readonly IDtoStore _dtoStore;
|
||||||
|
private readonly IAdvisoryStore _advisoryStore;
|
||||||
|
private readonly ISourceStateRepository _stateRepository;
|
||||||
|
private readonly RuNkckiOptions _options;
|
||||||
|
private readonly TimeProvider _timeProvider;
|
||||||
|
private readonly ILogger<RuNkckiConnector> _logger;
|
||||||
|
private readonly string _cacheDirectory;
|
||||||
|
|
||||||
|
private readonly HtmlParser _htmlParser = new();
|
||||||
|
|
||||||
|
public RuNkckiConnector(
|
||||||
|
SourceFetchService fetchService,
|
||||||
|
RawDocumentStorage rawDocumentStorage,
|
||||||
|
IDocumentStore documentStore,
|
||||||
|
IDtoStore dtoStore,
|
||||||
|
IAdvisoryStore advisoryStore,
|
||||||
|
ISourceStateRepository stateRepository,
|
||||||
|
IOptions<RuNkckiOptions> options,
|
||||||
|
TimeProvider? timeProvider,
|
||||||
|
ILogger<RuNkckiConnector> logger)
|
||||||
|
{
|
||||||
|
_fetchService = fetchService ?? throw new ArgumentNullException(nameof(fetchService));
|
||||||
|
_rawDocumentStorage = rawDocumentStorage ?? throw new ArgumentNullException(nameof(rawDocumentStorage));
|
||||||
|
_documentStore = documentStore ?? throw new ArgumentNullException(nameof(documentStore));
|
||||||
|
_dtoStore = dtoStore ?? throw new ArgumentNullException(nameof(dtoStore));
|
||||||
|
_advisoryStore = advisoryStore ?? throw new ArgumentNullException(nameof(advisoryStore));
|
||||||
|
_stateRepository = stateRepository ?? throw new ArgumentNullException(nameof(stateRepository));
|
||||||
|
_options = (options ?? throw new ArgumentNullException(nameof(options))).Value ?? throw new ArgumentNullException(nameof(options));
|
||||||
|
_options.Validate();
|
||||||
|
_timeProvider = timeProvider ?? TimeProvider.System;
|
||||||
|
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||||
|
_cacheDirectory = ResolveCacheDirectory(_options.CacheDirectory);
|
||||||
|
EnsureCacheDirectory();
|
||||||
|
}
|
||||||
|
|
||||||
|
public string SourceName => RuNkckiConnectorPlugin.SourceName;
|
||||||
|
|
||||||
|
public async Task FetchAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
var pendingDocuments = cursor.PendingDocuments.ToHashSet();
|
||||||
|
var pendingMappings = cursor.PendingMappings.ToHashSet();
|
||||||
|
var knownBulletins = cursor.KnownBulletins.ToHashSet(StringComparer.OrdinalIgnoreCase);
|
||||||
|
var now = _timeProvider.GetUtcNow();
|
||||||
|
var processed = 0;
|
||||||
|
|
||||||
|
IReadOnlyList<BulletinAttachment> attachments = Array.Empty<BulletinAttachment>();
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var listingResult = await FetchListingAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
if (!listingResult.IsSuccess || listingResult.Content is null)
|
||||||
|
{
|
||||||
|
_logger.LogWarning("NKCKI listing fetch returned no content (status={Status})", listingResult.StatusCode);
|
||||||
|
processed = await ProcessCachedBulletinsAsync(pendingDocuments, pendingMappings, knownBulletins, now, processed, cancellationToken).ConfigureAwait(false);
|
||||||
|
await UpdateCursorAsync(cursor
|
||||||
|
.WithPendingDocuments(pendingDocuments)
|
||||||
|
.WithPendingMappings(pendingMappings)
|
||||||
|
.WithKnownBulletins(NormalizeBulletins(knownBulletins))
|
||||||
|
.WithLastListingFetch(now), cancellationToken).ConfigureAwait(false);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
attachments = await ParseListingAsync(listingResult.Content, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException)
|
||||||
|
{
|
||||||
|
_logger.LogWarning(ex, "NKCKI listing fetch failed; attempting cached bulletins");
|
||||||
|
processed = await ProcessCachedBulletinsAsync(pendingDocuments, pendingMappings, knownBulletins, now, processed, cancellationToken).ConfigureAwait(false);
|
||||||
|
await UpdateCursorAsync(cursor
|
||||||
|
.WithPendingDocuments(pendingDocuments)
|
||||||
|
.WithPendingMappings(pendingMappings)
|
||||||
|
.WithKnownBulletins(NormalizeBulletins(knownBulletins))
|
||||||
|
.WithLastListingFetch(cursor.LastListingFetchAt ?? now), cancellationToken).ConfigureAwait(false);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (attachments.Count == 0)
|
||||||
|
{
|
||||||
|
_logger.LogDebug("NKCKI listing contained no bulletin attachments");
|
||||||
|
processed = await ProcessCachedBulletinsAsync(pendingDocuments, pendingMappings, knownBulletins, now, processed, cancellationToken).ConfigureAwait(false);
|
||||||
|
await UpdateCursorAsync(cursor
|
||||||
|
.WithPendingDocuments(pendingDocuments)
|
||||||
|
.WithPendingMappings(pendingMappings)
|
||||||
|
.WithKnownBulletins(NormalizeBulletins(knownBulletins))
|
||||||
|
.WithLastListingFetch(now), cancellationToken).ConfigureAwait(false);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
var newAttachments = attachments
|
||||||
|
.Where(attachment => !knownBulletins.Contains(attachment.Id))
|
||||||
|
.Take(_options.MaxBulletinsPerFetch)
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
if (newAttachments.Count == 0)
|
||||||
|
{
|
||||||
|
await UpdateCursorAsync(cursor
|
||||||
|
.WithPendingDocuments(pendingDocuments)
|
||||||
|
.WithPendingMappings(pendingMappings)
|
||||||
|
.WithKnownBulletins(NormalizeBulletins(knownBulletins))
|
||||||
|
.WithLastListingFetch(now), cancellationToken).ConfigureAwait(false);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
foreach (var attachment in newAttachments)
|
||||||
|
{
|
||||||
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var request = new SourceFetchRequest(RuNkckiOptions.HttpClientName, SourceName, attachment.Uri)
|
||||||
|
{
|
||||||
|
AcceptHeaders = BulletinAcceptHeaders,
|
||||||
|
TimeoutOverride = _options.RequestTimeout,
|
||||||
|
};
|
||||||
|
|
||||||
|
var attachmentResult = await _fetchService.FetchContentAsync(request, cancellationToken).ConfigureAwait(false);
|
||||||
|
if (!attachmentResult.IsSuccess || attachmentResult.Content is null)
|
||||||
|
{
|
||||||
|
if (TryReadCachedBulletin(attachment.Id, out var cachedBytes))
|
||||||
|
{
|
||||||
|
_logger.LogWarning("NKCKI bulletin {BulletinId} unavailable (status={Status}); using cached artefact", attachment.Id, attachmentResult.StatusCode);
|
||||||
|
processed = await ProcessBulletinEntriesAsync(cachedBytes, attachment.Id, pendingDocuments, pendingMappings, now, processed, cancellationToken).ConfigureAwait(false);
|
||||||
|
knownBulletins.Add(attachment.Id);
|
||||||
|
}
|
||||||
|
else
|
||||||
|
{
|
||||||
|
_logger.LogWarning("NKCKI bulletin {BulletinId} returned no content (status={Status})", attachment.Id, attachmentResult.StatusCode);
|
||||||
|
}
|
||||||
|
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
TryWriteCachedBulletin(attachment.Id, attachmentResult.Content);
|
||||||
|
processed = await ProcessBulletinEntriesAsync(attachmentResult.Content, attachment.Id, pendingDocuments, pendingMappings, now, processed, cancellationToken).ConfigureAwait(false);
|
||||||
|
knownBulletins.Add(attachment.Id);
|
||||||
|
}
|
||||||
|
catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException)
|
||||||
|
{
|
||||||
|
if (TryReadCachedBulletin(attachment.Id, out var cachedBytes))
|
||||||
|
{
|
||||||
|
_logger.LogWarning(ex, "NKCKI bulletin fetch failed for {BulletinId}; using cached artefact", attachment.Id);
|
||||||
|
processed = await ProcessBulletinEntriesAsync(cachedBytes, attachment.Id, pendingDocuments, pendingMappings, now, processed, cancellationToken).ConfigureAwait(false);
|
||||||
|
knownBulletins.Add(attachment.Id);
|
||||||
|
}
|
||||||
|
else
|
||||||
|
{
|
||||||
|
_logger.LogWarning(ex, "NKCKI bulletin fetch failed for {BulletinId}", attachment.Id);
|
||||||
|
await _stateRepository.MarkFailureAsync(SourceName, now, _options.FailureBackoff, ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
|
throw;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (processed >= _options.MaxVulnerabilitiesPerFetch)
|
||||||
|
{
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (_options.RequestDelay > TimeSpan.Zero)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await Task.Delay(_options.RequestDelay, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
catch (TaskCanceledException)
|
||||||
|
{
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var normalizedBulletins = NormalizeBulletins(knownBulletins);
|
||||||
|
|
||||||
|
var updatedCursor = cursor
|
||||||
|
.WithPendingDocuments(pendingDocuments)
|
||||||
|
.WithPendingMappings(pendingMappings)
|
||||||
|
.WithKnownBulletins(normalizedBulletins)
|
||||||
|
.WithLastListingFetch(now);
|
||||||
|
|
||||||
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task ParseAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
if (cursor.PendingDocuments.Count == 0)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
var pendingDocuments = cursor.PendingDocuments.ToList();
|
||||||
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
|
foreach (var documentId in cursor.PendingDocuments)
|
||||||
|
{
|
||||||
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
|
if (document is null)
|
||||||
|
{
|
||||||
|
pendingDocuments.Remove(documentId);
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!document.GridFsId.HasValue)
|
||||||
|
{
|
||||||
|
_logger.LogWarning("NKCKI document {DocumentId} missing GridFS payload", documentId);
|
||||||
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
|
pendingDocuments.Remove(documentId);
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
byte[] payload;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
payload = await _rawDocumentStorage.DownloadAsync(document.GridFsId.Value, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogError(ex, "NKCKI unable to download raw document {DocumentId}", documentId);
|
||||||
|
throw;
|
||||||
|
}
|
||||||
|
|
||||||
|
RuNkckiVulnerabilityDto? dto;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
dto = JsonSerializer.Deserialize<RuNkckiVulnerabilityDto>(payload, SerializerOptions);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogWarning(ex, "NKCKI failed to deserialize document {DocumentId}", documentId);
|
||||||
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
|
pendingDocuments.Remove(documentId);
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (dto is null)
|
||||||
|
{
|
||||||
|
_logger.LogWarning("NKCKI document {DocumentId} produced null DTO", documentId);
|
||||||
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
|
pendingDocuments.Remove(documentId);
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
var bson = MongoDB.Bson.BsonDocument.Parse(JsonSerializer.Serialize(dto, SerializerOptions));
|
||||||
|
var dtoRecord = new DtoRecord(Guid.NewGuid(), document.Id, SourceName, "ru-nkcki.v1", bson, _timeProvider.GetUtcNow());
|
||||||
|
await _dtoStore.UpsertAsync(dtoRecord, cancellationToken).ConfigureAwait(false);
|
||||||
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.PendingMap, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
|
pendingDocuments.Remove(documentId);
|
||||||
|
if (!pendingMappings.Contains(documentId))
|
||||||
|
{
|
||||||
|
pendingMappings.Add(documentId);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var updatedCursor = cursor
|
||||||
|
.WithPendingDocuments(pendingDocuments)
|
||||||
|
.WithPendingMappings(pendingMappings);
|
||||||
|
|
||||||
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
public async Task MapAsync(IServiceProvider services, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
|
||||||
|
var cursor = await GetCursorAsync(cancellationToken).ConfigureAwait(false);
|
||||||
|
if (cursor.PendingMappings.Count == 0)
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
var pendingMappings = cursor.PendingMappings.ToList();
|
||||||
|
|
||||||
|
foreach (var documentId in cursor.PendingMappings)
|
||||||
|
{
|
||||||
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
|
var document = await _documentStore.FindAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
|
if (document is null)
|
||||||
|
{
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
var dtoRecord = await _dtoStore.FindByDocumentIdAsync(documentId, cancellationToken).ConfigureAwait(false);
|
||||||
|
if (dtoRecord is null)
|
||||||
|
{
|
||||||
|
_logger.LogWarning("NKCKI document {DocumentId} missing DTO payload", documentId);
|
||||||
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
RuNkckiVulnerabilityDto dto;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
dto = JsonSerializer.Deserialize<RuNkckiVulnerabilityDto>(dtoRecord.Payload.ToString(), SerializerOptions) ?? throw new InvalidOperationException("DTO deserialized to null");
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogError(ex, "NKCKI failed to deserialize DTO for document {DocumentId}", documentId);
|
||||||
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var advisory = RuNkckiMapper.Map(dto, document, dtoRecord.ValidatedAt);
|
||||||
|
await _advisoryStore.UpsertAsync(advisory, cancellationToken).ConfigureAwait(false);
|
||||||
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Mapped, cancellationToken).ConfigureAwait(false);
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogError(ex, "NKCKI mapping failed for document {DocumentId}", documentId);
|
||||||
|
await _documentStore.UpdateStatusAsync(document.Id, DocumentStatuses.Failed, cancellationToken).ConfigureAwait(false);
|
||||||
|
pendingMappings.Remove(documentId);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var updatedCursor = cursor.WithPendingMappings(pendingMappings);
|
||||||
|
await UpdateCursorAsync(updatedCursor, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<int> ProcessCachedBulletinsAsync(
|
||||||
|
HashSet<Guid> pendingDocuments,
|
||||||
|
HashSet<Guid> pendingMappings,
|
||||||
|
HashSet<string> knownBulletins,
|
||||||
|
DateTimeOffset now,
|
||||||
|
int processed,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
if (!Directory.Exists(_cacheDirectory))
|
||||||
|
{
|
||||||
|
return processed;
|
||||||
|
}
|
||||||
|
|
||||||
|
var updated = processed;
|
||||||
|
var cacheFiles = Directory
|
||||||
|
.EnumerateFiles(_cacheDirectory, "*.json.zip", SearchOption.TopDirectoryOnly)
|
||||||
|
.OrderBy(static path => path, StringComparer.OrdinalIgnoreCase)
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
foreach (var filePath in cacheFiles)
|
||||||
|
{
|
||||||
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
|
var bulletinId = ExtractBulletinIdFromCachePath(filePath);
|
||||||
|
if (string.IsNullOrWhiteSpace(bulletinId) || knownBulletins.Contains(bulletinId))
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
byte[] content;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
content = File.ReadAllBytes(filePath);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogDebug(ex, "NKCKI failed to read cached bulletin at {CachePath}", filePath);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
updated = await ProcessBulletinEntriesAsync(content, bulletinId, pendingDocuments, pendingMappings, now, updated, cancellationToken).ConfigureAwait(false);
|
||||||
|
knownBulletins.Add(bulletinId);
|
||||||
|
|
||||||
|
if (updated >= _options.MaxVulnerabilitiesPerFetch)
|
||||||
|
{
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return updated;
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<int> ProcessBulletinEntriesAsync(
|
||||||
|
byte[] content,
|
||||||
|
string bulletinId,
|
||||||
|
HashSet<Guid> pendingDocuments,
|
||||||
|
HashSet<Guid> pendingMappings,
|
||||||
|
DateTimeOffset now,
|
||||||
|
int processed,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
if (content.Length == 0)
|
||||||
|
{
|
||||||
|
return processed;
|
||||||
|
}
|
||||||
|
|
||||||
|
var updated = processed;
|
||||||
|
using var archiveStream = new MemoryStream(content, writable: false);
|
||||||
|
using var archive = new ZipArchive(archiveStream, ZipArchiveMode.Read, leaveOpen: false);
|
||||||
|
|
||||||
|
foreach (var entry in archive.Entries.OrderBy(static e => e.FullName, StringComparer.OrdinalIgnoreCase))
|
||||||
|
{
|
||||||
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
|
if (!entry.FullName.EndsWith(".json", StringComparison.OrdinalIgnoreCase))
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
using var entryStream = entry.Open();
|
||||||
|
using var buffer = new MemoryStream();
|
||||||
|
await entryStream.CopyToAsync(buffer, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
|
if (buffer.Length == 0)
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
buffer.Position = 0;
|
||||||
|
|
||||||
|
using var document = await JsonDocument.ParseAsync(buffer, cancellationToken: cancellationToken).ConfigureAwait(false);
|
||||||
|
updated = await ProcessBulletinJsonElementAsync(document.RootElement, entry.FullName, bulletinId, pendingDocuments, pendingMappings, now, updated, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
|
if (updated >= _options.MaxVulnerabilitiesPerFetch)
|
||||||
|
{
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return updated;
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<int> ProcessBulletinJsonElementAsync(
|
||||||
|
JsonElement element,
|
||||||
|
string entryName,
|
||||||
|
string bulletinId,
|
||||||
|
HashSet<Guid> pendingDocuments,
|
||||||
|
HashSet<Guid> pendingMappings,
|
||||||
|
DateTimeOffset now,
|
||||||
|
int processed,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
var updated = processed;
|
||||||
|
|
||||||
|
switch (element.ValueKind)
|
||||||
|
{
|
||||||
|
case JsonValueKind.Array:
|
||||||
|
foreach (var child in element.EnumerateArray())
|
||||||
|
{
|
||||||
|
cancellationToken.ThrowIfCancellationRequested();
|
||||||
|
|
||||||
|
if (updated >= _options.MaxVulnerabilitiesPerFetch)
|
||||||
|
{
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (child.ValueKind != JsonValueKind.Object)
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (await ProcessVulnerabilityObjectAsync(child, entryName, bulletinId, pendingDocuments, pendingMappings, now, cancellationToken).ConfigureAwait(false))
|
||||||
|
{
|
||||||
|
updated++;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
break;
|
||||||
|
|
||||||
|
case JsonValueKind.Object:
|
||||||
|
if (await ProcessVulnerabilityObjectAsync(element, entryName, bulletinId, pendingDocuments, pendingMappings, now, cancellationToken).ConfigureAwait(false))
|
||||||
|
{
|
||||||
|
updated++;
|
||||||
|
}
|
||||||
|
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
return updated;
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<bool> ProcessVulnerabilityObjectAsync(
|
||||||
|
JsonElement element,
|
||||||
|
string entryName,
|
||||||
|
string bulletinId,
|
||||||
|
HashSet<Guid> pendingDocuments,
|
||||||
|
HashSet<Guid> pendingMappings,
|
||||||
|
DateTimeOffset now,
|
||||||
|
CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
RuNkckiVulnerabilityDto dto;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
dto = RuNkckiJsonParser.Parse(element);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogDebug(ex, "NKCKI failed to parse vulnerability in bulletin {BulletinId} entry {Entry}", bulletinId, entryName);
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
var payload = JsonSerializer.SerializeToUtf8Bytes(dto, SerializerOptions);
|
||||||
|
var sha = Convert.ToHexString(SHA256.HashData(payload)).ToLowerInvariant();
|
||||||
|
var documentUri = BuildDocumentUri(dto);
|
||||||
|
|
||||||
|
var existing = await _documentStore.FindBySourceAndUriAsync(SourceName, documentUri, cancellationToken).ConfigureAwait(false);
|
||||||
|
if (existing is not null && string.Equals(existing.Sha256, sha, StringComparison.OrdinalIgnoreCase))
|
||||||
|
{
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
var gridFsId = await _rawDocumentStorage.UploadAsync(SourceName, documentUri, payload, "application/json", null, cancellationToken).ConfigureAwait(false);
|
||||||
|
|
||||||
|
var metadata = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase)
|
||||||
|
{
|
||||||
|
["ru-nkcki.bulletin"] = bulletinId,
|
||||||
|
["ru-nkcki.entry"] = entryName,
|
||||||
|
};
|
||||||
|
|
||||||
|
if (!string.IsNullOrWhiteSpace(dto.FstecId))
|
||||||
|
{
|
||||||
|
metadata["ru-nkcki.fstec_id"] = dto.FstecId!;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!string.IsNullOrWhiteSpace(dto.MitreId))
|
||||||
|
{
|
||||||
|
metadata["ru-nkcki.mitre_id"] = dto.MitreId!;
|
||||||
|
}
|
||||||
|
|
||||||
|
var recordId = existing?.Id ?? Guid.NewGuid();
|
||||||
|
var lastModified = dto.DateUpdated ?? dto.DatePublished;
|
||||||
|
var record = new DocumentRecord(
|
||||||
|
recordId,
|
||||||
|
SourceName,
|
||||||
|
documentUri,
|
||||||
|
now,
|
||||||
|
sha,
|
||||||
|
DocumentStatuses.PendingParse,
|
||||||
|
"application/json",
|
||||||
|
Headers: null,
|
||||||
|
Metadata: metadata,
|
||||||
|
Etag: null,
|
||||||
|
LastModified: lastModified,
|
||||||
|
GridFsId: gridFsId,
|
||||||
|
ExpiresAt: null);
|
||||||
|
|
||||||
|
var upserted = await _documentStore.UpsertAsync(record, cancellationToken).ConfigureAwait(false);
|
||||||
|
pendingDocuments.Add(upserted.Id);
|
||||||
|
pendingMappings.Remove(upserted.Id);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<SourceFetchContentResult> FetchListingAsync(CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var request = new SourceFetchRequest(RuNkckiOptions.HttpClientName, SourceName, _options.ListingUri)
|
||||||
|
{
|
||||||
|
AcceptHeaders = ListingAcceptHeaders,
|
||||||
|
TimeoutOverride = _options.RequestTimeout,
|
||||||
|
};
|
||||||
|
|
||||||
|
return await _fetchService.FetchContentAsync(request, cancellationToken).ConfigureAwait(false);
|
||||||
|
}
|
||||||
|
catch (Exception ex) when (ex is HttpRequestException or TaskCanceledException)
|
||||||
|
{
|
||||||
|
_logger.LogError(ex, "NKCKI listing fetch failed for {ListingUri}", _options.ListingUri);
|
||||||
|
await _stateRepository.MarkFailureAsync(SourceName, _timeProvider.GetUtcNow(), _options.FailureBackoff, ex.Message, cancellationToken).ConfigureAwait(false);
|
||||||
|
throw;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<IReadOnlyList<BulletinAttachment>> ParseListingAsync(byte[] content, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
var html = Encoding.UTF8.GetString(content);
|
||||||
|
var document = await _htmlParser.ParseDocumentAsync(html, cancellationToken).ConfigureAwait(false);
|
||||||
|
var anchors = document.QuerySelectorAll("a[href$='.json.zip']");
|
||||||
|
|
||||||
|
var attachments = new List<BulletinAttachment>();
|
||||||
|
foreach (var anchor in anchors)
|
||||||
|
{
|
||||||
|
var href = anchor.GetAttribute("href");
|
||||||
|
if (string.IsNullOrWhiteSpace(href))
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!Uri.TryCreate(_options.BaseAddress, href, out var absoluteUri))
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
var id = DeriveBulletinId(absoluteUri);
|
||||||
|
if (string.IsNullOrWhiteSpace(id))
|
||||||
|
{
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
var title = anchor.GetAttribute("title");
|
||||||
|
if (string.IsNullOrWhiteSpace(title))
|
||||||
|
{
|
||||||
|
title = anchor.TextContent?.Trim();
|
||||||
|
}
|
||||||
|
|
||||||
|
attachments.Add(new BulletinAttachment(id, absoluteUri, title ?? id));
|
||||||
|
}
|
||||||
|
|
||||||
|
return attachments;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string DeriveBulletinId(Uri uri)
|
||||||
|
{
|
||||||
|
var fileName = Path.GetFileName(uri.AbsolutePath);
|
||||||
|
if (string.IsNullOrWhiteSpace(fileName))
|
||||||
|
{
|
||||||
|
return Guid.NewGuid().ToString("N");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (fileName.EndsWith(".zip", StringComparison.OrdinalIgnoreCase))
|
||||||
|
{
|
||||||
|
fileName = fileName[..^4];
|
||||||
|
}
|
||||||
|
|
||||||
|
if (fileName.EndsWith(".json", StringComparison.OrdinalIgnoreCase))
|
||||||
|
{
|
||||||
|
fileName = fileName[..^5];
|
||||||
|
}
|
||||||
|
|
||||||
|
return fileName.Replace('_', '-');
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string BuildDocumentUri(RuNkckiVulnerabilityDto dto)
|
||||||
|
{
|
||||||
|
if (!string.IsNullOrWhiteSpace(dto.FstecId))
|
||||||
|
{
|
||||||
|
var slug = dto.FstecId.Contains(':', StringComparison.Ordinal)
|
||||||
|
? dto.FstecId[(dto.FstecId.IndexOf(':') + 1)..]
|
||||||
|
: dto.FstecId;
|
||||||
|
return $"https://cert.gov.ru/materialy/uyazvimosti/{slug}";
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!string.IsNullOrWhiteSpace(dto.MitreId))
|
||||||
|
{
|
||||||
|
return $"https://nvd.nist.gov/vuln/detail/{dto.MitreId}";
|
||||||
|
}
|
||||||
|
|
||||||
|
return $"https://cert.gov.ru/materialy/uyazvimosti/{Guid.NewGuid():N}";
|
||||||
|
}
|
||||||
|
|
||||||
|
private string ResolveCacheDirectory(string? configuredPath)
|
||||||
|
{
|
||||||
|
if (!string.IsNullOrWhiteSpace(configuredPath))
|
||||||
|
{
|
||||||
|
return Path.GetFullPath(Path.IsPathRooted(configuredPath)
|
||||||
|
? configuredPath
|
||||||
|
: Path.Combine(AppContext.BaseDirectory, configuredPath));
|
||||||
|
}
|
||||||
|
|
||||||
|
return Path.Combine(AppContext.BaseDirectory, "cache", RuNkckiConnectorPlugin.SourceName);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void EnsureCacheDirectory()
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
Directory.CreateDirectory(_cacheDirectory);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogWarning(ex, "NKCKI unable to ensure cache directory {CachePath}", _cacheDirectory);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private string GetBulletinCachePath(string bulletinId)
|
||||||
|
{
|
||||||
|
var fileStem = string.IsNullOrWhiteSpace(bulletinId)
|
||||||
|
? Guid.NewGuid().ToString("N")
|
||||||
|
: Uri.EscapeDataString(bulletinId);
|
||||||
|
return Path.Combine(_cacheDirectory, $"{fileStem}.json.zip");
|
||||||
|
}
|
||||||
|
|
||||||
|
private static string ExtractBulletinIdFromCachePath(string path)
|
||||||
|
{
|
||||||
|
if (string.IsNullOrWhiteSpace(path))
|
||||||
|
{
|
||||||
|
return string.Empty;
|
||||||
|
}
|
||||||
|
|
||||||
|
var fileName = Path.GetFileName(path);
|
||||||
|
if (fileName.EndsWith(".zip", StringComparison.OrdinalIgnoreCase))
|
||||||
|
{
|
||||||
|
fileName = fileName[..^4];
|
||||||
|
}
|
||||||
|
|
||||||
|
if (fileName.EndsWith(".json", StringComparison.OrdinalIgnoreCase))
|
||||||
|
{
|
||||||
|
fileName = fileName[..^5];
|
||||||
|
}
|
||||||
|
|
||||||
|
return Uri.UnescapeDataString(fileName);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void TryWriteCachedBulletin(string bulletinId, byte[] content)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var cachePath = GetBulletinCachePath(bulletinId);
|
||||||
|
Directory.CreateDirectory(Path.GetDirectoryName(cachePath)!);
|
||||||
|
File.WriteAllBytes(cachePath, content);
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogDebug(ex, "NKCKI failed to cache bulletin {BulletinId}", bulletinId);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private bool TryReadCachedBulletin(string bulletinId, out byte[] content)
|
||||||
|
{
|
||||||
|
var cachePath = GetBulletinCachePath(bulletinId);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
if (File.Exists(cachePath))
|
||||||
|
{
|
||||||
|
content = File.ReadAllBytes(cachePath);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
catch (Exception ex)
|
||||||
|
{
|
||||||
|
_logger.LogDebug(ex, "NKCKI failed to read cached bulletin {BulletinId}", bulletinId);
|
||||||
|
}
|
||||||
|
|
||||||
|
content = Array.Empty<byte>();
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
private IReadOnlyCollection<string> NormalizeBulletins(IEnumerable<string> bulletins)
|
||||||
|
{
|
||||||
|
var normalized = (bulletins ?? Enumerable.Empty<string>())
|
||||||
|
.Where(static id => !string.IsNullOrWhiteSpace(id))
|
||||||
|
.Distinct(StringComparer.OrdinalIgnoreCase)
|
||||||
|
.OrderBy(static id => id, StringComparer.OrdinalIgnoreCase)
|
||||||
|
.ToList();
|
||||||
|
|
||||||
|
if (normalized.Count <= _options.KnownBulletinCapacity)
|
||||||
|
{
|
||||||
|
return normalized.ToArray();
|
||||||
|
}
|
||||||
|
|
||||||
|
var skip = normalized.Count - _options.KnownBulletinCapacity;
|
||||||
|
return normalized.Skip(skip).ToArray();
|
||||||
|
}
|
||||||
|
|
||||||
|
private async Task<RuNkckiCursor> GetCursorAsync(CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
var state = await _stateRepository.TryGetAsync(SourceName, cancellationToken).ConfigureAwait(false);
|
||||||
|
return state is null ? RuNkckiCursor.Empty : RuNkckiCursor.FromBson(state.Cursor);
|
||||||
|
}
|
||||||
|
|
||||||
|
private Task UpdateCursorAsync(RuNkckiCursor cursor, CancellationToken cancellationToken)
|
||||||
|
{
|
||||||
|
var document = cursor.ToBsonDocument();
|
||||||
|
var completedAt = cursor.LastListingFetchAt ?? _timeProvider.GetUtcNow();
|
||||||
|
return _stateRepository.UpdateCursorAsync(SourceName, document, completedAt, cancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
private readonly record struct BulletinAttachment(string Id, Uri Uri, string Title);
|
||||||
|
}
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
|
using StellaOps.Plugin;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Nkcki;
|
||||||
|
|
||||||
|
public sealed class RuNkckiConnectorPlugin : IConnectorPlugin
|
||||||
|
{
|
||||||
|
public const string SourceName = "ru-nkcki";
|
||||||
|
|
||||||
|
public string Name => SourceName;
|
||||||
|
|
||||||
|
public bool IsAvailable(IServiceProvider services) => services is not null;
|
||||||
|
|
||||||
|
public IFeedConnector Create(IServiceProvider services)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
return ActivatorUtilities.CreateInstance<RuNkckiConnector>(services);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
using Microsoft.Extensions.Configuration;
|
||||||
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
|
using StellaOps.DependencyInjection;
|
||||||
|
using StellaOps.Feedser.Core.Jobs;
|
||||||
|
using StellaOps.Feedser.Source.Ru.Nkcki.Configuration;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Nkcki;
|
||||||
|
|
||||||
|
public sealed class RuNkckiDependencyInjectionRoutine : IDependencyInjectionRoutine
|
||||||
|
{
|
||||||
|
private const string ConfigurationSection = "feedser:sources:ru-nkcki";
|
||||||
|
|
||||||
|
public IServiceCollection Register(IServiceCollection services, IConfiguration configuration)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
ArgumentNullException.ThrowIfNull(configuration);
|
||||||
|
|
||||||
|
services.AddRuNkckiConnector(options =>
|
||||||
|
{
|
||||||
|
configuration.GetSection(ConfigurationSection).Bind(options);
|
||||||
|
options.Validate();
|
||||||
|
});
|
||||||
|
|
||||||
|
services.AddTransient<RuNkckiFetchJob>();
|
||||||
|
services.AddTransient<RuNkckiParseJob>();
|
||||||
|
services.AddTransient<RuNkckiMapJob>();
|
||||||
|
|
||||||
|
services.PostConfigure<JobSchedulerOptions>(options =>
|
||||||
|
{
|
||||||
|
EnsureJob(options, RuNkckiJobKinds.Fetch, typeof(RuNkckiFetchJob));
|
||||||
|
EnsureJob(options, RuNkckiJobKinds.Parse, typeof(RuNkckiParseJob));
|
||||||
|
EnsureJob(options, RuNkckiJobKinds.Map, typeof(RuNkckiMapJob));
|
||||||
|
});
|
||||||
|
|
||||||
|
return services;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static void EnsureJob(JobSchedulerOptions schedulerOptions, string kind, Type jobType)
|
||||||
|
{
|
||||||
|
if (schedulerOptions.Definitions.ContainsKey(kind))
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
schedulerOptions.Definitions[kind] = new JobDefinition(
|
||||||
|
kind,
|
||||||
|
jobType,
|
||||||
|
schedulerOptions.DefaultTimeout,
|
||||||
|
schedulerOptions.DefaultLeaseDuration,
|
||||||
|
CronExpression: null,
|
||||||
|
Enabled: true);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
using System.Net;
|
||||||
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using StellaOps.Feedser.Source.Common.Http;
|
||||||
|
using StellaOps.Feedser.Source.Ru.Nkcki.Configuration;
|
||||||
|
|
||||||
|
namespace StellaOps.Feedser.Source.Ru.Nkcki;
|
||||||
|
|
||||||
|
public static class RuNkckiServiceCollectionExtensions
|
||||||
|
{
|
||||||
|
public static IServiceCollection AddRuNkckiConnector(this IServiceCollection services, Action<RuNkckiOptions> configure)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(services);
|
||||||
|
ArgumentNullException.ThrowIfNull(configure);
|
||||||
|
|
||||||
|
services.AddOptions<RuNkckiOptions>()
|
||||||
|
.Configure(configure)
|
||||||
|
.PostConfigure(static options => options.Validate());
|
||||||
|
|
||||||
|
services.AddSourceHttpClient(RuNkckiOptions.HttpClientName, (sp, clientOptions) =>
|
||||||
|
{
|
||||||
|
var options = sp.GetRequiredService<IOptions<RuNkckiOptions>>().Value;
|
||||||
|
clientOptions.BaseAddress = options.BaseAddress;
|
||||||
|
clientOptions.Timeout = options.RequestTimeout;
|
||||||
|
clientOptions.UserAgent = options.UserAgent;
|
||||||
|
clientOptions.AllowAutoRedirect = true;
|
||||||
|
clientOptions.DefaultRequestHeaders["Accept-Language"] = options.AcceptLanguage;
|
||||||
|
clientOptions.AllowedHosts.Clear();
|
||||||
|
clientOptions.AllowedHosts.Add(options.BaseAddress.Host);
|
||||||
|
clientOptions.ConfigureHandler = handler =>
|
||||||
|
{
|
||||||
|
handler.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
|
||||||
|
handler.AllowAutoRedirect = true;
|
||||||
|
handler.UseCookies = true;
|
||||||
|
handler.CookieContainer = new CookieContainer();
|
||||||
|
};
|
||||||
|
});
|
||||||
|
|
||||||
|
services.AddTransient<RuNkckiConnector>();
|
||||||
|
|
||||||
|
return services;
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,16 +1,22 @@
|
|||||||
<Project Sdk="Microsoft.NET.Sdk">
|
<Project Sdk="Microsoft.NET.Sdk">
|
||||||
|
|
||||||
<PropertyGroup>
|
<PropertyGroup>
|
||||||
<TargetFramework>net10.0</TargetFramework>
|
<TargetFramework>net10.0</TargetFramework>
|
||||||
<ImplicitUsings>enable</ImplicitUsings>
|
<ImplicitUsings>enable</ImplicitUsings>
|
||||||
<Nullable>enable</Nullable>
|
<Nullable>enable</Nullable>
|
||||||
</PropertyGroup>
|
</PropertyGroup>
|
||||||
|
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<ProjectReference Include="../StellaOps.Plugin/StellaOps.Plugin.csproj" />
|
<PackageReference Include="AngleSharp" Version="1.1.1" />
|
||||||
|
</ItemGroup>
|
||||||
<ProjectReference Include="../StellaOps.Feedser.Source.Common/StellaOps.Feedser.Source.Common.csproj" />
|
|
||||||
<ProjectReference Include="../StellaOps.Feedser.Models/StellaOps.Feedser.Models.csproj" />
|
<ItemGroup>
|
||||||
</ItemGroup>
|
<ProjectReference Include="../StellaOps.Plugin/StellaOps.Plugin.csproj" />
|
||||||
</Project>
|
<ProjectReference Include="../StellaOps.Feedser.Core/StellaOps.Feedser.Core.csproj" />
|
||||||
|
<ProjectReference Include="../StellaOps.Feedser.Normalization/StellaOps.Feedser.Normalization.csproj" />
|
||||||
|
<ProjectReference Include="../StellaOps.Feedser.Source.Common/StellaOps.Feedser.Source.Common.csproj" />
|
||||||
|
<ProjectReference Include="../StellaOps.Feedser.Models/StellaOps.Feedser.Models.csproj" />
|
||||||
|
<ProjectReference Include="../StellaOps.Feedser.Storage.Mongo/StellaOps.Feedser.Storage.Mongo.csproj" />
|
||||||
|
</ItemGroup>
|
||||||
|
|
||||||
|
</Project>
|
||||||
|
|||||||
@@ -2,10 +2,10 @@
|
|||||||
| Task | Owner(s) | Depends on | Notes |
|
| Task | Owner(s) | Depends on | Notes |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
|FEEDCONN-NKCKI-02-001 Research NKTsKI advisory feeds|BE-Conn-Nkcki|Research|**DONE (2025-10-11)** – Candidate RSS locations (`https://cert.gov.ru/rss/advisories.xml`, `https://www.cert.gov.ru/...`) return 403/404 even with `Accept-Language: ru-RU` and `--insecure`; site is Bitrix-backed and expects Russian Trusted Sub CA plus session cookies. Logged packet captures + needed cert list in `docs/feedser-connector-research-20251011.md`; waiting on Ops for sanctioned trust bundle.|
|
|FEEDCONN-NKCKI-02-001 Research NKTsKI advisory feeds|BE-Conn-Nkcki|Research|**DONE (2025-10-11)** – Candidate RSS locations (`https://cert.gov.ru/rss/advisories.xml`, `https://www.cert.gov.ru/...`) return 403/404 even with `Accept-Language: ru-RU` and `--insecure`; site is Bitrix-backed and expects Russian Trusted Sub CA plus session cookies. Logged packet captures + needed cert list in `docs/feedser-connector-research-20251011.md`; waiting on Ops for sanctioned trust bundle.|
|
||||||
|FEEDCONN-NKCKI-02-002 Fetch pipeline & state persistence|BE-Conn-Nkcki|Source.Common, Storage.Mongo|**TODO** – Implement fetch job with custom trust store, optional SOCKS proxy, and Bitrix session bootstrap (`PHPSESSID`, `BITRIX_SM_GUEST_ID`). Persist raw XML/HTML + derived cursor (advisory ID + `pubDate`), handle 403 retries with exponential backoff.|
|
|FEEDCONN-NKCKI-02-002 Fetch pipeline & state persistence|BE-Conn-Nkcki|Source.Common, Storage.Mongo|**DOING (2025-10-12)** – Listing fetch now expands `*.json.zip` bulletins into per-vulnerability JSON documents with cursor-tracked bulletin IDs and trust store wiring (`globalsign_r6_bundle.pem`). Parser/mapper emit canonical advisories; remaining work: strengthen pagination/backfill handling and add regression fixtures/telemetry. Offline cache helpers (ProcessCachedBulletinsAsync/TryReadCachedBulletin/TryWriteCachedBulletin) implemented.|
|
||||||
|FEEDCONN-NKCKI-02-003 DTO & parser implementation|BE-Conn-Nkcki|Source.Common|**TODO** – Build DTOs for NKTsKI advisories, sanitise HTML, extract vendors/products, CVEs, mitigation guidance.|
|
|FEEDCONN-NKCKI-02-003 DTO & parser implementation|BE-Conn-Nkcki|Source.Common|**DOING (2025-10-12)** – `RuNkckiJsonParser` extracts per-vulnerability JSON payloads (IDs, CVEs, CVSS, software text, URLs). TODO: extend coverage for optional fields (ICS categories, nested arrays) and add fixture snapshots.|
|
||||||
|FEEDCONN-NKCKI-02-004 Canonical mapping & range primitives|BE-Conn-Nkcki|Models|**TODO** – Map advisories into canonical records with aliases, references, and vendor range primitives. Coordinate normalized outputs and provenance per `../StellaOps.Feedser.Merge/RANGE_PRIMITIVES_COORDINATION.md`.<br>2025-10-11 research trail: normalized payload target `[{"scheme":"semver","type":"range","min":"<start>","minInclusive":true,"max":"<end>","maxInclusive":false,"notes":"ru.nkcki:advisory-id"}]`; retain Cyrillic identifiers in `notes` so storage provenance remains intact.|
|
|FEEDCONN-NKCKI-02-004 Canonical mapping & range primitives|BE-Conn-Nkcki|Models|**DOING (2025-10-12)** – `RuNkckiMapper` maps JSON entries to canonical advisories (aliases, references, vendor package, CVSS). Next steps: enrich package parsing (`software_text` tokenisation), consider CVSS v4 metadata, and backfill provenance docs before closing the task.|
|
||||||
|FEEDCONN-NKCKI-02-005 Deterministic fixtures & tests|QA|Testing|**TODO** – Add regression tests supporting `UPDATE_NKCKI_FIXTURES=1` for snapshot regeneration.|
|
|FEEDCONN-NKCKI-02-005 Deterministic fixtures & tests|QA|Testing|**DOING (2025-10-12)** – Added mocked listing/bulletin regression harness (`RuNkckiConnectorTests`) with fixtures + snapshot writer. Test run currently blocked on Mongo2Go dependency (libcrypto.so.1.1 missing); follow-up required to get embedded mongod running in CI before marking DONE.|
|
||||||
|FEEDCONN-NKCKI-02-006 Telemetry & documentation|DevEx|Docs|**TODO** – Add logging/metrics, document connector configuration, and close backlog entry after deliverable ships.|
|
|FEEDCONN-NKCKI-02-006 Telemetry & documentation|DevEx|Docs|**TODO** – Add logging/metrics, document connector configuration, and close backlog entry after deliverable ships.|
|
||||||
|FEEDCONN-NKCKI-02-007 Archive ingestion strategy|BE-Conn-Nkcki|Research|**TODO** – Once access restored, map Bitrix paging (`?PAGEN_1=`) and advisory taxonomy (alerts vs recommendations). Outline HTML scrape + PDF attachment handling for backfill and decide translation approach for Russian-only content.|
|
|FEEDCONN-NKCKI-02-007 Archive ingestion strategy|BE-Conn-Nkcki|Research|**TODO** – Once access restored, map Bitrix paging (`?PAGEN_1=`) and advisory taxonomy (alerts vs recommendations). Outline HTML scrape + PDF attachment handling for backfill and decide translation approach for Russian-only content.|
|
||||||
|FEEDCONN-NKCKI-02-008 Access enablement plan|BE-Conn-Nkcki|Source.Common|**DONE (2025-10-11)** – Documented trust-store requirement, optional SOCKS proxy fallback, and monitoring plan; shared TLS support now available via `SourceHttpClientOptions.TrustedRootCertificates` (`feedser:httpClients:source.nkcki:*`), awaiting Ops-sourced cert bundle before fetch implementation.|
|
|FEEDCONN-NKCKI-02-008 Access enablement plan|BE-Conn-Nkcki|Source.Common|**DONE (2025-10-11)** – Documented trust-store requirement, optional SOCKS proxy fallback, and monitoring plan; shared TLS support now available via `SourceHttpClientOptions.TrustedRootCertificates` (`feedser:httpClients:source.nkcki:*`), awaiting Ops-sourced cert bundle before fetch implementation.|
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user