Can't find latest SDK nuget packages in GH NuGet feed

[LuKePicci]

Maybe I'm making a dumb mistake, but I see the GH nuget feed from current README has indeed a lot more packages than the DevOps but not newer (they both apparently contains the same latest compiled for FreeBSD 12). I thought the GH feed also included newer packages built for FreeBSD 13.

A bit of context: I'm having issues using the feed from @sec as it is for FreeBSD 14 and current pfSense 2.7.x is not aligned to the same FreeBSD 14 release branch and package dependencies (which the sec's nuget app host package refers to).. Instead, I'm having no issues running binaries built for FreeBSD 12 because of the comatx13 package, but I wanted to update the SDK to latest one for FreeBSD 13, which would still work thanks to the same compatx13..

[sec]

Hey.
My feed have mainly binaries built under 13 or 14 - but for nugets, this should be much an issue, becuase that's managed code, what matter is your sdk version aligned with your FreeBSD version. If you want nuget's for 13 or 14 - you can extract them from Private Built Artifacts - you can find them on my release page.

What packages exactly are you having issues with?

[LuKePicci]

I'm building a. Net app from windows for freebsd. I publish it for freebsd as a single file (which implies self-contained runtime). As far as I can pinpoint, the issue with your feed is that the nuget packages for the native parts contains a native ELF stub which is linked to native libraries (eg. libssl, libcrypto,..) of versions that do exist in the corresponding Freebsd version you built them, but do not exist in the stupid deviation used by pfsense for the 2.7.x branch.

I can publish a working app binary by using dotnet CLI with a global.json overriding the SDK version down to 8.0.100 (8.0.0 packages) enabling the devops feed from this repo, which are targeted to freebsd 12, which runs fine even on pfsense 2.7.x for coincidence. This is fine but I would at least move to newer sdks versions because it is hard to keep 8.0.100 installed (it is known to be vulnerable and windows insists in auto upgrading it).

I see the nuget packages here in the releases, but I wonder if they should also be inside the feeds you kindly provided.

[sec]

For 8 this can be hard to achieve, as output rid was something like freebsd-x64. With 9 I saw that now we have freebsd.13-x64 or freebsd.14-x64 - but yes they are linked to base system libs.

Why you can't move to newer 8 versions? Ports have 8.0.6 - newer ones are hard to build for ports, becuase they require some nugets that are not part of artifacts (dunno why, maybe I've tried with wrong versions). But anyway, more work was focused on getting 9.

Is it possible to run your app on .net 9?

For my feed, I usually put nugets from 13 for x86 and 14 for arm64 - only for 9 I'm putting both versions from 13 and 14 where I can/have machines ready to build.

One way would be for you, to maybe have some spare machine with pfSense of your choice, with added extra libs somehow needed and compile your own version of SDK using some bootstrapped version?

All of my builds are mainly done on fresh 13/14 FreeBSD installs, without any external libraries, so they should work on every -RELEASE - at least in theory :)

[LuKePicci]

For 8 this can be hard to achieve, as output rid was something like freebsd-x64.

Yup, that's what I'm using

Why you can't move to newer 8 versions?

That's what I would like to do at the moment. I was just looking for such 8.0.8 nugets from thefrank feed because I though yours x86 were for 14. I didn't know about issues in getting newer 8.0.x , I saw them in this repo releases so I thought you got it, but I now understand they are built from 8.0.100, that could explain why VS is not finding correct nugets. I'm now looking around the feeds to find 8.0.6 nugets for Freebsd 13, will use the dotnet cli to override ths sdk down to that version instead.

Is it possible to run your app on .net 9?

They should run fine on .net 9 as well, it is all about getting an easy-to-live-with development experience which is fine for both pfSense 2.6.0 (yeah, I still have some around, shame on the broken cdce driver for older Sierra Wireless LTE modems) and newer 2.7.x
There is obviously no more support for FreeBSD 12, so it is not much convenient for me to switch to 9 right now.

For my feed, I usually put nugets from 13 for x86 and 14 for arm64

I see, this probably got me confused, I have to check this deeper.

so they should work on every -RELEASE - at least in theory :)

pfSense 2.7.x gone with 14-CURRENT before 14.0-RELEASE. They included some libs which are newer then packages available for released 14.0 and older then 14.1

[sec]

I have private nuget feed which you can use/connect to - https://github.com/sec/dotnet-freebsd-nuget-feed - maybe that would help. it should contain all current up-to-date nugets, only latest 8.x are missing (I have them on hdd, just need to upload them in free moment).

[Thefrank]

The NuGet files here are not flagged for specific version of FreeBSD as they are built via each dotNET repo and default to a portable build.
The VMR (and ports which uses it) are technically non-portable however, there should not be too many issues unless something library/symbol related issue comes up.

[LuKePicci]

So I made a few trials, with the nuget repo at https://fbsdnugetfeed.mooo.com/v3/index.json

I have both 8.0.100 and 8.0.112 SDK installed, set SDK version override from global.json file for the project, this file is only read by dotnet cli so I made the publish from CLI.

Here the results running the single-file output "MyAppBin" on pfSense 2.7.2 without the openssl package installed:

• 8.0.100 prints normal application console log, then as soon as it requires the libssl capabilities it prints:
  No usable version of libssl was found
  Not sure if this comes from the runtime or from the Azure IoT Hub SDK, but I guess it is looking for some of those:

# strings MyAppBin | grep libssl.so
libssl.so.3
libssl.so.11
libssl.so.1.0.0
libssl.so.10
libssl.so.1.1
libssl.so.1.0.2
libssl.so.111
libssl.so.

• 8.0.112: immediately prints:
  ld-elf.so.1: Shared object "libcrypto.so.12" not found, required by "MyNetBin"

pfSense already ships with libssl in its base OS files, even before installing openssl package, but:

# find / | grep libssl
/usr/libdata/pkgconfig/libssl.pc
/usr/lib/libssl.so
/usr/lib/debug/usr/lib/libssl.so.111.debug
/usr/lib/libssl.so.30
/usr/lib/libssl.a
/usr/local/lib/libssl3.so

In order to install the openssl package, which is not available in the pfSense repos, I have to pick it from the FreeBSD repos. Enabling FreeBSD repos in pfSense 2.7.2 is possible, but an update of the pkg tool is needed in order to use the online repo metadata, and it is not desirable as it breaks pkg and system updates. So I search manually inside the packagsite.yaml files at https://pkg.freebsd.org/FreeBSD:14:amd64/.

For FreeBSD:14:release_0 the available openssl package is at version 1.1.1w,1 and there also is a separate openssl30 package:

{"name":"openssl","origin":"security/openssl","version":"1.1.1w,1","comment":"TLSv1.3 capable SSL and crypto library","maintainer":"brnrd@FreeBSD.org","www":"https://www.openssl.org/","abi":"FreeBSD:14:amd64","arch":"freebsd:14:x86:64","prefix":"/usr/local","sum":"17b1bdfb47c1405eb72ebe24f109365a5227811a63537d6210f29ef3ba483645","flatsize":14205107,"path":"All/openssl-1.1.1w,1.pkg","repopath":"All/openssl-1.1.1w,1.pkg","licenselogic":"single","licenses":["OpenSSL"],"pkgsize":4333642,"desc":"The OpenSSL Project is a collaborative effort to develop a robust,\ncommercial-grade, full-featured, and Open Source toolkit implementing\nthe Secure Sockets Layer (SSL v3) and Transport Layer Security (TLS v1,\nv1.1, v1.2, v1.3) protocols with full-strength cryptography world-wide.\nThe project is managed by a worldwide community of volunteers that use\nthe Internet to communicate, plan, and develop the OpenSSL tookit\nand its related documentation.\n\nOpenSSL is based on the excellent SSLeay library developed by Eric\nA. Young and Tim J. Hudson. The OpenSSL toolkit is licensed under\nan Apache-style licence, which basically means that you are free\nto get and use it for commercial and non-commercial purposes subject\nto some simple license conditions.","categories":["security","devel"],"shlibs_provided":["libcrypto.so.11","libssl.so.11"],"options":{"ASYNC":"on","CT":"on","KTLS":"on","MAN3":"on","RFC3779":"off","SHARED":"on","ZLIB":"off","ARIA":"off","DES":"on","GOST":"on","IDEA":"off","SM4":"off","RC2":"on","RC4":"on","RC5":"off","WEAK-SSL-CIPHERS":"off","MD2":"off","MD4":"on","MDC2":"off","RMD160":"on","SM2":"off","SM3":"off","ASM":"on","SSE2":"on","THREADS":"on","EC":"on","NEXTPROTONEG":"on","SCTP":"on","SSL3":"off","TLS1":"on","TLS1_1":"on","TLS1_2":"on"},"annotations":{"cpe":"cpe:2.3:a:openssl:openssl:1.1.1w:::::freebsd14:x64","FreeBSD_version":"1400097"}}
{"name":"openssl30","origin":"security/openssl30","version":"3.0.11_1","comment":"TLSv1.3 capable SSL and crypto library","maintainer":"brnrd@FreeBSD.org","www":"https://www.openssl.org/","abi":"FreeBSD:14:amd64","arch":"freebsd:14:x86:64","prefix":"/usr/local","sum":"c9b889de7b645cf9675950fdff597da94943a80f09b802acbf695157a8823d8a","flatsize":23081822,"path":"All/openssl30-3.0.11_1.pkg","repopath":"All/openssl30-3.0.11_1.pkg","licenselogic":"single","licenses":["APACHE20"],"pkgsize":6763723,"desc":"The OpenSSL Project is a collaborative effort to develop a robust,\ncommercial-grade, full-featured, and Open Source toolkit implementing\nthe Secure Sockets Layer (SSL v3) and Transport Layer Security (TLS v1,\nv1.1, v1.2, v1.3) protocols with full-strength cryptography world-wide.\nThe project is managed by a worldwide community of volunteers that use\nthe Internet to communicate, plan, and develop the OpenSSL tookit\nand its related documentation.\n\nOpenSSL is based on the excellent SSLeay library developed by Eric\nA. Young and Tim J. Hudson. The OpenSSL toolkit is licensed under\nan Apache-style licence, which basically means that you are free\nto get and use it for commercial and non-commercial purposes subject\nto some simple license conditions.","categories":["security","devel"],"shlibs_provided":["libcrypto.so.12","libssl.so.12"],"options":{"ASYNC":"on","CT":"on","KTLS":"on","MAN3":"on","RFC3779":"on","SHARED":"on","ZLIB":"off","ARIA":"off","DES":"on","GOST":"on","IDEA":"off","SM4":"off","RC2":"on","RC4":"on","RC5":"off","WEAK-SSL-CIPHERS":"off","MD2":"off","MD4":"on","MDC2":"off","RMD160":"on","SM2":"off","SM3":"off","FIPS":"on","LEGACY":"off","ASM":"on","SSE2":"on","THREADS":"on","EC":"on","NEXTPROTONEG":"on","SCTP":"on","SSL3":"off","TLS1":"on","TLS1_1":"on","TLS1_2":"on"},"annotations":{"cpe":"cpe:2.3:a:openssl:openssl:3.0.11:::::freebsd14:x64:1","FreeBSD_version":"1400097"}}

For FreeBSD:14:release_1 openssl package is updated at version 3.0.13_4,1 but a compat13x package appears:

{"name":"openssl","origin":"security/openssl","version":"3.0.13_4,1","comment":"TLSv1.3 capable SSL and crypto library","maintainer":"brnrd@FreeBSD.org","www":"https://www.openssl.org/","abi":"FreeBSD:14:amd64","arch":"freebsd:14:x86:64","prefix":"/usr/local","sum":"3ace343ab7fb1491feab762af6d76499f19c09262afa885fdcba44106316db1d","flatsize":23138949,"path":"All/openssl-3.0.13_4,1.pkg","repopath":"All/openssl-3.0.13_4,1.pkg","licenselogic":"single","licenses":["APACHE20"],"pkgsize":6792547,"desc":"The OpenSSL Project is a collaborative effort to develop a robust,\ncommercial-grade, full-featured, and Open Source toolkit implementing\nthe Secure Sockets Layer (SSL v3) and Transport Layer Security (TLS v1,\nv1.1, v1.2, v1.3) protocols with full-strength cryptography world-wide.\nThe project is managed by a worldwide community of volunteers that use\nthe Internet to communicate, plan, and develop the OpenSSL tookit\nand its related documentation.\n\nOpenSSL is based on the excellent SSLeay library developed by Eric\nA. Young and Tim J. Hudson. The OpenSSL toolkit is licensed under\nan Apache-style licence, which basically means that you are free\nto get and use it for commercial and non-commercial purposes subject\nto some simple license conditions.","categories":["security","devel"],"shlibs_provided":["libcrypto.so.12","libssl.so.12"],"options":{"ASYNC":"on","CT":"on","KTLS":"on","MAN3":"on","RFC3779":"on","SHARED":"on","ZLIB":"off","ARIA":"off","DES":"on","GOST":"on","IDEA":"off","SM4":"off","RC2":"on","RC4":"on","RC5":"off","WEAK-SSL-CIPHERS":"off","MD2":"off","MD4":"on","MDC2":"off","RMD160":"on","SM2":"off","SM3":"off","FIPS":"on","LEGACY":"off","ASM":"on","SSE2":"on","THREADS":"on","EC":"on","NEXTPROTONEG":"on","SCTP":"on","SSL3":"off","TLS1":"on","TLS1_1":"on","TLS1_2":"on"},"annotations":{"build_timestamp":"2024-05-19T01:04:58+0000","ports_top_git_hash":"347f69f1d","ports_top_checkout_unclean":"no","port_git_hash":"30e246f34","port_checkout_unclean":"no","built_by":"poudriere-git-3.4.1-30-g79e3edcd","cpe":"cpe:2.3:a:openssl:openssl:3.0.13:::::freebsd14:x64:4","FreeBSD_version":"1400097"}}
{"name":"compat13x-amd64","origin":"misc/compat13x","version":"13.2.1302001.20231227","comment":"Convenience package to install the compat13x libraries","maintainer":"arrowd@FreeBSD.org","www":"UNKNOWN","abi":"FreeBSD:14:amd64","arch":"freebsd:14:x86:64","prefix":"/usr/local","sum":"9851d372455337080ebb7e098f783d7cca9cf94789fa8b5ebc05485474a1bf70","flatsize":6712568,"path":"All/compat13x-amd64-13.2.1302001.20231227.pkg","repopath":"All/compat13x-amd64-13.2.1302001.20231227.pkg","licenselogic":"single","licenses":["BSD2CLAUSE"],"pkgsize":2426699,"desc":"This package allows you to install the compat13x libraries on your\nsystem, so you can use legacy binaries that depend on them.\n\nPorts usage example:\n\n--\n.include <bsd.port.pre.mk>\n\n.if ${OSVERSION} >= 1400000\nLIB_DEPENDS+=\tlibcrypto.so.111:misc/compat13x\n.endif\n--","categories":["misc"],"shlibs_provided":["libcrypto.so.111","libssl.so.111"],"annotations":{"build_timestamp":"2024-04-09T01:08:21+0000","ports_top_git_hash":"c5cd82114","ports_top_checkout_unclean":"no","port_git_hash":"b3aa1ea86","port_checkout_unclean":"no","built_by":"poudriere-git-3.4.1-1-g1e9f97d6","FreeBSD_version":"1400097"}}

For FreeBSD:14:release_2 openssl package is even newer but compat13x is still there.

As you see from the shlibs_provided files, installing the openssl package for FreeBSD 14.0 makes the 8.0.100 compiled MyAppBin to work on pfSense 2.7.2. Installing the openssl30 makes the other 8.0.112 compiled app to work. The libs files from packages from FreeBSD 14.1 are not binary compatible and they make app crash with obvious errors aboit libc and similar.

If I switch to the nuget repo at https://pkgs.dev.azure.com/IFailAt/freebsd-dotnet-runtime-nightly/_packaging/freebsd-dotnet/nuget/v3/index.json I cannot find necessary packages for compiling with anything above 8.0.100.
Using 8.0.100 makes the dotnet publish output a much smaller binary file which runs fine on pfSense 2.7.2 with the same openssl 1.1.1w package from FreeBSD 14.0

So, trying to recap:

• the SDKs you compiled up to 8.0.100 (thefrank) and up to 8.0.101 (sec) ships with a runtime which is looking for libs that:
  -- are not part of pfSense 2.7.2
  -- can be installed in FreeBSD 14.0 (including pfSense 2.7.2) using openssl package from FreeBSD 14.0
  -- are no more provided by openssl package from FreeBSD 14.1+ where they are provided by compat13x from FreeBSD 14.1+, instead.
• the SDKs you compiled after 8.0.100, are missing nugets in TheFrank nuget repo (devops), and successfully producing AppHost binaries with the other sec nuget repo, which are LD-linked to a libssl version which is:
  -- not present in pfSense 2.7.2
  -- can be installed in FreeBSD 14.0 (including pfSense 2.7.2) using openssl30 package from FreeBSD 14.0
  -- is peacefully provided by openssl package from FreeBSD 14.1+

That said, it seems I can't get my app built by an SDK newer than 8.0.101 without making it require a libssl version (libssl.so.12) I can only get for FreeBSD 14.x (and pfSense 2.7.2) but not for FreeBSD 12.3 (pfSense 2.6.0).

[Thefrank]

What NuGets are missing from my feed? I tried to avoid uploading the dotNET 8 stuff unless I was sure the crossbuild did not output Linux ELFs into the NuGet. This did not impact anything before NET8 and anything after NET8 (e.g., NET9)

[sec]

IIRC, .net8 will require openssl 1.x (that's why compat pkg is needed). 9 should use/work with 3 (which 14 include as base now).
What might be confusing is that /usr/local/lib/libssl.so.12 was installed by package openssl-3.0.15_1,1 and there's only /usr/lib/libssl.so.30

BUT as general rule, the same is with clang that's in base - it should only be used to build world/kernel. I guess the same is with base openssl version. Installing the one from ports/pkg should be used by apps if you ask me - when you update base system, your apps shouldn't break because of clang/openssl version change. That's why you have option to use the one from ports/pkg.

Also FreeBSD 12.x is EOF, so I wouldn't put any effort on it anyway.

[Thefrank]

I will also add: I do not know about the original .NET8 SDK but for later ones, I backported a patch that allows the runtime to find/use OpenSSL 3.0 on FreeBSD

[LuKePicci]

I confirm the latest .NET8 from thefrank are now looking at openssl 3.0, so here is an updated recap (new parts in bold):

• the SDKs you compiled up to 8.0.100 (thefrank) and up to 8.0.101 (sec) ships with a runtime which is looking for libssl versions that:
  -- are not part of pfSense 2.7.2
  -- can be installed in FreeBSD 14.0 (including pfSense 2.7.2) using openssl package from FreeBSD 14.0
  -- are no more provided by openssl package from FreeBSD 14.1+ where they are provided by compat13x from FreeBSD 14.1+, instead.
• the SDKs sec compiled after 8.0.101 are successfully producing AppHost binaries by referencing his nuget repo, which are LD-linked to a libssl.so.12 version which:
  -- is not present in pfSense 2.7.2
  -- can be installed in FreeBSD 14.0 (including pfSense 2.7.2) using openssl30 package from FreeBSD 14.0
  -- is peacefully provided by openssl package from FreeBSD 14.1+
• the SDKs thefrank recently compiled, say after 8.0.114 I tested, are successfully producing AppHost binaries by manually importing his nugets (as per this issue title, nuget feeds from DevOps have no such packages) which are dynamically searching for usable libssl versions including:
  -- libssl.so.111 which is present by default in pfSense 2.7.2
  -- libssl.so.30 which is present by default in pfSense 2.7.2
  -- libssl.so.12 which can be installed in FreeBSD 14.0 (including pfSense 2.7.2) using openssl30 package from FreeBSD 14.0
  -- libssl.so.12 provided by openssl package from FreeBSD 14.1+

@Thefrank are you planning to also publish the new updates to the feeds? either here on GH or in DevOps?

[Thefrank]

@LuKePicci
https://dev.azure.com/IFailAt/freebsd-dotnet-runtime-nightly/_artifacts/feed/freebsd-dotnet/NuGet/Microsoft.NETCore.App.Host.freebsd-x64/versions/

That lists 8.015+ for the app host. The newest NuGets from June should up there too.

Missing items on DevOps are either a) never uploaded because I forgot or b) deleted to save space. I recently caved and decided to actually pay for space so going forward the b)-case should not happen. DevOps does not allow you to reupload items from NuGet once deleted :(

[LuKePicci]

Cool 😎 I think this will be more than fine. Living without an updated feed is a bit of a pain in Windows because WU keeps removing outdated vulnerable SDK versions. Thank you.

PS I think your README have a nice explanation about how the SDK have been built and what a native build is VS a crossbuild, however there are no info about the actual SDK output to expect. I guess the ability of loading any available libssl instead of fixing it like in @sec builds does not depend on how the SDK is being built. Is this configurable? I never used the official freebsd ports, how they behave?

[sec]

Your best bet would be either go with ports/packages - we try to keep them up-to-date (at least v9, I know v8 is little behind, but didn't had time to work on that to update ).

Cool 😎 I think this will be more than fine. Living without an updated feed is a bit of a pain in Windows because WU keeps removing outdated vulnerable SDK versions. Thank you.

What Windows have to do with FreeBSD builds? :)