Move the BSTR trailbyte tracking out of the sync block

[jkoritzinsky]

This corner case becomes less used with time, and it increases the size of every syncblock even though it's only used with strings.

This should also help reduce the number of syncblks we create for legacy applications that do use this feature a lot.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/interop-contrib
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull Request Overview

This pull request removes the legacy BSTR trail byte support from the CoreCLR, migrating it from a sync block-based implementation to a ConditionalWeakTable-based approach in managed code. This change aims to reduce memory overhead and simplify the string object implementation.

Key Changes

• Removed trail byte field and related methods from SyncBlock
• Removed trail byte methods from StringObject (HasTrailByte, GetTrailByte, SetTrailByte)
• Moved trail byte storage to managed code using ConditionalWeakTable in BSTRMarshaler
• Updated BSTR marshaling code to call managed methods instead of native interop

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/coreclr/vm/syncblk.h	Removed m_BSTRTrailByte field and related methods from SyncBlock
src/coreclr/vm/object.h	Removed trail byte methods and NewString overload from StringObject
src/coreclr/vm/object.cpp	Removed trail byte method implementations and NewString overload
src/coreclr/vm/stubhelpers.h	Removed native trail byte function declarations
src/coreclr/vm/stubhelpers.cpp	Removed native trail byte function implementations
src/coreclr/vm/qcallentrypoints.cpp	Removed QCall entry point for SetStringTrailByte
src/coreclr/vm/ecalllist.h	Removed FCall entry for TryGetStringTrailByte
src/coreclr/vm/olevariant.cpp	Updated to call managed methods for trail byte operations and fixed GC mode contracts
src/coreclr/vm/metasig.h	Added method signatures for new managed trail byte methods
src/coreclr/vm/corelib.h	Added method definitions for SetTrailByte and TryGetTrailByte
src/coreclr/vm/callhelpers.h	Added INT8_TO_ARGHOLDER macro for byte argument conversion
src/coreclr/System.Private.CoreLib/src/System/StubHelpers.cs	Implemented ConditionalWeakTable-based trail byte storage in BSTRMarshaler

Comments suppressed due to low confidence (1)

src/coreclr/vm/object.cpp:826

• These macros and functions (SPECIAL_STRING_VB_BYTE_ARRAY, MARKS_VB_BYTE_ARRAY, MAKE_VB_TRAIL_BYTE, GET_VB_TRAIL_BYTE) appear to be unused after the removal of trail byte functionality from StringObject and SyncBlock. Since the trail byte feature has been moved to managed code using ConditionalWeakTable, these definitions should be removed to avoid confusion and reduce technical debt.

//The special string helpers are used as flag bits for weird strings that have bytes
//after the terminating 0.  The only case where we use this right now is the VB BSTR as
//byte array which is described in MakeStringAsByteArrayFromBytes.
#define SPECIAL_STRING_VB_BYTE_ARRAY 0x100

FORCEINLINE BOOL MARKS_VB_BYTE_ARRAY(WCHAR x)
{
    return static_cast<BOOL>(x & SPECIAL_STRING_VB_BYTE_ARRAY);
}

FORCEINLINE WCHAR MAKE_VB_TRAIL_BYTE(BYTE x)
{
    return static_cast<WCHAR>(x) | SPECIAL_STRING_VB_BYTE_ARRAY;
}

FORCEINLINE BYTE GET_VB_TRAIL_BYTE(WCHAR x)
{
    return static_cast<BYTE>(x & 0xFF);
}

[AaronRobinsonMSFT]

Do we have a test for this already?

[jkoritzinsky]

I don't see one, I'll go add one.

[jkotas]

Thanks

[jkoritzinsky]

/ba-g android timeout