Error deserializing messages when using custom Akka serializers

[vadymberkut]

Version Information
Akka 1.4.31

Akka.Quartz.Actor 1.4.31`

Describe the bug
When using Akka custom message serializer and QuartzPersistentActor it fails to deserialize message from binary in QuartzPersistentJob.Execute, as it searches for serializer for typeof(object) which is Akka.Serialization.NewtonSoftJsonSerializer by default. However, initially, the message was serialized with a custom serializer in QuartzPersistentJob.CreateBuilderWithData.

I have custom serializers defined in Akka to serialize messages using protobuf. I inherit my serializers from SerializerWithStringManifest and use manifest that is based on the message type. When doing

var message = sys.Serialization.FindSerializerForType(typeof(object)).FromBinary(messageBytes, typeof(object));

in QuartzPersistentJob.Execute, it can't find proper serializer for type typeof(object) and returns Akka.Serialization.NewtonSoftJsonSerializer (default) which can't deserialize, because bytes are not json bytes, but protobuf bytes.

I tried to implement a custom serializer to handle deserializaion of typeof(object), but there is no way to figure out which exact type the bytes belong to, when the second argument type is always object type:

public class ObjectSerializer : Akka.Serialization.Serializer
{
  // ....
  public override object FromBinary(byte[] bytes, Type type) 
  {
     // typeof(type) == typeof(object)
     // how to figure out which class is encoded in bytes?
  }
}

FromBinary is called from here:

To Reproduce
Steps to reproduce the behavior:

1. Register a custom Akka binary serializer that is not JSON serializer.
2. Enable Quartz logs.
3. Create QuartzPersistentActor and schedule a job via it with a message that is serialized by serializer from 1..
4. When job is triggered you shoud see an error like this
   

Expected behavior
Pass correct Type type to FromBinary when executing a job, not typeof(object). Or introduce any other solution that will fix the issue.

Users of Akka.Quartz.Actor must be able to use any custom serializers.

Actual behavior
Serialized messages can't be deserialized back.

Environment
Windows 10, .NET 5

[Aaronontheweb]

cc @object I think the fix you made was just to make sure the System.Object serializer was used consistently on both ends - supporting custom serializers is still a problem, no?

[object]

@Aaronontheweb it should not be a problem but requires storing additional job metadata, specifically type information that can be passed as a manifest string.

[object]

In my pr I used object on both sides to provide minimal code changes to fix the problem. But the best would be to honor the actual serializer that is used for the given message type.

[Aaronontheweb]

yeah that's what I thought - ok, so this still needs fixing then.