Unmarshal remaining JSON after performing custom unmarshalling

Question

I have a JSON object That contains an implementation of an interface within it. I'm attempting to take that JSON and marshal it into a struct whilst creating the implementation of the interface.

I've managed to get it to implement the interface with a custom JSON unmarshal function however I'm struggling to piece together how to then marshal the rest of the fields

I've created an example in the Go playground

https://play.golang.org/p/ztF7H7etdjM

My JSON being passed into my application is

{

   "address":"1FYuJ4MsVmpzPoFJ6svJMJfygn91Eubid9",
   "nonce":13,
   "network_id":"qadre.demo.balance",
   "challenge":"f2b19e71876c087e681fc092ea3a34d5680bbfe772e40883563e1d5513bb593f",
   "type":"verifying_key",
   "verifying_key":{
      "verifying_key":"3b6a27bcceb6a42d62a3a8d02a6f0d73653215771de243a63ac048a18b59da29",
      "fqdn":"huski.service.key"
   },
   "signature":"a3bf8ee202a508d5a5632f50b140b70b7095d8836493dc7ac4159f6f3350280078b3a58b2162a240bc8c7485894554976a9c7b5d279d3f5bf49fec950f024e02",
   "fqdn":"huski.service.SingleKeyProof"
}

I've attempted to do a json.Unmarshal and pass in a new struct for the remaining fields however it seems to put me in an infinite loop, my application hangs and then crashes

The best solution I've come up with so far is to marshal the JSON into a `map[string]interface{} and do each field separately, this feels very clunky though

var m map[string]interface{}
if err := json.Unmarshal(data, &m); err != nil {
    return err
}

ad, ok := m["address"]
if ok {
    s.Address = ad.(string)
}
fqdn, ok := m["fqdn"]
if ok {
    s.FQDN = fqdn.(string)
}
n, ok := m["nonce"]
if ok {
    s.Nonce = int64(n.(float64))
}
c, ok := m["challenge"]
if ok {
    s.Challenge = []byte(c.(string))
}
network, ok := m["network_id"]
if ok {
    s.NetworkID = network.(string)
}
sig, ok := m["signature"]
if ok {
    s.Signature = []byte(sig.(string))
}

Answer 1

Looking at what you've done in your custom unmarshalling function, you seem to be passing in a map with the name of fields as index, and the reflect.Type you want to unmarshal said value into. That, to me, suggests that the keys might be different for different payloads, but that each key has a distinct type associated with it. You can perfectly handle data like this with a simple wrapper type:

type WrappedSingleKey struct {
    FQDN         string          `json:"fqdn"`
    Address      string          `json:"address"`
    Nonce        int64           `json:"nonce"`
    Challenge    []byte          `json:"challenge"`
    NetworkID    string          `json:"network_id"`
    Type         string          `json:"type"`
    VerifyingKey json.RawMessage `json:"verifying_key"`
    OtherKey     json.RawMessage `json:"other_key"`
    Signature    []byte          `json:"signature"`
}

type SingleKey struct {
    FQDN         string     `json:"fqdn"`
    Address      string     `json:"address"`
    Nonce        int64      `json:"nonce"`
    Challenge    []byte     `json:"challenge"`
    NetworkID    string     `json:"network_id"`
    Type         string     `json:"type"`
    VerifyingKey *PublicKey `json:"verifying_key,omitempty"`
    OtherType    *OtherKey  `json:"other_key,omitempty"`
    Signature    []byte     `json:"signature"`
}

So I've changed the type of your VerifyingKey field to a json.RawMessage. That's basically telling json.Unmarshal to leave that as raw JSON input. For every custom/optional field, add a corresponding RawMessage field.

In the unwrapped type, I've changed VerifyingKey to a pointer and added the omitempty bit to the tag. That's just to accomodate mutliple types, and not have to worry about custom marshalling to avoid empty fields, like the included OtherType field I have. To get what you need, then:

func (s *SingleKey) UnmarshalJSON(data []byte) error {
    w := WrappedSingleKey{} // create wrapped instance
    if err := json.Unmarshal(data, &w); err != nil {
        return err
    }
    switch w.Type {
    case "verifying_key":
       var pk PublicKey
       if err := json.Unmarshal([]byte(w.VerifyingKey), &pk); err != nil {
           return err
       }
       s.VerifyingKey = &pk // assign
    case "other_key":
        var ok OtherKey
        if err := json.Unmarshal([]byte(w.OtherKey), &ok); err != nil {
            return err
        }
        s.OtherKey = &ok
    }
    // copy over the fields that didn't require anything special
    s.FQDN = w.FQDN
    s.Address = w.Address
}

This is a fairly simple approach, does away with the reflection, tons of functions, and is quite commonly used. It's something that lends itself quite well to code generation, too. The individual assignment of the fields is a bit tedious, though. You might think that you can solve that by embedding the SingleKey type into the wrapper, but be careful: this will recursively call your custom unmarshaller function.

You could, for example, update all the fields in the WRapped type to be pointers, and have them point to fields on your actual type. That does away with the manual copying of fields... It's up to you, really.

Note

I didn't test this code, just wrote it as I went along. It's something I've used in the past, and I believe what I wrote here should work, but no guarantees (as in: you might need to debug it a bit)

Answer 2

The reason your code gets into an infinite loop when you try to unmarshal the rest of the fields is because, I presume, the implementation of UnmarshalJSON after its done unmarshaling the verifying key, calls json.Unmarshal with the receiver, which in turn calls the UnmarshalJSON method on the receiver and so they invoke each other ad infinitum.

What you can do is to create a temporary type using the existing type as its definition, this will "keep the structure" but "drop the methods", then unmarshal the rest of the fields into an instance of the new type, and, after unmarshal is done, convert the instance to the original type and assign that to the receiver.

While this fixes the infinite loop, it also re-introduces the original problem of json.Unmarshal not being able to unmarshal into a non-empty interface type. To fix that you can embed the new type in another temporary struct that has a field with the same json tag as the problematic field which will cause it to be "overshadowed" while json.Unmarshal is doing its work.

type SingleKey struct {
    FQDN         string    `json:"fqdn"`
    Address      string    `json:"address"`
    Nonce        int64     `json:"nonce"`
    Challenge    []byte    `json:"challenge"`
    NetworkID    string    `json:"network_id"`
    Type         string    `json:"type"`
    VerifyingKey PublicKey `json:"verifying_key"`
    Signature    []byte    `json:"signature"`
}

func (s *SingleKey) UnmarshalJSON(data []byte) error {
    type _SingleKey SingleKey
    var temp struct {
        RawKey json.RawMessage `json:"verifying_key"`
        _SingleKey
    }
    if err := json.Unmarshal(data, &temp); err != nil {
        return err
    }
    *s = SingleKey(temp._SingleKey)

    switch s.Type {
    case "verifying_key":
        s.VerifyingKey = &PublicKeyImpl{}
    // other cases ...
    }

    return json.Unmarshal([]byte(temp.RawKey), s.VerifyingKey)
}

https://play.golang.org/p/L3gdQZF47uN