Incompatibility with Avro >=1.9.0 (upgrade to Avro 1.11.3)

[Sage-Pierce]

Avro version 1.9.0 was released last month (https://mvnrepository.com/artifact/org.apache.avro/avro) and my team has run in to compatibility issues between jackson-dataformat-avro and this latest release.

In particular, it looks to be the case that Avro has switched from Codehaus Jackson to FasterXML Jackson, which results in NoSuchMethodError thrown when accessing utility methods.

The following test exposes the issue:

    @Test
    public void beanShouldBeSerializableAsAvroBinaryAndDeserializable() throws Exception {
        TestBean testBean = new TestBean();
        testBean.setData("Data");

        AvroMapper avroMapper = new AvroMapper();

        AvroSchema avroSchema = avroMapper.schemaFor(TestBean.class);

        ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
        avroMapper.writer().with(avroSchema).writeValue(outputStream, testBean);

        byte[] serialized = outputStream.toByteArray();

        assertNotNull(serialized);
        assertTrue(serialized.length > 0);

        TestBean deserialized = avroMapper.readerFor(TestBean.class).with(avroSchema).readValue(serialized);

        assertEquals(testBean, deserialized);
    }

    public static final class TestBean {

        private String data;

        @Override
        public boolean equals(Object o) {
            if (this == o)
                return true;
            if (o == null || getClass() != o.getClass())
                return false;
            TestBean testBean = (TestBean) o;
            return Objects.equals(data, testBean.data);
        }

        @Override
        public int hashCode() {
            return Objects.hash(data);
        }

        public String getData() {
            return data;
        }

        public void setData(String data) {
            this.data = data;
        }
    }

This test passes on v1.8.2 of Avro, but fails with v1.9.0 on the following:

java.lang.NoSuchMethodError: org.apache.avro.util.internal.JacksonUtils.toObject(Lorg/codehaus/jackson/JsonNode;)Ljava/lang/Object;

	at com.fasterxml.jackson.dataformat.avro.schema.RecordVisitor.schemaFieldForWriter(RecordVisitor.java:192)
	at com.fasterxml.jackson.dataformat.avro.schema.RecordVisitor.optionalProperty(RecordVisitor.java:121)
	at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.depositSchemaProperty(BeanPropertyWriter.java:839)
	at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.acceptJsonFormatVisitor(BeanSerializerBase.java:861)
	at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.acceptJsonFormatVisitor(DefaultSerializerProvider.java:566)
	at com.fasterxml.jackson.databind.ObjectMapper.acceptJsonFormatVisitor(ObjectMapper.java:3874)
	at com.fasterxml.jackson.databind.ObjectMapper.acceptJsonFormatVisitor(ObjectMapper.java:3853)
	at com.fasterxml.jackson.dataformat.avro.AvroMapper.schemaFor(AvroMapper.java:96)

Note that we are using com.fasterxml.jackson.dataformat:jackson-dataformat-avro:2.9.9

[cowtowncoder]

Hmmh. So the problem is that API of Apache Avro library is incompatible between 1.8.x and 1.9 -- one exposes Jackson 1.x JsonNode (1.8 and before), latter Jackson 2.x JsonNode. This is major incompatibility and something that Jackson Avro module probably can do nothing about, in the sense that if you use Jackson Avro Module 2.9 or earlier, you MUST use Apache Avro module version 1.8.x.
I can change this for Jackson 2.10 to upgrade to require 1.9.0 (with similar limitation that earlier versions can not be used).

Thank you for reporting the issue: I think it is good that apache avro library has upgraded to Jackson 2.x, but it is bit problematic as types are exposed in API on short term.

[salt-tony]

As of Apache Avro 1.9.0 they updated to depend on Jackson 2.9.7, but the problem is actually in the com.fasterxml.jackson.dataformat.avro.schema.RecordVisitor class which is depending on "org.apache.avro.util.internal.JacksonUtils", which used to reference the old org.codehaus.jackson.JsonNode.
The problem is the Jackson RecordVisitor class should not be relying on an 'internal' API from org.apache.avro.util.internal, and certainly should not be referring to classes from old versions of Jackson.

[cowtowncoder]

@salt-tony yes, I understand the mechanism pretty well and attempted to summarize above. But I am not sure of the point you are trying to make here?

My interest is not in figuring out what to blame (I agree that relying on internal parts of another library is not a good idea) but to figure out how to try to contain further version incompatibilities. Since Jackson versions up to 2.9.x will be bound to dependencies of Avro lib 1.8.x, it is not possible to upgrade to later Avro library there; and anything using Jackson 2.9.x Avro module can not use apache Avro 1.9.x or above.
But there is further problem that if with 2.10 we upgrade dependency here, it will then require upgrade of apache avro library for any mixed use cases.
Alternatively it would be possible keep status quo for Jackson 2.x and not upgrade to apache avro 1.9.x and beyond; and wait for Jackson 3.0 to make the jump.

Note, too, that Avro module is only relying on Jackson 1.x types because Avro 1.8.x exposes default values using 1.x JsonNode values; and internal JacksonUtils class seems to have been referenced for necessary conversions (I did not write code in question and failed to notice reference on code review); but even without it, reference to 1.x types is unavoidable before new version.

[cowtowncoder]

After investigating this for a bit now I am much less hopeful in figuring out a working way to upgrade Avro module to work with Apache Avro lib 1.9.x. Part of the problem is that the complicated way some of unit tests work, it is difficult to see what actually breaks: I can get code to compile fine, but many things do not seem to work the way they used to in avro lib.

So there is a distinct possibility that Avro module will require pre-1.9 version of Apache Avro lib, at least for 2.10.

[baharclerode]

Ohhhh man Avro has gone and created a right nasty mess. I've done a bit of digging, and there's some good news and some bad news:

1. Good: jackson-dataformat-avro can be made binary compatible with both 1.8.X and 1.9.X without too much work. We were leveraging some Avro helper utils to avoid duplicating the code and increase the likelihood of remaining api-compatible as things evolved. These just need to be replaced with fresh code that lives in jackson-dataformat-avro. Slightly increases the risk of api-compatibility (specifically around how default values are encoded and interpreted in schemas), but all of the regression tests still pass with 1.8.X and I expect the use-cases to be limited to a domain of types that shouldn't have compatability issues.

2. Bad: Avro changed the way they encode type information in schemas. In 1.8.X, nested classes were encoded as

    {"type":"record","namespace":"package.OuterClass$","name":"InnerClass", ....}
   

   Which was really helpful, because you could tell the difference between a nested class and a top-level class, and that made runtime type discovery relatively easy because these could be unambiguously concatenated and then fed into Class.forName() to find the type during deserialization. However, with 1.9.X, they've started serializing nested classes sans the trailing $ in the namespace:

    {"type":"record","namespace":"package.OuterClass","name":"InnerClass", ....}
   

   Which we can no longer simply feed into Class.forName(), and now have to do a "Guess and check" by probing both the non-nested and nested combinations of namespace+name. This makes 1.9.X not forwards compatible with 1.8.X over the wire.

The unit tests catch this pretty handily (all of the "Apache(1.9.X) -> Jackson w/ Apache(1.9.X) schema" round-trip tests start failing, while their counterparts do not), and once fixed reveal a handful of other edge cases that need to be looked at. It appears that most of the other failing unit tests are similarly related to this issue, in the forms of Enums not resolving correctly or Unions not resolving correctly because we had relied upon Avro helpers to resolve these situations in a compatible manner; It looks like they will have to be rewritten and kept in sync instead.

In general, I recommend staying away from 1.9.X unless you're prepared to upgrade your whole ecosystem (or at least every consumer) all at once because the schema format is not forwards compatible. I think these can all be worked around/fixed/patched so that Jackson can read from both versions and write from both versions without any major overhauls/work, but we do need to clone&own some pieces we were leveraging from Avro.

[cowtowncoder]

@baharclerode Thank you for digging bit deeper! Your comments make more sense to me, and I think that at least for 2.10 we'll better stay on 1.8.x. Even if it means we can't get rid of Jackson 1.x dependency.

I would like to see the minor cleanup wrt replacing use of internal.JacksonUtils, as that makes needed changes slightly simpler.

[baharclerode]

I'll see if I can get two PRs together:

1. Remove direct dependency on Jackson 1.X / internal.JacksonUtils
2. Support deserializing from Apache 1.9.X schemas

The irony is I'm pretty sure this behavior was changed because the $ in namespaces isn't allowed by the spec, but the java implementation used it accidentally and it was never caught because until now, namespaces were not validated by the java implementation (AVRO-2143); However, the java implementation will still generate invalid namespaces if you doubly-nest a class, i.e.:

package example;
public class Foo {
    public static class Bar {
        public static class Baz {}
    }
}

will still have a name of Baz and a namespace of example.Foo$Bar since they only fixed it for one level deep and didn't handle anything beyond that.

[cowtowncoder]

Heh. Oh boy. But yes, +1 for those PRs. I'll also need to find time to work on other submitted PRs, bugs against Avro module, to get things shipshape for 2.10.

[dani-e-l]

Hi cowtowncoder
are you working on this?

Avro 1.8.x has vulnerabilities and we need to update avro or remove jackson-dataformats-binary from our projects.
Is there any possibility to support 1.9.x in short term?

[cowtowncoder]

No, I do not plan to work on this. Avro 1.9.x seems fundamentally problematic wrt its dependencies.

[ZachManno]

@DanielCharczynski We fixed this vulnerability by using
tech.allegro.schema.json2avro converter version 0.2.9