Generalize-tsvs creating malformed enums causes crash

[amc-corey-cox]

Describe the bug
generalize-tsvs enum creation in some datasets causes exception crash in schema_as_dict due to enum type being string not dict.

To Reproduce
I'm not certain the specific details yet but my current data set causes the crash, I'm still hunting down the specific issue. A different data set does not have the issue.

Expected behavior
Create the schema without raising an exception and crashing.

Code snippets
This is where we're crashing.
obj = json_dumper.to_dict(schema)
if '@type' in obj:
del obj['@type']
obj['prefixes'] = {k: v['prefix_reference'] for k, v in obj.get('prefixes', {}).items()}
for k, v in obj.get('enums', {}).items():
for pv in v.get('permissible_values', {}).values():
del pv['text']

I'm going to try putting in a exception handler to give more information on what the data looks like.

[amc-corey-cox]

Here is the full trace-back.

Traceback (most recent call last):
File "/sbgenomics/workspace/dm-bip/.venv/bin/schemauto", line 8, in
sys.exit(main())
^^^^^^
File "/sbgenomics/workspace/dm-bip/.venv/lib/python3.11/site-packages/click/core.py", line 1161, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sbgenomics/workspace/dm-bip/.venv/lib/python3.11/site-packages/click/core.py", line 1082, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/sbgenomics/workspace/dm-bip/.venv/lib/python3.11/site-packages/click/core.py", line 1697, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sbgenomics/workspace/dm-bip/.venv/lib/python3.11/site-packages/click/core.py", line 1443, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sbgenomics/workspace/dm-bip/.venv/lib/python3.11/site-packages/click/core.py", line 788, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/sbgenomics/workspace/dm-bip/.venv/lib/python3.11/site-packages/schema_automator/cli.py", line 174, in generalize_tsvs

[amc-corey-cox]

Okay, so it's partially that my data is gross: Here is the exception handler to see the data.

    obj['prefixes'] = {k: v['prefix_reference'] for k, v in obj.get('prefixes', {}).items()}
#     for k, v in obj.get('enums', {}).items():
#         for pv in v.get('permissible_values', {}).values():

    for enum_name, enum_obj in obj.get('enums', {}).items():
        for pv_key, pv_val in enum_obj.get('permissible_values', {}).items():
            try:
                del pv_val['text']
            except Exception as e:
                print("\n🚨 Exception caught while processing permissible value:")
                print(f"  🔑 Enum: {enum_name}")
                print(f"  🔑 Permissible key: {pv_key}")
                print(f"  📦 Permissible value (pv_val): {pv_val} (type: {type(pv_val)})")
                import pprint
                pprint.pprint(pv_val)
                raise


#             del pv['text']
    obj = _remove_names(obj, None)
    return obj

And here is what we caught.

 Exception caught while processing permissible value:
   Enum: phv00107747_enum
   Permissible key: text
   Permissible value (pv_val): _&XX___&XX___& (type: <class 'str'>)
'_\x1e&\x02XX___\x1e&\x02XX___\x1e&\x02'

As you can see, we have control characters here so SchemaAutomator isn't sanitizing the keys before trying to create the dict. Obviously, this is a data problem but should we also be sanitizing here?