Something is triggering invisible work and sometimes re-analysis and leaking memory

[WeiN76LQh]

Version and Platform (required):

• Binary Ninja Version: 5.0.7195-dev (2abafca6)
• OS: macOS
• OS Version: 15.2
• CPU Architecture: M1

Bug Description:
Clicking around a bunch in HLIL of a DSC view seems to cause invisible work to be triggered (Binary Ninja doesn't tell you its doing anything but starts running a bunch of threads for a long perfiod of time). In the steps to reproduce below it doesn't seem to start doing re-analysis but I have had that happen before. Also it will start consuming a bunch of RAM and then not give it all back once its done.

This bug has been griefing quite a bit in larger database because the invisible work and re-analysis can take many hours and it can be triggered multiple times in the same session.

Steps To Reproduce:
Please provide all steps required to reproduce the behavior:

1. Open the DYLD Shared Cache from the IPSW for iPhone17,2 iOS 18.1.1 22B91
2. Wait for initial auto analysis to finish.
3. Load the image /System/Library/Frameworks/Foundation.framework/Foundation.
4. Wait for auto analysis to finish.
5. Go to the function at 0x182300964 called -[NSFilePresenterXPCMessenger _makePresenter:accommodateEvictionWithSubitemURL:completionHandler:] in HLIL. No particular reason for that function other than I was looking for references to objc_msgSend.
6. Click at the following locations in order: the red box (the reference to ___stack_chk_guard then the blue box (the reference to the field _queue) then the green box (the call to _objc_msgSend) and then the line below.
   
   I think the main problem here is _objc_msgSend which just clicking on it by itself seems to trigger the issue. I've found clicking around more makes things worse. So if you were to click on many different references to things and intermingle clicking on _objc_msgSend the duration of the invisible work and the memory consumption gets worse.
7. Observe Binary Ninja will start doing a lot of work in the background. bv.analysis_info will not show anything. It will also rapidly start consuming memory, some of which will be given back once it is done but a bunch will remain allocated. I observed 10s of GBs of memory being consumed after all analysis was completed just by clicking around like this.
8. Repeat the steps again and observe that again lots of background work will begin and eventually memory consumption will start to go up again.

Additional Steps:
Things get weirder

1. Do something in the Python console to trigger an analysis update which will result in extra work being done. For some reason the analysis stopped short. You'll notice in the screenshot above that the objc_msgSend calls have no parameters even though the function type is set to take 2. After analysis completes the function updates to look like this:
   
   Also these are the before and after memory usages when triggering the analysis update:

>>> get_memory_usage_info()
{'Active BasicBlock objects': 2312467, 'Active BinaryView objects': 6, 'Active Component objects': 0, 'Active FileMetadata objects': 3, 'Active FlowGraph objects': 0, 'Active Function objects': 162925, 'Active LowLevelILFunction objects': 13122, 'Active MediumLevelILFunction objects': 17400, 'Active Type objects': 3054578, 'Active TypeArchive objects': 0}
>>> get_memory_usage_info()
{'Active BasicBlock objects': 4606588, 'Active BinaryView objects': 6, 'Active Component objects': 0, 'Active FileMetadata objects': 3, 'Active FlowGraph objects': 0, 'Active Function objects': 162925, 'Active LowLevelILFunction objects': 52602, 'Active MediumLevelILFunction objects': 70040, 'Active Type objects': 3564068, 'Active TypeArchive objects': 0}

There's a huge number more IL function objects.
2. Do the clicking around again and observe the same invisible work issue
3. Call get_memory_usage_info() and see the stats have changed weirdly again:

>>> get_memory_usage_info()
{'Active BasicBlock objects': 2456535, 'Active BinaryView objects': 6, 'Active Component objects': 0, 'Active FileMetadata objects': 3, 'Active FlowGraph objects': 0, 'Active Function objects': 162925, 'Active LowLevelILFunction objects': 15453, 'Active MediumLevelILFunction objects': 20508, 'Active Type objects': 3099131, 'Active TypeArchive objects': 0}

Also this will trigger another analysis update that will do a meaningful amount of work. Memory usage after that analysis update looks like this:

>>> get_memory_usage_info()
{'Active BasicBlock objects': 2398970, 'Active BinaryView objects': 6, 'Active Component objects': 0, 'Active FileMetadata objects': 3, 'Active FlowGraph objects': 0, 'Active Function objects': 162925, 'Active LowLevelILFunction objects': 13353, 'Active MediumLevelILFunction objects': 17708, 'Active Type objects': 3104037, 'Active TypeArchive objects': 0}

4. You can keep doing this over and over; clicking around, waiting for the CPU usage to drop, triggering auto-analysis through the Python console, waiting for the analysis update to finish, rinse and repeat.

Additional Information:

• You can probably easily replicate this using any copy of the DSC.
• If it matters I have the DYLD Shared Cache in a project.
• I'm not sure the memory issue was quite as bad in prior dev versions but maybe it was.
• The more images loaded the longer the invisible work will run for.
• Would not surprise me if this is related to Types are not being applied to call sites #6458 and/or [ObjC] Certain functions stuck showing "analyzing function" spinner despite analysis showing as complete #6598

[WeiN76LQh]

Spamming get_memory_usage_info whilst Binary Ninja is doing the invisible work shows the following changes:

>>> get_memory_usage_info()
{'Active BasicBlock objects': 1776908, 'Active BinaryView objects': 2, 'Active Component objects': 0, 'Active FileMetadata objects': 1, 'Active FlowGraph objects': 0, 'Active Function objects': 53545, 'Active LowLevelILFunction objects': 11496, 'Active MediumLevelILFunction objects': 15333, 'Active Type objects': 1207217, 'Active TypeArchive objects': 0}
>>> get_memory_usage_info()
{'Active BasicBlock objects': 1975293, 'Active BinaryView objects': 2, 'Active Component objects': 0, 'Active FileMetadata objects': 1, 'Active FlowGraph objects': 0, 'Active Function objects': 53545, 'Active LowLevelILFunction objects': 13166, 'Active MediumLevelILFunction objects': 17570, 'Active Type objects': 1236381, 'Active TypeArchive objects': 0}
>>> get_memory_usage_info()
{'Active BasicBlock objects': 2118621, 'Active BinaryView objects': 2, 'Active Component objects': 0, 'Active FileMetadata objects': 1, 'Active FlowGraph objects': 0, 'Active Function objects': 53545, 'Active LowLevelILFunction objects': 15197, 'Active MediumLevelILFunction objects': 20298, 'Active Type objects': 1262898, 'Active TypeArchive objects': 0}
>>> get_memory_usage_info()
{'Active BasicBlock objects': 2256890, 'Active BinaryView objects': 2, 'Active Component objects': 0, 'Active FileMetadata objects': 1, 'Active FlowGraph objects': 0, 'Active Function objects': 53545, 'Active LowLevelILFunction objects': 17895, 'Active MediumLevelILFunction objects': 23845, 'Active Type objects': 1290231, 'Active TypeArchive objects': 0}
>>> get_memory_usage_info()
{'Active BasicBlock objects': 2404404, 'Active BinaryView objects': 2, 'Active Component objects': 0, 'Active FileMetadata objects': 1, 'Active FlowGraph objects': 0, 'Active Function objects': 53545, 'Active LowLevelILFunction objects': 20382, 'Active MediumLevelILFunction objects': 27194, 'Active Type objects': 1315600, 'Active TypeArchive objects': 0}
>>> get_memory_usage_info()
{'Active BasicBlock objects': 2528187, 'Active BinaryView objects': 2, 'Active Component objects': 0, 'Active FileMetadata objects': 1, 'Active FlowGraph objects': 0, 'Active Function objects': 53545, 'Active LowLevelILFunction objects': 22289, 'Active MediumLevelILFunction objects': 29708, 'Active Type objects': 1335070, 'Active TypeArchive objects': 0}
>>> get_memory_usage_info()
{'Active BasicBlock objects': 2626079, 'Active BinaryView objects': 2, 'Active Component objects': 0, 'Active FileMetadata objects': 1, 'Active FlowGraph objects': 0, 'Active Function objects': 53545, 'Active LowLevelILFunction objects': 24109, 'Active MediumLevelILFunction objects': 32137, 'Active Type objects': 1349765, 'Active TypeArchive objects': 0}
>>> get_memory_usage_info()
{'Active BasicBlock objects': 2694465, 'Active BinaryView objects': 2, 'Active Component objects': 0, 'Active FileMetadata objects': 1, 'Active FlowGraph objects': 0, 'Active Function objects': 53545, 'Active LowLevelILFunction objects': 25171, 'Active MediumLevelILFunction objects': 33570, 'Active Type objects': 1362670, 'Active TypeArchive objects': 0}
>>> get_memory_usage_info()
{'Active BasicBlock objects': 2767314, 'Active BinaryView objects': 2, 'Active Component objects': 0, 'Active FileMetadata objects': 1, 'Active FlowGraph objects': 0, 'Active Function objects': 53545, 'Active LowLevelILFunction objects': 26568, 'Active MediumLevelILFunction objects': 35457, 'Active Type objects': 1376309, 'Active TypeArchive objects': 0}
>>> get_memory_usage_info()
{'Active BasicBlock objects': 2890779, 'Active BinaryView objects': 2, 'Active Component objects': 0, 'Active FileMetadata objects': 1, 'Active FlowGraph objects': 0, 'Active Function objects': 53545, 'Active LowLevelILFunction objects': 28400, 'Active MediumLevelILFunction objects': 37864, 'Active Type objects': 1401737, 'Active TypeArchive objects': 0}

[Pixeluted]

Likely not related, but whenever binary is done analyzed and I choose to rebase it, in logs I get message "Analysis update was aborted" even though it was already completed.

[bpotchik]

I've recently updated the Abort handling. Aborts are now recoverable and we also log a message indicating completion of the abort action. However, we have always aborted any ongoing analysis before rebasing in the UI.

[bpotchik]

Also, I've tried to repro this issue and so far I've not had much luck. I even dragged around in feature map with linear view open to try and force as much scrolling/navigation and analysis as possible. No luck with that approach either.

[WeiN76LQh]

Also, I've tried to repro this issue and so far I've not had much luck. I even dragged around in feature map with linear view open to try and force as much scrolling/navigation and analysis as possible. No luck with that approach either.

Hmm, it was reproing 100% of the time for me. Maybe its something to do with my settings?

I have feature map disabled and cross references set to no limit. Maybe its one of them that makes the difference?

[WeiN76LQh]

I reset my cross references to the default amount (1000) and the issue still triggers but the amount of work its doing in the background is much lower and basically done immediately. I'm still seeing all the other stuff with memory usage and analysis each time after clicking around.

[bpotchik]

Yeah good point about the xrefs window. That actually ends up requesting HLIL for functions so that it can render IL in the preview. There might be something broken in that processing. Ideally, that logic should only request IL for functions that are in the viewport range.

[WeiN76LQh]

I'm starting to think this is actually 2 seperate issues.

1. A large number of cross references can cause Binary Ninja to spend a long time rendering IL previews which blocks other work that it might do like auto analysis.
2. Initial auto-analysis is not completing properly.