Framework Native/Universal build for Apple Silicon (x86_64 & arm64)

[Ishaanahuja7]

Summary

Support Universal build for macOS (x86_64 and arm64). If Universal cannot be done, then provide arm64 installer pkg file in addition to x86_64.

I did a complete dev environment setup of metasploit-framework described here on my M1 Max and it was successful. I don't know if there is a script for running tests to check everything but seems to be working with initial testing of like msfvenom, msfconsole session from macOS -> Windows 10, and msfdb.

Motivation

Running over Rosetta is really slow even on an M1 Max. Providing Universal or arm64 binaries would definitely speed up the framework.

[sempervictus]

It doesn't look like your platform even fully supports Ruby yet. Take a look at the Gemfile.lock to see all of the external dependencies Metasploit uses and then try to build the native extensions for all gems which use them for your platform. Until that works, the maintainers cannot create such a native package.
Strongly suggest using Linux for offsec purposes - Macs dont support all of the network protocols used by Framework and this sort of thing does not really pair well with offensive tooling.

[Ishaanahuja7]

It doesn't look like your platform even fully supports Ruby yet

In the same issue as above it says it was fixed in Ruby 3.1.3 and anyways metasploit-framework is run on Ruby 3.0.5 as suggested by the .ruby-version file which works fine on Apple Silicon.

Take a look at the Gemfile.lock to see all of the external dependencies Metasploit uses and then try to build the native extensions for all gems which use them for your platform. Until that works, the maintainers cannot create such a native package.

So I did bundle install with Ruby 3.0.5 and everything installed without any warnings or errors. I don't know if thats what you mean by that statement.

Strongly suggest using Linux for offsec purposes - Macs dont support all of the network protocols used by Framework and this sort of thing does not really pair well with offensive tooling.

I agree with Linux but I try to use my mac as an All in One kind of a solution. Anyways, so I took a close look at the link provided and its contents. I don't think we can come to the conclusion that Apple is scanning files and sending data back to its servers just by a call to an API. mediaanalysisd is probably downloading classifier or other AI models or something else but it doesn't suggest that its scanning files and sending data back in any way. I could be wrong but the author of that article also doesn't back it up with wireshark data which might suggest that its true. If you look in the comments, a lot of people who have done a little more digging also don't think thats true.

[Ishaanahuja7]

Any timeline for this yet?

[sempervictus]

So I did bundle install with Ruby 3.0.5 and everything installed without any warnings or errors. I don't know if thats what you mean by that statement.

Does that mean you were using the Ruby interepreter native to your architecture with native extensions built, or that you used the x86_64 ruby binary (and relevant environment) through rosetta? Might be a good idea to force-disable the emulation to verify native functionality.

[Ishaanahuja7]

Yes, its native. I installed Ruby 3.0.5 from rbenv, bundle install worked fine. Tried it with Homebrew Ruby (ruby@3.0) as well, no warnings or errors. Rosetta2 is not being used since both Ruby installations were compiled for arm64 arch which is native for Apple Silicon.

[sempervictus]

If that works, then at least for the time being, you should be able to run Metasploit from source on native binaries executing the Ruby scripts. Rather surprised that everything compiled as intended both due to the architecture and platform quirks - thats quite a win.

[Ishaanahuja7]

If that works, then at least for the time being, you should be able to run Metasploit from source on native binaries executing the Ruby scripts.

Thats what I have been doing since a year but wanted to see a pkg installer file for easy maintenance. Are there any issues adding an M1 build in your macOS automation for building pkg files?

Rather surprised that everything compiled as intended both due to the architecture and platform quirks - thats quite a win.

That is true and here is the log for environment, build and run:

Environment

ishaanahuja@Ishaan-Ahujas-MacBook-Pro ~/testing % git clone --recursive https://github.com/rapid7/metasploit-framework
Cloning into 'metasploit-framework'...
remote: Enumerating objects: 644428, done.
remote: Counting objects: 100% (3178/3178), done.
remote: Compressing objects: 100% (632/632), done.
remote: Total 644428 (delta 2462), reused 3103 (delta 2420), pack-reused 641250
Receiving objects: 100% (644428/644428), 803.30 MiB | 6.15 MiB/s, done.
Resolving deltas: 100% (475612/475612), done.
Updating files: 100% (12459/12459), done.
Submodule 'external/source/ReflectiveDLLInjection' (https://github.com/rapid7/ReflectiveDLLInjection.git) registered for path 'external/source/ReflectiveDLLInjection'
Cloning into '/Users/ishaanahuja/testing/metasploit-framework/external/source/ReflectiveDLLInjection'...
remote: Enumerating objects: 307, done.        
remote: Counting objects: 100% (94/94), done.        
remote: Compressing objects: 100% (17/17), done.        
remote: Total 307 (delta 84), reused 77 (delta 77), pack-reused 213        
Receiving objects: 100% (307/307), 333.95 KiB | 470.00 KiB/s, done.
Resolving deltas: 100% (181/181), done.
Submodule path 'external/source/ReflectiveDLLInjection': checked out '6bad4c49327ad3b7d9cce6e280d034b76dbec928'
ishaanahuja@Ishaan-Ahujas-MacBook-Pro ~/testing % cd metasploit-framework 
ishaanahuja@Ishaan-Ahujas-MacBook-Pro ~/testing/metasploit-framework % ruby -v                      
ruby 3.0.5p211 (2022-11-24 revision ba5cf0f7c5) [arm64-darwin22]
ishaanahuja@Ishaan-Ahujas-MacBook-Pro ~/testing/metasploit-framework % file /opt/homebrew/bin/ruby                                                          
/opt/homebrew/bin/ruby: Mach-O 64-bit executable arm64

Build

ishaanahuja@Ishaan-Ahujas-MacBook-Pro ~/testing/metasploit-framework % gem install bundler                                                 
Fetching bundler-2.4.7.gem
Successfully installed bundler-2.4.7
Parsing documentation for bundler-2.4.7
Installing ri documentation for bundler-2.4.7
Done installing documentation for bundler after 0 seconds
1 gem installed
ishaanahuja@Ishaan-Ahujas-MacBook-Pro ~/testing/metasploit-framework % bundle install             
Bundler 2.4.7 is running, but your lockfile was generated with 2.1.4. Installing Bundler 2.1.4 and restarting using that version.
Fetching gem metadata from https://rubygems.org/.
Fetching bundler 2.1.4
Installing bundler 2.1.4
Fetching gem metadata from https://rubygems.org/.........
Fetching rake 13.0.6
Installing rake 13.0.6
Fetching Ascii85 1.1.0
Installing Ascii85 1.1.0
Fetching concurrent-ruby 1.2.0
Installing concurrent-ruby 1.2.0
Fetching i18n 1.12.0
Installing i18n 1.12.0
Fetching minitest 5.17.0
Installing minitest 5.17.0
Fetching tzinfo 2.0.6
Installing tzinfo 2.0.6
Fetching activesupport 7.0.4.2
Installing activesupport 7.0.4.2
Fetching builder 3.2.4
Installing builder 3.2.4
Fetching erubi 1.12.0
Installing erubi 1.12.0
Fetching mini_portile2 2.8.1
Installing mini_portile2 2.8.1
Fetching racc 1.6.2
Installing racc 1.6.2 with native extensions
Fetching nokogiri 1.14.1 (arm64-darwin)
Installing nokogiri 1.14.1 (arm64-darwin)
Fetching rails-dom-testing 2.0.3
Installing rails-dom-testing 2.0.3
Fetching crass 1.0.6
Installing crass 1.0.6
Fetching loofah 2.19.1
Installing loofah 2.19.1
Fetching rails-html-sanitizer 1.5.0
Installing rails-html-sanitizer 1.5.0
Fetching actionview 7.0.4.2
Installing actionview 7.0.4.2
Fetching rack 2.2.6.2
Installing rack 2.2.6.2
Fetching rack-test 2.0.2
Installing rack-test 2.0.2
Fetching actionpack 7.0.4.2
Installing actionpack 7.0.4.2
Fetching activemodel 7.0.4.2
Installing activemodel 7.0.4.2
Fetching activerecord 7.0.4.2
Installing activerecord 7.0.4.2
Fetching public_suffix 5.0.1
Installing public_suffix 5.0.1
Fetching addressable 2.8.1
Installing addressable 2.8.1
Fetching afm 0.2.2
Installing afm 0.2.2
Fetching arel-helpers 2.14.0
Installing arel-helpers 2.14.0
Fetching ast 2.4.2
Installing ast 2.4.2
Fetching aws-eventstream 1.2.0
Installing aws-eventstream 1.2.0
Fetching aws-partitions 1.707.0
Installing aws-partitions 1.707.0
Fetching aws-sigv4 1.5.2
Installing aws-sigv4 1.5.2
Fetching jmespath 1.6.2
Installing jmespath 1.6.2
Fetching aws-sdk-core 3.170.0
Installing aws-sdk-core 3.170.0
Fetching aws-sdk-ec2 1.364.0
Installing aws-sdk-ec2 1.364.0
Fetching aws-sdk-iam 1.75.0
Installing aws-sdk-iam 1.75.0
Fetching aws-sdk-kms 1.62.0
Installing aws-sdk-kms 1.62.0
Fetching aws-sdk-s3 1.119.0
Installing aws-sdk-s3 1.119.0
Fetching bcrypt 3.1.18
Installing bcrypt 3.1.18 with native extensions
Fetching bcrypt_pbkdf 1.1.0
Installing bcrypt_pbkdf 1.1.0 with native extensions
Fetching bindata 2.4.15
Installing bindata 2.4.15
Fetching bson 4.15.0
Installing bson 4.15.0 with native extensions
Using bundler 2.1.4
Fetching byebug 11.1.3
Installing byebug 11.1.3 with native extensions
Fetching coderay 1.1.3
Installing coderay 1.1.3
Fetching cookiejar 0.3.3
Installing cookiejar 0.3.3
Fetching daemons 1.4.1
Installing daemons 1.4.1
Fetching io-console 0.6.0
Installing io-console 0.6.0 with native extensions
Fetching reline 0.3.2
Installing reline 0.3.2
Fetching irb 1.6.2
Installing irb 1.6.2
Fetching debug 1.7.1
Installing debug 1.7.1 with native extensions
Fetching diff-lcs 1.5.0
Installing diff-lcs 1.5.0
Fetching unf_ext 0.0.8.2
Installing unf_ext 0.0.8.2 with native extensions
Fetching unf 0.1.4
Installing unf 0.1.4
Fetching simpleidn 0.2.1
Installing simpleidn 0.2.1
Fetching dnsruby 1.61.9
Installing dnsruby 1.61.9
Fetching docile 1.4.0
Installing docile 1.4.0
Fetching domain_name 0.5.20190701
Installing domain_name 0.5.20190701
Fetching ed25519 1.3.0
Installing ed25519 1.3.0 with native extensions
Fetching eventmachine 1.2.7
Installing eventmachine 1.2.7 with native extensions
Fetching em-socksify 0.3.2
Installing em-socksify 0.3.2
Fetching http_parser.rb 0.8.0
Installing http_parser.rb 0.8.0 with native extensions
Fetching em-http-request 1.1.7
Installing em-http-request 1.1.7
Fetching factory_bot 6.2.1
Installing factory_bot 6.2.1
Fetching method_source 1.0.0
Installing method_source 1.0.0
Fetching thor 1.2.1
Installing thor 1.2.1
Fetching zeitwerk 2.6.6
Installing zeitwerk 2.6.6
Fetching railties 7.0.4.2
Installing railties 7.0.4.2
Fetching factory_bot_rails 6.2.0
Installing factory_bot_rails 6.2.0
Fetching faker 3.1.1
Installing faker 3.1.1
Fetching faraday-net_http 3.0.2
Installing faraday-net_http 3.0.2
Fetching ruby2_keywords 0.0.5
Installing ruby2_keywords 0.0.5
Fetching faraday 2.7.4
Installing faraday 2.7.4
Fetching faraday-retry 2.0.0
Installing faraday-retry 2.0.0
Fetching websocket-extensions 0.1.5
Installing websocket-extensions 0.1.5
Fetching websocket-driver 0.7.5
Installing websocket-driver 0.7.5 with native extensions
Fetching faye-websocket 0.11.1
Installing faye-websocket 0.11.1
Fetching ffi 1.15.5
Installing ffi 1.15.5 with native extensions
Fetching filesize 0.2.0
Installing filesize 0.2.0
Fetching fivemat 1.3.7
Installing fivemat 1.3.7
Fetching gssapi 1.3.1
Installing gssapi 1.3.1
Using rexml 3.2.5
Fetching gyoku 1.4.0
Installing gyoku 1.4.0
Fetching hashery 2.1.2
Installing hashery 2.1.2
Fetching hrr_rb_ssh 0.4.2
Installing hrr_rb_ssh 0.4.2
Fetching hrr_rb_ssh-ed25519 0.4.2
Installing hrr_rb_ssh-ed25519 0.4.2
Fetching http-cookie 1.0.5
Installing http-cookie 1.0.5
Fetching httpclient 2.8.3
Installing httpclient 2.8.3
Fetching rkelly-remix 0.0.7
Installing rkelly-remix 0.0.7
Fetching jsobfu 0.4.2
Installing jsobfu 0.4.2
Fetching json 2.6.3
Installing json 2.6.3 with native extensions
Fetching little-plugger 1.1.4
Installing little-plugger 1.1.4
Fetching multi_json 1.15.0
Installing multi_json 1.15.0
Fetching logging 2.3.1
Installing logging 2.3.1
Fetching memory_profiler 1.0.1
Installing memory_profiler 1.0.1
Fetching metasm 1.0.5
Installing metasm 1.0.5
Fetching metasploit-concern 5.0.1
Installing metasploit-concern 5.0.1
Fetching metasploit-model 5.0.1
Installing metasploit-model 5.0.1
Fetching pg 1.4.5
Installing pg 1.4.5 with native extensions
Fetching recog 3.0.3
Installing recog 3.0.3
Fetching webrick 1.7.0
Installing webrick 1.7.0
Fetching metasploit_data_models 6.0.2
Installing metasploit_data_models 6.0.2
Fetching net-ssh 7.0.1
Installing net-ssh 7.0.1
Fetching rex-core 0.1.30
Installing rex-core 0.1.30
Fetching rex-socket 0.1.47
Installing rex-socket 0.1.47
Fetching rubyntlm 0.6.3
Installing rubyntlm 0.6.3
Fetching rubyzip 2.3.2
Installing rubyzip 2.3.2
Fetching metasploit-credential 6.0.2
Installing metasploit-credential 6.0.2
Fetching metasploit-payloads 2.0.108
Installing metasploit-payloads 2.0.108
Fetching metasploit_payloads-mettle 1.0.20
Installing metasploit_payloads-mettle 1.0.20
Fetching mqtt 0.5.0
Installing mqtt 0.5.0
Fetching msgpack 1.6.0
Installing msgpack 1.6.0 with native extensions
Fetching nessus_rest 0.1.6
Installing nessus_rest 0.1.6
Fetching net-ldap 0.17.1
Installing net-ldap 0.17.1
Fetching timeout 0.3.1
Installing timeout 0.3.1
Fetching net-protocol 0.2.1
Installing net-protocol 0.2.1
Fetching net-smtp 0.3.3
Installing net-smtp 0.3.3
Fetching network_interface 0.0.2
Installing network_interface 0.0.2 with native extensions
Fetching nexpose 7.3.0
Installing nexpose 7.3.0
Fetching sawyer 0.9.2
Installing sawyer 0.9.2
Fetching octokit 4.25.1
Installing octokit 4.25.1
Fetching openssl-ccm 1.2.3
Installing openssl-ccm 1.2.3
Fetching openvas-omp 0.0.4
Installing openvas-omp 0.0.4
Fetching pcaprub 0.13.1
Installing pcaprub 0.13.1 with native extensions
Fetching packetfu 1.1.13
Installing packetfu 1.1.13
Fetching patch_finder 1.0.2
Installing patch_finder 1.0.2
Fetching ruby-rc4 0.1.5
Installing ruby-rc4 0.1.5
Fetching ttfunk 1.7.0
Installing ttfunk 1.7.0
Fetching pdf-reader 2.11.0
Installing pdf-reader 2.11.0
Fetching nio4r 2.5.8
Installing nio4r 2.5.8 with native extensions
Fetching puma 6.0.2
Installing puma 6.0.2 with native extensions
Fetching strptime 0.2.5
Installing strptime 0.2.5 with native extensions
Fetching rasn1 0.12.1
Installing rasn1 0.12.1
Fetching rb-readline 0.5.5
Installing rb-readline 0.5.5
Fetching redcarpet 3.6.0
Installing redcarpet 3.6.0 with native extensions
Fetching rex-text 0.2.49
Installing rex-text 0.2.49
Fetching rex-arch 0.1.14
Installing rex-arch 0.1.14
Fetching rex-struct2 0.1.3
Installing rex-struct2 0.1.3
Fetching rex-bin_tools 0.1.8
Installing rex-bin_tools 0.1.8
Fetching rex-encoder 0.1.6
Installing rex-encoder 0.1.6
Fetching rex-exploitation 0.1.37
Installing rex-exploitation 0.1.37
Fetching rex-java 0.1.6
Installing rex-java 0.1.6
Fetching rex-mime 0.1.7
Installing rex-mime 0.1.7
Fetching rex-nop 0.1.2
Installing rex-nop 0.1.2
Fetching rex-ole 0.1.7
Installing rex-ole 0.1.7
Fetching rex-random_identifier 0.1.10
Installing rex-random_identifier 0.1.10
Fetching rex-powershell 0.1.97
Installing rex-powershell 0.1.97
Fetching rex-registry 0.1.4
Installing rex-registry 0.1.4
Fetching rex-rop_builder 0.1.4
Installing rex-rop_builder 0.1.4
Fetching rex-sslscan 0.1.9
Installing rex-sslscan 0.1.9
Fetching rex-zip 0.1.4
Installing rex-zip 0.1.4
Fetching ruby-macho 3.0.0
Installing ruby-macho 3.0.0
Fetching openssl-cmac 2.0.2
Installing openssl-cmac 2.0.2
Fetching windows_error 0.1.5
Installing windows_error 0.1.5
Fetching ruby_smb 3.2.4
Installing ruby_smb 3.2.4
Fetching mustermann 3.0.0
Installing mustermann 3.0.0
Fetching rack-protection 3.0.5
Installing rack-protection 3.0.5
Fetching tilt 2.0.11
Installing tilt 2.0.11
Fetching sinatra 3.0.5
Installing sinatra 3.0.5
Fetching sqlite3 1.6.0 (arm64-darwin)
Installing sqlite3 1.6.0 (arm64-darwin)
Fetching sshkey 2.0.0
Installing sshkey 2.0.0
Fetching swagger-blocks 3.0.0
Installing swagger-blocks 3.0.0
Fetching thin 1.8.1
Installing thin 1.8.1 with native extensions
Fetching tzinfo-data 1.2022.7
Installing tzinfo-data 1.2022.7
Fetching unix-crypt 1.3.0
Installing unix-crypt 1.3.0
Fetching warden 1.2.9
Installing warden 1.2.9
Fetching win32api 0.1.0
Installing win32api 0.1.0
Fetching nori 2.6.0
Installing nori 2.6.0
Fetching winrm 2.3.6
Installing winrm 2.3.6
Fetching xdr 3.0.3
Installing xdr 3.0.3
Fetching xmlrpc 0.3.2
Installing xmlrpc 0.3.2
Using metasploit-framework 6.3.5 from source at `.`
Fetching parallel 1.22.1
Installing parallel 1.22.1
Fetching parser 3.2.0.0
Installing parser 3.2.0.0
Fetching pry 0.14.2
Installing pry 0.14.2
Fetching pry-byebug 3.10.1
Installing pry-byebug 3.10.1
Fetching rainbow 3.1.1
Installing rainbow 3.1.1
Fetching regexp_parser 2.6.2
Installing regexp_parser 2.6.2
Fetching rspec-support 3.12.0
Installing rspec-support 3.12.0
Fetching rspec-core 3.12.1
Installing rspec-core 3.12.1
Fetching rspec-expectations 3.12.2
Installing rspec-expectations 3.12.2
Fetching rspec-mocks 3.12.3
Installing rspec-mocks 3.12.3
Fetching rspec 3.12.0
Installing rspec 3.12.0
Fetching rspec-rails 6.0.1
Installing rspec-rails 6.0.1
Fetching rspec-rerun 1.1.0
Installing rspec-rerun 1.1.0
Fetching rubocop-ast 1.24.1
Installing rubocop-ast 1.24.1
Fetching ruby-progressbar 1.11.0
Installing ruby-progressbar 1.11.0
Fetching unicode-display_width 2.4.2
Installing unicode-display_width 2.4.2
Fetching rubocop 1.44.1
Installing rubocop 1.44.1
Fetching ruby-prof 1.4.2
Installing ruby-prof 1.4.2 with native extensions
Fetching simplecov-html 0.12.3
Installing simplecov-html 0.12.3
Fetching simplecov 0.18.2
Installing simplecov 0.18.2
Fetching timecop 0.9.6
Installing timecop 0.9.6
Fetching yard 0.9.28
Installing yard 0.9.28
Bundle complete! 16 Gemfile dependencies, 192 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
Post-install message from dnsruby:
Installing dnsruby...
  For issues and source code: https://github.com/alexdalitz/dnsruby
  For general discussion (please tell us how you use dnsruby): https://groups.google.com/forum/#!forum/dnsruby
Post-install message from rubyzip:
RubyZip 3.0 is coming!
**********************

The public API of some Rubyzip classes has been modernized to use named
parameters for optional arguments. Please check your usage of the
following classes:
  * `Zip::File`
  * `Zip::Entry`
  * `Zip::InputStream`
  * `Zip::OutputStream`

Please ensure that your Gemfiles and .gemspecs are suitably restrictive
to avoid an unexpected breakage when 3.0 is released (e.g. ~> 2.3.0).
See https://github.com/rubyzip/rubyzip for details. The Changelog also
lists other enhancements and bugfixes that have been implemented since
version 2.3.0.
Post-install message from openssl-ccm:
Thanks for installing!
Post-install message from openssl-cmac:
Thanks for installing!

msfdb and msfconsole initial run

ishaanahuja@Ishaan-Ahujas-MacBook-Pro ~/testing/metasploit-framework % ./msfdb init
[?] Would you like to init the webservice? (Not Required) [no]: no
Clearing http web data service credentials in msfconsole
Running the 'init' command for the database:
Creating database at /Users/ishaanahuja/.msf4/db
Creating db socket file at /var/folders/30/2vb0m9f92x3d2h6fz_vpp1cw0000gn/T
Starting database at /Users/ishaanahuja/.msf4/db...success
Creating database users
Writing client authentication configuration file /Users/ishaanahuja/.msf4/db/pg_hba.conf
Stopping database at /Users/ishaanahuja/.msf4/db
Starting database at /Users/ishaanahuja/.msf4/db...success
Creating initial database schema
Database initialization successful
ishaanahuja@Ishaan-Ahujas-MacBook-Pro ~/testing/metasploit-framework % ./msfconsole
                                                  
 _                                                    _
/ \    /\         __                         _   __  /_/ __
| |\  / | _____   \ \           ___   _____ | | /  \ _   \ \
| | \/| | | ___\ |- -|   /\    / __\ | -__/ | || | || | |- -|
|_|   | | | _|__  | |_  / -\ __\ \   | |    | | \__/| |  | |_
      |/  |____/  \___\/ /\ \\___/   \/     \__|    |_\  \___\


       =[ metasploit v6.3.5-dev-020d2d3302                ]
+ -- --=[ 2294 exploits - 1201 auxiliary - 410 post       ]
+ -- --=[ 968 payloads - 45 encoders - 11 nops            ]
+ -- --=[ 9 evasion                                       ]

Metasploit tip: Use the resource command to run 
commands from a file
Metasploit Documentation: https://docs.metasploit.com/

msf6 > 

[sempervictus]

Thanks for the detailed log outputs - nice of nokogiri to tell us its using the ARM64 native build 😄

[Ishaanahuja7]

Please review the above PRs