Technically Impossible

Lets look at the weak link in your statement. Anything "Technically Impossible" basically means we haven't figured out how yet.

トップ > Android > GPT on Android - build and run, 6GB RAM is not enough

GPT on Android - build and run, 6GB RAM is not enough

Android English post IT AI

Abstract

Some C/C++ programs can be compiled for Android, and building execution binary of chat in Alpaca.cpp is also possible. This post introduces its procedures to run it on Android.

Important note: 8~12GB RAM is required

One important note in advance of main topic. Check RAM capacity of a target smart phone or tablet. Executing chat, and it loads a language model to memory. Probably, more than 8GB or 12GB RAM is required to hold a model fully on memory.

The video at the end of this post shows forced termination by Android OS due to loading failure, not error or crash as programming bug. This is the case of Microsoft Surface Duo. It has 6GB RAM, but it is not sufficient to hold a language model on memory.
Although successful cases are reported on the net, some of them don't open usage model. Probably, they are recent high class model as Google Pixel 7 Pro.

Assumption

This post uses Linux as a build environment. Although Windows is applicable, using Linux on WSL is recommended. Preparation for both make and CMake on Windows is cumbersome.

Required source code and tools are

Download Alpaca 7B model with following the instruction on GitHub. The size of this model is 4GB. GitHub introduces following methods to download it. BitTorrent was used in this post.

Building execution binary

Android NDK is stored at the next folder.

/opt/android-ndk-r25c/

In this assumption, path of "android.toolchain.cmake" is 

/opt/android-ndk-r25c/build/cmake/android.toolchain.cmake

Path of source code is 

~/work/alpaca-android/

Execute commands to build binary at this folder, execution file "chat" is generated under "build" folder. Series of command is 

mkdir build
cd build
cmake -DCMAKE_TOOLCHAIN_FILE=/opt/android-ndk-r25c/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-23 -DCMAKE_C_FLAGS=-march=armv8.4a+dotprod ..
make

Termux

To run "chat" on Android, Terminal application is required. This post uses Termux.
This application should NOT be obtained from Google Play as described here*5. Download it at GitHub or F-Droid.

github.com f-droid.org

Termux needs permission to access storage. At first, assign storage permission at "Settings" of Android.
Basically, Termux can access and control under its folder, "/data/data/com.termux/files/home". Executing the next command at Termux, Android internal storage is mounted as "storage" under there.

termux-setup-storage

Copy "chat" and Alpaca model to Android smart phone, and run.

To execute "chat" from Termux, it should be located under Termux. Following procedure is required from copying files to run "chat".

  1. copy "chat" and Alpaca model to Android once. Example, "alpaca" folder under "Download".
  2. copy them under Termux.
  3. grant execution privileges to "chat"
  4. run "chat"

Series of commans is 

mkdir alpaca
cp storage/shared/Download/aplaca/* ./alpaca/.
cd alpaca/
chmod 777 chat
./chat

But unfortunately, this process is terminated by Android OS as the movie below. This is due to memory shortage to load a model to memory. In case of Linux, it can be avoided with swap*6, but Android can't take this solution due to missing root.

twitter.com

*1:clearlinux.org

*2:developer.android.com

*3:github.com

*4:github.com

*5:github.com

*6:impsbl.hatenablog.jp