Compile Erlang using Kerl with crypto module

When working with erlang, we can use old version to run our erlang app which are not yet upgraded.

This multiple erlang version run like python env if you come from python.

So in erlang we using Kerl which can switch to multiple erlang version at runtime. So here we go.

We are using Debian 10.x buster. Some prerequisite before able to finish are install the dev tools.

# apt-get install build-essential autoconf libncurses5-dev openssl libssl-dev fop xsltproc unixodbc-dev libz-dev

Then get the openssl version supported on your erlang version. Here i use 17.5.x release which require openssl-1.0.xx to compile properly with crypto module.

1 $ git clone git@github.com:openssl/openssl.git --branch OpenSSL_1_0_2-stable

2 $ cd openssl

3 $ mkdir __result

4 $ ./config --prefix="${HOME}/openssl" shared zlib -fPIC

5 $ make depend

6 $ make

7 $ make install INSTALL_PREFIX="/home/me/openssl/__result"

Then we can start the kerl compilation.

Get the kerl 

1 $ curl -0 https://raw.githubusercontent.com/kerl/kerl/master/kerl

2 $ chmod a+x kerl

Make the ssl available for kerl compiler

1 $ export KERL_CONFIGURE_OPTIONS="--with-ssl=/home/me/openssl-OpenSSL_1_0_2-stable/__result/home/me/openssl-OpenSSL_1_0_2-stable/openssl/"

2 $ mkdir .kerl

3 $ kerl build 17.5.3 17.5.3

This should be show which indicate SSL included.

 

Then continue with installation:

$ kerl install 17.5.3 ~/.kerl/17.5.3

To test run erlang.

$ . ~/.kerl/17.5.3/activate

$ erl

> crypto:start().

Should no error if install successfully.

And make that fault tolerant app with 0 downtime.

Read More

Environment variable with ChicagoBoss

On Chicagoboss I need to have some configuration variable set in the main configuration. Just like my experience with Django framework on settings.py

In Chicagoboss, we can do it also and add our on config variable in the configuration file, which are boss.config.

These below are the part of the config we can custom :

%% APPLICATION CONFIGURATIONS

%% domains - A list of domains to serve the application on
%% static_prefix - The URL prefix of static assets
%% doc_prefix - The URL prefix for developer documentation
{ myapp, [
    {path, "../myapp"},
    {base_url, "/"},

    {dummy, true}
]}

So we can add the variable for our application with below settings :

%% APPLICATION CONFIGURATIONS

%% domains - A list of domains to serve the application on
%% static_prefix - The URL prefix of static assets
%% doc_prefix - The URL prefix for developer documentation
{ myapp, [
    {path, "../myapp"},
    {base_url, "/"},

    {dummy, true}.
    {mail_from, "justme@mydomain.com"},
    {api_key, "324298nvUYSCN298snk92442ps"}

]}

Now how we can use it on the application. We can use :

application:get_env(myapp, api_key).

If we want to print out on console :

io:fwrite("~p", [application:get_env(myapp, api_key)]).

And if we want to assign the variable to our local variable in the app we can use :

{ok, apikey} = application:get_env(myapp, api_key).

With this we can add as many as variable we need on the config.

Read More

Generate a list from atom and string

On Erlang, there is no string data type, but its a list of strings. So different with other language.
Also some function return an atom, which usually we need to combine it with our string. For example was the need to generate an sql query from an input parameter and our hard coded string.

Let say we generate a year from erlang fun, then from that year, we generate the month and data.
To do this properly are :

{{Year,_,_},_} = calendar:universal_time(),
Y1 = io_lib:format("~p-01-01",[Year]),
Y2 = lists:flatten(Y1),

Y2 result will be  "2018-06-18"


Hope this helps, as i look all around the google for this.

Read More

Installing Multiple Erlang version in one systems

In python you can have multiple python version installed in your systems using virtualenv. I like to do this with Erlang, as need to test my code on several erlang versions. Its not effective if you install and reinstall the version you need if you want to test out.

And in Erlang of course there is such package to manage multiple instance of Erlang. This is used for developer of course. Come the rescue Kerl , an erlang application that make it  "Easy building and installing of Erlang/OTP instances" .

So with no further do, i installed in my debain dev machine. Here are my steps by steps to make use of kerl. You can also see the documentation from kerl on how to work with kerl. In this blog i just documenting what important to run it and ready for developement.

Install Kerl using easy way with curl.

$ curl -O https://raw.githubusercontent.com/kerl/kerl/master/kerl

Don't forget to set it executable .

$chmod a+x kerl

Then you can list what release supported by kerl by running :

$./kerl list releases

You will get all the supported erlang version from R14 to R20.0 in current version as for now.
But before can build it using Kerl, need to install some of the prerequisite like gcc, ncurses-dev.
In debian you can install with this command :

#apt-get install build-essential ncurses-dev libssl-dev 

And in ~/.kerlrc add this entry :

export CFLAGS="-DOPENSSL_NO_EC=1"

Then we need to build the erlang version we need.

$./kerl build 20.0 20.0

This will download and build erlang version 20.0 and put it in ~/.kerl/builds/20.0

If you have multiple builds can do :

   ./kerl list builds

To install the builds run :

./kerl install 20.0 ~/kerl/20.2

This will install in your home directory, inside kerl/20.2/ folder.

To see installed erlang release, you can run :

./kerl list installatins

And to use specific erlang version do this :

$. /installed/path/of/kerl/20.0/activate
$erl --version

To deactivate use this command :

$ kerl_deactivate

Happy coding with erlang / OTP.

Reference :
- Kerl source

Read More

CouchDB test performance using httperf

After test an erlang framework to serve JSON API from a postgreSQL database, i remember that had installed CouchDB in my laptop, and why not use it with same data and try the performance load using the same httperf command.

The result of json from couchdb :

{
  "total_rows": 2,
  "offset": 0,
  "rows": [
    {
      "id": "2f9bc9fb62f3e8fa19ace932b9000d9f",
      "key": "2f9bc9fb62f3e8fa19ace932b9000d9f",
      "value": {
        "_id": "2f9bc9fb62f3e8fa19ace932b9000d9f",
        "_rev": "1-0a77ba71f874dc7ca2b7d22893cf4882",
        "task": "learn",
        "status": "not done"
      }
    },
    {
      "id": "2f9bc9fb62f3e8fa19ace932b90013d9",
      "key": "2f9bc9fb62f3e8fa19ace932b90013d9",
      "value": {
        "_id": "2f9bc9fb62f3e8fa19ace932b90013d9",
        "_rev": "1-6127c1359f9d34d2733943876931e7d4",
        "task": "erlang",
        "status": "not done"
      }
    }
  ]
}

And the design document to retreive the data save with path /todo/_design/todo/_view/list

function(doc) {
  emit(doc._id, doc);
}


so here are the result with same data.


httperf --client=0/1 --server=127.0.0.1 --port=5984 --uri=/todo/_design/todo/_view/list --rate=150 --send-buffer=4096 --recv-buffer=16384 --num-conns=27000 --num-calls=1
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
Maximum connect burst length: 1

Total: connections 27000 requests 27000 replies 27000 test-duration 179.995 s

Connection rate: 150.0 conn/s (6.7 ms/conn, <=13 concurrent connections)
Connection time [ms]: min 0.6 avg 1.1 max 92.4 median 0.5 stddev 2.5
Connection time [ms]: connect 0.0
Connection length [replies/conn]: 1.000

Request rate: 150.0 req/s (6.7 ms/req)
Request size [B]: 90.0

Reply rate [replies/s]: min 149.8 avg 150.0 max 150.0 stddev 0.0 (36 samples)
Reply time [ms]: response 1.1 transfer 0.1
Reply size [B]: header 231.0 content 470.0 footer 2.0 (total 703.0)
Reply status: 1xx=0 2xx=27000 3xx=0 4xx=0 5xx=0

CPU time [s]: user 62.39 system 117.63 (user 34.7% system 65.4% total 100.0%)
Net I/O: 115.9 KB/s (0.9*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

As we can see the result is almost the same. no error happening, all request processed with 6.7ms / request.

Test done in 179 s or 3 minutes. I don't know if all erlang use same processing to dump json.

Anyhow, great result and positif about this.

Read More

Erlang Chicago Boss JSON API test performance

I am documenting my test on erlang web framework, Chicago Boss and use it for backend API.
My setting for the database using Postgresql 9.2 using 1 table with 2 rows and 4 fields only and return the request using JSON.

I test the performance load using httperf on my debian 8 laptop.

# httperf --server 127.0.0.1 --port 8001 --uri /todo/list --rate 150 --num-conn 27000 --num-call 1

In this simple test, the same page is retrieved repeatedly. The rate at which requests are issued is 150 per second. The test involves initiating a total of 27,000 TCP connections and on each connection one HTTP call is performed (a call consists of sending a request and receiving a reply)

 The result should be like this if access from browser :

{
  "todos": [
    {
      "id": "todo-1",
      "task": "learning",
      "status": "not done",
      "owner": "Voldomore"
    },
    {
      "id": "todo-2",
      "task": "erlang",
      "status": "not done",
      "owner": "Potter"
    }
  ]
}

and the result of httperf : 

httperf --client=0/1 --server=127.0.0.1 --port=8001 --uri=/todo/list --rate=150 --send-buffer=4096 --recv-buffer=16384 --num-conns=27000 --num-calls=1
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
Maximum connect burst length: 1

Total: connections 27000 requests 27000 replies 27000 test-duration 179.999 s

Connection rate: 150.0 conn/s (6.7 ms/conn, <=13 concurrent connections)
Connection time [ms]: min 3.9 avg 6.6 max 85.4 median 5.5 stddev 4.5
Connection time [ms]: connect 0.0
Connection length [replies/conn]: 1.000

Request rate: 150.0 req/s (6.7 ms/req)
Request size [B]: 71.0

Reply rate [replies/s]: min 149.8 avg 150.0 max 150.2 stddev 0.1 (36 samples)
Reply time [ms]: response 6.6 transfer 0.0
Reply size [B]: header 125.0 content 673.0 footer 0.0 (total 798.0)
Reply status: 1xx=0 2xx=27000 3xx=0 4xx=0 5xx=0

CPU time [s]: user 55.66 system 124.11 (user 30.9% system 68.9% total 99.9%)
Net I/O: 127.3 KB/s (1.0*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

What i see are 179 seconds (3 minutes) to complete all the request, and with basic setup. only have a model and controller to list. I want to try it using Django Rest Framework also later on to see how django rest framework performance. But surely Erlang faster. Just want to know how many can be handled by DRF.

Read More