Securing your Web services using Nginx

If we have an API service and want to publish to public, better use a reverse proxy like nginx to handle all the dirty traffic trying to taken down your services.

With Nginx you will make your API server live prosper and not minding the dirty request which should not coming on your server if not using Nginx in front of it.

So lets do the Nginx configuration for securing your backend API server.

    add_header Cache-Control public;

    add_header X-frame-Options "DENY";

    add_header X-Xss-Protection "1; mode=block" always;

    add_header Content-Security-Policy-Report-Only "script-src https://skyway.shineapi.net";

    add_header X-Content-Type-Options "nosniff" always;

    add_header Strict-Transport-Security 'max-age=31536000;includeSubDomains;preload;' always;

    add_header Referrer-Policy no-referrer-when-downgrade;

 
   if ($http_referer = "") {  return 403; }


With above configuration, any request incoming your API will be handled by Nginx and protected by the header config which is web security standards for securing any web app in the wild.

For the explanation i will put on another blog post for details.

Read More

Monitoring tools in the new age with Grafana

We know for old school monitoring tools are using Nagios, Cacti, CollectD. Well, that's what i use on my job to monitoring infrastructure.

No fancy graphic and display, no filter , only plain time range value with the monitored data.

Now, we see Grafana. I try grafana and its looks a like elastic kibana. Well, apparantly Grafana was a fork of Kibana 3.0 . Grafana is Free to use, but also have enterprise version which have support.

With grafana we can connect to multiple datastore. What i try was using prometheus, as it can be generate data also the TSDB it used. no need more DB for the storage.

Also we can also put our application metrics to prometheus to scrap and display it.

Now version 6.x , grafana have many plugin and chart template to be use.

It can also used for your monitoring dashboard to show alerts.

Alerts can be set using prometheus alert manager. all is there to be exploited.

Lets continue the journey with Grafana + prometheus + prometheus alert manager

Read More

Create a Services in linux box

In linux box, everything runs at startup is a service.

Service located on  /etc/systemd/system/servicename.service

Step to create a service are :
1. Create a service user account
2. Change ownership of config file to the service account
3. Create a service file to load when startup

Create service user account the secure way :

• $ sudo useradd --no-create-home --shell /bin/false blackbox_exporter

Here are template for create a service and use the created user above.

[Unit]
Description=Blackbox Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=blackbox_exporter
Group=blackbox_exporter
Type=simple
ExecStart=/usr/local/bin/blackbox_exporter --config.file /etc/blackbox_exporter/blackbox.yml

[Install]
WantedBy=multi-user.target

This will be inside the .service file.

Then we need to reload the daemon.

#systemctl daemon-reload

And service can run using below command :

#systemctl start service_name

To enable it do :

#systemctl enable service_name

Hope this helps

Read More

Exchange server error 452-4-3-1-insufficient-system-resources

When Microsoft Exchange have problem cannot receive any email from other party, there will be a lot of things can be happened. But make sure you pin point the exact issue by looking at the server event logs for exchange. Also if possible to get the NDR report from the sending party so we can know exactly what exchange spit out when error happening. Usually a mature mail systems will have specific error so administrator can resolve the problem.

On this case, we get the error message from the sender which are 452-4-3-1-insufficient-system-resources. What we got from this was :

1. Server is reachable
2. SMTP service is up
3. When complete the sending, it fail with insufficient system resources

On Exchange documentation, this error related to storage, so the situation was when exchange going to store the receive message to disk, its spit error because insufficient resources.

Microsoft Exchange Transport is rejecting message submissions because the available disk space has dropped below the configured threshold.The following resources are under pressure:

Queue database logging path (“C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\data\Queue\”) = 96% [High] [Normal=92% Medium=94% High=96%]

Physical memory load = 90% [limit is 94% to start dehydrating messages.]

The following components are disabled due to back pressure:

Inbound mail submission from Hub Transport servers

Inbound mail submission from the Internet

Mail submission from Pickup directory

Mail submission from Replay directory

Mail submission from Mailbox server

Content aggregation

The following resources are in normal state:

Queue database path (“C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\data\Queue\mail.que”) = 95% [Normal] [Normal=95% Medium=97% High=99%]

Version buckets = 0 [Normal] [Normal=80 Medium=120 High=200]

Private bytes = 14% [Normal] [Normal=71% Medium=73% High=75%]

Batch Point = 0 [Normal] [Normal=2000 Medium=4000 High=8000]

Submission Queue = 0 [Normal] [Normal=1000 Medium=2000 High=4000]

With Microsoft Exchange, it have a monitoring component which monitor available resources - Back Pressure, which is also tracking free space on a disk, where the Exchange Transport service queue are located.

Checking on the disk resources on servers, we have plenty disk 1.4TB with free 100GB of disk. This was strange.

Some behavior to note when the threshold limit exceeded, Exchange can  :

- Medium (90%) threshold - Stop receiving mail over SMTP from external senders (MAPI client e-mails are yet processed)

- High (99%) threshold - The mailflow stops to be processed completely

Dig down more on the event log, we found that the drive of Microsoft Exchange transport was because the available disk space has dropped below the configured threshold. And we read the documentation in Microsoft, it states that Microsoft Exchange transport service will need retain a min 10% free disk space where the transport Role folder resides.

So in this case the size of 1.4TB will need minimum 10% to be free, which are 140GB, and in this case not enough as it left only 100GB. This is by design to prevent disk full and Microsoft Exchange crash.

The solution to this was easy, just add more disk with expanding the drive, or just move transport role folder to another disk drive if you cannot expand it. After the disk size threshold surpassed, restart Microsoft exchange transport service and all will be running again.

If you want to move the transport queue to another disk , you can edit the config in exchange install located in 

$env:exchangeinstallpath\bin\EdgeTransport.exe.config  wiith the following changes

<add key=”QueueDatabasePath” value=”C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\data\Queue” />

<add key=”QueueDatabaseLoggingPath” value=”C:\Program Files\Microsoft\Exchange Server\V14\TransportRoles\data\Queue” />

When change the trasnport queue path, leave the content and just restart transport service as Exchange will recreate the folder automatically.

Note : Old directory can safely be removed

In my case, we just extend the disk without restart the servers.

Read More

Squid delay pools setup

Squid can do bandwith management and separate speed access based on group.

Below example of setup :

########################################### define networks
acl all src 0.0.0.0/0.0.0.0
acl unlimited src "/etc/squid/unlimited.txt"
acl our_1mbps src "/etc/squid/our_1mbps.txt"
acl our_512kbps src "/etc/squid/our_512kbps.txt"
acl our_256kbps src "/etc/squid/our_256kbps.txt"


######### give access

http_access allow unlimited
http_access allow our_1mbps
http_access allow our_512kbps
http_access allow our_256kbps
http_access deny all


##### define delay pools

delay_pools 4
delay_class 1 2
delay_access 1 allow unlimited
delay_access 1 deny all
delay_parameters 1 -1/-1 -1/-1


delay_class 2 2
delay_access 2 allow our_1mbps
delay_access 2 deny all
delay_parameters 2 -1/-1 131072/131072

delay_class 3 2
delay_access 3 allow our_512kbps
delay_access 3 deny all
delay_parameters 3 -1/-1 65536/65536

delay_class 4 2
delay_access 4 allow our_256kbps
delay_access 4 deny all
delay_parameters 4 -1/-1 32785/32786

With above setup, clients will limited to the above bandwith.

Read More

IT Asset Management Software

Do you have ever experience trouble in managing your Assets in your organizatoin, especially within the scope of IT Assets like PC, laptop, accessories, etc which related to day to day operation ?

Manual will be using spreadsheets application available. More advance are using a specialized software which manage your assets and logs where about the assets and status.

Come the rescue, Snipeit asset management. Its a software based asset management with full feature to manage your assets and also come as a free open source software. Of course with free meanings, you can do it all by yourself to make it works, besides your skill on translate the documentation to working application.

Here you need to know that Snipeit application is a :
1. PHP based application
2. Need a database to function properly, eg. Mariadb or Postgresql.
3. Multiuser applicatoin
4. Support LDAP integration for corporate environment.
5. Can run both on Linux / Windows server as a web services

With this information, you should know what basic prerequisite needed.

For my journey, i come to Snipeit because organization i work for need an asset management software to manage all operation related activity and keep track of the asset owned by organization. This will be a second step after doing manual labour works using spreadsheets.

What i recommend to setup are using High Availability setup for uninterupted service, as these are web service, its easy to do the High Availablility setup, especially the web service part.

What can we do are using this design :

User ---------   HA Proxy               --->   Snipeit App 01   ----->    |    Snipeit Database
                       Load Balancer        --->  Snipeit APP 02    ----->    |

With this setup, access from user are load balanced between Snipeit AP 01 and 02 to provide high availability and load balancing within the application.

Working Snipeit IT ASSET Management Systems

The software we used are :
1. Centos 7.x
2. HA Proxy
3. Nginx
4. PHP7.2
5. MariaDB 10
6. Memcached

Read More

keepalived issue with iptables

When talking about High Availability services, we can use keepalived to do a automatic failover between 2 host.

Keepalived is working like a charm by using a virtual ip. 1 Master elected for normal operation, and others as backup.
When master have issue, backup will take over the services.

Issue arise with configuration, where there is iptables entry to drop any vrrp traffic type. This was issue with the configuration.

To mitigate this, look for the config of keepalived.conf . change entry from :

vrrp_strict 

to become :

vrrp_accept

This entry will make keepalived will not use iptables.

Keepalived version we use was V1.3.5

Read More