Graphite & statsd installation and Configuration

How to install Graphite and statsd
-----------------------------------

yum install   python-devel.noarch
yum install   pycairo.x86_64 Django.noarch django-tagging.noarch  python-twisted.noarch python-zope-interface.x86_64 python-zope-interface httpd memcached  python-memcached
yum install   fontconfig.x86_64 fontconfig-devel.x86_64
yum install   mod_wsgi.x86_64
yum install   python-pip.noarch

pip-python install whisper
pip-python install carbon
pip-python install graphite-web



cp /opt/graphite/conf/carbon.conf.example /opt/graphite/conf/carbon.conf
 cp /opt/graphite/conf/storage-schemas.conf.example /opt/graphite/conf/storage-schemas.conf
 cp /opt/graphite/conf/graphite.wsgi.example /opt/graphite/conf/graphite.wsgi
 cp /opt/graphite/examples/example-graphite-vhost.conf /etc/httpd/conf.d/graphite.conf
 cp /opt/graphite/webapp/graphite/local_settings.py.example /opt/graphite/webapp/graphite/local_settings.py




 python /opt/graphite/webapp/graphite/manage.py syncdb
 chown -R apache:apache /opt/graphite/storage/



wget -O /etc/init.d/carbon https://gist.github.com/raw/3652720/6978bda604c794e21165a81c6b879528d19e8bb4/carbon.init.sh
 chmod 0755 /etc/init.d/carbon
chkconfig --add carbon



 /opt/graphite/bin/carbon-cache.py start
 /etc/init.d/httpd start



yum install nodejs npm


cd /opt/graphite
git clone https://github.com/etsy/statsd.git
cd statsd
cp exampleConfig.js local.js



/etc/init.d/carbon start

/etc/init.d/memcached  start
/etc/init.d/statsd start



Configurations
--------------------
Graphite Configuration: /opt/graphite
Statsd configugration: /opt/graphite/statsd




Graphite/Statsd Logs
---------------------------

Graphite: /opt/graphite/storage/log/webapp
Statsd:  /var/log/statsd/
statsderr.log  statsd.log

Bash script to mask credit card number or any data in logs.

create a temp test file for testing your data

cat test.txt

hi this is my credit card number 1234567891234567
i have 22 this 22 is just for testing 1234567890123456 and this one

Here is the format which varies according to your requirement .
In my case there are 16 digits . 
I want to mask first 6 number here is the regular expression or bash command 


 sed -e :a -e "s/[0-9]\{6\}\([0-9]\{10\}\)/\*\*\*\*\*\*\1/;ta" test.txt 


here 6 and 10 are number of masks and number of visible characters respectively.
you can alter them according to your needs. 

And the stars can be replaced by # or the value you want to mask with .
The number of starts should be equal to number of masks in my case it is 6

Linux list all processes based on memory consumption .

list top 5 processes
ps -eo pmem,pcpu,vsize,pid,cmd | sort -k 1 -nr | head -5

list all processes in assending order .
ps -e -orss=,args= | sort -b -k1,1n | pr -TW$COLUMNS


easy ntp time sync between servers

Choose one server from which time need to be synced .

then on clinet open  /etc/ntp.conf

#server 0.rhel.pool.ntp.org ( comment these three lines )
#server 1.rhel.pool.ntp.org
#server 2.rhel.pool.ntp.org
server 10.10.10.180  ( this is the server which i choose so all other servers will be in sync with this server )


 /usr/sbin/ntpdate -s -b -p 8 -u 10.10.10.180

/etc/init.d/ntpd restart


Other commands that are used for NTP are 


ntpdate -u 10.10.10.180

output : ntpdate[13786]: adjust time server 10.10.10.180 offset 0.000221 sec

ntpdate -d 10.10.10.180 

output : output will be more in detail

ntpdate[13806]: ntpdate 4.2.2p1@1.1570-o Thu Nov 26 11:34:35 UTC 2009 (1)
Looking for host 10.10.10.180 and service ntp
host found : 10.10.10.180
transmit(10.10.10.180)
receive(10.10.10.180)
transmit(10.10.10.180)
receive(10.10.10.180)
transmit(10.10.10.180)
receive(10.10.10.180)
transmit(10.10.10.180)
receive(10.10.10.180)
transmit(10.10.10.180)
server 10.10.10.180, port 123
stratum 3, precision -20, leap 00, trust 000
refid [10.10.10.180], delay 0.02583, dispersion 0.00000
transmitted 4, in filter 4
reference time:    d5b8497a.219236b6  Fri, Aug 16 2013 12:17:22.131
originate timestamp: d5b849c8.3769d786  Fri, Aug 16 2013 12:18:40.216
transmit timestamp:  d5b849c8.37669489  Fri, Aug 16 2013 12:18:40.216
filter delay:  0.02588  0.02585  0.02585  0.02583
         0.00000  0.00000  0.00000  0.00000
filter offset: -0.00006 -0.00007 -0.00006 -0.00007
         0.000000 0.000000 0.000000 0.000000
delay 0.02583, dispersion 0.00000
offset -0.000070

16 Aug 12:18:40 ntpdate[13806]: adjust time server 10.10.10.180 offset -0.000070 sec


ntpq -p

How to copy directory structure in linux

Sometimes we need to copy only directory structure leaving content ( files )

find . -type d >dirs.txt

xargs mkdir -p

Script to import csv into dhcpd.conf import multiple mac bindings

make sure your csv file is in this order MAC,IP,NAME and file name of csv is ips.csv
Once that is done just run the script and check mac_out.txt which is file to be imported in dhcpd.conf

!/bin/bash

INPUT=ips.csv
OUTPUT=mac_out.txt
OLDIFS=$IFS
IFS=","
[ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; }
[ -f $OUTPUT ] && { rm $OUTPUT; }
[ ! -f $OUTPUT ] && { touch $OUTPUT; }
while read mac ip uname
do
echo "host $uname {" >> $OUTPUT
echo -e "\thardware ethernet $mac;" >> $OUTPUT
echo -e "\tfixed-address $ip;" >> $OUTPUT
echo "}" >> $OUTPUT
done < $INPUT

IFS=$OLDIFS


How to drop cache or in-memory cache on linux and free up some memory ?

$ echo 1 | sudo tee /proc/sys/vm/drop_caches  # drop pagecache
$ echo 2 | sudo tee /proc/sys/vm/drop_caches  # drop dentries and inodes
$ echo 3 | sudo tee /proc/sys/vm/drop_caches  # drop pagecache, dentries and inod

Also we can use 

echo 1 > /proc/sys/vm/drop_caches
echo 2 > /proc/sys/vm/drop_caches
echo 3 > /proc/sys/vm/drop_caches



This is also used when we take benchmarks of server.

What Server remote management systems called ?

These are different remote server management tools

Intel uses RMM2 (remote management 2)
Dell uses DRAC (Dell Remote Access Control)
Sun or Oracle uses ILOM (Integrated Lights Out Manager)
IBM uses IMM (Integrated Management Module)
HP uses ILO (Integrated Lights-Out).

Simple encryption and decryption of files using GPG

gpg --yes --passphrase="test" -c test1.txt

encrypted file will be test1.txt.gpg and test1.txt will still exist.

gpg --decrypt  --passphrase "test" --output "test1.txt" "test1.txt.gpg"

Linux bash script for checking and sending email alerts on disk space issues

#!/bin/sh
FILE=/tmp/mailcontent.txt

> /tmp/mailcontent.txt

df -H | awk '{ print $5 " " $6 }'|head -6|tail -5|while read output;

do
#  echo $output
  usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1  )
  partition=$(echo $output | awk '{ print $2 }' )
echo $usep
  if [ $usep -ge 90 ]; then
   echo -e "Running out of space \n\nPartition:\"$partition ($usep%)\" \nHost:$(hostname) \nDate:$(date)\n" >> /tmp/mailcontent.txt

 fi

done

if [[ -s $FILE ]] ; then

mail -s "CRITICAL ALERT:Disk space of $hostname more that 90%" disk-alert@yourcompany.com
else
echo "$FILE is empty."
fi

exit

Install and configure OSSEC ( Host-based Intrusion Detection System ) - By Subhash C

OSSEC is  used to meet PCI Compliance central logging and intrusion monitoring requirements with a free and self-managed solution. OSSEC monitors all types of logs such as syslog, apache, maillogs, mysql logs, ftp logs, and more.

1.     Deploying OSSEC core server:



2.       Extract the compressed OSSEC package and run the “./install.sh” script (It will guide you through the installation).

# tar -xvf ossec-hids-2.7.tar.gz 
# cd ossec-hids-2.7
# ./install.sh


The interactive installation begins and during this process it prompts for installation type either server or agent or local. However, installation path for OSSEC can be defined but by default it installs in (/var/ossec) directory. It is advised to select default settings and follow accordingly till the end of the installation. Finally, it is compiled and installed successfully according to the specification provided.


Select the language (in this example it is EN)














Select the type of installation as server







Accept the default folder, unless you want to change it:








Enable email notification by specifying the email address which will be used to send email alerts FROM:






If setup fails to determine your SMTP gateway automatically, you may specify it manually (change the IP as necessary to reflect your environment):





Enable integrity check daemon:





Enable rootkit detection daemon:





Whether or not you want to have “Active response” enabled is up to you. Although in prod environment, it can be quite risky.












We will not be using Syslog daemon feature in this example, so disable it.











OSSEC core server installation is now completed.


1.     Generating client keys


OSSEC server with the agent names in order to generate unique authentication keys required to bind the agent to the corresponding OSSEC server.

a. Firstly, create agents.TXT file containing the IPs and names separated by comma, of all target Linux Servers:
192.168.9.101,rwca1
...
….
Save:wq!
b. Place this file to /var/ossec folder on OSSEC server.
c. Now execute the following command: /var/ossec/bin/manage_agents -f /agents.txt



1.    Start OSSEC Server


# /var/ossec/bin/ossec-control start
# /var/ossec/bin/ossec-control restart
#/var/ossec/bin/ossec-control stop
 


2.     Install the OSSEC Agents

Follow above server installation steps, instead of server choose type agent

            # tar -xvf ossec-hids-2.7.tar.gz 
# cd ossec-hids-2.7
# ./install.sh
Select the type of installation as agent

3.     Add Agents to OSSEC Manager


To add an agent to an OSSEC manager with manage_agents you need to follow the steps below.
  1. Run manage_agents on the OSSEC server.
  2. Add an agent.
  3. Extract the key for the agent.
  4. Copy that key to the agent.
  5. Run manage_agents on the agent.
  6. Import the key copied from the manager.
  7. Restart the manager’s OSSEC processes.
  8. Start and restart the agent.

4.     Verify successful installation of the agent


Then verify that the client has been installed successfully and communicating with the OSSEC server.

# tail –f /var/ossec/log/ossec.log

5.     Deploy OSSEC Web UI dashboard


Install PHP for OSSEC UI

#  yum install php-mysql php-devel php-gd php-pecl-memcache php-pspell php-snmp php-xmlrpc php-xml

# Start Apache:  /etc/init.d/httpd  start
# Configure Apache autostart: /sbin/chkconfig httpd on
# Restart Apache: /etc/init.d/httpd restart
Download OSSEC webUI module

# cd /var/ww/html/
# tar –xvf ossec-wui-0.8-alpha-0.tar.gz
# mv ossec-wui-0.8-alpha ossec-wui
# Switch to the ossec-wui and run
./setup.sh -  specify user and password to protect Web UI.


# Add apache user to ossec group: 
Change permissions for /var/ossec/tmp directory:
# chmod 770 /var/ossec/ tmp/
# chgrp apache  /var/ossec/tmp/
# /etcinit.d/httpd restart
# /var/ossec/bin/ossec-control restart (restart the OSSEC server)

Access the OSSEC web portal: http://your-ip-address/ossec-wui

in the above URL replace your-ip-address with ip address where you installed ossec-wui





Step by step tutorial to configure two node redhat clustering By -- Sojan VM



Two node Red Hat and Clustering
  node1
  node2
  management server
Configure the Shared Storage

Install the clustering software on the nodes
High availability application service “High Availability”
package group  and webserver
service ricci start
provide  passwd for ricci

Install the cluster management software on
the management server
yum groupinstall “High Availability Management”
chkconfig luci on
service luci start












Define a cluster
Node1 and Node2



Then create a partition and format


Define kvm Fence device
Cluster.conf--------------------cluster configuration fine


<?xml version="1.0"?>

<cluster config_version="6" name="cluster">

        <clusternodes>

                <clusternode name="cluster_node1" nodeid="1">

                        <fence>

                                <method name="kvm">

                                        <device domain="cluster_node1" name="kvm"/>

                                </method>

                        </fence>

                </clusternode>

                <clusternode name="cluster_node2" nodeid="2">

                        <fence>

                                <method name="kvm">

                                        <device domain="cluster_node2" name="kvm"/>

                                </method>

                        </fence>

                </clusternode>

        </clusternodes>

        <cman expected_votes="1" two_node="1"/>

        <fencedevices>

                <fencedevice agent="fence_xvm" name="kvm"/>

        </fencedevices>



</cluster>




Kvm Host configuration
Let's start Installing the necessary packages on the host:
yum install   fence-virt fence-virtd fence-virtd-libvirt fence-virtd-multicast
Then create the key needed for the host to authenticate all the fencing requests from the guest cluster:
dd if=/dev/urandom of=/etc/cluster/fence_xvm.key bs=4096 count=1
The key should then be copied to /etc/cluster/fence_xvm.key of every guest.
Now run the configuration tool:
fence_virtd –c
service fence_virtd start
 
check your fence is working or not:
fence_xvm -o reboot –H node2
 
Define Resources For Clustered Web Service
Shared Storage (if not in fstab)
IP address

Apache Resource






Define Failover Domains

Node2































Define Clustered Web Service
Define service
Add storage resource (if not in fstab)
Add ip address resource

add script resource








Fix : rpmdb: unable to lock mutex: Invalid argument

If you are getting this message printed out repeatedly on the terminal screen while trying to install some packages using yum
 
# rpmdb: unable to lock mutex: Invalid argument
 
yum -y update glibc ( This works sometimes )
 
If that doesn not work then execute below command 
 
rpmdb --rebuilddb -vv

Install configure sophos antivirus on linux servers


Download sav-linux-xx-xxx.tgz 
from
( http://www.sophos.com )

after untar we will get dir "sophos-av"


install.sh
sav.tar
talpa.tar
uncdownload.tar


Run ./install.sh to install

-------------------------------
Configuration
---------------------------------


/opt/sophos-av/bin/savdstatus


/opt/sophos-av/bin/savdctl enable


Start stop restart disable service as normal linux service

/etc/init.d/sav-protect start

(or)

service sav-protect start


Scanning your computer
-----------------------

savscan /  ( we can use sophos enterprise console to scan all computers at once )

Scanning bootsectors
---------------------

savscan -bs=drivename

scanning masterboot record

savscan -mbr


Enable on reboot
---------------------
/opt/sophos-av/bin/savdctl enableOnBoot savd

Enable onaccess scanning
---------------------------
/opt/sophos-av/bin/savconfig query EnableOnStart

to enable onaccess scanning after reboot

/opt/sophos-av/bin/savconfig set EnableOnStart true


chkconfig --list  ( check if is starting on reboot )



Update sophos antivirus
---------------------------
/opt/sophos-av/bin/savupdate

Get CPU RAM MEMORY of all the servers on the network


In order to accomplish this you have to have an account with password less login for all the server on the network you want to retrieve data

Here is my tutorial for password less login

Create a file "ips" which should contain all the ip's you want to retrive data for ( 1 ip per line )


for i in `cat ips`;do ssh -t $i 'echo -e -n `uname -n `"   '$i' " ;cat /proc/cpuinfo | grep "cpu cores" | uniq';done > cpucores

( This one will not work in single core cpu's or vm's )

for i in `cat ips`;do ssh -t $i 'echo -e -n `uname -n `"   '$i'" ;grep MemTotal /proc/meminfo   ';done > Mem_details


for i in `cat ips`;do ssh -t $i 'echo -e -n `uname -n `"   '$i'  " ; sudo /sbin/fdisk -l | grep Disk ';done >hdd_details 

( This one will not work in single core cup's or vm's )

After you run all the three commands you will have three files in your directory

1) cpucores - list servername ip-address and number of cores
2) Mem_details - list servername ip-address and memory details in kb
3) hdd_details - list servername ip-address and drive and size.

Free training for freshers in hyderabad.


Course Goals and Target Audience

1) Training for freshers in Hyderabad
---------------------------------------------
2) For someone new to IT field
--------------------------------------
3) Looking for career in IT field
---------------------------------------
4) If you are in IT field and you are afraid of it 
-------------------------------------------------
5) Any one with basic qualifications and want use computer
----------------------------------------------------------------------
6) Basic skills for every software professional
----------------------------------------------------
7) You don't know anything and want to buy computer
-----------------------------------------------------------------
8) Setting up home network with home theater by yourself
-------------------------------------------------------------------


Note: I will cover basic usage of all tools .
I will lay a geek path walking,running on it is up to you ( distance unlimited )

All the classes with be live graphic oriented i will use internet.

searching techniques to solve your problem on your own.

Start with CPU,RAM,HDD

download software , movies , books etc

ecommerce and shopping online 

Various payment gateways and how they work ( google checkout, paypal )

I will known your background, What are your interests
what kind of career you are looking for 



Operating system basics ( including installation and configuration )

Windows/Linux/unix/apple ios/android basics

how internet works 
Desktop applications vs web application vs webservices
RSS feeds,Social media
how to create websites
Basic tools ( MSpaint,Notepad )

I will cover all basic tools required in daily life
below is the sample list

image editing , video editing, audio editing , printing , CD/DVD burning

MS office

Basics of networking 
setting up wired and wireless networks

Securing your pc with antivirus and securing your data.

Choosing your path and what you want to become 

Expalin various roles and responsibilities .



This course is free and intended to build a good Community