Monday, February 1, 2016

Logging and Emailing EMC ReceverPoint Journal Lag Alerts with ksh Script

rpa_stats.ksh

Each value passed the high and low watermarks cause a mail to be sent and also a log message to be writen.

#####BEGIN#####
 #!/bin/ksh
RPAUSER=username
RPAIP=10.11.12.16
RPAPREFIX=CG_
STATUSFILE=/home/aixadm/bin/rpa_stats.status.txt
RUNDIR=/home/aixadm/bin
FILENAME=rpa_stats.log
LOGFILE=$RUNDIR/$FILENAME
TMPLOGFILE=$LOGFILE.tmp
HIMARK=3400
LOMARK=250
LASTLAG=`cat $STATUSFILE`
MAXLOGSIZE_inKB=10000
MAILTO="T-DM-Systems@anti.com.tr,SANTeam@anti.com.tr"

# exec 2>/dev/null
exec 2>>/home/aixadm/bin/rpa.err.log
unalias rm
du -k $LOGFILE|read LOGSIZE fn
if [[ $LOGSIZE -gt $MAXLOGSIZE_inKB ]] then
  find $RUNDIR -name "$FILENAME.[0-9]*_[0-9]*[0-9]" -exec gzip {} \;
  find $RUNDIR -name "$FILENAME.*.gz" -mtime +30 -exec rm {} \;
  mv $LOGFILE $LOGFILE.`date '+%Y%m%d_%H%M%S'|tr -d '\n'`
fi

total_lag_gb=0
echo "\n"|ssh $RPAUSER@$RPAIP get_group_statistics|egrep "$RPAPREFIX.*|Journal lag"|sed '/CG_.*:/N;s/\n/ /'|grep "GB$"|sort -k4n|
while read cg j l lag
do
  if [[ $cg == "Journal" ]] then
    continue
  fi
  print $cg "\t\t" $lag >> $TMPLOGFILE
  lag_gb="`echo $lag|sed -e 's/GB//'`"
  total_lag_gb=`echo "scale=2\n$total_lag_gb+$lag_gb"|bc`
done
print "Total Copy Journal Lag: " $total_lag_gb >> $TMPLOGFILE

if [[ $total_lag_gb -gt $HIMARK && $LASTLAG -le $HIMARK ]] then
  print "The total copy cournal lag is more than the HIMARK=" $HIMARK " GBs" >> $TMPLOGFILE
  date >> $TMPLOGFILE
  echo "##################################################################################" >> $TMPLOGFILE
  $RUNDIR/postmail_HTML_RPA.pl "WARNING: The RPA Total Copy Journal Lag Has Exceeded The High Watermark" $MAILTO < $TMPLOGFILE
  cat $TMPLOGFILE >> $LOGFILE
  rm $TMPLOGFILE
  echo $total_lag_gb > $STATUSFILE
  exit
fi
if [[ $total_lag_gb -lt $HIMARK && $LASTLAG -ge $HIMARK ]] then
  print "The total copy cournal lag has gone below the HIMARK=" $HIMARK " GBs" >> $TMPLOGFILE
  date >> $TMPLOGFILE
  echo "##################################################################################" >> $TMPLOGFILE
  $RUNDIR/postmail_HTML_RPA.pl "INFO: The RPA Total Copy Journal Lag Has Gone Below The High Watermark" $MAILTO < $TMPLOGFILE
  cat $TMPLOGFILE >> $LOGFILE
  rm $TMPLOGFILE
  echo $total_lag_gb > $STATUSFILE
  exit
fi
if [[ $total_lag_gb -lt $LOMARK && $LASTLAG -ge $LOMARK ]] then
  print "The total copy cournal lag is less than the LOMARK=" $LOMARK " GBs" >> $TMPLOGFILE
  date >> $TMPLOGFILE
  echo "##################################################################################" >> $TMPLOGFILE
  $RUNDIR/postmail_HTML_RPA.pl "OK: The RPA Total Copy Journal Lag Has Gone Below The Low Watermark" $MAILTO < $TMPLOGFILE
  cat $TMPLOGFILE >> $LOGFILE
  rm $TMPLOGFILE
  echo $total_lag_gb > $STATUSFILE
  exit
fi
if [[ $total_lag_gb -gt $LOMARK && $LASTLAG -le $LOMARK ]] then
  print "The total copy cournal lag has exceeded the LOMARK=" $LOMARK " GBs" >> $TMPLOGFILE
  date >> $TMPLOGFILE
  echo "##################################################################################" >> $TMPLOGFILE
  $RUNDIR/postmail_HTML_RPA.pl "INFO: The RPA Total Copy Journal Lag Has Exceeded The Low Watermark" $MAILTO < $TMPLOGFILE
  cat $TMPLOGFILE >> $LOGFILE
  rm $TMPLOGFILE
  echo $total_lag_gb > $STATUSFILE
  exit
fi
echo $total_lag_gb > $STATUSFILE
cat $TMPLOGFILE >> $LOGFILE
rm $TMPLOGFILE
date >> $LOGFILE
echo "##################################################################################" >> $LOGFILE
######END#####





postmail_HTML_RPA.pl

#####BEGIN#####
 #!/usr/bin/perl
#$ENV{'LANG'} = 'tr_TR';
#system("export LANG=tr_TR");
#   $_[0] --> Subject
#   $_[1] --> Mail Address
#   Stdin --> Body

  $SMTPSERVER = "tekno3.fw.tekno.com.tr";
  #$SMTPSERVER = "mailse3.anti.com.tr";
  $MAILFROM   = "T-DM Systems ";
  $_[0] = $ARGV[0];
  $_[1] = $ARGV[1];

  if ( $_[1] eq "" || $_[1] eq " " ) { exit;}
#  if ( ! -s $_[2] ) { exit(0) };
  $| = 1;
  open(STDERR, "> /dev/null");

  use Net::SMTP;
#  print "Content-type: text/html", "\n\n";

  $smtp = Net::SMTP->new($SMTPSERVER,Timeout => 15, Debug => 1);
  if ( ! $smtp ) { return; }
        die "Couldn't connect to server" unless $smtp;

  $MAILTO=$_[1];

  $smtp->mail( $MAILFROM );

  # .... Send Mail to related receivers
  @RCVR = split(/,/,$_[1]);
  foreach $receiver (@RCVR)
  {
    $smtp->to( $receiver );
  }


  # Start the mail
  $smtp->data();

  # Send the header
  $smtp->datasend("To: ".$MAILTO."\n");
  $smtp->datasend("From: ".$MAILFROM."\n");
  $smtp->datasend("Subject: ". $_[0] ."\n");
  $smtp->datasend("MIME-Version: 1.0\n");
  $smtp->datasend("Content-type: text/html\n\n");
  $smtp->datasend("\n");

  #$smtp->datasend( $_[2]);
  foreach $satir ( ) {                  #stdin'den geleni gonder
    $smtp->datasend( $satir."
\n" );
  }

  # Send the termination string
  $smtp->dataend();
  # Close the connection
  $smtp->quit();

######END#####

Printing EMC ReceverPoint Journal Lag with ksh Script

 rpa_stats_simple.ksh

#!/bin/ksh
RPAUSER=username
RPAIP=10.11.12.13
#RPAPREFIX is the standart prefix generally used before consistency group name
RPAPREFIX=CG_

total_lag_gb=0
echo "\n"|ssh $RPAUSER@$RPAIP get_group_statistics|egrep "$RPAPREFIX.*|Journal lag"|sed '/CG_.*:/N;s/\n/ /'|grep "GB$"|sort -k4n|
while read cg j l lag
do
  print $cg "\t\t" $lag
  lag_gb="`echo $lag|sed -e 's/GB//'`"
  total_lag_gb=`echo "scale=2\n$total_lag_gb+$lag_gb"|bc`
done
print "Total Copy Journal Lag: " $total_lag_gb
if [[ $total_lag_gb -gt 2900 ]]; then
  print "Total copy cournal lag is more than 2900 GBs"
fi
date

"FAILED pre-installation verification" during VIOS upgrade (updateios)

While trying to upgrade to VIOS 2.2.4.10 from 2.2.2.1 I got weird error messages about the missing filesets. There was two kinds of error messages on different VIO servers like those:

/usr/ios/cli/ioscli updateios -dev /mnt -accept -install


FAILURES
--------
  Filesets listed in this section failed pre-installation verification
  and will not be installed.

  Missing Filesets
  ----------------
  The following filesets could not be found on the installation media.
  If you feel these filesets really are on the media, check for typographical
  errors in the name specified or, if installing from directory, check for
  discrepancies between the Table of Contents file (.toc) and the images that
  reside in the directory.

    b                                       
    b                                       
    b                     

    p                                       
    p                                       

  << End of Failure Section >>

+-----------------------------------------------------------------------------+
                   BUILDDATE Verification ...
+-----------------------------------------------------------------------------+
Verifying build dates...done
FILESET STATISTICS
------------------
  228  Selected to be installed, of which:
      228  FAILED pre-installation verification

####################
/usr/ios/cli/ioscli updateios -dev /mnt -accept -install

0503-019 installp:  Implementation is an invalid level.
        ALL levels must be in one of the following formats: 
                vv.rr.mmmm.ffff OR vv.rr.mmmm.fffff.ppppppppp

    mit.script                              
    mit.script                              
    +                                       

  << End of Failure Section >>

+-----------------------------------------------------------------------------+
                   BUILDDATE Verification ...
+-----------------------------------------------------------------------------+
Verifying build dates...done
FILESET STATISTICS
------------------
  208  Selected to be installed, of which:
      208  FAILED pre-installation verification
  ----


SOLUTION:
I discovered two old timed files in /tmp. Removing them solved the problem in one type of systems. On the others removing all /tmp content cleared the problem.

cd /tmp
ls -l .list_filesets.*
-rw-r--r--    1 root     staff          1580 Dec 27 2011  .list_filesets.out
-rw-r--r--    1 root     staff          1238 Dec 27 2011  .list_filesets.toc

rm .list_file*
/usr/ios/cli/ioscli updateios -dev /mnt -accept -install