Managing VirtualBox VDI size for a Linux Guest

If you are just looking to save space on your operating drive, always remember to use the Description box for snapshots and delete those you no longer need.

And if the guest host is Ubuntu, see some good pre-wash steps at https://itsfoss.com/free-up-space-ubuntu-linux/.

Assuming you are using a dynamic disk you can perform the following steps below save more space:

Check for a dynamic disk VDI
Verify the VDI is a dynamic disk

From within the Linux guest, run the following in a terminal:

dd if=/dev/zero of=/var/tmp/bigemptyfile bs=4096k ; rm /var/tmp/bigemptyfile

It will take some time to complete. Depending on the size of the disk and the amount of empty space, it can be a long time, so be patient.

When the commands complete, shut down the guest.

The next step is to run VBoxManage.exe. First, locate where VirtualBox is installed. This is usually C:\Program Files\Oracle\VirtualBox, though you can also find it by checking the path of the VirtualBox launch icon:

Find the VirtualBox install location from the launch icon properties
Find the VirtualBox install location from the launch icon properties

Open a commend prompt (search for cmd.exe) and cd to where VirtualBox is installed:

cd "C:\Program Files\Oracle\VirtualBox"

Locate and copy the path to the VDI:

Locate and copy the path to the VDI
Then run VBoxManage as follows:
VBoxManage.exe modifymedium disk “[absolute path]” –compact
ex:

VBoxManage.exe modifymedium disk "T:\VirtualBoxes\Ubuntu_21.04_withVPN\Ubuntu_VPN_21.04-disk002.vdi " –compact

I also back mine up, which is why I need to reduce the size.

VirtualBox Export Appliance Menu

 

Facebooktwitterredditlinkedinmail
© Scott S. Nelson

Create a Reusable Informatica Cloud Secure Agent Dev Image

I am going to start this article with a confession. As a system integration and automation consultant I am continuously shifting from one product, language and platform to another. The frequent changes strengthen the short-term memory and change the long-term memory to manage concepts more than details. A complex set of steps, processes or systems commands that can be performed without having to refer to Google, Evernote, or vendor documentation today can often fade to nothing more than a clear memory of having done them without recall of the specifics within a few months. Sometimes in mere weeks (or even days) depending on how different the next project is.  So the main reason I write these detailed articles is so that I can repeat a process if ever needed again.  Being able to share these mental breadcrumbs is just a bonus.

Lately I have been working a great deal with Informatica Cloud, an excellent iPaaS solution. Informatica Cloud uses an on-premises application called a Secure Agent to facilitate secure interactions between internal systems and the Cloud. During training and development it is useful to have a Secure Agent running on a developer’s laptop. While the Secure Agent will install on the operating systems of most laptops, it runs better in an environment that is configured to support server operations rather than day-to-day work. There is also a simple way to switch the Informatica Cloud instance (known as an Org), though it can be a hassle for a developer or consultant that needs to interact with multiple Orgs. To make life easier in such circumstances, I have taken a baseline VirtualBox image and added some applications and scripts so that the image can be quickly configured for any Org I have access to. Following are the steps I took to build this image, plus instructions for running the scripts that will make it reusable.

Customize a Linux Image

If you aren’t familiar with VirtualBox, you can download it for free at https://www.virtualbox.org/wiki/Downloads.  Be sure to get the Guest Additions download, too, you will want it later. I don’t have a preference for a specific Linux distribution. This article uses a CentOS 7.0 image available from https://s3-eu-west-1.amazonaws.com/virtualboxes.org/CentOS7-Gnome.ova.torrent, a choice resulting from the first image found that was of recent vintage and provided sufficient support for the task (it also has a graphical desktop, which is useful when you want to share the image with people who may not be comfortable with command-line-only interfaces). That said, support for various installation processes vary widely between distributions and builds, so if you choose a different distro you may need to find alternative methods to some of the steps on your own.

The first step is to import the pre-built appliance you have downloaded.  The screenshots below will help if you have never done this before.

Import Appliance

Select Appliance File

Customize Configuration Values

I change the Name of the appliance to be descriptive of the purpose of the image after I have completed configuring it. I also increase the CPU count and RAM total so that it will not be too slow during development test runs. You may need to adjust these values according to your own machine. I recommend no more than half of your available resources.

Hard Disk Space

One reason I use this particular base image is that it has adequate hard drive space, which many lack.  If you have an appliance image you like but the drive space is too small, see my blog post about Resizing the Root Drive on a Linux VirtualBox Guest Image with a Windows Host for a solution to use it anyway.

One value of a VM is the ability to use Snapshots to go back to a working point.  During the process we will create one to make a reusable appliance.  Create others as you go according to your own sense of adventure or caution. I suggest you delete old snapshots when you know your current state is stable to save disk space. Doing so at the start saves re-importing the appliance if there is a mis-step earlier in the process.

Install the Informatica Cloud Secure Agent

The next thing to do is to install the Secure Agent. If you plan to re-use this image as a starting point, do not initialize the agent after installation, just start it and verify the install as follows (screen shots follow the steps for those unfamiliar with them):

Download the agent from Informatica Cloud

Download the Secure Agent

Select Linux Agent for this set up

Default save location is the user download directory

then:

[centos@localhost ics]$ cd ~/Downloads
[centos@localhost Downloads]$ chmod +x agent64_install.bin
[centos@localhost Downloads]$ ./agent64_install.bin

Default install options

Choosing the default install location is the simplest option.

[centos@localhost Downloads]$ cd ~/infaagent
[centos@localhost infaagent]$ ./infaagent startup
[centos@localhost infaagent]$ cd main/agentcore
[centos@localhost agentcore]$ ./consoleAgentManager.sh getstatus

Finally, add the following to ~/.bash_profile:

cd ~/infaagent/
./infaagent startup

Find your .bash_profile

Planning for Reuse

For this to be reusable, we will want to change the hostname, especially for a shared dev environment as we cannot have everyone using “localhost” as their hostname. The following is a bit of an overkill script that handles setting the hostname for a variety of distributions:

changehostname.sh:

#!/usr/bin/env bash

OLD_HOSTNAME="$( hostname )"
NEW_HOSTNAME="$1"

if [ -z "$NEW_HOSTNAME" ]; then
 echo -n "Please enter new hostname: "
 read NEW_HOSTNAME < /dev/tty
fi

if [ -z "$NEW_HOSTNAME" ]; then
 echo "Error: no hostname entered. Exiting."
 exit 1
fi

hostnamectl set-hostname "$NEW_HOSTNAME"
hostnamectl set-hostname "$NEW_HOSTNAME" --pretty
hostnamectl set-hostname "$NEW_HOSTNAME" --static
hostnamectl set-hostname "$NEW_HOSTNAME" --transient

echo "Changing hostname from $OLD_HOSTNAME to $NEW_HOSTNAME..."

hostname "$NEW_HOSTNAME"

if [ -n "$( grep "$OLD_HOSTNAME" /etc/sysconfig/network )" ]; then
 sed -i "s/HOSTNAME=.*/HOSTNAME=$NEW_HOSTNAME/g" /etc/sysconfig/network
else
 echo -e "HOSTNAME=$NEW_HOSTNAME" >> /etc/sysconfig/network
fi

if [ -n "$( grep "$OLD_HOSTNAME" /etc/hosts )" ]; then
 sed -i "s/$OLD_HOSTNAME/$NEW_HOSTNAME/g" /etc/hosts
else
 echo -e "$( hostname -I | awk '{ print $1 }' )\t$NEW_HOSTNAME" >> /etc/hosts
fi

echo "Re-boot for change to reflect correctly."

This is also a good time to place a readme on the desktop. Here is the one I use:

###### One time set up #######
Open a terminal and execute the folloiwng commands (excluding [examples])

cd ~/infaagent
sudo su
./changehostname.sh PL_USER_ID

[example:./changehostname.sh scott.nelson]

exit

Restart the VM

cd ~/infaagent/main/agentcore
./consoleAgentManager.sh configure USERNAME PASSWORD

[example:./consoleAgentManager.sh configure linux.virtualbox@primitivelogic.com pa55w0rd]


[LEAVE TERMINAL WINDOW OPEN]

Log in to Informatica Cloud with the same credentials registered above, go to Configure > Runtime Environments

Confirm that the PL_USER_ID agent has been created
wait for upgrade to complete (at least 10 minutes)
return to terminal and run:

reboot

Log in and should be good to go.

#################################
If you have questions: theitsolutionist@gmail.com

Useful commands:

cd ~/infaagent
./infaagent shutdown?
./infaagent startup?

cd ~/infaagent/main/agentcore
./consoleAgentManager.sh getstatus

Save the script in ~/infaagent, shut down the virtual machine and create a snapshot. Then start the machine and follow the readme. If all works well, restore the last snapshot and then export the appliance as your base image.

Export the VM Appliance

Select appliance to export

Set location and format

Add Guest Additions

Shared Folders require Guest Additions

I set up a shared folder to easily exchange files between the guest (Virtual Machine) and host (laptop), as well as to have a place to uses as a file location accessible to both while working with Informatica Cloud flat file connections and logs.

In order to access the shared folder (as well as have a shared clipboard), run the Guest Additions set up after logging in to the new VM.

CentOS7 is not all that easy to do that with. There is a good set of instructions at http://lifeofageekadmin.com/how-to-install-virtualbox-5-additions-on-centos-7/. One thing that is not clear in the instructions is that you will want to shut down and re-start the VM after the last yum update before running the Guest Additions installation. You may also have to run it twice to get everything working.

Insert and run the virtual CD

You will probably need to use Right-CTRL+C to release your mouse from the VM until the Guest Additions are installed.

Once the shared folder shows up under /media, add the centos user to the group for access:

sudo usermod -a -G vboxsf centos

Restart for change to take effect.

Installing MySQL

I often find that MySQL is handy to have on the VM. The following steps will work to get it installed on the CentOS 7.0 image available from https://s3-eu-west-1.amazonaws.com/virtualboxes.org/CentOS7-Gnome.ova.torrent.

First, remove the old install with

sudo yum -y remove mariadb-libs

Then download the latest install from https://dev.mysql.com/downloads/repo/yum/.

The Red Hat Enterprise Linux 7 version seems to work on Centos 7. Save the download rather than opening it, then go to your downloads folder and run the following commands:

sudo yum -y localinstall platform-and-version-specific-package-name.rpm
sudo yum -y install mysql-community-server
sudo service mysqld start

When done, verify with

sudo service mysqld status

Finally, find and update the database admin password and (optionally) install Workbench using:

grep 'temporary password' /var/log/mysqld.log
...
mysql_secure_installation
...
sudo yum -y install mysql-workbench-community

Be sure to add the new password to your readme.

Facebooktwitterredditlinkedinmail
© Scott S. Nelson

Resizing the Root Drive on a Linux VirtualBox Guest Image with a Windows Host

Most of the solutions I design and develop are deployed to a Linux server. Before “DevOps” became a thing, there were always server admins ready willing and able to help with setting up the deployment environment and handling the day-to-day maintenance afterwards. Lately I have been left to my own means to get these tasks done and have learned some commands, written a several bash scripts for repetitive or automated tasks and bookmarked enough reference sites to be productive while still not considering myself an expert and definitely not an administrator.

So, being cautious, I prefer to have a virtual machine that is close to the environment I will be deploying my work to, especially bash scripts that can bring things down faster than they build them up if there some errant typo in the right-wrong place. I once built the duplicate virtual machine image from scratch, which I found to be a painful and dissatisfying experience given that I wanted the machine chiefly due to my lack of expertise with the finer points of configuration and administration. The next best thing to building it yourself (or first best thing, in my case) is to find one that is already pre-built and then add the necessary customizations to it. There are a plethora of free Linxu VM images out there, and finding one that is fairly close to the enterprise standard of my current client is usually fairly simple. The one thing that is almost always an issue is that the free images have a small hard drive in the configuration. If it is a case where another drive can be created and mounted, great. But recently I ran across a production configuration where the everything was off the root mount and I finally figured out how to enlarge the drive on the VM image without too many headaches. Here is how I did.

First, this is based on using VirtualBox. I have not used VMWare in a long time, but I believe the first stage of enlarging the capacity on VMWare may be even easier than with VirtualBox, which is where we start in the slide show below.

To save writing down the command from the slide show, you can copy and modify the following:

“C:\Program Files\Oracle\VirtualBox\VBoxManage.exe” modifyhd “Ubuntu 15_40GB.vdi” –resize 40960

Facebooktwitterredditlinkedinmail
© Scott S. Nelson

A Simplified Oracle 11g Database Command Line Linux Basic Install

I have always struggled with setting up a fresh Oracle 11g Database install. I have made copious notes over the years on the steps to take each time I tackle the process, and when I go back to the notes they only serves as clues to the mystery. This time I had to do the whole thing on a Linux both without the use of the GUI installers and I have captured and scripted the process once and for all.

Of course, most new installs these days will be 12c and should be performed by someone with deep database skills. This article is intended for those that simply need a database for a proof-of-concept or development environment where 11g will suffice.

Acknowledgements

Before I get into the details, I want to point out that there are three key blog posts that I found most useful in this process. They are:

Step-By-Step

These steps are based on using the scripts in this article. Feel free to adjust to fit your own needs, which is how this process came about.

As a side note, while putting the process together and again while documenting it, I used a VM and took a snapshot at the completion of each step to save time if some issue arose in a particular step. You will probably benefit from this as well, since small differences can prevent fully using these steps exactly as described.

Downloads

The Database installs can be found at http://www.oracle.com/technetwork/database/enterprise-edition/downloads/index.html. If you have access to MOS, you can get a slightly newer version, and you will only need to first two files.

Environment Variables

The values need to be set in the environment. You can place them in a bash script under /etc/profile.d for all users, or in the oracle user home directory if necessary. In the /etc/profile.d folder, be sure to chmod 644 to ensure it will run properly.

export DB_USER=oracle
export DB_PASS=welcome1
export DB_USER_HOME=/home/${DB_USER}
export ORACLE_HOSTNAME=${HOSTNAME}
export ORACLE_UNQNAME=MY11GDB
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1
export ORACLE_SID=MY11GDB
export LD_LIBRARY_PATH=${ORACLE_HOME}/lib:/lib:/usr/lib;
export TMP=/tmp
export APP_DIR=/u01/app
export PATH=/usr/sbin:/sbin:/bin:/usr/sbin:/usr/bin:${ORACLE_HOME}/bin
export CLASSPATH=${ORACLE_HOME}/jlib:${ORACLE_HOME}/rdbms/jlib;
export SOFTWARE_HOME=/media/sf_software
export SCRIPT_HOME=/media/sf_scripts

Regardless of where you place these values, be sure they are set before continuing and placed where they will always be set when administering to the database in the future. This may require either sourcing the file, or rebooting.

Script Variables

db_installprep.sh will set all variables except for the name of the install zip files as those can change over time. Update as necessary in install_11gDB.sh.

Install Prep Script

The script below performs the following tasks:

Creates User and Groups

The _create_groups function takes an inelegant approach to create the oracle user and groups. For those of you better versed in Linux administration, feel free to create your own script. Please share if you do.

Creates ORACLE_HOME path and Update SELinux

For reasons I can’t fathom, the installer will not create the ORACLE_HOME path, so it must be created before installing. Once installed, SELinux enforcement can cause problems with getting the listeners running. Since I only use the database behind a firewall, turning it off works best for me. I would not take that approach with a production server, though.

Dependency Installs

As noted in the Acknowledgements, there is a useful script for making sure the dependencies are in place.

Set Response File Variables

If you use the response files provided in the Downloads section, this will update the values to match your Environment Variables. If you are using your own or from a different source, comment out the _create_rsp_files function.

db_installprep.sh

#!/bin/bash
# Script based on instructions at http://dbaora.com/install-oracle-in-silent-mode-11g-release-2-11-2/
export INSTALL_DATE=`/bin/date "+%Y%m%d-%H-%M"`

function_logcomment()
{
 echo "#########################################################################################"
 echo -e "$(date +"%I:%M:%S"): $1"
 echo "#########################################################################################"
}

_check_user() {
# "Borrowed from" https://danielwestermann.com/2013/08/11/a-simple-script-to-automate-the-oracle-12c-setup/
    if [ $(id -un) != "${1}" ]; then
        _log "you must run this as ${1}"
        exit 0
    fi
}

_create_groups(){
	groupNames=( oinstall dba oper asmdba asmoper asmadmin )
	groupIds=( 54321 54322 54323 54327 54328 54329 )
	
	for index in "${!groupNames[@]}"
	do
		function_logcomment "adding group ${groupNames[index]}"
		groupadd ${groupNames[index]} -f -g ${groupIds[index]}
	done
#NOT THE MOST ELEGANT APPROACH...
	echo "Check for $DB_USER user"
	if [ ! id -u $DB_USER > /dev/null 2>&1 ]; then
	echo "Adding $DB_USER user"
		useradd -u 500 $DB_USER
		echo $DB_PASS | passwd $DB_USER --stdin
	fi

	for index in "${!groupNames[@]}"
	do
		echo "Adding $DB_USER user ${groupNames[index]} group"
		usermod -aG ${groupNames[index]} $DB_USER 
	done
}

_pre_env_settings() {
	mkdir -p $ORACLE_HOME
	chown oracle:oinstall -R /u01
	sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
}

_package_updates(){
	cd /etc/yum.repos.d
#	cp $SCRIPT_HOME/build/database/public-yum-ol6.repo .
	wget http://public-yum.oracle.com/public-yum-ol6.repo
	wait $!
	yum -y install oracle-rdbms-server-11gR2-preinstall
}

_create_rsp_files(){
	cd $SCRIPT_HOME/build/database
	filenames=( db_install.rsp dbca.rsp )

	for index in "${!filenames[@]}"
	do
		sed 's#@ORACLE_BASE@#'"$ORACLE_BASE"'#g ; s#@ORACLE_HOME@#'"$ORACLE_HOME"'#g ; s#@ORACLE_SID@#'"$ORACLE_SID"'#g ; s#@DB_PASS@#'"$DB_PASS"'#g' ${filenames[index]}.template > ${filenames[index]}
	done
}

_check_user "root"
wait $!
_create_groups 2>&1 | tee ${APP_DIR}/db11g_install.log
wait $!
_pre_env_settings 2>&1 | tee ${APP_DIR}/db11g_install.log
wait $!
_create_rsp_files
wait $!
_package_updates 2>&1 | tee ${APP_DIR}/db11g_install.log
echo "DB Install Prep Complete"
#wait $!
#https://danielwestermann.com/2013/08/11/a-simple-script-to-automate-the-oracle-12c-setup/
#suggests the following firewall changes but have been successful so far without
#systemctl stop firewalld
#systemctl disable firewalld

Install the Database

As noted earlier, be sure that the install zip file names are correct in install_11gDB.sh. install_11gDB.sh will perform the following:

  1. Unzip the install files
  2. Perform a silent install of the database application
  3. Create a default listener
  4. Create a default database

install_11gDB.sh

#!/bin/bash
# Script based on instructions at http://dbaora.com/install-oracle-in-silent-mode-11g-release-2-11-2/
export INSTALL_DATE=`/bin/date "+%Y%m%d-%H-%M"`

function_logcomment()
{
 echo "#########################################################################################"
 echo -e "$(date +"%I:%M:%S"): $1"
 echo "#########################################################################################"
}

# check for the current os user
# "Borrowed from" https://danielwestermann.com/2013/08/11/a-simple-script-to-automate-the-oracle-12c-setup/
_check_user() {
    if [ $(id -un) != "${1}" ]; then
        _log "you must run this as ${1}"
        exit 0
    fi
}

_unzip_installs(){
	cd $DB_USER_HOME
	unzip -q $SOFTWARE_HOME/p13390677_112040_Linux-x86-64_1of7.zip
	unzip -q $SOFTWARE_HOME/p13390677_112040_Linux-x86-64_2of7.zip
}

_install_binaries(){
	cd $DB_USER_HOME/database
	./runInstaller -silent -responseFile $SCRIPT_HOME/build/database/db_install.rsp
}

_create_db(){
	mkdir $ORACLE_BASE/oradata
	mkdir $ORACLE_BASE/flash_recovery_area
}
_check_user $DB_USER
source ${DB_USER_HOME}/.bash_profile
_unzip_installs 2>&1 | tee ${APP_DIR}/db11g_install.log
wait $!
_install_binaries 2>&1 | tee ${APP_DIR}/db11g_install.log
wait $!
netca -silent -responseFile $SCRIPT_HOME/build/database/netca.rsp 2>&1 | tee ${APP_DIR}/db11g_install.log
wait $!
dbca -silent -responseFile $SCRIPT_HOME/build/database/dbca.rsp 2>&1 | tee ${APP_DIR}/db11g_install.log

Post Install Steps

After the installation, there are two scripts that are provided by Oracle that need to be run as root. Then there are some steps to perform so that the database is started every time the server is, because I have not had a case where this wasn’t what I needed. As noted in the Acknowledgements, How I Enable Autostarting of Oracle Database for Demonstrations and Development (by Christopher Jones) was my reference for this process and taken almost completely from that blog entry.

Once the above is completed, the install files are no longer necessary, so they are removed.

db_post_install.sh

#!/bin/bash

#script Distilled from https://blogs.oracle.com/opal/entry/how_i_enable_autostarting_of1

_check_user() {
# "Borrowed from" https://danielwestermann.com/2013/08/11/a-simple-script-to-automate-the-oracle-12c-setup/
    if [ $(id -un) != "${1}" ]; then
        echo "You must run this as ${1}!"
        exit 0
    fi
}
_update_oratab(){
	sed -i 's#'"$ORACLE_HOME"':N#'"$ORACLE_HOME"':Y#g' /etc/oratab 2>&1 | tee ${APP_DIR}/db11g_install.log
	#if above fails...
	grep -q -F ${ORACLE_SID}:${ORACLE_HOME}:Y /etc/oratab || echo ${ORACLE_SID}:${ORACLE_HOME}:Y >> /etc/oratab
}
_create_dbora(){
	cp $SCRIPT_HOME/build/database/dbora /etc/init.d/dbora
	sed -i 's#@ORACLE_HOME@#'"${ORACLE_HOME}"'#g' /etc/init.d/dbora
}
_update_dbora(){
	chgrp dba /etc/init.d/dbora
	chmod 750 /etc/init.d/dbora
	chkconfig --add /etc/init.d/dbora
	chkconfig dbora on
}
_check_user "root"
${ORACLE_BASE}/oraInventory/orainstRoot.sh
wait $!
${ORACLE_BASE}/product/11.2.0/db_1/root.sh
wait $!
_update_oratab
wait $!
_create_dbora
wait $!
_update_dbora
wait $!
rm -rf $DB_USER_HOME/database
echo "db_post_install.sh complete"

Verify the Installation

After running db_post_install.sh, you will either need to log out and log back in again as oracle, or simply reboot the server. First test with lsnrctl status, which should yield the following:

lsnctrl_success

On occasion, this may not work properly. Before sending me an email reminding me of my imperfections, you may first want to try:

sqlplus / as sysdba

startup

exit

And then either log out and back in or (preferably) reboot the server and try again. This generally fixed it for me while coming up with these steps.

Downloads

The scripts described here are available at https://drive.google.com/folderview?id=0B1L4vdibbjSXeUJ5Q0pURExfbXc&usp=sharing

One More Thing

I found installing sqldeveloper to be a great way to do more through testing. At the time of this writing it can be found at http://www.oracle.com/technetwork/developer-tools/sql-developer/downloads/index.html.

Facebooktwitterredditlinkedinmail
© Scott S. Nelson

Recovering Previous Versions from SharePoint

A few weeks ago I discovered that other people had mistakenly made changes to a draft of a technical document I was storing in SharePoint for which I had sole responsibility for. At the time we did not know how to access earlier versions of the document and I had to painstakingly review the entire draft and re-verify every detail.

Today I needed to compare revisions of a document that was stored in SharePoint with updates made on another machine. When I went to compare tab in this SharePoint managed document, I discovered how to find earlier versions!

Under the Review menu in Word, the Compare section has additional options for documents stored in SharePoint:

Compare Menu with SharePoint Library
Compare Menu with SharePoint Library

From here you can find the versions SharePoint has stored

SharePoint Library Revision Dialog
SharePoint Library Revision Dialog

And, once I knew what to look for, finding the Microsoft documentation was easy 🙂

Facebooktwitterredditlinkedinmail
© Scott S. Nelson