How to not be a supernova in the clouds - Run time

Run time and forecasting:

When running different applications with different behaviors in isolated or shared accounts,  vServers or domains, visibility is a key quality point of management. As I said in How to not be a supernova in the clouds - Design time, is very important to be prepared to scale out or scale down the environment, this can be based on demand (reactive) or on forecasting (proactive). Both approaches need as much information as possible to determine how load can it expect (the future), how is it doing now (present) and how it behaved under some types of loads (past). The present and the past we can get from Oracle Cloud Control, is very important to setup all layers monitoring in Exalogic (there is a very good blog from Donald Forbes Integrating Enterprise Manager 12c with Exalogic), not only the application (Weblogic,OHS,OTD,...) or vServer OS components. 

For forecasting the history is not enough, a more complex algorithm should be applied, we need to consider also limit dates, percentage of expected population attended, outages, promotions and behavior changing signals (I believe we can even use marketing forecasting  tools to try to figure out the future load). 

For long term forecasting business indicators are more useful than CPU,RAM, Network usage (measured as bits per second or requests per application), if we (badly) compare with meteorology climatic changes can be observed with using air temperature, air pressure and air humidity, but we cannot forecast climatic changes based only on those three measure elements (thanks to my father Eugenio Hackbart for precious meteorology concepts). For short term forecasting CPU,RAM ad Network usage measures  are quite useful for next 3 hours forecasting (for example), as I said in How to not be a supernova in the clouds - Design time, quick scaling will be predicted with short term forecasting and slow scaling will be predicted with long term forecasting, also for more hardware acquisition. 

Another very important run time key point is how the whole environment life-cycle is done, which assembly technology was used to help changes to occur without service interruption, avoiding single points of failure and replicas (all of this done in design time) and (in run time)  we are prepared to use those techniques and configurations in an visible, controlled and orchestrated way. Not all activities are automatized, many human activities are needed, and humans need philosophy to live, in our concrete case, the design time philosophy.   

Autonomic behavior (the (mental) stage I am trying to reach):

When have reach a stage (or maturity level) where the can have reactive scaling or proactive scaling well stabilized. We can start thinking in "why not let the system manage himself (under human supervision, of course)?", to implement autonomic behavior we need an 100% automated resource allocation procedures (obviously, I am not considering adding another Exalogic machine in the contingency data center ...) to support an autonomic behavior, those allocation procedures can be integrated with SOA composites and human tasks for evaluation and approval (this creates a very powerful control mechanism to control changes and track an tracing). Some activities that we can have in consideration

• Reactive resource allocation
• Configure monitoring to identifies components internal pressure
• Define demand increase curve versus allocation time curve
• Configure thresholds based on the SLAs for that application
• Allocates or release resources consonant component pressure
• Proactive resource allocation
• Record historic behavior
• Forecast future events (on forecasting allocation) 
• Allocates or release resources consonant component pressure history/forecast

Conclusion

All run time activities are related to a well design time architecture and "philosophy" definition, my aim with all specifications and tips is to reach an "Autonomic Behavior" state, this is not easy, should be constructed from the basis. The continuation of this blog will be many techniques and tips that help to reach this state, mostly related procedures automation, and commissioning/decommissioning  activities orchestration.

I hope they enjoy .

How to not be a supernova in the clouds - Design time

When designing an Exalogic based solution is fundamental to use redundancy as a key aspect to fault tolerance (as already assembled in Exalogic, with double power source, networking equipment or storage heads). When we consider that everything have two pathways load balanced, is expected that if one pathway fails the other will be forced to deal with the double of the common load. From this point of view we can consider that either, an external demand increase or an component failure will generate an quick demand increase. The failures can have origin in hardware failures but also in human interventions, like rolling upgrades/patches, or someone that erases all the content from one storage volume.

The first thing that should be defined is the isolation and availability of applications. In terms of isolation we can consider three main patterns:

• maximal: separated virtual datacenter accounts,servers,networks and storage volumes
• medium: shared virtual datacenter account,networks but separated servers and storage volumes
• low: shared accounts, servers, networks and storage volumes

Sure we can define more refine combinations depending on what we need, the Exalogic Elastic Cloud Software is very flexible when defining those kind of combinations.

From the point of view of availability, we can consider some main evaluation points, this will help us to understand how to design the environment inside Exalogic. All the steps defined here can be applied on an OVM installation or event other virtualization software installation, may be important to adapt to the in use technology. 

 The evaluation points:

1-Scaling patterns:

When we start preparing one elastic environment, is very important to take care about how much time we need to create a new resource and what downtime this allocation process can cause (the environment is over demanded and the resource is not yet available), to address this situation I defined two types of scaling to be used in conjunction to reduce the impact of an demand increase. Are the two pattern types:

• Quick scaling 
• small amount of  unnecessary (in normal situation) allocated resources
• those resources are expensive (we don't want to waste those resources)
• we have always some margin
• In weblogic servers permits ManagerServers creation without  vServers creation
• very fast start procedure
• provides the resource necessary to sustain the environment while allocation slow scaling resources
• demand minimal configuration changes and almost no human intervention
• normally no need to change vServers geometry (CPU,RAM,...)
• Slow scaling
• no allocated resources
• the resource are not used
• the resource to be used can be de-allocated from other (less important) application 
• slow (and maybe complex) start procedure
• may involve external resources like DNS, Firewall or load balancers
• may need more intense human intervention

The idea is to have always two sets of scaling plans, one for immediate reaction (and for more common situations) and the other for more durable reconfigurations. Those configuration changes can be durable or transient, is very important to have ways to measure if the allocated resources are enough or is necessary to allocate or release resources. 

2-In servers configuration

The power of Exalogic Elastic Cloud is based on the well understanding the philosophy of the product. Some good practices (already defined in EECS documentation) can make the environments administration much more easy. Some good practices:

• distribution groups
• one per service vServer set
• if defined 4 vServers for WLS ManagedServers for our application, but we are running into a 1/4 Exalogic, is good idea to define one distribution group for ManagedServer for this application with a maximum size of 8. When scaling this set of vServers we can count with up to 1 vServer per compute node.
• When reaching the limit of computing nodes is a good idea to start thinking in creating bigger vServers (in CPU and RAM).
• spare memory for more managed Servers inside vServers
• allow quick scaling to support slow scaling
• consider wasting some memory to peak moments, may help to survive while scaling.
• not software installed inside internal vServer disk
• Exalogic still not allowing the change of CPU and RAM of one vServer, when those changes are needed, the vServer must be recreated, if is there software inside the internal disk of the vServer the vServer recreation can become a nightmare (mainly when doing that under pressure).
• take control of singleton components
• singletons normally means single point of failure, that when possible should be avoided and when not possible automated.
• optimize your templates as much as possible (I will talk about using iaas to build templates in another post)
• avoid doing human tasks when commissioning new servers that can lead to very long scaling process.
• automate as much as possible servers creation using iaas CLI
• avoid human errors on server creation
• allocate quickly new servers 
• makes possible to create scripts that calculates automatically server names and IP addresses. 

3-In network configuration

• Start with a default pattern for networks
• service
• exposure of the application normally OTD or OHS
• normally as a datacenter network segment, with outside of Exalogic visible IP addresses. 
• supports normally HTTP
• web
• web traffic between OTD and OHS
• serves dynamic content and static content
• normally IPoIB network, that will be only used by vServers.
• support normally HTTP
• application
• application traffic between OTD or OHS and WLS
• is the HTTP channel for weblogic managed servers
• serves dynamic content generated by the managed servers
• supports normally HTTP
• cluster
• cluster broadcasting for WLS managed servers
• session replication for WLS managed servers (maybe over (SDP)) 
• coherence nodes intercommunication network
• for single Exalogic topology used as default channel
• management
• HTTP channel for AdminServers
• vServers only accept ssh from this network
• DNS,LDAP/NIS can be configured to traffegate over this network
• check where you can use SDP (currently only session replication and JDBC)
• be careful with SDP bugs 
• design realistic network masks
• considering the distribution groups sizes and the different sets of servers that will be connected in this network, can be easy to define the maximum number of vServers connected to this network. 
• automate as much as possible networks creation using iaas CLI 

4-In storage volumes 

• Considering isolation levels create per set of applications project (in ZFS) or per application. More projects more maintenance and bigger isolation
• Per node volumes
• consider one NFS volume created per node when each vServer should have ist own disk volume and should never be able to read or write from other vServer volume. This topology means a total storage volume isolation.
• Useful for WLS domain when we want the all vServer have exactly the same path for domains.
• duplicated volumes
• consider one pair of volumes divided between evens and odds vServers, when something goes wrong only 50% of the service is affected, when upgrading is very useful for rolling upgrades.
• Useful for binaries (that do not change frequently)
• Useful for configuration that can be reused by more than one instance of the software.
• single volumes
• consider um volume shared between all servers when something is not core or does not denial of service, normally with many directories inside, one per each vServer.
• Useful for logs,backups,shared staging,Oracle Cloud Control agent software 
• use NFSv4
• the behavior is different between NFSv3 and NFSv4 (Using LDAP for Shared Authentication from Donald Forbes)
• be careful to have NIS available to ZFS and vServers (if not available, everything will appear as nobody inside storage volumes :( and the applications will have very strange behavior). 

Planning the resource allocation procedure

• When create more machines when create bigger machines?
• when reaching the distribution group limit, makes no sense having two vServers inside one computing node to do the same thing, two kernels, paging systems, Java with the same Perm Gen contents, a waste. When reaching a sustained growing that reaches the maximum number of computing nodes available for the distribution group, start recreating the vServers with more RAM and/or CPU.
• industrialize the allocation procedure, let the Elastic Cloud Software work for you.
• use OVAB
• use scripting with substitution

-------------------------------------------------

Management aspects

• vServers lifecycle
• Oracle Fusion Middleware lifecycle
• Monitoring and alarmistic

Oracle Fusion Middleware aspects

• OTD
• use throttling
• use DOS detection mechanisms 
• OSB
• use throttling
• validate XML schemas before passing back
• WLS
• use the new 12.1.2 dynamic clusters when possible
• use Exalogic optimizations

The planning the cross datacenter clustering using Oracle MAA will be another post, does not change fundamentally how to design the environment but networking design patterns.

The next part will be about run time experiences and then I expect to start talking in more details about some items (like scripting, deployment automation, ...) that I have successfully used in some projects. 

How to not be a supernova in the clouds

In this post I will start a serial of posts about techniques successfully used by me in projects to avoid two mainly things, single component or resource failure and quick request demand increase. I called that a Supernova because the effects are similar (thanks for the brilliant concept Luis Botelho). The idea is to give some ideas in two phases of Exalogic setup, when designing you solution and when using in run time. 

In each phase my idea is to cover some topics that are:

• Capacity planning
• Scaling
• Resiliency planning
• Managing
• Throttling

And some components related do Exalogic:

• Servers
• Networks
• Storage volumes 

This is a very simple approach to construct a robust solution using Exalogic, not tight related with what is being installed inside Exalogic (middleware software), but how to define the infrastructure to reach the defined aims. 

The power of Linux + PXE + OVM

In one workshop I created, was necessary to make some practical work, to do that I defined a set of Labs using VirtualBox, OVM 3.2 and Oracle's Sun ZFS Storage Appliance Simulator. The idea is to simulate an OVM environment using VirtualBox, just to play with the technology (is not supposed that virtualization over virtualization will create something useful :)).

I created a set of .pdf files that installs and configures OVM using ZFS as the storage with NFS and iSCSI over VirtualBox. All the installations are done through PXE (to avoid installing many servers). The PXE is not very useful with Exalogic since the installation  method for Exalogic is using templates and JeOS. 

To execute the labs is necessary to have VirtualBox, ZFS appliance simulator, OVM 3.2 and Oracle Enterprise Linux isos.

The Labs are not very well documented (sorry), I strongly suggest the usage of OVM installation guide together with these files.

The link to the files is here.

I hope they enjoy.

No more waits for echoes, Exalogic Elastic Cloud Software 2.0.6 was released

Oracle had release the EECS 2.0.6, the main new functionality is OVAB 11.1.1.6.2 that is now available, but for some of our customers other new functionality are more interesting, below the list of new features extracted from the Oracle® Exalogic Elastic Cloud Release Notes Release EL X2-2 and EL X3-2 Part Number E18480-11.


1.1 New Features and Enhancements in EECS 2.0.6

    1.1.1 Deploying Assemblies in the Exalogic vDC Using OVAB Deployer
Totally new feature that comes to help the "industrialization" of Exalogic deployment, together with the new Weblogic 12.1.2 dynamic clusters allows to scale any Weblogic deployed application without service interruption. Using together with OVM based development environments makes possible to evolve or retrofit keeping the architectural patterns and templates. 

    1.1.2 Support for Hybrid Configurations
This ability to have partial visualized partial physical Exalogic is for me an "echo" for a customer question that have some software suppliers that do not support the current version of the embedded OVM in Exalogic, but want to use OFM with all the scaling (OVAB or not) functionality of virtualization. The most important part is the ability to convert later between physical to virtual and from virtual to physical. Really a very useful new functionality.

    1.1.3 Fewer VMs for the Exalogic Control Stack
The five vServers used by EECS 2.0.4 consumes really a good number os CPUS and RAM, when using inside on 1/4 or bigger Exalogic this does not represent a big deal, but when using inside an 1/8 Exalogic (with 4 X3-2 computing nodes) it makes difference. One really interesting note in the document: "Note that, in EECS 2.0.6, the Exalogic Control VMs are distributed over fewer compute nodes: two nodes in the case of a fresh EECS 2.0.6 installation, and three nodes if you upgrade from EECS 2.0.4 to 2.0.6. "

    1.1.4 LVM-Based Disk Partitioning on Guest vServers
In our vServer creation procedures we define a dis size considering the swap and log (for Linux kernel) inside vServer disk, this helps to manage in a more efficient way the internal disk configuration for vServers, I saw a very good from Oracle team in one customer doing the conversion from one disk with two partitions (/ and swap) to a very well defined LVM structure using the EECS 2.0.4 template. My approach is to keep as closer to the original template as possible to avoid surprises when upgrading, this means avoid installing rpms inside the vServer internal disk and use NFS (ZFS) volumes for everything (also for rotating internal Linux kernel logs) and never count with the swap for memory (we should ever consider the the vServers disk are over and NFS volume). 

    1.1.5 Exabus/IMB 1.1 and Exalogic Java 1.1.6

    1.1.6 Configuring DNS for EoIB Networks
Highly demanded, network name resolution inside Exalogic in previous versions  is by himself, we need to use /etc/hosts files (and the scale of the environment is tricky) or use public DNS servers (force name service resolution for not visible networks) both sounds a little bit strange for me. Now is possible to define one DNS server per network (Oracle® Exalogic Elastic Cloud Administrator's Guide Release EL X2-2 and X3-2 Part Number E25258-07)

    1.1.7 OSWatcher Included in the Base Image and Guest Template
Very useful tool, but uses internal disk for data storage(/opt/oswbb), now with LVM is more easy to manage. 

    1.1.8 imagehistory and imageinfo Commands Included in the Base Template


    1.1.9 Linux Server-Based Exalogic Guest Base Template
After configuring some vServer we realized that those Linux vServers was configured as desktop and run in runlevel 5 (with graphical interface), using the Exalogic philosophy, we created templates with everything that we need and running in runlevel 3. The new template is more aligned with the expected usage inside Exalogic. 

    1.1.10 Detailed View of Resource-Allocation by Account
Nice to have in the console more information about the accounts, personally I prefer IAAS CLI to manage the resources 

Some new things that are not in the release notes:

As per note The Command "Iaas-create-distribution-group" Does Not Allow To Define Number Of Elements (Doc ID 1569851.1) in MOS, the command iaas-create-distribution-group does not accept the already documented parameter size, the new IAAS CLI is available and now the parameter is there:

[exalogic@oel-exa-admin ~]$ iaas-create-distribution-group
Required option '--name' is missing (120029)
USAGE:
   Create a new distribution group.
 
   iaas-create-distribution-group|iaas-cdg  [--access-key-file <access-key-file>
                                            ] [--base-url <base_url>] [--debug]
                                            [--desc <descr>] [--header] [--help]
                                            --name <name> [--sep <separator>]
                                            [--size <size>]
                                            [--trust-store <truststore_file>]
                                            [--verbose] [--xml]
   Options:
 
   --access-key-file|-a <access-key-file>  Path to the access key file.
   --base-url <base_url>                   Base URL of the Oracle Enterprise
                                           Manager Ops Center vServer secure url
                                           (https://<ochost>).
   --debug|-D                              Start the command in debug mode.
   --desc|-d <descr>                       An (optional) description.
   --header|-H                             Output a header row (default no
                                           header).
   --help|-h                               Print this usage message.
   --name|-n <name>                        Name of the resource.
   --sep <separator>                       The column separator character(s),
                                           default TAB.
   --size|-s <size>                        size of the distribution group
                                           (default size is 50000).
   --trust-store <truststore_file>         Path to the file containing the
                                           trusted SSL certificates (default
                                           $HOME/.oracle_iaas/truststore).
   --verbose|-v                            Produce verbose output (for
                                           debugging).
   --xml                                   Output result in XML format (default
                                           table format).
Note: All commands are executed remotely as the specified or AccessKey user
      The for-user option may be used to execute on behalf of another user

There one thing that we are still waiting for:

1. Cross account networking over infiniband: in one project we thinked about having one account with common things (OTD, OSB or both for example) and those common things being able to forward requests to components in other accounts, but doing this over IB. Until EECS 2.0.4 (and I will check 2.0.6 in more detail) is not possible to do something like that. 

So, very good news from the echo I hope everyone can enjoy the powerful of the new Exalogic  Elastic Cloud Software 2.0.6.

P.S. (in 20.09.2013):

2. Today is not possible to change any configuration from an vServer without recreating the entire machine, this is related with the way that the vServer network interfaces over the xenbridge are created and cataloged over the two  catalogs (OVM and EMOC) and also over the vServer configurations file. My wish is to only CPU and RAM (that do not change complex xenbridge over infiniband partitions configurations) can be changed without recreating the entire machines.

P.S. (in 01.10.2013):
  
In EECS 2.0.6 Oracle VM Manager will be upgraded to v3.2.1, more functionality to colect information from OVM WS interface. For customers that want to run SAP inside a virtualized Exalogic now is possible.

Is all about philosophy

Hi all, this is my blog to share Oracle Exalogic Virtualized or Oracle VM experiences, looking how to use the Oracle engineered systems to achieve an "autonomic behavior". My approach is to as much as possible automate and simplify the Oracle virtualization/hardware solutions without loosing the ability to fine grain configure the environment as we want. 

The scope of this blog will be (more or less) Exalogic Elastic Cloud Software, OVM , ZFS, Infiniband Networking and Oracle Fusion Middlware software stack (that is a lot of things), and how to architect, manage and at the end just put your feet on the table and look at the graphics in Oracle Cloud Control (this actually don't work because a I was not able to grab the mouse to switch between CC12c tabs). 

So this will be a bunch of learned good practices, Elastic Cloud Philosophy and bit brushing.

I hope they enjoy it.