Solaris processor sets and resource pools

As a follow up on the earlier posting about setting up hard cpu limits under Solaris. That posting didn't address processor sets and resource pools, this will. The basic steps are as follows:

1. build a processor set with a number of CPU's assigned
2. bind the processor set to a resource pool
3. bind the zone to the same resource pool



1. I’ll create three non-global zones. The first is bound to a two processor set and the second two zones are bound to a quad processor set. In the global zone, list the available CPU’s:


root@x4140> psrinfo
0 on-line since 08/11/2009 13:18:42

1 on-line since 08/11/2009 13:18:48

2 on-line since 08/11/2009 13:18:50

3 on-line since 08/11/2009 13:18:52

4 on-line since 08/11/2009 13:18:54

5 on-line since 08/11/2009 13:18:56

6 on-line since 08/11/2009 13:18:58

7 on-line since 08/11/2009 13:19:00
2. Enable the resource pool facility on the global zone:

root@x4140> pooladm –e


3. Save the configuration:

root@x4140> pooladm –s


4. List the pools:

root@x4140> pooladm
system default
string system.comment
int system.version 1
boolean system.bind-default true
string system.poold.objectives wt-load

pool pool_default
int pool.sys_id 0
boolean pool.active true
boolean pool.default true
int pool.importance 1
string pool.comment
pset pset_default

pset pset_default
int pset.sys_id -1
boolean pset.default true
uint pset.min 1
uint pset.max 65536
string pset.units population
uint pset.load 6
uint pset.size 8
string pset.comment

cpu
int cpu.sys_id 7
string cpu.comment
string cpu.status on-line

cpu
int cpu.sys_id 4
string cpu.comment
string cpu.status on-line

cpu
int cpu.sys_id 1
string cpu.comment
string cpu.status on-line

cpu
int cpu.sys_id 6
string cpu.comment
string cpu.status on-line

cpu
int cpu.sys_id 3
string cpu.comment
string cpu.status on-line

cpu
int cpu.sys_id 0
string cpu.comment
string cpu.status on-line

cpu
int cpu.sys_id 5
string cpu.comment
string cpu.status on-line

cpu
int cpu.sys_id 2
string cpu.comment
string cpu.status on-line


5. Create the processor set that contains two CPU's:


root@x4140> poolcfg -c 'create pset duo-pset (uint pset.min=1; uint pset.max=4)'


6. Create a resource pool for the processor set:


root@x4140> poolcfg -c 'create pool duo-pool'


7. Assign the resource pool to the processor set:


root@x4140> poolcfg -c 'associate pool duo-pool (pset duo-pset)'


8. Enable the configuration:


root@x4140> pooladm –c


9. List the changes:

root@x4140> pooladm

system default

string system.comment

int system.version 1

boolean system.bind-default true

string system.poold.objectives wt-load

pool duo-pool

int pool.sys_id 1

boolean pool.active true

boolean pool.default false

int pool.importance 1

string pool.comment

pset duo-pset

pool pool_default

int pool.sys_id 0

boolean pool.active true

boolean pool.default true

int pool.importance 1

string pool.comment

pset pset_default

pset duo-pset

int pset.sys_id 1

boolean pset.default false

uint pset.min 1

uint pset.max 4

string pset.units population

uint pset.load 0

uint pset.size 2

string pset.comment

cpu

int cpu.sys_id 3

string cpu.comment

string cpu.status on-line

cpu
int cpu.sys_id 4

string cpu.comment

string cpu.status on-line

pset pset_default

int pset.sys_id -1

boolean pset.default true

uint pset.min 1

uint pset.max 65536

string pset.units population

uint pset.load 2

uint pset.size 4

string pset.comment

cpu
int cpu.sys_id 7

string cpu.comment

string cpu.status on-line

cpu
int cpu.sys_id 4

string cpu.comment

string cpu.status on-line

cpu

int cpu.sys_id 6

string cpu.comment

string cpu.status on-line

cpu

int cpu.sys_id 5

string cpu.comment

string cpu.status on-line


10. Make the changes to the zone:


root@x4140> zonecfg -z zone
zonecfg:zone> set pool=duo-pool
zonecfg:zone> verify
zonecfg:zone> commit
zonecfg:zone> exit


11. Create a second processor set with four CPU's and assign it to a new resource pool:


root@x4140> poolcfg -c 'create pset quad-pset (uint pset.min=1; uint pset.max=4)'
root@x4140> poolcfg -c 'create pool quad-pool'
root@x4140> poolcfg -c 'associate pool quad-pool (pset quad-pset)'
root@x4140> pooladm -c


12. Make changes to the second and third zones:


root@x4140> zonecfg -z zone2
zonecfg:zone> set pool=quad-pool
zonecfg:zone> verify
zonecfg:zone> commit
zonecfg:zone> exit

root@x4140> zonecfg -z zone3
zonecfg:zone3> set pool=quad-pool
zonecfg:zone3> verify
zonecfg:zone3> commit
zonecfg:zone3> exit


13. List the processors available to the zones:


root@x4140> zlogin zone

[Connected to zone 'zone' pts/3]

Last login: Mon Aug 17 16:25:23 on pts/3

Sun Microsystems Inc. SunOS 5.10 Generic January 2005

# psrinfo

3 on-line since 08/17/2009 16:46:44

4 on-line since 08/17/2009 16:46:46

root@x4140-a> zlogin zone2

[Connected to zone 'zone2' pts/3]

Last login: Mon Aug 17 16:25:39 on pts/3

Sun Microsystems Inc. SunOS 5.10 Generic January 2005

# psrinfo

0 on-line since 08/17/2009 16:46:34

1 on-line since 08/17/2009 16:46:40

2 on-line since 08/17/2009 16:46:42


root@x4140-a> zlogin zone3

[Connected to zone 'zone3' pts/4]

Sun Microsystems Inc. SunOS 5.10 Generic January 2005

# psrinfo

0 on-line since 08/17/2009 16:46:34

1 on-line since 08/17/2009 16:46:40

2 on-line since 08/17/2009 16:46:42


14. Then if we want to, we can enable the Fair Share Scheduler so that zone2 and zone3 share the processors in the quad-pool, while zone has dedicated access to two cpu's and doesn't share any cpu's with any other zones.


root@x4140> poolcfg -c 'modify pool quad-pool (string pool.scheduler="FSS")'
root@x4140> pooladm -c
[reboot zones or run this command]
root@x4140> priocntl -s -c FSS -i class TS
root@x4140> priocntl -s -c FSS -i pid 1

Assign three shares to zone2 and one share to zone3:

root@x4140> zonecfg -z zone2
zonecfg:zone2> add rctl
zonecfg:zone2:rctl> set name=zone.cpu-shares
zonecfg:zone2:rctl> add value (priv=privileged,limit=3,action=none)
zonecfg:zone2:rctl> end
zonecfg:zone2> exit

root@x4140> zonecfg -z zone3
zonecfg:zone3> add rctl
zonecfg:zone3:rctl> set name=zone.cpu-shares
zonecfg:zone3:rctl> add value (priv=privileged,limit=1,action=none)
zonecfg:zone3:rctl> end
zonecfg:zone3> exit


The zone "zone" now runs on it's own dedicated and guaranteed CPU's protected from the workload of the other zones. zone2 and zone3 now share the other quad resource pool with zone2 getting a higher weighting than zone3 and the global zone has the remaining CPU.

Setting hard cpu limits under Solaris zones using zonecfg

Very easy to do. There are two methods:

1) set hard cpu limits
2) set cpu pools

For ease of use I'd recommend going with option 1 and here are the steps to take:

bash-3.00# zlogin myserver
[Connected to zone 'myserver' pts/1]
Last login: Thu Jul 30 11:40:18 on pts/1
Sun Microsystems Inc. SunOS 5.10 Generic January 2005

# psrinfo
4 on-line since 07/04/2009 09:53:11
5 on-line since 07/04/2009 09:53:11
6 on-line since 07/04/2009 09:53:11
7 on-line since 07/04/2009 09:53:11
8 on-line since 07/04/2009 09:53:11
9 on-line since 07/04/2009 09:53:11
10 on-line since 07/04/2009 09:53:11
11 on-line since 07/04/2009 09:53:11
12 on-line since 07/04/2009 09:53:11
13 on-line since 07/04/2009 09:53:11
14 on-line since 07/04/2009 09:53:11
15 on-line since 07/04/2009 09:53:11
16 on-line since 07/04/2009 09:53:11
17 on-line since 07/04/2009 09:53:11
18 on-line since 07/04/2009 09:53:11
19 on-line since 07/04/2009 09:53:11
20 on-line since 07/04/2009 09:53:11
21 on-line since 07/04/2009 09:53:11
22 on-line since 07/04/2009 09:53:11
23 on-line since 07/04/2009 09:53:11
# exit

[Connection to zone 'myserver' pts/1 closed]

bash-3.00# zonecfg -z myserver
zonecfg:maass> add dedicated-cpu
zonecfg:maass:dedicated-cpu> setncpus=4
zonecfg:maass:dedicated-cpu> end
zonecfg:maass> exit



Reboot the zone and log back in:

bash-3.00# zlogin myserver
[Connected to zone 'myserver' pts/1]
Last login: Thu Jul 30 11:44:59 on pts/1
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
# psrinfo
4 on-line since 07/04/2009 09:53:11
5 on-line since 07/04/2009 09:53:11
6 on-line since 07/04/2009 09:53:11
7 on-line since 07/04/2009 09:53:11

Fibre Channel card initialization

I had a new SUN M4000 server with two identical QLOGIC FC cards installed. The first FC interface came up without any problem and I kind of forgot about the second one.

Well, in order to move the server into production I needed to multipath the two cards together using the stmsboot -e command, but the second card wasn't responding. I checked the physical connections between the FC switch and the server, but that wasn't it. I tried to poke the system using the cfgadm command -- no luck...


# cfgadm -c configure c2
# cfgadm -al
Ap_Id Type Receptacle Occupant Condition
[...]
c1 fc-fabric connected configured unknown
c2 fc connected unconfigured unknown

# fcinfo hba-port
HBA Port WWN: xxxxxxxx294
OS Device Name: /dev/cfg/c1
Manufacturer: QLogic Corp.
Model: 375-3355-02
Firmware Version: 4.04.01
FCode/BIOS Version: BIOS: 1.24; fcode: 1.24; EFI: 1.8;
Serial Number: 0402G00-0905674110
Driver Name: qlc
Driver Version: 20080617-2.29
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: 4Gb
Node WWN: xxxxxxxx294
HBA Port WWN: xxxxxxxx594
OS Device Name: /dev/cfg/c2
Manufacturer: QLogic Corp.
Model: 375-3355-02
Firmware Version: 4.04.01
FCode/BIOS Version: BIOS: 1.24; fcode: 1.24; EFI: 1.8;
Serial Number: xxx-xxx
Driver Name: qlc
Driver Version: 20080617-2.29
Type: unknown
State: offline
Supported Speeds: 1Gb 2Gb 4Gb
Current Speed: not established
Node WWN: xxxxxxxx594

# luxadm -e port
/devices/pci@3,700000/SUNW,qlc@0/fp@0,0:devctl CONNECTED
/devices/pci@2,600000/SUNW,qlc@0/fp@0,0:devctl NOT CONNECTED

No commands would bring the card to life so the FC switch wouldn't see anything attached to the port.

After a lot of troubleshooting I took it to basics and halted the system and reset the bus:

{1} ok
{1} ok reset-all
{1} ok probe-scsi-all

now the second card was responding, so I rebooted the server:

{1} ok boot -rv.

and now it's fine...

# cfgadm -al
[...]
c1 fc-fabric connected configured unknown
[bunch of disks]
c2 fc-fabric connected configured unknown
[bunch of disks]

Solaris 10 routes

If you have a Solaris 10 server with several NIC's and a different subnet plugged into each NIC, you might set up zones to use a specific NIC. In order for the non-global zone to route correctly, you will need to set up routing on the global zone.

I've found that putting in a startup script in /etc/rc3.d/S99add-route doesn't always work.

By the way, I use these commands in the script:

#!/bin/sh
/usr/sbin/ifconfig nxge0 plumb
/usr/sbin/ifconfig nxge1 plumb
/usr/sbin/route add default 10.10.10.1 -ifp nxge0
/usr/sbin/route add default 10.10.20.1 -ifp nxge1

It seems that SUN has added a "-p" flag to the route command that now makes the statement persistent across reboots, rendering the script moot:

# /usr/sbin/route -p add default 10.10.10.1 -ifp nxge0
add net default: gateway 10.10.10.1: entry exists
add persistent net default: gateway 10.10.10.1

# /usr/sbin/route -p add default 10.10.20.1 -ifp nxge1
add net default: gateway 10.10.20.1: entry exists
add persistent net default: gateway 10.10.20.1

The entries are written to the file: /etc/inet/static_routes

# File generated by route(1M) - do not edit.
default 10.10.10.1 -ifp nxge0
default 10.10.20.1 -ifp nxge1

Enable syslog for VMWare ESX3

# vi /etc/syslog.conf

add *.* @IPaddress_of_syslog_server

# esxcfg-firewall -o 514,udp,out,syslog
# service syslog restart
Shutting down kernel logger: [ OK ]
Shutting down system logger: [ OK ]
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]

Your syslog server should now be receiving the ESX3 server's syslog information.

ps on Solaris -- zone tip

This is a mini-tip. I was running a ps command to see if snmp was running on a particular host and got back way too much information:

#  ps -ef |grep snmp
   root   938     1   1   Jan 17 ?        2906:56 /usr/local/sbin/snmpd -r
   root 13536     1   0   Jan 25 ?          59:46 /usr/local/sbin/snmpd -r
   root  2545     1   0   Jan 17 ?          19:32 /usr/local/sbin/snmpd -r
   root  6050 26619   0 08:48:36 pts/2       0:00 grep snmp

At a glance I wanted to know what host the command was running on and it wasn't obvious, so I looked at the man page for the ps command and sure enough there was a new option that I didn't know about -- the "Z" option, which tells you which zone it's running on.

Very nice job SUN! Here's the new command:

# ps -efZ |grep snmp
  global    root   938     1   1   Jan 17 ?        2907:03 /usr/local/sbin/snmpd -r
   host1    root 13536     1   0   Jan 25 ?          59:46 /usr/local/sbin/snmpd -r
   host2    root  2545     1   0   Jan 17 ?          19:32 /usr/local/sbin/snmpd -r
  global    root  6189 26619   0 08:50:32 pts/2       0:00 grep snmp

Upgrading Solaris on hosts without a DVD or CDROM drive

http://docs.sun.com/app/docs/doc/817-5504/6mkv4nh96?a=view

This is the easiest way to upgrade hosts that don't have a local cdrom or DVD disk.

One thing to remember is that the two hosts need to be on a subnet that have no other tftpboot servers...