UberJS: A More Flexible Jumpstart System John Dickerson Brett Trotter Iowa State University Installing Solaris usually comes down to a choice between using CDs or Jumpstart. Nobody wants to install more than a couple of machines using CDs and a generic configuration, so most administrators use Sun's Jumpstart system if they want to customize their builds. Jumpstart is a powerful tool that makes unattended Solaris installs possible with fine-grained control over machine configurations, but it has a few shortcomings. Jumpstart is fairly limited in booting flexibility, but is still very capable when it comes to machine profiles. Jumpstart only works if the Jumpstart boot server is on the same network segment as the machine you're installing, and before you can do a network build of a machine you have to configure the Jumpstart server using Sun's add_install_client script. If you've made an error in the one of the many Jumpstart configuration files and you're down the hall from the server, you can burn through a lot of shoe leather because of a typo. In the College of Engineering at Iowa State University, we wanted to make Jumpstart a little easier. We install Sun boxes on many LANs where we can't afford a Jumpstart server or it isn't practical to manage one. For a while we built our machines using Jumpstart on an Intel laptop running Linux that we carried from building to building but the setup time got to be a headache. We needed a solution that was portable, easy to use, and didn't require touching the Jumpstart server for every new machine. We wanted to preserve Jumpstart's ability to apply different build profiles from a central location and we wanted unattended install capability. We've developed a system we call UberJS (for Uber JumpStart). We modeled it after RedHat's Kickstart so a person can boot a machine from CD, type a simple command and walk away. With some clever hacking of the Jumpstart install scripts and a CD burner, we created a bootable Jumpstart CD that handles the networking initialization usually provided by a Jumpstart boot server, but still allows us to access build profiles in a remote Jumpstart/Flashstart install repository. With the UberJS CD, we can jumpstart machines even when there is no Jumpstart server on the LAN. What's more, we incorporated more general configuration files that give us more flexibility and spontaneity than Sun's Jumpstart provides. To understand UberJS, let's first review what happens during a typical Jumpstart/Flashstart installs. (Kevin Amorin's excellent summary of the process is available from http://www.amorin.org/professional/jumpstart.php) When you boot a Sun box with 'boot net - install', it uses reverse ARP to get its IP address from a Jumpstart boot server which has that machine in its ethers list. Once its IP address is determined, the client sends a TFTP request to the server for a network boot kernel image. A symlink must be created in a TFTP directory on the server beforehand, linking the hex version of the IP address of the machine to the boot image for that specific hardware. The client boots and sends an rpc.bootparams request to the boot server that specifies which NFS servers the client is to use for Solaris install packages, sysidcfg (system configuration) settings, and the install profiles. The client then mounts the paths from the NFS servers, reads the sysidcfg settings, determines its build profile, and installs the packages. The rules.ok file on the install profile server tells the Sun installer what profile to use and each profile dictates either Jumpstart or Flashstart, which packages to install, and what pre and post installation scripts to run (if any). As you might expect, getting this all working can be a little tricky. UberJS gives us some key improvements over Jumpstart: >> UberJS switches the Sun installer from a CDROM-based install to a network-based install soon after it starts. More aptly, it fools Jumpstart into doing a network install by putting network mounts where it expects cdrom media. When combined with the network initialization code, this is THE major change in UberJS. >> UberJS doesn't require a reverse ARP server on the network because it looks up the IP address from a file on the CD, uses DHCP, or simply prompts you for the machine's IP address at the start. >> There is no need to TFTP the boot image since the system boots from a Solaris installation CD. >> The UberJS networking setup is more automated with features such as router auto-discovery, so it's not necessary to manage separate sysidcfg files for each network segment. >> UberJS can either read a bootparams file which is burned on the CD or wget'd once the network comes up which associates a client with a specific install and config server, or it can download a list of potential NFS server paths from an FTP server to give you a choice of install server mirrors. >> Even though you are initiating a CDROM install, you can still build according to specific machine profiles stored on a Jumpstart install server. Keep in mind that UberJS still assumes you have a Jumpstart or Flashstart install repository available via NFS that includes pre-defined build profiles. We won't go into detail about how to set that up here. Use Kevin Amorin's page or Jumpstart documentation from Sun for details. All of our scripts and methods were developed using the Solaris 9 08/03 Release. Here's a quick summary of what it takes to set up an UberJS build. 1. Extract the data from a Solaris 9 Install V1 CD and mount the slices for editing. 2. Add a script called uberjs-exec to the existing scripts. This script performs the essence of the behavior added by UberJS including prompting you for inputs, setting up the network, downloading configuration files from an FTP server, and processing of UberJS-specific configuration files. 3. Modify Sun's rcS and sysidfind scripts. Normally, the rcS and sysidfind scripts do most of the network setup and location of server paths. We modify rcS to call uberjs-exec instead to handle the networking and server path mounting. We also add a mechanism to turn off the twirling dial since it interferes with screen prompts. Sysidfind usually fetches the sysidcfg file from a network mount and copies it to the miniroot, but since uberjs-exec does all of the mounting and copying, we need to modify the sysidfind script to simply check to see if the file already exists. 4. Create a few configuration files and either copy them to the CD slices or to an accessible FTP server. The configuration files can be used to either hardcode the paths to the Jumpstart configuration files or be used to set up where those files can be downloaded at build time. 5. Make an ISO image out of the modified slices and burn a new CD. 6. Boot the client machine with 'boot cdrom - install' 7. Sit back and enjoy a hot beverage. In case you want to cut to the chase, we created a script called 'mkuber' which can automate the entire CD creation process for you. Simply place a Solaris 9 Install V1 CD in the drive and run the script. All that is left to do after the script runs is add any site-specific configuration files, as discussed below. Mkuber is available on the UberJS website. If you wish to perform the alterations manually and/or become more familiar with UberJS, the following methods will create a successful UberJS CD. Extracting the Slices from the CD (Creating a Work Space) Start by extracting the data from a Solaris 9 Install V1 CD. In case you're interested, you can view the layout of the CD beforehand to get a feeling for what's on the CD: # /etc/init.d/volmgt stop # prtvtoc /dev/dsk/c0t2d0s0 (or whatever your CD device is) The slices on the CD are as follows: Slice Contents ----- -------- 0 Software packages (The Solaris_9 directory which has the Product subdirectory) 1 Generic Kernel + root directory structure (includes install scripts) 2 boot for sun4c 3 boot for sun4m 4 boot for sun4d 5 boot for sun4u and sun4us # /etc/init.d/volmgt start Copy slice 0 from the CD to a directory on your system: # mkdir -p /MySolarisCD/s0 # cd /cdrom/cdrom0/s0 # find . -print | cpio -pdm /MySolarisCD/s0 Use dd to copy slices 1 through 5: # /etc/init.d/volmgt stop # for SLICE in 1 2 3 4 5; do dd if=/dev/dsk/c0t2d0s${SLICE} of=/MySolarisCD/sol9.s${SLICE} bs=512; done # cd / # /etc/init.d/volmgt start Note: The directory change to / is because volmgt will start out of the current directory, and if that is the cdrom directory, it will be busy if you try and un-mount it later. Modifying the Images For Solaris 9, all of the jumpstart functionality is now on slice 1. Sun moved the Solaris 9 boot tools out of slice 0 and now there's a symlink from s0/Solaris_9/Tools/Boot to slice 1 instead. With the slices extracted, we can use the Solaris lofiadm utility to mount the image of slice 1: # mkdir /mnt/slice # lofiadm -a /MySolarisCD/sol9.s1 /dev/lofi/1 # mount /dev/lofi/1 /mnt/slice /mnt/slice is just where we're mounting the slices on our Sun box while we retool the Solaris CD. Keep in mind that /mnt/slice will be mounted as / during installation. Also, Sun copies the directory /mnt/slice/.tmp_proto to the ramdisk /tmp and that's where a lot of critical files are stored and where a lot of symlinks are directed since /tmp is the only writeable filesystem on the CD. We'll put files unique to UberJS in /mnt/slice/.tmp_proto/uberjs Modifying the JumpStart Scripts The next stage of building UberJS is to modify Sun's rcS and sysidfind scripts. rcS Modification Change the following section around line 667 of /mnt/slice/sbin/rcS so that rcS calls the main UberJS script 'uberjs-exec' ---- if [ $USING_DHCP -eq 1 ] ; then echo "Using DHCP for network configuration information." dhcp_find_and_mount_cdrom else echo "Using RPC Bootparams for network configuration information." bootparams_find_and_mount_cdrom fi ---- to be ---- if [ -f /tmp/uberjs/uberjs-exec ]; then echo "Executing UberJS script: /tmp/uberjs/uberjs-exec" /sbin/sh /tmp/uberjs/uberjs-exec else if [ $USING_DHCP -eq 1 ]; then echo "Using DHCP for network configuration information." dhcp_find_and_mount_cdrom else echo "Using RPC Bootparams for network configuration information." bootparams_find_and_mount_cdrom fi fi ---- And change this section: ---- # # Start the twirling dial # if [ -x /sbin/dial ]; then dial & dial_pid=$! fi ---- to be ---- # # Start the twirling dial # touch /tmp/userjs/disable_dial to turn off the twirling # dial (it clutters up entering prompts). if [ -x /sbin/dial ]; then # Note: uses /.tmp_proto instead of /tmp since /tmp # isn't created until a few lines from here if [ -f /.tmp_proto/uberjs/disable_dial ]; then echo "Dial disabled" else dial & dial_pid=$! fi fi ---- sysidfind Modification: Sun uses the sysidfind script to fetch the sysidcfg file from the jumpstart NFS server and put it in /tmp/root/etc during install. The sysidcfg file contains general system configuration settings such as root password, language, timezone selection and other localization information. Normally, it also contains the default gateway (router) for the subnet, the netmask, and name service information. However, UberJS uses in.routed to discover the default router whenever DHCP is not used, so it is not necessary to have a different sysidcfg file on the Jumpstart server for each subnet where you build machines. UberJS lets you use a common sysidcfg configuration for your site and let the default gateway be determined automatically. UberJS will also provide the name service information from the resolv.conf that was already put on the CD. Perform the following edits to /mnt/slice/sbin/sysidfind : At line 342, change the following: ---- floppy_sysid_config if [ $? -ne 0 ]; then # Look for sysidcfg via DHCP only if the network is using DHCP if [ "X${_INIT_NET_STRATEGY}" = "Xdhcp" ]; then dhcp_sysid_config else bootparams_sysid_config fi fi ---- to be ---- if [ -f /etc/sysidcfg ]; then echo "Using sysidcfg already in /etc (or possibly a link from /tmp/uberjs)" return 0 else floppy_sysid_config if [ $? -ne 0 ]; then # Look for sysidcfg via DHCP only if the network is using DHCP if [ "X${_INIT_NET_STRATEGY}" = "Xdhcp" ]; then dhcp_sysid_config else bootparams_sysid_config fi fi fi ---- uberjs-exec The uberjs-exec script does the bulk of the work in UberJS. It is somewhat lengthy to list here in its entirety (roughly 700 lines with comments), but you can download the script from http://clue.eng.iastate.edu/uberjs. Copy the uberjs-exec script to /mnt/slice/.tmp_proto/uberjs Again, the mkuber script will do this for you. We'll just cover the key points of what uberjs-exec does here: 1. It checks to see if /tmp/uberjs/ipethers contains the MAC address of the client and will either use the IP assigned or DHCP if specified. 2. If DHCP was specified, the IP, netmask, and default route are set automatically per the DHCP protocol. 3. If no IP address was found in ipethers, DHCP will be used automatically. 4. If DHCP is used and fails, or the results are undesirable, the user can elect to input the IP and netmask. 5. If /tmp/uberjs/netmask exists, that netmask (dotted decimal) will be used, otherwise the user is prompted for it. 6. If DHCP is not used, in.routed is used to discover the default router. If this fails to determine a route, the user is prompted for it. 7. The CD should be seeded with a proper resolv.conf and nsswitch.conf, since nslookup is used to determine hostname, otherwise hostname defaults to localhost. DNS is also required if your NFS servers are not specified with IP addresses, the same for wget. 8. If wget exists, it copies any files it finds from the servers listed in wget_servers if that file exists as well. 9. If bootparams was downloaded by wget or already on the CD, it is parsed and the sysidcfg, install config, and install paths are thus obtained. 10. If bootparams does not have our hostname in it, /tmp/uberjs/nfs_servers will be checked and a menu will be presented the user can choose which set of servers to use, or choose to input manually (choosing to input manually essentially leaves the entries blank). 11. If no nfs_servers file exists, or any of the filesystem locations are left blank, the user will be prompted for the blank entries. 12. Next, the NFS filesystems are mounted and files put in place so that the Sun install itself can proceed when uberjs-exec exits back to rcS. 13. Finally, last second tweaks to sysidcfg are made so that the user is not prompted in the Suninstall phase for information about the network that has already been defined. Note that all the JumpStart scripts think we're doing a CD-based install because we booted with the command 'boot cdrom - install' and the scripts are aware of the boot arguments, so we simply unmount /cdrom and mount the NFS install server path in its place on /cdrom. We use the same general technique for the sysidcfg and profile directories. Install Configuration Files on the CD UberJS is designed to accommodate fully unattended builds as well as more ad hoc installations. The configuration files you choose to burn on the CD determine the behavior of an install. You can put all the information you'll need onto the CDs, or just some of it. Even if all the settings are present on the CD, you have some ability to override some data at the start of the build. If you have any questions about how the configuration files are used, read through the uberjs-exec script. The script is fairly well commented and (we think) easy to understand or alter for your own needs. Config Files for All Options During the actual build, it is important to remember that many files are symlinks to things in /tmp, especially in /etc where a lot of files need to be writeable. For UberJS, we did not need to create any new symlinks, but in your own modifications you may discover the need to put more in /tmp or a subdirectory. Remember to use .. to reference back to /tmp, so that the links remain relative to the cd when it is booted (eg in /mnt/slice/etc, ln -s ../tmp/root/etc/bootparams, and even if the file is never really created at that location, it will not appear as created due to the nature of symlinks). UberJS will attempt to use the DNS to resolve the client hostname. To ensure DNS works, we copy following files to /mnt/slice/.tmp_proto/root/etc, and since the /etc symlinks already exist we need not create new ones: resolv.conf A valid resolv.conf file for your network nsswitch.conf A valid nsswitch.conf file with 'hosts: files dns' (we recommend copying /mnt/slice/etc/nsswitch.dns to /mnt/slice/.tmp_proto/root/etc/nsswitch.conf) If there are any other files that you need to place in /etc, include them here as well and create symlinks in /etc when needed. Below, we present several types of install options, but there are many other variations that lie between the three generalized options given. There is a wide array of configurations from completely unattended, to entering a single 'y' or 'n', filling in a few simple blanks, or inputting nearly every option. All of which provide an unattended install once uberjs-exec exits and the sun install begins. Configuration File Options #1 - Complete Unattended Install, No FTP Server If you want the install process to run completely unattended and don't want to bother with placing config files on an FTP server, create the following files: In /mnt_slice/.tmp_proto/uberjs: ipethers File of tab-separated MAC address/IP address pairs. MAC addresses should not use leading zeros. For example: 8:0:20:c0:ff:ee 192.168.123.456 0:0:20:c0:ff:bb DHCP (The "DHCP" keyword tells UberJS to try to configure DHCP for that host) bootparams The /etc/bootparams file from your Jumpstart boot server. Used to specify paths to installation media configuration for a given host. This is only if you know the exact hostnames of every machine you'll be building. If you don't know, you'll want to use the nfs_servers file described below, but sacrifices a 100% unattended install to a single numerical selection. netmask Contains the default netmask. If you don't provide this, you will be prompted for a netmask at the start of install. disable_dial Touch this file to disable the ASCII twirling dial (highly recommended since the twirly dial gets in the way when typing prompted values.) Note: If DHCP is used, the user will need to accept the DHCP configured address. After one keystroke, the install becomes completely unattended if everything is configured correctly. Config Files Option #2 - Minimally Attended, No FTP Server The drawback to relying solely on the ipethers, netmask, and bootparams files on the CD is that every time you add more more machines to the list, you'll have to re-burn the CD, which is hardly any better than setting up a Jumpstart boot server to begin with. With UberJS, you can add some config files to the CD that will mean you'll probably have to burn a new CD less often, but you will have to provide more help at the beginning. To configure the CD for this minimally attended install, put the following files in /mnt/slice/.tmp_proto/uberjs: nfs_servers Contains a list of install, profile, and sysidcfg servers to select from. Multiple server lines are allowed. Each line is a triplet of the form: servername:/install_path servername:/profile_path servername:/sysidcfg_path If you include multiple triplets, UberJS will present them as a numbered menu and ask you which triplet you want to use for that build. netmask Sets the default netmask ( same as in Config Files Option #1 ) disable_dial Touch this file (same as in Config Files Option #1 ) Config Files Option #3 - Minimally Attended, With an FTP Server If you want the ultimate in flexibility and want to reduce as much as possible the need to re-burn the CD, you can have UberJS download most of the above files at build time from an FTP or HTTP server. This option also allows you to provide multiple mirrored servers from which to download the bootparams and/or nfs_servers file. Put the following files in /mnt/slice/.tmp_proto/uberjs: wget The GNU wget binary built for Solaris 9 (you can obtain the wget binary from http://www.sunfreeware.com. Or obtain the source from http://www.gnu.org/software/wget/wget.html and build it yourself) wget_servers A list of URLs from where it can download the config files. For example: ftp://ftpserver/my-uberjs-configs/ http://webserver/your-uberjs-configs/ disable_dial Touch this file. (Same as in Config File Options #3) If UberJS detects the wget_servers file on your CD, it will attempt to download files from each URL listed until it succeeds. Due to a limitation in the way Wget works, the path on the FTP or Web server to the config files must be in a directory in the document root of the HTTP/FTP server for UberJS to retrieve them properly. The UberJS sysidcfg File on the Jumpstart Server Since UberJS can determine the default route and domain name, you don't really need to have separate sysidcfg files for each subnet you're installing clients. You can use a generic sysidcfg file and UberJS will attempt to put proper settings for your network into the file. We recommend creating a directory on the Jumpstart sysidcfg server that contains a generic sysidcfg file for your network. For example: system_locale=en_US timezone=US/Central timeserver=123.123.123.123 terminal=sun-cmd security_policy=NONE root_password="cryptedpass" You would put this file on your Jumpstart server in a location something like: my.jump.start.server:/export/jumpstart/sysidcfg-uberjs Note: Be sure to use this as the sysidcfg path in bootparams or nfs_servers. Also, if the network_interface= or name_service= lines exist, UberJS will not attempt to modify them and you may end up being prompted for information, which is why we recommend using a file like the one above and UberJS will populate those two lines automatically. Similarly, we put our installation media in: my.jump.start.server:/export/solaris9/jumpstart (contains Solaris_9 directory) And we put our profiles/rules.ok in: my.jump.start.server:/export/jumpstart Burning the CDROM The final step in creating UberJS is to burn the modified contents back on to CD. When done creating the config files and editing/copying the scripts to the appropriate locations under /mnt/slice, un-mount the ISO with: # cd / # umount /mnt # lofiadm -d /dev/lofi/1 Mounting any other slice than s1 in this fashion produces only a file that is a boot redirector to slice 1. Creating an ISO Image There is a single line command that will produce an ISO of all the images. Another source mentions a way that involves several steps of cat'ing files together, which worked for Solaris 8 and earlier when everything was contained in slice 0 and you could simply remove the product directory, and then pad out the difference to the original size with zeros. A major problem we encountered was that we are changing slice 1 and making it bigger, so padding out the difference was not an option. We ended up using a solution that was posted in a discussion forum that uses Schily's mkisofs from the CDRTools package to create a Solaris bootable CD that has a valid VTOC. # cd /MySolarisCD # mkisofs -R -d -L -l -sparc-boot sol9.s1,sol9.s2,sol9.s3,sol9.s4,sol9.s5 -o sol9_boot.iso ./s0 Then, burn the ISO to CD using your favorite CD burning software (We copy everything to a Linux machine since we don't have a CD burner in a Sun machine). There are other means to create CD.s with valid VTOC.s. The mkisofs method seems to work for the vast majority of configurations, but it may not work for everyone. To quote the Sun Blueprint document http://www.sun.com/solutions/blueprints/0802/816-7587-10.pdf : 'Several choices are available if the slice layout of the CD being created needs to vary from that of the Software CD (for example, if the VTOC needs to be changed). Use CD creation software-such as the toolkit for building bootable CDs, available from Sun Professional Services, Gear Pro for UNIX, or Young Minds-to generate a correct and valid VTOC. Or create a new VTOC and disk label programmatically by creating and writing the dkl_vtoc and dk_label structures, respectively. See the Solaris system file /usr/include/sys/dklabel.h for more information on these structures.' Finally, to use the CD, pop it into a machine you need to build, hit stop-A, then: # boot cdrom - install We have used UberJS to install Ultra 5s, 10s, 60s, 70s, 80s, SunBlade 150s, 2050s, and SunFire V480s. If you try UberJS and have success (or problems) with other platforms, we'd love to hear from you. We've also created an alternate version of UberJS that allows you to boot a machine off a CD that provides a self-contained, fully functioning Jumpstart boot server. UberJS Server also implements the sysidcfg modification functionality of UberJS Standard, so you can use the same generic version and use all of the same paths. UberJS Server is handy if you're building a bunch of machines in a remote location where many of the clients don't have CDROM drives and you need a conventional Jumpstart boot server. More information is available on the UberJS web site. UberJS is fairly easy to modify. We put a lot of functionality into it that we think most people will find useful. But, if you find a better or more useful way to use it, feel free to modify it. We would appreciate hearing from you at our project email address: uberjs@iastate.edu. Credits While several people contributed ideas and laid groundwork that went into UberJS, Brett Trotter did all of the scripting development and vetting of Jumpstart's inner workings, plus all the web documentation. He has an uncanny ability to find ingenious problem solutions and the project could not have been done without him. Joe Mesterhazy established the initial Jumpstart and Flashstart build process that we used prior to UberJS and integrated cfengine and apt/rpm into our overall process. Matt Bradshaw created our UberKS RedHat Kickstart build CD that was the direct inspiration for UberJS. Finally, UberJS is an outgrowth from collaboration of the entire CLUE Network administration team at the College of Engineering at ISU. Thanks to them all. References The UberJS web site is: http://clue.eng.iastate.edu/uberjs. Scripts, notes and other goodies can be found there. Sun Blueprints: Customizing JumpStart Framework for Installation and Recovery - http://www.sun.com/solutions/blueprints/0802/816-7587-10.pdf Sun Blueprints: Deploying the Solaris Operating Environment Using a Solaris Security Toolkit CD - http://www.sun.com/solutions/blueprints/0903/817-3592.pdf Solaris Jumpstart Automated Installation - http://www.amorin.org/professional/jumpstart.php Creating self contained jumpstart/flash install DVD - http://dbforums.com/arch/128/2003/5/783690 Duke Sysadmins Talk - http://www.cs.duke.edu/~braun/jumpstart/index.html Bios John Dickerson is a Sr. UNIX Systems Administrator for the College of Engineering at Iowa State University. He has been working as a professional systems administrator for over fourteen years in industry and academia. Working on the CLUE Network has been a highlight of his career. He may be reached at jedicker@iastate.edu Brett Trotter is a student UNIX Systems Administrator for the College of Engineering at Iowa State University. He has had experience as a system administrator for an ISP and has been toying with various unixes for many years, but mostly he has been programming for most of his life in various languages. He may be reached at blt@iastate.edu.