Tutorial 7: Update a DNS Entry in the Service Director
This tutorial describes how to create a service and add a DNS entry for the service in the Platform EGO Service Director.
Using this tutorial, you will ...
- Open a connection to Platform EGO
- Print out cluster information
- Check if there are any registered clients connected to Platform EGO
- Log on to Platform EGO
- Register the client with Platform EGO
- Print out allocation and container reply info from a previous connection
- Print out host group information
- Request resource allocation from Platform EGO and print the allocation ID
- Create, run, and query a service
- Query for the IP address of the host where the service is running
- Update the Service Director with a new entry for service instance location.
Underlying principles
Platform EGO features a plug-in model for the Service Director. There is a default plug-in that comes with Platform EGO, however different Service Director elements can be installed. This sample uses the default plug-in, which implements a DNS server for storing location information about the service instances. At runtime, the Service Director's shared library can be loaded, and appropriate methods (declared in the sdplugin.h file) can be looked up and invoked.
In order to communicate with a service on a host cluster, its location must be known. Due to the nature of distributed computing, the service can be running on any host. Normally, when a service instance switches into the run state, the Service Controller sends a notification to the Service Director that includes location information of the service instance. The Service Director then adds the location record to its DNS server. In this sample, we create a service called "Sample7_mysql_service". When this service is running, the Service Director automatically updates its records to reflect the new service instance. However, to demonstrate how to manually update the Service Director records, we also create an alias to this service instance and query the service to reveal its network address.
Step 1: Preprocessor directives and method declarations
The first step is to include a reference to the system and API header files, followed by the declaration of methods and variables that are implemented in the sample.
#include <stdlib.h> #include <stdio.h> #include <unistd.h> #include <string.h> #include <errno.h> #include <pthread.h> #include <dlfcn.h> #include <netdb.h> #include "vem.api.h" #include "esc.api.h" #include "samples.h" #include "esd.h" #include "esdplugin.h" static int addResourceCB(vem_allocreply_t *areply); static int reclaimForceCB(vem_allocreclaim_t *areclaim); static int containerStateChgCB(vem_containerstatechg_t *cschange); static int hostStateChangeCB(vem_hoststatechange_t *hschange); static char *get_service_def_xml(); void printSDEntry(char *sname); service_state_t *service_stateP; esc_service_info_reply_t *service_info_reply;Step 2: Implement the principal method
Lines 4-7: define and initialize a data structure that is used to request a connection with the EGO host cluster. The data structure contains a reference to a configuration file where the master host name and port numbers are stored.
Line 8: pass the data structure as an argument to the vem_open () method, which opens a connection to the master host. If the connection attempt is successful, a handle is returned; otherwise the method returns NULL. The handle acts as a communication channel to the master host and all subsequent communication occurs through this handle.
Lines 14-15: the vem_name_t structure (defined as clusterName) is initialized with NULL. This structure holds the cluster name, system name, and version. The vem_uname () method is passed the communication handle and, if successful, returns a valid vem_name_t structure ; otherwise the method returns NULL.
Line 23: the cluster info is printed out to the screen.
Lines 26-43: define the client info structure. Use vem_locate() to get all registered clients. Since NULL is provided as the client name, all registered clients will be located and the method returns the number of registered clients. Note that Platform EGO is equipped with a number of default clients (services) such as the Service Controller, so as a minimum, the info relevant to these clients is printed out and the associated memory is released.
1 int 2 sample7() 3 { 4 vem_openreq_t orequest; 5 vem_handle_t *vhandle = NULL; 6 orequest.file = "ego.conf"; 7 orequest.flags=0; 8 vhandle = vem_open(&orequest); 9 if (vhandle == NULL) { 10 // error opening 11 fprintf(stderr, "Error opening cluster: %s\n", vem_strerror(vemerrno)); 12 return -1; 13 } 14 vem_name_t *clusterName = NULL; 15 clusterName = vem_uname(vhandle); 16 if (clusterName == NULL) { 17 // error connecting 18 fprintf(stderr, "Error connecting to cluster: %s\n", 19 vem_strerror(vemerrno)); 20 return -2; 21 } 22 23 fprintf(stdout, " Connected... %s %s %4.2f\n", clusterName->clustername, 24 clusterName->sysname, clusterName->version); 25 26 vem_clientinfo_t *clients; 27 int rc = vem_locate(vhandle, NULL, &clients); 28 if (rc >=0) { 29 if (rc == 0) { 30 printf("No registered clients exist\n"); 31 } else { 32 int i=0; 33 for (i=0; i<rc; i++) { 34 printf("%s %s %s\n", clients[i].name, clients[i].description, 35 clients[i].location); 36 } 37 // free 38 vem_clear_clientinfo(clients); 39 } 40 } else { 41 // error connecting 42 fprintf(stderr, "Error geting clients: %s\n", vem_strerror(vemerrno)); 43 }Lines 44-46: authenticate the user to Platform EGO.
Lines 48-52: define and initialize a structure for callback methods. These callback methods are invoked by Platform EGO when resources are added or reclaimed, or when a change occurs to host status or a container. When Platform EGO wants to communicate about these events, it invokes these methods thereby calling back to the client.
Lines 54-63: define the vem_allocation_info_reply_t and vem_container_info_reply_t structures. If a client gets disconnected and then re-registers, its existing allocations and containers are returned to these structures. If the client had never registered before, the structures would be empty. Define and initialize a structure (rreq) that holds client info for registration purposes. (This includes assigning the client callback structure (cbf) to the callback member of the rreq structure.) Register with Platform EGO via the open connection using vem_register().
44 if (login(vhandle, username, password)<0) { 45 fprintf(stderr, "Error logon: %s\n", vem_strerror(vemerrno)); 46 goto leave; 47 } 48 vem_clientcallback_t cbf; 49 cbf.addResource = addResourceCB; 50 cbf.reclaimForce = reclaimForceCB; 51 cbf.containerStateChg = containerStateChgCB; 52 cbf.hostStateChange = hostStateChangeCB; 53 54 vem_allocation_info_reply_t aireply; 55 vem_container_info_reply_t cireply; 56 vem_registerreq_t rreq; 57 rreq.name = "sample7_client"; 58 rreq.description = "Sample7 Client"; 59 rreq.flags = VEM_REGISTER_TTL; 60 rreq.ttl = 3; 61 rreq.cb = &cbf; 62 63 rc = vem_register(vhandle, &rreq, &aireply, &cireply); 64 if (rc < 0) { 65 fprintf(stderr, "Error registering: %s\n", vem_strerror(vemerrno)); 66 goto leave; 67 }Lines 68-71: print out information related to the allocation requests and containers. Once the info is printed out, the memory for the allocations is freed.
Lines 73-84: the vem_gethostgroupinfo() method collects the information for the requested hostgroup. In this case, the requested hostgroup in the input argument is set to NULL, which means that information about all hostgroups is requested. If the method call is successful, hostgroup information is printed out to the screen.
68 print_vem_allocation_info_reply(&aireply); 69 print_vem_container_info_reply(&cireply); 70 // freeup any previous allocations 71 release_vem_allocation(vhandle, &aireply); 72 73 vem_hostgroupreq_t hgroupreq; 74 hgroupreq.grouplist = NULL; 75 vem_hostgroup_t *hgroup; 76 77 rc = vem_gethostgroupinfo(vhandle, &hgroupreq, &hgroup); 78 if (rc < 0) { 79 fprintf(stderr, "Error getting hostgroup: %s\n", 80 vem_strerror(vemerrno)); 81 } else { 82 printf("%s %s %d %d\n", hgroup->groupName, hgroup->members, hgroup->free, 83 hgroup->allocated); 84 }Lines 85-88: define and initialize the service_stateP and client_stateP structures. These structures contain the respective mutex objects and condition variables.
Lines 90-91: get the service definition and store it in the service state structure. Refer to Step 3: Create the service definition.
Lines 93-97: create and run the service_thread. The service thread is responsible for defining and creating the Platform EGO service.
85 service_stateP = calloc(1, sizeof(service_state_t)); 86 initialize_service(service_stateP); 87 client_state_t *client_stateP = calloc(1, sizeof(client_state_t)); 88 initialize_client(client_stateP); 89 90 char *xml = get_service_def_xml(); 91 service_stateP->xml = xml; 92 93 pthread_t service_thread; 94 if (pthread_create(&service_thread, NULL, service_thread_fn, 95 service_stateP)) { 96 perror("Error creating service thread: "); 97 }Lines 98-113: use vem_locate() to get all registered clients. Since NULL is provided as the client name, all registered clients will be located and the method returns the number of registered clients. If successful, print out the client info (name, description, and location) and free the associated memory.
Lines 120-123 lock the service_mutex object and wait until the service is running. Set the service condition variable to unblock the service thread, which causes the service thread to query the service.
Lines 129-132: lock the client_mutex object and block the main thread with the client condition variable. When the service info is available, the service thread signals the main thread to resume execution using the client condition variable. The main thread prints out the service info reply.
98 rc = vem_locate(vhandle, NULL, &clients); 99 if (rc >=0) { 100 if (rc == 0) { 101 printf("No registered clients exist\n"); 102 } else { 103 int i=0; 104 for (i=0; i<rc; i++) { 105 printf("%s %s %s\n", clients[i].name, clients[i].description, 106 clients[i].location); 107 } 108 vem_clear_clientinfo(clients); 109 } 110 } else { 111 // error connecting 112 fprintf(stderr, "Error geting clients: %s\n", vem_strerror(vemerrno)); 113 } 114 sleep(60); 115 116 // get service info 117 service_info_reply = calloc(1, sizeof(esc_service_info_reply_t)); 118 int succeed=0; 119 while(!succeed) { 120 pthread_mutex_lock(&service_stateP->service_mutex); 121 if (service_stateP->ready && service_stateP->client == NULL) { 122 service_stateP->client = client_stateP; 123 pthread_cond_signal(&service_stateP->service_cond); 124 succeed = 1; 125 } 126 pthread_mutex_unlock(&service_stateP->service_mutex); 127 } 128 fprintf(stderr, "Sent Request to Service:\n"); 129 pthread_mutex_lock(&client_stateP->client_mutex); 130 pthread_cond_wait(&client_stateP->client_cond, 131 &client_stateP->client_mutex); 132 print_service_info_reply(service_info_reply); 133 pthread_mutex_unlock(&client_stateP->client_mutex);Lines 134-140: store the service info in the esc_service_info_t structure and retrieve the number of service instances. Assign the service instance info to the instance_info structure. Allocate and initialize an array in memory to hold the IP address associated with each service instance.
Lines 144- 149: for each service instance, retrieve the host name where the service instance is running and pass it to the gethostbyname() method. The method returns a pointer to a hostent structure (locationService), the members of which contain the fields of an entry in the network host database. The hostent structure is defined in the <netdb.h> header. Extract the host's IP address from the locationService structure and assign it to the ipAddresses array.
Lines 149-161: pass "mysql", an alias for "Sample7_mysql_service", to the printSDEntry() method. The method will query the service and print the IP address of the host that the service is running on. However, since there is currently no entry in the Service Director records for a service called "mysql", nothing is printed. (The printSDEntry() method is used again later in the sample to print out the IP address after the Service Director records have been properly updated.)
134 esc_service_info_t *service_info = &service_info_reply->serviceV[0]; 135 if (service_info == NULL) { 136 fprintf (stderr, "Could not get Service Info\n"); 137 goto done; 138 } 139 int i, num_instances; 140 num_instances = service_info->instC; 141 int ipaddrC = num_instances; 142 char **ipAddresses = calloc(ipaddrC, sizeof(char*)); 143 144 for(i=0; i < num_instances; i++) { 145 esc_service_instance_info_t *instance_info = &service_info->instV[i]; 146 char *location = instance_info->host; 147 struct hostent * locationService = gethostbyname(location); 148 ipAddresses[i] = locationService->h_addr_list[0]; 149 } char *sname = "mysql"; 150 printSDEntry(sname);
151 void 152 printSDEntry(char *sname) 153 { 154 int i=0; 155 struct hostent * dnsService = gethostbyname(sname); 156 fprintf(stderr, "Existing Entry for %s\n", sname); 157 if (dnsService != NULL) { 158 for (i=0; i<dnsService->h_length; i++) { 159 fprintf(stderr, "%s\n", dnsService->h_addr_list[i]); 160 } 161 }The following paragraphs demonstrate how to manually add a new entry in the Service Director that provides service instance location information.
Lines 162-167: define and initialize a structure (update) to hold service instance info that will be used to add a new record in the Service Director.
Lines 169-175: use dlopen() to open a shared library (esd_ego_default.so); this is the default Service Director plug-in. The dlopen() method returns a handle to the shared library. Define a structure to initialize all the plug-in variables. Pass the shared library handle and a function name (esd_plugin_initialize) to dlsym(). The method looks up the address of the function in the shared library handle obtained from dlopen.
Lines 179-189: set up the necessary plug-in variables and invoke the initialization method from the shared library.
Lines 190-191: pass the update structure to the update_service() method. The method adds a service location entry in the Service Directorconsisting of an alias (mysql) with the same IP address that was printed by the previous service query in the service thread; see Tutorial 6: Step 4: Query the service. The printSDEntry() method prints out the new service location entry.
162 si_updrec_t update; 163 sd_update_oper_t op = op_add; 164 update.name = sname; 165 update.ipaddrC = ipaddrC; 166 update.ipaddrV = ipAddresses; 167 update.oper = op; 168 169 void * esd_plugin = dlopen("esd_ego_default.so", RTLD_LAZY); 170 if (esd_plugin == NULL) { 171 fprintf (stderr, "%s\n", dlerror()); 172 } 173 esd_plugin_initialize_t esd_plugin_init; 174 175 void *fptr = dlsym(esd_plugin, "esd_plugin_initialize"); 176 if (fptr == NULL) { 177 fprintf (stderr, "%s\n", dlerror()); 178 } 179 * (void **) &esd_plugin_init = fptr; 180 181 esd_utility_funcs_t utility_functions; 182 utility_functions.calloc = calloc; 183 utility_functions.malloc = malloc; 184 utility_functions.free = free; 185 utility_functions.realloc = realloc; 186 utility_functions.strdup = strdup; 187 const char *param = ""; 188 esd_plugin_t sd_plugin; 189 (*esd_plugin_init)(&utility_functions, param, &sd_plugin); 190 sd_plugin.update_service(&update); 191 printSDEntry(sname);Step 3: Create the service definition
Create a service definition in XML format. Each service is described by an XML file that contains information about the service such as the type of resources required to run the service instances and how to start and monitor them.
char * get_service_def_xml() { char *xml = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\ <sc:ServiceDefinition xmlns:sc= \"http://www.platform.com/ego/2005/05/schema/sc\" xmlns:ego= \"http://www.platform.com/ego/2005/05/schema\" xmlns:xsi= \"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd= \"http://www.w3.org/2001/XMLSchema\" xsi:schemaLocation= \"http://www.platform.com/ego/2005/05/schema/sc ../sc.xsd http://www.platform.com/ego/2005/05/schema ../ego.xsd\" ServiceName= \"Sample7_mysql_service\">\ <sc:Description>\"Sample 7 MySQL Service\"</sc:Description>\ <sc:MinInstances>1</sc:MinInstances>\ <sc:MaxInstances>1</sc:MaxInstances>\ <sc:Priority>10</sc:Priority>\ <sc:MaxInstancesPerSlot>1</sc:MaxInstancesPerSlot>\ <sc:ControlPolicy>\ <sc:StartType>MANUAL</sc:StartType>\ <sc:MaxRestarts>10</sc:MaxRestarts>\ <sc:HostFailoverInterval>60s</sc:HostFailoverInterval>\ </sc:ControlPolicy>\ <ego:AllocationSpecification>\ <ego:ConsumerID>/SampleApplications/EclipseSamples</ego:ConsumerID>\ <!-- The ResourceType specifies a ""compute element"" identified by the URI used below -->\ <ego:ResourceSpecification ResourceType= \"http://www.platform.com/ego/2005/05/schema/ce\">\ <ego:ResourceGroupName>ComputeHosts</ego:ResourceGroupName>\ <ego:ResourceRequirement>LINUX86</ego:ResourceRequirement>\ </ego:ResourceSpecification>\ </ego:AllocationSpecification>\ <sc:ContainerDescription>\ <ego:Attribute name=\"hostType\" type=\"xsd:string \">LINUX86</ego:Attribute>\ <ego:ContainerSpecification>\ <ego:Command>bin/mysql --user mysql</ego:Command>\ <ego:RunAsOSUser>root</ego:RunAsOSUser>\ <ego:WorkingDirectory>/usr/local/mysql</ego:WorkingDirectory>\ <ego:Umask>0777</ego:Umask>\ </ego:ContainerSpecification>\ </sc:ContainerDescription>\ </sc:ServiceDefinition> \"; return xml; }Step 4: Create the service
In this sample, service creation is handled by the service thread. The service simply starts the mySQL program.
Lines 194-204: set the EGO_CONFDIR environment variable to the current working directory so that the Service Controller, which is just another Platform EGO client, can find the Platform EGO configuration. The configuration is stored in the ego.conf file.
Lines 205-208: define and initialize a security structure. The username and password variables have been initialized with "egoadmin" from sample # 2.
Line 210: create a service definition in XML format. Each service is described by an XML file that contains information about the service such as the type of resources required to run the service instances and how to start and monitor them.
Lines 212-219: pass the security structure and the service definition to the esc_createservice() method. This API call creates a new service object. If startType is automatic in the service definition, the service is enabled automatically. In this case, the startType is set to manual so the service must be enabled by the esc_enableservice() method. This API call starts the service. Once the service is started, the Service Controller allocates resources and starts service instances. A service can be started by esc_enableservice() or by starting a service that depends on it.
192 void *service_thread_fn(service_state_t *service_stateP) 193 { 194 int size=4096; 195 char *buf = calloc(size, sizeof(char)); 196 sprintf(buf, "EGO_CONFDIR="); 197 char *cwd = getcwd((buf+12), size-12); 198 int errn = errno; 199 if(cwd != NULL) { 200 putenv(buf); 201 } else { 202 fprintf(stderr, "Error getting CWD: %s\n", 203 strerror(errn)); 204 } 205 esc_security_def_t security; 206 security.username = username; 207 security.password = password; 208 security.credential = NULL; 209 char *sname = "sample6_service"; 210 char *xml = get_service_def_xml(); 211 212 if(esc_createservice(xml, &security)) { 213 fprintf(stderr, "Error creating service: %s\n", 214 esc_strerror(escerrno)); 215 //goto bailout; 216 } 217 if(esc_enableservice(sname, &security)) { 218 fprintf(stderr, "Error enabling service: %s\n", 219 esc_strerror(escerrno)); 220 }Step 5: Client callback methods
These callback methods are invoked by Platform EGO when resources are added or reclaimed, or when a change occurs to host status or a container. When Platform EGO wants to communicate about these events, it invokes these methods thereby calling back to the client.
Lines 221-227: this method is called by Platform EGO when resources have been added to an allocation in order to tell the client which resources have been provided for its use. This method prints out the allocation and consumer IDs, the number of hosts allocated, host names and number of slots, and host attributes.
Lines 229-235: this method is called by Platform EGO when resources need to be reclaimed. Resources may be reclaimed either for policy reasons, or because a resource has been found to be down or unavailable. The method prints out the host info including host name and slots for each host being reclaimed.
Lines 237-243: this method is called by Platform EGO in order to communicate status changes in containers to the clients that started them. The method prints out the container ID and its associated state.
Lines 245-251: this method is called by Platform EGO when a host changes state. The method prints out the host name and its new host state.
221 int 222 addResourceCB(vem_allocreply_t *areply) 223 { 224 printf("addResource Call Back\n"); 225 print_vem_allocreply(areply); 226 return 0; 227 } 228 229 int 230 reclaimForceCB(vem_allocreclaim_t *areclaim) 231 { 232 printf("reclaimForce Call Back\n"); 233 print_vem_allocreclaim(areclaim); 234 return 0; 235 } 236 237 int 238 containerStateChgCB(vem_containerstatechg_t *cschange) 239 { 240 printf("containerStateChg Call Back\n"); 241 printf("%s %d\n", cschange->containerId, cschange->newState); 242 return 0; 243 } 244 245 int 246 hostStateChangeCB(vem_hoststatechange_t *hschange) 247 { 248 printf("hostStateChange Call Back\n"); 249 printf("%s %d\n", hschange->name, hschange->newState); 250 return 0; 251 }Run the client application
- Select Run > Run.
The Run dialog appears.
- In the Configurations list, either select an EGO C Client Application or click New for a new configuration.
For a new configuration, enter the configuration name.
- Enter the project name and C/C++ Application name.
- Click Apply and then Run.
[ Top ]
[ Platform Documentation ]
Date Modified: July 12, 2006
Platform Computing: www.platform.com
Platform Support: support@platform.com
Platform Information Development: doc@platform.com
Copyright © 1994-2006 Platform Computing Corporation. All rights reserved.