Learn more about Platform products at http://www.platform.com



Tutorial 7: Update a DNS Entry in the Service Director

This tutorial describes how to create a service and add a DNS entry for the service in the Platform EGO Service Director.

Using this tutorial, you will ...

Underlying principles

Platform EGO features a plug-in model for the Service Director. There is a default plug-in that comes with Platform EGO, however different Service Director elements can be installed. This sample uses the default plug-in, which implements a DNS server for storing location information about the service instances. At runtime, the Service Director's shared library can be loaded, and appropriate methods (declared in the sdplugin.h file) can be looked up and invoked.

In order to communicate with a service on a host cluster, its location must be known. Due to the nature of distributed computing, the service can be running on any host. Normally, when a service instance switches into the run state, the Service Controller sends a notification to the Service Director that includes location information of the service instance. The Service Director then adds the location record to its DNS server. In this sample, we create a service called "Sample7_mysql_service". When this service is running, the Service Director automatically updates its records to reflect the new service instance. However, to demonstrate how to manually update the Service Director records, we also create an alias to this service instance and query the service to reveal its network address.


Step 1: Preprocessor directives and method declarations

The first step is to include a reference to the system and API header files, followed by the declaration of methods and variables that are implemented in the sample.

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <string.h>
#include <errno.h>
#include <pthread.h>
#include <dlfcn.h>
#include <netdb.h>

#include "vem.api.h"
#include "esc.api.h"
#include "samples.h"
#include "esd.h"
#include "esdplugin.h"

static int addResourceCB(vem_allocreply_t *areply);
static int reclaimForceCB(vem_allocreclaim_t *areclaim);
static int containerStateChgCB(vem_containerstatechg_t *cschange);
static int hostStateChangeCB(vem_hoststatechange_t *hschange);
static char *get_service_def_xml();

void printSDEntry(char *sname);
service_state_t *service_stateP;
esc_service_info_reply_t *service_info_reply;


Step 2: Implement the principal method

Lines 4-7: define and initialize a data structure that is used to request a connection with the EGO host cluster. The data structure contains a reference to a configuration file where the master host name and port numbers are stored.

Line 8: pass the data structure as an argument to the vem_open () method, which opens a connection to the master host. If the connection attempt is successful, a handle is returned; otherwise the method returns NULL. The handle acts as a communication channel to the master host and all subsequent communication occurs through this handle.

Lines 14-15: the vem_name_t structure (defined as clusterName) is initialized with NULL. This structure holds the cluster name, system name, and version. The vem_uname () method is passed the communication handle and, if successful, returns a valid vem_name_t structure ; otherwise the method returns NULL.

Line 23: the cluster info is printed out to the screen.

Lines 26-43: define the client info structure. Use vem_locate() to get all registered clients. Since NULL is provided as the client name, all registered clients will be located and the method returns the number of registered clients. Note that Platform EGO is equipped with a number of default clients (services) such as the Service Controller, so as a minimum, the info relevant to these clients is printed out and the associated memory is released.

1	 int 
2	 sample7() 
3	 {
4	   vem_openreq_t orequest;
5	   vem_handle_t *vhandle = NULL;
6	   orequest.file = "ego.conf";
7	   orequest.flags=0;
8	   vhandle = vem_open(&orequest);
9	   if (vhandle == NULL) {
10	   	 // error opening
11	    	 fprintf(stderr, "Error opening cluster: %s\n",  vem_strerror(vemerrno));
12	    	 return -1;
13	   }
14	   vem_name_t *clusterName = NULL;
15	   clusterName = vem_uname(vhandle);
16	   if (clusterName == NULL) {
17	   	 // error connecting
18	    	 fprintf(stderr, "Error connecting to cluster: %s\n", 
19	  vem_strerror(vemerrno));
20	    	 return -2;
21	   }
22	   
23	   fprintf(stdout, " Connected... %s %s %4.2f\n", clusterName->clustername,
24	  clusterName->sysname, clusterName->version);
25	 
26	   vem_clientinfo_t *clients;
27	   int  rc = vem_locate(vhandle, NULL, &clients); 
28	   if (rc >=0) {
29	     if (rc == 0) {
30	    	   printf("No registered clients exist\n");
31	     } else {
32	   	   int i=0;
33	   	   for (i=0; i<rc; i++) {
34	    	     printf("%s %s %s\n", clients[i].name, clients[i].description,
35	    	     clients[i].location);
36	   	   }
37	   	   // free
38	   	   vem_clear_clientinfo(clients);  	   
39	     }
40	   } else {
41	   	 // error connecting
42	    	 fprintf(stderr, "Error geting clients: %s\n",  vem_strerror(vemerrno));
43	   }

Lines 44-46: authenticate the user to Platform EGO.

Lines 48-52: define and initialize a structure for callback methods. These callback methods are invoked by Platform EGO when resources are added or reclaimed, or when a change occurs to host status or a container. When Platform EGO wants to communicate about these events, it invokes these methods thereby calling back to the client.

Lines 54-63: define the vem_allocation_info_reply_t and vem_container_info_reply_t structures. If a client gets disconnected and then re-registers, its existing allocations and containers are returned to these structures. If the client had never registered before, the structures would be empty. Define and initialize a structure (rreq) that holds client info for registration purposes. (This includes assigning the client callback structure (cbf) to the callback member of the rreq structure.) Register with Platform EGO via the open connection using vem_register().

44	 if (login(vhandle, username, password)<0) {
45	    	 fprintf(stderr, "Error logon: %s\n",  vem_strerror(vemerrno));
46	    	 goto leave;
47	   }
48	 vem_clientcallback_t cbf;
49	   cbf.addResource = addResourceCB;
50	   cbf.reclaimForce = reclaimForceCB;
51	   cbf.containerStateChg = containerStateChgCB;
52	   cbf.hostStateChange = hostStateChangeCB;
53	   
54	   vem_allocation_info_reply_t aireply;
55	   vem_container_info_reply_t  cireply;
56	   vem_registerreq_t rreq;
57	 rreq.name = "sample7_client";
58	   rreq.description = "Sample7 Client";
59	   rreq.flags = VEM_REGISTER_TTL;
60	   rreq.ttl = 3;
61	   rreq.cb = &cbf;
62	   
63	   rc = vem_register(vhandle, &rreq, &aireply, &cireply);
64	   if (rc < 0) {
65	     fprintf(stderr, "Error registering: %s\n",  vem_strerror(vemerrno));  	 
66	    	 goto leave;
67	   }

Lines 68-71: print out information related to the allocation requests and containers. Once the info is printed out, the memory for the allocations is freed.

Lines 73-84: the vem_gethostgroupinfo() method collects the information for the requested hostgroup. In this case, the requested hostgroup in the input argument is set to NULL, which means that information about all hostgroups is requested. If the method call is successful, hostgroup information is printed out to the screen.

68	 print_vem_allocation_info_reply(&aireply);
69	   print_vem_container_info_reply(&cireply);
70	   // freeup any previous allocations
71	   release_vem_allocation(vhandle, &aireply); 
72	   
73	   vem_hostgroupreq_t hgroupreq;
74	   hgroupreq.grouplist = NULL;
75	   vem_hostgroup_t *hgroup;
76	 
77	   rc = vem_gethostgroupinfo(vhandle, &hgroupreq, &hgroup); 
78	   if (rc < 0) {
79	     fprintf(stderr, "Error getting hostgroup: %s\n", 
80	  vem_strerror(vemerrno));  	 
81	   } else {
82	   	 printf("%s %s %d %d\n", hgroup->groupName, hgroup->members, hgroup->free,
83	  hgroup->allocated);
84	   }

Lines 85-88: define and initialize the service_stateP and client_stateP structures. These structures contain the respective mutex objects and condition variables.

Lines 90-91: get the service definition and store it in the service state structure. Refer to Step 3: Create the service definition.

Lines 93-97: create and run the service_thread. The service thread is responsible for defining and creating the Platform EGO service.

85	 service_stateP = calloc(1, sizeof(service_state_t));
86	   initialize_service(service_stateP);
87	   client_state_t *client_stateP = calloc(1, sizeof(client_state_t));
88	   initialize_client(client_stateP);
89	 
90	   char *xml = get_service_def_xml();
91	   service_stateP->xml = xml;
92	 
93	   pthread_t service_thread;
94	   if (pthread_create(&service_thread, NULL, service_thread_fn,
95	  service_stateP)) {
96	     perror("Error creating service thread: ");  	 	 	 	 	 	 	 	 	 
97	   }

Lines 98-113: use vem_locate() to get all registered clients. Since NULL is provided as the client name, all registered clients will be located and the method returns the number of registered clients. If successful, print out the client info (name, description, and location) and free the associated memory.

Lines 120-123 lock the service_mutex object and wait until the service is running. Set the service condition variable to unblock the service thread, which causes the service thread to query the service.

Lines 129-132: lock the client_mutex object and block the main thread with the client condition variable. When the service info is available, the service thread signals the main thread to resume execution using the client condition variable. The main thread prints out the service info reply.

98	  rc = vem_locate(vhandle, NULL, &clients); 
99	   if (rc >=0) {
100	     if (rc == 0) {
101	    	   printf("No registered clients exist\n");
102	     } else {
103	   	   int i=0;
104	   	   for (i=0; i<rc; i++) {
105	    	     printf("%s %s %s\n", clients[i].name, clients[i].description,
106	    	     clients[i].location);
107	   	   }
108	       vem_clear_clientinfo(clients);  
109	     }
110	   } else {
111	   	 // error connecting
112	    	 fprintf(stderr, "Error geting clients: %s\n",  vem_strerror(vemerrno));
113	   }
114	   sleep(60);
115	   
116	   // get service info
117	   service_info_reply = calloc(1, sizeof(esc_service_info_reply_t));
118	   int succeed=0;
119	   while(!succeed) {
120	     pthread_mutex_lock(&service_stateP->service_mutex);
121	     if (service_stateP->ready && service_stateP->client == NULL) {
122	       service_stateP->client = client_stateP;
123	       pthread_cond_signal(&service_stateP->service_cond);
124	       succeed = 1;
125	     }
126	     pthread_mutex_unlock(&service_stateP->service_mutex);
127	   }
128	   fprintf(stderr, "Sent Request to Service:\n");
129	   pthread_mutex_lock(&client_stateP->client_mutex);
130	   pthread_cond_wait(&client_stateP->client_cond,
131	  &client_stateP->client_mutex);
132	   print_service_info_reply(service_info_reply);
133	   pthread_mutex_unlock(&client_stateP->client_mutex);

Lines 134-140: store the service info in the esc_service_info_t structure and retrieve the number of service instances. Assign the service instance info to the instance_info structure. Allocate and initialize an array in memory to hold the IP address associated with each service instance.

Lines 144- 149: for each service instance, retrieve the host name where the service instance is running and pass it to the gethostbyname() method. The method returns a pointer to a hostent structure (locationService), the members of which contain the fields of an entry in the network host database. The hostent structure is defined in the <netdb.h> header. Extract the host's IP address from the locationService structure and assign it to the ipAddresses array.

Lines 149-161: pass "mysql", an alias for "Sample7_mysql_service", to the printSDEntry() method. The method will query the service and print the IP address of the host that the service is running on. However, since there is currently no entry in the Service Director records for a service called "mysql", nothing is printed. (The printSDEntry() method is used again later in the sample to print out the IP address after the Service Director records have been properly updated.)

134	  esc_service_info_t *service_info = &service_info_reply->serviceV[0];
135	   if (service_info == NULL) {
136	      fprintf (stderr, "Could not get Service Info\n");
137	      goto done;
138	   }
139	   int i, num_instances;
140	   num_instances = service_info->instC;
141	   int ipaddrC = num_instances;
142	   char **ipAddresses = calloc(ipaddrC, sizeof(char*));
143	   
144	   for(i=0; i < num_instances; i++) {
145	   	 esc_service_instance_info_t *instance_info = &service_info->instV[i];
146	     char *location = instance_info->host;    
147	     struct hostent * locationService = gethostbyname(location);
148	     ipAddresses[i] = locationService->h_addr_list[0];
149	   }	  char *sname = "mysql";
150	   printSDEntry(sname);

151	 void
152	 printSDEntry(char *sname)
153	 {
154	   int i=0;
155	   struct hostent * dnsService = gethostbyname(sname);
156	   fprintf(stderr, "Existing Entry for %s\n", sname);
157	   if (dnsService != NULL) {
158	   	 for (i=0; i<dnsService->h_length; i++) {
159	       fprintf(stderr, "%s\n", dnsService->h_addr_list[i]);
160	   	 } 
161	   }

The following paragraphs demonstrate how to manually add a new entry in the Service Director that provides service instance location information.

Lines 162-167: define and initialize a structure (update) to hold service instance info that will be used to add a new record in the Service Director.

Lines 169-175: use dlopen() to open a shared library (esd_ego_default.so); this is the default Service Director plug-in. The dlopen() method returns a handle to the shared library. Define a structure to initialize all the plug-in variables. Pass the shared library handle and a function name (esd_plugin_initialize) to dlsym(). The method looks up the address of the function in the shared library handle obtained from dlopen.

Lines 179-189: set up the necessary plug-in variables and invoke the initialization method from the shared library.

Lines 190-191: pass the update structure to the update_service() method. The method adds a service location entry in the Service Directorconsisting of an alias (mysql) with the same IP address that was printed by the previous service query in the service thread; see Tutorial 6: Step 4: Query the service. The printSDEntry() method prints out the new service location entry.

162	  si_updrec_t update;
163	   sd_update_oper_t op = op_add;
164	 update.name = sname;
165	   update.ipaddrC = ipaddrC;
166	   update.ipaddrV = ipAddresses;
167	   update.oper = op;
168	 
169	   void * esd_plugin = dlopen("esd_ego_default.so",  RTLD_LAZY);
170	   if (esd_plugin == NULL) {
171	      fprintf (stderr, "%s\n", dlerror());
172	   }
173	 esd_plugin_initialize_t esd_plugin_init;
174	   
175	   void *fptr = dlsym(esd_plugin, "esd_plugin_initialize");
176	   if (fptr == NULL) {
177	      fprintf (stderr, "%s\n", dlerror());
178	   }
179	   * (void **) &esd_plugin_init =  fptr;
180	   
181	   esd_utility_funcs_t utility_functions;
182	   utility_functions.calloc = calloc;
183	   utility_functions.malloc = malloc;
184	   utility_functions.free = free;
185	   utility_functions.realloc = realloc;
186	   utility_functions.strdup =  strdup;
187	 const char *param = "";
188	   esd_plugin_t sd_plugin;
189	 (*esd_plugin_init)(&utility_functions, param, &sd_plugin);
190	  sd_plugin.update_service(&update);
191	 printSDEntry(sname);


Step 3: Create the service definition

Create a service definition in XML format. Each service is described by an XML file that contains information about the service such as the type of resources required to run the service instances and how to start and monitor them.

char * get_service_def_xml()
{
	 char *xml = 
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\
<sc:ServiceDefinition xmlns:sc=
\"http://www.platform.com/ego/2005/05/schema/sc\" xmlns:ego=
\"http://www.platform.com/ego/2005/05/schema\" xmlns:xsi=
\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=
\"http://www.w3.org/2001/XMLSchema\" xsi:schemaLocation=
\"http://www.platform.com/ego/2005/05/schema/sc ../sc.xsd
 http://www.platform.com/ego/2005/05/schema ../ego.xsd\" ServiceName=
\"Sample7_mysql_service\">\
  <sc:Description>\"Sample 7 MySQL Service\"</sc:Description>\
  <sc:MinInstances>1</sc:MinInstances>\
  <sc:MaxInstances>1</sc:MaxInstances>\
  <sc:Priority>10</sc:Priority>\
  <sc:MaxInstancesPerSlot>1</sc:MaxInstancesPerSlot>\
  <sc:ControlPolicy>\
    <sc:StartType>MANUAL</sc:StartType>\
    <sc:MaxRestarts>10</sc:MaxRestarts>\
    <sc:HostFailoverInterval>60s</sc:HostFailoverInterval>\
  </sc:ControlPolicy>\
  <ego:AllocationSpecification>\
    <ego:ConsumerID>/SampleApplications/EclipseSamples</ego:ConsumerID>\
    <!-- The ResourceType specifies a ""compute element"" identified by the
 URI used below -->\
    <ego:ResourceSpecification ResourceType=
\"http://www.platform.com/ego/2005/05/schema/ce\">\
      <ego:ResourceGroupName>ComputeHosts</ego:ResourceGroupName>\
      <ego:ResourceRequirement>LINUX86</ego:ResourceRequirement>\
    </ego:ResourceSpecification>\
  </ego:AllocationSpecification>\
  <sc:ContainerDescription>\
    <ego:Attribute name=\"hostType\" type=\"xsd:string
\">LINUX86</ego:Attribute>\
    <ego:ContainerSpecification>\
      <ego:Command>bin/mysql --user mysql</ego:Command>\
      <ego:RunAsOSUser>root</ego:RunAsOSUser>\
      <ego:WorkingDirectory>/usr/local/mysql</ego:WorkingDirectory>\
      <ego:Umask>0777</ego:Umask>\
    </ego:ContainerSpecification>\
  </sc:ContainerDescription>\
</sc:ServiceDefinition>	 	 	 	 \";
return xml;
}


Step 4: Create the service

In this sample, service creation is handled by the service thread. The service simply starts the mySQL program.

Lines 194-204: set the EGO_CONFDIR environment variable to the current working directory so that the Service Controller, which is just another Platform EGO client, can find the Platform EGO configuration. The configuration is stored in the ego.conf file.

Lines 205-208: define and initialize a security structure. The username and password variables have been initialized with "egoadmin" from sample # 2.

Line 210: create a service definition in XML format. Each service is described by an XML file that contains information about the service such as the type of resources required to run the service instances and how to start and monitor them.

Lines 212-219: pass the security structure and the service definition to the esc_createservice() method. This API call creates a new service object. If startType is automatic in the service definition, the service is enabled automatically. In this case, the startType is set to manual so the service must be enabled by the esc_enableservice() method. This API call starts the service. Once the service is started, the Service Controller allocates resources and starts service instances. A service can be started by esc_enableservice() or by starting a service that depends on it.

192	 void *service_thread_fn(service_state_t *service_stateP)
193	 {
194	 int size=4096;
195	   char *buf = calloc(size, sizeof(char));
196	   sprintf(buf, "EGO_CONFDIR=");
197	   char *cwd = getcwd((buf+12), size-12);
198	   int errn =  errno;
199	   if(cwd != NULL) {
200	     putenv(buf);
201	   } else {
202	   	 fprintf(stderr, "Error getting CWD: %s\n",
203	  strerror(errn));
204	   }
205	   esc_security_def_t security;
206	   security.username = username;
207	   security.password = password;
208	   security.credential = NULL;
209	   char *sname = "sample6_service";
210	   char *xml = get_service_def_xml();
211	 
212	   if(esc_createservice(xml, &security)) {
213	     fprintf(stderr, "Error creating service: %s\n", 
214	  esc_strerror(escerrno));  	 
215	     //goto bailout;
216	   }
217	 if(esc_enableservice(sname, &security)) {
218	     fprintf(stderr, "Error enabling service: %s\n", 
219	  esc_strerror(escerrno));  	 
220	   }


Step 5: Client callback methods

These callback methods are invoked by Platform EGO when resources are added or reclaimed, or when a change occurs to host status or a container. When Platform EGO wants to communicate about these events, it invokes these methods thereby calling back to the client.

Lines 221-227: this method is called by Platform EGO when resources have been added to an allocation in order to tell the client which resources have been provided for its use. This method prints out the allocation and consumer IDs, the number of hosts allocated, host names and number of slots, and host attributes.

Lines 229-235: this method is called by Platform EGO when resources need to be reclaimed. Resources may be reclaimed either for policy reasons, or because a resource has been found to be down or unavailable. The method prints out the host info including host name and slots for each host being reclaimed.

Lines 237-243: this method is called by Platform EGO in order to communicate status changes in containers to the clients that started them. The method prints out the container ID and its associated state.

Lines 245-251: this method is called by Platform EGO when a host changes state. The method prints out the host name and its new host state.

221	 int 
222	 addResourceCB(vem_allocreply_t *areply)
223	 {
224	 	 printf("addResource Call Back\n");
225	     print_vem_allocreply(areply);
226	     return 0;
227	 }
228	 
229	 int 
230	 reclaimForceCB(vem_allocreclaim_t *areclaim)
231	 {
232	 	 printf("reclaimForce Call Back\n");
233	 	 print_vem_allocreclaim(areclaim);
234	 	 return 0;
235	 }
236	 
237	 int 
238	 containerStateChgCB(vem_containerstatechg_t *cschange)
239	 {
240	 	 printf("containerStateChg Call Back\n");
241	 	 printf("%s %d\n", cschange->containerId, cschange->newState);
242	     return 0;
243	 }
244	 
245	 int 
246	 hostStateChangeCB(vem_hoststatechange_t *hschange)
247	 {
248	 	 printf("hostStateChange Call Back\n");
249	 	 printf("%s %d\n", hschange->name, hschange->newState);
250	     return 0;
251	 }


Run the client application

  1. Select Run > Run.

    The Run dialog appears.

  2. In the Configurations list, either select an EGO C Client Application or click New for a new configuration.

    For a new configuration, enter the configuration name.

  3. Enter the project name and C/C++ Application name.
  4. Click Apply and then Run.

[ Top ]


[ Platform Documentation ]


      Date Modified: July 12, 2006
Platform Computing: www.platform.com

Platform Support: support@platform.com
Platform Information Development: doc@platform.com

Copyright © 1994-2006 Platform Computing Corporation. All rights reserved.