Why Enterprise Services / Part 3: Expensive resources

Why you want to use Enterprise Services for your .NET application
Part 3: Managing limited and expensive resources

A Scenario from Hell (and other places reigned by evil ;) - Mr. Miller and the Data-Center

(Parental Advisory: May contain ironic sequences, dramatizations, mild simplifications for educational purposes).

Meet Mr. Miller. Mr. Miller's job is to run a "pilot" .NET web-portal project that lets insurance clients and agents submit claims over the web, buy products and do some limited self-management and analysis of their current contracts portfolio. As with many major insurers, most of the actual data lives in an mainframe system and pretty much all business logic is implemented there, because ... it is that way (for many good current and historical reasons). So once he's getting to the mainframe integration part of his planning efforts, he picks up the phone and calls Mr. Petterson, who's managing the data center:

"Good morning, Mr. Petterson"
        "Good morning, Mr. Miller, how are you?"
"Quite well, thank you. As we've discussed via email, I am calling to discuss some options for integrating our new customer-self-service web-application with your mainframe."
        "That's this Internet solution you are building on this Microsoft stuff?"
"Yes, Sir."
        "I am curious whether that's going to be any better than what those other folks did with that Java project before they gave up a year ago. Anyways, so what's your plan there? What's your idea of getting the data to us and back from us?"
"Well, we were actually thinking to use Microsoft's Host Integration Server for that...."
        "Won't happen..."
"... but ..."
        "Look, the mainframe is the backbone of everything this company does. Do you really think, we'll let you attach some random software to that?"
"... but if we use COMTI, we'd just be like any other CICS client?! How about talking to you via MQSeries?"
        "Software that talks to our CICS transactions is written here and nowhere else. That means the even if we were using MQSeries here, we'd have to provide you with the client portion."
"So if we'd want to talk to your system, you'd provide us with some sort of client communication DLL?"
        "Theoretically, yes... we've done that before."
"Theoretically... ?"
        "We have no time for that now. We've got priority projects here."
"But we're supposed to get this going in 4 months? How are we going to get to the data?"
        "You can grab the data off a terminal data stream through one of our applications. My understanding is that some of the data that you are looking for is only available through interactive applications, anyways, so that transaction access wouldn't take you anywhere, no matter what."
"Terminal? We're expecting a couple hundred concurrent web-users here... how many concurrent terminal sessions are we talking about?"
        "Maybe five, maximum ten... hey, we really can't give you that much. We've got internal users to serve, too."
"Five terminal sessions?"
        "I guess you heard me. Any more questions?"

Unrealistic? Well, if this sounds completely out of this world for you, be glad. In a world where we're talking about XML anywhere, seamless integration of systems and all that, discussions like that take place and if you happen to be on Mr. Miller's end of this call, your options really only are to look for another job somewhere else or to deal with what you are getting. Needless to say: It isn't really much what Mr. Miller got here. It should be noted that Mr. Miller wouldn't have gotten anything more if his project wasn't a Microsoft .NET project but an IBM WebSphere project. Where he works, there's the mainframe and then there's just some other "decentralized IT" that doesn't matter all that much, at least when seen from the data-center view point.

So, there he is. Stuck with a web portal project that shall scale and whose backend logic is accessed through a terminal screen-scraper.

"Mr. Petterson?"
        "Is that you again, Miller? Any more questions?"
"I was thinking that there may be a better way to do this..."
        "I am listening."
"Well, isn't there a text-file import interface that you have there at least for submitting the insurance claims?"
        "Yes, we have that. We have a PC application -- I believe that runs under PC-DOS -- which will take a EBCDIC encoded text file and transfer it into the host-system. If you can write such an export file, you wouldn't have to go through the terminal. Would that help? "
"I guess so. Can you get me the specifications for the file format?"
        "Absolutely. I will have our assistant here copy the three binders right away. Tell me when you are done and we can arrange for testing."

So, at least for some part of the application, there is hope to get around the terminal screen-scraper, but essentially there are two "low tech" choices to deal with. A scriptable 3270 screen scraper ActiveX control (which is much better than nothing) to get data off and into a maximum of five concurrent terminal sessions and a PC-DOS app that will take one text-file at a time to export data to the mainframe. The good news is, though, that the latter allows batching of multiple jobs. On the "other end" of the application, he has an ASP.NET web-application, which serves a few hundred, virtually concurrent users. In fact, the number of really concurrent requests may max out at a few dozen, but it's still more than the five terminal sessions he's got.

So, the question is: How?

The terminal dilemma

There are quite a few problems to tackle regarding the terminal access. First, the terminal application that Mr. Miller's app needs to use to retrieve and submit data was built for interactive use. Therefore, the terminal session needs to be established, a user needs to be logged in and scripts need to be executed to navigate through the menu system to get to the right screens. There are 3 different "use-cases" (named A-C) that need to be handled through the terminal and therefore Mr. Miller decides to use 2 of his 5 allocated terminal logins to service use-case A and B respectively, and the 3 remaining logins are used to service use-case C, because the planning assumes that this use-case will be getting most traffic. The second problem is the simple fact that there are indeed many requesting threads competing for the very same set of terminal connections.

Here's Mr. Miller's plan for the terminal portion:

1. Write a .NET serviced component for each "use-case" A, B, and C. Each implementation will drive the screen scraper component to a well-known portion of the terminal application. Each "use-case" is indeed split up into a few operations, which are all implemented on the component and of which each will run a short scripting sequence to get data off the terminal screen buffer or into the terminal screen buffer to implement the functionality.
2. Enable object pooling for the components, with the minimum and maximum boundary for the number of pooled objects set to "1" for A and B and with a minimum of 1 and a maximum of 3 for the component wrapping the use-case C functionality. Make sure the components can reset themselves to a defined initial state and return true from CanBePooled().
3. Enable just-in-time activation and mark the serviced component methods with the AutoCompleteAttribute.
4. Set the service component's application to be activated as server and set the impersonation-level to "Identify", so that the caller's identity can be obtained but the caller isn't impersonated.
5. For Version 2, consider using the compensating resource manager to be able to participate in 2-phase commit transactions on the .NET side and do compensating rollback of operations though the terminal. (I'll get back to this in a later installment)

It's good that Mr. Miller knows his stuff. Wrapping each use-case with a dedicated component (1) is a good way of isolating the functionality to begin with, because there may be hope that in the future a more direct integration with the backend may be possible and therefore the web-application won't have to be changed once that happens. The reason why he's choosing a serviced component here is that this solves all of his other problems in this scenario as well.

He chooses to enable "object pooling" for all three components. The scenario here is a picture-perfect use-case for using object-pooling, indeed. Once the server application hosting the component spins up, one instance of each component is being instantiated (a minimum pool size of at least 1 causes that to happen) and therefore the component can initialize the terminal screen scraper, log in and navigate the menu-structure of the terminal application to get to the desired "application area". All of that can happen in the component's constructor. If it doesn't work, the component cannot be correctly initialized and therefore activation must fail.

The maximum pool size limits the number of objects that can be activated concurrently. For use-cases A and B, Enterprise Services essentially limits the components to be singletons because there can only be at most one instance due to the maximum pool size being set to 1. For use-case C, there can be at most 3 instances at a time and therefore object pooling automatically does the necessary throttling to deal with the 5 councurrent terminal sessions available. If more than one request is directed at A/B or more than 3 requests are directed at C, the excess requests are being queued up and will be served once the component is available again. Requests which haven't been served, yet, will time out after a timeout period which is configurable at the component level.

We speak of "requests" here, because just-in-time activation being enabled (3) and auto-completion is enabled. This means that any of the active instances will be "warned" of a connection by a client and imminent calls through a call to ServicedComponent.Activate() (in which case it may navigate the terminal to a defined initial position) and that the instance will be automatically disconnected and made available to the next client once the call has been served. Again, the component is being notified of the disconnect through a call to ServicedComponent.Deactivate() and can choose whether it wants to reset itself or wants to be destroyed and replaced by a fresh component by answering true or false to a call to ServicedComponent.CanBePooled() accordingly.

The advantage of just-in-time activation on the client-side is that the ASP.NET application can keep a pool of pre-initialized proxy objects around and not worry about the activation/deactivation/pooling happening on the server-side.

The whole concept of channeling all requests into 5 terminal logins this way works only well, because (4) the objects are activated in single, centralized server application. (If that sounds like a single-point-of-failure problem, there's COM+ 1.5 application recycling and MSCS hot-cold clustering safety-nets who'll take care of most of that risk). Using "Identify" as the impersonation option allows him to pick up the caller's identity while running the screen scraper with elevated privileges.

"So, in the end, it isn't all that bad. There would be better ways to do this", thinks Mr. Miller, "but there's always Version 2".

Of course, object pooling is not only good for this scenario. I am sure you'll find your own.

The export file

Access to the export file writer could be managed in the same way. Mr. Miller could implement the export component as a serviced component and enable object pooling to restrict the number of writes to the pre-defined export file and invocations to the PC-DOS helper application, but he chooses to do it differently: Using Queued Components.

Next installment: Queued Components.

Last update: 10.12.2002; 11:20:19.