Web caching in the Cloud

This is the buzz word now in enterprise if you are trying to implement private cloud. I think this can be confusing to an extend, is it application caching side we are talking about or are we talking about caching in cloud?

For Applications there are solution like Eh-cahce or open terracotta or Extreme scale from IBM are some good solution.

Web application performance can be improved by caching frequently‐accessed data in high‐speed memory, instead of in databases, they can dramatically reduce response times, economically scale their sites, and substantially reduce the load placed on their databases, file servers, and other information sources.

Now lets talk about web caching in the cloud and some player in this space. Memchached and Gear6 (which provides enhanced solution)

The open‐source distributed cache system Memcached has quickly become the standard for web caching, particularly for operators of dynamic, high‐growth web sites. Memcached stores frequently‐used data in DRAM instead of in databases, providing response time improvements of 100x or more.

It is not surprising that many large organizations are interested in both cloud computing and Memcached web caching, since they offer complementary advantages in cost savings, flexibility and ease of management. But they too have evolved in different communities, largely in isolation from one another, and they do not operate together as seamlessly as might be desired.

As an example, the ability to rapidly scale applications up or down in size is a key advantage of cloud computing, but standard Memcached does not accommodate it very well. Changing the size of a Memcached cache tier causes the cache data to be flushed and then rebuilt, this typically causes degradation in site performance, and causes spikes in the demands on source databases and file servers. Rebuilding the contents of a large cache tier can take hours, and site performance may not be restored to normal until the end of the rebuilding process.

You can read more about Memcahed and Gear6 and MemCahed can be dowloaded from

Download MemChached

MapReducer

Almost a year back I started looking in to multiple options as how to provide better search in quick time against large sets of data in enterprise. for example you might want to search some keyword in exceptions, it could be like sysOut or error logs for enterprise applications.

I think MapReducer is an answer to these problem, here's some info about MapReducer

MapReduce is an parallel and distributed solution approach developed by Google for processing large datasets. MapReduce is utilized by Google and Yahoo to power their websearch. MapReduce was first describes in a research paper from Google .

MapReduce has two key components. Map and Reduce. A map is a function which is used on a set of input values and calculates a set of key/value pairs. Reduce is a function which takes these results and applies another function to the result of the map function. Or with other words: Map transforms a set of data into key value pairs and Reduce aggregates this data into a scalar. A reducer receives all the data for a individual "key" from all the mappers.

The approach assumes that their are no dependencies between the input data. This make it easy to parallelize the problem. The number of parallel reduce task is limited by the number of distinct "key" values which are emitted by the map function.

MapReduce incorporates usually also a framework. A master controls the whole MapReduce process. The MapReduce framework is responsible for load balancing, re-issuing task if a worker as failed or is to slow, etc. The master divides the input data into separate units, send individual chunks of data to the mapper machines and collects the information once a mapper is finished. If the mapper are finished then the reducer machines will be assigned work. All key/value pairs with the same key will be send to the same reducer.

Here's nice video about MapReducer :

http://code.google.com/edu/submissions/mapreduce-minilecture/listing.htm

Apache Hadoop which start with one large open source project is getting very popular around this space.

Here's link if you want to know more about Hadoop:

Apache Hadoop



If you want paid solution and dont have time to develop, take a look at SPLUNK, its nice and easy and very powerful tool, you can feed any data against this. I have played with splunk and I believe Splunk usages Hadoop APIs.

Enterprise Architect vs Solution Architect

I find this subject very interesting and at least i didn't find any concrete definitions around these two different roles in IT, what are the main differences (in term of responsibility) between two practices/roles. I think TOGAF and Zackman architectural framework provide in detailed explanation about roles and responsibility for each role in and enterprise. I would like to share what i have got so far....

So what exactly is an Enterprise Architect vs a Solution Architect?

A Solution Architect may have a number of different types of architects working for him/her to help accomplish their task for delivering a high quality solution. For example, s/he might have an Infrastructure Architect to manage the architecture of the solution's hardware configuration so that the solution meets the Quality of Service business and IT requirements while at the same time represents the optimum way to deploy the solution into the target production environments.

There also might be a need for 'specialist' architects depending on the complexities of the business requirements or the target production environment. Such 'specialist' architects usually are called Domain Architects and examples include; Security Architect, Technology Architect (ie architects that specialize in a specific product or technology), Vertical Architects (ie architects that specialize in systems that are specialized in particular industry verticals such as Financial Services Industry, Telecommunicaitons, Public Sector, etc).

An Enterprise Architect may have also have a number of different types of architects in order to piece together a coherent, actionable future state architecture which can easily map to business strategy, be consumed by project teams and be a major contribution in governance activities.

For example, an Enterprise Architect may have Business Architects which document Business Strategies, Business Capabilities, Business Processes and Roles as well as a number of other artifacts.

Lastly, an Enterprise Architect may also have Solution Architects (aka Enterprise Solution Architects and Enterprise Application Architects) who focus on pulling all of the other Enterprise Architecture information together to shape the future state application system architecture.

Here is one place where I think there is confusion by many. You see, I described a Solution Architect at the project-level and an Enterprise Solution Architect across the enterprise and both share the words 'Solution Architect' in their role. They have VERY different purposes but are very complimentary. Could it be that simple of a reason why confustion exists? It just might be.

An Enterprise Solution Architect and project Solution Architect roles are very complimentary and because this relationship is of particular interest to me I'd like to take a moment and propose how they are to work together.

I think that the Enterprise Solution Architect be involved at the earliest point a new business initiative is created and do very high-level 'solutioning' via whiteboard and reference to future state architecture elements. Then, when the business initiative gains momentum is backed by Enterprise Architecture future state analysis the core project team is formed and the project Solution Architect takes over to be responsible for the solution. The Enterprise Solution Architect becomes a member of the project Solution Architects team and is responsible for the system integration architecture where the solution requires integration between enterprise systems as well as be responsible to provide future state system architecture guidance to help the project Solution Architect make decisions on which systems to commit to using in the solution. The Enterprise Solution Architect then is involved in governance boards to inform decision-makers with future-state architecture artifacts to help justify major technology, system, or business value decisions. Thats it.

As a quick summary, project Solution Architects and Enterprise Architects are different in that they have very different purposes. They are highly complimentary in that Solution Architects focus on delivery of solutions and Enterprise Architects focus on supporting them by documenting future state, participating on their teams and being involved in governance activities.