5 Ways to Scale Your Web Application

Writing web applications is a time consuming process, and when the application begins to get popular, it may not have been built to scale and handle volume. A great example of this is Twitter, a popular web 2.0 application that often has downtime. Here are some ways you can build your application with scaling in mind.

1. If you can, build your application with scaling in mind from the ground up. Write tighter code, test it against perceived loads, and continually try to lower the amount of time and resource usage required to run the application.
2. Build your application on cloud computing. Services like Amazon's Cloud computing service allow you to host your code, databases and files on their servers, and can scale instantly with your demand. It also saves you money in the long run, as you don't need to constantly plan to build new servers, database clusters and bandwidth.
3. If you're not using cloud computing, try using a grid hosting company. These are companies that can give you instant access to more resources as soon as the demand rolls in. The difference primarily lies in the pricing – cloud computing companies charge based on actual usage, while grid hosting companies will charge based on the amount of "slices" you need to meet your requirements over a month.
   
4. Optimize your databases and queries. If you're not a true expert in Databases, hire a freelancer to optimize your code. Correctly writing queries, caching queries that are run often, and correctly building the structure of your database can make or break a web application.
5. Use reusable, well documented code. Try not to reinvent the wheel –find classes and functions that have already been proven in the field, and use them for the heavy lifting part of your projects. Use frameworks and coding methods that have already been proven, and don't try building a heavy use platform on new or untested code.
   

This is a guest post authored by Maya Richard, who writes on the topic of high speed internet. You can email her at mayarichard@gmail.com with any feedback.

How to Secure Your Web Application

Disclaimer: This is a guest post written by Heather Johnson, who writes on the subject of dial up Internet.

Web applications will install firewall to restrict unauthorized access to ports 80 and 443. But these don’t exactly stop unwanted attacks that come over these ports. There is much more that a web professional should do in addition to installing firewall. We’ve come up with a list that should help you sew up any holes that may appear in your latest web application that you’re working on. Without further ado here’s our list that should help maintain the integrity of your newest program:

1. Limit access to your profile. One of the first things a hacker looks for is information regarding your web server. Be sure to take away the server header from the response of your web server. Go ahead and map out different paths to the file extensions of your dynamic pages. Take steps to ensure your machine isn’t named something that could identify its operating system. Take out your personal information from your WHOIS records that help someone in a social engineering attack.


2. Authenticate. Unfortunately you can’t always trust the users that visit your website. You need to have your users authenticate who they are and their intent for visiting your site. You want to try to separate legitimate traffic from traffic that can hurt you. We’re not saying that every user is evil, but you need to be aware that there are unsavory people out there that try to hurt people in your position.


3. Always be ready for the worst. If your server is compromised it’s important that you go offline and plug up any holes. You need to protect customer data such as credit card information and in order to do so you need to install security software that will protect this information.


4. Always be monitoring. Instead of waiting to take care of a problem after it occurs if you are constantly logging your website’s activity you can help avoid a problem instead of simply reacting to it. Make sure you pay careful attention to your server’s error log.


5. Don’t execute a poor request. There will be times when a user’s request is simply not good for you. Many attacks occur that attempt to modify the HTTP request that will end up create an averse effect. You should set up the positive model that will only allow what you want to allow; everything else should not be allowed to be processed on your site.
   

By-line:

This post was contributed by Heather Johnson, who writes on the subject of dial up Internet. She invites your feedback at heatherjohnson2323 at gmail dot com.

Flex: I'm lovin it so far!

After reading about the well praised Adobe Flex on several tech blogs, I gave it a shot last weekend. Its simplicity and ease of development took me by surprise. No prior experience with Flash is required to get started on Flex.

Adobe® Flex is a cross-platform development framework for creating rich Internet applications (RIAs). With flex, I can create powerful good looking webapps in less than an hour. I'm lovin it so far! Ajax is sooo old fashion. Quite a lot of web2.0 sites were developed in Flex. Check them out at Flex showcase.

The Flex SDK is free, but the FlexBuilder (an Eclipse based IDE for Flex) will cost us a bunch. I plan to get a lot done during the 60 day trial period of FlexBuilder. We can learn Flex in a couple of days using the Getting Started Guide from Adobe website. There is also a Video Tutorial on how to develop and debug a very basic application.

This weekend, I will probably get my hands wet in developing a flex version of my search mashup FundooWeb. I also found a great Javascript IDE called JSEclipse that is supported by Adobe, and is free for personal use. Great find Ah!

UPDATE (2/2/08)
Here is my first Flex application! It comes 2 sizes: Wide and Slim. Below is the wide one!

Feel free to grab this widget (Wide OR Slim), and embed it on your site.

Wana build a revolutionary web product?

This is a good start for developers who dream to revolutionize the web! However, to build a Next-Gen web application, it takes more than just brainstorming. Extend this list with your comments...

Mix n' Match
There are some killer combinations yet to be explored. For instance, how about "Blog + Social Network + Wiki". I'm sure several such combinations can be introduced by trying to mix 'n match the existing web.

Induce intelligence
Self learning & pattern-matching algorithms can be used to develop decision-capable applications. Such software can literally change the way we depend on computers for our daily lives. Imagine an application in which we can record our concrete experience & drop a line on what we learned each day. The application remembers the lessons learned and shall advice us in similar situations on demand.

Unify
Provide a solution to unify data & processes from several sources. Think of how MS Outlook provided a unified approach for communication at work. Get hold of a concept that is available in numerous forms... try to group them, aggregate them, and unify them! Recently, Plaxo aims to do just that with address book. What more? I can quickly think of Social networks which keeps growing in number and are hard to manage.

Collaborate
Extending what Web2.0 taught us, we should bring forward the concept of collective intelligence into several other online activities. Google has already shown us the way for collaboration in its Calendar, Spreadsheets & Docs applications. How about a collaboration to compose an email, or to make a decision. I remember reading about a Microsoft's research project that collects a set of location-based images taken by several individuals, and creates a wholesome image of that location. Great effort!

Connect
Our blogs live in their own worlds isolated from one another. How about a way to connect the web, and enhance the content we publish with additional links and pics. Imagine a web that relates our experience with other people's experience via semantic meta data.

Meta-Data will be the KEY for a Next-Gen Web company. Semantic-Web is the only technology contender so far to create a revolutionary web experience. If you are not impressed, you do the research on what you believe can enhance today's web. Of course, research is not everybody's cup of tea. But, its a sweet avenue to explore by PHD geeks and research addicts.

Ubiquitous Computing
Several electronics now include micro-chips and software to communicate with the web, create information-flow, and provide a utility value. PDAs, MP3 players, Cell-phones, GPS devices, and many more such products in the market provide computing outside the PC. Future software should cater to the needs of such products, communicate with Software-Agents, and instruct micro-robots.

How to quickly develop a Web application

Now, developing a web application has become a weekend activity for some... Gone were the days when you have to deal with complex code behind .NET or J2EE. So, what are the options to quickly build an industry standard webapp?

Use of Ready-made software

Easiest of all, you can use white-label frameworks to clone an existing webapp. Check out the chart published by Techcrunch on white-label social networks available. Below is a summary of clone projects mentioned by Read/WriteWeb.

Folkstr is a micro-blogging platform that mimics Twitter, but without the SMS features. It is a self-hosted and extensible social network developed in PHP and uses MySQL.

Pligg is a perfect replication of Digg, and adds some features such as tagging and the ability to automatically share links on other popular social news and bookmarking sites.

Scuttle is an open-source project that replicates the popular Delicious Bookmarking service of Yahoo. But, a new version hasn't been released in over a year.

Video Share Enterprise is a PHP/MySQL script that clones YouTube. Video Share duplicates most of YouTube's features pretty well, and powers a number of small-to-medium sized sites. Yet another You-Tube clone script is vShare.

One of the most popular and well-developed MySpace clone scripts out there is phpFoX. This is behind some fairly large niche MySpace clones, and has most of the features of MySpace.

AJAX DeskTop StartPage Enterprise is a PHP and MySQL based AJAX start page script that mimics the sites like Pageflakes ands Netvibes.

UPDATE: Check out more such frameworks listed by AjaxFlakes. There is a mention of NewsCloud, Dolphin, PHPizabi, Elgg, Mugshot, AroundMe, GetBoo, and more...


Using frameworks based on dynamic languages

Ruby on Rails (RoR)
The open-source RoR project innovated the coupling of the powerful Ruby language and the Rails framework. It aims to ease database-driven website development with a template pattern called 'Convention over Configuration'. But, many developers including me are not yet ready to get away from core Java. Luckily, Grails framework is now a good substitute to RoR.

Groovy & Grails
Groovy provides powerful features of a dynamic language, and proves to be a handy tool to Unit test Java applications. It offers a syntax similar to Java, supports Java libraries, and compiles to Java Byte Code. So, calling of methods between the two languages can be interoperable. Groovy can serve as a good alternative to Perl and Ruby.

Grails is a open source web application framework based on the Groovy. Similar to RoR, Grails also embraces convention over configuration. Additionally, Grails uses other proven tools to provide state-of-art technologies. In Grails, you will find ORM mapping provided by Hibernate, Dependency Injection and MVC architecture by Spring Framework, and layout management by SiteMesh.

PHP 2.0
We can create portals and simple web applications using PHP. Its the most preferred implementation language by all of Yahoo Portals.


Rich Internet Applications (RIA)

RIAs like JavaFX, Adobe AIR / Flex, and Microsoft Silverlight are slowly getting popular among developers. You may want to experiment with them.

UPDATE: The description of RoR & Grails framework is updated with original content.

New Trends in Programming

Here are a few technologies that are making a Buzz on the web, and every tech-blog is beginning to talk about them. These are just brief introductions to those non-Microsoft technologies that I'm excited about. Now, I am one in the crowd to experiment and tell the world.

GridGain provides computational grid platform for Java that is fun, simple and productive to use. As you would expect, it is open source and professionally supported. It's ideal for performing parallelizable tasks (i.e. Split the work, calculate, and aggregate the results). GridGain now supports integration with Jboss, Spring, AspectJ, Weblogic, Websphere, and many more... You can find good documentation and a quick demo at its homepage.

Erlang is a concurrent functional programming language designed at the Ericsson Computer Science Laboratory. Erlang processes are very lightweight (lighter than threads) and the system can support several thousands of processes.

Unlike most languages that use shared state concurrency, Erlang uses pure message passing concurrency. Thus, making things scalable and fault-tolerant is relatively easy. If this interests you, read the fine analysis of Erlang by Hendy Irawan. Here is an excerpt...

Erlang is built on the ideas of

• Share nothing : Process cannot share data in any way. Actually, this is not 100% true; there are some small exceptions.
• Pure message passing : Copy all data you need in the messages, no dangling pointers.
• Crash detection and recovery : Things will crash, so the best thing to do is let them crash and recover afterwards.

Google Gears is an open source browser extension that enables web applications to provide offline functionality. It provides Javascript hooks that will interface with a few locally running services such as Cache, Database, and asynchronous Thread pool. If you are a user of Google-Reader, you might have noticed the Offline support... This is made possible only with the help of Gears plugin on our browser.

This effort from Google is what lays the foundation, and gives fullness to the concept of Web O.S. As a developer, you’ll be able to make an application with the assurance that it will work offline and online across browsers. Check out the Google Gears API Developer's guide for tutorials. The version 0.2 release of Google Gears will support Cross Origin API, and will be a huge treat for Web application developers.

Java DB is Sun's supported distribution of the open source Apache Derby 100% Java technology database. Amongst the many changes in Java 6(u2), Sun introduces Java DB as a part of its SDK. This is great news for Java lovers who are looking for a light-weight database. I'm thrilled!

Interestingly, we can use it within browser, Web-2.0 applications for easy distribution, one-click install, secure local data storage, and data persistence if the Internet connection is lost or for use off-line. Further, it is easy to migrate an application using Java DB to other open standard databases.

Groovy provides powerful features of a dynamic language, and proves to be a handy tool to Unit test Java applications. It offers a syntax similar to Java, supports Java libraries, and compiles to Java Byte Code. So, calling of methods between the two languages can be interoperable. Groovy can serve as a good alternative to Perl and Ruby.

Scala is a general purpose programming language designed to express common programming patterns in a concise, elegant, and type-safe way. It smoothly integrates features of object-oriented and functional languages. It is also fully interoperable with Java. Check out the beginner's guide for getting started.

Success of lightweight JAVA

The Spring Framework and Hibernate revolutionized the concept of POJOs with the introduction of lightweight technology. Then came EJB 3.0 spec which was inspired from the Spring guys. Meanwhile, Hibernate won the hearts of millions of people for its excellent ORM support. Latest to the party is JBoss Seam 2.0.

Why?
The lightweight frameworks promote better and cleaner application architectures, and make it easier to reuse business components. The rise of lightweight technologies was largely due to developers' rebellion against the heavyweight of EJB 2.1 (and earlier).

The Central Theme?
The core principle shared by all lightweight enterprise Java frameworks is the use of plain old Java objects (POJOs) for the data access and business logic. There are no more infrastructure classes or interfaces to inherit or implement. You just create a POJO to model your data or to implement a business process using the data. Then the POJOs are "wired" together using metadata.

How it works?
A key technique in the wiring of POJOs is a design pattern called Dependency Injection (DI). DI uses the lightweight framework container (Eg. Spring container or an EJB 3.0 container) to inject services or other objects into a POJO. This way, all object instances are created and managed by the container. A POJO need not manage the life cycle of its service objects or to look up services.

The Distinguishing factor?
The major differences between lightweight frameworks are how they wire container services together and implement Dependency Injection.

Ref: Java Developer's Journal

Is MD5 the God of Hash?

It is 'Yes' according to Skrentablog. I found some interesting uses of MD5 in the article 'We Worship MD5, the GOD of HASH'. Here is a gist of it.

MD5 takes any length string of input bytes and outputs 128 bits. The bits are consistently random, based on the input string. But if you make even a tiny change to the input string, you'll get a completely different output hash.

MD5 tips & tricks

• Unique ID generation

  Say you want to create a set of fixed-sized IDs based on chunks of text -- urls, for example. MD5 (url) is 16 bytes, consistently, and you're unlikely to ever have a collision. So, it's safe to use the md5 as an ID for the URL.

• Checksums

  Don't trust your disk or your OS to properly detect errors for you. They CRC and protocol checksums they use are weak and bad data can get delivered.

  Instead, bring out an industrial strength checksum and protect your own data. MD5 your data before you stuff it onto the disk, check the MD5 when you read it.

      save_to_disk(data,md5(data))
      ...
      (data,md5) = read_from_disk()
      if (md5(data) != md5)
          read_error

  
  
• Password security
  

  You could store the password in your database, "in the clear". But this should be avoided. If your site is hacked, someone could get a giant list of usernames and passwords.
  

  So instead, store md5(password) in the database. When a user tries to login, take the password they entered, md5 it, and then check it against what is in the database. The process can then forget the cleartext password they entered.

  
  
• Hash table addressing
  

  MD5 isn't a weak hash function and you don't need to worry about that stuff. MD5 your key and have your table size be a power of 2. You will never have to worry about Hashtable bucket collisions and similar issues.

  
  
• Random number generation
  

  The typical library RNG available isn't generally very good. For the same reason that you want your hashes to be randomly distributed, you want your random numbers to actually be random, and not to have some underlying mathematical structure showing through.
  

  Having random numbers that can't be guessed or predicted can be surprisingly useful. MD5 based sequence numbers were a solution for the TCP sequence number guessing attacks.

SPADE for Developers

A "Single Page Application and Development Environment" (Spade) is a Spa application with an integrated development environment, all fitting onto a single web page. Popular among SPADE apps is Tiddlywiki, a wiki implemented in a single page. With Google Gears in place, a wide range of applications can sprout up with a little effort only.

Open Web Standards unify internet

Here is a note on some open standards that are intended to provide greater collaboration among websites, and create a unified Internet for the user. Instead of rephrasing the concepts, I'll borrow excerpts from Wikipedia to display clarity & correctness.

You may also checkout Videntity's social network that combines three recent web technologies: OpenID, XFN, and FOAF.

• OpenID allows people to login to a website using a global identity.
• XFN describes a standard set of relationships and a mechanism for embedding them into links in a web page.
• FOAF provides an RDF/XML format for downloading profile and relationship data.
  

OpenID is an open standard that lets you sign in to other sites on the Web using your openID account. This means less usernames and passwords to remember and less time spent signing up for new sites. How do I use my OpenID? Look for sites with sign in forms that look like this:



XHTML Friends Network (XFN) is an HTML microformat that provides a simple way to represent human relationships using links. It puts a human face on linking. XFN enables web authors to indicate relationships to the people in their blogrolls simply by adding one or more keywords as the 'rel' attribute to their links, as in this markup example:
<a href="http://jeff.example.org/" rel="friend met">...</a>

FOAF is a project under the umbrella of Semantic Web. The semantic web is an evolving extension of the internet in which web content can be expressed not only in natural language, but also in a format that can be read and used by software agents to find, share and integrate information more easily. A popular application of the semantic web is Friend of a Friend (or FoaF), which describes relationships among people and other agents in terms of RDF.

If you are interested in open standards, you may also want to check out Broadband Mechanics. The Broadband Mechanics’ strategy is to help change the world by providing the software and open standards necessary to inter-connect disparate social networks and blogging tools together.

Pick one: OpenID or Yahoo BBauth

OpenId is an impressive alternative for web-startups who really care about their users' convenience. For those who've never heard of OpenId, it is an open decentralized framework for user digital identity. You can create an OpenId once, and use the same credentials to log-in to multiple websites that support it.

Lately, I was trying to explore it for my upcoming web-application. After sufficient reading, I was able to appreciate its architecture and the API, and how simple it is for the user, as well as the developer. But before I dove in and got my hands dirty, I considered the HTTP authentication frameworks provided by Yahoo & Google.

Almost every web user has an account with Yahoo, and this means "effortless login" to my webapp. Yahoo BBAuth is indeed an excellent service but with one drawback. The drawback is that Yahoo does not reveal the UserID of the logged in user. Instead, it returns a Hash value that corresponds to the username. Hmm... But, I am not willing to use this hash value as the only key to associate the persisted user data and the users. Did you just ask me WHY? Well... becoz my webapp becomes unstable when Yahoo changes its hash algorithm later in course of time.

So, OpenId became my ultimate choice of implementation. There are plenty of OpenID Service providers available on the web. Check them out here... To make it easy for developers, there are several frameworks available for each implementation language. I was interested in Java libraries, and my options include IdPrism, NetMesh InfoGrid, OpenID4Java. Out of the lot, I found IdPrism to be very straightforward and easy to integrate with my application.

In my search for a good OpenId service provider, I came across an interesting approach taken by Simon Willison to integrate OpenId specs with Yahoo BBAuth API. You can check this out at IdProxy.net.

Phones serve as credit cards

Visa, the world's biggest credit card payment system, has partnered with Nokia handset makers to make mobile phones serve as credit cards. Here is an excerpt of the news...

Users can pay for groceries and other purchases by swiping a phone over a reader that electronically communicates with a microchip on the phone. Phone owners confirm the purchase with the push of a button and the deal is complete.

This opens the door for more security issues, stolen cell-phone cases, identity thefts, etc... This sure makes the cellphone a digital wallet, and hence more valuable than ever. I believe, the more valuable it gets, the less freedom I will have in my careless usage.

Ford reserves a place for Microsoft

Ford Motor Co. has teamed up with Microsoft to provide an excellent integration of in-car mobile and entertainment systems. The whole concept relies on Bluetooth or USB cable to serve as a communication medium. Here is an excerpt from ComputerWorld:

The Ford-exclusive technology -- based on Microsoft Auto software called Sync -- allows users to operate just about any mobile phone or digital media player using voice commands or the vehicle's steering wheel or radio controls, according to the companies.

I feel this more exciting than GM's onstar service. Of-course, they are completely different and each have their own purpose. But, this in-sync feature by Ford entices all its customers (youth, families, businessmen, techies, geeks, gals,...) in buying its automobiles. I wonder how much good could on-star service bring to GM. Its a smart investment by Ford.

Mac Pseudo-Transparent screens

Here is some interesting stuff... Check it out!


Click here for a video demonstrating the transparent screen in action...

Get to know how its done...

Yeah, it's a trick with setting your desktop to a picture of whatever is behind your screen. But it looks pretty cool, and apparently it really begins to feel to the user like your desktop is transparent and the stuff on your desktop is kind of floating in space.

This is an interesting pic that I found in the 'Transparent screens' photo-set. FYI, this pic is original (not manipulated). Any guess as how this shot was taken! Click here for the answer.

Mind Reader

Reading the mind is a central theme for many sci-fi fiction movies... Matrix and Minority-Report are some of them. Well... now, its not fiction anymore. Check out this article that has info on the 'Mental Typewriter', which is basically a mind reader. They help to carry out actions at the speed of thought..

Microsoft vs Rest of the world

We all know about the incompatible standards for the upcoming DVD generation (HD-DVD and Blu-ray). Microsoft takes the stand on HD-DVD, while many other companies take the opposite stand. Reasons are evident as Java Runtime Environment is used by the Blu-ray devices. If no single standard solution can be framed, it will then be interesting to let consumers decide which one to eliminate in near future.

Yet another battle of technologies... Its the Microsoft InfoCard versus Project Higgins that is lead by IBM and Novell. Project Higgins is an open-source effort managed by the eclipse foundation. It deals with the idea of managing personal identity of an internet user in a convenient way.

Interestingly, Windows servers for the first time topped Unix for the quarterly results. However, combining all flavors of Unix (Unix, Linux, Solaris) beats Windows without doubt. Check it out here. Moreover, the single-user nature of MS-DOS makes it really hard for Microsoft to compete with the multi-user nature of Unix.

Pens for Computer Scientists

Hey, you must have read several articles about these pens on the internet... They define how well can technology serve man... and how much technology can man invent. Click here for more info on these virtual laser keyboards. In brief, this is how it works...

A virtual keyboard is where a full-size image of a QWERTY keyboard is projected onto any surface. The virtual laser keyboard (VKB) works by using both infrared and laser technology to produce an invisible circuit and project a full-size virtual QWERTY keyboard on to any surface. The virtual PC keyboard behaves exactly like a real one: direction technology based on optical recognition enables the user to tap the images of the keys, complete with realistic tapping sounds(!), which feeds into the compatible PDA, Smartphone, laptop or PC.Touching the image of a key generates a unique electronic signal corresponding to a key's image.

They are not just pens with hidden cams... Seems like, they were once designed for James Bond. These virtual laser keyboards are already on sale... but their price prevents them from being ubiquitous. You may want to buy one of those ~ Check out the online stores. Though slightly old, here is a review of iTech's bluetooth-enabled virtual laser keyboard.

Category: [My Findings]

WHAT MORE CAN ROBOTS DO ?

I got hold of an interesting article on the web "Body of steel, heart of gold" that gives a short note on the works of Kim Jong-Hwan, director of ITRC (Intelligent Robot Research Centre). This gentleman is working on developing artificial chromosomes to create the feeling of lust in robots. I was wondering ... what exactly do we want robots to do? Whatz the purpose of giving them the ability to feel, reason and desire. Where in the society do they fit into?

Its good to imagine robots doing all the cleaning work at home and taking care of children. But imagine the situation like the one in "I-ROBOT" movie. Well... anything is possible, but not in other 25 years. Lets wait to see the robots being our slaves. Hope not the vice-versa.

IBM - THE INNOVATOR ALWAYS

I came across this article (click here)as I was browsing some tech-news at CNET. Well... this is a good article that focuses on the research works on the near-future trends that are carried out in IBM. As we all know, IBM always enjoys the pleasure of innovation.

THE INTERNET REBORN

Hey... i read an interesting technological update about replacing the Internet with something faster, more secure, and vastly smarter. Welcome to PlanetLab. The project is called PlanetLab, and within the next three years, researchers say, it will help revitalize the Internet, eventually enabling you to

• forget about hauling your laptop around. No matter where you go, you'll be able to instantly recreate your entire private computer workspace, program for program and document for document, on any Internet terminal
• escape the disruption caused by Internet worms and viruses-which inflicted an average of $81,000 in repair costs per company per incident in 2002-because the network itself will detect and crush rogue data packets before they get a chance to spread to your office or home;
• instantly retrieve video and other bandwidth-hogging data, no matter how many other users are competing for the same resources;
• archive your tax returns, digital photographs, family videos, and all your other data across the Internet itself, securely and indestructibly, for decades..

Its interesting .. isnt it? Awaiting to use it....