Why Web Scraping Software Won't Help

How to get continuous stream of data from these websites without getting stopped? Scraping logic depends upon the HTML sent out by the web server on page requests, if anything changes in the output, its most likely going to break your scraper setup.

If you are running a website which depends upon getting continuous updated data from some websites, it can be dangerous to reply on just a software.

Some of the challenges you should think:

1. Web masters keep changing their websites to be more user friendly and look better, in turn it breaks the delicate scraper data extraction logic.

2. IP address block: If you continuously keep scraping from a website from your office, your IP is going to get blocked by the "security guards" one day.

3. Websites are increasingly using better ways to send data, Ajax, client side web service calls etc. Making it increasingly harder to scrap data off from these websites. Unless you are an expert in programing, you will not be able to get the data out.

4. Think of a situation, where your newly setup website has started flourishing and suddenly the dream data feed that you used to get stops. In today's society of abundant resources, your users will switch to a service which is still serving them fresh data.

Getting over these challenges

Let experts help you, people who have been in this business for a long time and have been serving clients day in and out. They run their own servers which are there just to do one job, extract data. IP blocking is no issue for them as they can switch servers in minutes and get the scraping exercise back on track. Try this service and you will see what I mean here.




Source: http://ezinearticles.com/?Why-Web-Scraping-Software-Wont-Help&id=4550594

RFM - A Precursor to Data Mining

RFM in Action

RFM was initially utilized by marketers in the B-2-C space - specifically in industries like Cataloging, Insurance, Retail Banking, Telecommunications and others. There are a number of scoring approaches that can be used with RFM. We'll take a look at three:

RFM - Basic Ranking
RFM - Within Parent Cell Ranking
RFM - Weighted Cell Ranking

Each approach has experienced proponents that argue one over the other. The point is to start somewhere and experiment to find the one that works best for your company and your customer base. Let's look at a few examples.

RFM - Basic Ranking

This approach involves scoring customers based on each RFM factor separately. It begins with sorting your customers based on Recency, i.e., the number of days or months since their last purchase. Once sorted in ascending order (most recent purchasers at the top), the customers are then split into quintiles, or five equal groups. The customers in the top quintile represent the 20% of your customers that most recently purchased from you.

This process is then undertaken for Frequency and Monetary as well. Each customer is in one of the five cells for R, F, and M

Experience tells us that the best prospects for an upcoming campaign are those customers that are in Quintile 5 for each factor - those customers that have purchased most recently, most frequently and have spent the most money. In fact, a common approach to creating an aggregated score is to concatenate the individual RFM scores together resulting in 125 cells (5x5x5).

A customer's score can range from 555 being the highest, to 111 being the lowest.

RFM - Within Parent Cell Ranking

This approach is advocated by Arthur Middleton Hughes - one of the biggest proponents of RFM analysis. It begins like the one above, i.e., all customer are initially grouped into 5 cells based on Recency. The next step takes customers in a given Recency cell - say cell number 5, and then ranks those customers based on Frequency. Then customers in the 55 (RF) cell are ranked by monetary value.

RFM - Weighted Ranking

Weightings used by RFM practitioners vary. For example some advocate adding the RFM score together - thus giving equal weight to each factor. Consequently, scores can range from 15 (5+5+5) to 3 (1+1+1). Another weighting arrangement often used is, 3xR + 2xF + 1xM. In this case, scores can range from 30 to 3.

So which to use? In reality, there are many other permutations of approaches that are being used today. Best-practice marketing analytics requires a fine mix of mathematical and statistical science, creativity and experimentation. Bottom line, test multiple scoring methods to see which works best for your unique customer base.

Establishing a Score Threshold

After a test or production campaign, you will find that some of the cells were profitable while some were not. Let's turn to a case study to see how you can establish a threshold that will help maximize your profitability. This study comes from Professor Charlotte Mason of the Kenan-Flagler Business School and utilizes a real-life marketing study performed by The BookBinders Book Club (Source:Recency, Frequency and Monetary (RFM) Analysis, Professor Charlotte Mason, Kenan-Flagler Business School, University of North Carolina, 2003).

BookBinders is a specialty book seller that utilizes multiple marketing channels. BookBinders traditionally did mass marketing and wanted to test the power of RFM. To do so, they initially did a random mailing to 50,000 customers. The customers were mailed an offer to purchase The Art History of Florence. Response data was captured and a "post-RFM" analysis was completed. This "post analysis" was done by freezing the files of the 50,000 test customers prior to the actual test offer. Thus, the impact of this test campaign did not effect the analysis by coding many (the actual buyers) of the 50,000 test subjects as the most recent purchasers. The results firmly support the use of RFM as a highly effective segmentation approach.

Purchased the book = yes; months since last purchase = 8.61; total # purchases = 5.22; dollars spent = 234.30
Purchased the book = no; Months since last purchase = 12.73; total # purchases = 3.76; dollars spent = 205.74

Customers that purchased the book were more recent purchasers, more frequent purchasers and had spent the most with BookBinders.

The response rate for the top decile (18%) was twice the response rate associated with the 5th decile (9%).

Results from this test were then used by BookBinders to identify which of their remaining customers should receive the same mailing. BookBinders used a breakeven response rate calculation to determine the appropriate RFM cells to mail.

The following cost information was used as input:

Cost per Mail-piece $0.50

Selling Price $18.00

BookBinders Book Cost $9.00

Shipping Costs $3.00

Breakeven is achieved when the cost of the mailing is equal to the net profit from a sale. In this case:

Breakeven = (cost to mail the offer/net profit from a single sale)

= $0.50/($18-9-3)

= ($0.50/6)

= 8.3% = Breakeven Response rate

So, according to the test offer, profit can be obtained by mailing to cells that exhibited a response rate of greater than 8.3%

RFM dramatically improved profitability by capturing 71% of buyers (3,214/4,522) while mailing only 46% of their customers (22,731/50,000). And the return on marketing expenditures using RFM was more than eight times (69.7/8.5) that of a mass mailing.

Number of Cells and Cell Size Considerations

As previously mentioned, RFM was initially utilized by companies that operated in the B-to-C marketplace and generally possessed a very large number of customers. The idea of generating 125 cells using quintiles for R, F and M has been a very good practice as an initial modeling effort. But what if you are a B-to-B marketer with relatively fewer customers? Or, what if you are a B-to-C marketer with an extremely large file with millions of customers? The answer is to use the same approach that is used in data mining -- be flexible and experiment.

Establishing a minimum test cell size is a good place to start. Arthur Hughes recommends the following formula:

Test Cell Size = 4 / Breakeven Response Rate.

The Breakeven Response Rate was addressed above in the BookBinders case study. The number "4" is a number that Hughes has found works successfully based on many studies he has performed. BookBinders Breakeven Response Rate was 8.3%. Using the above formula, you would need a minimum of 48 customers in each cell (4/0.083). BookBinders actually had 400 customers per cell, so they had more than adequate comfort in the significance of their test. In reality, BookBinders could have created as many as 1,041 cells if they were comfortable using the minimum of 48 per cell. As an example, they could have used deciles as opposed to quintiles and established 1,000 cells (10 x 10 x 10). The more cells the finer the analysis, but of course the law of diminishing returns will arise.

Other weighting considerations can be used for small files. If your Breakeven Response Rate is 3%, your minimum cell size would be 133 customers (4/0.03). Therefore, if you have 12,000 customers you could have about 90 cells (12,000/133). As such, a 5 x 5 x 4 (100 cells) or a 5 x 4 x 4 (80 cells) approach may be appropriate.

Conclusions

RFM, BI and data mining are all part of an evolutionary path that is common to many marketing organizations. While RFM has been practiced for over 40 years, it still holds great value for many organizations. Its merits include:

- Simplicity - easy to understand and implement

- Relatively low cost

- Proven ROI

- The demand on data requirements are relatively low in terms of variables required and the number of records

- Once utilized, it sets up a broader foundation (from an infrastructure and business case perspective) to undertake more sophisticated data mining efforts

RFM's challenges include:

- Contact fatigue can be a problem for the higher scoring customers. A high level cross-campaign communication strategy can help prevent this.

- Your lowest scoring customers may never hear from you. Again, a cross-campaign communications plan should ensure that all of your customers are communicated with periodically to ensure low scoring customers are given the opportunity to meet their potential. Also, data mining and the prediction of customer lifetime value can help address this shortcoming.

- RFM includes only three variables. Data mining typically finds RFM-based variables to be quite important in response models. But there are additional variables that data mining typically use (e.g., detailed transaction, demographic and firmographic) that help produce improved results. Moreover, data mining techniques can also increase response rates via the development of richer segment/cell profiles that can be used to vary offer content and incentives.

As stated before, successful marketing efforts require analytics and experimentation. RFM has proven itself as an effective approach to predicting response and improving profitability. It can be an important stage in your company's evolution in marketing analytics.




Source: http://ezinearticles.com/?RFM---A-Precursor-to-Data-Mining&id=1962283

Data Discovery vs. Data Extraction

Looking at screen-scraping at a simplified level, there are two primary stages involved: data discovery and data extraction. Data discovery deals with navigating a web site to arrive at the pages containing the data you want, and data extraction deals with actually pulling that data off of those pages. Generally when people think of screen-scraping they focus on the data extraction portion of the process, but my experience has been that data discovery is often the more difficult of the two.

The data discovery step in screen-scraping might be as simple as requesting a single URL. For example, you might just need to go to the home page of a site and extract out the latest news headlines. On the other side of the spectrum, data discovery may involve logging in to a web site, traversing a series of pages in order to get needed cookies, submitting a POST request on a search form, traversing through search results pages, and finally following all of the "details" links within the search results pages to get to the data you're actually after. In cases of the former a simple Perl script would often work just fine. For anything much more complex than that, though, a commercial screen-scraping tool can be an incredible time-saver. Especially for sites that require logging in, writing code to handle screen-scraping can be a nightmare when it comes to dealing with cookies and such.

In the data extraction phase you've already arrived at the page containing the data you're interested in, and you now need to pull it out of the HTML. Traditionally this has typically involved creating a series of regular expressions that match the pieces of the page you want (e.g., URL's and link titles). Regular expressions can be a bit complex to deal with, so most screen-scraping applications will hide these details from you, even though they may use regular expressions behind the scenes.

As an addendum, I should probably mention a third phase that is often ignored, and that is, what do you do with the data once you've extracted it? Common examples include writing the data to a CSV or XML file, or saving it to a database. In the case of a live web site you might even scrape the information and display it in the user's web browser in real-time. When shopping around for a screen-scraping tool you should make sure that it gives you the flexibility you need to work with the data once it's been extracted.




Source: http://ezinearticles.com/?Data-Discovery-vs.-Data-Extraction&id=165396

Effectiveness of Web Data Mining Through Web Research

Web data mining is systematic approach to keyword based and hyperlink based web research for gaining business intelligence. It requires analytical skills to understand hyperlink structure of given website. Hyperlinks possess enormous amount of hidden human annotations that can help automatically understand the authority. If the webmaster provides a hyperlink pointing to another website or web page, this action is perceived as an endorsement to that webpage. Search engines highly focus on such endorsements to define the importance of the page and place them higher in organic search results.

However every hyperlink does not refer to the endorsement since the webmaster may have used it for other purposes, such as navigation or to render paid advertisements. It is important to note that authoritative pages rarely provide informative descriptions. For an instant, Google's homepage may not provide explicit self-description as "Web search engine."

These features of hyperlink systems have forced researchers to evaluate another important webpage category called hubs. A hub is a unique, informative webpage that offers collections of links to authorities. It may have only a few links pointing to other web pages but it links to a collection of prominent sites on a single topic. A hub directly awards authority status on sites that focus on a single topic. Typically, a quality hub points to many quality authorities, and, conversely, a web page that many such hubs link to can be deemed as a superior authority.

Such approach of identifying authoritative pages has resulted in the development of various popularity algorithms such as PageRank. Google uses PageRank algorithm to define authority of each webpage for a relevant search query. By analyzing hyperlink structures and web page content, these search engines can render better-quality search results than term-index engines such as Ask and topic directories such as DMOZ.




Source: http://ezinearticles.com/?Effectiveness-of-Web-Data-Mining-Through-Web-Research&id=5094403

Healthcare Marketing Series - Data Mining - The 21st Century Marketing Gold Rush

There is gold in them there hills! Well there is gold right within a few blocks of your office. Mining for patients, not unlike mining for gold or drilling for oil requires either great luck or great research.

It's all about the odds.

It's true that like old Jed from the Beverly Hillbillies, you might just take a shot and strike oil. But more likely you might drill a dry hole or dig a mine and find dirt not diamonds. Without research you might be a mere 2 feet from pay dirt, but drilling or mining in just the wrong spot.

Now oil companies and gold mining companies spend millions, if not, billions of dollars studying where and how to effectively find the "mother load". If market research is good enough for the big boys, it should be good enough for the healthcare provider. Remember as a health care professional you probably don't have the extras millions laying around to squander on trial and error marketing.

If you did there would be little need for you to market to find new patients to help.

In previous articles in the Health Care Marketing Series we talked about developing a marketing strategy, using metrics to measure the performance of your marketing execution, developing effective marketing warheads based on your marketing strategy, evaluating the most efficient ways to deliver those warheads, your marketing missile systems, and tying several marketing methods together into a marketing MIRV.

If you have been following along with our articles and starting to integrate the concepts detailed in them, by now you should have an excellent marketing infrastructure. Ready to launch laser guided marketing missiles tipped with nuclear marketing MIRVs. The better you have done your research, the more detailed your marketing strategy, the more effective and efficient your delivery systems, the bigger bang you'll receive from your marketing campaign. And ultimately the more lives you will help to change of patients that truly can benefit from your skills and talents as a doctor.

Sounds like you're ready for healthcare marketing shock and awe.

Everything is ready to launch, this is great, press the button and fire away!

Ah, but wait just a minute, General. What is the target? Where are they? What are the aiming coordinates?

The target? Why of course all those sick people out there.

Where are they? Well of course, out there!

The coordinates? Man just press the button, carpet bomb man. Carpet bomb!

This scenario is designed to show you how quickly the wheels can come off even the best intended marketing war machine. It brings us back full circle. We are right back to our original article on marketing strategy.

But this time we are going to introduce the concept of data mining. If you remember, our article on marketing strategy talked about doing research. We talked about research as the true cornerstone of all marketing efforts.

What is the target, General?

Answering this question is a little difficult and the truth is each healthcare provider needs to determine his or her high value target. And more importantly needs to know how to determine his or her high value targets.

Let's go back to our launch scenario to illustrate this point. Let's continue with our military analogy. Let's say we have several aircraft carriers, a few destroyers and a fleet of rowboats, making up our marketing battlefield.

As we have discussed previously, waging a marketing war, like any war, consumes resources. So do we want to launch our nuclear marketing MIRVs, the most valuable resources in our arsenal, and target the fleet of rowboats?

Or would it be wiser to target those aircraft carriers?

Well the obvious answer is "get those carriers".

But here is where things get a little tricky. One man's aircraft carrier is another man's rowboat.

You have to data mine your practice to determine which targets are high value targets.

What goes into that data mining process? Well first and foremost, what conditions do you 1.like to treat, 2. have a proven track record of treating and 3. obtain a reasonable reimbursement for treating.

In my own practice, I typically do not like or enjoy treating shoulder problems. I don't know if I don't like treating shoulders because I haven't had great results with them or if I haven't had great results, because I don't like treating them. Needless to say my reimbursement for treating shoulder cases is relatively low.

So do I really want to carpet bomb my marketing terrain and come up with 10 new cases of rotator cuff tears? These cases, for more than one reason, are my rowboats.

On the contrary, I like to treat neurological conditions like chronic pain; Neuropathy patients, Spinal Stenosis patients, Tinnitus patients, patients with Parkinson's Disease and Multiple Sclerosis patients. I've had results with these types of cases that have been good enough to publish. Because they are complex and difficult cases, I obtain a better than average reimbursement for my efforts. These cases are my aircraft carriers. If my marketing campaign brings me ten cases with these types of problems, chances are that the patient will obtain some great relief, I will find working with them an intellectual and stimulating challenge and my marketing efforts will bring me a handsome return on investment.

So the first lesson of data mining is to identify your aircraft carriers. They must be "your" aircraft carriers. You must have a good personal track record of helping these types of patients. You should enjoy treating these types of cases. And you should be rewarded for your time and expertise.

That's the first step in the process. Identifying your high value targets. The next step is THE most important aspect of healthcare marketing. As I discussed above, I enjoy working with complex neurological cases. But how many of these types of patients exist in my marketing terrain and are they looking for the type of help I can offer?

Being able to accurately answer these important questions is the single most valuable information I can extract using data mining.

It doesn't matter if I like treating these cases. It doesn't matter if I make a good living treating these cases. It doesn't matter if my success in treating these cases has made the local news. What matters is 1. do these types of cases exist in my neighborhood and 2. are they looking for the help I can provide to them?

You absolutely positively need to know who is looking for what in your marketing terrain and if what people are clamoring for is what you have to offer.

This knowledge is the most powerful tool in your marketing arsenal. It's your secret weapon. It is the foundation of your marketing strategy. It is so important that you should consider moving your office if the results of your data mining don't reveal an ocean full of aircraft carriers in your marketing terrain for you to target.

If your market research does not reveal an abundance of aircraft carriers on your horizon, you need to either 1. move to a new battlefield, 2. re-target your efforts towards the destroyers in your market or 3. try to create a market.

Let's look at your last choice. Trying to create a market. Unless you are Coke or Pepsi, your ability to create a market as a health care provider is extremely limited. To continue on with our analogy, to create a market requires converting rowboats into, at least, destroyers, but better yet aircraft carriers.

What would it cost if you took a rowboat to a ship yard and told them to rebuild it as an aircraft carrier?

This is what you face if you try to create a market where none exists. Unless you have a personality flaw and thrive on selling ice to Eskimos, creating a market is not a rewarding proposition.

So scratch this option off the table right now.

What about re-targeting your campaign towards destroyers? That's a viable option. It's a good option. It's probably your best option. It's an option that will likely give you your best return on investment. It is recommended that you focus your arsenal on the destroyers while at the same time never passing on an opportunity to sink an aircraft carrier.

So what is the secret? How do you data mine for aircraft carriers?

Well its quite simple in the internet age. Just use the services of a market research firm. I like http://www.marketresearch.com They will do the data mining for you.

They can provide market intelligence that will tell you not only what the health care aircraft carriers are, but also where they are.

With this information, you will have a competitive advantage in your marketing battlefield. You can segment, and target high value targets in your area while your competitors squander their marketing resources on rowboats. Or even worse carpet bomb and hit ocean water, not valuable targets.

Your marketing strategy should be highly targeted. Your marketing resources should be well spent. As we discussed in our very first article on true "Marketing Strategy" you should enter the battle against your competition already knowing your have won.

What gives you this dominant position in the market, is knowing ahead-of-time, who is looking for what in your marketing terrain. In other words, not trying to create a market, but rather identifying existing market niches, specifically targeting them with laser guided precision and having headlines and ad copy based on your strength versus the weakness of your competition within that niche.

This research-based marketing strategy is sure to cause a big bang with potential patients.

And leave your competition trying to sell ice to Eskimos.

I hope you see how important market research is and why it is a good thing to spend some of your marketing budget on research before you waste your marketing resources on poorly targeted low value or no-value targets. This article was intended to give you a glimpse at how to use data mining and consumer demographics information as a foundation for the development of a scientific research-based marketing strategy. This article shows you how to use existing resources to give your marketing efforts (and you) a competitive advantage.



Source: http://ezinearticles.com/?Healthcare-Marketing-Series---Data-Mining---The--21st-Century-Marketing-Gold-Rush&id=1486283

Online Data Entry and Data Mining Services

Data entry job involves transcribing a particular type of data into some other form. It can be either online or offline. The input data may include printed documents like Application forms, survey forms, registration forms, handwritten documents etc.

Data entry process is an inevitable part of the job to any organization. One way or other each organization demands data entry. Data entry skills vary depends upon the nature of the job requirement, in some cases data to be entered from a hard copy formats and in some other cases data to be entered directly into a web portal. Online data entry job generally requires the data to be entered in to any online data base.

For a super market, data associate might be required to enter the goods which have sold in a particular day and the new goods received in a particular day to maintain the stock well in order. Also, by doing this the concerned authorities will get an idea about the sale particulars of each commodity as they requires. In another example, an office the account executive might be required to input the day to day expenses in to the online accounting database in order to keep the account well in order.

The aim of the data mining process is to collect the information from reliable online sources as per the requirement of the customer and convert it to a structured format for the further use. The major source of data mining is any of the internet search engine like Google, Yahoo, Bing, AOL, MSN etc. Many search engines such as Google and Bing provide customized results based on the user's activity history. Based on our keyword search, the search engine lists the details of the websites from where we can gather the details as per our requirement.

Collect the data from the online sources such as Company Name, Contact Person, Profile of the Company, Contact Phone Number of Email ID Etc. are doing for the marketing activities. Once the data is gathered from the online sources into a structured format, the marketing authorities will start their marketing promotions by calling or emailing the concerned persons, which may result to create a new customer. So basically data mining is playing a vital role in today's business expansions. By outsourcing the data entry and its related works, you can save the cost that would be incurred in setting up the necessary infrastructure and employee cost.

E-dataentry is an offshore India based company providing superior quality data mining services to clients across the globe with high level of accuracy at reasonable price.



Source: http://ezinearticles.com/?Online-Data-Entry-and-Data-Mining-Services&id=7713395

Customer Relationship Management (CRM) Using Data Mining Services

In today's globalized marketplace Customer relationship management (CRM) is deemed as crucial business activity to compete efficiently and outdone the competition. CRM strategies heavily depend on how effectively you can use the customer information in meeting their needs and expectations which in turn leads to more profit.

Some basic questions include - what are their specific needs, how satisfied they are with your product or services, is there a scope of improvement in existing product/service and so on. For better CRM strategy you need a predictive data mining models fueled by right data and analysis. Let me give you a basic idea on how you can use Data mining for your CRM objective.

Basic process of CRM data mining includes:
1. Define business goal
2. Construct marketing database
3. Analyze data
4. Visualize a model
5. Explore model
6. Set up model & start monitoring

Let me explain last three steps in detail.

Visualize a Model:
Building a predictive data model is an iterative process. You may require 2-3 models in order to discover the one that best suit your business problem. In searching a right data model you may need to go back, do some changes or even change your problem statement.

In building a model you start with customer data for which the result is already known. For example, you may have to do a test mailing to discover how many people will reply to your mail. You then divide this information into two groups. On the first group, you predict your desired model and apply this on remaining data. Once you finish the estimation and testing process you are left with a model that best suits your business idea.

Explore Model:
Accuracy is the key in evaluating your outcomes. For example, predictive models acquired through data mining may be clubbed with the insights of domain experts and can be used in a large project that can serve to various kinds of people. The way data mining is used in an application is decided by the nature of customer interaction. In most cases either customer contacts you or you contact them.

Set up Model & Start Monitoring:
To analyze customer interactions you need to consider factors like who originated the contact, whether it was direct or social media campaign, brand awareness of your company, etc. Then you select a sample of users to be contacted by applying the model to your existing customer database. In case of advertising campaigns you match the profiles of potential users discovered by your model to the profile of the users your campaign will reach.

In either case, if the input data involves income, age and gender demography, but the model demands gender-to-income or age-to-income ratio then you need to transform your existing database accordingly.



Source: http://ezinearticles.com/?Customer-Relationship-Management-%28CRM%29-Using-Data-Mining-Services&id=4641198