Meta S. Brown

Cheat Sheet / Updated 02-17-2022

Data mining is the way that ordinary businesspeople use a range of data analysis techniques to uncover useful information from data and put that information into practical use. Data miners don’t fuss over theory and assumptions. They validate their discoveries by testing. And they understand that things change, so when the discovery that worked like a charm yesterday doesn’t hold up today, they adapt.

Step by Step / Updated 03-27-2016

University of Waikato faculty members develop tools as part of their work toward advancement of the field of machine learning. These tools are used in teaching, by scientists, and in industry. Weka is its general-purpose data-mining tool that offers a visual programming interface and a wide range of analytics capabilities. MOA is for real-time mining of data streams. To import the sample data in Weka, follow these steps:

Step by Step / Updated 03-27-2016

Using codes for data reduces data entry time, prevents errors, and reduces the memory requirements for storing the data. But the codes aren’t meaningful unless you have documentation, or labels, to explain their meaning. Some data formats enable you to enjoy the advantages of using codes while keeping the information about the meaning of the codes in the same file. These aren’t typical in data mining — you’re more likely to see them in statistical analysis products — but some data-mining applications can use these labeled data formats. Here’s how they work.

Step by Step / Updated 03-27-2016

Data miners work fast. One way to improve your productivity is to take full advantage of tools that let you do several things at once. It’s time-consuming (and boring) to set up a number of graphs separately, one at a time. So use these alternatives whenever you can:

Step by Step / Updated 03-27-2016

Because data miners lean heavily on basic graphs, some data-mining applications offer little or nothing more. Others provide a wide range of graph options, from the common to the exotic. It’s not necessary to use all of these, but you may benefit by selecting and using a few that suit your own needs. Data miners often use these graphs:

Step by Step / Updated 03-27-2016

Your first hands-on step with data is getting it from wherever it is to the place where you need it to be. Text formats are common, and you’re likely to encounter them often. One of the most common is comma-separated value (.csv) text. KNIME.com AG is a small software and services firm focused on data mining. It offers a data-mining product with a visual programming interface. To open the sample data in KNIME:

Step by Step / Updated 03-27-2016

RapidMiner is a small software and services firm focused on data mining. It offers a data-mining product with a visual programming interface. To open the sample data in RapidMiner, follow these steps:

Step by Step / Updated 03-27-2016

The Bioinformatics Laboratory of the Faculty of Computer and Information Science, University of Ljubljana, Slovenia, develops Orange in cooperation with an open source community. To open the sample data in Orange, follow these steps:

Article / Updated 03-26-2016

The Cross-Industry Standard Process for Data Mining (CRISP-DM) is the dominant data-mining process framework. It's an open standard; anyone may use it. The following list describes the various phases of the process. Business understanding: Get a clear understanding of the problem you're out to solve, how it impacts your organization, and your goals for addressing it. Tasks in this phase include: Identifying your business goals Assessing your situation Defining your data mining goals Producing your project plan Data understanding: Review the data that you have, document it, identify data management and data quality issues. Tasks for this phase include: Gathering data Describing Exploring Verifying quality Data preparation: Get your data ready to use for modeling. Tasks for this phase include: Selecting data Cleaning data Constructing Integrating Formatting Modeling: Use mathematical techniques to identify patterns within your data. Tasks for this phase include: Selecting techniques Designing tests Building models Assessing models Evaluation: Review the patterns you have discovered and assess their potential for business use. Tasks for this phase include: Evaluating results Reviewing the process Determining the next steps Deployment: Put your discoveries to work in everyday business. Tasks for this phase include: Planning deployment (your methods for integrating data mining discoveries into use) Reporting final results Reviewing final results

Article / Updated 03-26-2016

Data privacy is a big issue for data miners. News reports outlining the level of personal data in the hands of the US government's National Security Agency and breaches of commercial data sources have raised public awareness and concern. A central concept in data privacy is personally identifiable information (PII), or any data that can be traced to the individual person it describes. PII includes obvious identifiers such as names, credit card numbers, and social security numbers, and most data miners are well aware that this kind of data is private and must be handled with care. But PII refers to more than just these obvious identifiers. Any data that could be used to identify an individual, even if doing so requires using several fields in combination or manipulating the data in some way, is also PII. It's easy for data miners to overlook this kind of data, the kind that does not appear on the surface to be private, and yet could be sufficient for personal identification if it were manipulated for that purpose. If there is any way that data could be manipulated to identify individuals, it must be handled with the same precautions as you would give a list of credit card numbers. That's where data miners can easily get themselves in trouble. There are many ways to identify individuals if you make a little effort to do so. In one notable example, AOL Research released user search records for research use. The data was intended to be anonymous, there were no names in it, but The New York Times reported that it had been able to identify an individual from the search data by cross-referencing with phone listings. Later, Netflix made movie rating data available for use in a competition, and it was soon revealed that this data, too, could be used to identify individuals. In your work as a data miner, you may have experiences with prospective clients who shared data they claimed was anonymous (or even faked, to illustrate a point of discussion), but found that the data was nothing of the kind. Knowingly or not, these people are violating data privacy laws and exhibiting a lack of respect for their own customers. So, how can you prevent disasters like these? Don't try to do it alone. It's challenging to ensure compliance with all relevant data privacy laws, not to mention other good business practices. Jenny Juliany, Vice President of Solutions Architecture and Co-Founder of Intreis, a solutions integrator specializing in service management and compliance automation, describes the life cycle of data with an analogy to the four seasons: Spring: Inception, the data is created. Summer: Primetime, the data is in active use. Fall: Retirement, the data is no longer relevant or used, but there may be legal or other reasons to retain it. Winter: Removal, the data is destroyed. Each season has its own characteristics, with distinct requirements surrounding data privacy. Some are grounded in the law, others in common sense, and still others in individual agreements with clients and your own employer's business practices. It's not realistic to believe you can take on all these compliance details in addition to your primary role, so you must partner with your organization's data management professionals. You don't want to be the center of the next big data privacy scandal. Respect for data privacy and proper data management is the key to minimizing that risk. Don't wait until something goes wrong, contact the data privacy expert in your own organization today, and start building a working partnership to properly manage sensitive data. More details on the data lifecycle from Jenny Juliany on the Four Seasons of Data Management can be found here: 'Spring' Inception 'Summer' Primetime 'Fall' Retirement 'Winter' Removal