A few extra tips on data sculpting

Identify what changes in the situation: movement of units? Size? Value?

Identify the time scale. Does change happen over seconds? Minutes? Days? Months? Years? Centuries?

Identify how you can gather data about the situation. In some cases, the data may already be gathered for you: stock market rates, weather statistics, sports statistics, etc. are all available online, and you could use the available data as your source. Even in these cases, it is valuable to know how the data is gathered, so you know what is being left out, and how that omission may affect your work.

If you are gathering data yourself, the challenge is to identify what you can sense, measure, or quantify. Specific physical characteristics are easy to quantify: how much light, how far distant an object is, how many people are in a room, etc. Abstract qualities, while often more aesthetically interesting, are harder to measure: a person's attention, the tension in a room, activity in a lab, etc. If you are trying to investigate these more abstract qualities, you have to first identify some physical patterns or behaviors that you can identify consistently with those more abstract qualities.

When you identify a specific physical behavior, ask yourself whether it is unique to the quality you have in mind, or whether it can be associated with some other, non-related activity. For example, does proximity to an object mean I am paying attention to it, or merely that there is nowhere else to stand? Does the fact that my pulse is changing mean that I am lying, or just that I am aroused by the person next to me? This is an excellent opportunity for potential user surveys. We all have physical qualities that we may identify with mental states, but those identifications are not universal. Check yourself to see if your assumptions mirror those of your potential participants.

It may be that several physical characteristics combined give you the information you need, whereas one alone does not. For example, if I am facing an object, and I am a certain distance from it and I have not changed my position or facing for several seconds or minutes, the odds are good that I am paying attention to it. As long as there's nothing else in the same vicinity that I might be paying attention to.

Identify what constitutes the normal pattern of change. If you were examining the problem mathematically, this "normal" might constitute a formula characterizing the change. For example, if the pattern of movement of people in a room varies directly with the time of day, you've got a one-to-one relationship. If you're talking about how the tides vary with time of day, the relationship is sinusoidal, i.e. following a sinewave path.

Some patterns are not characterized by a specific formula, but they are still predictable within a certain range. For example, the stock market is not regular, but fluctuation within a two-percent range is considered "normal", while anything outside that range is considered an abnormal event.

Determine what in your data could constitute an abnormal event. For example, if you were reading an array of light sensors using a 10-bit ADC, you might expect to see a fluctuation of 10-20 points on each sensor. A sensor that's reading consistently outside the range of normal fluctuation around the average would be an abnormal event.

Compare your readings over time to determine what's normal and what's not. A sample of movement readings over a ten-minute period, for example, will give you a sense of what level of change is normal second-to-second, and what is not. Make sure "listen" long enough to tell what the norm is.

Be prepared for the norm to change. Taking the example above, the ten minutes of movement between classes in a schoolroom hallway will have a very different average level of movement than the ten minutes after the beginning of class. You might need to design a system that dynamically adjusts for such change, constantly comparing its past readings to its present one to determine what is "normal".