This is the latest in our ‘eat your greens’ series – a back to basics look at core statistical concepts that are often misunderstood or misapplied. In everyday speech the term ‘correlation’ refers to a mutual connection or relationship between two things. In statistics correlations are specific measures or values that attempt to quantify the …
This is the fifth post in our ‘eat your greens’ series – a back to basics look at some of the core concepts of statistics and analytics that, in our experience, are frequently misunderstood or misapplied. In this post we’ll look in more depth at the concept of the standard error and confidence intervals. In …
The gateway to inference – the standard error and confidence intervals Read More »
This is the fourth post in our ‘eat your greens’ series – a back to basics look at some of the core concepts of statistics and analytics that, in our experience, are frequently misunderstood or misapplied. In this post we’ll look in more depth at the concept of the normal distribution. One of the first …
Watch the other videos in this series
Watch the rest of the videos in this series
Watch the other videos in this series
Watch the rest of the videos in this series
None of us can have failed to notice the recent debacle over Ofqual’s (the Office of Qualifications and Examinations Regulation) use of an algorithm to predict pupil grades. Once again, ‘algorithm fever’ has generated a flurry of news articles questioning whether we are sleepwalking into a dystopian future where human expert decision-making is replaced with …
This is the third post in our ‘eat your greens’ series – a back to basics look at some of the core concepts of statistics and analytics that, in our experience, are frequently misunderstood or misapplied. In this post we’ll look in more depth at the concept of the standard distribution. I’m often struck by …
This is the second post in our ‘eat your greens’ series – a back to basics look at some of the core concepts of statistics and analytics that, in our experience, are frequently misunderstood or misapplied. In this post we’ll look in more depth at the concept of testing versus inferring. One of most daunting …
What do we mean when we say something is statistically significant? A guide to one of the most misused concepts in statistics.
In June of this year, IBM released the latest version of SPSS Statistics. Version 27 introduces several additional analysis procedures as well as new system enhancements. In this report we take a tour of some of the most valuable improvements that have been made. There’s a video of this tour here as well. Bootstrapping and …
PS Imago Pro is a statistical analysis and reporting solution based on IBM SPSS Statistics. Indeed, apart from the inclusion of an additional menu, users of SPSS Statistics may find that the data analysis module of PS Imago Pro looks almost identical to its SPSS counterpart. However, as Figure 1 shows, this additional menu contains …
You may have noticed that many media outlets are illustrating the tragic course of the coronavirus (COVID-19) pandemic by employing charts with log scales. These log values refer of course to the mathematical concept of logarithms. This is something that most of us learn about in school when we are taught that a logarithm is …
This infographic provides an overview of the four main families of approaches to predictive analytics. Prediction encompasses applications that aim to estimate or predict the values of a key target field. Segmentation refers to techniques such as cluster analysis which attempt to find the most ‘naturally occurring” groups within a dataset. Association modelling discovers groups …
An overview of the four main approaches to predictive analytics Read More »
At Smart Vision we’re in a pretty strong position to talk authoritatively about the reality of predictive analytics. That’s because we’re comprised of a team of veteran practitioners with decades of experience where we’ve all witnessed plenty of success stories but also one or two ‘data science’ train wrecks. Moreover, like anyone else, we’re exposed …
Like a lot of consultants working in the analytics industry, I’ve built up an extensive portfolio of materials to illustrate different kinds of applications and approaches. Some of these consist of files and slide decks used to explain quite esoteric procedures such as TURF analysis or Partial Least Squares. However, there are certain materials that …
Bootstrap aggregation, also called bagging, is a random ensemble method designed to increase the stability and accuracy of models. It involves creating a series of models from the same training data set by randomly sampling with replacement the data.
Boosting is another ensemble model-building method that was designed to help develop strong classification models from weak classifiers. Boosting methods focus on error (or misclassifications) that occur in prediction.
Feature Engineering is really just a fancy term for creating new data. Very often we can help an algorithm build better models by preparing the input data in a way that allows it to detect a clearer signal in the often noisy data. In machine learning variables are often referred to as ‘features’, so feature engineering refers to the transformation of variables or the creation of new ones.
Ensemble modelling refers to the practice of combining the predictions of separate models on the old principle that “two heads are better than one”. Ensemble methods can be particularly effective when combining models that have been created using completely different algorithms.
The idea of meta modelling is to build a predictive model using the predictions or scores generated by another model. By adding the predictive scores generated by an initial modelling algorithm to an existing pool of predictor fields, a second algorithm can then exploit these scores in to build a final more accurate model.
Split models or split population modelling is another technique that allows the user to build multiple models which can then be combined to create a single prediction. The idea with split modelling is that if the data represent different populations or contain separate groups that behave in very different ways, assuming that a single model can explain all the inherent variability across these distinct populations might be unreasonable.
In April of this year, IBM released the latest version of SPSS Statistics. Version 26 introduces a number of additional analysis procedures as well as new command enhancements. If you’re an existing SPSS user and you’d like to upgrade to v26 there’s more information about how to do that here. If you’re interested in trying SPSS …
The Regular Expressions for IBM SPSS Modeler node pack provides 4 nodes that integrate the power and flexibility of regular expression pattern matching into SPSS Modeler. However, some of these capabilities can be supported using the extension nodes built into SPSS Modeler and that begs the question – why buy the Regular Expression nodes? One …
Regular Expressions for IBM SPSS Modeler: performance comparison Read More »
In this video Jarlath Quinn takes a first look at SPSS Modeler v18.2 and demonstrates some of the new functionality that’s included within this release.
You may have noticed by now that there seem to be a couple of recurring themes in the plethora of articles and news programmes about artificial intelligence (AI). These themes can be summed up as a) “The dangers of AI” and b) “The limitations of AI”. Articles addressing the dangers of AI tend to focus …
Three questions to ask when reading articles about artificial intelligence Read More »
We’re often asked how you can change the appearance of the tables that SPSS generates as output. In this video Jarlath Quinn demonstrates two different ways to do this, either by choosing a different table look in the edit / options function, or by editing the table properties directly yourself.
In this video Jarlath Quinn demonstrates how to merge data files within SPSS Statistics using each of the two main methods, either adding cases (combining files with the same fields but additional rows) or adding variables (combining files by joining variables to a target file using something like an ID field as a ‘keyed variable’).
SPSS users often want to be able to create grouped or banded data from continuous fields such as, for example, creating age groups or income bands from continuous fields. In this video Jarlath Quinn demonstrates how to use the visual binning procedure within SPSS Statistics to do this.
Recoding your data means changing the values of a variable so that they represent something else. Within SPSS Statistics there is more than one type of recode that can be performed.
In this video Jarlath Quinn demonstrates how to use the functions within the explore command in SPSS Statistics to test for normality.
In this video Jarlath Quinn demonstrates how to work with date and time variables in SPSS using the SPSS date and time wizard.
In this video Jarlath Quinn demonstrates how to use SPSS Statistics to define data filters in order to select particular cases for analysis. This can be done either to create a temporary selection or to create a permanent new file with only a subsection of cases included within it.
In this video Jarlath Quinn demonstrates how to reverse the values of a rating scale (such as an agreement scale or a satisfaction scale) in SPSS Statistics, so that the highest value becomes the lowest value and vice versa.
SPSS users often want to know how they can combine variables together. In this video Jarlath Quinn demonstrates how to use the compute procedure to calculate the mean of a number of variables to create one combined variable, and also how to use the count values procedure to count how many times a particular value occurs across a series of variables in order to create an overall count.
In this video Jarlath Quinn introduces the popular TURF analysis technique and demonstrates how to apply it in IBM SPSS Statistics. TURF analysis is used in many industries to find the optimal sub-group of options from a wider portfolio in order to maximise their appeal to an audience or market.
In this video Jarlath Quinn explains what cluster analysis is, how it is applied in the real world and how easy it is create your own cluster analysis models in SPSS Statistics.
Sometimes you may have problems with your data issues not related so much to the values of the data but to the fields themselves, such as awkward field names. The filter node is a really useful tool that offers a bunch of tricks for dealing with awkward fields.
In this video we explore variable sets – a procedure in SPSS that allows you to generate subsets of variables or fields for display within dialogue boxes and in the data editor itself.
Used correctly, the generate menu offers analysts some substantial time saving benefits. Watch this video to learn more about how you can use the generate menu effectively.
We often talk to people who are unsure whether they need SPSS Statistics or whether SPSS Modeler might be more suited to their needs. In fact, it’s not always a clear cut choice as to which tool is more appropriate as it depends on the context in which the technology might be used. With that …
Do I need SPSS Statistics or Modeler? How to choose the right product for your needs Read More »
SPSS enables quite a high level of customisation so you can set up the software in a way that enables you to be a lot more productive, however many people are unaware of just how powerful these customisation options are. In this video we explore the options edit menu.
The data audit node is a powerful tool you can use to help understand the shape and structure of your data before your analysis begins. You can also make some decisions here regarding how you might want to clean up your data, for example by dealing with missing values or extremes and outliers.
Over the past two years I’ve noticed a steady stream of articles in the mainstream press and business journals centred on the themes of a) the dangers of machine learning 1 2 or b) the limitations of machine learning 3 4. Many of these articles refer to incidents where machine learning initiatives have echoed and …
Contemporary analytical platforms like SPSS and SAS represent the some of the earliest and yet longest-lived examples of proprietary software in the industry. When we think of the tectonic shifts the technology landscape has witnessed in last four decades, through the mainframe era, the rise of the PC, browser wars, the dotcom bubble, the smartphone …
How alternative interfaces can help you get more out of R Read More »
In part one of this video series Jarlath Quinn explains how operational analytics works, what data sources may be utilised and introduces the example case study that forms the basis of the subsequent videos in the series.
In part two of our predictive operational analytics video series, Jarlath Quinn introduces the IBM SPSS Modeler Software before showing how to explore the example data from the case study and carry out text analytics of engineer logs to categorise the information and create key fields for further analysis.
In part 3 of this predictive operational analytics guide Jarlath uses the results from the previous analysis stage to show how to build and assess a predictive model that identifies whether or not a part replacement will be required during a maintenance visit.
In this final part of our predictive operational analytics video series Jarlath shows how we can use the predictive technology to go further by predicting the actual outcome of the visit including which part is likely to require replacement.
