This is the fourth post in our ‘eat your greens’ series – a back to basics look at some of the core concepts of statistics and analytics that, in our experience, are frequently misunderstood or misapplied. In this post we’ll look in more depth at the concept of the normal distribution. One of the first …
Watch the other videos in this series
Watch the rest of the videos in this series
Watch the other videos in this series
Watch the rest of the videos in this series
None of us can have failed to notice the recent debacle over Ofqual’s (the Office of Qualifications and Examinations Regulation) use of an algorithm to predict pupil grades. Once again, ‘algorithm fever’ has generated a flurry of news articles questioning whether we are sleepwalking into a dystopian future where human expert decision-making is replaced with …
This is the third post in our ‘eat your greens’ series – a back to basics look at some of the core concepts of statistics and analytics that, in our experience, are frequently misunderstood or misapplied. In this post we’ll look in more depth at the concept of the standard distribution. I’m often struck by …
This is the second post in our ‘eat your greens’ series – a back to basics look at some of the core concepts of statistics and analytics that, in our experience, are frequently misunderstood or misapplied. In this post we’ll look in more depth at the concept of testing versus inferring. One of most daunting …
What do we mean when we say something is statistically significant? A guide to one of the most misused concepts in statistics.
In June of this year, IBM released the latest version of SPSS Statistics. Version 27 introduces several additional analysis procedures as well as new system enhancements. In this report we take a tour of some of the most valuable improvements that have been made. There’s a video of this tour here as well. Bootstrapping and …
PS Imago Pro is a statistical analysis and reporting solution based on IBM SPSS Statistics. Indeed, apart from the inclusion of an additional menu, users of SPSS Statistics may find that the data analysis module of PS Imago Pro looks almost identical to its SPSS counterpart. However, as Figure 1 shows, this additional menu contains …
Lorna: Could you start just by talking a little bit about your own background and how you’ve come to be working with data? Mark: I suppose my background’s different from a lot of people who do this kind of work. They often come from a maths, stats or science background, but I didn’t do A-level …
Expert insight – Mark Langer Crame, Senior Data Intelligence Team Lead, Jisc Read More »
You may have noticed that many media outlets are illustrating the tragic course of the coronavirus (COVID-19) pandemic by employing charts with log scales. These log values refer of course to the mathematical concept of logarithms. This is something that most of us learn about in school when we are taught that a logarithm is …
IBM SPSS Modeler supports Python scripting using Jython, a Java[tm] implementation of the Python language. Modeler versions 16 and 17 use Jython 2.5.1 which includes a number of useful and popular modules. However, many other modules are available and customers often want to use their own so a frequent question is how to include them. There …
Adding New Modules To Jython Scripting In IBM SPSS Modeler Read More »
Most SPSS Modeler scripts include code that locates an existing node e.g.: stream = modeler.script.stream() typenode = stream.findByType(“type”, None) However, some scripts need to search for all nodes – maybe by node type but also matching some other criteria. The Modeler scripting API documentation (PDF) mentions a findAll() function: d.findAll(filter, recursive): Collection filter (NodeFilter) : the node filter recursive …
Using the find all() Function to search for nodes Read More »
This infographic provides an overview of the four main families of approaches to predictive analytics. Prediction encompasses applications that aim to estimate or predict the values of a key target field. Segmentation refers to techniques such as cluster analysis which attempt to find the most ‘naturally occurring” groups within a dataset. Association modelling discovers groups …
An overview of the four main approaches to predictive analytics Read More »
There are a number of configuration settings associated with IBM SPSS Modeler Server that control its behaviour. The default settings aim to ensure that stream execution will complete successfully even if the host machine is being used by a number of other applications i.e. Modeler Server is trying to be a “good citizen”. However, if …
Using SPSS Modeler’s cache_compression setting to speed up your modelling Read More »
Gertrude Sai is Quantitative Manager, Evidence and Insights, at Girl Effect. Lorna: Can you start off by telling me a bit about your background and how you came to be where you are now? Gertrude: Yes, sure. My background is quite varied but mainly analytical. My degree in Psychology was definitely centered around statistics and …
Expert insight: Gertrude Sai, Quantitative Manager, Girl Effect Read More »
Lorna: Can you start by telling me something about how you came to be working with statistics and data? Hanna: I’ve always been interested in the psychology of why people do things and that led me to studying psychology at A-level, for my undergraduate degree and for my masters as well. Whilst doing my undergraduate …
Expert insight: Hanna Adeyinka, Customer Researcher, Hyde Housing Read More »
Inna Yordanova is Senior Researcher at IPSE – the Association of Independent Professionals and the Self-Employed. IPSE is the largest organisation representing freelancers and self-employed people. Its 78,000 members are freelancers, contractors, consultants and other self-employed people from all sectors of the economy. Lorna: Can you start off then just by telling me a little …
Expert insight: Inna Yordanova, Senior Researcher at IPSE Read More »
The good news is, it is a product that is still very much alive and in development, albeit it has had a long history of ownership and name changes! Back in the mists of time Data Collection Reports for Surveys was a staple component of the SPSS Market Research suite of tools. When the company …
What ever happened to Data Collection Reports for Surveys? Read More »
Through independent research, evaluation and strategic consulting support, SBD Automotive helps vehicle manufacturers and their partners create autonomous, more secure and better connected cars. Adam O’Shaughnessy is a Senior Specialist – Connected Car. Lorna: Can you tell me a little bit about your background and how you’ve come to be working with data and statistics …
Expert insight – Adam O’Shaughnessy, SBD Automotive Read More »
Rob Woods has worked in the analytics industry for over 20 years. He’s currently an Analytics Solutions Architect working on IBM’s Watson FSS suite of products. All views here are his own and not IBM’s. Describe your own background and how you came to be working with statistics / data / analytics I started as …
Expert insight: Rob Woods, Analytics Solution Architect/Data Scientist, Watson FSS Read More »
At Smart Vision we’re in a pretty strong position to talk authoritatively about the reality of predictive analytics. That’s because we’re comprised of a team of veteran practitioners with decades of experience where we’ve all witnessed plenty of success stories but also one or two ‘data science’ train wrecks. Moreover, like anyone else, we’re exposed …
Like a lot of consultants working in the analytics industry, I’ve built up an extensive portfolio of materials to illustrate different kinds of applications and approaches. Some of these consist of files and slide decks used to explain quite esoteric procedures such as TURF analysis or Partial Least Squares. However, there are certain materials that …
Bootstrap aggregation, also called bagging, is a random ensemble method designed to increase the stability and accuracy of models. It involves creating a series of models from the same training data set by randomly sampling with replacement the data.
Boosting is another ensemble model-building method that was designed to help develop strong classification models from weak classifiers. Boosting methods focus on error (or misclassifications) that occur in prediction.
Feature Engineering is really just a fancy term for creating new data. Very often we can help an algorithm build better models by preparing the input data in a way that allows it to detect a clearer signal in the often noisy data. In machine learning variables are often referred to as ‘features’, so feature engineering refers to the transformation of variables or the creation of new ones.
Ensemble modelling refers to the practice of combining the predictions of separate models on the old principle that “two heads are better than one”. Ensemble methods can be particularly effective when combining models that have been created using completely different algorithms.
The idea of meta modelling is to build a predictive model using the predictions or scores generated by another model. By adding the predictive scores generated by an initial modelling algorithm to an existing pool of predictor fields, a second algorithm can then exploit these scores in to build a final more accurate model.
Split models or split population modelling is another technique that allows the user to build multiple models which can then be combined to create a single prediction. The idea with split modelling is that if the data represent different populations or contain separate groups that behave in very different ways, assuming that a single model can explain all the inherent variability across these distinct populations might be unreasonable.
In April of this year, IBM released the latest version of SPSS Statistics. Version 26 introduces a number of additional analysis procedures as well as new command enhancements. If you’re an existing SPSS user and you’d like to upgrade to v26 there’s more information about how to do that here. If you’re interested in trying SPSS …
Lorna: Please could you start just by talking a little bit about your own background and how you’ve come to be working in data? Cliff: I went to university to do a mapping science degree, during which time I had a year out in industry. I worked as a surveyor for a company that was …
Expert insight: Clifford Budge, Head of Ecommerce Data Science, major toy manufacturer Read More »
Here is an interesting confession. I work for an analytics company but I would really rather not think about or talk about analytics. Let me explain. I once worked with an operations director. He had a great team of analysts. He valued their work and was interested in the information they gave him but the …
I work for an analytics company, but I don’t want to talk about analytics Read More »
The Regular Expressions for IBM SPSS Modeler node pack provides 4 nodes that integrate the power and flexibility of regular expression pattern matching into SPSS Modeler. However, some of these capabilities can be supported using the extension nodes built into SPSS Modeler and that begs the question – why buy the Regular Expression nodes? One …
Regular Expressions for IBM SPSS Modeler: performance comparison Read More »
I was talking to a potential client recently who was frustrated by the lack of ‘real analysis’ happening in his organisation. He had a team of analysts working for him, most of whom were using Excel when he really wanted them to be using Python. Although we don’t generally recommend Excel for advanced analytics – …
Don’t overlook the value of quick wins in your analytics project Read More »
In this video Jarlath Quinn takes a first look at SPSS Modeler v18.2 and demonstrates some of the new functionality that’s included within this release.
This year SPSS is 50 years old. Development started in 1965 by a team of political scientists, frustrated at how much time they had to spent on manual data cleaning before they could start their analysis. The first release was in 1968 and by the end of the 1960s SPSS was in use in over …
Expert insights: Major Lester, founder of SPSS UK talks about fifty years of SPSS Read More »
You may have noticed by now that there seem to be a couple of recurring themes in the plethora of articles and news programmes about artificial intelligence (AI). These themes can be summed up as a) “The dangers of AI” and b) “The limitations of AI”. Articles addressing the dangers of AI tend to focus …
Three questions to ask when reading articles about artificial intelligence Read More »
When we’re talking to potential clients about advanced analytics we often ask them what tools they’re currently using. More often than not they say they’re using spreadsheets. Spreadsheets are one of the most widely used tools for statistical analysis and of course, most businesses couldn’t run without them. However, when it comes to advanced analytics …
Thinking of using spreadsheets for advanced analytics? Think again. Read More »
Can you tell us a bit about yourself, your background and how you came to be where you are career-wise? I started my career in 2001 at Research International’s Marketing Science Centre, following a degree in Sociology and Social Policy. Back then I was working mainly on analysis of market research surveys. Over time I …
Expert insight – Paul Jackson, Head of Advanced Analytics, Bonamy Finch Read More »
Can you start off by telling us a bit about your background, and how you came to be working in the analytics field? I came to analytics via a somewhat circuitous route. My first degree was in psychology, then I did a postgraduate qualification in newspaper journalism. Obviously psychology is all about the science of …
Expert insight – John Gill, Head of Insight and Analytics, Betfred Read More »
We’re often asked how you can change the appearance of the tables that SPSS generates as output. In this video Jarlath Quinn demonstrates two different ways to do this, either by choosing a different table look in the edit / options function, or by editing the table properties directly yourself.
In this video Jarlath Quinn demonstrates how to merge data files within SPSS Statistics using each of the two main methods, either adding cases (combining files with the same fields but additional rows) or adding variables (combining files by joining variables to a target file using something like an ID field as a ‘keyed variable’).
SPSS users often want to be able to create grouped or banded data from continuous fields such as, for example, creating age groups or income bands from continuous fields. In this video Jarlath Quinn demonstrates how to use the visual binning procedure within SPSS Statistics to do this.
Recoding your data means changing the values of a variable so that they represent something else. Within SPSS Statistics there is more than one type of recode that can be performed.
In this video Jarlath Quinn demonstrates how to use the functions within the explore command in SPSS Statistics to test for normality.
In this video Jarlath Quinn demonstrates how to work with date and time variables in SPSS using the SPSS date and time wizard.
In this video Jarlath Quinn demonstrates how to use SPSS Statistics to define data filters in order to select particular cases for analysis. This can be done either to create a temporary selection or to create a permanent new file with only a subsection of cases included within it.
In this video Jarlath Quinn demonstrates how to reverse the values of a rating scale (such as an agreement scale or a satisfaction scale) in SPSS Statistics, so that the highest value becomes the lowest value and vice versa.
