There are several ways to obtain results for statistical significance on the aytm platform, including within charts and visualizations, with our Correlation Finder, and by exporting from the Crosstabs menu.
Real-time Significance Testing
By default, when you hover over your charts on your survey results page, you will see the data update in real-time. Learn more about how to read the charts.
To take your analysis a step further, or to utilize correlation scans (crosstab) on a single banner, try Correlation Finder from the Data Lab menu. Our Correlation Finder performs an analysis on the banner and attribute of your choice right in your results, including a chart. Learn how to use Correlation Finder or watch the video.
Data Lab Exports: Sig Tests and Correlation
To compare banners across multiple questions, or to obtain the specific p-values and full details of your survey's statistical tests, export a traditional Significance Test from the Analysis (microscope) tab. You can run a t- or z-test (significance test) or a chi-squared (correlation scan) for statistical analysis, or download raw numbers as shown below. You can adjust each slider to change the primary and secondary confidence levels.
After you click "Run Significance Test" your export will download as an excel file.
Note: if you choose the Raw Numbers export, no statistical analysis will be performed - it will only include the raw data.
Example: Significance Test Export
When you open the Excel file, the first sheet is the data with statistics included. The export shows the banner as columns, and as you scroll down you can see how the respondents who fell into each split of the banner answered all other survey questions.
Note: significance test, single banner (gender), 95%/90% CI, normal mode.
Cells with bold text are significant, and green fill color indicates significance at the primary confidence level you set. Letters within the cells indicate which cells in the row this cell holds significant difference from (greater-than). You can also hover over the notation triangles to see the exact details of the statistics for these cells. There is another tab labeled “legend” that will explain the notation and details shown in this data.
View of the Legend tab in the export:
Spend a few minutes with the legend to get a good grasp of this.
Example: Correlation Scan Export
You can run a correlation scan in the same dropdown with most of the same options as a significance test.
When your export finishes you'll have an Excel file that is setup with banner(s) as columns, and questions as rows just like a significance test. The difference is in the statistical test used by the analysis (chi-squared) and thus, the notations presented. Because chi-squared tests assume independence of cells, you’ll only see colored shading when the null hypothesis – which is that the banner and attribute are completely independent – is rejected.
In other words, there is statistical significance in the colored cells, which indicates the banner and attribute may be related. The coloring of the cells indicates which cell has a higher incidence (directionality) and the intensity of color indicates how strong the correlation is.
Note: correlation scan, single banner (gender), 95%/90% CI, normal mode.
Example: Condensed Mode
For both types of analysis, you can click the button for Condensed Mode to remove the totals from your export spreadsheet, leaving percentages only. This feature is useful if you're building your charts from scratch by copying and pasting rows from aytm's export.
Note: significance test, single banner (gender), 95%/90% CI. Normal mode on left, condensed mode on right.
Tips and Tricks for Analysis
We highly recommend using Virtual Questions to roll-up, organize, and more precisely slice-and-dice your data for use as banners! The resources below can show you how to do that using real-world examples.