The Information Visualization community has begun to pay attention to visualization literacy; however, researchers still lack instruments for measuring the visualization literacy of users. In order to address this gap, we systematically developed a visualization literacy assessment test (VLAT), especially for non-expert users in data visualization, by following the established procedure of test development in Psychological and Educational Measurement: (1) Test Blueprint Construction, (2) Test Item Generation, (3) Content Validity Evaluation, (4) Test Tryout and Item Analysis, (5) Test Item Selection, and (6) Reliability Evaluation. The VLAT consists of 12 data visualizations and 53 multiple-choice test items that cover eight data visualization tasks. The test items in the VLAT were evaluated with respect to their essentialness by five domain experts in Information Visualization and Visual Analytics (average content validity ratio = 0.66). The VLAT was also tried out on a sample of 191 test takers and showed high reliability (reliability coefficient omega = 0.76). In addition, we demonstrated the relationship between users' visualization literacy and aptitude for learning an unfamiliar visualization and showed that they had a fairly high positive relationship (correlation coefficient = 0.64). Finally, we discuss evidence for the validity of the VLAT and potential research areas that are related to the instrument.

VLAT : Development of a Visualization Literacy Assessment Test

The construct of test-wiseness (TW) has a relatively short history in educational research. Originally suggested as a possible effector of reliability (Thorndike, 1951), it has since become a frequently used term, but one not without its problems of interpretation. Early confusion over the precise components and functioning of TW stemmed from the fact that although most test-constructors testified to its existence, no empirical research was performed to thoroughly examine the construct. Thorndike's pioneering discussion of TW classified it as a possible source of variance in test scores, and described it as a persistent general trait of the test-taker. Although he did postulate possible characteristics of the test-wise individual, Thorndike's brief treatise of TW was descriptive in nature, and did not suggest any specific directions for research. Similarly, in another early appearance in the literature, TW was listed as a component of response variance in objective test questions, but no experimentation was cited or proposed (Ebel & Damrin, 1960). In providing the first thorough empirical investigation of TW, Gibb (1964) illustrated the dearth of research, by claiming that the most complete reference on the subject was the early work of Thorndike (1951). However, Gibb's own work altered that circumstance. In examining individual differences in TW skills, he provided an operational definition of TW, and also developed an instrument to measure the construct.

An Examination of Test-Wiseness In the Cognitive Test Domain

In a series of articles, Mondak and colleagues argue that the conventional way of measuring political knowledge in surveys is flawed. Personality related "propensity to guess" underestimates the level of political knowledge in the population and distorts estimates of between group differences, when a DK alternative is offered. This has led Mondak to recommend the use of closed-ended items on which DKs are not explicitly offered, following best practice in the field of educational testing. In this article, we present the results of an experimental study which calls into question the wisdom of this approach. Our results show little evidence of partial knowledge concealed within DK responses; when people who initially select a DK alternative are subsequently asked to provide a "best guess," they fare statistically no better than chance. We conclude that opinion researchers should be cautious about adopting Mondak's recommendations for the design of political knowledge items in surveys.

An experiment on the measurement of political knowledge in surveys

Objectives

In multiple-choice tests using a `don't-know' option the number of correct minus incorrect answers was used as the test score (formula scoring) in order to reduce the measurement error resulting from random guessing. In the literature diverging results are reported when comparing formula scoring and number-right scoring, the scoring method without the don't-know option.



Design

To investigate which method was most appropriate, both scoring methods were used in true–false tests (block tests) taken at the end of a second- and third-year educational module (block). The students were asked to answer each item initially by choosing from the response options true, false or don't know, and secondly to replace all don't-know answers by a true–false answer.



Setting

Maastricht University, The Netherlands.



Subjects

Medical students.



Results

The correct scores for the don't-know answered items were found to be 4·5% and 5·9%, respectively, higher than expected with pure random guesswork. This represents a source of bias with formula scoring, because students who were less willing to guess obtained lower scores. The average difference in the correct minus incorrect score for the two scoring methods (2·5%, P < 0·001, and 3·4%, P < 0·001, respectively) indicates the size of the bias (compare: the standard deviation of the score equals 11%). Test reliability was higher with formula scoring (0·72 vs. 0·66 and 0·74 vs. 0·66), but the difference decreased when the test was restricted to items which were close to the core content of the block (0·81 vs. 0·77, resp. 0·75 vs. 0·70).



Conclusions

In deciding what scoring method to use, less bias (number-right scoring) has to be weighed against higher reliability (formula scoring). Apart from these psychometric reasons educational factors must be considered.

The effect of a ‘don't know’ option on test scores: number‐right and formula scoring compared

An empirical methodology is proposed for determining optimal cutting scores for short-fixed-length criterion-referenced tests. Classification of outcome probabilities and validity coefficient approaches are developed using validation samples of instructed and uninstructed students. The optimal cutting score is selected according to the estimated probabilities of correct and incorrect mastery-nonmastery decisions. These probabilities, along with the gains and losses associated with the decisions, are then incorporated into an index of utility for identifying the cutting score which maximizes test usefulness in specific individual and group decision situations. The practical value of using the test compared to an alternative is also examined in terms of incremental validity.

Determination of Optional Cutting Scores in Criterion - Referenced Measurement,

Gibb (1964) defined test-wiseness (TW) as the ability to respond advantageously to multiple-choice items containing extraneous clues and therefore to obtain credit without knowledge of the subject matter being tested. This study investigated TW in a sample of 6th grade pupils. A test instrument was developed utilizing fictitious material similar to the strategy employed by Slakter, Koehler & Hampton (1970). The study examines the cognitive correlates of TW through the interpretation of correlational matrices. The results lend support to the notion that TW is not a general trait, but rather is clue-specific.

/pdf/an-investigation-of-the-cognitive-correlates-of-test-ib37vmo7uy.pdf

An investigation of the cognitive correlates of test‐wiseness 1,2

Many evaluation paradigms consider three rather similar concepts: input; implementation; and output. Until recently, little attention has been paid to the implementation phase of evaluation, which seems to be a critical area in need of assessment by both administrators and evaluators. The literature relating to implementation has tended to document the need for implementation studies, but has not been very useful in describing, in practical terms, how one may go about the process of assessing implementation or the subsequent use of such data for administrative feedback. This paper discusses the development and use of an implementation scale and describes basic characteristics of the instrument. Some obtained data are presented and possible interpretations of the data offered. The data would seem to indicate that instructional systems are not implemented fully in many schools. A second section of the paper presents correlational data obtained from the implementation instrument and variables that may be related to the degree of implementation of a classroom innovation. The innovative nature of the school, the climate established by the administrator, the existence of continuous training programs and staff and student attitudes were found to correlate significantly with degree of implementation.

Assessing the implementation of an innovative instructional system

With the increased use and availability of electronic computers, interest has been regenerated in item and choice weighting. A number of recent articles in confidence weighting (Shuford, Albert, & Massengill, 1966; Gannon & Sannito, 1968) and in Guttman choice weighting (Davis & Fifer, 1959; Stanley & Wang, 1968; Sabers & White, l969;Hendrickson, 1971) are evidence of this trend. In the days prior to computers, if any weighting of choices was done, it was typically the trichotomous system we know as the correction for guessing. The correction is of course familiar to many readers, but to clarify notation, what we mean by the correction for guessing is the use of the formula

The Correction for Guessing

Handle everyday research tasks with reliable, citation-backed results

Your personal Research Agent to handle research tasks with citation-backed results

Popular Tasks used by Researchers

How can I help with your research?

Meet SciSpace

Get more enhanced response by uploading the PDFs you want me to reference.

No relevant PDFs in your library

SciSpace is the AI research assistant for academics. Run systematic literature reviews on 280M+ papers, and write papers with cited sources. Trusted by 1M+ students, PhDs & researchers.

SciSpace | AI for Research

Analyze PDFs

Code & Manuscripts

Funding & Grants

Literature & Patents

Medical & Clinical Data

Systematic Review

Visualize & Present

Web & Data

Build a Google Scholar-like website for your research.

Build a website

Create charts and images for your research

Create a Chart

Write a paper for submission to a journal

Draft a manuscript

Patent Search

Design eye-catching scientific posters in minutes.

Scientific Poster Generation

Systematic Literature Review

One task is running at the moment. Your messages will be shown right after.

Drag and drop or click here to browse

Loved by <highlight>1 million+</highlight> researchers

Extract a list of specific topics and their sources from unstructured text

Topics

Compare and analyze relevant papers that matches with your search

Papers

Get insights from PDFs and bookmarked papers from your library

My library

Recent searches

Try searching for:

Catch AI-generated content in scholarly and non-scholarly content

{ai} Detector

Ai Writer

Get PDF Summaries, highlighted text explanations 

Chat with PDF

Effortlessly create in-text citations and bibliographies in APA and 2,500 other formats

Citation generator

Get explanations, summaries, and answers on academic papers

Ease up your research workflow with {scispace}'s cohort of exciting AI tools

Elevate your academic writing skills and convey your ideas the way you want

Paraphraser

Explore our range of reading and writing tools

Your file is being prepared and should be ready in a few minutes. If it's a large file, it might take a bit longer. You can close this window, and we'll email you the file when it's done.

You have reached a maximum limit of <strong>{limit}</strong> columns in the table. Remove at least <strong>1</strong> column to add or create another one.

William Evans

Author Tools

Chat about Author

Papers

An investigation of the cognitive correlates of test‐wiseness 1,2

Assessing the implementation of an innovative instructional system

The Correction for Guessing