The Remote Associates Test (RAT, CRA) is a classical creativity test used to measure creativity as a function of associative ability. The RAT has been administered in different languages. Nonetheless, because of how embedded in the language the test is, only a few items are directly translatable, and most of the time the RAT is created anew in each language. This process of manual (and in two cases computational) creation of RAT items is guided by the researchers’ understanding of the task. However, are the RAT items in different languages comparable? In this paper, different RAT stimuli datasets are analyzed qualitatively and quantitatively. Significant differences are observed between certain datasets in terms of solver performance. The potential sources of these differences are discussed, together with what this means for creativity psychometrics and computational vs. manual creation of stimuli.