What are we doing?
It's all about meaning! Here are brief descriptions of some of the areas we have been working on in recent years. Most, if not all, experimental and theoretical work bear on how the brain represents and processes meaning. We use a variety of experimental methods (eye-tracking, fMRI, psychophysical, behavioral), stimulus types (words, sentences, objects, faces, events), populations (neurotypical, Alzheimer's, individuals with aphasia), all aiming for convergence of results on how the brain makes sense of the world. See also Publications
The Nature of Compositionality: Investigating Semantic 'gaps' in Sentence Comprehension
How are sentences understood? And, more specifically, how do we attain the meaning of a sentence from its parts? At first, one might think that this simply requires understanding the sentence's individual words and how they combine. But oftentimes sentences have semantic 'gaps'. For instance, a sentence such as 'the boy finished the letter' might be understood as 'the boy finished [reading] the letter', while in reality the sentence could be about anything (even 'eating' the letter). Usually we understand these sentences in context, but some believe that regardless of context, there is a 'default' interpretation. We are investigating the default hypothesis (called 'coercion') using a variety of techniques and also advancing theoretical work aiming to understand the mechanisms responsible for filling the gap in--whether by contextual information (e.g., pragmatics) or whether by default semantic change ('coercion'). The illustration depicts partial results from our fMRI experiment showing right hemisphere involvement in sentences such as 'the boy finished the letter', suggesting that pragmatic processes are recruited in the interpretation of these sentences (see de Almeida, 2004), beyond their recruitment in other, fully specified sentences (such as 'the boy read the letter'). (See de Almeida & Riven, 2012, for discussion, and de Almeida, Riven, Manouilidou, Lungu, Dwivedi, Jarema, & Gillon, 2016, for a full set of results [link to article]). Two recent studies using a recognition memory paradigm and eye tracking investigate how these sentences are interpreted in context [link to article1] [link to article2]. Two other articles, one with Ernie Lepore (Rutgers University), and one with Caitlyn Antal, discuss theoretical issues related to this topic. [see Publications]
How are sentences understood? And, more specifically, how do we attain the meaning of a sentence from its parts? At first, one might think that this simply requires understanding the sentence's individual words and how they combine. But oftentimes sentences have semantic 'gaps'. For instance, a sentence such as 'the boy finished the letter' might be understood as 'the boy finished [reading] the letter', while in reality the sentence could be about anything (even 'eating' the letter). Usually we understand these sentences in context, but some believe that regardless of context, there is a 'default' interpretation. We are investigating the default hypothesis (called 'coercion') using a variety of techniques and also advancing theoretical work aiming to understand the mechanisms responsible for filling the gap in--whether by contextual information (e.g., pragmatics) or whether by default semantic change ('coercion'). The illustration depicts partial results from our fMRI experiment showing right hemisphere involvement in sentences such as 'the boy finished the letter', suggesting that pragmatic processes are recruited in the interpretation of these sentences (see de Almeida, 2004), beyond their recruitment in other, fully specified sentences (such as 'the boy read the letter'). (See de Almeida & Riven, 2012, for discussion, and de Almeida, Riven, Manouilidou, Lungu, Dwivedi, Jarema, & Gillon, 2016, for a full set of results [link to article]). Two recent studies using a recognition memory paradigm and eye tracking investigate how these sentences are interpreted in context [link to article1] [link to article2]. Two other articles, one with Ernie Lepore (Rutgers University), and one with Caitlyn Antal, discuss theoretical issues related to this topic. [see Publications]
The Nature of Concepts
Concepts are the most elementary mental representations bearing meaning. They enter into other higher cognitive processes such as thoughts (in fact, thoughts have concepts as their constituents just like sentences have words as their constituents). But how are concepts represented in the mind/brain? How do you represent, say, the concept DOG? Is it represented as a set of features (for instance, +barks, +animal, +domesticated, +furry)? How are concepts tied into categories? One line of investigation is on the nature of concepts that denote causation—such as the meaning of the verb to KILL. Some people think that verbs are represented in the brain by a "conceptual template" such as [[x ACT] CAUSE [y BECOME [dead]]] which contains primitive predicates (e.g., ACT, CAUSE, BECOME) and argument positions, x and y, specifying the kinds of elements the verb requires to form a grammatical sentence. We are investigating this hypothesis with causative verbs, employing different types of experimental techniques (memory for sentences, eye tracking) and populations (see our paper in Cognitive Neuropsychology). Another line of investigation on the nature of concepts involves a rapid presentation of words and pictures simultaneously (and, in one variation, separated by hemisphere) attempting to uncover how features contribute to the processing of object concepts. The figure shows part of the technique we developed to understand how and when object concepts are accessed in the brain. Results appeared in an article in the journal Cognitive Science [link to article]. A current discussion on the theory of conceptual atomism appears in several chapters of a recent book (de Almeida & Gleitman, 2018 [link to Oxford]) and in a recent chapter ( [.pdf] [book]).
Concepts are the most elementary mental representations bearing meaning. They enter into other higher cognitive processes such as thoughts (in fact, thoughts have concepts as their constituents just like sentences have words as their constituents). But how are concepts represented in the mind/brain? How do you represent, say, the concept DOG? Is it represented as a set of features (for instance, +barks, +animal, +domesticated, +furry)? How are concepts tied into categories? One line of investigation is on the nature of concepts that denote causation—such as the meaning of the verb to KILL. Some people think that verbs are represented in the brain by a "conceptual template" such as [[x ACT] CAUSE [y BECOME [dead]]] which contains primitive predicates (e.g., ACT, CAUSE, BECOME) and argument positions, x and y, specifying the kinds of elements the verb requires to form a grammatical sentence. We are investigating this hypothesis with causative verbs, employing different types of experimental techniques (memory for sentences, eye tracking) and populations (see our paper in Cognitive Neuropsychology). Another line of investigation on the nature of concepts involves a rapid presentation of words and pictures simultaneously (and, in one variation, separated by hemisphere) attempting to uncover how features contribute to the processing of object concepts. The figure shows part of the technique we developed to understand how and when object concepts are accessed in the brain. Results appeared in an article in the journal Cognitive Science [link to article]. A current discussion on the theory of conceptual atomism appears in several chapters of a recent book (de Almeida & Gleitman, 2018 [link to Oxford]) and in a recent chapter ( [.pdf] [book]).
The Language-Vision Interface
During the past couple of years we've been developing a series of studies on the interaction between linguistic and visual representations during language comprehension. These studies focus on how verb-semantic information may constrain the domain of visual attention to referents in the scene. Two theses (Di Nardo and Van de Velde) have explored this interface, both employing dynamic scenes and different verb classes. See our paper in Frontiers in Psychology (de Almeida, Di Nardo, Antal & von Grünau, 2019). Results from this study show that causative verbs yield faster—but not anticipatory—saccades (eye movements) to semantically related objects in the scene compared to verbs of perception. This suggests that information from vision and language interact quickly during naturalistic linguistic and dynamic scene processing, at a central conceptual system. The figure is a simplified version of the model we propose on how language and vision interact—via conceptual system. Current studies using a similar technique involve different types of linguistic and visual materials. [see our Materials page for samples]
During the past couple of years we've been developing a series of studies on the interaction between linguistic and visual representations during language comprehension. These studies focus on how verb-semantic information may constrain the domain of visual attention to referents in the scene. Two theses (Di Nardo and Van de Velde) have explored this interface, both employing dynamic scenes and different verb classes. See our paper in Frontiers in Psychology (de Almeida, Di Nardo, Antal & von Grünau, 2019). Results from this study show that causative verbs yield faster—but not anticipatory—saccades (eye movements) to semantically related objects in the scene compared to verbs of perception. This suggests that information from vision and language interact quickly during naturalistic linguistic and dynamic scene processing, at a central conceptual system. The figure is a simplified version of the model we propose on how language and vision interact—via conceptual system. Current studies using a similar technique involve different types of linguistic and visual materials. [see our Materials page for samples]
The Nature of Metaphors
Several studies with former graduate student Carlos Roncero focus on the representation and processing of metaphorical expressions (e.g, My lawyer is a shark). The studies employ different techniques (e.g., eye-tracking, priming, judgments, corpus data) and populations (healthy young adults, Alzheimer's patients). Metaphors are said to involve an "alternative meaning", one that goes beyond what is literally said. The current thinking is that because, say, lawyers cannot really be sharks, the interpretation of a word such as 'shark' involves the access to another meaning—sometimes even directly, without first accessing what it really means (viz., some kind of fish). In a recent eye-tracking study, we have shown that a metaphorical expression takes longer to process than an expression with the same basic constituents but conveying a (literal) similarity (such as My lawyer is like a shark). Our suggestion is that metaphors are not accessed directly qua metaphors but require the computation of the actual (literal) meaning. The graph shows partial results from one of our eye-tracking studies (Ashby, Roncero, de Almeida, & Agauas, 2018 [link to article]), with total reading time on the vehicle words ('shark'). A set of norms for metaphor and simile properties has also been published recently (Roncero & de Almeida, 2015 [link to article]). A recent study has investigated the interpretation of metaphors by Alzheimer's patients (Roncero & de Almeida, 2014 [link to article]). More recently, together with Laura Pissani, we have investigated how metaphors such as broken heart and sharp tongue are processed in sentence contexts; we found that although the metaphorical sense of broken heart can be quickly accessed, the literal meaning stays active for several seconds! Two articles have been published recently ([article 1], [article 2]). An article with our norming data, containing more than 300 of these expressions is in press in Behavior Research Methods [link soon].
Several studies with former graduate student Carlos Roncero focus on the representation and processing of metaphorical expressions (e.g, My lawyer is a shark). The studies employ different techniques (e.g., eye-tracking, priming, judgments, corpus data) and populations (healthy young adults, Alzheimer's patients). Metaphors are said to involve an "alternative meaning", one that goes beyond what is literally said. The current thinking is that because, say, lawyers cannot really be sharks, the interpretation of a word such as 'shark' involves the access to another meaning—sometimes even directly, without first accessing what it really means (viz., some kind of fish). In a recent eye-tracking study, we have shown that a metaphorical expression takes longer to process than an expression with the same basic constituents but conveying a (literal) similarity (such as My lawyer is like a shark). Our suggestion is that metaphors are not accessed directly qua metaphors but require the computation of the actual (literal) meaning. The graph shows partial results from one of our eye-tracking studies (Ashby, Roncero, de Almeida, & Agauas, 2018 [link to article]), with total reading time on the vehicle words ('shark'). A set of norms for metaphor and simile properties has also been published recently (Roncero & de Almeida, 2015 [link to article]). A recent study has investigated the interpretation of metaphors by Alzheimer's patients (Roncero & de Almeida, 2014 [link to article]). More recently, together with Laura Pissani, we have investigated how metaphors such as broken heart and sharp tongue are processed in sentence contexts; we found that although the metaphorical sense of broken heart can be quickly accessed, the literal meaning stays active for several seconds! Two articles have been published recently ([article 1], [article 2]). An article with our norming data, containing more than 300 of these expressions is in press in Behavior Research Methods [link soon].
Structure and Meaning in Word Recognition
How are words processed and represented in the brain? Regarding the structure of words, some people believe that we recognize and process words in full form while others believe that we recognize words by first breaking down their constituent morphemes (e.g., when you see a word such as remaking, you first decompose it into re, make, and ing). Together with Gary Libben (Brock University), we have been working on the nature of the lexical recognition and access codes employing different types of morphologically complex words. Our projects include experiments with inflected (e.g., barking) and compound (e.g., keyboard) words, as well as morphologically complex ambiguous words (e.g., undoable), and employ several experimental techniques (eye-tracking, priming, RSVP, etc.). A recent article shows that a word containing ambiguous root such as bark ('dog' or 'tree' related) access both its meanings in sentence contexts, even when the word is disambiguated by affixation (e.g., barking: 'dog' related), suggesting that lexical access is encapsulated, insensitive to context and even to morphological affixation. A recent articled appeared in the Journal of Experimental Psychology: Learning, Memory, and Cognition (open access). The figure depicts a recent technique we developed employing anaglyph glasses to understand the early moments of word recognition (LH: left hemisphere; RH: right hemisphere) via different visual pathways. In this technique, words are split into legal (FOOT+BALL — as shown) and illegal (FOO+TBALL) constituents to understand how the different hemispheres brain decodes early word constituents as well as how the word constituents are integrated [link to proceedings]. A full article has recently appeared also in the Journal of Experimental Psychology: LMC (open access).
How are words processed and represented in the brain? Regarding the structure of words, some people believe that we recognize and process words in full form while others believe that we recognize words by first breaking down their constituent morphemes (e.g., when you see a word such as remaking, you first decompose it into re, make, and ing). Together with Gary Libben (Brock University), we have been working on the nature of the lexical recognition and access codes employing different types of morphologically complex words. Our projects include experiments with inflected (e.g., barking) and compound (e.g., keyboard) words, as well as morphologically complex ambiguous words (e.g., undoable), and employ several experimental techniques (eye-tracking, priming, RSVP, etc.). A recent article shows that a word containing ambiguous root such as bark ('dog' or 'tree' related) access both its meanings in sentence contexts, even when the word is disambiguated by affixation (e.g., barking: 'dog' related), suggesting that lexical access is encapsulated, insensitive to context and even to morphological affixation. A recent articled appeared in the Journal of Experimental Psychology: Learning, Memory, and Cognition (open access). The figure depicts a recent technique we developed employing anaglyph glasses to understand the early moments of word recognition (LH: left hemisphere; RH: right hemisphere) via different visual pathways. In this technique, words are split into legal (FOOT+BALL — as shown) and illegal (FOO+TBALL) constituents to understand how the different hemispheres brain decodes early word constituents as well as how the word constituents are integrated [link to proceedings]. A full article has recently appeared also in the Journal of Experimental Psychology: LMC (open access).
Semantic Deficits in Alzheimer's Disease
Cases of category-specfic deficits (i.e., conceptual or semantic deficits caused by brain injuries or diseases) have been taken as evidence for the way concepts are represented and organized in the brain. Research on this topic has involved empirical investigation of verb-semantic deficits in individuals with Alzheimer's disease (Manouilidou, et al., 2009; Manouilidou & de Almeida, 2009); dissociation between verb classes employing dynamic scenes depicting different events and states (de Almeida, Mobayyen, Antal, Kehayia, Nair, & Schwartz, 2021 [proofs | .pdf]); and deficits in metaphor interpretation (Roncero & de Almeida, 2015; see above). The figure illustrates results from Manouilidou et al. (2009), showing that Alzheimer's patients have difficulty differentiating between two sub-classes of psychological verbs (e.g., fear, frighten), which don't have agents but experiencers (download .pdf ). A more recent study has shown that verb meaning in Alzheimer's can be impaired selectively, depending of verb category. See article in Cognitive Neuropsychology. Current related studies also investigate deficits in individuals with aphasia.
Cases of category-specfic deficits (i.e., conceptual or semantic deficits caused by brain injuries or diseases) have been taken as evidence for the way concepts are represented and organized in the brain. Research on this topic has involved empirical investigation of verb-semantic deficits in individuals with Alzheimer's disease (Manouilidou, et al., 2009; Manouilidou & de Almeida, 2009); dissociation between verb classes employing dynamic scenes depicting different events and states (de Almeida, Mobayyen, Antal, Kehayia, Nair, & Schwartz, 2021 [proofs | .pdf]); and deficits in metaphor interpretation (Roncero & de Almeida, 2015; see above). The figure illustrates results from Manouilidou et al. (2009), showing that Alzheimer's patients have difficulty differentiating between two sub-classes of psychological verbs (e.g., fear, frighten), which don't have agents but experiencers (download .pdf ). A more recent study has shown that verb meaning in Alzheimer's can be impaired selectively, depending of verb category. See article in Cognitive Neuropsychology. Current related studies also investigate deficits in individuals with aphasia.