Introduction

Working memory, the temporary maintenance and/or manipulation of information, is fundamental to various high-level cognitive processes and critical to producing goal-directed behavior. Simple daily tasks often require holding task-related information for seconds or minutes in working memory (WM) while concurrently processing new sensory input, and these two processes are known to interact with each other. Evidence for this interaction includes overlapping neural substrates for the processing and maintenance of visual information (e.g., Emrich et al., 2013; Harrison & Tong, 2009; Rademaker et al., 2019), and a bidirectional influence between WM content and a variety of perceptual tasks (e.g., Gayet et al., 2013; Olivers et al., 2006; Soto et al., 2005; Teng & Kravitz, 2019; Teng & Postle, 2021). Yet the temporal dynamics of this WM-perception interaction remain mostly unknown, despite the potential to facilitate an understanding of the interplay between these two fundamental cognitive functions.

When information is maintained in visual WM, it influences visual processing through enhancing bottom-up salience of a WM-matching stimulus (Gayet et al., 2013; Kiyonaga & Egner, 2016; Olivers et al., 2006), and distorts subjective perception in the form of an attractive or repulsive bias (Teng & Kravitz, 2019; Kang et al., 2011). Meanwhile, visual WM can also be disrupted by passive viewing of irrelevant distractors (Magnussen & Greenlee, 1992, 1999; Rademaker et al., 2015; Sun et al., 2017) or active visual processing in a secondary task (Fukuda et al., 2022; Teng & Kravitz, 2019; Teng & Postle, 2021; Teng et al., 2022; Saito et al., 2023). Consequences of distraction in WM are reduced precision (Rademaker et al., 2015; Sun et al., 2017; Bae & Luck, 2019) and/or presence of attractive (Rademaker et al., 2015; Teng & Kravitz, 2019) or repulsive (Bae & Luck, 2019) bias in memory recall. Further, the degree of WM-perception interaction depends on the similarity between WM content and visual stimuli in the feature space (Kiyonaga & Egner, 2016; Teng & Kravitz, 2019).

As WM is a dynamic process, information is maintained for only a brief duration of time, and then seemingly forgotten when it is no longer of use. One classic finding of forgetting in WM is the increase in error over time (Brown, 1958; Conrad, 1967; Cowan, 1988). Performance for color- and luminance-contrast WM drops rapidly with increasing interval size between encoding and probe (Cornelissen & Greenlee, 2000). Recognition accuracy for unfamiliar/unmeaningful shaped items held in WM also declined over longer intervals and eventually decreased completely, even when no distraction information was presented during the delay interval (Ricker & Cowan, 2010; Vergauwe et al., 2009). Types of forgetting errors include increased guess rate (Pertzov et al., 2017; Zhang & Luck, 2009), reduced precision (Rademaker et al., 2018; Shin et al., 2017), increased responses to non-target items (Pertzov et al., 2017), or target confusability (Schurgin et al., 2020). More recent insights come from network models that instead of a decline in representational strength, the increased variability in WM recall may be accounted for by diffusion (random drifts) in the encoded feature value over time (e.g., Panichello et al., 2019; Schneegans & Bays, 2018).

Beyond the behavioral, quantitative changes of WM representations over time (i.e., loss of information), there is also evidence to suggest that time may cause a qualitative change in the neural representation. On one hand, neuroimaging studies showed that information held in visual WM can be kept in visual cortex throughout the delay period up to 10 s in a code similar to that of perception (Albers et al., 2013; Harrison & Tong, 2009), even when facing passively presented distractors (Rademaker et al., 2019) or active perceptual discrimination (Hallenbeck et al., 2021). The recruitment of visual cortex for WM maintenance comes with a cost. Commensurate attractive biases in WM representations have been found in visual cortex (e.g., Hallenbeck et al., 2021; Lorenc et al., 2018). Yet on the other hand, there is evidence that under certain scenarios, WM representation may be transferred from a visual-like code into a more abstract code (e.g., verbal) to avoid interference. For example, the presence of visual distractors during a memory delay causes a decline in precision and an increase in categorical bias in orientation WM (Bae & Luck, 2019). In a different study, with increased stimulus onset asynchronicity (SOA) between memory encoding and the onset of visual search target, WM’s influence on visual search was found to be diminished (Han & Kim, 2009), which may be a result of increased cognitive control and/or a strategic usage of verbal code under the long SOA condition. Similar to these behavioral findings, physiological analyses proved that distractors caused a shift in the neural substrate recruited for maintaining orientation information from visual to parietal areas (Bettencourt & Xu, 2016), initiating a potential change in neural code for the representation in the parietal cortex (Rademaker et al., 2019). When a WM representation exists in different codes, it leads to unique consequences as result of the WM-perception interaction: when maintaining WM information in a verbal code was sufficient to complete the task, WM did not affect visual search performance; however, when the WM content must be maintained visually for successful future recognition, distractors matched with WM captured attention, causing interference (Olivers et al., 2006).

In the current study, we examined the time course of the interaction between visual WM and active ongoing perception, testing specific predictions of temporal decay and changes of representational code. The paradigm (adopted from Teng & Kravitz, 2019) consisted of a perceptual discrimination task embedded between WM encoding and report (Fig. 1). Participants memorized a low-level feature (color or orientation) and then made a same/different judgment on a set of two new features in the discrimination task. We previously found that the actively maintained information systematically altered the thresholds to reliably differentiate the two discrimination stimuli, and even task-irrelevant features of the discrimination stimuli created an attractive bias in WM report, providing support for the co-localization of perceptual processing and maintenance that lies at the heart of the sensorimotor recruitment framework. Here, we directly manipulated the ISI (inter-stimulus interval) between the memory cue and the discrimination to test the interaction between WM and perceptual processing at different stages of maintenance.

Fig. 1
figure 1

Measuring bidirectional interference between working memory and visual perception. a In Experiment 1a to 1c, participants performed a secondary color/orientation same/different discrimination task while holding a color/orientation in mind. The inter-stimulus interval (ISI) between the memory stimulus and the onset of the discrimination task was set to either 100, 550, or 1,000 ms. b The relationship between the memory color and the discrimination stimuli was manipulated. In the Middle condition, the memory color was between the two discrimination colors in the color space, drawing both towards itself and reducing the perceived distance between them. In the Side condition, the gap between the memory color and the nearby discrimination color was set to be 7° in color space. It was predicted that memory color attracts the nearby discrimination color to a greater extent than the other, increasing the perceived distance between them (adapted from Teng & Kravitz, 2019). The relationship in orientation between the memory sample and the discrimination stimuli was manipulated simultaneously in the same fashion (Middle and Side in orientation space)

Full size image

It is predicted that based on the temporal decay + visual code account, WM representation would be maintained in a visual format and persistently interact with perception through time. Thus, over short delays, the visual representation of the maintained information should be at its most stable and precise, exerting a strong impact on ongoing visual processing while being resistant to perceptual interference from concurrent sensory input (Fig. 2). With longer delays and decreased fidelity, the influence of the maintained information on perception would decrease and, critically, the memory representation also becomes more susceptible to perceptual interference. In contrast, based on the temporal decay + abstract code account, the visual representation degrades over time and then is shifted into an abstract/categorical code. Thus, the WM-perception interaction would only be observed at short ISIs but not for longer ISIs when the code changes, since WM maintained through verbal code does not influence visual processing (Olivers et al., 2006). The two accounts have two key distinctions in their predictions:

  1. 1)

    Previous research has shown that the interference between WM and external visual processing is domain-specific (Bae & Luck, 2019; Lorenc et al., 2021). Thus, the interference should be more severe when the two share the same code (both visual) compared to different codes (one visual, one abstract/verbal). Under the visual code account, both the perceptual threshold shift and the WM recall bias would persist even at longer delays because of the continued use of shared resources, whereas the abstract code account would predict a drastic drop of both effects at longer delay when WM and visual perception no longer share the same representational code.

  2. 2)

    While the visual code becomes more easily interfered with over long delays, the abstract code protects WM representations from interference. With the visual code deteriorating over time, visual input (the discrimination stimuli) entering the same system would be more dominating and cast greater interference on WM. A network model of WM (Bouchacourt & Buschman, 2019) showed that overlap of connections causes mutual interference among multiple WM items stored in a sensory network with recurrent connections to a random layer, and importantly, as memory degrades over time, WM representations drift towards each other to a greater extent. The increased attractive bias may serve as a mechanism to counteract the increasing noise towards a stable representation (Bouchacourt & Buschman, 2019; Chunharas et al., 2022). Empirical evidence also showed that serial dependence bias increases with longer delay (Bliss et al., 2017) and that higher noise leads to increasing attractive bias towards previously seen stimuli (e.g., Gallagher & Benton, 2022; van Bergen & Jehee, 2019). In comparison, the transfer of WM representation to a different coding scheme is hypothesized to increase distractor resistance in WM (e.g., Lorenc et al., 2021; Xu, 2017, 2020). Support for this prediction comes from evidence that WM representation in the parietal cortex was found to be unbiased by task-irrelevant distractors (Bettencourt & Xu, 2016; Lorenc et al., 2018) together with unimpaired behavior (Bettencourt & Xu, 2016). Thus, we predict a decrease of recall bias, rather than an increase of bias under the abstract code account.

Fig. 2
figure 2

Qualitative depictions of the visual code account versus the abstract/verbal code account are visualized here within the framing of the given experiment

Full size image

To foreshadow, we verified the predicted temporal changes of the temporal decay + visual code account in the context of color perception and memory, showing that (1) task-relevant content in WM persistently changed early color perception; (2) as the ISI increased, the memory representation decayed and was more prone to perceptual interference; and (3) task-irrelevant content in WM had a brief impact on perception but then decayed quickly after encoding. These results constitute further behavioral evidence for a temporal decay + visual code account, while additionally providing critical insight into the temporal dynamics of WM maintenance.

Experiment 1

Experiment 1a deployed a dual-task paradigm to capture the influence of WM on perception as well as the interference upon WM from concurrent perceptual processing. Participants maintained the color of a colored Gabor patch in mind while performing a color discrimination task on a set of two new colored Gabor patches (Fig. 1). The relationship between the maintained item and the discrimination stimuli was manipulated. In one condition (“Middle”), the maintained item was between the two discrimination stimuli in color space (Fig. 3b). We predicted that the perception of both discrimination stimuli will be equally shifted toward the maintained item, making it more difficult to tell the two apart and resulting in a higher discrimination threshold. In the other condition (“Side”), the maintained item was to one side of the two discrimination stimuli in color space. We previously found that distractors more similar to the WM representation caused more attentional capture and biased the WM recall to a greater extent than less similar ones (Teng & Kravitz, 2019; also see Kiyonaga & Egner, 2016). Therefore, we predict that WM should shift the perception of the closer discrimination stimuli more strongly, widening the gap between them and leading to a lower discrimination threshold. The magnitude of this threshold shift between Middle and Side conditions therefore reflects the influence of WM on perception. Concurrently, the memory representation should be interfered with by the discrimination stimuli. In the Side condition, both discrimination stimuli attract memory in the same direction in color space and cause a bias in the WM report, whereas in the Middle condition, the two discrimination stimuli would have opposite effects on the memory representation and cancel out the memory bias (see also Teng & Kravitz, 2019, for further explanation).

Fig. 3
figure 3

Testing the temporal dynamics of working memory. a Color discrimination performance in Experiment 1a plotted as a function of color relationship and the inter-stimulus interval (ISI). For all three ISIs, there was a significantly lower threshold for the Side condition than the Middle condition. b Bias in the memory report showed a greater bias in the Side condition than the Middle condition. c The difference in color threshold (Middle minus Side condition) and memory bias (Side minus Middle condition) plotted as a function of time. As threshold shift decreased from 100 ms to 550 and 1,000 ms, memory bias increased. These results are consistent with the decay of a sensory representation of working memory over time (sensorimotor recruitment) that predicts a decreased impact on perception and increased susceptibility to perceptual interference. * indicates p < .05 for pairwise comparisons. Error bars represent standard error

Full size image

To examine the temporal dynamics of this interaction, the ISI between the WM sample and the discrimination task was manipulated. The temporal decay + visual code account predicts that the maintained information will remain in its visual format throughout the delay period, decaying over time. Thus, the threshold shift between Middle and Side should be robust over short delays and reduced at longer delays. Further, the weakened WM representation would become more susceptible to incoming interference from the discrimination stimuli of the same code, and as a result, bias in the memory report should increase with time.

Experiments 1b and 1c served as control conditions to rule out alternative accounts. Experiment 1b tested whether the threshold shift could result from passive priming, as despite the irrelevance of these features, they are likely to cause short-lived perceptual effects, disappearing immediately without maintenance. Here, participants maintained the orientation rather than color of the Gabor, while performing a color discrimination task (Fig. 4a). Based on findings in Teng and Kravitz (2019), we predicted that the now task-irrelevant color information from the encoding stimulus would have only a brief transient effect on color discrimination, verifying the need for a maintained visual code in order to elicit the discrimination effect, as a transfer to an abstract code would create no interference whatsoever.

Fig. 4
figure 4

Task relevance differently impacts the interaction. a Discrimination performance in Experiment 1b where participants actively memorized orientation and performed the color discrimination task. When color of the memory sample was irrelevant, it briefly impacted color discrimination at 100-ms inter-stimulus interval (ISI), but not for longer delays. b Comparing the time courses of threshold shift caused by task-relevant (Exp. 1a) and task-irrelevant (Exp. 1b) information in working memory. While task-relevant information persistently changes color perception, task-irrelevant information had only a transient impact. c Bias in working memory report in Experiment 1c where participants actively memorized color and performed the orientation discrimination task. Although the colors of the two discrimination stimuli were task irrelevant, they caused a consistent change in the memory representation. d The time courses of memory bias caused by task-relevant (Exp. 1a) and task-irrelevant (Exp. 1c) information in the discrimination task. * indicates p < .05 for pairwise comparisons. Error bars represent standard error

Full size image

In Experiment 1c we examined whether the memory distortion bias requires active processing of the discrimination stimuli. Participants maintained color while performing an orientation discrimination (Fig. 4c). The visual code account predicts that the memory bias will occur even for the now task-irrelevant color of the discrimination stimuli because even passive perception of the stimuli engages the same perceptual modality in which the WM representation is maintained. Alternatively, a shift of code for maintained information would predict no such interference with low-level perception. Therefore, although the color of the discrimination stimuli is task-irrelevant, we still predict a significant bias in the Side condition.

Method

Participants

An a priori power analysis was conducted based on the smallest effect size reported in Teng and Kravitz (2019) using G*Power (Faul et al., 2009). To achieve 95% power and a two-tailed alpha of 0.05, the projected sample size was 45 participants to detect a Cohen’s d of 0.55 reported in the study. Thus, we decided to recruit 50 participants for each of the three experiments to be capable of detecting the predicted effects. Three separate sets of 50 Amazon Mechanical Turk workers (Exp. 1a: average age of 40.0 ± 9.5 years; 23 females; Exp. 1b: average age of 38.17 ± 12.28 years; 26 females; Exp. 1c: average age of 35.8 ± 7.1 years; 26 females) were recruited with compensation of $8 per hour for each of the three experiments. Data from participants who did not perform the discrimination task adequately (reached fewer than six reversals for any of the ISI conditions) were removed from further analysis. Further, we excluded data from participants whose absolute mean error in WM recall exceeded 90°. Participants gave informed consent approved by the Institutional Review Board of George Washington University and had normal color vision and normal or corrected-to-normal visual acuity.

Stimuli and procedure

Experiment 1a

A dual-task paradigm was used to test how information maintained in WM interferes with and is interfered with by an ongoing visual-processing task. While participants were memorizing the color of a Gabor patch, they performed a secondary color discrimination task. Each trial began with a colored Gabor patch (diameter of 125 pixels with five cycles) presented for 500 ms (Fig. 1a; memory sample). Subsequently, two new Gabors (diameter of 125 pixels with five cycles) were presented for 500 ms and participants made a judgment whether they were the same or a different in color by making a button press. The interval between the memory sample and the discrimination was set to be 100, 550, or 1,000 ms, leading to the three ISI conditions. A color wheel showed up 1,000 ms after the participants responded to the discrimination task, and they reported the memorized color by clicking on the wheel. The wheel was rotated randomly to avoid spatial coding of the memory color.

There were two key manipulations. First, we manipulated the relationship between the memory sample color and the discrimination colors with the Middle and Side conditions (Fig. 1b). The memory sample was set to be the center of the two discrimination colors in the Middle condition or on one side of the discrimination colors in the Side condition in the CIEL*a*b* color space (centered at L* of 70, a* of 20 and b* of 38, with a radius of 60). The distance between the two discrimination colors was determined by a staircase procedure (described in detail below). In the Side condition, the gap between the memory color and the close-by discrimination color was fixed to 9°. Second, we manipulated the ISI (100, 550, or 1,000 ms) between the memory cue and the discrimination stimuli to map out the time course of the interference effects between the memory cue and the discrimination task. This manipulation led to a two (Middle or Side) by three (ISI: 100, 550, or 1,000 ms) design. Trials for these six conditions were presented intermixed and participants completed 216 test trials in total (36 of each condition; the number of trials per condition was determined based on pilot testing and the data from the pilot was not included in our final dataset). To make the mechanics of the step function less apparent, 30 additional filler trials were added where the two discrimination stimuli had completely the same color and those trials were randomly placed in the experiment. These 30 trials served as the foil and were not part of the threshold calculation.

We used a staircase procedure separately for each of the six conditions to derive the discrimination threshold at which participants could reliably tell the two colors apart. The difference in color between the two stimuli was determined with a one-down, one-up stair-case method before the first reversal and then switched to a two-down, one-up to increase the efficiency in deriving the threshold. The color difference started at 18° in the color space. The size of the step (either decrease or increase) was determined as such: 3° for the first two reversals, 2° for the third to six reversals, and 1° for all the following trials. The upper limit for the color difference was 30° and the lower limit was 1°. It should be noted that we opted for a staircase procedure rather than the psychometric curve method in Teng and Kravitz (2019) because staircase is more efficient in estimating the threshold considering the additional conditions in the current study. Given the differences between the two measures, it is thus crucial to compare them side by side. For this reason, we included the same 1,000-ms ISI condition as a replication for the original study.

Experiments 1b and 1c

The displays for these two experiments were exactly the same as Experiment 1a except for the task instructions. In Experiment 1b, participants were told to memorize the orientation of the first Gabor and then perform a discrimination task on the colors of the following two Gabor patches (Fig. 3a). The color of the memory Gabor was task-irrelevant and was set to be in the center or to one side of the discrimination stimuli. The parameters of the color discrimination task were exactly the same as in Experiment 1a. The relationships between the memory sample and the discrimination stimuli in orientation and color space were manipulated simultaneously and in opposite directions: when the relationship in color was Middle, the relationship in orientation would be Side, and vice versa.

In Experiment 1c, the task dimensions were flipped so that participants memorized the color of the Gabor and then performed the discrimination task on the orientation of the two Gabor patches (Fig. 1a). The orientation of the memory Gabor could be in the center or to one side to the orientations of the discrimination stimuli. The color relationship was also manipulated and in the opposite direction to the orientation relationship. The staircase procedure described above was applied to the orientation difference between the two discrimination stimuli, which started at an 18° difference, with an upper bound of 30° and a lower bound of 1°.

Data analysis

Discrimination threshold

We followed the procedures in Leek (2001) to calculate the threshold independently for each condition. We removed trials before the second reversal and the threshold was taken as the average of color difference of all the rest trials. A reversal was defined as the point at which the staircase changes direction (either an incorrect response following more than two correct trials, or the second correct trial in a string of “wrong-correct-correct” trials).

Recall bias

We used a model-free approach to calculate bias in WM recall. For the Middle condition, bias was calculated as the mean error (signed) across trials. A positive bias means a general tendency of clockwise bias and a negative bias means a tendency of counterclockwise bias. For the side condition, we first sort trials by whether the discrimination stimuli were clockwise or counterclockwise to the memory color and then flip the counterclockwise trials and collapse them with the clockwise ones. Then we calculate the mean signed bias and a clockwise bias is deemed to be attractive towards the discrimination stimuli whereas a counterclockwise bias is deemed to be repulsive away from them.

Statistical analyses were performed using computer software JASP (JASP Team, 2023. Version 0.17.2). All multiple comparisons are Bonferroni–Holm corrected.

Results

Experiment 1a

As predicted, the discrimination threshold was affected by the memorized color (Fig. 3a). A two-way repeated-measures ANOVA with relationship (Middle, Side) and ISI (100, 550, and 1,000 ms) as factors revealed a significant main effect of relationship, with lower thresholds for the Side than Middle condition (F(1, 49) = 38.47, p < 0.001, ηp2 = 0.44). The main effect of ISI was not significant (F(2, 98) = 0.43, p = 0.65, ηp2 = 0.01), suggesting roughly equivalent discrimination performance across ISIs. Critically, the interaction between relationship and ISI was significant (F(2, 98) = 3.62, p = 0.03, ηp2 = 0.07), indicating that the difference in threshold was modulated by ISI. Pairwise comparisons showed a greater difference in threshold in the 100-ms compared with the 550-ms condition (t(49) = 2.82, p = 0.021, Cohen’s d = 0.40). The threshold shift in all three ISI conditions was greater than zero (all ts >2.29, ps < 0.049, Cohen’s ds < 0.32). These results are consistent with decreasing interference from the maintained content on ongoing perception as predicted by the temporal decay + visual code account.

In contrast to the threshold effect, bias in the memory report increased with time (Fig. 3b), consistent with the temporal decay + visual code account. A two-way repeated-measures ANOVA revealed greater bias in the Side condition than the Middle condition (F(1, 49) = 108.36, p < 0.001, ηp2 = 0.69). The main effect of ISI was also significant (F(2, 98) = 4.39, p = 0.02, ηp2 = 0.08), with larger biases at the longer than shorter ISIs. Further, there was a significant interaction between relationship and ISI (F(2, 98) = 3.27, p = 0.04, ηp2 = 0.06). Pairwise comparisons showed that the difference in bias between Middle and Side conditions was greater in the 1,000-ms condition than the 100-ms condition (t(49) = 3.28, p = 0.006, Cohen’s d = 0.46; all other ps > 0.43, Cohen’s ds > 0.15). The difference in memory bias in all three ISI conditions was greater than zero (all ts > 6.98, ps < 0.001, Cohen’s ds > 0.99). These results show that with a longer duration the memory representation weakened and was more likely to be affected by additional matching perceptual input. This result (summarized in Fig. 3c) suggests a long-lasting interaction between WM and ongoing perception and supports the predictions of a maintained visual format.

Experiment 1b

When color was task-irrelevant for WM, we still observed a significant main effect of color relationship (F(1, 49) = 9.25, p = 0.004, ηp2 = 0.16) in the discrimination threshold, as the threshold in the Side condition was lower than the Middle condition. This difference in threshold was further modulated by time (F(2, 98) = 4.02, p = 0.02, ηp2 = 0.08). Pairwise comparisons showed that the difference between Middle and Side conditions was greater in the 100-ms condition than in the 550-ms and 1,000-ms conditions (t(49) = 2.12, p = 0.04, Cohen’s d = 0.30; t(49) = 2.56, p = 0.01, Cohen’s d = 0.36, respectively). Importantly, the difference between Middle and Side conditions was only significant- for the 100-ms ISI (t(49) = 4.08, p < 0.001, Cohen’s d = 0.58), but not for 500 ms and 1,000 ms, (both p > 0.13, Cohen’s d < 0.22), demonstrating the transience of the effect (Fig. 4a).

To directly compare the effect of task-relevant and -irrelevant information on visual discrimination, a three-way mixed ANOVA was conducted for the combined data with task relevancy (relevant in Exp. 1a and irrelevant Exp. 1b) as a between-subject factor, and relationship and time as the within-subject factors. We found a significant main effect of relationship, as the threshold was lower in the Side condition than in the Middle condition (F(1, 98) = 44.48, p < 0.001, ηp2 = 0.31). Further, this difference between Middle and Side was modulated by task relevancy (F(2, 196) = 7.07, p = 0.01, ηp2 = 0.07), with a greater effect in the task-relevant condition (Exp. 1a) than the task-irrelevant condition (Exp. 1b). Time interacted with Middle versus Side (F(2, 196) = 7.29, p < 0.001, ηp2 = 0.07), such that the threshold difference was greater in the 100-ms condition than in the 550-ms and 1,000ms conditions (t(99) = 3.19, p = 0.002, Cohen’s d = 0.33; t(99) = 3.10, p = 0.003, Cohen’s d = 0.35). None of the other main effects or interactions were significant, (all ps > 0.35, ηp2 < 0.01). Therefore, consistent with our previous experiment, task relevancy modulated the effect on the difference in threshold (Fig. 4b).

For the memory report in Experiment 1b, as the memory dimension was orientation, the analysis was based on the orientation relationship between the memory sample and the discrimination stimuli. A two-way repeated ANOVA revealed a main effect of relationship that there was a significantly greater bias in the Side condition than the Middle condition, F(1, 49) = 10.84, p = 0.002, ηp2 = 0.18. The duration of ISI did not modulate the magnitude of bias and did not interact with the relationship. Thus, despite the general difference in bias between Side and Middle condition, its magnitude did not vary with time.

Experiment 1c

To examine how task-irrelevant color information during the orientation discrimination task influences color WM, we focused the analysis on the color memory report. There was a main effect of relationship that the bias in the Side condition was significantly greater than the Middle condition (F(1, 49) = 71.87, p < 0.001, ηp2 = 0.59). The main effect of time and the interaction were not significant (ps > 0.41, both ηp2 < 0.02). Therefore, regardless of task relevance, perceiving additional color input caused bias in the memory representation (Fig. 4c).

We then conducted a three-way mixed ANOVA for the combined data with task relevancy (relevant in Exp. 1a vs. irrelevant Exp. 1c) as a between-subject factor, and relationship and time as the within-subject factors. We found a significant main effect of relationship that the attraction bias was greater in the Side condition than the Middle condition (F(1, 98) = 176.29, p < 0.001, ηp2 = 0.64). Interestingly, the difference between Middle and Side was not modulated by task relevance (F(1, 98) = 0.55, p = 0.47, ηp2 = 0.02). None of the other main effects or interacts were significant (ps > 0.07, all ηp2 < 0.03; Fig. 4d).

For the threshold performance in the orientation discrimination task, there was a significant main effect of time (F(1, 49) = 6.97, p = 0.001, ηp2 = 0.13) that the threshold in the 100-ms condition was higher than the 550-ms and 1,000-ms conditions (t(49) = 3.03, p = 0.004, Cohen’s d = 0.43; t(49) = 2.95, p = 0.01, Cohen’s d = 0.42). The main effect of side was not significant (F(1, 49) = 3.05, p = 0.09, ηp2 = 0.06). The duration of ISI modulated the difference between the Middle and Side conditions (F(2, 98) = 4.22, p = 0.02, ηp2 = 0.08). Pairwise comparisons showed that the difference between the Middle and Side conditions were greater in the 100-ms condition than in the 1,000-ms condition (t(49) = 2.878, p = 0.01, Cohen’s d = 0.41). Further, the difference between the Middle and Side conditions was only significant for 100 ms ISI (t(49) = 2.71, p = 0.01, Cohen’s d = 0.38), but not for 500 ms and 1,000 ms (ps > 0.19, Cohen’s ds < 0.23), indicating that perceiving an orientation briefly altered the orientation discrimination threshold, consistent with the results in Experiment 1b (OSM Fig. 1b).

Discussion

Experiment 1a tested the temporal dynamics of the interaction between WM and perception. We found that the effect of WM upon perception decreased in its strength but remained robust across time, and as for the effect of perception upon WM, attraction bias in the memory report increased with time. Additionally, we observed a significant bidirectional interference confirming validity of the paradigm (Teng & Kravitz, 2019). These results indicate a decrease in stability in the memory representation over time so that its effect on perception becomes weaker and it is more susceptible to interference from additional perceptual input. They further suggest that information is maintained in a sensory code that persistently interacts with perception. Alternatively, if WM maintenance relies on an abstract code distinct from visual processing, this interaction should decrease in both directions over time, contradicted by the increase in memory bias we observed here.

The two control experiments, Experiments 1b and 1c, provided further insights by utilizing task-irrelevant discrimination stimuli with the goal of ruling out alternative interpretations for results in Experiment 1a that the observed effect could simply be attributed to factors such as priming or confusability. Experiment 1b demonstrated that irrelevant information did impact ongoing perception, but to a smaller extent. The influence was only robust at 100 ms but disappeared at 550 ms, suggesting that it is a transient effect. In addition, this finding speaks to previous studies that did not find WM’s influence on visual process (e.g., Bloem et al., 2018): if actively maintained memory dimension did not match that of the visual task, the irrelevant information in WM might decay more quickly than the actively maintained memory dimension and thus only had a brief effect on perception that disappeared with longer delay.

Experiment 1c, similarly to Experiment 1b, investigated the effect of task-irrelevant stimuli; however, Experiment 1c investigated whether or not passive perception could impact WM representation. It was found that attraction bias still existed throughout the time course, despite remaining unchanged, potentially due to the difficulty of the task. Since the interaction between task relevance and time did not reach statistical significance, we do not further speculate on the numerically different time courses between Experiments 1a and 1c (Fig. 4d). This result was congruent with previous findings (Teng & Kravitz, 2019) and provides additional support for the account that incoming sensory distraction alters the memory representation regardless of the level of processing and the amount of attention. These results provide further evidence against a separation in code between WM and visual perception because a separation would predict that irrelevant sensory input is less likely to influence WM.

Experiment 2

Method

Experiment 1 found evidence for the visual code account in the time course of the interaction between ongoing perception and maintained information shortly after encoding. Experiment 2 aimed to expand the quantification of the time course by including additional brief delays and delays closer to those found in neuroimaging studies (e.g., Harrison & Tong, 2009; Riggall & Postle, 2012; Serences et al., 2009). In Experiment 1a, the threshold shift decreased from 100 ms to 550 ms and remained stable for 1,000 ms. We thus added brief delays to quantify the changes between 100 ms and 550 ms. Moreover, as the time scales usually differ between different methods (e.g., neuroimaging studies typically rely on seconds of long delays), it is important to test that predictions of the visual code account hold at longer delays.

The paradigm was exactly the same as Experiment 1a, except for the ISIs between the memory cue and the discrimination task (Experiment 2a: 200, 316, 433, 550 ms; Experiment 2b: 2, 3, 4 s). Importantly, the purpose of these experiments is to provide (1) a quantification of the time course and (2) evidence that the interference effects are consistently present throughout delays from 100 ms to 4 s. We recruited two separate groups of Mechanical Turk workers (50 each) to complete the experiments. In Experiment 2a, the average age was 40.1 ± 12.8 years and there were 26 females. The average age in Experiment 2b was 37.3 ± 10.5 years and there were 31 females. Participants gave informed consent and reported normal color vision and normal or corrected-to-normal visual acuity.

Results

In Experiment 2a, using a repeated-measures ANOVA with relationship and ISI (200, 316, 433, 550 ms) as factors, we replicated the impact of maintained content on discrimination thresholds with a main effect of relationship (F(1, 49) = 30.86, p < 0.001, ηp2 = 0.39). No other main effects or interactions reached significance (all ps > 0.49, ηp2 < 0.02). An analogous ANOVA on bias revealed a significant main effect of relationship (F(1, 49) = 118.63, p < 0.001, ηp2 = 0.71). There was also a significant interaction between time and relationship (F(3, 147) = 2.90, p = 0.04, ηp2 = 0.06). Pairwise comparisons showed a trend of greater bias in the 316-ms condition than the 200-ms condition (t(49) = 2.52, p = 0.09, Cohen’s d = 0.43). For all these ISIs, we compared the threshold shift and difference in bias against zero and confirmed that they were all greater than zero (all ts > 2.42, ps < 0.03, Cohen’s ds > 0.34).

In Experiment 2b, a repeated-measures ANOVA on the discrimination thresholds with relationship and time (2, 3, 4s) revealed a significant main effect of relationship (F(1, 49) = 19.85, p < 0.001, ηp2 = 0.29). No other main effects or interactions reached significance (all ps > 0.27, ηp2 < 0.03). An analogous ANOVA on bias also showed main effects of relationship (F(1, 49) = 118.25, p < 0.001, ηp2 = 0.71) and time (F(2, 98) = 3.66, p = 0.03, ηp2 = 0.07). The interaction did not reach significance (F(2, 98)=2.49, p = 0.09, ηp2 = 0.04). For all these ISIs, the threshold shift and difference in bias were all greater than zero (all ts > 2.14, ps < 0.04, Cohen’s ds > 0.30).

Figure 5a and b provide a further quantification of the time course of the bidirectional interaction between working memory and visual perception, showing that both threshold and bias effects persisted at extended ISIs. In total, these results confirmed that WM content still maintained a bidirectional interaction with ongoing perceptual processing up to 4 s after encoding, inconsistent with a framework that suggests a shift of code. Moreover, the bias in the memory report actually increased over time, suggesting that the strength of the maintained information reduced and was more likely to be interfered with by incoming sensory input.

Fig. 5
figure 5

Time course of the interaction between working memory and ongoing visual perception. a The influence of working memory upon perception (reflected in the difference in threshold between middle and side conditions) and the influence of ongoing perception upon working memory (reflected in the memory bias caused by the discrimination stimuli) at inter-stimulus intervals (ISIs) of 200 ms, 316 ms, 463 ms, and 550 ms. Thresholds decrease consistently and rapidly over time following encoding. Similarly, bias increased immediately following the encoding period and remained throughout longer delays. b At longer ISIs (2,000 ms, 3,000 ms, and 4,000 ms), both threshold and bias effects not only persisted, but continued to increase and decrease, respectively. For all these time points, the threshold shift and memory bias were all greater than zero (*, p < .05; **, p < .01). Error bars represent standard error

Full size image

General discussion

In this set of experiments, we tested the temporal dynamics of visual WM representations over a wide range of times (from 0.1 to 4 s), by examining its interaction with ongoing visual perception. If the bidirectional interaction ceased to exist beyond the shorter intervals, evidence to suggest the temporal decay + abstract code account would hold weight. However, we found that the bidirectional interactions between the two processes persisted even with long intervals between memory encoding and the onset of the perceptual task, suggesting the temporal decay + visual code account. Additionally, for shorter intervals, changes in threshold shifts and bias went opposite directions: the influence of WM on perception (threshold shift) reduced with time, while the interference from perception on WM representation (bias) increased over time. This overall pattern is strikingly consistent with the temporal decay + visual code account that (1) the strength of the WM representation reduces over time and as a result, its influence on perceptual processing is weaker and becomes more susceptible to perceptual interference, and (2) we did not observe a sudden drop in the bidirectional interactions, suggesting that, at least in the context of this task, visual WM representation remains in a visual code and continually interacts with perception.

Previous studies have reported mixed results regarding the temporal decay in WM (Magnussen et al., 1990; Magnussen & Greenlee, 1992) and whether WM representations are robust against perceptual interference (Hallenbeck et al., 2021). One explanation for such discrepancy could be the flexible and strategic utilization of different codes of WM to optimize performance under variable task demands. The choice of code may be task-dependent, influenced by factors such as complexity and sensitivity of memory report method (e.g., change detection or continuous recall; whether the report requires fine perceptual report), salience of perceptual distractors, and memory load. Different neural codes of WM representations (e.g., sensory, categorical) have been demonstrated in a number of studies that there could be coexisting sensory representations and abstract, categorical representations along the visual hierarchy (Christophel et al., 2017; Kwak & Curtis, 2022; Rademaker et al., 2019). With the same memorandum, subjects relied on the abstract code in frontoparietal regions or the sensory code in visual processing regions depending on whether the WM report tested categorical information or fine visual detail, respectively (Lee et al., 2013). Thus, our current findings do not deny the possibility of recruiting an abstract/verbal code under certain task demands. However, our results do suggest that recoding of the visual WM representation into an abstract code is not obligatory: even when participants knew there was going to be a secondary task during the delay, making it optimal to proactively recruit the abstract code for the WM content, we still observed a persistent low-level interaction. It would be of interest for future studies to explore the factors that determine the format that WM representations are maintained in, and the cost and benefit of flexibly switching among different codes. In addition, continuous stimulus-specific information of features and objects in WM has been reported in frontal, parietal, and occipital regions in human (e.g., Cai et al., 2019; Christophel et al., 2017; Yu & Shim, 2017; and animal research (e.g., Buschman et al., 2011; Meyer et al., 2011) and one important future direction would be to determine the functional roles of these co-existing representations.

Regarding the mutual bias between WM and perception, the sources of attractive and repulsive biases and their functional relevance remains unclear. Using the dual-task paradigm in our current study and that of Teng and Kravitz (2019), we have observed attractive bias in all the experiments. However, both directions of bias have been observed in a wide literature of visual WM, visual perception, and WM-perception interaction across a variety of stimulus types and task structures (e.g., Bae & Luck, 2017; Brady & Alvarez, 2011; Chunharas et al., 2022; Fischer & Whitney, 2014; Kiyonaga et al., 2017). Functionally, attractive bias is proposed to stabilize visual perception/WM by integrating successive or simultaneously presented stimuli to counteract uncertainty arising from largest set sizes or noise accumulated across time, while repulsive bias is thought to differentiate similar visual or WM representations to increase their discriminability against each other (Chunharas et al., 2022; Czoschke et al., 2019; Kiyonaga et al., 2017). The two types of bias may reflect different computations within the visual system (Fritsche et al., 2020; Sheehan & Serences, 2022) and are sensitive to task demands (Bae & Luck, 2020; Teng et al., 2022). Attractive bias has been observed in both early sensory (Hallenbeck et al., 2021; Lorenc et al., 2018; Goettker & Stewart, 2022; John-Saaltink et al., 2016) and post-perceptual processing stages (Sheehan & Serences, 2022), whereas repulsive bias is mostly linked to low-level visual adaptation (Sheehan & Serences, 2022). One outstanding question is whether these two types of bias originate from perceptual or post-perceptual levels and how task demands may shape the direction of bias. Future studies may also investigate whether similar neural mechanisms subserve bias observed across different domains (i.e., visual perception and visual WM).

When encoding a specific feature of an item into WM, do task-irrelevant features of the same item exert an influence on sensory processing? This question touches upon an ongoing inquiry of whether task-irrelevant features are automatically encoded into WM alongside the task-relevant ones, as evidenced by some studies (Marshall & Bays, 2013). However, others suggest that task-irrelevant features may be encoded with lower fidelity and the process may not be fully automatic (e.g., Fougnie & Alvarez, 2011; Shin & Ma, 2016; Tam & Wyble, 2023). In the present study, results from Experiment 1b demonstrated that task-irrelevant features did influence perception shortly after encoding (100 ms after the offset of memory sample) but this effect waned over extended periods (550 ms and beyond). It is likely that the task-irrelevant feature is initially encoded into WM automatically and is subsequently discarded – either through a selectively removal process or due to withdrawn of attention. Nevertheless, because our study did not include a backward mask after the memory sample, the effect at 100 ms might equally be a reflection of perceptual priming, with no involvement of WM. Broadly speaking, these results are consistent with previous findings that did not find an influence of task-irrelevant feature on visual processing over longer delays (e.g., Olivers et al., 2006; Teng & Kravitz, 2019), suggesting that merely perceiving a feature may not be sufficient for its sustained maintenance within WM, and its subsequent influence on perception.

Another open question relates to prior TMS (transcranial magnetic stimulation) studies in which stimulating sensory regions failed to induce a substantial effect on WM performance (Rademaker et al., 2017; van Lamsweerde & Johnson, 2017), apparently counter to the sensory recruitment framework. However, if we consider the effect of TMS as adding random noise to the local circuit, the center of the populational response of WM would likely be unchanged. During retrieval/readout, it is still possible to reproduce the memorized orientation based on the peak of the activation. Nevertheless, this does not preclude the usage of TMS to examine the neural mechanisms of WM with a more sensitive approach, for the controversy surrounding the role of sensory regions in memory processes is often suggested to arise from methodological inconsistencies between relevant studies (Phylactou et al., 2022). With the current paradigm, adding noise to the sensory cortex would broaden the tuning of the population response and attenuate the difference in thresholds between the conditions. TMS could also be used to examine the interaction between the fronto-parietal cortex and visual cortex. Applying stimulation at different time points would allow for causal studies of when top-down signals are needed to stimulate WM maintenance.

To conclude, our previous work demonstrated that WM and perception are necessarily and intimately related through their shared representations. Here, we showed that the strength of WM’s influence on perception gradually decreases over time while it is simultaneously increasingly biased by perception. Our results support a temporal decay + visual code account, and that although multiple codes of visual WM representations may coexist, behavioral readout predominantly utilizes the visual code, even if it results in an increase in interference. Further, these results have potential implications for future EEG (Electroencephalogram) and TMS studies that require precise timing manipulation. Targeting different temporal windows might allow us to further tease apart the relative contributions of visual, parietal, and frontal cortices and the interactions among these circuits. As the timing of WM tasks is usually quite different across fMRI, EEG, and behavioral studies, understanding the changes of the WM representation over time is crucial to understand its mechanisms.